Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-11402

Modify the Dataset Service default thread pool sizes for increased throughput

    XMLWordPrintableJSON

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.2.0
    • Component/s: Datasets, Master
    • Labels:
      None
    • Release Notes:
      Increased the default threadpool size for Dataset Service, for increased throughput.
    • Rank:
      1|i000w7:

      Description

      If there are lot of clients (~1k) trying to talk to dataset service, some of the clients will have errors like:

      Caused by: co.cask.cdap.api.dataset.DatasetManagementException: Error making request to dataset.service service at...
      ...
      Caused by: java.net.SocketTimeoutException: Read timed out
      

      This can be improved by setting the following configuration:

      <property>
        <name>dataset.service.exec.threads</name>
        <value>300</value>
      </property>
      <property>
        <name>dataset.service.worker.threads</name>
        <value>30</value>
      </property>
      

      or by updating the defaults in CDAP to such values.

        Attachments

          Activity

            People

            • Assignee:
              ali.anwar Ali Anwar
              Reporter:
              ali.anwar Ali Anwar
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: