Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-2877

CDH 5.3 and 5.4 clusters not respecting memory allocation for containers

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 3.1.0
    • Fix Version/s: 3.1.0
    • Component/s: App Fabric
    • Labels:
      None
    • Rank:
      1|hzyvbb:

      Description

      On both CDH 5.3 and CDH 5.4 clusters, I'm seeing all containers being allocated with a minimum of 1024MB.

      /etc/hadoop/conf/yarn-site.xml contains the following:

        <property>
          <name>yarn.scheduler.minimum-allocation-mb</name>
          <value>256</value>
        </property>
      

      The AppMaster container log shows the request being made with a lower memory allocation:

      17:34:20.325 [ApplicationMasterService] INFO  o.a.t.i.a.ApplicationMasterService - Request 1 container with capability <memory:256, vCores:1> for runnable dataset.executor
      17:34:20.325 [ApplicationMasterService] INFO  o.a.t.i.a.ApplicationMasterService - Request 1 container with capability <memory:256, vCores:1> for runnable metrics.processor
      

      But the RM log shows container allocations like the following:

      2015-06-26 17:34:22,916 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1435005934607_0032_01_000007 Container Transitioned from NEW to ALLOCATED
      2015-06-26 17:34:22,916 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=cdap     OPERATION=AM Allocated Container        TARGET=SchedulerApp     RESULT=SUCCESS  APPID=application_1435005934607
      _0032    CONTAINERID=container_1435005934607_0032_01_000007
      2015-06-26 17:34:22,916 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Assigned container container_1435005934607_0032_01_000007 of capacity <memory:1024, vCores:1> on host cdh531803-100
      0.dev.continuuity.net:39222, which has 7 containers, <memory:7168, vCores:11> used and <memory:4307, vCores:21> available after allocation
      

      This could be due to a number of potential issues: cluster mis-configuration, YARN bug, or Twill bug, but I'm parking it here for further investigation.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                shankar Shankar Selvam
                Reporter:
                gary Gary Helmling
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: