Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-7820

Running hydrator pipeline spins up more containers than needed

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 4.0.0
    • Fix Version/s: None
    • Component/s: Pipelines
    • Labels:
    • Rank:
      1|hzzq6v:

      Description

      Running hydrator pipeline spins up more containers than needed. If we log hadoop logs at INFO level, we get following stacktrace, though it does not affect the functionality it would be good to figure out why does it happen:

      2016-12-13 22:07:46,299 - INFO  [RMCommunicator Allocator:o.a.h.m.v.a.r.RMContainerAllocator@1099] - Releasing unassigned container Container: [ContainerId: container_1481596700640_0026_01_000016, NodeId: host:port, NodeHttpAddress: host:port, Resource: <memory:1024, vCores:1>, Priority: 10, Token: Token { kind: ContainerToken, service: 10.128.0.34:44041 }, ]
      2016-12-13 22:07:46,300 - INFO  [RMCommunicator Allocator:o.a.h.m.v.a.r.RMContainerAllocator@1444] - After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:4 AssignedReds:3 CompletedMaps:8 CompletedReds:0 ContAlloc:15 ContRel:1 HostLocal:3 RackLocal:0
      2016-12-13 22:07:46,300 - INFO  [AsyncDispatcher event handler:o.a.h.m.v.a.j.i.TaskAttemptImpl@1095] - attempt_1481596700640_0026_r_000002_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
      2016-12-13 22:07:46,301 - INFO  [ContainerLauncher #1:o.a.h.m.v.a.l.ContainerLauncherImpl@363] - Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1481596700640_0026_01_000015 taskAttempt attempt_1481596700640_0026_r_000002_0
      2016-12-13 22:07:46,301 - INFO  [ContainerLauncher #1:o.a.h.m.v.a.l.ContainerLauncherImpl@127] - Launching attempt_1481596700640_0026_r_000002_0
      2016-12-13 22:07:46,302 - INFO  [ContainerLauncher #1:o.a.h.y.c.a.i.ContainerManagementProtocolProxy@260] - Opening proxy : host:port
      2016-12-13 22:07:46,345 - INFO  [ContainerLauncher #1:o.a.h.m.v.a.l.ContainerLauncherImpl@164] - Shuffle port returned by ContainerManager for attempt_1481596700640_0026_r_000002_0 : 13562
      2016-12-13 22:07:46,346 - INFO  [AsyncDispatcher event handler:o.a.h.m.v.a.j.i.TaskAttemptImpl@1404] - TaskAttempt: [attempt_1481596700640_0026_r_000002_0] using containerId: [container_1481596700640_0026_01_000015 on NM: [host:port]
      2016-12-13 22:07:46,347 - INFO  [AsyncDispatcher event handler:o.a.h.m.v.a.j.i.TaskAttemptImpl@1095] - attempt_1481596700640_0026_r_000002_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
      2016-12-13 22:07:46,348 - INFO  [AsyncDispatcher event handler:o.a.h.m.v.a.s.DefaultSpeculator@299] - ATTEMPT_START task_1481596700640_0026_r_000002
      2016-12-13 22:07:46,349 - INFO  [AsyncDispatcher event handler:o.a.h.m.v.a.j.i.TaskImpl@655] - task_1481596700640_0026_r_000002 Task Transitioned from SCHEDULED to RUNNING
      2016-12-13 22:07:46,412 - INFO  [IPC Server handler 0 on 43209:o.a.h.m.v.a.c.MRClientService@391] - Getting task report for MAP   job_1481596700640_0026. Report-size will be 11
      2016-12-13 22:07:46,479 - INFO  [IPC Server handler 0 on 43209:o.a.h.m.v.a.c.MRClientService@391] - Getting task report for REDUCE   job_1481596700640_0026. Report-size will be 3
      2016-12-13 22:07:47,305 - INFO  [RMCommunicator Allocator:o.a.h.m.v.a.r.RMContainerRequestor@209] - getResources() for application_1481596700640_0026: ask=1 release= 1 newContainers=1 finishedContainers=2 resourcelimit=<memory:1792, vCores:1> knownNMs=1
      2016-12-13 22:07:47,306 - INFO  [RMCommunicator Allocator:o.a.h.m.v.a.r.RMContainerAllocator@776] - Received completed container container_1481596700640_0026_01_000008
      2016-12-13 22:07:47,307 - INFO  [RMCommunicator Allocator:o.a.h.m.v.a.r.RMContainerAllocator@776] - Received completed container container_1481596700640_0026_01_000016
      2016-12-13 22:07:47,307 - ERROR [RMCommunicator Allocator:o.a.h.m.v.a.r.RMContainerAllocator@779] - Container complete event for unknown container id container_1481596700640_0026_01_000016

        Attachments

          Activity

            People

            • Assignee:
              nitin Nitin Motgi
              Reporter:
              vinisha Vinisha Shah
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: