Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-14779

Action only workflows do not filter out hadoop jars

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.0.0
    • Component/s: None
    • Labels:
      None
    • Release Notes:
      Fixed a bug that would cause pipelines to fail on remote clusters if the very first pipeline run was an action only pipeline.
    • Rank:
      1|i00kbz:

      Description

      The GCSToGCS integration test is failing.

      When I manually create a copy of the pipeline used in the test and run it on the cluster, it fails with:

      2019-01-18 19:23:02,737 - ERROR [MapReduceRunner-phase-1:c.c.c.i.a.r.ProgramControllerServiceAdapter@97] - MapReduce Program 'phase-1' failed.
      java.lang.Exception: java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.v2.util.MRApps.addLog4jSystemProperties(Lorg/apache/hadoop/mapred/Task;Ljava/util/List;Lorg/apache/hadoop/conf/Configuration;)V
      	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:369) ~[na:na]
      	at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) ~[com.google.guava.guava-13.0.1.jar:na]
      	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$2$1.run(MapReduceRuntimeService.java:450) [na:na]
      	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
      Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.v2.util.MRApps.addLog4jSystemProperties(Lorg/apache/hadoop/mapred/Task;Ljava/util/List;Lorg/apache/hadoop/conf/Configuration;)V
      	at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:409) ~[hadoop-mapreduce-client-jobclient-2.7.4.jar:na]
      	at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:295) ~[hadoop-mapreduce-client-jobclient-2.7.4.jar:na]
      	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:244) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.6.jar:na]
      	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.6.jar:na]
      	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.6.jar:na]
      	at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_181]
      	at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_181]
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1723) ~[hadoop-common-2.6.0-cdh5.5.6.jar:na]
      	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.6.jar:na]
      	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:355) ~[na:na]
      	... 3 common frames omitted
      2019-01-18 19:23:02,741 - ERROR [MapReduceRunner-phase-1:c.c.c.i.a.r.ProgramControllerServiceAdapter@98] - MapReduce program 'phase-1' failed with error: org.apache.hadoop.mapreduce.v2.util.MRApps.addLog4jSystemProperties(Lorg/apache/hadoop/mapred/Task;Ljava/util/List;Lorg/apache/hadoop/conf/Configuration;)V. Please check the system logs for more details.
      java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.v2.util.MRApps.addLog4jSystemProperties(Lorg/apache/hadoop/mapred/Task;Ljava/util/List;Lorg/apache/hadoop/conf/Configuration;)V
      	at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:409) ~[hadoop-mapreduce-client-jobclient-2.7.4.jar:na]
      	at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:295) ~[hadoop-mapreduce-client-jobclient-2.7.4.jar:na]
      	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:244) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.6.jar:na]
      	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.6.jar:na]
      	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.6.jar:na]
      	at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_181]
      	at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_181]
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1723) ~[hadoop-common-2.6.0-cdh5.5.6.jar:na]
      	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.6.jar:na]
      	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:355) ~[na:na]
      	at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) ~[com.google.guava.guava-13.0.1.jar:na]
      	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$2$1.run(MapReduceRuntimeService.java:450) [na:na]
      	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
      2019-01-18 19:23:02,759 - ERROR [WorkflowDriver:c.c.c.d.SmartWorkflow@544] - Pipeline 'test' failed.
      2019-01-18 19:23:02,799 - ERROR [WorkflowDriver:c.c.c.i.a.r.w.WorkflowProgramController@89] - Workflow service 'workflow.default.test.DataPipelineWorkflow.0b8231d3-1b56-11e9-8cd6-42010ad20023' failed.
      java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.v2.util.MRApps.addLog4jSystemProperties(Lorg/apache/hadoop/mapred/Task;Ljava/util/List;Lorg/apache/hadoop/conf/Configuration;)V
      	at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:409) ~[hadoop-mapreduce-client-jobclient-2.7.4.jar:na]
      	at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:295) ~[hadoop-mapreduce-client-jobclient-2.7.4.jar:na]
      	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:244) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.6.jar:na]
      	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.6.jar:na]
      	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.6.jar:na]
      	at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_181]
      	at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_181]
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1723) ~[hadoop-common-2.6.0-cdh5.5.6.jar:na]
      	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.6.jar:na]
      	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:355) ~[na:na]
      	at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) ~[com.google.guava.guava-13.0.1.jar:na]
      	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$2$1.run(MapReduceRuntimeService.java:450) ~[na:na]
      	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
      2019-01-18 19:23:02,805 - ERROR [WorkflowDriver:c.c.c.c.l.c.UncaughtExceptionHandler@34] - Uncaught error in thread Thread[WorkflowDriver,5,main], java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.v2.util.MRApps.addLog4jSystemProperties(Lorg/apache/hadoop/mapred/Task;Ljava/util/List;Lorg/apache/hadoop/conf/Configuration;)V
      2019-01-18 19:23:02,807 - ERROR [WorkflowDriver:c.c.c.c.l.c.UncaughtExceptionHandler@35] - Stacktrace for uncaught error in thread Thread[WorkflowDriver,5,main]
      java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.v2.util.MRApps.addLog4jSystemProperties(Lorg/apache/hadoop/mapred/Task;Ljava/util/List;Lorg/apache/hadoop/conf/Configuration;)V
      	at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:409) ~[hadoop-mapreduce-client-jobclient-2.7.4.jar:na]
      	at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:295) ~[hadoop-mapreduce-client-jobclient-2.7.4.jar:na]
      	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:244) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.6.jar:na]
      	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.6.jar:na]
      	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.6.jar:na]
      	at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_181]
      	at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_181]
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1723) ~[hadoop-common-2.6.0-cdh5.5.6.jar:na]
      	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) ~[hadoop-mapreduce-client-core-2.6.0-cdh5.5.6.jar:na]
      	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.startUp(MapReduceRuntimeService.java:355) ~[na:na]
      	at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:47) ~[com.google.guava.guava-13.0.1.jar:na]
      	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$2$1.run(MapReduceRuntimeService.java:450) ~[na:na]
      	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]
      

      This is because the twill.jar somehow contains the hadoop jars from the CDAP cluster, which is clashing with the hadoop jars from the remote cluster. For example, on the remote cluster I see the following in twill.jar/lib:

      hadoop-annotations-2.6.0-cdh5.5.6.jar
      hadoop-auth-2.6.0-cdh5.5.6.jar
      hadoop-common-2.6.0-cdh5.5.6.jar
      ...

      This happens every time the pipeline is run.

        Attachments

          Activity

            People

            • Assignee:
              ashau Albert Shau
              Reporter:
              ashau Albert Shau
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: