Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-14119

SparkWikipediaPipeline fails when building the jar from the examples directory of Sandbox distribution

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 5.1.0, 5.0.0
    • Fix Version/s: 5.1.0
    • Component/s: CDAP Examples
    • Labels:
      None
    • Rank:
      1|i00gkv:

      Description

      When downloading the Sandbox distribution and building the examples from the sources provided in the 'examples' directory in the distribution, the following jar is missing from the WikipediaPipeline's application's jar:

      lib/log4j-1.2.14.jar

      This results in the following failure when running the program in it:

      2018-08-14 15:06:32,393 - INFO  [Thread-246:c.c.c.i.a.r.b.MainOutputCommitter@94] - Setting up for MapReduce job: namespaceId=default, applicationId=WikipediaPipeline, program=WikiContentValidatorAndNormalizer, runid=4
      d32a0b7-a00e-11e8-bea3-acde48001122
      2018-08-14 15:06:32,781 - INFO  [Thread-246:c.c.c.i.a.r.b.MainOutputCommitter@181] - Invalidating transaction 1534284392394000000
      2018-08-14 15:06:32,791 - WARN  [Thread-246:o.a.h.m.LocalJobRunner$Job@587] - job_local823971620_0003
      java.lang.Exception: java.lang.NoClassDefFoundError: org/apache/log4j/Logger
              at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:489) ~[org.apache.hadoop.hadoop-mapreduce-client-common-2.8.0.jar:na]
              at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:549) ~[org.apache.hadoop.hadoop-mapreduce-client-common-2.8.0.jar:na]
      java.lang.NoClassDefFoundError: org/apache/log4j/Logger
              at org.sweble.wikitext.engine.WtEngineImpl.<clinit>(WtEngineImpl.java:62) ~[swc-engine-2.0.0.jar:2.0.0]
              at co.cask.cdap.examples.wikipedia.WikiContentValidatorAndNormalizer$FilterNormalizerMapper.toPlainText(WikiContentValidatorAndNormalizer.java:161) ~[unpacked/:na]
              at co.cask.cdap.examples.wikipedia.WikiContentValidatorAndNormalizer$FilterNormalizerMapper.map(WikiContentValidatorAndNormalizer.java:131) ~[unpacked/:na]
              at co.cask.cdap.examples.wikipedia.WikiContentValidatorAndNormalizer$FilterNormalizerMapper.map(WikiContentValidatorAndNormalizer.java:92) ~[unpacked/:na]
              at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) ~[org.apache.hadoop.hadoop-mapreduce-client-core-2.8.0.jar:na]
              at co.cask.cdap.internal.app.runtime.batch.MapperWrapper.run(MapperWrapper.java:135) ~[na:na]
              at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) ~[org.apache.hadoop.hadoop-mapreduce-client-core-2.8.0.jar:na]
              at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) ~[org.apache.hadoop.hadoop-mapreduce-client-core-2.8.0.jar:na]
              at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:270) ~[org.apache.hadoop.hadoop-mapreduce-client-common-2.8.0.jar:na]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_151]
              at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_151]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_151]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_151]
              at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_151]
      Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Logger
              at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_151]
              at co.cask.cdap.common.lang.InterceptableClassLoader.findClass(InterceptableClassLoader.java:46) ~[na:na]
              at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_151]
              at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_151]
              ... 14 common frames omitted 

       

      This might apply to other example applications also.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                terence Terence Yim
                Reporter:
                ali.anwar Ali Anwar
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: