Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-9142

Kafka source cannot use 'binary' format

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.1.1
    • Component/s: None
    • Labels:
      None
    • Release Notes:
      Fixed an issue with the binary format in the Kafka streaming source that prevented pipeline deployment.
    • Rank:
      1|hzzyhz:

      Description

      If you try to use the 'binary' format with the Kafka source, you get a really bizarre error (see attachment). The exception is:

      2017-03-23 17:55:41,531 - ERROR [appfabric-executor-19:c.c.c.g.h.AppLifecycleHttpHandler$1@404] - Deploy failure
      java.util.concurrent.ExecutionException: co.cask.cdap.common.InvalidArtifactException: Missing class binary. It may be caused by missing dependency jar(s) in the artifact jar.
      	at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:294) ~[com.google.guava.guava-13.0.1.jar:na]
      	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:267) ~[com.google.guava.guava-13.0.1.jar:na]
      	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96) ~[com.google.guava.guava-13.0.1.jar:na]
      	at co.cask.cdap.internal.app.deploy.pipeline.LocalArtifactLoaderStage.process(LocalArtifactLoaderStage.java:105) ~[na:na]
      	at co.cask.cdap.internal.app.deploy.pipeline.LocalArtifactLoaderStage.process(LocalArtifactLoaderStage.java:55) ~[na:na]
      	at co.cask.cdap.pipeline.AbstractStage.process(AbstractStage.java:52) ~[na:na]
      	at co.cask.cdap.internal.pipeline.SynchronousPipeline.execute(SynchronousPipeline.java:55) ~[na:na]
      	at co.cask.cdap.internal.app.deploy.LocalApplicationManager.deploy(LocalApplicationManager.java:133) ~[na:na]
      	at co.cask.cdap.internal.app.services.ApplicationLifecycleService.deployApp(ApplicationLifecycleService.java:608) ~[na:na]
      	at co.cask.cdap.internal.app.services.ApplicationLifecycleService.deployApp(ApplicationLifecycleService.java:425) ~[na:na]
      	at co.cask.cdap.gateway.handlers.AppLifecycleHttpHandler$1.onFinish(AppLifecycleHttpHandler.java:388) ~[na:na]
      	at co.cask.cdap.common.http.AbstractBodyConsumer.finished(AbstractBodyConsumer.java:65) [na:na]
      	at co.cask.http.HttpMethodInfo.bodyConsumerFinish(HttpMethodInfo.java:149) [co.cask.http.netty-http-0.16.0.jar:na]
      	at co.cask.http.HttpMethodInfo.invoke(HttpMethodInfo.java:93) [co.cask.http.netty-http-0.16.0.jar:na]
      	at co.cask.http.HttpDispatcher.messageReceived(HttpDispatcher.java:38) [co.cask.http.netty-http-0.16.0.jar:na]
      	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.6.6.Final.jar:na]
      	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.6.6.Final.jar:na]
      	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.6.6.Final.jar:na]
      	at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) [io.netty.netty-3.6.6.Final.jar:na]
      	at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) [io.netty.netty-3.6.6.Final.jar:na]
      	at org.jboss.netty.handler.execution.OrderedMemoryAwareThreadPoolExecutor$ChildExecutor.run(OrderedMemoryAwareThreadPoolExecutor.java:314) [io.netty.netty-3.6.6.Final.jar:na]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77]
      	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
      Caused by: co.cask.cdap.common.InvalidArtifactException: Missing class binary. It may be caused by missing dependency jar(s) in the artifact jar.
      	at co.cask.cdap.internal.app.deploy.InMemoryConfigurator.getSpecJson(InMemoryConfigurator.java:194) ~[na:na]
      	at co.cask.cdap.internal.app.deploy.InMemoryConfigurator.createResponse(InMemoryConfigurator.java:134) ~[na:na]
      	at co.cask.cdap.internal.app.deploy.InMemoryConfigurator.config(InMemoryConfigurator.java:124) ~[na:na]
      	at co.cask.cdap.internal.app.deploy.pipeline.LocalArtifactLoaderStage.process(LocalArtifactLoaderStage.java:104) ~[na:na]
      	... 20 common frames omitted
      Caused by: java.lang.RuntimeException: Exception while configuring plugin of type streamingsource and name Kafka for stage Kafka: Unable to instantiate a message parser from format 'binary' and message schema '{"type":"record","name":"kafka.message","fields":[{"name":"body","type":"bytes"}]}': binary
      	at co.cask.cdap.etl.spec.PipelineSpecGenerator.configurePlugin(PipelineSpecGenerator.java:240) ~[na:na]
      	at co.cask.cdap.etl.spec.PipelineSpecGenerator.configureStage(PipelineSpecGenerator.java:193) ~[na:na]
      	at co.cask.cdap.etl.spec.PipelineSpecGenerator.configureStages(PipelineSpecGenerator.java:123) ~[na:na]
      	at co.cask.cdap.datastreams.DataStreamsPipelineSpecGenerator.generateSpec(DataStreamsPipelineSpecGenerator.java:49) ~[na:na]
      	at co.cask.cdap.datastreams.DataStreamsPipelineSpecGenerator.generateSpec(DataStreamsPipelineSpecGenerator.java:29) ~[na:na]
      	at co.cask.cdap.datastreams.DataStreamsApp.configure(DataStreamsApp.java:44) ~[na:na]
      	at co.cask.cdap.api.app.AbstractApplication.configure(AbstractApplication.java:61) ~[na:na]
      	at co.cask.cdap.internal.app.deploy.InMemoryConfigurator.getSpecJson(InMemoryConfigurator.java:165) ~[na:na]
      	... 23 common frames omitted
      Caused by: java.lang.IllegalArgumentException: Unable to instantiate a message parser from format 'binary' and message schema '{"type":"record","name":"kafka.message","fields":[{"name":"body","type":"bytes"}]}': binary
      	at co.cask.hydrator.plugin.spark.KafkaConfig.validate(KafkaConfig.java:370) ~[na:na]
      	at co.cask.hydrator.plugin.spark.KafkaStreamingSource.configurePipeline(KafkaStreamingSource.java:78) ~[na:na]
      	at co.cask.cdap.etl.spec.PipelineSpecGenerator.configurePlugin(PipelineSpecGenerator.java:236) ~[na:na]
      	... 30 common frames omitted
      Caused by: java.lang.ClassNotFoundException: binary
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_77]
      	at co.cask.cdap.common.lang.InterceptableClassLoader.findClass(InterceptableClassLoader.java:46) ~[na:na]
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_77]
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_77]
      	at java.lang.Class.forName0(Native Method) ~[na:1.8.0_77]
      	at java.lang.Class.forName(Class.java:264) ~[na:1.8.0_77]
      	at co.cask.cdap.format.RecordFormats.createInitializedFormat(RecordFormats.java:87) ~[na:na]
      	at co.cask.hydrator.plugin.spark.KafkaConfig.validate(KafkaConfig.java:368) ~[na:na]
      	... 32 common frames omitted
      

      So this is really a combination of 2 errors. The first is that there is no 'binary' format class, so the plugin should look for 'binary' and interpret it appropriately. The second is that if the format class does not exist, the plugin should give a nicer error message instead of throwing an exception with ClassNotFoundException, which CDAP interprets as something wrong with the artifact.

        Attachments

          Activity

            People

            • Assignee:
              ashau Albert Shau
              Reporter:
              ashau Albert Shau
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: