Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-1873

Lots of warnings and errors when master starts up

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.1.0
    • Component/s: App Fabric
    • Labels:
      None
    • Rank:
      1|hzypgv:

      Description

      When you start up CDAP master, there are a ton of warnings and errors in the logs before the services come up.

      For example:

      15/03/20 00:46:21 ERROR async.DefaultEventHandler: Failed to send requests for topics logs.user-v2 with correlation ids in [481,488]
      15/03/20 00:46:21 ERROR async.ProducerSendThread: Error in handling batch of 2 events
      kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.
              at kafka.producer.async.DefaultEventHandler.handle(Unknown Source)
              at kafka.producer.async.ProducerSendThread.tryToHandle(Unknown Source)
              at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(Unknown Source)
              at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(Unknown Source)
              at scala.collection.immutable.Stream.foreach(Stream.scala:547)
              at kafka.producer.async.ProducerSendThread.processEvents(Unknown Source)
              at kafka.producer.async.ProducerSendThread.run(Unknown Source)
      2015-03-20 00:46:24,126 - ERROR [default-namespace-ensurer:c.c.c.i.a.s.DefaultStore$1@118] - Failed to access app.meta table
      co.cask.cdap.data2.dataset2.DatasetManagementException: Cannot discover dataset service
              at co.cask.cdap.data2.datafabric.dataset.DatasetServiceClient.resolve(DatasetServiceClient.java:308) ~[co.cask.cdap.cdap-data-fabric-2.8.0-SNAPSHOT.jar:na]
              at co.cask.cdap.data2.datafabric.dataset.DatasetServiceClient.doRequest(DatasetServiceClient.java:271) ~[co.cask.cdap.cdap-data-fabric-2.8.0-SNAPSHOT.jar:na]
              at co.cask.cdap.data2.datafabric.dataset.DatasetServiceClient.doRequest(DatasetServiceClient.java:294) ~[co.cask.cdap.cdap-data-fabric-2.8.0-SNAPSHOT.jar:na]
              at co.cask.cdap.data2.datafabric.dataset.DatasetServiceClient.doGet(DatasetServiceClient.java:234) ~[co.cask.cdap.cdap-data-fabric-2.8.0-SNAPSHOT.jar:na]
              at co.cask.cdap.data2.datafabric.dataset.DatasetServiceClient.getInstance(DatasetServiceClient.java:86) ~[co.cask.cdap.cdap-data-fabric-2.8.0-SNAPSHOT.jar:na]
              at co.cask.cdap.data2.datafabric.dataset.RemoteDatasetFramework.hasInstance(RemoteDatasetFramework.java:153) ~[co.cask.cdap.cdap-data-fabric-2.8.0-SNAPSHOT.jar:na]
              at co.cask.cdap.data2.datafabric.dataset.DatasetsUtil.createIfNotExists(DatasetsUtil.java:65) ~[co.cask.cdap.cdap-data-fabric-2.8.0-SNAPSHOT.jar:na]
              at co.cask.cdap.data2.datafabric.dataset.DatasetsUtil.getOrCreateDataset(DatasetsUtil.java:54) ~[co.cask.cdap.cdap-data-fabric-2.8.0-SNAPSHOT.jar:na]
              at co.cask.cdap.internal.app.store.DefaultStore$1.get(DefaultStore.java:113) [co.cask.cdap.cdap-app-fabric-2.8.0-SNAPSHOT.jar:na]
              at co.cask.cdap.internal.app.store.DefaultStore$1.get(DefaultStore.java:109) [co.cask.cdap.cdap-app-fabric-2.8.0-SNAPSHOT.jar:na]
              at co.cask.cdap.data2.dataset2.tx.Transactional.execute(Transactional.java:103) ~[co.cask.cdap.cdap-data-fabric-2.8.0-SNAPSHOT.jar:na]
              at co.cask.cdap.data2.dataset2.tx.Transactional.executeUnchecked(Transactional.java:77) ~[co.cask.cdap.cdap-data-fabric-2.8.0-SNAPSHOT.jar:na]
              at co.cask.cdap.internal.app.store.DefaultStore.getNamespace(DefaultStore.java:807) ~[co.cask.cdap.cdap-app-fabric-2.8.0-SNAPSHOT.jar:na]
              at co.cask.cdap.internal.app.namespace.DefaultNamespaceAdmin.getNamespace(DefaultNamespaceAdmin.java:99) ~[co.cask.cdap.cdap-app-fabric-2.8.0-SNAPSHOT.jar:na]
              at co.cask.cdap.internal.app.namespace.DefaultNamespaceAdmin.hasNamespace(DefaultNamespaceAdmin.java:114) ~[co.cask.cdap.cdap-app-fabric-2.8.0-SNAPSHOT.jar:na]
              at co.cask.cdap.internal.app.namespace.DefaultNamespaceAdmin.createNamespace(DefaultNamespaceAdmin.java:130) ~[co.cask.cdap.cdap-app-fabric-2.8.0-SNAPSHOT.jar:na]
              at co.cask.cdap.internal.app.namespace.DefaultNamespaceEnsurer.run(DefaultNamespaceEnsurer.java:49) ~[co.cask.cdap.cdap-app-fabric-2.8.0-SNAPSHOT.jar:na]
              at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_34]
      

      and

      15/03/20 00:46:21 WARN async.DefaultEventHandler: Failed to collate messages by topic,partition due to: Invalid partition id: 25 for topic logs.user-v2; Valid values are in the inclusive range of [0, 9]
      15/03/20 00:46:21 INFO async.DefaultEventHandler: Back off for 100 ms before retrying send. Remaining retries = 3
      15/03/20 00:46:21 INFO client.ClientUtils$: Fetching metadata from broker id:0,host:explore-secure-old-template1272-1000.dev.continuuity.net,port:9092 with correlation id 482 for 1 topic(s) Set(logs.user-v2)
      15/03/20 00:46:21 INFO producer.SyncProducer: Connected to explore-secure-old-template1272-1000.dev.continuuity.net:9092 for producing
      15/03/20 00:46:21 INFO producer.SyncProducer: Disconnecting from explore-secure-old-template1272-1000.dev.continuuity.net:9092
      15/03/20 00:46:21 WARN async.DefaultEventHandler: Failed to collate messages by topic,partition due to: Invalid partition id: 25 for topic logs.user-v2; Valid values are in the inclusive range of [0, 9]
      15/03/20 00:46:21 INFO async.DefaultEventHandler: Back off for 100 ms before retrying send. Remaining retries = 2
      15/03/20 00:46:21 INFO client.ClientUtils$: Fetching metadata from broker id:0,host:explore-secure-old-template1272-1000.dev.continuuity.net,port:9092 with correlation id 484 for 1 topic(s) Set(logs.user-v2)
      15/03/20 00:46:21 INFO producer.SyncProducer: Connected to explore-secure-old-template1272-1000.dev.continuuity.net:9092 for producing
      15/03/20 00:46:21 INFO producer.SyncProducer: Disconnecting from explore-secure-old-template1272-1000.dev.continuuity.net:9092
      15/03/20 00:46:21 WARN async.DefaultEventHandler: Failed to collate messages by topic,partition due to: Invalid partition id: 25 for topic logs.user-v2; Valid values are in the inclusive range of [0, 9]
      15/03/20 00:46:21 INFO async.DefaultEventHandler: Back off for 100 ms before retrying send. Remaining retries = 1
      15/03/20 00:46:21 INFO client.ClientUtils$: Fetching metadata from broker id:0,host:explore-secure-old-template1272-1000.dev.continuuity.net,port:9092 with correlation id 486 for 1 topic(s) Set(logs.user-v2)
      15/03/20 00:46:21 INFO producer.SyncProducer: Connected to explore-secure-old-template1272-1000.dev.continuuity.net:9092 for producing
      15/03/20 00:46:21 INFO producer.SyncProducer: Disconnecting from explore-secure-old-template1272-1000.dev.continuuity.net:9092
      15/03/20 00:46:21 WARN async.DefaultEventHandler: Failed to collate messages by topic,partition due to: Invalid partition id: 25 for topic logs.user-v2; Valid values are in the inclusive range of [0, 9]
      

        Attachments

          Activity

            People

            • Assignee:
              alexb Alex Baranau
              Reporter:
              ashau Albert Shau
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: