Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-3027

The DFSStreamHeartbeatsTest easily fail on cluster

    XMLWordPrintableJSON

    Details

    Error rendering 'com.atlassian.jira.jira-view-issue-plugin:details-module'. Please contact your Jira administrators.

      Description

      It seems the failure will hang the run. This is the log:

      015-07-15 22:55:31,023 - INFO  [ProcessThread(sid:0 cport:-1)::o.a.z.s.PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14e93ee0ebc0000 type:create cxid:0x3 zxid:0x2 txntype:-1 reqpath:n/a Error Path:/election/streams.coordinator Error:KeeperErrorCode = NoNode for /election/streams.coordinator
      2015-07-15 22:55:31,028 - INFO  [ProcessThread(sid:0 cport:-1)::o.a.z.s.PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14e93ee0ebc0000 type:create cxid:0x4 zxid:0x3 txntype:-1 reqpath:n/a Error Path:/election Error:KeeperErrorCode = NoNode for /election
      2015-07-15 22:55:31,044 - INFO  [leader-election-election-streams.coordinator:c.c.c.d.s.s.DistributedStreamService$5@406] - Became Stream handler leader. Starting resource coordinator.
      2015-07-15 22:55:31,066 - DEBUG [zk-client-EventThread:c.c.c.d.s.s.DistributedStreamService$7@477] - Adding namespace:default/stream:test_stream stream as a resource to the coordinator to manager streams leaders.
      2015-07-15 22:55:31,084 - INFO  [ProcessThread(sid:0 cport:-1)::o.a.z.s.PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14e93ee0ebc0000 type:create cxid:0xb zxid:0x7 txntype:-1 reqpath:n/a Error Path:/streams/coordination/requirements Error:KeeperErrorCode = NoNode for /streams/coordination/requirements
      2015-07-15 22:55:31,089 - INFO  [ProcessThread(sid:0 cport:-1)::o.a.z.s.PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14e93ee0ebc0000 type:create cxid:0xc zxid:0x8 txntype:-1 reqpath:n/a Error Path:/streams/coordination Error:KeeperErrorCode = NoNode for /streams/coordination
      2015-07-15 22:55:31,092 - INFO  [ProcessThread(sid:0 cport:-1)::o.a.z.s.PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14e93ee0ebc0000 type:create cxid:0xd zxid:0x9 txntype:-1 reqpath:n/a Error Path:/streams Error:KeeperErrorCode = NoNode for /streams
      2015-07-15 22:55:31,116 - INFO  [zk-client-EventThread:c.c.c.d.s.s.DistributedStreamService$6@434] - Stream resource requirement updated to ResourceRequirement{name=streams, partitions=[Partition{name=default.test_stream, replicas=1}]}
      2015-07-15 22:55:31,123 - INFO  [resource-coordinator:c.c.c.c.z.c.ResourceCoordinator$4@227] - Get requirement ResourceRequirement{name=streams, partitions=[Partition{name=default.test_stream, replicas=1}]}
      2015-07-15 22:55:31,167 - INFO  [ProcessThread(sid:0 cport:-1)::o.a.z.s.PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14e93ee0ebc0000 type:setData cxid:0x16 zxid:0xe txntype:-1 reqpath:n/a Error Path:/streams/coordination/assignments/streams Error:KeeperErrorCode = NoNode for /streams/coordination/assignments/streams
      2015-07-15 22:55:31,170 - INFO  [ProcessThread(sid:0 cport:-1)::o.a.z.s.PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14e93ee0ebc0000 type:create cxid:0x17 zxid:0xf txntype:-1 reqpath:n/a Error Path:/streams/coordination/assignments Error:KeeperErrorCode = NoNode for /streams/coordination/assignments
      2015-07-15 22:55:31,177 - DEBUG [resource-coordinator:c.c.c.c.z.c.ResourceCoordinator$6@377] - Resource assignment updated for streams. {"name":"streams","assignments":[[{"service":"streams","hostname":"bamboo-agent16.prod.continuuity.net","port":36153},{"name":"default.test_stream","replicaId":0}]]}
      2015-07-15 22:55:31,180 - DEBUG [zk-client-EventThread:c.c.c.c.z.c.ResourceCoordinatorClient$2@236] - Received resource assignment for streams. {co.cask.cdap.common.zookeeper.coordination.DiscoverableCodec$1@2b91887e=[PartitionReplica{partition=default.test_stream, replica=0}]}
      2015-07-15 22:55:31,181 - INFO  [resource-coordinator-client:c.c.c.d.s.s.DistributedStreamService$StreamsLeaderHandler@516] - Stream leader requirement has changed to [PartitionReplica{partition=default.test_stream, replica=0}]
      2015-07-15 22:55:31,183 - DEBUG [resource-coordinator-client:c.c.c.d.s.s.DistributedStreamService@495] - Stream writer is the leader of streams: [namespace:default/stream:test_stream]
      2015-07-15 22:55:31,240 - INFO  [New I/O worker #1:c.c.c.d.s.s.ConcurrentStreamWriter$StreamFileFactory@314] - Create stream writer for namespace:default/stream:test_stream with generation 0
      2015-07-15 22:55:31,244 - DEBUG [New I/O worker #1:c.c.c.d.s.TimePartitionedStreamFileWriter$StreamWriterFactory@160] - New stream file created at file:/tmp/CDAP-DUT-JOB1/junit6673014052725130276/junit4309628604121098557/namespaces/default/streams/test_stream/1436997600.03600/file.0.000000.dat
      2015-07-15 22:55:32,223 - WARN  [resource-coordinator-client:c.c.c.m.s.DefaultMetricDatasetFactory@127] - Cannot access or create table metrics.v2.table.ts.1, will retry in 1 sec.
      2015-07-15 22:55:33,002 - INFO  [heartbeats-scheduler:c.c.c.d.s.s.DFSStreamHeartbeatsTest$MockHeartbeatPublisher@228] - Received heartbeat StreamWriterHeartbeat{timestamp=1437000933002, instanceId=0, streamsSizes={namespace:default/stream:test_stream=20}} for Stream {}
      2015-07-15 22:55:34,224 - WARN  [resource-coordinator-client:c.c.c.m.s.DefaultMetricDatasetFactory@127] - Cannot access or create table metrics.v2.table.ts.1, will retry in 1 sec.
      2015-07-15 22:55:34,318 - INFO  [MockHeartbeatPublisher STOPPING:c.c.c.d.s.s.DFSStreamHeartbeatsTest$MockHeartbeatPublisher@223] - Stopping Publisher.
      2015-07-15 22:55:34,321 - INFO  [leader-election-election-streams.coordinator:c.c.c.d.s.s.DistributedStreamService$5@415] - Became Stream handler follower.
      2015-07-15 22:55:36,225 - WARN  [resource-coordinator-client:c.c.c.m.s.DefaultMetricDatasetFactory@127] - Cannot access or create table metrics.v2.table.ts.1, will retry in 1 sec.
      2015-07-15 22:55:38,226 - WARN  [resource-coordinator-client:c.c.c.m.s.DefaultMetricDatasetFactory@127] - Cannot access or create table metrics.v2.table.ts.1, will retry in 1 sec.
      

        Attachments

          Activity

            People

            • Assignee:
              terence Terence Yim
              Reporter:
              terence Terence Yim
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: