Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-11431

Spark streaming pipeline emits lots of logs after restarting.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 4.1.1
    • Fix Version/s: None
    • Component/s: Log, Spark
    • Labels:
      None
    • Rank:
      1|i0011b:

      Description

      If you stop spark streaming pipeline and start it again, lots of logs are emitted by the pipeline run. During one instance it was observed that, first run emits around 422 log lines. However, after stopping and starting the pipeline again, count of logs emitted was quickly increased to 38233.

      Following logs were observed frequently during the second run of the pipeline

      2017-05-08 08:41:54,129 - INFO  [block-manager-slave-async-thread-pool-2:o.a.s.s.BlockManager@58] - Removing RDD 50
      2017-05-08 08:41:54,130 - INFO  [block-manager-slave-async-thread-pool-6:o.a.s.e.CoarseGrainedExecutorBackend@93] - 08:41:54.130 [block-manager-slave-async-thread-pool-6] DEBUG o.a.s.s.BlockManagerSlaveEndpoint - Done removing RDD 50, response is 0
      2017-05-08 08:41:54,130 - DEBUG [block-manager-slave-async-thread-pool-6:o.a.s.s.BlockManagerSlaveEndpoint@62] - Done removing RDD 50, response is 0
      2017-05-08 08:41:54,130 - INFO  [block-manager-slave-async-thread-pool-6:o.a.s.e.CoarseGrainedExecutorBackend@93] - 08:41:54.130 [block-manager-slave-async-thread-pool-6] DEBUG o.a.s.s.BlockManagerSlaveEndpoint - Sent response: 0 to 10.250.0.29:40113
      2017-05-08 08:41:54,130 - DEBUG [block-manager-slave-async-thread-pool-6:o.a.s.s.BlockManagerSlaveEndpoint@62] - Sent response: 0 to 10.250.0.29:40113
      2017-05-08 08:41:54,130 - DEBUG [block-manager-slave-async-thread-pool-4:o.a.s.s.BlockManagerSlaveEndpoint@62] - Done removing RDD 51, response is 0
      2017-05-08 08:41:54,130 - INFO  [block-manager-slave-async-thread-pool-3:o.a.s.e.CoarseGrainedExecutorBackend@93] - 08:41:54.130 [block-manager-slave-async-thread-pool-3] DEBUG o.a.s.s.BlockManagerSlaveEndpoint - Done removing RDD 50, response is 0
      2017-05-08 08:41:54,130 - DEBUG [block-manager-slave-async-thread-pool-3:o.a.s.s.BlockManagerSlaveEndpoint@62] - Done removing RDD 50, response is 0
      2017-05-08 08:41:54,130 - INFO  [block-manager-slave-async-thread-pool-3:o.a.s.e.CoarseGrainedExecutorBackend@93] - 08:41:54.130 [block-manager-slave-async-thread-pool-3] DEBUG o.a.s.s.BlockManagerSlaveEndpoint - Sent response: 0 to 10.250.0.29:40113
      

        Attachments

          Activity

            People

            • Assignee:
              bhooshan Bhooshan Mogal
              Reporter:
              sagar Sagar Kapare
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: