Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-8701

Deadlock while stopping SDK due to log appender

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 4.2.0, 4.1.0
    • Fix Version/s: 4.3.0, 4.2.1, 4.1.0
    • Component/s: Log
    • Labels:
      None
    • Rank:
      1|hzzw5b:

      Description

      To reproduce the issue:
      1. Start SDK
      2. Run pipelines periodically for every 10 mins
      3. Shutdown the sdk after an hour.

      Based on the thread_dump, the Standalone main attempts to log a shutdown message and is blocked on that call and I don't see a thread that consumes the log which results in a deadlock state. Attached thread_dump of the process

      "Thread-3" prio=5 tid=0x00007fd6cf98e800 nid=0x267f3 waiting for monitor entry [0x00007000068cb000]
         java.lang.Thread.State: BLOCKED (on object monitor)
              at ch.qos.logback.core.AppenderBase.doAppend(AppenderBase.java:64)
              - waiting to lock <0x0000000799ecd6b0> (a co.cask.cdap.logging.framework.local.LocalLogAppender)
              at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:48)
              at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:280)
              at ch.qos.logback.classic.Logger.callAppenders(Logger.java:267)
              at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:449)
              at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:403)
              at ch.qos.logback.classic.Logger.info(Logger.java:607)
              at co.cask.cdap.StandaloneMain.shutDown(StandaloneMain.java:289)
              at co.cask.cdap.StandaloneMain$1.run(StandaloneMain.java:195)
      
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                terence Terence Yim
                Reporter:
                sree Sreevatsan Raman
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: