Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-12546

Running PySpark in SDK have threads and classloaders leakage

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.3.1
    • Component/s: None
    • Labels:
      None
    • Release Notes:
      Fixed a classloader leakage issue when PySpark is used in local sandbox
    • Rank:
      1|i007kf:

      Description

      Spark creates a thread that runs in infinite loop and never terminates.

        private class MonitorThread extends Thread(s"Idle Worker Monitor for $pythonExec") {
      
          setDaemon(true)
      
          override def run() {
            while (true) {
              synchronized {
                if (lastActivity + IDLE_WORKER_TIMEOUT_MS < System.currentTimeMillis()) {
                  cleanupIdleWorkers()
                  lastActivity = System.currentTimeMillis()
                }
              }
              Thread.sleep(10000)
            }
          }
        }
      

        Attachments

          Activity

            People

            • Assignee:
              terence Terence Yim
              Reporter:
              terence Terence Yim
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: