Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-11779

Cleanup directory created by usage of SparkInterpreter on closing the execution context

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.2.0
    • Component/s: None
    • Labels:
      None
    • Rank:
      1|i0031r:

      Description

      Ideally it's the user responsibility to call the close method to free up resources. However, due to limitation in Spark Streaming + Checkpointing, it's very difficult for user code to call the close method if the execution is restored from checkpoint.

      The solution is to maintain weakreference to SparkInterpreter created and strong references to the directory underneath and perform cleanup on closing the context.

        Attachments

          Activity

            People

            • Assignee:
              terence Terence Yim
              Reporter:
              terence Terence Yim
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: