Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-7725

Deleting a namespace in which a Spark streaming/checkpointing program ran leaves behind files

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: 4.0.0
    • Fix Version/s: 4.0.0
    • Component/s: Namespaces, Spark
    • Labels:
      None
    • Rank:
      1|hzzpef:

      Description

      Steps to reproduce:
      0. Create a namespace (such as 'ali'), with no special configurations.
      1. Launch a spark streaming/checkpointing program (such as one that consumes from Kafka).
      2. Stop it, and then delete the namespace.
      3. Observe that ONLY the checkpointing files remain in the namespace directory. I'm not sure why these are not deleted by the namespace delete operation. All other files in the namespace directory were deleted.
      4. Trying to create this namespace again will result in a failure due to the directory for that namespace already existing.

      [ali@[hostname] spark]$ hdfs dfs -ls /cdap/namespaces/bar
      Found 1 items
      drwxr-xr-x   - yarn supergroup          0 2016-12-01 09:47 /cdap/namespaces/bar/data
      [ali@[hostname] spark]$ hdfs dfs -ls /cdap/namespaces/bar/data
      Found 1 items
      drwxr-xr-x   - yarn supergroup          0 2016-12-01 09:47 /cdap/namespaces/bar/data/dataStreamsCheckpoints
      [ali@[hostname] spark]$ hdfs dfs -ls /cdap/namespaces/bar/data/dataStreamsCheckpoints
      Found 1 items
      drwxr-xr-x   - yarn supergroup          0 2016-12-01 09:47 /cdap/namespaces/bar/data/dataStreamsCheckpoints/kafkaStreaming
      [ali@[hostname] spark]$ hdfs dfs -ls /cdap/namespaces/bar/data/dataStreamsCheckpoints/kafkaStreaming
      Found 1 items
      drwxr-xr-x   - yarn supergroup          0 2016-12-01 09:47 /cdap/namespaces/bar/data/dataStreamsCheckpoints/kafkaStreaming/e1869550-37b3-4cdb-a1d3-ee9761bda120
      

        Attachments

          Activity

            People

            • Assignee:
              ali.anwar Ali Anwar
              Reporter:
              ali.anwar Ali Anwar
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: