Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-13120

PartitionedFileSet does not delete its directory when deleted

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.3.4
    • Component/s: None
    • Labels:
      None
    • Release Notes:
      Fixed a bug that caused directories to be left around if a workflow used a partitioned fileset as a local dataset
    • Rank:
      1|i00awn:

      Description

      When you create a local fileset in a workflow, the workflow driver deletes the files within the fileset directory, but leaves the directory around. This can be reproduced by running any mapreduce pipeline that uses a local dataset. For example, by having two aggregations in the pipeline.

      After a while, you end up with a bunch of empty directories of the form:

      [hdfs@perftests31151-1001 ~]$ hadoop fs -ls -R /cdap/namespaces/default/data
      drwxr-xr-x   - cdap cdap          0 2018-02-19 22:02 /cdap/namespaces/default/data/conn-0
      drwxr-xr-x   - cdap cdap          0 2018-02-19 18:28 /cdap/namespaces/default/data/conn-0/0147fb61-15a2-11e8-821e-42010afa000b
      drwxr-xr-x   - cdap cdap          0 2018-02-19 21:47 /cdap/namespaces/default/data/conn-0/5449eaa2-15be-11e8-82c4-42010afa000c
      drwxr-xr-x   - cdap cdap          0 2018-02-19 21:47 /cdap/namespaces/default/data/conn-0/5738a623-15be-11e8-a12d-42010afa000c
      drwxr-xr-x   - cdap cdap          0 2018-02-19 21:54 /cdap/namespaces/default/data/conn-0/68d8fa64-15be-11e8-9e4f-42010afa000c
      drwxr-xr-x   - cdap cdap          0 2018-02-19 21:15 /cdap/namespaces/default/data/conn-0/780840e2-15b9-11e8-a02d-42010afa000b
      drwxr-xr-x   - cdap cdap          0 2018-02-19 22:24 /cdap/namespaces/default/data/conn-0/7b0ffe75-15c0-11e8-98ba-42010afa000c
      drwxr-xr-x   - cdap cdap          0 2018-02-19 21:46 /cdap/namespaces/default/data/conn-0/876ac831-15bb-11e8-ad43-42010afa000c
      drwxr-xr-x   - cdap cdap          0 2018-02-19 22:02 /cdap/namespaces/default/data/conn-1
      drwxr-xr-x   - cdap cdap          0 2018-02-19 18:29 /cdap/namespaces/default/data/conn-1/0147fb61-15a2-11e8-821e-42010afa000b
      drwxr-xr-x   - cdap cdap          0 2018-02-19 21:47 /cdap/namespaces/default/data/conn-1/5449eaa2-15be-11e8-82c4-42010afa000c
      drwxr-xr-x   - cdap cdap          0 2018-02-19 21:47 /cdap/namespaces/default/data/conn-1/5738a623-15be-11e8-a12d-42010afa000c
      drwxr-xr-x   - cdap cdap          0 2018-02-19 21:54 /cdap/namespaces/default/data/conn-1/68d8fa64-15be-11e8-9e4f-42010afa000c
      drwxr-xr-x   - cdap cdap          0 2018-02-19 21:15 /cdap/namespaces/default/data/conn-1/780840e2-15b9-11e8-a02d-42010afa000b
      drwxr-xr-x   - cdap cdap          0 2018-02-19 22:25 /cdap/namespaces/default/data/conn-1/7b0ffe75-15c0-11e8-98ba-42010afa000c
      drwxr-xr-x   - cdap cdap          0 2018-02-19 21:47 /cdap/namespaces/default/data/conn-1/876ac831-15bb-11e8-ad43-42010afa000c
      drwxr-xr-x   - cdap cdap          0 2018-02-19 22:02 /cdap/namespaces/default/data/conn-2
      drwxr-xr-x   - cdap cdap          0 2018-02-19 18:29 /cdap/namespaces/default/data/conn-2/0147fb61-15a2-11e8-821e-42010afa000b
      drwxr-xr-x   - cdap cdap          0 2018-02-19 22:25 /cdap/namespaces/default/data/conn-2/7b0ffe75-15c0-11e8-98ba-42010afa000c
      drwxr-xr-x   - cdap cdap          0 2018-02-19 21:47 /cdap/namespaces/default/data/conn-2/876ac831-15bb-11e8-ad43-42010afa000c
      drwxr-xr-x   - cdap cdap          0 2018-02-19 22:02 /cdap/namespaces/default/data/conn-3
      drwxr-xr-x   - cdap cdap          0 2018-02-19 18:28 /cdap/namespaces/default/data/conn-3/0147fb61-15a2-11e8-821e-42010afa000b
      drwxr-xr-x   - cdap cdap          0 2018-02-19 22:25 /cdap/namespaces/default/data/conn-3/7b0ffe75-15c0-11e8-98ba-42010afa000c
      drwxr-xr-x   - cdap cdap          0 2018-02-19 21:47 /cdap/namespaces/default/data/conn-3/876ac831-15bb-11e8-ad43-42010afa000c
      drwxr-xr-x   - cdap cdap          0 2018-02-19 22:02 /cdap/namespaces/default/data/conn-4
      drwxr-xr-x   - cdap cdap          0 2018-02-19 18:28 /cdap/namespaces/default/data/conn-4/0147fb61-15a2-11e8-821e-42010afa000b
      drwxr-xr-x   - cdap cdap          0 2018-02-19 22:25 /cdap/namespaces/default/data/conn-4/7b0ffe75-15c0-11e8-98ba-42010afa000c
      drwxr-xr-x   - cdap cdap          0 2018-02-19 21:47 /cdap/namespaces/default/data/conn-4/876ac831-15bb-11e8-ad43-42010afa000c
      drwxr-xr-x   - cdap cdap          0 2018-02-19 22:02 /cdap/namespaces/default/data/conn-5
      drwxr-xr-x   - cdap cdap          0 2018-02-19 22:25 /cdap/namespaces/default/data/conn-5/7b0ffe75-15c0-11e8-98ba-42010afa000c
      drwxr-xr-x   - cdap cdap          0 2018-02-19 21:47 /cdap/namespaces/default/data/conn-5/876ac831-15bb-11e8-ad43-42010afa000c
      drwxr-xr-x   - cdap cdap          0 2018-02-19 22:02 /cdap/namespaces/default/data/conn-6
      drwxr-xr-x   - cdap cdap          0 2018-02-19 22:25 /cdap/namespaces/default/data/conn-6/7b0ffe75-15c0-11e8-98ba-42010afa000c
      drwxr-xr-x   - cdap cdap          0 2018-02-19 21:46 /cdap/namespaces/default/data/conn-6/876ac831-15bb-11e8-ad43-42010afa000c
      drwxr-xr-x   - cdap cdap          0 2018-02-19 22:02 /cdap/namespaces/default/data/conn-7
      drwxr-xr-x   - cdap cdap          0 2018-02-19 22:24 /cdap/namespaces/default/data/conn-7/7b0ffe75-15c0-11e8-98ba-42010afa000c
      drwxr-xr-x   - cdap cdap          0 2018-02-19 21:46 /cdap/namespaces/default/data/conn-7/876ac831-15bb-11e8-ad43-42010afa000c
      drwxr-xr-x   - cdap cdap          0 2018-02-19 22:02 /cdap/namespaces/default/data/conn-8
      drwxr-xr-x   - cdap cdap          0 2018-02-19 22:24 /cdap/namespaces/default/data/conn-8/7b0ffe75-15c0-11e8-98ba-42010afa000c
      drwxr-xr-x   - cdap cdap          0 2018-02-19 21:46 /cdap/namespaces/default/data/conn-8/876ac831-15bb-11e8-ad43-42010afa000c
      drwxr-xr-x   - cdap cdap          0 2018-02-19 22:02 /cdap/namespaces/default/data/conn-9
      drwxr-xr-x   - cdap cdap          0 2018-02-19 22:24 /cdap/namespaces/default/data/conn-9/7b0ffe75-15c0-11e8-98ba-42010afa000c
      drwxr-xr-x   - cdap cdap          0 2018-02-19 21:46 /cdap/namespaces/default/data/conn-9/876ac831-15bb-11e8-ad43-42010afa000c
      

      I'm not sure if this is a problem with just workflow local filesets or filesets in general, but the problem is more pronounced with local filesets since a directory is created for each workflow run.

        Attachments

          Activity

            People

            • Assignee:
              ashau Albert Shau
              Reporter:
              ashau Albert Shau
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: