Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-6829

Log Saver fails to cleanup log files when file metadata is present for a non-existent file

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.3.6
    • Fix Version/s: 3.5.0, 3.3.7, 3.4.4
    • Component/s: Log
    • Labels:
      None
    • Release Notes:
      Fixes issues that prevents log saver from performing cleanup when metadata is present for a non-existing file.
    • Rank:
      1|hzzihb:

      Description

      After creating an HDFS file for writing log events, Log Saver immediately adds it to the file metadata table. After deleting an HDFS file older than retention duration, it also removes the metadata from the table.

      However, since file operations and table operations are not transactional there can be cases where even after a file is deleted, its metadata continues to exist.

      During cleanup, when we try to get the modified time for the non-existent file an exception gets thrown, and the cleanup gets aborted from that point onward.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                poorna Poorna Chandra
                Reporter:
                poorna Poorna Chandra
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: