Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-6553

dropPartition is not transactional

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 3.5.0
    • Fix Version/s: None
    • Component/s: Datasets, Explore
    • Labels:
      None
    • Release Notes:
      Reopening to mark as Resolution: Duplicate, instead of Fixed
    • Rank:
      1|hzzghr:

      Description

      From CDAP-6062: Note: This protects a partition and its files from being deleted if a Hive query is concurrently reading it. However, it does not guarantee atomicity of drop operations across multiple partitions: If I try to delete 5 partitions in a single transaction, and the 4th partition is locked by Hive, then the first three are already deleted from Hive (and the file system) when we fail for number 4 (and the 5th will not even be attempted to delete, because the transaction fails). But the first three cannot be reinstated during tx rollback, because both Hive and the file system are not transactional and don't allow rollback. The partition meta data for these three will, however, be reinstated during rollback.

      Therefore this only protects against concurrency consistently if a single partition is deleted per transaction.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                nitin Nitin Motgi
                Reporter:
                andreas Andreas Neumann
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: