Tephra
  1. Tephra
  2. TEPHRA-35

Prune invalid transaction set once all data for a given invalid transaction has been dropped

    Details

    • Type: New Feature New Feature
    • Status: Open Open
    • Priority: Blocker Blocker
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • External issue ID:
      24
    • Rank:
      1|hzy913:

      Description

      In addition to dropping the data from invalid transactions we need to be able to prune the invalid set of any transactions where data cleanup has been completely performed. Without this, the invalid set will grow indefinitely and become a greater and greater cost to in-progress transactions over time.

      To do this correctly, the TransactionDataJanitor coprocessor will need to maintain some bookkeeping for the transaction data that it removes, so that the transaction manager can reason about when all of a given transaction's data has been removed. Only at this point can the transaction manager safely drop the transaction ID from the invalid set.

      One approach would be for the TransactionDataJanitor to update a table marking when a major compaction was performed on a region and what transaction IDs were filtered out. Once all regions in a table containing the transaction data have been compacted, we can remove the filtered out transaction IDs from the invalid set. However, this will need to cope with changing region names due to splits, etc.

        Issue Links

          Activity

          Hide
          James Taylor added a comment -

          This would seem to inherently limit the scalability of Tephra. Any insight on when this will be implemented?

          Show
          James Taylor added a comment - This would seem to inherently limit the scalability of Tephra. Any insight on when this will be implemented?
          Hide
          Alex Baranau added a comment -

          James Taylor There's no target fix version or date set for it.

          Note that in a healthy system it is very rare that transaction becomes invalid. Usually only if there's a client process crash or a datastore crash. In normal situation even if commit fails, the transaction gets rolled back and not put into invalid list. It is highly unlikely to accumulate a big size of invalid list. Though it can happen, hence the priority of fixing it is one of the highest.

          Show
          Alex Baranau added a comment - James Taylor There's no target fix version or date set for it. Note that in a healthy system it is very rare that transaction becomes invalid. Usually only if there's a client process crash or a datastore crash. In normal situation even if commit fails, the transaction gets rolled back and not put into invalid list. It is highly unlikely to accumulate a big size of invalid list. Though it can happen, hence the priority of fixing it is one of the highest.
          Hide
          James Taylor added a comment -

          Hmm. That's like saying transactions aren't important because usually everything works correctly. Unhealthy systems are one of the big reasons people want transactions.

          Show
          James Taylor added a comment - Hmm. That's like saying transactions aren't important because usually everything works correctly. Unhealthy systems are one of the big reasons people want transactions.
          Hide
          Gary Helmling added a comment -

          This is definitely a scalability concern and we are working on a plan for addressing it. However, we don't yet have a target date.

          There are two approaches we can take to mitigate this. The first is an operational approach, where Tephra would provide the ability for an administrator to truncate the invalid list up to a given point. The idea is that, as part of a normal operational policy handling major compactions, an admin would know up to what time all tables in the cluster have been major compacted. Since Tephra transaction IDs are time based, you could then manually issue a command to truncate the invalid list up to this time, since you know, by virtue of the major compactions completing, that any data from invalid transactions prior to that point have been purged. This isn't ideal, as it requires some operational coordination, but it is doable.

          The second approach would build this processing and tracking into Tephra itself, so that it could make the determination to automatically truncate the invalid list. This will require quite a bit more complexity to do the tracking, and needs a detailed design around it.

          Show
          Gary Helmling added a comment - This is definitely a scalability concern and we are working on a plan for addressing it. However, we don't yet have a target date. There are two approaches we can take to mitigate this. The first is an operational approach, where Tephra would provide the ability for an administrator to truncate the invalid list up to a given point. The idea is that, as part of a normal operational policy handling major compactions, an admin would know up to what time all tables in the cluster have been major compacted. Since Tephra transaction IDs are time based, you could then manually issue a command to truncate the invalid list up to this time, since you know, by virtue of the major compactions completing, that any data from invalid transactions prior to that point have been purged. This isn't ideal, as it requires some operational coordination, but it is doable. The second approach would build this processing and tracking into Tephra itself, so that it could make the determination to automatically truncate the invalid list. This will require quite a bit more complexity to do the tracking, and needs a detailed design around it.
          Hide
          Gary Helmling added a comment -

          James Taylor I don't think anyone is saying this issue isn't important. It is high priority.

          As I mentioned, there are two approaches with differing levels of complexity. We will likely implement this in those two stages.

          Show
          Gary Helmling added a comment - James Taylor I don't think anyone is saying this issue isn't important. It is high priority. As I mentioned, there are two approaches with differing levels of complexity. We will likely implement this in those two stages.
          Hide
          Alex Baranau added a comment -

          James Taylor I don't think anyone is saying this issue isn't important. It is high priority.

          Yes! Exactly:

          Though it can happen, hence the priority of fixing it is one of the highest.

          Show
          Alex Baranau added a comment - James Taylor I don't think anyone is saying this issue isn't important. It is high priority. Yes! Exactly: Though it can happen, hence the priority of fixing it is one of the highest.

            People

            • Assignee:
              Gary Helmling
              Reporter:
              Gary Helmling
            • Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated: