Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-1952

MapReduce runtime should call the Dataset's onSuccess/Failure inside an OutputCommitter

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 4.3.0
    • Component/s: App Fabric, Datasets
    • Labels:
    • Release Notes:
      Hide
      DatasetOutputCommiter's methods are now executed in the MapReduce ApplicationMaster, within OutputCommitter's commitJob/abortJob methods.
      The MapReduceContext.addOutput(Output.of(String, OutputFormatProvider)) API can no longer be used to add OutputFormatProviders that also implement the DatasetOutputCommitter interface.
      Show
      DatasetOutputCommiter's methods are now executed in the MapReduce ApplicationMaster, within OutputCommitter's commitJob/abortJob methods. The MapReduceContext.addOutput(Output.of(String, OutputFormatProvider)) API can no longer be used to add OutputFormatProviders that also implement the DatasetOutputCommitter interface.
    • Rank:
      1|hzypxb:

      Description

      Currently, it is called from the same context as the onFinish of the MapReduce. That means, onSuccess (for example, adding a partition to a PFS) does not happen in the same transaction as the MapReduce. Also, if onSuccess() fails, that does not fail and roll back the MapReduce.

      The way it is done in CDAP-1227 allows backward-compatibility with existing apps that add partitions in the onFinish() of their MapReduce. These apps will now produce warnings because the partition was already added. We should wait one or two minor releases before we move this in to the output committer of the MapReduce, because that will break such existing apps.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ali.anwar Ali Anwar
                Reporter:
                andreas Andreas Neumann
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: