Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-2894

API to get the system properties (e.g. MapReduce counters in case of MapReduce program) from the WorkflowToken

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.1.0
    • Component/s: Workflow
    • Labels:
      None
    • Sprint:
      Workflow/Spark ending 07/06
    • Rank:
      1|hzyc6n:

      Description

      We need to provide a better way to get the system properties from the WorkflowToken.
      Following approaches tried so far:

      1. In the "onFinish" method of the MapReduce program, user can add the counters to the WorkflowToken as required.
      Disadvantages: User will explicitly require to add the MapReduce counters, which will involve the fair amount of coding on the user's part.

      2. CDAP can provide the MapReduce counters implicitly with some prefix such as "mr.counters".
      Advantages: User can focus on map and reduce functionality and will get access to the counters from token out of the box.
      Disadvantages: User will need to remember the prefix "mr.counters", which can be prone to error.

      Another way to solve the problem is by having SCOPE.

        Attachments

          Activity

            People

            • Assignee:
              sagar Sagar Kapare
              Reporter:
              sagar Sagar Kapare
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: