Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-4026

Make the underlying processing framework's context available to ETL plugins

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 3.2.0
    • Fix Version/s: None
    • Component/s: ETL
    • Labels:
    • Rank:
      1|hzz1wf:

      Description

      Some plugins (sources, transforms, sinks) may want to access the context of the processing framework (e.g. MapReduce, Spark). They may want to use this for:
      1. Updating M/R counters
      2. Accessing the input split, so they can make decisions based on file paths (e.g. parse the file path and infer some properties)
      3. Accessing files in DistributedCache

      This should be implemented without a tight coupling with the underlying processing framework.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ashau Albert Shau
                Reporter:
                bhooshan Bhooshan Mogal
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: