Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-13011

Plugin Reference Name should be macro enabled

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 4.3.2
    • Fix Version/s: None
    • Component/s: Pipeline Plugins, Pipelines
    • Labels:
    • Rank:
      1|i00a93:

      Description

      If a pipeline developer uses macros to parametrize a pipeline, for example, the name of the dataset sink, or the location of the file source, then that is only useful if she can use a different reference name for each instance. Otherwise, the same dataset name will be used for all sources/sinks, for example, all file system locations that are ever passed in. But they may represent different data sources, hence meta data, lineage, audit etc. should be done on different entities.

      The caveat here is that the user may configure a different reference name for every run of the pipeline. That would lead to the creation of a large number of "external datasets" over time. One way to prevent that could be to require that the external dataset already exists if it was given as a macro.

        Attachments

          Activity

            People

            • Assignee:
              ashau Albert Shau
              Reporter:
              andreas Andreas Neumann
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: