Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-13011

Plugin Reference Name should be macro enabled


    • Type: Improvement
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 4.3.2
    • Fix Version/s: None
    • Component/s: Pipeline Plugins, Pipelines
    • Labels:
    • Rank:


      If a pipeline developer uses macros to parametrize a pipeline, for example, the name of the dataset sink, or the location of the file source, then that is only useful if she can use a different reference name for each instance. Otherwise, the same dataset name will be used for all sources/sinks, for example, all file system locations that are ever passed in. But they may represent different data sources, hence meta data, lineage, audit etc. should be done on different entities.

      The caveat here is that the user may configure a different reference name for every run of the pipeline. That would lead to the creation of a large number of "external datasets" over time. One way to prevent that could be to require that the external dataset already exists if it was given as a macro.




            • Assignee:
              ashau Albert Shau
              andreas Andreas Neumann
            • Votes:
              0 Vote for this issue
              1 Start watching this issue


              • Created: