Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-13326

Pipelines should propagate dynamic schemas

    XMLWordPrintableJSON

    Details

    • Rank:
      1|i00c4n:

      Description

      More and more, we see use cases where schema is macro-enabled. Today, the plugins can set their output schema given their config properties and given their input schema. This schema is then accessible at runtime through the context's getInputSchema() method. However, setting the output schema is only done in the configurePipeline method, which means it only happens for static schemas. Any plugin that wants to be used in a pipeline with dynamic schema must specify its own schema property.

      There should also be a way to set output schema at runtime, after the macros have been resolved. Then context.getInputSchema() would always be accurate, and plugins would not need to have an extra schema property just because some upstream pipeline stage is using dynamic schema.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ashau Albert Shau
                Reporter:
                ashau Albert Shau
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: