Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-10649

Ability to perform point lookup of data from a dataset in Transforms

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Pipelines, Spark
    • Labels:
    • Rank:
      1|hzzoun:

      Description

      We need the ability to do point lookups in Transform plugins. We have that feature for KVTables but this feature will not work in Spark due to closure limitations. Also, note that the dataset could be in a different namespace.

      The use case is when a record comes in and we want to lookup a property for a field in that record from a dataset. The workaround in Batch pipelines is to have another source that reads from the dataset and performs a join to enrich the data. For this workaround will not work for realtime streaming pipelines.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                gokul Gokul Gunasekaran
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: