Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-13088

Metadata for schema fields and custom non-entities in CDAP

    Details

    • Type: New Feature
    • Status: IN PROGRESS
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 5.0.0
    • Fix Version/s: None
    • Component/s: Metadata
    • Labels:
      None
    • Sprint:
      Metadata 5.0
    • Rank:
      1|hzz9iz:

      Description

      This is the umbrella jira for Metadata changes in 5.0
      Please note that this jira does not capture any tasks/requirement related to capturing and storing field level lineage information.

      Desing doc: https://wiki.cask.co/display/CE/Metadata+Custom+Entities+and+Authorization
      User stories: https://wiki.cask.co/pages/viewpage.action?pageId=4365177 (Note: Not all the user stories are part of this jira)

      This jira and its subtasks will captures the following requirements from the list here: https://wiki.cask.co/pages/viewpage.action?pageId=4365177#Metadata5.X+-Requirements
      Any other requirements not listed here is not captured and is not in scope for 5.0 as of now.

      Store

      • associate meta data with a file
      • associate meta data with a field of a dataset (’s schema)
      • retrieve meta data for non-CDAP entities
      • search meta data for non-CDAP entities

      Pipeline

      • explicitly set meta data for an entity (note: in our current implementation the support will only be for entity associated with source and sink plugin or its sub-entities. For example if source is a DatasetSource then the plugin will be able to add metadata to the dataset or any of its sub-entity for example field)

      Integrations

      • all meta data operations via message bus
      • authorization for meta data through Ranger/Sentry/external auth provider

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sree Sreevatsan Raman
                Reporter:
                rsinha Rohit Sinha
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: