Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.2.0, 4.1.1
    • Component/s: Pipelines, UI
    • Labels:
    • Sprint:
      CDAP UI 02/27 (4.2)
    • Release Notes:
      Removed the "__ui__" property which was being used only by CDAP UI while exporting a pipeline configuration.
    • Rank:
      1|hzy307:

      Description

      The UI is storing a lot of information in a special _ui_ section in each Hydrator config. This has been the cause of a lot bugs related to import/export, and pipelines created through market, CLI, or REST.

      From what I can tell, most of the information is redundant. Much of what is stored there can be taken from the 'pipeline.spec' property that the programs set. For example, this is the result when I export a simple pipeline:

      {
          "name": "simple_copy",
          "artifact": {
              "name": "cdap-data-pipeline",
              "version": "4.0.1",
              "scope": "SYSTEM"
          },
          "__ui__": {
              "nodes": [
                  {
                      "name": "Stream",
                      "plugin": {
                          "name": "Stream",
                          "type": "batchsource",
                          "label": "Stream",
                          "artifact": {
                              "name": "core-plugins",
                              "version": "1.5.1",
                              "scope": "SYSTEM"
                          },
                          "properties": {
                              "schema": "{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"email\",\"type\":\"string\"}]}",
                              "format": "csv",
                              "name": "streem",
                              "duration": "1d"
                          }
                      },
                      "outputSchema": "{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"email\",\"type\":\"string\"}]}",
                      "type": "batchsource",
                      "label": "Stream",
                      "icon": "icon-streams",
                      "_uiPosition": {
                          "top": "341px",
                          "left": "524.5px"
                      },
                      "$$hashKey": "object:2994"
                  },
                  {
                      "name": "Projection",
                      "plugin": {
                          "name": "Projection",
                          "type": "transform",
                          "label": "Projection",
                          "artifact": {
                              "name": "core-plugins",
                              "version": "1.5.1",
                              "scope": "SYSTEM"
                          },
                          "properties": {
                              "drop": "ts,headers"
                          }
                      },
                      "outputSchema": "{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"email\",\"type\":\"string\"}]}",
                      "inputSchema": [
                          {
                              "name": "Stream",
                              "schema": "{\"name\":\"etlSchemaBody\",\"type\":\"record\",\"fields\":[{\"name\":\"ts\",\"type\":\"long\"},{\"name\":\"headers\",\"type\":{\"type\":\"map\",\"keys\":\"string\",\"values\":\"string\"}},{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"email\",\"type\":\"string\"}]}"
                          }
                      ],
                      "type": "transform",
                      "label": "Projection",
                      "icon": "icon-projection",
                      "_uiPosition": {
                          "top": "341px",
                          "left": "824.5px"
                      },
                      "$$hashKey": "object:2995"
                  },
                  {
                      "name": "TPFSAvro",
                      "plugin": {
                          "name": "TPFSAvro",
                          "type": "batchsink",
                          "label": "TPFSAvro",
                          "artifact": {
                              "name": "core-plugins",
                              "version": "1.5.1",
                              "scope": "SYSTEM"
                          },
                          "properties": {
                              "compressionCodec": "None",
                              "schema": "{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"email\",\"type\":\"string\"}]}",
                              "name": "avrogado"
                          }
                      },
                      "outputSchema": "{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"email\",\"type\":\"string\"}]}",
                      "inputSchema": [
                          {
                              "name": "Projection",
                              "schema": "{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"email\",\"type\":\"string\"}]}"
                          }
                      ],
                      "type": "batchsink",
                      "label": "TPFSAvro",
                      "icon": "icon-tpfsavro",
                      "_uiPosition": {
                          "top": "341px",
                          "left": "1124.5px"
                      },
                      "$$hashKey": "object:2996"
                  }
              ],
              "connections": [
                  {
                      "from": "Stream",
                      "to": "Projection"
                  },
                  {
                      "from": "Projection",
                      "to": "TPFSAvro"
                  }
              ]
          },
          "config": {
              "batchInterval": "1s",
              "resources": {
                  "memoryMB": 1024,
                  "virtualCores": 1
              },
              "driverResources": {
                  "memoryMB": 1024,
                  "virtualCores": 1
              },
              "schedule": "0 * * * *",
              "connections": [
                  {
                      "from": "Stream",
                      "to": "Projection"
                  },
                  {
                      "from": "Projection",
                      "to": "TPFSAvro"
                  }
              ],
              "comments": [],
              "postActions": [],
              "engine": "mapreduce",
              "stages": [
                  {
                      "name": "Stream",
                      "plugin": {
                          "name": "Stream",
                          "type": "batchsource",
                          "label": "Stream",
                          "artifact": {
                              "name": "core-plugins",
                              "version": "1.5.1",
                              "scope": "SYSTEM"
                          },
                          "properties": {
                              "schema": "{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"email\",\"type\":\"string\"}]}",
                              "format": "csv",
                              "name": "streem",
                              "duration": "1d"
                          }
                      },
                      "outputSchema": "{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"email\",\"type\":\"string\"}]}",
                      "type": "batchsource",
                      "label": "Stream",
                      "icon": "icon-streams",
                      "_uiPosition": {
                          "top": "341px",
                          "left": "524.5px"
                      },
                      "$$hashKey": "object:2994"
                  },
                  {
                      "name": "Projection",
                      "plugin": {
                          "name": "Projection",
                          "type": "transform",
                          "label": "Projection",
                          "artifact": {
                              "name": "core-plugins",
                              "version": "1.5.1",
                              "scope": "SYSTEM"
                          },
                          "properties": {
                              "drop": "ts,headers"
                          }
                      },
                      "outputSchema": "{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"email\",\"type\":\"string\"}]}",
                      "inputSchema": [
                          {
                              "name": "Stream",
                              "schema": "{\"name\":\"etlSchemaBody\",\"type\":\"record\",\"fields\":[{\"name\":\"ts\",\"type\":\"long\"},{\"name\":\"headers\",\"type\":{\"type\":\"map\",\"keys\":\"string\",\"values\":\"string\"}},{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"email\",\"type\":\"string\"}]}"
                          }
                      ],
                      "type": "transform",
                      "label": "Projection",
                      "icon": "icon-projection",
                      "_uiPosition": {
                          "top": "341px",
                          "left": "824.5px"
                      },
                      "$$hashKey": "object:2995"
                  },
                  {
                      "name": "TPFSAvro",
                      "plugin": {
                          "name": "TPFSAvro",
                          "type": "batchsink",
                          "label": "TPFSAvro",
                          "artifact": {
                              "name": "core-plugins",
                              "version": "1.5.1",
                              "scope": "SYSTEM"
                          },
                          "properties": {
                              "compressionCodec": "None",
                              "schema": "{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"email\",\"type\":\"string\"}]}",
                              "name": "avrogado"
                          }
                      },
                      "outputSchema": "{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"email\",\"type\":\"string\"}]}",
                      "inputSchema": [
                          {
                              "name": "Projection",
                              "schema": "{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"email\",\"type\":\"string\"}]}"
                          }
                      ],
                      "type": "batchsink",
                      "label": "TPFSAvro",
                      "icon": "icon-tpfsavro",
                      "_uiPosition": {
                          "top": "341px",
                          "left": "1124.5px"
                      },
                      "$$hashKey": "object:2996"
                  }
              ]
          }
      }
      

      and this is the value of the 'pipeline.spec' property, available through

      $ curl localhost:11015/v3/namespaces/default/apps/simple/workflows/DataPipelineWorkflow
      
      {
          "clientResources": {
              "memoryMB": 1024,
              "virtualCores": 1
          },
          "connections": [
              {
                  "from": "Stream",
                  "to": "Projection"
              },
              {
                  "from": "Projection",
                  "to": "TPFSAvro"
              }
          ],
          "driverResources": {
              "memoryMB": 1024,
              "virtualCores": 1
          },
          "endingActions": [],
          "numOfRecordsPreview": 100,
          "resources": {
              "memoryMB": 1024,
              "virtualCores": 1
          },
          "stageLoggingEnabled": true,
          "stages": [
              {
                  "inputSchemas": {
                      "Stream": {
                          "fields": [
                              {
                                  "name": "ts",
                                  "type": "long"
                              },
                              {
                                  "name": "headers",
                                  "type": {
                                      "keys": "string",
                                      "type": "map",
                                      "values": "string"
                                  }
                              },
                              {
                                  "name": "id",
                                  "type": "long"
                              },
                              {
                                  "name": "name",
                                  "type": "string"
                              },
                              {
                                  "name": "email",
                                  "type": "string"
                              }
                          ],
                          "name": "event",
                          "type": "record"
                      }
                  },
                  "inputs": [
                      "Stream"
                  ],
                  "name": "Projection",
                  "outputSchema": {
                      "fields": [
                          {
                              "name": "id",
                              "type": "long"
                          },
                          {
                              "name": "name",
                              "type": "string"
                          },
                          {
                              "name": "email",
                              "type": "string"
                          }
                      ],
                      "name": "event.projected",
                      "type": "record"
                  },
                  "outputs": [
                      "TPFSAvro"
                  ],
                  "plugin": {
                      "artifact": {
                          "name": "core-plugins",
                          "scope": "SYSTEM",
                          "version": {
                              "fix": 1,
                              "major": 1,
                              "minor": 5,
                              "version": "1.5.1"
                          }
                      },
                      "name": "Projection",
                      "properties": {
                          "drop": "ts,headers"
                      },
                      "type": "transform"
                  }
              },
              {
                  "inputSchemas": {
                      "Projection": {
                          "fields": [
                              {
                                  "name": "id",
                                  "type": "long"
                              },
                              {
                                  "name": "name",
                                  "type": "string"
                              },
                              {
                                  "name": "email",
                                  "type": "string"
                              }
                          ],
                          "name": "event.projected",
                          "type": "record"
                      }
                  },
                  "inputs": [
                      "Projection"
                  ],
                  "name": "TPFSAvro",
                  "outputs": [],
                  "plugin": {
                      "artifact": {
                          "name": "core-plugins",
                          "scope": "SYSTEM",
                          "version": {
                              "fix": 1,
                              "major": 1,
                              "minor": 5,
                              "version": "1.5.1"
                          }
                      },
                      "name": "TPFSAvro",
                      "properties": {
                          "compressionCodec": "None",
                          "name": "avrogado",
                          "schema": "{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"email\",\"type\":\"string\"}]}"
                      },
                      "type": "batchsink"
                  }
              },
              {
                  "inputSchemas": {},
                  "inputs": [],
                  "name": "Stream",
                  "outputSchema": {
                      "fields": [
                          {
                              "name": "ts",
                              "type": "long"
                          },
                          {
                              "name": "headers",
                              "type": {
                                  "keys": "string",
                                  "type": "map",
                                  "values": "string"
                              }
                          },
                          {
                              "name": "id",
                              "type": "long"
                          },
                          {
                              "name": "name",
                              "type": "string"
                          },
                          {
                              "name": "email",
                              "type": "string"
                          }
                      ],
                      "name": "event",
                      "type": "record"
                  },
                  "outputs": [
                      "Projection"
                  ],
                  "plugin": {
                      "artifact": {
                          "name": "core-plugins",
                          "scope": "SYSTEM",
                          "version": {
                              "fix": 1,
                              "major": 1,
                              "minor": 5,
                              "version": "1.5.1"
                          }
                      },
                      "name": "Stream",
                      "properties": {
                          "duration": "1d",
                          "format": "csv",
                          "name": "streem",
                          "schema": "{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"email\",\"type\":\"string\"}]}"
                      },
                      "type": "batchsource"
                  }
              }
          ]
      }
      

      This property is also more reliable, as the schemas, properties, and plugin artifact information is exactly what is being used by the app.

        Attachments

          Activity

            People

            • Assignee:
              ajai Ajai Narayan
              Reporter:
              ashau Albert Shau
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: