Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-11097

Database Source plugin needs to FAIL a batch pipeline run when database returns ERRORs

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Critical
    • Resolution: Done
    • Affects Version/s: 4.0.1
    • Fix Version/s: None
    • Component/s: Pipelines
    • Labels:
    • Rank:
      1|hzzul3:

      Description

      Our Vertica database sometimes throws insufficient resource ERRORs and provides no (or apparently sometimes partial) results to Hydrator Database Source plugins. I have four Hydrator Database Source plugins in my pipeline and they ignore the ERRORs and proceed. Therefore the results are invalid yet the batch pipeline run declares SUCCESS. I need CDAP to declare the batch pipeline run FAILED instead of SUCCEEDED when any one of the Database Source plugins receives an ERROR from the Vertica database. This way partial results will not be committed by the pipeline to the sink CDAP Table Datasets.

      Logs are attached from one run of my pipeline in which all four database sources received ERRORs from Vertica, yet the pipeline declared the run SUCCEEDED.

        Attachments

          Activity

            People

            • Assignee:
              ts862m Terry Schmalzried
              Reporter:
              ts862m Terry Schmalzried
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: