Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-7208

Improve CDAP Master logging of events

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 3.5.0, 3.3.7, 3.4.3
    • Fix Version/s: 6.0.0
    • Component/s: Master
    • Labels:
    • Release Notes:
      Improved CDAP Master logging of events related to programs that it launches.
    • Rank:
      1|hzy5sn:

      Description

      Traceability of CDAP programs to YARN applications is hard to do via the CDAP master log. The following is needed to debug issues:

      • Each CDAP application that is launched should log the YARN Id or MapReduce Job Id
      • Each statement regarding a CDAP program should include the run-id. Often just the name is used and sometimes the name and namespace is used. For example multiple instances of a workflow can run at the same time or the same worker or service is running in multiple namespaces, it is important to know which program is being referenced.
      • Log relevant YARN information about each submitted program, number of YARN containers requested and amount obtained, requested memory/VCPUs
      • Log the state of a workflow
      • Workflow Y1 started by schedule X1
      • Workflow Y1 submitted to YARN
      • Workflow Y1 submitted MapReduce job MR1 (yarn resources used - number of containers requested, number of mappers/reducers)
      • Workflow Y1 completed MapReduce job MR1 (success or failure of job, number of retries, job duration)
      • Workflow Y1 ended - total x seconds to complete was workflow successful or failed

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                andreas Andreas Neumann
                Reporter:
                jd565c Jeff Dix
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: