Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-13318

Recursive logging in spark programs

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Log, Spark
    • Labels:
    • Rank:
      1|i00c2n:

      Description

      I took the example application SparkPageRank and packaged a custom logback.xml under cdap/cdap-examples/SparkPageRank/src/main/resources/logback.xml.
      I've attached this file as logback.xml.

      The main change in this logback.xml file is to change spark's log level from WARN to INFO:

      <logger name="org.apache.spark" level="INFO"/> 

      I then uploaded the application and run the PageRankWorkflow. The spark job gets submitted, but the AM contains a spam of logs, of which I've attached a snippet of just below:

       INFO [main:c.c.c.c.l.RedirectedPrintStream@128] - 2018-04-17 05:22:21,859 - INFO [main:c.c.c.c.l.RedirectedPrintStream@128] - 2018-04-17 05:22:21,859 - INFO [main:c.c.c.c.l.RedirectedPrintStream@128] - 2018-04-17 05:22:21,859 - INFO [main:c.c.c.c.l.RedirectedPrintStream@128] - 2018-04-17 05:22:21,858 - INFO [main:c.c.c.c.l.RedirectedPrintStream@128] - 2018-04-17 05:22:21,858 - INFO [main:c.c.c.c.l.RedirectedPrintStream@128] - 2018-04-17 05:22:21,858 - INFO [main:c.c.c.c.l.RedirectedPrintStream@128] - 2018-04-17 05:22:21,858 - INFO [main:c.c.c.c.l.RedirectedPrintStream@128] - 2018-04-17 05:22:21,857 - INFO [main:c.c.c.c.l.RedirectedPrintStream@128] - 2018-04-17 05:22:21,857 - INFO [main:c.c.c.c.l.RedirectedPrintStream@128] - 2018-04-17 05:22:21,856 - INFO [main:c.c.c.c.l.RedirectedPrintStream@128] - 2018-04-17 05:22:21,856 - INFO [main:c.c.c.c.l.RedirectedPrintStream@128] - 2018-04-17 05:22:21,855 - INFO [main:c.c.c.c.l.RedirectedPrintStream@128] - 2018-04-17 05:22:21,855 - INFO [main:c.c.c.c.l.RedirectedPrintStream@128] - 2018-04-17 05:22:21,855 - INFO [main:c.c.c.c.l.RedirectedPrintStream@128] - 2018-04-17 05:22:21,854 - INFO [main:c.c.c.c.l.RedirectedPrintStream@128] - 2018-04-17 05:22:21,854 - INFO [main:c.c.c.c.l.RedirectedPrintStream@128] - 2018-04-17 05:22:21,853 - INFO [main:c.c.c.c.l.RedirectedPrintStream@128] - 2018-04-17 05:22:21,853 - INFO [main:c.c.c.c.l.RedirectedPrintStream@128] - 2018-04-17 05:22:21,852 - INFO [main:c.c.c.c.l.RedirectedPrintStream@128] - 2018-04-17 05:22:21,852 - INFO [main:c.c.c.c.l.RedirectedPrintStream@128] - 2018-04-17 05:22:21,852 - INFO [main:c.c.c.c.l.RedirectedPrintStream@128] -

       
      A workaround to this issue is to set the spark log level back to WARN.
      Commenting out the ConsoleLogAppender from the logback.xml also fixed the issue.

        Attachments

          Activity

            People

            • Assignee:
              bhooshan Bhooshan Mogal
              Reporter:
              ali.anwar Ali Anwar
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: