Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-9191

SparkWikipediaClustering in WikipediaPipeline out of memory in PermGen space

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 4.1.0
    • Fix Version/s: None
    • Component/s: CDAP Examples
    • Labels:
      None
    • Rank:
      1|hzzyrz:

      Description

      Spark program SparkWikipediaClustering in WikipediaPipeline example failed with exceptions in driver:

      Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "sparkDriver-akka.remote.default-remote-dispatcher-5"
      
      Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "sparkDriver-akka.remote.default-remote-dispatcher-6"
      Exception in thread "Driver" 
      Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Driver"
      Exception in thread "LeaseRenewer:cdap@10.128.0.24" 
      

      and following gc logs:

      Heap
       PSYoungGen      total 290304K, used 246687K [0x00000000eaa80000, 0x0000000100000000, 0x0000000100000000)
        eden space 230912K, 100% used [0x00000000eaa80000,0x00000000f8c00000,0x00000000f8c00000)
        from space 59392K, 26% used [0x00000000fc600000,0x00000000fd567ef8,0x0000000100000000)
        to   space 59392K, 0% used [0x00000000f8c00000,0x00000000f8c00000,0x00000000fc600000)
       ParOldGen       total 321024K, used 137415K [0x00000000c0000000, 0x00000000d3980000, 0x00000000eaa80000)
        object space 321024K, 42% used [0x00000000c0000000,0x00000000c8631d40,0x00000000d3980000)
       PSPermGen       total 93184K, used 93099K [0x00000000b8000000, 0x00000000bdb00000, 0x00000000c0000000)
        object space 93184K, 99% used [0x00000000b8000000,0x00000000bdaeace0,0x00000000bdb00000)
      

      After increasing Spark driver PermGen size in SparkWikipediaClustering with

      @Override
        protected void initialize() throws Exception {
          SparkConf sparkConf = new SparkConf()
            .set("spark.driver.extraJavaOptions", "-XX:MaxPermSize=128m");
          getContext().setSparkConf(sparkConf);
        }
      

      driver stops having such exceptions but the same exceptions happen to executors:

      Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Executor task launch worker-0"
      
      Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-10"
      Exception when shutting down services
      
      Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-6"
      

      with gc logs:

      Heap
       PSYoungGen      total 290304K, used 246687K [0x00000000eaa80000, 0x0000000100000000, 0x0000000100000000)
        eden space 230912K, 100% used [0x00000000eaa80000,0x00000000f8c00000,0x00000000f8c00000)
        from space 59392K, 26% used [0x00000000fc600000,0x00000000fd567ef8,0x0000000100000000)
        to   space 59392K, 0% used [0x00000000f8c00000,0x00000000f8c00000,0x00000000fc600000)
       ParOldGen       total 321024K, used 137415K [0x00000000c0000000, 0x00000000d3980000, 0x00000000eaa80000)
        object space 321024K, 42% used [0x00000000c0000000,0x00000000c8631d40,0x00000000d3980000)
       PSPermGen       total 93184K, used 93099K [0x00000000b8000000, 0x00000000bdb00000, 0x00000000c0000000)
        object space 93184K, 99% used [0x00000000b8000000,0x00000000bdaeace0,0x00000000bdb00000)
      
      @Override
        protected void initialize() throws Exception {
          SparkConf sparkConf = new SparkConf()
            .set("spark.driver.extraJavaOptions", "-XX:MaxPermSize=128m")
            .set("spark.executor.extraJavaOptions", "-XX:MaxPermSize=128m");
          getContext().setSparkConf(sparkConf);
        }
      

      solves the exceptions

        Attachments

          Activity

            People

            • Assignee:
              mao Chengfeng Mao
              Reporter:
              mao Chengfeng Mao
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: