Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-7361

Corrupted log file if log.saver container crashed due to kill by YARN

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.5.1
    • Fix Version/s: 4.0.0, 3.6.0, 3.5.2
    • Component/s: Log
    • Labels:
    • Release Notes:
      Fixed a problem with log file corruption if the log saver container crashes due to being killed by YARN.
    • Rank:
      1|hzzmgv:

      Description

      Here is the exception when trying to fetch the corrupted log file.

      2016-09-23 09:50:16,716 - ERROR [CommonNettyHttpServiceBuilder-executor-1316:o.a.a.f.DataFileStream@102] - Got exception while reading log file hdfs://BoldSandbox/project/ecpsandingest/cdap/logs/avro/EcpNovusBulkIngest/workflow-BulkLoadWorkflow/2016-09-22/1474577020788.avro
      java.io.IOException: Not a data file.
      	at org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:102)
      	at org.apache.avro.file.DataFileReader.<init>(DataFileReader.java:97)
      	at co.cask.cdap.logging.read.AvroFileReader.createReader(AvroFileReader.java:233)
      	at co.cask.cdap.logging.read.AvroFileReader.readLog(AvroFileReader.java:61)
      	at co.cask.cdap.logging.read.FileLogReader.getLog(FileLogReader.java:169)
      	at co.cask.cdap.logging.read.DistributedLogReader.getLog(DistributedLogReader.java:114)
      	at co.cask.cdap.logging.gateway.handlers.LogHandler.doGetLogs(LogHandler.java:127)
      	at co.cask.cdap.logging.gateway.handlers.LogHandler.getRunIdLogs(LogHandler.java:105)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:497)
      	at co.cask.http.HttpMethodInfo.invoke(HttpMethodInfo.java:80)
      	at co.cask.http.HttpDispatcher.messageReceived(HttpDispatcher.java:38)
      	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
      	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
      	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:124)
      	at co.cask.cdap.common.http.AuthenticationChannelHandler.messageReceived(AuthenticationChannelHandler.java:64)
      	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
      	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
      	at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
      	at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
      	at org.jboss.netty.handler.execution.OrderedMemoryAwareThreadPoolExecutor$ChildExecutor.run(OrderedMemoryAwareThreadPoolExecutor.java:314)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                poorna Poorna Chandra
                Reporter:
                terence Terence Yim
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: