Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-12989

Add rate limited logging for zookeeper.KeeperException

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.3.2
    • Fix Version/s: 5.0.0
    • Component/s: CDAP, Log
    • Labels:
    • Release Notes:
      Added rate limiting to router logs in the event of zookeeper outages
    • Rank:
      1|i00a5b:

      Description

      We have seen when zookeeper crashes and recovers CDAP can log the following up to 43 million times of a 48 hour period. We should add a rate limit to this.

      2017-12-13 11:27:52,080 - WARN [leader-election-leader:o.a.t.i.z.LeaderElection$5@218] - Got exception during children fetch for /leader. Retry. 
      org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /cdap/security/auth/leader 
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:123) ~[zookeeper-3.4.6.2.6.2.0-205.jar:3.4.6-205--1] 
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.2.6.2.0-205.jar:3.4.6-205--1] 
      at org.apache.twill.internal.zookeeper.DefaultZKClientService$Callbacks$4.processResult(DefaultZKClientService.java:619) ~[org.apache.twill.twill-zookeeper-0.12.0.jar:0.12.0] 
      at org.apache.zookeeper.ClientCnxn$E2017-12-13 11:27:56,288 - WARN [leader-election-leader:o.a.t.i.z.LeaderElection$5@218] - Got exception during children fetch for /leader. Retry. 
      org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /cdap/security/auth/leader 
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:123) ~[zookeeper-3.4.6.2.6.2.0-205.jar:3.4.6-205--1] 
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.2.6.2.0-205.jar:3.4.6-205--1] 
      at org.apache.twill.internal.zookeeper.DefaultZKClientService$Callbacks$4.processResult(DefaultZKClientService.java:619) ~[org.apache.twill.twill-zookeeper-0.12.0.jar:0.12.0] 
      at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:605) ~[zookeeper-3.4.6.2.6.2.0-205.jar:3.4.6-205--1] 
      at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) ~[zookeeper-3.4.6.2.6.2.0-205.jar:3.4.6-205--1] 
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sree Sreevatsan Raman
                Reporter:
                mattwuenschel Matt Wuenschel
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: