Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-7697

Add kafka.topic.prefix configuration property to prepend all Kafka topics

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: CDAP
    • Labels:
      None
    • Rank:
      1|hzzp6v:

      Description

      A new configuration property kafka.topic.prefix will be used to prepend all the topics prefixes below. A use case is to differentiate topics in the same Zookeeper namespace but from different Kafka instance with the same topic name.

        <property>
          <name>root.namespace</name>
          <value>env1</value>
          <!-- Variable defined to be used to have a separation in HDFS and Zookeeper-->
          <!-- If multiple instance of CDAP share the same hadoop cluster, then the root.namespace should differ-->
        </property>
      
        <property>
          <name>hdfs.namespace</name>
          <value>/cdap/${root.namespace}</value>
          <!-- HDFS top level directory inside with CDAP instance level data will reside on HDFS--->
        </property>
        <property>
          <name>zookeeper.quorum</name>
          <value><hostname>/{root.namespace}</value>
          <!-- zookeeper quorum which is used for service registration and discovery, if multiple instance of CDAP share the same zookeeper instance then /${root.namespace} should be suffixed, other wise the service discovery and registration will not work as intended--> 
        </property>
      
        <property>
          <name>log.kafka.topic</name>
          <value>${root.namespace}.logs.user-v2</value>
          <!-- Topic name for log publishing-->
        </property>
      
        <property>
          <name>notification.kafka.topic</name>
          <value>${root.namespace}.notifications</value>
          <!-- Topic name for notification publishing-->
        </property>
      
        <property>
          <name>metadata.updates.kafka.topic</name>
          <value>${root.namesspace}.cdap-metadata-updates</value>
          <!-- Topic name for metadata publishing-->
        </property>
      
        <property>
          <name>audit.kafka.topic</name>
          <value>${root.namespace}.audit</value>
          <!-- Topic name for audit publishing-->
        </property>
      
        <property>
          <name>metrics.kafka.topic.prefix</name>
          <value>${root.namespace}.metrics</value>
          <!-- Topic prefix for metrics publishing-->
        </property>
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                mao Chengfeng Mao
                Reporter:
                mao Chengfeng Mao
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: