Interconnecting MRS Kafka with Kafka Eagle

Prerequisites

• You have bought an MRS 3.1.0 cluster that contains the Kafka component and enabled Kerberos authentication for the cluster. For details, see Buying a Custom Cluster.
• The MRS cluster client has been installed. For details, see Installing a Client.

Introduction to Kafka Eagle

Kafka Eagle is a distributed and highly available Kafka monitoring software. It provides a range of Kafka monitoring metrics, such as the number of brokers, topics, consumers, Topic LogSize Top10, Topic Capacity Top10, Lag squeeze, and CPU/memory of Kafka clusters.

Eagle is renamed EFAK in the new version.

Kafka Eagle Installation Procedure

1. Download Kafka Eagle. The following uses EFAK3.0.1 as an example.

   

   For example, download the kafka-eagle-bin-3.0.1.tar.gz software package.

2. Log in to FusionInsight Manager, choose Cluster > Services > Kafka, click Configurations, and then All Configurations. Search for KAFKA_JMX_IP and change the value to ${BROKER_IP}.
   Figure 1 Modifying Kafka parameters
   

3. Click Save in the upper left corner. In the displayed dialog box, click OK.
4. Click the Dashboard tab and choose More > Restart Service in the upper right corner to restart the Kafka service.
5. Log in to the active node of the cluster as the root user, save the obtained EFAK installation package kafka-eagle-bin-3.0.1.tar.gz to the cluster directory, for example, /opt, and run the following command to decompress the package:

   cd /opt

   tar -xvf kafka-eagle-bin-3.0.1.tar.gz

   cd kafka-eagle-bin-3.0.1

   tar -xvf efak-web-3.0.1-bin.tar.gz

6. Create a directory in the opt directory, for example, efak, and copy efak-web-3.0.1 to the /opt/efak directory.

   mkdir /opt/efak

   cp -r /opt/kafka-eagle-bin-3.0.1/efak-web-3.0.1 /opt/efak/

7. Add environment variables.

   vi /etc/profile

   Add the export KE_HOME parameter. The parameter value is the path of the efak-web-3.0.1 file (example value: /opt/efak/efak-web-3.0.1). Add $KE_HOME/bin to the end of the export PATH value. The following is an example:

   export KE_HOME=/opt/efak/efak-web-3.0.1
   export PATH=$PATH:$KE_HOME/bin

8. Modify the system-config.properties configuration file.

   cd /opt/efak/efak-web-3.0.1/conf/

   vi system-config.properties

   # Configure a cluster.
   kafka.eagle.zk.cluster.alias=cluster1
   cluster1.zk.list=10.20.90.24:2181
   #cluster2.zk.list=xdn10:2181,xdn11:2181,xdn12:2181
   # Modify kafka jmx uri.
   cluster1.efak.jmx.uri=service:jmx:rmi:///jndi/rmi://%s/kafka 
   # Modify the database configuration.
   efak.driver=com.mysql.cj.jdbc.Driver
   efak.url=jdbc:mysql://IP:Port/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
   efak.username=root
   efak.password=XXX

   

   • The value of cluster1.zk.list is the value of the Kafka component parameter metrics.reporter.zookeeper.url. To obtain the value, you can log in to FusionInsight Manager, choose Cluster > Services > Kafka, click Configurations, and then All Configurations. Search for metrics.reporter.zookeeper.url.
   • The value of efak.url is the string representation of the MySQL JDBC connection.
   • The value of efak.username is the username for connecting to the database.
   • The value of efak.password is the password of the username for connecting to the database.

9. Start the EFAK service.

   sh /opt/efak/efak-web-3.0.1/bin/ke.sh start

   Obtain the EFAK web UI login address from the command output.

   

10. Log in to the EFAK web UI.
    

    The default initial username for logging in to the EFKA web UI is admin and the password is 123456.

    You can access the Kafka cluster monitoring page, topic monitoring page, and consumer monitoring page.

    Figure 2 Cluster monitoring
    
    Figure 3 Topic monitoring
    
    Figure 4 Consumer monitoring
    

FAQs

Symptom:

java.io.IOException cannot be cast to javax.management.remote.JMXConnector

Possible Cause

cluster1.efak.jmx.uri=service:jmx:rmi:///jndi/rmi://%s/jmxrmi

Solution

cluster1.efak.jmx.uri=service:jmx:rmi:///jndi/rmi://%s/kafka