Updated on 2024-08-12 GMT+08:00

Interconnecting MRS Kafka with Kafka Eagle

Introduction to Kafka Eagle

Kafka Eagle is a distributed and highly available Kafka monitoring software. It provides a range of Kafka monitoring metrics, such as the number of brokers, topics, consumers, Topic LogSize Top10, Topic Capacity Top10, Lag squeeze, and CPU/memory of Kafka clusters.

Eagle is renamed EFAK in the new version.

Prerequisites

  • You have bought an MRS 3.1.0 cluster that contains the Kafka component and enabled Kerberos authentication for the cluster. For details, see Buying a Custom Cluster.
  • The MRS cluster client has been installed. For details, see Installing a Client.

Kafka Eagle Installation Procedure

  1. Download Kafka Eagle. The following uses EFAK3.0.1 as an example.

    For example, download the kafka-eagle-bin-3.0.1.tar.gz software package.

  1. Log in to FusionInsight Manager, choose Cluster > Services > Kafka, click Configurations, and then All Configurations. Search for KAFKA_JMX_IP and change the value to ${BROKER_IP}.

    Figure 1 Modifying Kafka parameters

  2. Click Save in the upper left corner. In the displayed dialog box, click OK.
  3. Click the Dashboard tab and choose More > Restart Service in the upper right corner to restart the Kafka service.
  4. Log in to the active node of the cluster as the root user, save the obtained EFAK installation package kafka-eagle-bin-3.0.1.tar.gz to the cluster directory, for example, /opt, and run the following command to decompress the package:

    cd /opt

    tar -xvf kafka-eagle-bin-3.0.1.tar.gz

    cd kafka-eagle-bin-3.0.1

    tar -xvf efak-web-3.0.1-bin.tar.gz

  5. Create a directory in the opt directory, for example, efak, and copy efak-web-3.0.1 to the /opt/efak directory.

    mkdir /opt/efak

    cp -r /opt/kafka-eagle-bin-3.0.1/efak-web-3.0.1 /opt/efak/

  6. Add environment variables.

    vi /etc/profile

    Add the export KE_HOME parameter. The parameter value is the path of the efak-web-3.0.1 file (example value: /opt/efak/efak-web-3.0.1). Add $KE_HOME/bin to the end of the export PATH value. The following is an example:

    export KE_HOME=/opt/efak/efak-web-3.0.1
    export PATH=$PATH:$KE_HOME/bin

  7. Modify the system-config.properties configuration file.

    cd /opt/efak/efak-web-3.0.1/conf/

    vi system-config.properties

    # Configure a cluster.
    kafka.eagle.zk.cluster.alias=cluster1
    cluster1.zk.list=10.20.90.24:2181
    #cluster2.zk.list=xdn10:2181,xdn11:2181,xdn12:2181
    # Modify kafka jmx uri.
    cluster1.efak.jmx.uri=service:jmx:rmi:///jndi/rmi://%s/kafka 
    # Modify the database configuration.
    efak.driver=com.mysql.cj.jdbc.Driver
    efak.url=jdbc:mysql://IP:Port/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
    efak.username=root
    efak.password=XXX
    • The value of cluster1.zk.list is the value of the Kafka component parameter metrics.reporter.zookeeper.url. To obtain the value, you can log in to FusionInsight Manager, choose Cluster > Services > Kafka, click Configurations, and then All Configurations. Search for metrics.reporter.zookeeper.url.
    • The value of efak.url is the string representation of the MySQL JDBC connection.
    • The value of efak.username is the username for connecting to the database.
    • The value of efak.password is the password of the username for connecting to the database.

  8. Start the EFAK service.

    sh /opt/efak/efak-web-3.0.1/bin/ke.sh start

    Obtain the EFAK web UI login address from the command output.

  9. Log in to the EFAK web UI.

    The default initial username for logging in to the EFKA web UI is admin and the password is 123456.

    You can access the Kafka cluster monitoring page, topic monitoring page, and consumer monitoring page.

    Figure 2 Cluster monitoring
    Figure 3 Topic monitoring
    Figure 4 Consumer monitoring

FAQs

Symptom:

The Kafka CPU and memory monitoring information cannot be obtained.
java.io.IOException cannot be cast to javax.management.remote.JMXConnector

Possible Cause

The JMX URI is incorrectly configured. The default JMX URI is as follows:
cluster1.efak.jmx.uri=service:jmx:rmi:///jndi/rmi://%s/jmxrmi

Solution

If the Kafka JMX name in MRS is kafka, set the URI parameter as follows:
cluster1.efak.jmx.uri=service:jmx:rmi:///jndi/rmi://%s/kafka