Help Center/ MapReduce Service/ Getting Started/ Using Kafka from Scratch
Updated on 2022-09-14 GMT+08:00
View PDF

Using Kafka from Scratch

MapReduce Service (MRS) provides Hadoop-based high-performance big data components, such as Spark, HBase, Kafka, and Storm.

This section uses a cluster with Kerberos authentication disabled as an example to describe how to generate and consume messages in a Kafka topic.

You can get started by reading the following steps:
  1. Purchasing a Cluster
  2. Installing the Kafka Client
  3. Logging In to a Master Node Using VNC
  4. Creating a Topic Using the Kafka Client
  5. Managing Messages in Kafka Topics

Purchasing a Cluster

  1. Purchase an MRS cluster.

    1. Log in to the Huawei Cloud management console.
    2. Choose Service List > Analytics > MapReduce Service.

    3. Choose Clusters > Active Clusters. On the Active Clusters page that is displayed, click Buy Cluster. On the displayed page, click the Custom Config tab.

  2. Configure the following parameters and click Next.

    • Region: Select a region as required.
    • Cluster Name: Enter mrs_demo or specify a name according to naming rules.
    • Cluster Version: Select MRS 3.1.0.

    • Cluster Type: Select Streaming Cluster, select all streaming cluster components, and click Next.

  3. On the Configure Hardware page, configure the parameters by referring to Table 1, and click Next.

    Table 1 MRS hardware configuration parameters

    Parameter

    Example Value

    Billing Mode

    Pay-per-use

    AZ

    AZ2

    VPC

    Retain the default value. You can also click View VPC to create one.

    EIP

    You can select an existing EIP from the drop-down list. If no EIPs are available, click Manage EIP to access the EIPs page to create one.

    Enterprise Project

    default

    Cluster Node

    Retain the default values. Do not add task nodes.
    Figure 1 Configure Hardware

  4. Configure advanced options.

    1. On the Set Advanced Options page, configure the parameters by referring to Table 2 and retain the default values for other parameters.
      Table 2 MRS cluster advanced parameters

      Parameter

      Example Value

      Kerberos Authentication

      Toggle the slider off.

      Username

      admin

      Password

      Configure the password for logging in to the cluster management page, for example, Test!@12345.

      Confirm Password

      Enter the password again.

      Login Mode

      Password

      Username

      root

      Password

      Configure a password for remotely logging in to ECSs or BMSs, for example, Test!@12345.

      Confirm Password

      Enter the password again.

      Secure Communications

      Select Enable.
      Figure 2 Set Advanced Options
    2. Click Buy Now. The page is displayed showing that the task has been submitted.
    3. Click Back to Cluster List. You can view the status of the cluster on the Active Clusters page.
    4. Wait for the cluster creation to complete. The initial status of the cluster is Starting. After the cluster has been created, the cluster status becomes Running.

Installing the Kafka Client

  1. Choose Clusters > Active Clusters. On the Active Clusters page, click the cluster named mrs_demo to go to its details page.
  2. Click Access Manager next to MRS Manager. On the page that is displayed, configure the EIP information and click OK. Enter the username and password to access FusionInsight Manager.

  3. Choose Cluster > Services > HBase. On the page displayed, choose More > Download Client. In the Download Cluster Client dialog box, select Complete Client for Select Client Type, select a platform type, select Save to Path, and click OK. The Kafka client software package, for example, FusionInsight_Cluster_1_Kafka_Client.tar, is downloaded.

  4. Log in to the active node as user root.
  5. Go to the directory where the software package is stored and run the following commands to decompress and verify the software package, and decompress the obtained installation file:

    cd /tmp/FusionInsight-Client

    tar -xvf FusionInsight_Cluster_1_Kafka_Client.tar

    sha256sum -c FusionInsight_Cluster_1_Kafka_ClientConfig.tar.sha256

    tar -xvf FusionInsight_Cluster_1_Kafka_ClientConfig.tar

  6. Go to the directory where the installation package is stored, and run the following command to install the client to a specified directory (absolute path), for example, /opt/hadoopclient:

    cd /tmp/FusionInsight-Client/FusionInsight_Cluster_1_Kafka_ClientConfig

    Run the ./install.sh /opt/hadoopclient command and wait until the client installation is complete.

  7. Check whether the client is installed.

    cd /opt/hadoopclient

    source bigdata_env

    Run the klist command to query and confirm authentication details. If the command is executed, the Kafka client is installed.

Logging In to a Master Node Using VNC

  1. Choose Clusters > Active Clusters. On the Active Clusters page that is displayed, click the cluster named mrs_demo. On the cluster details page that is displayed, click the Nodes tab. On this tab page, locate the node whose type is Master1 and click the node name to go to the ECS details page.

  2. Click Remote Login in the upper right corner of the page to remotely log in to the master node. Log in using the username root and the password configured during cluster purchase.

Creating a Topic Using the Kafka Client

  1. Configure environment variables. For example, if the Kafka client installation directory is /opt/hadoopclient, run the following command:

    source /opt/hadoopclient/bigdata_env

  2. Choose Clusters > Active Clusters. On the Active Clusters page, click the cluster named mrs_demo to go to the Dashboard tab page. On this page, click Synchronize next to IAM User Sync.
  3. After the synchronization is complete, click the Components tab. On this tab page, select ZooKeeper. On the page that is displayed, click the Instances tab. Record the IP address of any ZooKeeper instance, for example, 192.168.7.35.

  4. Run the following command to create a Kafka topic:

    kafka-topics.sh --create --zookeeper <IP address of the node where the ZooKeeper instance resides:2181/kafka> --partitions 2 --replication-factor 2 --topic <Topic name>

Managing Messages in Kafka Topics

  1. Click the Components tab. On this tab page, select Kafka. On the page that is displayed, click the Instances tab. On the Instances tab page, view the IP addresses of Kafka instances. Record the IP address of any Kafka instance, for example, 192.168.7.15.

  2. Log in to the master node and run the following command to generate messages in a topic test:

    kafka-console-producer.sh --broker-list <IP address of the node where the Kafka instance resides:9092> --topic <Topic name> --producer.config /opt/hadoopclient/Kafka/kafka/config/producer.properties

    Enter the specified content as the messages generated by the producer and press Enter to send the messages. To stop generating messages, press Ctrl+C to exit.

  3. Consume messages in the topic test.

    kafka-console-consumer.sh --topic <Topic name> --bootstrap-server <IP address of the node where the Kafka instance resides:9092> --consumer.config /opt/hadoopclient/Kafka/kafka/config/consumer.properties