Installing Kafka

Introduction

Kafka is distributed pub/sub messaging middleware with high throughput, data persistence, horizontal scalability, and stream data processing. Common applications include log collection, data streaming, online/offline system analytics, and real-time monitoring.

This tutorial describes how you can deploy Kafka in HCE 2.0.

Preparations

• Prepare an ECS and assign a public IP address or EIP to the ECS.

Procedure

1. Install Kafka.

   Run the following command to install Kafka:

   dnf install kafka

   After the command is executed, Kafka is installed in the /opt/kafka directory.

2. Configure Kafka.
   1. Open the /opt/kafka/config/server.properties file and modify the following attributes:

      listeners=PLAINTEXT://<Private IP address>:9092
      advertised.listeners=PLAINTEXT://<Public IP address>:9092

   2. Create the /lib/systemd/system/zookeeper.service file and enter the following information:

      [Unit]
      Description=Zookeeper service
      After=network.target
      
      [Service]
      Type=simple
      Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
      User=root
      Group=root
      ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
      ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh
      Restart=on-failure
      SuccessExitStatus=143
      
      [Install]
      WantedBy=multi-user.target

   3. Create the /lib/systemd/system/kafka.service file and enter the following information:

      [Unit]
      Description=Apache Kafka server (broker)
      After=network.target  zookeeper.service
      
      [Service]
      Type=simple
      Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
      User=root
      Group=root
      ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
      ExecStop=/opt/kafka/bin/kafka-server-stop.sh
      Restart=on-failure
      SuccessExitStatus=143
      
      [Install]
      WantedBy=multi-user.target

   4. Run the following commands in sequence to start Kafka and ZooKeeper:

      systemctl daemon-reload
      systemctl start zookeeper
      systemctl start kafka

3. Verify Kafka.
   1. Run the following command to create a topic:

       /opt/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

   2. Run the following command to view the created topic:

      /opt/kafka/bin/kafka-topics.sh --list --zookeeper localhost:2181

      If test is displayed, Kafka is deployed.