Updated on 2023-05-31 GMT+08:00

Step 1: Prepare the Environment

VPC

A VPC provides an isolated virtual network for your Kafka instances. You can configure and manage the network as required.

  1. Before creating a Kafka instance, ensure that a VPC and a subnet are available.

    For details, see Creating a VPC. If you already have an available VPC and subnet, you do not need to create new ones.

    Note the following when creating a VPC and subnet:

    • The VPC and the Kafka instance must be in the same region.
    • Use the default settings when creating a VPC and subnet.

  2. Before creating a Kafka instance, ensure that a security group is available.

    For details, see Creating a Security Group. If you already have an available security group, you do not need to create a new one.

    Note the following when creating a security group:

    • Set Template to Custom.
    • To use Kafka instances, add the security group rules described in Table 1. Other rules can be added based on site requirements.
      Table 1 Security group rules

      Direction

      Protocol

      Port

      Source

      Description

      Inbound

      TCP

      9094

      0.0.0.0/0

      Access a Kafka instance through the public network (without SSL encryption).

      Inbound

      TCP

      9092

      0.0.0.0/0

      Access a Kafka instance within a VPC (without SSL encryption).

      Inbound

      TCP

      9095

      0.0.0.0/0

      Access a Kafka instance through the public network (with SSL encryption).

      Inbound

      TCP

      9093

      0.0.0.0/0

      Access a Kafka instance within a VPC (with SSL encryption).

      Inbound

      TCP

      9999

      0.0.0.0/0

      Access Kafka Manager.

      Inbound

      TCP

      9011

      198.19.128.0/17

      Access a Kafka instance across VPCs using a VPC endpoint (with or without SSL).

      Inbound

      TCP

      9011

      0.0.0.0/0

      Access a Kafka instance using DNAT (with or without SSL).

      After a security group is created, it has a default inbound rule that allows communication among ECSs within the security group and a default outbound rule that allows all outbound traffic. If you access your Kafka instance within a VPC, you do not need to add the rules described in Table 1.

(Optional) EIP

To access a Kafka instance over a public network, prepare an elastic IP address (EIP) in advance.

For details, see Assigning an EIP.

Note the following when creating EIPs:

  • The EIPs must be created in the region the Kafka instance is in.
  • The number of EIPs must be the same as the number of Kafka instance brokers.

ECS

Before connecting to a Kafka instance, ensure that you have purchased an ECS, installed the JDK, configured environment variables, and downloaded an open-source Kafka client. The following steps describe how to complete these preparations. A Linux ECS is taken as an example. For more information on how to install JDK and configure the environment variables for a Windows ECS, please search the Internet.

  1. Log in to the management console. In the upper left corner, hover the mouse pointer over . Under Compute, click Elastic Cloud Server, and then create an ECS.

    For details, see Purchasing an ECS. If you already have an available ECS, skip this step.

  2. Log in to the ECS.
  3. Install JDK or JRE, and add the following contents to .bash_profile in the home directory to configure the environment variables JAVA_HOME and PATH. In this command, /opt/java/jdk1.8.0_151 is the JDK installation path. Change it to the path where you install JDK or JRE.

    export JAVA_HOME=/opt/java/jdk1.8.0_151 
    export PATH=$JAVA_HOME/bin:$PATH

    Run the source .bash_profile command for the modification to take effect.

    Use Oracle JDK instead of ECS's default JDK (for example, OpenJDK), because ECS's default JDK may not be suitable. Obtain Oracle JDK 1.8.111 or later from Oracle's official website.

  4. Download an open-source Kafka client.

    If the version of the Kafka instance is 1.1.0, download the client at https://archive.apache.org/dist/kafka/1.1.0/kafka_2.11-1.1.0.tgz.

    wget https://archive.apache.org/dist/kafka/1.1.0/kafka_2.11-1.1.0.tgz

    If the version of the Kafka instance is 2.3.0, download the client at https://archive.apache.org/dist/kafka/2.3.0/kafka_2.11-2.3.0.tgz.

    wget https://archive.apache.org/dist/kafka/2.3.0/kafka_2.11-2.3.0.tgz

    If the version of the Kafka instance is 2.7, download the client at https://archive.apache.org/dist/kafka/2.7.2/kafka_2.12-2.7.2.tgz.

    wget https://archive.apache.org/dist/kafka/2.7.2/kafka_2.12-2.7.2.tgz

  5. Run the following command to decompress the package:

    tar -zxf ${kafka_tar}

    In the preceding command, kafka_tar indicates the name of the client package. For example:

    tar -zxf kafka_2.12-2.7.2.tgz

Follow-Up Procedure

Step 2: Create a Kafka Instance