Updated on 2024-04-02 GMT+08:00

Development Process

Kafka client roles include Producer and Consumer, which share the same application development process.

Figure 1 and Table 1 show each phase of the development process.

Figure 1 Kafka client application development process
Table 1 Kafka client application development process description

Phase

Description

Reference Document

Prepare the development and operating environment.

The Java language is recommended for the development of Java applications. The IntelliJ IDEA tool can be used.

The Kafka running environment is the Kafka client. Install and configure the client according to the guide.

Preparing for Development and Operating Environment

Preparing the configuration files for connecting to the cluster

During the development or a test run of the program, you need to use the cluster configuration files to connect to an MRS cluster. The configuration files usually contain the cluster component information file and user files used for security authentication. You can obtain the required information from the created MRS cluster.

Nodes used for program debugging or running must be able to communicate with the nodes within the MRS cluster, and the hosts domain name must be configured.

Preparing the Configuration Files for Connecting to the Cluster

Configuring and importing sample projects

Kafka provides sample projects in various scenarios. sample projects can be imported for studying.

Configuring and Importing a Sample Project

Configuring security authentication

If you are using an MRS cluster with Kerberos authentication enabled, security authentication is required.

Preparing for Security Authentication

Writing program code for a service scenario

Producer and Consumer API usage samples are provided and old APIs, new APIs, and multi-thread usage scenarios are covered, helping you quickly know Kafka APIs well.

Compile and run the program. You can debug and run the program in the local Windows development environment, or compile the program into a JAR package and submit it to a Linux node.

Developing an Application

Compiling and running the program

This phase provides guidance for users to submit and run a developed program and then view the result.

Application Commissioning