Help Center> Cloud Stream Service> Eco Development> Cloud Ecosystem: MRS> Interaction with Kafka Using User-Defined Jobs
View PDF

Interaction with Kafka Using User-Defined Jobs

Overview

You can perform secondary development based on Flink and Spark APIs to build your own JAR packages and submit them to the CS cluster to implement interactions with CS and MRS Kafka clusters.

Apache Kafka is a fast, scalable, and fault-tolerant distributed message publishing and subscription system. It delivers high throughput and built-in partitions and provides data replicas and fault tolerance. Apache Kafka is applicable to scenarios of handling massive messages. Kafka clusters are deployed and hosted on MRS that is powered on Apache Kafka.

Prerequisites

  • To use Kafka in an MRS cluster, you need to use the VPC peering connection to interconnect CS with the MRS cluster.

    For details about how to set up the VPC peering connection, see VPC Peering Connection in the Cloud Stream Service User Guide.

  • If the Kafka server listens on the port using hostname, you need to add the mapping between the hostname and IP address of the Kafka Broker node to the CS cluster. Contact the Kafka service deployment personnel to obtain the hostname and IP address of the Kafka Broker node. For details about how to add an IP-domain mapping, see the description of Adding an IP-Domain Mapping in Cluster Management in the Cloud Stream Service User Guide.

Procedure

Create and submit a user-defined Flink job. For details, see Creating a User-Defined Flink Job in the Cloud Stream Service User Guide.

Create and submit a user-defined Spark job. For details, see Creating a User-Defined Spark Job in the Cloud Stream Service User Guide.

Parent topic: Cloud Ecosystem: MRS