Scenarios
Scenarios
Assume that a Flink service receives one word record every 1 second.
Develop a Flink application that can generate output of prefixed message contents.
Data Planning
Sample project data of Flink is stored in Kafka. A user with Kafka permission can send data to Kafka and receive data from it.
- Ensure that clusters, including HDFS, YARN, Flink, and Kafka are successfully installed.
- Create a topic.
The format of the command is following:
bin/kafka-topics.sh --create --zookeeper {zkQuorum}/kafka --partitions {partitionNum} --replication-factor {replicationNum} --topic {Topic}
Table 1 parameter
Description
{zkQuorum}
ZooKeeper cluster information. The format is IP:port.
{PartitionNum}
The number of partitions for the topic.
{ReplicationNum}
The number of copies of each partition for the topic.
{Topic}
The topic name.
Assume that the IP:ports of ZooKeeper clusters are 10.96.101.32:2181, 10.96.101.251:2181, 10.96.101.177:2181, and 10.91.8.160:2181, and the topic named is topic1. The command for creating a topic is as follows:bin/kafka-topics.sh --create --zookeeper 10.96.101.32:2181,10.96.101.251:2181,10.96.101.177:2181,10.91.8.160:2181/kafka --partitions 5 --replication-factor 1 --topic topic1
Development Approach
- Start the Flink Kafka Producer to send data to Kafka.
- Start Flink Kafka Consumer to receive data from Kafka. Ensure that topics of Kafka Consumer are consistent with that of Kafka Producer.
- Add prefix to the data content and print the result.
Last Article: Interconnecting with Kafka
Next Article: Java Sample Code
Did this article solve your problem?
Thank you for your score！Your feedback would help us improve the website.