Relationship Between Flink and Other Components

Relationship Between Flink and YARN

Flink supports YARN-based cluster management mode. In this mode, Flink serves as an application of YARN and runs on YARN.

Figure 1 shows the YARN-based Flink cluster deployment.

Figure 1 YARN-based Flink cluster deployment

1. The Flink YARN Client first checks whether there are sufficient resources for starting the YARN cluster. If yes, the Flink YARN client uploads JAR files and configuration files to HDFS.
2. Flink YARN client communicates with YARN ResourceManager to request a container for starting ApplicationMaster. After all YARN NodeManagers finish downloading the JAR file and configuration files, the ApplicationMaster is started.
3. During the startup, the ApplicationMaster interacts with the YARN ResourceManager to request the container for starting a TaskManager. After the container is ready, the TaskManager process is started.
4. In the Flink YARN cluster, the ApplicationMaster and Flink JobManager are running in the same container. The ApplicationMaster informs each TaskManager of the RPC address of the JobManager. After TaskManagers are started, they register with the JobManager.
5. After all TaskManagers has registered with the JobManager, Flink starts up in the YARN cluster. Then, the Flink YARN client can submit Flink jobs to the JobManager, and Flink can perform mapping, scheduling, and computing for the jobs.