Help Center> MapReduce Service> Developer Guide (3.x)> Flink Development Guide (Security Mode)> Overview> Application Development
View PDF

Application Development

Overview

Flink is a unified computing framework that supports both batch processing and stream processing. It provides a stream data processing engine that supports data distribution and parallel computing. Flink features stream processing and is a top open-source stream processing engine in the industry.

Flink provides high-concurrency pipeline data processing, millisecond-level latency, and high reliability, making it suitable for low-latency data processing.

Figure 1 shows the technology stack of Flink.

Figure 1 Technology stack of Flink

The following lists the key features of Flink in the current version:

  • DataStream
  • Checkpoint
  • Window
  • Job Pipeline
  • Configuration Table

For details about other Flink features, seehttps://ci.apache.org/projects/flink/flink-docs-release-1.12.

Architecture

Figure 2 shows the architecture of Flink.

Figure 2 Flink architecture

As shown in Figure 2, the entire Flink system consists of three parts:

  • Client

    Flink client is used to submit jobs (streaming jobs) to Flink.

  • TaskManager

    TaskManager (also called worker) is a service execution node of Flink. It executes specific tasks. A Flink system could have multiple TaskManagers. These TaskManagers are equivalent to each other.

  • JobManager

    JobManager (also called master) is a management node of Flink. It manages all TaskManagers and schedules tasks submitted by users to specific TaskManagers. In high-availability (HA) mode, multiple JobManagers are deployed. Among these JobManagers, one of which is selected as the leader, and the others are standby.

Flink provides the following features:

  • Low latency

    Millisecond-level processing capability.

  • Exactly once

    Asynchronous snapshot mechanism, ensuring that all data is processed only once.

  • High availability

    Leader/Standby JobManagers, preventing single point of failure (SPOF).

  • Scale out

    Manual scale out supported by TaskManagers.

Flink Development APIs

Flink DataStream API can be developed using Scala and Java languages, as shown in Table 1.

Table 1 Flink DataStream API

Function

Description

Scala API

API in Scala, which can be used for data processing, such as filtering, joining, windowing, and aggregation.

Java API

API in Java, which can be used for data processing, such as filtering, joining, windowing, and aggregation.
Parent topic: Overview