Help Center/ Log Tank Service/ User Guide/ Log Ingestion/ Self-Built Middleware (Beta)/ Flink Logs
Updated on 2025-04-03 GMT+08:00
View PDF
Share

Flink Logs

Flink, developed by Apache, is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink features high throughput, low latency (millions of events processed per second and millisecond-level latency), and result accuracy (by defining event-time and processing-time).

Flink system logs contain various information generated during the Flink running, such as task execution status, error information, and resource usage.

Collection Method

Install ICAgent and configure ECS text log ingestion by referring to Installing ICAgent (Intra-Region Hosts) and Ingesting ECS Text Logs to LTS. Then, ICAgent will collect ECS text logs and report them to LTS.

Flink Log Examples and Fields

System log example

2023-10-23 14:35:00,000 INFO  org.apache.flink.runtime.taskmanager.Task  - Source: Custom Source -> Map (1/1) (83529a6b73e4a3f7046f695a27f67f32) switched from RUNNING to FINISHED.  
2023-10-23 14:35:01,500 WARN  org.apache.flink.runtime.checkpoint.CheckpointCoordinator  - Checkpoint 1 for job 83529a6b73e4a3f7046f695a27f67f32 timed out after 60000 milliseconds. This indicates a problem with the job, as the checkpoints were not completed in time.  
2023-10-23 14:35:05,000 ERROR org.apache.flink.runtime.executiongraph.ExecutionGraph  - Job 83529a6b73e4a3f7046f695a27f67f32 failed.
Table 1 Flink system log fields

Field

Description

Example

Timestamp

Time when the log was generated.

2023-10-23 14:35:00,000

Log level

Log importance or severity, such as INFO, WARN, or ERROR.

INFO

Log source

Component or task that generated the log.

org.apache.flink.runtime.taskmanager.Task

Task/Job information

Logs may contain task or job information, such as the task name, task ID, and job ID. In this example, the task name is Source: Custom Source -> Map (1/1) and the job ID is 83529a6b73e4a3f7046f695a27f67f32.

-

Status change

Task/Job status change.

For example, switched from RUNNING to FINISHED indicates that the task is switched from the running state to the completed state.

Error message

If the log level is ERROR, the log may contain details about the error.

For example, "Job 83529a6b73e4a3f7046f695a27f67f32 failed." indicates that the job failed.

Log Location

  1. Flink jobs running on YARN:

    Logs of jobs in Flink-on-YARN mode are usually stored in **$HADOOP_HOME/logs/userlogs/<APPLICATION_ID>/<CONTAINER_ID>/**. $HADOOP_HOME indicates the Hadoop installation directory, and <APPLICATION_ID> and <CONTAINER_ID> indicate identifiers assigned by YARN to Flink jobs. You can modify the YARN log directory by setting <property><name>yarn.nodemanager.log-dirs</name><value>/your/custom/log/path</value></property> in the yarn-site.xml configuration file.

  2. Independently deployed Flink clusters: If a Flink cluster is deployed in independent mode, log files are stored in the log or logs folder in the Flink installation directory. For example, if Flink is installed in /usr/local/flink, log files may be located in /usr/local/flink/log or /usr/local/flink/logs.