Flink Log Overview

Updated on 2024-11-29 GMT+08:00

View PDF

Log Description

Log path:

Run logs of a Flink job: ${BIGDATA_DATA_HOME}/hadoop/data${i}/nm/containerlogs/application_${appid}/container_{$contid}
NOTE:

The logs of executing tasks are stored in the preceding path. After the execution is complete, the Yarn configuration determines whether these logs are gathered to the HDFS directory.
FlinkResource run logs: /var/log/Bigdata/flink/flinkResource
FlinkServer run logs: /var/log/Bigdata/flink
FlinkServer audit logs: /var/log/Bigdata/audit/flink/flinkserver
Run logs related to FlinkServer HA scripts: /var/log/Bigdata/audit/flink/flinkserver/ha

Log archive rules:

FlinkResource run logs:

By default, service logs are backed up each time when the log size reaches 20 MB. A maximum of 20 logs can be reserved without being compressed.
You can set the log size and number of compressed logs on the Manager page or modify the corresponding configuration items in log4j-cli.properties, log4j.properties, and log4j-session.properties in Client installation directory/Flink/flink/conf/ on the client.

**Table 1** FlinkResource log list
Type	Name	Description
FlinkResource run logs	checkService.log	Health check log
	kinit.log	Initialization log
	postinstall.log	Service installation log
	prestart.log	Prestart script log
	start.log	Startup log

FlinkServer service logs, HA-related logs, and audit logs.

By default, FlinkServer service logs, HA-related logs, and audit logs are backed up each time when the log size reaches 100 MB. The service logs are stored for a maximum of 30 days, and audit logs are stored for a maximum of 90 days.
You can set the log size and number of compressed logs on the Manager page or modify the corresponding configuration items in log4j-cli.properties, log4j.properties, and log4j-session.properties in Client installation directory/Flink/flink/conf/ on the client.

**Table 2** FlinkServer log list
Type	Name	Description
FlinkServer run logs	checkService.log	Health check log
	checkFlinkServer.log	Health check log of FlinkServer
	localhost_access_log..yyyy-mm-dd.txt	URL log of FlinkServer
	start_thrift_server.out	Thrift server startup log
	thrift_server_thriftServer_xxx.log.last	Thrift server startup log
	cleanup.log	Cleanup log file for instance installation and uninstallation
	flink-omm-client-IP.log	Job startup log
	flinkserver_yyyymmdd-x.log.gz	Service archive log
	flinkserver.log	Service log
	flinkserver---pidxxxx-gc.log.x.current	GC log
	kinit.log	Initialization log
	postinstall.log	Service installation log
	prestart.log	Prestart script log
	start.log	Startup log
	stop.log	Stop log
	catalina.yyyy-mm-dd.log	Tomcat run log
	catalina.out
	host-manager.yyyy-mm-dd.log
	localhost.yyyy-mm-dd.log
	manager.yyyy-mm-dd.log
Run log file related to FlinkServer HA scripts	ha.log	HA run log
	ha_monitor.log	HA process monitoring log
	floatip_ha.log	Floating IP address resource script log
	rcommflinkserver.log	FlinkServer resource script log
	checkHaStatus.log	HA process log
	checknode.log	HA health status log
	rs-sendAlarm.log	HA alarm sending log
	flink_roll.log	FlinkServer active/standby switchover log (active/standby switchover required)
FlinkServer audit logs	flinkserver_audit_yyyymmdd-x.log.gz	Audit archive log
FlinkServer audit logs	flinkserver_audit.log	Audit log
Stack information log	threadDump-<DATE>.log	Log printed when instances are restarted or stopped

Log Level

Table 3 describes the log levels supported by Flink. The priorities of log levels are ERROR, WARN, INFO, and DEBUG in descending order. Logs whose levels are higher than or equal to the specified level are printed. The number of printed logs decreases as the specified log level increases.

**Table 3** Log levels
Level	Description
ERROR	Error information about the current event processing
WARN	Exception information about the current event processing
INFO	Normal running status information about the system and events
DEBUG	System information and system debugging information

To modify log levels, perform the following steps:

Go to the All Configurations page of Flink by referring to Modifying Cluster Service Configuration Parameters.
On the menu bar on the left, select the log menu of the target role.
Select a desired log level.
Save the configuration. In the displayed dialog box, click OK to make the configurations take effect.

NOTE:

After the configuration is complete, you do not need to restart the service. Download the client again for the configuration to take effect.
You can also change the configuration items corresponding to the log level in log4j-cli.properties, log4j.properties, and log4j-session.properties in Client installation directory/Flink/flink/conf/ on the client.
When a job is submitted using a client, a log file is generated in the log folder on the client. The default umask value is 0022. Therefore, the default log permission is 644. To change the file permission, you need to change the umask value. For example, to change the umask value of user omm:
- Add umask 0026 to the end of the /home/omm/.baskrc file.
- Run the source /home/omm/.baskrc command to make the file permission take effect.

Log Format

**Table 4** Log formats
Type	Format	Example
Run log	<yyyy-MM-dd HH:mm:ss,SSS>\|<Log level>\|<Name of the thread that generates the log>\|<Message in the log>\|<Location where the log event occurs>	2019-06-27 21:30:31,778 \| INFO \| [flink-akka.actor.default-dispatcher-3] \| TaskManager container_e10_1498290698388_0004_02_000007 has started. \| org.apache.flink.yarn.YarnFlinkResourceManager (FlinkResourceManager.java:368)

Parent topic: Using Flink

Previous topic: Interconnecting FlinkServer with Redis

Next topic: Flink Performance Tuning

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot