Help Center > > User Guide> MRS Cluster Component Operation Guide> Using Flink> Introduction to Logs

Introduction to Logs

Updated at: Mar 31, 2020 GMT+08:00

Log Description

Log paths:
  • Executor run log: "${BIGDATA_DATA_HOME}/hadoop/data${i}/nm/containerlogs/application_${appid}/container_{$contid}"

    The logs of running tasks are stored in the preceding path. After the running is complete, the system determines whether to aggregate the logs to an HDFS directory based on the Yarn configuration.

  • Other logs: "/var/log/Bigdata/flink/flinkResource"

Log archive rules:

  • Executor log files are stored each time when the size of the log files reaches 50 MB by default. A maximum of 100 log files are retained without being compressed.
  • You can configure the log file size and the number of compressed files to be retained on MRS Manager.
Table 1 Flink log list

Log Type

Log File Name

Description

Flink service log

postinstall.log

Service installation log

prestart.log

Log file of the prestart script

cleanup.log

Cleanup log file for instance installation and uninstallation

Log Levels

Table 2 describes the log levels supported by Flink. The priorities of log levels are ERROR, WARN, INFO, and DEBUG in descending order. Logs whose levels are higher than or equal to the specified level are printed. The number of printed logs decreases as the specified log level increases.

Table 2 Log levels

Log Level

Description

ERROR

Logs of this level record error information about the current event processing.

WARN

Logs of this level record error information about the current event processing.

INFO

Logs of this level record normal running status information about the system and events.

DEBUG

Logs of this level record the system information and system debugging information.

To modify log levels, perform the following operations:

  1. Go to the cluster details page and choose Components > Flink > Service Configuration.

    For MRS 1.8.10 or earlier, log in to MRS Manager. For details, see Accessing MRS Manager. Then, choose Services > Flink > Service Configuration.

  2. Set Type to All.
  3. On the menu bar on the left, select the log menu of the target role.
  4. Select a desired log level.
  5. Click Save Configuration. In the displayed dialog box, click OK to make configurations take effect.

    The configurations take effect immediately without the need of restarting the service.

Log Format

Table 3 Log format

Log Type

Format

Example

Run log

<yyyy-MM-dd HH:mm:ss,SSS>|<Log Level>|<Name of the thread that generates the log>|<Message in the log>|<Location of the log event>

2019-06-27 21:30:31,778 | INFO | [flink-akka.actor.default-dispatcher-3] | TaskManager container_e10_1498290698388_0004_02_000007 has started. | org.apache.flink.yarn.YarnFlinkResourceManager (FlinkResourceManager.java:368)

Did you find this page helpful?

Submit successfully!

Thank you for your feedback. Your feedback helps make our documentation better.

Failed to submit the feedback. Please try again later.

Which of the following issues have you encountered?







Please complete at least one feedback item.

Content most length 200 character

Content is empty.

OK Cancel