Failed to Customize the Flink Log4j Log Level

Issue

The customized level for Flink Log4j logs of an MRS 3.1.0 cluster does not take effect.

Symptom

1. When analyzing data using Flink of an MRS 3.1.0 cluster, a user changes the log level in the log4j.properties file in the $Flink_HOME/conf directory to INFO.
2. However, after the task is submitted successfully, the log level displayed on the console is still ERROR, rather than INFO.

Cause Analysis

The log4j.properties file in the $Flink_HOME/conf directory controls the log output of in JobManager and TaskManager operators, and the logs are printed to the corresponding Yarn containers. You can view the logs on the Yarn web UI.

In MRS 3.1.0 and later versions, the default log framework of Flink 1.12.0 is Log4j2. The configuration method is different from that of Log4j. For example, Log4j log rules do not take effect.

Procedure

For details about configuring Log4j2 log specifications, see the official open-source document at http://logging.apache.org/log4j/2.x/manual/configuration.html#Properties.