Changing Spark Log Levels

Scenarios

In some scenarios, to locate problems or check information by changing the log level,

you can add the -Dlog4j.configuration.watch=true parameter to the JVM parameter of a process before the process is started. After the process is started, you can modify the log4j configuration file corresponding to the process to change the log level.

The following processes support the dynamic setting of log levels: driver, executor, ApplicationMaster, JobHistory and JDBCServer.

Allowed log levels are as follows: FATAL, ERROR, WARN, INFO, DEBUG, TRACE, and ALL.

Configuration Description

Add the following parameters to the JVM parameter corresponding to a process.

**Table 1** Parameter description
Parameter	Description	Default Value
-Dlog4j.configuration.watch	Indicates a JVM parameter of a process. If this parameter is set to true, the dynamic configuration of log levels is enabled.	Left blank, indicating that the dynamic configuration of log levels is disabled

Table 2 lists the JVM parameters of the driver, executor, and ApplicationMaster processes. Configure the following parameters in the spark-defaults.conf file on the Spark client. Set the log levels of the driver, executor, and ApplicationMaster processes in the log4j configuration file specified by the -Dlog4j.configuration parameter.

**Table 2** JVM parameters of processes (1)
Parameter	Description	Default Log Level
spark.driver.extraJavaOptions	Indicates the JVM parameter of the driver process.	INFO
spark.executor.extraJavaOptions	Indicates the JVM parameter of the executor process.	INFO
spark.yarn.am.extraJavaOptions	Indicates the JVM parameter of the ApplicationMaster process.	INFO

Table 3 describes the JVM parameters of JobHistory Server and JDBCServer. Set the parameters in the ENV_VARS configuration file. Set the log levels of JobHistory Server and JDBCServer in the log4j.properties configuration file.

**Table 3** JVM parameters of processes (2)
Parameter	Description	Default Log Level
GC_OPTS	Indicates the JVM parameter of the JobHistory Server process.	INFO
SPARK_SUBMIT_OPTS	Indicates the JVM parameter of JDBCServer.	INFO

Example:

To change the log level of the executor process to DEBUG dynamically, modify the spark.executor.extraJavaOptions JVM parameter of the executor process in the spark-defaults.conf file and run the following command to add the following configuration before the process is started:

-Dlog4j.configuration.watch=true

After the user application is submitted, change the log level in the Log4j configuration file (for example, -Dlog4j.configuration=file:${BIGDATA_HOME}/FusionInsight_Spark2x_8.1.0.1/install/FusionInsight-Spark2x-*/spark/conf/log4j-executor.properties) specified by the -Dlog4j.configuration parameter in spark.executor.extraJavaOptions to DEBUG: