Yarn Log Overview

Log Description

The default paths for saving Yarn logs are as follows:

ResourceManager: /var/log/Bigdata/yarn/rm (run logs) and /var/log/Bigdata/audit/yarn/rm (audit logs)
NodeManager: /var/log/Bigdata/yarn/nm (run logs) and /var/log/Bigdata/audit/yarn/nm (audit logs)
TimelineServer: /var/log/Bigdata/yarn/tls (run logs) and /var/log/Bigdata/audit/yarn/tls (audit logs)

Log archive rule: The automatic compression and archive function is enabled for Yarn logs. By default, when the size of a log file exceeds 50 MB, the log file is automatically compressed. The naming rule of the compressed log file is as follows: <Original log file name>-<yyyy-mm-dd_hh-mm-ss>.[ID].log.zip. A maximum of 100 latest compressed files are retained. The number of compressed files can be configured on Manager.

Log archive rule:

**Table 1** Yarn log list
Log Type	Log File Name	Description
Run log	hadoop-<SSH_USER>-<process_name>-<hostname>.log	Yarn component log file, which records most of the logs generated when the Yarn component is running
	hadoop-<SSH_USER>-<process_name>-<hostname>.out	Log file that records Yarn running environment information
	<process_name>-<SSH_USER>-<DATE>-<PID>-gc.log	Garbage collection log file
	yarn-haCheck.log	ResourceManager active/standby status detection log file
	yarn-service-check.log	Log file that records the health check details of the Yarn service
	yarn-start-stop.log	Log file that records the startup and stop of the Yarn service
	yarn-prestart.log	Log file that records cluster operations before the Yarn service startup
	yarn-postinstall.log	Work log file after installation and before startup of the Yarn service
	hadoop-commission.log	Yarn service entry log file
	yarn-cleanup.log	Log file that records the cleanup operation during uninstallation of the Yarn service
	yarn-refreshqueue.log	Yarn queue refresh log file
	upgradeDetail.log	Upgrade log file
	stderr/stdin/syslog	Container log file of the applications running on the Yarn service
	yarn-application-check.log	Check log file of applications running on the Yarn service
	yarn-appsummary.log	Running result log file of applications running on the Yarn service
	yarn-switch-resourcemanager.log	Run log file that records the Yarn active/standby switchover
	yarn-az-state.log	AZ status log of Yarn
	yarn-az-disaster-exercise.log	AZ DR drill log of Yarn
	yarn-az-check.log	Yarn AZ check log file
	ranger-yarn-plugin-enable.log	Log file that records the enabling of Ranger authentication for Yarn
	yarn-nodemanager-period-check.log	Periodic check log of Yarn NodeManager
	yarn-resourcemanager-period-check.log	Periodic check log of Yarn ResourceManager
	hadoop.log	Hadoop client logs
	env.log	Environment information log file before the instance is started or stopped.
	tls-daemon-start-stop.log	YARN daemon startup and stop log
	tls-leveldb-sync.log	Log that records LevelDB synchronization between the active and standby TimelineServer nodes
	threadDump-<process_name>-<thread pid>-<timestamp>.log	Dump log generated when YARN is stopped
Audit logs	yarn-audit-<process_name>.log ranger-plugin-audit.log	Yarn operation audit log file
Audit logs	SecurityAuth.audit	Yarn security audit log file

Log Level

Table 2 describes the log levels supported by Yarn, including OFF, FATAL, ERROR, WARN, INFO, and DEBUG, from high priority to low. Logs whose levels are higher than or equal to the specified level are printed. The number of printed logs decreases as the specified log level increases.

**Table 2** Log levels
Level	Description
FATAL	Logs of this level record critical error information about the current event processing.
ERROR	Logs of this level record error information about the current event processing.
WARN	Logs of this level record exception information about the current event processing.
INFO	Logs of this level record normal running status information about the system and events.
DEBUG	Logs of this level record the system as well as system debugging information.

To modify log levels, perform the following operations:

Go to the All Configurations page of the Yarn service by referring to Modifying Cluster Service Configuration Parameters.
On the menu bar on the left, select the log menu of the target role.
Select a desired log level.
Click Save Configuration. In the dialog box that is displayed, click OK to make the setting take effect.

The configurations take effect immediately without the need to restart the service.

Log Format

The following table lists the Yarn log formats.

**Table 3** Log formats
Log Type	Format	Example
Run log	<yyyy-MM-dd HH:mm:ss,SSS>\|<Log Level>\|<Thread that generates the log>\|<Message in the log>\|<Location of the log event>	2014-09-26 14:18:59,109 \| INFO \| main \| Client environment:java.compiler=<NA> \| org.apache.zookeeper.Environment.logEnv(Environment.java:100)
Audit log	<yyyy-MM-dd HH:mm:ss,SSS>\|<Log Level>\|<Thread that generates the log>\|<Message in the log>\|<Location of the log event>	2014-09-26 14:24:43,605 \| INFO \| main-EventThread \| USER=omm OPERATION=refreshAdminAcls TARGET=AdminService RESULT=SUCCESS \| org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger$LogLevel$6.printLog(RMAuditLogger.java:91)