Help Center/ MapReduce Service/ Component Operation Guide (Normal)/ Using Storm/ Storm Log Introduction
Updated on 2022-09-15 GMT+08:00
View PDF

Storm Log Introduction

This section applies to MRS 3.x or later.

Log Description

Log paths: The default paths of Storm log files are /var/log/Bigdata/storm/Role name (run logs) and /var/log/Bigdata/audit/storm/Role name (audit logs).

  • Nimbus: /var/log/Bigdata/storm/nimbus (run logs) and /var/log/Bigdata/audit/storm/nimbus (audit logs)
  • Supervisor: /var/log/Bigdata/storm/supervisor (run logs) and /var/log/Bigdata/audit/storm/supervisor (audit logs)
  • UI: /var/log/Bigdata/storm/ui (run logs) and /var/log/Bigdata/audit/storm/ui (audit logs)
  • Logviewer: /var/log/Bigdata/storm/logviewer (run logs) and /var/log/Bigdata/audit/storm/logviewer (audit logs)

Log archive rule: The automatic Storm log compression function is enabled. By default, when the size of logs exceeds 10 MB, logs are automatically compressed into a log file named in the following format: <Original log name>.log.[ID].gz. A maximum of 20 latest compressed files are reserved by default. You can configure the number of compressed files and the compression threshold.

Names of compressed audit log files are in the format of audit.log.[yyyy-MM-dd].[ID].zip. These files permanently exist.

Table 1 Storm log list

Log Type

Log File Name

Description

Run log

nimbus/access.log

Nimbus user access log

nimbus/nimbus-<PID>-gc.log

GC log of the Nimbus process

nimbus/checkavailable.log

Nimbus availability check log

nimbus/checkService.log

Nimbus serviceability check log

nimbus/metrics.log

Nimbus monitoring statistics log

nimbus/nimbus.log

Run log of the Nimbus process

nimbus/postinstall.log

Work log after Nimbus installation

nimbus/prestart.log

Work log before Nimbus startup

nimbus/start.log

Work log of Nimbus startup

nimbus/stop.log

Work log of Nimbus shutdown

supervisor/access.log

Supervisor access log

supervisor/metrics.log

Supervisor monitoring statistics log

supervisor/postinstall.log

Work log after supervisor installation

supervisor/prestart.log

Work log before supervisor startup

supervisor/start.log

Work log of supervisor startup

supervisor/stop.log

Work log of supervisor shutdown

supervisor/supervisor.log

Run log of the supervisor process

supervisor/supervisor-<PID>-gc.log

GC log of the supervisor process

ui/access.log

UI access log

ui/metric.log

UI monitoring statistics log

ui/ui-<PID>-gc.log

GC log of the UI process

ui/postinstall.log

Work log after UI installation

ui/prestart.log

Work log before UI startup

ui/start.log

Work log of UI startup

ui/stop.log

Work log of UI shutdown

ui/ui.log

Run log of the UI process

logviewer/access.log

Logviewer access log

logviewer/metric.log

Logviewer monitoring statistics log

logviewer/logviewer-<PID>-gc.log

GC log file of the logviewer process

logviewer/logviewer.log

Run log of the logviewer process

logviewer/postinstall.log

Work log after logviewer installation

logviewer/prestart.log

Work log before logviewer startup

logviewer/start.log

Work log of logviewer startup

logviewer/stop.log

Work log of logviewer shutdown

supervisor/[topologyId]-worker-[Port number].log

Run log of the Worker process. One port occupies one log file. By default, the system contains five ports: 29100, 29101, 29102, 29103 and 29304.

supervisor/metadata/[topologyid]-worker-[Port number].yaml

Worker log metadata file, which is used by logviewer to delete logs. This file is automatically deleted by the logviewer log deletion thread based on certain conditions.

nimbus/cleanup.log

Cleanup log of Nimbus uninstallation

logviewer/cleanup.log

Cleanup log of logviewer uninstallation

ui/cleanup.log

Cleanup log of UI uninstallation

supervisor/cleanup.log

Cleanup log of supervisor uninstallation

leader_switch.log

Run log file that records the Storm active/standby switchover

Audit log

nimbus/audit.log

Nimbus audit log

ui/audit.log

UI audit log

supervisor/audit.log

Supervisor audit log

logviewer/audit

Logviewer audit log

Log Levels

Table 2 describes the log levels supported by Storm.

Levels of run logs and audit logs are ERROR, WARN, INFO, and DEBUG from the highest to the lowest priority. Run logs of equal or higher levels are recorded. The higher the specified log level, the fewer the logs recorded.

Table 2 Log levels

Level

Description

ERROR

Logs of this level record error information about system running.

WARN

Logs of this level record exception information about the current event processing.

INFO

Logs of this level record normal running status information about the system and events.

DEBUG

Logs of this level record the system information and system debugging information.

To modify log levels, perform the following operations:

  1. Go to the All Configurations page of Storm by referring to Modifying Cluster Service Configuration Parameters.
  2. On the menu bar on the left, select the log menu of the target role.
  3. Select a desired log level.
  4. Save the configuration. In the displayed dialog box, click OK to make the configurations take effect.

Log Format

The following table lists the Storm log formats:

Table 3 Log Formats

Log Type

Format

Example

Run log

%d{yyyy-MM-dd HH:mm:ss,SSS} | %-5p | [%t] | %m | %logger (%F:%L) %n

2015-03-11 23:04:00,241 | INFO | [RMI TCP Connection(2646)-10.0.0.2] | The baseSleepTimeMs [1000] the maxSleepTimeMs [1000] the maxRetries [1] | backtype.storm.utils.StormBoundedExponentialBackoffRetry (StormBoundedExponentialBackoffRetry.java:46)

<yyyy-MM-dd HH:mm:ss,SSS><HostName><RoleName><logLevel><Message>

2017-03-28 02:57:52 493 10-5-146-1 storm- INFO Nimbus start normally

Audit log

<Username><User IP address><Time><Operation><Operation object><Operation result>

UserName=storm/hadoop, UserIP=10.10.0.2, Time=Tue Mar 10 01:15:35 CST 2015, Operation=Kill, Resource=test, Result=Success
Parent topic: Using Storm