Configuring Container Log Aggregation
Scenario
- After the application is complete, collect container logs to HDFS at a time.
- During application running, periodically collect log segments generated by containers and save them to HDFS.
Configuring Container Log Aggregation
- Log in to FusionInsight Manager.
For details about how to log in to FusionInsight Manager, see Accessing MRS FusionInsight Manager.
- Choose Cluster > Services > Yarn > Configurations > All Configurations.
- Search for the following parameters and change their values as required.
- Modify and save the configuration. On the Dashboard tab page, choose More > Synchronize Configuration. After the synchronization is complete, restart the YARN service.
Configuring the Container Log Retention Duration
- Log in to FusionInsight Manager.
- Choose Cluster > Services > Mapreduce > Configurations > All Configurations.
- Search for the following parameters and change their values as required.
Table 2 Parameter description Parameter
Description
Default Value
yarn.log-aggregation.retain-seconds
Duration for retaining aggregated logs, in second
- If this parameter is set to –1, the container logs will be retained permanently in the HDFS.
- If this parameter is set to 0 or a positive integer, container logs will be stored for such a period and deleted after the period expires.
- A short period may increase load of the NameNode. Therefore, you are advised to set this parameter to a proper value.
1296000
yarn.log-aggregation.retain-check-interval-seconds
Interval for storing container logs in HDFS, in second
- If this parameter is set to -1 or 0, the interval will be one tenth of the period specified by yarn.log-aggregation.retain-seconds.
If yarn.log-aggregation.retain-seconds is set to 0, this parameter cannot be set to 0 or -1.
- If this parameter is set to a positive number, container logs in HDFS will be scanned at such an interval.
- A short interval may increase load of the NameNode. Therefore, you are advised to set this parameter to a proper value.
86400
- Save the modified configuration. Restart the expired service or instance for the configuration to take effect.
Configuring the Rolling Output of MapReduce Application Log Files
The periodic log collection function applies only to MapReduce applications, for which rolling output of log files must be configured. Table 3 describes the configurations in the Client installation path/Yarn/config/mapred-site.xml configuration file on the MapReduce client node.
- Log in to FusionInsight Manager.
For details about how to log in to FusionInsight Manager, see Accessing MRS FusionInsight Manager.
- Choose Cluster > Services > Yarn > Configurations > All Configurations.
- Search for the following parameters and change their values as required.
Table 3 Configuring rolling output of MapReduce application log files Parameter
Description
Default Value
mapreduce.task.userlog.limit.kb
Maximum size of a single task log file of the MapReduce application, in KB.
- The value must be greater than or equal to 0.
- When the maximum size of the log file has been reached, another log file is generated.
- The value 0 indicates that the size of the log file is not limited.
51200
yarn.app.mapreduce.task.container.log.backups
Maximum number of task logs that can be retained for the MapReduce application.
Number of task log backup files when ContainerRollingLogAppender (CRLA) is used.
- By default, the ContainerLogAppender (CLA) is used and container logs are not rolled back.
- When both mapreduce.task.userlog.limit.kb and yarn.app.mapreduce.task.container.log.backups are greater than 0, CRLA is enabled.
- If this parameter is set to 0, rolling output is disabled.
- The value ranges from 0 to 999.
10
yarn.app.mapreduce.am.container.log.limit.kb
Maximum size of a single ApplicationMaster log file of the MapReduce application, in KB.
- When the maximum size of the log file has been reached, another log file is generated.
- The value must be greater than or equal to 0.
- The value 0 indicates that the size of a single ApplicationMaster log file is not limited.
51200
yarn.app.mapreduce.am.container.log.backups
Number of ApplicationMaster log backup files when CRLA is used.
- When both yarn.app.mapreduce.am.container.log.limit.kb and yarn.app.mapreduce.am.container.log.backups are greater than 0, CRLA is enabled for the ApplicationMaster.
- By default, CLA is used and container logs are not rolled back.
- The value ranges from 0 to 999.
- If this parameter is set to 0, rolling output is disabled.
20
yarn.app.mapreduce.shuffle.log.backups
Maximum number of shuffle logs that can be retained for an MR application.
- When both yarn.app.mapreduce.shuffle.log.limit.kb and yarn.app.mapreduce.shuffle.log.backups are greater than 0, syslog.shuffle uses CRLA.
- The value ranges from 0 to 999.
- If this parameter is set to 0, rolling output is disabled.
10
yarn.app.mapreduce.shuffle.log.limit.kb
Maximum size of a single shuffle log file of a MapReduce application, in KB.
- When the maximum size of the log file has been reached, another log file is generated.
- The value must be greater than or equal to 0.
- The value 0 indicates that the size of a single shuffle log file is not limited.
51200
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot