Why Are Local Logs Not Deleted After YARN Restarts?

Question

If YARN is restarted in either of the following scenarios, local logs will not be deleted as scheduled and will be retained permanently:

• When YARN is restarted during task running, local logs are not deleted.
• When the task is complete and logs fail to be collected, restart Yarn before the logs are cleared as scheduled. In this case, local logs are not deleted.

Answer

NodeManager supports a restart recovery mechanism. For details, visit the following:

Versions earlier than MRS 3.2.0: https://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/NodeManager.html#NodeManager_Restart

MRS 3.2.0 or later: https://hadoop.apache.org/docs/r3.3.1/hadoop-yarn/hadoop-yarn-site/NodeManager.html#NodeManager_Restart

Perform the following operations to set yarn.nodemanager.recovery.enabled of NodeManager to true to make the configuration take effect. The default value is true. In this way, redundant local logs are periodically deleted when the YARN restarts.

1. Log in to FusionInsight Manager.

   For details about how to log in to FusionInsight Manager, see Accessing MRS FusionInsight Manager.

2. Choose Cluster > Services > Yarn > Configurations > All Configurations.
3. Search for yarn.nodemanager.recovery.enabled and change its value to true.

   This parameter controls whether to collect failure logs when NodeManager restarts and whether to recover the unfinished application. The default value is true.

4. Save the settings. Restart the expired service or instance for the configuration to take effect.