Task Execution Fails After Host Isolation and De-isolation

Issue

Task execution fails after a host is isolated and then de-isolated.

Symptom

When a host in an MRS cluster becomes unresponsive and triggers host isolation, after de-isolating the host and starting all instances on the node, the NodeManager instance starts successfully, but all YARN containers on that node fail to start.

Upon reviewing the NodeManager logs, the following error is continuously reported: container-log4j.properties (Permission denied), exit code 255. This causes services that depend on container startup, such as SparkJDBCServer, to fail to run properly.

Cause Analysis

When the host becomes unresponsive and triggers isolation, the command to stop NodeManager fails to be delivered because the host is unresponsive. The instance status remains recorded as STARTED.

When de-isolation occurs, NodeAgent reads the STARTED state and directly starts the instance, bypassing the permission correction steps in the startup script. It only renames the etc_UPDATE directory to the etc directory (where file permissions are set to 600), resulting in abnormal configuration file permissions.

Handling Procedure

1. After the administrator has resolved the host anomaly or fault, log in to the MRS cluster Manager. Choose Hosts, select the isolated host, choose More > Cancel Isolation, and wait for the de-isolation process to complete.
2. On the Hosts page, click the name of the de-isolated host, click Start All Instances, and start the instances as prompted.
3. Choose Cluster > Services > Yarn > Instances. Select the NodeManager instance that matches the name of the de-isolated host, choose More > Restart Instance, and restart the NodeManager instance on that node.
4. Manually submit a job and observe the NodeManager logs to confirm that YARN containers are starting normally.