Workload Cannot Be Started After a Node Is Restarted (ASM 1.28+)

Symptom

In ASM 1.28 or later, workloads fail to start after a mesh node is restarted. The event log shows the following information.

Root Cause

By default, graceful node shutdown is not enabled. As a result, the istio-cni component configuration file is not properly cleared due to abnormal shutdown. For details about graceful node shutdown, see Node Shutdowns.

Enabling Graceful Node Shutdown

Procedure

1. Back up the original configuration file.

   cp /opt/cloud/cce/kubernetes/kubelet/kubelet_config.yaml /opt/cloud/cce/kubernetes/kubelet/kubelet_config.yaml.bak

2. Edit the kubelet configuration file on the node.

   vim /opt/cloud/cce/kubernetes/kubelet/kubelet_config.yaml

3. Add the following graceful shutdown configuration to the end of the file:

   shutdownGracePeriod: 60s   # Total duration for waiting for graceful pod termination before the node is shut down
   shutdownGracePeriodCriticalPods: 57s     # Graceful termination duration for critical pods

4. Enter :wq to save the changes and exit, and restart kubelet.

   systemctl restart kubelet

Solution: Restore the Abnormal Node

Application scenario: Graceful termination is not enabled and a workload exception has occurred.

Procedure

1. Back up the original configuration file.

   cp /etc/cni/net.d/cni.conflist /etc/cni/net.d/cni.conflist.bak

2. Edit the CNI configuration file.

   vim /etc/cni/net.d/cni.conflist

3. Delete the istio-cni configuration section under plugins (the following uses typical configurations as an example):
   • Delete the istio-cni configuration section: In the plugins array, find the configuration section whose name is istio-cni and delete the entire section. This configuration section usually contains multiple key-value pairs, such as ambient_enabled, cni_agent_run_dir, and dns.
   • Delete the comma at the end of the previous configuration item: After deleting the istio-cni configuration section, check the end of the previous configuration item and delete the comma to ensure that the JSON format is correct.

     For example, in the following configuration:

   

   Delete the entire configuration section starting from "ambient_enabled": false to "operator": "NotIn", and delete the comma at the end of the previous configuration item ("type": "eni-neutron").

4. Enter :wq to save the changes and exit, and restart the workload.

   kubectl rollout restart deployment <workload-name>