How Do I Locate the Fault When a Cluster Is Unavailable?

Troubleshooting

Possible causes are listed in order of likelihood.

Check these causes one by one until you find the cause of the fault.

• Check Item 1: Security Group Modification
• Check Item 2: Cluster Overloaded
• Check Item 3: KMS Key for Secret Encryption at Rest
• Check Item 4: Cluster Certificate

If the fault persists, submit a service ticket and contact the customer service to help you locate the fault.

Check Item 1: Security Group Modification

1. Log in to the management console and choose Service List > Networking > Virtual Private Cloud. In the navigation pane, choose Access Control > Security Groups and find the security group of the master node in the cluster.

   The name of this security group is in the format of {cluster_name}-cce-control-{ID}.

2. Click the security group. On the details page displayed, ensure that the security group rules of the master node are correct.

   For details, see How Do I Modify Cluster Security Group Rules?

Check Item 2: Cluster Overloaded

Symptom

The resource usage on the master nodes in the cluster reaches 100%.

Possible Cause

When a cluster has a large number of resources created simultaneously, it causes an overload on the API server. This, in turn, overloads the master nodes and leads to OOM issues.

Solution

Increase the cluster management scale. A larger cluster management scale means higher capacity and improved performance of the master nodes. For details, see Changing a Cluster Scale.

If a cluster is overloaded, you can submit a service ticket for technical support.

Check Item 3: KMS Key for Secret Encryption at Rest

Symptom

If a cluster is unavailable, you can check the cluster event to locate the fault.

If KMS key status abnormal is displayed in the events, check whether the key used by the cluster is in the Disabled or Pending deletion state.

Solution

1. Log in to the DEW console.
2. In the custom key list, find the KMS key used by the cluster.
   • For a key in the Pending deletion state, click Cancel Deletion in the Operation column. If the key remains in a Disabled state even after cancellation, then cancel the disabling.
   • For a key in the Disabled state, click Enable in the Operation column.

3. Verify whether the key has been enabled and wait for the cluster to be automatically restored. The restoration process should take about 5 to 10 minutes.

Check Item 4: Cluster Certificate

Symptom

If the region where a cluster is located requires a transition between daylight saving time (DST) and standard time (ST), there may be a period of unavailability during the overlapping time. For example, if you apply to create a cluster at 02:00 in the morning, the time will shift to 01:00 in the morning when DST changes to ST. This can potentially result in the cluster being unavailable.

Possible Cause

A cluster certificate cannot work immediately. This will affect the cluster performance. Each Kubernetes component uses a certificate to access kube-apiserver, which then verifies the certificate validity. If the verification fails, the request will be rejected.

Solution

• Delete the cluster and create a new one.
• Wait until the certificate takes effect, at which point the cluster will automatically become available.
• Submit a service ticket to contact O&M personnel.

Check Item 4: Cluster Deletion Failed (Status Changed to Error)

Symptom

If cluster deletion fails, the status changes to Error, and a message indicates that the cluster is in an abnormal state and its functions are unavailable.

Possible Cause

Review the error details to identify the root cause. For example, the deletion task may fail because dependent services are temporarily unavailable.

Solution

• Retry the deletion.
• If the fault persists, submit a service ticket for technical support.