What Can I Do If the kube-prometheus-stack Add-on Instance Fails to Be Scheduled?

Symptom

During the installation of kube-prometheus-stack, the add-on remains in the partially ready state. The message "0/x nodes are available: x node(s) had volume node affinity conflict." is displayed in the event of the prometheus pod.

The same problem may occur during the installation of grafana.

Possible Causes

The PV required by the prometheus pod already exists in the cluster, but the corresponding EVS disk is not in the same AZ as the node where the prometheus pod resides. As a result, the pod scheduling fails. This may be because kube-prometheus-stack is not installed for the first time in the cluster.

• If kube-prometheus-stack is installed for the first time, the attaching of an EVS disk (with the PVC named pvc-prometheus-server-0) to the prometheus pod will be delayed. When the EVS disk is created, it will automatically be in the same AZ as the node where the prometheus pod resides. For example, if the AZ of the node where the pod is running is AZ 1, the disk will be automatically created in AZ 1.
• When kube-prometheus-stack is uninstalled from the cluster, the PV mounted to the prometheus pod will not be deleted and the existing monitoring data will be retained. If the add-on is installed again, the nodes in the cluster may be newly created. If none of them is in AZ 1, the prometheus pod cannot run.

The causes may also result in the scheduling failure of a grafana pod.

Solution

Check the AZ of the EVS disk corresponding to the existing PV mounted to the pod and create a node in the same AZ as this EVS disk.

1. Log in to the CCE console and click the cluster name to access the cluster console.
2. In the navigation pane, choose Storage. Click the PVs tab, locate the row that contains the pvc-prometheus-server-0 PVC in the PVC column, and click the volume name in the Volume column to go to the EVS disk details page.
   Figure 2 Locating the target volume
   

3. In the Basic Information area, view the AZ of the EVS disk.
   Figure 3 Viewing the details of the EVS disk
   

4. On the CCE console, click the cluster name to access the cluster console. Choose Nodes in the navigation pane, click the Nodes tab, and click Create Node to create a node in the same AZ as the EVS disk.
   Figure 4 Creating a node in a specified AZ
   

5. Reschedule the node. This operation is automatically performed by the workload scheduler.