Fault Locating

If the pod status is ImagePullBackOff, the image fails to be pulled. For details about how to view Kubernetes events, see Viewing Pod Events.

Troubleshooting Process

Determine the cause based on the event information, as listed in Table 1.

Figure 1 Troubleshooting process

Check Item 1: Whether imagePullSecret Is Specified When You Use kubectl to Create a Workload

If the workload status is abnormal and a Kubernetes event is displayed indicating that the pod fails to pull the image, check whether the imagePullSecrets field exists in the YAML file.

Items to Check

• If an image needs to be pulled from HUAWEI CLOUD SWR, the name parameter must be set to default-secret.

  apiVersion: extensions/v1beta1
  kind: Deployment
  metadata:
    name: nginx
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: nginx
    strategy:
      type: RollingUpdate
    template:
      metadata:
        labels:
          app: nginx
      spec:
        containers:
        - image: nginx 
          imagePullPolicy: Always
          name: nginx
        imagePullSecrets:
        - name: default-secret

• If an image needs to be pulled from a third-party image repository, the imagePullSecrets parameter must be set to the created secret name.

  When you use kubectl to create a workload from a third-party image, specify the imagePullSecret field, in which name indicates the name of the secret used to pull the image. For details about how to create a secret, see Using kubectl.

Check Item 2: Whether the Image Address Is Correct When a Third-Party Image Is Used

CCE can create workloads using images pulled from a third-party image repository in addition to SWR and open source image registries.

Enter the third-party image address according to requirements. The format must be ip:port/path/name:version or name:version. If no tag is specified, latest is used by default.

• For a private repository, enter an image address in the format of ip:port/path/name:version.
• For an open-source Docker repository, enter an image address in the format of name:version, for example, nginx:latest.
  Figure 2 Using a third-party image
  

The following information is displayed when you fail to pull an image due to incorrect image address provided.

Failed to pull image "nginx:v1.1": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io: no such host

Solution

You can either edit your YAML file to modify the image address or log in to the CCE console to replace the image on the Upgrade tab page on the workload details page.

Check Item 3: Whether an Incorrect Secret Is Used When a Third-Party Image Is Used

Generally, a third-party image repository can be accessed only after authentication (using your account and password). CCE uses the secret authentication mode to pull images. Therefore, you need to create a secret for an image repository before pulling images from the repository.

Solution

If your secret is incorrect, images will fail to be pulled. In this case, create a new secret.

To create a secret, see Using kubectl.

Check Item 4: Whether the Node Disk Space Is Insufficient

After a node is created in a cluster of v1.7.3-r7 or a later version, a 100 GB data disk dedicated for Docker is bound to the node. If the data disk space is insufficient, image fails to be pulled.

Figure 3 Data disk capacity (GB)

If the Kubernetes event contains the following information, the node has no disk space left for storing images. You need to clean up images or expand the disk capacity.

Run the lvs command to check the disk space for storing images on the node.

Run the following command to clean up images:

docker rmi -f {Image ID}

To expand the disk capacity, perform the following steps:

1. Expand the capacity of the Docker disk on the EVS console.
2. Log in to the target node.
3. Run the following commands on the node to add the new disk capacity to the Docker disk:

   pvresize /dev/vdb
   lvextend -l+100%FREE -n vgpaas/thinpool

Check Item 5: Whether the Remote Image Repository Uses an Unknown or Insecure Certificate

When a pod pulls an image from a third-party image repository that uses an unknown or insecure certificate, the image fails to be pulled from the node. The pod event list contains the event "Failed to pull the image" with the cause "x509: certificate signed by unknown authority", as shown in the following figure.

The security of EulerOS 2.9 images is enhanced. Some insecure or expired certificates are removed from the system. It is normal that this error is reported in EulerOS 2.9 but not or some third-party images on other types of nodes. You can also perform the following operations to rectify the fault.

Solution

1. Check the IP address and port number of the third-party image server for which the error message "unknown authority" is displayed.
   You can see the IP address and port number of the third-party image server for which the error is reported in the event information "Failed to pull image".

   Failed to pull image "bitnami/redis-cluster:latest": rpc error: code = Unknown desc = error pulling image configuration: Get https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/e8/e83853f03a2e792614e7c1e6de75d63e2d6d633b4e7c39b9d700792ee50f7b56/data?verify=1636972064-AQbl5RActnudDZV%2F3EShZwnqOe8%3D: x509: certificate signed by unknown authority

   The IP address of the third-party image server is production.cloudflare.docker.com, and the default HTTPS port number is 443.

2. Load the root certificate of the third-party image server to the node where the third-party image is to be downloaded.

   Run the following commands on the EulerOS and CentOS nodes with {server_url}:{server_port} replaced with the IP address and port number obtained in Step 1, for example, production.cloudflare.docker.com:443:

   If the container engine of the node is containterd, replace systemctl restart docker with systemctl restart containerd.

   openssl s_client -showcerts -connect {server_url}:{server_port} < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /etc/pki/ca-trust/source/anchors/tmp_ca.crt
   update-ca-trust
   systemctl restart docker

   Run the following command on Ubuntu nodes:

   openssl s_client -showcerts -connect {server_url}:{server_port} < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /usr/local/share/ca-certificates/tmp_ca.crt
   update-ca-trust
   systemctl restart docker

Check Item 6: Whether the Image Size Is Too Large

The pod event list contains the event "Failed to pull image". This may be caused by a large image size.

Failed to pull image "XXX": rpc error: code = Unknown desc = context canceled

Log in to the node and run the docker pull command to manually pull the image. The image is successfully pulled.

Root Cause

The default value of image-pull-progress-deadline is 1 minute. If the image pull progress is not updated within 1 minute, image pull is canceled. If the node performance is poor or the image size is too large, the image may fail to be pulled and the workload may fail to be started.

Solution

• (Recommended) Method 1: Log in to the node, run the docker pull command to manually pull the image, and check whether imagePullPolicy of the workload is IfNotPresent (default policy configuration). In this case, the image that has been pulled to the local host is used to create the workload.
• Method 2: Modify the kubelet configuration parameters.

  For a cluster of v1.15 or later, run the following command:

  vi /opt/cloud/cce/kubernetes/kubelet/kubelet

  For a cluster earlier than v1.15, run the following command:

  vi /var/paas/kubernetes/kubelet/kubelet

  Add --image-pull-progress-deadline=30m to the end of the DAEMON_ARGS parameter. 30m indicates 30 minutes. You can change the value as required. The added configuration and the existing configuration are separated by a space.

  

  Run the following command to restart kubelet:

  systemctl restart kubelet

  Wait for a while and check whether the kubelet status is running.

  systemctl status kubelet

  

  The workload is started properly, and the image is successfully pulled.