What Should I Do If a Pod Fails to Pull the Image?

Fault Locating

When a workload enters the state of "Pod not ready: Back-off pulling image "xxxxx", a Kubernetes event of PodsFailed to pull image or Failed to re-pull image will be reported. For details about how to view Kubernetes events, see Viewing Pod Events.

Troubleshooting Process

Determine the cause based on the event information, as listed in Table 1.

Check Item 1: Whether imagePullSecret Is Specified When You Use kubectl to Create a Workload

If the workload status is abnormal and a Kubernetes event is displayed indicating that the pod fails to pull the image, check whether the imagePullSecrets field exists in the YAML file.

Items to Check

• If an image needs to be pulled from SWR, the name parameter must be set to default-secret.

  apiVersion: extensions/v1beta1
  kind: Deployment
  metadata:
    name: nginx
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: nginx
    strategy:
      type: RollingUpdate
    template:
      metadata:
        labels:
          app: nginx
      spec:
        containers:
        - image: nginx 
          imagePullPolicy: Always
          name: nginx
        imagePullSecrets:
        - name: default-secret

• If an image needs to be pulled from a third-party image repository, the imagePullSecrets parameter must be set to the created secret name.

  When you use kubectl to create a workload from a third-party image, specify the imagePullSecret field, in which name indicates the name of the secret used to pull the image.

Check Item 2: Whether the Image Address Is Correct When a Third-Party Image Is Used

CCE allows you to create workloads using images pulled from third-party image repositories.

Enter the third-party image address according to requirements. The format must be ip:port/path/name:version or name:version. If no tag is specified, latest is used by default.

• For a private repository, enter an image address in the format of ip:port/path/name:version.
• For an open-source Docker repository, enter an image address in the format of name:version, for example, nginx:latest.

The following information is displayed when you fail to pull an image due to incorrect image address provided.

Failed to pull image "nginx:v1.1": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io: no such host

Solution

You can either edit your YAML file to change the image address or log in to the CCE console to replace the image on the Upgrade tab on the workload details page.

Check Item 3: Whether an Incorrect Secret Is Used When a Third-Party Image Is Used

Generally, a third-party image repository can be accessed only after authentication (using your account and password). CCE uses the secret authentication mode to pull images. Therefore, you need to create a secret for an image repository before pulling images from the repository.

Solution

If your secret is incorrect, images will fail to be pulled. In this case, create a new secret.

Check Item 4: Whether the Node Disk Space Is Insufficient

If the Kubernetes event contains information "no space left on device", there is no disk space left for storing the image. As a result, the image will fail to be pulled. In this case, clear the image or expand the disk space to resolve this issue.

Failed create pod sandbox: rpc error: code = Unknown desc = failed to create a sandbox for pod "nginx-6dc48bf8b6-l8xrw": Error response from daemon: mkdir xxxxx: no space left on device

Run the following command to obtain the disk space for storing images on a node:

lvs

Solution 1: Clearing images

Perform the following operations to clear unused images:

• Nodes that use containerd
  1. Obtain local images on the node.

     crictl images -v

  2. Delete the images that are not required by image ID.

     crictl rmi Image ID

• Nodes that use Docker
  1. Obtain local images on the node.

     docker images

  2. Delete the images that are not required by image ID.

     docker rmi Image ID

Do not delete system images such as the cce-pause image. Otherwise, pods may fail to be created.

Solution 2: Expanding the disk capacity

To expand a disk capacity, perform the following steps:

1. Expand the capacity of the data disk on the EVS console.
2. Log in to the CCE console and click the cluster. In the navigation pane, choose Nodes. Locate the row containing the target node and choose More > Sync Server Data in the Operation column.
3. Log in to the target node.
4. Run lsblk to view the block device information of the node.

   A data disk is divided depending on the container storage Rootfs:

   • Overlayfs: No independent thin pool is allocated. Image data is stored in the dockersys disk.

     # lsblk
     NAME                MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
     sda                   8:0    0   50G  0 disk 
     └─sda1                8:1    0   50G  0 part /
     sdb                   8:16   0  200G  0 disk 
     ├─vgpaas-dockersys  253:0    0   90G  0 lvm  /var/lib/docker               # Space used by the container engine
     └─vgpaas-kubernetes 253:1    0   10G  0 lvm  /mnt/paas/kubernetes/kubelet  # Space used by Kubernetes

     Run the following commands on the node to add the new disk capacity to the dockersys disk:

     pvresize /dev/sdb 
     lvextend -l+100%FREE -n vgpaas/dockersys
     resize2fs /dev/vgpaas/dockersys

   • Devicemapper: A thin pool is allocated to store image data.

     # lsblk
     NAME                                MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
     sda                                   8:0    0   50G  0 disk 
     └─sda1                                8:1    0   50G  0 part /
     sdb                                   8:16   0  200G  0 disk 
     ├─vgpaas-dockersys                  253:0    0   18G  0 lvm  /var/lib/docker    
     ├─vgpaas-thinpool_tmeta             253:1    0    3G  0 lvm                   
     │ └─vgpaas-thinpool                 253:3    0   67G  0 lvm                   # Thin pool space.
     │   ...
     ├─vgpaas-thinpool_tdata             253:2    0   67G  0 lvm  
     │ └─vgpaas-thinpool                 253:3    0   67G  0 lvm  
     │   ...
     └─vgpaas-kubernetes                 253:4    0   10G  0 lvm  /mnt/paas/kubernetes/kubelet

     • Run the following commands on the node to add the new disk capacity to the thinpool disk:

       pvresize /dev/sdb 
       lvextend -l+100%FREE -n vgpaas/thinpool

     • Run the following commands on the node to add the new disk capacity to the dockersys disk:

       pvresize /dev/sdb 
       lvextend -l+100%FREE -n vgpaas/dockersys
       resize2fs /dev/vgpaas/dockersys

Check Item 5: Whether the Remote Image Repository Uses an Unknown or Insecure Certificate

When a pod pulls an image from a third-party image repository that uses an unknown or insecure certificate, the image fails to be pulled from the node. The pod event list contains the event "Failed to pull the image" with the cause "x509: certificate signed by unknown authority".

The security of EulerOS 2.9 images is enhanced. Some insecure or expired certificates are removed from the system. It is normal that this error is reported in EulerOS 2.9 but not or some third-party images on other types of nodes. You can also perform the following operations to rectify the fault.

Solution

1. Check the IP address and port number of the third-party image server for which the error message "unknown authority" is displayed.
   You can see the IP address and port number of the third-party image server for which the error is reported in the event information "Failed to pull image".

   Failed to pull image "bitnami/redis-cluster:latest": rpc error: code = Unknown desc = error pulling image configuration: Get https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/e8/e83853f03a2e792614e7c1e6de75d63e2d6d633b4e7c39b9d700792ee50f7b56/data?verify=1636972064-AQbl5RActnudDZV%2F3EShZwnqOe8%3D: x509: certificate signed by unknown authority

   The IP address of the third-party image server is production.cloudflare.docker.com, and the default HTTPS port number is 443.

2. Load the root certificate of the third-party image server to the node where the third-party image is to be downloaded.

   Run the following commands on the EulerOS and CentOS nodes with {server_url}:{server_port} replaced with the IP address and port number obtained in Step 1, for example, production.cloudflare.docker.com:443:

   If the container engine of the node is containerd, replace systemctl restart docker with systemctl restart containerd.

   openssl s_client -showcerts -connect {server_url}:{server_port} < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /etc/pki/ca-trust/source/anchors/tmp_ca.crt
   update-ca-trust
   systemctl restart docker

   Run the following command on Ubuntu nodes:

   openssl s_client -showcerts -connect {server_url}:{server_port} < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /usr/local/share/ca-certificates/tmp_ca.crt
   update-ca-trust
   systemctl restart docker

Check Item 6: Whether the Image Size Is Too Large

The pod event list contains the event "Failed to pull image". This may be caused by a large image size.

Failed to pull image "XXX": rpc error: code = Unknown desc = context canceled

Log in to the node and run the docker pull command to manually pull the image. The image is successfully pulled.

Possible Causes

The default value of image-pull-progress-deadline is 1 minute. If the image pull progress is not updated within 1 minute, image pull is canceled. If the node performance is poor or the image size is too large, the image may fail to be pulled and the workload may fail to be started.

Solution

• (Recommended) Method 1: Log in to the node, run the docker pull command to manually pull the image, and check whether imagePullPolicy of the workload is IfNotPresent (default policy configuration). In this case, the image that has been pulled to the local host is used to create the workload.
• Method 2: Modify the kubelet configuration parameters.

  For a cluster of v1.15 or later, run the following command:

  vi /opt/cloud/cce/kubernetes/kubelet/kubelet

  For a cluster earlier than v1.15, run the following command:

  vi /var/paas/kubernetes/kubelet/kubelet

  Add --image-pull-progress-deadline=30m to the end of the DAEMON_ARGS parameter. 30m indicates 30 minutes. You can change the value as required. The added configuration and the existing configuration are separated by a space.

  

  Run the following command to restart kubelet:

  systemctl restart kubelet

  Wait for a while and check whether the kubelet status is running.

  systemctl status kubelet

  

  The workload is started properly, and the image is successfully pulled.

Check Item 7: Connection to the Image Repository

Symptom

The following error message is displayed during workload creation:

Failed to pull image "docker.io/bitnami/nginx:1.22.0-debian-11-r3": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Possible Causes

Failed to connect to the image repository due to the disconnected network. SWR allows you to pull images only from the official Docker repository. For image pulls from other repositories, you need to access the Internet.

Solution

• Bind a public IP address to the node which needs to pull the images.
• Upload the image to SWR and then pull the image from SWR.

Check Item 8: Whether the Number of Public Image Pull Times Reaches the Upper Limit

Symptom

The following error message is displayed during workload creation:

ERROR: toomanyrequests: Too Many Requests.

Or

you have reached your pull rate limit, you may increase the limit by authenticating an upgrading: https://www.docker.com/increase-rate-limits.

Possible Causes

Docker Hub sets the maximum number of container image pull requests. For details, see Understanding Your Docker Hub Rate Limit.

Solution

Push the frequently used image to SWR and then pull the image from SWR.