What Do I Do If a Container Image Fails to Be Pulled?

Symptom

An image in SWR cannot be pulled to an edge node.

Fault Locating

Figure 1 shows the major causes of the failure to pull the container image. You can locate the fault based on Table 1.

Figure 1 Fault locating

Image Address Is Incorrect

Perform the following operations to check whether the image address is correct. If the address is incorrect, change it to a correct one.

1. Log in to the SWR console.
2. In the navigation pane, choose My Images. On the page displayed, click the image that fails to be pulled.
3. On the Image Tags tab page, in the same row as the image tag, click to copy its pull command.
   Figure 2 Confirming the pull command
   

User Has No Permission to Pull the Image

Obtain and run the login command to pull the image. If the login fails or the image cannot be pulled, you do not have the permission.

Figure 3 Obtaining a login command

Edge Node Cannot Resolve the SWR and OBS Domain Names

As SWR images are actually stored in OBS, check that the edge node has permissions to access both SWR and OBS. Otherwise, images cannot be pulled.

Check whether your node can resolve the SWR and OBS domain names. If the domain names cannot be resolved, images fail to be pulled.

• Check method 1:

  View logs on the edge node.

  cat /var/IEF/sys/log/edge_core.log| grep lookup

• Check method 2:

  Run the following commands on the edge node:

  ping swr.cn-north-4.myhuaweicloud.com

  ping obs.cn-north-4.myhuaweicloud.com

  Check whether the domain names can be resolved into corresponding IP addresses. If the domain names can be resolved but the login to SWR is suspended, check the domain name resolution mode of your container engine. If the domain names cannot be resolved into corresponding IP addresses, perform the following steps to check whether the DNS server configuration was modified:

  • If the mapping between the domain names and IP addresses is configured in the local hosts file, check whether the configuration is correct.

    cat /etc/hosts | grep swr

    cat /etc/hosts | grep obs

    If the hosts file configuration is incorrect, the domain name resolution will fail. In this case, resolve the domain name on the host whose hosts file is correct to obtain a correct IP address and then configure the IP address in the hosts file on your edge node.

  • If you use the DNS server configured in the /etc/resolv.conf file to resolve domain names, check whether the DNS server configuration is correct and whether the DNS server can resolve the preceding SWR and OBS domain names.

    Note that if IEF and the container engine are accessed through a public network, the DNS server with the IP address set to 114.114.114.114 is used in Chinese mainland. Ensure that your DNS server is correctly configured so that the domain names can be resolved within the validity period.

    If multiple DNS server IP addresses (including 114.114.114.114) are configured but the container engine still cannot resolve domain names, you are advised to retain only 114.114.114.114 and comment out the other DNS server IP addresses.

  • If a proxy server is configured, check whether the proxy server can resolve the SWR and OBS domain names.
    • Open two terminals on your edge node.

      On one terminal, run ping swr.cn-north-4.myhuaweicloud.com.

      On the other terminal, run tcpdump -nn -i eth0 udp port 53 to capture packets.

      Check the sequence of using DNS servers and the domain name resolution result to determine the DNS server by which the domain name is resolved. (If the hosts file is configured, domain name resolution is not performed.)

    • On one terminal, run docker login XXX (temporary login command copied from SWR).

      On the other terminal, run tcpdump -nn -i eth0 udp port 53 to capture packets.

      Check the sequence of using DNS servers and the domain name resolution result to determine the DNS server by which the domain name is resolved. (If the hosts file is configured, domain name resolution is not performed.)

    • Check whether the DNS servers used in the preceding two steps are the same and whether the domain names are resolved. If the DNS servers used in the two steps are different, modify the /etc/resolv.conf file and configure the hosts file.

Edge Node Cannot Connect to SWR

1. Check whether the network connection between the edge node and SWR is normal.

   On the edge node, run the following commands:

   curl -i -k -v swr.cn-north-4.myhuaweicloud.com

   curl -i -k -v obs.cn-north-4.myhuaweicloud.com

   NOTE:

   The domain name varies depending on regions.

   If the network is inaccessible, check your network policies to determine whether the edge node can connect to external networks. If the network is accessible but the Docker login command is suspended, repeat Edge Node Cannot Resolve the SWR and OBS Domain Names to check the resolution of the SWR domain name.

   If the SWR domain name can be resolved, check the external network access mode configured for the edge node to determine whether your container can access external networks. Confirm the following problems:

   1. Check whether a proxy is configured for the edge node.

      env| grep proxy

      env| grep PROXY

   2. Check whether a proxy is configured for the container engine.

      systemctl show --property=Environment docker

      If a proxy is used for external network access, ensure that the proxy configuration of your node and container engine is correct.

2. Log in to SWR again and pull the image.

   If a message indicating an authentication failure is displayed, submit a service ticket.

Insufficient Disk Space

Run the following command to query the disk usage:

df -h

Check the available disk space in /var/lib/docker/* on the edge node. Then, compare the available disk space with the size of your image (the actual image size is generally greater than that displayed on the SWR console) to determine whether the space is sufficient for pulling the image. If the disk space is insufficient, an error is reported during image pull.

Image Is Too Large but the Bandwidth Is Insufficient

Check the image size and external network bandwidth to determine whether the image pull will time out.

For example, if the image size is 1 GB and the download speed of the edge node is about 200 kbit/s, it takes about 85.3 minutes (1024 MB/0.2 MB = 5,129s) to pull the image. This duration is much longer than the normal delivery duration of an application. In this case, the application cannot obtain the image.

To solve this problem, you are advised to pull the image to the edge node and then deliver the application.

Container Engine Is Abnormal

If an error is reported when you run the docker pull command, you can search for the solution to solve the error on the Internet. This is because the container engine versions installed by users may be different and have some performance defects.

You can also restart the container engine to work around this problem.

systemctl restart docker

To avoid this problem, you are advised to install the container engine versions recommended by IEF. For details about the container engine version requirements, see Constraints.