How Do I Locate a Workload Networking Fault?

Troubleshooting

Possible causes are listed in order of likelihood.

Check these causes one by one until you find the cause of the fault.

• Check Item 1: Container and Container Port
• Check Item 2: Node IP Address and NodePort
• Check Item 3: Load Balancer IP Address and Service Port
• Check Item 4: NAT Gateway and Port
• Check Item 5: Whether the Security Group of the Node Where the Container Is Located Allows Access

Check Item 1: Container and Container Port

Log in to the CCE console or use kubectl to obtain the pod IP address. Then, log in to the node or the pod and run curl or use another method to manually call the API and check whether the expected response is returned.

If the API is inaccessible externally, log in to the service container and attempt to access 127.0.0.1:{port}.

Common Issues

1. The container port is incorrectly configured (the container does not listen to the access port).
2. The URL does not exist (no related path exists in the container).
3. A Service exception (a Service bug in the container) occurs.
4. Check whether the cluster network's kernel component is functioning correctly (Open vSwitch for the container tunnel network model, IPvlan for the VPC network model, and the component for the Cloud Native Network 2.0 model).

Check Item 2: Node IP Address and NodePort

Only NodePort or LoadBalancer Services can be accessed externally using the node IP address and NodePort.

• NodePort Services:

  The NodePort is exposed through the node's security group.

• LoadBalancer Service:

  The NodePort can be viewed in the Service YAML.

Example:

nodePort: 30637 indicates the exposed node port. targetPort: 80 indicates the exposed pod port. port: 123 is the exposed Service port. LoadBalancer Services also use this port to configure the ELB listener.

After identifying the NodePort, access <node-IP-address>:<NodePort> of the node where the container is running and check whether the expected response is returned.

Common Issues

1. The NodePort is not allowed in the inbound rules of the node's security group.
2. A custom route is incorrectly configured for the node.
3. Pod labels do not match the Service selector (regardless of whether the Service was created via kubectl or API).

Check Item 3: Load Balancer IP Address and Service Port

There are several possible causes if <load-balancer-IP-address>:<Service-port> is inaccessible while <node-IP-address>:<NodePort> works.

Possible causes:

• The backend server group of the port or URL does not meet the expectations.
• The security group on the node has not exposed the related protocol or port to the ELB.
• The health check of the Layer 4 load balancing is not enabled.
• The certificate used for Services of Layer 7 load balancing has expired.

Common Issues

1. When exposing a Layer 4 ELB load balancer, if you have not enabled health check on the console, the load balancer may route requests to abnormal nodes.
2. For UDP access, the ICMP port of the node has not been allowed in the inbound rules.
3. Pod labels do not match the Service selector (regardless of whether the Service was created via kubectl or API).

Check Item 4: NAT Gateway and Port

Generally, no EIP is configured for the backend server of NAT. Otherwise, exceptions such as network packet loss may occur.

Check Item 5: Whether the Security Group of the Node Where the Container Is Located Allows Access

Log in to the management console and choose Service List > Networking > Virtual Private Cloud. On the Network console, choose Access Control > Security Groups, locate the security group rule of the CCE cluster, and modify and strengthen the security group rule.

• CCE cluster:

  The security group name of the node is {Cluster name}-cce-node-{Random characters}.

• CCE Turbo cluster:

  The security group name of the node is {Cluster name}-cce-node-{Random characters}.

  The name of the security group associated with the containers is {Cluster name}-cce-eni-{Random characters}.

Check the following:

• IP address, port, and protocol of an external request to access the workloads in the cluster. They must be allowed in the inbound rules of the cluster security group.
• IP address, port, and protocol of a request sent by a workload to visit external applications outside the cluster. They must be allowed in the outbound rules of the cluster security group.

For details about security group configuration, see How Do I Modify Cluster Security Group Rules?