Copied.
Why Can't the ELB Address Be Used to Access Workloads in a Cluster?
Symptom
In a cluster (on a node or in a container), the ELB address cannot be used to access workloads.
Scenario 1: Service Affinity Is Node Level
Possible Cause
upstream connect error or disconnect/reset before headers. reset reason: connection failure Or curl: (7) Failed to connect to **.**.**.** port 900: Connection refused Or curl: (28) Failed to connect to **.**.**.** port 1122: Connection timed out
- For a multi-pod workload, ensure that all pods are accessible. Otherwise, there is a possibility that the access to the workload fails.
- The table lists only the scenarios where the access may fail. Other scenarios that are not listed in the table indicate that the access is normal.
- For CCE clusters using IPVS, set externalTrafficPolicy to Local for node-level Service affinity. For CCE Turbo clusters with DataPlane V2 disabled, node-level affinity is supported only when the Service backend uses hostNetwork.
Show Details
-
Server Service Type
Access Type
Client Request Source
Tunnel Network Cluster
VPC Network Cluster (DataPlane V2 disabled. If enabled, see Clusters with DataPlane V2 Enabled.)
CCE Turbo Cluster (DataPlane V2 disabled. If enabled, see Clusters with DataPlane V2 Enabled.)
NodePort Service
Public/Private network
Same node as service pod
Access via node IP + NodePort on the server node: Success
Access via node IP + NodePort on a non-server node: Failure
Access via node IP + NodePort on the server node: Success
Access via node IP + NodePort on a non-server node: Failure
Accessible
Different node from service pod
Access via node IP + NodePort on the server node: Success
Access via node IP + NodePort on another node: Failure
Access via node IP + NodePort on the server node: Success
Access via node private IP + NodePort on the client node: Success
Access via node EIP + NodePort on the client node: Failure
Access via node IP + NodePort on another node: Failure
Accessible
Other containers on the same node as the service pod
Access via node IP + NodePort on the server node: Success
Access via node IP + NodePort on a non-server node: Failure
Access via node EIP + NodePort on the server node: Success
Access via node private EIP + NodePort on the server node: Failure
Access via node IP + NodePort on a non-server node: Failure
Accessible
Other containers on a different node from the service pod
Access via node IP + NodePort on the server node: Success
Access via node private IP + NodePort on the client node: Success
Access via node IP + NodePort on another node: Failure
Access via node IP + NodePort on the server node: Success
Access via node private IP + NodePort on the client node: Success
Access via node IP + NodePort on another node: Failure
Accessible
LoadBalancer Service using a dedicated load balancer
Private network
Same node as service pod
Inaccessible
Inaccessible
Accessible
Other containers on the same node as the service pod
Inaccessible
Inaccessible
Accessible
-
- For CCE clusters using iptables, set externalTrafficPolicy to Local for node-level Service affinity. For CCE Turbo clusters with DataPlane V2 disabled, node-level affinity is supported only when the Service backend uses hostNetwork.
Show Details
-
Server Service Type
Access Type
Client Request Source
Tunnel Network Cluster
VPC Network Cluster (DataPlane V2 disabled. If enabled, see Clusters with DataPlane V2 Enabled.)
CCE Turbo Cluster (DataPlane V2 disabled. If enabled, see Clusters with DataPlane V2 Enabled.)
NodePort Service
Public/Private network
Same node as service pod
Access via node IP + NodePort on the server node: Success
Access via node IP + NodePort on a non-server node: Failure
Access via node IP + NodePort on the server node: Success
Access via node IP + NodePort on a non-server node: Failure
Accessible
Different node from service pod
Access via node IP + NodePort on the server node: Success
Access via node private IP + NodePort on the client node: Success
Access via node EIP + NodePort on the client node: Failure
Access via node IP + NodePort on another node: Failure
Access via node IP + NodePort on the server node: Success
Access via node private IP + NodePort on the client node: Success
Access via node EIP + NodePort on the client node: Failure
Access via node IP + NodePort on another node: Failure
Accessible
Other containers on the same node as the service pod
Access via node IP + NodePort on the server node: Success
Access via node IP + NodePort on a non-server node: Failure
Access via node EIP + NodePort on the server node: Success
Access via node private EIP + NodePort on the server node: Failure
Access via node IP + NodePort on a non-server node: Failure
Accessible
Other containers on a different node from the service pod
Access via node IP + NodePort on the server node: Success
Access via node IP + NodePort on a non-server node: Failure
Access via node IP + NodePort on the server node: Success
Access via node IP + NodePort on a non-server node: Failure
Accessible
LoadBalancer Service using a dedicated load balancer
Private network
Same node as service pod
Inaccessible
Inaccessible
Accessible
Other containers on the same node as the service pod
Inaccessible
Inaccessible
Accessible
-
- For clusters with DataPlane V2 enabled, set externalTrafficPolicy to Local for node-level Service affinity.
DataPlane V2 is supported only in CCE VPC-network clusters and Turbo clusters. Once enabled, it replaces kube-proxy entirely, eliminating the need to configure a service forwarding mode. For details, see DataPlane V2.
Show Details
-
Server Service Type
Access Type
Client Request Source
VPC Network Cluster (DataPlane V2 Enabled to Replace kube-proxy)
CCE Turbo Cluster (DataPlane V2 Enabled to Replace kube-proxy)
NodePort Service
Public/Private network
Same node as service pod
Access via node IP + NodePort on the server node: Success
Access via node private IP + NodePort on a non-server node: Success
Access via node EIP + NodePort on a non-server node: Failure
Access via node IP + NodePort on the server node: Success
Access via node private IP + NodePort on a non-server node: Success
Access via node EIP + NodePort on a non-server node: Failure
Different node from service pod
Access via node IP + NodePort on the server node: Success
Access via node private IP + NodePort on a non-server node: Success
Access via node EIP + NodePort on a non-server node: Failure
Access via node IP + NodePort on the server node: Success
Access via node private IP + NodePort on a non-server node: Success
Access via node EIP + NodePort on a non-server node: Failure
Other containers on the same node as the service pod
Access via node IP + NodePort on the server node: Success
Access via node IP + NodePort on another node: Failure
Access via node IP + NodePort on the server node: Success
Access via node IP + NodePort on another node: Failure
Other containers on a different node from the service pod
Access via node IP + NodePort on the server node: Success
Access via node private IP + NodePort on the client node: Success
Access via node IP + NodePort on another node: Failure
Access via node IP + NodePort on the server node: Success
Access via node private IP + NodePort on the client node: Success
Access via node IP + NodePort on another node: Failure
LoadBalancer Service using a dedicated load balancer
Private network
Same node as service pod
Inaccessible
Inaccessible
Other containers on the same node as the service pod
Inaccessible
Accessible
-
Solution
The following methods can be used to solve this problem:
- (Recommended) In the cluster, use the ClusterIP Service or service domain name for access.
- Set externalTrafficPolicy of the Service to Cluster, which means cluster-level service affinity. However, this affects source IP address preservation because the cluster performs NAT. As a result, backend applications are unable to obtain the client's actual IP address.
apiVersion: v1 kind: Service metadata: annotations: kubernetes.io/elb.class: union kubernetes.io/elb.autocreate: '{"type":"public","bandwidth_name":"cce-bandwidth","bandwidth_chargemode":"traffic","bandwidth_size":5,"bandwidth_sharetype":"PER","eip_type":"5_bgp","name":"james"}' labels: app: nginx name: nginx spec: externalTrafficPolicy: Cluster ports: - name: service0 port: 80 protocol: TCP targetPort: 80 selector: app: nginx type: LoadBalancer - Leveraging the pass-through feature of the Service, kube-proxy is bypassed when the ELB address is used for access. The ELB load balancer is accessed first, and then the workload. For details, see Configuring Passthrough Networking for a LoadBalancer Service.
- In a CCE standard cluster, after passthrough networking is configured using a dedicated load balancer, the private IP address of the load balancer cannot be accessed from the node where a workload pod resides or other pods on the same node as the workload.
- Passthrough networking is not supported for clusters of 1.15 or earlier.
- In IPVS, the passthrough settings of Services connected to the same load balancer must be the same.
- If node-level (local) service affinity is used, kubernetes.io/elb.pass-through is automatically set to onlyLocal to enable passthrough networking.
apiVersion: v1 kind: Service metadata: annotations: kubernetes.io/elb.pass-through: "true" kubernetes.io/elb.class: union kubernetes.io/elb.autocreate: '{"type":"public","bandwidth_name":"cce-bandwidth","bandwidth_chargemode":"traffic","bandwidth_size":5,"bandwidth_sharetype":"PER","eip_type":"5_bgp","name":"james"}' labels: app: nginx name: nginx spec: externalTrafficPolicy: Local ports: - name: service0 port: 80 protocol: TCP targetPort: 80 selector: app: nginx type: LoadBalancer
Scenario 2: A Load Balancer in a Cluster That Uses IPVS Is Reused
Possible Cause
In a cluster that uses IPVS for service forwarding, a load balancer address may become inaccessible under certain conditions.
- If a LoadBalancer ingress and a Service share the same load balancer, when the cluster tries to access the ingress, the ipvs-0 bridge will intercept the traffic and redirect it to the Service instead, causing access failures.
- If a Service in the local cluster and a Service in another cluster share the same load balancer, when the local cluster tries to access the Service in another cluster, the ipvs-0 bridge will intercept the traffic and redirect it to the Service in the local cluster instead, causing access failures.
Solution
Avoid these scenarios whenever possible. If a load balancer must be shared, enable passthrough networking for all Services associated with that load balancer so that traffic bypasses kube-proxy's IPVS forwarding. For details, see Configuring Passthrough Networking for a LoadBalancer Service.
Scenario 3: A LoadBalancer Service in a Cluster That Uses IPVS Listens on Ports
Possible Cause
A Service can use a load balancer to listen on a range of ports (for example, 30000–30005). However, inside the cluster, only the port defined in spec.ports (30000) is accessible. Requests to the other ports (30001–30005) fail because they are intercepted by the ipvs-0 bridge and redirected incorrectly.
Solution
If access to ports of a Service from within the cluster is needed, enable passthrough networking for the LoadBalancer Service that listens on ports so that traffic bypasses kube-proxy's IPVS forwarding. For details, see Configuring Passthrough Networking for a LoadBalancer Service.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot