Compute
Elastic Cloud Server
Huawei Cloud Flexus
Bare Metal Server
Auto Scaling
Image Management Service
Dedicated Host
FunctionGraph
Cloud Phone Host
Huawei Cloud EulerOS
Networking
Virtual Private Cloud
Elastic IP
Elastic Load Balance
NAT Gateway
Direct Connect
Virtual Private Network
VPC Endpoint
Cloud Connect
Enterprise Router
Enterprise Switch
Global Accelerator
Management & Governance
Cloud Eye
Identity and Access Management
Cloud Trace Service
Resource Formation Service
Tag Management Service
Log Tank Service
Config
OneAccess
Resource Access Manager
Simple Message Notification
Application Performance Management
Application Operations Management
Organizations
Optimization Advisor
IAM Identity Center
Cloud Operations Center
Resource Governance Center
Migration
Server Migration Service
Object Storage Migration Service
Cloud Data Migration
Migration Center
Cloud Ecosystem
KooGallery
Partner Center
User Support
My Account
Billing Center
Cost Center
Resource Center
Enterprise Management
Service Tickets
HUAWEI CLOUD (International) FAQs
ICP Filing
Support Plans
My Credentials
Customer Operation Capabilities
Partner Support Plans
Professional Services
Analytics
MapReduce Service
Data Lake Insight
CloudTable Service
Cloud Search Service
Data Lake Visualization
Data Ingestion Service
GaussDB(DWS)
DataArts Studio
Data Lake Factory
DataArts Lake Formation
IoT
IoT Device Access
Others
Product Pricing Details
System Permissions
Console Quick Start
Common FAQs
Instructions for Associating with a HUAWEI CLOUD Partner
Message Center
Security & Compliance
Security Technologies and Applications
Web Application Firewall
Host Security Service
Cloud Firewall
SecMaster
Anti-DDoS Service
Data Encryption Workshop
Database Security Service
Cloud Bastion Host
Data Security Center
Cloud Certificate Manager
Edge Security
Situation Awareness
Managed Threat Detection
Blockchain
Blockchain Service
Web3 Node Engine Service
Media Services
Media Processing Center
Video On Demand
Live
SparkRTC
MetaStudio
Storage
Object Storage Service
Elastic Volume Service
Cloud Backup and Recovery
Storage Disaster Recovery Service
Scalable File Service Turbo
Scalable File Service
Volume Backup Service
Cloud Server Backup Service
Data Express Service
Dedicated Distributed Storage Service
Containers
Cloud Container Engine
SoftWare Repository for Container
Application Service Mesh
Ubiquitous Cloud Native Service
Cloud Container Instance
Databases
Relational Database Service
Document Database Service
Data Admin Service
Data Replication Service
GeminiDB
GaussDB
Distributed Database Middleware
Database and Application Migration UGO
TaurusDB
Middleware
Distributed Cache Service
API Gateway
Distributed Message Service for Kafka
Distributed Message Service for RabbitMQ
Distributed Message Service for RocketMQ
Cloud Service Engine
Multi-Site High Availability Service
EventGrid
Dedicated Cloud
Dedicated Computing Cluster
Business Applications
Workspace
ROMA Connect
Message & SMS
Domain Name Service
Edge Data Center Management
Meeting
AI
Face Recognition Service
Graph Engine Service
Content Moderation
Image Recognition
Optical Character Recognition
ModelArts
ImageSearch
Conversational Bot Service
Speech Interaction Service
Huawei HiLens
Video Intelligent Analysis Service
Developer Tools
SDK Developer Guide
API Request Signing Guide
Terraform
Koo Command Line Interface
Content Delivery & Edge Computing
Content Delivery Network
Intelligent EdgeFabric
CloudPond
Intelligent EdgeCloud
Solutions
SAP Cloud
High Performance Computing
Developer Services
ServiceStage
CodeArts
CodeArts PerfTest
CodeArts Req
CodeArts Pipeline
CodeArts Build
CodeArts Deploy
CodeArts Artifact
CodeArts TestPlan
CodeArts Check
CodeArts Repo
Cloud Application Engine
MacroVerse aPaaS
KooMessage
KooPhone
KooDrive
Help Center/ Cloud Container Engine/ FAQs/ Networking/ Network Exception Troubleshooting/ Why the Ingress Cannot Be Accessed Outside the Cluster?

Why the Ingress Cannot Be Accessed Outside the Cluster?

Updated on 2024-11-13 GMT+08:00

Ingresses forward requests based on layer-7 HTTP and HTTPS protocols. As an entry of cluster traffic, ingresses use domain names and paths to achieve finer granularities. After an ingress is added to a cluster, the cluster may fail to be accessed. This section describes how to locate the fault when an ingress fails to be added or cannot be accessed. Before rectifying ingress issues, read the following precautions and perform a self-check:

NOTICE:
  • If the host address is specified in the ingress, the IP address cannot be used for access.
  • Check the node security group of the cluster and ensure that the service ports in the range of 30000 to 32767 are accessible to all network segments for inbound traffic.

Troubleshooting Process

This section provides an overview of troubleshooting ingress external access exceptions, as shown in Figure 1.

Figure 1 Overview of troubleshooting ingress external access exceptions
  1. Checking Whether the Exception Is Caused by the Ingress.

    Check whether the problem is caused by the ingress. Ensure that the external domain name resolution is normal, the security group rules are correct, and the service and workload corresponding to the ingress are working properly.

  2. Checking the Ingress Status.

    When the service and workload are normal, ensure that the load balancer on which the ingress depends is normal. If the ingress is of the Nginx type, ensure that the nginx-ingress add-on is normal.

  3. Checking Whether the Ingress Is Configured Correctly.
    If the preceding check results are normal, the ingress configuration may be incorrect.
    • Check whether the parameters for interconnecting with the load balancer are correct.
    • Check whether the Service configuration is correct.
    • Check whether the forwarding configuration is correct.
  4. Checking Certificate.

    If HTTPS access is enabled on the ingress, you also need to check whether the fault is caused by incorrect certificate configuration. You can use the same load balancer to create an HTTP ingress. If the access is normal, the HTTPS certificate may be faulty.

  5. If the fault persists, capture packets for analysis or submit a service ticket for help.

Checking Whether the Exception Is Caused by the Ingress

Check whether the access exception is caused by the ingress. If the domain name resolution exception, security group rule error, service exception, or workload exception occurs, the ingress access may fail.

The following check sequence complies with the rules from outside to inside:

  1. Check whether the domain name resolution or security group rules are normal.

    1. Run the following command to check whether record sets of the domain name take effect on the authoritative DNS server:
      nslookup -qt= Type Domain name Authoritative DNS address
    2. Check the security group rules of the cluster nodes and ensure that the service ports in the 30000–32767 range are accessible to all network segments for inbound traffic. For details about how to harden the security group, see How Can I Configure a Security Group Rule in a Cluster?.

  2. Check whether the Service can access services in the container.

    You can create a pod in the cluster and use the cluster IP address to access the Service. If the Service type is NodePort, you can also use EIP:Port to access the service over the Internet.
    1. Use kubectl to connect to the cluster and query the Service in the cluster.
      # kubectl get svc
      NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
      kubernetes   ClusterIP   10.247.0.1       <none>        443/TCP        34m
      nginx        ClusterIP   10.247.138.227   <none>        80/TCP         30m
    2. Create a pod and log in to the container.
      kubectl run -i --tty --image nginx:alpine test --rm /bin/sh
    3. Run the curl command to access ClusterIP address:Port of the Service to check whether the Service in the cluster is accessible.
      curl 10.247.138.227:80

      If the Service can be accessed, the backend workload status is normal. It can be preliminarily determined that the exception is caused by the ingress. For details, see Checking the Ingress Status.

      If the Service access is abnormal, check the workload status to determine the cause.

  3. Check whether the workload status is normal.

    If the workload is normal but the Service cannot be accessed, the exception may be caused by the Service. Check the Service configuration. For example, check whether the container port is correctly configured to an open service port of the container.

    If the workload is normal but the access result is not as expected, check the service code running in the container.

Checking the Ingress Status

CCE supports two types of ingresses. The Ngnix Ingress Controller is provided by the open source community and needs to be maintained by installing the add-on in the cluster. The ELB Ingress Controller runs on the master node and is maintained by a dedicated Huawei Cloud team.

  1. If you use an Nginx ingress, you need to install the NGINX Ingress Controller add-on in the cluster. If you use a LoadBalancer ingress, skip this step.

    Go to Add-ons and check whether the NGINX Ingress Controller add-on runs properly. Ensure that there are enough node resources in the cluster. If not, the add-on pods cannot be scheduled to nodes.

  2. Go to the ELB console to check the ELB status.

    • ELB ingress

      The access port can be customized. Check whether the listener and backend server group created on the ELB are not deleted or modified.

      When creating an ELB ingress, you can choose Auto Create on the console to automatically create a load balancer. Do not modify the load balancer. Otherwise, ingress exceptions may be caused.

    • Nginx ingress

      The access ports are fixed to 80 and 443. Custom ports are not supported. Installing the nginx-ingress add-on occupies both ports 80 and 443. Do not delete them. Otherwise, you need to reinstall the add-on.

    You can also determine whether the fault is caused by the load balancer based on the error code. If the following error code is displayed, there is a high probability that the fault is caused by the load balancer. In this case, you need to pay special attention to the load balancer.

Checking Whether the Ingress Is Configured Correctly

If the preceding check items are normal, check whether the exception is caused by parameter settings. When using kubectl to create an ingress, a large number of parameters need to be set, which is prone to errors. You are advised to use the console to create ingresses and set parameters as required to automatically filter out load balancers and Services that do not meet requirements. This effectively prevents incorrect formats or missing of key parameters.

Check the ingress configuration according to the following steps:
  • Check whether the parameters for interconnecting with the load balancer are correct.

    Load balancers are defined by parameters in the annotations field. Kubernetes does not verify the parameters in the annotations field when creating resources. If key parameters are incorrect or missing, an ingress can be created but cannot be accessed.

    The following problems frequently occur:

    • The interconnected ELB load balancer is not in the same VPC as the cluster.
    • Key fields kubernetes.io/elb.id, kubernetes.io/elb.ip, kubernetes.io/ingress.class, and kubernetes.io/elb.port in annotations are missing when an ELB ingress is added to connect to an existing ELB load balancer.
    • When you add an Nginx ingress, the nginx-ingress add-on is not installed. As a result, the ELB connection is unavailable.
    • When you add an Nginx ingress, key fields kubernetes.io/ingress.class and kubernetes.io/elb.port are missing in annotations.
    • When you add an Nginx ingress, the kubernetes.io/elb.port field does not support custom ports. If HTTP is used, the value is fixed to 80. If HTTPS is used, the value is fixed to 443.
  • Check whether Service is configured correctly.
    • Check whether the Service type connected to the ingress is correct. For details about the Service types supported by the ingress, see the following table.
      Table 1 Services supported by LoadBalancer ingresses

      Cluster Type

      ELB Type

      ClusterIP

      NodePort

      CCE standard cluster

      Shared load balancer

      Not supported

      Supported

      Dedicated load balancer

      Not supported (Failed to access the dedicated load balancers because no ENI is bound to the associated pod of the ClusterIP Service.)

      Supported

      CCE Turbo cluster

      Shared load balancer

      Not supported

      Supported

      Dedicated load balancer

      Supported

      Not supported (Failed to access the dedicated load balancers because an ENI has been bound to the associated pod of the NodePort Service.)

      Table 2 Services supported by Nginx ingress

      Cluster Type

      ELB Type

      ClusterIP

      NodePort

      CCE standard cluster

      Shared load balancer

      Supported

      Supported

      Dedicated load balancer

      Supported

      Supported

      CCE Turbo cluster

      Shared load balancer

      Supported

      Supported

      Dedicated load balancer

      Supported

      Supported

    • Check whether the access port number of the Service is correct. The access port number (port field) of the Service must be different from the container port number (targetPort field).
  • Check whether the forwarding configuration is correct.
    • The forwarding URL added must exist in the backend application. Otherwise, the forwarding fails.

      For example, the default access URL of the Nginx application is /usr/share/nginx/html. When adding /test to the ingress forwarding policy, ensure that your Nginx application contains the same URL, that is, /usr/share/nginx/html/test, otherwise, 404 is returned.

      NOTE:

      When using the Nginx Ingress Controller, you can add the rewrite comment to the annotations field for redirection to rewrite the path that does not exist in the Service to avoid the error that the access path does not exist. For details, see Rewrite.

    • If the domain name (host) is specified when an ingress is created, the ingress cannot be accessed using an IP address.

Checking Certificate

The ingress secret certificate type of CCE is IngressTLS or kubernetes.io/tls. If the certificate type is incorrect, the ingress cannot create a listener on the load balancer. As a result, the ingress access becomes abnormal.

  1. Remove HTTPS parameters from YAML and create an HTTP ingress to check whether the ingress can be accessed.

    If the HTTP access is normal, check whether the HTTPS secret certificate is correct.

  2. Check whether the secret type is correct. Check whether the secret type is IngressTLS or kubernetes.io/tls.

    # kubectl get secret
    NAME                  TYPE                                  DATA   AGE
    ingress               IngressTLS                            2      36m

  3. Create test certificates to rectify the certificate fault.

    # openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN={YOUR_HOST}/O={YOUR_HOST}"

  4. Use the test certificates tls.key and tls.crt to create a secret and check whether the secret can be accessed normally. The following example shows how to create an IngressTLS secret.

    Specifically, the IngressTLS secret is created using kubectl:

    kind: Secret
    apiVersion: v1
    type: IngressTLS
    metadata:
      name: ingress
      namespace: default
    data:
      tls.crt: LS0tLS1CRU*****FURS0tLS0t
      tls.key: LS0tLS1CRU*****VZLS0tLS0=
    NOTE:

    In the preceding information, tls.crt and tls.key are only examples. Replace them with the actual files. The values of tls.crt and tls.key are the content encrypted using Base64.

We use cookies to improve our site and your experience. By continuing to browse our site you accept our cookie policy. Find out more

Feedback

Feedback

Feedback

0/500

Selected Content

Submit selected content with the feedback