Deploying NGINX Ingress Controller in Custom Mode

Background

Nginx Ingress Controller is a popular open source ingress controller in the industry and is widely used. Large-scale clusters require multiple ingress controllers to distinguish different traffic. For example, if some services in a cluster need to be accessed through a public network ingress, but some internal services cannot be accessed through a public network and can only be accessed by other services in the same VPC, you can deploy two independent Nginx Ingress Controllers and bind two different load balancers.

Figure 1 Application scenario of multiple Nginx ingresses

Solution

You can use either of the following solutions to deploy multiple NGINX ingress controllers in the same cluster.

• (Recommended) Install the NGINX Ingress Controller add-on and deploy multiple instances in the same cluster with just a few clicks. For details, see Nginx Ingress Controller.

  For clusters 1.23, the add-on of 2.2.52 or later must be installed. For clusters 1.23 or later, the add-on of 2.5.4 or later must be installed.

• Install the open source Helm package. The parameters to be configured in this solution are complex. You need to configure the ingress-class parameter (default value: nginx) to declare the listening ranges of different NGINX ingress controllers. In this way, when creating an ingress, you can select different NGINX ingress controllers to distinguish traffic.

Prerequisites

• Public images may need to be pulled during the installation. Therefore, bind an EIP to the node.

Constraints

• If multiple Nginx Ingress Controllers are deployed, each Controller needs to interconnect with a load balancer. Ensure that the load balancer has at least two listeners and ports 80 and 443 are not occupied by listeners. If dedicated load balancers are used, specify the network type.
• When the nginx-ingress template and image provided by the community are used, CCE does not provide additional maintenance for service loss caused by community software defects. Exercise caution when serving commercial purposes.

Deploying an Nginx Ingress Controller

You can perform the following steps to deploy another NGINX Ingress Controller in the cluster.

1. Obtain a chart.

   Go to the chart page, select a proper version, and download the Helm chart in .tgz format. This section uses the chart of version 4.4.2 as an example. This chart applies to CCE clusters of v1.21 or later. The configuration items in the chart may vary according to the version. The configuration in this section takes effect only for the chart of 4.4.2 version.

2. Upload the chart.
   1. Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane, choose App Templates and click Upload Chart in the upper right corner.
   2. Click Select File, select the chart to be uploaded, and click Upload.

   

3. Customize the value.yaml file.

   You can create a value.yaml configuration file on the local PC to configure workload installation parameters. During workload installation, you only need to import this configuration file for customized installation. Other unspecified parameters will use the default settings.

   The configuration content is as follows:

   controller:
     image:
       repository: registry.k8s.io/ingress-nginx/controller
       registry: ""
       image: ""
       tag: "v1.5.1"  # Controller version
       digest: ""
     ingressClassResource:
       name: ccedemo         # The name of each Ingress Controller in the same cluster must be unique and cannot be nginx or cce.
       controllerValue: "k8s.io/ingress-nginx-demo"  # The listening identifier of each Ingress Controller in the same cluster must be unique and cannot be set to k8s.io/ingress-nginx.
     ingressClass: ccedemo   # The name of each Ingress Controller in the same cluster must be unique and cannot be nginx or cce.
     service: 
       annotations: 
         kubernetes.io/elb.id: 5083f225-9bf8-48fa-9c8b-67bd9693c4c0     #ELB ID
         kubernetes.io/elb.class: performance  # This annotation is required only for dedicated load balancers.
     config:
       keep-alive-requests: 100
     extraVolumeMounts: # Mount the /etc/localtime file on the node to synchronize the time zone.
       - name: localtime
         mountPath: /etc/localtime
         readOnly: true
     extraVolumes:
       - name: localtime
         type: Hostpath
         hostPath:
           path: /etc/localtime 
     admissionWebhooks: # Disable webhook authentication.
       enabled: false
       patch:
         enabled: false
     resources: # Set the controller's resource limit, which can be customized.
       requests:
         cpu: 200m
         memory: 200Mi
   defaultBackend: # Set defaultBackend.
     enabled: true
     image: 
       repository: registry.k8s.io/defaultbackend-amd64
       registry: ""
       image: ""
       tag: "1.5"
       digest: ""

   For details about the preceding parameters, see Table 1.

4. Create a release.
   1. Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane, choose App Templates.
   2. In the list of uploaded charts, click Install.
   3. Set Release Name, Namespace, and Version.
   4. Click Select File next to Configuration File, select the YAML file created locally, and click Install.
   5. On the Releases tab page, you can view the installation status of the release.

      

Performing Verification

Deploy a workload and configure the newly deployed Nginx Ingress Controller to provide network access for the workload.

1. Create an Nginx workload.
   1. Log in to the CCE console, click the created cluster, choose Workloads in the navigation pane, and click Create Workload in the upper right corner.
   2. Enter the following information and click OK.

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: nginx
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: nginx
        strategy:
          type: RollingUpdate
        template:
          metadata:
            labels:
              app: nginx
          spec:
            containers:
            - image: nginx    # If an image from an open-source image registry is used, enter the image name. If you use an image in My Images, obtain the image path from SWR.
              imagePullPolicy: Always
              name: nginx
            imagePullSecrets:
            - name: default-secret
      ---
      apiVersion: v1
      kind: Service
      metadata:
        labels:
          app: nginx
        name: nginx
      spec:
        ports:
        - name: service0
          port: 80                 # Port for accessing a Service.
          protocol: TCP           # Protocol used for accessing a Service. The value can be TCP or UDP.
          targetPort: 80           # Port used by the service to access the target container. In this example, the Nginx image uses port 80 by default.
        selector:                   # Label selector. A Service selects a pod based on the label and forwards the requests for accessing the Service to the pod.
          app: nginx
        type: ClusterIP            # Type of a Service. ClusterIP indicates that a Service is only reachable from within the cluster.

2. Create an ingress and use the newly deployed Nginx Ingress Controller to provide network access.
   1. In the navigation pane, choose Services & Ingresses. Click the Ingresses tab and click Create from YAML in the upper right corner.
      

      When interconnecting with Nginx Ingress Controller that is not deployed using an add-on, you can create an ingress only through YAML.

   2. Enter the following information and click OK.
      For clusters of v1.23 or later:

      apiVersion: networking.k8s.io/v1 
      kind: Ingress 
      metadata: 
        name: ingress-test
        namespace: default 
      spec: 
        ingressClassName: ccedemo  # Enter the ingressClass of the newly created Nginx Ingress Controller.
        rules: 
        - host: foo.bar.com
          http: 
            paths: 
            - path: / 
              pathType: ImplementationSpecific   # The matching depends on IngressClass.
              backend: 
                service: 
                  name: nginx    # Replace it with the name of your target Service.
                  port: 
                    number: 80   # Replace it with the port of your target Service.
              property: 
                ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH  

      For clusters earlier than v1.23:

      apiVersion: networking.k8s.io/v1beta1
      kind: Ingress 
      metadata: 
        name: tomcat-t1 
        namespace: test 
        annotations: 
          kubernetes.io/ingress.class: ccedemo  # Enter the ingressClass of the newly created Nginx Ingress Controller.
      spec: 
        rules: 
          - host: foo.bar.com
            http: 
              paths: 
                - path: / 
                  pathType: ImplementationSpecific 
                  backend: 
                    serviceName: nginx  # Replace it with the name of your target Service.
                    servicePort: 80     # Replace it with the port of your target Service.
                  property: 
                    ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH

3. Log in to the cluster node and access the application through the Controller in the nginx-ingress add-on of the cluster and the newly deployed Nginx Ingress Controller service, respectively.
   • Use the new Nginx Ingress Controller service to access the application (the Nginx page is expected to be displayed). 192.168.114.60 is the ELB address of the new Nginx Ingress Controller service.

     curl -H "Host: foo.bar.com" http://192.168.114.60

     

   • Use the Controller service in the nginx-ingress add-on (404 is expected to be returned). 192.168.9.226 is the ELB address of the nginx-ingress add-on.

     curl -H "Host: foo.bar.com" http://192.168.9.226

     

Parameter Description

For details about more parameters, see ingress-nginx.