Configuring an Advanced Forwarding Policy for a LoadBalancer Ingress
When a LoadBalancer ingress is created in a CCE cluster, CCE automatically provisions a listener for the load balancer associated with the ingress on ELB and adds forwarding policies. By default, the ingress routes incoming client requests based on the domain name and path and directs the requests through the listener to the appropriate backend server, either a node or a pod, for processing. During LoadBalancer ingress creation, CCE allows you to define custom forwarding rules and actions for flexible, fine-grained traffic distribution.
An advanced forwarding policy operates as follows:
- The client initiates a request to the LoadBalancer ingress.
- The load balancer evaluates the request against the forwarding rules defined in the advanced forwarding policy.
- The load balancer forwards the request to the appropriate backend server based on the action specified in the forwarding rule.
- The backend server returns a response to the client.
For details about the forwarding rules and actions supported by advanced forwarding policies, see Forwarding Rules and Forwarding Actions.
Prerequisites
- A CCE standard or Turbo cluster is available, and the cluster version meets the requirements.
- The cluster can be accessed using kubectl. For details, see Accessing a Cluster Using kubectl.
Notes and Constraints
- This feature is only available when dedicated load balancers are used.
- This feature relies on the ELB advanced forwarding policies. Once enabled, the priority is not determined by the domain name or path matching. You can customize the forwarding policy priority as needed. For details about the forwarding policy priority, see Forwarding Policy Priorities of LoadBalancer Ingresses.
CCE allows you to configure header writing/removal, rewriting, and limiting on requests, based on ELB's advanced forwarding for LoadBalancer ingresses. These functions are currently being rolled out, and availability depends on the actual regions that have been officially opened. To use some advanced forwarding actions that are not available on the console yet, submit a service ticket to enable required functions.
Forwarding Rules
LoadBalancer ingresses support advanced traffic routing by allowing you to configure forwarding rules using the kubernetes.io/elb.conditions.<svc-name> annotation. Incoming requests are evaluated against these rules. If the conditions are met, the requests are forwarded accordingly.
If you set up different rules for the same Service using the same annotation, they must all be matched for a request to go through. But if you set multiple values in the same rule, then the request only needs to match one of them. For example, if both Method and QueryString are configured, the traffic can be distributed only when both rules are met. However, if the Method value includes both GET and POST, the traffic can be distributed only when both rules are met and the request method is either GET or POST.
You can configure up to 10 advanced forwarding conditions (including domain names and paths) per ingress. Each individual value within a rule counts as one forwarding condition.
For example, if you define a domain name and a path for an ingress, there are two forwarding conditions, leaving eight remaining. If you then specify two HTTP methods, for example, GET and POST, two more forwarding conditions are used.
|
Forwarding Rule |
Description |
Reference |
Cluster Version |
|---|---|---|---|
|
HTTP request method |
Route HTTP requests based on the request type (such as GET and POST). If a request matches the specified rule, the defined forwarding action is used. An annotation example is as follows: kubernetes.io/elb.conditions.<svc-name>: |
[{
"type": "Method",
"methodConfig": {
"values": [
"GET",
"POST"
]
}
}] |
For details about the configuration example and parameters of advanced forwarding rules, see Scenario 1: Configuring Advanced Forwarding Rules for a LoadBalancer Ingress. |
|
|
HTTP request header |
Route requests based on the specific information (such as User-Agent and Content-Type) in HTTP request headers. If a request matches the specified rule, the defined forwarding action is used. An annotation example is as follows: kubernetes.io/elb.conditions.<svc-name>: |
[{
"type": "Header",
"headerConfig": {
"key": "gray-hello",
"values": [
"value1",
"value2"
]
}
}] |
||
|
Cookie |
Route requests based on the specified keys and values in cookies. If a request matches the specified rule, the defined forwarding action is used. An annotation example is as follows: kubernetes.io/elb.conditions.<svc-name>: |
[{
"type": "Header",
"headerConfig": {
"key": "gray-hello",
"values": [
"value1",
"value2"
]
}
}] |
||
|
Query string |
Route requests based on the specific parameters and values of the query strings in the URLs. If a request matches the specified rule, the defined forwarding action is used. An annotation example is as follows: kubernetes.io/elb.conditions.<svc-name>: |
[{
"type": "QueryString",
"queryStringConfig": {
"key": "querykey",
"values": [
"queryvalue"
]
}
}] |
||
|
CIDR block |
Route requests based on the source IP addresses from where the requests originate. If a request matches the specified rule, the defined forwarding action is used. An annotation example is as follows: kubernetes.io/elb.conditions.<svc-name>: |
[{
"type": "SourceIp",
"sourceIpConfig": {
"values": [
"192.168.0.0/16",
"172.16.0.0/16"
]
}
}] |
Forwarding Actions
LoadBalancer ingresses allow you to define multiple forwarding actions through annotations. This gives you flexible control over how requests are handled. The supported forwarding actions include:
- (Default) Forward to a backend server group: Additional actions like rewrite, CORS, write/remove header, and limit request are supported. Load balancers handle requests based on these additional actions before forwarding them to backend server groups. Among all the additional actions, limit request has the highest priority.
- Redirect to another listener: HTTP requests can be redirected to an HTTPS listener.
- Return a specific response body: Additional actions such as limit request are supported.
- Forward to a custom backend server group: This action cannot be used together with grayscale release, return a specific response body, or redirect to another URL.
|
Forwarding Action |
Additional Action |
Description |
Reference |
Cluster Version |
|---|---|---|---|---|
|
(Default) Forward to a backend server group |
Rewrite |
Rewrite the request paths. An annotation example is as follows: kubernetes.io/elb.rewrite-target: /rewritepath |
For details about the configuration example and parameters, see Configuring URL Rewriting for a LoadBalancer Ingress. |
|
|
CORS |
Support cross-origin access of resources. An annotation example is as follows: kubernetes.io/elb.cors-allow-origin: 'http://example.com' # The origin that can be accessed kubernetes.io/elb.cors-allow-headers: 'fake-header-1' # The allowed request header kubernetes.io/elb.cors-expose-headers: 'fake-header-2' # The response header to be exposed kubernetes.io/elb.cors-allow-methods: 'GET,POST' # The allowed HTTP request methods kubernetes.io/elb.cors-allow-credentials: 'true' # Credentials can be sent. kubernetes.io/elb.cors-max-age: '3600' # The cache duration of a pre-check request |
For details about the configuration example and parameters, see Configuring Cross-Origin Access for LoadBalancer Ingresses. |
|
|
|
Write/Remove header |
Write or delete the configured header in a request before accessing the backend server. An annotation example is as follows: kubernetes.io/elb.actions.<svc-name>: | [{ "type": "InsertHeader", "InsertHeaderConfig": { "key": "insertkey", "value_type": "USER_DEFINED", "value": "insertvalue" } }, { "type": "RemoveHeader", "RemoveHeaderConfig": { "key": "removekey" } } }] |
For details about the configuration example and parameters, see Example 1: Configuring Write/Remove Header for a LoadBalancer Ingress. |
||
|
Limit request |
Support rate limiting on requests. An annotation example is as follows: kubernetes.io/elb.actions.<svc-name>: |
[{
"type": "TrafficLimit",
"trafficLimitConfig": {
"QPS": 4,
"perSourceIpQps": 2,
"burst": 2
}
}] |
For details about the configuration example and parameters, see Example 3: Configuring Limit Request for a LoadBalancer Ingress. |
|
|
|
Redirect to another listener (HTTP requests redirected to an HTTPS listener) |
None |
Forward HTTP access requests to an HTTPS listener. An annotation example is as follows: kubernetes.io/elb.ssl-redirect: 'true' kubernetes.io/elb.listen-ports: '[{"HTTP":80},{"HTTPS":443}]' # Forward HTTP requests to an HTTPS port. |
For details about the configuration example and parameters, see Redirecting HTTP to HTTPS for a LoadBalancer Ingress. |
|
|
Redirect to another URL |
None |
Redirect specific requests to a specified URL and return a 3xx return code. An annotation example is as follows: kubernetes.io/elb.redirect-url: https://example.com/testa # Information about the redirection to a URL kubernetes.io/elb.redirect-url-code: '301' # Code returned after the ingress is redirected to a URL |
For details about the configuration example and parameters, see Configuring URL Redirection for a LoadBalancer Ingress. |
|
|
Return a specific response body |
None |
Directly return a specific response body and do not continue to forward the response to the backend server. An annotation example is as follows: kubernetes.io/elb.actions.<svc-name>: |
[{
"type": "FixedResponse",
"fixedResponseConfig": {
"contentType": "text/plain",
"statusCode": "503",
"messageBody": "503 error text"
}
}] |
For details about the configuration example and parameters, see Example 2: Configuring Return a Specific Response Body for a LoadBalancer Ingress. |
|
|
Limit request |
Support rate limiting on requests. An annotation example is as follows: kubernetes.io/elb.actions.<svc-name>: |
[{
"type": "FixedResponse",
"fixedResponseConfig": {
"contentType": "text/plain",
"statusCode": "503",
"messageBody": "503 error text"
}
},
{
"type": "TrafficLimit",
"trafficLimitConfig": {
"QPS": 4,
"perSourceIpQps": 2,
"burst": 2
}
}] |
For details about the configuration example and parameters, see Example 3: Configuring Limit Request for a LoadBalancer Ingress. |
|
|
|
Forward to a custom backend server group |
Forward LoadBalancer ingress requests to both pods in the CCE cluster (default backend server group) and an ECS group (custom backend server group) located outside the cluster. An annotation example is as follows: kubernetes.io/elb.actions.<svc-name>: | [{ "type": "ForwardPool", "forwardConfig": [{ "serviceName": "test", "servicePort": 80, "weight": 90, "type": "service" },{ "poolID": "53d8516e-xxxx-xxxx-xxxx-b15ffd6b3cca", "weight": 70, "type": "pool" }] }] |
For details about the configuration example and parameters, see Example 4: Configuring the Forward to a Custom Backend Server Group Action for a LoadBalancer Ingress. |
|
|
Scenario 1: Configuring Advanced Forwarding Rules for a LoadBalancer Ingress
LoadBalancer ingresses can route traffic based on custom HTTP request methods, request headers, query strings, and source CIDR blocks. In this example, traffic is routed to the svc-hello1 Service only when the request path is /hello1 and the following advanced forwarding rules are met:
- The request method is GET or POST.
- The key in the request header is gray-hello, and the value is value1 or value2.
- The key of the cookie is cookiekey1 and the value is cookievalue1, or the key of the cookie is cookiekey2 and the value is cookievalue2.
- The key of the query string is querykey, and the value is queryvalue.
- The source CIDR block is 192.168.0.0/16 or 172.16.0.0/16.
The operations and parameter settings are as follows:
- Use kubectl to access the cluster. For details, see Accessing a Cluster Using kubectl.
- Create a YAML file named ingress-test.yaml. The file name can be customized.
vi ingress-test.yaml
An example YAML file of an ingress created using an existing load balancer is as follows:apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: kubernetes.io/elb.class: performance kubernetes.io/elb.id: ab53c3b2-xxxx-xxxx-xxxx-5ac3eb2887be kubernetes.io/elb.port: '80' # The following annotation indicates that the advanced forwarding rules are configured for the svc-hello1 Service. The Service name must be the same as that in the backend field. kubernetes.io/elb.conditions.svc-hello1: | [ { "type": "Method", "methodConfig": { "values": [ "GET", "POST" ] } }, { "type": "Header", "headerConfig": { "key": "gray-hello", "values": [ "value1", "value2" ] } }, { "type": "Cookie", "cookieConfig": { "values": [ { "key": "cookiekey1", "value": "cookievalue1" }, { "key": "cookiekey2", "value": "cookievalue2" } ] } }, { "type": "QueryString", "queryStringConfig": { "key": "querykey", "values": [ "queryvalue" ] } }, { "type": "SourceIp", "sourceIpConfig": { "values": [ "192.168.0.0/16", "172.16.0.0/16" ] } } ] name: ingress-test spec: ingressClassName: cce rules: - http: paths: - path: /hello1 pathType: ImplementationSpecific backend: service: name: svc-hello1 port: number: 80Table 3 Annotation for advanced forwarding rules Parameter
Type
Description
kubernetes.io/elb.conditions.${svc_name}
String
Configure advanced forwarding rules. ${svc_name} indicates the Service name, which can contain a maximum of 48 characters.
If the annotation value is set to [], the advanced forwarding rules will be deleted.
The annotation value is a JSON array. For details, see Table 4.
CAUTION:If you set up different rules for the same condition array, they must all be matched for a request to go through. But if you set multiple values in the same rule, then the request only needs to match one of them. For example, if both Method and QueryString are configured, the traffic can be distributed only when both rules are met. However, if the Method value includes both GET and POST, the traffic can be distributed only when both rules are met and the request method is either GET or POST.
- Create an ingress.
kubectl create -f ingress-test.yaml
If information similar to the following is displayed, the ingress has been created:
ingress/ingress-test created
- Check the created ingress.
kubectl get ingress
If information similar to the following is displayed, the ingress has been created:NAME CLASS HOSTS ADDRESS PORTS AGE ingress-test cce * 121.**.**.** 80 10s
Scenario 2: Configuring Advanced Forwarding Actions for a LoadBalancer Ingress
The examples describe how to configure actions like write/remove header, return a specific response body, limit request, and forward to a custom backend server group for a LoadBalancer ingress.
Example 4: Configuring the Forward to a Custom Backend Server Group Action for a LoadBalancer Ingress
- Configuring a custom backend server group for a LoadBalancer ingress is subject to the capabilities of your ELB service. Before using this feature, submit a service ticket to enable the required ELB capabilities.
- The forward to a custom backend server group action cannot be used together with grayscale release, return a specific response body, or redirect to another URL.
- If you configure the forward to a custom backend server group action for a LoadBalancer ingress, the number field for the backend Services of the ingress will be invalid and the Service port name must be set to use-annotation. For details, see the example provided in this section.
- When configuring the forward to a custom backend server group action for a LoadBalancer ingress, ensure that each Service name and backend server group ID are unique.
- If you configure the kubernetes.io/elb.slowstart or kubernetes.io/elb.pool-protocol annotation when configuring the forward to a custom backend server group action for a LoadBalancer ingress, the annotation applies only to the default backend server groups automatically created by CCE. It does not apply to any custom backend server groups configured on ELB.
You can forward requests from an ingress to multiple custom backend server groups by weight. In this example, when a user accesses example.com, traffic is distributed between two backend server groups. Some is distributed to the backend server group associated with the test Service, assigned a weight of 90. The remaining traffic is directed to the load balancer's backend server group, which has a weight of 70.
The operations and parameter settings are as follows:
- Use kubectl to access the cluster. For details, see Accessing a Cluster Using kubectl.
- Create a YAML file named ingress-test.yaml. The file name can be customized.
vi ingress-test.yaml
An example YAML file of an ingress created using an existing load balancer is as follows:apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: kubernetes.io/elb.class: performance kubernetes.io/elb.id: ***** kubernetes.io/elb.port: "80" # The following annotation is used to configure a backend server group associated with a Service and a load balancer's backend server group for the ingress. forward-demo is the Service name in the backend field. kubernetes.io/elb.actions.forward-demo: | [{ "type": "ForwardPool", "forwardConfig": [{ "serviceName": "test", "servicePort": 80, "weight": 90, "type": "service" },{ "poolID": "*****", "weight": 70, "type": "pool" }] }] name: ingress-test namespace: default spec: ingressClassName: cce rules: - host: example.com http: paths: - backend: service: name: forward-demo # When configuring a custom backend server group, ensure that it has the same name as the Service specified in the annotation. In this example, the Service name is forward-demo. port: name: use-annotation # When you configure a custom backend server group, the original port.number field becomes invalid. You must instead specify the port.name parameter and set it to use-annotation. path: / pathType: ImplementationSpecific property: ingress.beta.kubernetes.io/url-match-mode: STARTS_WITHTable 11 Annotations for advanced forwarding actions Parameter
Type
Description
kubernetes.io/elb.actions.${svc_name}
String
Configure an advanced forwarding action for an ingress. ${svc_name} indicates the Service name, which can contain a maximum of 51 characters.
If the annotation value is set to [], all advanced forwarding actions will be deleted.
The value of an annotation for configuring a custom backend server group is a JSON string array. For details, see Table 12.
Table 12 Parameters for configuring a custom backend server group Parameter
Description
Example
type
The value is fixed at ForwardPool, indicating that a custom backend server group is configured for a load balancer.
NOTE:A maximum of one custom backend server group can be added to an advanced forwarding action annotation.
None
forwardConfig
NOTE:A maximum of five custom backend server groups can be added to this parameter.
- type: type of a custom backend server group, which can be service or pool.
- service: CCE automatically creates a backend server group for each Service and manages its lifecycle.
- pool: You need to create and maintain the backend server group on the ELB console.
- poolID: mandatory when type is set to pool.
This parameter specifies the ID of the backend server group under the target ingress's load balancer. The ID must be unique for each group.
You can go to the backend server group list, search for the backend server group of a load balancer using the name or ID of the load balancer associated with the ingress, and obtain the group ID.
- serviceName: mandatory when type is set to service.
Before configuring a Service name, ensure that it is unique and the Service exists in the cluster.
- servicePort: mandatory when type is set to service.
This parameter specifies a Service port.
- weight: weight for distributing traffic to each backend server group, ranging from 0 to 100.
{ "type": "ForwardPool", "forwardConfig": [{ "serviceName": "test", "servicePort": 80, "weight": 90, "type": "service" },{ "poolID": "53d8516e-xxxx-xxxx-xxxx-b15ffd6b3cca", "weight": 70, "type": "pool" }] } - type: type of a custom backend server group, which can be service or pool.
- Create an ingress.
kubectl create -f ingress-test.yaml
If information similar to the following is displayed, the ingress has been created:
ingress/ingress-test created
- Check the created ingress.
kubectl get ingress
If information similar to the following is displayed, the ingress has been created:NAME CLASS HOSTS ADDRESS PORTS AGE ingress-test cce example.com 121.**.**.** 80 10s
Combined Application Scenarios
Advanced forwarding rules and actions can be combined to meet various requirements. Some examples are shown in the table below.
|
Rules to Be Applied When an Ingress Is Accessed |
Actions to Be Performed |
Target Service (Backend Service) |
|---|---|---|
|
The following actions will be performed during forwarding:
|
test |
|
The following actions will be performed during forwarding:
|
test2 |
|
The following actions will be performed during forwarding:
|
test and test2 |
The YAML file example is as follows:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-demo
namespace: default
annotations:
kubernetes.io/elb.class: performance
kubernetes.io/elb.id: *****
kubernetes.io/elb.listener-master-ingress: default/ingress-demo
kubernetes.io/elb.port: '80'
# The following annotation applies to the test Service and is used to add multiple load balancer forwarding matching conditions to it.
kubernetes.io/elb.conditions.test: |
[
{
"type": "Method",
"methodConfig": {
"values": [
"GET",
"POST"
]
}
},
{
"type": "Header",
"headerConfig": {
"key": "gray-hello",
"values": [
"value1",
"value2"
]
}
},
{
"type": "Cookie",
"cookieConfig": {
"values": [
{
"key": "cookiekey1",
"value": "cookievalue1"
},
{
"key": "cookiekey2",
"value": "cookievalue2"
}
]
}
},
{
"type": "QueryString",
"queryStringConfig": {
"key": "querykey",
"values": [
"queryvalue"
]
}
},
{
"type": "SourceIp",
"sourceIpConfig": {
"values": [
"192.168.0.0/16",
"172.16.0.0/16"
]
}
}
]
# The following annotation applies to the test Service and is used to configure the write/remove header action for it.
kubernetes.io/elb.actions.test: |
[{
"type": "InsertHeader",
"InsertHeaderConfig": {
"key": "aa",
"value_type": "USER_DEFINED",
"value": "aa"
}
},{
"type": "RemoveHeader",
"RemoveHeaderConfig": {
"key": "dd"
}
}]
# The following annotation applies to the test2 Service and is used to configure the return a specific response body and limit request actions for it.
kubernetes.io/elb.actions.test2: |
[{
"type": "FixedResponse",
"fixedResponseConfig": {
"contentType": "text/plain",
"statusCode": "503",
"messageBody": "503 error text"
}},{
"type": "TrafficLimit",
"TrafficLimItConfig": {
"perSourceIpQps": 3,
"qps": 67
}}
]
# The following annotation applies to the test3 Service and is used to configure the write header, limit request, and forward to custom backend server groups (associated with test and test2) actions for it.
kubernetes.io/elb.actions.test3: |
[{
"type": "InsertHeader",
"InsertHeaderConfig": {
"key": "aa",
"value_type": "USER_DEFINED",
"value": "aa"
}
},{
"type": "ForwardPool",
"forwardConfig": [{
"serviceName": "test",
"servicePort": 80,
"weight": 90,
"type": "service"
},{
"serviceName": "test2",
"servicePort": 80,
"weight": 10,
"type": "service"
}]
},{
"type": "TrafficLimit",
"TrafficLimItConfig": {
"perSourceIpQps": 3,
"qps": 67
}}
]
spec:
ingressClassName: cce
rules:
- host: example.com
http:
paths:
- path: /path1
pathType: ImplementationSpecific
backend:
service:
name: test
port:
number: 80
property:
ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
- path: /path2
pathType: ImplementationSpecific
backend:
service:
name: test2
port:
number: 80
property:
ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
- path: /path3
pathType: ImplementationSpecific
backend:
service:
name: test3
port:
name: use-annotation # If the forward to custom backend server group ("type": "ForwardPool") action is specified, port.name must be set to use-annotation.
property:
ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot
