Load-aware Scheduling

Resource metrics can be provided by Metrics API only when local data storage is enabled for the Cloud Native Cluster Monitoring add-on.

Check whether Metrics API has been enabled for the cluster. If it is enabled, you can skip this step.

kubectl get APIServices | grep v1beta1.metrics.k8s.io

If any command output is displayed, Metrics API is enabled. Skip this step and go to the next step to add metric collection rules.

If no Metrics API is found, you can manually create an APIService object to enable it.

1. Create a file named metrics-apiservice.yaml. Example file content:

   apiVersion: apiregistration.k8s.io/v1
   kind: APIService
   metadata:
     labels:
       app: custom-metrics-apiserver
       release: cceaddon-prometheus
     name: v1beta1.metrics.k8s.io
   spec:
     group: metrics.k8s.io
     groupPriorityMinimum: 100
     insecureSkipTLSVerify: true
     service:
       name: custom-metrics-apiserver
       namespace: monitoring
       port: 443
     version: v1beta1
     versionPriority: 100

2. Create an APIService object.

   kubectl create -f metrics-apiservice.yaml

3. Check whether Metrics API is enabled for the cluster.

   kubectl get APIServices | grep v1beta1.metrics.k8s.io

   

   After Metrics API is enabled, if you need to uninstall the Cloud Native Cluster Monitoring add-on, run the following kubectl and delete the APIService object. Otherwise, the residual APIService resources will cause the Kubernetes Metrics Server add-on to fail to be installed.

   kubectl delete APIService v1beta1.metrics.k8s.io

1. Modify the user-adapter-config configuration item.

   kubectl edit configmap user-adapter-config -n monitoring

2. Add collection rules to the rules field.
   In the following example rules, the rules in red are new ones and those in black are existing ones:

   ...
   data:
     config.yaml: >
       rules:
       - seriesQuery: '{__name__=~"node_cpu_seconds_total"}'
         resources:
           overrides:
             instance:
               resource: node
         name:
           matches: node_cpu_seconds_total
           as: node_cpu_usage_avg
         metricsQuery: avg_over_time((1 - avg (irate(<<.Series>>{mode="idle"}[5m])) by (instance))[10m:30s])
       - seriesQuery: '{__name__=~"node_memory_MemTotal_bytes"}'
         resources:
           overrides:
             instance:
               resource: node
         name:
           matches: node_memory_MemTotal_bytes
           as: node_memory_usage_avg
         metricsQuery: avg_over_time(((1-node_memory_MemAvailable_bytes/<<.Series>>))[10m:30s])
       resourceRules:
         cpu:
           containerQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>,container!="",pod!=""}[1m])) by (<<.GroupBy>>)
           nodeQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>, id='/'}[1m])) by (<<.GroupBy>>)
           resources:
             overrides:
               instance:
                 resource: node
               namespace:
                 resource: namespace
               pod:
                 resource: pod
           containerLabel: container
         memory:
           containerQuery: sum(container_memory_working_set_bytes{<<.LabelMatchers>>,container!="",pod!=""}) by (<<.GroupBy>>)
           nodeQuery: sum(container_memory_working_set_bytes{<<.LabelMatchers>>,id='/'}) by (<<.GroupBy>>)
           resources:
             overrides:
               instance:
                 resource: node
               namespace:
                 resource: namespace
               pod:
                 resource: pod
           containerLabel: container
         window: 1m
   ...

   • Rules for collecting the average CPU usage
     • node_cpu_usage_avg: average CPU usage of nodes. The name of this metric cannot be changed.
     • metricsQuery: avg_over_time((1 - avg (irate(<<.Series>>{mode="idle"}[5m])) by (instance))[10m:30s]): statement for obtaining nodes' average CPU usage.

       metricsQuery indicates to obtain the average CPU usage of all nodes in the target cluster in the last 10 minutes. To change the period, for example, to the last 5 or 30 minutes, change 10m in red to 5m or 30m.

   • Rules for collecting the average memory usage
     • node_memory_usage_avg: average memory usage of nodes. The name of this metric cannot be changed.
     • metricsQuery: avg_over_time(((1-node_memory_MemAvailable_bytes/<<.Series>>))[10m:30s]): statement for obtaining nodes' average memory usage.

       metricsQuery indicates to obtain the average memory usage of all nodes in the target cluster in the last 10 minutes. To change the period, for example, to the last 5 or 30 minutes, change 10m in red to 5m or 30m.

3. Redeploy the custom-metrics-apiserver workload in the monitoring namespace.

   kubectl rollout restart deployment custom-metrics-apiserver -n monitoring

4. Verify that the custom rules are configured successfully.
   1. Run the following command. If the custom metric information is returned, the metric collection configuration on Prometheus is successful.

      kubectl get --raw=/apis/custom.metrics.k8s.io/v1beta1

      

   2. Query information about nodes in the cluster.

      kubectl get nodes 

      Run the following command on any node. Replace xxxx with the obtained value of node_name. If you want to query the resource information of all nodes, replace xxxx with an asterisk (*).

      kubectl get --raw=/apis/custom.metrics.k8s.io/v1beta1/nodes/xxxx/node_cpu_usage_avg

      Information similar to the following is displayed.

      

1. Run the following commands to install prometheus-adapter:

   git clone https://github.com/kubernetes-sigs/prometheus-adapter.git
   cd prometheus-adapter/deploy/manifests
   kubectl apply -f .

2. Modify the configuration for prometheus-adapter to connect to prometheus-server.

   kubectl edit deployment prometheus-adapter -n monitoring

   Change the value of prometheus-url as follows:

   • Change HTTPS to HTTP.
   • Change the default domain name to the IP address and port of Prometheus Service. You can run the kubectl get service -n monitoring command to query the IP address and port.

   ...
         containers:
           - name: prometheus-adapter
             image: registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.12.0
             args:
               - --cert-dir=/var/run/serving-cert
               - --config=/etc/adapter/config.yaml
               - --prometheus-url=http://10.21.72.124:9090/
               - --secure-port=6443
               - --tls-cipher-suites=TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA
   ...

3. Verify that the native node_cpu_seconds_total and node_memory_MemAvailable_bytes metrics can be obtained on Prometheus.

   

Check whether Custom Metrics API has been enabled for the cluster. If it is enabled, you can skip this step.

kubectl get APIServices | grep v1beta1.custom.metrics.k8s.io

If any command output is displayed, Custom Metrics API has been enabled. If no command output is displayed, perform the following operations to register the APIService object. (Note that the Service name and namespace must be consistent with those in the actual installation environment.) The following uses the default values of kube-prometheus.

1. Create a file named custom-metrics-apiservice.yaml. Example file content:

   apiVersion: apiregistration.k8s.io/v1
   kind: APIService
   metadata:
     labels:
       app.kubernetes.io/component: metrics
       app.kubernetes.io/name: prometheus-adapter
       app.kubernetes.io/part-of: prometheus-adapter
     name: v1beta1.custom.metrics.k8s.io
   spec:
     group: custom.metrics.k8s.io
     groupPriorityMinimum: 100
     insecureSkipTLSVerify: true
     service:
       name: prometheus-adapter
       namespace: monitoring
       port: 443
     version: v1beta1
     versionPriority: 100

2. Create an APIService object.

   kubectl create -f custom-metrics-apiservice.yaml

3. Check whether Custom Metrics API is enabled for the cluster.

   kubectl get APIServices | grep v1beta1.custom.metrics.k8s.io

1. Modify the adapter-config configuration item.

   kubectl edit configmap adapter-config -n monitoring

2. Add collection rules to the rules field.
   Example collection rules:

   ...
   data:
     config.yaml: >
       rules:
       - seriesQuery: '{__name__=~"node_cpu_seconds_total"}'
         resources:
           overrides:
             instance:
               resource: node
         name:
           matches: node_cpu_seconds_total
           as: node_cpu_usage_avg
         metricsQuery: avg_over_time((1 - avg (irate(<<.Series>>{mode="idle"}[5m])) by (instance))[10m:30s])
       - seriesQuery: '{__name__=~"node_memory_MemTotal_bytes"}'
         resources:
           overrides:
             instance:
               resource: node
         name:
           matches: node_memory_MemTotal_bytes
           as: node_memory_usage_avg
         metricsQuery: avg_over_time(((1-node_memory_MemAvailable_bytes/<<.Series>>))[10m:30s])
   ...

3. Redeploy the prometheus-adapter workload in the monitoring namespace.

   kubectl rollout restart deployment prometheus-adapter -n monitoring

4. Verify that the custom rules are configured successfully.
   1. Run the following command. If the custom metric information is returned, the metric collection configuration on Prometheus is successful.

      kubectl get --raw=/apis/custom.metrics.k8s.io/v1beta1

      

   2. Query information about nodes in the cluster.

      kubectl get nodes 

      Run the following command on any node. Replace xxxx with the obtained value of node_name. If you want to query the resource information of all nodes, replace xxxx with an asterisk (*).

      kubectl get --raw=/apis/custom.metrics.k8s.io/v1beta1/nodes/xxxx/node_cpu_usage_avg

      Information similar to the following is displayed.

      

1. Log in to the CCE console.
2. Click the cluster name to access the cluster console. Choose Settings in the navigation pane. In the right pane, click the Scheduling tab.
3. In Expert mode, click Try Now.

   

4. Modify the parameter settings in expert mode and configure the load-aware scheduling policy. The parameters in red are added to enable load-aware scheduling. The details are as follows:

   admission_kube_api_qps: 200
   admissions: /jobs/mutate,/jobs/validate,/podgroups/mutate,/pods/validate,/pods/mutate,/queues/mutate,/queues/validate,/eas/pods/mutate,/eas/pods/validate,/npu/jobs/validate,/resource/validate,/resource/mutate,/workloadbalancer/balancer/validate,/workloadbalancer/balancerpolicytemplate/validate
   annotations: {}
   colocation_enable: "true"
   controller_kube_api_qps: 200
   default_scheduler_conf:
     actions: allocate, backfill, preempt
     metrics:
       interval: 30s
       type: prometheus_adaptor
     tiers:
       - plugins:
           - name: priority
           - enableJobStarving: false
             enablePreemptable: false
             name: gang
           - name: conformance
           - name: oversubscription
       - plugins:
           - enablePreemptable: false
             name: drf
           - name: predicates
           - name: nodeorder
           - arguments:
                cpu.weight: 1
                memory.weight: 1
                thresholds:
                   cpu: 80
                   mem: 80
               usage.weight: 5
             enablePredicate: true
             name: usage
       - plugins:
           - name: cce-gpu-topology-predicate
           - name: cce-gpu-topology-priority
           - name: xgpu
       - plugins:
           - name: nodelocalvolume
           - name: nodeemptydirvolume
           - name: nodeCSIscheduling
           - name: networkresource
   deschedulerPolicy:
     profiles:
       - name: ProfileName
         pluginConfig:
           - args:
               nodeFit: true
             name: DefaultEvictor
           - args:
               evictableNamespaces:
                 exclude:
                   - kube-system
               thresholds:
                 cpu: 20
                 memory: 20
             name: HighNodeUtilization
           - args:
               evictableNamespaces:
                 exclude:
                   - kube-system
               metrics:
                 type: prometheus_adaptor
               nodeFit: true
               targetThresholds:
                 cpu: 80
                 memory: 85
               thresholds:
                 cpu: 30
                 memory: 30
             name: LoadAware
         plugins:
           balance:
             enabled: null
   descheduler_enable: "false"
   deschedulingInterval: 10m
   enable_workload_balancer: false
   oversubscription_method: nodeResource
   oversubscription_profile_period: 300
   oversubscription_ratio: 60
   recommendation_enable: ""
   scheduler_kube_api_qps: 200
   update_pod_status_qps: 50
   workload_balancer_score_annotation_key: ""
   workload_balancer_third_party_types: ""

   The key parameters for cluster load-aware scheduling are as follows:

   • usage.weight: indicates the weight of the load-aware scheduling policy. A larger value indicates a higher weight of the load-aware policy in overall scheduling.
   • cpu.weight: indicates the CPU weight. A larger value indicates CPU resources will be preferentially balanced.
   • memory.weight: indicates the memory weight. A larger value indicates memory resources will be preferentially balanced.
   • thresholds:
     • cpu: indicates the actual CPU load threshold.
     • mem: indicates the actual memory load threshold.
   • enablePredicate: indicates the actual effective mode of the load threshold.
     • If the value is true, hard constraints are enabled. When the actual CPU or memory load of a node reaches the threshold, no new tasks can be scheduled to this node.
     • If the value is false, soft constraints are enabled. When the actual CPU or memory load of a node reaches the threshold, new tasks will be preferentially allocated to underutilized nodes, but this node can still be scheduled.