Load-aware Scheduling

Resource metrics can be provided by Metrics API only when local data storage is enabled for the Cloud Native Cluster Monitoring add-on.

Check whether Metrics API has been enabled for the cluster. If it is enabled, you can skip this step.

kubectl get APIServices | grep v1beta1.metrics.k8s.io

If any command output is displayed, Metrics API is enabled. Skip this step and go to the next step to add metric collection rules.

If no Metrics API is found, you can manually create an APIService object to enable it.

1. Create a file named metrics-apiservice.yaml. Example file content:

   apiVersion: apiregistration.k8s.io/v1
   kind: APIService
   metadata:
     labels:
       app: custom-metrics-apiserver
       release: cceaddon-prometheus
     name: v1beta1.metrics.k8s.io
   spec:
     group: metrics.k8s.io
     groupPriorityMinimum: 100
     insecureSkipTLSVerify: true
     service:
       name: custom-metrics-apiserver
       namespace: monitoring
       port: 443
     version: v1beta1
     versionPriority: 100

2. Create an APIService object.

   kubectl create -f metrics-apiservice.yaml

3. Check whether Metrics API is enabled for the cluster.

   kubectl get APIServices | grep v1beta1.metrics.k8s.io

   

   After Metrics API is enabled, if you need to uninstall the Cloud Native Cluster Monitoring add-on, run the following kubectl and delete the APIService object. Otherwise, the residual APIService resources will cause the Kubernetes Metrics Server add-on to fail to be installed.

   kubectl delete APIService v1beta1.metrics.k8s.io

1. Modify the user-adapter-config configuration item.

   kubectl edit configmap user-adapter-config -n monitoring

2. Add collection rules to the rules field.
   In the following example rules, the rules in red are new ones and those in black are existing ones:

   ...
   data:
     config.yaml: >
       rules:
       - seriesQuery: '{__name__=~"node_cpu_seconds_total"}'
         resources:
           overrides:
             instance:
               resource: node
         name:
           matches: node_cpu_seconds_total
           as: node_cpu_usage_avg
         metricsQuery: avg_over_time((1 - avg (irate(<<.Series>>{mode="idle"}[5m])) by (instance))[10m:30s])
       - seriesQuery: '{__name__=~"node_memory_MemTotal_bytes"}'
         resources:
           overrides:
             instance:
               resource: node
         name:
           matches: node_memory_MemTotal_bytes
           as: node_memory_usage_avg
         metricsQuery: avg_over_time(((1-node_memory_MemAvailable_bytes/<<.Series>>))[10m:30s])
       resourceRules:
         cpu:
           containerQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>,container!="",pod!=""}[1m])) by (<<.GroupBy>>)
           nodeQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>, id='/'}[1m])) by (<<.GroupBy>>)
           resources:
             overrides:
               instance:
                 resource: node
               namespace:
                 resource: namespace
               pod:
                 resource: pod
           containerLabel: container
         memory:
           containerQuery: sum(container_memory_working_set_bytes{<<.LabelMatchers>>,container!="",pod!=""}) by (<<.GroupBy>>)
           nodeQuery: sum(container_memory_working_set_bytes{<<.LabelMatchers>>,id='/'}) by (<<.GroupBy>>)
           resources:
             overrides:
               instance:
                 resource: node
               namespace:
                 resource: namespace
               pod:
                 resource: pod
           containerLabel: container
         window: 1m
   ...

   • Rules for collecting the average CPU usage
     • node_cpu_usage_avg: average CPU usage of nodes. The name of this metric cannot be changed.
     • metricsQuery: avg_over_time((1 - avg (irate(<<.Series>>{mode="idle"}[5m])) by (instance))[10m:30s]): statement for obtaining nodes' average CPU usage.

       metricsQuery indicates to obtain the average CPU usage of all nodes in the target cluster in the last 10 minutes. To change the period, for example, to the last 5 or 30 minutes, change 10m in red to 5m or 30m.

   • Rules for collecting the average memory usage
     • node_memory_usage_avg: average memory usage of nodes. The name of this metric cannot be changed.
     • metricsQuery: avg_over_time(((1-node_memory_MemAvailable_bytes/<<.Series>>))[10m:30s]): statement for obtaining nodes' average memory usage.

       metricsQuery indicates to obtain the average memory usage of all nodes in the target cluster in the last 10 minutes. To change the period, for example, to the last 5 or 30 minutes, change 10m in red to 5m or 30m.

3. Redeploy the custom-metrics-apiserver workload in the monitoring namespace.

   kubectl rollout restart deployment custom-metrics-apiserver -n monitoring

4. Verify that the custom rules are configured successfully.
   1. Run the following command. If the custom metric information is returned, the metric collection configuration on Prometheus is successful.

      kubectl get --raw=/apis/custom.metrics.k8s.io/v1beta1

      

   2. Query information about nodes in the cluster.

      kubectl get nodes 

      Run the following command on any node. Replace xxxx with the obtained value of node_name. If you want to query the resource information of all nodes, replace xxxx with an asterisk (*).

      kubectl get --raw=/apis/custom.metrics.k8s.io/v1beta1/nodes/xxxx/node_cpu_usage_avg

      Information similar to the following is displayed.