Monitoring Metrics of the Master Node Components
Viewing Metrics of the Master Node Components
The kube-prometheus-stack add-on (kube-prometheus-stack) of version 3.5.0 or later can monitor and collect metrics of kube-apiserver, kube-controller, kube-scheduler and etcd-server on the master nodes.
- Install the kube-prometheus-stack of version 3.5.0 or later in your cluster. For details, see kube-prometheus-stack.
- After the installation, wait until all add-on instances are running.
- Create a public network LoadBalancer Service for Prometheus. Enter load balancer public IP: Service port in the address box of the browser to access Prometheus. For details, see Accessing Prometheus.
- Choose Status > Targets. The preceding master node components are displayed.
Collecting Metrics of the Master Node Components Using Self-built Prometheus
This section describes how to collect the metrics of master node components using the self-built prometheus.
![](https://support.huaweicloud.com/intl/en-us/my-kualalumpur-1-usermanual-cce/public_sys-resources/notice_3.0-en-us.png)
- The cluster version must be 1.19 or later.
- The prometheus-operator must be installed in the cluster. For details, see Prometheus Operator.
- Self-built prometheus must be installed in the cluster. For details, see Prometheus community Helm charts. The prometheus (prometheus) add-on is end of maintenance and does not support this function. Therefore, avoid using this add-on.
- Use kubectl to connect to the cluster.
- Modify the ClusterRole of Prometheus.
kubectl edit ClusterRole prometheus -n {namespace}
Add the following content under the rules field:rules: ... - apiGroups: - proxy.exporter.k8s.io resources: - "*" verbs: ["get", "list", "watch"]
- Create a file named kube-apiserver.yaml and edit it.
vi kube-apiserver.yaml
Example file content:apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: app.kubernetes.io/name: apiserver name: kube-apiserver namespace: monitoring # Change it to the namespace where Prometheus will be installed. spec: endpoints: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token interval: 30s metricRelabelings: - action: keep regex: (aggregator_unavailable_apiservice|apiserver_admission_controller_admission_duration_seconds_bucket|apiserver_admission_webhook_admission_duration_seconds_bucket|apiserver_admission_webhook_admission_duration_seconds_count|apiserver_client_certificate_expiration_seconds_bucket|apiserver_client_certificate_expiration_seconds_count|apiserver_current_inflight_requests|apiserver_request_duration_seconds_bucket|apiserver_request_total|go_goroutines|kubernetes_build_info|process_cpu_seconds_total|process_resident_memory_bytes|rest_client_requests_total|workqueue_adds_total|workqueue_depth|workqueue_queue_duration_seconds_bucket|aggregator_unavailable_apiservice_total|rest_client_request_duration_seconds_bucket) sourceLabels: - __name__ - action: drop regex: apiserver_request_duration_seconds_bucket;(0.15|0.25|0.3|0.35|0.4|0.45|0.6|0.7|0.8|0.9|1.25|1.5|1.75|2.5|3|3.5|4.5|6|7|8|9|15|25|30|50) sourceLabels: - __name__ - le port: https scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt serverName: kubernetes jobLabel: component namespaceSelector: matchNames: - default selector: matchLabels: component: apiserver provider: kubernetes
Create a ServiceMonitor:
kubectl apply -f kube-apiserver.yaml
- Create a file named kube-controller.yaml and edit it.
vi kube-controller.yaml
Example file content:apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: app.kubernetes.io/name: kube-controller name: kube-controller-manager namespace: monitoring # Change it to the namespace where Prometheus will be installed. spec: endpoints: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token interval: 15s honorLabels: true port: https relabelings: - regex: (.+) replacement: /apis/proxy.exporter.k8s.io/v1beta1/kube-controller-proxy/${1}/metrics sourceLabels: - __address__ targetLabel: __metrics_path__ - regex: (.+) replacement: ${1} sourceLabels: - __address__ targetLabel: instance - replacement: kubernetes.default.svc.cluster.local:443 targetLabel: __address__ scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt jobLabel: app namespaceSelector: matchNames: - kube-system selector: matchLabels: app: kube-controller-proxy version: v1
Create a ServiceMonitor:
kubectl apply -f kube-controller.yaml
- Create a file named kube-scheduler.yaml and edit it.
vi kube-scheduler.yaml
Example file content:apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: app.kubernetes.io/name: kube-scheduler name: kube-scheduler namespace: monitoring # Change it to the namespace where Prometheus will be installed. spec: endpoints: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token interval: 15s honorLabels: true port: https relabelings: - regex: (.+) replacement: /apis/proxy.exporter.k8s.io/v1beta1/kube-scheduler-proxy/${1}/metrics sourceLabels: - __address__ targetLabel: __metrics_path__ - regex: (.+) replacement: ${1} sourceLabels: - __address__ targetLabel: instance - replacement: kubernetes.default.svc.cluster.local:443 targetLabel: __address__ scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt jobLabel: app namespaceSelector: matchNames: - kube-system selector: matchLabels: app: kube-scheduler-proxy version: v1
Create a ServiceMonitor:
kubectl apply -f kube-scheduler.yaml
- Create a file named etcd-server.yaml and edit it.
vi etcd-server.yaml
Example file content:apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: app.kubernetes.io/name: etcd-server name: etcd-server namespace: monitoring # Change it to the namespace where Prometheus will be installed. spec: endpoints: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token interval: 15s honorLabels: true port: https relabelings: - regex: (.+) replacement: /apis/proxy.exporter.k8s.io/v1beta1/etcd-server-proxy/${1}/metrics sourceLabels: - __address__ targetLabel: __metrics_path__ - regex: (.+) replacement: ${1} sourceLabels: - __address__ targetLabel: instance - replacement: kubernetes.default.svc.cluster.local:443 targetLabel: __address__ scheme: https tlsConfig: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt jobLabel: app namespaceSelector: matchNames: - kube-system selector: matchLabels: app: etcd-server-proxy version: v1
Create a ServiceMonitor:
etcd-server.yaml
- After the creation, if you have created a public network LoadBalancer Service for Prometheus, you can access load balancer public IP: Service port to access Prometheus.
- Choose Status > Targets. The preceding master node components are displayed.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot