How Do I Modify the Collection Configuration of the kube-state-metrics Component?

Symptom

The kube-state-metrics component of the kube-prometheus-stack add-on converts the metrics data format of Prometheus into the format that can be identified by Kubernetes APIs. By default, the kube-state-metrics component does not collect all labels and annotations of Kubernetes resources. To collect these labels and annotations, you need to modify the collection configuration in the startup parameter and check whether the corresponding metrics are added to the collection whitelist of ServiceMonitor named kube-state-metrics.

Procedure

1. Run the following command to open the YAML file corresponding to the workload kube-state-metrics:

   kubectl edit deployment kube-state-metrics -nmonitoring

2. Modify the startup parameter of kube-state-metrics.

   For example, to collect all labels of a pod, modify the startup parameter of kube-state-metrics as follows:

   --metric-labels-allowlist=pods=[*],nodes=[node,failure-domain.beta.kubernetes.io/zone,topology.kubernetes.io/zone]

   kube-state-metrics starts to collect labels of pods and nodes and uses kubectl edit servicemonitor kube-state-metrics -nmonitoring to check whether kube_pod_labels is in the collection task of Prometheus.

   To collect annotations, add the parameter --metric-annotations-allowlist to the startup parameter in the same way.

   For details, see https://github.com/kubernetes/kube-state-metrics/blob/v2.2.3/docs/cli-arguments.md.