Configuring Monitoring and Alarm Rules Using PrometheusRule

How Do I Configure Monitoring Rules Using PrometheusRule

1. Create monitoring rules to query metrics using PrometheusRule.

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: demo
     namespace: monitoring
     labels:
       role: operator-prometheus   # (Mandatory) The value must be the same as that of the ruleSelector field configured in the Prometheus CR.
   spec: 
     groups: 
     - name:  demo
       interval: 15s
       rules:
       - record: cpu_request
         expr:   kube_pod_container_resource_requests{resource="cpu",unit="core"}
       - record: cpu_limit
         expr:   kube_pod_container_resource_limits{resource="cpu",unit="core"}
       - record: memory_request
         expr:   kube_pod_container_resource_requests{resource="memory",unit="byte"}
       - record: memory_limit
         expr:   kube_pod_container_resource_limits{resource="memory",unit="byte"}

2. Access the Prometheus web page and choose Status > Rules. On the displayed page, query the configured monitoring rules.

   

How Do I Configure Alarm Rules Using PrometheusRule?

You can configure a Prometheus custom resource (CR) that includes a PrometheusRule object to create alarm rules. The following uses the cluster CPU usage as an example to describe how to create an alarm rule template. For details, see https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/.

1. Create an alarm rule template.

   kubectl apply -f PrometheusRule.yaml 

   Example configuration of PrometheusRule.yaml:

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     labels:
       role: operator-prometheus # (Mandatory) The value must be the same as that of the ruleSelector field configured in the Prometheus CR.
     name: demo
     namespace: monitoring
   spec:
     groups:
     - name: alert-cluster-demo
       rules:
       - alert: The cluster CPU usage exceeds 50%.
         expr: 100 - (avg  (irate(node_cpu_seconds_total{mode="idle"}[2m])) * 100) >=50
         for: 2m
         labels:
           severity: critical
           cce_alert_kind: resources
           alertname: Cluster CPU usage exceeds 50%
           kind: resources
           resource_kind: Cluster
           resourceType: Cluster
           source: prometheus
         annotations:
           info: "The actual CPU usage of the cluster exceeds 50%. The current CPU usage of the cluster is {{ printf \"%.2f\" $value }}%".
           description: "The actual CPU usage of the cluster exceeds 50%. The current CPU usage of the cluster is {{ printf \"%.2f\" $value }}%".

2. Access the Prometheus web page and choose Alerts. On the displayed page, check whether the alarm rules are triggered or take effect.

   

3. The Prometheus add-on automatically pushes alerts to the Alertmanager. If you want to configure the alert recipient, you can configure the key named alertmanager in the monitoring namespace. For details, see the Alertmanager document at https://prometheus.io/docs/alerting/latest/configuration/.

   

   

   As shown in the alertmanager-alertmanager StatefulSet YAML file, the alarm data is stored in the pod disk. However, if the pod is restarted, the alarm data will be lost. If you want to persistently store the data, create a PVC and mount it to the pod, and modify the CR of the Alertmanager.