配置监控告警（中间件类可选）
本章节主要介绍部分监控指标的告警策略以及配置操作。在实际业务中，建议按照以下告警策略，配置监控指标的告警规则。
监控指标
| 监控项说明 | 涉及指标 | 采集周期 | 告警名称 | 告警级别 | 告警阈值 | 告警阈值、PromQL说明 |
|---|---|---|---|---|---|---|
| 节点读请求失败个数 | EMS_GET_KV_STAT | 30s | ems_server_io_read_status_abnormal | 重要级别 | 5% | 连续3个周期内出现报错触发；连续3个周期内未报恢复 PromQL： avg(EMS_GET_KV_STAT{namespace="ems", container="ems-server", item="BackBad"}) by (pid) > 0 |
| 节点写请求失败个数 | EMS_PUT_KV_STAT | 30s | ems_server_io_write_status_abnormal | 重要级别 | 5% | 连续3个周期内出现报错触发；连续3个周期内未报恢复 PromQL： avg(EMS_PUT_KV_STAT{namespace="ems", container="ems-server", item="BackBad"}) by (pid) > 0 |
| 请求成功率 | EMS_PUT_KV_STAT、EMS_GET_KV_STAT | 30s | ems_server_io_success_rate_low | 紧急级别 | 0.95 | 连续5min内成功率小于0.95触发（单个周期内总请求数大于100），5min内成功率大于0.95恢复 (avg(EMS_PUT_KV_STAT{namespace="ems", container="ems-server", item="BackGood"})+avg(EMS_GET_KV_STAT{namespace="ems", container="ems-server", item="BackGood"}))/(avg(EMS_PUT_KV_STAT{namespace="ems", container="ems-server", item="Start"})+avg(EMS_GET_KV_STAT{namespace="ems", container="ems-server", item="Start"}))<0.95 |
| 节点读写QPS | ems_server_kv_concurrent_num_used、ems_server_kv_concurrent_num_limit | 30s | ems_server_io_concurrent_high | 重要级别 | 1000 key/s | 连续3个周期内请求数大于1000 key/s并发限制触发，连续3个周期内小于1000 key/s恢复 PromQL： avg(ems_server_kv_concurrent_num_used{namespace="ems", pod=~"ems-server.*"} / ems_server_kv_concurrent_num_limit{namespace="ems", pod=~"ems-server.*"}) by (pod) > 0.8 |
| 节点读请求时延 | EMS_GET_KV_STAT | 30s | ems_server_io_read_latency_high | 重要级别 | 100ms | 连续3个周期内存在时延大于100ms的读请求触发；连续3个周期内未出现大于100ms的读请求恢复 PromQL： avg_over_time(EMS_GET_KV_STAT{namespace="ems", kubernetes_pod=~"ems-server.*", container="ems-server", item="Av"}[90s])>100 |
| 节点写请求时延 | EMS_PUT_KV_STA | 30s | ems_server_io_write_latency_high | 重要级别 | 200ms | 连续3个周期内存在时延大于200ms的写请求触发；连续3个周期内未出现大于200ms的写请求恢复 PromQL： avg_over_time(EMS_PUT_KV_STAT{namespace="ems", kubernetes_pod=~"ems-server.*", container="ems-server", item="Av"}[90s])>200 |
| ems server连接zk异常 | ems_server_service_zk_status | 30s | ems_server_zk_status_abnormal | 重要级别 | 0 | 连续3个周期内出现状态异常触发；连续3个周期内状态正常则恢复 PromQL： avg_over_time(ems_server_service_zk_status{namespace="ems", kubernetes_pod=~"ems-server.*", container="ems-server"}[90s]) != 0 |
| ems-server cpu使用率 | ems_server_process_cpu_usage | 30s | ems_server_cpu_usage_high | 重要级别 | 0.8 | 连续3个周期内大于阈值触发，连续3个周期内小于阈值恢复 avg_over_time(ems_server_process_cpu_usage{namespace="ems", kubernetes_pod=~"ems-server.*", container="ems-server"}[1m30s])>80 |
| ems-server 句柄数 | ems_server_fd_high | 30s | ems_server_fd_high | 重要级别 | 20000 | 连续3个周期内大于阈值触发，连续3个周期内小于阈值恢复 avg_over_time(ems_server_all_process_fds{namespace="ems", kubernetes_pod=~"ems-server.*", container="ems-server"}[1m30s])>20000 |
| ems-server 节点间带宽 | ems_server_inter_io_bandwidth_high | 30s | ems_server_inter_io_bandwidth_high | 重要级别 | 5GB/s | 连续3个周期内大于阈值触发，连续3个周期内小于阈值恢复 avg_over_time(MFV_CE_TASK_PULL_FLOW_CONTROL_s_all{namespace="ems", kubernetes_pod=~"ems-server.*", container="ems-server", item="Total"}[90s])/30/1000000000>5 |
| ems-server 节点间带宽 | ems_server_inter_io_bandwidth_high | 30s | ems_server_inter_io_bandwidth_high | 重要级别 | 5GB/s | 连续3个周期内大于阈值触发，连续3个周期内小于阈值恢复 avg_over_time(MFV_CE_TASK_PUSH_FLOW_CONTROL_s_all{namespace="ems", kubernetes_pod=~"ems-server.*", container="ems-server", item="Total"}[90s])/30/1000000000>5 |
| ems-server 节点间PUT IO时延 | ems_server_inter_io_latency_high | 30s | ems_server_inter_io_latency_high | 重要级别 | 50ms | 连续3个周期内大于阈值触发，连续3个周期内小于阈值恢复 avg_over_time(MFV_REM_KV_CLI_PUT_VAL{namespace="ems", kubernetes_pod=~"ems-server.*", container="ems-server", item="Av"}[90s])>50000 |
| ems-server 节点间GET IO时延 | ems_server_inter_io_latency_high | 30s | ems_server_inter_io_latency_high | 重要级别 | 50ms | 连续3个周期内大于阈值触发，连续3个周期内小于阈值恢复 avg_over_time(MFV_REM_KV_CLI_GET_VAL{namespace="ems", kubernetes_pod=~"ems-server.*", container="ems-server", item="Av"}[90s])>50000 |
| ems-server 大页内存有效利用率 | ems_server_pool_memory_used/ems_server_pool_memory_total | 30s | ems_server_hugepage_usage_high | 重要级别 | 0.95 | 连续3个周期内大于阈值触发，连续3个周期内小于阈值恢复 avg((ems_server_pool_memory_used{namespace="ems", pod=~"ems-server.*"}) /(ems_server_pool_memory_total{namespace="ems", pod=~"ems-server.*"})) by (pod) >0.95 |
| ems-server 网卡带宽监控 | ems_server_nic_bandwidth | 30s | ems_server_nic_bandwidth_high | 重要级别 | 0.5 | 连续3个周期内大于阈值触发，连续3个周期内小于阈值恢复 avg_over_time(ems_server_nic_bandwidth{namespace="ems", pod=~"ems-server.*",container = "ems-server"}[90s]) >0.5 |
| ems-ctrl 句柄数 | ems_controller_all_process_fds | 30s | ems_ctrl_fd_high | 重要级别 | 20000 | 连续3个周期内大于阈值触发，连续3个周期内小于阈值恢复 avg_over_time(ems_controller_all_process_fds{namespace="ems", kubernetes_pod=~"ems-controller.*", container="ems-controller"}[1m30s])>20000 |
| ems-ctrl cpu使用率 | ems_controller_process_cpu_usage | 30s | ems_ctrl_cpu_usage_high | 重要级别 | 0.8 | 连续3个周期内大于阈值触发，连续3个周期内小于阈值恢复 avg_over_time(ems_controller_process_cpu_usage{namespace="ems", kubernetes_pod=~"ems-controller.*", container="ems-controller"}[1m30s]) > 80 |
| zk进程状态 cpu使用率 | dmk_zk_cpu_usage | 30s | ems_zk_cpu_usage_high | 重要级别 | 0.8 | 连续3个周期内大于阈值触发，连续3个周期内小于阈值恢复 avg_over_time(dmk_zk_cpu_usage{namespace="ems", pod=~"ems-zookeeper.*"}[90s])>80 |
| controller pod状态 | kube_pod_status_ready | 30s | ems_controller_pod_status_abnormal | 重要级别 | 1 | ems controller pod status == notok（CCE集群监控），故障时间>5min PromQL： avg_over_time(kube_pod_status_ready{namespace="ems", condition="true", pod=~"ems-controller.*"}[90s]) != 1 |
| server pod状态 | kube_pod_status_ready | 30s | ems_server_pod_status_abnormal | 重要级别 | 1 | ems server pod status == notok （CCE集群监控），故障时间>5min PromQL： avg_over_time(kube_pod_status_ready{namespace="ems", condition="true", pod=~"ems-server.*"}[90s]) != 1 |
| controller连接zk状态 | ems_controller_service_zk_status | 30s | ems_controller_zk_status_abnormal | 重要级别 | -1 | controller_service_zk_status == -1，故障时间>90s PromQL： avg_over_time(ems_controller_service_zk_status{namespace="ems", kubernetes_pod=~"ems-controller.*", container="ems-controller"}[90s]) != 0 |
| 计费信息上报功能状态 | ems_controller_service_charge_report_status | 30s | ems_controller_charge_report_status_abnormal | 重要级别 | 0 | controller_service_charge_report_status == -1 PromQL： avg_over_time(ems_controller_network_loss_packet_rate_qdisc{namespace="ems", kubernetes_pod=~"ems-controller.*", container="ems-controller"}[90s]) != 0 |
| ems-ctrl容器网卡丢包 | ems_controller_network_loss_packet_rate_qdisc | 30s | ems_controller_network_loss_packet_rate_qdisc | 重要级别 | 0% | 连续3个周期内大于阈值触发，连续3个周期内小于阈值恢复 avg_over_time(ems_controller_network_loss_packet_rate_qdisc{namespace="ems", kubernetes_pod=~"ems-controller.*", container="ems-controller"}[90s])!=0 |
| ems-server容器网卡丢包 | ems_server_network_loss_packet_rate_qdisc | 30s | ems_server_network_loss_packet_rate_qdisc | 重要级别 | 0% | 连续3个周期内大于阈值触发，连续3个周期内小于阈值恢复 avg_over_time(ems_server_network_loss_packet_rate_qdisc{namespace="ems", kubernetes_pod=~"ems-server.*", container="ems-server"}[90s])!=0 |
| 业务进程状态异常 | ems_server_process_status | 30s | ems_server_process_status | 紧急级别 | 1 | 连续3个周期内不等于0/1阈值触发，连续3个周期内小于阈值恢复 avg_over_time(ems_server_process_status{namespace="ems", kubernetes_pod=~"ems-server.*", container="ems-server"} [90s]) !=1 |
| ems-ctrl主节频繁故障 | ems_controller_master_change | 30s | ems_controller_master_change | 紧急级别 | 3次 | 连续3个周期内切主三次 sum( changes(ems_controller_master_change{namespace="ems", kubernetes_pod=~"ems-controller.*", container="ems-controller"}[5m]) > 0 ) > 3 |
| 配置文件效验 | ems_server_config_error_code | 30s | ems_kv_config_error | 紧急级别 | 0 | 连续3个周期内不等于0阈值触发，连续3个周期内小于阈值恢复 avg_over_time(ems_server_config_error_code{namespace="ems", kubernetes_pod=~"ems-server.*", container="ems-server"} [90s]) !=0 |
操作步骤
配置监控告警步骤如下：
- 登录AOM控制台。
- 选择，进入告警规则页面。
- 单击“创建告警规则”，根据表1 ems指标告警内容填写规则，填写后单击“确定”即可。
其中规则填写要求如下：
- 规则名称对应表1 ems指标告警告警名称。
- 规则显示名称、描述为可选项。
- 规则类型选择指标告警规则。
- 配置方式选择语法模式。
- prometheus实例选择收集运维指标中步骤四所创建的实例。
- 普罗语句对应表1 ems指标告警中的promQL内容。
- 告警级别对应表1 ems指标告警中的告警级别。 图1 创建告警规则
- 反复执行3，直到所有告警配置完毕。
