Synchronizing Alerts and Metrics to Prometheus

How the Feature Works

Elasticsearch business metrics must be pushed by the alerting plugin, whereas Prometheus uses a pull-based model by default. The Prometheus Pushgateway bridges this gap: it receives push-based metrics and exposes them via a scrape endpoint for Prometheus. The Open Distro Alerting plugin built into CSS Elasticsearch clusters provides a Prometheus-compatible export channel connected to the Pushgateway.

Figure 1 Alert & metric synchronization link

1. Monitor: Defines the aggregation query (which metrics to compute), the target indexes, and the execution schedule (query frequency).
2. Trigger: Evaluates query results against threshold conditions to determine if and when an alert should be generated. It also triggers an action. Action: Transforms Elasticsearch aggregation results (in document form) into standard Prometheus JSON format.
3. Destination: A message channel that sends Prometheus-compatible JSON packets to the Pushgateway.
4. Pushgateway: A temporary storage area that receives push-based metrics from Elasticsearch.
5. Prometheus: Periodically pulls data from the Pushgateway and stores it in its time-series database, where Grafana queries the data and visualizes metrics on unified dashboards.

For more about the Open Distro alerting plugin, see Open Distro Alerting. For more about Prometheus Pushgateway, see Prometheus Pushgateway.

Constraints

• This feature is available only for Elasticsearch clusters whose image version is no earlier than 7.10.2_24.3.3_xxx.
• The Elasticsearch cluster must be able to access the Prometheus Pushgateway endpoint. Prometheus and Elasticsearch must be connected. Otherwise, alerts cannot be sent.
• Only Gauge-type metrics (used in dashboards) can be synchronized. They include statistical or percentage values that can both increase and decrease.

Configuring Alert Synchronization

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch.
3. In the cluster list, find the target cluster, and click Kibana in the Operation column to log in to the Kibana console.
4. On the Kibana page, choose Open Distro for Elasticsearch > Alerting in the navigation pane on the left.
5. Create a Prometheus destination to establish a physical connection between Elasticsearch and Pushgateway.
   1. On the Alerting page, click the Destinations tab, and click Add destination to configure destination information.

      Table 1 Destinations parameters

      Parameter	Description
      Name	User-defined destination name
      Type	Type of the notification. Select PROMETHEUS.
      Settings	Enter the Prometheus Pushgateway endpoint address. Currently, dashboards can only visualize Gauge-type metrics. These include statistical and percentage values that can both increase and decrease. Metrics are queried using specific statements and numeric values are synchronized to Pushgateway for monitoring via Prometheus. Two types of Pushgateway addresses are supported: HTTP and HTTPS.

      Figure 2 Add destination
      
   2. Click Create.
   3. Return to the Destinations page. If the new destination is displayed, it has been created.
      Figure 3 Destinations list
      
6. Create a monitor and trigger to define the alarm triggering conditions and monitor interval.
   1. On the Alerting page, click the Monitors tab. Then click Create monitor, and configure the data source and query frequency.

      Table 2 Monitor parameters

      Parameter		Description
      Configure monitor	Monitor name	User-defined monitor name
      	Monitor state	Whether to disable the monitor. Select Disable monitor: Disable the monitor. (Recommended) Deselect Disable monitor: Enable the monitor.
      Define monitor	Method of definition	Select a method to define the monitor. You are advised to use Define using extraction query. Define using visual graph: use a visual query Define using extraction query: use a specific query
      	Index	Index to be monitored
      	Time field	Timestamp used for time-based aggregations, such as count. This parameter is required only when you select Define using visual graph to define the monitor.
      Monitor schedule	Frequency	Select the monitor frequency and set the monitor interval. The options include: By interval Daily Weekly Monthly Custom cron expression

   2. Click Create. The Create trigger page is displayed.
   3. On the Create trigger page, define how to convert data into Prometheus-readable formats.

      Table 3 Trigger parameters

      Parameter		Description
      Define trigger	Trigger name	User-defined trigger name.
      	Severity level	Sensitivity of the trigger, that is, how many alerts need to be triggered before an alert is actually sent. 1 indicates the highest sensitivity.
      	Trigger condition	Trigger condition. An alert is triggered when the trigger condition is met. You are advised to set a trigger condition that can almost always be triggered (for example, 1 > 0) so that the query results will always be synchronized to the Pushgateway.
      Configure actions	Action name	Name of the triggered action.
      	Destination	Select the Prometheus destination created in 5.
      	Message	Defines the body of the message to be published, which must use the JSON format. Message template: { "metricsName":"hits_total_value", //Prometheus metric name "metricsLabel": {"label_key1":"label_value1","label_key2":"label_value2"}, //Prometheus labels "metricsValue":{{ctx.results.0.hits.total.value}}, //Prometheus metric values "jobName":"job_name", //Prometheus monitor task name "metricsHelp":"***" //Metric explanation. Optional. }
      	Action throttling	Specify the message frequency to limit the number of notifications you receive within a given span of time. Without it, high-frequency or low-severity triggers may cause information overload or result in unexpected cloud costs. For example, if this parameter is set to 10 minutes, Prometheus sends only one alert notification in the next 10 minutes even if the trigger condition is met multiple times. After 10 minutes, Prometheus sends another alert notification if the trigger condition is met again.

   4. Click Send test message to test the link to Prometheus.
      Figure 4 Sending a test message
      
   5. As shown in Figure 5, Prometheus can receive a triggered message, meaning the trigger is set successfully.
      Figure 5 Message received successfully
      
   6. Click Create to go to the monitor details page.