Using Cloud Eye to Monitor Clusters

In production environments, OpenSearch clusters run continuously, and issues such as full disks, JVM out-of-memory errors, and query queue backlogs often emerge before causing failures. However, relying only on cluster logs makes it difficult to detect these issues early and analyze metric trends visually. To address this, CSS integrates with Cloud Eye to collect core OpenSearch cluster metrics, such as disk usage, CPU usage, JVM heap usage, and read/write latency, in real time. Cloud Eye provides visual monitoring dashboards to display metric trends and sends alarm notifications through Simple Message Notification (SMN) when metrics exceed configured thresholds. This helps operations personnel shift from post-failure troubleshooting to proactive issue prevention.

How the Feature Works

Figure 1 Monitoring data flow
Click to enlarge

The monitoring data flow is as follows:

Clusters automatically report monitoring data to Cloud Eye.
You can set alarm rules for CSS on Cloud Eye. You can configure alarm notifications via SMN.
You can then view monitoring metrics on the console.

Cloud Eye provides the following monitoring dimensions for OpenSearch clusters.

**Table 1** Monitoring dimensions
Dimension	Description	Applicable Scenario
Cluster	Aggregates metrics at the OpenSearch cluster level, such as cluster health status and cluster-level disk usage.	Overall health checks and cluster-level alerting
Node	Collects resource metrics at the node level, such as the CPU usage and JVM heap usage of individual nodes.	Identifying hot nodes and troubleshooting node-level performance issues

Recommended Monitoring Metrics

**Table 2** Recommended metrics
Dimension	Recommended Metrics	Description
Disk and cluster health (routine monitoring)	Disk usage, cluster health	Full disks are a primary cause of cluster unavailability. You are advised to set an alarm threshold of 85% for disk usage.
CPU and JVM	Average JVM heap usage, maximum JVM heap usage, average CPU usage, and maximum CPU usage	Continuously high JVM heap usage (> 75%) may cause frequent GC and affect query performance.
Read/Write latency and throughput	Average indexing latency, average indexing rate, average query latency, and average query rate	Continuously increasing indexing/query latency usually indicates excessive cluster load or the presence of slow queries.
Write/Search queues and rejected requests	Total queued tasks in the Write queue, total queued tasks in the Search queue, total rejected tasks in the Write queue, and total rejected tasks in the Search queue	Rejected requests indicate that the cluster is overloaded and requests are being rejected. You are advised to configure an alarm rule that triggers when the number of rejected requests exceeds 0.

Constraints

Runtime duration: The cluster must run for more than 10 minutes before Cloud Eye can collect valid monitoring data.
Operation permissions: The account used for operations must have the required Cloud Eye permissions. Otherwise, you cannot view monitoring information or configure alarm rules.

Prerequisites

The target OpenSearch cluster is available.

Viewing Monitoring Metrics

You can view the real-time monitoring metrics of clusters and nodes on the built-in monitoring page of the CSS console. You can compare data across multiple clusters and perform period-over-period analysis.

Log in to the CSS management console.
In the navigation pane on the left, choose Clusters > OpenSearch.
In the cluster list, find the target cluster, and click Monitoring Metrics in the Operation column.
View cluster-level or node-level monitoring data as needed.
Figure 2 Viewing monitoring metrics
- Viewing cluster-level monitoring metrics
  On the monitoring metrics page, select the Instances tab.
  - Instances: Select the cluster whose monitoring data you want to view. Data for the current cluster is displayed by default. You can select multiple clusters for horizontal comparison.
  - You can set the monitoring time range and view period-over-period comparisons for the same cluster to identify periodic patterns or abnormal deviations.
  - Create Graph Group: Add custom groups on top of the default groups to group related metrics for focused analysis. When you move the pointer over a group name on the left, the icons for changing the group name and deleting the group are displayed on the right. You can modify the group.
  - Select Metric: In the Select Metric dialog box, you can select both All Metrics (single line charts) and Top N Metrics (comparison of the top N nodes by metric value).
- Viewing node-level monitoring metrics
  On the monitoring metrics page, select the CSS Instances tab.
  - CSS Instances: Select the cluster nodes whose monitoring data you want to view. Data for the first node in the current cluster is displayed by default. You can select multiple nodes for horizontal comparison.
  - You can also set monitoring periods so that you can compare metrics across different periods for the same cluster.
  - Create Graph Group: Add custom groups on top of the default groups to group related metrics for focused analysis. When you move the pointer over a group name on the left, the icons for changing the group name and deleting the group are displayed on the right. You can modify the group.
  - Select Metric: In the displayed Select Metric dialog box, set All Metrics (per-metric charts).
(Optional) To view monitoring dashboards, see Method 1: Monitoring Trends on the Monitoring Page (Monitoring Dashboards).
Monitoring dashboards display time-series trend charts for multiple metrics in one place, allowing you to quickly understand the overall cluster status during routine checks.

Configuring Alarm Rules

You can configure alarm thresholds and notification methods for monitoring metrics. When metrics exceed configured thresholds, notifications are automatically sent to specified destinations, enabling operations personnel to shift from reactive troubleshooting to proactive monitoring.

Log in to the Cloud Eye console.
In the navigation pane on the left, choose Alarm Management > Rule Settings.

In the Resource Type column, search for alarm rules whose Resource Type is Cloud Search Service, and check whether these rules meet your requirements.

To create new alarm rules, see Creating an Alarm Rule. Table 3 describes key parameters. Configure other parameters based on your requirements.

**Table 3** Key parameters for configuring an alarm rule
Parameter	Description
Alarm Type	Select Metric.
Cloud Product	Select Cloud Search Service - CSS Clusters.
Resource Level	Select Specific dimension. In the drop-down list box on the right, select a specific dimension. CSS Clusters: Creates an alarm rule at the cluster level. CSS Clusters - CSS Instances: Creates an alarm rule at the individual node level.