Using Cloud Eye to Monitor Clusters

Logstash clusters serve as core components in data processing pipelines, continuously handling high-load tasks such as log ingestion, format conversion, and data distribution. If a node experiences CPU overload or insufficient JVM memory, pipeline processing delays and data backlogs can occur. In severe cases, nodes may exit unexpectedly, resulting in data loss. Relying on operations personnel to periodically log in to clusters and manually check metrics is inefficient and makes it difficult to identify issues at an early stage. To address this, CSS integrates with Cloud Eye to collect and visualize key runtime metrics, such as CPU usage and JVM memory metrics, for Logstash nodes in real time. When metrics exceed configured thresholds, Cloud Eye sends alarm notifications through Simple Message Notification (SMN). This helps operations personnel shift from post-failure troubleshooting to proactive issue prevention.

How the Feature Works

Figure 1 Monitoring data flow
Click to enlarge

The monitoring data flow is as follows:

Clusters automatically report monitoring data to Cloud Eye.
You can set alarm rules for CSS on Cloud Eye. You can configure alarm notifications via SMN.
You can then view monitoring metrics on the console.

Cloud Eye provides the following four monitoring dimensions for Logstash clusters.

**Table 1** Monitoring dimensions
Dimension	Description	Applicable Scenario
Cluster	Aggregates metrics at the Logstash cluster level to reflect overall cluster resource utilization.	Overall health checks and cluster-level alerting
Node	Collects resource metrics for individual Logstash nodes (VMs).	Identifying hot nodes and troubleshooting CPU/JVM bottlenecks on individual nodes
Pipeline	Aggregates runtime metrics for all pipelines in a cluster, such as event throughput and processing latency.	Monitoring overall pipeline performance and throughput and detecting global pipeline issues
Single-node pipeline	Collects runtime metrics for pipelines running on an individual node.	Precisely locating processing bottlenecks in specific pipelines on a specific node

Recommended Monitoring Metrics

**Table 2** Recommended metrics
Dimension	Recommended Metrics	Description
CPU and JVM	Average JVM heap usage, maximum JVM heap usage, average CPU usage, and maximum CPU usage	Continuously high JVM heap usage (> 75%) may cause frequent GC and affect data processing performance.
Pipeline throughput	logstash_pipeline_events_in, logstash_pipeline_events_filtered, logstash_pipeline_events_out	A continuously increasing difference between input and output events usually indicates insufficient processing capacity or downstream blocking.

Constraints

Runtime duration: The cluster must run for more than 10 minutes before Cloud Eye can collect valid monitoring data.
Operation permissions: The account used for operations must have the required Cloud Eye permissions. Otherwise, you cannot view monitoring information or configure alarm rules.

Prerequisites

The Logstash cluster is available.

Viewing Cluster/Node Monitoring Metrics

You can view the real-time monitoring metrics of clusters and nodes on the built-in monitoring page of the CSS console. You can compare data across multiple clusters and perform period-over-period analysis.

Log in to the CSS management console.
In the navigation pane on the left, choose Clusters > Logstash.
In the cluster list, find the target cluster, and click Monitoring Metrics in the Operation column.
View cluster-level or node-level monitoring data as needed.
Figure 2 Monitoring metrics
- Viewing cluster-level monitoring metrics
  On the monitoring metrics page, select the Instances tab.
  - Instances: Select the cluster whose monitoring data you want to view. Data for the current cluster is displayed by default. You can select multiple clusters for horizontal comparison.
  - You can set the monitoring time range and view period-over-period comparisons for the same cluster to identify periodic patterns or abnormal deviations.
  - Create Graph Group: Add custom groups on top of the default groups to group related metrics for focused analysis. When you move the pointer over a group name on the left, the icons for changing the group name and deleting the group are displayed on the right. You can modify the group.
  - Select Metric: In the Select Metric dialog box, you can select both All Metrics (single line charts) and Top N Metrics (comparison of the top N nodes by metric value).
- Viewing node-level monitoring metrics
  On the monitoring metrics page, select the CSS Instances tab.
  - CSS Instances: Select the cluster nodes whose monitoring data you want to view. Data for the first node in the current cluster is displayed by default. You can select multiple nodes for horizontal comparison.
  - You can also set monitoring periods so that you can compare metrics across different periods for the same cluster.
  - Create Graph Group: Add custom groups on top of the default groups to group related metrics for focused analysis. When you move the pointer over a group name on the left, the icons for changing the group name and deleting the group are displayed on the right. You can modify the group.
  - Select Metric: In the displayed Select Metric dialog box, set All Metrics (per-metric charts).
(Optional) To view monitoring dashboards, see Method 1: Monitoring Trends on the Monitoring Page (Monitoring Dashboards).
Monitoring dashboards display time-series trend charts for multiple metrics in one place, allowing you to quickly understand the overall cluster status during routine checks.

Viewing Pipeline Monitoring Metrics

You can view cluster-level or node-level pipeline runtime metrics through the Configuration Center to identify pipeline processing bottlenecks.

Log in to the CSS management console.
In the navigation pane on the left, choose Clusters > Logstash.
In the cluster list, find the target cluster and click Configuration Center in the Operation column.
In the pipeline list, click Monitoring Metrics in the Operation column. On the displayed Cloud Eye monitoring metrics page, you can view the aggregated monitoring metrics for all pipelines in the current cluster.
Click the back button in the upper part of the page to go to the custom monitoring page.
View cluster-level or node-level pipeline monitoring data as needed.
1. Viewing cluster-level pipeline metrics
  On the custom monitoring page, click the cluster_id,pipeline_name tab, select the monitoring object, and click View Metric in the Operation column to view metrics.
2. Viewing node-level pipeline metrics
  On the custom monitoring page, click the cluster_id,instance_id,pipeline_name tab, select the monitoring object, and click View Metric in the Operation column to view metrics.

Configuring Alarm Rules

You can configure alarm thresholds and notification methods for monitoring metrics. When metrics exceed configured thresholds, notifications are automatically sent to specified destinations, enabling operations personnel to shift from reactive troubleshooting to proactive monitoring.

Log in to the Cloud Eye console.
In the navigation pane on the left, choose Alarm Management > Rule Settings.

In the alarm rule list, search for alarm rules whose Resource Type is Cloud Search Service (cluster or node level) or CSS.CUSTOM (pipeline level), and check whether these rules meet your requirements.

To create new alarm rules, see Creating an Alarm Rule. The following describes key parameters. Configure other parameters based on your requirements.

Cluster/Node-level alarm rules

**Table 3** Key parameters for configuring a cluster/node-level alarm rule
Parameter	Description
Alarm Type	Select Metric.
Cloud Product	Select Cloud Search Service - CSS Clusters.
Resource Level	Select Specific dimension. In the drop-down list box on the right, select a specific dimension. CSS Clusters: Creates an alarm rule at the cluster level. CSS Clusters - CSS Instances: Creates an alarm rule at the individual node level.

Figure 3 Configuring a cluster/node-level alarm rule

Pipeline-level alarm rules

**Table 4** Key parameters for configuring a pipeline-level alarm rule
Parameter	Description
Alarm Type	Select Metric.
Cloud Product	Select CSS.CUSTOM.
Resource Level	Select a specific dimension to configure the alarm rule. cluster_id,pipeline_name: Creates an alarm rule based on aggregated metrics of all pipelines in a cluster. cluster_id,instance_id,pipeline_name: Creates an alarm rule based on pipelines running on an individual node in a cluster.