Updated on 2025-07-29 GMT+08:00

Planning Node Specifications and Capacity

This topic provides suggestions on selecting node specifications and configuring the node storage type, storage capacity, and node quantity for a Logstash cluster, helping you properly plan the capacities of your cluster.

Node Configuration Suggestions

Table 1 Node configuration

Parameter

Configuration Suggestions

Node Specifications

In the node flavor list, vCPUs | Memory indicate the number of vCPUs and memory capacity available for each flavor, and Recommended Storage indicates the supported storage capacity range. We recommend that you select node specifications based on core metrics of your Logstash cluster, such as the CPU load, memory requirements, and I/O features.

Logstash Node Specifications describes the application scenarios and core features of different node specifications. It can help you properly plan your cluster.

For more information about different node specifications, see ECS Types.

Node Storage Type and Capacity

Select an appropriate storage type and capacity for cluster nodes.

  • Select an EVS disk type:
    • If you select persisted (disk storage) for event buffering, you are advised to select EVS disks with better performance, such as Ultra-high I/O and Extreme SSD.
    • If you use memory queues, select high I/O EVS disks.
    NOTE:

    By default, Logstash uses in-memory bounded queues to buffer events. The corresponding configuration item is queue.type, which is available when you configure pipelines in the configuration file.

    For more on EVS disk performance, see Disk Types and Performance.

  • The node storage capacity is fixed at 40 GB.

Nodes

The number of nodes in a Logstash cluster ranges from 1 to 100.

Logstash nodes are used to ingest, parse, process, and transfer data. The number of nodes determines the data migration speed. Select the number of Logstash nodes based on service requirements.

When the Logstash cluster has two or more nodes, all nodes use the same configuration files. This mode works when Logstash is a consumer of Kafka data.

Logstash Node Specifications

Logstash nodes support EVS disks only. EVS disks are a virtual block storage service that is independent of ECSs. They provide high reliability and fast elasticity, making them ideal for workloads that require high data reliability and highly scalable storage capacity.

The table below describes the application scenarios and core features of different node specifications. It can help you properly plan your cluster.
Table 2 Comparing different node specifications

CPU Architecture

Node Flavor

Description

x86

Compute-intensive

Core advantages

  • High-performance CPU: designed for high computational load, ideal for CPU-intensive tasks.
  • Optimized network I/O: supports high throughput in both the inbound and outbound directions (such as network plugins).

Application scenarios

  • CPU-intensive plugins: plugins that involve heavy CPU computation, such as grok (regular expression parsing) and dissect (structured log parsing).
  • Hybrid load tasks: tasks that involve both heavy CPU computation and network I/O loads, such as real-time log ingestion and data cleaning.
  • Large-scale data processing at high speed, such as log aggregation and event stream processing.

Precautions

  • Set the number of task threads (pipeline.workers) to equal the number of vCPUs. This optimizes CPU utilization while avoiding resource contention.
  • Pay attention to warnings on I/O performance bottlenecks. If the load of network plugins (such as beats and http) is heavy, make sure there is sufficient network bandwidth.

General computing-plus - AC

Core advantages

  • Dedicated CPU: no resource contention between different instances, stable performance at a relatively low cost, ideal for high-priority tasks.
  • Low latency: guaranteed efficiency for CPU-intensive plugins.

Application scenarios

  • High CPU-load tasks: real-time log parsing and complex field extraction (grok and ruby)
  • Mission-critical service processes: reliable performance required (such as financial transaction log processing)
  • Multi-thread processing: high-concurrency tasks

Precautions

If tasks mainly involve high network I/O, you are advised to use ultra-high I/O disks.

General computing

Core advantages

Balanced configuration: default specifications, suitable for medium-scale data processing tasks.

Application scenarios

  • Medium-scale log processing, such as enterprise log ingestion and monitoring data aggregation.
  • Low CPU-load tasks: mainly network I/O (such as file and kafka plugins).
  • Standard deployment: meets the needs of general use cases, no need for special tuning.

Memory usage evaluation

Estimate the needed memory capacity using this formula: Average size of each piece of data processed by Logstash x (pipeline.workers x pipeline.batch.size)

Example: If the average data size is 1 KB, pipeline.workers = 4, and pipeline.batch.size = 1000, the memory size is ~4 MB.

Memory-optimized

Core advantages

  • Large memory capacity: suitable for memory-intensive tasks, reduced disk I/O pressure.
  • Optimized memory queues: data cached in memory queues, more efficient data processing.

Application scenarios

  • Large-scale log aggregation: log analytics platform, security information and event management (SIEM), etc.
  • Complex data transformation: tasks that involve temporary storage of large amounts of data (for example, the aggregate plugin).

Precautions

  • Monitor the memory usage in real time to avoid out of memory (OOM).
  • This cost is high. Use this flavor for memory-intensive tasks that have high priorities.

Kunpeng

Kunpeng general computing

Core advantages

  • Cost-effective: In general, the Arm architecture is more cost-effective than x86, with lower power consumption.
  • Memory-efficient: Use this flavor for workloads that are cost-sensitive and require large memory capacities.

Application scenarios

  • Cost-sensitive workloads, such as log ingestion for small and medium enterprises and testing environments.
  • Arm ecosystem compatibility: use when your workloads must be compatible with Kunpeng servers.

Precautions

Make sure your Logstash plugins (including Java virtual machine and third-party plugins) are compatible with the Arm architecture.