Help Center/ Cloud Search Service/ User Guide/ Using Logstash for Data Migration/ Creating a Logstash Cluster (New Version)
Updated on 2025-09-05 GMT+08:00
View PDF
Share

Creating a Logstash Cluster (New Version)

Logstash clusters created in CSS can be used for efficient ingestion, parsing, processing, and transmission of vast quantities of data. They support real-time log analytics, data integration and transformation, and cross-system data transfers while providing capabilities for data monitoring, alarming, as well as in-depth analysis and visualization.

You can create a Logstash cluster in either of the following ways:

  • Method 1: Create a Logstash cluster on the CSS management console. This topic provides the operation guide for the default UI (that is, the new UI). For the operation guide for the old UI, see Creating a Logstash Cluster (Old Version).

    To improve efficiency in creating clusters, CSS has made several enhancements on the cluster creation page in terms of ease-of-use. Being more intuitive, the new UI streamlines cluster creation procedures.

  • Method 2: Create a Logstash cluster using CSS APIs. For details, see Creating a Cluster.

Prerequisites

  • You have planned the Logstash cluster configuration by following the guidelines in Logstash Cluster Planning Suggestions, including the cluster size, node configuration, and storage capacity requirements.
  • You have created a VPC, subnet, and security group beforehand, or you can create them when creating the cluster. For more information, see Virtual Private Cloud User Guide.

Opening the Create Cluster Page

  1. Log in to the CSS management console.
  2. In the navigation pane on the left, choose Clusters > Logstash.
  3. In the upper right corner, click Create Cluster. The new-version UI for creating a cluster is displayed by default.
    Figure 1 Create Cluster (new version)

Cluster Configuration

Select the cluster type and version.

Table 1 Cluster Configuration

Parameter

Description

Cluster Type

Select Logstash.

Cluster Version

Select a cluster version from the drop-down list.

Basic Settings

Select the region, billing mode, and AZ of the cluster.

Table 2 Basic settings

Parameter

Description

Region

Select the region where the cluster is located. A region is the location of a physical data center. Regions are defined based on their geographical location and network latency. For lower network latency and quicker resource access, select the nearest region.

AZ

Select one or more AZs associated with the cluster region. An AZ is a physical region where resources use independent power supplies and networks. AZs are physically isolated but interconnected through an internal network.

A maximum of two AZs can be configured. For details about the use of multiple AZs, see Suggestions on Multi-AZ Deployment.

Billing Mode

Billing mode for the cluster, which can be Yearly/Monthly or Pay-per-use.

  • Yearly/Monthly: You prepay for a yearly or monthly subscription.

    This mode is cost-effective when the duration of your service usage is predictable. It is recommended for long-term users. For more information, see Yearly/Monthly.

    A yearly/monthly subscription cannot be deleted. If you no longer need it, unsubscribe from it.

  • Pay-per-use (postpaid): You will be billed hourly by actual duration of use. Any partial hour of usage will be rounded up to one hour.

    This mode is ideal for short-term or unpredictable workloads. For more information, see Pay-per-Use.

    Pay-per-use resources can be provisioned and deleted at any time.

Configuring Logstash Nodes

Configure the nodes of the Logstash cluster.

Table 3 Configuring Logstash nodes

Parameter

Description

CPU Architecture

Select the CPU architecture of the Logstash nodes. x86 and Kunpeng nodes are supported. The architectures actually supported may vary depending on the regional environment.

Node Specifications

Select the specifications of the Logstash nodes. Click Available. On the displayed page, select a flavor that suits your needs.

The nodes of a Logstash cluster support EVS disks only. For more information about different node specifications, see ECS Types.

In the node flavor list, vCPUs | Memory indicate the number of vCPUs and memory capacity available for each flavor, and Recommended Storage indicates the supported storage capacity range.

The node flavors available may vary depending on the region you select.

Node Storage Type and Capacity

Select the storage type and capacity of the Logstash nodes.

  • Available EVS disk types vary depending on your region. For more on EVS disk performance, see Disk Types and Performance.
  • The node storage capacity is fixed at 40 GB.

Nodes

Set the number of nodes for the Logstash cluster.

The number of nodes determines the data migration speed. Select the number of Logstash nodes based on service needs.

Value range: 1 to 100.

Network Settings

Configure the VPC, subnet, IP addresses, and security group of the cluster.

Table 4 Configuring network settings

Parameter

Description

VPC

Select a VPC for the cluster for proper network isolation.

You can select a shared VPC, which is indicated by Shared VPC in the VPC name. To create a CSS cluster in a shared VPC, the default vpc subnet statement permission is required.

VPC sharing allows you to centrally manage resources across multiple accounts, helping to improve resource management efficiency and reduce O&M costs. For more information about shared VPCs, see VPC Sharing.

If none of the existing VPCs meets your requirements, click Create VPC to create one. (Make sure you have the permission to create VPCs.) For details, see Creating a VPC and Subnet.

Subnet

Select a subnet for the cluster. A subnet improves network security by providing exclusive network resources that are isolated from other networks.

Select a subnet in the current VPC. Or you can select a subnet under a shared VPC. The owner of a VPC can share the VPC to specified users.

IPv4 Address

Assign IPv4 addresses to cluster nodes.

  • Assign automatically: Private IPv4 addresses will be automatically assigned to cluster nodes.
  • Assign manually: Manually assign private IPv4 addresses to cluster nodes. Before assigning an IP address, click View In-Use IP Address to check IP addresses that are already in use.

    If you manually specify fewer IP addresses than the total number of cluster nodes, the system will automatically assign random private IPv4 addresses to the remaining nodes.

Default value: Assign automatically

Security Group

Select a security group for the cluster. A security group serves as a virtual firewall that provides access control policies for clusters.

To select a security group that meets your requirements, click View Security Group to go to the security group list, where you can check the details of each security group.

The selected security group must allow port 9600 in the inbound direction. Otherwise, the cluster may be inaccessible to external services.

If none of the existing security groups meets your requirements, click Create Security Group to create one. (Make sure you have the permission to create security groups.) For details, see Creating a Security Group.

Cluster Management

Configure the cluster name, enterprise project, and more.
Table 5 Cluster management information

Parameter

Description

Cluster Name

User-defined cluster name.

The cluster name must start with a letter and can contain 4 to 32 characters. Only letters, digits, hyphens (-), and underscores (_) are allowed.

Add Description

Add a description for the cluster for easy recognition.

Click + Add Description and then enter a description of the cluster. The value can contain 0 to 128 characters.

Enterprise Project

Associate the cluster with an enterprise project.

An enterprise project groups cloud resources, so you can manage resources and members by project. The default project is default.

If enterprise projects are enabled, you can select an enterprise project from the drop-down list. You can click View Enterprise Projects to go to the Enterprise Project Management Service console and check enterprise project details. For more information, see Accessing the Enterprise Center.

Tags

Adding tags to clusters helps you identify and manage your cluster resources.

Click + Add. You can select existing tag keys and values from the drop-down lists or enter new ones.

If you want to use the same tag to identify multiple cloud resources for better resource grouping, we recommend that you predefine tags in Tag Management Service (TMS). For details, see Predefined Tags.

Each cluster can have a maximum of 20 tags.

If your organization has configured tag policies for CSS, add tags to clusters based on these policies. If a tag does not comply with the tag policies, cluster creation may fail. Contact the administrator to learn more about tag policies.

Quantity

Configure the cluster's subscription duration and auto-renewal option. This parameter is available only when Billing Mode is set to Yearly/Monthly.
Table 6 Configuring the subscription duration

Parameter

Description

Required Duration

Set the duration of your subscription in the range of 1 month to 3 years. If you plan to use the cluster for more than nine months, you should choose a yearly subscription for a better price.

Auto-renew

Whether to automatically renew the subscription. By default, auto-renewal is disabled. To enable it, select Auto-renew.

When Auto-renew is selected, a yearly/monthly subscription is automatically renewed upon expiry.

  • Monthly subscription: auto-renews for 1 month each time.
  • Yearly subscription: auto-renews for 1 year each time.

For more information, see Auto-Renewal Rules.

Submit

  1. In the Configuration Summary panel on the right, review the cluster configuration details.

    Any mandatory fields that are missing are displayed in red, and you have to set them in the configuration area.

  2. Click Create Now.

    Cluster creation time depends on the number of nodes. Typically, this process takes less than 60 minutes, though clusters with a large number of nodes may require additional time.

  3. Return to the cluster list and check the newly created cluster.

    If the cluster is created successfully, Cluster Status changes to Available.

Related Documents