Help Center/ Cloud Search Service/ User Guide/ Using Logstash for Data Migration/ Logstash Cluster Planning Suggestions
Updated on 2024-09-14 GMT+08:00
View PDF
Share

Logstash Cluster Planning Suggestions

Planning Cluster AZs

By deploying a cluster across multiple AZs, you can increase the cluster's availability, lower the likelihood of data loss, and minimize service downtime. You can select two different AZs in the same region to deploy a cluster.

If you select two AZs when creating a cluster, CSS automatically enables cross-AZ HA, ensuring cluster nodes distribute evenly across both AZs. Even distribution of cluster nodes across AZs means the difference between node quantities in different AZs does not exceed 1. For details, see Table 1.
  • When creating a multi-AZ cluster, select at least two nodes. Otherwise, multi-AZ cluster deployment will fail.
  • When a multi-AZ cluster is deployed, nodes are evenly distributed across different AZs.
Table 1 Node quantities and AZ distribution

Node Quantity

Single AZ

Two AZs

AZ1

AZ1

AZ2

1

1

Not supported

2

2

1

1

3

3

2

1

4

4

2

2

...

...

...

...

Planning Node Storage

  • Planning node models

    Logstash clusters do not need a large storage capacity, so you are advised to choose compute-intensive ECSs.

  • Planning node specifications

    Given the expected data handling capacities, it is always preferable to use a smaller number of nodes with larger specifications rather than a larger number of nodes with smaller specifications. For example, a cluster consisting of three nodes each with 32 CPU cores and 64 GB memory is usually better than a cluster consisting of 12 nodes each with 8 CPU cores and 16 GB memory in terms of stability and scalability.

    The specific advantages are as follows:

    • Cluster stability: High-specs nodes provide more powerful data processing capabilities and larger memory space, leading to higher overall cluster stability.
    • Improved scalability: When a cluster consisting of high-specs nodes encounters a performance bottleneck, you simply add more of these high-specs nodes. This is easier than increasing the specifications of existing nodes.
    • Easier maintenance: A smaller number of nodes means easier maintenance and less complex management.

    In contrast, when a cluster consisting of low-specs nodes needs extra capacity, usually a vertical scale-up is performed, meaning to increase the specifications of existing nodes. This may entail not only more complex, challenging migration and upgrade processes, but also additional maintenance costs.

    To sum up, when planning a cluster, you must fully consider performance, costs, maintenance, and scalability, and choose the node specifications that best suit your needs.

  • Planning storage capacity

    The default node storage capacity for a Logstash cluster is 40 GB and cannot be changed.

Planning VPCs and Subnets

There are two types of VPCs: shared and non-shared.

Compared with a non-shared VPC, a shared VPC has the following advantages:

  • You can create resources in a VPC under one account and share the resources with other accounts. This way, the other accounts do not need to resources. Fewer resources and simplified network architecture improves management efficiency and reduces costs.

    If there are VPCs under different accounts, VPC peering connections will be needed to connect these VPCs. With VPC sharing, different accounts can create resources within one VPC. This eliminates the need to create VPC peering connections and simplifies the network structure.

  • Resources can be centrally managed in one account, which helps enterprises configure security policies in a centralized manner and better monitor and audit resource usage for higher security.
If you choose to create a cluster using a shared VPC, make sure the VPC has already been created. For details, see Table 2. For details about constraints and procedures for using a shared VPC, see VPC Sharing.
Table 2 Procedure for sharing a subnet

Method

Description

Operation Guide

Method A:
  1. On the RAM console, the owner creates a resource share.
    1. Select a subnet to be shared.
    2. Select permissions to grant to principals on the shared subnet.

      To create a CSS cluster in a shared VPC, the default vpc subnet statement permission is required.

    3. Specify principals that will have access to the shared subnet.
  2. On the RAM console, principals accept or reject the resource share.
    • If principals accept the resource share, they can use the shared subnet.

      If principals do not want to use the shared subnet, they can leave the resource share.

    • If principals reject the resource share, they cannot use the subnet.
  1. Creating a Resource Share
  2. Responding to a Resource Sharing Invitation

    Leaving a Resource Share

Method B:
  1. On the RAM console, the owner creates a resource share.
    1. Select a subnet to be shared.
    2. Select permissions to grant to principals on the shared subnet.

      To create a CSS cluster in a shared VPC, the default vpc subnet statement permission is required.

    3. Specify principals that will have access to the shared subnet.
  2. On the VPC console, the owner creates a subnet and adds it to a resource share created earlier.
  3. On the RAM console, principals accept or reject the resource share.
    • If principals accept the resource share, they can use the shared subnet.

      If principals do not want to use the shared subnet, they can leave the resource share.

    • If principals reject the resource share, they cannot use the subnet.
  1. Creating a Resource Share
  2. Sharing a Subnet with Other Accounts
  3. Responding to a Resource Sharing Invitation

    Leaving a Resource Share