Scaling Out/Up an Elasticsearch Cluster

Prerequisites

• The cluster status is Available, and there are no ongoing tasks.
• Your CSS resource quotas are sufficient for the capacity expansion you are about to perform. You can check available resources on the Modify Configuration page.
  Figure 1 Checking available resources
  

Impact on Billing

For a pay-per-use cluster, you can see its new price when confirming the scale-out or scale-up on the console. After the scale-out or scale-up is complete, the new price will apply. For pricing details, see Cloud Search Service Price Calculator.

For a yearly/monthly cluster, a new order will be placed after you confirm the scale-out or scale-up. Then you can check the new price.

Constraints

• The node storage capacity of a cluster can only be increased—not decreased. Select an appropriate node storage capacity based on the data volume and projected data growth.
• For a yearly/monthly cluster, you cannot increase the number of nodes and node storage capacity at the same time. For a pay-per-use cluster, however, you can do that.
• The storage capacity of the master and client nodes in a cluster cannot be expanded.
• The storage capacity of data nodes that use local disks cannot be expanded.
• For the range of node quantities supported by each node type, see Table 2.

  Table 2 Node quantity ranges

  Node Type	Value Range
  Data nodes	Without master nodes: 1 to 32 With master nodes: 1 to 200
  Master nodes	3, 5, 7, or 9 (must be an odd number from 3 to 9)
  Client nodes	1–32
  Cold data nodes	1–32

Change Impact

Before the change, learn about possible impacts and operation suggestions, and develop a plan to minimize these impacts.

Expanding the storage capacity of cluster nodes (vertical scaling) does not impact services, whereas adding new nodes or new node types (horizontal scaling) may have the following impacts:

• Impact on performance

  Adding new nodes does not interrupt services. However, after new nodes are added, data shards need to be redistributed to these nodes to balance the load, and this process will consume I/O performance. This is why you are advised to perform the operation during off-peak hours.

  To minimize this impact, it is advisable to adjust the data migration rate based on the cluster's traffic cycle: increase the data migration rate during off-peak hours to shorten the task duration, and decrease it before peak hours arrive to ensure optimal cluster performance. The data migration rate is determined by the indices.recovery.max_bytes_per_sec parameter. The default value of this parameter is the number of vCPUs multiplied by 32 MB. For example, for four vCPUs, the data migration rate is 128 MB. Set this parameter to a value between 40 MB and 1000 MB based on your service requirements.

  PUT /_cluster/settings  
  {  
    "transient": {  
      "indices.recovery.max_bytes_per_sec": "1000MB"  
    }  
  }

• Characteristics of this process

  Once started, a scaling task cannot be stopped until it succeeds or fails.

Duration

• It takes 10 to 30 minutes to add new nodes or new node types (horizontal scaling), depending on the cluster's scheduling capabilities.
• It takes 10 to 15 minutes to expand the storage capacity of cluster nodes (vertical scaling).

Adding More Nodes or Increasing Node Storage Capacity

If a cluster experiences rapid data growth or sustained high memory usage, you may add more nodes to it or increase the cluster nodes' storage capacity. This helps to improve service and data availability and durability.

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch to show the Elasticsearch cluster list.
3. Find the target cluster, and choose More > Modify Configuration in the Operation column. The Modify Configuration page is displayed.
4. Click the Scale Cluster tab.
5. Set the necessary parameters.

   Table 3 Adding more nodes or increasing node storage capacity

   Parameter	Description
   Action	Select Scale out.
   Resources	Quantities of resources added.
   Nodes	Increase the number of nodes and node storage capacity in the Nodes and Node Storage Type columns. You can change multiple node types at the same time. For the range of node quantities supported by each node type, see Constraints. The value range of node storage capacity is determined by the node flavor you select. The value must be divisible by 20. NOTE: For a yearly/monthly cluster, you cannot increase the number of nodes and node storage capacity at the same time. For a pay-per-use cluster, however, you can do that.

   Figure 2 Scaling out a cluster
   
6. Click Next.
7. Confirm the information and click Submit.
8. Click Back to Cluster List to switch to the Clusters page. The Task Status is Scaling out. When Cluster Status changes to Available, the cluster has been successfully scaled out.

Adding New Node Types

For a cluster that has no master or client nodes, as the load on its data plane increases, you may add master or client nodes to it to share the load of the data nodes.

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch to show the Elasticsearch cluster list.
3. In the cluster list, find the target cluster, and choose More > Modify Configuration in the Operation column. The Modify Configuration page is displayed.
4. On the Modify Configuration page, choose the Add Master/Client Nodes tab.

   If a cluster already has both master and client nodes, the Add Master/Client Node tab is unavailable.

5. Select the target node type and set the node specifications, quantity, and storage.

   Table 4 Adding master or client nodes

   Parameter	Description
   Node types	Select the type of nodes you want to add. Only one node type can be selected at a time. If you need to add both Master and Client nodes, you need to perform this task twice. If the cluster already has Master or Client nodes, only the other node type is displayed here.
   Node Specifications	Select node specifications based on site requirements.
   Nodes	Set the number of nodes to add. For the value range, see Constraints.
   Node Storage Type	Set the node storage type. The storage capacity per master or client node is fixed to 40 GB.

   Figure 3 Adding master/client nodes
   
6. Click Next.
7. Confirm the information and click Submit.

   Return to the cluster list page. The Task Status of the cluster is Scaling out.

   • If you added master nodes, you can tell they are added successfully when Cluster Status changes to Available.

     If the cluster version is earlier than 7.x, when the Cluster Status changes to Available, you need to restart all data nodes and cold data nodes in the cluster to make the new nodes take effect. Otherwise, the cluster may be reported as unavailable. (The cluster services still run properly.) For details, see Restarting an Elasticsearch Cluster.

   • If you added client nodes, you can tell they are added successfully when Cluster Status changes to Available. You can restart data nodes and cold data nodes to stop Cerebro and Kibana processes on these nodes.

Related Documents

• You can also upgrade the node specifications and change the EVS disk type of an Elasticsearch cluster. For details, see Changing the Node Specifications and Storage Type of an Elasticsearch Cluster.
• During a capacity expansion, if the resources in an AZ become insufficient, you can migrate the nodes in this AZ to another that has sufficient resources, and then perform the capacity expansion in the new AZ. For details, see Switching AZs for an Elasticsearch Cluster.