Scaling In an Elasticsearch Cluster

Scenario

CSS supports multiple scale-in methods, as described in Table 1.

Constraints

• To reduce the impact on services, you are advised to perform scale-in during off-peak hours.
• During a scale-in, the data on the nodes to be removed needs to be migrated to other nodes. The timeout threshold for data migration is five hours. If data migration is not complete within 5 hours, the cluster scale-in fails. If the cluster stores large amounts of data, you are advised to break down the scale-in task into multiple batches, and complete each batch one at a time.
• Ensure that the disk usage after scale-in is less than 80% and each AZ has at least one node of each type.
• In a cross-AZ cluster, the difference between the numbers of the same type nodes in different AZs cannot exceed 1.
• For a cluster without master nodes, the number of removed data nodes plus cold data nodes in a scale-in must be fewer than half of the original data nodes plus cold data nodes, and the number of remaining data nodes plus cold data nodes after a scale-in must be greater than the maximum number of index replicas.
• For a cluster with master nodes, the number of removed master nodes in a scale-in must be fewer than half of the original master nodes. After scale-in, there has to be an odd number of master nodes, and there has to be at least three of them.
• For a cluster without master nodes, scale-in is allowed only if the number of data nodes plus cold data nodes is at least three. For a cluster with cluster nodes, this number should be at least two. If you want to reduce cluster nodes even though your cluster does not meet these criteria, you are advised to create a new cluster. You can migrate the data of your cluster by following the procedures described inBacking up and Restoring the Data of an Elasticsearch Cluster.
• In each cluster, the number of nodes supported by each node type varies, depending on the types of nodes used in that cluster. For details, see Table 2.

  Table 2 Number of nodes of different types allowed in a single cluster

  Node Type	Node Quantity
  ess	ess: 1-32
  ess, ess-master	ess: 1-200 ess-master: an odd number ranging from 3 to 9
  ess, ess-client	ess: 1-32 ess-client: 1-32
  ess, ess-cold	ess: 1-32 ess-cold: 1-32
  ess, ess-master, ess-client	ess: 1-200 ess-master: an odd number ranging from 3 to 9 ess-client: 1-32
  ess, ess-master, ess-cold	ess: 1-200 ess-master: an odd number ranging from 3 to 9 ess-cold: 1-32
  ess, ess-client, ess-cold	ess: 1-32 ess-client: 1-32 ess-cold: 1-32
  ess, ess-master, ess-client, ess-cold	ess: 1-200 ess-master: an odd number ranging from 3 to 9 ess-client: 1-32 ess-cold: 1-32
  NOTE: ess: data node, which is the default node type that is mandatory for cluster creation. The other three node types are optional. ess-master: master node ess-client: client node ess-cold: cold data node

Prerequisites

The target cluster is available and has no tasks in progress.

Reducing Nodes Randomly

1. Log in to the CSS management console.
2. In the navigation pane, choose a cluster type. The cluster management page is displayed.
3. In the cluster list, select the target cluster, and choose More > Modify Configuration in the Operation column. The Modify Configuration page is displayed.
4. Click the Scale Cluster tab.
5. Click Scale in to set parameters.

   Table 3 Reducing nodes randomly

   Parameter	Description
   Action	Select Scale in.
   Resources	Shows the change of resources for this scale-in operation.
   Agency	When a node is deleted, NICs are released. This means you need to have VPC permissions. Select an IAM agency to grant the current account the permission to access and use VPC. This parameter is available only when the new IAM plane is connected. If you are configuring an agency for the first time, click Automatically Create IAM Agency to create css-upgrade-agency. If there is an IAM agency automatically created earlier, you can click One-click authorization to delete the VPC Administrator role or the VPC FullAccess system policy, and add the following custom policies instead to implement more refined permissions control. "vpc:subnets:get", "vpc:ports:*" To use Automatically Create IAM Agency and One-click authorization, the following minimum permissions are needed: "iam:agencies:listAgencies", "iam:roles:listRoles", "iam:agencies:getAgency", "iam:agencies:createAgency", "iam:permissions:listRolesForAgency", "iam:permissions:grantRoleToAgency", "iam:permissions:listRolesForAgencyOnProject", "iam:permissions:revokeRoleFromAgency", "iam:roles:createRole" To use an IAM agency, the following minimum permissions are needed: "iam:agencies:listAgencies", "iam:agencies:getAgency", "iam:permissions:listRolesForAgencyOnProject", "iam:permissions:listRolesForAgency"
   Nodes	Reduce the number of nodes in the Nodes column. You can change multiple node types at the same time. For the value range of node quantity for each node type, see Table 2.

   Figure 1 Scaling in a cluster
   
6. Click Next.
7. Confirm the information and click Submit.
8. Click Back to Cluster List to switch to the Clusters page. The Task Status is Scaling in. When Cluster Status changes to Available, the cluster has been successfully scaled in.

Removing Specified Nodes

1. Log in to the CSS management console.
2. In the navigation pane, choose a cluster type. The cluster management page is displayed.
3. In the cluster list, select the target cluster, and choose More > Modify Configuration in the Operation column. The Modify Configuration page is displayed.
4. On the Modify Configuration page, click the Scale In tab.
5. Set scale-in parameters.

   Table 4 Removing specified nodes

   Parameter	Description
   Agency	When a node is deleted, NICs are released. This means you need to have VPC permissions. Select an IAM agency to grant the current account the permission to access and use VPC. This parameter is available only when the new IAM plane is connected. If you are configuring an agency for the first time, click Automatically Create IAM Agency to create css-upgrade-agency. If there is an IAM agency automatically created earlier, you can click One-click authorization to delete the VPC Administrator role or the VPC FullAccess system policy, and add the following custom policies instead to implement more refined permissions control. "vpc:subnets:get", "vpc:ports:*" To use Automatically Create IAM Agency and One-click authorization, the following minimum permissions are needed: "iam:agencies:listAgencies", "iam:roles:listRoles", "iam:agencies:getAgency", "iam:agencies:createAgency", "iam:permissions:listRolesForAgency", "iam:permissions:grantRoleToAgency", "iam:permissions:listRolesForAgencyOnProject", "iam:permissions:revokeRoleFromAgency", "iam:roles:createRole" To use an IAM agency, the following minimum permissions are needed: "iam:agencies:listAgencies", "iam:agencies:getAgency", "iam:permissions:listRolesForAgencyOnProject", "iam:permissions:listRolesForAgency"
   Node Type	Expand the node type that needs be changed to show all nodes under it. Select the nodes you want to remove.

6. Click Next.
7. Confirm the change information and click Submit. In the confirm dialog box, choose to migrate data, which helps to prevent data loss, and click OK.

   During data migration, the system migrates all data from the to-be-removed nodes to the remaining nodes, and removes these nodes upon completion of the data migration. If the data on the to-be-removed nodes has replicas on other nodes, data migration can be skipped and the cluster change can be completed faster.

8. Click Back to Cluster List to switch to the Clusters page. The Task Status is Scaling in. When Cluster Status changes to Available, the cluster has been successfully scaled in.