Switching AZs for an Elasticsearch Cluster

Description

Prerequisites

• Ensure that an AZ with sufficient resources exists.
• The target cluster is available and has no tasks in progress.
• Make sure that no non-standard operations have been performed in the cluster. If you have made non-standard modifications, such as modifying return routes, system parameters, and Kibana configurations, these modifications will be lost after the AZ change and your services may be affected.

Constraints

• To ensure service continuity, the total number of data nodes and cold data nodes in a cluster must be greater than or equal to 3.
• During the change, nodes are brought offline one by one and then new nodes are created. Ensure that the disk capacity of other nodes can store all the data of the node after a single node is brought offline.
• To prevent backup allocation failures after a node is brought offline during the change, ensure that the maximum number of primary and standby index shards of an index can be allocated to the remaining data nodes and cold data nodes. That is, the maximum number of primary and standby shards of an index plus 1 is less than or equal to the total number of data nodes and cold data nodes in the current cluster.
• You are advised to back up data before the change to prevent data loss caused by upgrade faults.
• Before a change completes, some nodes may have been moved to a new AZ. In this case, the AZs before and after the change are both displayed. After the change succeeds, the new AZs and their nodes will be displayed properly.
• When adding AZs, the current AZ must be retained in the change. When adding one or two AZs to a single-AZ cluster, you must change AZs for all nodes at the same time. When adding an AZ to a dual-AZ cluster, you can change AZs for a single type of nodes or all nodes in a cluster at a time. For example, in a cluster using the dual-AZ architecture, you can use the three-AZ architecture for master nodes alone. During HA modification, the nodes with the smallest configurations are modified to rebuild the cluster. After the HA modification is complete, the YML configuration of the nodes that are not modified is also updated. You need to restart the cluster to make the modification take effect.
• When migrating an AZ, you can select only one target AZ. You can migrate AZs for a single type of nodes or all nodes in a cluster at a time. For example, in a cluster with two AZs, you can migrate the AZ of the master node to the other AZ. After adding AZs, you need to restart the cluster to make the modification take effect.

Changing AZs

1. Log in to the CSS management console.
2. In the navigation pane, choose a cluster type. The cluster management page is displayed.
3. Choose More > Modify Configuration in the Operation column of the target cluster. The Modify Configuration page is displayed.
4. Click the Change AZ tab.
5. On the Change AZ page, set parameters.

   Table 1 Parameters for changing AZs

   Parameter	Description
   Operation Type	Add AZ: Add one or two AZs to a single-AZ cluster, or add an AZ to a dual-AZ cluster. During HA modification, the nodes with the smallest configurations are modified to rebuild the cluster. After the HA modification is complete, the YML configuration of the nodes that are not modified is also updated. You need to restart the cluster to make the modification take effect. Migrate AZ: Migrate data from one AZ to another. After adding AZs, you need to restart the cluster to make the modification take effect.
   Node Type	Select a type of node or All nodes to change their AZ. NOTE: When adding one or two AZs to a single-AZ cluster, you can only select All nodes to change AZs for all nodes at a time.
   Current AZ	Current AZ of a cluster
   Target AZ	Target AZ. Add AZ: Select up to three AZs, which must include all your current AZs. Migrate AZ: Select only one target AZ, which cannot be your current AZ.
   Agency	When a node is deleted, NICs are released. This means you need to have VPC permissions. Select an IAM agency to grant the current account the permission to access and use VPC. This parameter is available only when the new IAM plane is connected. If you are configuring an agency for the first time, click Automatically Create IAM Agency to create css-upgrade-agency. If there is an IAM agency automatically created earlier, you can click One-click authorization to delete the VPC Administrator role or the VPC FullAccess system policy, and add the following custom policies instead to implement more refined permissions control. "vpc:subnets:get", "vpc:ports:*" To use Automatically Create IAM Agency and One-click authorization, the following minimum permissions are needed: "iam:agencies:listAgencies", "iam:roles:listRoles", "iam:agencies:getAgency", "iam:agencies:createAgency", "iam:permissions:listRolesForAgency", "iam:permissions:grantRoleToAgency", "iam:permissions:listRolesForAgencyOnProject", "iam:permissions:revokeRoleFromAgency", "iam:roles:createRole" To use an IAM agency, the following minimum permissions are needed: "iam:agencies:listAgencies", "iam:agencies:getAgency", "iam:permissions:listRolesForAgencyOnProject", "iam:permissions:listRolesForAgency"

6. Click Submit. Determine whether to check for the backup of all indexes and click OK to start the change.
   Figure 1 Checking full index snapshots
   
7. The current AZ change task is displayed in the task list. If the task status is Running, expand the task list and click View Progress to view the progress details.

   If the task status is Failed, you can retry or terminate the task.

   • Retry a task: Click Retry in the Operation column of a task.
   • Terminate a task: Click Terminate in the Operation column of a task.

     If the AZ of the original node is not changed after the task is terminated, you can recover the node by referring to Replacing Specified Nodes for an Elasticsearch Cluster.

     

     If the AZ of some nodes has been changed and the AZ form of the cluster has changed, stopping the switchover task may make the delivery of the previous switchover request fail. Exercise caution when stopping the switchover task.