Scaling In a Cluster

Function

Cluster scale-in refers to reducing the number of nodes in a cluster to release excess resources and reduce costs when there are more compute or storage resources in the cluster than needed.

Application Scenarios

• During off-peak hours, cluster node resources are idle for a long time.
• Workloads have been migrated or reduced, so there are significantly more compute or storage resources in the cluster than needed.
• There is a need to reduce cluster costs by releasing unnecessary node resources.

Impact on the System

A scale-in is irreversible. After a scale-in, nodes will be removed and data will be redistributed. Before performing this operation, read the following impact and notes carefully and back up data:

• Before performing a scale-in, you must disable connections to the clients with temporary tables. During the scale-in, temporary tables will become invalid, and operations on the temporary tables will fail. Temporary tables created after the scale-in are not affected.
• If you start a scale-in, an automated snapshot will be created for the cluster before the scale-in is performed. If you do not need the snapshot, you can disable the automated backup function on the scale-in page.
• Ensure that the skew rate is below 10% before a scale-in. It is recommended that the skew rate of tables larger than 50 GB be 20 to 30%. There are no strict requirements for the dirty page rate, but a high rate can affect the redistribution efficiency.
• When scaling in a cluster, several functions are disabled, including cluster restart, cluster scale-out, snapshot creation, node management, intelligent O&M, workload management, parameter modification, security configurations, log service, database administrator password resetting, and cluster deletion.
• During offline scale-in, stop all services or run only a few query statements. During table redistribution, a shared lock is added to tables. All INSERT, UPDATE, and DELETE operations as well as DDL operations on the tables are blocked for a long time, which may cause a lock wait timeout. After a table is redistributed, you can access the table. Do not perform queries that take more than 20 minutes during the redistribution (the default time for applying for the write lock during redistribution is 20 minutes). Otherwise, data redistribution may fail due to lock wait timeout.
• During redistribution of a table in an online scale-in, you can perform INSERT, UPDATE, and DELETE operations on the table. However, the redistribution may block data updates and affect statement execution performance for a short period of time. Redistribution during a scale-in consumes a large number of CPU and I/O resources, which significantly affects job performance. Perform redistribution when services are stopped or lightly loaded.
• If a node is deleted while DDL statements are being executed (to create a schema or function) during online scale-in, errors may occur because the DN cannot be found. To resolve this issue, you can simply re-execute the statements.
• If a cluster scale-in fails, the database does not automatically roll back the scale-in operation, and no O&M operations can be performed. In this case, you need to click the Scale In on the console to try again.
• In the cloud native 9.0.2 scale-in scenario, if the number of data buckets allocated to each DN in the new cluster is not from 3 to 20, automatic scale-in of buckets will be triggered. The number of buckets determines the data distribution granularity. Ensure that the number of buckets allocated to each DN after the scale-in falls within an appropriate range. You can view the number of buckets using the GUC parameter table_buckets.
  • A bucket is the minimum logical unit for data distribution. Data is divided by bucket and then distributed to each DN. Currently, buckets can only be scaled in offline. The procedure is the same as that of the existing scale-in procedure. The system automatically determines and executes the bucket scale-in process.
  • During scaling, the cluster will be restarted and all connections will be closed. The restart takes several minutes.
  • After the restart is complete, the database can be read but cannot be written until data redistribution is complete.

Notes and Constraints

• Scale-in is supported only by pay-per-use clusters of version 8.1.1.300 or later. For clusters billed in yearly/monthly mode, the function is supported only in version 8.2.1 or later. By default, scaled in nodes are charged by quantity.
• The cluster is in Available state, is not read-only, and there is no data being redistributed in the cluster.
• When you scale in a storage-compute coupled cluster, you can only remove nodes with the same storage specifications as the cluster.
• Only clusters with more than three nodes can be scaled in. Basic nodes cannot be removed.
• You must ensure that a cluster configuration file has been generated and that configuration information is consistent with the current cluster configuration.
• Before scaling in a cluster, ensure that the value of the default_storage_nodegroup parameter is installation. This parameter specifies the node group where tables are created by default. Currently, this parameter applies only to common tables. Value installation indicates that tables are created in the node group where the cluster is installed by default.
• A cluster is deployed in ring mode. A ring is the smallest unit for scale-in. For example, four or five hosts form a ring, and the primary, standby, and secondary DNs on these hosts are deployed in this ring. If the current cluster has only one ring, scale-in is not supported and the Scale In button is unavailable.
• The hosts to be removed cannot contain the ETCD, GTM, or CM Server component.
• The nodes to be removed cannot contain CNs. If a CN is deployed on a node, delete the CN before performing a scale-in.
• A scale-in cannot be rolled back, but can be retried. A data redistribution failure after a scale-in does not affect services. You can complete scale-in at other appropriate time. Otherwise, unbalanced data distribution will persist for a long time.
• Before redistribution, ensure that the data_redis schema in the corresponding database is reserved for redistribution and that no operations are performed on it or its tables. The data_redis schema will be used during redistribution and deleted afterwards. User tables (if any) in the schema may be deleted by mistake.
• The gs_cgroup tool (used for resource group management) cannot be used during a scale-in.
• Before a scale-in, check the remaining capacity of the cluster. The nodes to be retained must have sufficient space to store the data of the entire cluster. Otherwise, the scale-in cannot be properly performed.
  • The used physical disk space on each node is less than 80%.
  • All the users and roles use less than 80% of resource quota in total.
  • The estimated space usage after scale-in must be less than 80%.
  • The available space is 1.5 times larger than the maximum size of a single table.
• Automatic removal of faulty CNs is temporarily disabled during a scale-in and is automatically enabled after the scale-in is complete.

Procedure

1. Log in to the DWS console.
2. In the navigation pane on the left, choose Cluster > Cluster List. By default, all clusters of the user are displayed.
3. In the Operation column of the target cluster, choose More > Scale Node > Scale In.
4. In the slide-out panel on the right, Current Resource Details, Scale-in Configuration, and Confirm Configuration areas are displayed.

   If the cluster meets inspection conditions before the scale-in, the Inspection page is displayed. Click Start Inspection to perform an inspection. (To stop an inspection task, click Terminate Task.) When an inspection is ongoing, the inspection progress is displayed. Ensure that the inspection is passed and click Next to perform the change. For details, see Viewing Inspection Results.

5. You can configure scale-in information in the Scale-in Configuration area by referring to the information in the Current Resource Details area.

   Table 1 Scale-in parameters

   Parameter	Description
   Virtual Warehouse/Logical Cluster	The logic cluster/virtual warehouse to be scaled in. This parameter is displayed for a logical cluster/virtual warehouse.
   Nodes to Remove	Number of nodes to be removed
   Scale-in Mode	Online and Offline are supported. Online is enabled by default. During an online scale-in, services may be interrupted briefly, and DDL statements (except data addition, deletion, update, and query statements) may return errors.
   Redistribution Concurrency	You can set a value from 1 to 200. The default value is 4.

6. Confirm that all information in New Specifications is correct and select I acknowledge the potential risks and agree to proceed with the scale-in.
7. Click Submit and return to the cluster list. The cluster status will change to Scaling in. Wait until the scale-in is completed.
8. (Optional) If the scale-in fails, click More in the Operation column and choose Scale Node > Retry Scale-in. In the displayed dialog box, confirm the scale-in policy and click Yes.
9. (Yearly/Monthly billing mode) After the cluster scale-in is complete, you will be prompted to delete idle nodes. Click OK to delete idle nodes. After the scale-in of a pay-per-use cluster is complete, specified nodes will be automatically removed in the background. For a yearly/monthly cluster, you need to manually delete the nodes.
10. On the Delete Node page, view the resource information and click Submit.
11. (Yearly/Monthly billing mode) On the displayed resource confirmation page, confirm the refund information and click Submit.