Scaling Out a Cluster
Function
Cluster scale-out means adding nodes to a cluster to improve the compute and storage capacity of the cluster. If the resources in your cluster cannot keep up with your growing data volume or concurrent query requirements, you can scale out your cluster to improve its processing capability.
The cluster scale-out process is as follows:
- Adding nodes: Add DNs to the cluster to expand compute and storage resources.
- Redistributing data: Redistribute data evenly among all nodes after the scale-out. Data redistribution can be automatically performed or manually triggered after nodes are added. The method is configurable.
Suggestions Before Scale-out
If you want to scale out a cluster to expand its storage capacity, you are advised to run the VACUUM command first to clear and reclaim the storage space. In DWS, after you run the DELETE command, the disk space occupied by the deleted data may not be released immediately, resulting in dirty data (dead tuples) and wasting disk space.
Suggestion: Run the VACUUM command to reclaim space. If the used storage capacity is still too high after the reclamation, expand the capacity. For details about the VACUUM syntax, see VACUUM in the SQL Syntax Reference.
Scale-out Methods
DWS supports online and offline scale-out. For details, see Table 1.
| Scale-out Method | Description | Use Case |
|---|---|---|
| Online scale-out | During a scale-out, a cluster is available and supports addition, deletion, modification, query, and some DDL operations. Write operations are temporarily blocked only during data redistribution. Online scale-out is enabled by default for the following cluster versions (disabled for other cluster versions by default):
| Production environments where services cannot be interrupted |
| Offline scale-out | During a scale-out, the cluster needs to be restarted and is in the Unavailable state. Write operations will be blocked for a long time during data redistribution. During a scale-out, the cluster automatically restarts and becomes Unavailable for a period of time. After the cluster is restarted, the cluster becomes Available. At the end of the scale-out, if you select automatic redistribution, the system dynamically redistributes user data in the cluster to all nodes. Otherwise, you need to start data redistribution. | A maintenance window that allows a short downtime |
Impact on the System
The impact of a scale-out on the system is as follows.
Impact of an Offline Scale-out
- Cluster restart: During an offline scale-out, the cluster automatically restarts. Therefore, the cluster becomes Unavailable for a period of time. After the cluster is restarted, its status becomes Available again. At the end of the scale-out, if you select automatic redistribution, the system dynamically redistributes user data in the cluster to all nodes. Otherwise, you need to start data redistribution.
- Service interruption: During an offline scale-out, stop all services or run only a few query statements. During table redistribution, a shared lock is added to tables. All insert, update, and delete operations as well as DDL operations on the tables are blocked for a long time, which may cause a lock wait timeout. After a table is redistributed, you can access the table.
- Long query restrictions: During redistribution, avoid performing a query task that lasts for more than 20 minutes (a write lock will be placed if a query task is not completed within 20 minutes). Otherwise, a redistribution failure may occur due to lock timeout.
Impact of an Online Scale-out
- Node addition: The cluster is locked, and database objects are checked. To ensure that the cluster is successfully locked, do not execute CREATE/DROP DATABASE or CREATE/DROP TABLESPACE statements during node addition.
- Data redistribution: You can perform insert, update, and delete operations on tables, but data updates are still blocked for a short period of time.
- Resource consumption: Redistribution during a scale-out consumes a large number of CPU and I/O resources, which significantly affects job performance. Perform redistribution when services are stopped or lightly loaded.
- Phase-by-phase redistribution: A phase-by-phase redistribution policy is recommended for cluster scale-out. It means that high-concurrency redistribution is performed during periods of light load, and that redistribution is stopped or low-concurrency redistribution is performed during periods of heavy load.
General Impact
- Function restrictions: Certain cluster functions, including restarting, stopping, and starting, modifying specifications, adding or removing CNs, creating snapshots, and resetting the database administrator's password, cannot be performed during a cluster scale-out.
- Invalid temporary tables: Before a scale-out, you must disable the connections to clients that have created temporary tables. Temporary tables created during or before the scale-out will become invalid, and operations on the temporary tables will fail. Temporary tables created after the scale-out are not affected.
- UNION ALL query error: During scale-out, if two tables belong to different node groups and the COLLATE values specified by the distribution columns of the two tables are different, data redistribution triggered in the UNION ALL scenario may cause a query error. To resolve this issue, explicitly specify the same COLLATE value for the two tables.
- Snapshot changes: Snapshots created after a scale-out will contain data on the new nodes.
- Rollback upon a scale-out failure: If a cluster scale-out fails, the database automatically performs a rollback operation so that the number of nodes in the cluster can be restored to that before the scale-out.
- If the rollback is successful, the cluster is still running properly. You can perform the scale-out again. If the scale-out fails, contact technical support.
- If the rollback fails due to an exception, the cluster may become Unavailable. In this case, you cannot scale out or restart the cluster. Contact technical support.
Impact of the Scale-out of a Storage-Compute Decoupled Cluster on Bucket Adjustment
If the number of buckets assigned to each DN during the scale-out of a Storage-Compute Decoupled cluster is not from 3 to 20, automatic adjustment of the number of buckets will be activated. You can view the number of buckets using the GUC parameter table_buckets.
- Bucket scale-out supports only the offline mode. The system automatically determines the trigger conditions and executes the scale-out.
- During the scale-out, the cluster will be restarted and all connections will be disabled. The restart takes a few minutes.
- After the restart is complete, the database can be read but cannot be written until data redistribution is complete.
Example: There are 32 buckets on the current node and nine DNs in the logical cluster. The number of DNs needs to be increased to 15. As the result of 32 divided by 15 is 2 (rounded down), two buckets are allocated to each DN, which does not fall into 3 to 20. Therefore, the number of buckets will be increased.
Prerequisites
- The cluster must be in the Available, Read-only, or Unbalanced state.
- The number of nodes to be added must be less than or equal to the available nodes. Otherwise, scale-out is not allowed.
- To scale out a cluster as an IAM user, ensure that the IAM user has permissions for VPC, EVS, and BMS.
- To ensure reliability, prepare idle ECS nodes in advance and use them for scale-out. (In anti-affinity deployment mode, the number of idle nodes used for a scale-out must be an integer multiple of the security ring.) If there are no idle nodes, click Add Idle Node to add such nodes. For details, see Adding Nodes.
Notes and Constraints
Specifications and billing constraints
- If a cluster is billed in yearly/monthly mode, nodes added to the cluster will also be billed in yearly/monthly mode.
- When you scale out a storage-compute coupled data warehouse cluster, use the same storage specifications as the cluster.
- If you create a BMS-hosted cluster, contact technical support for cross-flavor scale-out. Resource Flavor will be displayed on the Add Node page.
- If the number of subnet IP addresses is insufficient, cross-subnet scale-out is allowed.
Operation risks
- A cluster becomes read-only during scale-out. Exercise caution when performing this operation.
- The cluster will be intermittently disconnected during scale-out. Exercise caution when performing this operation.
- You are advised to create a manual snapshot before scale-out to ensure data security. For details about how to create a snapshot, see Backing Up a DWS Cluster.
- After you start a scale-out, the system first checks for scale-out prerequisites. If your cluster fails the check, modify configurations as prompted and try again. For details, see What Do I Do If the Scale-out Check Fails?
- Disable automatic redistribution when you scale out a large-scale cluster to facilitate retries upon failures for improved reliability. After the scale-out is complete, manually perform redistribution on the cluster to ensure that redistribution can be retried multiple times if a failure occurs.
Scaling Out a Cluster
- Log in to the DWS console.
- In the navigation pane on the left, choose Cluster > Cluster List. All clusters are displayed by default.
- On the displayed Clusters page, locate the row that contains the target cluster and choose More > Scale Node > Scale Out.
- On the displayed scale-out page, Current Resource Details, Capacity Expansion Configuration, and Confirm the change operation are displayed.
If the cluster meets inspection conditions before the scale-out, the Inspection page is displayed. Click Start Inspection to perform an inspection. (To stop an inspection task, click Terminate Task.) When an inspection is ongoing, the inspection progress is displayed. Ensure that the inspection is passed and click Next to perform the change. For details, see Viewing Inspection Results.
- If you create a BMS-hosted cluster, contact technical support for cross-flavor scale-out. Resource Flavor will be displayed on the Add Node page.
- If the cluster is billed on a yearly/monthly basis, the remaining time and expiration time are displayed.
- If the IP addresses of the original subnet are insufficient, you can scale out the cluster across subnets.
- You can configure scale-out information in the Scale-out Configuration area by referring to the information in the Current Resource Details area.
Table 2 Scale-out parameters Parameter
Description
Precautions
DNs to Add
For an ECS cluster, the number of idle nodes in the cluster is used by default. For details about how to add idle nodes, see Adding Nodes. For a BMS cluster, you can manually set the number of nodes to be added. It must be less than or equal to the number of idle nodes in the cluster.
- DNs are added during scale-out. For details about how to add CNs, see Adding or Deleting a CN in a DWS Cluster.
- At least three nodes must be added in a scale-out. The maximum number of nodes that can be added depends on your available quota. A maximum of 256 nodes can be added.
- For a multi-AZ cluster, the number of nodes to be added must be an integer multiple of 3.
Scale to logical cluster/virtual warehouse
This parameter is displayed for a logical cluster/virtual warehouse. Select the target logical cluster/virtual warehouse to which you want to add nodes. You can only select a logical cluster/virtual warehouse that can be read or written.
To add nodes to a new logical cluster or virtual warehouse, click Scale to New Logical Cluster or Scale to New Virtual Warehouse. The page for adding a logical cluster/virtual warehouse is automatically displayed.
Online scale-out
It is enabled by default. During online scale-out, data can be added, deleted, modified, and queried in the database; and some DDL syntaxes are supported. Errors will be reported for unsupported syntaxes.
-
Terminate Job
This parameter can be set when Online Scale-out is enabled. If this function is enabled and there are jobs that block an online scale-out, the jobs will be automatically terminated after the specified time period.
Clusters of version 8.2.1.100 or later support job termination.
Time Before Blocked Job Termination (s)
This parameter takes effect when Terminate Job is enabled. After the wait time, the jobs blocking the scale-out will be terminated. Enter an integer ranging from 30 to 1,200.
This parameter is displayed when Terminate Job is enabled.
- Configure redistribution information in the Redistribution Configuration area.
Table 3 Redistribution parameters Parameter
Description
Auto Redistribution
It is enabled by default. If automatic redistribution is enabled, data will be redistributed immediately after the scale-out is complete. If this function is disabled, only the scale-out is performed. In this case, to redistribute data, select a cluster and choose More > Scale Node > Redistribute.
Redistribution Concurrency
If automatic redistribution is enabled, you can set the number of concurrent redistribution tasks. The value range is 1 to 200. The default value is 4.
Redistribution Mode
You can select Online or Offline. The default value is Online. In online mode, you can perform addition, deletion, modification, and query operations during the redistribution (which affects performance). In offline mode, the cluster is read-only during redistribution.
- Confirm that all information is correct, select I acknowledge the potential risks and agree to proceed with the change, and click Submit.
- If nodes outside a logical cluster are to be added to the cluster, a dialog box is displayed showing the data distribution information before and after the scale-out. The information includes the number of DNs and buckets, bucket tilt rates, and bucket changes before and after the scale-out. After the scale-out, the number of buckets may increase, so data redistribution is required. After data redistribution, data is evenly distributed, improving database performance. Confirm the information and click OK.
- After you submit the scale-out application, task information of the cluster changes to Scaling out and the process will take several minutes.
- During the scale-out, the cluster automatically restarts. Therefore, the cluster status will stay Unavailable for a while. After the cluster is restarted, the status will change to Available.
- After the scale-out is complete, the system dynamically redistributes user data in the cluster, during which the cluster is in the Read-only state.
- A cluster is successfully scaled out only when the cluster is in the Available state and task information Scaling out is not displayed. Then you can use the cluster.
- If Scale-out failed is displayed, the cluster fails to be scaled out.
Viewing Scaling Details
- Log in to the DWS console.
- In the navigation pane on the left, choose Cluster > Cluster List. By default, all clusters of the user are displayed.
- In the Task Information column of a cluster, click View Details.
- Check the scale-out status of the cluster on the scaling details page. Figure 1 Viewing scale-out details
What is your overall rating for this page?
Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot
