How Do I Back Up Large Volumes of Data Using Snapshots?

When you have large volumes of data and backing up the data may take more than one day, you can use the following methods to improve backup efficiency:

Back up specified indexes in batches: Manually create snapshots on the console and specify a subset of indexes each time to reduce the amount of data backed up in a single operation.
Customize the backup rate: Modify the snapshot configuration on the console to adjust the maximum backup rate and improve the efficiency of each backup operation.
Use a custom snapshot repository: Create a custom repository through Kibana to flexibly configure backup rates and chunk sizes for efficient backup.

Method 1: Backing Up Specified Indexes in Batches

When the data volume is extremely large, you can manually create snapshots by specifying a subset of indexes each time to back up indexes in batches.

On the Dev Tools page of Kibana or OpenSearch Dashboards, run the following command to query the names of all indexes in the cluster:
```
GET /_cat/indices
```
Divide indexes into multiple batches based on index size and priority.
Log in to the CSS management console.
In the navigation pane on the left, expand Clusters. Select a cluster type based on the target cluster. The cluster list is displayed.
In the cluster list, click the name of the target cluster. The cluster information page is displayed.
Choose the Cluster Snapshots tab and enable snapshots.

Under Cluster Snapshot Tasks, click Create Snapshot Manually. In the displayed dialog box, configure the snapshot policy.

**Table 1** Manually creating a snapshot
Parameter	Description
Snapshot Name	Set the snapshot name. You can use names like snapshot-batch1 or snapshot-batch2 to distinguish different batches.
Index	Specify the indexes to be backed up in this snapshot. If you do not specify this parameter, all indexes in the cluster are backed up by default.
Snapshot Description	Add a snapshot description.

Click OK to start creating snapshots.
In the cluster snapshot task list, when Snapshot Task Status changes to Available, backup for the current batch is complete.
Click the expand icon in front of the snapshot name to view the backed-up shards and indexes and verify successful backup.
Repeat steps 7 to 9 until all indexes are backed up.

Method 2: Customizing the Backup Rate

You can modify the snapshot configuration on the console to adjust the maximum backup rate and improve the backup speed of each node.

Increasing the backup rate increases the CPU and disk I/O load on the cluster. It is recommended that you perform this operation during off-peak hours and reduce this rate before peak hours.

Log in to the CSS management console.
In the navigation pane on the left, expand Clusters. Select a cluster type based on the target cluster. The cluster list is displayed.
In the cluster list, click the name of the target cluster. The cluster information page is displayed.

Choose the Cluster Snapshots tab, and click Modify Settings to increase the maximum backup rate.

**Table 2** Modifying the backup rate
Parameter	Description
Maximum Backup Rate (/s)	The parameter sets the maximum backup rate per node. When it is exceeded, flow control is triggered to prevent excessive resource usage and ensure system stability. Value format: number (0–9999) + unit (KB, MB, GB, TB, PB, or B) Default value: 40 MB The value 0MB means there is no rate limit. An excessively high backup rate may lead to excessive resource usage, which may impact cluster stability. Configure this parameter carefully to maintain optimal performance. The actual backup rate may not reach the configured value, as it depends on factors such as OBS performance and disk I/O.
Maximum Recovery Rate (/s)	The parameter sets the maximum recovery rate per node. When it is exceeded, flow control is triggered to prevent excessive resource usage and ensure system stability. Value format: number (0–9999) + unit (KB, MB, GB, TB, PB, or B) Default value: 40 MB for Elasticsearch 7.6.2 or earlier; and 0 MB (no rate limit) for OpenSearch and Elasticsearch versions later than 7.6.2. An excessively high recovery rate may lead to excessive resource usage, which may impact cluster stability. Configure this parameter carefully to maintain optimal performance. For OpenSearch clusters and Elasticsearch versions later than 7.6.2, the actual recovery rate is also affected by the value of indices.recovery.max_bytes_per_sec. Whichever is smaller between the two parameters will be used. Additionally, the actual recovery rate may not reach the configured value, as it depends on factors such as OBS performance and disk I/O.

Click OK to save the change.
Manually create a snapshot or wait for an automatic snapshot to be triggered. The backup will be performed based on the new rate.

Method 3: Using a Custom Snapshot Repository

In addition to the repo_auto repository provided by CSS, you can create a custom snapshot repository to flexibly configure parameters such as backup rate and chunk size for more efficient backup.

Create a custom snapshot repository.

PUT _snapshot/my_backup
{
  "type": "obs",
  "settings": {
    "bucket": "<OBS bucket name>",
    "base_path": "<Backup path>",
    "chunk_size": "2g",
    "endpoint": "obs.<Region name>.example.com",
    "region": "<Region name>",
    "compress": "true",
    "access_key": "<AK>",
    "secret_key": "<SK>",
    "max_restore_bytes_per_sec": "100mb",
    "max_snapshot_bytes_per_sec": "100mb"
  }
}

**Table 3** Custom snapshot repository parameters
Parameter	Mandatory	Description
bucket	Yes	OBS bucket name. The bucket must be in the same region as the cluster, and its storage class must be Standard.
base_path	Yes	Path for storing snapshots in the OBS bucket. The path cannot start with / or ..
chunk_size	No	Chunk size for backup. The default value is 2g. Retain the default value for large data volumes.
endpoint	Yes	OBS endpoint in the format obs.<Region name>.example.com.
region	Yes	Region where the cluster is located.
compress	No	Whether to enable data compression. The default value is true. Data compression reduces OBS storage usage.
access_key	Yes	AK for accessing OBS. For details about how to obtain the AK, see How Do I Obtain an Access Key (AK/SK)?
secret_key	Yes	SK for accessing OBS. For details about how to obtain the SK, see How Do I Obtain an Access Key (AK/SK)?
max_snapshot_bytes_per_sec	No	Maximum backup rate per node. You can increase this value during off-peak hours (for example, 100 MB) and reduce it during peak hours.
max_restore_bytes_per_sec	No	Maximum restore rate per node. Adjust this value based on cluster performance.

Create a snapshot using a custom repository.

PUT _snapshot/my_backup/<snapshot name>
{
  "indices": "*",
  "ignore_unavailable": true,
  "include_global_state": false
}

**Table 4** Parameters for creating a snapshot
Parameter	Mandatory	Description
<snapshot name>	Yes	Snapshot name. Use names like snapshot-batch1 to distinguish different batches.
indices	No	Specifies the indexes to back up. Separate multiple indexes with commas. Wildcards () are supported. The default value is , meaning to back up all indexes.
ignore_unavailable	No	Whether to ignore indexes that cannot be found. The default value is false. true: Ignore indexes that cannot be found and continue backing up other indexes. false: Return a failure when a target index cannot be found.
include_global_state	No	Whether to save the global cluster state. The default value is false. You are advised to retain the default value to prevent the global state from affecting restoration.

Query the snapshot status.
```
GET _snapshot/my_backup/<snapshot name>/_status
```
If the value of state in the returned result is SUCCESS, the backup is successful.

Parent topic: CSS Cluster Backup and Restoration

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

For any further questions, feel free to contact us through the chatbot.

Chatbot