New Incremental Backup and Restoration Solution

Scenarios

As business scales, database backups must evolve beyond basic availability to offer higher reliability, fewer constraints, and faster recovery efficiency. For DDS instances, Huawei Cloud has introduced a new incremental backup and restoration solution. This solution optimizes backup capabilities for sharded clusters. Compared to the old solution, the new solution offers fewer constraints, faster incremental backup speeds, and a more comprehensive recovery workflow, delivering significant performance advantages particularly in clusters with a large number of shards.

Overview

DDS backup includes full backup and incremental backup.

● Full backup: It saves complete data at a specific point in time.

● Incremental backup: It records data changes that have occurred since the last full backup.

● Restoration: Instances can be restored to a specific point in time using both full and incremental backups.

Old solution:

● Replica set instances perform incremental backups using the oplog.

● Cluster instances perform incremental backups using change streams.

New solution:

● Replica set instances still perform incremental backups using the oplog.

● Cluster instances now back up their oplogs separately using the hidden nodes of each shard and the hidden nodes of the config server.

The main change in the new solution is the upgrade of the incremental backup architecture for cluster instances.

Differences Between the Old and New Solutions

In the old solution, incremental backups of cluster instances rely on change streams to obtain data changes. In the new solution, it no longer relies on change streams. Instead, the hidden nodes of each shard and the hidden nodes of the config server back up their own oplogs separately.

This can be understood as follows:

● The old solution uses change streams to obtain data changes of cluster instances.

● In the new solution, each component collects incremental data using its own oplog.

The new solution aligns incremental backups of cluster instances more closely with the underlying log structure, resulting in a more streamlined backup process.

Advantages of the New Solution

After the new solution is launched, cluster instances will benefit in the following ways:

1. There are fewer constraints.

   In the old solution, incremental backups of cluster instances rely on change streams, which are subject to functional limitations in certain scenarios. The new solution directly performs incremental backups based on the oplog of each shard and the config server, making it more widely applicable and less restrictive.

2. The incremental backup speed is significantly improved.

   The new solution lets each shard back up its own oplog in parallel, significantly improving incremental backup efficiency. For a cluster instance, more shards mean higher parallelism and faster incremental backups. Therefore, the new solution is more advantageous in cluster instances with many shards.

3. The restoration process is more complete.

   The new solution covers not only data changes of each shard, but also metadata changes of the config server. During restoration, the system performs unified orchestration based on both full and incremental backups to restore cluster instances more completely.

Restoration Process

The system restores data using backups. The main process includes:

1. Restoring data from a full backup
2. Replaying incremental logs in sequence
3. Collaboratively restoring data changes of each shard and the config server

The new solution not only improves incremental backup efficiency but also provides a more complete restoration foundation for cluster instances.

Switching from the Old Solution to the New One

For instances using the old incremental backup solution, you can switch to the new solution by performing the following steps.

1. This function is now in OBT. To use it, submit a service ticket to enable the new incremental backup function for the account that has created the target DDS instance.
2. After the new incremental backup solution is enabled, upgrade your instance to the latest patch version that supports the new solution. For details, see Upgrading a Minor Engine Version.
3. Re-enable incremental backup.

   In your backup policy:

   • Disable incremental backup first.
   • Enable incremental backup again.

   After that, your instance now uses the new incremental backup solution.

Precautions for Switching the Solutions

Precautions:

1. Disabling incremental backup will delete historical incremental backups.

2. A new incremental backup chain will be set up after the switch.

3. Switch the solutions when no restoration is required.

4. If you may need to restore data using historical incremental backups in the near future, wait until an appropriate time window to switch the solutions.

Precautions for Using Backups for Restoration

To improve restoration efficiency and minimize failure risks, take the following precautions:

1. Before performing destructive operations, create a manual full backup.

   Before performing any of the following operations, create a manual full backup:

   • dropCollection
   • dropDatabase
   • shardCollection

   These operations usually have a great impact on data or data distribution. Creating a new full backup before performing such operations can significantly reduce recovery time in the event of misoperations or if restoration is required.

2. When performing PITR to a new instance, ensure that the specifications of the target instance are not significantly lower than those of the source instance.

   If you choose a target instance whose specifications are much lower than the source instance during PITR, the following problems may occur:

   • The restoration time will be much longer.
   • The resource pressure during the restoration will be high.

   In extreme cases, the restoration may fail due to out of memory (OOM) issues. Therefore, when performing PITR to a new instance, ensure that the specifications of the target instance are not significantly lower than those of the source instance. Especially when there are large amounts of data and many incremental logs, properly evaluate the CPU and memory resources of the target instance.

3. In the new incremental backup solution, PITR performance is related to the target instance's CPU specifications.

   In the new solution, incremental data is restored by replaying oplogs using multiple threads. Therefore, during a PITR to a new instance, a larger number of CPU cores on the target instance usually results in faster incremental log replay. In scenarios requiring high restoration efficiency, select a target instance with higher CPU specifications to complete PITR as soon as possible and locate the critical data you need. When the restoration is complete and the required data is verified, you can downsize the instance specifications as needed to balance efficiency and costs.