Cutover with Downtime

Service Downtime Evaluation

According to statistics from Huawei Cloud's past projects, the downtime for most applications during a cutover is 0.5 to 3.5 hours, as analyzed in the following table.

**Table 1** Service downtime breakdown
Service downtime evaluation for cloud migration
Total Downtime (36–211 min)
Stopping the source system (12–75 min)			Incremental data synchronization and verification (6–40 min)			Starting the destination system (18 to 96 minutes)
Stopping the source system	Duration (min)	How to reduce the duration	Incremental data synchronization	Duration (min)	How to reduce the duration	Starting the destination system	Duration (min)	How to reduce the duration
Block inbound traffic at the access layer (gateway/ELB)	1~5	1. Call APIs or perform batch operations using scripts to reduce operation time.	Last incremental synchronization	1~10	1. Perform the cutover during off-peak hours to reduce incremental data.	Allow writes to databases	1	1. Run scripts to allow write operations.
Stop applications	1~30	1. Stop non-critical services in advance to reduce the workload. 2. Shut down services in waves to reduce operation time. 3. Use a unified log platform to reduce the time needed to check application logs.	Data verification	5~30	1. Enable dynamic verification for tools to reduce the verification time.	Start applications	1~30	1. Use a unified O&M platform to stop and start services in batches. 2. Use a unified log platform to reduce the time needed to check application logs.
Stop messaging middleware (after all messages are consumed)	5~30	1. Stop non-critical services in advance to reduce the number of messages. 2. Use a unified monitoring platform to reduce inspection time.	-	-	-	Test applications	15~60	1. Use automated test cases. 2. Execute only critical test cases.
Stop the data layer (writes blocked for check)	5~10	1. Use a unified monitoring platform to reduce the check time.	-	-	-	Cut over inbound traffic	1~5	1. Call APIs or perform batch operations using scripts to reduce operation time.

Exceptions of Service Duration

Downtime < 30 min: If the total service downtime is required to be less than 30 minutes, you can migrate data in waves (dividing data or workloads into different waves) or perform the cutover layer by layer. For example, cut over the application layer and then the data layer. Additionally, if scripts and tools can be used to automate most of the operations, the total service downtime may be reduced to less than 30 minutes.
Downtime > 3.5 hours: When there are large amounts of data and workloads to migrate; batching is impossible due to complex relationships between different components; or the runbook cutover is complex and not sufficiently automated, the total service downtime may be longer than 3.5 hours, or even up to 8–10 hours. For example, with a large organization that has over 600 microservices, over 100 middleware services, over 80 databases, over 1000 batch processing tasks, and over 4000 test cases, the total downtime is around 8 hours.

Ways to Reduce Downtime (in Minutes)

The service downtime depends on multiple factors. Downtime can be reduced during cutover through batch operations, automation, multiple drills, and service adaptation and changes. The figure below shows four ways to reduce downtime during cutover.

Figure 1 Ways to reduce downtime during cutover

Four Ways of Cutover with Downtime

**Table 2** Four ways of cutover with downtime
Cutover Method	Scenarios	Downtime	Number of Service Interruptions	Impact Scope
One-off cutover (stopping the application and data layers all at once)	The application system can tolerate long downtime; there are complex relationships between applications and data and between different applications.	Long	1	All services
Cutover in 3–4 waves: Perform gray cutover of the application layer (1%, 30%…100%) before stopping services. Then, cut over the entire data layer. Then switch internal and external domain names to the destination system.	The downtime is short. Cross-cloud access is allowed in a short period of time, and cross-cloud bandwidth and latency meet the service requirements.	Medium	1	All services
Cutover in 5–10 waves: Perform gray cutover of the application layer (1%, 30%, …100%). Then cut over the data layer in waves (for example, cache + database in wave 1, cache + database in wave 2, and middleware message queues in wave 3).	The downtime is short. Cross-cloud access is allowed in a short period of time, cross-cloud bandwidth and latency meet the service requirements, and the impact of the cutover on internal and external systems is controllable.	Short	Multiple times	Some services
Cutover by service domain	Service domains are relatively independent, with simple relationships between them. They can be migrated to the cloud separately.	Short	Multiple times	Some services

One-Off Cutover (Stopping the Application and Data Layers All at Once)

Preparations

The applications and data of the source system have been migrated to Huawei Cloud.
The applications and data have been verified to function properly in the destination system on Huawei Cloud.

Service cutover:

Stop the applications and batch processing tasks at the source end to prevent new data from being generated. Check that no new messages are generated in message queues and no new data is generated at the data layer.
Complete incremental data synchronization between the data layers at the source end and Huawei Cloud, compare the data for consistency, and then disconnect the data synchronization link.
Change the internal domain names used at the application and data layers on Huawei Cloud, and restart applications and services on Huawei Cloud.
Switch the addresses corresponding to external domain names from the source system to Huawei Cloud to redirect access traffic to Huawei Cloud.

Figure 2 One-off cutover

Gray Cutover of the Application Layer and One-Off Cutover of the Data Layer

Before gray cutover of the application layer, ensure that:

The applications of the source system have been migrated or deployed to Huawei Cloud.
The applications on Huawei Cloud can access the databases at the source end. Functionality and performance have been verified.

After the preparations are complete, redirect traffic from the source-end access layer to Huawei Cloud gradually (1%~30%, …100%).

Figure 3 Gray cutover of the application layer

Then, cut over the data layer one-off by performing the following steps:

Stop applications and batch processing tasks in both the source and destination systems to prevent new data from being written in. This is when services become unavailable.
After all messages in the middleware message queue are consumed, synchronize the incremental data to Huawei Cloud. Verify data consistency between the source and destination systems.
Modify configuration to redirect the applications on Huawei Cloud to the data that is also on Huawei Cloud, and start the applications.
Switch the addresses corresponding to external domain names from the source system to Huawei Cloud to redirect access traffic to Huawei Cloud.

Figure 4 One-off cutover of the data layer

Gray Cutover of the Application Layer and Wave-by-Wave Cutover of the Data Layer

Before gray cutover of the application layer, ensure that:

The applications of the source system have been migrated or deployed to Huawei Cloud.
The applications on Huawei Cloud can access the databases at the source end. Functionality and performance have been verified.

After the preparations are complete, redirect traffic from the source-end access layer to Huawei Cloud gradually (1%~30%, …100%).

Figure 5 Gray cutover of the application layer

Then, cut over the data layer in waves by performing the following steps:

Stop the applications and batch processing tasks related to each wave of the data to be migrated to freeze the data. (All messages in message queues have been consumed, and no new data is written into databases.)
Verify data consistency and then cut over the data layer.
Modify related configurations and start the applications and batch processing tasks related each wave of data. Verify performance and functionality.
Switch the addresses corresponding to external domain names from the source system to Huawei Cloud to redirect access traffic to Huawei Cloud for each application.

Figure 6 Cutover of the data layer in waves

This solution has the following advantages:

Application cutover is simple. When redirecting traffic, you only need to adjust traffic weights on gateways and load balancers.
Traffic redirecting on gateways and load balancers occurs gradually. In case of a problem, the traffic can be switched back to the source system.
During application switchover, applications are still available. During data switchover, writing was disabled only for some of the applications, but the data can still be read.
Data nodes are switched over in waves, and the operations are simple. While some nodes are being switched over, other nodes can still provide services.
This solution has the following disadvantages:
Significant time and manpower are required to analyze and map the dependencies between the application and data layers.
After cross-cloud migration, it is difficult to roll back the data layer.
Cross-cloud migration requires dedicated connections for accessing and transmitting large amounts of data between clouds, which may lead to increased latency.

Cutover by Service Domain

Preparations

The applications and data of the service domain to be cut over have been deployed and migrated to Huawei Cloud.
The applications and data have been verified for functionality and performance.

Per-service domain cutover:

Stop applications and batch processing tasks for one service domain at the source end to prevent new data from being generated. Also, stop applications and batch processing tasks that share middleware and databases with this service domain to prevent new data from being generated.
Check that no new data is generated. Verify data consistency, and then disconnect the data synchronization link.
Modify configurations on Huawei Cloud to start applications and batch processing tasks for the migrated service domain. Then, start applications and batch processing tasks that share middleware and databases with this service domain at the source end.
Cut over DNS domain names to Huawei Cloud and verify functionality and performance.

Figure 7 Cutover by service domain