Real Cutover

The organization, preparations, and roles and responsibilities of the real cutover are largely the same as those of the mock cutover. Nevertheless, during the real cutover, operations must strictly follow the real cutover runbook. The cutover solution and runbook vary from service system to service system. The following cutover steps are for reference only.

1. Pre-cutover preparations and checks

   Before the cutover, complete all pre-cutover preparations and checks as specified in the runbook checklist. The runbook preparation and check steps vary according to the service system. The following steps are for reference only.

   Table 1 Preparations and check items before the cutover

   Phase	Task	Department	Activity	Involved or Not	Completed or Not
   Organizational and assurance preparation	Confirming the cutover window	Enterprise project manager	Determine that the cutover window starts at MM, DD, YYYY HH:MM.	Yes	Yes
   	Confirming the images and wording for the downtime notice	Enterprise project manager	Check that the images and wording of the downtime notice have been updated.	Yes	Yes
   	Notifying related personnel to release the notice on the official website	Enterprise project manager	Send an email to related personnel for releasing the notice on the official website.	Yes	Yes
   	Reserving a meeting room	Enterprise project manager	Reserve a meeting room.	Yes	Yes
   	Notifying and checking the personnel involved in the cutover	Enterprise project manager	Check whether the cutover participants can attend the meeting.	Yes	No
   		Enterprise project manager	Confirm the third-party personnel involved in the real cutover and their contact information.	Yes	Yes
   		Enterprise project manager	Confirm that the personnel in the operations center are on duty during the cutover.	Yes	Yes
   	Establishment of the enterprise's internal communication channel	Enterprise project manager	Establish an internal communication channel for the cutover.	Yes	Yes
   	Backend assurance team establishment	Cloud vendor's project manager	Set up a co-managed cutover assurance team with the customer. Establish a communication channel to ensure seamless assurance coordination between the backend O&M and R&D teams.	Yes	Yes
   Communication with the third-party/business segments	Items to be communicated with the third-party/business segments	Enterprise project manager	Communicate with the business segments about the impact of downtime and solutions.	Yes	No
   		O&M team	In cases where a third party uses fixed IP addresses, check the steps for modifying the configurations.	Yes	No
   Environment checklist check	Determining the cut-off date for release pause	R&D team	Determine the cut-off date for release pause.	Yes	No
   	Application inventory checking and start/stop scripts refreshing	R&D team	Check the inventory of applications to be migrated to the cloud.	Yes	No
   	Job inventory checking and script refreshing	R&D team	Check the latest job inventory.	Yes	No
   		R&D team	Check whether the inventory in the script is up-to-date.	Yes	No
   Environment check (including source and target environments, migration tasks, and execution scripts)	Basic cloud service check	O&M team	Check whether the test Wi-Fi provided by the O&M team is available.	Yes	No
   		O&M team	Check whether there is an alarm for exceeding the maximum bandwidth of the Huawei Cloud Direct Connect for synchronization.	Yes	No
   		O&M team	Perform routine status check.	Yes	No
   		O&M team	Check the high availability of cloud services and check whether there is a single-AZ or single-node fault on the target cloud resources.	Yes	No
   	Database check	Database-related departments	Check whether the Huawei Cloud database port is the same as that in the production environment.	Yes	No
   		Database-related departments	Check whether the NTP clock settings are consistent.	Yes	No
   		Database-related departments	Check whether the Redis data migration task is normal and that no error or alarm (including the rollback task) is reported.	Yes	No
   		Database-related departments	Check whether the DRS-MySQL data migration task is in Incremental migration in progress state, with no error or alarm (including the rollback task) reported and the dynamic data comparison task configured.	Yes	No
   		Database-related departments	Check whether the DRS-MongoDB data migration task is in the Incremental migration in progress state, with no error or alarm (including the rollback task) reported.	Yes	No
   		Database-related departments	Check whether character sets are consistent between the source and target MySQL databases.	Yes	Yes
   		Database-related departments	Check whether users are consistent between the source and target databases.	Yes	Yes
   	Supporting system check	Big data-related departments	Change the address of the database for big data extraction to the address of the standby IDC database.	Yes	No
   	Script check	O&M team	Store the application service startup script on the executor.	Yes	No
   		O&M team	Store the application heartbeat check script on the executor.	Yes	No
   	Log system check	O&M team	Check whether ELK can handle the massive amounts of logs generated when multiple applications are started at the same time.	Yes	No
   	Alarm monitoring system check	O&M team	Check whether the monitoring system is normal.	Yes	Yes
   	Disk cleanup	O&M team	Check the disk usage in the production environment and execute the script to clean up disk spaces in batches.	Yes	Yes
   Preparations of the operation guide, tools, terminals, and the login platform	Updating the runbooks for all participants to the latest version	Project manager	Synchronize the latest production runbook address to all participants (including personnel of the business segments).	Yes	No
   	Personnel preparation	Project manager	Organize the participants to review the overall cutover process and provide guidance on their own tasks.	Yes	No
   	Personnel operation permissions check	ALL	Log in to the operation environment and check the operation permissions (such as the login system, OS, and operation GUI).	Yes	No
   		Test team	Check whether ITSM can be logged in. Check whether issues of cloud migration projects can be recorded.	Yes	No
   		ALL	Check whether the operators have the permission to operate the executor on the batch task platform on the day of real cutover.	Yes	No
   	Operation terminal check	ALL	Check the laptops and environments on the evening before the mock or real cutover and ensure that they run properly (DBAs should have a dedicated network cable, and a large switch should be prepared in advance).	Yes	No
   	Test client check	Test team	Clear the client and browser cache.	Yes	No

1. Cutover Runbook

   Once preparations and checks are complete, perform the real cutover according to the steps in the runbook. Each task must strictly follow the instructions. Note that the cutover steps in the runbook may vary depending on the business system. The following steps are for reference only. Steps with the same sequence number are executed concurrently.

   If there are many batch processing tasks and the cutover time window is limited, you can start the tasks based on the priority.

   Table 2 Example cutover procedure

   Task	Step	Subtask
   Forwarding source service traffic to the maintenance notice page	1.1	Changing the status of a CMDB service to Maintaining
   	1.2	Forwarding external access traffic to the maintenance announcement page
   Stopping the scheduled tasks on the source server	2.1	Stopping the scheduled tasks on the source server
   	2.1	Stopping scheduled database tasks on the source server
   Stopping the application services and configuration center on the source server	3.1	Stopping xxx application services on the source server
   	3.1	Stopping the configuration center on the source server
   Migrating message queue data	4.1	Migrating MQ data
   	4.2	Waiting and confirming that Kafka consumption is complete
   Confirming that the data at the data layer on the source server is static	5.1	Confirming that the Redis data on the source server is static
   	5.1	Confirming that the MySQL data on the source server is static
   	5.1	Confirming that the MongoDB data on the source server is static
   Data consistency check	6.1	Checking Redis data consistency and stopping synchronization tasks
   	6.1	Checking MongoDB data consistency and stopping synchronization tasks
   	6.1	Checking MySQL data consistency and stopping synchronization tasks
   Modifying the private DNS resolution at the data layer	7.1	Resolving the private domain name for application-to-application access to the IP address of the application instance on Huawei Cloud
   	7.1	Resolving the private domain name of the target Redis to the IP address of the Huawei Cloud instance
   	7.1	Resolving the private domain name of the target MySQL to the IP address of the Huawei Cloud instance
   	7.1	Resolving the private domain name of the target MongoDB to the IP address of the Huawei Cloud instance
   	7.1	Changing the private domain name of the target MQ to the IP address of the Huawei Cloud instance
   	7.1	Resolving the private domain name of the target Kafka to the IP address of the Huawei Cloud instance
   Starting the configuration center and scheduled task scheduling service, registering jobs, and enabling Kafka consumption	8.1	Starting the configuration center
   	8.2	Starting the scheduled task scheduling service
   	8.3	Releasing configurations in the configuration center in batches (registering jobs)
   	8.3	Releasing configurations in the configuration center in batches (enabling Kafka consumption)
   	8.4	Checking whether the configuration center and scheduled task scheduling service are enabled.
   Intranet downtime notice revocation, and application startup and checks on the target server	9.1	Starting the target MQ
   	9.2	Starting xxx application services on the target server
   	9.3	Heartbeat check
   	9.4	Basic service check
   	9.5	Taking down the intranet maintenance announcement page
   Starting the scheduled tasks on the target database and the scheduled tasks with the highest priority	10.1	Starting scheduled database tasks
   	10.1	Starting the first batch of batch processing tasks on the target server
   Main process test (P0 case)	11.1	Main process test (P0 case) Verify that the application is running properly in the target cloud environment. Verify that core functions and key service processes are the same as those before the migration. Monitor logs and metrics to ensure that the system is running properly.
   Internet downtime notice revocation	12.1	Taking down the Internet maintenance announcement page
   Starting the second batch of batch processing tasks on the target server	13.1	Starting the second batch of batch processing tasks on the target server
   	13.2
   P1 service verification (After the job is started, verify the P1 case.)	14.1	Verifying the service functions of the target server
   Starting the third batch of batch processing tasks on the target server	15.1	Starting the third batch of batch processing tasks on the target server
   Starting the second batch of batch processing tasks on the target server	13.1	Starting the second batch of batch processing tasks on the target server