Scenario

CDM supports entire DB migration between homogeneous and heterogeneous data sources. The migration principles are the same as those in Table/File Migration Jobs. Each type of Elasticsearch or each collection of MongoDB can be executed concurrently as a subtask.

The source databases can be deployed in on-premises data centers or built on ECSs, or third-party database services.

Field Mapping in Automatic Table Creation

CDM automatically creates tables at the destination during database migration. Figure 1 describes the field mapping between the DWS tables created by CDM and source tables. For example, if you use CDM to migrate the Oracle database to DWS, CDM automatically creates a table on DWS and maps the NUMBER(3,0) field of the Oracle database to the SMALLINT field of DWS.

Figure 1 Field mapping in automatic table creation on DWS

Prerequisites

• You have created links according to Creating Links.
• The CDM cluster can communicate with the data source.

Procedure

1. Log in to the CDM management console.
2. In the left navigation pane, click Cluster Management. Locate the target cluster and click Job Management.
3. Choose Entire DB Migration > Create Job. The page for configuring the job is displayed.
   Figure 2 Creating an entire DB migration job
   

4. Configure the related parameters of the source database according to Table 2.

   Table 2 Parameter description

   Source Database	Parameter	Description	Example Value
   Oracle MySQL PostgreSQL Microsoft SQL Server	Schema/Tablespace	Name of the schema or tablespace from which data will be extracted. This parameter is displayed when Use SQL Statement is set to No. Click the icon next to the text box to go to the page for selecting a schema or directly enter a schema or tablespace. If the desired schema or tablespace is not displayed, check whether the login account has the permissions required to query metadata.	schema
   	WHERE Clause	WHERE clause used to specify the tables to be extracted. This parameter applies to all subtables in the entire DB migration. If this parameter is not set, the entire table is extracted. If the table to be migrated does not contain the fields specified by the WHERE clause, the migration will fail. This parameter can be configured as a macro variable of date and time to extract data generated at a specific date. For details, see WHERE Clause.	age > 18 and age <= 60
   Elasticsearch	Index	Index of the data to be extracted. The value can be a wildcard character. Multiple indexes that meet the wildcard condition can be migrated at a time. For example, if this parameter is set to cdm*, CDM migrates all indexes starting with cdm, such as cdm01, cdmB3, cdm_45 and so on. If multiple indexes are migrated at the same time, Index cannot be configured at the migration destination.	cdm*
   MongoDB	Database Name	Name of the database from which data is to be migrated. The user configured in the source link must have the permission to read the database.	mongodb

5. Configure the related parameters, from Table 3, for the destination cloud service.

   Table 3 Parameter description

   Cloud Service	Parameter	Description
   MRS Hive RDS for MySQL	Schema/Tablespace	Database name
   	Auto Table Creation	The options are as follows: Non-auto creation: CDM will not automatically create a table. Auto creation: If no corresponding table exists in the destination database, CDM will automatically create one. Deletion before creation: If a table with the same name exists in the destination database, CDM will delete the table first and create another one with the same name.
   	Clear Data Before Import	Whether to clear the data in the destination table before data import. The options are as follows: Do not clear: The data in the destination table is not cleared before data import. The imported data is just added to the table. Clear all data: All data is cleared from the destination table before data import. Clear part of data: Part of the data in the destination table is cleared before data import. If you select Clear part of data, you must configure WHERE Clause to specify the data to be deleted from the destination table.
   	WHERE Clause	Used to specify the data to be deleted from the destination table before data import, for example, age > 18 and age <= 60
   CSS	Index	Index to which data is written. If multiple indexes are migrated at a time, this parameter cannot be configured. CDM automatically creates indexes at the migration destination.
   	Clear Data Before Import	Whether to clear data of the target type before data is written
   DWS	-	For details about the destination job parameters required for entire DB migration to DWS, see To a Relational Database.
   OBS	-	For details about the destination job parameters required for entire DB migration to OBS, see To OBS.

6. If a relational database is migrated, after job parameters are configured, click Next to access the page for selecting tables. You can select the tables to be migrated to the migration destination based on your requirements.
7. Click Next and set job parameters.
   Table 4 describes related parameters.

   Table 4 Task configuration parameters

   Parameter	Description	Example Value
   Write Dirty Data	Whether to record dirty data. By default, this parameter is set to No.	Yes
   Write Dirty Data Link	This parameter is only displayed when Write Dirty Data is set to Yes. Only links to OBS support dirty data writes.	obs_link
   OBS Bucket	This parameter is only displayed when Write Dirty Data Link is a link to OBS. Name of the OBS bucket to which the dirty data will be written.	dirtydata
   Dirty Data Directory	This parameter is only displayed when Write Dirty Data is set to Yes. Directory for storing dirty data on OBS. Dirty data is saved only when this parameter is configured. You can go to this directory to query data that fails to be processed or is filtered out during job execution, and check the source data that does not meet conversion or cleaning rules.	/user/dirtydir
   Max. Error Records in a Single Shard	This parameter is only displayed when Write Dirty Data is set to Yes. When the number of error records of a single map exceeds the upper limit, the job will automatically terminate and the imported data cannot be rolled back. You are advised to use a temporary table as the destination table. After the data is imported, rename the table or combine it into the final data table.	0

8. Click Save or Save and Run.

   When the job starts running, a sub-job will be generated for each table. You can click the job name to view the sub-job list.