Updated on 2023-03-03 GMT+08:00

Step 3: DataArts Migration

This topic describes how to use DataArts Studio DataArts Migration to migrate source data to the cloud in batches.

Creating a Cluster

For details about how to buy DataArts Studio incremental packages, see Buying a DataArts Studio Incremental Package.

Creating Source and Destination Links for Data Migration

  1. Log in to the DataArts Studio console. Locate an instance and click Access. On the displayed page, locate a workspace and click DataArts Migration.
  2. In the left navigation pane, choose Cluster Management. In the cluster list, locate the required cluster and click Job Management.

    Figure 1 Cluster management

  3. On the Job Management page, click Links.

    Figure 2 Links

  4. Create two links, one connecting to OBS to read source data stored on OBS, and the other connecting to MRS Hive to write data to the MRS Hive database.

    Click Create Link. On the page displayed, select Object Storage Service (OBS) and click Next. Then, set the link parameters and click Save.

    Figure 3 Creating an OBS link
    Table 1 Parameter description



    Example Value


    Link name, which should be defined based on the data source type, so it is easier to remember what the link is for


    OBS Endpoint

    An endpoint is the request address for calling an API. Endpoints vary depending on services and regions. You can obtain the OBS bucket endpoint by either of the following means:

    To obtain the endpoint of an OBS bucket, go to the OBS console and click the bucket name to go to its details page.



    Data transmission port. The HTTPS port number is 443 and the HTTP port number is 80.


    OBS Bucket Type

    Select a value from the drop-down list, generally, Object Storage.

    Object Storage


    AK and SK are used to log in to the OBS server.

    You need to create an access key for the current account and obtain an AK/SK pair.

    To obtain an access key, perform the following steps:
    1. Log in to the management console, move the cursor to the username in the upper right corner, and select My Credentials from the drop-down list.
    2. On the My Credentials page, choose Access Keys, and click Create Access Key. See Figure 4.
      Figure 4 Clicking Create Access Key
    3. Click OK and save the access key file as prompted. The access key file will be saved to your browser's configured download location. Open the credentials.csv file to view Access Key Id and Secret Access Key.
      • Only two access keys can be added for each user.
      • To ensure access key security, the access key is automatically downloaded only when it is generated for the first time and cannot be obtained from the management console later. Keep them properly.




    On the Links tab page, click Create Link again. On the page displayed, select MRS Hive and click Next. Then, set the link parameters and click Save.

    Figure 5 Creating an MRS Hive link
    Table 2 MRS Hive link parameters



    Example Value


    Link name, which should be defined based on the data source type, so it is easier to remember what the link is for


    Manager IP

    Floating IP address of MRS Manager. Click Select next to the Manager IP text box to select an MRS cluster. CDM automatically fills in the authentication information.

    Authentication Method

    Authentication method used for accessing MRS
    • SIMPLE: Select this for non-security mode.
    • KERBEROS: Select this for security mode.


    HIVE Version

    Set this to the Hive version on the server.



    If Authentication Method is set to KERBEROS, you must provide the username and password used for logging in to MRS Manager. If you need to create a snapshot when exporting a directory from HDFS, the user configured here must have the administrator permission on HDFS.

    To create a data connection for an MRS security cluster, do not use user admin. The admin user is the default management page user and cannot be used as the authentication user of the security cluster. You can create an MRS user and set Username and Password to the username and password of the created MRS user when creating an MRS data connection.
    • If the CDM cluster version is 2.9.0 or later and the MRS cluster version is 3.1.0 or later, the created user must have the permissions of the Manager_viewer role to create links on CDM. To perform operations on databases, tables, and columns of an MRS component, you also need to add the database, table, and column permissions of the MRS component to the user by following the instructions in the MRS documentation.
    • If the CDM cluster version is earlier than 2.9.0 or the MRS cluster version is earlier than 3.1.0, the created user must have the permissions of Manager_administrator or System_administrator to create links on CDM.
    • A user with only the Manager_tenant or Manager_auditor permission cannot create connections.



    Password used for logging in to MRS Manager


    OBS storage support

    The server must support OBS storage. When creating a Hive table, you can store the table in OBS.


    Run Mode

    This parameter is used only when the Hive version is HIVE_3_X. Possible values are:
    • EMBEDDED: The link instance runs with CDM. This mode delivers better performance.
    • Standalone: The link instance runs in an independent process. If CDM needs to connect to multiple Hadoop data sources (MRS, Hadoop, or CloudTable) with both Kerberos and Simple authentication modes, Standalone prevails.

      Note: The STANDALONE mode is used to solve the version conflict problem. If the connector versions of the source and destination ends of the same link are different, a JAR file conflict occurs. In this case, you need to place the source or destination end in the STANDALONE process to prevent the migration failure caused by the conflict.


    Check Hive JDBC Connectivity

    Whether to check the Hive JDBC connectivity


    Use Cluster Config

    You can create cluster configurations on the Links page to simplify the configuration of Hadoop link parameters.


    Hive Properties

    Other parameters for the Hive client


Creating a Table/File Migration Job

  1. On the DataArts Migration console, click Cluster Management in the left navigation pane, locate the required cluster in the cluster list, and click Job Management.
  2. On the Job Management page, click Table/File Migration and click Create Job.

    Figure 6 Table/File Migration

  3. Set job parameters:

    1. Configure the job name, source job parameters, and destination job parameters, and click Next. See Figure 7.
      • Job Name: source-sdi
      • Source Job Configuration
        • Source Link Name: obs-link
        • Bucket Name: fast-demo
        • Source Directory/File: /2017_Yellow_Taxi_Trip_Data.csv
        • File Format: CSV
        • Show Advanced Attributes: Click Show Advanced Attributes. The system provides default values for advanced attributes. Set parameters based on the actual data format.
          Pay attention to the settings of the following parameters based on the sample data format in Preparing a Data Source. For other parameters, retain the default values.
          • Field Delimiter: Retain the default value (,) in this example.
          • First N Rows As Header: Set this parameter to Yes because the first row is the title row in this example.
          • The Number of Header Rows: Enter 1.
          • Encode Type: Retain the default value UTF-8 in this example.
      • Destination Job Configuration
        • Destination Link Name: mrs-link
        • Database Name: demo_sdi_db
        • Table Name: sdi_taxi_trip_data
        • Clear Data Before Import

          In this example, Clear Data Before Import is set to Yes, indicating that data will be cleared before being imported each time a job is executed. In actual services, set this parameter based on the site requirements to prevent data loss.

        Figure 7 Configuring basic job information
    2. In the Map Field step, configure field mappings and the time format of date fields, as shown in Figure 8. After the configuration is complete, click Next.
      • Field Mapping: In this example, the field sequence in the destination table is the same as that of source data. Therefore, you do not need to adjust the field mapping sequence.

        If the field sequence in the destination table is different from that of source data, map the source fields one by one to the destination fields with the same meaning. Move the cursor to the start point of the arrow of a field. When the cursor is displayed as a plus sign (+), press and hold the mouse button, point the arrow to the destination field with the same meaning, and then release the button.

      • Time Format: The second and third fields in the sample data are time fields. The data format is 02/14/2017 04:08:11 PM. Therefore, set Time Format to MM/dd/yyyy hh:mm:ss a for these two fields. You can also manually enter this format in the text box.

        Select the time format based on the actual data format. For example:

        yyyy/MM/dd HH:mm:ss indicates that the time is converted to the 24-hour format, for example, 2019/08/18 15:35:45.

        yyyy/MM/dd hh:mm:ss a indicates that the time is converted to the 12-hour format, for example, 2019/06/27 03:24:21 PM.

      Figure 8 Mapping fields
    3. Set Retry if failed and Schedule Execution of the task as required.
      Figure 9 Configuring the task

      Click Show Advanced Attributes and set Concurrent Extractors and Write Dirty Data, as shown in Figure 10.

      • Concurrent Extractors: Set this parameter based on the service volume. If the data source is of the file type and there are multiple files, you can increase the value of Concurrent Extractors to improve the extraction speed.
      • Write Dirty Data: You are advised to set this parameter to Yes and set related parameters by referring to Figure 10. Dirty data refers to the data that does not match the fields at the migration destination. Such data will be recorded to a specified OBS bucket. After dirty data writing is configured, normal data will be written to the destination, and migration jobs will not be interrupted due to dirty data. In this example, set OBS Bucket to fast-demo created in Preparing a Data Source. Go to the OBS console, click Create Folder to create a directory, for example, error-data, in the fast-demo bucket, and configure the dirty data directory in Figure 10 as the directory.
      Figure 10 Advanced attributes

  4. Click Save.

    On the Table/File Migration tab page, you can view the created job in the job list.

    Figure 11 Execution result of the migration task