Migrating Data from Elasticsearch to DLI

Prerequisites

• You have created a DLI SQL queue.
  

  When you create a queue, set its Type to For SQL.

• You have created a CSS Elasticsearch cluster.

  In this example, the version of the created CSS cluster is 7.6.2, and security mode is disabled for the cluster.

• You have created a CDM cluster.
  

  • If the destination data source is an on-premises database, you need the Internet or Direct Connect. When using the Internet, ensure that an EIP has been bound to the CDM cluster, the security group of CDM allows outbound traffic from the host where the off-cloud data source is located, the host where the data source is located can access the Internet, and the connection port has been enabled in the firewall rules.
  • If the data source is CSS on a cloud, the network must meet the following requirements:

    i. If the CDM cluster and the cloud service are in different regions, a public network or a dedicated connection is required for enabling communication between the CDM cluster and the cloud service. If the Internet is used for communication, ensure that an EIP has been bound to the CDM cluster, the host where the data source is located can access the Internet, and the port has been enabled in the firewall rules.

    ii. If the CDM cluster and the cloud service are in the same region, VPC, subnet, and security group, they can communicate with each other by default. If the CDM cluster and the cloud service are in the same VPC but in different subnets or security groups, you must configure routing rules and security group rules.

    iii. The cloud service instance and the CDM cluster belong to the same enterprise project. If they do not, you can modify the enterprise project of the workspace.

  In this example, the VPC, subnet, and security group of the CDM cluster are the same as those of the CSS cluster.

Step 1: Prepare Data

• Create an index for the CSS cluster and import data.
  1. Log in to the CSS management console and choose Clusters > Elasticsearch from the navigation pane on the left.
  2. On the Clusters page, click Access Kibana in the Operation column of the created CSS cluster.
  3. In the navigation pane of Kibana, choose Dev Tools. The Console page is displayed.
  4. On the displayed Console page, run the following command to create index my_test:

     PUT /my_test
     {
       "settings": {
         "number_of_shards": 1
       },
       "mappings": {
               "properties": {
             "productName": {
               "type": "text",
               "analyzer": "ik_smart"
             },
             "size": {
               "type": "keyword"
             }
           }
         }
       }

  5. Run the following command to import data to the my_test index:

     POST /my_test/_doc/_bulk
     {"index":{}}
     {"productName":"2017 Autumn New Shirts for Women", "size":"L"}
     {"index":{}}
     {"productName":"2017 Autumn New Shirts for Women", "size":"M"}
     {"index":{}}
     {"productName":"2017 Autumn New Shirts for Women", "size":"S"}
     {"index":{}}
     {"productName":"2018 Spring New Jeans for Women","size":"M"}
     {"index":{}}
     {"productName":"2018 Spring New Jeans for Women","size":"S"}
     {"index":{}}
     {"productName":"2017 Spring Casual Pants for Women","size":"L"}
     {"index":{}}
     {"productName":"2017 Spring Casual Pants for Women","size":"S"}

     If errors is false in the command output, the data is imported.

• Create a database and table on DLI.
  1. Log in to the DLI management console and click SQL Editor. On the displayed page, set Engine to spark and Queue to the created SQL queue.

     Enter the following statement in the editing window to create a database, for example, the migrated DLI database testdb:

     create database testdb;

  2. Create a table in the database.

     create table tablecss(size string, productname string);

Step 2: Migrate Data

1. Create a CDM connection to MRS Hive.
   1. Create a connection to link CDM to the data source CSS.
      1. Log in to the CDM console, choose Cluster Management. On the displayed page, locate the created CDM cluster, and click Job Management in the Operation column.
      2. On the Job Management page, click the Links tab, and click Create Link. On the displayed page, select Cloud Search Service and click Next.
         Figure 1 Selecting the CSS connector
         
      3. Configure the connection. The following table describes the required parameters.

         Table 1 CSS data source configuration

         Parameter	Value.
         Name	Name of the CSS data source, for example, source_css.
         Elasticsearch Server List	Click Select next to the text box and select the CSS cluster. The Elasticsearch server list is automatically displayed.
         Security mode Authentication	If you have enabled the security mode for the CSS cluster, set this parameter to Yes. Otherwise, set this parameter to No. In this example, set this parameter to No.

         Figure 2 Configuring the CSS connection
         
      4. Click Save to complete the configuration.
   2. Create a connection to link CDM to DLI.
      1. Log in to the CDM console, choose Cluster Management. On the displayed page, locate the created CDM cluster, and click Job Management in the Operation column.
      2. On the Job Management page, click the Links tab, and click Create Link. On the displayed page, select Data Lake Insight and click Next.
         Figure 3 Selecting the DLI connector
         
      3. Configure the connection parameters.
         Figure 4 Configuring connection parameters
         
      4. After the configuration is complete, click Save.
2. Create a CDM migration job.
   1. Log in to the CDM console, choose Cluster Management. On the displayed page, locate the created CDM cluster, and click Job Management in the Operation column.
   2. On the Job Management page, choose the Table/File Migration tab and click Create Job.
   3. On the Create Job page, specify job information.
      Figure 5 Configuring the CDM job
      
      1. Job Name: Name of the data migration job, for example, css_to_dli
      2. Set parameters required for Source Job Configuration.

         Table 2 Source job configuration parameters

         Parameter	Value
         Source Link Name	Select the name of the data source created in 1.a.
         Index	Select the Elasticsearch index created for the CSS cluster. In this example, the my_test index created in Create an index for the CSS cluster and import data is used. The index can contain only lowercase letters.
         Type	Elasticsearch type, which is similar to the table name of a relational database. The type name can contain only lowercase letters. Example: _doc.

         For details about other parameters, see From Elasticsearch or CSS.

      3. Set parameters required for Destination Job Configuration.

         Table 3 Destination job configuration parameters

         Parameter	Value
         Destination Link Name	Select the DLI data source connection created in 1.b.
         Resource Queue	Select a created DLI SQL queue.
         Database	Select a created DLI database. In this example, database testdb created in Create a database and table on DLI is selected.
         Table	Select the name of a table in the database. In this example, table tablecss created in Create a database and table on DLI is created.
         Clear data before import	Whether to clear data in the destination table before data import. In this example, set this parameter to No. If this parameter is set to Yes, data in the destination table will be cleared before the task is started.

3. Click Next. The Map Field page is displayed. CDM automatically matches the source and destination fields.
   • If the field mapping is incorrect, you can drag the fields to adjust the mapping.
   • If the type is automatically created at the migration destination, you need to configure the type and name of each field.
   • CDM allows for field conversion during migration. For details, see Field Conversion.
     Figure 6 Field mapping
     
4. Click Next and set task parameters. Generally, retain the default values of all parameters.

   In this step, you can configure the following optional functions:

   • Retry Upon Failure: If the job fails to be executed, you can determine whether to automatically retry. Retain the default value Never.
   • Group: Select the group to which the job belongs. The default group is DEFAULT. On the Job Management page, jobs can be displayed, started, or exported by group.
   • Scheduled Execution: Retain the default value No.
   • Concurrent Extractors: Enter the number of extractors to be concurrently executed. Retain the default value 1.
   • Write Dirty Data: Specify this parameter if data that fails to be processed or filtered out during job execution needs to be written to OBS. Before writing dirty data, create an OBS link. You can view the data on OBS later. Retain the default value No so that dirty data is not recorded.
5. Click Save and Run. On the Job Management page, you can view the job execution progress and result.
   Figure 7 Job progress and execution result
   

Step 3: Query Results

After the migration job is complete, log in to the DLI management console and click SQL Editor. In the displayed page, set Engine to spark, Queue to the created SQL queue, and Database to the database created in 1. Execute the following query statement and check whether the CSS table data has been migrated to the tablecss table:

select * from tablecss;

Figure 8 Querying migrated data