Example Typical Scenario: Migrating Data from Elasticsearch to DLI

Updated on 2024-11-08 GMT+08:00

View PDF

This section describes how to use CDM's data synchronization to migrate data from a CSS Elasticsearch cluster to DLI. Data in a self-built Elasticsearch cluster can also be bidirectionally synchronized with DLI using CDM.

Prerequisites

You have created a DLI SQL queue. For how to create a DLI queue, see Creating a Queue.
CAUTION:

Set Type to For SQL when buying a queue.
You have created a CSS Elasticsearch cluster. For how to create a CSS cluster, see Creating a CSS Cluster.
In this example, the version of the created CSS cluster is 7.6.2, and the cluster is a non-security cluster.
You have created a CDM cluster. For how to create a CDM cluster, see Creating a CDM Cluster.
NOTE:
- To connect the cluster to an on-premises database as the destination data source, you can use either Internet or Direct Connect. If the Internet is used, make sure that an EIP has been bound to the CDM cluster, the security group of CDM allows outbound traffic from the host where the on-premises data source is located, the host where the data source is located can access the Internet, and the connection port has been enabled in the firewall rules.
- To successfully connect to the cloud service CSS as a data source, the following requirements must be met:
  i. If the CDM cluster and the cloud service are in different regions, they must be connected through either the Internet or Direct Connect. If the Internet is used, make sure that an EIP has been bound to the CDM cluster, the host where the data source is located can access the Internet, and the port has been enabled in the firewall rules.
  
  ii. If the CDM cluster and the cloud service are in the same region, instances in the same VPC, subnet, and security group can communicate with each other by default. If the CDM cluster and the cloud service are in the same VPC but in different subnets or security groups, you must configure routing and security group rules.
  
  For how to configure routing rules, see Configure routes. For how to configure security group rules, see Security Group Configuration Examples.
  
  iii. The cloud service instance and the CDM cluster belong to the same enterprise project. If they do not, change the enterprise project of the workspace.
In this example, the VPC, subnet, and security group of the CDM cluster match those of the CSS cluster.

Step 1: Prepare Data

Create an index for the CSS cluster and import data.

Log in to the CSS management console and choose Clusters > Elasticsearch from the navigation pane on the left.
On the Clusters page, locate the created CSS cluster and click Access Kibana in the Operation column.
In the navigation pane of Kibana, choose Dev Tools.

On the displayed Console page, run the following command to create index my_test:

PUT /my_test
{
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
          "properties": {
        "productName": {
          "type": "text",
          "analyzer": "ik_smart"
        },
        "size": {
          "type": "keyword"
        }
      }
    }
  }

Import data to the my_test index.

POST /my_test/_doc/_bulk
{"index":{}}
{"productName":"2017 Autumn New Shirts for Women", "size":"L"}
{"index":{}}
{"productName":"2017 Autumn New Shirts for Women", "size":"M"}
{"index":{}}
{"productName":"2017 Autumn New Shirts for Women", "size":"S"}
{"index":{}}
{"productName":"2018 Spring New Jeans for Women","size":"M"}
{"index":{}}
{"productName":"2018 Spring New Jeans for Women","size":"S"}
{"index":{}}
{"productName":"2017 Spring Casual Pants for Women","size":"L"}
{"index":{}}
{"productName":"2017 Spring Casual Pants for Women","size":"S"}

If errors is false in the command output, the data is imported.

Create a database and table on DLI.
1. Log in to the DLI management console. In the navigation pane on the left, choose SQL Editor. On the displayed page, set Engine to Spark and Queues to the created SQL queue.
  Create a database, for example, testdb. For the syntax to create a DLI database, see Creating a Database.
```
create database testdb;
```
2. Create a table in the database. For the table creation syntax, see Creating a DLI Table Using the DataSource Syntax.
```
create table tablecss(size string, productname string);
```

Step 2: Migrate Data

Create a CDM connection.

Create a connection to link CDM to the data source CSS.

Log in to the CDM console. In the navigation pane on the left, choose Cluster Management. On the displayed page, locate the created CDM cluster and click Job Management in the Operation column.
On the Job Management page, click the Links tab, and click Create Link. On the displayed page, select Cloud Search Service and click Next.
Figure 1 Selecting the CSS connector

Configure the connection. For details about parameter settings, see Link to Elasticsearch/CSS.

**Table 1** CSS data source configuration
Parameter	Value
Name	Name of the CSS data source, for example, source_css.
Elasticsearch Server List	Elasticsearch server list, which is automatically displayed after you click Select next to the text box and select the CSS cluster.
Security Mode Authentication	If you have enabled the security mode for the CSS cluster, set this parameter to Yes. Otherwise, set this parameter to No. In this example, set it to No.

Figure 2 Configuring the CSS connection

Click Save.

Create a connection to link CDM to DLI.
1. Log in to the CDM console. In the navigation pane on the left, choose Cluster Management. On the displayed page, locate the created CDM cluster and click Job Management in the Operation column.
2. On the Job Management page, click the Links tab, and click Create Link. On the displayed page, select Data Lake Insight and click Next.
  Figure 3 Selecting the DLI connector
3. Set the connection parameters. For details about parameter settings, see Link to DLI.
  Figure 4 Setting connection parameters
4. Click Save.

Create a CDM migration job.

Log in to the CDM console. In the navigation pane on the left, choose Cluster Management. On the displayed page, locate the created CDM cluster and click Job Management in the Operation column.
On the Job Management page, click the Table/File Migration tab. On the displayed tab, click Create Job.

On the Create Job page, set job parameters.

Figure 5 Setting CDM job parameters

Job Name: Name of the data migration job, for example, css_to_dli.

Set the parameters in the Source Job Configuration area as follows:

**Table 2** Source job parameters
Parameter	Value
Source Link Name	Select the name of the data source created in 1.a.
Index	Select the Elasticsearch index created for the CSS cluster. In this example, the my_test index created in Create an index for the CSS cluster and import data is used. Only lowercase letters are allowed.
Type	Elasticsearch type, which is similar to the table name of a relational database. Only lowercase letters are allowed. Example: _doc.

For details about other parameters, see From Elasticsearch or CSS.

Set the parameters in the Destination Job Configuration area as follows:

**Table 3** Destination job parameters
Parameter	Value
Destination Link Name	Select the DLI data source connection created in 1.b.
Resource Queue	Select a created DLI SQL queue.
Database Name	Select a created DLI database. In this example, the database testdb created in Create a database and table on DLI is selected.
Table Name	Select the name of a table in the database. In this example, the table tablecss created in Create a database and table on DLI is used.
Clear data before import	Whether to clear data in the destination table before data import. In this example, set it to No. If set to Yes, data in the destination table will be cleared before the task is started.

For details about parameter settings, see To DLI.

Click Next. The Map Field page is displayed. CDM automatically matches the source and destination fields.
- You can drag any unmatched fields to match them.
- If the type is automatically created at the migration destination, you need to configure the type and name of each field.
- CDM allows for field conversion during migration. For details, see Field Conversion.
  Figure 6 Field mapping
Click Next and set task parameters. Typically, retain the default values for all parameters.
In this step, you can configure the following optional features:
- Retry Upon Failure: If the job fails to be executed, you can determine whether to automatically retry. Retain the default value Never.
- Group: Select the group to which the job belongs. The default group is DEFAULT. On the Job Management page, jobs can be displayed, started, or exported by group.
- Scheduled Execution: For how to configure scheduled execution, see Scheduling Job Execution. Retain the default value No.
- Concurrent Extractors: Enter the number of extractors to be concurrently executed. Retain the default value 1.
- Write Dirty Data: Set this parameter if data that fails to be processed or filtered out during job execution needs to be written to OBS. Before writing dirty data, create an OBS link. You can view the data on OBS later. Retain the default value No, meaning dirty data is not recorded.
Click Save and Run. On the Job Management page, you can view the job execution progress and result.
Figure 7 Job progress and execution result

Step 3: Query Results

Once the migration job is complete, check whether the CSS table data has been migrated to the tablecss table. Specifically, do as follows: Log in to the DLI management console and choose SQL Editor. On the displayed page, set Engine to Spark, Queues to the created SQL queue, and Databases to the database created in 1. Then, execute the following query statement: