Bu sayfa henüz yerel dilinizde mevcut değildir. Daha fazla dil seçeneği eklemek için yoğun bir şekilde çalışıyoruz. Desteğiniz için teşekkür ederiz.

Compute
Elastic Cloud Server
Huawei Cloud Flexus
Bare Metal Server
Auto Scaling
Image Management Service
Dedicated Host
FunctionGraph
Cloud Phone Host
Huawei Cloud EulerOS
Networking
Virtual Private Cloud
Elastic IP
Elastic Load Balance
NAT Gateway
Direct Connect
Virtual Private Network
VPC Endpoint
Cloud Connect
Enterprise Router
Enterprise Switch
Global Accelerator
Management & Governance
Cloud Eye
Identity and Access Management
Cloud Trace Service
Resource Formation Service
Tag Management Service
Log Tank Service
Config
OneAccess
Resource Access Manager
Simple Message Notification
Application Performance Management
Application Operations Management
Organizations
Optimization Advisor
IAM Identity Center
Cloud Operations Center
Resource Governance Center
Migration
Server Migration Service
Object Storage Migration Service
Cloud Data Migration
Migration Center
Cloud Ecosystem
KooGallery
Partner Center
User Support
My Account
Billing Center
Cost Center
Resource Center
Enterprise Management
Service Tickets
HUAWEI CLOUD (International) FAQs
ICP Filing
Support Plans
My Credentials
Customer Operation Capabilities
Partner Support Plans
Professional Services
Analytics
MapReduce Service
Data Lake Insight
CloudTable Service
Cloud Search Service
Data Lake Visualization
Data Ingestion Service
GaussDB(DWS)
DataArts Studio
Data Lake Factory
DataArts Lake Formation
IoT
IoT Device Access
Others
Product Pricing Details
System Permissions
Console Quick Start
Common FAQs
Instructions for Associating with a HUAWEI CLOUD Partner
Message Center
Security & Compliance
Security Technologies and Applications
Web Application Firewall
Host Security Service
Cloud Firewall
SecMaster
Anti-DDoS Service
Data Encryption Workshop
Database Security Service
Cloud Bastion Host
Data Security Center
Cloud Certificate Manager
Edge Security
Situation Awareness
Managed Threat Detection
Blockchain
Blockchain Service
Web3 Node Engine Service
Media Services
Media Processing Center
Video On Demand
Live
SparkRTC
MetaStudio
Storage
Object Storage Service
Elastic Volume Service
Cloud Backup and Recovery
Storage Disaster Recovery Service
Scalable File Service Turbo
Scalable File Service
Volume Backup Service
Cloud Server Backup Service
Data Express Service
Dedicated Distributed Storage Service
Containers
Cloud Container Engine
Software Repository for Container
Application Service Mesh
Ubiquitous Cloud Native Service
Cloud Container Instance
Databases
Relational Database Service
Document Database Service
Data Admin Service
Data Replication Service
GeminiDB
GaussDB
Distributed Database Middleware
Database and Application Migration UGO
TaurusDB
Middleware
Distributed Cache Service
API Gateway
Distributed Message Service for Kafka
Distributed Message Service for RabbitMQ
Distributed Message Service for RocketMQ
Cloud Service Engine
Multi-Site High Availability Service
EventGrid
Dedicated Cloud
Dedicated Computing Cluster
Business Applications
Workspace
ROMA Connect
Message & SMS
Domain Name Service
Edge Data Center Management
Meeting
AI
Face Recognition Service
Graph Engine Service
Content Moderation
Image Recognition
Optical Character Recognition
ModelArts
ImageSearch
Conversational Bot Service
Speech Interaction Service
Huawei HiLens
Video Intelligent Analysis Service
Developer Tools
SDK Developer Guide
API Request Signing Guide
Terraform
Koo Command Line Interface
Content Delivery & Edge Computing
Content Delivery Network
Intelligent EdgeFabric
CloudPond
Intelligent EdgeCloud
Solutions
SAP Cloud
High Performance Computing
Developer Services
ServiceStage
CodeArts
CodeArts PerfTest
CodeArts Req
CodeArts Pipeline
CodeArts Build
CodeArts Deploy
CodeArts Artifact
CodeArts TestPlan
CodeArts Check
CodeArts Repo
Cloud Application Engine
MacroVerse aPaaS
KooMessage
KooPhone
KooDrive
Help Center/ Data Lake Insight/ Best Practices/ Data Migration/ Migrating Data from Elasticsearch to DLI

Migrating Data from Elasticsearch to DLI

Updated on 2024-09-20 GMT+08:00

This section describes how to use the CDM data synchronization function to migrate data from a CSS Elasticsearch cluster to DLI. Data in a self-built Elasticsearch cluster can also be bidirectionally synchronized between CDM and DLI.

Prerequisites

  • You have created a DLI SQL queue. For details about how to create a DLI queue, see Creating a Queue.
    CAUTION:

    When you create a queue, set its Type to For SQL.

  • You have created a CSS Elasticsearch cluster. For how to create a CSS cluster, see Creating a CSS Cluster.

    In this example, the version of the created CSS cluster is 7.6.2, and security mode is disabled for the cluster.

  • You have created a CDM cluster. For details about how to create a CDM cluster, see Creating a CDM Cluster.
    NOTE:
    • If the destination data source is an on-premises database, you need the Internet or Direct Connect. When using the Internet, ensure that an EIP has been bound to the CDM cluster, the security group of CDM allows outbound traffic from the host where the off-cloud data source is located, the host where the data source is located can access the Internet, and the connection port has been enabled in the firewall rules.
    • If the data source is CSS on a cloud, the network must meet the following requirements:

      i. If the CDM cluster and the cloud service are in different regions, a public network or a dedicated connection is required for enabling communication between the CDM cluster and the cloud service. If the Internet is used for communication, ensure that an EIP has been bound to the CDM cluster, the host where the data source is located can access the Internet, and the port has been enabled in the firewall rules.

      ii. If the CDM cluster and the cloud service are in the same region, VPC, subnet, and security group, they can communicate with each other by default. If the CDM cluster and the cloud service are in the same VPC but in different subnets or security groups, you must configure routing rules and security group rules.

      For details about how to configure routes, see Configure routes. For details about how to configure security groups, see section Security Group Configuration Examples.

      iii. The cloud service instance and the CDM cluster belong to the same enterprise project. If they do not, you can modify the enterprise project of the workspace.

    In this example, the VPC, subnet, and security group of the CDM cluster are the same as those of the CSS cluster.

Step 1: Prepare Data

  • Create an index for the CSS cluster and import data.
    1. Log in to the CSS management console and choose Clusters > Elasticsearch from the navigation pane on the left.
    2. On the Clusters page, click Access Kibana in the Operation column of the created CSS cluster.
    3. In the navigation pane of Kibana, choose Dev Tools. The Console page is displayed.
    4. On the displayed Console page, run the following command to create index my_test:
      PUT /my_test
      {
        "settings": {
          "number_of_shards": 1
        },
        "mappings": {
                "properties": {
              "productName": {
                "type": "text",
                "analyzer": "ik_smart"
              },
              "size": {
                "type": "keyword"
              }
            }
          }
        }
    5. Run the following command to import data to the my_test index:
      POST /my_test/_doc/_bulk
      {"index":{}}
      {"productName":"2017 Autumn New Shirts for Women", "size":"L"}
      {"index":{}}
      {"productName":"2017 Autumn New Shirts for Women", "size":"M"}
      {"index":{}}
      {"productName":"2017 Autumn New Shirts for Women", "size":"S"}
      {"index":{}}
      {"productName":"2018 Spring New Jeans for Women","size":"M"}
      {"index":{}}
      {"productName":"2018 Spring New Jeans for Women","size":"S"}
      {"index":{}}
      {"productName":"2017 Spring Casual Pants for Women","size":"L"}
      {"index":{}}
      {"productName":"2017 Spring Casual Pants for Women","size":"S"}

      If errors is false in the command output, the data is imported.

  • Create a database and table on DLI.
    1. Log in to the DLI management console and click SQL Editor. On the displayed page, set Engine to spark and Queue to the created SQL queue.

      Enter the following statement in the editing window to create a database, for example, the migrated DLI database testdb: For details about the syntax for creating a DLI database, see Creating a Database.

      create database testdb;
    2. Create a table in the database. For details about the table creation syntax, see Creating a DLI Table Using the DataSource Syntax.
      create table tablecss(size string, productname string);

Step 2: Migrate Data

  1. Create a CDM connection to MRS Hive.
    1. Create a connection to link CDM to the data source CSS.
      1. Log in to the CDM console, choose Cluster Management. On the displayed page, locate the created CDM cluster, and click Job Management in the Operation column.
      2. On the Job Management page, click the Links tab, and click Create Link. On the displayed page, select Cloud Search Service and click Next.
        Figure 1 Selecting the CSS connector
      3. Configure the connection. The following table describes the required parameters. For details about parameter settings, see Link to Elasticsearch/CSS.
        Table 1 CSS data source configuration

        Parameter

        Value.

        Name

        Name of the CSS data source, for example, source_css.

        Elasticsearch Server List

        Click Select next to the text box and select the CSS cluster. The Elasticsearch server list is automatically displayed.

        Security mode Authentication

        If you have enabled the security mode for the CSS cluster, set this parameter to Yes. Otherwise, set this parameter to No.

        In this example, set this parameter to No.

        Figure 2 Configuring the CSS connection
      4. Click Save to complete the configuration.
    2. Create a connection to link CDM to DLI.
      1. Log in to the CDM console, choose Cluster Management. On the displayed page, locate the created CDM cluster, and click Job Management in the Operation column.
      2. On the Job Management page, click the Links tab, and click Create Link. On the displayed page, select Data Lake Insight and click Next.
        Figure 3 Selecting the DLI connector
      3. Configure the connection parameters. For details about parameter settings, see Link to DLI.
        Figure 4 Configuring connection parameters
      4. After the configuration is complete, click Save.
  2. Create a CDM migration job.
    1. Log in to the CDM console, choose Cluster Management. On the displayed page, locate the created CDM cluster, and click Job Management in the Operation column.
    2. On the Job Management page, choose the Table/File Migration tab and click Create Job.
    3. On the Create Job page, specify job information.
      Figure 5 Configuring the CDM job
      1. Job Name: Name of the data migration job, for example, css_to_dli
      2. Set parameters required for Source Job Configuration.
        Table 2 Source job configuration parameters

        Parameter

        Value

        Source Link Name

        Select the name of the data source created in 1.a.

        Index

        Select the Elasticsearch index created for the CSS cluster. In this example, the my_test index created in Create an index for the CSS cluster and import data is used.

        The index can contain only lowercase letters.

        Type

        Elasticsearch type, which is similar to the table name of a relational database. The type name can contain only lowercase letters. Example: _doc.

        For details about other parameters, see From Elasticsearch or CSS.

      3. Set parameters required for Destination Job Configuration.
        Table 3 Destination job configuration parameters

        Parameter

        Value

        Destination Link Name

        Select the DLI data source connection created in 1.b.

        Resource Queue

        Select a created DLI SQL queue.

        Database

        Select a created DLI database. In this example, database testdb created in Create a database and table on DLI is selected.

        Table

        Select the name of a table in the database. In this example, table tablecss created in Create a database and table on DLI is created.

        Clear data before import

        Whether to clear data in the destination table before data import. In this example, set this parameter to No.

        If this parameter is set to Yes, data in the destination table will be cleared before the task is started.

        For details about parameter settings, see To DLI.

  3. Click Next. The Map Field page is displayed. CDM automatically matches the source and destination fields.
    • If the field mapping is incorrect, you can drag the fields to adjust the mapping.
    • If the type is automatically created at the migration destination, you need to configure the type and name of each field.
    • CDM allows for field conversion during migration. For details, see Field Conversion.
      Figure 6 Field mapping
  4. Click Next and set task parameters. Generally, retain the default values of all parameters.

    In this step, you can configure the following optional functions:

    • Retry Upon Failure: If the job fails to be executed, you can determine whether to automatically retry. Retain the default value Never.
    • Group: Select the group to which the job belongs. The default group is DEFAULT. On the Job Management page, jobs can be displayed, started, or exported by group.
    • Scheduled Execution: For details about how to configure scheduled execution, see Scheduling Job Execution. Retain the default value No.
    • Concurrent Extractors: Enter the number of extractors to be concurrently executed. Retain the default value 1.
    • Write Dirty Data: Specify this parameter if data that fails to be processed or filtered out during job execution needs to be written to OBS. Before writing dirty data, create an OBS link. You can view the data on OBS later. Retain the default value No so that dirty data is not recorded.
  5. Click Save and Run. On the Job Management page, you can view the job execution progress and result.
    Figure 7 Job progress and execution result

Step 3: Query Results

After the migration job is complete, log in to the DLI management console and click SQL Editor. In the displayed page, set Engine to spark, Queue to the created SQL queue, and Database to the database created in 1. Execute the following query statement and check whether the CSS table data has been migrated to the tablecss table:
select * from tablecss;
Figure 8 Querying migrated data

Sitemizi ve deneyiminizi iyileştirmek için çerezleri kullanırız. Sitemizde tarama yapmaya devam ederek çerez politikamızı kabul etmiş olursunuz. Daha fazla bilgi edinin

Feedback

Feedback

Feedback

0/500

Selected Content

Submit selected content with the feedback