Updated on 2025-09-19 GMT+08:00

Preparing Data in an MRS Cluster

Before importing data from MRS to a DWS cluster, you must have:

  1. You have created an MRS cluster and a LakeFormation instance, and linked them together. For details, see Configuring a LakeFormation Data Connection During Cluster Creation.
  2. You have created a Spark Hudi table in an MRS cluster, and stored table data in the corresponding OBS path.

If you have completed the preparations, skip this section.

In this tutorial, the Spark Hudi table will be created in the MRS cluster as an example to complete the preparation work.

Creating a Spark Hudi Table in an MRS Cluster

  1. Create an MRS cluster.

    For details, see "Creating a Cluster > Creating a User-Defined Cluster" in the MapReduce Service User Guide.

  2. Download the client.

    1. Go back to the MRS cluster page. Click the cluster name. On the Dashboard tab page of the cluster details page, click Access Manager. If a message is displayed indicating that EIP needs to be bound, bind an EIP first.
    2. Enter the username admin and its password for logging in to MRS Manager. The password is the one you entered when creating the MRS cluster.
    3. Choose Cluster > Name of the desired cluster > Dashboard. On the page that is displayed, choose More > Download Client. The Download Cluster Client dialog box is displayed.

      To obtain the client of an earlier version, choose Services > Download Client and set Select Client Type to Configuration Files Only.

  3. Log in to the Spark-sql client of the MRS cluster.

    1. Log in to a Master node.

      For details, see "Logging in to a cluster > Logging In to an ECS" in the MapReduce Service User Guide.

    2. Run the following command to switch the user:
      sudo su - omm
    3. Run the following command to go to the client directory:
      cd /opt/client
    4. Run the following command to configure the environment variables:
      source bigdata_env 
      source ./Hudi/component_env
    5. If Kerberos authentication has been enabled for the current cluster, run the following command to authenticate the current user. The current user must have the permission to create Hive tables. For details, see "Creating a Role" in MapReduce Service User Guide Configure roles with corresponding permissions. For details, see section "Creating a Role" in the MapReduce Service User Guide. Bind roles to users. If the Kerberos authentication is disabled for the current cluster, skip this step.
      kinit MRS cluster user

      Example: kinit hiveuser

    6. Run the following command to start the Hive client:
      spark-sql

  4. Create a Hudi table in Spark-sql.

    Run the following commands to create a Hudi table and insert data into the table:

    CREATE TABLE test_hudi
     (  
           product_price                int            , 
           product_id                   char(30)       ,
           product_time                 date           , 
           product_level                char(10)       ,
           product_name                 varchar(200)   , 
           product_type1                varchar(20)    ,  
           product_type2                char(10)       ,  
           product_monthly_sales_cnt    int            ,  
           product_comment_time         date           ,  
           product_comment_num          int        , 
           product_comment_content      varchar(200)   
      ) 
    using hudi options ( type = 'mor', primaryKey = 'product_price',payloadClass = 'org.apache.hudi.common.model.DefaultHoodieRecordPayload',preCombineField = 'product_id' );  
    insert into test_hudi values 
    (100,XHDK-A-1293-#fJ3,2017-09-01,A,2017 Autumn New Shirt Women,red,M,328,2017-09-04,715,good), 
    (205,KDKE-B-9947-#kL5,2017-09-01,A,2017 Autumn New Knitwear Women,pink,L,584,2017-09-05,406,very good!), 
    (300,JODL-X-1937-#pV7,2017-09-01,A,2017 autumn new T-shirt men,red,XL,1245,2017-09-03,502,Bad.), 
    (310,QQPX-R-3956-#aD8,2017-09-02,B,2017 autumn new jacket women,red,L,411,2017-09-05,436,It's really super nice), 
    (250,EABE-D-1476-#oB1,2017-09-10,A,2017 autumn new dress women,black,S,841,2017-09-15,299,Follow the store for a long time.), 
    (300,HWEC-L-2531-#xP8,2017-09-23,A,2017 autumn new shoes women,brown,M,403,2017-09-26,607,good)