MRS Iceberg

Preparation and Constraints

• Network requirements

  The MRS Iceberg data source can communicate with CDM. This ensures smooth data transmission. For details, see Enabling Network Connectivity.

• Required permissions
  • Iceberg read and write permissions
    • Read permission: Grant the read-only permission of Iceberg to the IAM user or user group of DataArts Migration through a system policy such as MRS ReadOnlyAccess. You can also create a custom policy to grant read permissions such as SELECT.
    • Write permission: In addition to the preceding OBS permissions, you need to grant the write permission of Iceberg to the IAM user or user group of DataArts Migration through a system policy such as MRS CommonOperations and MRS FullAccess. You can also create a custom policy to grant write permissions such as INSERT INTO TABLE and CREATE TABLE.
    • OBS permissions (storage and compute analysis): When storage and compute analysis is enabled for MRS Iceberg, DataArts Migration reads files from and writes files to OBS. In this case, you must have the read and write permissions on OBS files.
  • Enabling access ports: When configuring the MRS Iceberg data source, ensure that the following ports have been enabled in the security group or network so that DataArts Migration can access MRS.

    Table 1 Service ports

    Service	Port Type	Port Number	Description
    Spark	TCP	22550	Spark JDBC Thrift port, which is used for the communications between the Spark client and Spark server
    MRS Manager	TCP	28443	Used to download the MRS cluster configuration
    		20009	Used for CAS authentication
    		20029	Used by Manager to communicate with and manage other components
    KDC	TCP&&UDP	21730	Used for Kerberos authentication
    		21731	Used for Kerberos authentication
    HDFS	TCP	8020	HDFS NameNode service port
    Hive	TCP	10000	HiveServer port used for communication between the client and HiveServer
    		9083	Hive Metastore service port used to store and manage Hive metadata
    Zookeeper	TCP	2181	ZooKeeper service port used for communication between the client and ZooKeeper cluster
    JobGateway	TCP	29974	JobBalancer port used for communication between the client and JobBalancer

Supported Data Types

DataArts Migration can read the following types of fields from MRS Iceberg based on the Spark read capability.

Supported Migration Scenarios

DataArts Migration supports the following modes for synchronizing on-premises data:

Database and table shard synchronization and entire DB migration are not supported in all regions. The following table lists the migration scenarios supported by MRS Iceberg.

Creating an Offline Data Migration Job

Create an MRS Iceberg migration job in DataArts Factory. For details, see Creating an Offline Processing Migration Job.