DWS

Preparation and Constraints

• Network requirements

  The DWS data source can communicate with CDM. This ensures smooth data transmission.

• Required permissions
  DWS read and write permissions:

  • Read permission:
    • To read data from DWS, you can grant the DWS ReadOnlyAccess system policy to the IAM user or user group of DataArts Migration. This policy allows users to view DWS resources.
    • For more refined permission control, you can create custom policies to grant users read permissions, such as the permissions to query databases and tables.
  • Write permission:
    • To write data to DWS, you can grant the DWS FullAccess or DWS Administrator system policy to the IAM user or user group of DataArts Migration. This policy has all permissions for DWS.
    • You can create custom policies to grant users write permissions, such as the permissions to insert data, create tables, and update data.

Supported Data Types

The following table lists the field types supported for reading and writing DWS data. The field types supported by DWS vary depending on the DWS version. DataArts Migration uses DWS 8.1.X as the baseline version and adapts to mainstream types and their variants.

Supported Migration Scenarios

DataArts Migration supports the following offline synchronization modes:

• Single table synchronization

  DataArts Migration supports table/file synchronization in data ingestion into a data lake or data migration to the cloud.

• Database and table shard synchronization

  DataArts Migration supports synchronization of data from multiple databases and tables in data ingestion into a data lake or data migration to the cloud.

• Entire DB migration

  DataArts Migrations supports synchronization of data from an on-premises database in data ingestion into a data lake or data migration to the cloud.

Database and table shard synchronization and entire DB migration are not supported in all regions. The following table lists the supported DWS migration scenarios.

Core Capabilities

• Connection configuration

  Configuration Item	Supported	Description
  User/AK	Yes	User AK/SK authentication ensures connection security.
  SSL encryption	Yes	SSL encryption ensures secure data transmission. Currently, SSL authentication can be enabled only for RDS.
  SSL authentication	One-way authentication	Currently, SSL authentication can be enabled only for RDS. The standard Huawei Cloud CA certificate is used for authentication.
  Connection configuration optimization	Yes	Connection configuration such as connectTimeout can be optimized to improve connection performance.
  Custom driver	No	Custom drivers provide better flexibility. They are not supported for now.

• Read capabilities

  Configuration Item	Supported	Description
  Shard concurrency	Yes	Horizontal sharding based on primary keys or common fields and multi-thread concurrent extraction significantly improve the throughput and efficiency.
  Dirty data processing	Yes	Abnormal data can be written to the dirty data bucket to prevent job failures caused by a small amount of abnormal data.
  Custom fields	Yes	You can add computed columns, constant columns, or masking functions for tasks to meet personalized service requirements.
  Incremental read	Yes	Where conditions and the SQL mode enable incremental data reading.
  Stream and batch reading	Batch reading	Batch reading improves efficiency when there is a small or medium amount of data.
  Optimization of the number of rows read	Yes	You can set Fetch Size in the connection to properly control the amount of data to be transmitted. This improves performance and prevents a transmission delay or the system from being overloaded when there is a large amount of data.
  View reading	Yes	Data can be read from views. This enables flexible data integration and processing.

• Write capabilities

  Configuration Item	Supported	Description
  Conflict resolution	Yes	Multiple write modes, such as copy and upsert, can flexibly handle data conflicts.
  Pre- and post-import processing	Yes	Operations such as preSql and delete can clean and process data before and after data import.
  Concurrent write	Yes	Concurrent write improves efficiency.
  Optimization of the number of written rows	Yes	You can set the number of rows written by each request in the connection to properly control the amount of data to be transmitted. This improves performance and prevents a transmission delay or the system from being overloaded when there is a large amount of data.
  Dirty data processing	Yes	Abnormal data can be written to the dirty data bucket to prevent job failures caused by a small amount of abnormal data.
  Table creation in editing state	Yes	You can create a table when editing a job using the table structure and field types you have defined based on the data structure and requirements.

Creating a Data Source

Create a data source in Management Center. For details, see Configuring Data Connection Parameters.

Creating an Offline Data Migration Job

Create a DWS migration job in DataArts Factory. For details, see Creating an Offline Processing Migration Job.