DataArts Fabric SQL (In Internal Test)

Huawei Cloud DataArts Fabric SQL is a fully managed serverless data platform that can analyze terabytes to petabytes of data in lakehouses in seconds. It is free of O&M and provides a set of SQL that streamlines AI and big data processing.

DataArts Migration can efficiently migrate data from and to DataArts Fabric SQL (Fabric in short).

How It Works

FabricWriter uses a two-level commit mechanism, including file flushing to disks and data import to atomic libraries. First, a temporary Fabric table is created, and upstream data is written to the Huawei Cloud OBS directory in the file format corresponding to the temporary table. Then, the INSERT/OVERWRITE SQL statement is executed to load the data in the temporary table to the formal FabricSQL table in transaction mode at a time. Data is directly written to OBS files, which provides excellent write performance and achieves high-throughput batch data integration that features less table locking and can be rolled back.

Preparation and Constraints

Network requirements
The DataArts Fabric SQL data source can communicate with CDM. This ensures smooth data transmission. For details, see Enabling Network Connectivity and Configuring Networks for the DataArts Fabric SQL Data Source.
Required permissions
- Fabric write permission: To write data to Fabric, assign the DataArtsFabricFullPolicy policy (including INSERT and CREATE TABLE permissions).
- LakeFormation metadata read and write permissions: Fabric uses LakeFormation as the metadata component. To enable Fabric to read and write LakeFormation data, assign the LakeFormation CommonOperations or LakeFormation FullAccess system policy.
- OBS write permission: DataArts Migration reads files from and writes files to OBS. You need to assign the OBS OperateAccess or OBS Administrator system policy.

Supported Data Types

The following table lists the types of data that can be written to Fabric.

Data Type	Fabric Data Type	Write
Numeric	SMALLINT	√
	INTEGER	√
	BIGINT	√
	NUMERIC	√
	DECIMAL	√
	REAL	√
	FLOAT4	√
	FLOAT8	√
	DOUBLE PRECISION	√
	FLOAT[(p)]	√
Boolean	BOOLEAN	√
Character	CHAR	√
	CHARACTER	√
	NCHAR	√
	VARCHAR(n)	√
	CHARACTER VARYING	√
	TEXT	√
Date/Time	DATE	√
Date/Time	TIMESTAMP	√
Binary	BYTEA	√

Supported Migration Scenarios

DataArts Migration supports the following modes for synchronizing on-premises data:

Single table synchronization
DataArts Migration supports table/file synchronization in data ingestion into a data lake or data migration to the cloud.
Database and table shard synchronization
DataArts Migration supports synchronization of data from multiple databases and tables in data ingestion into a data lake or data migration to the cloud.
Entire DB migration
DataArts Migrations supports synchronization of data from an on-premises database in data ingestion into a data lake or data migration to the cloud.

Database and table shard synchronization and entire DB migration are not supported in all regions. The following table lists the supported DataArts Fabric SQL migration scenarios.

Supported Migration Scenario	Single Table Read	Single Table Write	Database/Table Shard Read	Database/Table Shard Write	Entire DB Read	Entire DB Write
Supported	x	√	x	√	x	x

Core Capabilities

Connection configuration

Configuration Item	Supported	Description
AK/SK authentication	√	AK/SK authentication is used to access Fabric.
Agency authentication	x	An IAM agency authorizes service roles to access the service. Currently, this function is not supported.

Write capabilities

Configuration Item	Supported	Description
Write mode	Insert into Insert overwrite	Two write modes are supported: INSERT INTO and INSERT OVERWRITE. INSERT INTO adds data to a destination table and is applicable to writing incremental data. INSERT OVERWRITE overwrites the data of a destination table or partition and is applicable to updating full data.
Dirty data processing	x	Abnormal data cannot be written to the dirty data bucket to prevent job failures caused by a small amount of abnormal data.
Concurrent write	√	Concurrent write can fully utilize cluster resources to improve the data write speed.
Table creation in editing state	√	A destination table can be created during the configuration of a job migrating data from a semi-structured or structured data source to Fabric.