DataArts Studio
DataArts Studio
All results for "
" in this service
All results for "
" in this service
What's New
Function Overview
Service Overview
DataArts Studio Infographics
What Is DataArts Studio?
Basic Concepts
Functions
Advantages
Application Scenarios
Versions
Billing
Permission Management
Permissions
Notes and Constraints
Related Services
Data Governance Methodology
Purpose
Intended Audience
Data Governance Framework
Framework
Data Governance Modules
Module Relationships
Data Governance Organizational Architecture
Framework
Responsibilities
Measurement and Evaluation System
Data Governance Methods
Measurement Dimensions
Measurement and Scoring Rules
Huawei Data Governance Cases
Thoughts
Practices
Effects
Thoughts on Data Governance and the Impact of COVID-19
Implementation of the Data Governance Methodology
Getting Started
Quick Start Guide
Beginners: DLI-powered Data Development Based on E-commerce BI Reports
Scenario
Step 1: Prepare Data
Step 2: Develop Data
Step 3: Unsubscribe from Services
Novices: DWS-powered Data Integration and Development Based on Movie Scores
Scenario
Step 1: Prepare Data
Step 2: Integrate Data
Step 3: Develop Data
Step 4: Unsubscribe from Services
Experienced Users: MRS Hive-powered Data Governance Based on Taxi Trip Data
Example Scenario
Step 1: Design a Process
Step 2: Prepare Data
Step 3: DataArts Migration
Step 4: Metadata Collection
Step 5: Design Data Architecture
Step 6: Develop Data
Step 7: DataArts Quality
Step 8: View Data Assets
Step 9: Unsubscribe from Services
User Guide
DataArts Studio development process
Buying and Configuring a DataArts Studio Instance
Buying a DataArts Studio Instance
Buying a DataArts Studio Incremental Package
Introduction to Incremental Packages
Buying a DataArts Migration Incremental Package
Buying a DataArts Migration Resource Group Incremental Package
Buying a DataArts DataService Exclusive Cluster Incremental Package
Accessing the DataArts Studio Instance Console
Creating and Configuring a Workspace in Simple Mode
Creating a Workspace in Simple Mode
Setting Workspace Quotas
(Optional) Changing the Job Log Storage Path
(Optional) Creating and Using a Workspace in Enterprise Mode
Introduction to the Enterprise Mode
Creating a Workspace in Enterprise Mode
Operations Supported for Different Roles in Enterprise Mode
Service Process in Enterprise Mode
Admin Operations
Developer Operations
Deployer Operations
Operator Operations
Managing DataArts Studio Resources
Associating a Real-Time Migration Resource Group with Workspaces
Authorizing Users to Use DataArts Studio
Creating an IAM User and Assigning DataArts Studio Permissions
Authorizing the Use of Real-Time Data Migration
Adding Workspace Members and Assigning Roles
Management Center
Data Sources Supported by DataArts Studio
Creating a DataArts Studio Data Connection
Configuring DataArts Studio Data Connection Parameters
DWS Connection Parameters
DLI Connection Parameters
MRS Hive Connection Parameters
MRS HBase Connection Parameters
MRS Kafka Connection Parameters
MRS Spark Connection Parameters
MRS ClickHouse Connection Parameters
MRS Hetu Connection Parameters
MRS Impala Connection Parameters
MRS Ranger Connection Parameters
MRS Presto Connection Parameters
Doris Connection Parameters
OpenSource ClickHouse Connection Parameters
RDS Connection Parameters
Oracle Connection Parameters
Host Connection Parameters
Rest Client Connection Parameters
Redis Connection Parameters
SAP HANA Connection Parameters
LTS Connection Parameters
Configuring DataArts Studio Resource Migration
Configuring Environment Isolation for a DataArts Studio Workspace in Enterprise Mode
Typical Scenarios for Using Management Center
Creating a Connection Between DataArts Studio and an MRS Hive Data Lake
Creating a Connection Between DataArts Studio and a GaussDB(DWS) Data Lake
Creating a Connection Between DataArts Studio and a MySQL Database
DataArts Migration (CDM Jobs)
Overview
Notes and Constraints
Supported Data Sources
Supported Data Sources (2.10.0.300)
Supported Data Sources (2.9.3.300)
Supported Data Sources (2.9.2.200)
Supported Data Types
Creating and Managing a CDM Cluster
Creating a CDM Cluster
Binding or Unbinding an EIP
Restarting a CDM Cluster
Deleting a CDM Cluster
Downloading CDM Cluster Logs
Viewing and Modifying CDM Cluster Configurations
Managing and Viewing CDM Metrics
CDM Metrics
Configuring CDM Alarm Rules
Querying CDM Metrics
Creating a Link in a CDM Cluster
Creating a Link Between CDM and a Data Source
Configuring Link Parameters
OBS Link Parameters
PostgreSQL/SQLServer Link Parameters
GaussDB(DWS) Link Parameters
RDS for MySQL/MySQL Database Link Parameters
Oracle Database Link Parameters
DLI Link Parameters
Hive Link Parameters
HBase Link Parameters
HDFS Link Parameters
FTP/SFTP Link Parameters
Redis Link Parameters
DDS Link Parameters
CloudTable Link Parameters
MongoDB Link Parameters
Cassandra Link Parameters
Kafka Link Parameters
DMS Kafka Link Parameters
CSS Link Parameters
Elasticsearch Link Parameters
Dameng Database Link Parameters
SAP HANA Link Parameters
Shard Link Parameters
MRS Hudi Link Parameters
MRS ClickHouse Link Parameters
ShenTong Database Link Parameters
LogHub (SLS) Link Parameters
CloudTable OpenTSDB Link Parameters
GBASE Link Parameters
YASHAN Link Parameters
Uploading a CDM Link Driver
Creating a Hadoop Cluster Configuration
Creating a Job in a CDM Cluster
Table/File Migration Jobs
Creating an Entire Database Migration Job
Configuring CDM Source Job Parameters
From OBS
From HDFS
From HBase/CloudTable
From Hive
From DLI
From FTP/SFTP
From HTTP
From PostgreSQL/SQL Server
From DWS
From SAP HANA
From MySQL
From Oracle
From a Database Shard
From MongoDB/DDS
From Redis
From Kafka/DMS Kafka
From Elasticsearch or CSS
From OpenTSDB
From MRS Hudi
From MRS ClickHouse
From LogHub (SLS)
From a ShenTong Database
From a Dameng Database
From YASHAN
Configuring CDM Destination Job Parameters
To OBS
To HDFS
To HBase/CloudTable
To Hive
To MySQL/SQL Server/PostgreSQL
To Oracle
To DWS
To DDS
To Redis
To Elasticsearch/CSS
To DLI
To OpenTSDB
To MRS Hudi
To MRS ClickHouse
To MongoDB
Configuring CDM Job Field Mapping
Configuring a Scheduled CDM Job
Managing CDM Job Configuration
Managing a CDM Job
Managing CDM Jobs
Using Macro Variables of Date and Time
Improving Migration Performance
How Migration Jobs Work
Performance Tuning
Reference: Job Splitting Dimensions
Reference: CDM Performance Test Data
Key Operation Guide
Incremental Migration
Incremental File Migration
Incremental Migration of Relational Databases
HBase/CloudTable Incremental Migration
MongoDB/DDS Incremental Migration
Migration in Transaction Mode
Encryption and Decryption During File Migration
MD5 Verification
Configuring Field Converters
Adding Fields
Migrating Files with Specified Names
Regular Expressions for Separating Semi-structured Text
Recording the Time When Data Is Written to the Database
File Formats
Converting Unsupported Data Types
Auto Table Creation
Tutorials
Creating an MRS Hive Link
Creating a MySQL Link
Migrating Data from MySQL to MRS Hive
Migrating Data from MySQL to OBS
Migrating Data from MySQL to DWS
Migrating an Entire MySQL Database to RDS
Migrating Data from Oracle to CSS
Migrating Data from Oracle to DWS
Migrating Data from OBS to CSS
Migrating Data from OBS to DLI
Migrating Data from MRS HDFS to OBS
Migrating the Entire Elasticsearch Database to CSS
Error Codes
DataArts Migration (Offline Jobs)
Overview of Offline Jobs
Supported Data Sources
Creating an Offline Processing Migration Job
Configuring an Offline Processing Migration Job
Configuring Source Job Parameters
From MySQL
From Hive
From HDFS
From Hudi
From PostgreSQL
From SQLServer
From Oracle
From DLI
From OBS
From SAP HANA
From Kafka
From Rest Client
From DWS
From FTP/SFTP
From Doris
From HBase
From ClickHouse
From Elasticsearch
From MongoDB
From RestApi
From GBase
From Redis
From LTS
Configuring Destination Job Parameters
To PostgreSQL
To Oracle
To MySQL
To SQLServer
To Hudi
To Hive
To DLI
To Elasticsearch
To DWS
To OBS
To SAP HANA
To ClickHouse
To Doris
To HBase
To MongoDB
To MRS Kafka
To GBase
To Redis
To HDFS
Configuring Field Converters
Adding Fields
DataArts Migration (Real-Time Jobs)
Overview of Real-Time Jobs
Notes and Constraints
Supported Data Sources
Check Before Use
Enabling Network Communications
Database Deployed in an On-premises IDC
Using Direct Connect to Enable Network Communications
Using VPN to Enable Network Communications
Using a Public Network to Enable Network Communications
Database Deployed on Another Cloud
Using Direct Connect to Enable Network Communications
Using VPN to Enable Network Communications
Using a Public Network to Enable Network Communications
Database Deployed on Huawei Cloud
Enabling Network Communications Directly for the Same Region and Tenant
Using a VPC Peering Connection to Enable Network Communications for the Same Region but Different Tenants
Using an Enterprise Router to Enable Network Communications for the Same Region but Different Tenants
Using a Cloud Connection to Enable Cross-Region Network Communications
Creating a Real-Time Migration Job
Configuring a Real-Time Migration Job
Real-Time Migration Job O&M
Viewing Monitoring Metrics
Viewing Synchronization Logs
Creating an Alarm Rule
Modifying Job Configurations
Field Type Mapping
Mapping Between MySQL and MRS Hudi Field Types
Mapping Between MySQL and GaussDB(DWS) Field Types
Mapping Between PostgreSQL and GaussDB(DWS) Field Types
Mapping Between PostgreSQL and MRS Hudi Field Types
Mapping Between Centralized/Distributed GaussDB and GaussDB(DWS) Field Types
Mapping Between Centralized/Distributed GaussDB and MRS Hudi Field Types
Mapping Between SQL Server and GaussDB(DWS) Field Types
Mapping Between SQL Server and Hudi Field Types
Mapping Between Oracle and MRS Hudi Field Types
Job Performance Optimization
Overview
Optimizing Job Parameters
Optimizing the Parameters of a Job for Migrating Data from MRS Kafka to MRS Hudi
Optimizing the Parameters of a Job for Migrating Data from MySQL to MRS Hudi
Optimizing the Parameters of a Job for Migrating Data from MySQL to GaussDB(DWS)
Optimizing the Parameters of a Job for Migrating Data from MySQL to
Optimizing the Parameters of a Job for Migrating Data from DMS for Kafka to OBS
Optimizing the Parameters of a Job for Migrating Data from Apache Kafka to MRS Kafka
Optimizing the Parameters of a Job for Migrating Data from SQL Server to MRS Hudi
Optimizing the Parameters of a Job for Migrating Data from PostgreSQL to GaussDB(DWS)
Optimizing the Parameters of a Job for Migrating Data from PostgreSQL to MRS Hudi
Optimizing the Parameters of a Job for Migrating Data from Oracle to GaussDB(DWS)
Optimizing the Parameters of a Job for Migrating Data from Oracle to MRS Hudi
Optimizing the Parameters of a Job for Migrating Data from SQL Server to GaussDB(DWS)
Optimizing the Parameters of a Job for Migrating Data from a Centralized/Distributed GaussDB to GaussDB(DWS)
Optimizing the Parameters of a Job for Migrating Data from a Centralized/Distributed GaussDB to MRS Hudi
Tutorials
Overview
Migrating a DRS Task to DataArts Migration
Configuring a Job for Synchronizing Data from MySQL to MRS Hudi
Configuring a Job for Synchronizing Data from MySQL to GaussDB(DWS)
Configuring a Job for Synchronizing Data from MySQL to Kafka
Configuring a Job for Synchronizing Data from DMS for Kafka to OBS
Configuring a Job for Synchronizing Data from Apache Kafka to MRS Kafka
Configuring a Job for Synchronizing Data from to Hudi
Configuring a Job for Synchronizing Data from SQL Server to MRS Hudi
Configuring a Job for Synchronizing Data from SQL Server to GaussDB(DWS)
Configuring a Job for Synchronizing Data from PostgreSQL to GaussDB(DWS)
Configuring a Job for Synchronizing Data from PostgreSQL to MRS Hudi
Configuring a Job for Synchronizing Data from PostgreSQL to
Configuring a Job for Synchronizing Data from Oracle to GaussDB(DWS)
Configuring a Job for Synchronizing Data from Oracle to MRS Hudi
Configuring a Job for Synchronizing Data from Oracle to
Configuring a Job for Synchronizing Data from MongoDB to GaussDB(DWS)
Configuring a Job for Migrating Data from a Centralized/Distributed GaussDB to GaussDB(DWS)
Configuring a Job for Migrating Data from a Centralized/Distributed GaussDB to MRS Hudi
Configuring a Job for Synchronizing Data from a Centralized/Distributed GaussDB to
DataArts Architecture
Overview
DataArts Architecture Use Process
Adding Reviewers
Data Survey
Designing Processes
Designing Subjects
Logical Models
Standards Design
Creating a Lookup Table
Creating Data Standards
Model Design
Data Warehouse Planning
ER Modeling
Dimensional Modeling
Creating Dimensions
Managing Dimension Tables
Creating Fact Tables
Data Mart
Metric Design
Business Metrics
Technical Metrics
Creating Atomic Metrics
Creating Derivative Metrics
Creating Compound Metrics
Creating Time Filters
Common Operations
Reversing a Database (ER Modeling)
Reversing a Database (Dimensional Modeling)
Importing/Exporting Data
Associating Quality Rules
Viewing Tables
Modifying Subjects, Directories, and Processes
Managing the Configuration Center
Review Center
Tutorials
DataArts Architecture Example
DataArts Factory
Overview
Data Management
Data Management Process
Creating a Data Connection
Creating a Database
(Optional) Creating a Database Schema
Creating a Table
Script Development
Script Development Process
Creating a Script
Developing Scripts
Developing an SQL Script
Developing a Shell Script
Developing a Python Script
Submitting a Version
Releasing a Script Task
(Optional) Managing Scripts
Copying a Script
Copying the Script Name and Renaming a Script
Moving a Script or Script Directory
Exporting and Importing Scripts
Viewing Script References
Deleting a Script
Unlocking a Script
Changing the Script Owner
Unlocking Scripts
Job Development
Job Development Process
Creating a Job
Developing a Pipeline Job
Developing a Batch Processing Single-Task SQL Job
Developing a Real-Time Processing Single-Task MRS Flink SQL Job
Developing a Real-Time Processing Single-Task MRS Flink Jar Job
Developing a Real-Time Processing Single-Task DLI Spark Job
Setting Up Scheduling for a Job
Submitting a Version
Releasing a Job Task
(Optional) Managing Jobs
Copying a Job
Copying the Job Name and Renaming a Job
Moving a Job or Job Directory
Exporting and Importing Jobs
Configuring Jobs
Deleting a Job
Unlocking a Job
Viewing a Job Dependency Graph
Changing the Job Owner
Unlocking Jobs
Going to Monitor Job page
Solution
Execution History
O&M and Scheduling
Overview
Monitoring a Job
Monitoring a Batch Job
Monitoring a Real-Time Job
Instance Monitoring
Monitoring PatchData
Notification Management
Managing Notifications
Cycle Overview
Managing Terminal Subscriptions
Managing Backups
Operation History
Configuration and Management
Configuring Resources
Configuring Environment Variables
Configuring an OBS Bucket
Managing Job Tags
Configuring a Scheduling Identity
Configuring the Number of Concurrently Running Nodes
Configuring a Template
Configuring a Scheduling Calendar
Configuring a Default Item
Configuring Task Groups
Managing Resources
Review Center
Download Center
Node Reference
Node Overview
Node Lineages
Data Lineage Overview
Configuring Data Lineages
Viewing Data Lineages
CDM Job
Data Migration
Rest Client
Import GES
MRS Kafka
Kafka Client
ROMA FDI Job
DLI Flink Job
DLI SQL
DLI Spark
DWS SQL
MRS Spark SQL
MRS Hive SQL
MRS Presto SQL
MRS Spark
MRS Spark Python
MRS ClickHouse
MRS Impala SQL
MRS Flink Job
MRS MapReduce
CSS
Shell
RDS SQL
ETL Job
Python
DORIS SQL
ModelArts Train
Create OBS
Delete OBS
OBS Manager
Open/Close Resource
Data Quality Monitor
Subjob
For Each
SMN
Dummy
EL Expression Reference
Expression Overview
Basic Operators
Date and Time Mode
Env Embedded Objects
Job Embedded Objects
StringUtil Embedded Objects
DateUtil Embedded Objects
JSONUtil Embedded Objects
Loop Embedded Objects
OBSUtil Embedded Objects
Examples of Common EL Expressions
EL Expression Use Examples
Simple Variable Set
Usage Guidance
Referencing Parameters in Scripts and Jobs
Setting the Job Scheduling Time to the Last Day of Each Month
Configuring a Yearly Scheduled Job
Using PatchData
Obtaining the Output of an SQL Node
Obtaining the Maximum Value and Transferring It to a CDM Job Using a Query SQL Statement
IF Statements
Obtaining the Return Value of a Rest Client Node
Using For Each Nodes
Using Script Templates and Parameter Templates
Developing a Python Job
Developing a DWS SQL Job
Developing a Hive SQL Job
Developing a DLI Spark Job
Developing an MRS Flink Job
Developing an MRS Spark Python Job
DataArts Quality
Metric Monitoring (Pending Offline)
Overview
Creating a Metric
Creating a Rule
Creating a Scenario
Viewing a Scenario Instance
Monitoring Data Quality
Overview
Creating a Data Quality Rule
Creating a Data Quality Job
Creating a Data Comparison Job
Viewing Job Instances
Viewing Data Quality Reports
Tutorials
Creating a Business Scenario
Creating a Quality Job
Creating a Comparison Job
DataArts Catalog
Viewing the Workspace Data Map
Viewing Data Assets in a Workspace
Viewing the Asset Overview
Viewing Data Assets
Managing Asset Tags
Configuring Data Access Permissions (To Be Removed)
Introduction to Data Permissions (To Be Removed)
Configuring Data Catalog Permissions (To Be Removed)
Configuring Table Permissions (To Be Removed)
Managing Review Center (To Be Removed)
Configuring Data Security Policies (To Be Removed)
Introduction to Data Security (To Be Removed)
Creating a Data Security Level (To Be Removed)
Creating a Data Classification (To Be Removed)
Creating a Data Masking Policy (To Be Removed)
Collecting Metadata of Data Sources
Overview
Configuring a Metadata Collection Task
Viewing Task Monitoring Information
Tutorial for Typical Scenarios of DataArts Catalog
Configuring an Incremental Metadata Collection Task
Viewing Data Lineages Through DataArts Catalog
Data Lineage Overview
Configuring Data Lineages
Viewing Data Lineages
DataArts Security
Overview
Dashboard
Unified Permission Governance
Permission Governance Process
Authorizing dlg_agency
Checking the Cluster Version and Permissions
Synchronizing IAM Users to the Data Source
Controlling Data Access Using Permissions
Configuring Workspace Permission Sets
Configuring Permission Sets
Configuring Roles
Managing Members
Configuring Row-level Access Control
Synchronizing MRS Hive and Hetu Permissions
Applying for Permissions and Approving Permissions
Enabling Fine-grained Authentication
Controlling Service Resource Access
Configuring Queue Permissions
Configuring Workspace Resource Permission Policies
Controlling Ranger Access Using Permissions
Configuring Resource Permissions
Viewing Permission Reports
Sensitive Data Governance
Sensitive Data Governance Process
Creating Data Security Levels
Creating Data Classifications
Creating Identification Rules
Creating Identification Rule Groups
Discovering Sensitive Data
Viewing Sensitive Data Distribution
Managing Sensitive Data
Sensitive Data Protection
Overview
Static Masking Tasks
Managing Masking Algorithms
Managing Sample Libraries
Managing Masking Policies
Managing Static Masking Tasks
Dynamic Masking Tasks
Managing Dynamic Masking Policies
Subscribing to Dynamic Masking Policies
Data Watermarks
Embedding Data Watermarks
Tracing Data Using Watermarks
File Watermarks
Dynamic Watermarks
Data Security Operations
Viewing Audit Logs
Diagnosing Data Security Risks
Managing the Recycle Bin
DataArts DataService
Overview
Specifications
Developing APIs in DataArts DataService
Buying and Managing an Exclusive Cluster
Creating a Reviewer in DataArts DataService
Creating an API
Generating an API Using Configuration
Generating an API Using a Script or MyBatis
Debugging an API
Publishing an API
Managing APIs
Managing API Versions
Displaying an API
Suspending/Restoring an API
Unpublishing/Deleting APIs
Copying an API
Synchronizing APIs
Exporting All/Exporting/Importing APIs
Orchestrating APIs
Overview
Configuring an Entry API Operator
Configuring a Conditional Branch Operator
Configuring a Parallel Processing Operator
Configuring an Output Processing Operator
Typical API Orchestration Configuration
Configuring a Throttling Policy for API Calling
Authorizing API Calling
Authorizing an API Which Uses App Authentication to Apps
Authorizing an API Which Uses IAM Authentication to Apps
Authorizing an API Which Uses IAM Authentication Through a Whitelist
Calling APIs in DataArts DataService
Applying for API Authorization
Calling APIs Using Different Methods
API Calling Methods
(Recommended) Using an SDK to Call an API Which Uses App Authentication
Using an API Tool to Call an API Which Uses App Authentication
Using an API Tool to Call an API Which Uses IAM Authentication
Using an API Tool to Call an API Which Requires No Authentication
Using a Browser to Call an API Which Requires No Authentication
Viewing API Access Logs
Configuring Review Center
Audit Log
Viewing Traces
Key Operations Recorded by CTS
Management Center Operations
Key CDM Operations Recorded by CTS
DataArts Architecture Operations
DataArts Factory Operations
DataArts Quality Operations
DataArts Catalog Operations
DataArts DataService Operations
Best Practices
Advanced Data Migration Guidance
Incremental Migration
Incremental File Migration
Incremental Migration of Relational Databases
HBase/CloudTable Incremental Migration
MongoDB/DDS Incremental Migration
Using Macro Variables of Date and Time
Migration in Transaction Mode
Encryption and Decryption During File Migration
MD5 Verification
Configuring Field Converters
Adding Fields
Migrating Files with Specified Names
Regular Expressions for Separating Semi-structured Text
Recording the Time When Data Is Written to the Database
File Formats
Converting Unsupported Data Types
Advanced Data Development Guidance
Dependency Policies for Periodic Scheduling
Comparison Between Traditional Periodic Scheduling Dependency and Natural Periodic Scheduling Dependency
Traditional Periodic Scheduling
Natural Periodic Scheduling
Natural Periodic Scheduling: Same-Period Dependency
Natural Periodic Scheduling: Dependency on the Previous Period
Scheduling by Discrete Hours and Scheduling by the Nearest Job Instance
Using PatchData
Setting the Job Scheduling Time to the Last Day of Each Month
Obtaining the Output of an SQL Node
IF Statements
Obtaining the Return Value of a Rest Client Node
Using For Each Nodes
Invoking DataArts Quality Operators Using DataArts Factory and Transferring Quality Parameters During Job Running
Scheduling Jobs Across Workspaces
Developing a Flink Jar Job
DataArts Studio Data Migration Configuration
Overview
Management Center Data Migration Configuration
DataArts Migration Data Migration Configuration
DataArts Architecture Data Migration Configuration
DataArts Factory Data Migration Configuration
DataArts Quality Data Migration Configuration
DataArts Catalog Data Migration Configuration
DataArts Security Data Migration Configuration
DataArts DataService Data Migration Configuration
Least Privilege Authorization
How Do I View the Number of Table Rows and Database Size?
Comparing Data Before and After Data Migration Using DataArts Quality
Configuring Alarms for Jobs in DataArts Factory of DataArts Studio
Scheduling a CDM Job by Transferring Parameters Using DataArts Factory
Enabling Incremental Data Migration Through DataArts Factory
Creating Table Migration Jobs in Batches Using CDM Nodes
Automatic Construction and Analysis of Graph Data
Scenario
Operating Environment and Data Preparation
Creating a Data Integration Job
Developing and Scheduling an Import GES Job
Analyzing Graph Data
Simplified Migration of Trade Data to the Cloud and Analysis
Scenario
Analysis Process
Using CDM to Upload Data to OBS
Uploading Inventory Data
Uploading Incremental Data
Analyzing Data
SDK Reference
SDK Overview
REST API SDK Reference
DataArts DataService SDK Reference
DataArts DataService SDK Overview
Preparations for Using an SDK
Common Error Codes and Messages for SDK Invocation
Calling APIs Through App Authentication
Preparation
Java
Go
Python
C#
JavaScript
PHP
C++
C
Android
curl
Other Programming Languages
API Reference
Before You Start
Overview
API Calling
Concepts
Endpoints
Project ID and Account ID
DataArts Studio Instance ID and Workspace ID
Data Development Job ID
DataArts Catalog GUID
Constraints
API Overview
Calling APIs
Making an API Request
Authentication
Response
Application Cases
Application Example
Example of Using Data Development APIs
DataArts Migration APIs
Cluster Management
Querying Cluster Details
Deleting a Cluster
Restarting a Cluster
Starting a Cluster
Stopping a Cluster
Creating a Cluster
Querying the Cluster List
Job Management
Querying a Job
Deleting a Job
Modifying a Job
Creating and Executing a Job in a Random Cluster
Stopping a Job
Creating a Job in a Specified Cluster
Starting a Job
Querying Job Status
Querying Job Execution History
Link Management
Creating a Link
Querying a Link
Deleting a Link
Modifying a Link
Public Data Structures
Link Parameter Description
Link to a Relational Database
Link to OBS
Link to HDFS
Link to HBase
Link to CloudTable
Link to Hive
Link to an FTP or SFTP Server
Link to MongoDB
Link to Redis/DCS (to Be Brought Offline)
Link to Kafka
Link to Elasticsearch/Cloud Search Service
Link to DLI
Link to DMS Kafka
Source Job Parameters
From a Relational Database
From Object Storage
From HDFS
From Hive
From HBase/CloudTable
From FTP/SFTP/NAS (to Be Brought Offline)/SFS (to Be Brought Offline)
From HTTP/HTTPS
From MongoDB/DDS
From Redis/DCS (to Be Brought Offline)
From DIS
From Kafka
From Elasticsearch/Cloud Search Service
Destination Job Parameters
To a Relational Database
To OBS
To HDFS
To Hive
To HBase/CloudTable
To FTP/SFTP/NAS (to Be Brought Offline)/SFS (to Be Brought Offline)
To DDS
To DCS
To Elasticsearch/Cloud Search Service
To DLI
To DIS
Job Parameter Description
DataArts Factory APIs
Connection Management APIs
Creating a Connection
Querying a Connection List
Querying Connection Details
Editing a Connection
Deleting a Connection
Exporting a Connection
Script Development APIs
Deleting a Script
Executing a Script
Stopping Executing a Script Instance
Resource Management APIs
Deleting a Resource
Job Development APIs
Viewing a Job File
Stopping a Job
Deleting a Job
Stopping a Job Instance
Retrying a Job Instance
APIs to Be Taken Offline
Creating a Job
Editing a Job
Viewing a Job List
Viewing Job Details
Exporting a Job
Batch Exporting Jobs
Importing a Job
Executing a Job Immediately
Starting a Job
Viewing Running Status of a Real-Time Job
Viewing a Job Instance List
Viewing Job Instance Details
Querying a System Task
Creating a Script
Modifying a Script
Querying a Script
Querying a Script List
Querying the Execution Result of a Script Instance
Creating a Resource
Modifying a Resource
Querying a Resource
Querying a Resource List
Importing a Connection
DataArts Architecture APIs
Overview
Obtaining Statistics
Obtaining Model Statistics
Obtaining ER Model Statistics
Obtaining Standard Coverage Statistics
Data Standard APIs
Querying the Data Standard List
Creating a Data Standard
Deleting Data Standards
Modifying a Data Standard
Obtaining Data Standard Details
Associating Attributes with a Data Standard
Lookup Table Management APIs
Querying the Lookup Table List
Creating a Lookup Table
Deleting Lookup Tables
Modifying a Lookup Table
Querying Lookup Table Details
Obtaining Field Values in a Lookup Table
Editing Field Values in a Lookup Table
Catalog APIs
Obtaining a Tree of All Catalogs
Querying the Catalog List
Creating a Catalog
Modifying a Catalog
Deleting Catalogs
Obtaining Catalog Details
Data Standard Template APIs
Querying Data Standard Templates
Creating a Data Standard Template
Modifying a Data Standard Template
Deleting Data Standard Templates
Initializing a Data Standard Template
Performing a Batch Publish Operation
Approval Management APIs
Querying Approvals
Withdrawing Approvals
Processing Approvals
Performing a Batch Offline Operation
Creating an Approver
Querying the Approver List
Deleting Approvers
Subject Management APIs
Querying the Subject List
Deleting Topics
Creating or Modifying a Subject
Obtaining a Subject Tree
Subject Level APIs
Obtaining Subject Levels
Modifying or Deleting Subject Levels
Atomic Metric APIs
Querying Atomic Metrics
Querying Atomic Metric Details
Derivative Metric APIs
Querying Derivative Metrics
Querying Derivative Metric Details
Compound Metric APIs
Querying Compound Metrics
Querying Compound Metric Details
Dimension APIs
Querying Dimensions
Querying Dimension Details
Querying Dimension Groups
Querying Dimension Table Reverse Tasks
Filter APIs
Querying Filters
Querying Filter Details
Dimension Table APIs
Querying Dimension Tables
Querying Dimension Table Details
Fact Table APIs
Querying Fact Tables
Querying Fact Table Details
Querying Fact Table Reverse Tasks
Summary Table APIs
Querying Summary Tables
Querying Summary Table Details
Business Metric APIs
Querying Business Metrics
Querying Metric Details
Querying Metric Dimensions
Querying Metric Owners
Obtaining Relationship Information of a Metric
ER Modeling APIs
Obtaining the Model Entity List
Querying Relationships
Querying Relationship Details
Querying All Relationships in a Model
Obtaining Details of a Table Model
Querying the Model List
Creating Workspaces
Updating Workspaces
Deleting Workspaces
Querying Details About a Model
Import/Export APIs
Querying the Import Result
Catalog Management APIs
Obtaining All Directories
Version Information APIs
Querying Versions
DataArts Quality APIs
Catalog APIs
Obtaining the Catalog List
Rule Template APIs
Obtaining the Rule Template List
Obtaining Rule Template Details
Quality Job APIs
Obtaining the Quality Job List
Obtaining Quality Job Details
Comparison Job APIs
Obtaining the Comparison Job List
Obtaining Comparison Job Details
O&M APIs
Obtaining the Task Execution Result List
Obtaining the Instance Result
DataArts Catalog APIs
Logical Asset APIs
Querying Logical Assets
Metric Asset APIs
Querying the Metric Asset Directory Tree
Querying Metric Assets
Asset Statistics APIs
Obtaining Technical Asset Statistics
Obtaining Logical Asset Statistics
Asset Management APIs
Querying Technical Assets
Obtaining Asset Details by GUID
Asset Classification APIs
Associating a Classification with Assets
Associating a Classification with One or More Assets
Dissociating a Classification from an Asset
Asset Security Level APIs
Associating a Security Level with Assets
Associating a Security Level with an Asset
Dissociate a Security Level from an Asset
Data Lake Mall APIs
API Management
Creating an API
Debugging an API
Querying the API List
Updating an API
Querying API Information
Deleting APIs
Publishing an API
Unpublishing, Suspending, and Restoring an API
Authorizing an API to Apps
Performing API Authorization Operations
Querying API Publishing Messages in DLM Exclusive
Querying API Debugging Messages in DLM Exclusive
Querying Instances for API Operations in DLM Exclusive
Authorization Management
Querying Apps Bound to an API
Querying Authorized APIs of an App
Application Management
Querying the Application List
Reviewing Applications
Obtaining Application Details
Message Management
Querying the Message List
Processing Messages
Obtaining Message Details
Service Catalog Management
Creating a Service Catalog
Updating a Service Catalog
Querying a Service Catalog
Deleting Service Catalogs
Moving a Catalog to Another Catalog
Moving APIs to Another Catalog
Obtaining the List of APIs and Catalogs in a Catalog
Obtaining the List of APIs in a Catalog
Obtaining the List of Sub-Catalogs in a Catalog
Obtaining the ID of a Catalog Through Its Path
Obtaining the Path of a Catalog Through Its ID
Obtaining the Paths to a Catalog Through Its ID
Gateway Management
Obtaining the List of Gateway Instances
Obtaining the List of Gateway Groups
App Management
Creating an App
Querying the App List
Updating an App
Deleting an App
Querying App Details
Overview APIs
Querying API Overview
Querying App Overview
Querying Top N Services Called by an API
Querying Top N Services Used by an App
Querying API Statistics Details
Querying App Statistics Details
Querying API Dashboard Data Details
Querying Data Details of a Specified API Dashboard
Querying App Dashboard Data Details
Querying Top N Apps Called by a Specified API
Appendix
Common Message Headers
Parsing a Stream in a Response Message
Status Codes
Error Codes
CDM Error Codes
DLF Error Codes
FAQs
Consultation and Billing
How Do I Select a Region and an AZ?
What Is the Relationship Between DataArts Studio and Huawei Horizon Digital Platform?
What Are the Differences Between DataArts Studio and ROMA?
Can DataArts Studio Be Deployed in a Local Data Center or on a Private Cloud?
How Do I Create a Fine-Grained Permission Policy in IAM?
How Do I Isolate Workspaces So That Users Cannot View Unauthorized Workspaces?
What Should I Do If a User Cannot View Workspaces After I Have Assigned the Required Policy to the User?
What Should I Do If Insufficient Permissions Are Prompted When I Am Trying to Perform an Operation as an IAM User?
Can I Delete DataArts Studio Workspaces?
Can I Transfer a Purchased or Trial Instance to Another Account?
Does DataArts Studio Support Version Upgrade?
Does DataArts Studio Support Version Downgrade?
How Do I View the DataArts Studio Instance Version?
Why Can't I Select a Specified IAM Project When Purchasing a DataArts Studio Instance?
What Is the Session Timeout Period of DataArts Studio? Can the Session Timeout Period Be Modified?
Will My Data Be Retained If My Package Expires or My Pay-per-Use Resources Are in Arrears?
How Do I Check the Remaining Validity Period of a Package?
Why Isn't the CDM Cluster in a DataArts Studio Instance Billed?
Why Does the System Display a Message Indicating that the Number of Daily Executed Nodes Has Reached the Upper Limit? What Should I Do?
Management Center
Which Data Sources Can DataArts Studio Connect To?
What Are the Precautions for Creating Data Connections?
What Should I Do If Database or Table Information Cannot Be Obtained Through a GaussDB(DWS)/Hive/HBase Data Connection?
Why Are MRS Hive/HBase Clusters Not Displayed on the Page for Creating Data Connections?
What Should I Do If a GaussDB(DWS) Connection Test Fails When SSL Is Enabled for the Connection?
Can I Create Multiple Connections to the Same Data Source in a Workspace?
Should I Select the API or Proxy Connection Type When Creating a Data Connection in Management Center?
How Do I Migrate the Data Development Jobs and Data Connections from One Workspace to Another?
DataArts Migration (CDM Jobs)
What Are the Differences Between CDM and Other Data Migration Services?
What Are the Advantages of CDM?
What Are the Security Protection Mechanisms of CDM?
How Do I Reduce the Cost of Using CDM?
Will I Be Billed If My CDM Cluster Does Not Use the Data Transmission Function?
Why Am I Billed Pay per Use When I Have Purchased a Yearly/Monthly CDM Incremental Package?
How Do I Check the Remaining Validity Period of a Package?
Can CDM Be Shared by Different Tenants?
Can I Upgrade a CDM Cluster?
How Is the Migration Performance of CDM?
What Is the Number of Concurrent Jobs for Different CDM Cluster Versions?
Does CDM Support Incremental Data Migration?
Does CDM Support Field Conversion?
What Component Versions Are Recommended for Migrating Hadoop Data Sources?
What Data Formats Are Supported When the Data Source Is Hive?
Can I Synchronize Jobs to Other Clusters?
Can I Create Jobs in Batches?
Can I Schedule Jobs in Batches?
How Do I Back Up CDM Jobs?
What Should I Do If Only Some Nodes in a HANA Cluster Can Communicate with the CDM Cluster?
How Do I Use Java to Invoke CDM RESTful APIs to Create Data Migration Jobs?
How Do I Connect the On-Premises Intranet or Third-Party Private Network to CDM?
Does CDM Support Parameters or Variables?
How Do I Set the Number of Concurrent Extractors for a CDM Migration Job?
Does CDM Support Real-Time Migration of Dynamic Data?
Can I Stop CDM Clusters?
How Do I Obtain the Current Time Using an Expression?
What Should I Do If the Log Prompts that the Date Format Fails to Be Parsed?
What Can I Do If the Map Field Tab Page Cannot Display All Columns?
How Do I Select Distribution Columns When Using CDM to Migrate Data to GaussDB(DWS)?
What Do I Do If the Error Message "value too long for type character varying" Is Displayed When I Migrate Data to DWS?
What Can I Do If Error Message "Unable to execute the SQL statement" Is Displayed When I Import Data from OBS to SQL Server?
What Should I Do If the Cluster List Is Empty, I Have No Access Permission, or My Operation Is Denied?
Why Is Error ORA-01555 Reported During Migration from Oracle to DWS?
What Should I Do If Migration Using a MongoDB Connection Fails?
What Should I Do If a Hive Migration Job Is Suspended for a Long Period of Time?
What Should I Do If an Error Is Reported Because the Field Type Mapping Does Not Match During Data Migration Using CDM?
What Should I Do If a JDBC Connection Timeout Error Is Reported During MySQL Migration?
What Should I Do If a CDM Migration Job Fails After a Link from Hive to GaussDB(DWS) Is Created?
How Do I Use CDM to Export MySQL Data to an SQL File and Upload the File to an OBS Bucket?
What Should I Do If CDM Fails to Migrate Data from OBS to DLI?
What Should I Do If a CDM Connector Reports the Error "Configuration Item [linkConfig.iamAuth] Does Not Exist"?
What Should I Do If Error "Configuration Item [linkConfig.createBackendLinks] Does Not Exist" or "Configuration Item [throttlingConfig.concurrentSubJobs] Does Not Exist" Is Reported?
What Should I Do If Message "CORE_0031:Connect time out. (Cdm.0523)" Is Displayed During the Creation of an MRS Hive Link?
What Should I Do If Message "CDM Does Not Support Auto Creation of an Empty Table with No Column" Is Displayed When I Enable Auto Table Creation?
What Should I Do If I Cannot Obtain the Schema Name When Creating an Oracle Relational Database Migration Job?
What Should I Do If invalid input syntax for integer: "true" Is Displayed During MySQL Database Migration?
DataArts Architecture
What Is the Relationship Between Lookup Tables and Data Standards?
What Are the Differences Between ER Modeling and Dimensional Modeling?
What Data Modeling Methods Are Supported by DataArts Architecture?
How Can I Use Standardized Data?
Does DataArts Architecture Support Database Reversing?
What Are the Differences Between the Metrics in DataArts Architecture and DataArts Quality?
Why Doesn't the Table in the Database Change After I Have Modified Fields in an ER or Dimensional Model?
Can I Configure Lifecycle Management for Tables?
How Should I Select a Subject When a Public Dimension (Date, Region, Supplier, or Product) Is Shared by Multiple Subject Areas?
How Can I Create an Atomic Metric Using a Dimension Table Since Only a Fact Table Can Be Selected for Creating an Atomic Metric?
DataArts Factory
How Many Jobs Can Be Created in DataArts Factory? Is There a Limit on the Number of Nodes in a Job?
Does DataArts Studio Support Custom Python Scripts?
How Can I Quickly Rectify a Deleted CDM Cluster Associated with a Job?
Why Is There a Large Difference Between Job Execution Time and Start Time of a Job?
Will Subsequent Jobs Be Affected If a Job Fails to Be Executed During Scheduling of Dependent Jobs? What Should I Do?
What Should I Pay Attention to When Using DataArts Studio to Schedule Big Data Services?
What Are the Differences and Relationships Between Environment Variables, Job Parameters, and Script Parameters?
What Should I Do If a Job Log Cannot Be Opened and Error 404 Is Reported?
What Should I Do If the Agency List Fails to Be Obtained During Agency Configuration?
Why Can't I Select Specified Peripheral Resources When Creating a Data Connection in DataArts Factory?
Why Can't I Receive Job Failure Alarm Notifications After I Have Configured SMN Notifications?
Why Is There No Job Running Scheduling Log on the Monitor Instance Page After Periodic Scheduling Is Configured for a Job?
Why Isn't the Error Cause Displayed on the Console When a Hive SQL or Spark SQL Scripts Fails?
What Should I Do If the Token Is Invalid During the Execution of a Data Development Node?
How Do I View Run Logs After a Job Is Tested?
Why Does a Job Scheduled by Month Start Running Before the Job Scheduled by Day Is Complete?
What Should I Do If Invalid Authentication Is Reported When I Run a DLI Script?
Why Cannot I Select a Desired CDM Cluster in Proxy Mode When Creating a Data Connection?
Why Is There No Job Running Scheduling Record After Daily Scheduling Is Configured for the Job?
What Do I Do If No Content Is Displayed in Job Logs?
Why Do I Fail to Establish a Dependency Between Two Jobs?
What Should I Do If an Error Is Reported During Job Scheduling in DataArts Studio, Indicating that the Job Has Not Been Submitted?
What Should I Do If an Error Is Reported During Job Scheduling in DataArts Studio, Indicating that the Script Associated with Node XXX in the Job Has Not Been Submitted?
What Should I Do If a Job Fails to Be Executed After Being Submitted for Scheduling and an Error Displayed: Depend Job [XXX] Is Not Running Or Pause?
How Do I Create Databases and Data Tables? Do Databases Correspond to Data Connections?
Why Is No Result Displayed After a Hive Task Is Executed?
Why Is the Last Instance Status On the Monitor Instance Page Either Successful or Failed?
How Do I Configure Notifications for All Jobs?
What Is the Maximum Number of Nodes That Can Be Executed Simultaneously?
Can I Change the Time Zone of a DataArts Studio Instance?
How Do I Synchronize the Changed Names of CDM Jobs to DataArts Factory?
Why Does the Execution of an RDS SQL Statement Fail and an Error Is Reported Indicating That hll Does Not Exist?
What Should I Do If Error Message "The account has been locked" Is Displayed When I Am Creating a DWS Data Connection?
What Should I Do If a Job Instance Is Canceled and Message "The node start execute failed, so the current node status is set to cancel." Is Displayed?
What Should I Do If Error Message "Workspace does not exists" Is Displayed When I Call a DataArts Factory API?
Why Don't the URL Parameters for Calling an API Take Effect in the Test Environment When the API Can Be Called Properly Using Postman?
What Should I Do If Error Message "Agent need to be updated?" Is Displayed When I Run a Python Script?
Why Is an Execution Failure Displayed for a Node in the Log When the Node Status Is Successful?
What Should I Do If an Unknown Exception Occurs When I Call a DataArts Factory API?
Why Is an Error Message Indicating an Invalid Resource Name Is Displayed When I Call a Resource Creation API?
Why Does a PatchData Task Fail When All PatchData Job Instances Are Successful?
Why Is a Table Unavailable When an Error Message Indicating that the Table Already Exists Is Displayed During Table Creation from a DWS Data Connection?
What Should I Do If Error Message "The throttling threshold has been reached: policy user over ratelimit,limit:60,time:1 minute." Is Displayed When I Schedule an MRS Spark Job?
What Should I Do If Error Message "UnicodeEncodeError: 'ascii' codec can't encode characters in position 63-64: ordinal not in range(128)" Is Displayed When I Run a Python Script?
What Should I Do If an Error Message Is Displayed When I View Logs?
What Should I Do If a Shell/Python Node Fails and Error "session is down" Is Reported?
What Should I Do If a Parameter Value in a Request Header Contains More Than 512 Characters?
What Should I Do If a Message Is Displayed Indicating that the ID Does Not Exist During the Execution of a DWS SQL Script?
How Do I Check Which Jobs Invoke a CDM Job?
What Should I Do If Error Message "The request parameter invalid" Is Displayed When I Use Python to Call the API for Executing Scripts?
What Should I Do If the Default Queue of a New DLI SQL Script in DataArts Factory Has Been Deleted?
Does the Event-based Scheduling Type in DataArts Factory Support Offline Kafka?
What Should I Do If an Error Is Reported When I Submit an MRS Job with an Agency Through an MRS Cluster Connected Using an MRS API or the MRS Tenant Plane?
DataArts Quality
What Are the Differences Between Quality Jobs and Comparison Jobs?
How Can I Confirm that a Quality Job or Comparison Job Is Blocked?
How Do I Manually Restart a Blocked Quality Job or Comparison Job?
How Do I View Jobs Associated with a Quality Rule Template?
What Should I Do If the System Displays a Message Indicating that I Do Not Have the MRS Permission to Perform a Quality Job?
DataArts Catalog
What Are the Functions of the DataArts Catalog Module?
What Assets Can Be Collected by DataArts Catalog?
What Is Data Lineage?
How Do I Visualize Data Lineages in a Data Catalog?
DataArts Security
Why Isn't Data Masked Based on a Specified Rule After a Data Masking Task Is Executed?
What Should I Do If a Message Is Displayed Indicating that Necessary Request Parameters Are Missing When I Approve a GaussDB(DWS) Permission Application?
What Should I Do If Error Message "FATAL: Invalid username/password,login denied" Is Displayed During the GaussDB(DWS) Connectivity Check When Fine-grained Authentication Is Enabled?
What Should I Do If Error Message "Failed to obtain the database" Is Displayed When I Select a Database in DataArts Factory After Fine-grained Authentication Is Enabled?
Why Does the System Display a Message Indicating Insufficient Permissions During Permission Synchronization to DLI?
DataArts DataService
What Languages Do DataArts DataService SDKs Support?
What Can I Do If the System Displays a Message Indicating that the Proxy Fails to Be Invoked During API Creation?
What Should I Do If the Background Reports an Error When I Access the Test App Through the Data Service API and Set Related Parameters?
How Many Times Can a Subdomain Name Be Accessed Using APIs Every Day?
Can Operators Be Transferred When API Parameters Are Transferred?
What Should I Do If No More APIs Can Be Created When the API Quota in the Workspace Is Used Up?
How Can I Access APIs of DataArts DataService Exclusive from the Internet?
How Can I Access APIs of DataArts DataService Exclusive Using Domain Names?
What Should I Do If It Takes a Long Time to Obtain the Total Number of Data Records of a Table Through an API If the Table Contains a Large Amount of Data?
Videos