All Documentation
DataArts Studio
DataArts Studio
All results for "
" in this service
All results for "
" in this service
What's New
Function Overview
Service Overview
DataArts Studio Infographics
What Is DataArts Studio?
Basic Concepts
Functions
Advantages
Application Scenarios
Versions
Billing
Security
Shared Responsibilities
Asset Identification and Management
Identity Authentication and Access Control
Data Protection Technologies
Audit and Logging
Service Resilience
Security Risk Monitoring
Recovery from Failures
Update Management
Certificates
Permission Management
Permissions
Constraints and Restrictions
Related Services
Data Governance Methodology
Purpose
Intended Audience
Data Governance Framework
Framework
Data Governance Modules
Module Relationships
Data Governance Organizational Architecture
Framework
Responsibilities
Measurement and Evaluation System
Data Governance Methods
Measurement Dimensions
Measurement and Scoring Rules
Huawei Data Governance Cases
Thoughts
Practices
Effects
Thoughts on Data Governance and the Impact of COVID-19
Implementation of the Data Governance Methodology
Preparations
Preparations
Buying DataArts Studio Instances
Buying a DataArts Studio Basic Package
(Optional) buy an Incremental Package
(Optional) Creating Instance Tags
Managing a Workspace
Creating and Managing a Workspace
(Optional) Changing the Job Log Storage Path
Authorizing Users to Use DataArts Studio
Creating an IAM User and Assigning DataArts Studio Permissions
Adding a Member and Assigning a Role
(Optional) Obtaining Authentication Information
Getting Started
Overview
Starters: Data Development Based on E-commerce BI Reports
Scenario
Step 1: Make Preparations
Step 2: Develop Data
(Optional) Step 3: Unsubscribe from Services
Beginners: Data Integration and Development Based on Movie Scores
Scenario
Step 1: Make Preparations
Step 2: Integrate Data
Step 3: Develop Data
(Optional) Step 4: Unsubscribe from Services
Advanced Users: Data Governance Based on Taxi Trip Data
Example Scenario
Step 1: Process Design
Step 2: Preparations
Step 3: DataArts Migration
Step 4: Metadata Collection
Step 5: DataArts Architecture
Step 6: DataArts Factory
Step 7: DataArts Quality
Step 8: DataArts Catalog
(Optional) Step 9: Service Unsubscription
Practices
User Guide
DataArts Studio Introduction
Management Console
Tags
Enterprise Mode
DataArts Studio Enterprise Mode Overview
Service Process in Enterprise Mode
Creating a Workspace in Enterprise Mode
Admin Operations
Developer Operations
Deployer Operations
Operator Operations
Management Center
Data Sources
Managing Data Connections
Creating a Data Connection
Configuring a DWS Connection
Configuring a DLI Connection
Configuring an MRS Hive Connection
Configuring an MRS HBase Connection
Configuring an MRS Kafka Connection
Configuring an MRS Spark Connection
Configuring an MRS ClickHouse Connection
Configuring an MRS Hetu Connection
Configuring an MRS Impala Connection
Configuring an MRS Ranger Connection
Configuring an MRS Presto Connection
Configuring an MRS Doris Connection
Configuring an RDS Connection
Configuring an Oracle Connection
Configuring a DIS Connection
Configuring a Host Connection
Migrating Resources
Configuring Environment Isolation for a Workspace in Enterprise Mode
Tutorials
Creating an MRS Hive Connection
Creating a DWS Connection
Creating a MySQL Connection
DataArts Migration
Overview
Constraints
Supported Data Sources
Supported Data Sources (2.9.3.300)
Supported Data Sources (2.9.2.200)
Supported Data Types
Managing Clusters
Creating a CDM Cluster
Binding or Unbinding an EIP
Restarting a Cluster
Deleting a Cluster
Downloading Cluster Logs
Viewing Basic Cluster Information and Modifying Cluster Configurations
Managing Cluster Tags
Viewing Metrics
CDM Metrics
Configuring Alarm Rules
Querying Metrics
Managing Links
Creating a Link
Managing Drivers
Managing Agents
Managing Cluster Configurations
Link to OBS
Link to PostgreSQL/SQLServer
Link to DWS
Link to an RDS for MySQL/MySQL Database
Link to an Oracle Database
Link to DLI
Link to Hive
Link to HBase
Link to HDFS
Link to an FTP or SFTP Server
Link to Redis
Link to DDS
Link to CloudTable
Link to MongoDB
Link to Cassandra
Link to DIS
Link to Kafka
Link to DMS Kafka
Link to CSS
Link to Elasticsearch
Link to a Dameng Database
Link to SAP HANA
Link to a Database Shard
Link to MRS Hudi
Link to MRS ClickHouse
Link to a ShenTong Database
Link to CloudTable OpenTSDB
Managing Jobs
Table/File Migration Jobs
Creating an Entire Database Migration Job
Source Job Parameters
From OBS
From HDFS
From HBase/CloudTable
From Hive
From DLI
From FTP/SFTP
From HTTP
From PostgreSQL/SQL Server
From DWS
From SAP HANA
From MySQL
From Oracle
From a Database Shard
From MongoDB/DDS
From Redis
From DIS
From Kafka/DMS Kafka
From Elasticsearch or CSS
From OpenTSDB
From MRS Hudi
From MRS ClickHouse
From a ShenTong Database
From a Dameng Database
Destination Job Parameters
To OBS
To HDFS
To HBase/CloudTable
To Hive
To MySQL/SQL Server/PostgreSQL
To Oracle
To DWS
To DDS
To Redis
To Elasticsearch/CSS
To DLI
To OpenTSDB
To MRS Hudi
To MRS ClickHouse
To MongoDB
Configuring Field Mapping
Scheduling Job Execution
Job Configuration Management
Managing a Single Job
Managing Jobs in Batches
Improving Migration Performance
How Migration Jobs Work
Performance Tuning
Reference: Job Splitting Dimensions
Reference: CDM Performance Test Data
Error Codes
Key Operation Guide
Incremental Migration
Incremental File Migration
Incremental Migration of Relational Databases
HBase/CloudTable Incremental Migration
MongoDB/DDS Incremental Migration
Using Macro Variables of Date and Time
Migration in Transaction Mode
Encryption and Decryption During File Migration
MD5 Verification
Configuring Field Converters
Adding Fields
Migrating Files with Specified Names
Regular Expressions for Separating Semi-structured Text
Recording the Time When Data Is Written to the Database
File Formats
Converting Unsupported Data Types
Auto Table Creation
Tutorials
Creating an MRS Hive Link
Creating a MySQL Link
Migrating Data from MySQL to MRS Hive
Migrating Data from MySQL to OBS
Migrating Data from MySQL to DWS
Migrating an Entire MySQL Database to RDS
Migrating Data from Oracle to CSS
Migrating Data from Oracle to DWS
Migrating Data from OBS to CSS
Migrating Data from OBS to DLI
Migrating Data from MRS HDFS to OBS
Migrating the Entire Elasticsearch Database to CSS
More Cases and Practices
DataArts Architecture
Overview
DataArts Architecture Use Process
Preparations
Adding Reviewers
Managing the Configuration Center
Data Survey
Designing Processes
Designing Subjects
Standards Design
Creating Lookup Tables
Creating Data Standards
Model Design
ER Modeling
Designing Logical Models
Designing Physical Models
Dimensional Modeling
Creating Dimensions
Managing Dimension Tables
Creating Fact Tables
Metric Design
Business Metrics
Technical Metrics
Creating Atomic Metrics
Creating Derivative Metrics
Creating Compound Metrics
Creating Time Filters
Data Mart Building
Creating Summary Tables
Common Operations
Reversing a Database (ER Modeling)
Reversing a Database (Dimensional Modeling)
Importing/Exporting Data
Associating Quality Rules
Viewing Tables
Modifying Subjects, Directories, and Processes
Review Center
Tutorials
DataArts Architecture Example
DataArts Factory
Overview
Data Management
Data Management Process
Creating a Data Connection
Creating a Database
(Optional) Creating a Database Schema
Creating a Table
Script Development
Script Development Process
Creating a Script
Developing Scripts
Developing an SQL Script
Developing a Shell Script
Developing a Python Script
Submitting a Version
Releasing a Script Task
(Optional) Managing Scripts
Copying a Script
Copying the Script Name and Renaming a Script
Moving a Script or Script Directory
Exporting and Importing a Script
Viewing Script References
Deleting a Script
Unlocking a Script
Changing the Script Owner
Unlocking Scripts
Job Development
Job Development Process
Creating a Job
Developing a Pipeline Job
Developing a Batch Processing Single-Task SQL Job
Developing a Real-Time Processing Single-Task Flink SQL Job
Developing a Real-Time Processing Single-Task Flink JAR Job
Developing a Real-Time Processing Single-Task DLI Spark Job
Setting Up Scheduling for a Job
Submitting a Version
Releasing a Job Task
(Optional) Managing Jobs
Copying a Job
Copying the Job Name and Renaming a Job
Moving a Job or Job Directory
Exporting and Importing a Job
Configuring Jobs
Deleting a Job
Unlocking a Job
Viewing a Job Dependency Graph
Changing the Job Owner
Unlocking Jobs
Going to Monitor Job page
Solution
Execution History
O&M and Scheduling
Overview
Monitoring a Job
Monitoring a Batch Job
Monitoring a Real-Time Job
Instance Monitoring
Monitoring PatchData
Baseline O&M
Overview
Restrictions
Baseline Instances
Baseline Management
Event Management
Properly Configuring the Promised Completion Time and Time Left Before Promise Breakdown
Managing Notifications
Managing Notifications
Cycle Overview
Managing Terminal Subscriptions
Managing Backups
Operation History
Configuration and Management
Configuring Resources
Configuring Environment Variables
Configuring an OBS Bucket
Managing Job Tags
Configuring a Scheduling Identity
Configuring the Number of Concurrently Running Nodes
Configuring a Template
Configuring a Scheduling Calendar
Configuring a Default Item
Configuring Task Groups
Managing Resources
Review Center
Download Center
Node Reference
Node Overview
Node Lineages
Overview
Configuring Data Lineages
Viewing Data Lineages
CDM Job
DIS Stream
DIS Dump
DIS Client
Rest Client
Import GES
MRS Kafka
Kafka Client
ROMA FDI Job
DLI Flink Job
DLI SQL
DLI Spark
DWS SQL
MRS Spark SQL
MRS Hive SQL
MRS Presto SQL
MRS Spark
MRS Spark Python
MRS ClickHouse
MRS Flink Job
MRS MapReduce
CSS
Shell
RDS SQL
ETL Job
Python
ModelArts Train
Http Trigger
Create OBS
Delete OBS
OBS Manager
Open/Close Resource
Data Quality Monitor
Subjob
For Each
SMN
Dummy
EL Expression Reference
Expression Overview
Basic Operators
Date and Time Mode
Env Embedded Objects
Job Embedded Objects
StringUtil Embedded Objects
DateUtil Embedded Objects
JSONUtil Embedded Objects
Loop Embedded Objects
OBSUtil Embedded Objects
Examples of Common EL Expressions
EL Expression Use Examples
Simple Variable Set
Usage Guidance
Referencing Parameters in Scripts and Jobs
Setting the Job Scheduling Time to the Last Day of Each Month
Configuring a Yearly Scheduled Job
Using PatchData
Obtaining the Output of an SQL Node
Obtaining the Maximum Value and Transferring It to a CDM Job Using a Query SQL Statement
IF Statements
Obtaining the Return Value of a Rest Client Node
Using For Each Nodes
Using Script Templates and Parameter Templates
Developing a Python Job
Developing a DWS SQL Job
Developing a Hive SQL Job
Developing a DLI Spark Job
Developing an MRS Flink Job
Developing an MRS Spark Python Job
More Cases for Reference
DataArts Quality
Metric Monitoring (Unavailable Soon)
Overview
Creating a Metric
Creating a Rule
Creating a Scenario
Viewing a Scenario Instance
Monitoring Data Quality
Overview
Creating Rule Templates
Creating Quality Jobs
Creating a Comparison Job
Viewing Job Instances
Viewing Quality Reports
Tutorials
Creating a Business Scenario
Creating a Quality Job
Creating a Comparison Job
DataArts Catalog
Data Maps
Overview
Dashboard
Data Catalogs
Tags
Data Permissions
Overview
Data Catalog Permissions
Table Permissions
Review Center
Data Security
Overview
Data Security Levels
Data Classifications
Masking Policies
Metadata Collection
Overview
Task Management
Task Monitoring
Tutorials
Developing an Incremental Metadata Collection Task
Viewing Data Lineages Through the Data Map
Overview
Configuring Data Lineages
Viewing Data Lineages
DataArts Security
Overview
Dashboard
Unified Permission Governance
Process
Preparation 1: Authorizing dlg_agency
Preparation 2: Checking Configuration Based on the Checklist
Synchronizing Users
Configuring Workspace Permission Sets
Configuring Permission Sets
Configuring Roles
Managing Members
Applying for Permissions and Reviewing Permission Requests
Configuring Row-level Access Control
Enabling Permission Applications
Configuring Queue Permissions
Configuring Workspace Resource Permission Policies
Configuring Resource Permissions
Viewing Permission Reports
Sensitive Data Governance
Overview
Creating Data Security Levels
Creating Data Classifications
Creating Identification Rules
Creating Identification Rule Groups
Discovering Sensitive Data
Viewing Sensitive Data Distribution
Managing Sensitive Data
Privacy Protection and Management
Overview
Managing Masking Algorithms
Managing Masking Policies
Managing Static Masking Tasks
Managing Dynamic Masking Policies
Subscribing to Dynamic Masking Policies
Managing Data Watermarks
Embedding Data Watermarks
Tracing Data Using Watermarks
Managing File Watermarks
Managing Dynamic Watermarking Policies
Data Security Operations
Auditing Data Access
Diagnosing Data Security Risks
DataArts DataService
Overview
Specifications
API Development
Preparations
Buying an Exclusive DataArts DataService instance
Adding Reviewers
Creating an API
Generating an API Using Configuration
Generating an API Using a Script or MyBatis
Registering APIs
Debugging an API
Publishing an API
Managing APIs
Setting an API to Be Visible
Suspending/Restoring an API
Unpublishing/Deleting APIs
Copying an API
Synchronizing APIs
Exporting All/Exporting/Importing APIs
Creating Throttling Policies
Calling APIs
Configuring Log Dump and Viewing Logs on LTS
Performing Operations in Review Center
Audit Log
Viewing Traces
Key Operations Recorded by CTS
Management Center Operations
DataArts Migration Operations
DataArts Architecture Operations
DataArts Factory Operations
DataArts Quality Operations
DataArts Catalog Operations
DataArts DataService Operations
Best Practices
Advanced Data Migration Guidance
Incremental Migration
Incremental File Migration
Incremental Migration of Relational Databases
HBase/CloudTable Incremental Migration
MongoDB/DDS Incremental Migration
Using Macro Variables of Date and Time
Migration in Transaction Mode
Encryption and Decryption During File Migration
MD5 Verification
Configuring Field Converters
Adding Fields
Migrating Files with Specified Names
Regular Expressions for Separating Semi-structured Text
Recording the Time When Data Is Written to the Database
File Formats
Converting Unsupported Data Types
Advanced Data Development Guidance
Dependency Policies for Periodic Scheduling
Comparison Between Traditional Periodic Scheduling Dependency and Natural Periodic Scheduling Dependency
Traditional Periodic Scheduling
Natural Periodic Scheduling
Natural Periodic Scheduling: Same-Period Dependency
Natural Periodic Scheduling: Dependency on the Previous Period
Using PatchData
Setting the Job Scheduling Time to the Last Day of Each Month
Obtaining the Output of an SQL Node
IF Statements
Obtaining the Return Value of a Rest Client Node
Using For Each Nodes
Invoking DataArts Quality Operators Using DataArts Factory and Transferring Quality Parameters During Job Running
Scheduling Jobs Across Workspaces
Cross-Workspace DataArts Studio Data Migration
Overview
Management Center Data Migration
DataArts Migration Data Migration
Data Migration for DataArts Architecture
Data Migration for DataArts Factory
Data Migration for DataArts Quality
DataArts Catalog Data Migration
DataArts Security Data Migration
DataArts DataService Data Migration
Authorizing Users to Use DataArts Studio by Complying with the Principle of Least Privilege
How Do I View the Number of Table Rows and Database Size?
Comparing Data Before and After Data Migration Using DataArts Quality
Scheduling a CDM Job by Transferring Parameters Using DataArts Factory
Enabling Incremental Data Migration Through DataArts Factory
Creating Table Migration Jobs in Batches Using CDM Nodes
Building Graph Data Based on MRS Hive Tables and Automatically Importing the Data to GES
Scenario
Making Preparations
Creating a Data Integration Job
Developing and Scheduling an Import GES Job
Analyzing Graph Data
Case: Trade Data Statistics and Analysis
Scenario
Analysis Process
Using CDM to Upload Data to OBS
Uploading Inventory Data
Uploading Incremental Data
Analyzing Data
Case: IoV Big Data Service Migration to Cloud
Scenario
Migration Preparation
Using CDM to Migrate Data of the Last Month
Using DES to Migrate Historical Data Generated One Month Ago
Restoring the HBase Table on MRS
Case: Building a Real-Time Alarm Platform
SDK Reference
SDK Overview
REST API SDK Reference
DataArts DataService SDK Reference
Overview
Preparations for Using an SDK
Common Error Codes and Messages for SDK Invocation
Calling APIs Through App Authentication
Preparation
Java
Go
Python
C#
JavaScript
PHP
C++
C
Android
curl
Other Programming Languages
API Reference
Before You Start
Overview
API Calling
Concepts
Endpoints
Project ID and Account ID
Instance ID and Workspace ID
Constraints
API Overview
Calling APIs
Making an API Request
Authentication
Response
Application Cases
Example of Using DataArts Migration APIs
Example of Using DataArts Factory APIs
DataArts Migration APIs
Cluster Management
Querying Cluster Details
Deleting a Cluster
Restarting a Cluster
Starting a Cluster
Stopping a Cluster (To Be Taken Offline)
Creating a Cluster
Querying the Cluster List
Job Management
Querying a Job
Deleting a Job
Modifying a Job
Creating and Executing a Job in a Random Cluster
Stopping a Job
Creating a Job in a Specified Cluster
Starting a Job
Querying Job Status
Querying Job Execution History
Link Management
Creating a Link
Querying a Link
Deleting a Link
Modifying a Link
Public Data Structures
Link Parameter Description
Link to a Relational Database
Link to OBS
Link to HDFS
Link to HBase
Link to CloudTable
Link to Hive
Link to an FTP or SFTP Server
Link to MongoDB
Link to Redis
Link to Kafka
Link to DIS
Link to Elasticsearch/Cloud Search Service
Link to DLI
Link to DMS Kafka
Source Job Parameters
From a Relational Database
From Object Storage
From HDFS
From Hive
From HBase/CloudTable
From FTP/SFTP
From HTTP/HTTPS
From MongoDB/DDS
From Redis
From DIS
From Kafka
From Elasticsearch/Cloud Search Service
Destination Job Parameters
To a Relational Database
To OBS
To HDFS
To Hive
To HBase/CloudTable
To DDS
To Elasticsearch/Cloud Search Service
To DLI
To DIS
Job Parameter Description
DataArts Factory APIs
Script Development APIs
Creating a Script
Modifying a Script
Querying Script Details
Querying a Script List
Querying the Execution Result of a Script Instance
Deleting a Script
Executing a Script
Stopping Executing a Script Instance
Resource Management APIs
Creating a Resource
Modifying a Resource
Querying Resource Details
Deleting a Resource
Querying a Resource List
Job Development APIs
Creating a Job
Modifying a Job
Viewing a Job List
Viewing Job Details
Viewing a Job File
Exporting a Job
Batch Exporting Jobs
Importing a Job
Executing a Job Immediately
Starting a Job
Stopping a Job
Deleting a Job
Stopping a Job Instance
Rerunning a Job Instance
Viewing Running Status of a Real-Time Job
Viewing a Job Instance List
Viewing Job Instance Details
Querying System Task Details
Creating a PatchData Instance
Querying PatchData Instances
Stopping a PatchData Instance
Changing a Job Name
Querying Release Packages
Querying Details About a Release Package
Configuring Job Tags
Querying Alarm Notifications
Releasing Task Packages
Canceling Task Packages
Querying the Instance Execution Status
Querying Completed Tasks
Querying Instances of a Specified Job
Connection Management APIs (To Be Taken Offline)
Creating a Connection (to Be Taken Offline)
Querying a Connection List (to Be Taken Offline)
Querying Connection Details (to Be Taken Offline)
Modifying a Connection (to Be Taken Offline)
Deleting a Connection (to Be Taken Offline)
Exporting Connections (to Be Taken Offline)
Importing Connections (to Be Taken Offline)
Appendix
Common Message Headers
Parsing a Stream in a Response Message
Status Codes
Error Codes
DataArts Migration Error Codes
Data Development Error Codes
FAQs
Consultation and Billing
Regions and AZs
Database, Data Warehouse, Data Lake, and Huawei FusionInsight Intelligent Data Lake
What Is the Relationship Between DataArts Studio and Huawei Horizon Digital Platform?
What Is the Relationship Between DataArts Studio and ROMA?
Can DataArts Studio Be Deployed in a Local Data Center or on a Private Cloud?
How Do I Create a Fine-Grained Permission Policy in IAM?
What Should I Do If a User Cannot View Workspaces After I Have Assigned the Required Policy to the User?
What Should I Do If Insufficient Permissions Are Prompted When I Am Trying to Perform an Operation as an IAM User?
Can I Delete DataArts Studio Workspaces?
Can I Transfer a Purchased or Trial Instance to Another Account?
Does DataArts Studio Support Version Upgrade?
Does DataArts Studio Support Version Downgrade?
How Do I View the DataArts Studio Instance Version?
What Can I Do If I Cannot Select an IAM Project When I Buy a DataArts Studio Instance?
What Is the Session Timeout Period of DataArts Studio? Can the Session Timeout Period Be Modified?
Will My Data Be Retained If My Package Expires or My Pay-per-Use Resources Are in Arrears?
How Do I Check the Remaining Validity Period of a Package?
Why Isn't the CDM Cluster in a DataArts Studio Instance Billed?
Why Does the System Display a Message Indicating that the Number of Daily Executed Nodes Has Reached the Upper Limit? What Should I Do?
Management Center
Which Data Connections Are Supported by DataArts Studio?
What Are the Precautions for Creating Data Connections?
Why Do DWS/Hive/HBase Data Connections Fail to Obtain the Information About Database or Tables?
Why Are MRS Hive/HBase Clusters Not Displayed on the Page for Creating Data Connections?
What Should I Do If the Connection Test Fails When I Enable the SSL Connection During the Creation of a DWS Data Connection?
Can I Create Multiple Data Connections in a Workspace in Proxy Mode?
Should I Choose a Direct or a Proxy Connection When Creating a DWS Connection?
How Do I Migrate the Data Development Jobs and Data Connections from One Workspace to Another?
Can I Delete Workspaces?
DataArts Migration
What Are the Differences Between CDM and Other Data Migration Services?
What Are the Advantages of CDM?
What Are the Security Protection Mechanisms of CDM?
How Do I Reduce the Cost of Using CDM?
Will I Be Billed If My CDM Cluster Does Not Use the Data Transmission Function?
Why Am I Billed Pay per Use When I Have Purchased a Yearly/Monthly CDM Incremental Package?
How Do I Check the Remaining Validity Period of a Package?
Can CDM Be Shared by Different Tenants?
Can I Upgrade a CDM Cluster?
How Is the Migration Performance of CDM?
What Is the Number of Concurrent Jobs for Different CDM Cluster Versions?
Does CDM Support Incremental Data Migration?
Does CDM Support Field Conversion?
What Component Versions Are Recommended for Migrating Hadoop Data Sources?
What Data Formats Are Supported When the Data Source Is Hive?
Can I Synchronize Jobs to Other Clusters?
Can I Create Jobs in Batches?
Can I Schedule Jobs in Batches?
How Do I Back Up CDM Jobs?
How Do I Configure the Connection If Only Some Nodes in the HANA Cluster Can Communicate with the CDM Cluster?
How Do I Use Java to Invoke CDM RESTful APIs to Create Data Migration Jobs?
How Do I Connect the On-Premises Intranet or Third-Party Private Network to CDM?
Does CDM Support Parameters or Variables?
How Do I Set the Number of Concurrent Extractors for a CDM Migration Job?
Does CDM Support Real-Time Migration of Dynamic Data?
Can I Stop CDM Clusters?
How Do I Obtain the Current Time Using an Expression?
What Should I Do If the Log Prompts that the Date Format Fails to Be Parsed?
What Can I Do If the Map Field Tab Page Cannot Display All Columns?
How Do I Select Distribution Columns When Using CDM to Migrate Data to DWS?
What Do I Do If the Error Message "value too long for type character varying" Is Displayed When I Migrate Data to DWS?
What Can I Do If Error Message "Unable to execute the SQL statement" Is Displayed When I Import Data from OBS to SQL Server?
What Should I Do If the Cluster List Is Empty, I Have No Access Permission, or My Operation Is Denied?
Why Is Error ORA-01555 Reported During Migration from Oracle to DWS?
What Should I Do If the MongoDB Connection Migration Fails?
What Should I Do If a Hive Migration Job Is Suspended for a Long Period of Time?
What Should I Do If an Error Is Reported Because the Field Type Mapping Does Not Match During Data Migration Using CDM?
What Should I Do If a JDBC Connection Timeout Error Is Reported During MySQL Migration?
What Should I Do If a CDM Migration Job Fails After a Link from Hive to DWS Is Created?
How Do I Use CDM to Export MySQL Data to an SQL File and Upload the File to an OBS Bucket?
What Should I Do If CDM Fails to Migrate Data from OBS to DLI?
What Should I Do If a CDM Connector Reports the Error "Configuration Item [linkConfig.iamAuth] Does Not Exist"?
What Should I Do If Error Message "Configuration Item [linkConfig.createBackendLinks] Does Not Exist" Is Displayed During Data Link Creation or Error Message "Configuration Item [throttlingConfig.concurrentSubJobs] Does Not Exist" Is Displayed During Job Creation?
What Should I Do If Message "CORE_0031:Connect time out. (Cdm.0523)" Is Displayed During the Creation of an MRS Hive Link?
What Should I Do If Message "CDM Does Not Support Auto Creation of an Empty Table with No Column" Is Displayed When I Enable Auto Table Creation?
What Should I Do If I Cannot Obtain the Schema Name When Creating an Oracle Relational Database Migration Job?
What Should I Do If invalid input syntax for integer: "true" Is Displayed During MySQL Database Migration?
DataArts Architecture
What Is the Relationship Between Lookup Tables and Data Standards?
What Is the Difference Between ER Modeling and Dimensional Modeling?
What Data Modeling Methods Are Supported by DataArts Architecture?
How Can I Use Standardized Data?
Does DataArts Architecture Support Database Reverse?
What Are the Differences Between the Metrics in DataArts Architecture and DataArts Quality?
Why Doesn't the Table in the Database Change After I Have Modified Fields in an ER or Dimensional Model?
Can I Configure Lifecycle Management for Tables?
DataArts Factory
How Many Jobs Can Be Created in DataArts Factory? Is There a Limit on the Number of Nodes in a Job?
Does DataArts Studio Support Custom Python Scripts?
How Can I Quickly Rectify a Deleted CDM Cluster Associated with a Job?
Why Is There a Large Difference Between Job Execution Time and Start Time of a Job?
Will Subsequent Jobs Be Affected If a Job Fails to Be Executed During Scheduling of Dependent Jobs? What Should I Do?
What Should I Pay Attention to When Using DataArts Studio to Schedule Big Data Services?
What Are the Differences and Connections Among Environment Variables, Job Parameters, and Script Parameters?
What Should I Do If Job Logs Cannot Be Opened and Error 404 Is Reported?
What Should I Do If the Agency List Fails to Be Obtained During Agency Configuration?
Why Cannot Specified Peripheral Resources Be Selected When a Data Connection Is Created in Data Development?
Why Cannot I Receive a Job Failure Alarm Notification After SMN Is Configured?
Why Is There No Job Running Scheduling Log on the Monitor Instance Page After Periodic Scheduling Is Configured for a Job?
Why does the GUI display only the failure result but not the specific error cause after Hive SQL and Spark SQL scripts fail to be executed?
What Do I Do If the Token Is Invalid During the Running of a Data Development Node?
How Do I View Run Logs After a Job Is Tested?
Why Does a Job Scheduled by Month Start Running Before the Job Scheduled by Day Is Complete?
What Should I Do If Invalid Authentication Is Reported When I Run a DLI Script?
Why Cannot I Select the Desired CDM Cluster in Proxy Mode When Creating a Data Connection?
Why Is There No Job Running Scheduling Record After Daily Scheduling Is Configured for the Job?
What Do I Do If No Content Is Displayed in Job Logs?
Why Do I Fail to Establish a Dependency Between Two Jobs?
What Should I Do If an Error Is Displayed During DataArts Studio Scheduling: The Job Does Not Have a Submitted Version?
What Do I Do If an Error Is Displayed During DataArts Studio Scheduling: The Script Associated with Node XXX in the Job Is Not Submitted?
What Should I Do If a Job Fails to Be Executed After Being Submitted for Scheduling and an Error Displayed: Depend Job [XXX] Is Not Running Or Pause?
How Do I Create a Database And Data Table? Is the database a data connection?
Why Is No Result Displayed After an HIVE Task Is Executed?
Why Does the Last Instance Status On the Monitor Instance page Only Display Succeeded or Failed?
How Do I Create a Notification for All Jobs?
What Is the Maximum Number of Nodes That Can Be Executed Simultaneously?
Can I Change the Time Zone of a DataArts Studio Instance?
How Do I Synchronize the Changed Names of CDM Jobs to DataArts Factory?
Why Does the Execution of an RDS SQL Statement Fail and an Error Is Reported Indicating That hll Does Not Exist?
What Should I Do If Error Message "The account has been locked" Is Displayed When I Am Creating a DWS Data Connection?
Why Is a Job Instance Canceled and Message "The node start execute failed, so the current node status is set to cancel." Displayed?
What Should I Do If Error Message "Workspace does not exists" Is Displayed When I Call a DataArts Factory API?
Why Don't the URL Parameters for Calling an API Take Effect in the Test Environment When the API Can Be Called Properly Using Postman?
What Should I Do If Error Message "Agent need to be updated?" Is Displayed When I Run a Python Script?
Why Is an Execution Failure Displayed for a Node in the Log When the Node Status Is Successful?
What Should I Do If an Unknown Exception Occurs When I Call a DataArts Factory API?
Why Is an Error Message Indicating an Invalid Resource Name Is Displayed When I Call a Resource Creation API?
Why Does a PatchData Task Fail When All PatchData Job Instances Are Successful?
Why Is a Table Unavailable When an Error Message Indicating that the Table Already Exists Is Displayed During Table Creation from a DWS Data Connection?
What Should I Do If Error Message "The throttling threshold has been reached: policy user over ratelimit,limit:60,time:1 minute." Is Displayed When I Schedule an MRS Spark Job?
What Should I Do If Error Message "UnicodeEncodeError: 'ascii' codec can't encode characters in position 63-64: ordinal not in range(128)" Is Displayed When I Run a Python Script?
What Should I Do If an Error Message Is Displayed When I View Logs?
What Should I Do If a Shell/Python Node Fails and Error "session is down" Is Reported?
What Should I Do If a Parameter Value in a Request Header Contains More Than 512 Characters?
What Should I Do If a Message Is Displayed Indicating that the ID Does Not Exist During the Execution of a DWS SQL Script?
How Do I Check Which Jobs Invoke a CDM Job?
What Should I Do If Error Message "The request parameter invalid" Is Displayed When I Use Python to Call the API for Executing Scripts?
DataArts Quality
What Are the Differences Between Quality Jobs and Comparison Jobs?
How Can I Confirm that a Quality Job or Comparison Job Is Blocked?
How Do I Manually Restart a Blocked Quality Job or Comparison Job?
How Do I View Jobs Associated with a Quality Rule Template?
What Should I Do If the System Displays a Message Indicating that I Do Not Have the MRS Permission to Perform a Quality Job?
DataArts Catalog
What Are the Functions of the DataArts Catalog Module?
What Assets Can Be Collected by DataArts Catalog?
What Is Data Lineage?
How Do I Visualize Data Lineages in a Data Catalog?
DataArts DataService
What Languages Do Data Lake Mall SDKs Support?
What Can I Do If the System Displays a Message Indicating that the Proxy Fails to Be Invoked During API Creation?
What Should I Do If the Background Reports an Error When I Access the Test App Through the Data Service API and Set Related Parameters?
What Can I Do If an Error Is Reported When I Use an API?
Can Operators Be Transferred When API Parameters Are Transferred?
What Should I Do If the API Quota Provided by DataArts DataService Exclusive Has Been Used up?
How Do I Bind an EIP and a Domain Name to an API Published in DataArts DataService?
What Should I Do If It Takes a Long Time to Obtain the Total Number of Data Records of a Table Through an API If the Table Contains a Large Amount of Data?
More Documents
User Guide (Kuala Lumpur Region)
Service Overview
What Is DataArts Studio?
Basic Concepts
Functions
Advantages
Application Scenarios
DataArts Studio Permissions Management
DataArts Studio Permissions
Constraints
Related Services
Preparations
Preparations
Creating DataArts Studio Instances
Creating a DataArts Studio Basic Package
(Optional) Creating a DataArts Studio Incremental Package
Managing a Workspace
Creating and Managing a Workspace
(Optional) Changing the Job Log Storage Path
Authorizing Users to Use DataArts Studio
Creating an IAM User and Assigning DataArts Studio Permissions
Adding a Member and Assigning a Role
(Optional) Obtaining Authentication Information
User Guide
Preparations Before Using DataArts Studio
Management Center
Data Sources
Creating Data Connections
Migrating Resources
Tutorials
Creating an MRS Hive Connection
Creating a DWS Connection
Creating a MySQL Connection
DataArts Migration
Overview
Constraints
Supported Data Sources
Managing Clusters
Creating a CDM Cluster
Binding or Unbinding an EIP
Restarting a Cluster
Deleting a Cluster
Downloading Cluster Logs
Viewing Basic Cluster Information and Modifying Cluster Configurations
Viewing Metrics
CDM Metrics
Configuring Alarm Rules
Querying Metrics
Managing Links
Creating Links
Managing Drivers
Managing Agents
Managing Cluster Configurations
Link to a Common Relational Database
Link to a Database Shard
Link to MyCAT
Link to a Dameng Database
Link to a MySQL Database
Link to an Oracle Database
Link to DLI
Link to Hive
Link to HBase
Link to HDFS
Link to OBS
Link to an FTP or SFTP Server
Link to Redis/DCS
Link to DDS
Link to CloudTable
Link to CloudTable OpenTSDB
Link to MongoDB
Link to Cassandra
Link to Kafka
Link to DMS Kafka
Link to Elasticsearch/CSS
Managing Jobs
Table/File Migration Jobs
Creating an Entire Database Migration Job
Source Job Parameters
From OBS
From HDFS
From HBase/CloudTable
From Hive
From DLI
From FTP/SFTP
From HTTP
From a Common Relational Database
From MySQL
From Oracle
From a Database Shard
From MongoDB/DDS
From Redis
From Kafka/DMS Kafka
From Elasticsearch or CSS
From OpenTSDB
Destination Job Parameters
To OBS
To HDFS
To HBase/CloudTable
To Hive
To a Common Relational Database
To DWS
To DDS
To DCS
To CSS
To DLI
To OpenTSDB
Scheduling Job Execution
Job Configuration Management
Managing a Single Job
Managing Jobs in Batches
Auditing
Key CDM Operations Recorded by CTS
Viewing Traces
Tutorials
Creating an MRS Hive Link
Creating a MySQL Link
Migrating Data from MySQL to MRS Hive
Migrating Data from MySQL to OBS
Migrating Data from MySQL to DWS
Migrating an Entire MySQL Database to RDS
Migrating Data from Oracle to CSS
Migrating Data from Oracle to DWS
Migrating Data from OBS to CSS
Migrating Data from OBS to DLI
Migrating Data from MRS HDFS to OBS
Migrating the Entire Elasticsearch Database to CSS
Advanced Operations
Incremental Migration
Incremental File Migration
Incremental Migration of Relational Databases
Using Macro Variables of Date and Time
HBase/CloudTable Incremental Migration
Migration in Transaction Mode
Encryption and Decryption During File Migration
MD5 Verification
Field Conversion
Migrating Files with Specified Names
Regular Expressions for Separating Semi-structured Text
Recording the Time When Data Is Written to the Database
File Formats
DataArts Factory
Overview
Data Management
Data Management Process
Creating a Data Connection
Creating a Database
(Optional) Creating a Database Schema
Creating a Table
Script Development
Script Development Process
Creating a Script
Developing Scripts
Developing an SQL Script
Developing a Shell Script
Developing a Python Script
Submitting a Version and Unlocking the Script
(Optional) Managing Scripts
Copying a Script
Copying the Script Name and Renaming a Script
Moving a Script or Script Directory
Exporting and Importing a Script
Viewing Script References
Deleting a Script
Changing the Script Owner
Unlocking Scripts
Job Development
Job Development Process
Creating a Job
Developing a Job
Setting Up Scheduling for a Job
Submitting a Version and Unlocking the Script
(Optional) Managing Jobs
Copying a Job
Copying the Job Name and Renaming a Job
Moving a Job or Job Directory
Exporting and Importing a Job
Deleting a Job
Changing the Job Owner
Unlocking Jobs
Solution
Execution History
O&M and Scheduling
Overview
Monitoring a Job
Monitoring a Batch Job
Monitoring a Real-Time Job
Monitoring an Instance
Monitoring PatchData
Managing Notifications
Managing a Notification
Cycle Overview
Managing Backups
Configuration and Management
Configuring Resources
Configuring Environment Variables
Configuring an OBS Bucket
Managing Job Labels
Configuring Agencies
Configuring a Default Item
Managing Resources
Node Reference
Node Overview
CDM Job
Rest Client
Import GES
MRS Kafka
Kafka Client
ROMA FDI Job
DLI Flink Job
DLI SQL
DLI Spark
DWS SQL
MRS Spark SQL
MRS Hive SQL
MRS Presto SQL
MRS Spark
MRS Spark Python
MRS Flink Job
MRS MapReduce
CSS
Shell
RDS SQL
ETL Job
Python
Create OBS
Delete OBS
OBS Manager
Open/Close Resource
Subjob
For Each
SMN
Dummy
EL Expression Reference
Expression Overview
Basic Operators
Date and Time Mode
Env Embedded Objects
Job Embedded Objects
StringUtil Embedded Objects
DateUtil Embedded Objects
JSONUtil Embedded Objects
Loop Embedded Objects
OBSUtil Embedded Objects
Expression Use Example
Usage Guidance
Job Dependency
IF Statements
Obtaining the Return Value of a Rest Client Node
Using For Each Nodes
Developing a Python Script
Developing a DWS SQL Job
Developing a Hive SQL Job
Developing a DLI Spark Job
Developing an MRS Flink Job
Developing an MRS Spark Python Job
FAQs
Consultation
Regions
What Should I Do If a User Cannot View Existing Workspaces After I Have Assigned the Required Policy to the User?
Can I Delete DataArts Studio Workspaces?
Can I Transfer a Trial Instance to Another Account?
Does DataArts Studio Support Version Downgrade?
Management Center
What Are the Precautions for Creating Data Connections?
Why Do DWS/Hive/HBase Data Connections Fail to Obtain the Information About Database or Tables?
Why Are MRS Hive/HBase Clusters Not Displayed on the Page for Creating Data Connections?
What Should I Do If the Connection Test Fails When I Enable the SSL Connection During the Creation of a DWS Data Connection?
Can I Create Multiple Data Connections in a Workspace in Proxy Mode?
Should I Choose a Direct or a Proxy Connection When Creating a DWS Connection?
How Do I Migrate the Data Development Jobs and Data Connections from One Workspace to Another?
Can I Delete Workspaces?
DataArts Migration
General
What Are the Advantages of CDM?
What Are the Security Protection Mechanisms of CDM?
How Do I Reduce the Cost of Using CDM?
Can I Upgrade a CDM Cluster?
How Is the Migration Performance of CDM?
What Is the Number of Concurrent Jobs for Different CDM Cluster Versions?
Functions
Does CDM Support Incremental Data Migration?
Does CDM Support Field Conversion?
What Component Versions Are Recommended for Migrating Hadoop Data Sources?
What Data Formats Are Supported When the Data Source Is Hive?
Can I Synchronize Jobs to Other Clusters?
Can I Create Jobs in Batches?
Can I Schedule Jobs in Batches?
How Do I Back Up CDM Jobs?
How Do I Configure the Connection If Only Some Nodes in the HANA Cluster Can Communicate with the CDM Cluster?
How Do I Use Java to Invoke CDM RESTful APIs to Create Data Migration Jobs?
How Do I Connect the On-Premises Intranet or Third-Party Private Network to CDM?
How Do I Set the Number of Concurrent Extractors for a CDM Migration Job?
Does CDM Support Real-Time Migration of Dynamic Data?
Troubleshooting
What Can I Do If Error Message "Unable to execute the SQL statement" Is Displayed When I Import Data from OBS to SQL Server?
Why Is Error ORA-01555 Reported During Migration from Oracle to DWS?
What Should I Do If the MongoDB Connection Migration Fails?
What Should I Do If a Hive Migration Job Is Suspended for a Long Period of Time?
What Should I Do If an Error Is Reported Because the Field Type Mapping Does Not Match During Data Migration Using CDM?
What Should I Do If a JDBC Connection Timeout Error Is Reported During MySQL Migration?
What Should I Do If a CDM Migration Job Fails After a Link from Hive to DWS Is Created?
How Do I Use CDM to Export MySQL Data to an SQL File and Upload the File to an OBS Bucket?
What Should I Do If CDM Fails to Migrate Data from OBS to DLI?
What Should I Do If a CDM Connector Reports the Error "Configuration Item [linkConfig.iamAuth] Does Not Exist"?
What Should I Do If Error Message "Configuration Item [linkConfig.createBackendLinks] Does Not Exist" Is Displayed During Data Link Creation or Error Message "Configuration Item [throttlingConfig.concurrentSubJobs] Does Not Exist" Is Displayed During Job Creation?
What Should I Do If Message "CORE_0031:Connect time out. (Cdm.0523)" Is Displayed During the Creation of an MRS Hive Link?
What Should I Do If Message "CDM Does Not Support Auto Creation of an Empty Table with No Column" Is Displayed When I Enable Auto Table Creation?
What Should I Do If I Cannot Obtain the Schema Name When Creating an Oracle Relational Database Migration Job?
DataArts Factory
How Many Jobs Can Be Created in DataArts Factory? Is There a Limit on the Number of Nodes in a Job?
Why Is There a Large Difference Between Job Execution Time and Start Time of a Job?
Will Subsequent Jobs Be Affected If a Job Fails to Be Executed During Scheduling of Dependent Jobs? What Should I Do?
What Should I Pay Attention to When Using DataArts Studio to Schedule Big Data Services?
What Are the Differences and Connections Among Environment Variables, Job Parameters, and Script Parameters?
What Do I Do If Node Error Logs Cannot Be Viewed When a Job Fails?
What Should I Do If the Agency List Fails to Be Obtained During Agency Configuration?
How Do I Locate Job Scheduling Nodes with a Large Number?
Why Cannot Specified Peripheral Resources Be Selected When a Data Connection Is Created in Data Development?
Why Is There No Job Running Scheduling Log on the Monitor Instance Page After Periodic Scheduling Is Configured for a Job?
Why Does the GUI Display Only the Failure Result but Not the Specific Error Cause After Hive SQL and Spark SQL Scripts Fail to Be Executed?
What Do I Do If the Token Is Invalid During the Running of a Data Development Node?
How Do I View Run Logs After a Job Is Tested?
Why Does a Job Scheduled by Month Start Running Before the Job Scheduled by Day Is Complete?
What Should I Do If Invalid Authentication Is Reported When I Run a DLI Script?
Why Cannot I Select the Desired CDM Cluster in Proxy Mode When Creating a Data Connection?
Why Is There No Job Running Scheduling Record After Daily Scheduling Is Configured for the Job?
What Do I Do If No Content Is Displayed in Job Logs?
Why Do I Fail to Establish a Dependency Between Two Jobs?
What Should I Do If an Error Is Displayed During DataArts Studio Scheduling: The Job Does Not Have a Submitted Version?
What Do I Do If an Error Is Displayed During DataArts Studio Scheduling: The Script Associated with Node XXX in the Job Is Not Submitted?
What Should I Do If a Job Fails to Be Executed After Being Submitted for Scheduling and an Error Displayed: Depend Job [XXX] Is Not Running Or Pause?
How Do I Create a Database And Data Table? Is the database a data connection?
Why Is No Result Displayed After an HIVE Task Is Executed?
Why Does the Last Instance Status On the Monitor Instance page Only Display Succeeded or Failed?
How Do I Create a Notification for All Jobs?
How Many Nodes Can Be Executed Concurrently in Each DataArts Studio Version?
What Is the Priority of the Startup User, Execution User, Workspace Agency, and Job Agency?
API Reference (Kuala Lumpur Region)
Before You Start
Overview
API Calling
Concepts
Endpoints
Project ID and Account ID
DataArts Studio Instance ID and Workspace ID
Data Development Job ID
Constraints
API Overview
Calling APIs
Making an API Request
Authentication
Response
Application Cases
Application Example
Example of Using Data Development APIs
DataArts Migration APIs
Cluster Management
Querying Cluster Details
Deleting a Cluster
Restarting a Cluster
Starting a Cluster
Stopping a Cluster
Creating a Cluster
Querying the Cluster List
Job Management
Querying a Job
Deleting a Job
Modifying a Job
Creating and Executing a Job in a Random Cluster
Stopping a Job
Creating a Job in a Specified Cluster
Starting a Job
Querying Job Status
Querying Job Execution History
Link Management
Creating a Link
Querying a Link
Deleting a Link
Modifying a Link
Public Data Structures
Link Parameter Description
Link to a Relational Database
Link to OBS
Link to HDFS
Link to HBase
Link to CloudTable
Link to Hive
Link to an FTP or SFTP Server
Link to MongoDB
Link to Redis/DCS (to Be Brought Offline)
Link to Kafka
Link to Elasticsearch/Cloud Search Service
Link to DLI
Link to CloudTable OpenTSDB
Link to Amazon S3
Link to DMS Kafka
Source Job Parameters
From a Relational Database
From Object Storage
From HDFS
From Hive
From HBase/CloudTable
From FTP/SFTP/NAS (to Be Brought Offline)/SFS (to Be Brought Offline)
From HTTP/HTTPS
From MongoDB/DDS
From Redis/DCS (to Be Brought Offline)
From DIS
From Kafka
From Elasticsearch/Cloud Search Service
From OpenTSDB
Destination Job Parameters
To a Relational Database
To OBS
To HDFS
To Hive
To HBase/CloudTable
To FTP/SFTP/NAS (to Be Brought Offline)/SFS (to Be Brought Offline)
To DDS
To DCS
To Elasticsearch/Cloud Search Service
To DLI
To DIS
To OpenTSDB
Job Parameter Description
DataArts Factory APIs
Connection Management APIs
Creating a Connection
Querying a Connection List
Viewing Connection Details
Editing a Connection
Deleting a Connection
Exporting a Connection
Script Development APIs
Deleting a Script
Executing a Script
Stopping Executing a Script Instance
Resource Management APIs
Deleting a Resource
Job Development APIs
Viewing a Job File
Stopping a Job
Deleting a Job
Stopping a Job Instance
Retrying a Job Instance
Data Structure
APIs to Be Taken Offline
Creating a Job
Editing a Job
Viewing a Job List
Viewing Job Details
Exporting a Job
Batch Exporting Jobs
Importing a Job
Executing a Job Immediately
Starting a Job
Viewing Running Status of a Real-Time Job
Viewing a Job Instance List
Viewing Job Instance Details
Querying a System Task
Creating a Script
Modifying a Script
Querying a Script
Querying a Script List
Querying the Execution Result of a Script Instance
Creating a Resource
Modifying a Resource
Querying a Resource
Querying a Resource List
Importing a Connection
Appendix
Common Message Headers
Status Codes
Error Codes
CDM Error Codes
DLF Error Codes
Parsing a Stream in a Response Message
General Reference
Glossary
Service Level Agreement
White Papers
Endpoints
Permissions