هذه الصفحة غير متوفرة حاليًا بلغتك المحلية. نحن نعمل جاهدين على إضافة المزيد من اللغات. شاكرين تفهمك ودعمك المستمر لنا.
- What's New
- Function Overview
- Service Overview
- Data Governance Methodology
- Preparations
- Getting Started
-
User Guide
- DataArts Studio development process
-
Buying and Configuring a DataArts Studio Instance
- Buying a DataArts Studio Instance
-
Buying a DataArts Studio Incremental Package
- Introduction to Incremental Packages
- Buying a DataArts Migration Incremental Package
- Buying a DataArts Migration Resource Group Incremental Package
- Buying a DataArts DataService Exclusive Cluster Incremental Package
- Buying an Incremental Package for Job Node Scheduling Times/Day
- Buying an Incremental Package for Technical Asset Quantity
- Buying an Incremental Package for Data Model Quantity
- Accessing the DataArts Studio Instance Console
- Creating and Configuring a Workspace in Simple Mode
- (Optional) Creating and Using a Workspace in Enterprise Mode
- Managing DataArts Studio Resources
- Authorizing Users to Use DataArts Studio
-
Management Center
- Data Sources Supported by DataArts Studio
- Creating a DataArts Studio Data Connection
-
Configuring DataArts Studio Data Connection Parameters
- DWS Connection Parameters
- DLI Connection Parameters
- MRS Hive Connection Parameters
- MRS HBase Connection Parameters
- MRS Kafka Connection Parameters
- MRS Spark Connection Parameters
- MRS ClickHouse Connection Parameters
- MRS Hetu Connection Parameters
- MRS Impala Connection Parameters
- MRS Ranger Connection Parameters
- MRS Presto Connection Parameters
- Doris Connection Parameters
- OpenSource ClickHouse Connection Parameters
- RDS Connection Parameters
- Oracle Connection Parameters
- DIS Connection Parameters
- Host Connection Parameters
- Rest Client Connection Parameters
- Redis Connection Parameters
- SAP HANA Connection Parameters
- LTS Connection Parameters
- Configuring DataArts Studio Resource Migration
- Configuring Environment Isolation for a DataArts Studio Workspace in Enterprise Mode
- Typical Scenarios for Using Management Center
-
DataArts Migration (CDM Jobs)
- Overview
- Notes and Constraints
- Supported Data Sources
- Creating and Managing a CDM Cluster
-
Creating a Link in a CDM Cluster
- Creating a Link Between CDM and a Data Source
-
Configuring Link Parameters
- OBS Link Parameters
- PostgreSQL/SQLServer Link Parameters
- GaussDB(DWS) Link Parameters
- RDS for MySQL/MySQL Database Link Parameters
- Oracle Database Link Parameters
- DLI Link Parameters
- Hive Link Parameters
- HBase Link Parameters
- HDFS Link Parameters
- FTP/SFTP Link Parameters
- Redis Link Parameters
- DDS Link Parameters
- CloudTable Link Parameters
- MongoDB Link Parameters
- Cassandra Link Parameters
- DIS Link Parameters
- Kafka Link Parameters
- DMS Kafka Link Parameters
- CSS Link Parameters
- Elasticsearch Link Parameters
- Dameng Database Link Parameters
- SAP HANA Link Parameters
- Shard Link Parameters
- MRS Hudi Link Parameters
- MRS ClickHouse Link Parameters
- ShenTong Database Link Parameters
- CloudTable OpenTSDB Link Parameters
- GBASE Link Parameters
- YASHAN Link Parameters
- Uploading a CDM Link Driver
- Creating a Hadoop Cluster Configuration
-
Creating a Job in a CDM Cluster
- Table/File Migration Jobs
- Creating an Entire Database Migration Job
-
Configuring CDM Source Job Parameters
- From OBS
- From HDFS
- From HBase/CloudTable
- From Hive
- From DLI
- From FTP/SFTP
- From HTTP
- From PostgreSQL/SQL Server
- From DWS
- From SAP HANA
- From MySQL
- From Oracle
- From a Database Shard
- From MongoDB/DDS
- From Redis
- From DIS
- From Kafka/DMS Kafka
- From Elasticsearch or CSS
- From OpenTSDB
- From MRS Hudi
- From MRS ClickHouse
- From a ShenTong Database
- From a Dameng Database
- From YASHAN
- Configuring CDM Destination Job Parameters
- Configuring CDM Job Field Mapping
- Configuring a Scheduled CDM Job
- Managing CDM Job Configuration
- Managing a CDM Job
- Managing CDM Jobs
- Using Macro Variables of Date and Time
- Improving Migration Performance
-
Key Operation Guide
- Incremental Migration
- Migration in Transaction Mode
- Encryption and Decryption During File Migration
- MD5 Verification
- Configuring Field Converters
- Adding Fields
- Migrating Files with Specified Names
- Regular Expressions for Separating Semi-structured Text
- Recording the Time When Data Is Written to the Database
- File Formats
- Converting Unsupported Data Types
- Auto Table Creation
-
Tutorials
- Creating an MRS Hive Link
- Creating a MySQL Link
- Migrating Data from MySQL to MRS Hive
- Migrating Data from MySQL to OBS
- Migrating Data from MySQL to DWS
- Migrating an Entire MySQL Database to RDS
- Migrating Data from Oracle to CSS
- Migrating Data from Oracle to DWS
- Migrating Data from OBS to CSS
- Migrating Data from OBS to DLI
- Migrating Data from MRS HDFS to OBS
- Migrating the Entire Elasticsearch Database to CSS
- Error Codes
- DataArts Migration (Offline Jobs)
-
DataArts Migration (Real-Time Jobs)
- Overview of Real-Time Jobs
- Supported Data Sources
- Check Before Use
-
Enabling Network Communications
- Database Deployed in an On-premises IDC
- Database Deployed on Another Cloud
-
Database Deployed on Huawei Cloud
- Enabling Network Communications Directly for the Same Region and Tenant
- Using a VPC Peering Connection to Enable Network Communications for the Same Region but Different Tenants
- Using an Enterprise Router to Enable Network Communications for the Same Region but Different Tenants
- Using a Cloud Connection to Enable Cross-Region Network Communications
- Creating a Real-Time Migration Job
- Configuring a Real-Time Migration Job
- Real-Time Migration Job O&M
- Field Type Mapping
-
Job Performance Optimization
- Overview
- Optimizing Job Parameters
- Optimizing the Parameters of a Job for Migrating Data from MySQL to MRS Hudi
- Optimizing the Parameters of a Job for Migrating Data from MySQL to GaussDB(DWS)
- Optimizing the Parameters of a Job for Migrating Data from MySQL to DMS for Kafka
- Optimizing the Parameters of a Job for Migrating Data from DMS for Kafka to OBS
- Optimizing the Parameters of a Job for Migrating Data from Apache Kafka to MRS Kafka
- Optimizing the Parameters of a Job for Migrating Data from SQL Server to MRS Hudi
- Optimizing the Parameters of a Job for Migrating Data from PostgreSQL to GaussDB(DWS)
- Optimizing the Parameters of a Job for Migrating Data from Oracle to GaussDB(DWS)
- Optimizing the Parameters of a Job for Migrating Data from Oracle to MRS Hudi
-
Tutorials
- Overview
- Migrating a DRS Task to DataArts Migration
- Configuring a Job for Synchronizing Data from MySQL to MRS Hudi
- Configuring a Job for Synchronizing Data from MySQL to GaussDB(DWS)
- Configuring a Job for Synchronizing Data from MySQL to Kafka
- Configuring a Job for Synchronizing Data from DMS for Kafka to OBS
- Configuring a Job for Synchronizing Data from Apache Kafka to MRS Kafka
- Configuring a Job for Synchronizing Data from SQL Server to MRS Hudi
- Configuring a Job for Synchronizing Data from PostgreSQL to GaussDB(DWS)
- Configuring a Job for Synchronizing Data from Oracle to GaussDB(DWS)
- Configuring a Job for Synchronizing Data from Oracle to MRS Hudi
- Configuring a Job for Synchronizing Data from MongoDB to GaussDB(DWS)
- DataArts Architecture
-
DataArts Factory
- Overview
- Data Management
- Script Development
-
Job Development
- Job Development Process
- Creating a Job
- Developing a Pipeline Job
- Developing a Batch Processing Single-Task SQL Job
- Developing a Real-Time Processing Single-Task MRS Flink SQL Job
- Developing a Real-Time Processing Single-Task MRS Flink Jar Job
- Developing a Real-Time Processing Single-Task DLI Spark Job
- Setting Up Scheduling for a Job
- Submitting a Version
- Releasing a Job Task
- (Optional) Managing Jobs
- Notebook Development
- Solution
- Execution History
- O&M and Scheduling
- Configuration and Management
- Review Center
- Download Center
-
Node Reference
- Node Overview
- Node Lineages
- CDM Job
- Data Migration
- DIS Stream
- DIS Dump
- DIS Client
- Rest Client
- Import GES
- MRS Kafka
- Kafka Client
- ROMA FDI Job
- DLI Flink Job
- DLI SQL
- DLI Spark
- DWS SQL
- MRS Spark SQL
- MRS Hive SQL
- MRS Presto SQL
- MRS Spark
- MRS Spark Python
- MRS ClickHouse
- MRS Impala SQL
- MRS Flink Job
- MRS MapReduce
- CSS
- Shell
- RDS SQL
- ETL Job
- Python
- DORIS SQL
- ModelArts Train
- Create OBS
- Delete OBS
- OBS Manager
- Open/Close Resource
- Data Quality Monitor
- Subjob
- For Each
- SMN
- Dummy
- EL Expression Reference
- Simple Variable Set
-
Usage Guidance
- Referencing Parameters in Scripts and Jobs
- Setting the Job Scheduling Time to the Last Day of Each Month
- Configuring a Yearly Scheduled Job
- Using PatchData
- Obtaining the Output of an SQL Node
- Obtaining the Maximum Value and Transferring It to a CDM Job Using a Query SQL Statement
- IF Statements
- Obtaining the Return Value of a Rest Client Node
- Using For Each Nodes
- Using Script Templates and Parameter Templates
- Developing a Python Job
- Developing a DWS SQL Job
- Developing a Hive SQL Job
- Developing a DLI Spark Job
- Developing an MRS Flink Job
- Developing an MRS Spark Python Job
- DataArts Quality
- DataArts Catalog
-
DataArts Security
- Overview
- Dashboard
- Unified Permission Governance
- Sensitive Data Governance
- Sensitive Data Protection
- Data Security Operations
- Managing the Recycle Bin
-
DataArts DataService
- Overview
- Specifications
- Developing APIs in DataArts DataService
-
Calling APIs in DataArts DataService
- Applying for API Authorization
-
Calling APIs Using Different Methods
- API Calling Methods
- (Recommended) Using an SDK to Call an API Which Uses App Authentication
- Using an API Tool to Call an API Which Uses App Authentication
- Using an API Tool to Call an API Which Uses IAM Authentication
- Using an API Tool to Call an API Which Requires No Authentication
- Using a Browser to Call an API Which Requires No Authentication
- Viewing API Access Logs
- Configuring Review Center
- Audit Log
-
Best Practices
-
Advanced Data Migration Guidance
- Incremental Migration
- Using Macro Variables of Date and Time
- Migration in Transaction Mode
- Encryption and Decryption During File Migration
- MD5 Verification
- Configuring Field Converters
- Adding Fields
- Migrating Files with Specified Names
- Regular Expressions for Separating Semi-structured Text
- Recording the Time When Data Is Written to the Database
- File Formats
- Converting Unsupported Data Types
-
Advanced Data Development Guidance
- Dependency Policies for Periodic Scheduling
- Scheduling by Discrete Hours and Scheduling by the Nearest Job Instance
- Using PatchData
- Setting the Job Scheduling Time to the Last Day of Each Month
- Obtaining the Output of an SQL Node
- IF Statements
- Obtaining the Return Value of a Rest Client Node
- Using For Each Nodes
- Invoking DataArts Quality Operators Using DataArts Factory and Transferring Quality Parameters During Job Running
- Scheduling Jobs Across Workspaces
-
DataArts Studio Data Migration Configuration
- Overview
- Management Center Data Migration Configuration
- DataArts Migration Data Migration Configuration
- DataArts Architecture Data Migration Configuration
- DataArts Factory Data Migration Configuration
- DataArts Quality Data Migration Configuration
- DataArts Catalog Data Migration Configuration
- DataArts Security Data Migration Configuration
- DataArts DataService Data Migration Configuration
- Least Privilege Authorization
- How Do I View the Number of Table Rows and Database Size?
- Comparing Data Before and After Data Migration Using DataArts Quality
- Configuring Alarms for Jobs in DataArts Factory of DataArts Studio
- Scheduling a CDM Job by Transferring Parameters Using DataArts Factory
- Enabling Incremental Data Migration Through DataArts Factory
- Creating Table Migration Jobs in Batches Using CDM Nodes
- Automatic Construction and Analysis of Graph Data
- Simplified Migration of Trade Data to the Cloud and Analysis
- Migration of IoV Big Data to the Lake Without Loss
- Real-Time Alarm Platform Construction
-
Advanced Data Migration Guidance
- SDK Reference
-
API Reference
- Before You Start
- API Overview
- Calling APIs
-
DataArts Migration APIs
-
Cluster Management
- Querying Cluster Details
- Deleting a Cluster
- Querying All AZs
- Querying Supported Versions
- Querying Version Specifications
- Querying Details About a Flavor
- Querying the Enterprise Project IDs of All Clusters
- Querying the Enterprise Project ID of a Specified Cluster
- Query a Specified Instance in a Cluster
- Modifying a Cluster
- Restarting a Cluster
- Starting a Cluster
- Stopping a Cluster (To Be Taken Offline)
- Creating a Cluster
- Querying the Cluster List
- Job Management
- Link Management
- Public Data Structures
-
Cluster Management
-
DataArts Factory APIs (V1)
- Script Development APIs
- Resource Management APIs
-
Job Development APIs
- Creating a Job
- Modifying a Job
- Viewing a Job List
- Viewing Job Details
- Viewing a Job File
- Exporting a Job
- Batch Exporting Jobs
- Importing a Job
- Executing a Job Immediately
- Starting a Job
- Stopping a Job
- Deleting a Job
- Stopping a Job Instance
- Rerunning a Job Instance
- Viewing Running Status of a Real-Time Job
- Viewing a Job Instance List
- Viewing Job Instance Details
- Querying System Task Details
-
Connection Management APIs (To Be Taken Offline)
- Creating a Connection (to Be Taken Offline)
- Querying a Connection List (to Be Taken Offline)
- Querying Connection Details (to Be Taken Offline)
- Modifying a Connection (to Be Taken Offline)
- Deleting a Connection (to Be Taken Offline)
- Exporting Connections (to Be Taken Offline)
- Importing Connections (to Be Taken Offline)
-
DataArts Factory APIs (V2)
-
Job Development APIs
- Creating a PatchData Instance
- Querying PatchData Instances
- Stopping a PatchData Instance
- Changing a Job Name
- Querying Release Packages
- Querying Details About a Release Package
- Configuring Job Tags
- Querying Alarm Notifications
- Releasing Task Packages
- Canceling Task Packages
- Querying the Instance Execution Status
- Querying Completed Tasks
- Querying Instances of a Specified Job
-
Job Development APIs
-
DataArts Architecture APIs
- Overview
- Information Architecture
- Data Standards
- Data Sources
- Process Architecture
- Data Standard Templates
- Approval Management
- Subject Management
- Subject Levels
- Catalog Management
- Atomic Metrics
- Derivative Metrics
- Compound Metrics
- Dimensions
- Filters
- Dimension Tables
- Fact Tables
- Summary Tables
- Business Metrics
- Version Information
-
ER Modeling
- Lookup Table Model List
- Creating a Table Model
- Updating a Table Model
- Deleting a Table Model
- Querying a Relationship
- Viewing Relationship Details
- Querying All Relationships in a Model
- Viewing Table Model Details
- Obtaining a Model
- Creating a Model Workspace
- Updating the Model Workspace
- Deleting a Model Workspace
- Viewing Details About a Model
- Querying Destination Tables and Fields (To Be Offline)
- Exporting DDL Statements of Tables in a Model
- Converting a Logical Model to a Physical Model
- Obtaining the Operation Result
- Import and Export
- Customized Items
- Quality Rules
- Tag API
- Lookup Table Management
- DataArts Quality APIs
-
DataArts DataService APIs
-
API Management
- Create an API
- Querying an API List
- Updating an API
- Querying API Information
- Deleting APIs
- Publishing an API
- API operations (offline/suspension/resumption)
- Batch Authorization API (Exclusive Edition)
- Debugging an API
- API authorization operations (authorization/authorization cancellation/application/renewal)
- Querying API Publishing Messages in DLM Exclusive
- Querying Instances for API Operations in DLM Exclusive
- Querying API Debugging Messages in DLM Exclusive
- Importing an Excel File Containing APIs
- Exporting an Excel File Containing APIs
- Exporting a .zip File Containing All APIs
- Downloading an Excel Template
- Application Management
- Message Management
- Authorization Management
-
Service Catalog Management
- Obtaining the List of APIs and Catalogs in a Catalog
- Obtaining the List of APIs in a Catalog
- Obtaining the List of Sub-Catalogs in a Catalog
- Updating a Service Catalog
- Query the service catalog
- Creating a Service Catalog
- Deleting Directories in Batches
- Moving a Catalog to Another Catalog
- Moving APIs to Another Catalog
- Obtaining the ID of a Catalog Through Its Path
- Obtaining the Path of a Catalog Through Its ID
- Obtaining the Paths to a Catalog Through Its ID
- Querying the Service Catalog API List
- Gateway Management
- App Management
-
Overview
- Querying and Collecting Statistics on User-related Overview Development Indicators
- This API is used to query and collect statistics on user-related overview invoking metrics.
- Querying Top N API Services Invoked
- Querying Top N Services Used by an App
- Querying API Statistics Details
- Querying App Statistics
- Querying API Dashboard Data Details
- Querying Data Details of a Specified API Dashboard
- Querying App Dashboard Data Details
- Querying Top N APIs Called by a Specified API Application
- Cluster Management
-
API Management
- Application Cases
- Appendix
-
FAQs
-
Consultation and Billing
- How Do I Select a Region and an AZ?
- What Is a Database, Data Warehouse, Data Lake, and Huawei FusionInsight Intelligent Data Lake? What Are the Differences and Relationships Between Them?
- What Is the Relationship Between DataArts Studio and Huawei Horizon Digital Platform?
- What Are the Differences Between DataArts Studio and ROMA?
- Can DataArts Studio Be Deployed in a Local Data Center or on a Private Cloud?
- How Do I Create a Fine-Grained Permission Policy in IAM?
- How Do I Isolate Workspaces So That Users Cannot View Unauthorized Workspaces?
- What Should I Do If a User Cannot View Workspaces After I Have Assigned the Required Policy to the User?
- What Should I Do If Insufficient Permissions Are Prompted When I Am Trying to Perform an Operation as an IAM User?
- Can I Delete DataArts Studio Workspaces?
- Can I Transfer a Purchased or Trial Instance to Another Account?
- Does DataArts Studio Support Version Upgrade?
- Does DataArts Studio Support Version Downgrade?
- How Do I View the DataArts Studio Instance Version?
- Why Can't I Select a Specified IAM Project When Purchasing a DataArts Studio Instance?
- What Is the Session Timeout Period of DataArts Studio? Can the Session Timeout Period Be Modified?
- Will My Data Be Retained If My Package Expires or My Pay-per-Use Resources Are in Arrears?
- How Do I Check the Remaining Validity Period of a Package?
- Why Isn't the CDM Cluster in a DataArts Studio Instance Billed?
- Why Does the System Display a Message Indicating that the Number of Daily Executed Nodes Has Reached the Upper Limit? What Should I Do?
-
Management Center
- Which Data Sources Can DataArts Studio Connect To?
- What Are the Precautions for Creating Data Connections?
- What Should I Do If Database or Table Information Cannot Be Obtained Through a GaussDB(DWS)/Hive/HBase Data Connection?
- Why Are MRS Hive/HBase Clusters Not Displayed on the Page for Creating Data Connections?
- What Should I Do If a GaussDB(DWS) Connection Test Fails When SSL Is Enabled for the Connection?
- Can I Create Multiple Connections to the Same Data Source in a Workspace?
- Should I Select the API or Proxy Connection Type When Creating a Data Connection in Management Center?
- How Do I Migrate the Data Development Jobs and Data Connections from One Workspace to Another?
-
DataArts Migration (CDM Jobs)
- What Are the Differences Between CDM and Other Data Migration Services?
- What Are the Advantages of CDM?
- What Are the Security Protection Mechanisms of CDM?
- How Do I Reduce the Cost of Using CDM?
- Will I Be Billed If My CDM Cluster Does Not Use the Data Transmission Function?
- Why Am I Billed Pay per Use When I Have Purchased a Yearly/Monthly CDM Incremental Package?
- How Do I Check the Remaining Validity Period of a Package?
- Can CDM Be Shared by Different Tenants?
- Can I Upgrade a CDM Cluster?
- How Is the Migration Performance of CDM?
- What Is the Number of Concurrent Jobs for Different CDM Cluster Versions?
- Does CDM Support Incremental Data Migration?
- Does CDM Support Field Conversion?
- What Component Versions Are Recommended for Migrating Hadoop Data Sources?
- What Data Formats Are Supported When the Data Source Is Hive?
- Can I Synchronize Jobs to Other Clusters?
- Can I Create Jobs in Batches?
- Can I Schedule Jobs in Batches?
- How Do I Back Up CDM Jobs?
- What Should I Do If Only Some Nodes in a HANA Cluster Can Communicate with the CDM Cluster?
- How Do I Use Java to Invoke CDM RESTful APIs to Create Data Migration Jobs?
- How Do I Connect the On-Premises Intranet or Third-Party Private Network to CDM?
- Does CDM Support Parameters or Variables?
- How Do I Set the Number of Concurrent Extractors for a CDM Migration Job?
- Does CDM Support Real-Time Migration of Dynamic Data?
- Can I Stop CDM Clusters?
- How Do I Obtain the Current Time Using an Expression?
- What Should I Do If the Log Prompts that the Date Format Fails to Be Parsed?
- What Can I Do If the Map Field Tab Page Cannot Display All Columns?
- How Do I Select Distribution Columns When Using CDM to Migrate Data to GaussDB(DWS)?
- What Do I Do If the Error Message "value too long for type character varying" Is Displayed When I Migrate Data to DWS?
- What Can I Do If Error Message "Unable to execute the SQL statement" Is Displayed When I Import Data from OBS to SQL Server?
- What Should I Do If the Cluster List Is Empty, I Have No Access Permission, or My Operation Is Denied?
- Why Is Error ORA-01555 Reported During Migration from Oracle to DWS?
- What Should I Do If the MongoDB Connection Migration Fails?
- What Should I Do If a Hive Migration Job Is Suspended for a Long Period of Time?
- What Should I Do If an Error Is Reported Because the Field Type Mapping Does Not Match During Data Migration Using CDM?
- What Should I Do If a JDBC Connection Timeout Error Is Reported During MySQL Migration?
- What Should I Do If a CDM Migration Job Fails After a Link from Hive to GaussDB(DWS) Is Created?
- How Do I Use CDM to Export MySQL Data to an SQL File and Upload the File to an OBS Bucket?
- What Should I Do If CDM Fails to Migrate Data from OBS to DLI?
- What Should I Do If a CDM Connector Reports the Error "Configuration Item [linkConfig.iamAuth] Does Not Exist"?
- What Should I Do If Error "Configuration Item [linkConfig.createBackendLinks] Does Not Exist" or "Configuration Item [throttlingConfig.concurrentSubJobs] Does Not Exist" Is Reported?
- What Should I Do If Message "CORE_0031:Connect time out. (Cdm.0523)" Is Displayed During the Creation of an MRS Hive Link?
- What Should I Do If Message "CDM Does Not Support Auto Creation of an Empty Table with No Column" Is Displayed When I Enable Auto Table Creation?
- What Should I Do If I Cannot Obtain the Schema Name When Creating an Oracle Relational Database Migration Job?
- What Should I Do If invalid input syntax for integer: "true" Is Displayed During MySQL Database Migration?
-
DataArts Migration (Real-Time Jobs)
- Overview
- How Do I Troubleshoot a Network Disconnection Between the Data Source and Resource Group?
- Which Ports Must Be Allowed by the Data Source Security Group So That DataArts Migration Can Access the Data Source?
- How Do I Configure a Spark Periodic Task for Hudi Compaction?
- What Should I Do If an Error Is Reported During DDL Synchronization of New Columns in a Real-Time MySQL-to-DWS Synchronization Job?
- Why Does DWS Filter the Null Value of the Primary Key During Real-Time Synchronization from MySQL to DWS?
- What Should I Do If a Job for Synchronizing Data from Kafka to DLI in Real Time Fails and "Array element access needs an index starting at 1 but was 0" Is Displayed?
- How Do I Grant the Log Archiving, Query, and Parsing Permissions of an Oracle Data Source?
- How Do I Manually Delete Replication Slots from a PostgreSQL Data Source?
-
DataArts Architecture
- What Is the Relationship Between Lookup Tables and Data Standards?
- What Are the Differences Between ER Modeling and Dimensional Modeling?
- What Data Modeling Methods Are Supported by DataArts Architecture?
- How Can I Use Standardized Data?
- Does DataArts Architecture Support Database Reversing?
- What Are the Differences Between the Metrics in DataArts Architecture and DataArts Quality?
- Why Doesn't the Table in the Database Change After I Have Modified Fields in an ER or Dimensional Model?
- Can I Configure Lifecycle Management for Tables?
- How Should I Select a Subject When a Public Dimension (Date, Region, Supplier, or Product) Is Shared by Multiple Subject Areas?
- How Can I Create an Atomic Metric Using a Dimension Table Since Only a Fact Table Can Be Selected for Creating an Atomic Metric?
-
DataArts Factory
- How Many Jobs Can Be Created in DataArts Factory? Is There a Limit on the Number of Nodes in a Job?
- Does DataArts Studio Support Custom Python Scripts?
- How Can I Quickly Rectify a Deleted CDM Cluster Associated with a Job?
- Why Is There a Large Difference Between Job Execution Time and Start Time of a Job?
- Will Subsequent Jobs Be Affected If a Job Fails to Be Executed During Scheduling of Dependent Jobs? What Should I Do?
- What Should I Pay Attention to When Using DataArts Studio to Schedule Big Data Services?
- What Are the Differences and Relationships Between Environment Variables, Job Parameters, and Script Parameters?
- What Should I Do If a Job Log Cannot Be Opened and Error 404 Is Reported?
- What Should I Do If the Agency List Fails to Be Obtained During Agency Configuration?
- Why Can't I Select Specified Peripheral Resources When Creating a Data Connection in DataArts Factory?
- Why Can't I Receive Job Failure Alarm Notifications After I Have Configured SMN Notifications?
- Why Is There No Job Running Scheduling Log on the Monitor Instance Page After Periodic Scheduling Is Configured for a Job?
- Why Isn't the Error Cause Displayed on the Console When a Hive SQL or Spark SQL Scripts Fails?
- What Should I Do If the Token Is Invalid During the Execution of a Data Development Node?
- How Do I View Run Logs After a Job Is Tested?
- Why Does a Job Scheduled by Month Start Running Before the Job Scheduled by Day Is Complete?
- What Should I Do If Invalid Authentication Is Reported When I Run a DLI Script?
- Why Cannot I Select a Desired CDM Cluster in Proxy Mode When Creating a Data Connection?
- Why Is There No Job Running Scheduling Record After Daily Scheduling Is Configured for the Job?
- What Do I Do If No Content Is Displayed in Job Logs?
- Why Do I Fail to Establish a Dependency Between Two Jobs?
- What Should I Do If an Error Is Reported During Job Scheduling in DataArts Studio, Indicating that the Job Has Not Been Submitted?
- What Should I Do If an Error Is Reported During Job Scheduling in DataArts Studio, Indicating that the Script Associated with Node XXX in the Job Has Not Been Submitted?
- What Should I Do If a Job Fails to Be Executed After Being Submitted for Scheduling and an Error Displayed: Depend Job [XXX] Is Not Running Or Pause?
- How Do I Create Databases and Data Tables? Do Databases Correspond to Data Connections?
- Why Is No Result Displayed After a Hive Task Is Executed?
- Why Is the Last Instance Status On the Monitor Instance Page Either Successful or Failed?
- How Do I Configure Notifications for All Jobs?
- What Is the Maximum Number of Nodes That Can Be Executed Simultaneously?
- Can I Change the Time Zone of a DataArts Studio Instance?
- How Do I Synchronize the Changed Names of CDM Jobs to DataArts Factory?
- Why Does the Execution of an RDS SQL Statement Fail and an Error Is Reported Indicating That hll Does Not Exist?
- What Should I Do If Error Message "The account has been locked" Is Displayed When I Am Creating a DWS Data Connection?
- What Should I Do If a Job Instance Is Canceled and Message "The node start execute failed, so the current node status is set to cancel." Is Displayed?
- What Should I Do If Error Message "Workspace does not exists" Is Displayed When I Call a DataArts Factory API?
- Why Don't the URL Parameters for Calling an API Take Effect in the Test Environment When the API Can Be Called Properly Using Postman?
- What Should I Do If Error Message "Agent need to be updated?" Is Displayed When I Run a Python Script?
- Why Is an Execution Failure Displayed for a Node in the Log When the Node Status Is Successful?
- What Should I Do If an Unknown Exception Occurs When I Call a DataArts Factory API?
- Why Is an Error Message Indicating an Invalid Resource Name Is Displayed When I Call a Resource Creation API?
- Why Does a PatchData Task Fail When All PatchData Job Instances Are Successful?
- Why Is a Table Unavailable When an Error Message Indicating that the Table Already Exists Is Displayed During Table Creation from a DWS Data Connection?
- What Should I Do If Error Message "The throttling threshold has been reached: policy user over ratelimit,limit:60,time:1 minute." Is Displayed When I Schedule an MRS Spark Job?
- What Should I Do If Error Message "UnicodeEncodeError: 'ascii' codec can't encode characters in position 63-64: ordinal not in range(128)" Is Displayed When I Run a Python Script?
- What Should I Do If an Error Message Is Displayed When I View Logs?
- What Should I Do If a Shell/Python Node Fails and Error "session is down" Is Reported?
- What Should I Do If a Parameter Value in a Request Header Contains More Than 512 Characters?
- What Should I Do If a Message Is Displayed Indicating that the ID Does Not Exist During the Execution of a DWS SQL Script?
- How Do I Check Which Jobs Invoke a CDM Job?
- What Should I Do If Error Message "The request parameter invalid" Is Displayed When I Use Python to Call the API for Executing Scripts?
- What Should I Do If the Default Queue of a New DLI SQL Script in DataArts Factory Has Been Deleted?
- Does the Event-based Scheduling Type in DataArts Factory Support Offline Kafka?
-
DataArts Quality
- What Are the Differences Between Quality Jobs and Comparison Jobs?
- How Can I Confirm that a Quality Job or Comparison Job Is Blocked?
- How Do I Manually Restart a Blocked Quality Job or Comparison Job?
- How Do I View Jobs Associated with a Quality Rule Template?
- What Should I Do If the System Displays a Message Indicating that I Do Not Have the MRS Permission to Perform a Quality Job?
- DataArts Catalog
-
DataArts Security
- Why Isn't Data Masked Based on a Specified Rule After a Data Masking Task Is Executed?
- What Should I Do If a Message Is Displayed Indicating that Necessary Request Parameters Are Missing When I Approve a GaussDB(DWS) Permission Application?
- What Should I Do If Error Message "FATAL: Invalid username/password,login denied" Is Displayed During the GaussDB(DWS) Connectivity Check When Fine-grained Authentication Is Enabled?
- What Should I Do If Error Message "Failed to obtain the database" Is Displayed When I Select a Database in DataArts Factory After Fine-grained Authentication Is Enabled?
- Why Does the System Display a Message Indicating Insufficient Permissions During Permission Synchronization to DLI?
-
DataArts DataService
- What Languages Do DataArts DataService SDKs Support?
- What Can I Do If the System Displays a Message Indicating that the Proxy Fails to Be Invoked During API Creation?
- What Should I Do If the Background Reports an Error When I Access the Test App Through the Data Service API and Set Related Parameters?
- How Many Times Can a Subdomain Name Be Accessed Using APIs Every Day?
- Can Operators Be Transferred When API Parameters Are Transferred?
- What Should I Do If No More APIs Can Be Created When the API Quota in the Workspace Is Used Up?
- How Can I Access APIs of DataArts DataService Exclusive from the Internet?
- How Can I Access APIs of DataArts DataService Exclusive Using Domain Names?
- What Should I Do If It Takes a Long Time to Obtain the Total Number of Data Records of a Table Through an API If the Table Contains a Large Amount of Data?
-
Consultation and Billing
-
More Documents
-
User Guide (Kuala Lumpur Region)
- Service Overview
- Preparations
-
User Guide
- Preparations Before Using DataArts Studio
- Management Center
-
DataArts Migration
- Overview
- Constraints
- Supported Data Sources
- Managing Clusters
-
Managing Links
- Creating Links
- Managing Drivers
- Managing Agents
- Managing Cluster Configurations
- Link to a Common Relational Database
- Link to a Database Shard
- Link to MyCAT
- Link to a Dameng Database
- Link to a MySQL Database
- Link to an Oracle Database
- Link to DLI
- Link to Hive
- Link to HBase
- Link to HDFS
- Link to OBS
- Link to an FTP or SFTP Server
- Link to Redis/DCS
- Link to DDS
- Link to CloudTable
- Link to CloudTable OpenTSDB
- Link to MongoDB
- Link to Cassandra
- Link to Kafka
- Link to DMS Kafka
- Link to Elasticsearch/CSS
- Managing Jobs
- Auditing
-
Tutorials
- Creating an MRS Hive Link
- Creating a MySQL Link
- Migrating Data from MySQL to MRS Hive
- Migrating Data from MySQL to OBS
- Migrating Data from MySQL to DWS
- Migrating an Entire MySQL Database to RDS
- Migrating Data from Oracle to CSS
- Migrating Data from Oracle to DWS
- Migrating Data from OBS to CSS
- Migrating Data from OBS to DLI
- Migrating Data from MRS HDFS to OBS
- Migrating the Entire Elasticsearch Database to CSS
- Advanced Operations
-
DataArts Factory
- Overview
- Data Management
- Script Development
- Job Development
- Solution
- Execution History
- O&M and Scheduling
- Configuration and Management
-
Node Reference
- Node Overview
- CDM Job
- Rest Client
- Import GES
- MRS Kafka
- Kafka Client
- ROMA FDI Job
- DLI Flink Job
- DLI SQL
- DLI Spark
- DWS SQL
- MRS Spark SQL
- MRS Hive SQL
- MRS Presto SQL
- MRS Spark
- MRS Spark Python
- MRS Flink Job
- MRS MapReduce
- CSS
- Shell
- RDS SQL
- ETL Job
- Python
- Create OBS
- Delete OBS
- OBS Manager
- Open/Close Resource
- Subjob
- For Each
- SMN
- Dummy
- EL Expression Reference
- Usage Guidance
-
FAQs
- Consultation
-
Management Center
- What Are the Precautions for Creating Data Connections?
- Why Do DWS/Hive/HBase Data Connections Fail to Obtain the Information About Database or Tables?
- Why Are MRS Hive/HBase Clusters Not Displayed on the Page for Creating Data Connections?
- What Should I Do If the Connection Test Fails When I Enable the SSL Connection During the Creation of a DWS Data Connection?
- Can I Create Multiple Data Connections in a Workspace in Proxy Mode?
- Should I Choose a Direct or a Proxy Connection When Creating a DWS Connection?
- How Do I Migrate the Data Development Jobs and Data Connections from One Workspace to Another?
- Can I Delete Workspaces?
-
DataArts Migration
- General
-
Functions
- Does CDM Support Incremental Data Migration?
- Does CDM Support Field Conversion?
- What Component Versions Are Recommended for Migrating Hadoop Data Sources?
- What Data Formats Are Supported When the Data Source Is Hive?
- Can I Synchronize Jobs to Other Clusters?
- Can I Create Jobs in Batches?
- Can I Schedule Jobs in Batches?
- How Do I Back Up CDM Jobs?
- How Do I Configure the Connection If Only Some Nodes in the HANA Cluster Can Communicate with the CDM Cluster?
- How Do I Use Java to Invoke CDM RESTful APIs to Create Data Migration Jobs?
- How Do I Connect the On-Premises Intranet or Third-Party Private Network to CDM?
- How Do I Set the Number of Concurrent Extractors for a CDM Migration Job?
- Does CDM Support Real-Time Migration of Dynamic Data?
-
Troubleshooting
- What Can I Do If Error Message "Unable to execute the SQL statement" Is Displayed When I Import Data from OBS to SQL Server?
- Why Is Error ORA-01555 Reported During Migration from Oracle to DWS?
- What Should I Do If the MongoDB Connection Migration Fails?
- What Should I Do If a Hive Migration Job Is Suspended for a Long Period of Time?
- What Should I Do If an Error Is Reported Because the Field Type Mapping Does Not Match During Data Migration Using CDM?
- What Should I Do If a JDBC Connection Timeout Error Is Reported During MySQL Migration?
- What Should I Do If a CDM Migration Job Fails After a Link from Hive to DWS Is Created?
- How Do I Use CDM to Export MySQL Data to an SQL File and Upload the File to an OBS Bucket?
- What Should I Do If CDM Fails to Migrate Data from OBS to DLI?
- What Should I Do If a CDM Connector Reports the Error "Configuration Item [linkConfig.iamAuth] Does Not Exist"?
- What Should I Do If Error Message "Configuration Item [linkConfig.createBackendLinks] Does Not Exist" Is Displayed During Data Link Creation or Error Message "Configuration Item [throttlingConfig.concurrentSubJobs] Does Not Exist" Is Displayed During Job Creation?
- What Should I Do If Message "CORE_0031:Connect time out. (Cdm.0523)" Is Displayed During the Creation of an MRS Hive Link?
- What Should I Do If Message "CDM Does Not Support Auto Creation of an Empty Table with No Column" Is Displayed When I Enable Auto Table Creation?
- What Should I Do If I Cannot Obtain the Schema Name When Creating an Oracle Relational Database Migration Job?
-
DataArts Factory
- How Many Jobs Can Be Created in DataArts Factory? Is There a Limit on the Number of Nodes in a Job?
- Why Is There a Large Difference Between Job Execution Time and Start Time of a Job?
- Will Subsequent Jobs Be Affected If a Job Fails to Be Executed During Scheduling of Dependent Jobs? What Should I Do?
- What Should I Pay Attention to When Using DataArts Studio to Schedule Big Data Services?
- What Are the Differences and Connections Among Environment Variables, Job Parameters, and Script Parameters?
- What Do I Do If Node Error Logs Cannot Be Viewed When a Job Fails?
- What Should I Do If the Agency List Fails to Be Obtained During Agency Configuration?
- How Do I Locate Job Scheduling Nodes with a Large Number?
- Why Cannot Specified Peripheral Resources Be Selected When a Data Connection Is Created in Data Development?
- Why Is There No Job Running Scheduling Log on the Monitor Instance Page After Periodic Scheduling Is Configured for a Job?
- Why Does the GUI Display Only the Failure Result but Not the Specific Error Cause After Hive SQL and Spark SQL Scripts Fail to Be Executed?
- What Do I Do If the Token Is Invalid During the Running of a Data Development Node?
- How Do I View Run Logs After a Job Is Tested?
- Why Does a Job Scheduled by Month Start Running Before the Job Scheduled by Day Is Complete?
- What Should I Do If Invalid Authentication Is Reported When I Run a DLI Script?
- Why Cannot I Select the Desired CDM Cluster in Proxy Mode When Creating a Data Connection?
- Why Is There No Job Running Scheduling Record After Daily Scheduling Is Configured for the Job?
- What Do I Do If No Content Is Displayed in Job Logs?
- Why Do I Fail to Establish a Dependency Between Two Jobs?
- What Should I Do If an Error Is Displayed During DataArts Studio Scheduling: The Job Does Not Have a Submitted Version?
- What Do I Do If an Error Is Displayed During DataArts Studio Scheduling: The Script Associated with Node XXX in the Job Is Not Submitted?
- What Should I Do If a Job Fails to Be Executed After Being Submitted for Scheduling and an Error Displayed: Depend Job [XXX] Is Not Running Or Pause?
- How Do I Create a Database And Data Table? Is the database a data connection?
- Why Is No Result Displayed After an HIVE Task Is Executed?
- Why Does the Last Instance Status On the Monitor Instance page Only Display Succeeded or Failed?
- How Do I Create a Notification for All Jobs?
- How Many Nodes Can Be Executed Concurrently in Each DataArts Studio Version?
- What Is the Priority of the Startup User, Execution User, Workspace Agency, and Job Agency?
-
API Reference (Kuala Lumpur Region)
- Before You Start
- API Overview
- Calling APIs
- Application Cases
-
DataArts Migration APIs
- Cluster Management
- Job Management
- Link Management
-
Public Data Structures
-
Link Parameter Description
- Link to a Relational Database
- Link to OBS
- Link to HDFS
- Link to HBase
- Link to CloudTable
- Link to Hive
- Link to an FTP or SFTP Server
- Link to MongoDB
- Link to Redis/DCS (to Be Brought Offline)
- Link to Kafka
- Link to Elasticsearch/Cloud Search Service
- Link to DLI
- Link to CloudTable OpenTSDB
- Link to Amazon S3
- Link to DMS Kafka
-
Source Job Parameters
- From a Relational Database
- From Object Storage
- From HDFS
- From Hive
- From HBase/CloudTable
- From FTP/SFTP/NAS (to Be Brought Offline)/SFS (to Be Brought Offline)
- From HTTP/HTTPS
- From MongoDB/DDS
- From Redis/DCS (to Be Brought Offline)
- From DIS
- From Kafka
- From Elasticsearch/Cloud Search Service
- From OpenTSDB
- Destination Job Parameters
- Job Parameter Description
-
Link Parameter Description
-
DataArts Factory APIs
- Connection Management APIs
- Script Development APIs
- Resource Management APIs
- Job Development APIs
- Data Structure
-
APIs to Be Taken Offline
- Creating a Job
- Editing a Job
- Viewing a Job List
- Viewing Job Details
- Exporting a Job
- Batch Exporting Jobs
- Importing a Job
- Executing a Job Immediately
- Starting a Job
- Viewing Running Status of a Real-Time Job
- Viewing a Job Instance List
- Viewing Job Instance Details
- Querying a System Task
- Creating a Script
- Modifying a Script
- Querying a Script
- Querying a Script List
- Querying the Execution Result of a Script Instance
- Creating a Resource
- Modifying a Resource
- Querying a Resource
- Querying a Resource List
- Importing a Connection
- Appendix
-
User Guide (Kuala Lumpur Region)
- General Reference
Show all
Function Overview
-
DataArts Migration
-
Cloud DataArts Migration enables batch data migration between 30+ homogeneous and heterogeneous data sources. You can use it to ingest data from both on-premises and cloud-based data sources, including file systems, relational databases, data warehouses, NoSQL databases, big data services, and object storage.
DataArts Migration uses a distributed compute framework and concurrent processing techniques to help you migrate data in batches without any downtime and rapidly build desired data structures.
Available in all regions
-
Cluster Management
-
The following cluster management capabilities are available:
- Creating a cluster
- Binding or unbinding an EIP
- Modifying cluster configurations
- Viewing cluster configurations, logs, and monitoring data
- Configuring monitoring metrics
-
-
Link Management
-
The following link management capabilities are available:
- Managing links to DLI, MRS Hive, Spark SQL, DWS, MySQL, and hosts
- Supporting various link modes, such as agent links, direct links, and MRS API links
-
-
Job Management
-
CDM can migrate tables or files between homogeneous and heterogeneous data sources. For details about data sources that support table/file migration, see Supported Data Sources.
CDM is applicable to data migration to the cloud, data exchange on the cloud, and data migration to on-premises service systems.
-
-
-
DataArts Factory
-
The DataArts Factory module of DataArts Studio is a one-stop agile big DataArts Factory platform. It provides a visualized graphical development interface, rich DataArts Factory types (script development and job development), fully-hosted job scheduling and O&M monitoring capabilities, built-in industry data processing pipeline, one-click development, full-process visualization, and online collaborative development by multiple people, as well as supports management of multiple big data cloud services, greatly lowering the threshold for using big data and helping you quickly build big data processing centers.
Available in all regions
-
Data Management
-
The data management function helps you quickly establish data models and provides you with data entities for script and job development. With data management, you can:
- Manage multiple types of data warehouses, such as DWS and MRS Hive.
- Use the GUI and DDL to manage database tables.
-
-
Script Development
-
The following script development capabilities are available:
- An online script editor that allows more than one operator to collaboratively develop and debug SQL and Shell scripts online
- Variables and functions
- Script version management
-
-
Job Development
-
The following job development capabilities are available:
- A graphical designer that allows you to quickly build a data processing workflow by drag-and-drop
- Presetting multiple job types, such as data integration, computing and analysis, resource management, and data monitoring, and completing complex data analysis and processing based on dependencies between jobs
- Various scheduling modes
- Importing and exporting jobs
- Monitoring job status and sending job result notifications
- Managing job versions
-
-
O&M and Scheduling
-
You can view the statistics of job instances in charts. Currently, you can view four types of statistics:
- Today's Job Instance Scheduling
- Latest 7 Days' Job Instance Scheduling
- Latest 30 Days' Top 10 Ranking in Job Instance Execution Duration: View the detailed running records of the job instance with a long execution time.
- Latest 30 Days' Top 10 Ranking in Job Instance Running Failed: View the detailed running records of the job instance that is running abnormally.
-
-
Configuration and Management
-
The following configuration and management capabilities are available:
- Managing a host connection
- Managing resources
- Configuring environment variables
- Managing job labels
- Configuring agencies
- Backing up and restoring assets
-
-
-
Management Center
-
DataArts Studio Management Center provides instance management, workspace management, data connection management, and resource migration functions.
Available in all regions
-
Instance Management
-
You can create an instance and configure the enterprise project, VPC, subnet, and security group on which the instance depends.
-
-
Workspace Management
-
A workspace enables its administrator to manage user (member) permissions, resources, and underlying compute engines of DataArts Studio.
The workspace is a basic unit for member management as well as role assignment. Each team has an independent workspace.
After an admin adds an account to a workspace and assigns the required permissions, the account user can access Management Center, DataArts Catalog, DataArts Quality, DataArts Architecture, DataArts DataService, DataArts Factory, and Data Integration modules.
-
-
Data Connection Management
-
You can create data connections by configuring data sources. Metadata management allows you to create, edit, and delete data connections, as well as test their connectivity. Data connections apply to collection tasks, business metrics, and data quality. If there are any changes made to the saved information, update the related data connections.
-
-
Resource Migration
-
To migrate rules created for an environment to another, you can enable resource migration of DataArts Studio to import and export resources. Resources that can be migrated include data services, metadata categories, metadata tags, metadata collection tasks, and data connections.
-
-
-
DataArts Architecture
-
DataArts Studio DataArts Architecture incorporates data governance methods. You can use it to visualize data governance operations, connect data from different layers, formulate data standards, and generate DataArts Catalog. You can standardize your data through ER modeling and dimensional modeling. DataArts Architecture is a good option for unified construction of metric platforms. With DataArts Architecture, you can build standard metric systems to eliminate data ambiguity and facilitate communications between different departments. In addition to unifying computing logic, you can use it to query data and explore data value by subject.
Available in AP-Singapore, AP-Bangkok, AP-Jakarta, and AF-Johannesburg
-
Information Architecture
-
An information architecture is a set of component specifications that describe various types of information required for business operations and management decision-making as well as the relationships of business entities. On the Information Architecture page, you can view and manage business tables, dimension tables, fact tables, and summary tables.
-
-
Process Design
-
Business Process Architecture (BPA) is developed based on value streams, and is used to guide and standardize the management of BT&IT requirements and ensure the efficiency of business requirement handling, analysis, and delivery. BPA prioritizes high-value requirements, which maximizes the business value, assists in business operations, and facilitates goal achievement.
-
-
Subject Design
-
A subject is a hierarchical architecture that classifies and defines data to help clarify DataArts Catalog and specify relationships between subject areas and business objects.
You can design subjects in either of the following ways:
- Creating a subject
Manually create a subject.
- Importing a subject
If the subject information is complex, you are advised to import subjects in batches.
- You can download the provided subject design template, fill in the content, and upload the file to import the subjects in batches.
- You can export the subjects created in DataArts Architecture of a DataArts Studio instance to an Excel file. Then, import the Excel file.
After creating a subject, you can search for, edit, or delete it.
-
-
Lookup Table Management
-
A lookup table is also called a data dictionary table. It consists of enumerable data names and codes and stores the relationships between them. A lookup table provides the following functions:
- Standardizes business data and supplements mapping fields during data cleansing.
- Monitors the value range of business data during data quality monitoring.
- Enumerates dimensions during dimensional modeling.
-
-
Data Standards
-
Data standards describe data meanings and business rules that are stipulated and commonly recognized by enterprises and must be complied with by the enterprises.
A data standard, also called a data element, is the smallest unit of data used. It cannot be further divided. A data standard is a data unit whose definition, identifiers, representations, and allowed values are specified by a group of properties. You can associate data standards with databases of a wide range of businesses. The identifier, data type, expression format, and value range are the basis of data exchange. They are used to describe field metadata of a table and standardize data information stored in a field.
This section describes how to create a data standard. A created data standard can be associated with fields in a business table created during ER modeling, ensuring that fields in the business table comply with the specified data standards.
-
-
ER Modeling
-
ER modeling supports logical model design, physical model design, reverse database, quality rule association, table import and export, and table viewing.
-
-
Dimensional Modeling
-
A dimension is the perspective to observe and analyze business data and assist in data aggregation, drilling, slicing, and analysis, and used as a GROUP BY condition in SQL statements. Most dimensions have hierarchical structures, such as geographic dimensions (including countries, regions, provinces/states, and cities) and time dimensions (including annually, quarterly, and monthly dimensions). Creating a dimension is a way to standardize the existence and uniqueness of business entities (also called primary data) from the top down.
-
-
Business Metrics
-
After data survey and requirement analysis, you must implement metrics. A metric is a statistical value that measures the overall characteristic of a target and reflects the business situation in a business activity of an enterprise. A metric consists of its name and value. The metric name and its definition reflect the quality and quantity of the metric. The metric value reflects the quantifiable values of the specified time, location, and condition of the metric. Business metrics are used to guide technical metrics, and technical metrics are used to implement business metrics.
-
-
Technical Metrics
-
You can create atomic metrics, derivative metrics, compound metrics, and time filters.
-
-
Review Center
-
After the modeling and data processing tasks generated in the development environment are submitted, they are stored in the review center. After the tasks are approved on the Review Center page, these tasks are available in the production environment.
-
-
Configuration Center
-
Configuration Center supports standard template management, function configuration, field type management, DDL template management, and metric encoding rules.
-
-
-
DataArts Quality
-
DataArts Quality can monitor your metrics and data quality, and screen out unqualified data in a timely manner.
Available in AP-Singapore, AP-Bangkok, AP-Jakarta, and AF-Johannesburg
-
Monitoring Business Metrics
-
You can use DQC to monitor the quality of data in your databases. You can create metrics, rules, or scenarios that meet your requirements and schedule them in real time or recursively.
-
-
Monitoring Data Quality
-
DQC is a type of quality management tool used to manage the quality of data in databases. You can filter out unqualified data in a single column or across columns, rows, and tables from the following perspectives: integrity, validity, timeliness, consistency, accuracy, and uniqueness. It can also be used for data standardization, automatic generation of standardization rules based on data standards, and periodic monitoring.
-
-
Viewing Quality Reports
-
A five-point scale is used for quality scoring based on table-associated rules. The scores in different dimensions, such as tables, business objects, and subject areas, are calculated based on the weighted average values of rule scores in different dimensions.
You can query the quality scores of subject area groups, subject areas, business objects, tables, and table-associated rules.
-
-
-
DataArts Catalog
-
DataArts Studio provides enterprise-class metadata management to clarify information assets. It also supports data drilling and source tracing. It uses a data map to display a data lineage and panorama of DataArts Catalog for intelligent data search, operations, and monitoring.
Available in AP-Singapore, AP-Bangkok, AP-Jakarta, and AF-Johannesburg
-
Data Maps
-
Data maps facilitate data search, analysis, development, mining, and operations. With data maps, you can search for data quickly and perform lineage and impact analysis with ease.
- Search: Before data analysis, a data map can be used to search for keywords to narrow down the scope of data to be analyzed.
- Details: A data map can be used to query table details by table names, letting you know how to use a table.
- Lineage: Through lineage analysis, a data map displays you how a table is generated and where it is applied, and the logic used for processing table fields.
-
-
Data Permissions
-
To ensure data security and controllability, you need to apply for permissions before using data tables. The Data Permissions module facilitates permission control, provides visualized application and approval processes, and supports permission audit and management. Data is secure and data permission control is convenient.
The Data Permissions module consists of Data Catalog Permissions, Data Table Permissions, and Review Center. The following functions are provided:
- Self-service permission application: You can select a data table and quickly apply for the needed permissions online.
- Permission audit: Administrators can quickly and easily view the personnel with the corresponding database table permissions and perform audit management.
- Permission revoking and returning: Administrators can revoke user permissions in a timely manner. Users can also proactively return unnecessary permissions.
- Permission approval and management: A visualized and process-based management and authorization mechanism facilitates post-event tracing.
-
-
Metadata Collection
-
Metadata is data about data. Metadata streamlines source data, data warehouses, and data applications, and records the entire process from data generation to data consumption. Metadata mainly refers to model definitions in the data warehouse and mappings between layers. It also describes the monitoring data status of the data warehouse and running status of ETL tasks. In the data warehouse system, metadata helps data warehouse administrators and developers easily locate the data they are looking for, improving the efficiency of data management and development.
Metadata is classified into technical metadata and business metadata by function.
- Technical metadata is data that stores technical details of a data warehouse system and is used to develop and manage data warehouses.
- Business metadata describes data in a data warehouse from the business perspective. It provides a semantic layer between users and actual systems, enabling business personnel who do not understand computer technologies to understand data in the data warehouse.
The metadata management module is the cornerstone of data lake governance. It allows you to create collection tasks by custom collection policies to collect technical metadata from data sources, customize business metamodels to batch import business metadata, associate business metadata with technical metadata, and manage and apply linkages throughout the entire link.
-
-
-
DataArts DataService
-
DataService aims to build a unified data service bus for enterprises to centrally manage internal and external API services. You can use DataService to generate APIs and register the APIs with DataService for unified management and publication.
DataService adjusts and controls API access requests based on throttling policies to provide multi-dimensional protection for backend services. API throttling allows you to limit the number of API calls by user, application, or time period. You can select a policy based on your service requirements.
DataService uses a serverless architecture. You only need to focus on the API query logic and do not need to worry about infrastructure such as the runtime environment. DataService supports elastic scaling of compute resources, significantly reducing O&M costs.
Available in AP-Singapore, AP-Bangkok, AP-Jakarta, and AF-Johannesburg
-
Generating APIs
-
DataService supports API generation in the wizard or script mode.
DataService can quickly generate data APIs based on data source tables in the wizard mode. You can configure a data API within several minutes without coding.
To meet personalized query requirements, DataService also supports API generation in the SQL script mode. It allows you to compile API query SQL statements and provides multi-table join, complex query conditions, and aggregation functions.
-
-
Publishing APIs
-
This section describes how to publish APIs on DataService to the service market.
DataService provides API hosting services through API Gateway, including API publishing, management, O&M, and sales. It helps you implement microservice aggregation, frontend and backend separation, and system integration in an easy, quick, cost-effective, and low-risk manner. With DataService, you can make your functions and data accessible to your partners and developers.
-
-
Reviewing APIs
-
The review center of DataService is designed to approve the applications of publishing APIs, suspending APIs, applying for authorization, renewal, and other operations.
- If an API developer wants to publish an API to the service market, remove an API from the service market, and reclaim the authorization of an application, these operations take effect only after being approved by the reviewers.
- If an API caller wants to apply for API authorization or renewal, these operations take effect only after being approved by the reviewers.
- An API developer or caller can cancel an API application to be reviewed in the review center.
-
-
Calling APIs
-
You can create an application and get authorized, and authorize an application to use an API. To call an API, perform the following operations:
- Obtain an API from the service market.
- Create an application and get authorized.
- After completing the preceding operations, you can call the API.
-
-
Operating APIs
-
You can create and delete throttling policies and bind a throttling policy to an API.
-
-
-
DataArts Security
-
DataArts Security protects data lake security and meets the data security and governance requirements of different roles, such as data development engineers, data security administrators, data security auditors, and data security operators.
Available in AP-Singapore, AP-Bangkok, AP-Jakarta, and AF-Johannesburg
-
Unified Permission Governance
-
DataArts Security provides unified management of data permissions based on MRS, DLI, and GaussDB(DWS). You can create workspace permission sets, permission sets, or roles, and use them to control access to MRS, DLI, and GaussDB(DWS) data, assign the minimum permissions to users and user groups on demand, and reduce data security risks.
-
-
Sensitive Data Governance
-
You can create sensitive data identification rules (or rule groups), or use the built-in identification rules (or rule groups), to detect, classify, and grade sensitive data.
-
-
Sensitive Data Protection
-
You can use static and dynamic data masking, and data, file, and dynamic watermarking to prevent your data from being misused, disclosed, or stolen intentionally or unintentionally. In this way, your sensitive data is secure, complete, and safe to use.
-
-
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot