Function Overview

The following cluster management capabilities are available:

• Creating a cluster
• Binding or unbinding an EIP
• Modifying cluster configurations
• Viewing cluster configurations, logs, and monitoring data
• Configuring monitoring metrics

The following link management capabilities are available:

• Managing links to DLI, MRS Hive, Spark SQL, DWS, MySQL, and hosts
• Supporting various link modes, such as agent links, direct links, and MRS API links

CDM can migrate tables or files between homogeneous and heterogeneous data sources. For details about data sources that support table/file migration, see Supported Data Sources.

CDM is applicable to data migration to the cloud, data exchange on the cloud, and data migration to on-premises service systems.

The data management function helps you quickly establish data models and provides you with data entities for script and job development. With data management, you can:

• Manage multiple types of data warehouses, such as DWS and MRS Hive.
• Use the GUI and DDL to manage database tables.

The following script development capabilities are available:

• An online script editor that allows more than one operator to collaboratively develop and debug SQL and Shell scripts online
• Variables and functions
• Script version management

The following job development capabilities are available:

• A graphical designer that allows you to quickly build a data processing workflow by drag-and-drop
• Presetting multiple job types, such as data integration, computing and analysis, resource management, and data monitoring, and completing complex data analysis and processing based on dependencies between jobs
• Various scheduling modes
• Importing and exporting jobs
• Monitoring job status and sending job result notifications
• Managing job versions

You can view the statistics of job instances in charts. Currently, you can view four types of statistics:

• Today's Job Instance Scheduling
• Latest 7 Days' Job Instance Scheduling
• Latest 30 Days' Top 10 Ranking in Job Instance Execution Duration: View the detailed running records of the job instance with a long execution time.
• Latest 30 Days' Top 10 Ranking in Job Instance Running Failed: View the detailed running records of the job instance that is running abnormally.

The following configuration and management capabilities are available:

• Managing a host connection
• Managing resources
• Configuring environment variables
• Managing job labels
• Configuring agencies
• Backing up and restoring assets

You can create an instance and configure the enterprise project, VPC, subnet, and security group on which the instance depends.

A workspace enables its administrator to manage user (member) permissions, resources, and underlying compute engines of DataArts Studio.

The workspace is a basic unit for member management as well as role assignment. Each team has an independent workspace.

After an admin adds an account to a workspace and assigns the required permissions, the account user can access Management Center, DataArts Catalog, DataArts Quality, DataArts Architecture, DataArts DataService, DataArts Factory, and Data Integration modules.

You can create data connections by configuring data sources. Metadata management allows you to create, edit, and delete data connections, as well as test their connectivity. Data connections apply to collection tasks, business metrics, and data quality. If there are any changes made to the saved information, update the related data connections.

To migrate rules created for an environment to another, you can enable resource migration of DataArts Studio to import and export resources. Resources that can be migrated include data services, metadata categories, metadata tags, metadata collection tasks, and data connections.

An information architecture is a set of component specifications that describe various types of information required for business operations and management decision-making as well as the relationships of business entities. On the Information Architecture page, you can view and manage business tables, dimension tables, fact tables, and summary tables.

Business Process Architecture (BPA) is developed based on value streams, and is used to guide and standardize the management of BT&IT requirements and ensure the efficiency of business requirement handling, analysis, and delivery. BPA prioritizes high-value requirements, which maximizes the business value, assists in business operations, and facilitates goal achievement.

A subject is a hierarchical architecture that classifies and defines data to help clarify DataArts Catalog and specify relationships between subject areas and business objects.

You can design subjects in either of the following ways:

• Creating a subject

Manually create a subject.

• Importing a subject

If the subject information is complex, you are advised to import subjects in batches.

• You can download the provided subject design template, fill in the content, and upload the file to import the subjects in batches.
• You can export the subjects created in DataArts Architecture of a DataArts Studio instance to an Excel file. Then, import the Excel file.

After creating a subject, you can search for, edit, or delete it.

A lookup table is also called a data dictionary table. It consists of enumerable data names and codes and stores the relationships between them. A lookup table provides the following functions:

• Standardizes business data and supplements mapping fields during data cleansing.
• Monitors the value range of business data during data quality monitoring.
• Enumerates dimensions during dimensional modeling.

Data standards describe data meanings and business rules that are stipulated and commonly recognized by enterprises and must be complied with by the enterprises.

A data standard, also called a data element, is the smallest unit of data used. It cannot be further divided. A data standard is a data unit whose definition, identifiers, representations, and allowed values are specified by a group of properties. You can associate data standards with databases of a wide range of businesses. The identifier, data type, expression format, and value range are the basis of data exchange. They are used to describe field metadata of a table and standardize data information stored in a field.

This section describes how to create a data standard. A created data standard can be associated with fields in a business table created during ER modeling, ensuring that fields in the business table comply with the specified data standards.

ER modeling supports logical model design, physical model design, reverse database, quality rule association, table import and export, and table viewing.

A dimension is the perspective to observe and analyze business data and assist in data aggregation, drilling, slicing, and analysis, and used as a GROUP BY condition in SQL statements. Most dimensions have hierarchical structures, such as geographic dimensions (including countries, regions, provinces/states, and cities) and time dimensions (including annually, quarterly, and monthly dimensions). Creating a dimension is a way to standardize the existence and uniqueness of business entities (also called primary data) from the top down.

After data survey and requirement analysis, you must implement metrics. A metric is a statistical value that measures the overall characteristic of a target and reflects the business situation in a business activity of an enterprise. A metric consists of its name and value. The metric name and its definition reflect the quality and quantity of the metric. The metric value reflects the quantifiable values of the specified time, location, and condition of the metric. Business metrics are used to guide technical metrics, and technical metrics are used to implement business metrics.

You can create atomic metrics, derivative metrics, compound metrics, and time filters.

After the modeling and data processing tasks generated in the development environment are submitted, they are stored in the review center. After the tasks are approved on the Review Center page, these tasks are available in the production environment.

Configuration Center supports standard template management, function configuration, field type management, DDL template management, and metric encoding rules.

You can use DQC to monitor the quality of data in your databases. You can create metrics, rules, or scenarios that meet your requirements and schedule them in real time or recursively.

DQC is a type of quality management tool used to manage the quality of data in databases. You can filter out unqualified data in a single column or across columns, rows, and tables from the following perspectives: integrity, validity, timeliness, consistency, accuracy, and uniqueness. It can also be used for data standardization, automatic generation of standardization rules based on data standards, and periodic monitoring.

A five-point scale is used for quality scoring based on table-associated rules. The scores in different dimensions, such as tables, business objects, and subject areas, are calculated based on the weighted average values of rule scores in different dimensions.

You can query the quality scores of subject area groups, subject areas, business objects, tables, and table-associated rules.

Data maps facilitate data search, analysis, development, mining, and operations. With data maps, you can search for data quickly and perform lineage and impact analysis with ease.

• Search: Before data analysis, a data map can be used to search for keywords to narrow down the scope of data to be analyzed.
• Details: A data map can be used to query table details by table names, letting you know how to use a table.
• Lineage: Through lineage analysis, a data map displays you how a table is generated and where it is applied, and the logic used for processing table fields.

To ensure data security and controllability, you need to apply for permissions before using data tables. The Data Permissions module facilitates permission control, provides visualized application and approval processes, and supports permission audit and management. Data is secure and data permission control is convenient.

The Data Permissions module consists of Data Catalog Permissions, Data Table Permissions, and Review Center. The following functions are provided:

• Self-service permission application: You can select a data table and quickly apply for the needed permissions online.
• Permission audit: Administrators can quickly and easily view the personnel with the corresponding database table permissions and perform audit management.
• Permission revoking and returning: Administrators can revoke user permissions in a timely manner. Users can also proactively return unnecessary permissions.
• Permission approval and management: A visualized and process-based management and authorization mechanism facilitates post-event tracing.

Metadata is data about data. Metadata streamlines source data, data warehouses, and data applications, and records the entire process from data generation to data consumption. Metadata mainly refers to model definitions in the data warehouse and mappings between layers. It also describes the monitoring data status of the data warehouse and running status of ETL tasks. In the data warehouse system, metadata helps data warehouse administrators and developers easily locate the data they are looking for, improving the efficiency of data management and development.

Metadata is classified into technical metadata and business metadata by function.

• Technical metadata is data that stores technical details of a data warehouse system and is used to develop and manage data warehouses.
• Business metadata describes data in a data warehouse from the business perspective. It provides a semantic layer between users and actual systems, enabling business personnel who do not understand computer technologies to understand data in the data warehouse.

The metadata management module is the cornerstone of data lake governance. It allows you to create collection tasks by custom collection policies to collect technical metadata from data sources, customize business metamodels to batch import business metadata, associate business metadata with technical metadata, and manage and apply linkages throughout the entire link.

DataService supports API generation in the wizard or script mode.

DataService can quickly generate data APIs based on data source tables in the wizard mode. You can configure a data API within several minutes without coding.

To meet personalized query requirements, DataService also supports API generation in the SQL script mode. It allows you to compile API query SQL statements and provides multi-table join, complex query conditions, and aggregation functions.

This section describes how to register APIs, manage APIs generated based on data tables, and publish APIs to API Gateway.

DataService supports the registration of RESTful APIs using GET and POST methods.

This section describes how to publish APIs on DataService to the service market.

DataService provides API hosting services through API Gateway, including API publishing, management, O&M, and sales. It helps you implement microservice aggregation, frontend and backend separation, and system integration in an easy, quick, cost-effective, and low-risk manner. With DataService, you can make your functions and data accessible to your partners and developers.

DataService is the last defense for opening up or calling APIs. It provides services such as permission management, traffic control, and access control. For security reasons, the generated and registered APIs on DataService must be published to the service market before they can be provided.

The review center of DataService is designed to approve the applications of publishing APIs, suspending APIs, applying for authorization, renewal, and other operations.

• If an API developer wants to publish an API to the service market, remove an API from the service market, and reclaim the authorization of an application, these operations take effect only after being approved by the reviewers.
• If an API caller wants to apply for API authorization or renewal, these operations take effect only after being approved by the reviewers.
• An API developer or caller can cancel an API application to be reviewed in the review center.

You can create an application and get authorized, and authorize an application to use an API. To call an API, perform the following operations:

• Obtain an API from the service market.
• Create an application and get authorized.
• After completing the preceding operations, you can call the API.

You can create and delete throttling policies and bind a throttling policy to an API.