Creating Data Standards

Creating a Data Standard Directory

1. On the DataArts Studio console, locate an instance and click Access. On the displayed page, locate a workspace and click DataArts Architecture.
   Figure 1 DataArts Architecture
   

2. On the DataArts Architecture page, choose Standards > Data Standards in the left navigation pane.
3. When you access the Data Standards page for the first time, the page where you can customize a standard template is displayed. Select the required options for Optional, add custom items, and click Update.

   After saving the template settings, you can modify it on the Standard Templates tab page of Configuration Center. For details, see Standard Templates. When creating a data standard, you must set the selected options in the template.

4. On the Data Standards page, select a directory and click to create a directory under the selected one. When creating a directory for the first time, you can create a directory under the root directory.
   Figure 2 Data Standards page
   
5. In the dialog box displayed, set the parameters and click OK.
   Figure 3 Create Directory dialog page
   

   Table 1 Parameters for creating directories

   Parameter	Description
   Name	Only letters, numbers, and underscores (_) are allowed.
   Select Directory	Select an existing directory, and create a subdirectory under it.

Creating a Data Standard

1. On the Data Standards page, select a directory and click Create.
2. Set the parameters based on Table 2 and click Publish.

   On the page for creating a data standard, only the selected parameters and custom parameters that have been added on the Standard Templates tab page of the Configuration Center are displayed. Table 2 lists all parameters that are available in a data standard template. For details on how to configure a data standard template, see Standard Templates.

   Table 2 Parameters for creating a data standard

   Parameter	Description
   Standard Name	The name can contain only letters, digits, brackets, spaces, hyphens (-), and underscores (_), and must start with letters. If Data Standard Allows Duplicate Names is disabled, ensure that the standard name is unique in the current workspace. To check whether Data Standard Allows Duplicate Names is enabled, go to DataArts Architecture > Configuration Center > Functions.
   Standard Code	The value can be Auto Generate or Custom. The value must be unique in the current workspace. It is used to identify a data standard record. For details, see Table 2.
   Data Type	The possible values are STRING, BIGINT, DOUBLE, TIMESTAMP, DATE, BOOLEAN, and DECIMAL. The data type varies according to the system. The system converts the data type internally. If the required data type does not exist, you can add one. See Data Types.
   Data Length	Data length You can leave this parameter blank. If it is left blank, there is no limit to the data length. You can enter a value from 1 to 10000. You can also set a range by entering the minimum and maximum values. If you set this parameter and select STRING for Data Type, a data quality job will be created for the attribute matching the data standard. If you select any other data type, no data quality job will be created.
   Allowed Value Exist	If Allowed Value Exist is enabled, you can specify one or more allowed values.
   Lookup Table	Select a created lookup table and the corresponding table fields. In this way, the lookup table fields can be associated with data standard. If no lookup table is created, create one. See Creating Lookup Tables. If Create Data Quality Jobs is selected for Model Design Process on the Function Settings tab page of Configuration Center, and the data standard of the referenced lookup tables is associated with the business tables in ER modeling, the system will automatically create quality jobs in DataArts Quality when the business tables are published, and generate quality rules based on the associated data standard and lookup tables. If the quality jobs have already been published, the system will automatically update the quality jobs and add the quality rules generated based on the data standard and lookup tables.
   Quality Rule	If Create Data Quality Jobs is selected for Model Design Process on the Function Settings tab page of Configuration Center, your data standard template includes the parameter Quality Rule, and the data standard is associated with a table, the system will automatically create a quality job in DataArts Quality after the table is published. The created quality job contains the quality rule added here. If the table associated with the data standard to be created has been published, the system will automatically update a quality job. Click . In the dialog box displayed, click Add Rule. For example, add a rule named Unique value, select the rule, click OK, enter an alarm condition expression in the Alarm Condition text box, add other rules in the same way, and click OK. An alarm condition expression consists of alarm parameters and logical operators. When a quality job is running, the system calculates the result of the alarm condition expression and determines whether to trigger the alarm based on the result of the expression. If the expression result is true, the alarm will be triggered. Otherwise, no quality alarm will be triggered. The alarm parameters of each data quality rule are listed as buttons. Figure 4 Associate Quality Rule dialog box
   Rule Designer	Select a rule designer from the drop-down list box. This owner is responsible for making quality rules. You can enter an owner name or select an existing owner.
   Rule Implementer	Select a rule implementer from the drop-down list box. This owner is responsible for implementing quality rules. You can enter an owner name or select an existing owner.
   Level	global indicates the global level. domain indicates non-global level.
   Custom Item	A custom item added on the Standard Templates tab page in Metrics > Configuration Center. You can add one or more custom items based on project requirements. For more information about adding custom items, see Standard Templates.
   Description	A description of the data standard to create. Up to 600 characters are supported.

3. Click Save.

Importing a Data Standard

1. On the DataArts Architecture page, choose Standards > Data Standards in the left navigation pane.
2. In the directory structure of data standards, select a directory and choose More > Import.
   Figure 5 Import Data Standard dialog box
   

3. In the Import Data Standard dialog box, determine whether to update the existing data. Existing data is uniquely identified by a standard code. If a standard code in the import template already exists in the current workspace, the system considers that the group of data to which the standard code in the import template belongs already exists.
4. On the Import tab page, click Data standard import template to download the template. Open the template, set the parameters in the template based on service requirements, and save the settings.

   Table 3 and Table 4 describe the parameters required for importing a data standard. Parameters whose names start with an asterisk (*) are mandatory, and other parameters are optional.

   Table 3 Parameters in the Standards sheet

   Parameter	Description
   Directory	The directory that the imported data standard belongs to.
   *Standard Name	The name of the data standard to import. It must start with letters. Only letters, digits, brackets, spaces, hyphens (-), and underscores (_) are allowed.
   *Standard Code	The value can be Auto Generate or Custom. The value must be unique in the workspace. It is used to identify a data standard record. For details, see Table 2.
   *Data Type	The possible values are STRING, BIGINT, DOUBLE, TIMESTAMP, DATE, BOOLEAN, and DECIMAL. The data type varies according to the system. The system converts the data type internally. If the required data type does not exist, you can add one. See Data Types.
   Data Length	You can enter a value ranging from 1 to 10,000. If it is left blank, there is no limit to the data length. If you enter a value and select STRING for Data Type, a data quality job will be created for the attribute matching the data standard. If you select any other data type, no data quality job will be created.
   Allowed Value	The value true indicates that there are allowed values, and the value false indicates that there are no allowed values.
   Allowed Value List	If you select true for Allowed Value, you must enter an allowed value. You can add up to 20 values. Multiple values must be separated by commas (,), for example, 1,2,3.
   Lookup Table	Set this parameter to the name of a created lookup table.
   Lookup Table Field	If Lookup Table is not left blank, you must set Lookup Table Field. In this way, the code table field can be associated with the data standard.
   Owner of Business Rules	Enter the business rule owner. You can enter the name of an owner or select an existing owner.
   Owner of Data Monitoring	Enter the data monitoring owner. You can enter the name of an owner or select an existing owner.
   Standard Level	global indicates the global level. domain indicates non-global level.
   Description	A description of the data standard to import. Up to 600 characters are supported.
   (Optional) Custom Item	If you have added one or more custom fields when customizing a data standard template, you must also fill in the corresponding fields in the import template. If no custom field is added, you do not need to fill in the fields. For details on how to customize a data standard template, see Standard Templates.

   In the Quality Rules sheet, you can configure quality rules to be added for data standards. If Create Data Quality Jobs is selected for Model Design Process on the Function Settings tab page of Configuration Center, your data standard template includes the parameter Quality Rule, and the data standard is associated with a table, the system will automatically create a quality job in DataArts Quality after the table is published. The created quality job contains the quality rule added here. If the table associated with the data standard to be created has been published, the system will automatically update a quality job.

   Table 4 Parameters in the Quality Rules sheet

   Parameter	Description
   *Code	The code of the data standard that a quality rule is added to.
   Rule Name	Enter an existing rule name. In the upper left corner of the DataArts Studio console, select DataArts Quality from the drop-down list box. Then, you can view the existing rule names on the Rule Templates page.
   Alarm Config	An alarm condition expression consists of alarm parameters and logical operators. When a quality job is running, the system calculates the result of the alarm condition expression and determines whether to trigger the alarm based on the result of the expression. If the expression result is true, the alarm will be triggered. Otherwise, no quality alarm will be triggered. In the alarm condition expression, alarm parameters are represented by variables such as ${1}, ${2}, and ${3}. The variable name indicates the alarm parameter of the specified quality rule. The variable $1 indicates the first alarm parameter, $2 indicates the second alarm parameter, and so on. In the upper left corner of the DataArts Studio console, select DataArts Quality from the drop-down list box. Access the Rule Templates page and view the alarm parameters supported by the data quality rule in the Result Description column. Example: ${1} > 100
   Expression	A regular expression must be configured when Rule Name is set to Regular Expression or Validity Verification.

5. Return to the Import Data Standard dialog box, select the data standard template file configured in the previous step, and click Upload.

   If the uploaded template file fails the verification, modify the file and upload it again.

6. In the Import Data Standard dialog box, the import result is displayed on the Last Import tab page. If the import is successful, click Close. If the import fails, you can view the failure cause, correct the template file, and upload it again.
   Figure 6 Last Import tab page
   

Managing a Data Standard

On the DataArts Architecture page, choose Standards > Data Standards in the left navigation pane. On the page displayed, you can manage data standards as required.

Figure 7 List of data standards

Exporting a Data Standard

1. On the DataArts Architecture page, choose Standards > Data Standards in the left navigation pane.
2. In the data standard directory structure, right-click a directory name and choose Export.