Configuring a Default Item

Scenario

If a parameter is invoked by multiple jobs, you can use this parameter as the default configuration item. In this way, you do not need to set this parameter for each job.

Configuring Periodic Scheduling

To configure the default action on the current job when the job it depends on fails, perform the following operations:

1. In the navigation pane, choose Configuration > Specifications.
2. Choose Default Configuration.
   

   Three options are available. The default value is Terminate.

   • Suspend: The current job is suspended.
   • Continue: The current job continues to be executed.
   • Cancel: The current job is canceled.

3. Click Save to save the settings. This parameter takes effect only for new jobs.

Configuring the Multi-IF Policy

To configure the policy for executing nodes with multiple IF conditions, perform the following operations:

1. In the navigation pane, choose Configuration > Specifications.
2. Choose Default Configuration.
   

   The following two options are available:

   • OR: Nodes are executed if an IF condition is met.
   • AND: Nodes are executed if all IF conditions are met.

   For details, see Configuring the Policy for Executing a Node with Multiple IF Statements.

3. Click Save to save the settings.

Configuring the Hard and Soft Lock Policy

The policy determines how you can grab the lock of a job or script. If you use a soft lock, you can grab the lock of a job or script regardless of whether you have the lock. If you use a hard lock, you can only unlock or grab the lock of a job or script for which you have the lock. Operations such as publish, execution, and scheduling are not restricted by locks.

You can configure the hard/soft policy based on your needs.

1. In the navigation pane, choose Configuration > Specifications.
2. Choose Default Configuration.
   

   The default policy is Soft Lock.

   • Soft lock: You can lock or unlock jobs or scripts, regardless of whether they are locked by others.
   • Hard Lock: You can lock jobs or scripts only after they have been unlocked by other users. The space administrator and the DAYU Administrator can lock and unlock jobs or scripts without any limitations.

3. Click Save to save the settings.

Configuring Script Variables

Variables of an SQL script can be in ${} or ${dlf.} format. You can configure either type as needed. The configured variable format applies to SQL scripts, SQL statements in jobs, single-node jobs, and environment variables.

1. In the navigation pane, choose Configuration > Configure.
2. Click Default Configuration and set Script Variable Definition.
   

   The default variable format is ${}.

   • ${}: Identify the definition of the ${} format in the script and parse the field as the variable name. For example, variable name xxx is identified from ${xxx}.
   • ${dlf.}: Identify the definition of the ${dlf.} format in the script and parse the dlf. field as the variable name. Other ${} format definitions are not recognized as variables. For example, variable name dlf.xxx is identified from ${dlf.xxx}.

3. Click Save to save the settings.

Configuring a Data Export Policy

By default, all users can download and dump the execution results of SQL scripts. If you do not want all users to have this permission, perform the following steps to configure a data export policy:

1. In the navigation pane, choose Configuration > Configure.
2. Choose Default Configuration and set Data Export Policy.
   

   The default data export policy is All User Can.

   • All User Can: All users can download and dump SQL execution results.
   • All User Cannot: No user can download or dump SQL execution results.
   • Only Workspace Manager Can: Only workspace administrators can download and dump SQL execution results.

3. Click Save.

Disabling Auto Node Name Change

On the Develop Job page, when you select a script for a node or associate a node with the function of another cloud service, the node name will be automatically changed to the script name or function name. You can disable this function.

1. In the navigation pane, choose Configuration > Specifications.
2. Choose Default Configuration. Find Disable Auto Node Name Change and select job nodes.
   

   • You can disable automatic name change for the following nodes: CDM Job, DIS Stream, DLI SQL, DWS SQL, MRS Spark SQL, MRS Hive SQL, MRS Presto SQL, MRS HetuEngine, MRS ClickHouse, MRS Impala SQL, Shell, RDS SQL, Subjob, For Each, or Python.
   • No job nodes are selected by default.
   • Names of the selected nodes will not be automatically changed when a script is selected or a function is associated with them.

3. Click Save.

Use Simple Variable Set

The simple variable set provides a series of customized variables to dynamically replace parameters during task scheduling.

1. In the navigation pane on the Data Development page, choose Configuration > Configure.
2. Choose Default Configuration and set Use Simple Variable Set.
   

   • Yes: Simple variable sets are supported. A series of customized variables provided by the simple variable set. Customized parameters are automatically replaced with specific values based on the service date, plan time, and parameter value format of task scheduling. In this way, parameters can be dynamically replaced during task scheduling.
   • No: Simple variable sets are not supported.

3. Click Save to save the settings.

Setting the Notification Policy for Jobs in Failure Ignored Status

To configure the notification type for jobs whose status is failure ignored, perform the following steps:

1. In the navigation pane on the Data Development page, choose Configuration > Configure.
2. Choose Default Configuration and set Notification Policy for Jobs in Failure Ignored Status.
3. Select a notification type for jobs whose status is failure ignored.
   

   • Jobs whose status is failure ignored are those whose Policy for Handling Subsequent Nodes If the Current Node Fails is set to Go to the next node. By default, such jobs are deemed successful by the system.
   • You can configure either of the following notification types for such jobs:

     Abnormal

     Successful (default)

4. Click Save.

Setting Retry Node upon Timeout

You can set this parameter to specify whether a node will be re-executed if it fails upon timeout.

1. In the navigation pane on the Data Development page, choose Configuration > Configure.
2. Choose Default Configuration.
3. Set Retry Node upon Timeout.
   

   • No: A node will not be re-executed if it fails upon timeout.
   • Yes: A node will be re-executed if it fails upon timeout.

4. Click Save to save the settings.

Exclude Waiting Time from Instance Timeout Duration

You can specify whether to exclude waiting time from instance timeout duration.

1. In the navigation pane, choose Configuration > Specifications.
2. Choose Default Configuration and set Exclude Waiting Time from Instance Timeout Duration.
3. Select Yes or No.
   

   Yes: The waiting time before an instance starts running is excluded from the instance timeout duration.

   No: The waiting time before an instance starts running is included in the instance timeout duration.

4. Click Save to save the settings.

Rules for Splitting MRS JAR Package Parameters

You can set the rule for splitting the string parameters (enclosed by "") in the JAR package parameters of MRS MapReduce and MRS Spark operators.

1. In the navigation pane, choose Configuration > Specifications.
2. Choose Default Configuration and set Rules for Splitting MRS JAR Package Parameters.
3. Select a rule.
   

   Split String Arguments by Space: For example, "select * from table" is split into four parameters by space: select, *, from, and table.

   Do not split string arguments: For example, "select * from table" is regarded as one parameter and is not split.

4. Click Save to save the settings.

Synchronization of Job Version by Waiting Instance

You can specify whether a waiting instance can synchronize the latest job version.

1. In the navigation pane, choose Configuration > Specifications.
2. Choose Default Configuration and set Synchronization of Job Version by Waiting Instance.
3. Select Yes or No.
   

   Yes: The waiting instance uses the latest job version.

   No: The waiting instance still uses the existing job version.

4. Click Save to save the settings.

Execution Mode for Hive SQL and Spark SQL Statements

When Hive SQL and Spark SQL statements are executed, DGCDataArts Studio can place SQL statements in OBS or in the request body.

1. In the navigation pane, choose Configuration > Configure.
2. Choose Default Configuration.
3. Set Execution Mode for Hive SQL and Spark SQL Statements.
   

   In OBS: Hive SQL and Spark SQL statements are put in OBS, and the OBS is returned to MRS.

   In the request message body: Hive SQL and Spark SQL statements are put in the request message body, and the script content is returned to MRS.

4. Click Save to save the settings.
   

   This configuration supports Hive SQL and Spark SQL scripts, and pipeline and single-task jobs.

Setting PatchData Priority

You can set the priority of a PatchData job. When system resources are insufficient, computing resources are preferentially allocated to jobs with higher priorities. A larger number indicates a higher priority. Currently, only the priorities of DLI SQL operators can be set.

1. In the left navigation pane on the DataArts Factory console, choose Configuration > Configure.
2. Choose Default Configuration and set PatchData Job Priority.
3. Set the patch data priority policy.
4. Click Save to save the settings.
   

   The mapping between the PatchData Job Priority and spark.sql.dli.job.priority of DLI is as follows:

   If PatchData Job Priority is set to 1, spark.sql.dli.job.priority of DLI is 1.

   If PatchData Job Priority is set to 2, spark.sql.dli.job.priority of DLI is 3.

   If PatchData Job Priority is set to 3, spark.sql.dli.job.priority of DLI is 5.

   If PatchData Job Priority is set to 4, spark.sql.dli.job.priority of DLI is 8.

   If PatchData Job Priority is set to 5, spark.sql.dli.job.priority of DLI is 10.

Historical Job Instance Cancellation Policy

You can set the number of retention days for waiting job instances. If the waiting time of a job instance exceeds the configured retention days, the job instance is canceled. The minimum number of retention days is 2, that is, a job instance which is not executed can be canceled after at least two days. The default number of retention days is 60.

1. In the left navigation pane on the DataArts Factory console, choose Configuration > Configure.
2. Choose Default Configuration.
3. Set the number of retention days for waiting job instances.
4. Click Save to save the settings.

Send Alarm Upon Instance Cancellation If you select Yes for this parameter and configure a cancellation notification for a job, an alarm notification will be sent when a historical job instance is canceled upon timeout. If you select No, no alarm notification will be sent.

1. In the left navigation pane on the DataArts Factory console, choose Configuration > Configure.
2. Choose Default Configuration.
3. Set Send Alarm Upon Instance Cancellation.
4. Click Save to save the settings.

Historical Job Instance Alarm Policy

You can set the number of days during which alarms can be generated for monitored job instances. The default value is seven days. Alarms cannot be sent for job instances beyond the seven-day period.

For example, if you set the value of this parameter to 2, alarms can be generated for the job instances of yesterday and today, but cannot be generated for the job instances of the day before yesterday and of an earlier time even if the triggering conditions are met.

1. In the navigation pane, choose Configuration > Configure.
2. Choose Default Configuration and locate Historical Job Instance Alarm Policy.
3. Set the number of days during which alarms can be generated for monitored job instances.
   

   The default value is 7. Set a value from 1 to 270.

   After you set this parameter, alarms are generated only for the job instances which are created after this parameter is set and not for historical instances.

4. Click Save to save the settings.

Job Alarm Notification Topic

You can set the topic used to send notifications by owner.

1. In the navigation pane, choose Configuration > Configure.
2. Choose Default Configuration.
3. Set Job Alarm Notification Topic. Click View Topic to go to the SMN console to view available topics.
   

   You can only select a topic that you created on the SMN console (to prevent conflict with any existing topic). Only the workspace administrator can configure topic.

4. Click Save to save the settings.

Default Retry Policy upon Job Operator Failure

This policy takes effect only for new job operators in the current workspace. The default policy for the operators in historical jobs is not affected. The default value is No.

1. In the left navigation pane on the DataArts Factory console, choose Configuration > Configure.
2. Choose Default Configuration.
3. Set Default Retry Policy upon Job Operator Failure.
   

   If this parameter is set to Yes, new job operators can be retried once, and the retry interval is 120 seconds by default.

4. Click Save to save the settings.

Generate Alarm Upon Job Retry Failure

If you enable this function, an alarm is generated each time a job fails to be retried.

1. In the left navigation pane on the DataArts Factory console, choose Configuration > Configure.
2. Choose Default Configuration.
3. Set Generate Alarm Upon Job Retry Failure.
   

   • If you select All jobs, Real-time jobs, or Batch jobs, an alarm is generated each time a job fails to be retried.
   • If you select Disable, an alarm is generated only when the maximum number of retries has been reached for the job.

4. Click Save to save the settings.

Automatic Script Name Transfer During Job Execution

If this function is enabled, set mapreduce.job.name="Script name" of the Hive SQL script is automatically transferred to MRS during job execution in the current workspace.

This function takes effect only if the preceding parameter value has not been set for the script. If the parameter value has been set for the script, the value set is preferentially read and transferred to MRS. This function is unavailable for MRS clusters in security mode. To enable this function for such clusters, set them to non-security mode.

1. In the navigation pane, choose Configuration > Specifications.
2. Choose Default Configuration.
3. Set Automatic Script Name Transfer During Job Execution.
   

   • Yes: The system automatically transfers the Hive SQL script name to MRS during job execution.
   • No: The system does not automatically transfer the Hive SQL script name to MRS during job execution.

4. Click Save to save the settings.

Job Dependency Rule

Jobs can be depended on by jobs in other workspaces (requires the permission to query the job list in the workspace). All default roles in the workspace have this permission. Custom roles must have the job query permission in DataArts Factory.

1. In the left navigation pane on the DataArts Factory console, choose Configuration > Configure.
2. Choose Default Configuration.
3. Configure Job Dependency Rule.
   

   • Jobs cannot be depended on by jobs in other workspaces: Jobs in this workspace cannot be depended on by jobs in other workspaces.
   • Jobs can be depended on by jobs in other workspaces: Jobs in this workspace can be depended on by jobs in other workspaces, without requiring the permissions of this workspace.
   • Jobs can be depended on by jobs in other workspaces (requires the permission to query the job list in the workspace): Jobs in this workspace can be depended on by jobs in other workspaces, requiring the permissions of this workspace. If you do not have the permissions, the system displays a message indicating that you do not have the permission to obtain the job list in workspace xxx when you configure job dependencies across workspaces.

4. Click Save to save the settings.

Script Execution History

You can set this parameter to control the permissions to view the script execution history.

1. In the left navigation pane on the DataArts Factory console, choose Configuration > Configure.
2. Choose Default Configuration.
3. Set Script Execution History.
   

   • Myself: The script execution history for only myself is displayed.
   • All users: The script execution history for all users is displayed.

4. Click Save to save the settings.

Identity for Job Tests

After configuring this parameter, you can specify the identity used to test jobs.

1. In the left navigation pane on the DataArts Factory console, choose Configuration > Configure.
2. Choose Default Configuration.
3. Set Identity for Job Tests.
   

   • Public agency or IAM account: A public agency or IAM account is used to execute jobs.
   • Personal account: The user who clicks Test is used to execute jobs.

     If no workspace agency or IAM account is available, a personal account is used for job tests.

     If you are using a federated account, you must set this parameter to Public agency or IAM account.

4. Click Save to save the settings.

SparkSqlJob/Script Default Template Configuration

You can set this parameter to determine whether any parameters can be set to overwrite the default parameters of the template.

In the MRS API connection mode, default parameters can be configured for Spark SQL scripts. For proxy connections, this function is not supported.

1. In the navigation pane, choose Configuration > Specifications.
2. Choose Default Configuration.
3. Set SparkSqlJob/Script Default Template Configuration.
   

   • Yes: You can set any parameters for jobs and scripts.
   • No: You must select a template for jobs and scripts, and cannot overwrite the parameters in the template when configuring jobs and scripts. If you select No, select a default parameter template that has been configured. For details about how to configure a template, see Configuring a Template.

     Then go to the basic information page of the Spark SQL job or Spark SQL script page and click in the upper right corner to view the configured default program parameters. The preset default parameters are unavailable and cannot be modified.

     You can also customize program parameters. When a Spark SQL job or script is executed, the unavailable parameters in the template prevail.

HiveSqlJob/Script Default Template Configuration

You can set this parameter to determine whether parameters can be set to overwrite the default parameters of the template.

In the MRS API connection mode, default parameters can be configured for Hive SQL scripts. For proxy connections, this function is not supported.

1. In the navigation pane, choose Configuration > Specifications.
2. Choose Default Configuration.
3. Set HiveSqlJob/Script Default Template Configuration.
   

   • Yes: You can set any parameters for jobs and scripts.
   • No: You must select a template for jobs and scripts, and cannot overwrite the parameters in the template when configuring jobs and scripts. If you select No, select a default parameter template that has been configured. For details about how to configure a template, see Configuring a Template.

     Then go to the basic information page of the Hive SQL job or Hive SQL script page and click in the upper right corner to view the configured default program parameters. The preset default parameters are unavailable and cannot be modified.

     You can also customize program parameters. When a Hive SQL job or script is executed, the unavailable parameters in the template prevail.

4. Click Save to save the settings.

Job/Script Change Management

If you enable this function, you can export job/script changes (addition, modification, and deletion) in a workspace to a .zip file, and import the file to another workspace.

1. In the left navigation pane on the DataArts Factory console, choose Configuration > Configure.
2. Click Default Configuration.
3. Set Job/Script Change Management.
   

   • Yes: Events are recorded for job and script changes. All the changed jobs and scripts can be incrementally exported and imported by time.
   • No: No events are recorded for job and script changes. Only selected jobs and scripts can be exported and imported.

4. Click Save to save the settings.
   

   You can export and import jobs and scripts in the workspace only if you have set Job/Script Change Management to Yes.