Configuring Hourly Scheduling

Basic Concepts

• Job (in DataArts Factory): A job is composed of one or more nodes that run together to complete data operations.
• Node: A node defines the actions to be performed on data. For example, the MRS Hive SQL node can execute the Hive SQL scripts predefined in DataArts Factory. DML and DDL SQL statements can be transferred to the MRS Hive SQL node for execution.
• Periodic scheduling: Jobs are automatically executed at a specified interval. An appropriate scheduling period is required. A maximum of five instances can be concurrently executed in a job. If the start time of a job instance is later than the configured job execution time, the job instances in the subsequent batch will be queued. As a result, the job execution costs a longer time than expected.
• Job dependency: You can select jobs that are executed periodically in different workspaces as dependency jobs. The current job starts only after its dependency jobs are executed.

Operation Process

Preparations

• Create a proxy data connection named hive_000001 in Management Center. For details, see MRS Hive Connection Parameters.
• Create a database named default. For details, see Creating a Database.
• Create a Hive SQL script named hive_0520. For details, see Developing an SQL Script.

Procedure

1. In the left navigation pane of DataArts Factory, choose Development > Develop Job.
2. Right-click the directory and select Create Job.
3. Select Batch processing for Job Type and enter job_8750 for Job Name.
4. Set other basic job parameters and click OK to access the job development page.
5. Drag the MRS Hive SQL node to the canvas, click the node name, and set node parameters. For details about the parameters of the MRS Hive SQL node, see MRS Hive SQL.

   The key parameters are as follows:

   • SQL script: Select the hive_0520 script you have created.
   • Script Parameter: Set parameters as needed. If the SQL script does not use any parameters, Script Parameter is not displayed.
   • Data Connection: Select the hive_000001 data connection you have created (the data connection selected in the script).
     Figure 1 Setting node parameters
     

     Set other parameters as needed. Advanced parameters are set by default. You can choose not to modify them.

6. Click Scheduling Setup on the right, set Scheduling Type to Run periodically, and set Scheduling Frequency to Hours.
   • Interval Hour

     Set Start Time to 00:00, Interval to 2 Hours, and End Time to 23:59.

     Figure 2 Scheduling by interval hours
     
   • Discrete Hour

     Set the job scheduling time, for example, 09:00, 10:00, 15:00, 16:00, and 19:00 every day.

     Figure 3 Scheduling by discrete hours
     
     

     You can specify the time points for running the job every day. In the discrete hour scheduling mode, you can specify the minute.

7. Configure the job dependencies. The current job starts after its dependencies are complete. For details about the dependency policies for periodic scheduling, see Dependency Policies for Periodic Scheduling.
   Figure 4 Configuring job dependencies
   

8. After configuring the scheduling and dependency parameters, retain the default values for other job parameters.
9. After configuring the job, save and submit the version.
   Figure 5 Submitting a job version
   

10. Click Execute to start the job.
    

    • The current job starts after its dependencies are complete. Before that, the current job is in waiting state.
    • An appropriate scheduling period is required. A maximum of five instances can be concurrently executed in a job. If the start time of a job instance is later than the configured job execution time, the job instances in the subsequent batch will be queued. As a result, the job execution costs a longer time than expected. In addition, if the scheduling period of the dependency differs significantly from that of the current job, the job may not run properly. Ensure that the scheduling periods meet service rules.
    • In the discrete-hour scheduling mode, there are a wide range of dependency choices, for example, dependency on the latest job. For details, see Scheduling by Discrete Hours and Scheduling by the Nearest Job Instance.

11. In the navigation pane on the left, choose Monitoring > Job Monitoring. On the Batch Jobs page, view the job execution result.
    Figure 6 Batch Jobs
    

Reference

• Creating data connections: You can create data connections by configuring data sources. Based on the data connections in Management Center, DataArts Studio connects to the data lake foundation and provides data development, governance, services, and operations. When the data connection of a script or job is released in the development environment, the connection is automatically switched to the data connection of the production environment. For details, see Creating a DataArts Studio Data Connection.
• Creating tables: You can create tables on the DataArts Factory console, or using DDL or a SQL script. You are advised to create tables on the DataArts Factory console with no code. For details, see Creating a Table.
• Configuring job node parameters: A node defines the actions to be performed on data. DataArts Factory provides nodes used for data integration, compute and analysis, database operations, and resource management. You can choose your desired nodes. For details, see Node Parameter Configuration.
• Dependency policies for periodic scheduling: Currently, DataArts Factory supports two types of dependency policies, that is, dependency between jobs whose scheduling periods are traditional periods and dependency between jobs whose scheduling periods are natural periods. For details, see Dependency Policies for Periodic Scheduling.
• Scheduling by discrete hours and scheduling by the nearest job instance: Many new rules are available for natural periodic scheduling, such as scheduling by discrete hours and scheduling by the nearest job instance. For details, see Scheduling by Discrete Hours and Scheduling by the Nearest Job Instance.