Creating an Offline Processing Migration Job
Notes and Constraints
Procedure
- Log in to the DataArts Studio console by following the instructions in Accessing the DataArts Studio Instance Console.
- On the DataArts Studio console, locate a workspace and click DataArts Factory.
- In the left navigation pane of DataArts Factory, choose .
- Create a migration job using either of the following methods:
Method 1: On the Develop Job page, click Create Data Migration Job.
Figure 1 Creating a migration job (method 1)
Method 2: In the directory list, right-click a directory and select Create Data Migration Job.
Figure 2 Creating a migration job (method 2)
- In the displayed Create Data Migration Job dialog box, configure job parameters. Table 1 describes the job parameters.
Figure 3 Configuring data migration job parameters
Table 1 Job parameters Parameter
Description
Job Name
Name of the job. The name must contain 1 to 128 characters, including only letters, numbers, hyphens (-), underscores (_), and periods (.).
Job Type
Job type. Select Offline processing.
- Offline processing: A large amount of collected data is processed and analyzed in batches. These tasks usually use optimized computing and storage resources to ensure efficient data processing and analysis. These tasks are usually executed periodically (for example, every day or every week) to process a large amount of historical data for batch analysis and data warehouses.
- Real-time processing: New data generated continuously is processed and analyzed in real time to meet the requirements for data timeliness. This mode requires instant processing of data upon generation and returns the result or triggers operations.
Select Directory
Directory to which the job belongs. The root directory is selected by default.
- Click OK.
Configuring Basic Job Information
After you configure the owner and priority for a job, you can search for the job by the owner and priority. The procedure is as follows:
Click the Basic Info tab on the right of the canvas to expand the configuration page and configure job parameters, as listed in Table 2.
Parameter |
Description |
---|---|
Owner |
An owner configured during job creation is automatically matched. This parameter value can be modified. |
Executor |
This parameter is available when Scheduling Identities is set to Yes. User that executes the job. When you enter an executor, the job is executed by the executor. If the executor is left unspecified, the job is executed by the user who submitted the job for startup. |
Job Agency |
This parameter is available when Scheduling Identities is set to Yes. After an agency is configured, the job interacts with other services as an agency during job execution. |
Priority |
Priority configured during job creation is automatically matched. This parameter value can be modified. |
Execution Timeout |
Timeout of the job instance. If this parameter is set to 0 or is not set, this parameter does not take effect. If the notification function is enabled for the job and the execution time of the job instance exceeds the preset value, the system sends a specified notification. |
Exclude Waiting Time from Instance Timeout Duration |
Whether to exclude the wait time from the instance execution timeout duration If you select this option, the time to wait before an instance starts running is excluded from the timeout duration. You can modify this setting on the Default Configuration page. If you do not select this option, the time to wait before an instance starts running is included in the timeout duration. |
Custom Parameter |
Set the name and value of the parameter. |
Job Tag |
Configure job tags to manage jobs by category. Click Add to add a tag to the job. You can also select a tag configured in Managing Job Tags. |
Node Status Polling Interval (s) |
How often the system checks completeness of the node task. The value ranges from 1 to 60 seconds. |
Max. Node Execution Duration |
Execution timeout interval for the node. If retry is configured and the execution is not complete within the timeout interval, the node will be executed again. |
Retry upon Failure |
You can select Retry 3 times or Never. Never is recommended. You are advised to configure automatic retry for only file migration jobs or database migration jobs with Import to Staging Table enabled to avoid data inconsistency caused by repeated data writes.
NOTE:
If you want to set parameters in DataArts Studio DataArts Factory to schedule the CDM migration job, do not configure this parameter. Instead, set parameter Retry upon Failure for the CDM node in DataArts Factory. |
Policy for Handling Subsequent Nodes If the Current node Fails |
Policy for handling subsequent nodes if the current node fails
|
Configuring Job Parameters
Job parameters can be globally used in any node in jobs. The procedure is as follows:
Click Parameter Setup on the right of the editor and set the parameters described in Table 3.
Functions |
Description |
---|---|
Variables |
|
Add |
Click Add and enter the variable parameter name and parameter value in the text boxes.
After the parameter is configured, it is referenced in the format of ${parameter name} in the job. |
Edit Parameter Expression |
Click next to the parameter value text box. In the displayed dialog box, edit the parameter expression. For more expressions, see Expression Overview. |
Modifying a Job |
Change the parameter name or value in the corresponding text boxes. |
Mask |
If the parameter value is a key, click to mask the value for security purposes. |
Delete |
Click next to the parameter name and value text boxes to delete the job parameter. |
Constant Parameter |
|
Add |
Click Add and enter the constant parameter name and parameter value in the text boxes.
After the parameter is configured, it is referenced in the format of ${parameter name} in the job. |
Edit Parameter Expression |
Click next to the parameter value text box. In the displayed dialog box, edit the parameter expression. For more expressions, see Expression Overview. |
Modifying a Job |
Modify the parameter name and parameter value in text boxes and save the modifications. |
Delete |
Click next to the parameter name and value text boxes to delete the job parameter. |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot