Updated on 2026-11-16 GMT+08:00

Node Overview

What Is a Node?

A node defines the operations performed on data and is the basic unit of a data development job. A pipeline job consists of one or more nodes connected by lines. The execution sequence of nodes is determined by the dependencies between them. A directed acyclic graph (DAG) shows the dependencies between nodes. You can select required nodes based on your service model and orchestrate the dependencies between nodes in the job canvas to design a data processing process. For details about how to orchestrate job nodes, see Developing a Pipeline Job.

The execution sequence of nodes is determined by the dependencies between them. A downstream node starts only after all its upstream nodes have been successfully executed. Proper dependency configuration is necessary for successful job execution.

Node Types

DataArts Factory provides nodes used for data integration, compute and analysis, resource management, and data monitoring. You can choose your desired nodes. The following table describes the functions and typical use cases of each type of node.

Table 1 Node types

Node Type

Node Name

Description

Data integration

CDM Job, Data Migration, Rest Client, and more

These nodes are used to migrate data from the source to a lake, for example, synchronize data from a service database to a data warehouse or data lake. They are suitable for data migration and data ingestion into a lake.

Compute & Analysis

DLI Spark, DLI SQL, DWS SQL, MRS Spark, MRS Hive SQL, MRS Spark SQL, and more

These nodes are used for data cleansing, transformation, computation, and analysis. They are suitable for ETL as well as stream and batch data processing.

Resource management

Create OBS, Delete OBS, OBS Manager, and more

These nodes are used to manage job resources.

Data monitoring

Data Quality Monitor and DRS Monitor

These nodes are used to manage job monitoring tasks. They are suitable for job quality monitoring.

Other

SubJob, For Each, SMN, Dummy, and more

These nodes are suitable for more complex job execution scenarios, for example, specifying a subjob to be executed cyclically.

Node Connection Modes

Nodes can be connected in serial or parallel mode.

  • Serial connection: Nodes are executed one by one in sequence. Node B is executed only after node A is executed. This mode is suitable when a node depends on the output of its previous node.
  • Parallel connection: Different nodes are executed at the same time. This mode is suitable when nodes are independent of each other and can process data in parallel. It shortens the job execution time.
Figure 1 Connection diagram
  • If there is data dependency between the nodes executed in parallel (for example, a node uses the calculation result of its previous node), use a serial connection to connect the nodes. Otherwise, the execution may fail due to data unavailability.
  • During job execution, the system automatically schedules nodes based on their dependencies. You do not need to manually control the execution sequence.

Node Parameters and Expressions

Node parameters support EL expressions, which can be used as variables in node parameters and job parameters. This enables parameterized configuration and reuse of the same job in different scenarios.

  • When a job is started, the EL expression is replaced with the actual value, and then the node is executed.
  • Parameterized configuration eliminates the need to define duplicate jobs and improves development efficiency.

For details about the syntax and examples of EL expressions, see Expression Overview.

Nodes and Lineages

Data lineage tracks the full lifecycle of data, including generation, processing, convergence, transfer, circulation, and deletion. In DataArts Studio, data lineages are generated between nodes. The relationship between lineages and nodes is as follows:

  • Lineage diagram elements: A data lineage diagram consists of data table objects and job node objects. A node is connected to a data table through an arrow, which indicates that the data in the table is generated by the node.
  • Lineage parsing granularity: Data lineages are parsed based on the nodes in a job. Lineages are established between the input and output tables of each node in a job. You are advised to design the nodes of a job based on the service processing granularity to accurately display lineages.
  • Automatic lineage parsing: Lineages are automatically generated after the system parses the data processing and data migration nodes in data development jobs. No manual configuration is required. For details about the node types and scenarios that support automatic lineage parsing, see Automatic Lineage Parsing.
  • Manual lineage configuration: You can customize the input and output tables of lineages in the nodes of data development jobs when automatic lineage parsing is not supported. For details about the node types that support manual lineage configuration, see Manually Configuring a Lineage.
  • Prerequisites for viewing lineages: You need to complete a metadata collection task in DataArts Catalog. When a job meets the requirements for automatic lineage parsing, or lineages have been manually configured and the job has been successfully scheduled, you can view the lineages in DataArts Catalog.

Data lineages allow you to quickly trace the data source and processing link, locate problematic nodes during data processing, and meet data audit and compliance requirements.

Helpful Links