Updated on 2024-12-02 GMT+08:00

From Doris

If the source link of a job is a Doris link, configure the source job parameters based on Table 1.

Table 1 Parameter description

Type

Parameter

Description

Example Value

Basic parameters

Use SQL Statement

Whether to use SQL statements to extract source data

No

SQL Statement

When Use SQL Statement is set to Yes, enter an SQL statement here. CDM exports data based on the SQL statement.

NOTE:
  • SQL statements can only be used to query data. Join and nesting are supported, but multiple query statements are not allowed, for example, select * from table a; select * from table b.
  • With statements are not supported.
  • Comments, such as -- and /*, are not supported.
  • Addition, deletion, and modification operations are not supported, including but not limited to the following:
    • load data
    • delete from
    • alter table
    • create table
    • drop table
    • into outfile
  • If the SQL statement is too long, the request fails to be delivered. If you continue to create a job, the system displays an error message indicating that the request is incorrect. In this case, you need to simplify or clear the SQL statement and try again.

select id,name from sqoop.user;

Schema/Tablespace

Name of the database to which data will be written. The schema can be automatically created. Click the icon next to the text box to select a schema or tablespace.

schema

Table Name

Name of the table from which data will be read. Click the button next to the text box. The dialog box for selecting the table is displayed.

This parameter can be configured as a macro variable of date and time and a path name can contain multiple macro variables. When the macro variable of date and time works with a scheduled job, the incremental data can be synchronized periodically. For details, see Incremental Synchronization Using the Macro Variables of Date and Time.

NOTE:

If you have configured a macro variable of date and time and schedule a CDM job through DataArts Factory of DataArts Studio, the system replaces the macro variable of date and time with (Planned start time of the data development jobOffset) rather than (Actual start time of the CDM jobOffset).

table

Advanced attributes

Where Clause

If you set Use SQL Statement to No, you can add a where clause to add filter criteria.

age > 18 and age <= 60

Retain One Decimal Place for Date Values

Whether to retain one decimal place for date values

No

Partition Column

Column used to split data during data extraction to implement parallel extraction

id

Null in Partition Column

During concurrent extraction, if the partition column does not contain null, set this parameter to No to improve performance. If you are not sure whether the partition column contains null, set this parameter to Yes to avoid data loss.

No