To HDFS

Basic parameters

Write Directory

HDFS directory to which data will be written.

/user/cdm/output

File Format

Format used for transmitting data. The CSV and JSON formats are supported for migration to tables, and the binary format is supported for file migration.

CSV

Newline character processing mode

Policy for processing newline characters when the data written to a text file table contains newline characters (\n|\r|\r\n)

• Delete
• Ignore
• Replace with another string

Delete

Newline Replacement String

It indicates the string that will replace newline characters.

N/A

Advanced attributes

Write to Temporary File

This parameter is displayed only when File Format is set to Binary.

It indicates writing the binary file to a temporary file whose name extension is .tmp.

No

Line Separator

This parameter is displayed only when File Format is set to CSV.

It indicates the line feed character in a file. By default, the system automatically identifies \n, \r, and \r\n. You can configure special characters. For spaces and carriage returns, encode them with URL. You can also configure them by editing the job JSON, in which case URL encoding is not required.

\n

Field Delimiter

This parameter is displayed only when File Format is set to CSV.

Field delimiter in the file. Special characters must be encoded using URLs.

,

Job Success Marker File

Marker file name

When the job is successful, a marker file is generated in the destination directory. If this parameter is left blank, no marker file will be generated.

finish.txt

Use Quote Character

This parameter is displayed only when File Format is set to CSV.

Enclose a string using quote characters. Field separators in the quote characters are regarded as a part of the string value. Only quotation marks (") can be used as quote characters.

No

Customize Hierarchical Directory

Users can customize the directory hierarchy of files.

Example: [Table name]/[Year]/[Month]/[Day]/[Data file name]. csv

No

Hierarchical Directory

This parameter is displayed when Customize Hierarchical Directory is set to Yes.

Used to specify the directory level of a file, with time macro supported (the time format is yyyy/MM/dd). If the source link is a relational database link, the directory name consists of the source table name and a custom directory name. In other scenarios, the directory is a custom directory.

${dateformat(yyyy/MM/dd, -1, DAY)}

File Name Prefix

This parameter is displayed only when File Format is set to CSV.

Prefix of the file name.

File name format: prefix-jobname-timestamp-index

data

Compression Format

This parameter is displayed only when File Format is set to CSV.

Compression format of the file to be written

• NONE
• DEFLATE
• GZIP
• BZIP2
• SNAPPY

SNAPPY

Encryption

This parameter is displayed only when File Format is set to Binary.

Encryption mode for the uploaded data

• None
• AES-256-GCM

None

DEK

This parameter is displayed when File Format is set to Binary and Encryption Mode is set.

It indicates the data encryption key. The AES-256-GCM key consists of 64 hexadecimal digits.

DD0AE00DFECD78BF051BCFDA25BD4E320DB0A7AC75A1F3FC3D3C56A457DCDC1B

IV

This parameter is displayed when File Format is set to Binary and Encryption Mode is set.

It indicates the initialization vector, which consists of 32 hexadecimal digits.

5C91687BA886EDCD12ACBC3FF19A3C3F