From FTP/SFTP

Basic parameters

Source Directory/File

Directory or a single file path to be transferred

FROM_DIRECTORY/ or FROM_DIRECTORY/example.csv

File Format

Format used for transmitting data.

CSV, JSON, and binary formats are supported.

The CSV and JSON formats are supported for migration to tables, and the binary format is supported for file migration.

CSV

JSON Type

This parameter is available when File Format is set to JSON.

Type of a JSON object stored in a JSON file. The options are JSON object and JSON array.

JSON object

JSON Reference Node

This parameter is available when JSON Type is set to JSON object.

It indicates the root node that records data. The data recorded is a JSON array. The system extracts data from the array in the same mode. Use periods (.) to separate multi-layer nested JSON nodes.

data.list

Advanced attributes

Use rfc4180 Parser

This parameter is available when File Format is set to CSV.

It specifies whether to use the rfc4180 parser to parse CSV files.

No

Line Separator

This parameter is available when File Format is set to CSV.

It indicates the line feed character in a file. By default, the system automatically identifies \n, \r, and \r\n.

\n

Use Quote Char

This parameter is available when File Format is set to CSV.

Quote characters are used to enclose a string value. Field separators in the quote characters are regarded as a part of the string value. Only quotation marks (") can be used as quote characters.

No

Use Escape Char

This parameter is available when File Format is set to CSV.

CSV supports only the backslash (\) as the escape character.

If you select Yes, the backslash (\) in the data row is used as an escape character.

If you select No, the backslash (\) in the CSV file will not be escaped.

Yes

Using RE to separate fields

This parameter is available when File Format is set to CSV.

It specifies whether to use a regular expression to separate fields.

Yes

Regular Expression

This parameter is available when File Format is set to CSV and Use RE to Separate Fields is set to Yes.

It indicates the regular expression used to separate fields.

^(\d.*\d) (\w*) \[(.*)\] ([\w\.]*) (\w.*).*

File Separator

This parameter is available when File Format is set to CSV and Use RE to Separate Fields is set to No.

It indicates the field delimiter in the file.

,

First Row As Header

This parameter is available when File Format is set to CSV.

If you set this parameter, the program reads the first row as the header row when extracting data.

No

Encode Type

This parameter is available when File Format is set to CSV or JSON.

It indicates the file encoding type.

The encoding type can be set only for text files. Otherwise, the setting is invalid.

Two file encoding types are supported: UTF-8 and GBK.

UTF-8

Compression Format

Compression format

Default value: none. The following compression formats are supported: GZIP, ZIP, and TAR.GZ.

GZIP

Compressed File Extension

This parameter is available when the compression format is GZIP, ZIP, or TAR.GZ.

It indicates the extension of the file to be decompressed.

The decompression operation is performed only when the file name extension is used in a batch of files. Otherwise, files are transferred in the original format. If you enter * or leave this parameter blank, all files will be decompressed.

tar.gz

File Separator

Character used to separate files

|

Start Job by Marker File

A job is started only when there is a marker file for starting the job in the source path. Otherwise, the job will be suspended for a period of time.

No

Marker File

This parameter is available when Start Job by Marker File is set to Yes.

It indicates the name of the marker file for starting a job. If you enter the name of the marker file, the migration job is executed only when the marker file exists in the source path. The marker file will not be migrated.

ok.txt

Wait Time

This parameter is available when Start Job by Marker File is set to Yes.

It indicates the maximum time for detecting a marker file.

If the wait time ends and no marker file is detected, the job fails. If the wait time is set to 0 and there is no marker file in the source path, the job fails immediately. The unit is second.

60

Marker File Type

This parameter is available when Start Job by Marker File is set to Yes.

It indicates the marker file type.

• MARK_DONE: The migration job is executed only when the marker file exists in the source path.
• MARK_DOING: The migration job is executed only when the marker file does not exist in the source path.

MARK_DONE

Filter Type

Type of the file that will be transmitted.

The following filter criteria are supported: none, wildcard, and regular expression.

None

Directory Filter

This parameter is available when Filter Type is set to Wildcard or Regular expression.

It filters one or multiple levels of directories in the input path.

• input*/test* for a wildcard
• intput.*/test.* for a regular expression.

File Filter

This parameter is available when Filter Type is set to Wildcard or Regular expression.

It filters files in the input path.

• *csv for a wildcard
• .*\.csv for a regular expression

Time Filter

Filters files that meet a specified time range.

• Files modified after the start time or before the end time will be transferred.
• If both the start time and end time are specified, files modified within this time range will be transferred.

No

Minimum Timestamp

This parameter is available only when Time Filter is set to Yes.

Files modified after the specified time will be transferred. The specified time must be earlier than the current timestamp and cannot be later than the end time. The time is in yyyy-MM-dd HH:mm:ss format.

2018-01-01 00:00:00

Maximum Timestamp

This parameter is available only when Time Filter is set to Yes.

Files modified before the specified time will be transferred. The specified time must be earlier than the current timestamp but cannot be earlier than the start time. The time is in yyyy-MM-dd HH:mm:ss format.

2018-01-01 00:00:00

Disregard Non-existent Path or File

Whether to proceed when the selected file does not exist in the source path. If you select Yes, the job can be successfully executed even if the file does not exist in the source path.

No

Whether to Skip Empty Lines

This parameter is available when File Format is set to CSV.

It specifies whether to skip an empty line.

No

Null Value

This parameter is available when File Format is set to CSV.

No string can be used to define a null value in text files. This parameter specifies the string to be identified as a null value.

If this parameter is set to null and the value of a column in a row is null, the value will be parsed as null.

N/A

MD5 File Extension

This parameter is available when File Format is set to Binary.

Check whether the files extracted by CDM are consistent with source files.

.md5