From OBS
Type |
Parameter |
Description |
Example Value |
---|---|---|---|
Basic parameters |
Bucket Name |
Name of the bucket from which data will be migrated |
BUCKET_2 |
File Format |
Format used for transmitting data.
|
CSV |
|
Source Directory/File |
Directory or file path from which data will be extracted. You can enter a maximum of 50 file paths. By default, the file paths are separated by vertical bars (|). You can also customize a file separator. For details, see Migration of a List of Files. This parameter can be configured as a macro variable of date and time and a path name can contain multiple macro variables. When the macro variable of date and time works with a scheduled job, the incremental data can be synchronized periodically. For details, see Incremental Synchronization Using the Macro Variables of Date and Time.
NOTE:
If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset). |
FROM/example.csv |
|
Entries Files |
This parameter is displayed only when File Format is set to Binary. If the pull list file function is enabled, the content of a file (such as a .txt file) in an OBS bucket can be read as the list of files to be migrated. The content in the file must be the absolute path of the file to be migrated (rather than a directory). For example, the content is as follows: /052101/DAY20211110.data /052101/DAY20211111.data |
Yes |
|
OBS Link of entries files |
This parameter is available only when Entries Files is set to Yes. You can select the OBS link where the list file is located. |
OBS_test_link |
|
OBS Bucket of entries files |
This parameter is available only when Entries Files is set to Yes. It indicates the name of the OBS bucket where the list file is located. |
01 |
|
Path/Directory of entries files |
This parameter is available only when Entries Files is set to Yes. It indicates the absolute path or directory of the list file in the OBS bucket. You are advised to select the absolute path of the file. If you select a directory, files in subdirectories can also be migrated. However, if the number of files in the directory is too large, the cluster memory may become insufficient. |
/0521/Lists.txt |
|
JSON Type |
This parameter is displayed only when File Format is set to JSON. Type of a JSON object stored in a JSON file. The options are JSON object and JSON array. |
JSON object |
|
JSON Reference Node |
This parameter is available only when File Format is set to JSON and JSON Type is set to JSON object. CDM parses the data under the JSON node. If the node's corresponding data is a JSON array, the system will extract data from the array in the same pattern. Use periods (.) to separate multi-layer nested JSON nodes. |
data.list |
|
Advanced attributes |
Line Separator |
Lind feed character in a file. By default, the system automatically identifies \n, \r, and \r\n. This parameter is displayed only when File Format is set to CSV. |
\n |
Use Quote Char |
If you set this parameter to Yes, the field delimiters in the encircling symbol are regarded as a part of the string value. Currently, the default encircling symbol of CDM is ". |
No |
|
Using Escape Char |
If you select Yes, the backslash (\) in the data row is used as an escape character. If you select No, the backslash (\) in the CSV file will not be escaped. CSV supports only the backslash (\) as the escape character. |
Yes |
|
Using RE to separate fields |
Whether to use regular expressions to separate fields. If you set this parameter to Yes, Field Delimiter becomes invalid. This parameter is displayed only when File Format is set to CSV. |
Yes |
|
Regular Expression |
Regular expression used to separate fields. For details about regular expressions, see Regular Expressions for Separating Semi-structured Text. |
^(\d.*\d) (\w*) \[(.*)\] ([\w\.]*) (\w.*).* |
|
First N Rows As Header |
This parameter is displayed only when File Format is set to CSV. When you migrate a CSV file to a table, CDM writes all data to the table by default. If you set this parameter to Yes, CDM uses the first N rows of the CSV file as the heading row and does not write the row to the destination table. |
No |
|
The Number of Header Rows |
This parameter is available when First N Rows As Header is set to Yes. It specifies the number of header rows to be skipped during data extraction.
NOTE:
The number of header rows cannot be empty. The value is an integer from 1 to 99. |
1 |
|
Extract first row as columns |
This parameter is available when First N Rows As Header is set to Yes. It specifies whether to parse the first row of the header as a column name. The column name is displayed in the source field during field mapping configuration.
NOTE:
|
Yes |
|
Encode type |
Encoding type, for example, UTF-8 or GBK. You can set the encoding type for text files only. This parameter is invalid when File Format is set to Binary. |
GBK |
|
Compression Format |
The options are as follows:
|
None |
|
Compressed File Extension |
Extension of the files to be decompressed. The decompression operation is performed only when the file name extension is used in a batch of files. Otherwise, files are transferred in the original format. If you enter * or leave the parameter blank, all files are decompressed. |
* |
|
Start Job by Marker File |
Whether to start a job by a marker file. A job is only started if there is a marker file for starting the job in the source path. If there is no marker file, the job will be suspended for a period of time specified by Suspension Period. |
No |
|
File Separator |
File separator. If you enter multiple file paths in Source Directory/Files, CDM uses the file separator to identify files. The default value is |. |
| |
|
Marker File |
Name of the marker file for starting a job. If you specify a marker file, the migration job is executed only when the marker file exists in the source path. The marker file will not be migrated. |
ok.txt |
|
Wait Time |
Waiting period for a marker file. If you set Start Job by Marker File to Yes but there is no marker file in the source path, the job fails when the suspension period times out. If you set this parameter to 0 and there is no marker file in the source path, the job will fail immediately. Unit: second |
10 |
|
Filter Type |
Only paths or files that meet the filtering conditions are transferred. The options are None, Wildcard, and Regex. For details, see Incremental File Migration. |
Wildcard |
|
Directory Filter |
If you set Filter Type to Wildcard or Regex, enter a wildcard character to filter paths. The paths that meet the filtering condition are migrated. You can configure multiple paths separated by commas (,).
NOTE:
If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset). |
*input |
|
File Filter |
If you set Filter Type to Wildcard or Regex, you can enter a wildcard character to search for files in a specified path. The files that meet the search criteria are migrated. You can configure multiple files separated by commas (,).
NOTE:
If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset). |
*.csv,*.txt |
|
Time Filter |
If you select Yes, files are transferred based on their modification time. |
Yes |
|
Minimum Timestamp |
If you set Filter Type to Time Filter, and specify a point in time for this parameter, only the files modified after the specified time are transferred. The time format must be yyyy-MM-dd HH:mm:ss. This parameter can be set to a macro variable of date and time. For example, ${timestamp(dateformat(yyyy-MM-dd HH:mm:ss,-90,DAY))} indicates that only files generated within the latest 90 days are migrated.
NOTE:
If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset). |
2019-06-01 00:00:00 |
|
Maximum Timestamp |
If you set Filter Type to Time Filter, and specify a point in time for this parameter, only the files modified before the specified time are transferred. The time format must be yyyy-MM-dd HH:mm:ss. This parameter can be set to a macro variable of date and time. For example, ${timestamp(dateformat(yyyy-MM-dd HH:mm:ss))} indicates that only the files whose modification time is earlier than the current time are migrated.
NOTE:
If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset). |
2019-07-01 00:00:00 |
|
Disregard Non-existent Path/File |
Whether to proceed when the selected file does not exist in the source path. If you select Yes, the job can be successfully executed even if the file does not exist in the source path. |
No |
|
MD5 File Extension |
This parameter is displayed only when File Format is set to Binary. This parameter is used to check whether the files extracted by CDM are consistent with source files. For details, see MD5 Verification. |
.md5 |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot