From HTTP
If the source link of a job is an HTTP link, configure the source job parameters based on Table 1. Currently, data can only be exported from the HTTP URLs.
Parameter |
Description |
Example Value |
---|---|---|
File URL |
Use the GET method to obtain data from the HTTP/HTTPS URL. These connectors are used to read files with an HTTP/HTTPS URL, such as reading public files on the third-party object storage system and web disks. |
https://bucket.obs.myhuaweicloud.com/object-key |
Pull List File |
If this parameter is set to Yes, the system pulls the files corresponding to the URLs in the text file to be uploaded and stores them on OBS. The text file records the file paths on HDFS. |
Yes |
OBS Link of List File |
Select an existing OBS link. |
obs_link |
OBS Bucket of entries files |
Name of the OBS bucket that stores the text file |
obs-cdm |
Path/Directory of entries files |
Custom OBS directories that store the text file. Use slashes (/) to separate different directories. |
test1 |
File Format |
CDM supports Binary only, which indicates that files (even not in binary format) will be directly transferred. |
Binary |
Compression Format |
Compression format of the source files. The options are as follows:
|
NONE |
Compressed File Suffix |
This parameter is displayed when Compression Format is not NONE. This parameter specifies the extension of the files to be decompressed. The decompression operation is performed only when the file name extension is used in a batch of files. Otherwise, files are transferred in the original format. If you enter * or leave the parameter blank, all files are decompressed. |
* |
File Separator |
File separator. When multiple files are transferred, CDM uses the file separator to identify files. The default value is |. This parameter is not displayed if Pull List File is set to Yes. |
| |
Query Parameter |
|
No |
Disregard Non-existent Path or File |
If this is set to Yes, the job can be successfully executed even if the source path does not exist. |
No |
MD5 File Extension |
This parameter is used to check whether the files extracted by CDM are consistent with source files. For details, see MD5 Verification. |
.md5 |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot