Source Link Configurations of Loader Jobs
Overview
When Loader jobs obtain data from different data sources, a link corresponding to a data source type needs to be selected and the link properties need to be configured.
This section applies to versions earlier than MRS 3.x.
obs-connector
Parameter |
Description |
---|---|
Bucket Name |
OBS file system for storing source data. |
Source Directory/File |
Actual storage form of source data. It can be either all data files in a directory or a single data file contained in the file system. |
File Format |
Loader supports the following file formats of data stored in OBS:
|
Line Separator |
Identifier of each line end of source data |
Field Separator |
Identifier of each field end of source data |
Encoding Type |
Text encoding type of source data. It takes effect on text files only. |
File Split Type |
The following types are supported:
|
generic-jdbc-connector
Parameter |
Description |
---|---|
Schema/Tablespace |
Name of the database storing source data. You can query and select it on the interface. |
Table Name |
Data table storing the source data. You can query and select it on the interface. |
Partition Column |
If multiple columns need to be read, use this column to split the result and obtain data. |
Where Clause |
Query statement used when accessing the database |
ftp-connector or sftp-connector
Parameter |
Description |
---|---|
Source Directory/File |
Actual storage form of source data. It can be either all data files in a directory or single data file contained in the file server. |
File Format |
Loader supports the following file formats of data stored in the file server:
|
Line Separator |
Identifier of each line end of source data
NOTE:
If FTP or SFTP serves as a source link and File Format is set to BINARY_FILE, the value of Line Separator in the advanced properties is invalid. |
Field Separator |
Identifier of each field end of source data
NOTE:
If FTP or SFTP serves as a source link and File Format is set to BINARY_FILE, the value of Field Separator in the advanced properties is invalid. |
Encoding Type |
Text encoding type of source data. It takes effect on text files only. |
File Split Type |
The following types are supported:
|
hbase-connector
Parameter |
Description |
---|---|
Table Name |
HBase table storing source data |
hdfs-connector
Parameter |
Description |
---|---|
Source Directory/File |
Actual storage form of source data. It can be either all data files in a directory or single data file contained in HDFS. |
File Format |
Loader supports the following file formats of data stored in HDFS:
|
Line Separator |
Identifier of each line end of source data
NOTE:
If HDFS serves as a source link and File Format is set to BINARY_FILE, the value of Line Separator in the advanced properties is invalid. |
Field Separator |
Identifier of each field end of source data
NOTE:
If HDFS serves as a source link and File Format is set to BINARY_FILE, the value of Field Separator in the advanced properties is invalid. |
File Split Type |
The following types are supported:
|
hive-connector
Parameter |
Description |
---|---|
Database Name |
Name of the Hive database storing the data source. You can query and select it on the interface. |
Table |
Name of the Hive table storing the data source. You can query and select it on the interface. |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.