Overview of Importing and Exporting Loader Data

Updated on 2024-10-09 GMT+08:00

View PDF

Introduction to Loader Data Import

Loader is an ETL tool that enables MRS to exchange data and files with external data sources, such as relational databases, SFTP servers, and FTP servers. It allows data or files to be imported from relational databases or file systems to MRS.

Loader supports the following data import modes:

Importing data from a relational database to HDFS or OBS
Importing data from a relational database to HBase
Importing data from a relational database to Phoenix tables
Importing data from a relational database to Hive tables
Importing data from an SFTP server to HDFS or OBS
Importing data from an SFTP server to HBase
Importing data from an SFTP server to Phoenix tables
Importing data from an SFTP server to Hive tables
Importing data from an FTP server to HDFS or OBS
Importing data from an FTP server to HBase
Importing data from an FTP server to Phoenix tables
Importing data from an FTP server to Hive tables
Importing data from HDFS or OBS to HBase in the same cluster

MRS needs to connect to an external data source to exchange data and files with the data source. The following connectors are used to configure connection parameters for different types of data sources

generic-jdbc-connector: relational database connector
ftp-connector: FTP data source connector
hdfs-connector: HDFS data source connector
oracle-connector: dedicated connector for Oracle databases. row_id serves as partition columns. Compared with generic-jdbc-connector, Map jobs are more evenly distributed on oracle-connector, and whether indexes have been created for the partition columns does not matter.
mysql-fastpath-connector: dedicated connector for MySQL databases. Data is imported and exported by using the mysqldump and mysqlimport tools of MySQL. Compared with generic-jdbc-connector, mysql-fastpath-connector delivers a faster data import and export speed.
sftp-connector: SFTP data source connector
oracle-partition-connector: connector that supports the Oracle partition feature, which is used to optimize the import and export of Oracle partition tables.

NOTE:

When an FTP data source connector is used, data is not encrypted. This may result in security risks. You are advised to use an SFTP data source connector.
You are advised to deploy the SFTP server, FTP server, database server, and Loader into separate subnets to ensure secure data import.
For connection to relational databases, general database connectors (generic-jdbc-connector) or dedicated database connectors (oracle-connector, oracle-partition-connector, and mysql-fastpath-connector) are available. However, compared with general database connectors, dedicated database connectors perform better in data import and export because they are optimized for specific database types.
When mysql-fastpath-connector is used, the mysqldump and mysqlimport commands of MySQL must be available on NodeManager nodes, and the MySQL client version to which the two commands belong must be compatible with the MySQL server version. If the two commands are unavailable or the versions are incompatible, install the MySQL client applications and tools following the instructions at http://dev.mysql.com/doc/refman/5.7/en/linux-installation-rpm.html.
When oracle-connector is used, the connection user must be granted the select permission on the following system catalogs or views:
dba_tab_partitions, dba_constraints, dba_tables, dba_segments, v$version, dba_objects, v$instance, SYS_CONTEXT, dba_extents, and dba_tab_subpartitions
When oracle-partition-connector is used, the connection user must be granted the select permission on the following system catalogs: dba_objects and dba_extents.

Introduction to Loader Data Export

Loader is an extract, transform, and load (ETL) tool for exchanging data and files between MRS and relational databases and file systems. You can use the Loader to export data or files from MRS to the relational databases or file systems.

Loader supports the following data export modes:

Exporting data from HDFS or OBS to an SFTP server
Exporting data from HDFS or OBS to a relational database
Exporting data from HBase to an SFTP server
Exporting data from HBase to a relational database
Exporting data from Phoenix tables to an SFTP server
Exporting data from Phoenix tables to a relational database
Exporting data from Hive to an SFTP server
Exporting Data from Hive to a Relational Database
Exporting data from HBase to HDFS or OBS in the same cluster

The MRS needs to connect to the data source to exchange data and files with the external data source. The following connectors are used to configure connection parameters for different types of data sources:

generic-jdbc-connector: relational database connector
hdfs-connector: HDFS data source connector
oracle-connector: specifies a dedicated connector for Oracle databases. row_id serves as partition columns. Compared with generic-jdbc-connector, Map tasks are more evenly distributed on oracle-connector, and whether the partition columns have created indexes does not affect oracle-connector.
mysql-fastpath-connector: specifies a dedicated connector for MySQL databases. Data is imported and exported by using the mysqldump and mysqlimport tools of MySQL. Compared with generic-jdbc-connector, the data import and export speed is faster.
sftp-connector: SFTP data source connector
oracle-partition-connector: connector that supports the Oracle partition feature, which is used to optimize the import and export of Oracle partition tables.

Import Process

You can import and export data on the Loader page. Figure 1 shows the data import process.

Figure 1 Import process
Click to enlarge

You can also use shell scripts to update and run Loader jobs. To use this method, you need to configure the installed Loader client.

Parent topic: Using Loader

Previous topic: Using Loader

Next topic: Loader User Permission Management

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot