Help Center/ MapReduce Service/ Component Operation Guide (LTS)/ Using Loader/ Managing Loader Links
Updated on 2022-11-18 GMT+08:00
View PDF

Managing Loader Links

Scenarios

You can create, view, edit, and delete links on the Loader page.

Prerequisites

Before using a relational database, perform the following operations:
  1. Obtain the JAR package of the relational database driver and save it to the following directory on the active and standby Loader nodes: ${BIGDATA_HOME}/FusionInsight_Porter_8.1.2.2/install/FusionInsight-Sqoop-1.99.3/FusionInsight-Sqoop-1.99.3/server/webapps/loader/WEB-INF/ext-lib.

    The version 8.1.2.2 is used as an example. Replace it with the actual version number.

  2. Run the following command on the active and standby nodes as user root to modify the permission:

    cd ${BIGDATA_HOME}/FusionInsight_Porter_8.1.2.2/install/FusionInsight-Sqoop-1.99.3/FusionInsight-Sqoop-1.99.3/server/webapps/loader/WEB-INF/ext-lib

    chown omm:wheel JAR package name

    chmod 600 JAR package name

  3. Log in to FusionInsight Manager. Choose Cluster > Name of the desired cluster > Services > Loader > More > Restart Service. Enter the password of the administrator to restart the Loader service.

Creating a Connection

  1. Log in to FusionInsight Manager. For details, see Accessing FusionInsight Manager. Choose Cluster > Services.
  2. Select Loader. On the right of Loader WebUI, click the link to open the Loader web UI.
  3. On the Loader page, click New job.
  4. Click Add next to Connection and set connection parameters.

    For details about the parameters, see Loader Connection Configuration.

  5. Click OK.

    If connection configurations, for example, IP address, port, and access user information, are incorrect, the connection will fail to be verified and saved.

    You can click Test to immediately check whether the connection is available.

Viewing a Connection

  1. On the Loader page, click New job.
  2. Click the drop-down list of Connection to view the connections you have created.

Editing a Connection

  1. On the Loader page, click New job.
  2. Select the name of the connection to be edited from the Connection drop-down list.
  3. Click Edit next to Connection.
  4. On the dialog box displayed, modify the connection parameters based on service requirements.
  5. Click Test.

    • If the test is successful, go to 6.
    • If the test fails, repeat 4.

  6. Click Save.

    If a Loader job has integrated into a Loader link, editing the link parameters may affect Loader running.

Deleting a Connection

  1. On the Loader page, click New job.
  2. Select the name of the connection to be deleted from the Connection drop-down list.
  3. Click Delete.
  4. In the displayed dialog box, click OK.

    If a Loader job has integrated a Loader connection, the connection cannot be deleted.

Loader Connection Configuration

Loader supports the following connections:

  • generic-jdbc-connector: For details about parameter settings, see Table 1.
  • ftp-connector: For details about parameter settings, see Table 2.
  • sftp-connector: For details about parameter settings, see Table 3.
  • hdfs-connector: For details about parameter settings, see Table 4.
  • oracle-connector: For details about parameter settings, see Table 5.
  • mysql-fastpath-connector: For details about parameter settings, see Table 7.
  • oracle-partition-connector: For details about parameter settings, see Table 6.
Table 1 generic-jdbc-connector configuration

Parameter

Description

Name

Name of a Loader connection

Connector

Select generic-jdbc-connector.

JDBC Driver Class

JDBC driver classes are as follows:

  • oracle: oracle.jdbc.driver.OracleDriver
  • SQLServer: com.microsoft.jdbc.sqlserver.SQLServerDriver
  • mysql: com.mysql.jdbc.Driver
  • postgresql: org.postgresql.Driver
  • gaussdb200: com.huaweixxx.gauss200.jdbc.Driver

JDBC Connection String

Database access address, which can be an IP address or domain name.

Enter the database connection string. In the following examples, 10.10.10.10 indicates the IP address and test indicates the database name.

  • oracle: jdbc:oracle:thin:@10.10.10.10:1521:orcl
  • SQLServer: jdbc:microsoft:sqlserver://10.10.10.10:1433;DatabaseName=test
  • mysql: jdbc:mysql://10.10.10.10/test?&useUnicode=true&characterEncoding=GBK
  • postgresql: jdbc:postgresql://10.10.10.10:5432/test
  • gaussdb200: jdbc:gaussdb://10.10.10.10:15400/test (15400 is an example port number.)

Username

Username for accessing the database

Password

Password of the user. Use the actual password.
Table 2 ftp-connector configuration

Parameter

Description

Name

Name of a Loader connection

Connector

Select ftp-connector.

FTP Mode

Select ACTIVE or PASSIVE.

FTP Protocol

Select:

  • FTP
  • SSL_EXPLICIT
  • SSL_IMPLICIT
  • TLS_EXPLICIT
  • TLS_IMPLICIT

File Name Encoding Type

Encoding type of the file name or file path.
Table 3 sftp-connector configuration

Parameter

Description

Name

Name of a Loader connection

Connector

Select sftp-connector.
Table 4 hdfs-connector configuration

Parameter

Description

Name

Name of a Loader connection

Connector

Select hdfs-connector.
Table 5 oracle-connector configuration

Parameter

Description

Name

Name of a Loader connection

Connector

Select oracle-connector.

JDBC Connection String

Connection string used for connecting to the database

Username

Username for accessing the database

Password

Password of the user. Use the actual password.
Table 6 oracle-partition-connector configuration

Parameter

Description

Name

Name of a Loader connection

Connector

Select oracle-partition-connector.

JDBC Driver Class

Enter com.microsoft.jdbc.sqlserver.SQLServerDriver.

JDBC Connection String

Connection string used for connecting to the database

Username

Username for accessing the database

Password

Password of the user. Use the actual password.
Table 7 mysql-fastpath-connector configuration

Parameter

Description

Name

Name of a Loader connection

Connector

Select mysql-fastpath-connector.

NOTICE:

When mysql-fastpath-connector is used, the mysqldump and mysqlimport commands of MySQL must be available on NodeManagers, and the MySQL client version to which the two commands belong must be compatible with the MySQL server version. If the two commands are unavailable or the versions are incompatible, see http://dev.mysql.com/doc/refman/5.7/en/linux-installation-rpm.html. Install the MySQL client applications and tools.

For example, the following RPM packages need to be installed on the RHEL-x86 system (select a version based on the site requirements):

  • mysql-community-client-5.7.23-1.el7.x86_64.rpm
  • mysql-community-common-5.7.23-1.el7.x86_64.rpm
  • mysql-community-devel-5.7.23-1.el7.x86_64.rpm
  • mysql-community-embedded-5.7.23-1.el7.x86_64.rpm
  • mysql-community-libs-5.7.23-1.el7.x86_64.rpm
  • mysql-community-libs-compat-5.7.23-1.el7.x86_64.rpm

JDBC Connection String

Connection string used for connecting to the database

Username

Username for accessing the database

Password

Password of the user. Use the actual password.
Parent topic: Using Loader