An Error Is Reported When Database Data Is Migrated Using Sqoop

Issue

• The following error is reported when database data is migrated using MRS Sqoop:

  Communications link failure;
  The driver has not received any packets from the server;

• If the migration source is a GaussDB(DWS) database, the following error message is displayed:

  got exception running sqoop .java.lang.Runtime.Exception, could not load db driver class.

Figure 1 Error reported when database data is migrated using Sqoop

Figure 2 Error reported when the migration source is GaussDB(DWS)

Cause Analysis

According to the preceding error information, the possible cause is that the JDBC driver is not correctly loaded on the server or the network is faulty. During data migration using Sqoop, MapReduce jobs run on analysis core nodes. Therefore, ensure that the driver class is properly loaded and the network communication between the source database and cluster nodes is normal.

Procedure

1. Check whether the source database port is enabled for all cluster nodes.

   If EIPs are used, EIPs must be bound to all cluster nodes, especially the analysis nodes, to ensure normal job running.

   

   Generally, Sqoop instances are distributed on the master node, but jobs are running on the analysis nodes. Therefore, creating a database connection is to communicate with the master node. Jobs are distributed to the analysis nodes through Yarn tasks.

2. Check whether the JAR file of the database driver exists in the installation directory of the Sqoop instances. Ensure that the path of the JAR file is correct, the file permission, owner, and owner group are correct, and the configuration file is correctly configured. Then, restart the Sqoop instances. In this way, the driver can be loaded properly.
   • Driver path for MRS 3.x clusters: /opt/Bigdata/FusionInsight_Current/1_xx_LoaderServer/install/FusionInsight-Sqoop-1.99.3/server/webapps/loader/WEB-INF/ext-lib/
   • Driver path for versions earlier than MRS 3.x: /opt/Bigdata/MRS_XXX/install/FusionInsight-Sqoop-1.99.7/FusionInsight-Sqoop-1.99.7/server/jdbc/
     

     For versions earlier than 3.x, you need to modify the configuration file.

3. If you submit jobs using the CLI, you are advised to specify the --driver parameter.

   If the data source is GaussDB(DWS), add --driver com.xxx.gauss200.jdbc.Driver to the CLI.

   Figure 3 Configuration of the --driver parameter when the data source is GaussDB(DWS)
   

   For other databases, configure the --driver parameter based on site requirements.