How Do I Do If An Error Is Reported During Database Data Migration Using Sqoop?

Question

• An error is reported when database data is migrated using MRS Sqoop.

  For example, the following error messages are displayed:

  Communications link failure;

  The driver has not received any packets from the server;

• If the data source is GaussDB(DWS), an error message similar to "got exception running sqoop .java.lang.Runtime.Exception, could not load db driver class" is displayed.

Figure 1 Error messages displayed during database data migration using Sqoop

Figure 2 Error messages displayed when the data source is GaussDB(DWS)

Fault Locating

According to the preceding error information, the possible cause is that the JDBC driver is not correctly loaded on the server or the network is faulty. During data migration using Sqoop, MapReduce jobs run on analysis core nodes. Therefore, ensure that the driver class is properly loaded and the network communication between the source database and cluster nodes is normal.

Troubleshooting

1. Check whether the source database port is enabled for all cluster nodes.

   If EIPs are used, EIPs must be bound to all cluster nodes, especially the analysis nodes, to ensure normal job running.

   

   Sqoop instances are typically located on master nodes, but jobs run on analysis core nodes. Therefore, database connections are used for communication with master nodes.

   Jobs are distributed to analysis nodes through YARN tasks.

2. Check whether the JAR file of the database driver exists in the installation directory of the Sqoop instances. Ensure that the path of the JAR file is correct, the file permission, owner, and owner group are correct, and the configuration file is correctly configured. Then, restart the Sqoop instances. In this way, the driver can be loaded properly.
   • Driver path for MRS 3.x clusters: /opt/Bigdata/FusionInsight_Current/1_xx_LoaderServer/install/FusionInsight-Sqoop-1.99.3/server/webapps/loader/WEB-INF/ext-lib/
   • Driver path for versions earlier than MRS 3.x: /opt/Bigdata/MRS_XXX/install/FusionInsight-Sqoop-1.99.7/FusionInsight-Sqoop-1.99.7/server/jdbc/
     

     For versions earlier than 3.x, you need to modify the configuration file.

3. If you submit jobs using the CLI, you are advised to specify the --driver parameter.

   If the data source is GaussDB(DWS), add --driver com.huawei.gauss200.jdbc.Driver to the CLI.

   Figure 3 Configuration of the --driver parameter when the data source is GaussDB(DWS)
   

   For other databases, configure the --driver parameter based on site requirements.