An Error Is Reported When the sqoop import Command Is Executed to Extract Data from PgSQL to Hive

Background

A user runs the sqoop import command to extract data from the open-source PgSQL database to MRS HDFS or Hive.

Issue

The sqoop command can be executed to query the PgSQL database table, but an error is reported when the sqoop import command is executed to import data.

The authentication type 5 is not supported. Check that you have configured the pg_hba.conf file to include the client's IP address or subnet.

Cause Analysis

1. MD5 authentication for connecting to PgSQL fails. A whitelist needs to be configured in the pg_hba.conf file.
2. When the sqoop import command is executed, a MapReduce job is started. The PgSQL driver package gsjdbc4-*.jar exists in the MRS Hadoop installation directory, for example, /opt/Bigdata/FusionInsight_HD_*/1_*_NodeManager/install/hadoop/share/hadoop/common/lib, which is incompatible with the open-source PgSQL service. As a result, an error is reported.

Procedure

1. Configure a whitelist in the pg_hba.conf file.
2. Delete the gsjdbc4 jar packages from all core nodes, and add the PgSQL JAR package to sqoop/lib.

   mv /opt/Bigdata/FusionInsight_HD_*/1_*_NodeManager/install/hadoop/share/hadoop/common/lib/gsjdbc4-*.jar /tmp