Checking ShareLib

Prerequisites

• You have installed the HDFS and Oozie clients.
• To check Spark ShareLib, you need to install the Spark client on the node where the Oozie client is located.
• The user who performs the check must have the common user permission of Oozie and the permission to access the /user/oozie directory of HDFS.

Procedure

1. Log in to the node where the client is installed as the client installation user.
2. Run the following command to go to the client installation directory:

   cd Client installation directory

3. Run the following commands to configure environment variables and authenticate the user:

   source bigdata_env

   kinit User who submits Oozie tasks (Skip this step for normal clusters.)

4. Check ShareLib by checking the client or server. Spark ShareLib can be checked only by checking the client.
   • Checking the client:
     • Check Oozie ShareLib and ensure that an Oozie instance is installed on the node where the Oozie client to be checked resides.

       oozie -validatesharelib -oozie.core.path=Oozie instance installation path

       The following is an example:

       oozie -validatesharelib -oozie.core.path=${BIGDATA_HOME}/FusionInsight_Porter_*/install/FusionInsight-Oozie-*/oozie-*

     • Check Spark ShareLib.

       oozie -validatesharelib -spark.client.path=Spark client installation directory

       The following is an example:

       oozie -validatesharelib -spark.client.path=/opt/client/Spark/

   • Checking the server:

     Run the following command to check Oozie ShareLib:

     oozie job -oozie https://Host name of the Oozie role:21003/oozie -validatesharelib

     To view the host name of the oozie role, choose Cluster > Services > Oozie and click the Instance tab on FusionInsight Manager.

     21003 is the running port of Oozie HTTPS requests. To view the port, log in to FusionInsight Manager, choose Cluster > Services > Oozie and click the Configuration tab. Search for OOZIE_HTTPS_PORT.

5. View check results. The following circumstances are included:
   • Some ShareLib JAR packages are missing.

     If some JAR packages are missing, message "Share Lib jar file(s) not found on hdfs:" and information about the missing JAR packages are displayed.

     If no ShareLib JAR package is missing, message "All Share Lib jar file(s) found on hdfs." is displayed.

   • Some JAR packages are damaged.

     If damaged JAR packages are detected, message "Share Lib jar file(s) mismatch on hdfs:" and information about the damaged JAR packages are displayed.

     If no ShareLib JAR package is damaged, message "All Share Lib jar file(s) on hdfs match." is displayed.

   • Custom JAR packages are uploaded.

     If custom JAR packages are detected, message "Extra Share Lib jar file(s) found on hdfs:" and information about the custom JAR packages are displayed.

     If no custom JAR package is detected, message "No extra Share Lib jar file(s) found on hdfs." is displayed.

6. Rectify the fault based on the check results.

   If the detection result obtained in 5 contains information indicating JAR packages are missing or damaged, perform the following operations:

   • Spark ShareLib:

     Upload the Spark JAR package in the Spark client installation directory/spark/jars directory to the HDFS path in the check result.

     hdfs dfs -put -f Local JAR package path HDFS path that Spark JAR packages are missing or damaged

   • Oozie ShareLib:
     1. Decompress the oozie-sharelib-*.tar.gz file under Oozie installation path ${BIGDATA_HOME}/FusionInsight_Porter_*/install/FusionInsight-Oozie-*/oozie-*/ and find the ShareLib JAR package.

        tar -zxf oozie-sharelib-*.tar.gz

     2. Upload the obtained Oozie JAR package to the HDFS path in the check result.

        hdfs dfs -put -f Local JAR package path HDFS path that Oozie JAR packages are missing or damaged