Configuring Spark JDBC to Load Third-Party JAR Packages

Scenarios

When you execute Spark jobs using JDBC, you might use third-party JAR packages not integrated by default in the MRS cluster; in such cases, the JDBC job logs will display the "ClassNotFoundException" error.

Notes and Constraints

This section applies only to MRS 3.6.0-LTS and later.

Prerequisites

• Only JAR packages of the same version can be configured for the same Spark service or Spark instance.
• Non-customized, open-source JAR files, including but not limited to org.apache.*, com.google.*, and javax.*, cannot be stored in the directory.

Procedure

1. Log in to FusionInsight Manager.

   For details, see Accessing FusionInsight Manager.

2. Choose Cluster > Services > Spark, click Instances, and record the IP addresses of all JDBCServer roles.
3. Log in to each node where the JDBCServer role is deployed and upload the dependent third-party JAR packages to the ${BIGDATA_HOME}/third_lib/Spark${idx} directory on the node.
   

   • ${idx} indicates the sequence number for the multi-instance or multi-service.
   • The Spark multi-instance uses 1 to 4.
   • The Spark multi-service uses -1, -2, -3, and others.

4. To apply the settings:
   • Multi-tenant mode: Do not need to restart the JDBCServer role instance. The number of JDBC applications is adjusted based on the driver load. For a newly started JDBC application, the settings take effect immediately. For the already started JDBC application, the settings do not take effect; you need to manually stop the application and start a new one.
   • Multi-instance mode: Restart the JDBCServer role instance on Manager to apply the changes.