Help Center/ MapReduce Service/ Developer Guide (Normal_3.x)/ Spark2x Development Guide (Security Mode)/ FAQs About Spark Application Development/ Common JAR File Conflicts
Updated on 2024-10-23 GMT+08:00
View PDF
Share

Common JAR File Conflicts

Symptom

Spark can interconnect with many third-party tools. Therefore, it depends on a large number of third-party packages. Some packages are provided by MRS. As a result, the versions of the JAR files used by the code may be different from those of the JAR files provided by the cluster. In this case, JAR file conflicts may occur.

Common JAR file conflict errors are as follows:

1. A class cannot be found: java.lang.NoClassDefFoundError

2. A method cannot be found: java.lang.NoSuchMethodError

Cause Analysis

The following uses UDF customization as an example:

If the class cannot be found, do as follows:

  1. Check whether the JAR file of the class is in the classpath of the JVM. The JAR files of the Spark are stored in the Spark client directory/jars/ directory.
  2. Check whether multiple JAR files contain the class.
  3. Check whether other dependencies are added to the task by using --jars.
  4. If the dependencies are added to the task but cannot be found, the driver of the configuration file or the classpath of the executor may be incorrectly configured. View logs to check whether the configuration file is loaded to the environment.
  5. Check whether the error is caused by a class initialization failure that occurs before using the class.

If the method cannot be found, do as follows:

  1. Check whether the JAR file of the class corresponding to the method is loaded to the classpath of the JVM. The classes of the Spark are stored in the Spark client directory/jars/ directory.
  2. Check whether multiple JAR files contain the class. (Pay special attention to different versions of the same tool.)
  3. If the error is reported for a Hadoop package, the possible cause is that the Hadoop version is inconsistent. As a result, some methods have been modified.
  4. If the error is reported for a third-party package, the possible cause is that the Spark has the related JAR file, but the version is different from that in the code.

Procedure

Solution 1:

Check whether third-party tool packages are required. If the packages can be replaced with the packages of the same version in the cluster, modify dependency versions.

You are advised to use the dependencies provided by the MRS cluster.

Solution 2:

The following shows how to modify a JAR file version:

MRS_2.1 is used as an example.

  1. Add the <properties> parameter to the pom.xml file and set the variables to facilitate subsequent version modification.

  2. Use the preceding defined variables to set the version of each JAR file in the dependencies parameter.

    If other third-party packages conflict, check whether the same package of different versions exists by querying the dependency relationship. If yes, replace the JAR file version with the version of the JAR file provided by the cluster.

    You can obtain sample projects from Huawei Mirrors by referring to the pom.xml file of the MRS sample project.

  3. Print the dependency tree.

    Run the mvn dependency:tree command in the directory where the pom.xml file is stored.