Common JAR File Conflicts

Updated on 2024-10-23 GMT+08:00

View PDF

Symptom

Spark can interconnect with many third-party tools. Therefore, it depends on a large number of third-party packages. Some packages are provided by MRS. As a result, the versions of the JAR files used by the code may be different from those of the JAR files provided by the cluster. In this case, JAR file conflicts may occur.

Common JAR file conflict errors are as follows:

1. A class cannot be found: java.lang.NoClassDefFoundError

2. A method cannot be found: java.lang.NoSuchMethodError

Cause Analysis

The following uses UDF customization as an example:

If the class cannot be found, do as follows:

Check whether the JAR file of the class is in the classpath of the JVM. The JAR files of the Spark are stored in the Spark client directory/jars/ directory.
Check whether multiple JAR files contain the class.
Check whether other dependencies are added to the task by using --jars.
If the dependencies are added to the task but cannot be found, the driver of the configuration file or the classpath of the executor may be incorrectly configured. View logs to check whether the configuration file is loaded to the environment.
Check whether the error is caused by a class initialization failure that occurs before using the class.

If the method cannot be found, do as follows:

Check whether the JAR file of the class corresponding to the method is loaded to the classpath of the JVM. The classes of the Spark are stored in the Spark client directory/jars/ directory.
Check whether multiple JAR files contain the class. (Pay special attention to different versions of the same tool.)
If the error is reported for a Hadoop package, the possible cause is that the Hadoop version is inconsistent. As a result, some methods have been modified.
If the error is reported for a third-party package, the possible cause is that the Spark has the related JAR file, but the version is different from that in the code.

Procedure

Solution 1:

Check whether third-party tool packages are required. If the packages can be replaced with the packages of the same version in the cluster, modify dependency versions.

NOTE:

You are advised to use the dependencies provided by the MRS cluster.

Solution 2:

The following shows how to modify a JAR file version:

MRS_2.1 is used as an example.

Add the <properties> parameter to the pom.xml file and set the variables to facilitate subsequent version modification.
Use the preceding defined variables to set the version of each JAR file in the dependencies parameter.

If other third-party packages conflict, check whether the same package of different versions exists by querying the dependency relationship. If yes, replace the JAR file version with the version of the JAR file provided by the cluster.

You can obtain sample projects from Huawei Mirrors by referring to the pom.xml file of the MRS sample project.
Print the dependency tree.
Run the mvn dependency:tree command in the directory where the pom.xml file is stored.

Parent topic: FAQs About Spark Application Development

Previous topic: What Should I Do If the Structured Streaming Task Submission Way Is Changed?

Next topic: Spark2x Development Guide (Normal Mode)

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot