Help Center>
Data Lake Insight>
FAQs>
Flink Jobs>
Flink Jar Jobs>
Why Does the Flink Submission Fail Due to Hadoop JAR File Conflict?
Updated on 2023-05-19 GMT+08:00
Why Does the Flink Submission Fail Due to Hadoop JAR File Conflict?
Symptom
Flink Job submission failed. The exception information is as follows:
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.obs.metrics.OBSAMetricsProvider not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2664) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2688) ... 31 common frames omitted Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.obs.metrics.OBSAMetricsProvider not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2568) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2662) ... 32 common frames omitted
Cause Analysis
Flink JAR files conflicted. The submitted Flink JAR file conflicted with the HDFS JAR file of the DLI cluster.
Procedure
- Configure hadoop-hdfs in the POM file as follows:
<dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs</artifactId> <version>${hadoop.version}</version> <scope> provided </scope> </dependency>
Alternatively, use the exclusions tag to exclude the association.
- To use HDFS configuration files, change core-site.xml, hdfs-site.xml, and yarn-site.xml to mrs-core-site.xml, mrs-hdfs-site.xml and mrs-hbase-site.xml, respectively.
conf.addResource(HBaseUtil.class.getClassLoader().getResourceAsStream("mrs-core-site.xml"), false); conf.addResource(HBaseUtil.class.getClassLoader().getResourceAsStream("mrs-hdfs-site.xml"), false); conf.addResource(HBaseUtil.class.getClassLoader().getResourceAsStream("mrs-hbase-site.xml"), false);
Parent topic: Flink Jar Jobs
Flink Jar Jobs FAQs
- How Do I Configure Checkpoints for Flink Jar Jobs and Save the Checkpoints to OBS?
- Can I Upload Configuration Files for Flink Jar Jobs?
- Why Does a Flink Jar Package Conflict Result in Submission Failure?
- Why Does a Flink Jar Job Fail to Access GaussDB(DWS) and a Message Is Displayed Indicating Too Many Client Connections?
- Why Is Error Message "Authentication failed" Displayed During Flink Jar Job Running?
- Why Is Error Invalid OBS Bucket Name Reported After a Flink Job Submission Failed?
- Why Does the Flink Submission Fail Due to Hadoop JAR File Conflict?
- How Do I Connect a Flink jar Job to SASL_SSL?
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
The system is busy. Please try again later.
For any further questions, feel free to contact us through the chatbot.
Chatbotmore