What Can I Do When Receiving java.lang.AbstractMethodError in the Spark Job?

The Spark 2.3 has changed the behavior of the internal interface Logging. If the user code directly inherits the Logging and the earlier version Spark is used during compilation, the java.lang.AbstractMethodError is reported when the application runs in the Spark 2.3 environment.

Solutions are as follows:

• You can recompile the application based on Spark 2.3.
• You can use the sl4j+log4j to implement the log function instead of inheriting the internal interface Logging of the Spark. Details are described as follows:

  <dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-api</artifactId>
    <version>1.7.16</version>
  </dependency>
  <dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-log4j12</artifactId>
    <version>1.7.16</version>
  </dependency>
  <dependency>
    <groupId>log4j</groupId>
    <artifactId>log4j</artifactId>
    <version>1.2.17</version>
  </dependency>
  
  private val logger = LoggerFactory.getLogger(this.getClass)
  logger.info("print log with sl4j+log4j")