Error Message "Code of method ... grows beyond 64 KB" Is Displayed When a Complex SQL Statement Is Executed

Question

When executing a deeply nested or highly complex SQL query, such as one with multiple layers of subqueries or extensive use of CASE WHEN logic, the system displays the following error log. The error indicates that the code of a method exceeds 64 KB (allowed code size limit).

java.util.concurrent.ExecutionException: java.lang.Exception: failed to compile: org.codehaus.janino.JaninoRuntimeException: Code of method "(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificUnsafeProjection;Lorg/apache/spark/sql/catalyst/InternalRow;)V" of class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection" grows beyond 64 KB

Answer

When Spark's Tungsten engine is enabled, it uses Whole-Stage Java Code Generation to compile parts of the execution plan into optimized Java code. However, there is a limit that each compiled method must not exceed 64 KB in bytecode size. When executing a deeply nested or highly complex SQL query, such as one with multiple layers of subqueries or extensive use of CASE WHEN logic, a very large Java method may be generated using Whole-Stage Java Code Generation. If the compiled method exceeds 64 KB, the compilation fails.

Workaround:

If this issue occurs, you can disable the Whole-Stage Java Code Generation function by disabling the Tungsten engine to ensure that queries can executes successfully. Set spark.sql.codegen.wholeStage to false in the {Client installation directory}/Spark/spark/conf/spark-defaults.conf configuration file on the client.