Help Center/ Data Lake Insight/ Product Bulletin/ Version Support Bulletin/ What's New in Spark 3.1.1
Updated on 2024-08-19 GMT+08:00
View PDF

What's New in Spark 3.1.1

DLI complies with the release consistency of the open source Spark compute engine. This section describes the updates in Spark 3.1.1.

For more information about Spark 3.1.1, see Spark Release Notes.

Spark 3.1.1 Release Date

Version

Release Date

Status

EOM Date

EOS Date

DLI Spark 3.1.1

December 2021

Released

December 31, 2023

December 31, 2024

For more version support information, see Lifecycle of DLI Compute Engine Versions.

Spark 3.1.1 Description

The following lists the main features of Spark 3.1.1.

For more new features, see Release Notes - Spark 3.1.1.

  • [SPARK-33050]: Upgraded Apache ORC to version 1.5.12.
  • [SPARK-33092]: Improved subexpression elimination.
  • [SPARK-33480]: Added support for the char/varchar data type.
  • [SPARK-32302]: Optimized the pushdown of some predicates.
  • [SPARK-30648]: Added support for the pushdown of predicates in JSON datasource tables.
  • [SPARK-32346]: Added support for the pushdown of predicates in Avro datasource tables.
  • [SPARK-32461]: Optimized the Shuffle Hash Join algorithm.
  • [SPARK-32272]: Added the SQL-standard command SET TIME ZONE.
  • [SPARK-21492]: Fixed memory leak caused by the sort-merge join algorithm.
  • [SPARK-27812]: Upgraded the Kubernetes client to version 4.6.1.

    DLI does not support built-in geospatial query functions since Spark 3.x.

Parent topic: Version Support Bulletin