EOS Announcement for DLI Spark 2.3.2

Description

Huawei Cloud schedules an end of service (EOS) for DLI Spark 2.3.2 at 00:00 (GMT+08:00) on December 31, 2023.

Impact

After the EOS, no technical support will be provided for DLI Spark 2.3.2. You are advised to select the Spark engine of the latest version when executing jobs. DLI Spark 2.4.5 is recommended.

For jobs that are using DLI Spark 2.3.2, switch to the Spark engine of the latest version as soon as possible. Otherwise, no technical support will be provided if an error occurs during job execution.

If you have any questions or suggestions, please submit a service ticket or call us on +86-4000-955-988 or +86-950-808.

FAQ

• How does the EOS affect the jobs that are using DLI Spark 2.3.2?

  If a queue is created after the EOS of Spark 2.3.2, the compute engine that has reached EOS cannot be selected during job execution.

  Historical queues can still use Spark 2.3.2 to execute jobs. However, if an error occurs during job execution, no technical support is provided. Replace the compute engine with a new version as soon as possible.

• Which version can be used as a replacement after the EOS?

  DLI Spark 2.4.5 is recommended.

• What are the advantages of DLI Spark 2.4.5?

  Table 1 Advantages of Spark 2.4.5

  Feature	Description
  Merging small files	If a large number of small files are generated during SQL execution, job execution and table query will take a long time. In this case, you are advised to merge small files. Merge small files by referring to How Do I Merge Small Files?
  Modifying column comments of non-partitioned or partitioned tables	You can modify the column comments of non-partitioned or partitioned tables.
  Collecting statistics on the CPU usage of SQL jobs	You can view the total CPU used on the console.
  Viewing Spark logs of container clusters	You need to view logs in the container.
  Dynamic UDF loading (OBT)	The UDF takes effect without restarting the queue.
  Supporting flame graphs on the Spark UI	Flame graphs can be created on the Spark UI.
  Optimizing the query performance of the NOT IN statement for SQL jobs	The query performance of the NOT IN statement is improved.
  Optimizing the query performance of the Multi-INSERT statement	The query performance of the Multi-INSERT statement is improved.

  For more advantages, see Spark SQL Upgrade Guide.

• Does the upgrade affect the DLI resource price?

  DLI bills you based on the amount of compute and storage resources consumed by jobs, regardless of the compute engine version.

• How do I upgrade DLI Spark to version 2.4.5?
  1. Log in to the DLI management console. In the navigation pane on the left, choose Job Management > Spark Jobs.
  2. On the Spark Jobs page, locate the row that contains the target job and click Edit in the Operation column.
  3. On the page displayed, select the latest Spark version. Spark 2.4.5 is recommended.