On this page

What Are DLI Datasource V1 and V2 Tables?
Compatibility of DLI Spark Versions with V1 and V2 Tables
How Do I Confirm If a User-Created Table is a V1 or V2 Table?
Example Upgrade

Show all

Help Center/ Data Lake Insight/ Product Bulletin/ Version Support Bulletin/ Differences Between Spark 2.4.x and Spark 3.3.x/ DLI Datasource V1 Table and Datasource V2 Table

DLI Datasource V1 Table and Datasource V2 Table

Updated on 2025-01-09 GMT+08:00

View PDF

What Are DLI Datasource V1 and V2 Tables?

DLI datasource V1 table (referred to as V1 table): This is a DLI-specific datasource table format. DLI's custom create/insert/truncate commands are used, and the data path of the table is $tablepath/UUID/Data file.
Figure 1 DLI datasource v1 table
DLI datasource V2 table (referred to as V2 table): This is the open-source datasource table format of Spark. Spark's open-source create/insert/truncate commands are used, and the data path of the table is $tablepath/Data file.
Figure 2 DLI datasource v2 table

Compatibility of DLI Spark Versions with V1 and V2 Tables

**Table 1** Compatibility of DLI Spark versions with v1 and v2 tables
Table Type	Spark 2.3 SQL Queue	Spark 2.3 General-Purpose Queue	Spark 2.4 SQL Queue	Spark 2.4 General-Purpose Queue	Spark 3.1 SQL Queue	Spark 3.1 General-Purpose Queue	Spark 3.3 SQL Queue	Spark 3.3 General-Purpose Queue
V1 table	√	√	√	√	√	√	√	Partially supported
V2 table	×	×	√	√	×	×	√	√

**Table 2** Syntax support list for Spark 3.3 general-purpose queues
Table Type	select	create table	create table like	CTAS	insert into	insert overwrite	load data	alter table set location	truncate table
V1 table	√	√	√	×	×	×	×	×	×
V2 table	√	√	√	√	√	√	√	√	√

How Do I Confirm If a User-Created Table is a V1 or V2 Table?

1. Use the datasource syntax to create a table:

CREATE TABLE IF NOT EXISTS table_name (id STRING) USING parquet;

2. Run show create table to check the value of the version field under TBLPROPERTIES.

If v1, it is a V1 table; if v2, it is a V2 table.

To change a V1 table to a V2 table, submit a service ticket to contact customer support.

Example Upgrade

NOTE:

Upgrading the Spark engine and modifying data tables may cause changes in the cost of billed resources if the type of compute resource changes when creating a queue.

If the original queue uses compute resources of the elastic resource pool type, creating a queue does not involve changes in the cost of compute resources.
If the original queue uses compute resources of a non-elastic resource pool type, creating a queue within an elastic resource pool will change the cost of compute resources. Refer to the price details of compute resources for specifics.

Example 1: Does upgrading Spark from version 2.4.x to Spark 3.3.1 affect the version of data tables when using a SQL queue?
No, SQL queues in Spark 2.4.x support V1 and V2 tables, so upgrading Spark only requires considering the compatibility of the Spark version with SQL syntax.

Example 2: Does upgrading Spark from version 2.4.x to Spark 3.3.1 affect the version of data tables when using a general-purpose queue?

General-purpose queues in Spark 2.4.x support V1 and V2 tables, but general-purpose queues in Spark 3.3.x do not support V1 tables.

Therefore, to upgrade Spark from version 2.4.x to 3.3.1, follow these steps:

Change V1 tables in Spark 2.4.x to V2 tables.

Upgrade V2 tables in Spark 2.4.x to V2 tables in Spark 3.3.1.

Consider the compatibility of Spark Jar job API syntax as well.

**Table 3** Compatibility of DLI Spark versions with v1 and v2 tables
Table Type	Spark 2.4 General-Purpose Queue	Spark 3.3 General-Purpose Queue
V1 table	√	Partially supported
V2 table	√	√

Example 3: How do I upgrade V1 tables in Spark 2.3.2 to V2 tables in Spark 3.3.1 using a general-purpose queue?

General-purpose queues in Spark 2.3.2 do not support V2 tables, and general-purpose queues in Spark 3.3.1 do not support V1 tables.

Upgrade V1 tables in Spark 2.3.2 to V1 tables in Spark 2.4.5.
Change V1 tables in Spark 2.4.5 to V2 tables.
Upgrade V2 tables in Spark 2.4.5 to V2 tables in Spark 3.3.1.
Consider the compatibility of Spark Jar job API syntax as well.

**Table 4** Compatibility of DLI Spark versions with v1 and v2 tables
Table Type	Spark 2.3 General-Purpose Queue	Spark 2.4 General-Purpose Queue	Spark 3.3 General-Purpose Queue
V1 table	√	√	Partially supported
V2 table	×	√	√

Parent topic: Differences Between Spark 2.4.x and Spark 3.3.x

Previous topic: Differences in General-Purpose Queues Between Spark 2.4.x and Spark 3.3.x

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot