DWS-Connector Version Description

1.0

Initial version.

dws-connector-flink only releases Scala2.11 Flink 1.12.

1.0.2

Optimized the exception retry logic of dwsclient. The retry mode is changed from retry upon all exceptions to retry only upon five types of exceptions: connection exception, database read-only, timeout, excessive connections, and lock exception.

Compatible versions of dws-connector-flink:

Scala2.11: Flink 1.12 and 1.13

Scala2.12: Flink 1.12, 1.13, and 1.15

1.0.3

1. Resolved known issues and optimized performance.
2. Added the update write mode.
3. Added the use of unique indexes.
4. The update mode is supported. To avoid ambiguity of the upsert interface in dwsClient, the write interface is used for write operations. The write interface is recommended for both the two writes.

-

1.0.4

Increased the SQL execution timeout interval to avoid long-time blocking.

-

1.0.5

Resolved the problem that duplicated data is lost when being written to a table without a primary key.

-

1.0.6

1. Optimized the cache saving logic to improve the throughput when the CPU is insufficient.
2. Temporary tables are reused to prevent frequent creation of temporary tables in COPY MERGE/UPSERT scenarios.
3. The CSV format is added for COPY to avoid that complex data cannot be imported to the database due to special characters.

-

1.0.7

1. Retry is supported after data fails to be written during database restart.
2. The AS mode is added to create temporary tables to solve the problem that COPY MERGE/UPSERT cannot be used in tables with primary keys.
3. The database fields are case-insensitive by default.
4. The primary key printing parameter is added to Flink SQL statements to locate problems when data is missing.

-

1.0.8

1. Fixed the problem that the case of the Flink SQL primary key must be the same as that in the database.
2. Added the parameter for setting sink concurrency.

-

1.0.9

Optimized the import of data of the time type.

-

1.0.10

1. Resolved the issue of data loss caused by concurrent delete and insert operations on the client. This could happen when the insert operation ran before the delete operation, and the same primary key was deleted and then inserted in the same cache batch.
2. Resolved the issue that data is occasionally lost after an exception occurs when Kafka writes data to DWS.
3. The connector parameter ignoreUpdateBefore is added. Some main parameters are compatible with flink-connector-jdbc.

-

1.0.11

1. The DWS client write API validates database fields against the input schema, automatically adding any missing fields.
2. Configuration options are available for comparison fields, and new data is only updated if it exceeds the existing database value.
3. Flink SQL also enables logical deletion.

Scala 2.12 is added to connector-flink: Flink 1.17

1.1.0

Optimized the cache write performance of the DWS client.

-

1.1.0.1

1. The client adjusts to the bit type and saves it to the database.
2. The client sets the nvarchar value to null and saves it to the database.

-

1.1.0.2

Fixed issues:

1. Fixed the issue where an error occurred when a table column was in uppercase.
2. Resolved the problem with upsert causing errors during data import to a database with a bigint type, specifically when the first data record was of int type and subsequent records were of long type exceeding the int range.

-

1.1.0.3

Fixed issues:

1. Fixed the error when reading binlog with capitalized table names.
2. Resolved the issue of errors occurring when the source data contained \u0000 in copy mode.

-

2.0.0-r0~r2

In the 2.x version initialization, DWS made the following enhancements:

1. Enabled direct import of connected DNs to the database.
2. Revamped the cache model by introducing partition-level cache at the table level, allowing multiple caches per table.
3. Implemented support for the properties configuration file mode during client initialization.
4. Ensured that operations like upsert and merge can only proceed after the previous data batch is fully imported to maintain the sequence of data import to the database consistent with writing data to the cache.
5. Implemented data type conversion migration to the write cache, enabling both copy and upsert functionalities.
6. Schemas, table names, table columns, and uppercase characters could not be read.

-

2.0.0-r3

Fixed issues:

1. The table name read by Flink SQL binlog did not exist, leading to the exhaustion of DWS cluster connections.
2. A single data record could be lost in case of a DWS fault during the import of Flink SQL statements to the database.

Improved features:

1. Introduced import delay and import speed indicators for Flink SQL statements and APIs.
2. Implemented automatic generation of a unique index when no primary key is present but a unique index exists.
3. Added a label for JDBC connections.
4. Enhanced the logic for converting time type fields.
5. Updated to utilize the Map interface within the context of the DwsInvokeFunction interface.

-

2.0.0.r4

Resolved the issue that an error is reported during data format parsing when the field sequence in Flink SQL is inconsistent with that in the physical table.

-

2.0.0.5

Fixed issues:

1. An error is reported during dimension table serialization.
2. The lookupBatchScanEnable parameter is added to support batch read of dimension tables.
3. An error occurs in binlog-based aggregation calculation due to repeated consumption at the t point.

Improved features:

1. The Flink SQL binlog capability is enhanced. The synchronization point is registered in a timely manner, the BinlogQueueSize size is printed periodically, the binlogSkipFullSync parameter is provided to skip the full synchronization phase, and the binlogCondition parameter is provided as a filter condition.

The version number rules of Maven are used. The version number does not contain letters and is named using digits and dots.

2.0.0.6

Fixed issues:

1. The initialization is suspended when cacheType is set to ALL for dimension tables.
2. When DNs are directly connected, table metadata is not queried using indexes, resulting in poor performance.
3. Flink checkpoint takes a long time.

-

2.1.0.1

Fixed issues:

1. The PgConnection connection type is incorrectly converted.
2. After direct connection to DNs is enabled, the error message "relation 'public.pgxc_node_ips' does not exist" is displayed.
3. Some types (such as nvarchar2) are incorrectly converted.
4. An error is reported for the Flink TIMESTAMP_LTZ type.
5. An error is reported when the source table contains timestamp data to be synchronized.
6. Duplicate data exists because the temporary table fails to be truncated.
7. No node group is specified when a temporary table is merged.
8. The performance is poor when a temporary table lacks orientation and distribution columns.
9. The DELETE operation is overwritten by the UPSERT operation after being imported to the downstream.
10. In COPY import mode, data inconsistency occurs when the syntax null 'null' is used.
11. When a table with a primary key is imported using COPY, the import mode is changed to COPY_UPSERT.

Improved features:

1. During full binlog synchronization, if executeDirectOn is enabled, tables are directly queried using SQL statements.
2. The direct connection to DNs feature has been refactored to use a dynamic library for the process of hash calculation and mapping to DNs.
3. Dimension tables support condition pushdown.

Direct connection to DNs no longer depends on the UDF capability.

1.1.0.1 and earlier

8.1.3

1.1.0.2, 1.1.0.3

8.2.1.300-200

2.0.0-r0~2.0.0.6

8.2.1.300-200

2.1.0.1

8.6.1-200

1.0

Flink SQL + Stream API

Scala 2.11: Flink 1.12

dws-connector-flink_${scala.version}_${flink.version}

Not supported

Addresses for Downloading Dependency Packages

1.0.2-1.0.10

Flink SQL + Stream API

• Scala 2.11: Flink 1.12, 1.13
• Scala 2.12: Flink 1.12, 1.13, 1.15

dws-connector-flink_${scala.version}_${flink.version}

In 1.0.7 and later versions, only database fields are case-insensitive.

1.0.11-1.1.0.3

Flink SQL + Stream API

• Scala 2.11: Flink 1.12, 1.13
• Scala 2.12: Flink 1.12, 1.13, 1.15, 1.17

dws-connector-flink_${scala.version}_${flink.version}

Only database fields are case-insensitive.

2.0.0-r0

Only Stream API/Flink SQL, compatible with Stream API

Scala 2.12: 1.15

• dws-connector-flink
• dws-connector-flink-sql-${flink.version}

Database fields, table names, and schemas can be read in uppercase.

2.0.0-r1 to r4

Only Stream API/Flink SQL, compatible with Stream API

• Scala 2.11: Flink 1.12, 1.13
• Scala 2.12: Flink 1.12, 1.13, 1.15, 1.17, 1.20

• dws-connector-flink
• dws-connector-flink-sql-${flink.version}

Database fields, table names, and schemas can be read in uppercase.

2.0.0.5-2.1.0.1

Only Stream API/Flink SQL, compatible with Stream API

Scala 2.12: Flink 1.12, 1.13, 1.15, 1.17, 1.20

• dws-connector-flink
• dws-connector-flink-sql-${flink.version}

Database fields, table names, and schemas can be read in uppercase.