New Features in 9.1.0.x

The beta features discussed below are not available for commercial use. Search for technical support before utilizing these features.

Version 9.1.0.100 (August 12, 2024)

Elastic architecture

1. Architecture upgrade: The storage-compute decoupling architecture 3.0, based on OBS, introduces layered and elastic computing and storage, with on-demand storage charging to reduce costs and improve efficiency. Multiple virtual warehouses (VWs) can be deployed to enhance service isolation and resolve resource contention.
2. The elastic VW feature, which is stateless and supports read/write acceleration, addresses issues like insufficient concurrent processing, unbalanced peak and off-peak hours, and resource contention for data loading and analytics. For details, see Elastically Adding or Deleting a Logical Cluster.
3. Both auto scale-out and classic scale-out are supported when adding or deleting DNs. Auto scale-out does not redistribute data on OBS, while classic scale-out redistributes all data. The system automatically selects the scale-out mode based on the total number of buckets and DNs.
4. The storage-compute decoupling architecture (DWS 3.0) enhances performance with disk cache and asynchronous I/O read/write. When the disk cache is fully utilized, performance matches that of the storage-compute integration architecture (DWS 2.0).

Figure 1 Decoupled storage and compute

Real-time processing

1. Launched the vectorized Turbo acceleration engine, doubling the performance of tpch 1000x.
2. Upgraded version of hstore, called hstore_opt, offers a higher compression ratio and works in conjunction with the Turbo engine to reduce storage space by 40% when compared to column storage.
3. With Flink, you can connect directly to DNs to import data into the database. This results in linear performance improvement in batch data import scenarios. For details, see Real-Time Binlog Consumption by Flink.
4. GaussDB(DWS) supports Binlog (currently in beta) and can be used in conjunction with Flink to enable incremental computing. For details, see Subscribing to Hybrid Data Warehouse Binlog.
5. This update significantly improves full-column performance while reducing resource consumption.
6. GaussDB(DWS) supports materialized views (currently in beta). For details, see CREATE MATERIALIZED VIEW.
7. To improve coarse filtering, the Varchar/text column now supports bitmap index and bloom filter. When creating a table, you must specify them explicitly. For details, see CREATE TABLE.
8. To enhance performance in topK and join scenarios, the runtime filter feature is now supported. You can learn more about GUC parameters runtime_filter_type and runtime_filter_ratiox in Other Optimizer Options.
9. GaussDB(DWS) supports asynchronous sorting to enhance the min-max coarse filtering effect of PCK columns.
10. The performance in the IN scenario is greatly improved.
11. ANALYZE supports incremental merging of partition statistics, collecting only statistics on changed partitions and reusing historical data, which improves execution efficiency. It collects statistics only on predicate columns.
    • The CREATE TABLE syntax now includes the incremental_analyze parameter to control whether to enable incremental ANALYZE mode for partitioned tables. For details, see CREATE TABLE.
    • The enable_analyze_partition GUC parameter determines whether to collect statistics on a partition of a table. For details, see Other Optimizer Options.
    • The enable_expr_skew_optimization GUC parameter controls whether to use expression statistics in the skew optimization policy. For details, see Optimizer Method Configuration.
    • ANALYZE | ANALYSE
12. GaussDB(DWS) supports large and wide tables, with a maximum of 5,000 columns.
13. Create index/reindex supports parallel processing.
14. The pgxc_get_cstore_dirty_ratio function is added to obtain the dirty page rate of CU, Delta, and CUDesc in the target table (only hstore_opt is supported).

[Convergence and unification]

1. One-click lakehouse: You can use create external schema to connect to the HiveMetaStore metadata, avoiding complex create foreign table operations and reducing maintenance costs. For details, see Accessing HiveMetaStore Across Clusters.
2. GaussDB(DWS) allows for reading and writing in Parquet/ORC format, as well as overwriting, appending, and multi-level partition read and write.
3. GaussDB(DWS) allows for reading in Hudi format.
4. Foreign tables support concurrent execution of ANALYZE, significantly improving the precision and speed of statistics collection. However, foreign tables do not support AutoAnalyze capabilities, so it is recommended to manually perform ANALYZE after data import.
5. Foreign tables can use the local disk cache for read acceleration.
6. Predicates such as IN and NOT IN can be pushed down for foreign tables to enhance partition pruning.
7. Foreign tables now support complex types such as map, struct, and array, as well as bytea and blob types.
8. Foreign tables support data masking and row-level access control.
9. GDS now supports the fault tolerance parameter compatible_illegal_char for exporting foreign tables.
10. The read_foreign_table_file function is added to parse ORC and Parquet files, facilitating fault demarcation.

High availability

1. The fault recovery speed of the unlogged table is greatly improved.
2. Backup sets support cross-version restoration. Fine-grained table-level restoration supports restoration of backup sets generated by clusters of earlier versions (8.1.3 and later versions).
3. Fine-grained table-level restoration supports restoration to a heterogeneous cluster (the number of nodes, DNs, and CNs can be different).
4. Fine-grained restoration supports permissions and comments. Cluster-level and schema-level physical fine-grained backups support backup permissions and comments, as do table-level restorations and schema-level DR.

Space saving

1. Column storage now supports JSONB and JSON types, allowing JSON tables to be created as column-store tables, unlike earlier versions which only supported row-store tables.
2. Hot and cold tables support partition-level index unusable, saving local index space for cold partitions.
3. The upgraded hstore_opt provides a higher compression ratio and, when used with the Turbo engine, saves 40% more space compared to column storage.

O&M and stability improvement

1. The query filter is enhanced to support interception by SQL feature, type, source, and processed data volume. For details, see CREATE BLOCK RULE.
2. GaussDB(DWS) now automatically frees up memory resources by reclaiming idle connections in a timely manner. You can specify the syscache_clean_policy parameter to set the policy for clearing the memory and number of idle DN connections. For details, see Connection Pool Parameters.
3. The gs_switch_respool function is added for dynamic switching of the resource pool used by queryid and threadid. This enables dynamic adjustment of the resources used by SQL. For details, see Resource Management Functions.
4. The pg_sequences view is added to display the attributes of sequences accessible to the current user.
5. The following functions are added to allow you to query information about all chunks requested by the memory in a specified shared memory:
   • pg_shared_chunk_detail
   • pv_session_chunk_detail
   • pg_shared_chunk_dump
   • pv_session_chunk_dump
6. The pgxc_query_resource_info function is added to display the resource usage of the SQL statement corresponding to a specified query ID on all DNs. For details, see pgxc_query_resource_info.
7. The pgxc_stat_get_last_data_access_timestamp function is added to return the last access time of a table. This helps the service to identify and clear tables that have not been accessed for a long time. For details, see pgxc_stat_get_last_data_access_timestamp.
8. SQL hints support more hints that provide better control over the generation of execution plans. For details, see Configuration Parameter Hints.
9. Performance fields are added to top SQL statements that are related to syntax parsing and disk cache. This makes it easier to identify performance issues. For details, see Real-time Top SQL.
10. The preset data masking administrator has the authority to create, modify, and delete data masking policies.
11. Audit logs can record objects that are deleted in cascading mode.
12. Audit logs can be dumped to OBS.

Ecosystem compatibility

1. if not exists can be included in the create schema, create index, and create sequence statements.
2. The merge into statement now allows for specified partitions to be merged. For details, see MERGE INTO.
3. In Teradata-compatible mode, trailing spaces in strings can be ignored when comparing them.
4. GUC parameters can be used to determine if the n in varchar(n) will be automatically converted to nvarchar2.
5. PostGIS has been upgraded to version 3.2.2.

Restrictions

1. A maximum of 256 VWs are supported, each with 1,024 DNs. It is recommended to have no more than 32 VWs and 128 DNs.
2. DR is not supported by OBS tables that have decoupled storage and compute. Only full backup and restoration are available.

Behavior changes

1. VACUUM FULL, ANALYZE, and CLUSTER are only supported for individual tables, not the entire database. Even though there are no syntax errors, the commands will not be executed.
2. OBS tables with decoupled storage and compute do not support delta tables. If enable_delta is set to on, no error is reported, but delta tables do not take effect. If a delta table is required, use the hstore-opt table instead.
3. By default, NUMA core binding is enabled and can be turned off dynamically using the enable_numa_bind parameter.
4. Upgrading from version 8.3.0 Turbo to version 910 changes the numeric(38) data type in Turbo tables to numeric(39), without affecting display width. Rolling back to the previous version will not reverse this change.
5. Due to the decoupling of storage and compute, the EVS storage space in DWS 3.0 is half that of DWS 2.0 by default. For example, purchasing 1 TB of EVS storage provides 500 GB in DWS 3.0 for active/standby mode, compared to 1 TB in DWS 2.0. When migrating data from DWS 2.0 to DWS 3.0, the EVS storage space required in DWS 3.0 is twice that of DWS 2.0.