What's New
Function Overview
Product Bulletin
- Product Notice
- Version Support Notes
Service Overview
- GaussDB(DWS) Infographics
- What Is GaussDB(DWS)?
- Data Warehouse Types
- Data Warehouse Flavors
- Advantages
- Application Scenarios
- Functions
- Concepts
- Related Services
- Security
- GaussDB(DWS) Permissions Management
- GaussDB(DWS) Access
- Restrictions
- Technical Support
- Service Quotas
- GaussDB(DWS) Technical Specifications
Billing
- GaussDB(DWS) Billing Overview
- Billing Modes
- Item
- Billing Examples
- Billing Mode Change
- Renewal
  - Overview
  - Manual Renewal
- Bills
- Arrears
- Stopping Billing
- Cost Management
- Billing FAQs
Getting Started
- Checkpoint Vehicle Analysis
- Supply Chain Requirement Analysis of a Company
- Operations Status Analysis of a Retail Department Store
- Creating a Time Series Table
- Best Practices of Hot and Cold Data Management
- Best Practices for Automatic Partition Management
- Creating a Cluster and Connecting to It
- Using CDM to Migrate MySQL Data to the GaussDB(DWS) Cluster
- Using DLI Flink Jobs to Write Kafka Data to GaussDB(DWS) in Real Time
- Basic SQL Operations
- Database Quick Start
- Getting Started with Common Practices
User Guide
- Using GaussDB(DWS)
- Preparations
- Creating a GaussDB(DWS) Cluster
- Connecting to a GaussDB(DWS) Cluster
- Creating a GaussDB(DWS) Database and User
- Migrating Service Data to a GaussDB(DWS) Cluster
  - Migrating Data to a GaussDB(DWS) Cluster Using GDS-Kafka
  - Data Source Management
- GaussDB(DWS) Cluster Data Security and Encryption
- GaussDB(DWS) Cluster Management
- GaussDB(DWS) Cluster O&M
Best Practices
- Import and Export
- Data Migration
- Data Analytics
- Decoupled Storage and Compute
  - DWS 3.0 Decoupled Storage and Compute Usage Suggestions and Performance Optimization
- Data Development
- Database Management
- Performance Tuning
  - Optimizing Table Structure Design to Enhance GaussDB(DWS) Query Performance
  - Analyzing SQL Statements That Are Being Executed to Handle GaussDB(DWS) Performance Issues
- Cluster Management
  - Binding Different Resource Pools to Two Types of Jobs to Balance Load for GaussDB(DWS)
  - Scaling Options for GaussDB(DWS) with a Coupled Storage-Compute Architecture
Data Migration and Synchronization
- Data Migration to GaussDB(DWS)
- Importing Data
- Full Database Migration
  - Using CDM to Migrate Data to GaussDB(DWS)
  - Using DSC to Migrate SQL Scripts
- Real-time Import
  - Using DRS to Import Data to GaussDB(DWS)
  - Real-time Data Import From Kafka GaussDB(DWS)
- Metadata Migration
  - Using gs_dump and gs_dumpall to Export Metadata
  - Using gs_restore to Import Data
- Exporting Data
- Other Operations
Developer Guide
- Standard Data Warehouse (9.1.0.x)
- Standard data warehouse (8.1.3.x)
- Hybrid Data Warehouse (9.1.0.x）
- Hybrid Data Warehouse (8.1.3.x）
- Historical Versions
SQL Syntax Reference
- SQL Syntax Reference (9.1.0.x)
- SQL Syntax Reference (8.1.3.x)
- Historical Versions
Performance Tuning
- Overview of Query Performance Optimization
- Query Execution Process
- SQL Execution Plan
- SQL Optimization Guide
- Optimization Cases
- SQL Execution Troubleshooting
- query_band Load Identification
- Common Performance Parameter Optimization Design
Tool Guide
- Overview
- Downloading Related Tools
- gsql
- Data Studio
- GDS
- DSC
- DataCheck
- DWS-Connector
- Server Tool
API Reference
- Before You Start
- API Overview
- Calling APIs
- Getting Started
- API Description
- Application Cases
  - Using Postman to Call the API for Creating a Cluster
  - Using Postman to Call the API for Creating a Snapshot
- Introduction
- Appendix
SDK Reference
- SDK Overview
FAQs
- Top FAQs
- Product Consulting
- Database Connections
- Data Migration
- Database Usage
- Cluster Management
- Account Permissions
- Database Performance
- Backup and Restoration
  1. Why Does It Take a Long Time to Create an Automated Snapshot in GaussDB(DWS)?
  2. Does a GaussDB(DWS) Snapshot Have the Same Function as an EVS Snapshot?
Troubleshooting
- Database Connections
- JDBC/ODBC
- Data Import and Export
- Database Parameter Modification
- Account/Permission/Password
- Cluster Performance
- Cluster Exceptions
  - The Disk Usage Alarm Is Frequently Generated
- Database Use
Videos
Performance White Paper
- Overview
- Test Result
  - TPC-H Single Query Test
  - TPC-DS Single Query Test
- Test Methods
- Appendixes
  - TPC-H Test Sets
  - TPC-DS Test Sets
Technical White Paper
- GaussDB(DWS)
- Platforms and Technical Specifications Supported by GaussDB(DWS)
  - Technical Specifications
- GaussDB(DWS) Core Technologies
- GaussDB(DWS) Tools
  - Client Tools
  - Database Monitoring Tool
- External APIs
Error Code Reference
- 8.2.0 and earlier versions
  - Management Console Error Code
  - Data Warehouse Service Error Codes
- 8.2.1 or later versions
Glossary
More Documents
- User Guide
- API Reference (ME-Abu Dhabi Region)
- Developer Guide (ME-Abu Dhabi Region)
- SQL Syntax Reference (ME-Abu Dhabi Region)
- Tool Guide (ME-Abu Dhabi Region)
- Error Code Reference (ME-Abu Dhabi Region)
  - Management Console Error Code
- User Guide (Paris Region)
- API Reference (Paris Region)
- Developer Guide (Paris Region)
- SQL Syntax Reference (Paris Region)
- Tool Guide (Paris Region)
- Error Code Reference (Paris Region)
  - Management Console Error Code
- User Guide (Kuala Lumpur Region)
- API Reference (Kuala Lumpur Region)
- Developer Guide (Kuala Lumpur Region)
- SQL Syntax Reference (Kuala Lumpur Region)
- Tool Guide (Kuala Lumpur Region)
- Error Code Reference (Kuala Lumpur Region)
  - Management Console Error Code
General Reference
- Glossary
- Service Level Agreement
- White Papers
- Endpoints
- Permissions

On this page

Working Principles
Use Cases
Creating HStore Tables and Related Views
Usage Practice
Requirements and Suggestions for Using HStore Tables

Show all

Help Center/ GaussDB(DWS)/ Best Practices/ Data Development/ Best Practices for Using HStore Tables

Best Practices for Using HStore Tables

Updated on 2024-10-29 GMT+08:00

View PDF

Working Principles

In GaussDB(DWS), a CU is the smallest unit used to store data in a column-store table. By default, each column in the table stores 60,000 rows of data as a CU. Once generated, the data in a CU cannot be altered. A single CU is generated regardless of whether 1 or 60,000 records are inserted into a column-store table. Frequent small data insertions hinder effective compression, leading to data expansion, which negatively impacts query performance and disk usage.

CU file data can only be appended, not modified. Deleting data marks it as invalid in the dictionary, while updating marks old data as deleted and writes new data to a new CU. Repeated updates or deletions cause space expansion and inefficient space usage.

The column-store Delta table addresses the issue of small CUs from frequent small data imports but does not resolve lock conflicts from concurrent updates on the same CU. A hybrid data warehouse needs to work with data sources, such as upstream databases or applications. Therefore, in real-time import scenarios, concurrent insert, update, and delete operations are necessary for timely data import and high query efficiency.

HStore tables use additional delta tables. Batch-inserted data is written directly to CUs, maintaining the compression benefits of column-store tables. Updated columns and small batch inserts are serialized, compressed, and periodically merged into primary table CUs.

Use Cases

GaussDB(DWS) uses column storage format for HStore tables to minimize disk usage, enable high-concurrency updates, and enhance query performance. HStore tables are ideal for scenarios that demand real-time data import and query capabilities, as well as the ability to process traditional TP transactions.

To enhance performance, GaussDB(DWS) 8.3.0.100 has optimized HStore tables and kept the old ones for compatibility purposes. The optimized tables are known as HStore_opt tables. HStore tables can be replaced by HStore_opt tables for better performance, except in scenarios requiring high performance without micro-batch updates.

Creating HStore Tables and Related Views

To create an HStore table, you need to specify the enable_hstore table-level parameter.

     
         CREATE TABLE test1 (i int,j text) with (orientation = column,enable_hstore=on);

To create an HStore_opt table, you need to specify the enable_hstore_opt table-level parameter.

    
        CREATE TABLE test2 (i int,j text) with (orientation = column,enable_hstore_opt=on);

Check the number of tuples in the Delta table and the expansion status of the Delta table in the view.

   
       SELECT * FROM pgxc_get_hstore_delta_info('tableName');

Use functions to perform lightweight cleanup and full cleanup on the Delta table.

After the lightweight merge accumulates 60,000 I records and deletion information on the CU, the level-4 lock ceases to hinder the addition, deletion, modification, and querying of services. Nevertheless, the space is not freed up for the system.
1

select hstore_light_merge('tableName');
Merging all records and truncating the Delta table is necessary to free up space for the system. Nonetheless, holding a level-8 lock will impede services.
1

select hstore_full_merge('tableName');

Insert 100 data records into the HStore table in batches. A record whose type is I (n_i_tup is 1) is generated.

   
       CREATE TABLE data(a int primary key, b int);
 NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "data_pkey" for table "data"
 CREATE TABLE

 INSERT INTO data values(generate_series(1,100),1);
 INSERT 0 100

 CREATE TABLE hs(a int primary key, b int)with(orientation=column, enable_hstore=on);
 NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "hs_pkey" for table "hs"
 CREATE TABLE

 INSERT INTO hs SELECT * FROM data;
 INSERT 0 100

 SELECT * FROM pgxc_get_hstore_delta_info('hs'); 
  node_name |      part_name      | live_tup | n_i_type | n_d_type | n_x_type | n_u_type | n_m_type | data_size
 -----------+---------------------+----------+----------+----------+----------+----------+----------+-----------
  dn_1      | non partition table |        1 |        1 |        0 |        0 |        0 |        0 |      8192
 (1 row)

After hstore_full_merge is executed, no tuple exists in the Delta table (the value of live_tup is 0), and the value of data_size is 0.

   
      SELECT hstore_full_merge('hs');
  hstore_full_merge
 -------------------
                  1
 (1 row)
 ​
SELECT * FROM pgxc_get_hstore_delta_info('hs'); 
  node_name |      part_name      | live_tup | n_i_type | n_d_type | n_x_type | n_u_type | n_m_type | data_size
 -----------+---------------------+----------+----------+----------+----------+----------+----------+-----------
  dn_1      | non partition table |        0 |        0 |        0 |        0 |        0 |        0 |         0
 (1 row)

Perform the deletion. The Delta table contains a record whose type is D (n_d_tup is 1).

   
       DELETE hs where a = 1;
 DELETE 1
SELECT * FROM pgxc_get_hstore_delta_info('hs'); 
  node_name |      part_name      | live_tup | n_i_type | n_d_type | n_x_type | n_u_type | n_m_type | data_size
 -----------+---------------------+----------+----------+----------+----------+----------+----------+-----------
  dn_1      | non partition table |        1 |        0 |        1 |        0 |        0 |        0 |      8192
 (1 row)

Usage Practice

For optimal performance of HStore tables, it is crucial to configure the following parameter settings:

Set autovacuum_max_workers_hstore to 3, autovacuum_max_workers to 6, autovacuum to true, and enable_col_index_vacuum to on.

Concurrent update

Once a batch of data is inserted into a column-store table, two sessions are initiated. In session 1, a piece of data is deleted, and the transaction is not terminated.

       
           CREATE TABLE col(a int , b int)with(orientation=column);
 CREATE TABLE

 INSERT INTO col select * from data;
 INSERT 0 100

 BEGIN;
 BEGIN

 DELETE col where a = 1;
 DELETE 1

When session 2 attempts to delete more data, it becomes evident that session 2 can only proceed after session 1 is committed. This scenario imitates the CU lock issue in column storage.

       
           BEGIN;
 BEGIN
 DELETE col where a = 2;

Repeat the previous experiment using the HStore table. Session 2 can be executed successfully without any lock wait.

       
           BEGIN;
 BEGIN
 DELETE hs where a = 2;
 DELETE 1

Compression efficiency

Create a data table with 3 million data records.

       
        
          
           CREATE TABLE data( a int, b bigint, c varchar(10), d varchar(10));
 ​
 CREATE TABLE
 INSERT INTO data values(generate_series(1,100),1,'asdfasdf','gergqer');
 INSERT 0 100
 INSERT INTO data select * from data;
 INSERT 0 100
 INSERT INTO data select * from data;
 INSERT 0 200
 ​
 ---Insert data cyclically until the data volume reaches 3 million.
 ​
 INSERT INTO data select * from data;
 INSERT 0 1638400
 select count(*) from data;
   count
 ---------
  3276800
 (1 row)

         

       
      

Import data to a row-store table in batches and check whether the size is 223 MB.

      
          CREATE TABLE row (like data including all);
 CREATE TABLE
 INSERT INTO row select * from data;
 INSERT 0 3276800
 select pg_size_pretty(pg_relation_size('row'));
  pg_size_pretty
 ----------------
  223 MB
 (1 row)

Import data to a column-store table in batches and check whether the size is 3.5 MB.

      
          CREATE TABLE hs(a int, b bigint, c varchar(10),d varchar(10))with(orientation= column, enable_hstore=on);
 CREATE TABLE
 INSERT INTO hs select * from data;
 INSERT 0 3276800
 select pg_size_pretty(pg_relation_size('hs'));
  pg_size_pretty
 ----------------
  3568 KB
 (1 row)

HStore tables have a good compression effect because of their simple table structure and duplicate data. They are usually compressed three to five times more than row-store tables.

Batch query performance

It takes approximately four seconds to query the fourth column of the row-store table using the created table.

       
           explain analyze select d from data;
 explain analye                                                               QUERY PLAN
 -----------------------------------------------------------------------------------------------------------------------------------------
   id |          operation           |        A-time        | A-rows  | E-rows  | Peak Memory  | E-memory | A-width | E-width | E-costs
  ----+------------------------------+----------------------+---------+---------+--------------+----------+---------+---------+----------
    1 | ->  Streaming (type: GATHER) | 4337.881             | 3276800 | 3276800 | 32KB         |          |         |       8 | 61891.00
    2 |    ->  Seq Scan on data      | [1571.995, 1571.995] | 3276800 | 3276800 | [32KB, 32KB] | 1MB      |         |       8 | 61266.00

It takes about 300 milliseconds to query the fourth column of the HStore table.

       
           explain analyze select d from hs;
                                                                     QUERY PLAN
 ---------------------------------------------------------------------------------------------------------------------------------------------------
   id |               operation                |       A-time       | A-rows  | E-rows  |  Peak Memory   | E-memory | A-width | E-width | E-costs
  ----+----------------------------------------+--------------------+---------+---------+----------------+----------+---------+---------+----------
    1 | ->  Row Adapter                        | 335.280            | 3276800 | 3276800 | 24KB           |          |         |       8 | 15561.80
    2 |    ->  Vector Streaming (type: GATHER) | 111.492            | 3276800 | 3276800 | 96KB           |          |         |       8 | 15561.80
    3 |       ->  CStore Scan on hs            | [111.116, 111.116] | 3276800 | 3276800 | [254KB, 254KB] | 1MB      |         |       8 | 14936.80

Only the batch query scenario has been tested, and in this scenario, the stored tables and HStore tables perform better than row-store tables in terms of query performance.

Requirements and Suggestions for Using HStore Tables

Configure the parameters.
To optimize query performance and compression efficiency, the background thread should perform MERGE clearing on HStore tables. Prior to utilizing HStore tables, ensure that the relevant GUC parameters are configured correctly. The suggested parameter values can be found below.

autovacuum_max_workers_hstore: 3

autovacuum_max_workers: 6

autovacuum: true

enable_col_index_vacuum: on
Suggestions on importing data to the database (The HStore_opt table is recommended.)
Suggestions for importing the HStore_opt table to the database:
1. The performance of importing data using UPDATE is poor. You are advised to use UPSERT to import data.
2. When using DELETE to import data, use index scanning. The JDBC batch method is recommended.
3. When using UPSERT to import data, enable enable_hstore_nonconflict_upsert_optimization when there is no concurrency conflict and disable it in other scenarios. The optimal path is automatically selected.
4. Use merge into only when importing over one million data records per DN and there is no concurrent data to prevent duplicate data.
Point query suggestion (The HStore_opt table is recommended.)
Suggestions for querying the HStore_opt table:
1. Create a level-2 partition on columns with evenly distributed distinct values and frequent equivalent filter criteria. Avoid level-2 partitions on columns with skewed or few distinct values.
2. When dealing with fixed filter criteria columns (excluding level-2 partitions), use the cbtree index (up to 5 columns).
3. When dealing with variable filter criteria columns (excluding level-2 partitions), use the GIN index (up to 5 columns).
4. For all string columns involving equivalent filtering, bitmap indexes can be specified during table creation. The number of columns is not limited, but cannot be modified later.
5. Specify columns that can be filtered by time range as the partition columns.
6. If point queries return over 100,000 records per DN, index scanning may outperform non-index scanning. Use the GUC parameter enable_seqscan to compare performance.
Index-related
Indexes occupy extra space and offer limited point query performance improvement. Create primary key or B-tree indexes only for upsert or unique/near-unique point queries.

MERGE-related
HStore tables rely on background autovacuum for MERGE operations. Ensure data import speed does not exceed MERGE speed to prevent delta table expansion. Control import speed by managing concurrent data import tasks. Delta table space reuse is affected by oldestXmin. Old transactions can delay space reuse, causing expansion.

Parent topic: Data Development

Previous topic: Improving Development Efficiency by Leveraging GaussDB(DWS)'s View Decoupling and Rebuilding Function

Next topic: Best Practices of GIN Index