Logical Decoding Options

General Options

These options can be configured for both serial decoding and parallel decoding, but may be invalid. For details, see the description of related options.

• include-xids:

  Specifies whether the decoded data column contains XID information.

  Value range: Boolean. The default value is true.

  • false: The decoded data column does not contain XID information.
  • true: The decoded data column contains XID information.
• skip-empty-xacts:

  Specifies whether to ignore empty transaction information during decoding.

  Value range: Boolean. The default value is false.

  • false: The empty transaction information is not ignored during decoding.
  • true: The empty transaction information is ignored during decoding.
• include-timestamp:

  Specifies whether decoded information contains the commit timestamp.

  Value range: Boolean. The default value is false in parallel decoding scenarios or true in SQL function decoding and serial decoding scenarios.

  • false: The decoded information does not contain the commit timestamp.
  • true: The decoded information contains the commit timestamp.
• only-local:

  Specifies whether to decode only local logs.

  Value range: Boolean. The default value is true.

  • false: Non-local logs and local logs are decoded.
  • true: Only local logs are decoded.
• white-table-list:

  Specifies a whitelist, including the schemas and tables to be decoded.

  Value range: a string that contains table names in the whitelist. Different tables are separated by commas (,). An asterisk (*) is used to fuzzily match all tables. Schema names and table names are separated by periods (.). No space character is allowed. For example:

  select * from pg_logical_slot_peek_changes('slot1', NULL, 4096, 'white-table-list', 'public.t1,public.t2,*.t3,my_schema.*');

• max-txn-in-memory:

  Memory control parameter. The unit is MB. If the memory occupied by a single transaction is greater than the value of this parameter, data is flushed to disks.

  For serial decoding, the value range is an integer ranging from 0 to 100. The default value is 0, indicating that memory control is disabled.

  For parallel decoding, the value ranges from 0 to 25% of the value of max_process_memory. The default value is max_process_memory/4/1024, where 1024 indicates the conversion from KB to MB. The value 0 indicates that this memory control is disabled.

• max-reorderbuffer-in-memory:

  Memory control parameter. The unit is GB. If the total memory (including the cache) of transactions being concatenated in the sender thread is greater than the value of this parameter, the current decoding transaction is flushed to disks.

  For serial decoding, the value range is an integer ranging from 0 to 100. The default value is 0, indicating that memory control is disabled.

  For parallel decoding, the value ranges from 0 to 50% of the value of max_process_memory. The default value is max_process_memory/2/1048576, where 1048576 indicates the conversion from KB to GB. The value 0 indicates that this memory control is disabled.

  

  Function decoding is serial decoding. For streaming decoding, setting parallel-decode-num to 1 indicates serial decoding; setting it to a value greater than 1 indicates parallel decoding.

• desc-memory-limit

  A memory control parameter, in MB. When the total memory of table metadata maintained by a logical decoding task is greater than the value of this parameter, some table metadata will be evicted.

  Value range: an integer ranging from 10 to 1024. The default value is 100.

• include-user:

  Specifies whether the BEGIN logical log of a transaction records the username of the transaction. The username of a transaction refers to the authorized user, that is, the login user who executes the session corresponding to the transaction. The username does not change during the execution of the transaction.

  Value range: Boolean. The default value is false.

  • false: The BEGIN logical log of a transaction does not record the username of the transaction.
  • true: The BEGIN logical log of a transaction records the username of the transaction.
• exclude-userids:

  OIDs of blacklisted users. This parameter can be configured only for decoding tasks on directly connected DNs and does not apply to decoding tasks on CNs with strong consistency in distributed mode.

  Value range: a string, which specifies the OIDs of blacklisted users. Multiple OIDs are separated by commas (,). The system does not check whether the OIDs exist.

• exclude-users:

  Specifies the name list of blacklisted users.

  Value range: a string, which specifies the names of blacklisted users. Multiple names are separated by commas (,). The system does not check whether the names exist.

• dynamic-resolution:

  Specifies whether to dynamically parse the names of blacklisted users. If no user is created when an Xlog is written, the system considers that the log user does not exist when the Xlog is decoded.

  Value range: Boolean. The default value is true.

  • false: An error is reported and logical decoding exits if a user who does not exist in the blacklist specified by exclude-users is detected. If the user exists in the blacklist, operations of the user can be filtered out.
  • true: No error is reported and the decoding is normal if a user who does not exist in the blacklist specified by exclude-users is detected. If the user exists in the blacklist, operations of the user can be filtered out.
• standby-connection:

  Specifies whether to restrict decoding only on the standby node. Because no standby CNs are configured, this parameter applies only to DNs.

  Value range: Boolean. The default value is false.

  • true: Only the standby node can be connected for decoding. When the primary node is connected for decoding, an error is reported and the system exits.
  • false: The primary or standby node can be connected for decoding.
  

  If the resource usage of the primary node is high and services are insensitive to real-time incremental data synchronization, you are advised to perform decoding on the standby node. If services have high requirements on real-time incremental data synchronization and the service pressure on the primary node is low, you are advised to perform decoding on the primary node.

• sender-timeout:

  Specifies the heartbeat timeout threshold between the GaussDB and the client. This option is valid only for streaming decoding. If no message is received from the client within the period, the logical decoding stops and disconnects from the client. The unit is ms.

  Value range: an integer ranging from 0 to 2147483647. The default value depends on the value of the GUC parameter logical_sender_timeout. The value 0 indicates that logical decoding does not proactively disconnect from the client. A small value, for example, 1 ms, indicates that decoding tasks may be interrupted.

• change-log-max-len:

  Specifies the maximum length of the logical log buffer, in bytes. This option is valid only for parallel decoding of connected DNs and is invalid for distributed strong consistency decoding, serial decoding, and SQL function decoding. If the length of a single decoding result exceeds the upper limit, the memory will be destroyed and another memory whose size is 1024 bytes is allocated for caching. If the value is too large, the memory usage increases. If the value is too small, the memory allocation and release operations are frequently triggered. Therefore, you are advised not to set it to a value less than 1024.

  Value range: 1 to 65535. The default value is 4096.

• max-decode-to-sender-cache-num:

  Specifies the threshold of the number of cached parallel decoding logs. This option is valid only for parallel decoding of connected DNs and is invalid for distributed strong consistency decoding, serial decoding, and SQL function decoding. If the number of locally cached logs is insufficient, more logs will be retrieved from the global cache.

  Value range: 1 to 65535. The default value is 4096.

• enable-heartbeat:

  Specifies whether to generate heartbeat logs. This option is valid only for streaming decoding.

  Value range: Boolean. The default value is false.

  • true: Heartbeat logs are generated.
  • false: Heartbeat logs are not generated.
  

  If the heartbeat log output option is enabled, heartbeat logs will be generated. The following uses parallel decoding as an example to describe how to parse heartbeat logs: For a binary heartbeat log message, it starts with a character 'h' and then the heartbeat log content: an 8-byte uint64 string, an 8-byte uint64 string, and an 8-byte int64 string. For the first 8-byte uint64 string, in the decoding scenario where DNs are directly connected, this string is an LSN, indicating the end position of the WAL read when the heartbeat logical log is sent; in the decoding scenario where distributed strong consistency is required, this string is a CSN, indicating the decoding log transaction CSN that has been sent when the heartbeat logical log is sent. For the second 8-byte uint64 string, in the decoding scenario where DNs are directly connected, this string is an LSN, indicating the location of the WAL that has been flushed to disks when the heartbeat logical log is sent; in the decoding scenario where distributed strong consistency is required, this string is a CSN, indicating the CSN to be obtained by the next transaction committed by the cluster. The last 8-byte int64 string indicates the generation timestamp (starting from January 1, 1970) of the latest decoded transaction log or checkpoint log. Then, it ends with character 'F'. TEXT/JSON heartbeat log messages that are sent in batches end with '0'. There is no such terminator for each TEXT/JSON heartbeat log message. The message content is transmitted in big-endian mode. The following figure shows the format. (In consideration of forward compatibility, the LSN naming mode is retained. The actual meaning depends on the specific scenario.)

  

• parallel-decode-num:

  Specifies the number of decoder threads for parallel decoding. This option is valid only for streaming decoding. When the system function is called, this option is invalid and only the value range is verified.

  Value range: an integer ranging from 1 to 20. The value 1 indicates that decoding is performed based on the original serial logic. Other values indicate that parallel decoding is enabled. The default value is 1.

  

  If parallel-decode-num is not set (the default value is 1) or is explicitly set to 1, the options in the following "Parallel decoding" cannot be configured.

• output-order:

  Specifies whether to output decoding results based on CSNs. This option is valid only for streaming decoding. When the system function is called, this option is invalid and only the value range is verified.

  Valid value: 0 or 1 of the int type. The default value is 0.

  • 0: The decoding results are sorted by transaction COMMIT LSN. This mode can be used only when the value of confirmed_csn of the decoding replication slot is set to 0 (not displayed). Otherwise, an error is reported.
  • 1: The decoding results are sorted by transaction CSN. This mode can be used only when the value of confirmed_csn of the decoding replication slot is not set to 0. Otherwise, an error is reported.
  

  • When output-order is not configured (that is, the default value 0 is used and the order is based on the COMMIT LSN) or is explicitly configured to 0, the options in section "Distributed Strong-Consistency Decoding" cannot be configured.
  • In streaming decoding scenarios, when a DN receives a logical decoding connection from a CN, the output-order option is invalid and CSN decoding is performed by default.

• auto-advance:

  Specifies whether to automatically update the logical replication slot number. This option is valid only for streaming decoding.

  Value range: Boolean. The default value is false.

  • true: The logical replication slot is advanced to the current decoding position when all sent logs are confirmed and there is no transaction to be sent.
  • false: The replication service calls the log confirmation API to advance the logical replication slot.
• skip-generated-columns:

  Specifies whether to skip stored generated columns in the logical decoding result. This parameter is invalid for UPDATE and DELETE on old tuples, and the corresponding tuples always output the stored generated columns. Stored generated columns are not supported in a distributed system and therefore, this parameter has no actual impact.

  Value range: Boolean. The default value is false or off.

  • true/on: The decoding result of stored generated columns is not output.
  • false/off: The decoding result of stored generated columns is output.

  Virtual generated columns are not controlled by this parameter. The DML decoding result does not output virtual generated columns.

• enable-ddl-decoding:

  Specifies whether to enable logical decoding for DDL statements.

  Value range: Boolean. The default value is false.

  • true: Logical decoding of DDL statements is enabled.
  • false: Logical decoding of DDL statements is disabled.
• enable-ddl-json-format:

  Specifies the DDL statement reverse parsing process and output format for logical decoding.

  Value range: Boolean. The default value is false.

  • true: The DDL statement reverse parsing result is output in JSON format.
  • false: The DDL statement reverse parsing result is output in the format specified by decode-style.
• timezone-is-utc:

  Specifies the logical decoding control parameter, which is used to control the output of time type data with time zones (for example, timestamptz type in ORA-/MySQL-compatible mode, and timestamp type in M-compatible mode). This parameter is valid only for streaming decoding and does not take effect for function decoding.

  Value range: Boolean. The default value is false.

  • true: The time of time zone 0 is forcibly output during decoding.
  • false: The time of the current database time zone is output during decoding.
• decode-sequence:

  Specifies whether to output the decoding result of the change log of the sequence value, which is a logical decoding control parameter.

  Value range: Boolean. The default value is false.

  • The value cannot be set to true currently.
  • false: The decoding result of the change log of the sequence value is not output.
  

  Currently, decode-sequence can only be set to false. If decode-sequence is set to true, an error is reported when decoding is started and the decoding process exits.

• data-limit

  Controls the data volume output by logical decoding.

  When the GUC parameter logical_decode_options_default is used, the value is an integer in the range [0,100]. Unit: GB Default value: 10 The value 0 indicates that the size of the decoding result is not limited.

  The GUC parameter setting must be used together with the data-limit input parameter of the pg_logical_get_area_changes function. For details, see the description of the pg_logical_get_area_changes function in "SQL Reference > Functions and Operators > System Management Functions > Logical Replication Functions" in Developer Guide.

Distributed Strong-Consistency Decoding

• logical-receiver-num:

  Specifies the number of logical receivers started for distributed decoding. This option is valid only for streaming decoding. When the system function is called, this option is invalid and only the value range is verified.

  Value range: an integer ranging from 1 to 20. The default value is 1. If this option is set to a value greater than the number of shards in the current cluster, the value is changed to the number of shards.

• slice-id:

  Specifies the ID of the shard where the current DN is located. This option is set only when DNs are connected for decoding. It is used to decode replication tables.

  Value range: an integer ranging from 0 to 8192. The default value is -1, indicating that the shard ID is not specified. However, an error is reported when the data is decoded to the replication table.

This configuration option is used when the DN attempts to use the CSN logical replication slot (confirmed_csn is a non-zero replication slot) for decoding. It is used to indicate the shard ID (that is, the sequence number of the shard. Enter 0 for the first shard). If this option is not set (that is, the default value -1 is used), an error is reported when data is decoded to the replication table. This parameter cannot be specified when a CN is connected for decoding. The program obtains the DN shard number, and the CN collects only the decoding result of the replication table of the DN shard.

• start-position:

  Filters out transactions whose CSNs are less than the specified CSN, and filters out logs whose LSNs are less than the specified LSN for the transaction with the specified CSN. This option is set only when DNs are connected. BEGIN logs of the transaction with the specified CSN must be filtered out.

  Value range: a string of two uint64 characters separated by a slash (/). The left and right sides indicate the CSN and LSN, respectively.

  

  This parameter cannot be specified when a CN is connected for decoding. The program uses this option to filter logs that may have been received when a decoding request is sent after the CN is connected to a DN.

Serial Decoding

• force-binary:

  Specifies whether to output the decoding result in binary format and display different behaviors in different scenarios.

  • For system functions pg_logical_slot_get_binary_changes and pg_logical_slot_peek_binary_changes:

    Value range: Boolean. The default value is false. The value is meaningless. The decoding result is always output in binary format.

  • For system functions pg_logical_slot_get_changes, pg_logical_slot_peek_changes, and pg_logical_get_area_changes:

    Value range: Boolean. The value is fixed at false. The decoding result is always output in text format.

  • For streaming decoding (supported only when DNs are connected):

    Value range: Boolean. The default value is false. The value is meaningless. The decoding result is always output in text format.

Parallel Decoding

The binary encoding rules are as follows:

1. The first four bytes represent the total number of bytes of the decoding result of statements following the statement-level delimiter letter P (excluded) or the batch end character F (excluded). If the value is 0, the decoding of this batch ends.
2. The next eight bytes (uint64) indicate the corresponding LSN (begin corresponds to first_lsn, commit corresponds to end_lsn, and other values correspond to the LSN of the statement).
3. The next 1-byte letter can be B, C, I, U, or D, representing BEGIN, COMMIT, INSERT, UPDATE, or DELETE, respectively.
4. If B is used in Step 3:
   1. The next eight bytes (uint64) indicate the CSN.
   2. The next eight bytes (uint64) indicate first_lsn.
   3. (Optional) If the next 1-byte letter is T, the following four bytes (uint32) indicate the timestamp length for committing the transaction. The following characters with the same length are the timestamp character string.
   4. (Optional) If the next one-byte letter is N, the following four bytes (uint32) indicate the length of the transaction username. The following characters with the same length are the transaction username.
   5. Because there may still be a decoding statement subsequently, a 1-byte letter P or F is used as a separator between statements. P indicates that there are still decoding statements in this batch, and F indicates that decoding in this batch is complete.
5. If C is used in 3:
   1. (Optional) If the next 1-byte letter is X, the following eight bytes (uint64) indicate XID.
   2. (Optional) If the next 1-byte letter is T, the following four bytes (uint32) indicate the timestamp length. The following characters with the same length are the timestamp character string.
   3. When logs are sent in batches, decoding results of other transactions may still exist after a COMMIT log is decoded. If the next 1-byte letter is P, the batch still needs to be decoded. If the letter is F, the batch decoding ends.
6. If I, U, or D is used in 3:
   1. The next two bytes (uint16) indicate the length of the schema name.
   2. The schema name is read based on the preceding length.
   3. The next two bytes (uint16) indicate the length of the table name.
   4. The table name is read based on the preceding length.
   5. (Optional) If the next 1-byte letter is N, it indicates a new tuple. If the letter is O, it indicates an old tuple. In this case, the new tuple is sent first.
      1. The following two bytes (uint16) indicate the number of columns to be decoded for the tuple, which is recorded as attrnum.
      2. The following procedure is repeated for attrnum times.
         1. The next two bytes (uint16) indicate the length of the column name.
         2. The column name is read based on the preceding length.
         3. The following 4 bytes (uint32) indicate the OID of the current column type.
         4. The next 4 bytes (uint32) indicate the length of the value (stored in string format) in the current column. If the value is 0xFFFFFFFF, it indicates null. If the value is 0, it indicates a string whose length is 0.
         5. The column value is read based on the preceding length.
   6. Because there may still be a decoding statement subsequently, if the next 1-byte letter is P, it indicates that the batch still needs to be decoded, and if the next 1-byte letter is F, it indicates that decoding of the batch ends.

• sending-batch:

  Specifies whether to send messages in batches.

  Valid value: 0 or 1 of the int type. The default value is 0.

  • 0: The decoding results are sent one by one.
  • 1: When the accumulated size of decoding results reaches 1 MB, decoding results are sent in batches.

  In the scenario where batch sending is enabled, if the decoding format is 'j' or 't', before each original decoding statement, a uint32 number is added indicating the length of the decoding result (excluding the current uint32 number), and a uint64 number is added indicating the LSN corresponding to the current decoding result.

In the CSN-based decoding scenario (that is, output-order is set to 1), batch sending is limited to a single transaction (that is, if a transaction has multiple small statements, the statements can be batch sent). That is, multiple transactions are not sent in the same batch, and BEGIN and COMMIT statements are not batch sent.

• parallel-queue-size:

  Specifies the length of the queue for interaction between parallel logical decoding threads.

  Value range: an integer ranging from 2 to 1024. The value must be an integer power of 2. The default value is 128.

  The queue length is positively correlated with the memory usage during decoding.