TaurusDB Primary/Standby Replication Principles

Background

In database architectures, primary nodes, standby nodes, and read replicas are core components for achieving high availability, load balancing, and data backup.

MySQL uses a shared-nothing architecture for primary/standby replication, which is implemented through binlogs. TaurusDB uses a shared storage architecture, which is different from the primary/standby architecture of MySQL. In TaurusDB, synchronization between the primary and standby nodes is achieved through redo logs, and only a small amount of metadata needs to be transferred. The standby node reads redo logs from the shared storage and performs operations such as invalidating old data pages, replaying metadata locks, updating transaction snapshots, and caching logs to achieve fast primary/standby synchronization.

This section describes the implementation principles and main process of primary/standby replication in TaurusDB.

TaurusDB Primary/Standby Architecture

TaurusDB uses a shared storage architecture, and does not use binlogs for primary/standby synchronization.

In the MySQL community, a standby node refers to the secondary database in traditional primary/secondary replication. It receives binary logs from the primary database and applies them to maintain data consistency. It is mainly used for failover and data redundancy. A read replica is a MySQL instance that is specifically configured to accept no write operations. It is typically used to scale out read capabilities and share the read load of the primary database. It can be a standby node, a cascaded secondary database, or even an independent node that does not participate in HA switchover.

For TaurusDB, standby nodes and read replicas are not distinguished.

As shown in the figure, TaurusDB embraces a shared everything philosophy. Redo logs are used to synchronize data between the primary node and read replicas. Redo logs are not directly transmitted between the primary node and read replicas. Instead, only a small amount of necessary metadata is synchronized. This metadata is mainly used to correctly read data from the shared storage system.

Only redo logs are written by the primary node to the storage pool. Data pages are generated and modified by the shared storage system. The primary node writes redo logs to the shared storage system, and read replicas read the redo logs from the shared storage system to achieve primary/standby synchronization. Both the primary node and read replicas read data pages from the shared storage system.

This architecture has several advantages. First, read replicas can be quickly added regardless of the data volume. Second, the latency between the primary node and read replicas is low, and the bandwidth usage for data transmission is low. Third, if the primary node fails, a read replica can be quickly promoted to primary to restore services.

Figure 1 TaurusDB primary/standby architecture

Primary/Standby Replication Process

1. Primary/Standby Communication

   TaurusDB read replicas synchronize data with the primary node by processing redo logs. Primary/standby synchronization enables read replicas to query new or modified data on the primary node as quickly as possible. Since TaurusDB uses a shared storage architecture, redo logs are read from the shared storage instead of the primary node. Only necessary metadata is transmitted between the primary node and read replicas.

   Such metadata includes the following:

   • Metadata sent from the primary node to read replicas:
     1. Metadata such as the location and size of redo logs in the shared storage system, which is used to read redo logs.
     2. Mapping between tablespace data pages and shared storage system shards, which is used to read data pages.
     3. Initial transaction snapshot information, which is used to determine the visibility of transactions and is continuously updated by parsing redo logs.
   • Metadata sent from read replicas to the primary node:

     Recycle LSNs of redo logs that are no longer used by read replicas. The primary node uses this data to determine the range of data to be recycled, preventing the recycling of historical data that may still be used by read replicas.

   Each read replica starts a thread named handle_slave_sync to communicate with the primary node. This thread is used to receive and send required metadata from and to the primary node.

2. Processing Redo Logs

   Through the primary/standby communication thread, read replicas obtain the required information and can read and process redo logs.

   1. Primary/standby replication background threads of read replicas

      To speed up redo log processing, read replicas start multiple groups of background threads for parallel processing in a pipeline manner. The number of threads is configurable.

      These threads include reader, dispatcher, parser, advancer, and impeller threads. They are notified and woken up through events to collaboratively complete processing. When the primary/standby communication thread detects new redo logs through synchronization information, it instructs the reader thread to start reading redo logs from the shared storage and then initiates a series of processing workflows.

      Figure 2 Primary/standby replication background threads of TaurusDB read replicas
      
      • Log reader thread: waits for notifications from the primary/standby communication thread handle_slave_sync and reads redo log data from the shared storage system.
      • Log dispatcher thread: waits for notifications from the log reader thread and distributes logs to each log parser thread.
      • Log parser thread: waits for notifications from the log dispatcher thread and parses redo logs. It performs two main tasks. First, it groups redo logs for specific data pages by page ID. Second, it parses other types of redo logs for future use, such as summarizing committed transaction IDs.
      • Log impeller thread: processes redo logs for modified data pages. It combines the logs processed by the log parser thread into a single log for each page ID. It also invalidates data pages in the buffer pool and caches redo logs into the corresponding log directory structure for subsequently generating new versions of data pages.
      • Advancer thread: handles other primary/standby synchronization actions, including processing metadata locks (MDLs) to prevent conflicts between metadata modifications and data queries, and updating transaction snapshots to incorporate new active and committed transactions for determining transaction visibility.
   2. Read replica parsing redo logs

      There are multiple types of redo logs. In addition to modifications to data page content, redo logs also contain other information that needs to be parsed and processed by read replicas to maintain synchronization with the primary node. Each redo log read from the shared storage system undergoes the processing steps shown in the following figure.

      Figure 3 TaurusDB read replica parsing and processing logs
      

      Redo logs generated from modifications to data page content are summarized in a hash table using page IDs as keys. On one hand, this mechanism invalidates pages with the same ID in the buffer pool, signaling the read replica that a new version of the data page must be fetched when next accessed. On the other hand, these logs are cached in the global log directory so that when the read replica requires a new version of a data page, it can replay the corresponding redo logs based on the old version of the page in the buffer pool to generate the new version of the data page. This reduces the overhead of reading pages from the shared storage, thereby improving performance.

      The read replica also needs to parse and process the following information from various types of redo logs:

      • Committed transaction IDs

        The committed transaction IDs in redo logs can be used to update the active transaction list for visibility determination. For example, at the repeatable read isolation level, the modifications of committed transactions are visible. However, if the ID of a transaction is between the IDs of two committed transactions and the redo log does not record that the transaction has been committed, the transaction is considered active and its modifications are invisible.

      • Transaction purge No.

        It is used to obtain the maximum transaction ID and determine the low limit number of the transaction snapshot on the read replica.

      • Information about updated and deleted tablespaces

        It is used to update the space cache, for example, when a tablespace has been deleted or its size has been increased.

      • MDL requests

        Redo logs record the request and release of MDL locks by DDL statements. The read replica simulates the same request and release process to prevent conflicts between metadata modifications and queries, for example, when an index currently in use is being deleted.

      • Memory objects that need to be invalidated, such as the query cache
      • Undo tablespace information that needs to be invalidated

        It is used to process operations such as undo truncate performed on the primary node.

      • Statistics that need to be updated

      After the preceding operations are complete, the read replica complete synchronization with the primary node and can provide query services based on the latest data. However, TaurusDB does not provide strong consistency. In principle, there is a certain delay between the read replica and the primary node. In the redo log processing workflow described above, before the current round is complete, the read replica can only provide data queries up to the log sequence number (LSN) corresponding to the completion of the previous synchronization round.