Changing the Consistency Level of a Proxy Instance

After a proxy instance is created, requests for SELECT operations are routed to different nodes based on their read weights. Because there is a replication latency between the primary node and each read replica and the replication latency varies for different read replicas, the result returned by each SELECT statement may be different when you repeatedly execute a SELECT statement within a session. In this case, only eventual consistency is ensured.

You can select eventual consistency if you want to offload read requests from the primary node to read replicas or if read requests far outnumber write requests in your workloads.

To eliminate data inconsistencies caused by eventual consistency, session consistency is provided. Session consistency ensures the result returned by each SELECT statement in a session is the data that was updated after the last write request.

Proxy instances record the log sequence number (LSN) of each node and session. When data in a session is updated, a proxy instance records the LSN of the primary node as a session LSN. When a read request arrives subsequently, the proxy instance compares the session LSN with the LSN of each node and routes the request to a node whose LSN is at least equal to the session LSN. This ensures session consistency.

In session consistency, if there is significant replication latency between the primary node and read replicas and the LSN of each read replica is smaller than the session LSN, requests for SELECT operations will be routed to the primary node. In this case, the primary node is heavily loaded and instance performance suffers.

Figure 1 Principle of session consistency

The higher the consistency level, the heavier the load on the primary node and the lower the cluster performance. If your application requires that data written within a session be immediately available for query, you can select session consistency.

Session consistency ensures that data updated before a read request is executed can be queried in a given session. For example, in Figure 2, read request R(x1) depends on the updated data of write request W(x1,4). However, in some situations, there are dependencies between sessions. For example, in Figure 3, R(x1) over Conn_2 depends on the updated data of W(x1,4) over Conn_1. In this case, session consistency cannot ensure the consistency of query results. TaurusDB provides global consistency to solve this problem, which ensures data consistency in different sessions. It means that once data is written, you can immediately query the updated data.

Figure 2 Intra-session dependency

Figure 3 Inter-session dependency

Global consistency works as follows:

TaurusDB utilizes minimal replication latency and shared distributed log storage technology to provide strong consistency read for RO nodes at the pure kernel level. When global consistency is enabled, the LSN of the primary node is updated to the latest one after a data update operation is complete in a session. When a read request is received, a proxy instance routes the request to an expected node based on its routing policy. Due to the replication latency between the primary node and read replicas, the LSNs of read replicas may still lag that of the primary node. In this case, the kernel immediately accelerates the LSN updates of the read replicas and completes the LSN synchronization between the read replicas and the primary node within milliseconds. Once synchronization is complete, the read request is then responded to. This ensures that the read request can access any data that has been updated up to the moment the request was initiated.

Figure 4 Principle of global consistency

As shown in the figure, after an UPDATE statement is sent to the primary node, the primary node's LSN increases from 6 to 10. For the first SELECT statement sent to a read replica, the latest results are only returned after the read replica's LSN increases from 2 to 10. Similarly, for the second SELECT statement sent to another read replica, the latest results are only returned after the read replica's LSN increases from 5 to 10.

When there is notable latency between the primary node and read replicas, choosing global consistency may cause more requests to be routed to the primary node. This increased load on the primary node may lead to higher workload latency. So, you are advised to select global consistency when there are more read operations than write operations.

There are two parameters related to global consistency.

• Global consistency takes effect only for new connections and supports only proxy connections.
• The global consistency switch is bound to a proxy instance. You can enable global consistency for one or more proxy instances (if any).
• Once global consistency is enabled for a read-only proxy instance, you cannot modify consistTimeoutPolicy. If you try to modify this parameter, an error message will be sent to the client.
• Global consistency is not available when query cache is enabled for read replicas.
• If global consistency is enabled, the default value of consistTimeoutPolicy is used, and there is notable latency between the primary node and read replicas, SELECT requests will be sent to the primary node. This will increase load on the primary node and reduce read/write performance of the entire database cluster.
• If global consistency is enabled, read replicas lag the primary node, and the read replicas are heavily loaded, the number of active connections on the read replicas increases and the read performance of the DB instance deteriorates. You are advised to set the routing policy of the proxy instance to load balancing to distribute read requests based on the number of active connections. In this way, the read performance of the DB instance can be fully utilized.
• If global consistency is enabled and the CPU is fully utilized due to heavy load, workloads may be intermittently interrupted.