How Is a DDS Node Going to Be Disconnected and What Can I Do?

A replica set consists of three nodes: primary, secondary, and hidden. The three-node architecture is set up automatically, and the three nodes automatically synchronize data with each other to ensure data reliability. Replica sets are recommended for small- and medium-sized service systems that require high availability.

• Primary node: Primary nodes are used to process both read and write requests.
• Secondary node: Secondary nodes are used to process read requests only.
• Hidden node: Hidden nodes are used to back up service data.

You can perform operations on the primary and secondary nodes. If the primary node is faulty, the system automatically selects a new primary node. The following figure shows the replica set architecture.

Figure 1 Three-node replica set architecture

DDS can write data only on the primary node. When data is written to the primary node, oplogs are generated. The secondary and hidden nodes read oplogs from the primary node for replay to ensure data consistency.

The storage capacity of oplogs is determined by the value of oplogSize (10% of the default disk capacity).

• How is the primary/secondary latency generated?

  If the write speed of the primary node is too fast and exceeds the replay speed of the oplog read on the secondary node, the primary/secondary latency occurs.

• When is a node going to be disconnected?

  The storage capacity of oplogs is limited. If the capacity reaches the upper limit, the earliest oplog will be deleted. The secondary node reads oplogs and records the last oplog each time. If the primary/secondary latency reaches a certain value, the secondary node finds that the last oplog point has been deleted. In this case, the secondary node cannot continue to read oplogs, and the secondary node is disconnected.

• How to effectively prevent the secondary node from being disconnected?
  • Set writeConcern to majority. In this way, data is written to a majority of nodes, ensuring data consistency.
  • Increase the oplog storage capacity by changing the value of oplogSizePercent on the console. For details, see Modifying DDS DB Instance Parameters.
  • Perform time-consuming DDL operations (such as index creation) and data backup operations during off-peak hours to avoid burst addition, deletion, and modification operations.
    

    If writeConcern is not set to majority, the data that is not synchronized to the secondary node may be lost when a primary/secondary switchover occurs.