DDS Transactions and Read/Write Concerns

Transactions

Overview
In DDS, an operation on a single document is atomic. Because you can use embedded documents and arrays to capture relationships between data in a single document structure instead of normalizing across multiple documents and collections, this single-document atomicity obviates the need for multi-document transactions for many practical use cases.

For situations that require atomicity of multiple document updates or consistency between multiple document reads: Starting from version 4.0, DDS is able to execute multi-document transactions for replica sets.

Multi-document transactions can be used across multiple operations, collections, databases, and documents, providing an "All-Or-Nothing" proposition. Transactions either apply all data changes or roll back the changes. If a transaction commits, all data changes made in the transaction are saved. If any operation in the transaction fails, the transaction aborts and all data changes made in the transaction are discarded without ever becoming visible. Until a transaction commits, write operations in the transaction are not visible outside the transaction.

In most cases, a multiple-document transaction incurs a greater performance cost over single document writes, and the availability of multiple-document transactions should not be a replacement for effective schema design. For many scenarios, the denormalized data model (embedded documents and arrays) will continue to be optimal for your data and use cases. That is, for many scenarios, modeling your data appropriately will minimize the need for multi-document transactions.

Transaction APIs

The following mongo shell code shows the key APIs for using DDS transactions:

// Runs the txnFunc and retries if TransientTransactionError encountered
function runTransactionWithRetry(txnFunc, session) {
     while (true) { 
        try {  
             txnFunc(session);  // performs transaction 
             break;
        } catch (error) { 
            // If transient error, retry the whole transaction 
            if ( error.hasOwnProperty("errorLabels") && error.errorLabels.includes("TransientTransactionError")  ) {               
                 print("TransientTransactionError, retrying transaction ...");
                 continue;
            } else {  
                throw error;   
            } 
        } 
     } 
}

// Retries commit if UnknownTransactionCommitResult encountered 
function commitWithRetry(session) {
     while (true) { 
          try { 
               session.commitTransaction(); // Uses write concern set at transaction start.
               print("Transaction committed.");
               break;
          } catch (error) { 
              // Can retry commit  
              if (error.hasOwnProperty("errorLabels") && error.errorLabels.includes("UnknownTransactionCommitResult") ) { 
                  print("UnknownTransactionCommitResult, retrying commit operation ..."); 
                  continue;
              } else {  
                  print("Error during commit ..."); 
                  throw error;      
              } 
          } 
     } 
}  

// Updates two collections in a transactions
function updateEmployeeInfo(session) {
     employeesCollection = session.getDatabase("hr").employees;
     eventsCollection = session.getDatabase("reporting").events; 

     session.startTransaction( { readConcern: { level: "snapshot" }, writeConcern: { w: "majority" } } ); 

     try{      
          employeesCollection.updateOne( { employee: 3 }, { $set: { status: "Inactive" } } ); 
          eventsCollection.insertOne( { employee: 3, status: { new: "Inactive", old: "Active" } } );
          print(tojson(employeesCollection.find({ employee: 3 }).toArray())); 
          print(tojson(eventsCollection.find({ employee: 3 }).toArray())); 
          print("countDocuments = " + eventsCollection.countDocuments({}));
     } catch (error) {  
          print("Caught exception during transaction, aborting.", error);
          session.abortTransaction(); 
          throw error; 
     }     
 
     commitWithRetry(session);
}  

// insert data 
employeesCollection = db.getSiblingDB("hr").employees; eventsCollection = db.getSiblingDB("reporting").events; 
employeesCollection.drop();
eventsCollection.drop();
for (var i = 0; i < 10; ++i) {   
     employeesCollection.insertOne({employee: i});
     eventsCollection.insertOne({employee: i});
}  

// Start a session.
session = db.getMongo().startSession( { readPreference: { mode: "primary" } } );
try{ 
    runTransactionWithRetry(updateEmployeeInfo, session);
} catch (error) {  
   // Do something with error
} finally {  
   session.endSession();
}

Transactions are associated with a session. You must start a session before starting a transaction. You can have at most one open transaction at a time for a session. If a session ends and it has an open transaction, the transaction aborts.

When using the drivers, each operation in the transaction must be associated with the session.

Transactions and Atomicity
Multi-document transactions are atomic.
- If a transaction commits, all data changes made in the transaction are saved and are visible outside the transaction. Until a transaction commits, the data changes made in the transaction are not visible outside the transaction.
- When a transaction aborts, all data changes made in the transaction are discarded without ever becoming visible. For example, if any operation in the transaction fails, the transaction aborts and all data changes made in the transaction are discarded without ever becoming visible.
Restrictions on Transaction Operations
For transactions:
- You can specify Create/Retrieve/Update/Delete (CRUD) operations on existing collections. The collections used in a transaction can be in different databases.
- You cannot read from or write to collections in the config, admin, or local databases.
- You cannot write to system.* collections.
- You cannot return the supported operation's query plan using explain.
- For cursors created outside of a transaction, you cannot call getMore inside the transaction.
- For cursors created in a transaction, you cannot call getMore outside the transaction.
- You cannot execute non-CURD commands, including listCollections, listIndexes, createUser, getParameter, and count.
- To perform a count operation within a transaction, use the $count aggregation stage or the $group (with a $sum expression) aggregation stage. Starting from DDS 4.0, mongo shell provides the db.collection.countDocuments() method that uses the $group with a $sum expression to perform a count.
- A transaction cannot contain an insert operation that would result in the creation of a new collection, for example, an operation for implicitly creating a collection. A transaction cannot contain operations that affect the database catalog, for example, creating or dropping collections or indexes.
Precautions for Using Transactions
- Runtime Limit
  By default, a transaction must have a runtime of less than 1 minute. You can modify this limit using transactionLifetimeLimitSeconds for the DDS instances. Transactions that exceed this limit are considered expired and will be aborted by a periodic cleanup process.
- Oplog Size Limit
  If a committed transaction contains any write operations, a single oplog entry is created. That is, each operation in the transaction does not have a corresponding oplog entry. Instead, a single oplog entry encapsulates all write operations in a transaction. Each oplog entry must be within the BSON document size limit of 16 MB.
- Cache
  To prevent storage cache pressure from negatively impacting the performance:
  - When you abandon a transaction, abort the transaction.
  - When you encounter an error during individual operation in the transaction, abort and retry the transaction.
  The transactionLifetimeLimitSeconds parameter also ensures that expired transactions are aborted periodically to relieve storage cache pressure.
- Transactions and Locks
  By default, transactions wait up to 5 milliseconds to acquire locks required by the operations in the transaction. If the transaction cannot acquire its required locks within the 5 milliseconds, the transaction aborts. Transactions release all locks upon abort or commit.
  
  You can use the maxTransactionLockRequestTimeoutMillis parameter to adjust how long transactions wait to acquire locks. Increasing maxTransactionLockRequestTimeoutMillis allows operations in the transactions to wait the specified time to acquire the required locks. This can help obviate transaction aborts on momentary concurrent lock acquisitions, like fast-running metadata operations. However, this could possibly delay the abort of deadlocked transaction operations.
- Pending DDL Operations and Transactions
  If a multi-document transaction is in progress, new DDL operations that affect the same database wait behind the transaction. While these pending DDL operations exist, new transactions that access the same database as the pending DDL operations cannot obtain the required locks and will abort after waiting maxTransactionLockRequestTimeoutMillis. In addition, new non-transaction operations that access the same database will block until they reach their maxTimeMS limit.
  
  Consider the following scenarios:
  
  While an in-progress transaction is performing various CRUD operations on the employees collection in the hr database, you can start and complete a separate transaction to access the foobar collection in the hr database.
  
  While an in-progress transaction is performing various CRUD operations on the employees collection in the hr database and a separate DDL operation is issued to create an index on the employees collection in the hr database, the DDL operation must wait for the transaction to complete.
  
  When the DDL operation is pending, a new transaction attempts to access the foobar collection in the hr database. If the DDL operation remains pending for more than maxTransactionLockRequestTimeoutMillis, the new transaction will abort.
  - In-progress Transactions and Write Conflicts
    If a multiple-document transaction is in progress and a write outside the transaction modifies a document that an operation in the transaction later tries to modify, the transaction aborts because of a write conflict. If a multi-document transaction is in progress and has taken a lock to modify a document, when a write outside the transaction tries to modify the same document, the write waits until the transaction ends.
  - In-progress Transactions and Stale Reads
    Read operations inside a transaction can return old data, which is known as a stale read. Read operations inside a transaction are not guaranteed to see writes performed by other committed transactions or non-transactional writes.
    
    For example, consider the following sequence:
    - A transaction is in-progress.
    - A write outside the transaction deletes a document.
    - A read operation inside the transaction can read the now-deleted document since the operation uses a snapshot from before the write operation.
    To avoid stale reads inside transactions for a single document, you can use the db.collection.findOneAndUpdate() method. Example:
```
// insert data
employeesCollection = db.getSiblingDB("hr").employees;
employeesCollection.drop(); 
employeesCollection.insertOne({ _id: 1, employee: 1, status: "Active" });

// Start a session.
session = db.getMongo().startSession( { readPreference: { mode: "primary" } } ); 

session.startTransaction( { readConcern: { level: "snapshot" }, writeConcern: { w: "majority" } } ); 
 
employeesCollection = session.getDatabase("hr").employees; 

employeeDoc = employeesCollection.findOneAndUpdate( 
   { _id: 1, employee: 1, status: "Active" },
   { $set: { employee: 1 } }, 
   { returnNewDocument: true }
); 
print(tojson(employeeDoc));
```
    If the employee document is changed outside the transaction, the transaction aborts.
    
    If the employee document is not changed, the transaction returns and locks the document.
Best Practices for Using Multi-Document Transactions
- By default, DDS automatically aborts multiple-document transactions that run for more than 60 seconds. Note that if a server write volume is low, you can flexibly adjust a transaction to obtain a longer execution time. To resolve timeout issues, split a transaction into smaller parts to ensure that it can be executed within a specified period of time. Make sure that query statements are optimized and provide appropriate index coverage to quickly access data in transactions.
- There is no limit on the number of documents that can be read in a transaction. In this best practice, no more than 1,000 documents should be modified in a transaction. If you want to modify more than 1,000 documents in a transaction, we recommend that you split the transaction into multiple parts and they can be executed in batches.
- In DDS 4.0, a transaction is represented by a single oplog entry. The entry must fall within 16 MB in size. In DDS, oplogs record the incremental content in the case of an update operation, and record the entire document in the case of an insert operation. Therefore, all oplog entries for all statements in a transaction must fall within 16 MB in size. If this limit is exceeded, the transaction is aborted and completely rolled back. We recommend that you split a large transaction into smaller operation sets that each are 16 MB or less in size.
- When a transaction is abnormally aborted, an exception is returned to the driver and the transaction is fully rolled back. You can add application logic to catch and retry transactions that are aborted due to temporary exceptions, such as primary/secondary switchovers or network faults. Drivers provided by DDS use retryable writes to automatically retry to commit transactions.
- DDL operations, such as createIndex or dropDatabase, block transactions running on a namespace. All transactions that attempt to access the namespace during DDL suspension cannot obtain a lock within a specified time, which causes new transactions to be aborted.

Transactions and Read Concern

Operations in a transaction use the transaction-level read concern. This means a read concern set at the collection or database level is ignored inside the transaction. You can set the transaction-level read concern at the transaction start.

If the transaction-level read concern is unset, the transaction-level read concern defaults to the session-level read concern.

If the transaction-level and session-level read concerns are unset, the transaction-level read concern defaults to the client-level read concern. By default, the client-level read concern is "local" for reads on the primary node.

Multi-document transactions support the following read concern levels:
- "local"
  
  In most cases, the "majority" read concern is recommended.
- "majority"
  If the transaction commits with write concern "majority", read concern "majority" returns data that has been acknowledged by a majority of the replica set members and cannot be rolled back.
  
  If the transaction does not commit with write concern "majority", read concern "majority" provides no guarantees that read operations read majority-committed data.
- "snapshot"
  Read concern "snapshot" returns data from a snapshot of majority committed data if the transaction commits with write concern "majority".
  
  If the transaction does not use write concern "majority" for the commit, the "snapshot" read concern provides no guarantee that read operations used a snapshot of majority-committed data.
Suggestions on Using Transactions and Read Concern
- In most cases, the "majority" read concern is recommended.
  This setting ensures data isolation and consistency. Applications can read data only when the data is replicated to most nodes in a replica set. In this way, data will not be rolled back even if a new primary node is elected.
- In scenarios where the "read your own write" function is required, you need to read data directly from the primary node and use the "local" read concern.
  In this way, the latest update can be read as soon as possible after the write operation is complete. If the transaction commits with write concern "majority", read concern "majority" can also be used.

Transactions and Write Concern

Transactions use the transaction-level write concern to commit the write operations. Any write concerns set inside a transaction are ignored. Do not explicitly set the write concern for the individual write operations inside a transaction. You can set the transaction-level write concern at the transaction start.

If the transaction-level write concern is unset, the transaction-level write concern defaults to the session-level write concern for the commit.

If the transaction-level and session-level write concerns are unset, the transaction-level write concern defaults to the client-level write concern. By default, the client-level write concern is w: 1.

Multi-document transactions support all write concern w values, including:
- w: 1
  Write concern w: 1 returns acknowledgment after the commit is applied to the primary node. When you commit with w: 1, your transaction can be rolled back if there is a failover.
  
  When you commit with w: 1 write concern, transaction-level "majority" read concern provides no guarantees that read operations in the transaction read majority-committed data.
  
  When you commit with w: 1 write concern, transaction-level "snapshot" read concern provides no guarantee that read operations in the transaction used a snapshot of majority-committed data.
- w: "majority"
  Write concern w: "majority" returns acknowledgment after the commit has been applied to a majority of (M) voting members. That is, the commit has been applied to the primary node and (M-1) voting secondary nodes.
  
  When you commit with w: "majority" write concern, transaction-level "majority" read concern guarantees that operations have read majority-committed data.
  
  When you commit with w: "majority" write concern, transaction-level "snapshot" read concern guarantees that operations have read from a synchronized snapshot of majority-committed data.

Suggestions on Using Transactions and Write Concern
- In most cases, the "majority" write concern is recommended.
  This setting ensures that write operations are acknowledged by a majority of nodes in a replica set, thereby avoiding data loss or rollback risks even in the event of node failures or abnormal switchovers.
- In scenarios where high write performance is required, you can use write concern w: 1 and pay close attention to the replication delay of secondary nodes.
  Write concern w: 1 can provide better write performance and is applicable to write-intensive scenarios. In addition, you must monitor the replication delay of secondary nodes. If the delay is too long, the primary node may be rolled back unexpectedly. If the replication delay of a secondary node exceeds the oplog retention period, the secondary node may enter the RECOVERING state and cannot be automatically restored, reducing the instance availability. Therefore, you should pay attention to and handle this problem first.
- Set an appropriate write concern based on different operation requirements.
  You can flexibly adjust the write concern to meet diversified service requirements. For example, financial transaction data can be written using transactions with write concern to ensure atomicity. Core player data of gaming services can be written using w: "majority" write concern to ensure that the data will not be rolled back. Log data can be written using the default or w: 1 write concern to acquire better write performance.

Previous topic: DDS Query Plans and Query Replanning

Next topic: DDS Metric Alarm Configuration Suggestions

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot