Why a File Is Not Found or Is Incomplete When Multiple Clients Operate the File at the Same Time?

Symptom

There are two clients (client A and client B). They mount the same SFS Turbo file system and operate the same file in the file system at the same time. The following typical scenarios may occur:

• Scenario 1: Client A adds or renames a file. Then, client B reads the file. The file cannot be found.
• Scenario 2: Both clients write the same file at the same time. The file is incomplete after the writes are complete.
• Scenario 3: Both clients write the same file at the same time. Then, an error is reported.

Fault Diagnosis

An SFS Turbo file system can provide shared access for multiple clients. Because there are client-side cache and server-side cache, SFS Turbo only ensures eventual consistency.

If the clients mount the file system using NFS, the NFS mechanism controls their caches, and the behaviors vary depending on the OS version. The SFS Turbo background is a distributed system and uses multiple NFS servers to provide services. When client mounts are distributed on different NFS servers, data synchronization may be slow.

• Scenario 1: After client A writes a new file, the cache is not updated. So, client B does not have the information about the new file in the directory entry cache, and the message indicating that the file was not found is returned.
• Scenario 2: Both clients append data to the same file and begin appending at the current file size. After client A writes data, the file size in the cache of client B is not updated. As a result, when client B writes data, some data is overwritten and lost.
• Scenario 3: Both clients overwrite the same file and truncate the file to zero before writing. When client A writes data, client B truncates the file size to zero. As a result, the write position of client A exceeds the file length, and an error is reported.

Solution

1. Avoid having multiple clients write the same file at the same time. Adjust the services if required. For example, adjust to have different clients to write different files and then combine the files. Or, have clients negotiate with each other and specify a primary client to perform the write operation.
2. After different clients write the same file, to avoid the cache from affecting the read results, you can disable the client cache by referring to What Can I Do If Data of My SFS Turbo File System Is Not the Same When Accessed from Two Client Servers? or mount the clients to the same NFS server using the IP address.
   

   Note that disabling the client cache and mounting via the IP address will affect the SFS Turbo performance. Evaluate the impacts before doing so.