Troubleshooting Shard Deletion Failures

Scenarios

This section describes how to diagnose and resolve shard deletion failures.

Troubleshooting Failures Because the Target Shard Is a Primary Shard

1. If there are unsharded collections on the shard to be deleted, use DRS to migrate the collections to a new instance.
2. If there are no unsharded collections on the target shard, run the movePrimary command to switch the primary shard. (This command blocks normal services when migrating unsharded collections. Therefore, it is disabled by default.)

Troubleshooting Failures Because the Target Shard Contains Jumbo Chunks

Recommended solutions:

1. Use DRS to migrate data to a new instance and redesign shard keys to prevent jumbo chunks from being generated on the new instance.
2. If possible, delete some data from the jumbo chunks to reduce the size.

Troubleshooting Failures Due to duplicated key(_id)

1. This is usually caused by orphaned documents. Delete orphaned documents first. For details, see Deleting Orphaned Documents.
2. If the failure persists after orphaned documents are deleted, run the following command for each collection to check for conflicting key values. If possible, delete unnecessary documents and retry the task.

   db.getSiblingDB("test").coll.aggregate([{$group: { _id: "$_id",  count: { $sum: 1 }}},{$match: { count: { $gt: 1 } }}], {allowDiskUse: true})

   The execution plan of the preceding command is a full collection scan (COLLSCAN), which consumes a large number of CPU and memory resources. The execution time is directly proportional to the volume of data. Therefore, run this command during off-peak hours.

3. Based on the result returned in 2, delete conflicting documents as required.

Checking the Balancer Status

1. Connect to a cluster instance using Mongo Shell.
2. Run the following command to query the balancer status:

   sh.startBalancer()

3. If true is returned, the balancer is enabled. To disable it, see Disabling the Balancer.

Disabling the Balancer

1. Connect to a cluster instance using Mongo Shell.
2. After connecting to a dds mongos node, run the following command to switch to the config database:

   use config

3. Run the following command to check the status of the balancer:

   while(sh.isBalancerRunning()){
              print("waiting...");           
              sleep(500); 
   }

   • If the command output is empty, the balancer is not executing any task. In this case, you can proceed to disable the balancer.
   • If the command output returns waiting, the balancer is migrating chunks. Do not run the command to disable the balancer. Otherwise, data inconsistency may occur.
4. If the command output in 3 is empty, run the following command to disable the balancer:

   sh.stopBalancer()

5. To enable the balancer, run the following command:

   sh.startBalancer()

Checking the Progress of the Shard Deletion Task

1. Connect to a cluster instance using Mongo Shell.
2. Check the number of chunks to be migrated from the shard to be deleted.

   db.getSiblingDB("config").chunks.aggregate([{$match: {ns: "test.coll", shard: "shard_3"}},{$group: {_id: "$ns", count: {$sum: 1}}}])
   { "_id" : "db.coll", "count" : 1 }

3. Run the following command to query the migration rate and the number of chunks that have been migrated in the last 24 hours:

   sh.status()

   

4. In ideal conditions (the workload is stable and the total number of chunks does not change), you can estimate the time based on the preceding information:

   Estimated time ≈ Number of chunks to be migrated/Number of chunks migrated in the last 24 hours

   For example, if the number of chunks to be migrated is 1,000 and the number of chunks migrated in the last 24 hours is 500, the estimated time is 2 days (1000/500 = 2).