High CPU Usage

Scenarios

If your CPU usage reaches 80%, a CPU bottleneck exists. In this case, data reads and writes are slow, affecting your services.

The following describes how to analyze current slow queries. After the analysis and optimization, query performance will be improved and indexes will be used more efficiently.

Procedure

Analyzing Current Queries

Connect to an instance using Mongo Shell.
For instances with public access enabled, see:
For instances without public access enabled, see:

Run the following command to view the operations being performed on the database:

db.currentOp()

Command output:

{
        "raw" : {
                "shard0001" : {
                        "inprog" : [
                                {
                                        "desc" : "StatisticsCollector",
                                        "threadId" : "140323686905600",
                                        "active" : true,
                                        "opid" : 9037713,
                                        "op" : "none",
                                        "ns" : "",
                                        "query" : {
 
                                        },
                                        "numYields" : 0,
                                        "locks" : {
 
                                        },
                                        "waitingForLock" : false,
                                        "lockStats" : {
 
                                        }
                                },
                                {
                                        "desc" : "conn2607",
                                        "threadId" : "140323415066368",
                                        "connectionId" : 2607,
                                        "client" : "172.16.36.87:37804",
                                        "appName" : "MongoDB Shell",
                                        "active" : true,
                                        "opid" : 9039588,
                                        "secs_running" : 0,
                                        "microsecs_running" : NumberLong(63),
                                        "op" : "command",
                                        "ns" : "admin.",
                                        "query" : {
                                                "currentOp" : 1
                                   },
                                        "numYields" : 0,
                                        "locks" : {
 
                                        },
                                        "waitingForLock" : false,
                                        "lockStats" : {
 
                                        }
                                }
                        ],
                        "ok" : 1
                },
    ...
}

client: IP address of the client that sends the request
opid: unique operation ID
secs_running: elapsed time for execution, in seconds. If the returned value of this field is too large, check whether the request is reasonable.
microsecs_running: elapsed time for execution, in microseconds. If the returned value of this field is too large, check whether the request is reasonable.
op: operation type. The value can be query, insert, update, delete, or command.
ns: target collection
For details, see the db.currentOp() command in the official documentation.

Based on the command output, check whether there are requests that take a long time to process.
If the CPU usage is low while services are being processed but then becomes high during just certain operations, analyze the requests that take a long time to execute.

If an abnormal query is found, find the opid corresponding to the operation and run db.killOp(opid) to kill it.

Analyzing Slow Queries

You can view slow query logs on the DDS console. You can start from the slowest operation recorded in the log and optimize the operations one by one.

Analyze slow query logs to find the cause of the high CPU usage.

The following is an example of a slow query log. The log shows that a request performed a full table scan, scanning 1,561,632 documents without using an index.

{
        "op" : "query",
        "ns" : "taiyiDatabase.taiyiTables$10002e",
        "query" : {
                "find" : "taiyiTables",
                "filter" : {
                        "filed19" : NumberLong("852605039766")
                },
                "shardVersion" : [
                        Timestamp(1, 1048673),
                        ObjectId("5da43185267ad9c374a72fd5")
                ],
                "chunkId" : "10002e"
        },
        "keysExamined" : 0,
        "docsExamined" : 1561632,
        "cursorExhausted" : true,
        "numYield" : 12335,
        "locks" : {
                "Global" : {
                        "acquireCount" : {
                                "r" : NumberLong(24672)
                        }
                },
                "Database" : {
                        "acquireCount" : {
                                "r" : NumberLong(12336)
                        }
                },
                "Collection" : {
                        "acquireCount" : {
                                "r" : NumberLong(12336)
                        }
                }
        },
        "nreturned" : 0,
        "responseLength" : 157,
        "protocol" : "op_command",
        "millis" : 44480,
        "planSummary" : "COLLSCAN",
        "execStats" : {
              "stage" : "SHARDING_FILTER",
                "nReturned" : 0,
                "executionTimeMillisEstimate" : 43701,
                "works" : 1561634,
                "advanced" : 0,
                "needTime" : 1561633,
                "needYield" : 0,
                "saveState" : 12335,
                "restoreState" : 12335,
                "isEOF" : 1,
                "invalidates" : 0,
                "chunkSkips" : 0,
                "inputStage" : {
                        "stage" : "COLLSCAN",
                        "filter" : {
                                "filed19" : {
                                        "$eq" : NumberLong("852605039766")
                                }
                        },
                        "nReturned" : 0,
                        "executionTimeMillisEstimate" : 43590,
                        "works" : 1561634,
                        "advanced" : 0,
                        "needTime" : 1561633,
                        "needYield" : 0,
                        "saveState" : 12335,
                        "restoreState" : 12335,
                        "isEOF" : 1,
                        "invalidates" : 0,
                        "direction" : "forward",
                        "docsExamined" : 1561632
                }
        },
        "ts" : ISODate("2019-10-14T10:49:52.780Z"),
        "client" : "172.16.36.87",
        "appName" : "MongoDB Shell",
        "allUsers" : [
                {
                        "user" : "__system",
                        "db" : "local"
                }
        ],
       "user" : "__system@local"
}

The following stages can be causes for a slow query:

COLLSCAN involves a full collection (full table) scan.
When a request (such as query, update, and delete) requires a full table scan, it consumes a large amount of CPU resources. If you find COLLSCAN in the slow query log, CPU resources may be occupied.

If such requests are frequent, create indexes for the fields to be queried.
docsExamined involves a full collection (full table) scan.
You can view the value of docsExamined to check the number of documents scanned. A larger value indicates a higher CPU usage.
The index may be suboptimal when IXSCAN and keysExamined are used.
- Too many indexes can affect write and update performance.
- If your application has more write operations, creating indexes may increase write latency.
You can view the value of keysExamined to see how many indexes are scanned in a query. A larger value indicates a higher CPU usage.

If an index is not properly created or there are many matching results, the CPU usage does not decrease greatly and the execution speed is slow.

Example: For the data of a collection, the number of values of the a field is small (only 1 and 2), but the b field has more values.
```
{ a: 1, b: 1 }
{ a: 1, b: 2 }
{ a: 1, b: 3 }
......
{ a: 1, b: 100000} 
{ a: 2, b: 1 }
{ a: 2, b: 2 }
{ a: 2, b: 3 }
......
{ a: 1, y: 100000}
```
The following shows how to implement the {a: 1, b: 2} query.
```
db.createIndex({a: 1}): The query is not effective because the a field has too many same values.
```
```
db.createIndex({a: 1, b: 1}): The query is not effective because the a field has too many same values.
```
```
db.createIndex({b: 1}): The query is effective because the b field has a few same values.
```
```
db.createIndex({b: 1, a: 1}): The query is not effective because the a field has a few same values.
```
For the differences between {a: 1} and {b: 1, a: 1}, see the official documentation.
SORT and hasSortStage may involve sorting a large amount of data.
When a query contains a sorting operation, the value of hasSortStage in the slow query log is true. If the sorting cannot be performed using an index, the query results are sorted. The sorting consumes a large amount of CPU resources, so you need to create indexes on fields that are frequently used for sorting.

When you find the keyword SORT in the slow query log, you can consider optimizing the sorting operation by creating an index.

Other operations, such as index creation and aggregation (combinations of traversal, query, update, and sorting), also apply to the scenarios mentioned earlier because they are also CPU-intensive operations. For more information about profiling, see the official documentation.

Analysis Capability

After the analysis and optimization of the requests that are being executed and slow requests, all requests use proper indexes, and the CPU usage becomes stable. If the CPU usage remains high after the analysis and troubleshooting, the current instance may have reached the performance bottleneck and cannot meet service requirements. In this case, you can perform the following operations to solve the problem:

View monitoring information to analyze instance resource usage. For details, see Viewing Monitoring Metrics.
Change the DDS instance class or add shard nodes.