Troubleshooting High Memory Usage of a DCS Redis Instance

Symptom

Redis provides fast database services. If the memory is insufficient, keys may be frequently evicted, the response time may increase, and the QPS may be unstable, affecting service running. This is normal due to Redis functions (such as master/replica replication and lazyfree). When the memory becomes full, scale up the instance or remove unnecessary data. Generally, you need to be alerted when the memory usage exceeds 95%.

Fault Locating

1. Query the memory usage in a specified period. For details, see Viewing Metrics. Check whether the value of Memory Usage is close to 100% continuously.
2. If the values of Evicted Keys and Maximum Command Latency increase significantly during the period when the memory usage exceeds 95%, the memory is insufficient.

   In this case, log in to the console and analyze big keys and slow queries by referring to Cache Analysis and Viewing Redis Slow Queries. If no expiration is set for the instance, too much data will be stored in the instance, using up the memory.

3. If the memory of a Redis instance is full but there are not many keys, the output buffer may have occupied an excessive amount of memory.

   In this case, run the redis-cli --bigkeys command to scan for big keys after connecting to the instance using redis-cli. Then, run the info command to check the output buffer size.

Solution

1. Perform big key and hot key analysis on the DCS console, and take measures accordingly. For details, see Analyzing Big Keys and Hot Keys.
   

   You can configure alarms to detect hot keys for DCS Redis 3.0 instances, which do not support hot key analysis.

2. Scan for expired keys and release them, or manually delete unnecessary keys.
3. Other suggestions:
   • The value of String must be less than or equal to 10 KB.
   • For data structures such as Hash, List, Set, and Zset, it is recommended that the number of elements in a single key be less than or equal to 5000.
   • When naming keys, use the service name abbreviation as the prefix and do not use special characters such as spaces, line brakes, single or double quotation marks, and other escape characters.
   • Do not rely too much on Redis transactions.
   • The performance of short connections ("connect" in Redis terminology) is poor. Use clients with connection pools.
   • If the data is used only for data cache and data loss is tolerated, you are advised to disable the persistence function. (Change the value of appendonly to no in the instance parameter configuration to disable the AOF persistence function.)
   • Configure alarms to detect big keys and hot keys in advance.
     • Configure an alarm for node-level memory usage. For details, see Configuring Alarm Rules for Critical Metrics.

       If a node has a big key, the memory usage of the node is much higher than that of other nodes. In this case, an alarm is triggered to help you find the problematic key.

     • Configure alarms for node-level Maximum Inbound Bandwidth, Maximum Outbound Bandwidth, and CPU Usage. For details, see Configuring Alarm Rules for Critical Metrics.

       If a node has a hot key, the bandwidth and CPU usage of the node is much higher than that of other nodes. In this case, an alarm is triggered to help you find the problematic key.

4. If the instance memory usage remains high after you take the preceding measures, expand the instance specifications during off-peak hours. For details, see Modifying Specifications.