Troubleshooting High Memory Usage of a DCS Redis Instance

Symptom

Redis provides fast database services. If the memory is insufficient, keys may be frequently evicted, the response time may increase, and the QPS may be unstable, affecting service running. This is normal due to Redis functions (such as master/replica replication and lazyfree). When the memory becomes full, scale up the instance or remove unnecessary data. Generally, you need to be alerted when the memory usage exceeds 95%.

Fault Locating

1. Query the memory usage in a specified period. For details, see Viewing DCS Monitoring Metrics. Check whether the value of Memory Usage is close to 100% continuously.
2. If the values of Evicted Keys and Maximum Command Latency increase significantly during the period when the memory usage exceeds 95%, the memory is insufficient.

   In this case, log in to the console and analyze big keys and slow queries. If no expiration is set for the instance, too much data will be stored in the instance, using up the memory.

3. If the memory of a Redis instance is full but there are not many keys, the output buffer may have occupied an excessive amount of memory.

   In this case, run the redis-cli --bigkeys command to scan for big keys after connecting to the instance using redis-cli. Then, run the info command to check the output buffer size.

Solution

• Check for big and hot keys.
  • You can analyze big keys and hot keys on the DCS console. For details, see Analyzing Big Keys and Hot Keys.
  • Use the bigkeys and hotkeys options on redis-cli.
    • redis-cli uses the bigkeys option to traverse all keys in a Redis instance and returns the overall key statistics and the biggest key of six data types: Strings, Lists, Hashes, Sets, Zsets, and Streams. The command is redis-cli -h <Instance connection address> -p <Port number> -a <Password> --bigkeys.
    • In Redis 4.0 and later, you can use the hotkeys option to quickly find hot keys in redis-cli. Run this command during service running to find hot keys: redis-cli -h <Instance connection address> -p <Port number> -a <Password> --hotkeys. The hot key details can be obtained from the summary part in the returned result.
• Detect big keys and hot keys in advance.
  • Configure an alarm for node-level memory usage. For details, see Configuring Alarm Rules for Critical Metrics.

    If a node has a big key, the memory usage of the node is much higher than that of other nodes. In this case, an alarm is triggered to help you find the problematic key.

  • Configure alarms for node-level Maximum Inbound Bandwidth, Maximum Outbound Bandwidth, and CPU Usage. For details, see Configuring Alarm Rules for Critical Metrics.

    If a node has a hot key, the bandwidth and CPU usage of the node is much higher than that of other nodes. In this case, an alarm is triggered to help you find the problematic key.

    

    You can use the preceding method to detect hot keys for DCS Redis 3.0 instances which do not support hot key analysis.

• Other suggestions:
  • Keep the size of each String within 10 KB and the number of Hashes, Lists, Sets, or Zsets within 5000.
  • When naming keys, use the service name abbreviation as the prefix and do not use special characters such as spaces, line brakes, single or double quotation marks, and other escape characters.
  • Do not rely too much on Redis transactions.
  • The performance of short connections ("connect" in Redis terminology) is poor. Use clients with connection pools.
  • Do not enable data persistence if you use Redis just for caching and can tolerate data loss.
• If the instance memory usage remains high after you take the preceding measures, expand the instance specifications during off-peak hours. For details, see Modifying DCS Instance Specifications.