Detecting and Handling Big Keys and Hot Keys

Big key

There are two types of big keys:

• Keys that have a large value. If the size of a single String key exceeds 10 KB, or if the size of all elements of a key combined exceeds 50 MB, the key is defined as a big key.
• Keys that have a large number of elements. If the number of elements in a key exceeds 5000, the key is defined as a big key.

Hot key

A hot key is most frequently accessed, or consumes significant resources. For example:

• In a cluster instance, a shard processes 10,000 requests per second, among which 3000 are performed on the same key.
• In a cluster instance, a shard uses a total of 100 Mbits/s inbound and outbound bandwidth, among which 80 Mbits/s is used by the HGETALL operation on a Hash key.

Big key

Instance specifications fail to be modified.

Specification modification of a Redis Cluster instance involves rebalancing (data migration between nodes). Redis has a limit on key migration. If the instance has any single key bigger than 512 MB, the modification will fail when big key migration between nodes times out. The bigger the key, the more likely the migration will fail.

Data migration fails.

During data migration, if a key has many elements, other keys will be blocked and will be stored in the memory buffer of the migration ECS. If they are blocked for a long time, the migration will fail.

Cluster shards are unbalanced.

• The memory usage of shards is unbalanced. For example, if a shard uses a large memory or even uses up the memory, keys on this shard are evicted, and resources of other shards are wasted.
• The bandwidth usage of shards is unbalanced. For example, flow control is repeatedly triggered on a shard.

Latency of client command execution increases.

Slow operations on a big key block other commands, resulting in a large number of slow queries.

Flow control is triggered on the instance.

Frequently reading data from big keys exhausts the outbound bandwidth of the instance, triggering flow control. As a result, a large number of commands time out or slow queries occur, affecting services.

Master/standby switchover is triggered.

If the high-risk DEL operation is performed on a big key, the master node may be blocked for a long time, causing a master/standby switchover.

Hot key

Cluster shards are unbalanced.

If only the shard where the hot key is located is busy processing service queries, there may be performance bottlenecks on a single shard, and the compute resources of other shards may be wasted.

CPU usage surges.

A large number of operations on hot keys may cause high CPU usage. If the operations are on a single cluster shard, the CPU usage of the shard where the hot key is located will surge. This will slow down other requests and the overall performance. If the service volume increases sharply, a master/standby switchover may be triggered.

Cache breakdown may occur.

If Redis cannot handle the pressure on hot keys, requests will hit the database. The database may break down as its load increases dramatically, affecting other services.

Through Big Key Analysis and Hot Key Analysis on the DCS console

See Analyzing Big Keys and Hot Keys.

By using the bigkeys and hotkeys options on redis-cli

• redis-cli uses the bigkeys option to traverse all keys in a Redis instance and returns the overall key statistics and the biggest key of six data types: Strings, Lists, Hashes, Sets, Zsets, and Streams. The command is redis-cli -h <Instance connection address> -p <Port number> -a <Password> --bigkeys.
• In Redis 4.0 and later, you can use the hotkeys option to quickly find hot keys in redis-cli. Run this command during service running to find hot keys: redis-cli -h <Instance connection address> -p <Port number> -a <Password> --hotkeys. The hot key details can be obtained from the summary part in the returned result.

Searching for big keys using Redis commands

If there is a pattern of big keys, for example, the prefix is cloud:msg:test, you can use a program to scan for keys that match the prefix, and then run commands to query the number of members in the key and query the key sizes to find big keys.

• Commands for querying the number of members: LLEN, HLEN, XLEN, ZCARD, SCARD
• Commands for querying the memory usage of keys: DEBUG OBJECT, MEMORY USAGE

Searching for big keys using redis-rdb-tools

redis-rdb-tools is an open-source tool for analyzing Redis RDB snapshot files. You can use it to analyze the memory usage of all keys in a Redis instance.

To use this method, you must export the RDB file of an instance on the Backups & Restorations page of the DCS console.

Big key

Split big keys.

Scenarios:

• If the big key is a String, you can split it into several key-value pairs and use MGET or a pipeline consisting of multiple GET operations to obtain the values. In this way, the pressure of a single operation can be split. For a cluster instance, the operation pressure can be evenly distributed to multiple shards, reducing the impact on a single shard.
• If the big key contains multiple elements, and the elements must be operated together, the big key cannot be split. You can remove the big key from Redis and store it on other storage media instead. This scenario should be avoided by design.
• If the big key contains multiple elements, and only some elements need to be operated each time, separate the elements. Take a Hash key as an example. Each time you run the HGET or HSET command, the result of the hash value modulo N (customized on the client) determines which key the field falls on. This algorithm is similar to that used for calculating slots in Redis Cluster.

Store big keys on other storage media.

If a big key cannot be split, it is not suitable to be stored in Redis. You can store it on other storage media, such as SFS or other NoSQL databases, and delete the big key from Redis.

Set appropriate expiration and delete expired data regularly.

Appropriate expiration prevents expired data from remaining in Redis. Due to Redis's lazy free, expired data may not be deleted in time. If this occurs, scan expired keys.

Hot key

Split read and write requests.

If a hot key is frequently read, configure read/write splitting on the client to reduce the impact on the master node. You can also add replicas to meet the read requirements, but there cannot be too many replicas. In DCS, replicas replicate data from the master in parallel. The replicas are independent of each other and the replication delay is short. However, if there is a large number of replicas, CPU usage and network load on the master node will be high.

Use the client cache or local cache.

If you know what keys are frequently used, you can design a two-level cache architecture (client/local cache and remote Redis). Frequently used data is obtained from the local cache first. The local cache and remote cache are updated with data writes at the same time. In this way, the read pressure on frequently accessed data can be separated. This method is costly because it requires changes to the client architecture and code.

Design a circuit breaker or degradation mechanism.

Hot keys can easily result in cache breakdown. During peak hours, requests are passed through to the backend database, causing service avalanche. To ensure availability, the system must have a circuit breaker or degradation mechanism to limit the traffic and degrade services if breakdown occurs.