New Features of DCS for Redis 5.0

Stream Data Structure

Stream is a new data type introduced with Redis 5.0. It supports message persistence and multicast.

Figure 1 shows the structure of a Redis stream, which allows messages to be appended to the stream.

Figure 1 Stream data structure

Stream commands

Figure 2 Stream commands

Message (stream item) acknowledgement

Compared with Pub/Sub, streams not only support consumer groups, but also message acknowledgement.

When a consumer invokes the XREADGROUP command to read or invokes the XCLAIM command to take over a message, the server does not know whether the message is processed at least once. Therefore, once having successfully processed a message, the consumer should invoke the XACK command to notify the stream so that the message will not be processed again. In addition, the message is removed from PEL and the memory will be released from the Redis server.

In some cases, such as network faults, the client does not invoke XACK after consumption. In such cases, the item ID is retained in PEL. After the client is reconnected, set the start message ID of XREADGROUP to 0-0, indicating that all PEL messages and messages after last_id are read. In addition, repeated message transmission must be supported when consumers consume messages.

Figure 3 Acknowledgment mechanism

Memory Usage Optimization

The memory usage of Redis 5.0 is optimized based on the previous version.

• Active defragmentation

  If a key is modified frequently and the value length changes constantly, Redis will allocate additional memory for the key. To achieve high performance, Redis uses the memory allocator to manage memory. Memory is not always freed up to the OS. As a result, memory fragments occur. If the fragmentation ratio (used_memory_rss/used_memory) is greater than 1.5, the memory usage is inefficient.

  To reduce memory fragments, properly plan and use cache data and standardize data writing.

  For Redis 3.0 and earlier versions, memory fragmentation problems are resolved by restarting the process regularly. It is recommended that the actual cache data does not exceed 50% of the available memory.

  For Redis 4.0, active defragmentation is supported, and memory is defragmented while online. In addition, Redis 4.0 supports manual memory defragmentation by running the memory purge command.

  For Redis 5.0, improved active defragmentation is supported with the updated Jemalloc, which is faster, more intelligent, and provides lower latency.

• HyperLogLog implementation improvements

  A HyperLogLog is a probabilistic data structure used to calculate the cardinality of a set while consuming little memory. Redis 5.0 improves HyperLogLog by further optimizing its memory usage.

  For example: the B-tree is efficient in counting, but consumes a lot of memory. By using HyperLogLog, a lot of memory can be saved. While the B-tree requires 1 MB memory for counting, HyperLogLog needs only 1 KB.

• Enhanced memory statistics

  The information returned by the INFO command is more detailed.

New and Better Commands

1. Enhanced client management
   • redis-cli supports cluster management.

     In Redis 4.0 and earlier versions, the redis-trib module needs to be installed to manage clusters.

     Redis 5.0 optimizes redis-cli, integrating all cluster management functions. You can run the redis-cli --cluster help command for more information.

   • The client performance is enhanced in frequent connection and disconnection scenarios.

     This optimization is valuable when your application needs to use short connections.

2. Simpler use of sorted sets

   ZPOPMIN and ZPOPMAX commands are added for sorted sets.

   • ZPOPMIN key [count]

     Removes and returns up to count members with the lowest scores in the sorted set stored at key. When returning multiple elements, the one with the lowest score will be the first, followed by the elements with higher scores.

   • ZPOPMAX key [count]

     Removes and returns up to count members with the highest scores in the sorted set stored at key. When returning multiple elements, the one with the lowest score will be the first, followed by the elements with lower scores.

3. More sub-commands added to the help command

   The help command can be used to view help information, saving you the trouble of visiting redis.io every time. For example, run the following command to view the stream help information: xinfo help

   127.0.0.1:6379> xinfo help
   1) XINFO <subcommand> arg arg ... arg. Subcommands are:
   2) CONSUMERS <key> <groupname>  -- Show consumer groups of group <groupname>.
   3) GROUPS <key>                 -- Show the stream consumer groups.
   4) STREAM <key>                 -- Show information about the stream.
   5) HELP                         -- Print this help.
   127.0.0.1:6379> 

4. redis-cli command input tips

   After you enter a complete command, redis-cli displays a parameter tip to help you memorize the syntax format of the command.

   As shown in the following figure, run the zadd command, and redis-cli displays zadd syntax in light color.

   

RDB Storing LFU and LRU Information

In Redis 5.0, storage key eviction policies LRU and LFU were added to the RDB snapshot file.

• FIFO: First in, first out. The earliest stored data is evicted first.
• LRU: Least recently used. Data that is not used for a long time is evicted first.
• LFU: Least frequently used. Data that is least frequently used is evicted first.

The RDB file format of Redis 5.0 is modified and is backward compatible. Therefore, if a snapshot is used for migration, data can be migrated from the earlier Redis versions to Redis 5.0, but cannot be migrated from the Redis 5.0 to the earlier versions.