New Features of DCS for Redis 5.0

DCS for Redis 5.0 is compatible with the new features of the open-source Redis 5.0, in addition to all the improvements and new commands in Redis 4.0.

Stream Data Structure

Stream is a new data type introduced with Redis 5.0. It supports message persistence and multicast.

Figure 1 shows the structure of a Redis stream, which allows messages to be appended to the stream.

Streams have the following features:

A stream can have multiple consumer groups.
Each consumer group contains a Last_delivered_id that points to the last consumed item (message) in the consumer group.
Each consumer group contains multiple consumers. All consumers share the last_delivered_id of the consumer group. A message can be consumed by only one consumer.
pending_ids in the consumer can be used to record the IDs of items that have been sent to the client, but have not been acknowledged.
For detailed comparison between stream and other Redis data structures, see Table 1.

Figure 1 Stream data structure

**Table 1** Differences between streams and existing Redis data structures
Item	Stream	List, Pub/Sub, Zset
Complexity of seeking items	O(log(N))	List: O(N)
Offset	Supported. Each item has a unique ID. The ID is not changed as other items are added or evicted.	List: Not supported. If an item is evicted, the latest item cannot be located.
Persistence	Supported. Streams are persisted to AOF and RDB files.	Pub/Sub: Not supported.
Consumer group	Supported.	Pub/Sub: Not supported.
Acknowledgment	Supported.	Pub/Sub: Not supported.
Performance	Not related to the number of consumers.	Pub/Sub: Positively related to the number of clients.
Eviction	Streams are memory efficient by blocking to evict the data that is too old and using a radix tree and listpack.	Zset consumes more memory because it does not support inserting same items, blocking, or evicting data
Randomly deleting items	Not supported.	Zset: Supported.

Stream commands

Stream commands are described below in the order they are used. For details, see Table 2.

Run the XADD command to add a stream item, that is, create a stream. The maximum number of messages that can be saved can be specified when adding the item.
Create a consumer group by running the XGROUP command.
A consumer uses the XREADGROUP command to consume messages.
After the consumption, the client runs the XACK command to confirm that the consumption is successful.

Figure 2 Stream commands

**Table 2** Stream commands description
Command	Description	Syntax
XACK	Deletes one or multiple messages from the pending entry list (PEL) a consumer group of the stream.	XACK key group ID [ID ...]
XADD	Adds a specified entry to the stream at a specified key. If the key does not exist, running this command will result in a key to be automatically created based on the entry.	XADD key ID field string [field string ...]
XCLAIM	Changes the ownership of a pending message, so that the new owner is the consumer specified as the command argument.	XCLAIM key group consumer min-idle-time ID [ID ...] [IDLE ms] [TIME ms-unix-time] [RETRYCOUNT count] [FORCE] [JUSTID]
XDEL	Removes the specified entries from a stream, and returns the number of entries deleted, that may be different from the number of IDs passed to the command in case certain IDs do not exist.	XDEL key ID [ID ...]
XGROUP	Manages the consumer groups associated with a stream You can use XGROUP to: Create a new consumer group associated with a stream. Destroy a consumer group. Remove a specified consumer from a consumer group. Set the consumer group last delivery ID to something else.	XGROUP [CREATE key groupname id-or-$] [SETID key id-or-$] [DESTROY key groupname] [DELCONSUMER key groupname consumername]
XINFO	Retrieves different information about the streams and associated consumer groups.	XINFO [CONSUMERS key groupname] key key [HELP]
XLEN	Returns the number of entries in a stream. If the specified key does not exist, 0 is returned, indicating an empty stream.	XLEN key
XPENDING	Obtains data from a stream through a consumer group. This command is the interface to inspect the list of pending messages in order to observe and understand what clients are active, what messages are pending to be consumed, or to see if there are idle messages.	XPENDING key group [start end count] [consumer]
XRANGE	Returns entries matching a given range of IDs.	XRANGE key start end [COUNT count]
XREAD	Reads data from one or multiple streams, only returning entries with an ID greater than the last received ID reported by the caller.	XREAD [COUNT count] [BLOCK milliseconds] STREAMS key [key ...] ID [ID ...]
XREADGROUP	A special version of the XREAD command, which is used to specify a consumer group to read from.	XREADGROUP GROUP group consumer [COUNT count] [BLOCK milliseconds] STREAMS key [key ...] ID [ID ...]
XREVRANGE	This command is exactly like XRANGE, but with the notable difference of returning the entries in reverse order, and also taking the start-end range in reverse order.	XREVRANGE key end start [COUNT count]
XTRIM	Trims the stream to a specified number of items, if necessary, evicting old items (items with lower IDs).	XTRIM key MAXLEN [~] count

Message (stream item) acknowledgement

Compared with Pub/Sub, streams not only support consumer groups, but also message acknowledgement.

When a consumer invokes the XREADGROUP command to read or invokes the XCLAIM command to take over a message, the server does not know whether the message is processed at least once. Therefore, once having successfully processed a message, the consumer should invoke the XACK command to notify the stream so that the message will not be processed again. In addition, the message is removed from PEL and the memory will be released from the Redis server.

In some cases, such as network faults, the client does not invoke XACK after consumption. In such cases, the item ID is retained in PEL. After the client is reconnected, set the start message ID of XREADGROUP to 0-0, indicating that all PEL messages and messages after last_id are read. In addition, repeated message transmission must be supported when consumers consume messages.

Figure 3 Acknowledgment mechanism

Memory Usage Optimization

The memory usage of Redis 5.0 is optimized based on the previous version.

Active defragmentation
If a key is modified frequently and the value length changes constantly, Redis will allocate additional memory for the key. To achieve high performance, Redis uses the memory allocator to manage memory. Memory is not always freed up to the OS. As a result, memory fragments occur. If the fragmentation ratio (used_memory_rss/used_memory) is greater than 1.5, the memory usage is inefficient.

To reduce memory fragments, properly plan and use cache data and standardize data writing.

For Redis 3.0 and earlier versions, memory fragmentation problems are resolved by restarting the process regularly. It is recommended that the actual cache data does not exceed 50% of the available memory.

For Redis 4.0, active defragmentation is supported, and memory is defragmented while online. In addition, Redis 4.0 supports manual memory defragmentation by running the memory purge command.

For Redis 5.0, improved active defragmentation is supported with the updated Jemalloc, which is faster, more intelligent, and provides lower latency.
HyperLogLog implementation improvements
A HyperLogLog is a probabilistic data structure used to calculate the cardinality of a set while consuming little memory. Redis 5.0 improves HyperLogLog by further optimizing its memory usage.

For example: the B-tree is efficient in counting, but consumes a lot of memory. By using HyperLogLog, a lot of memory can be saved. While the B-tree requires 1 MB memory for counting, HyperLogLog needs only 1 KB.
Enhanced memory statistics
The information returned by the INFO command is more detailed.

New and Better Commands

Enhanced client management
- redis-cli supports cluster management.
  In Redis 4.0 and earlier versions, the redis-trib module needs to be installed to manage clusters.
  
  Redis 5.0 optimizes redis-cli, integrating all cluster management functions. You can run the redis-cli --cluster help command for more information.
- The client performance is enhanced in frequent connection and disconnection scenarios.
  This optimization is valuable when your application needs to use short connections.
Simpler use of sorted sets
ZPOPMIN and ZPOPMAX commands are added for sorted sets.
- ZPOPMIN key [count]
  Removes and returns up to count members with the lowest scores in the sorted set stored at key. When returning multiple elements, the one with the lowest score will be the first, followed by the elements with higher scores.
- ZPOPMAX key [count]
  Removes and returns up to count members with the highest scores in the sorted set stored at key. When returning multiple elements, the one with the lowest score will be the first, followed by the elements with lower scores.

More sub-commands added to the help command

The help command can be used to view help information, saving you the trouble of visiting redis.io every time. For example, run the following command to view the stream help information: xinfo help

127.0.0.1:6379> xinfo help
1) XINFO <subcommand> arg arg ... arg. Subcommands are:
2) CONSUMERS <key> <groupname>  -- Show consumer groups of group <groupname>.
3) GROUPS <key>                 -- Show the stream consumer groups.
4) STREAM <key>                 -- Show information about the stream.
5) HELP                         -- Print this help.
127.0.0.1:6379>

redis-cli command input tips
After you enter a complete command, redis-cli displays a parameter tip to help you memorize the syntax format of the command.

As shown in the following figure, run the zadd command, and redis-cli displays zadd syntax in light color.

RDB Storing LFU and LRU Information

In Redis 5.0, storage key eviction policies LRU and LFU were added to the RDB snapshot file.

FIFO: First in, first out. The earliest stored data is evicted first.
LRU: Least recently used. Data that is not used for a long time is evicted first.
LFU: Least frequently used. Data that is least frequently used is evicted first.

The RDB file format of Redis 5.0 is modified and is backward compatible. Therefore, if a snapshot is used for migration, data can be migrated from the earlier Redis versions to Redis 5.0, but cannot be migrated from the Redis 5.0 to the earlier versions.

Parent topic: Instance Types/Versions

Previous topic: New Features of DCS for Redis 4.0

Next topic: New Features of DCS for Redis 6.0