Optimizing RabbitMQ Performance
Keep the queue length as short as possible.
Many messages may stack in queues, causing high memory usage. To release memory, RabbitMQ dumps these messages to disks, which is time-consuming. As a result, the message processing becomes slow or the production is blocked. Therefore, message stack may cause negative effects on brokers. Moreover, if a broker crashes and restarts, rebuilding indexes for much data and broker synchronization in the cluster mode are time-consuming.
Use lazy queues for stability.
Lazy queues is a new feature for RabbitMQ 3.6 and later. Messages in lazy queues are automatically stored to disks, which reduces memory usage but increases I/O workload and affects throughput. Lazy queues can better control performance and stabilize clusters. In contrast to non-lazy queues, their messages are stacked in memory and flushed to disks at a time when the memory is insufficient, causing unstable queue performance. To send a large number of messages at a time, or require the consumption speed to catch up with the production speed, lazy queues are recommended. Otherwise, do not use lazy queues when: a. high performance is needed b. queues are not stacked c. queues are configured with the max-length policy
Limit the queue length using TTL or max-length.
Message or queue TTL, and max-length can be used to limit the queue length. If the queue length reaches the max-length, the messages at the queue head are discarded or enter dead letter queues. Expired messages are also discarded or enter dead letter queues.
Note the queue quantity.
In RabbitMQ, a queue is processed by a thread. To obtain a high throughput, create multiple queues using the multi-core and distributed features of a server, and distribute different queues to different CPUs or different nodes. In the meantime, many queues may overload the CPU and memory, or slow down the response of RabbitMQ management APIs.
Allocate a queue name to a temporary queue automatically.
To use a temporary queue (e.g., an exclusive queue, automatic deletion queue, or non-persistent queue), RabbitMQ can automatically allocate a queue name by calling queueDeclare().
Use automatic deletion queues as required.
If queues that are no longer used are stored on a server for a long time, RabbitMQ performance may be affected. They can be automatically deleted by TTL, auto-delete, or exclusive settings.
Use priority queues under control.
Each priority uses an internal queue in the Erlang VM, which consumes some resources. Up to five priorities are enough in most scenarios.
Determine the message size.
Specifying the length of messages sent to RabbitMQ is a common problem. Note that the number of messages sent per second is more likely to hit the bottleneck than the message size. Either sending large messages, or sending multiple small messages is a good choice. It is better that the producer encapsulates multiple small messages into a large one, and then the consumer splits the message for processing. But processing a large message that contains many small messages is slow. In addition, if a small message fails to be processed, the entire large message needs to be resent. Therefore, consider the bandwidth and service architecture in determining the message size.
Connections and channels
A connection occupies about 100 KB memory (100+ KB when TLS is enabled). Thousands of connections cause heavy load on RabbitMQ servers. In extreme cases, RabbitMQ servers may break down due to OOM. The AMQP protocol can implement multiplexing over a single TCP connection. Create one TCP connection for a process. Each thread reuses this connection to create its own channel. The connection must have a long lifecycle because its creation is costly (of at least seven TCP packets). On the contrary, a channel can often be enabled and disabled. But sending messages reusing a channel is a good practice. Do not open a new channel every time a message is sent. Note the following:
- For thread safety, do not share channels among multiple threads.
- For proper latency, do not open and close connections or channels frequently.
- For high throughput, producers and consumers use separate connections.
- To avoid request timeout, do not use many connections and channels because they may affect API performance.
Message acknowledgment
To avoid message loss due to connection issues, consumers use the acknowledgment mechanism. Clients can return an ack message to servers after receiving or processing messages. This mechanism affects performance. If high throughput is required, disable manual acknowledgment. If messages are important to services, the messages should be acknowledged after they are processed. Similarly, producers use the confirm mechanism. When a server receives a message from a producer, the server returns an ack message to implement at-least-once delivery. Note, the confirm mechanism also affects performance.
Control the number of unacknowledged messages.
All unacknowledged messages are temporarily stored in the memory. Many unacknowledged messages may cause service OOM. To limit the number of unacknowledged messages, you can enable the prefetch function on the consumer which limits the maximum number of messages that can be pulled.
Persistent resources
To prevent message loss from service breakdown, restart, or hardware faults, use persistent queues and messages. Persistent messages involve disk writes. When lazy queues are used, all messages including non-persistent messages are written to disks. If high performance is required, non-persistent messages can be used.
Enable TLS connections.
On AMQP, you can use TLS to connect to RabbitMQ. TLS encrypts and decrypts data, which affects performance.
Select proper QoS.
- When there is single or a few consumers and the consumption is fast, use large QoS to keep the client busy.
- If the message processing speed and bandwidth on the client remain unchanged, estimate proper QoS using formula RTT/Processing duration per message.
- When there are many consumers and the message processing is fast, use small QoS.
- When there are many consumers and the message processing is slow, you are advised to set QoS to 1 so that messages are evenly distributed to all consumers.
Note, if the client is pull-based or has auto-ack enabled, the prefetch function will not work. There is a common error: The number of prefetched messages is not limited. All messages are sent to one consumer. As a result, consumer OOM happens and messages are repeatedly delivered. For more information about the RabbitMQ prefetch function, see here.
Performance metrics
Metric ID |
Metric Name |
Description |
channels |
Channels |
Number of channels in the RabbitMQ instance |
queues |
Queues |
Number of queues in the RabbitMQ instance |
connections |
Connections |
Number of connections in the RabbitMQ instance |
connections_usage |
Connection Usage |
Percentage of current connections to the maximum number of connections |
rabbitmq_disk_usage |
Disk Capacity Usage |
Disk usage of the RabbitMQ VM |
rabbitmq_cpu_usage |
CPU Usage |
CPU usage of the RabbitMQ VM |
rabbitmq_memory_usage |
Memory Usage |
Memory usage of the RabbitMQ VM |
rabbitmq_cpu_core_load |
Average Load per CPU Core |
Average load of each CPU core of the RabbitMQ VM |
Learn more about RabbitMQ Metrics.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot