Methods for Improving the Message Processing Efficiency

Paying Attention to the Confirmation Process of Message Production and Consumption

Message production (sending)

After messages are sent, the producer checks the message sending confirmation returned from DMS. If messages fail to be sent, the producer re-sends the messages.

After messages are produced, the producer waits for the acknowledgement (ACK) returned by the message sending API to determine whether messages are sent successfully. During message sending, if any abnormality occurs and the producer fails to receive the ACK, the producer determines whether to re-send messages. If the producer receives the ACK indicating that the message has been successfully sent, the message is reliably stored by DMS.

Message consumption

When a message is consumed, the consumer needs to check whether the message consumption is successful.

Produced messages are stored in the storage medium of DMS. During message consuming, the consumer obtains messages stored in DMS. Then the consumer consumes the messages, records message consumption status (successful or failed), and submits the consumption status to DMS. Based on the consumption status, DMS determines whether to consume next batch of messages or re-consume the messages that failed to be consumed.

Throughout the entire message consuming process, if the message consumption status fails to be submitted due to some abnormalities, this batch of messages will be re-obtained by the consumer in the subsequent message consumption requests.

Idempotent Transferring of Message Production and Consumption

DMS provides a series of reliability measures to ensure that messages are not lost. For example, the message synchronization storage mechanism is used to prevent the system and server from being abnormally restarted or powered off. The ACK mechanism is used to solve the exceptions that occur during message transmission.

Considering the extreme conditions such as network exceptions, you need to cooperate with DMS to design message sending and consumption in addition to confirming message production and consumption.

• If the message fails to be confirmed, the producer needs to send the message to DMS repeatedly.
• After receiving a message that has been processed, the consumer needs to notify that DMS consumption is successful and ensure that the message is not processed repeatedly.

Producing and Consuming Messages in Batches

To improve the message sending and consumption efficiency, consumers are advised to use the batch message sending and consumption mode, which can effectively lower the number of API calls and minimize service fees,

as shown in the following figures.

Figure 1 Message production (sending) and consumption in batches

A maximum of 10 messages can be sent in batches. The total size cannot exceed 512 KB.

Batch production (sending) messages can be flexibly used. When there are a large number of concurrent messages, the messages are sent in batches. When the number of concurrent messages is small, the messages are sent one by one. In this way, the number of API calls is reduced and the real-time message sending is ensured.

Figure 2 Message production (sending) and consumption one by one

In addition, consumers need to process and confirm messages in the sequence of receiving messages in batches. Therefore, when a consumer fails to consume a message, you are advised to stop the consumer from consuming the rest messages, and directly submit consumption confirmations of the successfully consumed messages to DMS.

Using Consumer Groups to Assist O&M

You can use DMS as the message management system. Viewing the message content of a queue is crucial for locating problems and debugging services.

When problems occur during message production and consumption, you can create different consumer groups to locate and analyze problems or debug service interconnection. You can create a consumer group to consume messages in queues and analyze the consumption process. This does not affect the message processing of other services.