Appendix: Methods for Improving the Message Processing Efficiency

Optimizing the Acknowledgment Process of Message Production and Consumption

Message Production

The producer decides whether to re-send the message based on the ROMA Connect response.

Each time the producer sends a message, it waits for an API response to confirm that the message is successfully sent. If an exception occurs when sending the message, the producer will not receive a success response and must decide whether to re-send the message. If a success response is received, it indicates that the message has been stored in ROMA Connect.

Message Consumption

The consumer acknowledges successful message retrieval.

Messages are stored in ROMA Connect in the order that they are created. During message retrieval, the consumer obtains messages stored in ROMA Connect in the order that they are stored. After the consumer retrieves the messages, the message retrieval status is recorded as successful or failed. The status is then submitted to ROMA Connect. Based on the retrieval status, ROMA Connect determines whether to retrieve the next batch of messages or retrieve the messages that failed to be retrieved.

During this process, the message retrieval status may not be successfully submitted due to an exception. In this case, the corresponding messages will be re-obtained by the consumer in the next message retrieval request.

Idempotent Transferring of Message Production and Consumption

ROMA Connect provides a series of reliability measures to ensure that messages are not lost. For example, the message synchronization storage mechanism is used to prevent the system and server from being abnormally restarted or powered off. The ACK mechanism is used to solve the exceptions that occur during message transmission.

Considering the extreme conditions such as network exceptions, you need to use ROMA Connect to design message sending and consumption in addition to confirming message production and consumption.

• If the message sending cannot be confirmed, the producer needs to send the message to ROMA Connect repeatedly.
• After consuming a message that has been processed, the consumer needs to notify that ROMA Connect consumption is successful and ensure that the message is not processed repeatedly.

Producing and Consuming Messages in Batches

To improve the message sending and consumption efficiency, consumers are advised to use the batch message sending and consumption mode. Generally, messages are consumed in batches by default, and messages are sent in batches if possible, which effectively reduces the number of API calls.

Refer to the following two figures.

Figure 1 Producing and consuming messages in batches

A maximum of 10 messages can be sent in batches. The total size cannot exceed 512 KB.

Message production (sending) in batches can be flexibly used. When there are a large number of concurrent messages, the messages are sent in batches. When the number of concurrent messages is small, the messages are sent one by one. This is done to reduce the number of API calls and ensure real-time message sending.

Figure 2 Producing and consuming messages one by one

When consuming messages in batches, consumers need to process and confirm messages in the sequence of receiving messages. Therefore, when a message in the batch fails to be consumed, the consumer does not need to consume the rest messages, and directly submit consumption confirmations of the successfully consumed messages to ROMA Connect.

Using Consumer Groups to Assist O&M

You can use ROMA Connect as a message management system. Reading message content from queues is helpful to fault locating and service debugging.

When problems occur during message production and consumption, you can create different consumer groups to locate and analyze problems or debug services for interconnecting with other services. To ensure that other services can continue to process messages in topics, you can create a consumer group to retrieve and analyze the messages.