Comparing Kafka Instances and DMS Advanced Queues

Open-Source Compatibility

• DMS advanced queues:

  Kafka 0.10.2.1

• Kafka premium instances:

  Kafka 1.1.0, 2.3.0, and 2.7

  With each version upgrade, Apache Kafka introduces new features, improves APIs, and updates producer and consumer configuration files. To check whether your application features and APIs are compatible with your Kafka clients, see the upgrade notes on the official Apache Kafka website.

Creation

• DMS advanced queues:

  An advanced queue (equivalent to a topic) is created on the DMS console. You do not need to configure the storage space or the bandwidth because these resources are allocated by the system.

• Kafka premium instances:

  A Kafka premium instance is created on the DMS for Kafka console. Before creating a Kafka premium instance, determine the required bandwidth and storage space based on your service expectations for the next one or two years. You also need to prepare a VPC and security group for the instance.

  After the instance has been created, you must create topics in the instance and configure the number of partitions and replicas for the topics.

Usage

• DMS advanced queues:

  Advanced queues are compatible with Kafka APIs. DMS provides SDKs in Java, Python, Lua, C, and Go languages.

  To use open-source Kafka clients, see "Using the Enhanced Java SDK" in Developer Guide. Add the enhanced Kafka Java SDK provided by DMS for Kafka to the directory of the open-source client package and then pass the security authentication.

• Kafka premium instances:

  DMS for Kafka is fully compatible with open-source Kafka. You can access Kafka premium instances and topics using open-source Kafka clients. If SASL access is enabled, you must use the SSL certificate provided by Kafka.

Performance

• DMS advanced queues:

  There are two queue modes: high-throughput and high-reliability. In the high-throughput mode, messages are flushed to disk asynchronously, ensuring high concurrency.

• Kafka premium instances:

  Compute, bandwidth, and storage resources are physically isolated for each instance. Determine the required bandwidth and storage space when creating an instance. For storage space, you can choose Ultra-high I/O, which indicates that messages are stored on SSDs.

Other Dimensions

You can customize the number of partitions and replicas for a Kafka premium instance. Each topic can have 1 to 100 partitions and 1 to 3 replicas.

By default, an advanced queue has three partitions and three replicas.

Divide a topic into a certain number of partitions so that messages can be evenly distributed to partitions, enabling load balancing and horizontal scalability. Different consumers can retrieve messages from one or more partitions, improving message processing performance.

With more replicas come higher reliability. However, synchronizing messages between replicas consumes bandwidth and offsets compute performance.