Help Center/ Distributed Message Service for Kafka/ User Guide (ME-Abu Dhabi Region)/ Service Overview/ Specifications
Updated on 2023-05-06 GMT+08:00
View PDF
Share

Specifications

Kafka Instance Specifications

Kafka instances are compatible with open-source Kafka v1.1.0, v2.3.0, and v2.7. The instance specifications are represented by the ECS flavor and the number of brokers. Available options are kafka.2u4g.cluster, kafka.4u8g.cluster, kafka.8u16g.cluster, kafka.12u24g.cluster, and kafka.16u32g.cluster.

In the following table, transactions per second (TPS) are calculated assuming that the size of a message is 1 KB.

Table 1 Kafka instance specifications

Flavor

Brokers

Maximum TPS per Broker

Maximum Partitions per Broker

Maximum Consumer Groups per Broker

Maximum Client Connections per Broker

Storage Space

kafka.2u4g.cluster

3–30

30,000

250

20

2000

300 GB–300,000 GB

kafka.4u8g.cluster

3–30

100,000

500

100

4000

300 GB–600,000 GB

kafka.8u16g.cluster

3–30

150,000

1000

150

4000

300 GB–900,000 GB

kafka.12u24g.cluster

3–30

200,000

1500

200

4000

300 GB–900,000 GB

kafka.16u32g.cluster

3–30

250,000

2000

200

4000

300 GB–900,000 GB

Flavor Selection

  • kafka.2u4g.cluster with 3 brokers

    Recommended for up to 3000 client connections, 60 consumer groups, and 100,000 TPS

  • kafka.4u8g.cluster with 3 brokers

    Recommended for up to 10,000 client connections, 300 consumer groups, and 300,000 TPS

  • kafka.8u16g.cluster with 3 brokers

    Recommended for up to 20,000 client connections, 600 consumer groups, and 600,000 TPS

  • kafka.12u24g.cluster with 3 brokers

    Recommended for up to 20,000 client connections, 600 consumer groups, and 900,000 TPS

  • kafka.16u32g.cluster with 3 brokers

    Recommended for up to 20,000 client connections, 600 consumer groups, and 1,200,000 TPS

Storage Space Selection

Kafka instances support multi-replica storage. The storage space is consumed by all replicas. When creating an instance, specify its storage space based on the expected service message size and the number of replicas.

For example, if the estimated message size is 100 GB, the disk capacity must be at least: 100 GB x Number of replicas + 100 GB (reserved space).

The storage space can be expanded as your service grows.

Topic Quantity

There are limits on the topic quantity and the aggregate number of partitions in the topics. When the partition quantity limit is reached, you can no longer create topics.

The number of topics is related to the maximum number of partitions allowed (see Figure 1) and the specified number of partitions in each topic (see Table 1).

Figure 1 Setting the number of partitions

The maximum number of partitions allowed for an instance with kafka.2u4g.cluster and 3 brokers is 750.

  • If the number of partitions of each topic in the instance is 3, the maximum number of topics is 750/3 = 250.
  • If the number of partitions of each topic in the instance is 1, the maximum number of topics is 750/1 = 750.
Parent topic: Service Overview