Kafka Application Development Rules

Create Topics by calling Kafka APIs (AdminZkClient.createTopic)

• For Java programming languages, correct examples are as follows:

  import kafka.zk.AdminZkClient;
  import kafka.zk.KafkaZkClient;
  import kafka.admin.RackAwareMode;
  … 
  KafkaZkClient kafkaZkClient = KafkaZkClient.apply(zkUrl, JaasUtils.isZkSecurityEnabled(), zkSessionTimeoutMs, zkConnectionTimeoutMs, Int.MaxValue(), Time.SYSTEM, "", "", null);
  AdminZkClient adminZkClient = new AdminZkClient(kafkaZkClient);
  adminZkClient.createTopic(topic, partitions, replicas, new Properties(), RackAwareMode.Enforced$.MODULE$);
  …

• For Scala programming languages, correct examples are as follows:

  import kafka.zk.AdminZkClient;
  import kafka.zk.KafkaZkClient;
  … 
  val kafkaZkClient: KafkaZkClient = KafkaZkClient.apply(zkUrl, JaasUtils.isZkSecurityEnabled(), zkSessionTimeoutMs, zkConnectionTimeoutMs, Int.MaxValue, Time.SYSTEM, "", "")
  val adminZkClient: AdminZkClient = new AdminZkClient(kafkaZkClient)
  adminZkClient.createTopic(topic, partitions, replicas)

The number of Partition copies must be less than or equal to the number of nodes

Copies of Topic Partitions in Kafka are used for improving data reliability. Copies of the same Partition are distributed on different nodes. Therefore, the number of Partition copies must be less than or equal to the number of nodes.

Set the fetch.message.max.bytes parameter of the Consumer client

The value of fetch.message.max.bytes muster be equal to or greater than the maximum number of bytes of messages that the Producer client generates each time. If the value is too small, the messages generated by the Producer client cannot be consumed successfully by the Consumer client.