文档首页/ MapReduce服务 MRS/ 故障排除/ 使用Kafka/ 执行Kafka Topic删除操作时发现无法删除
更新时间:2023-11-18 GMT+08:00
分享

执行Kafka Topic删除操作时发现无法删除

问题背景与现象

在使用Kafka客户端命令删除Topic时,发现Topic无法被删除。

kafka-topics.sh --delete --topic test --zookeeper 192.168.234.231:2181/kafka

可能原因

  • 客户端命令连接ZooKeeper地址错误。
  • Kafka服务异常Kafka部分节点处于停止状态。
  • Kafka服务端配置禁止删除。
  • Kafka配置自动创建,且Producer未停止。

原因分析

  1. 客户端命令,打印ZkTimeoutException异常。
    [2016-03-09 10:41:45,773] WARN Can not get the principle name from server 192.168.234.231 (org.apache.zookeeper.ClientCnxn)
    Exception in thread "main" org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 30000
    at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:880)
    at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98)
    at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84)
    at kafka.admin.TopicCommand$.main(TopicCommand.scala:51)
    at kafka.admin.TopicCommand.main(TopicCommand.scala)

    解决方法参考1

  2. 客户端查询命令。
    kafka-topics.sh --list --zookeeper 192.168.0.122:2181/kafka
    test - marked for deletion

    通过Manager查看Kafka Broker实例的运行状态。

    通过cd /var/log/Bigdata/kafka/broker命令进入RunningAsController节点日志目录,在controller.log发现ineligible for deletion: test。

    2016-03-09 11:11:26,228 | INFO  | [main] | [Controller 1]: List of topics to be deleted:  | kafka.controller.KafkaController (Logging.scala:68)
    2016-03-09 11:11:26,229 | INFO  | [main] | [Controller 1]: List of topics ineligible for deletion: test | kafka.controller.KafkaController (Logging.scala:68)
  3. 通过Manager查询Broker删除Topic相关配置。

    解决方法参考2

  4. 客户端查询命令:
    kafka-topics.sh --describe -topic test --zookeeper 192.168.0.122:2181/kafka

    进入RunningAsController节点日志目录,在controller.log发现marked ineligible for deletion。

    2016-03-10 11:11:17,989 | INFO  | [delete-topics-thread-3] | [delete-topics-thread-3], Handling deletion for topics test | kafka.controller.TopicDeletionManager$DeleteTopicsThread (Logging.scala:68)
    2016-03-10 11:11:17,990 | INFO  | [delete-topics-thread-3] | [delete-topics-thread-3], Not retrying deletion of topic test at this time since it is marked ineligible for deletion | kafka.controller.TopicDeletionManager$DeleteTopicsThread (Logging.scala:68)
  5. 通过Manager查询Broker状态。

    其中一个Broker处于停止或者故障状态。Topic进行删除必须保证该Topic的所有Partition所在的Broker必须处于正常状态。

    解决方法参考3

  6. 进入RunningAsController节点日志目录,在controller.log发现Deletion successfully,然后又出现New topics: [Set(test)],表明被再次创建。
    2016-03-10 11:33:35,208 | INFO  | [delete-topics-thread-3] | [delete-topics-thread-3], Deletion of topic test successfully completed | kafka.controller.TopicDeletionManager$DeleteTopicsThread (Logging.scala:68)
    
    2016-03-10 11:33:38,501 | INFO  | [ZkClient-EventThread-19-192.168.0.122:2181,160.172.0.52:2181,160.172.0.51:2181/kafka] | [TopicChangeListener on Controller 3]: New topics: [Set(test)], deleted topics: [Set()], new partition replica assignment
  7. 通过Manager查询Broker创建Topic相关配置。

    经确认,对该Topic操作的应用没有停止。

    解决方法参考4

解决办法

  1. ZooKeeper连接失败导致。

    Kafka客户端连接ZooKeeper服务超时。检查客户端到ZooKeeper的网络连通性。

    网络连接失败,通过Manager界面查看Zookeeper服务信息。

    图1 Zookeeper服务信息

    配置错误,修改客户端命令中ZooKeeper地址。

  2. Kafka服务端配置禁止删除。

    通过Manager界面修改delete.topic.enable为true。保存配置并重启服务。

    图2 修改delete.topic.enable

    客户端查询命令,无Topic:test。

    kafka-topics.sh --list --zookeeper 192.168.0.122:24002/kafka

    进入RunningAsController节点日志目录,在controller.log发现Deletion of topic test successfully。

    2016-03-10 10:39:40,665 | INFO  | [delete-topics-thread-3] | [Partition state machine on Controller 3]: Invoking state change to OfflinePartition for partitions [test,2],[test,15],[test,6],[test,16],[test,12],[test,7],[test,10],[test,13],[test,9],[test,19],[test,3],[test,5],[test,1],[test,0],[test,17],[test,8],[test,4],[test,11],[test,14],[test,18] | kafka.controller.PartitionStateMachine (Logging.scala:68)
    2016-03-10 10:39:40,668 | INFO  | [delete-topics-thread-3] | [Partition state machine on Controller 3]: Invoking state change to NonExistentPartition for partitions [test,2],[test,15],[test,6],[test,16],[test,12],[test,7],[test,10],[test,13],[test,9],[test,19],[test,3],[test,5],[test,1],[test,0],[test,17],[test,8],[test,4],[test,11],[test,14],[test,18] | kafka.controller.PartitionStateMachine (Logging.scala:68)
    2016-03-10 10:39:40,977 | INFO  | [delete-topics-thread-3] | [delete-topics-thread-3], Deletion of topic test successfully completed | kafka.controller.TopicDeletionManager$DeleteTopicsThread (Logging.scala:68)
    

  3. Kafka部分节点处于停止或者故障状态。

    启动停止的Broker实例。

    客户端查询命令,无Topic:test。

    kafka-topics.sh --list --zookeeper 192.168.0.122:24002/kafka

    进入RunningAsController节点日志目录,在controller.log发现Deletion of topic test successfully。

    2016-03-10 11:17:56,463 | INFO  | [delete-topics-thread-3] | [Partition state machine on Controller 3]: Invoking state change to NonExistentPartition for partitions [test,4],[test,1],[test,8],[test,2],[test,5],[test,9],[test,7],[test,6],[test,0],[test,3] | kafka.controller.PartitionStateMachine (Logging.scala:68)
    2016-03-10 11:17:56,726 | INFO  | [delete-topics-thread-3] | [delete-topics-thread-3], Deletion of topic test successfully completed | kafka.controller.TopicDeletionManager$DeleteTopicsThread (Logging.scala:68)
    

  4. Kafka配置自动创建,且Producer未停止。

    停止相关应用,通过Manager界面修改“auto.create.topics.enable”“false”,保存配置并重启服务。

    图3 修改auto.create.topics.enable

  5. 再次执行delete操作。

相关文档