Relationships Between ZooKeeper and Other Components

Hadoop Distributed File System (HDFS)

Figure 1 shows the relationship between ZooKeeper and HDFS.

Figure 1 Relationship between ZooKeeper and HDFS

As the client of a ZooKeeper cluster, ZKFailoverController (ZKFC) monitors the status of NameNode. ZKFC is deployed only in the node where NameNode is deployed, and in both the active and standby HDFS NameNodes.

1. The ZKFC connects to ZooKeeper and saves information such as host names to ZooKeeper under the znode directory /hadoop-ha. NameNode that creates the directory first is considered as the active node, and the other is the standby node. NameNodes read the NameNode information periodically through ZooKeeper.
2. When the process of the active node ends abnormally, the standby NameNode detects changes in the /hadoop-ha directory through ZooKeeper, and then takes over the service of the active NameNode.

YARN

Figure 2 shows the relationship between ZooKeeper and YARN.

Figure 2 Relationship Between ZooKeeper and YARN

1. When the system is started, ResourceManager attempts to write state information to ZooKeeper. ResourceManager that first writes state information to ZooKeeper is selected as the active ResourceManager, and others are standby ResourceManagers. The standby ResourceManagers periodically monitor active ResourceManager election information in ZooKeeper.
2. The active ResourceManager creates the Statestore directory in ZooKeeper to store application information. If the active ResourceManager is faulty, the standby ResourceManager obtains application information from the Statestore directory and restores the data.

HBase

Figure 3 shows the relationship between ZooKeeper and HBase.

Figure 3 Relationship between ZooKeeper and HBase

1. RegionServer registers itself to ZooKeeper on Ephemeral node. ZooKeeper stores the HBase information, including the HBase metadata and HMaster addresses.
2. HMaster detects the health status of each RegionServer using ZooKeeper, and monitors them.
3. HBase supports multiple HMaster nodes (like HDFS NameNodes). When the active HMatser is faulty, the standby HMaster obtains the state information about the entire cluster using ZooKeeper. That is, using ZooKeeper can avoid HBase SPOFs.

Kafka

Figure 4 shows the relationship between ZooKeeper and Kafka.

Figure 4 Relationship between ZooKeeper and Kafka

1. Broker uses ZooKeeper to register broker information and elect a partition leader.
2. The consumer uses ZooKeeper to register consumer information, including the partition list of consumer. In addition, ZooKeeper is used to discover the broker list, establish a socket connection with the partition leader, and obtain messages.