Other Services Are Unstable When Too Many HBase Connections Occupy the Network Ports

Question

What should I do if HBase connections occupy too many network ports and other services become unstable.

Answer

Run the OS command lsof or netstat. A large number of TCP connections are in the CLOSE_WAIT state and the connection holder is HBase RegionServer. As a result, network ports may be used up or the number of HDFS connections may exceed the threshold, causing service instability. HBase connections can get stuck in a CLOSE_WAIT state due to the HBase mechanism.

HBase creates an HDFS connection when it creates or loads a StoreFile, which is HBase data stored in HFlile. When the creation or load task is complete, HDFS can close the connection. However, to ensure real-time response, HBase keeps the connection open to access the data file when needed. This results in the CLOSE_WAIT status, indicating that the client is waiting for a close request.

A StoreFile is created when HBase executes flush.

Data is written to HBase memory first, stored in the memstore. The flush operation is triggered when memory usage reaches the threshold or a manual flush command is executed, writing data to HDFS.

Solutions:

To reduce HBase port usage, control the number of StoreFiles by triggering HBase compaction, which combines files. Trigger HBase compaction by either of the following methods:

Method 1: Use the HBase shell client and perform the major_compact operation on the client.

Method 2: Write the HBase client code and call the compact method in the HBaseAdmin class to trigger the compaction in HBase.

If the compaction cannot solve HBase port occupation, the HBase usage has reached the limit. Check the following items:

• Whether the initial number of regions of the table is proper
• Whether useless data exists

If useless data exists, delete the data to reduce the number of files stored in HBase. If the problem persists, scale-out is required.