Setting the Number Limit of HBase and HDFS Handles

Scenario

When an HDFS file is opened and the number of handles is limited, the following error occurs:

IOException (Too many open files)

Setting the Number Limit of HBase and HDFS Handles

To increase the number of handles for each user, contact the cluster administrator. This is a configuration on the OS instead of that for HBase or HDFS. It is recommended that the cluster administrator set the number of handles based on the service volume of HBase and HDFS and the permissions of each user. If a user needs to frequently perform many operations on the HDFS with heavy service volume, set a large number of handles for the user to avoid the preceding error.

1. Log in to operating systems of all nodes or clients in the cluster as user root and go to the /etc/security directory.
2. Run the following command to edit the limits.conf file:

   vi limits.conf

   Added the following contents:

   hdfs  -       nofile  32768 
   hbase -       nofile  32768

   hdfs and hbase indicate the OS user names used in services.

   

   • Only user root has the permission to edit the limits.conf file.
   • If the modification does not take effect, check whether there are other nofile values for the OS user in the /etc/security/limits.d directory. Such values may overwrite the value configured in /etc/security/limits.conf.
   • If a user needs to perform operations on HBase, it is recommended that the handle count of the user be set to more than 10,000. If a user needs to perform operations on HDFS, it is recommended that the handle count be set based on the service volume. (A small value is not recommended.) If a user needs to perform operations on HBase and HDFS, it is recommended that the handle count be set to a large value, for example, 32,768.

3. Run the following command to view the maximum number of handles of a user:

   su - user_name

   ulimit -n

   The command output displays the maximum number of handles of the user. The following is an example.

   8194