Configuring ulimit for HBase and HDFS

Symptom

When an HDFS file is opened and the number of handles is limited, the following error occurs:

IOException (Too many open files)

Procedure

You can contact the cluster administrator to increase the number of handles for each user. This is a configuration on the OS instead of that for HBase or HDFS. It is recommended that the cluster administrator set the number of handles based on the service volume of HBase and HDFS and the permissions of each user. If a user needs to frequently perform many operations on the HDFS with heavy service volume, set a large number of handles for the user to avoid the preceding error.

Log in to operating systems of all nodes or clients in the cluster as user root and go to the /etc/security directory.
Run the following command to edit the limits.conf file:

vi limits.conf

Added the following contents:
```
hdfs  -       nofile  32768 
hbase -       nofile  32768
```
hdfs and hbase indicate the OS user names used in services.
- Only user root has the permission to edit the limits.conf file.
- If the modification does not take effect, check whether there are other nofile values for the OS user in the /etc/security/limits.d directory. Such values may overwrite the value configured in /etc/security/limits.conf.
- If a user needs to perform operations on HBase, it is recommended that the handle count of the user be set to more than 10,000. If a user needs to perform operations on HDFS, it is recommended that the handle count be set based on the service volume. (A small value is not recommended.) If a user needs to perform operations on HBase and HDFS, it is recommended that the handle count be set to a large value, for example, 32,768.
You can run the following command to view the maximum number of handles of a user:

su - user_name

ulimit -n

The command output displays the maximum number of handles of the user. The following is an example.
```
8194
```