Configuring ulimit for HBase and HDFS

Updated on 2024-11-29 GMT+08:00

View PDF

Symptom

When an HDFS file is opened and the number of handles is limited, the following error occurs:

IOException (Too many open files)

Procedure

You can contact the cluster administrator to increase the number of handles for each user. This is a configuration on the OS instead of that for HBase or HDFS. It is recommended that the cluster administrator set the number of handles based on the service volume of HBase and HDFS and the permissions of each user. If a user needs to frequently perform many operations on the HDFS with heavy service volume, set a large number of handles for the user to avoid the preceding error.

Log in to operating systems of all nodes or clients in the cluster as user root and go to the /etc/security directory.
Run the following command to edit the limits.conf file:

vi limits.conf

Added the following contents:
```
hdfs  -       nofile  32768 
hbase -       nofile  32768
```
hdfs and hbase indicate the OS user names used in services.
NOTE:
- Only user root has the permission to edit the limits.conf file.
- If the modification does not take effect, check whether there are other nofile values for the OS user in the /etc/security/limits.d directory. Such values may overwrite the value configured in /etc/security/limits.conf.
- If a user needs to perform operations on HBase, it is recommended that the handle count of the user be set to more than 10,000. If a user needs to perform operations on HDFS, it is recommended that the handle count be set based on the service volume. (A small value is not recommended.) If a user needs to perform operations on HBase and HDFS, it is recommended that the handle count be set to a large value, for example, 32,768.
You can run the following command to view the maximum number of handles of a user:

su - user_name

ulimit -n

The command output displays the maximum number of handles of the user. The following is an example.
```
8194
```

Parent topic: Using HDFS

Previous topic: Planning HDFS Capacity

Next topic: Configuring HDFS DataNode Data Balancing

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot