ALM-25005 nscd Service Exception

Alarm Description

The system checks the status of the nscd service every 60 seconds. This alarm is generated when the nscd process fails to be queried for four consecutive times (three minutes) or users in LdapServer cannot be obtained.

This alarm is cleared when the process is restored and users in LdapServer can be obtained.

Alarm Attributes

Alarm Parameters

Impact on the System

The alarmed node may not be able to synchronize data from LdapServer. The id command may fail to obtain the LDAP data, affecting upper-layer services.

Possible Causes

• The nscd service is not started.
• The network is faulty, and cannot access the LDAP server.
• NameService is abnormal.
• Users cannot be queried because the OS executes commands too slowly.

Handling Procedure

Check whether the nscd service is started.

1. Log in to FusionInsight Manager and choose O&M > Alarm > Alarms. Record the IP address of HostName in Location of the alarm as IP1 (if multiple alarms exist, record the IP addresses as IP1, IP2, and IP3 respectively).
2. Contact the O&M engineers to access the node using IP1 as user root. Run the ps -ef | grep nscd command on the node and check whether the /usr/sbin/nscd process is started.
   • If yes, go to 5.
   • If no, go to 3.

3. Run the service nscd restart command as user root to restart the nscd service. Then run the ps -ef | grep nscd command to check whether the nscd service is started.
   • If yes, go to 4.
   • If no, go to 15.

4. Wait for 5 minutes and run the ps -ef | grep nscd command again as user root. Check whether the service exists.
   • If yes, go to 11.
   • If no, go to 15.

Check whether the network is faulty, and whether the LDAP server can be accessed.

5. Log in to the alarmed node as user root and run the ping command to check whether the network connectivity between this node and the LdapServer node. is normal.
   • If yes, go to 6.
   • If no, contact network administrators to troubleshoot the fault.

Check whether the NameService is normal.

6. Log in to the alarmed node as user root. Run the cat /etc/nsswitch.conf command to check whether the passwd, group, services, netgroup, and aliases of NameService are correctly configured.

   The correct parameter configurations are as follows:

   passwd: compat ldap; group: compat ldap; services: files ldap; netgroup: files ldap; aliases: files ldap

   • If yes, go to 7.
   • If no, go to 9.

7. Log in to the alarmed node as user root. Run the cat /etc/nscd.conf command to check whether the enable-cache passwd, positive-time-to-live passwd, enable-cache group, and positive-time-to-live group in the configuration file are correctly configured.

   The correct parameter configurations are as follows:

   enable-cache passwd: yes; positive-time-to-live passwd: 600; enable-cache group: yes; positive-time-to-live group: 3600

   • If yes, go to 8.
   • If no, go to 10.

8. Run the /usr/sbin/nscd -i group and /usr/sbin/nscd -i passwd commands as user root. Wait for 2 minutes and run the id admin and id backup/manager commands to check whether results can be queried.
   • If yes, go to 11.
   • If no, go to 15.

9. Run the vi /etc/nsswitch.conf command as user root. Correct the configurations in 6 and save the file. Run the service nscd restart command to restart the nscd service. Wait for 2 minutes and run the id admin and id backup/manager commands to check whether results can be queried.
   • If yes, go to 11.
   • If no, go to 15.

10. Run the vi /etc/nscd.conf command as user root. Correct the configurations in 7 and save the file. Run the service nscd restart command to restart the nscd service. Wait for 2 minutes and run the id admin and id backup/manager commands to check whether results can be queried.
    • If yes, go to 11.
    • If no, go to 15.

11. Log in to the FusionInsight Manager portal. Wait for 5 minutes and check whether the nscd Service Exception alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 12.

Check whether frame freezing occurs when running a command in the operating system.

12. Log in to the faulty node as user root, run the id admin command, and check whether the command execution takes a long time. If the command execution takes more than 3 seconds, the command execution is deemed to be slow.
    • If yes, go to 13.
    • If no, go to 15.

13. Run the cat /var/log/messages command to check whether the nscd frequently restarts or the error information "Can't contact LDAP server" exists.

    nscd exception example:

    Feb 11 11:44:42 10-120-205-33 nscd: nss_ldap: failed to bind to LDAP server ldaps://10.120.205.55:21780: Can't contact LDAP server
    Feb 11 11:44:43 10-120-205-33 ntpq: nss_ldap: failed to bind to LDAP server ldaps://10.120.205.55:21780: Can't contact LDAP server
    Feb 11 11:44:44 10-120-205-33 ntpq: nss_ldap: failed to bind to LDAP server ldaps://10.120.205.92:21780: Can't contact LDAP server

    • If yes, go to 14.
    • If no, go to 15.

14. Run the vi$BIGDATA_HOME/tmp/random_ldap_ip_order command to modify the number at the end. If the original number is an odd number, change it to an even number. If the number is an even number, change it to an odd number.

    Run the vi /etc/ldap.conf command to enter the editing mode, press Insert to start editing, and then change the first two IP addresses of the URI configuration item.

    After the modification is complete, press Esc to exit the editing mode and enter :wq! to save the settings and exit.

    Run the service nscd restart command to restart the nscd service. Wait 5 minutes and run the id admin command again. Check whether the command execution is slow.

    • If yes, go to 15.
    • If no, log in to other faulty nodes and repeat 12 to 14 to check whether the first LdapServer node in the URI before modifying /etc/ldap.conf is faulty. For example, check whether the service IP address is unreachable, the network delay is too long, or other abnormal software is deployed.

Collect the fault information.

15. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
16. Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select LdapClient for the target cluster.
17. Click in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
18. Contact O&M engineers and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None.