Help Center/ MapReduce Service/ User Guide (Kuala Lumpur Region)/ FusionInsight Manager Operation Guide (Applicable to 3.x)/ Alarm Reference (Applicable to MRS 3.x)/ ALM-25005 nscd Service Exception
Updated on 2022-08-12 GMT+08:00
View PDF
Share

ALM-25005 nscd Service Exception

Description

The system checks the status of the nscd service every 60 seconds. This alarm is generated when the nscd process fails to be queried for four consecutive times (three minutes) or users in LdapServer cannot be obtained.

This alarm is cleared when the process is restored and users in LdapServer can be obtained.

Attribute

Alarm ID

Alarm Severity

Auto Clear

25005

Major

Yes

Parameters

Name

Meaning

Source

Specifies the cluster for which the alarm is generated.

ServiceName

Specifies the service name for which the alarm is generated.

HostName

Specifies the object (host ID) for which the alarm is generated.

Impact on the System

The alarmed node may not be able to synchronize data from LdapServer. The id command may fail to obtain the LDAP data, affecting upper-layer services.

Possible Causes

  • The nscd service is not started.
  • The network is faulty and cannot access the LDAP server.
  • NameService is abnormal.
  • Users cannot be queried because the OS executes commands too slowly.

Procedure

Check whether the nscd service is started.

  1. On the FusionInsight Manager portal, choose O&M > Alarm > Alarms. Find the IP address of HostName in Location of the alarm and record it as IP1 (if multiple alarms exist, record the IP addresses as IP1, IP2, and IP3 respectively).
  2. Contact the O&M personnel to access the node using IP1 as user root. Run the ps -ef | grep nscd command and check whether the /usr/sbin/nscd process is started.

    • If the process is started, go to 5.
    • If the process is not started, go to 3.

  3. Run the service nscd restart command as user root to restart the nscd service. Then run the ps -ef | grep nscd command to check whether the nscd service is started.

    • If it is started, go to 4.
    • If it is not started, go to 15.

  4. Wait for 5 minutes and run the ps -ef | grep nscd command again as user root. Check whether the service exists.

    • If it exists, go to 11.
    • If it does not exist, go to 15.

Check whether the LDAP server can be accessed.

  1. Log in to the alarmed node as user root. Run the ping command to check the network connectivity between this node and the LdapServer node.

    • If the network is normal, go to 6.
    • If the network is faulty, contact network administrators to troubleshoot the fault.

Check whether NameService is normal.

  1. Log in to the alarmed node as user root. Run the cat /etc/nsswitch.conf command and check the passwd, group, services, netgroup, and aliases configurations of NameService.

    The correct parameter configurations are as follows: passwd: compat ldap, group: compat ldap, services: files ldap, netgroup: files ldap, and aliases: files ldap.

    • If the configurations are correct, go to 7.
    • If the configurations are incorrect, go to 9.

  2. Log in to the alarmed node as user root. Run the cat /etc/nscd.conf command and check the enable-cache passwd,positive-time-to-live passwd,enable-cache group and positive-time-to-live group configurations of NameService.

    The correct parameter configurations are as follows: enable-cache passwd yes, positive-time-to-live passwd 600, enable-cache group yes and positive-time-to-live group 3600.
    • If the configurations are correct, go to 8.
    • If the configurations are incorrect, go to 10.

  3. Run the /usr/sbin/nscd -i group and /usr/sbin/nscd -i passwd commands as user root. Wait for 2 minutes and run the id admin and id backup/manager commands to check whether results can be queried.

    • If results are queried, go to 11.
    • If no result is queried, go to 15.

  4. Run the vi /etc/nsswitch.conf command as user root. Correct the configurations in 6 and save the file. Run the service nscd restart command to restart the nscd service. Wait for 2 minutes and run the id admin and id backup/manager commands to check whether results can be queried.

    • If results are queried, go to 11.
    • If no result is queried, go to 15.

  5. Run the vi /etc/nscd.conf command as user root. Correct the configurations in 7 and save the file. Run the service nscd restart command to restart the nscd service. Wait for 2 minutes and run the id admin and id backup/manager commands to check whether results can be queried.

    • If results are queried, go to 11.
    • If no result is queried, go to 15.

  6. Log in to the FusionInsight Manager portal. Wait for 5 minutes and check whether the nscd Service Exception alarm is cleared.

    • If the alarm is cleared, no further action is required.
    • If the alarm persists, go to 12.

Check whether frame freezing occurs when running a command in the operating system.

  1. Log in to the faulty node as user root, run the id admin command, and check whether the command execution takes a long time. If the command execution takes more than 3 seconds, the command execution is deemed to be slow.

    • If yes, go to 13.
    • If no, go to 15.

  2. Run the cat /var/log/messages command to check whether the nscd frequently restarts or the error information Can't contact LDAP server exists.

    nscd exception example:

    Feb 11 11:44:42 10-120-205-33 nscd: nss_ldap: failed to bind to LDAP server ldaps://10.120.205.55:21780: Can't contact LDAP server
Feb 11 11:44:43 10-120-205-33 ntpq: nss_ldap: failed to bind to LDAP server ldaps://10.120.205.55:21780: Can't contact LDAP server
Feb 11 11:44:44 10-120-205-33 ntpq: nss_ldap: failed to bind to LDAP server ldaps://10.120.205.92:21780: Can't contact LDAP server
    • If yes, go to 14.
    • If no, go to 15.

  3. Run the vi $BIGDATA_HOME/tmp/random_ldap_ip_order command to modify the number at the end. If the original number is an odd number, change it to an even number. If the number is an even number, change it to an odd number.

    Run the vi /etc/ldap.conf command to enter the editing mode, press Insert to start editing, and then change the first two IP addresses of the URI configuration item.

    After the modification is complete, press Esc to exit the editing mode and enter :wq to save the settings and exit.

    Run the service nscd restart command to restart the nscd service. Wait 5 minutes and run the id admin command again. Check whether the command execution is slow.

    • If yes, go to 15.
    • If no, log in to other faulty nodes and run 12 to 14 and check whether the first ldapserver node in the URI before modifying /etc/ldap.conf is faulty. For example, check whether the service IP address is unreachable, the network delay is too long, or other abnormal software is deployed.

Collect fault information.

  1. On the FusionInsight Manager portal, choose O&M > Log > Download.
  2. Select LdapClient in the required cluster from the Service.
  3. Click in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
  4. Contact the O&M personnel and send the collected fault logs.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

Related Information

None