ALM-25006 Sssd Service Exception

Alarm Description

The system checks the status of the sssd service every 60 seconds. This alarm is generated when the sssd process fails to be queried for four consecutive times (three minutes) or users in LdapServer cannot be obtained.

This alarm is cleared when the process is restored and users in LdapServer can be obtained.

Alarm Attributes

Alarm Parameters

Impact on the System

The alarmed node may not be able to synchronize data from LdapServer. The id command may fail to obtain the LDAP data, affecting upper-layer services.

Possible Causes

• The sssd service is not started or is incorrectly started.
• The network is faulty and cannot access the LDAP server.
• NameService is abnormal.

Handling Procedure

Check whether the sssd service is correctly started.

1. On the FusionInsight Manager portal, choose O&M > Alarm > Alarms. Find the IP address of HostName in Location of the alarm and record it as IP1 (if multiple alarms exist, record the IP addresses as IP1, IP2, and IP3 respectively).
2. Contact the O&M engineers to access the node using IP1 as user root. Run the ps -ef | grep sssd command and check whether the /usr/sbin/sssd process is started.
   • If the process is started, go to 3.
   • If the process is not started, go to 4.

3. Check whether the sssd process queried in 2 has three subprocesses.
   • If yes, go to 5.
   • If no, go to 4.

4. Run the service sssd restart command as user root to restart the sssd service. Then run the ps -ef | grep sssd command to check whether the sssd process is normal.

   In the normal state, the /usr/sbin/sssd process has three subprocesses: /usr/libexec/sssd/sssd_be, /usr/libexec/sssd/sssd_nss, and /usr/libexec/sssd/sssd_pam.

   • If it exists, go to 9.
   • If it does not exist, go to 13.

Check whether the LDAP server can be accessed.

5. Log in to the alarmed node as user root. Run the ping command to check the network connectivity between this node and the LdapServer node.
   • If the network is normal, go to 6.
   • If the network is faulty, contact network administrators to troubleshoot the fault.

Check whether NameService is normal.

6. Log in to the alarmed node as user root. Run the cat /etc/nsswitch.conf command and check the passwd and group configurations of NameService.

   The correct parameter configurations are as follows: passwd: compat ldap and group: compat ldap.

   • If the configurations are correct, go to 7.
   • If the configurations are incorrect, go to 8.

7. Run the /usr/sbin/sss_cache -G and /usr/sbin/sss_cache -U commands as user root. Wait for 2 minutes and run the id admin and id backup/manager commands to check whether results can be queried.
   • If results are queried, go to 9.
   • If no result is queried, go to 13.

8. Run the vi /etc/nsswitch.conf command as user root. Correct the configurations in 6 and save the file. Run the service sssd restart command to restart the sssd service. Wait for 2 minutes and run the id admin and id backup/manager commands to check whether results can be queried.
   • If results are queried, go to 9.
   • If no result is queried, go to 13.

9. Log in to the FusionInsight Manager portal. Wait for 5 minutes and check whether the sssd Service Exception alarm is cleared.
   • If the alarm is cleared, no further action is required.
   • If the alarm persists, go to 10.

Check whether frame freezing occurs when running a command in the operating system.

10. Log in to the faulty node as user root, run the id admin command, and check whether the command execution takes a long time. If the command execution takes more than 3 seconds, the command execution is deemed to be slow.
    • If yes, go to 11.
    • If no, go to 13.

11. Run the cat /var/log/messages command to check whether the sssd frequently restarts or the error information Can't contact LDAP server exists.

    sssd restart example:

    Feb  7 11:38:16 10-132-190-105 sssd[pam]: Shutting down
    Feb  7 11:38:16 10-132-190-105 sssd[nss]: Shutting down
    Feb  7 11:38:16 10-132-190-105 sssd[nss]: Shutting down
    Feb  7 11:38:16 10-132-190-105 sssd[be[default]]: Shutting down
    Feb  7 11:38:16 10-132-190-105 sssd: Starting up
    Feb  7 11:38:16 10-132-190-105 sssd[be[default]]: Starting up
    Feb  7 11:38:16 10-132-190-105 sssd[nss]: Starting up
    Feb  7 11:38:16 10-132-190-105 sssd[pam]: Starting up

    • If yes, go to 12.
    • If no, go to 13.

12. Run the vi $BIGDATA_HOME/tmp/random_ldap_ip_order command to modify the number at the end. If the original number is an odd number, change it to an even number. If the number is an even number, change it to an odd number.

    Run the vi /etc/sssd/sssd.conf command to reverse the first two IP addresses of the ldap_uri configuration item, save the settings, and exit.

    Run the ps -ef | grep sssd command to query the ID of the sssd process, kill it, and run the /usr/sbin/sssd -D -f command to restart the sssd service. Wait 5 minutes and run the id admin command again.

    Check whether the command execution is slow.

    • If yes, go to 13.
    • If no, log in to other faulty nodes and run 10 to 12. Collect logs and check whether the first ldapserver node in the ldap_uri before modifying /etc/sssd/sssd.conf is faulty. For example, check whether the service IP address is unreachable, the network latency is too long, or other abnormal software is deployed.

Collect fault information.

13. On the FusionInsight Manager portal, choose O&M > Log > Download.
14. Select LdapClient in the required cluster from the Service.
15. Click in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
16. Contact the O&M engineers and send the collected fault logs.

Alarm Clearance

After the fault is rectified, the system automatically clears this alarm.

Related Information

None.