ALM-12066 Trust Relationships Between Nodes Become Invalid

Alarm Description

The system checks whether the trust relationship between the active OMS node and other Agent nodes is normal every hour. The alarm is generated if the mutual trust fails. This alarm is automatically cleared after the fault is rectified.

Alarm Attributes

Alarm Parameters

Impact on the System

Some operations (such as restart, configuration synchronization, and instance status query) that need to connect to the node may fail. If the trust relationships between nodes are invalid, services may be affected.

Possible Causes

• The /etc/ssh/sshd_config configuration file is damaged.
• The password of user omm has expired.

Handling Procedure

Check the status of the /etc/ssh/sshd_config configuration file.

1. In the alarm list on FusionInsight Manager, locate the row that contains the alarm and click to view the host list in the alarm details.
2. Log in to the active OMS node as user omm.
3. Run the ssh command, for example, ssh host2, on each node in the alarm details to check whether the connection fails. (host2 is a node other than the OMS node in the alarm details.)
   • If yes, go to 4.
   • If no, go to 6.

4. Open the /etc/ssh/sshd_config configuration file on host2 and check whether AllowUsers or DenyUsers is configured for other nodes.
   • If yes, go to 5.
   • If no, contact OS experts.

5. Modify the whitelist or blacklist to ensure that user omm is in the whitelist or not in the blacklist. Check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 6.

Check the status of the password of user omm.

6. Check the interaction information of the ssh command.
   • If the password of user omm is required, go to 7.
   • If message "Enter passphrase for key '/home/omm/.ssh/id_rsa':" is displayed, go to 9.
   • If the connection is set up successfully, run the following command to restart ssh-agent and then go to 3:

     ps -ef|grep ssh-agent |grep -v grep |awk '{print $2}' | xargs kill -9

7. Check the trust list (/home/omm/.ssh/authorized_keys) of user omm on the OMS node and host2 node. Check whether the trust list contains the public key file (/home/omm/.ssh/id_rsa.pub) of user omm on the peer host.
   • If yes, contact OS experts.
   • If no, add the public key of user omm of the peer host to the trust list of the local host.

8. Add the public key of user omm of the peer host to the trust list of the local host. Run the ssh command, for example, ssh host2, on each node in the alarm details to check whether the connection fails. (host2 is a node other than the OMS node in the alarm details.)
   • If yes, go to 9.
   • If no, check whether the alarm is cleared. If the alarm is cleared, no further action is required; otherwise, go to 9.

Collect the fault information.

9. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
10. Select Controller for Service and click OK.
11. Click in the upper right corner to set the log collection time range. Generally, the time range is 10 minutes before and after the alarm generation time. Click Download.
12. Contact O&M personnel and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

Perform the following steps to handle abnormal trust relationships between nodes:

• Perform this operation as user omm.
• If the network between nodes is disconnected, rectify the network fault first. Check whether the two nodes are connected to the same security group and whether hosts.deny and hosts.allow are set.

1. Run the ssh-add -l command on both nodes to check whether any identities exist.

   

   • If yes, go to 4.
   • If no, go to 2.
2. If no identities are displayed, run the ps -ef|grep ssh-agent command to find the ssh-agent process, stop the process, and wait for the process to automatically restart.

   

3. Run the ssh-add -l command to check whether the identities have been added. If yes, manually run the ssh command to check whether the trust relationship is normal.

   

4. If identities exist, check whether the /home/omm/.ssh/authorized_keys file contains the information in the /home/omm/.ssh/id_rsa.pub file of the peer node. If it does not, manually add the information.
5. Check whether the permissions on the files in the /home/omm/.ssh directory are modified.
6. Check the /var/log/Bigdata/nodeagent/scriptlog/ssh-agent-monitor.log file.
7. If the /home directory of user omm is deleted, contact MRS support personnel for assistance.