ALM-12011 Manager Data Synchronization Exception Between the Active and Standby Nodes

Alarm Description

The system checks data synchronization between the active and standby Manager nodes every 60 seconds. This alarm is generated when the standby Manager fails to synchronize files with the active Manager.

This alarm is cleared when the standby Manager synchronizes files with the active Manager.

Alarm Attributes

Alarm Parameters

Impact on the System

The configuration file of the standby Manager is not updated. When an active/standby switchover occurs, the configuration file that fails to be synchronized may be lost. Manager and some components may not run properly.

Possible Causes

• The link between the active and standby Managers is interrupted or The storage space of the /srv/BigData/LocalBackup directory is full.
• The synchronization file does not exist or the file permission is incorrect.

Handling Procedure

Check whether the network between the active Manager server and the standby Manager server is normal.

1. In the FusionInsight Manager portal, click O&M > Alarm > Alarms, click in the row where the alarm is located and obtain the standby Manager server IP address (Peer Manager IP address) in the alarm details.
2. Log in to the active Manager server as user root.
3. Run the ping standby Manager IP address command to check whether the standby Manager server is reachable.
   • If yes, go to 6.
   • If no, go to 4.

4. Contact the network administrator to check whether the network is faulty.
   • If yes, go to 5.
   • If no, go to 6.

5. Rectify the network fault and check whether the alarm is cleared from the alarm list.
   • If yes, no further action is required.
   • If no, go to 6.

Check whether the storage space of the /srv/BigData/LocalBackup directory is full.

6. Run the following command to check whether the storage space of the /srv/BigData/LocalBackup directory is full:

   df -hl /srv/BigData/LocalBackup

   • If yes, go to 7.
   • If no, go to 10.

7. Run the following command to clear unnecessary backup files:

   rm -rf Directory to be cleared

   Example:

   rm -rf /srv/BigData/LocalBackup/0/default-oms_20191211143443

8. On FusionInsight Manager, choose O&M > Backup and Restoration > Backup Management.

   In the Operation column of the backup task to be performed, click Configure and change the value of Maximum Number of Backup Copies to reduce the number of backup file sets.

9. Wait about 1 minute and check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 10.

Check whether the synchronization file exists and whether the file permission is normal.

10. Run the following command to check whether the synchronization file exists.

    find /srv/BigData/ -name "sed*"

    find /opt -name "sed*"

    • If yes, go to 11.
    • If no, go to 12.

11. Run the following command to view the synchronization file information and permission obtained in 10.

    ll path of the file to be found

    • If the size of the file is 0 and the permission column is -, the file is a junk file. Run the following command to delete it.

      rm -rf files to be deleted

      Wait for several minutes and check whether the alarm is cleared. If the alarm persists, go to 12.

    • If the file size is not 0, go to 12.

12. View the log files generated when the alarm is generated.
    1. Run the following command to switch to the HA run log file path.

       cd /var/log/Bigdata/omm/oms/ha/runlog/

    2. Decompress and view the log files generated when the alarm is generated.

       For example, if the name of the file to be viewed is ha.log.2021-03-22_12-00-07.gz, run the following command:

       gunzip ha.log.2021-03-22_12-00-07.gz

       vi ha.log.2021-03-22_12-00-07

       Check whether error information is reported before and after the alarm generation time.

       • If yes, rectify the fault based on the error information. Then go to 13.

         For example, if the following error information is displayed, the directory permission is insufficient. In this case, change the directory permission to be the same as that on the normal node.

         

       • If no, go to 14.

13. Wait about 10 minute and check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 14.

Collect fault information.

14. On the FusionInsight Manager, choose O&M > Log > Download.
15. Select the following nodes from the Service and click OK:
    • OmmServer
    • Controller
    • NodeAgent

16. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
17. Contact the O&M engineers and send the collected log information.

Alarm Clearance

After the fault is rectified, the system automatically clears this alarm.

Related Information

None.