ALM-45641 Data Synchronization Exception Between the Active and Standby FlinkServer Nodes

Alarm Description

The system checks data synchronization between the active and standby FlinkServer nodes every 60 seconds. This alarm is generated when the standby FlinkServer node fails to synchronize files with the active FlinkServer node.

This alarm is cleared when the standby FlinkServer synchronizes files with the active FlinkServer.

Alarm Attributes

Alarm Parameters

Impact on the System

After an active/standby switchover, some configurations may be lost. Some jobs and connections of the FlinkServer are interrupted, but the FlinkServer can still provide services properly.

Possible Causes

• The link between the active and standby FlinkServer nodes is interrupted.
• The synchronization file does not exist or the file permission is required.

Handling Procedure

Check whether the network between the active and standby FlinkServer is in normal state.

1. On FusionInsight Manager, choose Cluster > Services > ClickHouse > Instance. View and record the IP addresses of active and standby FlinkServer.
2. Log in to the active FlinkServer node as user root.
3. Run the following command to check whether the standby FlinkServer is reachable:

   ping IP address of the standby FlinkServer

   • If yes, go to 6.
   • If no, go to 4.

4. Contact the network administrator to check whether the network is faulty.
   • If yes, go to 5.
   • If no, go to 6.

5. Rectify the network fault and check whether the alarm is cleared from the alarm list.
   • If yes, no further action is required.
   • If no, go to 6.

Check whether the storage space of the /srv/BigData/LocalBackup directory is full.

6. Run the following command to check whether the storage space of the /srv/BigData/LocalBackup directory is full:
   df -hl /srv/BigData/LocalBackup

   • If yes, go to 7.
   • If no, go to 10.

7. Run the following command to clear unnecessary backup files:

   rm -rf Directory to be cleared

   The following are two examples:

   rm -rf /srv/BigData/LocalBackup/0/default-oms_20191211143443

8. On FusionInsight Manager, choose O&M > Backup and Restoration > Backup Management.

   In the Operation column of the backup task, click Configure and change the value of Maximum Number of Backup Copies to reduce the number of backup file sets.

9. Wait for 1 minute and check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 10.

Check whether the synchronization file exists and whether the file permission is valid.

10. Run the following command to check whether the synchronization file exists:

    find /srv/BigData/ -name "sed*"

    find /opt -name "sed*"

    • If yes, go to 11.
    • If no, go to 12.

11. Run the following command to check the synchronization file information and permission queried in 10:

    ll Path of the file you want to search for

    • If the file size is 0 and all values in the permission column are -, the file is a junk file. Run the following command to delete it:

      rm -rf Files to be deleted

      Wait for several minutes and check whether the alarm is cleared. If the alarm persists, go to 12.

    • If the file size is not 0, go to 12.

12. View the log file generated when the alarm is reported.
    1. Run the following command to go to the HA run log file path of the current cluster:

       cd /var/log/Bigdata/flink/flinkserver/ha/runlog

    2. Decompress log file and view the logs generated when the alarm is reported.

       For example, if the name of the file is ha.log.2021-03-22_12-00-07.gz, run the following command:

       gunzip ha.log.2021-03-22_12-00-07.gz

       vi ha.log.2021-03-22_12-00-07

       Check whether error information is displayed before and after the alarm generation time in the logs.

       • If it is displayed, rectify the fault based on the error information. Go to 13.

         For example, if the following error information is displayed, the directory permission is required. In this case, obtain the directory permission that is the same as the permission on a normal node.

         

       • If no, go to 14.

13. Wait for about 10 minutes and check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 14.

Collect fault information.

14. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
15. Select FlinkServer information from Services and click OK.
16. Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the hosts to which the role belongs, and click OK.
17. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
18. Contact O&M engineers and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None.