On this page

Show all

Help Center/ MapReduce Service/ User Guide (Kuala Lumpur Region)/ FusionInsight Manager Operation Guide (Applicable to 3.x)/ Alarm Reference (Applicable to MRS 3.x)/ ALM-19006 HBase Replication Sync Failed

ALM-19006 HBase Replication Sync Failed

Updated on 2022-08-12 GMT+08:00

View PDF

Description

The alarm module checks the HBase DR data synchronization status every 30 seconds. When disaster recovery (DR) data fails to be synchronized to a standby cluster, the alarm is triggered.

When DR data synchronization succeeds, the alarm is cleared.

NOTE:

If the multi-instance function is enabled in the cluster and multiple HBase service instances are installed, you need to determine the HBase service instance where the alarm is generated based on the value of ServiceName in Location. For example, if the HBase1 service is unavailable, ServiceName=HBase1 is displayed in Location, and the operation object in the procedure needs to be changed from HBase to HBase1.

Attribute

Alarm ID	Alarm Severity	Automatically Cleared
19006	Critical	Yes

Parameters

Name	Meaning
Source	Specifies the cluster for which the alarm is generated.
ServiceName	Specifies the service for which the alarm is generated.
RoleName	Specifies the role for which the alarm is generated.
HostName	Specifies the host for which the alarm is generated.
Trigger Condition	Specifies the threshold triggering the alarm. If the current indicator value exceeds this threshold, the alarm is generated.

Impact on the System

HBase data in a cluster fails to be synchronized to the standby cluster, causing data inconsistency between active and standby clusters.

Possible Causes

The HBase service on the standby cluster is abnormal.
A network exception occurs.

Procedure

Observe whether the system automatically clears the alarm.

On the FusionInsight Manager portal of the active cluster, click O&M > Alarm > Alarms.
In the alarm list, click the alarm to obtain alarm generation time from Generated of the alarm. Check whether the alarm has existed for five minutes.
- If yes, go to 4.
- If no, go to 3.
Wait five minutes and check whether the system automatically clears the alarm.
- If yes, no further action is required.
- If no, go to 4.

Check the HBase service status of the standby cluster.

Log in to the FusionInsight Manager portal of the active cluster, and click O&M > Alarm > Alarms.
In the alarm list, click the alarm to obtain HostName from Location.
Access the node where the HBase client of the active cluster resides as user omm.

If the cluster uses a security mode, perform security authentication first and then access the hbase shell interface as user hbase.

cd /opt/Bigdata/client

source ./bigdata_env

kinit hbaseuser

Run the status 'replication', 'source' command to check the DR synchronization status of the faulty node.

The DR synchronization status of a node is as follows.

10-10-10-153: 
SOURCE: PeerID=abc, SizeOfLogQueue=0, ShippedBatches=2, ShippedOps=2, ShippedBytes=320, LogReadInBytes=1636, LogEditsRead=5, LogEditsFiltered=3, SizeOfLogToReplicate=0, TimeForLogToReplicate=0, ShippedHFiles=0, SizeOfHFileRefsQueue=0, AgeOfLastShippedOp=0, TimeStampsOfLastShippedOp=Mon Jul 18 09:53:28 CST 2016, Replication Lag=0, FailedReplicationAttempts=0 
SOURCE: PeerID=abc1, SizeOfLogQueue=0, ShippedBatches=1, ShippedOps=1, ShippedBytes=160, LogReadInBytes=1636, LogEditsRead=5, LogEditsFiltered=3, SizeOfLogToReplicate=0, TimeForLogToReplicate=0, ShippedHFiles=0, SizeOfHFileRefsQueue=0, AgeOfLastShippedOp=16788, TimeStampsOfLastShippedOp=Sat Jul 16 13:19:00 CST 2016, Replication Lag=16788, FailedReplicationAttempts=5

Obtain PeerID corresponding to a record whose FailedReplicationAttempts value is greater than 0.

In the preceding step, data on the faulty node 10-10-10-153 fails to be synchronized to a standby cluster whose PeerID is abc1.
Run the list_peers command to find the cluster and the HBase instance corresponding to the PeerID value.
```
PEER_ID CLUSTER_KEY STATE TABLE_CFS 
abc1 10.10.10.110,10.10.10.119,10.10.10.133:2181:/hbase2 ENABLED  
abc 10.10.10.110,10.10.10.119,10.10.10.133:2181:/hbase ENABLED 
```
In the preceding information, /hbase2 indicates that data is synchronized to the HBase2 instance of the standby cluster.
In the service list of FusionInsight Manager of the standby cluster, check whether the running status of the HBase instance obtained by using 9 is Normal.
- If yes, go to 14.
- If no, go to 11.
In the alarm list, check whether the ALM-19000 HBase Service Unavailable alarm is generated.
- If yes, go to 12.
- If no, go to 14.
Follow troubleshooting procedures in ALM-19000 HBase Service Unavailable to rectify the fault.
Wait for a few minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 14.

Check network connections between RegionServers on active and standby clusters.

Log in to the FusionInsight Manager portal of the active cluster, and click O&M > Alarm > Alarms.
In the alarm list, click the alarm to obtain HostName from Location.
Use the IP address obtained in 15 to log in to a faulty RegionServer node as user omm.
Run the ping command to check whether network connections between the faulty RegionServer node and the host where RegionServer of the standby cluster resides are in the normal state.
- If yes, go to 20.
- If no, go to 18.
Contact the network administrator to restore the network.
After the network is running properly, check whether the alarm is cleared in the alarm list.
- If yes, no further action is required.
- If no, go to 20.

Collect fault information.

On the FusionInsight Manager interface of active and standby clusters, choose O&M > Log > Download.
In the Service drop-down list box, select HBase in the required cluster.
Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
Contact the O&M personnel and send the collected fault logs.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

Related Information

None

Parent topic: Alarm Reference (Applicable to MRS 3.x)

Previous topic: ALM-19000 HBase Service Unavailable

Next topic: ALM-19007 HBase GC Time Exceeds the Threshold

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot