Configuring HBase DR

Scenario

HBase disaster recovery (DR), a key feature that is used to ensure high availability (HA) of the HBase cluster system, provides the real-time remote DR function for HBase. HBase DR provides basic O&M tools, including tools for maintaining and re-establishing DR relationships, verifying data, and querying data synchronization progress. To implement real-time DR, back up data of an HBase cluster to another HBase cluster. DR in the HBase table common data writing and BulkLoad batch data writing scenarios is supported.

This section applies to MRS 3.x or later clusters.

1. Log in to Manager of the active cluster.
2. Choose Cluster > Name of the desired cluster > Services > HBase > Configurations and click All Configurations. The HBase configuration page is displayed.
3. (Optional) Table 1 describes the optional configuration items during HBase DR. You can set the parameters based on the description or use the default values.

   Table 1 Optional configuration items

   Navigation Path	Parameter	Default Value	Description
   HMaster > Performance	hbase.master.logcleaner.ttl	600000	Specifies the retention period of HLog. If the value is set to 604800000 (unit: millisecond), the retention period of HLog is 7 days.
   	hbase.master.cleaner.interval	60000	Interval for the HMaster to delete historical HLog files. The HLog that exceeds the configured period will be automatically deleted. You are advised to set it to the maximum value to save more HLogs.
   RegionServer > Replication	replication.source.size.capacity	16777216	Maximum size of edits, in bytes. If the edit size exceeds the value, HLog edits will be sent to the standby cluster.
   	replication.source.nb.capacity	25000	Maximum number of edits, which is another condition for triggering HLog edits to be sent to the standby cluster. After data in the active cluster is synchronized to the standby cluster, the active cluster reads and sends data in HLog according to this parameter value. This parameter is used together with replication.source.size.capacity.
   	replication.source.maxretriesmultiplier	10	Maximum number of retries when an exception occurs during replication.
   	replication.source.sleepforretries	1000	Retry interval (Unit: ms)
   	hbase.regionserver.replication.handler.count	6	Number of replication RPC server instances on RegionServer

Configuring the BulkLoad batch data writing DR parameters for the active cluster

4. Determine whether to enable the BulkLoad batch data writing DR function.

   If yes, go to 5.

   If no, go to 8.

5. Choose Cluster > Name of the desired cluster > Services > HBase > Configurations and click All Configurations. The HBase configuration page is displayed.
6. Search for hbase.replication.bulkload.enabled and change its value to true to enable the BulkLoad batch data writing DR function.
7. Search for hbase.replication.cluster.id and change the HBase ID of the active cluster. The ID is used by the standby cluster to connect to the active cluster. The value can contain uppercase letters, lowercase letters, digits, and underscores (_), and cannot exceed 30 characters.

Restarting the HBase service and install the client

8. Click Save. In the displayed dialog box, click OK. Restart the HBase service.
9. In the active and standby clusters, choose Cluster > Name of the desired cluster > Service > HBase > More > Download Client to download the client and install it.

Adding the DR relationship between the active and standby clusters

10. Log in as user hbase to the HBase shell page of the active cluster.
11. Run the following command on HBase Shell to create the DR synchronization relationship between the active cluster HBase and the standby cluster HBase.

    add_peer 'Standby cluster ID', CLUSTER_KEY => "ZooKeeper service IP address in the standby cluster ", CONFIG => {"hbase.regionserver.kerberos.principal" => "Standby cluster RegionServer principal", "hbase.master.kerberos.principal" => "Standby cluster HMaster principal"}

    • The standby cluster ID indicates the ID for the active cluster to recognize the standby cluster. Enter an ID. The value can be specified randomly. Digits are recommended.
    • The ZooKeeper address of the standby cluster includes the service IP address of ZooKeeper, the port for listening to client connections, and the HBase root directory of the standby cluster on ZooKeeper.
    • Search for hbase.master.kerberos.principal and hbase.regionserver.kerberos.principal in the HBase hbase-site.xml configuration file of the standby cluster.

    For example, to add the DR relationship between the active and standby clusters, run the add_peer 'Standby cluster ID', CLUSTER_KEY => "192.168.40.2,192.168.40.3,192.168.40.4:24002:/hbase", CONFIG => {"hbase.regionserver.kerberos.principal" => "hbase/hadoop.hadoop.com@HADOOP.COM", "hbase.master.kerberos.principal" => "hbase/hadoop.hadoop.com@HADOOP.COM"}

12. (Optional) If the BulkLoad batch data write DR function is enabled, the HBase client configuration of the active cluster must be copied to the standby cluster.
    • Create the /hbase/replicationConf/hbase.replication.cluster.id of the active cluster directory in the HDFS of the standby cluster.
    • HBase client configuration file, which is copied to the /hbase/replicationConf/hbase.replication.cluster.id of the active cluster directory of the HDFS of the standby cluster.

      Example: hdfs dfs -put HBase/hbase/conf/core-site.xml HBase/hbase/conf/hdfs-site.xml HBase/hbase/conf/yarn-site.xml hdfs://NameNode IP:25000/hbase/replicationConf/source_cluster

Enabling HBase DR to synchronize data

13. Check whether a naming space exists in the HBase service instance of the standby cluster and the naming space has the same name as the naming space of the HBase table for which the DR function is to be enabled.
    • If the same namespace exists, go to 14.
    • If no, create a naming space with the same name in the HBase shell of the standby cluster and go to 14.

14. In the HBase shell of the active cluster, run the following command as user hbase to enable the real-time DR function for the table data of the active cluster to ensure that the data modified in the active cluster can be synchronized to the standby cluster in real time.

    You can only synchronize the data of one HTable at a time.

    enable_table_replication 'table name'

    

    • If the standby cluster does not contain a table with the same name as the table for which real-time synchronization is to be enabled, the table is automatically created.
    • If a table with the same name as the table for which real-time synchronization is to be enabled exists in the standby cluster, the structures of the two tables must be the same.
    • If the encryption algorithm SMS4 or AES is configured for 'Table name', the function for synchronizing data from the active cluster to the standby cluster cannot be enabled for the HBase table.
    • If the standby cluster is offline or has tables with the same name but different structures, the DR function cannot be enabled.
    • If the DR data synchronization function is enabled for some Phoenix tables in the active cluster, the standby cluster cannot have common HBase tables with the same names as the Phoenix tables in the active cluster. Otherwise, the DR function fails to be enabled or the tables with the names in the standby cluster cannot be used properly.
    • If the DR data synchronization function is enabled for Phoenix tables in the active cluster, you need to enable the DR data synchronization function for the metadata tables of the Phoenix tables. The metadata tables include SYSTEM.CATALOG, SYSTEM.FUNCTION, SYSTEM.SEQUENCE, and SYSTEM.STATS.
    • If the DR data synchronization function is enabled for HBase tables of the active cluster, after adding new indexes to HBase tables, you need to manually add secondary indexes to DR tables in the standby cluster to ensure secondary index consistency between the active and standby clusters.
    • The HBase multi-instance function also supports DR. You need to modify the parameters on the HBase service instance that corresponds to the standby cluster and run the commands on the clients of multiple instances. When adding the DR relationship, you need to select the directory, such as hbase1, for ZooKeeper of the standby cluster to store HBase multi-instance data.

15. (Optional) If HBase does not use Ranger, run the following command as user hbase in the HBase shell of the active cluster to enable the real-time permission to control data DR function for the HBase tables in the active cluster.

    enable_table_replication 'hbase:acl'

Creating Users

16. Log in to FusionInsight Manager of the standby cluster, choose System > Permission > Role > Create Role to create a role, and add the same permission for the standby data table to the role based on the permission of the HBase source data table of the active cluster.
17. Choose System > Permission > User > Create to create a user. Set the User Type to Human-Machine or Machine-Machine based on service requirements and add the user to the created role. Access the HBase DR data of the standby cluster as the newly created user.
    

    • After the permission of the active HBase source data table is modified, to ensure that the standby cluster can properly read data, modify the role permission for the standby cluster.
    • If the current component uses Ranger for permission control, you need to configure permission management policies based on Ranger. For details, see Adding a Ranger Access Permission Policy for HBase.

Synchronizing the table data of the active cluster

18. After HBase DR is configured and data synchronization is enabled, check whether tables and data exist in the active cluster and whether the historical data needs to be synchronized to the standby cluster.
    • If yes, a table exists and data needs to be synchronized. Log in as the HBase table user to the node where the HBase client of the active cluster is installed and run the kinit username to authenticate the identity. The user must have the read and write permissions on tables and the execute permission on the hbase:meta table. Then go to 19.
    • If no, no further action is required.

19. The HBase DR configuration does not support automatic synchronization of historical data in tables. You need to back up the historical data of the active cluster and then manually restore the historical data in the standby cluster.

    Manual recovery refers to the recovery of a single table, which can be performed through Export, DistCp, or Import.

    To manually recover a single table, perform the following steps:

    1. Export table data from the active cluster.

       hbase org.apache.hadoop.hbase.mapreduce.Export -Dhbase.mapreduce.include.deleted.rows=true Table name Directory where the source data is stored

       Example: hbase org.apache.hadoop.hbase.mapreduce.Export -Dhbase.mapreduce.include.deleted.rows=true t1 /user/hbase/t1

    2. Copy the data that has been exported to the standby cluster.

       hadoop distcp directory where the source data is stored on the active cluster hdfs://ActiveNameNodeIP:8020/directory where the source data is stored on the standby cluster

       ActiveNameNodeIP indicates the IP address of the active NameNode in the standby cluster.

       Example: hadoop distcp /user/hbase/t1 hdfs://192.168.40.2:8020/user/hbase/t1

    3. Import data to the standby cluster as the HBase table user of the standby cluster.

       On the HBase shell screen of the standby cluster, run the following command as user hbase to retain the data writing status:

       set_clusterState_active

       The command is run successfully if the following information is displayed:

       hbase(main):001:0> set_clusterState_active
       => true

       hbase org.apache.hadoop.hbase.mapreduce.Import -Dimport.bulk.output=Directory where the output data is stored in the standby cluster Table name Directory where the source data is stored in the standby cluster

       hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles Directory where the output data is stored in the standby cluster Table name

       Example:

       hbase(main):001:0> set_clusterState_active
       => true

       hbase org.apache.hadoop.hbase.mapreduce.Import -Dimport.bulk.output=/user/hbase/output_t1 t1 /user/hbase/t1

       hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles /user/hbase/output_t1 t1

20. Run the following command on the HBase client to check the synchronized data of the active and standby clusters. After the DR data synchronization function is enabled, you can run this command to check whether the newly synchronized data is consistent.

    hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication --starttime=Start time --endtime=End time Column family name ID of the standby cluster Table name

    

    • The start time must be earlier than the end time.
    • The values of starttime and endtime must be in the timestamp format. You need to run date -d "2015-09-30 00:00:00" +%s to change a common time format to a timestamp format.

Specify the data writing status for the active and standby clusters.

21. On the HBase shell screen of the active cluster, run the following command as user hbase to retain the data writing status:

    set_clusterState_active

    The command is run successfully if the following information is displayed:

    hbase(main):001:0> set_clusterState_active
    => true

22. On the HBase shell screen of the standby cluster, run the following command as user hbase to retain the data read-only status:

    set_clusterState_standby

    The command is run successfully if the following information is displayed:

    hbase(main):001:0> set_clusterState_standby
    => true