Migrating HBase Data to an MRS Cluster with Replication

Scenarios

HBase's replication feature enables asynchronous data synchronization across multiple HBase clusters. It is primarily used to enhance data availability, support disaster recovery, and facilitate cross-region data distribution.

HBase replication is built on the write-ahead log (WAL) mechanism. The core process works as follows:

1. Data write and WAL

   When a client writes data, HBase first logs the operation in the RegionServer's write-ahead log (WAL), then stores the data in the MemStore.

2. WAL replication

   The RegionServer in the active (source) cluster asynchronously transmits WAL logs to the standby (destination) cluster.

3. WAL replay

   After receiving the logs, the standby cluster replays the operations in order to synchronize data with the corresponding tables, ensuring consistency with the active cluster.

HBase replication enables seamless data migration from an open-source big data platform or a self-managed HBase cluster to an MRS HBase cluster.

• Restore snapshots in the destination cluster to migrate both metadata and historical data.
• Migrate data from WAL files in the source cluster to the destination cluster to complete incremental data migration.

Figure 1 HBase data migration

Migration Survey

Before migrating HBase data, you need to conduct a survey on the source HBase component to evaluate the risks that may occur during the migration and impact on the system. The survey covers the HBase component version, deployment mode, data storage, and performance optimization. For details, see Table 1.

Networking Types

The migration solution supports various networking types, such as the public network, VPN, and Direct Connect. Select a networking type based on the site requirements. The migration can be performed only when the source and destination networks can communicate with each other.

Notes and Constraints

• During historical data migration using snapshots, the metadata must remain unchanged; otherwise, incremental data from WAL files cannot be written to the original table.
• Replication adds workload to the source cluster. Before snapshot-based data migration is complete, incremental data accumulates in WAL files, which may exhaust disk space in the source cluster.
• Migration is only supported between HBase clusters running the same kernel version, for example, from HBase 1.x to HBase 1.x or from HBase 2.x to HBase 2.x.

Preparations

• You have created an HBase MRS cluster and installed the cluster client.
• The source cluster can communicate with the destination cluster.
• You have permissions required for the migration.

Establishing a Replication Relationship Between Source and Destination Tables

1. In the /etc/hosts file of the source cluster, configure the mapping between the host name and IP address for the node hosting the HBase instance in the destination cluster.
2. Access the HBase shell command interface of the source cluster.
3. Establish a peer relationship between the source and destination clusters.

   add_peer 'peer_name', CLUSTER_KEY => 'ZooKeeper_IP:Port:/hbase'

   • peer_name: Unique identifier of the replication relationship, which is used for subsequent management. Choose a name that is easy to remember.
   • ZooKeeper_IP:Port: ZooKeeper configuration of the destination cluster. This information is essential for the source cluster to locate the metadata of the destination cluster.

     For details about how to obtain the ZooKeeper IP address and port number of the MRS cluster, see How Can I Obtain the IP Address and Port Number of a ZooKeeper Instance?

   

   Run the following command to check whether the peer relationship is established:

   list_peers

   

   If the peer relationship is incorrectly configured, run the following command to delete it:

   remove_peer 'peer_name'

4. Set the replication attributes for the table in the source cluster.

   Run the following command in HBase shell:

   alter 'Destination table name',{NAME =>'Column family name',REPLICATION_SCOPE => '1'}

   For example, if the destination table name is student and the column family name is f, run the following command:

   alter 'student',{NAME => 'f',REPLICATION_SCOPE => '1'}

   

   You can also run the following command to enable data synchronization for all column families of a specified table:

   enable_table_replication 'Destination table name' 

   Run the following command to disable data synchronization:

   disable_table_replication 'Destination table name'

Migrating Historical HBase Data

1. Create a snapshot for the HBase table in the source cluster.

   Run the following command in the HBase shell to create a table snapshot:

   snapshot 'Table name','Snapshot name'

   For example, create a snapshot student_snapshot for table student.

   snapshot 'student','student_snapshot'

   Check whether the snapshot is generated.

   list_snapshots

   

2. Migrate the snapshot from the source cluster to the destination MRS cluster.

   hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot \ 
     -snapshot <snapshot_name> \
     -copy-to <destination_path> \
     [-mappers <num_mappers>] \ 
     [-bandwidth <mb_per_sec>] \
     [-diff <previous_snapshot>] \
     [-overwrite]

   • -snapshot <snapshot_name>: Name of the snapshot you want to export.
   • -copy-to <destination_path>: Destination HDFS path. If the source and destination clusters are different, the format is hdfs://<Destination NameNode>:<NameNode port>/path (the HDFS of the destination cluster must be accessible).
     Log in to MRS Manager of the destination cluster and check the IP address and port number of the active HDFS NameNode.

     1. Choose Cluster > Services > HDFS and click Instances. On the displayed page, view the IP address of the NameNode (hacluster,Active) instance in the instance list.
     2. Click Configurations, and search for and view the NameNode port parameter dfs.namenode.rpc.port.
   • -mappers <num_mappers>: (Optional) specifies the number of Mapper tasks in a MapReduce task. You can adjust the number based on cluster resources and increase it when there is a large volume of data.
   • -diff <previous_snapshot>: (Optional) copies only the data that has changed since a specified previous snapshot. This parameter is used for incremental backup.
   • -bandwidth <mb_per_sec>: (Optional) sets a maximum data transfer rate (in MB/s). This helps prevent the task from consuming excessive network resources.
   • -overwrite: (Optional) indicates that any existing files in the destination directory with the same name as exported files will be forcibly replaced.

   For example, to migrate snapshot student_snapshot to the /hbase-snapshot directory of the destination MRS cluster, exit the HBase shell and run the following command:

   hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot student_snapshot -copy-to hdfs://192.168.1.200:8020/hbase-snapshot

3. Save the snapshot data to the corresponding directory of the destination MRS cluster.
   1. HBase uses a scheduled cleanup mechanism to manage its archive directory. Check the cleanup interval configured for the HMaster in the destination cluster to clear historical HLog files and increase the interval appropriately to prevent data from being cleared during data replication.

      Log in to MRS Manager of the destination cluster, and choose Cluster > Services > HBase. On the displayed HBase service page, choose Configurations > All Configurations, search for hbase.master.cleaner.interval, and modify the settings.

      The unit of this parameter is milliseconds. For example, setting it to 600000 configures a cleanup interval of 10 minutes.

   2. Log in to the destination MRS cluster client, and store metadata to the /hbase/.hbase-snapshot directory and original data to the /hbase/archive directory.

      If the destination cluster does not have the directories, run the following commands to create them:

      hdfs dfs -mkdir -p /hbase/.hbase-snapshot

      hdfs dfs -mkdir -p /hbase/archive/data/default/

      Move the exported snapshot file to the snapshot directory.

      hdfs dfs -mv Directory storing exported snapshots/.hbase-snapshot/Snapshot name /hbase/.hbase-snapshot/

      hdfs dfs -mv Directory storing exported snapshots/archive/data/default/Table name /hbase/archive/data/default/

      For example, move the snapshot file of table Student to the specified snapshot directory.

      hdfs dfs -mv /hbase-snapshot/.hbase-snapshot/student_snapshot /hbase/.hbase-snapshot/

      hdfs dfs -mv /hbase-snapshot/archive/data/default/student /hbase/archive/data/default/

4. Restore the table from the snapshot in the destination cluster.

   Access the HBase shell command interface of the destination MRS cluster.

   1. Disable a table.

      disable 'student'

   2. Restore the table from a snapshot.

      restore_snapshot 'Snapshot name'

      For example, restore the table from snapshot student_snapshot.

      restore_snapshot  'student_snapshot'

   3. Enable the table after it is restored from the snapshot.

      enable 'student'

Migrating Incremental HBase Data

After you restore a table from a snapshot, the corresponding table is created in the destination cluster. Based on the replication relationship configured in the source cluster, the retained WALs are automatically applied to the restored table in the destination cluster, enabling incremental data synchronization.

Run the following command on the HBase shell command interface to check the synchronization status:

status 'replication'

Verifying Data

Check whether the data in the source and destination clusters is consistent.

For example, you can run the select count (1) command to compare the number of rows in the table.

select count (1) from student;

Deleting the Settings After Data Migration

1. In the /etc/hosts file of the source cluster, delete the mapping between the host name and IP address for the node hosting the HBase instance in the destination cluster.
2. Access the HBase shell command interface of the source cluster.
3. Delete the replication relationship.
   1. Check the established replication relationship.

      list_peers

   2. Delete the specified replication relationship.

      remove_peer 'peer_name'

   3. Cancel table synchronization:

      Check whether synchronization has been configured for the table.

      desc 'Destination table name' 

      Run the following command to cancel the synchronization:

      alter 'Destination table name',{NAME => 'Column family name of the destination table',REPLICATION_SCOPE => '0'}

      Alternatively, run the following command:

      disable_table_replication 'Destination table name'