On this page

Show all

Help Center/ MapReduce Service/ Best Practices/ Data Migration/ Using BulkLoad to Import Data to HBase in Batches

Using BulkLoad to Import Data to HBase in Batches

Updated on 2024-09-10 GMT+08:00

View PDF

Application Scenarios

When batch importing a large amount of data to HBase, you have many choices, for example, calling the put method of HBase to insert data or using MapReduce to load data from HDFS. However, the two methods cause high pressure on the RegionServer and consume a large number of CPU and network resources because of frequent flush, compact, and split operations of HBase, thereby resulting in low efficiency.

This practice describes how to import local data to HBase in batches using BulkLoad after you create an MRS cluster. This method greatly improves the write efficiency and reduces the write pressure on RegionServer nodes.

Solution Architecture

HBase provides a data import tool called BulkLoad, which imports and directly writes data to underlying data files and WAL logs, greatly improving data loading speed and efficiency. ‌

BulkLoad uses MapReduce jobs to directly convert data into HFiles that comply with the internal data format of HBase, and then loads the generated StoreFiles to the corresponding nodes in a cluster. This method requires no flush, compact, or split operations, occupies no region resources, and generates little write requests. Fewer CPU and network resources are required.

Inapplicable scenarios of BulkLoad:

Large amounts of data needs to be loaded to HBase in the one-off manner.
When data is loaded to HBase, requirements on reliability are not high and WAL files do not need to be generated.
When the put method is used to load large amounts of data to HBase, data loading and query will be slow.
The size of an HFile generated after data loading is similar to the size of HDFS blocks.

Creating an MRS Offline Query Cluster

Go to the Buy Cluster page.

Click the Quick Config tab and set configuration parameters.

**Table 1** Software configurations
Parameter	Description	Value
Region	MRS clusters in different regions cannot communicate with each other over an intranet. For lower network latency and quick resource access, select the region nearest to you.	CN-Hong Kong
Billing Mode	MRS provides two billing modes. Yearly/Monthly Pay-per-use A prepaid balance will be frozen. For details, see Billing.	Pay-per-use
Cluster Name	The cluster name must be unique. A cluster name can contain 1 to 64 characters. Only letters, digits, hyphens (-), and underscores (_) are allowed.	MRS_hbase
Cluster Type	Available cluster types are as follows: Analysis cluster Streaming cluster Hybrid cluster Custom cluster	Custom
Version Type	Available version types are as follows: Normal LTS	Normal
Cluster Version	Available MRS versions	MRS 3.1.0
Component	MRS cluster components. For details about component versions supported by different versions of MRS clusters, see List of MRS Component Versions.	HBase Query Cluster
AZ	An availability zone (AZ) is a physical area that uses independent power and network resources. AZs are physically isolated but interconnected through the internal network. This improves the availability of applications. You are advised to create clusters in different AZs.	AZ1
Enterprise Project	The Enterprise Management console is designed for resource management. It helps you manage cloud-based personnel, resources, permissions, and finance in a hierarchical manner, such as management of companies, departments, and projects.	default
VPC	A Virtual Private Cloud (VPC) is a secure, isolated, and logical network environment.	vpc-01
Subnet	A subnet provides dedicated network resources that are logically isolated from other networks for network security.	subnet-01
Kerberos Authentication	If Kerberos authentication is enabled for a cluster, check whether Kerberos authentication is required. If yes, click Continue. If no, click Back to disable Kerberos authentication and then create a cluster. After a cluster is purchased, this configuration cannot be modified.	Toggle the slider on.
Username	The default value is root/admin. User root is used to remotely log in to ECS nodes, and user admin is used to access the cluster management page.	root/admin
Password	Password for users root/admin.	Set the password for logging in to the cluster management page and ECS node, for example, Test!@12345.
Confirm Password	-	Enter the password again.
Secure Communications	In an MRS cluster, you can provision, manage, and use big data components through the management console. Big data components are deployed in users' VPCs. To allow the MRS console to directly access big data components, you must enable the corresponding security group rules after granting authorization. This authorization process is called secure communications. If the secure communications function is not enabled, MRS clusters cannot be created.	Select Enable.

Figure 1 Creating an HBase query cluster
Click to enlarge

Click Buy Now and wait until the MRS cluster is created.

Importing Local Data to HDFS

Prepare a student information file info.txt on the local host.

The fields include student ID, name, birthday, gender, and address. An example file is as follows:

20200101245, Zhang xx, 20150324, Male, City 1
20200101246, Li xx, 20150202, Male, City 2
20200101247, Yang xx, 20151101, Female, City 3
20200101248, Chen xx, 20150218, Male, City 4
20200101249, Li xx, 20150801, Female, City 5
20200101250, Wang xx, 20150315, Male, City 6
20200101251, Li xx, 20151201, Male, City 7
20200101252, Sun xx, 20150916, Female, City 8
20200101253, Lin xx, 20150303, Male, City 9

Log in to OBS Console, click Parallel File Systems in the navigation pane. On the displayed page, click Create Parallel File System, set the following parameters, and click Create Now.

**Table 2** Parallel file system parameters
Parameter	Value
Region	CN-Hong Kong
File System Name	mrs-hbase
Data Redundancy Policy	Single-AZ storage
Policy	Private
Direct Reading	Disable
Enterprise Project	default
Tags	-

Click the name of the created bucket and click Upload File in the Files tab to upload the data file to the OBS bucket.

Switch back to the MRS console and click the name of the created MRS cluster. On the Dashboard page, click Synchronize next to IAM User Sync. The synchronization takes about five minutes.
Upload the data file to the HDFS.
1. On the Files page, click the HDFS File List and go to the data storage directory, for example, /tmp/test.
  The /tmp/test directory is only an example. You can use any directory on the page or create a new one.
2. Click Import Data.
  - OBS Path: Find the info.txt file in the created OBS parallel file system and click Yes.
  - HDFS Path: Select an HDFS path, for example, /tmp/test, and click Yes.
3. Click OK and wait until the data file is imported.
Figure 2 Importing data

Creating an HBase Table

Log in to FusionInsight Manager of the cluster (if no elastic IP address is available, purchase one), create a user named hbasetest, and bind it to the user group supergroup and role System_administrator.
Download the cluster client, and install it, for example, in the /opt/client directory of the active master node. For details, see Installing a Client.
You can also use the cluster client provided by the Master node. The installation directory is /opt/Bigdata/client.
Run the following commands to bind an elastic IP address to the active Master node, log in to the active Master node as user root, go to the directory where the client is located, and authenticate the user.
cd /opt/client

source bigdata_env

kinit hbasetest
Run the hbase shell command to go to the HBase shell page.
Plan the table name, rowkey, column family, and column of the HBase data table based on the imported data. Ensure that the rowkey is pre-split during table creation.

Run the following command to create the student_info table:

create 'student_info', {NAME => 'base',COMPRESSION => 'SNAPPY', DATA_BLOCK_ENCODING => 'FAST_DIFF'},SPLITS => ['1','2','3','4','5','6','7','8']
- NAME => 'base': Column family name of the HBase table
- COMPRESSION: Compression mode
- DATA_BLOCK_ENCODING: encoding algorithm
- SPLITS: Region pre-splitting
Check whether the table is created and then exit the HBase shell page.
list

Generating an HFile and Importing It to HBase

Create a custom template file, for example, /opt/configuration_index.xml. You can obtain the template file example from Client installation directory/HBase/hbase/conf/index_import.xml.template.

vi /opt/configuration_index.xml

An example template file is as follows:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<!--The value of column_num must be consistent with the number of columns in the data file: 5 columns -->
 <import column_num="5" id="first">
  <columns>
   <column type="string" index="1">P_ID</column>
   <column type="string" index="2">P_NAME</column>
   <column type="string" index="3">P_BIRTH</column>
   <column type="string" index="4">P_GENDER</column>
   <column type="string" index="5">P_DISTRICT</column>
  </columns>
<!--reverse(P_BIRTH): Reverse the birth date to avoid hotspotting. -->
<!--substring(P_NAME,0,1): Filter out the student information based on the last name. -->
<!--substring(P_ID,0,6): Filter out the student information based on the first six digits of a student ID. -->
   <rowkey>
    reverse(P_BIRTH)+'_'+substring(P_NAME,0,1)+'_'+substring(P_ID,0,6)
   </rowkey>
  <qualifiers>
  <!--The specified family must correspond to the column family of the table. -->
   <normal family="base">
    <qualifier column="P_ID">H_ID</qualifier>
    <qualifier column="P_NAME">H_NAME</qualifier>
    <qualifier column="P_BIRTH">H_BIRTH</qualifier>
    <qualifier column="P_GENDER">H_GENDER</qualifier>
    <qualifier column="P_DISTRICT">H_DISTRICT</qualifier>
   </normal>
  </qualifiers>
 </import>
</configuration>

Run the following commands to generate an HFile file:
hbase com.huawei.hadoop.hbase.tools.bulkload.ImportData -Dimport.separator=',' -Dimport.hfile.output=/tmp/test/hfile /opt/configuration_index.xml student_info /tmp/test/info.txt
- -Dimport.separator: indicates a separator.
- -Dimport.hfile.output: indicates the output path of the execution result.
- /opt/configuration_index.xml: indicates a custom template file.
- student_info: indicates the name of the HBase table to be operated.
- /tmp/test/info.txt: indicates the HDFS data directory to which data is to be uploaded in batches.
- com.huawei.hadoop.hbase.tools.bulkload.IndexImportData: indicates IndexImportData used to create a secondary index during data import. If no secondary index needs to be created, ImportData is used.
After the MapReduce job is successfully executed, an HFile file is generated in the output path.

hdfs dfs -ls /tmp/test/hfile
```
Found 2 items
-rw-r--r--   3 hbasetest hadoop          0 2021-05-14 11:39 /tmp/test/hfile/_SUCCESS
drwxr-xr-x   - hbasetest hadoop          0 2021-05-14 11:39 /tmp/test/hfile/base
```
Run the following command to import the HFile to the HBase table:
hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles /tmp/test/hfile student_info

Run the following commands to go to the HBase shell page and view the table content:

hbase shell

scan 'student_info', {FORMATTER => 'toString'}

ROW                                               COLUMN+CELL
 10115102_Yang_202001                    column=base:H_BIRTH, timestamp=2021-05-14T15:28:56.755, value=20151101
 10115102_Yang_202001                    column=base:H_DISTRICT, timestamp=2021-05-14T15:28:56.755, value=City3
 10115102_Yang_202001                    column=base:H_GENDER, timestamp=2021-05-14T15:28:56.755, value=female
 10115102_Yang_202001                    column=base:H_ID, timestamp=2021-05-14T15:28:56.755, value=20200101247
 10115102_Yang_202001                    column=base:H_NAME, timestamp=2021-05-14T15:28:56.755, value=Yang xx
 10215102_Li_202001                    column=base:H_BIRTH, timestamp=2021-05-14T15:28:56.755, value=20151201
 10215102_Li_202001                    column=base:H_DISTRICT, timestamp=2021-05-14T15:28:56.755, value=City7
...