Using BulkLoad to Import Data to HBase in Batches

Application Scenarios

When batch importing a large amount of data to HBase, you have many choices, for example, calling the put method of HBase to insert data or using MapReduce to load data from HDFS. However, the two methods cause high pressure on the RegionServer and consume a large number of CPU and network resources because of frequent flush, compact, and split operations of HBase, thereby resulting in low efficiency.

This practice describes how to import local data to HBase in batches using BulkLoad after you create an MRS cluster. This method greatly improves the write efficiency and reduces the write pressure on RegionServer nodes.

Solution Architecture

HBase provides a data import tool called BulkLoad, which imports and directly writes data to underlying data files and WAL logs, greatly improving data loading speed and efficiency. ‌

BulkLoad uses MapReduce jobs to directly convert data into HFiles that comply with the internal data format of HBase, and then loads the generated StoreFiles to the corresponding nodes in a cluster. This method requires no flush, compact, or split operations, occupies no region resources, and generates little write requests. Fewer CPU and network resources are required.

Creating an MRS Offline Query Cluster

1. Go to the Buy Cluster page.
2. Click the Quick Config tab and set configuration parameters.

   Table 1 Software configurations

   Parameter	Description	Value
   Region	MRS clusters in different regions cannot communicate with each other over an intranet. For lower network latency and quick resource access, select the region nearest to you.	CN-Hong Kong
   Billing Mode	MRS provides two billing modes. Yearly/Monthly Pay-per-use A prepaid balance will be frozen. For details, see Billing.	Pay-per-use
   Cluster Name	The cluster name must be unique. A cluster name can contain 1 to 64 characters. Only letters, digits, hyphens (-), and underscores (_) are allowed.	MRS_hbase
   Cluster Type	Available cluster types are as follows: Analysis cluster Streaming cluster Hybrid cluster Custom cluster	Custom
   Version Type	Available version types are as follows: Normal LTS	Normal
   Cluster Version	Available MRS versions	MRS 3.1.0
   Component	MRS cluster components. For details about component versions supported by different versions of MRS clusters, see List of MRS Component Versions.	HBase Query Cluster
   AZ	An availability zone (AZ) is a physical area that uses independent power and network resources. AZs are physically isolated but interconnected through the internal network. This improves the availability of applications. You are advised to create clusters in different AZs.	AZ1
   Enterprise Project	The Enterprise Management console is designed for resource management. It helps you manage cloud-based personnel, resources, permissions, and finance in a hierarchical manner, such as management of companies, departments, and projects.	default
   VPC	A Virtual Private Cloud (VPC) is a secure, isolated, and logical network environment.	vpc-01
   Subnet	A subnet provides dedicated network resources that are logically isolated from other networks for network security.	subnet-01
   Kerberos Authentication	If Kerberos authentication is enabled for a cluster, check whether Kerberos authentication is required. If yes, click Continue. If no, click Back to disable Kerberos authentication and then create a cluster. After a cluster is purchased, this configuration cannot be modified.	Toggle the slider on.
   Username	The default value is root/admin. User root is used to remotely log in to ECS nodes, and user admin is used to access the cluster management page.	root/admin
   Password	Password for users root/admin.	Set the password for logging in to the cluster management page and ECS node, for example, Test!@12345.
   Confirm Password	-	Enter the password again.
   Secure Communications	In an MRS cluster, you can provision, manage, and use big data components through the management console. Big data components are deployed in users' VPCs. To allow the MRS console to directly access big data components, you must enable the corresponding security group rules after granting authorization. This authorization process is called secure communications. If the secure communications function is not enabled, MRS clusters cannot be created.	Select Enable.

   Figure 1 Creating an HBase query cluster
   
3. Click Buy Now and wait until the MRS cluster is created.

   

Importing Local Data to HDFS

1. Prepare a student information file info.txt on the local host.
   The fields include student ID, name, birthday, gender, and address. An example file is as follows:

   20200101245, Zhang xx, 20150324, Male, City 1
   20200101246, Li xx, 20150202, Male, City 2
   20200101247, Yang xx, 20151101, Female, City 3
   20200101248, Chen xx, 20150218, Male, City 4
   20200101249, Li xx, 20150801, Female, City 5
   20200101250, Wang xx, 20150315, Male, City 6
   20200101251, Li xx, 20151201, Male, City 7
   20200101252, Sun xx, 20150916, Female, City 8
   20200101253, Lin xx, 20150303, Male, City 9

2. Log in to OBS Console, click Parallel File Systems in the navigation pane. On the displayed page, click Create Parallel File System, set the following parameters, and click Create Now.

   Table 2 Parallel file system parameters

   Parameter	Value
   Region	CN-Hong Kong
   File System Name	mrs-hbase
   Data Redundancy Policy	Single-AZ storage
   Policy	Private
   Direct Reading	Disable
   Enterprise Project	default
   Tags	-

   Click the name of the created bucket and click Upload File in the Files tab to upload the data file to the OBS bucket.

3. Switch back to the MRS console and click the name of the created MRS cluster. On the Dashboard page, click Synchronize next to IAM User Sync. The synchronization takes about five minutes.
4. Upload the data file to the HDFS.
   1. On the Files page, click the HDFS File List and go to the data storage directory, for example, /tmp/test.

      The /tmp/test directory is only an example. You can use any directory on the page or create a new one.

   2. Click Import Data.
      • OBS Path: Find the info.txt file in the created OBS parallel file system and click Yes.
      • HDFS Path: Select an HDFS path, for example, /tmp/test, and click Yes.
   3. Click OK and wait until the data file is imported.
   Figure 2 Importing data
   

Creating an HBase Table

1. Log in to FusionInsight Manager of the cluster (if no elastic IP address is available, purchase one), create a user named hbasetest, and bind it to the user group supergroup and role System_administrator.

   

2. Download the cluster client, and install it, for example, in the /opt/client directory of the active master node. For details, see Installing a Client.

   You can also use the cluster client provided by the Master node. The installation directory is /opt/Bigdata/client.

3. Run the following commands to bind an elastic IP address to the active Master node, log in to the active Master node as user root, go to the directory where the client is located, and authenticate the user.

   cd /opt/client

   source bigdata_env

   kinit hbasetest

4. Run the hbase shell command to go to the HBase shell page.

   Plan the table name, rowkey, column family, and column of the HBase data table based on the imported data. Ensure that the rowkey is pre-split during table creation.

   Run the following command to create the student_info table:

   create 'student_info', {NAME => 'base',COMPRESSION => 'SNAPPY', DATA_BLOCK_ENCODING => 'FAST_DIFF'},SPLITS => ['1','2','3','4','5','6','7','8']

   • NAME => 'base': Column family name of the HBase table
   • COMPRESSION: Compression mode
   • DATA_BLOCK_ENCODING: encoding algorithm
   • SPLITS: Region pre-splitting
5. Check whether the table is created and then exit the HBase shell page.

   list

Generating an HFile and Importing It to HBase

1. Create a custom template file, for example, /opt/configuration_index.xml. You can obtain the template file example from Client installation directory/HBase/hbase/conf/index_import.xml.template.

   vi /opt/configuration_index.xml

   An example template file is as follows:

   <?xml version="1.0" encoding="UTF-8"?>
   <configuration>
   <!--The value of column_num must be consistent with the number of columns in the data file: 5 columns -->
    <import column_num="5" id="first">
     <columns>
      <column type="string" index="1">P_ID</column>
      <column type="string" index="2">P_NAME</column>
      <column type="string" index="3">P_BIRTH</column>
      <column type="string" index="4">P_GENDER</column>
      <column type="string" index="5">P_DISTRICT</column>
     </columns>
   <!--reverse(P_BIRTH): Reverse the birth date to avoid hotspotting. -->
   <!--substring(P_NAME,0,1): Filter out the student information based on the last name. -->
   <!--substring(P_ID,0,6): Filter out the student information based on the first six digits of a student ID. -->
      <rowkey>
       reverse(P_BIRTH)+'_'+substring(P_NAME,0,1)+'_'+substring(P_ID,0,6)
      </rowkey>
     <qualifiers>
     <!--The specified family must correspond to the column family of the table. -->
      <normal family="base">
       <qualifier column="P_ID">H_ID</qualifier>
       <qualifier column="P_NAME">H_NAME</qualifier>
       <qualifier column="P_BIRTH">H_BIRTH</qualifier>
       <qualifier column="P_GENDER">H_GENDER</qualifier>
       <qualifier column="P_DISTRICT">H_DISTRICT</qualifier>
      </normal>
     </qualifiers>
    </import>
   </configuration>

2. Run the following commands to generate an HFile file:

   hbase com.huawei.hadoop.hbase.tools.bulkload.ImportData -Dimport.separator=',' -Dimport.hfile.output=/tmp/test/hfile /opt/configuration_index.xml student_info /tmp/test/info.txt

   • -Dimport.separator: indicates a separator.
   • -Dimport.hfile.output: indicates the output path of the execution result.
   • /opt/configuration_index.xml: indicates a custom template file.
   • student_info: indicates the name of the HBase table to be operated.
   • /tmp/test/info.txt: indicates the HDFS data directory to which data is to be uploaded in batches.
   • com.huawei.hadoop.hbase.tools.bulkload.IndexImportData: indicates IndexImportData used to create a secondary index during data import. If no secondary index needs to be created, ImportData is used.

   After the MapReduce job is successfully executed, an HFile file is generated in the output path.

   hdfs dfs -ls /tmp/test/hfile

   Found 2 items
   -rw-r--r--   3 hbasetest hadoop          0 2021-05-14 11:39 /tmp/test/hfile/_SUCCESS
   drwxr-xr-x   - hbasetest hadoop          0 2021-05-14 11:39 /tmp/test/hfile/base

3. Run the following command to import the HFile to the HBase table:

   hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles /tmp/test/hfile student_info

4. Run the following commands to go to the HBase shell page and view the table content:

   hbase shell

   scan 'student_info', {FORMATTER => 'toString'}

   ROW                                               COLUMN+CELL
    10115102_Yang_202001                    column=base:H_BIRTH, timestamp=2021-05-14T15:28:56.755, value=20151101
    10115102_Yang_202001                    column=base:H_DISTRICT, timestamp=2021-05-14T15:28:56.755, value=City3
    10115102_Yang_202001                    column=base:H_GENDER, timestamp=2021-05-14T15:28:56.755, value=female
    10115102_Yang_202001                    column=base:H_ID, timestamp=2021-05-14T15:28:56.755, value=20200101247
    10115102_Yang_202001                    column=base:H_NAME, timestamp=2021-05-14T15:28:56.755, value=Yang xx
    10215102_Li_202001                    column=base:H_BIRTH, timestamp=2021-05-14T15:28:56.755, value=20151201
    10215102_Li_202001                    column=base:H_DISTRICT, timestamp=2021-05-14T15:28:56.755, value=City7
   ...

5. Analyze and process data based on the upper-layer applications of the big data platform after data is imported to the cluster.