Using BulkLoad to Import Data to HBase in Batches

Scenario

BulkLoad uses MapReduce jobs to directly convert data into HFiles that comply with the internal data format of HBase, and then loads the generated StoreFiles to the corresponding nodes in a cluster. This method requires no flush, compact, or split operations, occupies no region resources, and generates little write requests. Fewer CPU and network resources are required.

Creating an MRS Offline Query Cluster

1. Go to the Buy Cluster page.
2. Click the Quick Config tab and set configuration parameters.

   Table 1 Software configurations

   Parameter	Value
   Region	EU-Dublin
   Billing Mode	Pay-per-use
   Cluster Name	MRS_hbase
   Version Type	Normal
   Cluster Version	MRS 3.1.0
   Component	HBase Query Cluster
   AZ	AZ1
   Enterprise Project	default
   VPC	vpc-01
   Subnet	subnet-01
   Kerberos Authentication	Toggle the slider on.
   Username	root/admin
   Password	Set the password for logging in to the cluster management page and ECS node, for example, Test!@12345.
   Confirm Password	Enter the password again.
   Secure Communications	Select Enable.

   Figure 1 Creating an HBase query cluster
   
3. Click Buy Now and wait until the MRS cluster is created.

   

Importing Local Data to HDFS

1. Prepare a student information file info.txt on the local host.
   The fields include student ID, name, birthday, gender, and address. An example file is as follows:

   20200101245, Zhang xx, 20150324, Male, City 1
   20200101246, Li xx, 20150202, Male, City 2
   20200101247, Yang xx, 20151101, Female, City 3
   20200101248, Chen xx, 20150218, Male, City 4
   20200101249, Li xx, 20150801, Female, City 5
   20200101250, Wang xx, 20150315, Male, City 6
   20200101251, Li xx, 20151201, Male, City 7
   20200101252, Sun xx, 20150916, Female, City 8
   20200101253, Lin xx, 20150303, Male, City 9

2. Log in to the OBS console, click Create Bucket, set the following parameters, and click Create Now.

   Table 2 Bucket parameters

   Parameter	Value
   Region	EU-Dublin
   Bucket Name	mrs-hbase
   Data Redundancy Policy	Single-AZ storage
   Default Storage Class	Standard
   Bucket Policy	Private
   Default Encryption	Disabled
   Direct Reading	Disable
   Enterprise Project	default
   Tags	-

   After the bucket is created, click the bucket name. In the navigation pane on the left, choose Objects and click Upload Object to upload the data file.

   Figure 2 Uploading an object
   
3. Switch back to the MRS console and click the name of the created MRS cluster. On the Dashboard page, click Synchronize next to IAM User Sync. The synchronization takes about five minutes.
4. Upload the data file to the HDFS.
   1. On the Files page, click the HDFS File List and go to the data storage directory, for example, /tmp/test.

      The /tmp/test directory is only an example. You can use any directory on the page or create a new one.

   2. Click Import Data.
      • OBS Path: Find the info.txt file in the created OBS bucket and click Yes.
      • HDFS Path: Select an HDFS path, for example, /tmp/test, and click Yes.
   3. Click OK and wait until the data file is imported.
   Figure 3 Importing data
   

Creating an HBase Table

1. Log in to FusionInsight Manager of the cluster (if no elastic IP address is available, purchase one), create a user named hbasetest, and bind it to the user group supergroup and role System_administrator.

   

2. Download the cluster client, and install it, for example, in the /opt/client directory of the active master node.

   You can also use the cluster client provided by the Master node. The installation directory is /opt/Bigdata/client.

3. Run the following commands to bind an elastic IP address to the active Master node, log in to the active Master node as user root, go to the directory where the client is located, and authenticate the user.

   cd /opt/client

   source bigdata_env

   kinit hbasetest

4. Run the hbase shell command to go to the HBase shell page.

   Plan the table name, rowkey, column family, and column of the HBase data table based on the imported data. Ensure that the rowkey is pre-split during table creation.

   Run the following command to create the student_info table:

   create 'student_info', {NAME => 'base',COMPRESSION => 'SNAPPY', DATA_BLOCK_ENCODING => 'FAST_DIFF'},SPLITS => ['1','2','3','4','5','6','7','8']

   • NAME => 'base': Column family name of the HBase table
   • COMPRESSION: Compression mode
   • DATA_BLOCK_ENCODING: encoding algorithm
   • SPLITS: Region pre-splitting
5. Run the following command to check whether the table is created and exit the HBase shell page:

   list

Generating an HFile and Importing It to HBase

1. Create a custom template file, for example, /opt/configuration_index.xml. You can obtain the template file example from Client installation directory/HBase/hbase/conf/index_import.xml.template.

   vi /opt/configuration_index.xml

   An example template file is as follows:

   <?xml version="1.0" encoding="UTF-8"?>
   <configuration>
   <!--The value of column_num must be consistent with the number of columns in the data file: 5 columns -->
    <import column_num="5" id="first">
     <columns>
      <column type="string" index="1">P_ID</column>
      <column type="string" index="2">P_NAME</column>
      <column type="string" index="3">P_BIRTH</column>
      <column type="string" index="4">P_GENDER</column>
      <column type="string" index="5">P_DISTRICT</column>
     </columns>
   <!--reverse(P_BIRTH): Reverse the birth date to avoid hotspotting. -->
   <!--substring(P_NAME,0,1): Filter out the student information based on the last name. -->
   <!--substring(P_ID,0,6): Filter out the student information based on the first six digits of a student ID. -->
      <rowkey>
       reverse(P_BIRTH)+'_'+substring(P_NAME,0,1)+'_'+substring(P_ID,0,6)
      </rowkey>
     <qualifiers>
     <!--The specified family must correspond to the column family of the table. -->
      <normal family="base">
       <qualifier column="P_ID">H_ID</qualifier>
       <qualifier column="P_NAME">H_NAME</qualifier>
       <qualifier column="P_BIRTH">H_BIRTH</qualifier>
       <qualifier column="P_GENDER">H_GENDER</qualifier>
       <qualifier column="P_DISTRICT">H_DISTRICT</qualifier>
      </normal>
     </qualifiers>
    </import>
   </configuration>

2. Run the following commands to generate an HFile file:

   hbase com.huawei.hadoop.hbase.tools.bulkload.ImportData -Dimport.separator=',' -Dimport.hfile.output=/tmp/test/hfile /opt/configuration_index.xml student_info /tmp/test/info.txt

   • -Dimport.separator: indicates a separator.
   • -Dimport.hfile.output: indicates the output path of the execution result.
   • /opt/configuration_index.xml: indicates a custom template file.
   • student_info: indicates the name of the HBase table to be operated.
   • /tmp/test/info.txt: indicates the HDFS data directory to which data is to be uploaded in batches.
   • com.huawei.hadoop.hbase.tools.bulkload.IndexImportData: indicates IndexImportData used to create a secondary index during data import. If no secondary index needs to be created, ImportData is used.

   After the MapReduce job is successfully executed, run the following command to an HFile file in the output path.

   hdfs dfs -ls /tmp/test/hfile

   Found 2 items
   -rw-r--r--   3 hbasetest hadoop          0 2021-05-14 11:39 /tmp/test/hfile/_SUCCESS
   drwxr-xr-x   - hbasetest hadoop          0 2021-05-14 11:39 /tmp/test/hfile/base

3. Run the following command to import the HFile to the HBase table:

   hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles /tmp/test/hfile student_info

4. Run the following commands to go to the HBase shell page and view the table content:

   hbase shell

   scan 'student_info', {FORMATTER => 'toString'}

   ROW                                               COLUMN+CELL
    10115102_Yang_202001                    column=base:H_BIRTH, timestamp=2021-05-14T15:28:56.755, value=20151101
    10115102_Yang_202001                    column=base:H_DISTRICT, timestamp=2021-05-14T15:28:56.755, value=City3
    10115102_Yang_202001                    column=base:H_GENDER, timestamp=2021-05-14T15:28:56.755, value=female
    10115102_Yang_202001                    column=base:H_ID, timestamp=2021-05-14T15:28:56.755, value=20200101247
    10115102_Yang_202001                    column=base:H_NAME, timestamp=2021-05-14T15:28:56.755, value=Yang xx
    10215102_Li_202001                    column=base:H_BIRTH, timestamp=2021-05-14T15:28:56.755, value=20151201
    10215102_Li_202001                    column=base:H_DISTRICT, timestamp=2021-05-14T15:28:56.755, value=City7
   ...

5. Analyze and process data based on the upper-layer applications of the big data platform after data is imported to the cluster.