Using BulkLoad to Import Data to HBase in Batches
Application Scenarios
When batch importing a large amount of data to HBase, you have many choices, for example, calling the put method of HBase to insert data or using MapReduce to load data from HDFS. However, the two methods cause high pressure on the RegionServer and consume a large number of CPU and network resources because of frequent flush, compact, and split operations of HBase, thereby resulting in low efficiency.
This practice describes how to import local data to HBase in batches using BulkLoad after you create an MRS cluster. This method greatly improves the write efficiency and reduces the write pressure on RegionServer nodes.
Solution Architecture
HBase provides a data import tool called BulkLoad, which imports and directly writes data to underlying data files and WAL logs, greatly improving data loading speed and efficiency.
BulkLoad uses MapReduce jobs to directly convert data into HFiles that comply with the internal data format of HBase, and then loads the generated StoreFiles to the corresponding nodes in a cluster. This method requires no flush, compact, or split operations, occupies no region resources, and generates little write requests. Fewer CPU and network resources are required.
- Large amounts of data needs to be loaded to HBase in the one-off manner.
- When data is loaded to HBase, requirements on reliability are not high and WAL files do not need to be generated.
- When the put method is used to load large amounts of data to HBase, data loading and query will be slow.
- The size of an HFile generated after data loading is similar to the size of HDFS blocks.
Creating an MRS Offline Query Cluster
- Go to the Buy Cluster page.
- Click the Quick Config tab and set configuration parameters.
Table 1 Software configurations Parameter
Description
Value
Region
MRS clusters in different regions cannot communicate with each other over an intranet. For lower network latency and quick resource access, select the region nearest to you.
CN-Hong Kong
Billing Mode
MRS provides two billing modes.- Yearly/Monthly
- Pay-per-use
A prepaid balance will be frozen. For details, see Billing.
Pay-per-use
Cluster Name
The cluster name must be unique. A cluster name can contain 1 to 64 characters. Only letters, digits, hyphens (-), and underscores (_) are allowed.
MRS_hbase
Cluster Type
Available cluster types are as follows:- Analysis cluster
- Streaming cluster
- Hybrid cluster
- Custom cluster
Custom
Version Type
Available version types are as follows:
- Normal
- LTS
Normal
Cluster Version
Available MRS versions
MRS 3.1.0
Component
MRS cluster components. For details about component versions supported by different versions of MRS clusters, see List of MRS Component Versions.
HBase Query Cluster
AZ
An availability zone (AZ) is a physical area that uses independent power and network resources. AZs are physically isolated but interconnected through the internal network. This improves the availability of applications. You are advised to create clusters in different AZs.
AZ1
Enterprise Project
The Enterprise Management console is designed for resource management. It helps you manage cloud-based personnel, resources, permissions, and finance in a hierarchical manner, such as management of companies, departments, and projects.
default
VPC
A Virtual Private Cloud (VPC) is a secure, isolated, and logical network environment.
vpc-01
Subnet
A subnet provides dedicated network resources that are logically isolated from other networks for network security.
subnet-01
Kerberos Authentication
If Kerberos authentication is enabled for a cluster, check whether Kerberos authentication is required. If yes, click Continue. If no, click Back to disable Kerberos authentication and then create a cluster. After a cluster is purchased, this configuration cannot be modified.
Toggle the slider on.
Username
The default value is root/admin. User root is used to remotely log in to ECS nodes, and user admin is used to access the cluster management page.
root/admin
Password
Password for users root/admin.
Set the password for logging in to the cluster management page and ECS node, for example, Test!@12345.
Confirm Password
-
Enter the password again.
Secure Communications
In an MRS cluster, you can provision, manage, and use big data components through the management console. Big data components are deployed in users' VPCs. To allow the MRS console to directly access big data components, you must enable the corresponding security group rules after granting authorization. This authorization process is called secure communications.
If the secure communications function is not enabled, MRS clusters cannot be created.
Select Enable.
Figure 1 Creating an HBase query cluster
- Click Buy Now and wait until the MRS cluster is created.
Importing Local Data to HDFS
- Prepare a student information file info.txt on the local host.
The fields include student ID, name, birthday, gender, and address. An example file is as follows:
20200101245, Zhang xx, 20150324, Male, City 1 20200101246, Li xx, 20150202, Male, City 2 20200101247, Yang xx, 20151101, Female, City 3 20200101248, Chen xx, 20150218, Male, City 4 20200101249, Li xx, 20150801, Female, City 5 20200101250, Wang xx, 20150315, Male, City 6 20200101251, Li xx, 20151201, Male, City 7 20200101252, Sun xx, 20150916, Female, City 8 20200101253, Lin xx, 20150303, Male, City 9
- Log in to OBS Console, click Parallel File Systems in the navigation pane. On the displayed page, click Create Parallel File System, set the following parameters, and click Create Now.
Table 2 Parallel file system parameters Parameter
Value
Region
CN-Hong Kong
File System Name
mrs-hbase
Data Redundancy Policy
Single-AZ storage
Policy
Private
Direct Reading
Disable
Enterprise Project
default
Tags
-
Click the name of the created bucket and click Upload File in the Files tab to upload the data file to the OBS bucket.
- Switch back to the MRS console and click the name of the created MRS cluster. On the Dashboard page, click Synchronize next to IAM User Sync. The synchronization takes about five minutes.
- Upload the data file to the HDFS.
- On the Files page, click the HDFS File List and go to the data storage directory, for example, /tmp/test.
The /tmp/test directory is only an example. You can use any directory on the page or create a new one.
- Click Import Data.
- OBS Path: Find the info.txt file in the created OBS parallel file system and click Yes.
- HDFS Path: Select an HDFS path, for example, /tmp/test, and click Yes.
- Click OK and wait until the data file is imported.
Figure 2 Importing data
- On the Files page, click the HDFS File List and go to the data storage directory, for example, /tmp/test.
Creating an HBase Table
- Log in to FusionInsight Manager of the cluster (if no elastic IP address is available, purchase one), create a user named hbasetest, and bind it to the user group supergroup and role System_administrator.
- Download the cluster client, and install it, for example, in the /opt/client directory of the active master node. For details, see Installing a Client.
You can also use the cluster client provided by the Master node. The installation directory is /opt/Bigdata/client.
- Run the following commands to bind an elastic IP address to the active Master node, log in to the active Master node as user root, go to the directory where the client is located, and authenticate the user.
cd /opt/client
source bigdata_env
kinit hbasetest
- Run the hbase shell command to go to the HBase shell page.
Plan the table name, rowkey, column family, and column of the HBase data table based on the imported data. Ensure that the rowkey is pre-split during table creation.
Run the following command to create the student_info table:
create 'student_info', {NAME => 'base',COMPRESSION => 'SNAPPY', DATA_BLOCK_ENCODING => 'FAST_DIFF'},SPLITS => ['1','2','3','4','5','6','7','8']
- NAME => 'base': Column family name of the HBase table
- COMPRESSION: Compression mode
- DATA_BLOCK_ENCODING: encoding algorithm
- SPLITS: Region pre-splitting
- Check whether the table is created and then exit the HBase shell page.
Generating an HFile and Importing It to HBase
- Create a custom template file, for example, /opt/configuration_index.xml. You can obtain the template file example from Client installation directory/HBase/hbase/conf/index_import.xml.template.
vi /opt/configuration_index.xml
An example template file is as follows:
<?xml version="1.0" encoding="UTF-8"?> <configuration> <!--The value of column_num must be consistent with the number of columns in the data file: 5 columns --> <import column_num="5" id="first"> <columns> <column type="string" index="1">P_ID</column> <column type="string" index="2">P_NAME</column> <column type="string" index="3">P_BIRTH</column> <column type="string" index="4">P_GENDER</column> <column type="string" index="5">P_DISTRICT</column> </columns> <!--reverse(P_BIRTH): Reverse the birth date to avoid hotspotting. --> <!--substring(P_NAME,0,1): Filter out the student information based on the last name. --> <!--substring(P_ID,0,6): Filter out the student information based on the first six digits of a student ID. --> <rowkey> reverse(P_BIRTH)+'_'+substring(P_NAME,0,1)+'_'+substring(P_ID,0,6) </rowkey> <qualifiers> <!--The specified family must correspond to the column family of the table. --> <normal family="base"> <qualifier column="P_ID">H_ID</qualifier> <qualifier column="P_NAME">H_NAME</qualifier> <qualifier column="P_BIRTH">H_BIRTH</qualifier> <qualifier column="P_GENDER">H_GENDER</qualifier> <qualifier column="P_DISTRICT">H_DISTRICT</qualifier> </normal> </qualifiers> </import> </configuration>
- Run the following commands to generate an HFile file:
hbase com.huawei.hadoop.hbase.tools.bulkload.ImportData -Dimport.separator=',' -Dimport.hfile.output=/tmp/test/hfile /opt/configuration_index.xml student_info /tmp/test/info.txt
- -Dimport.separator: indicates a separator.
- -Dimport.hfile.output: indicates the output path of the execution result.
- /opt/configuration_index.xml: indicates a custom template file.
- student_info: indicates the name of the HBase table to be operated.
- /tmp/test/info.txt: indicates the HDFS data directory to which data is to be uploaded in batches.
- com.huawei.hadoop.hbase.tools.bulkload.IndexImportData: indicates IndexImportData used to create a secondary index during data import. If no secondary index needs to be created, ImportData is used.
After the MapReduce job is successfully executed, an HFile file is generated in the output path.
hdfs dfs -ls /tmp/test/hfile
Found 2 items -rw-r--r-- 3 hbasetest hadoop 0 2021-05-14 11:39 /tmp/test/hfile/_SUCCESS drwxr-xr-x - hbasetest hadoop 0 2021-05-14 11:39 /tmp/test/hfile/base
- Run the following command to import the HFile to the HBase table:
hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles /tmp/test/hfile student_info
- Run the following commands to go to the HBase shell page and view the table content:
scan 'student_info', {FORMATTER => 'toString'}
ROW COLUMN+CELL 10115102_Yang_202001 column=base:H_BIRTH, timestamp=2021-05-14T15:28:56.755, value=20151101 10115102_Yang_202001 column=base:H_DISTRICT, timestamp=2021-05-14T15:28:56.755, value=City3 10115102_Yang_202001 column=base:H_GENDER, timestamp=2021-05-14T15:28:56.755, value=female 10115102_Yang_202001 column=base:H_ID, timestamp=2021-05-14T15:28:56.755, value=20200101247 10115102_Yang_202001 column=base:H_NAME, timestamp=2021-05-14T15:28:56.755, value=Yang xx 10215102_Li_202001 column=base:H_BIRTH, timestamp=2021-05-14T15:28:56.755, value=20151201 10215102_Li_202001 column=base:H_DISTRICT, timestamp=2021-05-14T15:28:56.755, value=City7 ...
- Analyze and process data based on the upper-layer applications of the big data platform after data is imported to the cluster.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot