BulkLoad Configuration File

Updated on 2024-10-09 GMT+08:00

View PDF

This section describes how to use the BulkLoad tool to obtain required data.

Configure a User-Defined Combined Rowkey

When using the BulkLoad tool to import HBase data in batches, users can customize combined rowkeys. Combining rowkeys using BulkLoad is to process some column names in a customized manner using some rules and combine them to generate a new rowkey.

NOTE:

The column name consists of letters, digits, and underscores and cannot contain any special characters.

Details about how to set configuration.xml to combine rowkeys are as follows. For example, combine column SMS_ID, the second to fourth characters of column SMS_NAME, and the reverse of column SMS_SERAIL (The parts are connected by underscores (_)).

 <columns> 
                         <column index="1" type="int">SMS_ID</column> 
                         <column index="2" type="string">SMS_NAME</column> 
                         <column index="3" type="string">SMS_ADDRESS</column> 
                 </columns> 
                 <rowkey> 
                        SMS_ID+'_'+substring(SMS_NAME,1,4)+'_'+reverse(SMS_ADDRESS)
                 </rowkey>

**Table 1** Rowkey segment process functions
Function Prototype	Description	Example
format(data,"DataType")	Used to format string data.	For example, format(data,"0.000") is used to input data in "0.000" format.
converse(data,"yyyy-MM-dd","yyyyMMdd")	Used to convert the date format.	For example, converse(data,"yyyy-MM-dd","yyyyMMdd") is used change the date format from "yyyy-MM-dd" to "yyyyMMdd".
rand	Used to generate a random number. Only the int type is supported.	None
replace(data,"A","B")	Used to replace data.	replace(data,"A","B") is used to replace A with B.
reverse(data)	Used to reverse a character string.	For example, reverse(ABC) is used reverse "ABC" to "CBA".
substring(data,Length1,Length2), or substring(data,Length3)	Used to subtract a character string.	For example, substring(data,1,5), or substring(data,3) is used to subtract [1,5) or [3,data.length) from the data character string.
to_number("data")	Used to convert a character string into a numeric value. The Long type numeric value is supported.	For example, to_number("123") is used to convert "123" into 123. Note that "data" must be a numerical value.

Configuring Customized Rowkey Implementation

When the BulkLoad tool is used to import HBase data in batches, user-defined combined rowkeys are supported. You can compile RowKey implementation code and import combined RowKeys based on the code logic.

To configure a customized row key, perform the following steps:

When you compile the implementation class of a custom RowKey, inherit the interface. The JAR file path of the interface is Client installation directory/HBase/hbase/lib/hbase-it-bulk-load-*.jar.
```
[com.huawei.hadoop.hbase.tools.bulkload.RowkeyHandlerInterface]
```
Interface implementation method:
```
byte[] getRowkeyBytes(String[] colsValues, RegulationDomain regulation)
```
Where
- colsValues indicates a collection of one original data row. Each element is a column.
- regulation indicates information about the configuration file to be imported. (Typically, this parameter is not used.)
Compress the implementation class and its dependent package into a JAR file, save the file to any location on the node where the HBase client resides, and ensure that the user who executes the command has the permission to read and execute the JAR file.
When you run the import command, add the following two configuration items:

-Dimport.rowkey.jar=Full path of the JAR file in step 2

-Dimport.rowkey.class=Full class name of the user implementation class

Configuring Custom Combination Fields

You can combine fields in a customized manner using BulkLoad. Multiple columns are combined into one column in append mode.

NOTE:

The column name consists of letters, digits, and underscores and cannot contain any special characters.

Details about how to combine fields into H_COMBINE_1 are as follows. For example, combine fields SMS_SERIAL, SMS_ADDRESS, and SMS_SNAME into H_COMBINE_1.

<!-- Define composite columns --> 
                         <composite family="f2"> 
                                 <!-- define composited class name, and this class must not exists -->
                                 <qualifier class="com.huawei.H_COMBINE_1">H_COMBINE_1</qualifier> 
                                 <columns> 
                                         <column>SMS_ADDRESS</column> 
                                         <column>SMS_NAME</column> 
                                 </columns> 
                         </composite>

Specifying the Field Data Type

HBase BulkLoad can read native data files, map fields in the data files to fields defined by HBase, and define data types of these fields.

You can define multiple methods in configuration.xml for importing data in batches.

NOTE:

The column name consists of letters, digits, and underscores and cannot contain any special characters.

Details about how to specify the field data type are as follows. For example, specify data types for columns SMS_ID, SMS_NAME, SMS_ADDRESS, and SMS_SERIAL.

                <columns> 
                         <column index="1" type="int">SMS_ID</column> 
                         <column index="2" type="string">SMS_NAME</column> 
                         <column index="3" type="string">SMS_ADDRESS</column> 
                 </columns>

NOTE:

The supported data types include short, int, long, float, double, boolean, and string.

Defining Inapplicable Data Row

BulkLoad supports the function of defining inapplicable data rows. The inapplicable data rows are not stored in HBase. Instead, these data rows are stored in a specific file.

You can define multiple methods in configuration.xml for importing data in batches.

NOTE:

The column name consists of letters, digits, and underscores and cannot contain any special characters.

Details about how to define inapplicable data rows are as follows:

<!-- Define bad line filter rule --> 
<badlines>SMS_ID &lt; 7000 &amp;&amp; SMS_NAME == 'HBase'</badlines>

NOTE:

SMS_ID < 7000 && SMS_NAME == 'HBase'

Table 2 lists the operators in <badlines> and corresponding parameter types.

**Table 2** Operators and corresponding parameter types
Operator	Parameter Type
&&	The parameter type is Boolean.
&	The parameter type is integer.
\|	The parameter type is integer.
^	The parameter type is integer.
/	The parameter type is digit.
==	The parameter type is string.
>=	The parameter type is digit.
>	The parameter type is digit.
<<	The parameter type is integer.
<=	The parameter type is digit.
<	The parameter type is digit.
%	The parameter type is digit.
*	The parameter type is digit.
!=	The parameter type is string.
\|\|	The parameter type is Boolean.
+	The parameter type is digit and string.
>>	The parameter type is integer.
-	The parameter type is string.
>>>	The parameter type is integer.