Source Table

Function

Create a source stream to obtain data from HBase as input for jobs. HBase is a column-oriented distributed cloud storage system that features enhanced reliability, excellent performance, and elastic scalability. It applies to the storage of massive amounts of data and distributed computing. You can use HBase to build a storage system capable of storing TB- or even PB-level data. With HBase, you can filter and analyze data with ease and get responses in milliseconds, rapidly mining data value. DLI can read data from HBase for filtering, analysis, and data dumping.

Prerequisites

• An enhanced datasource connection has been created for DLI to connect to HBase, so that jobs can run on the dedicated queue of DLI and you can set the security group rules as required.
  • For details about how to set up an enhanced datasource connection, see Enhanced Datasource Connections.
  • For details about how to configure security group rules, see Security Group Overview.
• If MRS HBase is used, IP addresses of all hosts in the MRS cluster have been added to host information of the enhanced datasource connection.

  For details, see Modifying Host Information.

Caveats

• When you create a Flink OpenSource SQL job, set Flink Version to 1.15 in the Running Parameters tab. Select Save Job Log, and specify the OBS bucket for saving job logs.
• Storing authentication credentials such as usernames and passwords in code or plaintext poses significant security risks. It is recommended using DEW to manage credentials instead. Storing encrypted credentials in configuration files or environment variables and decrypting them when needed ensures security. For details, see Flink OpenSource SQL Jobs Using DEW to Manage Access Credentials.
• The column families in created HBase source table must be declared as the ROW type, the field names map the column family names, and the nested field names map the column qualifier names.

  There is no need to declare all the families and qualifiers in the schema. Users can declare what is used in the query. Except the ROW type fields, the single atomic type field (for example, STRING or BIGINT) will be recognized as the HBase rowkey. The rowkey field can be an arbitrary name, but should be quoted using backticks if it is a reserved keyword.

Syntax

create table hbaseSource (
  attr_name attr_type 
  (',' attr_name attr_type)* 
  (',' watermark for rowtime_column_name as watermark-strategy_expression)
  ','PRIMARY KEY (attr_name, ...) NOT ENFORCED)
)
with (
  'connector' = 'hbase-2.2',
  'table-name' = '',
  'zookeeper.quorum' = ''
);

Parameters

Data Type Mapping

HBase stores all data as byte arrays. The data needs to be serialized and deserialized during read and write operations.

When serializing and de-serializing, Flink HBase connector uses utility class org.apache.hadoop.hbase.util.Bytes provided by HBase (Hadoop) to convert Flink data types to and from byte arrays.

Flink HBase connector encodes null values to empty bytes, and decodes empty bytes to null values for all data types except the string type. For string type, the null literal is determined by the null-string-literal option.

Example

In this example, data is read from the HBase data source and written to the Print result table. (The HBase version used in this example is 2.2.3.)

1. Create an enhanced datasource connection in the VPC and subnet where HBase locates, and bind the connection to the required Flink queue. For details, see Enhanced Datasource Connections. Add MRS host information for the enhanced datasource connection by referring to Modifying Host Information.
2. Set HBase cluster security groups and add inbound rules to allow access from the Flink job queue. Test the connectivity using the HBase address by referring to Testing Address Connectivity. If the connection passes the test, it is bound to the queue.
3. Use the HBase shell to create HBase table order that has only one column family detail. For details, see Using HBase from Scratch. The creation statement is as follows:

   create 'order', {NAME => 'detail'}

4. Run the following command in the HBase shell to insert a data record:

   put 'order', '202103241000000001', 'detail:order_channel','webShop'
   put 'order', '202103241000000001', 'detail:order_time','2021-03-24 10:00:00'
   put 'order', '202103241000000001', 'detail:pay_amount','100.00'
   put 'order', '202103241000000001', 'detail:real_pay','100.00'
   put 'order', '202103241000000001', 'detail:pay_time','2021-03-24 10:02:03'
   put 'order', '202103241000000001', 'detail:user_id','0001'
   put 'order', '202103241000000001', 'detail:user_name','Alice'
   put 'order', '202103241000000001', 'detail:area_id','330106'

5. Create a Flink OpenSource SQL job. Enter the following job script and submit the job. The job script uses the HBase data source and the Print result table.
   When you create a job, set Flink Version to 1.15 in the Running Parameters tab. Select Save Job Log, and specify the OBS bucket for saving job logs. Change the values of the parameters in bold as needed in the following script.

   create table hbaseSource (
     order_id string,-- Indicates the unique rowkey.
     detail Row( -- Indicates the column family.
       order_channel string,
       order_time string,
       pay_amount string,
       real_pay string,
       pay_time string,
       user_id string,
       user_name string,
       area_id string),
     primary key (order_id) not enforced
   ) with (
     'connector' = 'hbase-2.2',
      'table-name' = 'order',
      'zookeeper.quorum' = 'ZookeeperAddress:ZookeeperPort'
   ) ;
   
   create table printSink (
     order_id string,
     order_channel string,
     order_time string,
     pay_amount string,
     real_pay string,
     pay_time string,
     user_id string,
     user_name string,
     area_id string
   ) with (
    'connector' = 'print'
   );
   
   insert into printSink select order_id, detail.order_channel,detail.order_time,detail.pay_amount,detail.real_pay,
   detail.pay_time,detail.user_id,detail.user_name,detail.area_id from hbaseSource;

6. Perform the following operations to view the data result in the taskmanager.out file:
   1. Log in to the DLI console. In the navigation pane, choose Job Management > Flink Jobs.
   2. Click the name of the corresponding Flink job, choose Run Log, click OBS Bucket, and locate the folder of the log you want to view according to the date.
   3. Go to the folder of the date, find the folder whose name contains taskmanager, download the taskmanager.out file, and view result logs.

   The data result is as follows:

   +I(202103241000000001,webShop,2021-03-24 10:00:00,100.00,100.00,2021-03-24 10:02:03,0001,Alice,330106)

FAQ

• Q: What should I do if the Flink job execution fails and the log contains the following error information?

  java.lang.IllegalArgumentException: offset (0) + length (8) exceed the capacity of the array: 6

  A: If data in the HBase table is imported in other modes, the data is represented in the string format. Therefore, this error is reported when other data formats are used. Change the type of the non-string fields in the HBase source table created by Flink to the string format.