Help Center> Cloud Stream Service> Eco Development> Cloud Ecosystem: MRS> HBase Sink Stream
View PDF

HBase Sink Stream

Overview

CS exports the job output data to HBase of MRS.

Prerequisites

  • An MRS cluster has been created by using your account. Currently, CS can only interconnect with the MRS cluster disabled with Kerberos authentication.
  • In this scenario, the job needs to run in an exclusive cluster. Ensure that an exclusive cluster has been created.

    For details about how to create an exclusive cluster, see Cluster Management in the Cloud Stream Service User Guide.

  • Ensure that a VPC peering connection has been set up between the exclusive cluster and the MRS cluster, and security group rules have been configured based on the site requirements.

    For details about how to set up the VPC peering connection, see VPC Peering Connection in the Cloud Stream Service User Guide.

    For details about how to configure security group rules, see Security Group in the Virtual Private Cloud User Guide.

Syntax

Syntax

CREATE SINK STREAM stream_id (attr_name attr_type (',' attr_name attr_type)* )WITH (type = "mrs_hbase",region = "",cluster_address = "",table_name = "",table_columns = "",illegal_data_table = "",batch_insert_data_num = "",action = "")

Description

Table 1 Syntax description

Parameter

Mandatory

Description

type

Yes

Output channel type. Value mrs_hbase indicates that data is stored to HBase of MRS.

region

Yes

Region where MRS resides.

cluster_address

Yes

ZooKeeper address of the cluster to which the table where data is to be inserted belongs. The value is in the format of ip1,ip2:port.

table_name

Yes

Name of the table where data is to be inserted.

It can be specified through parameter configurations. For example, if you want one or more certain columns as part of the table name, use car_pass_inspect_with_age_${car_age}, where car_age is the column name.

table_columns

Yes

Columns to be inserted. The parameter value is the following format: rowKey, f1:c1, f1:c2, f2:c1, where rowKey must be specified. If you do not want to add a column, for example the third column, to the database, set this parameter to rowKey,f1:c1,,f2:c1.

illegal_data_table

No

If this parameter is specified, abnormal data (for example, rowKey does not exist) will be written into the table. If not specified, abnormal data will be discarded. The rowKey value is taskNo_Timestamp followed by six random digits, and the schema is info:data, info:reason.

batch_insert_data_num

No

Number of data records to be written in batches at a time. The value must be a positive integer. The upper limit is 1000. The default value is 10.

action

No

Whether data is added or deleted. Available options include add and delete. The default value is add.

Precautions

None

Example

Output data to HBase of MRS.

CREATE SINK STREAM qualified_cars (
  car_id STRING,
  car_owner STRING,
  car_age INT,
  average_speed INT,
  total_miles INT
)
WITH (
  type = "mrs_hbase",
  region = "cn-north-1" ,
  cluster_address = "192.16.0.88,192.87.3.88:2181",
  table_name = "car_pass_inspect_with_age_${car_age}",
  table_columns = "rowKey,info:owner,,car:speed,car:miles",
  illegal_data_table = "illegal_data",
  batch_insert_data_num = "20",
  action = "add"
);
Parent topic: Cloud Ecosystem: MRS