Updated on 2022-08-17 GMT+08:00

To HBase/CloudTable

Sample JSON File

"to-config-values": {
        "configs": [
          {
            "inputs": [
              {
                "name": "toJobConfig.table",
                "value": "rf_to"
              },
              {
                "name": "toJobConfig.storageType",
                "value": "PUTLIST"
              },
              {
                "name": "toJobConfig.columns",
                "value": "AA:AA&BB:BB&CC:CC&DD:DD"
              },
              {
                "name": "toJobConfig.rowKeyColumn",
                "value": "AA:AA"
              },
              {
                "name": "toJobConfig.isOverride",
                "value": "false"
              },
              {
                "name": "toJobConfig.isRowkeyRedundancy",
                "value": "false"
              },
              {
                "name": "toJobConfig.algorithm",
                "value": "NONE"
              },
              {
                "name": "toJobConfig.writeToWAL",
                "value": "true"
              },
              {
                "name": "toJobConfig.transType",
                "value": "false"
              }
            ],
            "name": "toJobConfig"
          }
        ]
      }

Parameter Description

Parameter

Mandatory

Type

Description

toJobConfig.table

Yes

String

Name of the table to which data is written. For example, TBL_EXAMPLE.

toJobConfig.storageType

Yes

Enumeration

Mode for writing data to an HBase table. The options are as follows:
  • BULKLOAD: The BULKLOAD mode is recommended to improve the loading performance.
  • PUTLIST: The PUTLIST mode is recommended only when the data volume is small.

toJobConfig.columns

No

String

Columns to be extracted. Use & to separate column numbers and : to separate column families and columns. For example, cf1:c1&cf2:c2.

toJobConfig.rowKeyColumn

Yes

String

Columns serve as rowkeys. Use & to separate column numbers and : to separate column families and columns. For example, cf1:c1&cf2:c2.

toJobConfig.isOverride

No

Boolean

Whether to clear data when data is imported in BULKLOAD mode. For example, true.

toJobConfig.delimiter

No

String

Delimiter used for separating columns when multiple columns are used as rowkeys. For example, vertical bars (|).

toJobConfig.isRowkeyRedundancy

No

Boolean

Whether to write rowkey data to the HBase column at the same time

toJobConfig.algorithm

No

Enumeration

Compression algorithm used when a new HBase table is created. The Snappy and GZ algorithms are supported. The default value is None.

toJobConfig.writeToWAL

No

Boolean

Whether to enable Write Ahead Log (WAL) of HBase. The options are as follows:

  • Yes: If the HBase server breaks down after the function is enabled, you can replay the operations that have not been performed in WAL.
  • No: If you set this parameter to No, the write performance is improved. However, if the HBase server breaks down, data may be lost.

toJobConfig.transType

No

Boolean

  • true: Data of the Short, Int, Long, Float, Double, and Decimal columns in the source database is converted into Byte[] arrays (binary) and written into HBase. Other types of data are written as character strings. If several types of data mentioned above are combined as rowkeys, they will be written as character strings.

    This function saves storage space. In specific scenarios, the rowkey distribution is evener.

  • false: All types of data in the source database are written into HBase as character strings.