Updated on 2022-09-15 GMT+08:00

Link to Hive

Description

By creating a Hive link, you can extract data from or load data to Hive of MRS.

Sample Link

{
  "links": [
    {
      "link-config-values": {
        "configs": [
          {
            "inputs": [
              {
                "name": "linkConfig.host",
                "value": "10.120.205.230"
              },
			  {
                "name": "linkConfig.authType",
                "value": "KERBEROS"
              },
              {
                "name": "linkConfig.user",
                "value": "cdm"
              },
              {
                "name": "linkConfig.password",
                "value": "Add password here"
              } 
            ],
            "name": "linkConfig"
          }
        ]
      },
      "name": "hive_link",
      "connector-name": "hive-connector"
    }
  ]
}

Link Parameters

Parameter

Mandatory

Type

Description

llinkConfig.host

Yes

String

IP address of MRS Manager

linkConfig.authType

Yes

Enumeration

Authentication method of MRS. The options are as follows:
  • SIMPLE: for non-security mode
  • KERBEROS: for security mode

linkConfig.principal

No

String

Account principal required for Kerveros authentication. You can contact the administrator to obtain the account. Before using the cluster configuration, you must set this parameter in cluster configuration management.

linkConfig.keytab

No

FileContent

Local absolute path of the keytab file required for Kerveros authentication. You can also contact the administrator to obtain the file. Before using the cluster configuration, you must set this parameter in cluster configuration management.

linkConfig.hiveVersion

Yes

Enumeration

Hive version

  • HIVE_1_X
  • HIVE_3_X

linkConfig.user

No

String

Username for logging in to Manager. This parameter is not required if the cluster configuration is used.

linkConfig.password

No

String

Password for logging in to Manager. This parameter is not required if the cluster configuration is used.

linkConfig.uri

No

String

NameNode URI required for the link to Apache Hadoop. The format is ip:port.

linkConfig.hiveMsUris

No

String

Hive metadata address for connecting to Apache Hadoop. For details, see the hive.metastore.uris configuration item. Example: thrift://host-192-168-1-212:9083

linkConfig.obsSupport

Yes

Boolean

The server must support OBS storage. When creating a Hive table, you can store the table in OBS.

linkConfig.runMode

Yes

Enumeration

This parameter is used only when the Hive version is HIVE_3_X. Possible values are:

  • EMBEDDED: The link instance runs with CDM. This mode delivers better performance.
  • STANDALONE: The link instance runs in an independent process. If CDM needs to connect to multiple Hadoop data sources (MRS, Hadoop, or CloudTable) with both Kerberos and Simple authentication modes, select STANDALONE or configure different agents.

    Note: The STANDALONE mode is used to solve the version conflict problem. If the connector versions of the source and destination ends of the same link are different, a JAR file conflict occurs. In this case, you need to place the source or destination end in the STANDALONE process to prevent the migration failure caused by the conflict.

linkConfig.accessKey

No

String

AK This parameter is mandatory when OBS is supported.

linkConfig.securityKey

No

String

SK This parameter is mandatory when OBS is supported.

linkConfig.properties

No

Map

Properties. You can add configuration properties of the client. Each property must contain a name and a value.