Creating a Hadoop Cluster Configuration

On the Cluster Configurations page, you can create, edit, or delete Hadoop cluster configurations.

When creating a Hadoop link, the Hadoop cluster configurations can simplify the link creation. See Figure 1 for details.

Figure 1 Comparison before and after using the cluster configurations

CDM supports the following types of Hadoop links:

MRS clusters: MRS HDFS, MRS HBase, and MRS Hive
FusionInsight clusters: FusionInsight HDFS, FusionInsight HBase, and FusionInsight Hive
Apache clusters: Apache HDFS, Apache HBase, and Apache Hive

Scenario

Before creating a Hadoop link, you are advised to create cluster configurations to simplify the link parameter configurations.

Prerequisites

A cluster has been created.
You have obtained the Hadoop cluster configuration file and keytab file. See Table 1 for details.

Obtaining the Cluster Configuration File and Keytab File

The methods for obtaining the Hadoop cluster configuration file and keytab file vary depending on the Hadoop cluster type. For details, see Table 1.

**Table 1** Obtaining the cluster configuration file and keytab file
Hadoop Link	Obtaining the Cluster Configuration File	Obtaining the Keytab File
MRS cluster MRS HDFS MRS HBase MRS Hive MRS Hudi MRS ClickHouse	For clusters of MRS 3.x: Log in to FusionInsight Manager. Choose Cluster > Name of the desired cluster > Dashboard > More > Download Client. In the dialog box that is displayed, select Configuration Files Only. The platform type must be the same as that on the server. Retain the default values of other parameters and click OK to download the configuration file to the local host. Obtain the downloaded TAR package, which is the FusionInsight cluster configuration file. For clusters of MRS 2.x or earlier: Log in to the MRS console. Choose Clusters > Active Clusters and click a cluster name to go to the cluster details page. Click the Components tab. Click Download Client. Set Client Type to Only configuration files, set Download To to Server or Remote host, customize the client path, and click OK to generate the client configuration file. Save the generated configuration file to a local path. See MRS documentation for details.	For clusters of MRS 3.x: Log in to FusionInsight Manager. Choose System > Permission > User, locate the row that contains the target user, and choose More > Download Authentication Credential to download the authentication credential file. Obtain the downloaded TAR package, which is the keytab file of the FusionInsight cluster. For clusters of MRS 2.x or earlier: Log in to MRS Manager and click System. In the Permission area, click Manage User. In the row of the user for whom you want to export the keytab file, choose More > Download authentication credential to download the authentication file. After the file is automatically generated, save it to a specified path and keep it properly. See MRS documentation for details.
FusionInsight clusters: FusionInsight HDFS FusionInsight HBase FusionInsight Hive	Log in to FusionInsight Manager. Choose Cluster > Name of the desired cluster > Dashboard > More > Download Client. In the dialog box that is displayed, select Configuration Files Only. The platform type must be the same as that on the server. Retain the default values of other parameters and click OK to download the configuration file to the local host. Obtain the downloaded TAR package, which is the FusionInsight cluster configuration file. See the FusionInsight documentation for details.	Log in to FusionInsight Manager. Choose System > Permission > User, locate the row that contains the target user, and choose More > Download Authentication Credential to download the authentication credential file. Obtain the downloaded TAR package, which is the keytab file of the FusionInsight cluster. See the FusionInsight documentation for details.
Apache clusters: Apache HDFS Apache HBase Apache Hive	In the Apache cluster scenario, only the required configuration files and packaging rules are described. For details about how to obtain each configuration file, see the corresponding documentation. HDFS needs to compress the following files into a .zip package without the directory format: hosts core-site.xml hdfs-site.xml yarn-site.xml mapred-site.xml krb5.conf (optional, for clusters in security mode) HBase needs to compress the following files into a .zip package without the directory format: hosts core-site.xml hdfs-site.xml yarn-site.xml mapred-site.xml hbase-site.xml krb5.conf (optional, for clusters in security mode) Hive needs to compress the following files into a .zip package without the directory format: hosts core-site.xml hdfs-site.xml yarn-site.xml mapred-site.xml hive-site.xml hivemetastore-site.xml krb5.conf (optional, for clusters in security mode)	In the Apache cluster scenario, only the principles for packaging authentication credential files are required. For details about how to obtain the authentication credential files, see the corresponding documentation. Rename the user's authentication credential file as user.keytab. Compress the user.keytab file into a .zip package without the directory format: user.keytab.zip.

A cluster configuration file contains the configuration parameters of the cluster. If the cluster configuration parameters are modified, you need to obtain the configuration file again.
The keytab file is the authentication credential file. Before obtaining the keytab file, you need to change the password of this user at least once in the cluster. Otherwise, the downloaded keytab file may be unavailable. After a user password is changed, the exported keytab file becomes invalid, and you need to export a keytab file again.
The keytab file is used only in a cluster in security mode. In other cases, you do not need to prepare the keytab file.

Procedure

On the CDM console, choose Cluster Management in the left navigation pane. Locate the row that contains a cluster and choose Job Management > Links > Cluster Configurations.
On the Cluster Configurations page, click Create Cluster Configuration and set the parameters as prompt.
Figure 2 Creating cluster configurations
- Configuration Name: Enter a cluster configuration name that is easy to remember and distinguish based on the type of the data source to be connected.
- Configuration File: Click Select File to select a local cluster configuration file, and then click Upload on the right to upload the file.
- Principal: This parameter is required only for clusters in security mode. Principal is the username in Kerberos security mode and must be the same as that in the keytab file.
- Keytab File: Upload the keytab file only for clusters in security mode. Click Select File to select a local keytab file, and then click Upload on the right to upload the file.
- Description: Add a description to identify and distinguish the cluster configuration.
Click OK. When creating a Hadoop link, set Authentication Method as required, Use Cluster Config to Yes, and then select the corresponding cluster configuration name to quickly create a Hadoop link.
Figure 3 Use Cluster Config