Preparing the Configuration Files for Connecting to the Cluster

Updated on 2024-10-23 GMT+08:00

View PDF

NOTE:

If you want to use the multi-component access sample program, ensure that the Hive and HBase services have been installed in the cluster.

User Information for Cluster Authentication

For an MRS cluster with Kerberos authentication enabled, you need to prepare a user who has the operation permission on related components and download the authentication credential file for program authentication.

The following MapReduce permission configuration example is for reference only. You can flexibly adjust the configuration based on service requirements in actual service scenarios.

Log in to FusionInsight Manager.
Choose System > Permission > Role. On the displayed page, click Create Role.
Enter the role name, for example, developrole.
In the Configure Resource Permission area, set the following parameters and click OK.

Select Name of the desired cluster > HDFS > File system > hdfs://hacluster/, and select Read, Write, Execute, and Recursive for the user.

To execute a multi-component test case, you also need to:
- Choose Name of the desired cluster > HDFS > File System > hdfs://hacluster/ > tmp, and select Read, Write and Execute for hive-scratch. If examples exist, select Read, Write, Execute, and recursion for example.
- Choose Name of the desired cluster > HBase > HBase Scope > global and select the default option create.
- Choose Name of the desired cluster > HBase > HBase Scope > global > hbase, select hbase:meta, and click Execute.
- Choose Name of the desired cluster > Hive > Hive Read Write Privileges and select Query, Insert, Create and recursion for default.
Edit the role. In the Configure Resource Permission table, select Name of the desired cluster > Yarn > Scheduling queue > root, select Submit for default, and click OK.
Choose User in the navigation pane and click Create on the displayed page. Create a machine-machine user, for example, developuser.
- Add the hadoop user group to User Group.
- Add the new role to Role, for example, developrole.
Log in to FusionInsight Manager as user admin and choose System > Permission > User. In the Operation column of developuser, choose More > Download Authentication Credential. Save the file and decompress it to obtain the user.keytab and krb5.conf files of the user.

Preparing the Configuration Files of the Running Environment

During the development or a test run of the program, you need to use cluster configuration files to connect to an MRS cluster. The configuration files usually contain the cluster component information file and user files used for security authentication. You can obtain the required information from the created MRS cluster.

Nodes used for program debugging or running must be able to communicate with the nodes within the MRS cluster, and the hosts domain name must be configured.

Scenario 1: Prepare the configuration files required for debugging in the local Windows development environment.

Log in to FusionInsight Manager, choose Cluster > Overview > More > Download Client, set Select Client Type to All Client Files, select the correct platform type based on the node type of the node where the client is to be installed, and click OK, after the client file package is generated, download the client to the local PC as prompted and decompress it.
For example, if the client file package is FusionInsight_Cluster_1_Services_Client.tar, decompress it to obtain FusionInsight_Cluster_1_Services_ClientConfig.tar. Then, decompress this file

Go to the FusionInsight_Cluster_1_Services_ClientConfig folder and obtain the configuration files listed in Table 1.

**Table 1** **File**
How to Obtain	File	Function
Yarn\config	core-site.xml	Hadoop Core parameters
	hbase-site.xml	HBase parameters
	hdfs-site.xml	HDFS parameters
	mapred-site.xml	Hadoop MapReduce configuration file
	yarn-site.xml	Configures YARN parameters.
HBase\config	hbase-site.xml	HBase parameters
Hive\config	hive-site.xml	Configures Hive parameters.
Hive\config	hiveclient.properties	Configures Hive parameters.

NOTE:

If you do not run the MapReduce multi-component access sample program, you do not need to obtain the hbase-site.xml, hive-site.xml, and hiveclient.properties configuration files of HBase and Hive.

Copy all items from the hosts file in the decompression directory to the hosts file on the node where the client is installed. Ensure that the network communication between the local PC and hosts listed in the hosts file in the decompression directory is normal.
NOTE:
- If the host where the client is installed is not a node in the cluster, configure network connections for the client to prevent errors when you run commands on the client.
- The local hosts file in a Windows environment is stored, for example, in C:\WINDOWS\system32\drivers\etc\hosts.

Scenario 2: Prepare the configuration files required for running the program in a Linux environment.
1. Install the client on a node.
  If MapReduce Accessing Multi-Component Example Project is needed, ensure that the Hive and HBase services have been installed in the cluster.
  
  For example, the client installation directory is /opt/client.
2. Log in to FusionInsight Manager, choose Cluster > Overview > More > Download Client, set Select Client Type to Only Configuration Files, select Save to the following path only, and click OK to download the client configuration file to the active OMS node of the cluster.
3. Log in to the active OMS node as user root, go to the directory (/tmp/FusionInsight-Client/ by default) where the client configuration file is stored, and decompress the software package.
  For example, if the client software package is FusionInsight_Cluster_1_Services_Client.tar and the download path is /tmp/FusionInsight-Client on the active management node, run the following command to decompress the package:
  
  cd /tmp/FusionInsight-Client
  
  tar -xvf FusionInsight_Cluster_1_Services_Client.tar
  
  tar -xvf FusionInsight_Cluster_1_Services_ClientConfig_ConfigFiles.tar
  
  cd /tmp/FusionInsight-Client/FusionInsight_Cluster_1_Services_ClientConfig_ConfigFiles
4. Upload related configuration files to the conf directory (the JAR packages compiled later also need to be stored in this directory) by referring to Table 1 for subsequent commissioning. For example, /opt/hadoopclient/conf. This directory must be created on the client node in advance.
  For example, to upload the core-site.xml file of the Yarn client, run the following command:
  
  scp Yarn/config/core-site.xml root@Client node IP address:/opt/hadoopclient/conf
  
  Run the preceding commands to upload all configuration files listed in Table 1 in sequence.
5. Check the network connection of the client node.
  During the client installation, the system automatically configures the hosts file on the client node. You are advised to check whether the /etc/hosts file contains the host names of the nodes in the cluster. If there is no required information, copy the content of the hosts file in the decompression directory to the hosts file on the node where the client is deployed, to ensure that the local host can communicate with each host in the cluster.