Preparing the Configuration Files for Connecting to the Cluster

Updated on 2023-08-31 GMT+08:00

View PDF

User Information for Cluster Authentication

For an MRS cluster with Kerberos authentication enabled, you need to prepare a user who has the operation permission on related components for program authentication.

The following Hive permission configuration example is for reference only. You can modify the configuration as you need.

Log in to FusionInsight Manager.
Choose Cluster > Services > Hive. On the displayed page, click More > Enable Ranger in the upper right corner. Check whether the button is grayed out.
- If it is grayed out, create a user and assign related operation rights to the user in Ranger.
  1. Choose System > Permission > User. On the displayed page, click Create. On the displayed page, create a machine-machine user, for example, developuser.
    Add the hive user group to User Group.
  2. Log in to the Ranger management page as the Ranger administrator rangeradmin.
    The default password of user rangeradmin is Rangeradmin@123. For details, see User Account List.
  3. On the homepage, click Settings and choose Roles.
  4. Click the role whose Role Name is admin. In the Users area, click Select User and select the user name created in 2.a.
  5. Click Add Users, select Is Role Admin in the row where the username is located, and click Save.
    NOTE:
    
    If an HDFS path is specified when you perform operations on a database or table, grant the HDFS path permission to the user in Ranger by referring to Table 2.
- If the button is available, create a user and grant related operation permissions to the user on Manager.
  1. Choose System > Permission > Role. On the displayed page, click Create Role.
    1. Enter the role name, for example, developrole.
    2. In the Configure Resource Permission table, choose Name of the desired cluster > Yarn > Scheduler Queue > root. Select Submit and Admin for default, and click OK.
      NOTE:
      
      If an HDFS path is specified when you perform operations on a database or table, grant the HDFS path permission to the user in Manager by referring to Table 2.
  2. Choose User in the navigation pane and click Create on the displayed page. Create a machine-machine user, for example, developuser.
    - Add the hive user group to User Group.
    - Add the new role created in 2.a to Role.
Log in to FusionInsight Manager as user admin and choose System > Permission > User. In the Operation column of developuser, choose More > Download Authentication Credential. Save the file and decompress it to obtain the user.keytab and krb5.conf files of the user.

Preparing the Configuration Files of the Running Environment

During the development or a test run of the program, you need to use cluster configuration files to connect to an MRS cluster. The configuration files usually contain the cluster component information file and user files used for security authentication. You can obtain the required information from the created MRS cluster.

Nodes used for program debugging or running must be able to communicate with the nodes within the MRS cluster, and the hosts domain name must be configured.

Scenario 1: Prepare the configuration files required for debugging in the local Windows development environment.

Log in to FusionInsight Manager and choose Cluster > Dashboard > More > Download Client. Set Select Client Type to Configuration Files Only. Select the platform type based on the type of the node where the client is to be installed (select x86_64 for the x86 architecture and aarch64 for the Arm architecture) and click OK. After the client files are packaged and generated, download the client to the local PC as prompted and decompress it.
For example, if the client file package is FusionInsight_Cluster_1_Services_Client.tar, decompress it to obtain FusionInsight_Cluster_1_Services_ClientConfig_ConfigFiles.tar. Then, decompress this file.

Go to the Hive\config directory where the client is decompressed and obtain the configuration files listed in Table 1.

**Table 1** Configuration files
File	Description
hiveclient.properties	Configuration parameters for Hive client connection
core-site.xml	Hadoop client configuration parameters

Copy the hosts file content from the decompression directory to the hosts file of the local PC.
NOTE:
- If you need to debug the application in the local Windows environment, ensure that the local PC can communicate with the hosts listed in the hosts file.
- If your PC cannot communicate with the network plane where the MRS cluster is deployed, you can bind an EIP to access the MRS cluster. For details, see How Do I Access Hive of the Cluster in Security Mode on Windows Using EIPs?.
- The local hosts file in a Windows environment is stored, for example, in C:\WINDOWS\system32\drivers\etc\hosts.

Scenario 2: Prepare the configuration files required for running the program in a Linux environment.
1. Install the client on a node.
  For example, the client installation directory is /opt/client.
  
  The difference between the client time and the cluster time must be less than 5 minutes.
2. Obtain the configuration files.
  1. Log in to FusionInsight Manager and choose Cluster > Dashboard > More > Download Client. Set Select Client Type to Configuration Files Only. Select the platform type based on the type of the node where the client is to be installed (select x86_64 for the x86 architecture and aarch64 for the Arm architecture), select Save to Path, and click OK. Download the client configuration file to the active OMS node of the cluster.
  2. Log in to the active OMS node as user root, go to the directory where the client configuration file is stored (/tmp/FusionInsight-Client/ by default), decompress the software package, and obtain the configuration files listed in Table 1 from the Hive/config directory.
    For example, if the client software package is FusionInsight_Cluster_1_Services_Client.tar and the download path is /tmp/FusionInsight-Client on the active OMS node, run the following commands:
    
    cd /tmp/FusionInsight-Client
    
    tar -xvf FusionInsight_Cluster_1_Services_Client.tar
    
    tar -xvf FusionInsight_Cluster_1_Services_ClientConfig_ConfigFiles.tar
    
    cd FusionInsight_Cluster_1_Services_ClientConfig_ConfigFiles
3. Check the network connection of the client node.
  During the client installation, the system automatically configures the hosts file on the client node. You are advised to check whether the /etc/hosts file contains the host names of the nodes in the cluster. If no, manually copy the content of the hosts file in the decompression directory to the hosts file on the node where the client is located, to ensure that the local host can communicate with each host in the cluster.

Hive Operation Permissions

Before application development, the user must have been added to the Hive group. Additional operation permissions must be obtained from the system administrator. For details about permission requirements, see Table 2. To run example programs, you must have the CREATE permission for the default database.

**Table 2** Required permissions
Operation Type/Object	Operation	Required Permissions
DATABASE	CREATE DATABASE dbname [LOCATION "hdfs_path"]	If the HDFS path hdfs_path is specified, the ownership and RWX permission of hdfs_path are required.
	DROP DATABASE dbname	The ownership of database dbname is required.
	ALTER DATABASE dbname SET OWNER user_or_role	The admin permission is required.
TABLE	CREATE TABLE table_a	The CREATE permission on databases is required.
	CREATE TABLE table_a AS SELECT table_b	The CREATE permission on databases, and the SELECT permission on table_b are required.
	CREATE TABLE table_a LIKE table_b	The CREATE permission on databases is required.
	CREATE [EXTERNAL] TABLE table_a LOCATION "hdfs_path"	The CREATE permission for databases, and the ownership and RWX permission of hdfs_path in HDFS are required.
	DROP TABLE table_a	The ownership of table_a is required.
	ALTER TABLE table_a SET LOCATION "hdfs_path"	The ownership of table_a, and the ownership and RWX permission of hdfs_path in HDFS are required.
	ALTER TABLE table_a SET FILEFORMAT	The ownership of table_a is required.
	TRUNCATE TABLE table_a	The ownership of table_a is required.
	ANALYZE TABLE table_a COMPUTE STATISTICS	The SELECT and INSERT permission on table_a is required.
	SHOW TBLPROPERTIES table_a	The SELECT permission on table_a is required.
	SHOW CREATE TABLE table_a	The SELECT permission WITH GRANT OPTION on table_a is required.
Alter	ALTER TABLE table_a ADD COLUMN	The ownership of table_a is required.
	ALTER TABLE table_a REPLACE COLUMN	The ownership of table_a is required.
	ALTER TABLE table_a RENAME	The ownership of table_a is required.
	ALTER TABLE table_a SET SERDE	The ownership of table_a is required.
	ALTER TABLE table_a CLUSTER BY	The ownership of table_a is required.
PARTITION	ALTER TABLE table_a ADD PARTITION partition_spec LOCATION "hdfs_path"	The INSERT permission on table_a, and the ownership and RWX permission of hdfs_path in HDFS are required.
	ALTER TABLE table_a DROP PARTITION partition_spec	The DELETE permission on table_a is required.
	ALTER TABLE table_a PARTITION partition_spec SET LOCATION "hdfs_path"	The ownership of table_a, and the ownership and RWX permission of hdfs_path in HDFS are required.
	ALTER TABLE table_a PARTITION partition_spec SET FILEFORMAT	The ownership of table_a is required.
LOAD	LOAD INPATH 'hdfs_path' INTO TABLE table_a	The INSERT permission on table_a, and the ownership and RWX permission of hdfs_path in HDFS are required.
INSERT	INSERT TABLE table_a SELECT FROM table_b	The INSERT permission on table_a and SELECT permission on table_b are required. The SUBMIT permission on the default Yarn queue is required.
SELECT	SELECT * FROM table_a	The SELECT permission on table_a is required.
	SELECT FROM table_a JOIN table_b	The SELECT permission on table_a and table_b, and the SUBMIT permission on the default Yarn queue are required.
	SELECT FROM (SELECT FROM table_a UNION ALL SELECT FROM table_b)	The SELECT permission on table_a and table_b. The SUBMIT permission on the default Yarn queue is required.
EXPLAIN	EXPLAIN [EXTENDED\|DEPENDENCY] query	The RX permission on related table directories is required.
VIEW	CREATE VIEW view_name AS SELECT ...	The SELECT permission WITH GRANT OPTION on related tables is required.
	ALTER VIEW view_name RENAME TO new_view_name	The ownership of view_name is required.
	DROP VIEW view_name	The ownership of view_name is required.
FUNCTION	CREATE [TEMPORARY] FUNCTION function_name AS 'class_name'	The admin permission is required.
FUNCTION	DROP [TEMPORARY] function_name	The admin permission is required.
MACRO	CREATE TEMPORARY MACRO macro_name ...	The admin permission is required.
MACRO	DROP TEMPORARY MACRO macro_name	The admin permission is required.

NOTE:

You can perform all the previous operations when owning the admin permission of Hive and the corresponding HDFS directory permissions.
If the component uses Ranger for permission control, you need to configure Ranger policies for permission management.

Parent topic: Preparing the Environment

Previous topic: Preparations

Next topic: Configuring and Importing Sample Projects