Preparing the Configuration Files for Connecting to the Cluster

Updated on 2023-08-31 GMT+08:00

View PDF

User Information for Cluster Authentication

For an MRS cluster with Kerberos authentication enabled, you need to prepare a user who has the operation permission on related components for program authentication.

The following Flink permission configuration example is for reference only. You can modify the configuration as you need.

Log in to FusionInsight Manager.
Choose System > Permission > Role, and click Create Role to add required permissions based on the permission control type of the service.
1. Enter a role name, for example, developrole.
2. In the Configure Resource Permission table, choose Name of the desired cluster > HDFS > File System > hdfs://hacluster/ > flink, select Read, Write, and Execute, and click Service in the Configure Resource Permission table to return.
3. In the Configure Resource Permission table, choose Name of the desired cluster > Yarn > Scheduler Queue > root, select Submit for default, and click OK.
  NOTE:
  
  If state backend is set to a path on HDFS, for example, hdfs://hacluster/flink-checkpoint, the read, write, and execute permissions on the hdfs://hacluster/flink-checkpoint directory are required.
Choose System > Permission > User Group, and click Create User Group. Enter a group name, for example, developgroup.
Choose System > Permission > User. In the right pane, click Create. On the displayed page, create a human-machine user, for example, developuser.
- User Group: developgroup (primary group) and hadoop
  NOTE:
  
  If a user wants to interconnect with Kafka, a hybrid cluster with Flink and Kafka components is required, or cross-cluster mutual trust needs to be configured for the cluster with Flink and the cluster with Kafka components. Additionally, the created Flink user must be added to the kafkaadmin user group.
- Role: developrolecreated in 2.
Log in to FusionInsight Manager as developuser. Change the password as prompted upon the first login and log in to FusionInsight Manager again.
Choose System > Permission > User. In the Operation column of developuser, choose More > Download Authentication Credential. Save the file and decompress it to obtain user.keytab and krb5.conf files.

Configuration Files of the Running Environment

During the development or a test run of the program, you need to use cluster configuration files to connect to an MRS cluster. The configuration files usually contain the cluster component information file and user files used for security authentication. You can obtain the required information from the created MRS cluster.

Nodes used for program debugging or running must be able to communicate with the nodes within the MRS cluster, and the hosts domain name must be configured.

Prepare the configuration files required for running the program in a Linux environment.

Install the MRS cluster client on the node. For example, the client installation directory can be /opt/client.
NOTE:
- The difference between the client time and the cluster time must be less than 5 minutes.
- The configuration options in the Flink client configuration file flink-conf.yaml must be correctly configured. For details, see Preparing for Security Authentication.
- In security mode, you need to append the service IP address of the node where the client is installed and floating IP address of Manager to the jobmanager.web.allow-access-address configuration item in the flink-conf.yaml file. Use commas (,) to separate IP addresses.

Obtain the configuration files.

Log in to FusionInsight Manager and download the client. Set Select Client Type to Configuration Files Only, select Save to Path, and click OK to download the client configuration file to the active OMS node of the cluster.

Log in to the active OMS node as user root, go to the directory where the client configuration files are stored (/tmp/FusionInsight-Client by default), decompress the software package, and obtain all configuration files in the FusionInsight_Cluster_x_Services_ClientConfig/Flink/config directory. Place the files in the conf directory where the compiled JAR package is stored on the client node, for example, /opt/client/conf.

For example, if the client software package is FusionInsight_Cluster_1_Services_Client.tar and the download path is /tmp/FusionInsight-Client on the active OMS node, run the following commands:

cd /tmp/FusionInsight-Client

tar -xvf FusionInsight_Cluster_1_Services_Client.tar

tar -xvf FusionInsight_Cluster_1_Services_ClientConfig.tar

cd FusionInsight_Cluster_1_Services_ClientConfig

scp Flink/config/* root@IP address of the client node:/opt/client/conf

The keytab file obtained during the User Information for Cluster Authentication is also stored in this directory. Table 1 describes the main configuration files.

**Table 1** Configuration files
File	Function
core-site.xml	Configures Flink parameters.
hdfs-site.xml	Configures HDFS parameters.
yarn-site.xml	Configures YARN parameters.
flink-conf.yaml	Configures the Flink client.
user.keytab	Provides HDFS user information for Kerberos security authentication.
krb5.conf	Provides server configuration information.

Check the network connection of the client node.
During the client installation, the system automatically configures the hosts file on the client node. You are advised to check whether the /etc/hosts file contains the host names of the nodes in the cluster. If no, manually copy the content of the hosts file in the decompression directory to the hosts file on the node where the client is located, to ensure that the local host can communicate with each host in the cluster.