Preparing the Configuration File for Connecting Spark to the Cluster

Updated on 2024-10-23 GMT+08:00

View PDF

Preparing User Information for Cluster Authentication

For an MRS cluster with Kerberos authentication enabled, prepare a user who has the operation permission on related components for program authentication.

The following Spark2x permission configuration example is for reference only. You can modify the configuration as you need.

Log in to FusionInsight Manager.
Choose Cluster > Services > Spark2x. On the displayed page, click More > Enable Ranger in the upper right corner. Check whether the button is grayed out.
- If it is grayed out, create a user and assign related operation rights to the user in Ranger.
  1. Choose System > Permission > User. On the displayed page, click Create. On the displayed page, create a machine-machine user, for example, developuser.
    The developgroup group needs to be added. If you need to interconnect with Kafka, add the Kafkaadmin user group.
  2. Log in to the Ranger management page as the Ranger administrator rangeradmin.
  3. On the home page, click the component plug-in name in the HADOOP SQL area, for example, Hive.
  4. Click in the Action column of the row containing the all - database, table, column policy.
  5. In the Allow Conditions area, add an allow condition. Select the user created in 2.a for Select User, and select Select All for Permissions.
  6. Click Save.
- If the button is available, create a user and grant related operation permissions to the user on Manager.
  1. Choose System > Permission > Role. On the displayed page, click Create Role.
    1. Enter the role name, for example, developrole.
    2. (Configure this parameter if HBase is installed.) In Configure Resource Permission, click the name of the desired cluster and choose HBase > HBase Scope > global, select the default option create, and click OK.
    3. (Configure this parameter if HBase is installed.) Edit the role. In Configure Resource Permission, click the name of the desired cluster and choose HBase > HBase Scope > global > hbase, select execute for hbase:meta, and click OK.
    4. Edit the role. In Configure Resource Permission, click the name of the desired cluster, choose HDFS > File System, and choose hdfs://hacluster/ > user. Select Execute for hive, and click OK.
    5. Edit the role. In Configure Resource Permission, click the name of the desired cluster, choose HDFS > File System and choose hdfs://hacluster/ > user > hive. Select Read, Write, and Execute for warehouse, and click OK.
    6. Edit the role. In Configure Resource Permission, click the name of the desired cluster, choose Hive > Hive Read Write Privileges, select the default option Create, and click OK.
    7. Edit the role. In Configure Resource Permission, click the name of the desired cluster, choose Yarn > Scheduler Queue > root, select the default option Submit, and click OK.
  2. Choose User in the navigation pane and click Create on the displayed page. Create a machine-machine user, for example, developuser.
    - Add the hadoop user group to User Group.
    - Add the new role created in 2.a to Role.
Log in to FusionInsight Manager as user admin and choose System > Permission > User. In the Operation column of developuser, choose More > Download Authentication Credential. Save the file and decompress it to obtain the user.keytab and krb5.conf files of the user.

Preparing the Configuration Files of the Running Environment

During application development, you need to prepare the environment for code running and commissioning to verify that the application is running properly.

Scenario 1: Prepare the configuration files required for debugging in the local Windows development environment.
1. Log in to FusionInsight Manager, choose Cluster > Dashboard, click More, and select Download Client. Set Select Client Type to Configuration Files Only. Select the platform type based on the type of the node where the client is to be installed (select x86_64 for the x86 architecture and aarch64 for the Arm architecture) and click OK. After the client files are packaged and generated, download the client to the local PC as prompted and decompress it.
  For example, if the client file package is FusionInsight_Cluster_1_Services_Client.tar, decompress it to obtain FusionInsight_Cluster_1_Services_ClientConfig_ConfigFiles.tar. Then, decompress this file.
2. Go to the *\Spark\config directory where the client configuration file is decompressed, obtain the Spark configuration files, and import all the configuration files to the configuration file directory (usually the resources folder) of the Spark sample project.
  The keytab file obtained in Preparing User Information for Cluster Authentication is also stored in this directory.
3. Copy the hosts file content from the decompression directory to the hosts file on the node where the client is located.
  NOTE:
  - If you need to debug the application in the local Windows environment, ensure that the local PC can communicate with the hosts listed in the hosts file.
  - If your PC cannot communicate with the network plane where the MRS cluster is deployed, you can bind an EIP to access the MRS cluster.
  - The local hosts file in a Windows environment is stored in, for example, C:\WINDOWS\system32\drivers\etc\hosts.

Scenario 2: Prepare the configuration files required for running the program in a Linux environment.
1. Install the MRS cluster client in a directory, for example, /opt/client, on the node.
2. Obtain the configuration files.
  1. Log in to FusionInsight Manager, click More and select Download Client in the upper right corner of Homepage. Set Select Client Type to Configuration Files Only, select Save to Path, and click OK to download the client configuration file to the active OMS node of the cluster.
  2. Log in to the active OMS node as user root, go to the directory where the client configuration file is stored (/tmp/FusionInsight-Client/ by default), decompress the software package, and obtain the configuration files in the *\Spark\config directory. Place all configuration files in the conf directory (for example, /opt/client /conf) where the compiled JAR package is stored for subsequent commissioning.
    For example, if the client software package is FusionInsight_Cluster_1_Services_Client.tar and the download path is /tmp/FusionInsight-Client on the active OMS node, run the following commands:
    
    cd /tmp/FusionInsight-Client
    
    tar -xvf FusionInsight_Cluster_1_Services_Client.tar
    
    tar -xvf FusionInsight_Cluster_1_Services_ClientConfig.tar
    
    cd FusionInsight_Cluster_1_Services_ClientConfig
    
    cp -r Spark/config/* /opt/client/conf
    
    The keytab file obtained in Preparing User Information for Cluster Authentication is also stored in this directory.
3. Check the network connection of the client node.
  During the client installation, the system automatically configures the hosts file on the client node. You are advised to check whether the /etc/hosts file contains the host names of the nodes in the cluster. If they are not contained, manually copy the content in the hosts file in the decompression directory to the hosts file on the node where the client is deployed.