Help Center/ MapReduce Service/ Developer Guide (Normal_3.x)/ Spark2x Development Guide (Normal Mode)/ Preparing a Spark Application Development Environment/ Preparing the Configuration File for Connecting Spark to the Cluster
Updated on 2024-10-23 GMT+08:00
View PDF
Share

Preparing the Configuration File for Connecting Spark to the Cluster

During application development, you need to prepare the environment for code running and commissioning to verify that the application is running properly.

  • Scenario 1: Prepare the configuration files required for debugging in the local Windows development environment.
    1. Log in to FusionInsight Manager, choose Cluster > Dashboard, click More, and select Download Client. Set Select Client Type to Configuration Files Only. Select the platform type based on the type of the node where the client is to be installed (select x86_64 for the x86 architecture and aarch64 for the Arm architecture) and click OK. After the client files are packaged and generated, download the client to the local PC as prompted and decompress it.

      For example, if the client file package is FusionInsight_Cluster_1_Services_Client.tar, decompress it to obtain FusionInsight_Cluster_1_Services_ClientConfig_ConfigFiles.tar. Then, decompress FusionInsight_Cluster_1_Services_ClientConfig_ConfigFiles.tar

    2. Go to the *\Spark\config directory where the client configuration file is decompressed, obtain the Spark configuration files, and import all the configuration files to the configuration file directory (usually the resources folder) of the Spark sample project.
    3. Copy the hosts file content from the decompression directory to the hosts file on the node where the client is located.
      • If you need to debug the application in the local Windows environment, ensure that the local PC can communicate with the hosts listed in the hosts file.
      • If your PC cannot communicate with the network plane where the MRS cluster is deployed, you can bind an EIP to access the MRS cluster.
      • The local hosts file in a Windows environment is stored in, for example, C:\WINDOWS\system32\drivers\etc\hosts.
  • Scenario 2: Prepare the configuration files required for running the program in a Linux environment.
    1. Install the MRS cluster client in a directory, for example, /opt/client, on the node.
    2. Obtain the configuration files.
      1. Log in to FusionInsight Manager, click More and select Download Client in the upper right corner of Homepage. Set Select Client Type to Configuration Files Only, select Save to Path, and click OK to download the client configuration file to the active OMS node of the cluster.
      2. Log in to the active OMS node as user root, go to the directory where the client configuration file is stored (/tmp/FusionInsight-Client/ by default), decompress the software package, and obtain the configuration files in the *\Spark\config directory. Place all configuration files in the conf directory (for example, /opt/client /conf) where the compiled JAR package is stored for subsequent commissioning.

        For example, if the client software package is FusionInsight_Cluster_1_Services_Client.tar and the download path is /tmp/FusionInsight-Client on the active OMS node, run the following commands:

        cd /tmp/FusionInsight-Client

        tar -xvf FusionInsight_Cluster_1_Services_Client.tar

        tar -xvf FusionInsight_Cluster_1_Services_ClientConfig.tar

        cd FusionInsight_Cluster_1_Services_ClientConfig

        cp -r Spark/config/* /opt/client/conf

    3. Check the network connection of the client node.

      During the client installation, the system automatically configures the hosts file on the client node. You are advised to check whether the /etc/hosts file contains the host names of the nodes in the cluster. If they are not contained, manually copy the content in the hosts file in the decompression directory to the hosts file on the node where the client is deployed.