Preparing a Local Application Development Environment

Preparing the Development Environment

Table 1 lists the development and running environment to be prepared for application development.

Preparing the Running Environment

During application development, you need to prepare the code running and debugging environment to verify that the application is running properly.

• If you use the Linux environment for commissioning, prepare the Linux node where the cluster client is to be installed and obtain related configuration files.
  1. Install the client on the node. For example, install the client in the /opt/client directory.

     The difference between the client time and the cluster time must be less than 5 minutes.

     For details about how to use a client on the master or core nodes inside a cluster, see Using an MRS Client on Nodes Inside a Cluster. For details about how to use a client outside a cluster, see Using an MRS Client on Nodes Outside a Cluster.

  2. Log in to the FusionInsight Manager portal. Download the cluster client software package to the active management node and decompress it. Then, log in to the active management node as user root. Go to the decompression path of the cluster client and copy all configuration files in the FusionInsight_Cluster_1_Services_ClientConfig\Flink\config directory to the conf directory where the compiled JAR package is stored for subsequent commissioning, for example, /opt/client/conf.

     For example, if the client software package is FusionInsight_Cluster_1_Services_Client.tar and the download path is /tmp/FusionInsight-Client on the active management node, run the following commands:

     cd /tmp/FusionInsight-Client

     tar -xvf FusionInsight_Cluster_1_Services_Client.tar

     tar -xvf FusionInsight_Cluster_1_Services_ClientConfig.tar

     cd FusionInsight_Cluster_1_Services_ClientConfig

     scp Flink/config/* root@IP address of the client node:/opt/client/conf

     Table 2 describes the main configuration files.

     Table 2 Configuration files

     File	Function
     core-site.xml	Configures Flink parameters.
     hdfs-site.xml	Configures HDFS parameters.
     yarn-site.xml	Configures YARN parameters.
     flink-conf.yaml	Configures the Flink client.

  3. Check the network connection of the client node.

     During the client installation, the system automatically configures the hosts file on the client node. You are advised to check whether the /etc/hosts file contains the host names of the nodes in the cluster. If they are not contained, manually copy the content in the hosts file in the decompression directory to the hosts file on the node where the client is installed to ensure that the local host can communicate with each host in the cluster.

  4. (Optional) To run a Python job, perform the following additional configurations (applicable to MRS 3.3.0 or later):
     1. Log in to the node where the Flink client is installed as user root. Then run the following command to check whether Python 3.7 or a later version has been installed:

        python3 -V

     2. Go to the Python 3 installation path, for example, /srv/pyflink-example, and install virtualenv:

        cd /srv/pyflink-example

        virtualenv venv --python=python3.x

        source venv/bin/activate

     3. Copy the Flink/flink/opt/python/apache-flink-*.tar.gz file from the client installation directory to /srv/pyflink-example:

        cp Client installation directory/Flink/flink/opt/python/apache-flink-*.tar.gz /srv/pyflink-example

     4. Install the dependency package. If the following command output is displayed, the installation is successful:

        python -m pip install apache-flink-libraries-*.tar.gz

        python -m pip install apache-flink-Version number.tar.gz

        ...
        Successfully built apache-flink
         Installing collected packages: apache-flink
          Attempting uninstall: apache-flink
           Found existing installation: apache-flink x.xx.x
           Uninstalling apache- flink-x.xx.x:
            Successfully uninstalled apache-flink-x.xx.x
        Successfully installed apache-flink-x.xx.x