Updated on 2025-04-14 GMT+08:00

Preparing a Local Application Development Environment

Preparing the Development Environment

This section describes the development and running environment to be prepared for application development.

Table 1 Development environment

Item

Description

Operating system (OS)

  • Development environment: Windows 7 or later
  • Running environment: Linux

    If the program needs to be commissioned locally, the running environment must be able to communicate with the cluster service plane.

JDK installation

Basic configuration of the development and running environment. The version requirements are as follows:

The server and client support only the built-in OpenJDK. Other JDKs cannot be used.

If the JAR packages of the SDK classes that need to be referenced by the customer applications run in the application process, the JDK requirements are as follows:

  • For x86 nodes that run clients, use the following JDKs:
    • Oracle JDK 1.8
    • IBM JDK 1.8.0.7.20 and 1.8.0.6.15
  • For Arm nodes that run clients, use the following JDKs:
    • OpenJDK 1.8.0_272 (built-in JDK, which can be obtained from the JDK folder in the cluster client installation directory.)
    • BiSheng JDK 1.8.0_272
NOTE:

IDEA installation and configuration

Tool used for developing Flink applications. IDEA 2019.1 or other compatible versions are required.

Scala installation

Basic configuration for the Scala development environment.

The required versions are as follows:

  • Scale 2.11.7 for versions earlier than MRS 3.2.0
  • Scale 2.12.7 for MRS 3.2.0 to MRS 3.5.0

Scala plug-in installation

Basic configuration for the Scala development environment. Version 1.5.4 is required.

Maven installation

Basic configurations for the development environment. Maven is used for project management throughout the lifecycle of software development.

Developer account

Cluster user for application development. Create the user and grant permissions to the user by referring to Preparing MRS Application Development User.

7-Zip

Tool used to decompress *.zip and *.rar files. 7-Zip 16.04 is supported.

Python3

Tool used to run Flink Python jobs. Python 3.7 to Python 3.10 is required.

Preparing the Running Environment

This section describes the running and debugging environment to be prepared for application development. The environments are used to verify that applications can run properly.

  • If you use the Linux environment for debugging, prepare a Linux node for installing the cluster client and obtain related configuration files.
    1. Install the client on the node, for example, in the /opt/hadoopclient directory. For details about how to install a client, see Installing the Client and Preparing for Security Authentication.

      The difference between the client time and the cluster time must be less than 5 minutes.

      For details about how to use a client on the master or core nodes inside a cluster, see Using an MRS Client on Nodes Inside a Cluster. For details about how to use a client outside a cluster, see Using an MRS Client on Nodes Outside a Cluster.

      • Ensure that the authentication-related configuration items in the flink-conf.yaml configuration file of the Flink client are correctly configured. For details, see step 5 in Installing the Client and Preparing for Security Authentication.
      • In security mode, append the service IP address of the node where the client is installed and floating IP address of Manager to the jobmanager.web.allow-access-address configuration item in the flink-conf.yaml file. Use commas (,) to separate IP addresses.
    2. Log in to FusionInsight Manager. Download the cluster client software package to the active management node and decompress it. Then, log in to the active management node as user root. Go to the decompression path of the cluster client and copy all configuration files in the FusionInsight_Cluster_1_Services_ClientConfig\Flink\config directory to the conf directory where the compiled JAR file is stored for subsequent commissioning, for example, /opt/hadoopclient/conf.

      For example, if the client software package is FusionInsight_Cluster_1_Services_Client.tar and the download path is /tmp/FusionInsight-Client on the active management node, run the following commands:

      cd /tmp/FusionInsight-Client

      tar -xvf FusionInsight_Cluster_1_Services_Client.tar

      tar -xvf FusionInsight_Cluster_1_Services_ClientConfig.tar

      cd FusionInsight_Cluster_1_Services_ClientConfig

      scp Flink/config/* root@IP address of the client node:/opt/hadoopclient/conf

      The keytab file obtained during the Preparing MRS Application Development User is also stored in this directory. Table 2 describes the main configuration files.

      Table 2 Configuration files

      File

      Function

      core-site.xml

      Configures Flink parameters.

      hdfs-site.xml

      Configures HDFS parameters.

      yarn-site.xml

      Configures YARN parameters.

      flink-conf.yaml

      Configures the Flink client.

      user.keytab

      Provides user information for Kerberos security authentication.

      krb5.conf

      Provides Kerberos server configuration information.

    3. Check the network connection of the client node.

      During the client installation, the system automatically configures the hosts file on the client node. You are advised to check whether the /etc/hosts file contains the host names of the nodes in the cluster. If they are not contained, manually copy the content in the hosts file in the decompression directory to the hosts file on the node where the client is installed to ensure that the local host can communicate with each host in the cluster.

    4. (Optional) To run a Python job, perform the following additional configurations (applicable to MRS 3.3.0 or later):
      1. Log in to the node where the Flink client is installed as user root. Then run the following command to check whether Python 3.7 or a later version has been installed:

        python3 -V

      2. Go to the Python 3 installation path, for example, /srv/pyflink-example, and install virtualenv:

        cd /srv/pyflink-example

        virtualenv venv --python=python3.x

        source venv/bin/activate

      3. Copy the Flink/flink/opt/python/apache-flink-*.tar.gz file from the client installation directory to /srv/pyflink-example:

        cp Client installation directory/Flink/flink/opt/python/apache-flink-*.tar.gz /srv/pyflink-example

      4. Install the dependency package. If the following command output is displayed, the installation is successful:

        python -m pip install apache-flink-libraries-*.tar.gz

        python -m pip install apache-flink-Version number.tar.gz
        ...
        Successfully built apache-flink
         Installing collected packages: apache-flink
          Attempting uninstall: apache-flink
           Found existing installation: apache-flink x.xx.x
           Uninstalling apache- flink-x.xx.x:
            Successfully uninstalled apache-flink-x.xx.x
        Successfully installed apache-flink-x.xx.x