Updated on 2023-08-31 GMT+08:00

Preparing the Development Environment

Spark2x applications can be developed in Scala, Java, and Python. Table 1 describes the the development and running environment to be prepared.
Table 1 Development environment

Preparation Item

Description

OS

  • Development environment: Windows OS. Windows 7 or later is supported.
  • Running environment: Windows OS or Linux OS.

    If the program needs to be commissioned locally, the running environment must be able to communicate with the cluster service plane network.

JDK installation

Basic configuration for the Java/Scala development and running environment. The version requirements are as follows:

The server and client support only built-in OpenJDK, version: 1.8.0_272, and therefore a JDK replacement is not allowed.

Customers' applications that need to reference the JAR packages of SDK to run in the application processes support Oracle JDK, IBM JDK, and OpenJDK.

  • For x86 nodes that run clients, the following JDKs can be used:
    • Oracle JDK versions: 1.8
    • IBM JDK versions: 1.8.5.11
  • For TaiShan nodes that run clients, the following JDK can be used:
    • OpenJDK: 1.8.0_272
NOTE:

The server supports only TLS V1.2 or later to ensure security.

By default, IBM JDK supports only TLS V1.0. If IBM JDK is used, setting com.ibm.jsse2.overrideDefaultTLS to true enables TLS V1.0, V1.1, and V1.2. For details, see https://www.ibm.com/support/knowledgecenter/en/SSYKE2_8.0.0/com.ibm.java.security.component.80.doc/security-component/jsse2Docs/matchsslcontext_tls.html#matchsslcontext_tls.

IntelliJ IDEA installation and configuration

It is a tool used to develop Spark applications. Version 2019.1 or other compatible versions are recommended.

NOTE:
  • If the IBM JDK is used, ensure that the JDK configured in IntelliJ IDEA is the IBM JDK.
  • If the Oracle JDK is used, ensure that the JDK configured in IntelliJ IDEA is the Oracle JDK.
  • If the Open JDK is used, ensure that the JDK configured in IntelliJ IDEA is the Open JDK.
  • Do not use the same workspace and the sample project in the same path for different IntelliJ IDEA programs.

Installation of Maven

Basic configurations of the development environment. It can be used for project management throughout the lifecycle of software development.

Scala installation

It is the basic configuration for the Scala development environment. The required version is 2.12.10.

Scala plug-in installation

It is the basic configuration for the Scala development environment. The required version is 2018.2.11 or other compatible versions.

Editra installation

Editra is an editor in the Python development environment and is used to compile Python programs. You can also use other IDEs for Python programming.

7-zip

Used to decompress .zip and .rar packages. The 7-Zip 16.04 is supported.

Python installation

Its version must be 2.6.6 or later.