Help Center/ MapReduce Service/ Developer Guide (Normal_Earlier Than 3.x)/ Spark Development Guide/ Preparing a Spark Application Development Environment/ Preparing a Java Development Environment for Spark
Updated on 2024-08-16 GMT+08:00
View PDF
Share

Preparing a Java Development Environment for Spark

Scenario

The Java development environment can be set up on Windows, but the operating environment (client) can be deployed on Linux only.

Procedure

  1. The IDEA tool is recommended for the Java development environment. The installation requirements are as follows:

    • JDK 1.7 or 1.8 is required.
    • IntelliJ IDEA 13.1.6 is required.
    • Spark tasks cannot be submitted to a server in Yarn-client mode when client applications are running on IBM JDK 1.7.
    • Oracle JDK requires security hardening. The operations are as follows:
    1. Obtain the Java Cryptography Extension (JCE) file whose version matches that of JDK from the Oracle official website. After decompression, the JCE file contains local_policy.jar and US_export_policy.jar. Copy the JAR file to the following directory:

      Linux: JDK installation directory/jre/lib/security

      Windows: JDK installation directory\jre\lib\security

    2. Copy SMS4JA.jar in the Client installation directory/JDK/jdk/jre/lib/ext/ directory to the following directory:

      Linux: JDK installation directory/jre/lib/ext/

      Windows: JDK installation directory\jre\lib\ext\

  2. Install and configure the IntelliJ IDEA and JDK tools.

    1. Install the JDK.
    2. Install IntelliJ IDEA.
    3. Configure the JDK in the IntelliJ IDEA.
      1. Start the IntelliJ IDEA and select Configure.
        Figure 1 Quick start
      2. Choose Project Defaults on the Configure page.
        Figure 2 Configure page
      3. Choose Project Structure on the Project Defaults page.
        Figure 3 Project Defaults page
      4. On the Project Structure page, select SDKs and click the green plus sign to add the JDK.
        Figure 4 Adding the JDK
      5. On the Select Home Directory for JDK page that is displayed, select the JDK directory and click OK.
        Figure 5 Selecting the JDK directory
      6. After selecting the JDK, click OK to complete the configuration.
        Figure 6 Completing the JDK configuration