Help Center/ MapReduce Service/ Developer Guide (Normal_Earlier Than 3.x)/ Spark Development Guide/ Preparing a Spark Application Development Environment/ Preparing a Scala Development Environment for Spark
Updated on 2024-08-16 GMT+08:00
View PDF
Share

Preparing a Scala Development Environment for Spark

Scenario

The Scala development environment can be set up on Windows, but the operating environment (client) can be deployed on Linux only.

Procedure

  1. The IDEA tool is recommended for the Scala development environment. The installation requirements are as follows:

    • JDK 1.7 or 1.8 is required.
    • IntelliJ IDEA 13.1.6 is required.
    • Scala 2.11.8 is required.
    • Scala plugin 0.35.683 is required.
    • Spark tasks cannot be submitted to a server in Yarn-client mode when client applications are running on IBM JDK 1.7.
    • Oracle JDK requires security hardening. The operations are as follows:
    1. Obtain the JCE file whose version matches that of JDK from the Oracle official website. After decompression, the JCE file contains local_policy.jar and US_export_policy.jar. Copy the JAR file to the following directory:

      Linux: JDK installation directory/jre/lib/security

      Windows: JDK installation directory\jre\lib\security

    2. Copy SMS4JA.jar in the Client installation directory/JDK/jdk/jre/lib/ext/ directory to the following directory:

      Linux: JDK installation directory/jre/lib/ext/

      Windows: JDK installation directory\jre\lib\ext\

  2. Install and configure the IntelliJ IDEA, JDK, and Scala tools.

    1. Install the JDK.
    2. Install the IntelliJ IDEA.
    3. Install the Scala.
    4. Configure the JDK in the IntelliJ IDEA.
      1. Start the IntelliJ IDEA and select Configure.
        Figure 1 Quick Start
      2. Choose Project Defaults on the Configure page.
        Figure 2 Configure
      3. Choose Project Structure on the Project Defaults page.
        Figure 3 Project Defaults
      4. On the Project Structure page, select SDKs and click the green plus sign to add the JDK.
        Figure 4 Adding the JDK
      5. On the Select Home Directory for JDK page that is displayed, select the JDK directory and click OK.
        Figure 5 Selecting the JDK directory
      6. After selecting the JDK, click OK to complete the configuration.
        Figure 6 Completing the JDK configuration
    5. Install the Scala plugin in the IntelliJ IDEA.
      1. On the Configure page, select Plugins.
        Figure 7 Plugins
      2. On the Plugins page, select Install plugin from disk.
        Figure 8 Install plugin from disk
      3. On the Choose Plugin File page, select the Scala plugin file of the corresponding version and click OK.

      4. On the Plugins page, click Apply to install the Scala plugin.
      5. On the Plugins Changed page that is displayed, click Restart to make the configurations take effect.
        Figure 9 Plugins Changed