Importing and Configuring Spark Sample Projects

Overview

Spark provides sample projects for different scenarios, including Java and Scala projects, to help users quickly understand Spark projects.

The import methods for Java and Scala projects are the same Python sample projects do not need to be imported and you only need to open the Python file (*.py).

The import of Java sample codes is used as a sample in the following procedure. Figure 1 shows the procedure of importing sample projects.

Figure 1 Procedure of importing sample projects

Procedure

1. Obtain multiple sample projects such as Scala and Spark Streaming in the sparknormal-examples folder within the spark-examples directory where the sample code is decompressed. For details, see Obtaining the MRS Application Development Sample Project.
2. After installing the IntelliJ IDEA and JDK, configure the JDK in IntelliJ IDEA.
   1. Start IntelliJ IDEA and select Configure.
      Figure 2 Quick Start
      
   2. Choose Project Defaults from the Configure drop-down list.
      Figure 3 Configure
      
   3. Select Project Structure from Project Defaults.
      Figure 4 Project Defaults
      
   4. On the Project Structure page, select SDKs and click the plus sign to add the JDK.
      Figure 5 Adding the JDK
      
   5. On the displayed Select Home Directory for JDK page, select the JDK directory and click OK.
      Figure 6 Selecting a home directory for the JDK
      
   6. After selecting the JDK, click OK to complete the configuration.
      Figure 7 Completing the configuration
      

3. (Optional) If the Scala development environment is used, install the Scala plugin in IntelliJ IDEA.
   1. Select Plugins from the Configure drop-down list.
      Figure 8 Plugins
      
   2. On the Plugins page, select Install plugin from disk.
      Figure 9 Install plugin from disk
      
   3. On the Choose Plugin File page, select the Scala plugin file of the corresponding version and click OK.

      

   4. On the Plugins page, click Apply to install the Scala plugin.
   5. On the displayed Plugins Changed page, click Restart for the configuration to take effect.
      Figure 10 Plugins Changed
      

4. Import the Java sample projects to IntelliJ IDEA.
   1. Start IntelliJ IDEA. On the Quick Start page, select Import Project.
      Alternatively, for the used IDEA tool, add the project directly from the IDEA home page. Select File > Import project... to import projects.

      Figure 11 Importing a project (on the Quick Start page)
      
   2. Select the path and folder for the to-be-imported project and click OK.
      Figure 12 Select File or Directory to Import
      
   3. Confirm the location and name of the project and click Next.
      Figure 13 Import Project from Maven
      
   4. Select the project to be imported and click Next.
   5. Confirm the project JDK and click Next.
      Figure 14 Select project SDK
      
   6. Confirm the project name and project file location, and click Finish to complete the import.
      Figure 15 Confirm the project name and file location
      
   7. After the import, the imported projects are displayed on the IDEA home page.
      Figure 16 Imported projects
      

5. (Optional) If a sample project developed in Scala is imported, configure the language for the project.
   1. On the main page of the IDEA, choose File > Project Structures... to access the Project Structure page.
   2. Choose Modules, right-click the project name, and choose Add > Scala.
      Figure 17 Selecting the Scala language
      
   3. Wait until IDEA identifies Scala SDK, select the dependency JAR packages in the Add Scala Support dialog box, and click OK.
      Figure 18 Add Scala Support
      
   4. If IDEA fails to identify Scala SDK, create a Scala SDK.
      1. Click Create....
         Figure 19 Create...
         
      2. On the Select JAR's for the new Scala SDK page, click Browse....
         Figure 20 Select JAR's for the new Scala SDK
         
      3. On the Scala SDK files page, select the scala sdk directory, and click OK.
         Figure 21 Scala SDK files
         
   5. Click OK.
      Figure 22 Successful configuration
      

6. Set the file encoding format of IntelliJ IDEA to avoid garbled characters.
   1. On the IDEA homepage, choose File > Settings....
      Figure 23 Choosing Settings
      
   2. Configure encoding.
      1. On the Settings page, choose Editor > File Encodings.
      2. In the Global Encoding and Project Encoding drop-down lists, select UTF-8, respectively.
      3. Click Apply.
      4. Click OK.

      

Sample Code Path Description