Configuring and Importing Sample Projects

Scenario

Flink provides sample projects for multiple scenarios, including Java and Scala sample projects, to help you quickly learn Flink projects.

Methods to import Java and Scala projects are the same.

The following example describes how to import Java sample code. Figure 1 shows the operation process.

Figure 1 Process of importing sample projects

Procedure

1. Obtain the sample project folder flink-examples-normal in the src\flink-examples directory where the sample code is decompressed. For details, see Obtaining the MRS Application Development Sample Project.
   

   • In security mode, obtain the sample project flink-examples-security from the src\flink-examples folder.
   • In normal mode, obtain the sample project flink-examples-normal from the src\flink-examples folder.

2. Before importing the sample project, configure JDK for IntelliJ IDEA.
   1. Start IntelliJ IDEA and click Configure.
      Figure 2 Choosing Configure
      
   2. Choose Project Defaults from the Configure drop-down list.
      Figure 3 Choosing Project Defaults
      
   3. Choose Project Structure from the Project Defaults submenu.
      Figure 4 Project Defaults
      
   4. On the Project Structure page, select SDKs and click the plus sign to add the JDK.
      Figure 5 Adding the JDK
      
   5. On the Select Home Directory for JDK page that is displayed, select the JDK directory and click OK.
      Figure 6 Selecting the JDK directory
      
   6. After the JDK is selected, click OK to complete the configuration.
      Figure 7 Completing the JDK configuration
      

3. (Optional) If a Scala sample project is imported, install Scala plug-ins in IntelliJ IDEA.
   1. Choose Plugins from the Configure drop-down list.
      Figure 8 Plugins
      
   2. On the Plugins page, click Install plugin from disk.
      Figure 9 Installing plugins from disk
      
   3. On the Choose Plugin File page, select the Scala plugin file of the corresponding version and click OK.
      Figure 10 Choose Plugin File
      
   4. On the Plugins page, click Apply to install the Scala plugins.
   5. On the Platform and Plugin Updates page that is displayed, click Restart to make the configurations take effect.
      Figure 11 Platform and Plugin Updates
      

4. Import the Java sample project to IDEA.
   1. Start IntelliJ IDEA. On the Quick Start page, select Import Project.

      Alternatively, for the used IDEA tool, add the project directly from the IDEA home page. Choose File > Import project... to import a project.

      Figure 12 Importing a project (on the Quick Start page)
      
   2. Select the directory for storing the imported projects and click OK.
      Figure 13 Select File or Directory to Import
      
   3. Select Create project from existing sources and click Next.
      Figure 14 Create project from existing sources
      
   4. Confirm the project location and project name, and click Next.
      Figure 15 Import Project
      
   5. Retain the default value of the root directory for the project to be imported and click Next.
      Figure 16 Import Project
      
   6. Retain the default dependency library that is automatically recognized by IDEA and the suggested module structure, and click Next.
   7. Confirm the JDK to be used by the project and click Next.
      Figure 17 Select project SDK
      
   8. After the import is complete, click Finish. The imported sample project is displayed on the IDEA home page.
      Figure 18 Completing the import
      
      Figure 19 Imported project
      

5. Import the dependency JAR file for the sample project.
   If the sample project code is obtained from an open-source image site, dependency JAR files are automatically downloaded after Maven is configured.

   

   If other FusionInsight components such as Kafka are used in the sample code, obtain them from the installation directory of these FusionInsight components. For details about dependency packages of sample projects, see Reference Information.

6. (Optional) If a Scala sample application is imported, configure a language for the project.
   1. On the IDEA home page, choose File > Project Structures... to go to the Project Structure page.
   2. Choose Modules, right-click a project name, and choose Add > Scala.
      Figure 20 Selecting Scala
      
   3. Wait until IDEA identifies Scala SDK, select the dependency JAR files in the Add Scala Support dialog box, and then click OK.
      Figure 21 Add Scala Support
      
   4. If IDEA fails to identify Scala SDK, create a Scala SDK.
      1. Click Create....
         Figure 22 Create...
         
      2. On the Select JAR's for the new Scala SDK page, click Browse....
         Figure 23 Select JAR's for the new Scala SDK
         
      3. On the Scala SDK files page, select the scala sdk directory and click OK.
         Figure 24 Scala SDK files
         
   5. Click OK.
      Figure 25 Configuration successful
      

7. Configure the text file encoding format of IDEA to prevent garbled characters.
   1. On the IDEA home page, choose File > Settings....
      Figure 26 Choosing Settings
      
   2. Configure encoding.
      1. On the Settings page, unfold Editor, and choose File Encodings.
      2. Select UTF-8 from the IDE Encoding and Project Encoding drop-down lists on the right.
      3. Click Apply.
      4. Click OK to complete the encoding settings.
      Figure 27 Settings
      
   

   • Obtain related dependency packages from the Flink server installation directory.
   • Obtain Kafka dependency packages from the Kafka environment.
   • For details about the dependency packages, see Reference Information.

8. Configure Maven.
   1. Add the configuration information such as the address of the open-source image repository to the setting.xml configuration file of the local Maven by referring to Configuring Huawei Open-Source Mirrors.
   2. On the IntelliJ IDEA page, select File > Settings > Build, Execution, Deployment > Build Tools > Maven, select Override next to User settings file, and change the value of User settings file to the directory where the settings.xml file is stored. Ensure that the directory is <Local Maven installation directory>\conf\settings.xml.
      Figure 28 Directory for storing the settings.xml file
      
   3. Click the drop-down list next to Maven home directory and select the Maven installation directory.
   4. Click Apply, and then click OK.
   5. On the right of the IntelliJ IDEA home page, click Maven Projects. On the Maven Projects page, choose Project name > Lifecycle and run the clean and compile scripts.
      Figure 29 Maven Projects page
      

Reference Information

The lib and opt directories of the Flink client contain Flink JAR files. By default, the lib directory contains the Flink core JAR file, and the opt directory contains the JAR file (for example, flink-connector-kafka*.jar) for connecting to external components. If it is required during application development, manually copy the related JAR files to the lib directory.

The dependency packages of the sample projects provided by Flink are as follows:

• Sample project of the DataStream application (in Java or Scala)
  • flink-dist_*.jar
    

    flink-dist_*.jar can be obtained from lib in the installation directory of the Flink client or server.

• Sample projects of the application for producing and consuming data in Kafka (in Java or Scala)
  • kafka-clients-*.jar
  • flink-connector-kafka_*.jar
  

  • flink-connector-kafka_*.jar can be obtained from opt in the installation directory of the Flink client or server.
  • kafka-clients-*.jar is provided by Kafka and can be obtained from the lib directory in the installation directory of the Kafka client or server.

• Sample project of the asynchronous checkpoint mechanism application (in Java or Scala)
  • flink-dist_*.jar
• Sample projects of pipeline (in Java or Scala)
  • flink-connector-netty_*.jar
  • flink-dist_*.jar
  • flink-shaded-curator-*.jar
  • curator-client-2.12.0.jar
  • curator-framework-2.12.0.jar
  

  • flink-shaded-curator-*.jar can be obtained from opt in the installation directory of the Flink client or server.
  • flink-connector-netty_*.jar, curator-client-2.12.0.jar, and curator-framework-2.12.0.jar can be obtained from the lib folder generated after the secondary development sample code is compiled.
  • flink-shaded-curator-*.jar applies only to MRS 3.0.X clusters.
  • curator-client-2.12.0.jar and curator-framework-2.12.0.jar apply only to MRS 3.1.X clusters.
• Sample projects of Stream SQL JOIN (in Java)
  • kafka-clients-*.jar
  • flink-connector-kafka_2.11*.jar
  • flink-connector-kafka-base_*.jar
  • flink-connector-kafka_*.jar
  • flink-dist_2.11*.jar
  • flink-table_2.11*.jar
  

  • flink-connector-kafka-base_*.jar and flink-connector-kafka_*.jar can be obtained from the opt directory in the installation directory of the Flink client or server.
  • flink-connector-kafka-base_*.jar and flink-connector-kafka_*.jar apply only to MRS 3.0.X clusters.
  • flink-connector-kafka_2.11*.jar, flink-dist_2.11*.jar, and flink-table_2.11*.jar apply only to MRS 3.1.X clusters.