Updated on 2022-09-14 GMT+08:00

Development Process

All stages of the development process are shown and described in Figure 1 and  Table 1.

Figure 1 MapReduce development process
Table 1 Description of MapReduce development process

Stage

Description

Reference

Understand the basic concepts.

Before the application development, the basic concepts of MapReduce are required to be understood.

Basic Concepts

Prepare the development and operating environment.

Use IntelliJ IDEA and configure the development environment based on the reference.

The running environment of MapReduce is the MapReduce client. Install and configure the client based on the reference.

Preparing for Development and Operating Environment

Prepare projects

MapReduce provides sample projects in various scenarios. Sample projects can be imported for studying. Or you can create a new MapReduce project based on the reference.

Configuring and Importing Sample Projects

Creating a New Project (Optional)

Prepare for safety certification.

If a safe cluster is used, the safety certification must be performed.

Preparing the Authentication Mechanism

Develop projects based on scenarios.

Provide the example project.

This helps users to learn about the programming interfaces of all Spark components quickly.

Developing the Project

Compile and run the application.

Users compile the developed application and deliver it for running based on the reference.

Commissioning the Application

Check the application running results.

Application running results are stored in the directory specified by users. Users can also check the running results through the UI.

Commissioning the Application