Development Process
Development Process of a Spark Application
Spark includes Spark Core, Spark SQL and Spark Streaming, whose development processes are the same.
All stages of the development process are shown and described in Figure 1 and Table 1.
|
Stage |
Description |
Reference |
|---|---|---|
|
Understand the basic concepts. |
Before the application development, the basic concepts of Spark are required to be understood. Choose the concepts required to be understood based on the actual scenario. The basic concepts include the basic concept of Spark Core, basic concept of Spark SQL and basic concept of Spark Streaming. |
|
|
Prepare the development and operating environment. |
The Spark application is developed in Scala, Java, and Python. The IDEA tool is recommended to prepare development environments in different languages based on the reference. The running environment of Spark is the Spark client. Install and configure the client based on the reference. |
|
|
Developer account preparation |
Only the developer user with permission to access HDFS, YARN, Kafka, and Hive is allowed to run Spark sample projects. |
|
|
Prepare projects. |
Spark provides sample projects in various scenarios. Sample projects can be imported for studying. Or you can create a new Spark project based on the reference. |
|
|
Prepare for safety authentication. |
If a safe cluster is used, the safety authentication must be performed. |
|
|
Develop projects based on scenarios. |
Sample projects in different languages including Scala, Java, and Python are provided. Sample projects in different scenarios including Streaming, SQL, JDBC client program, and Spark on HBase are also provided. This helps users to better understand the programming interfaces of all Spark components quickly. |
|
|
Compile and run the application. |
Users compile the developed application and deliver it for running based on the reference. |
|
|
Check the application running results. |
Application running results are stored in the directory specified by users. Users can also check the running results through the UI. |
|
|
Tune the application. |
Based on the application running results, tun the application to meet the requirements of the service scenario. After application tuning, compile and run the application again. |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.
