Spark Streaming Performance Tuning

Updated on 2024-10-09 GMT+08:00

View PDF

Scenario

Streaming is a mini-batch streaming processing framework that features second-level delay and high throughput. To optimize Streaming is to improve its throughput while maintaining second-level delay so that more data can be processed per unit time.

NOTE:

This section applies to the scenario where the input data source is Kafka.

Procedure

A simple streaming processing system consists of a data source, a receiver, and a processor. The data source is Kafka, the receiver is the Kafka data source receiver of Streaming, and the processor is Streaming.

Enhance the performance of the three components for Streaming tuning.

Data source optimization
In actual application scenarios, the data source stores the data in the local disks to ensure the error tolerance of the data. However, the calculation results of the Streaming are stored in the memory, and the data source may become the largest bottleneck of the streaming system.

Kafka can be optimized from the following aspects:
- Use Kafka-0.8.2 or later version that allows you to use new Producer APIs in asynchronous mode.
- Configure multiple Broker directories, multiple I/O threads, and a proper number of partitions for a topic.
For details, see section Performance Tuning in the Kafka open source documentation at http://kafka.apache.org/documentation.html.
Receiver optimization
Streaming has multiple data source receivers, such as Kafka, Flume, MQTT, and ZeroMQ. Kafka has the most receiver types and is the most mature receiver.

Kafka provides three types of receiver APIs:
- KafkaReceiver directly receives Kafka data. If the process is abnormal, data may be lost.
- ReliableKafkaReceiver receives data displacement through ZooKeeper records.
- DirectKafka reads data from each partition of Kafka through the RDD, ensuring high reliability.
According to the implementation mechanism and test results, DirectKafka provides better performance than the other two APIs. Therefore, the DirectKafka API is recommended to implement the receiver.

Kafka receivers function as Kafka consumers. For details about how to optimize them, see the Kafka open source documentation at http://kafka.apache.org/documentation.html.
Processor optimization
The bottom layer of Spark Streaming is executed by Spark. Therefore, most optimization measures for Spark can also be applied to Spark Streaming. The following is an example:
- Data serialization
- Memory configuration
- Configuring DOP
- Using the external shuffle service to improve performance
NOTE:

Higher performance of Spark Streaming indicates lower overall reliability. Examples:

If spark.streaming.receiver.writeAheadLog.enable is set to false, disk I/Os are reduced and performance is improved. However, because WAL is disabled, data is lost during fault recovery.

Therefore, do not disable configuration items that ensure data reliability in production environments during Spark Streaming tuning.
Log archive optimization
The spark.eventLog.group.size parameter is used to group JobHistory logs of an application based on the specified number of jobs. Each group creates a file recording log to prevent JobHistory reading failures caused by an oversized log generated during the long-term running of the application. If this parameter is set to 0, logs are not grouped.

Most Spark Streaming jobs are small jobs and are generated at a high speed. As a result, frequent grouping is performed and a large number of small log files are generated, consuming disk I/O resources. You are advised to increase the parameter value to, for example, 1000 or greater.

Parent topic: Using Spark/Spark2x

Previous topic: SQL Optimization for Multi-level Nesting and Hybrid Join

Next topic: Spark on OBS Performance Tuning

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot