Configuring Local Disk Cache for JobHistory
Scenario
JobHistory can use local disks to cache historical data of Spark applications to prevent large volumes of application data from being loaded to the JobHistory memory and reduce memory usage. In addition, the cached data can be reused to accelerate access to the same application.
Parameters
Log in to FusionInsight Manager and choose Cluster > Services > Spark. Click Configurations then All Configurations, and search for the following parameters:
|
Parameter |
Description |
Default Value |
|---|---|---|
|
spark.history.store.path |
Local directory for JobHistory to cache historical data. If this parameter is configured, JobHistory caches historical application data in local disks instead of the memory. |
${BIGDATA_HOME}/tmp/spark_JobHistory |
|
spark.history.store.maxDiskUsage |
Maximum available space for JobHistory to caching data in local disks |
10g |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot
