Configuring Local Disk Cache for JobHistory

Scenario

JobHistory can use local disks to cache historical data of Spark applications to prevent large volumes of application data from being loaded to the JobHistory memory and reduce memory usage. In addition, the cached data can be reused to accelerate access to the same application.

Configuring Parameters

1. Log in to FusionInsight Manager.

   For details, see Accessing FusionInsight Manager.

2. Choose Cluster > Services > Spark2x or Spark > Configurations, click All Configurations, and search for the following parameters and adjust their values:

   Parameter	Description	Example Value
   spark.history.store.path	Local directory for JobHistory to cache historical data. If this parameter is configured, JobHistory caches historical application data in local disks instead of the memory.	${BIGDATA_HOME}/tmp/spark2x_JobHistory
   spark.history.store.maxDiskUsage	Maximum available space for JobHistory to caching data in local disks	10 GB

3. After the parameter settings are modified, click Save, perform operations as prompted, and wait until the settings are saved successfully.
4. After the Spark server configurations are updated, if Configure Status is Expired, restart the component for the configurations to take effect.
   Figure 1 Modifying Spark configurations
   
   On the Spark dashboard page, choose More > Restart Service or Service Rolling Restart, enter the administrator password, and wait until the service restarts.

   

   Components are unavailable during the restart, affecting upper-layer services in the cluster. To minimize the impact, perform this operation during off-peak hours or after confirming that the operation does not have adverse impact.