Insufficient Disk Space on the JDBCServer Node

Question

After a YARN task initiated by JDBCServer runs for an extended period of time, disk space on certain nodes becomes exhausted. This leads to insufficient disk capacity, causing the JDBCServer instance on the affected node to behave abnormally.

Solution

• Versions earlier than MRS 3.1.2-LTS:

  Log in to FusionInsight Manager, choose Cluster > Services > Spark, click Instances, and restart the JDBCServer instance to clear the disk.

• MRS 3.1.2-LTS or later:

  Log in to FusionInsight Manager, choose Cluster > Services > Spark, and click Configurations and then All Configurations. Choose JDBCServer(Role) > Customization, add the spark.shuffle.service.removeShuffle parameter in the custom area, set the parameter value to true, save the settings, and restart the instance.

  Figure 1 Adding the parameter
  

  Table 1 Parameter description

  Parameter	Description	Example Value
  spark.shuffle.service.removeShuffle	Whether to use the ExternalShuffleService for deleting shuffle blocks for deallocated executors when the shuffle is no longer needed. Without this enabled, shuffle data on executors that are deallocated will remain on disk until the application ends. The default value is false, indicating that this function is disabled.	true