How Do I Handle a Large Number of Residual Large Files in the /tmp Directory on the Node Where the Driver Is Located?

Question

How do I solve the problem that after some PySpark tasks are executed, a large number of residual large files exist in the /tmp directory on the Driver node, occupying too much disk storage space? The following is an example of the residual directories:

/tmp/prophet_modelf28943ij
/tmp/4xgpw7sc.json
/tmp/9zsivwu3.json

Answer

When a PySpark task is submitted, if the prophet or fbprophet library (or other third-party libraries) is imported to the code, the library creates temporary directories and files in the working directory (generally the /tmp directory by default) during model training or prediction. If these temporary files are not deleted in a timely manner after the task is complete, they will consume disk space heavily.

Solution

1. Stop the abnormal task and release the occupied resources.
2. Before executing the task, specify the environment variables in the service code to store the cached data of the third-party libraries to other directories instead of the default /tmp directory.

   For example, you can set the PROPHET_CACHE_DIR environment variable as follows:

   import os
   os.environ['PROPHET_CACHE_DIR'] = '/path/to/alternative/cache/directory'

   In this way, cached files can be stored in the specified directory, preventing the /tmp directory from being occupied.