How to Use Cache Table?

Question

What is cache table used for? Which point should I pay attention to while using cache table?

Answer

Spark SQL can cache tables to the memory and store the cached tables in a compressed format, minimizing the memory usage. Once a table is cached, data can be directly read from the memory, reducing the I/O overhead.

Note that cached tables consume Executor's memory. This means that caching large or many tables compromises Executor's stability even if compressed storage has been used to reduce memory overhead as much as possible.

If it is no longer necessary to accelerate data query by means of cache table, run the following command to uncache tables to free up memory: You can run the uncache table table_name command to uncache tables.

The Storage tab page of the Spark Driver user interface displays the cached tables.