What is Operator Spilling in GaussDB(DWS)?

Operator Spilling to Disks

Any computing consumes memory space. If too much memory is consumed, the memory for running other jobs will be insufficient, causing unstable job running. Therefore, you need to limit the memory usage of query statements to ensure job running stability.

If a job running requires 500 MB memory but only 300 MB memory is allocated to it, data that is not used temporarily needs to be written to disks, and only data that is being used is retained in the memory. This is called data spilling. It is also called operator spilling. Large size of data spilled to disk may cause 100% disk usage. If it happens, the database will turn to read-only and query performance will be greatly affected. Therefore, GaussDB(DWS) imposes a limit on operator spilling. If the operator spilling exceeds the limit, an error is reported and the query exits.

Which operators can spill?

Operators that can spill to disk include Hash(VecHashJoin), Agg(VecAgg), Sort(VecSort), Material(VecMaterial), SetOp(VecSetOp), and WindowAgg(VecWindowAgg). They can be vectorized or non-vectorized.

Which parameters can be used to control the operator spilling?

• work_mem: sets spilling threshold. Disk usage exceeding this parameter will cause operator spilling. This parameter takes effect only when the memory is not self-adaptive. (enable_dynamic_workload=off). It ensures concurrent throughput and the performance of a single query job. Therefore, you need to optimize the parameter based on the output of Explain Performance.
• temp_file_limit: limits the size of files spilled to disks. You are advised to set this parameter based on the site requirements to prevent spilled files from using up the disk space. Spilled files exceeding the value of this parameter will cause an error.

How Do I Know Whether a Statement Is Spilled to Disks?

• Check the spill files. The spill files are stored in the base/pgsql_tmp directory of the instance directory. The spill files are named pgsql_tmp$queryid_$pid. You can determine which SQL statement is spilled to the disk based on the queryid.
• Check the waiting view (pgxc_thread_wait_status). If write file is displayed in the waiting view, there are temporary results spilled to disks.
• Check the execution plan (EXPLAIN PERFORMANCE). Keywords such as spill, written disk, and temp file num indicates that there is an operator spilling.
• Check whether the spill_info column in Real-time TopSQL or Historical TopSQL contains spill information. If this column is not empty, data has been spilled to disks on DNs. (Prerequisite: The topsql function has been enabled.)

How Do I Avoid Spilling?

When operators are spilled to disks, operator calculation data is written to disks. Compared with memory access, disk operations are slow, causing performance deterioration and query response time deterioration. Therefore, try to avoid operator spilling during query execution. You are advised to use the following methods:

• Reduce the intermediate result set: If the intermediate result set is too large, you can add filter criteria to reduce the size of the intermediate result set.
• Avoid data skew: If there is a severe data skew, spilling will occur on a DN that has a large amount of data.
• Perform ANALYZE in a timely manner: When the statistics are inaccurate, the number of rows may be estimated to be smaller. As a result, the plan is not optimal and data is spilled to disks.
• Perform single-point optimization: Perform optimization on single SQL statements.
• If the memory is not self-adaptive, and the intermediate result set cannot be reduced, increase the value of work_mem as proper.
• If the memory is self-adaptive, increase the available memory of the database as much as possible to reduce the probability of data spilling.