Near Data Processing (NDP) is a computing pushdown solution to improve data query efficiency. For data-intensive queries, operations such as column extraction, aggregation calculation, and condition filtering are pushed down to multiple nodes on a distributed storage layer for parallel execution. This reduces query processing pressure on compute nodes, improves parallel processing capabilities, and saves network traffic.
GaussDB(for MySQL) uses an architecture with decoupled storage and compute to reduce network traffic. Based on this architecture, NDP is used to accelerate data queries. Without NDP, all raw data needs to be transmitted from storage nodes to compute nodes for query processing. NDP pushed the most I/O-intensive and CPU-intensive query tasks down to storage nodes. Only the required columns and filtered rows or aggregated results are sent back to compute nodes, greatly reducing network traffic. Additionally, parallel processing across storage nodes reduces the CPU usage of compute nodes and improves the query efficiency.
NDP is integrated with parallel query. Batch pages are prefetched to realize the entire process in parallel. The query execution efficiency is greatly improved.
In terms of functions, NDP is suitable for the following scenarios:
- SELECT - WHERE clause for filtering
A filter expression is executed on the storage nodes. Only the rows that meet the conditions are sent to the compute node.
- InnoDB tables.
- Tables with rows in the COMPACT or DYNAMIC format.
- Primary keys or B-tree indexes. Hash and full-text indexes are not supported.
- SELECT statements among the DML statements. INSERT INTO SELECT statements and SELECT statements that will lock rows (such as SELECT FOR SHARE/UPDATE) are not supported.
- Expressions with numeric, log, time, or partial string types (CHAR and VARCHAR). The utf8mb4 and utf8 character sets are supported.
- Expression predicates with comparison operators (<,>,=,<=,>=,!=), IN, NOT IN, LIKE, NOT LIKE, BETWEEN AND, and AND/OR.
Value: off or on
Default value: off