System Rules

Updated on 2024-08-20 GMT+08:00

View PDF

Table 1 lists the system inspection rules supported by DLI.

Default system rules are SQL inspection rules automatically created by DLI when a queue is created. These rules are bound to the queue and cannot be deleted.

Default system rules include Scan files number, Scan partitions number, Count(distinct) occurrences, and Not in<Subquery>.

Only one inspection rule can be created for an action or a queue.

For every supported action, a default system rule is created. For example, when a queue is created, a Scan files number rule is automatically created for both the Info and Block actions.

**Table 1** System inspection rules supported by DLI
Rule ID	Rule Name	Description	Type	Applicable Engine	Action	Value	Default System Rule	Example SQL Statement
dynamic_0001	Scan files number	Maximum number of files to be scanned	Dynamic	Spark trino	Info Block	Value range: 1–2000000 Default value: 200000	Yes	N/A
dynamic_0002	Scan partitions number	Maximum number of partitions involved in the operations (select, delete, update, and alter) that can be performed on a table	Dynamic	Spark	Info Block	Value range: 1–500000 Default value: 5000	Yes	DELETE FROM table_name WHERE column_name = value
running_0002	Memory used(MB)	Peak memory usage of the SQL statement	Running	Spark	Circuit Breaker	Value range: 1–8388608	No	N/A
running_0003	Run time(S)	Maximum running duration of the SQL statement	Running	Spark	Circuit Breaker	Unit: second Value range: 1–43200	No	N/A
running_0004	Scan data(GB)	Maximum amount of data to be scanned	Running	Spark	Circuit Breaker	Unit: GB Value range: 1–10240	No	N/A
static_0001	Count(distinct) occurrences	Maximum number of occurrences of count(distinct) in the SQL statement	Static	Spark	Info Block	Value range: 1–100 Default value: 10	Yes	SELECT COUNT(DISTINCT deviceId), COUNT(DISTINCT collDeviceId) FROM table GROUP BY deviceName, collDeviceName, collCurrentVersion;
static_0002	Not in<Subquery>	Check whether not in <subquery> is used in the SQL statement.	Static	Spark	Info Block	Value range: Yes, No Default value: Yes	Yes	SELECT * FROM Orders o WHERE Orders.Order_ID not in (Select Order_ID FROM HeldOrders h where h.order_id = o.order_id);
static_0003	Join occurrences	Maximum number of joins in the SQL statement	Static	Spark	Info Block	Value range: 1–50	No	SELECT name, text FROM table_1 JOIN table_2 ON table_1.Id = table_2.Id
static_0004	Union occurrences	Maximum number of union all times in the SQL statement	Static	Spark	Info Block	Value range: 1–100	No	select * from tables t1 union all select * from tables t2 union all select * from tables t3
static_0005	Subquery nesting layers	Maximum number of subquery nesting layers	Static	Spark	Info Block	Value range: 1–20	No	select * from ( with temp1 as (select * from tables) select * from temp1);
static_0006	Sql size(KB)	Maximum size of a SQL file	Static	Spark	Info Block	Unit: KB Value range: 1–1024	No	N/A
static_0007	Cartesian product	Limitation of Cartesian products when multiple tables are being associated	Static	Spark	Info Block	Value range: 0–1	No	select * from A,B;