Overview

What Is SQL Inspection?

There are numerous SQL engines in the current big data field, which bring diversity to the solutions but also expose some issues such as varying quality of SQL input statements, difficult SQL problem localization, and excessive resource consumption by large SQL statements.

Poor quality SQL can have unforeseeable impacts on data analysis platforms, affecting system performance or platform stability.

Only Spark 3.3 or later supports this function.

Functions

DLI offers this feature to allow you to create inspection rules for the Spark SQL engine. This helps prevent common issues like large or low-quality SQL statements by providing pre-event information, blocking, and in-event circuit breaker. You do not need to change your SQL submission method or syntax, so it is easy to implement without affecting service operations.

You can configure SQL inspection rules in a visualized manner and also have the ability to query and modify these rules.
During query execution and response, each SQL engine proactively inspects SQL statements based on the rules.
Administrators can choose to display hints on, intercept, or block SQL statements. The system logs SQL inspection events in real time for SQL audit. O&M engineers can analyze the logs, assess the quality of SQL statements on the live network, identify potential risks, and take preventive measures.

SQL inspection rules contain the following types of actions:

Info: The system provides hints or blocks SQL statements based on SQL syntax rules when a SQL request meets certain conditions.
Block: The system blocks SQL statements based on rules of data table statistics and metadata information. The system will cease processing a SQL request when certain conditions are met.
Circuit Breaker: The system performs circuit breaking on SQL statements based on system states (such as CPU, memory, and I/O) during the runtime of the SQL statements. The system will perform circuit breaking on a SQL task when certain conditions are met.