What Are the Advantages of the New Pipe-Character-Based SQL Engine?

Features and Advantages

Features:

Combined search and SQL analysis: The new engine allows you to combine a search statement and a SQL analysis statement using the pipe character (|).

For example, to count logs that contain the keyword error, you can use the following syntax:

error* | select count(*) as cnt

Advantages:

• Better performance: The new SQL engine dramatically improves calculation efficiency, especially with a high volume of raw logs.

  The old engine was inefficient, as it performed a full fuzzy match on all logs. For example, to count logs that contain the keyword error, you can use the following syntax:

  select count(*) as cnt where status like "%error%"

  In contrast, the new engine uses an inverted index to quickly filter logs with the keyword error before counting, making it significantly faster. The syntax is:

  status : "error" | select count(*) as pv 

  Using the pipe character (|) to search for a specific keyword like error is related to the delimiters set in your index settings. The new engine can only find the keywords that are fully retained after being delimited by these delimiters. For more rules, see Using Search Syntax.

• More SQL functions: The new SQL engine supports over 300 SQL functions, compared to just over 100 in the old engine.
• More standard SQL syntax: The new engine's SQL syntax is fully compatible with SQL-92 standards, making it more standard and easier to understand.

LTS Console Entries

The entries for the old and new engines on the LTS console have been updated for different scenarios.