Configuring Enhanced Aggregation for an Elasticsearch Cluster

Scenario

In the case of data clustering, enhanced aggregation uses the vectorization technology to process data in batches, improving aggregation performance and facilitating aggregated analysis for faster time to insight.

In large-scale dataset aggregation and analysis scenarios, a major chunk of the time is spent on data grouping and aggregation.

Enhancing data grouping and aggregation relies on two things:

• Sorting key: Data is sorted based on the sorting key.
• Cluster key: It is contained in the sorting key. Data is clustered based on the clustering key.

Table 1 describes common scenarios that involve enhanced aggregation.

Constraints

Only Elasticsearch 7.10.2 clusters support aggregation enhancement.

Grouping and Aggregation of Low-cardinality Fields

Generally, low-cardinality fields use grouping as a way of aggregation. With an appropriate sorting key, grouping prepares the data for batch vectorization.

For example, the query statement is as follows:

POST testindex/_search
{
  "size": 0,
  "aggs": {
    "groupby_region": {
      "terms": {
        "field": "region"
      },
      "aggs": {
        "groupby_host": {
          "terms": {
            "field": "host"
          },
          "aggs": {
            "avg_cpu_usage": {
              "avg": {
                "field": "cpu_usage"
              }
            }
          }
        }
      }
    }
  }
}

Assume that region and host are low-cardinality fields. To use enhanced aggregation, set parameters as follows:

//Configure an index
"settings" : {
    "index" : {
        "search" : {
            "turbo" : {
                "enabled" : "true" //Enable optimization
            }
        },
        "sort" : { //Specify a sorting key
            "field" : [
                "region",
                "host",
                "other"
            ]
        },
        "cluster" : {
            "field" : [ //Specify a clustering key
                "region",
                "host"
            ]
        }
    }
}

The clustering key must be a subset of the sorting key.

High-cardinality Field Histogram Aggregation

High-cardinality fields commonly use histogram aggregation, which facilitates data processing per range.

For example, the query statement is as follows: This query groups the field timestamp using a histogram and calculates the average score.

POST testindex/_search?pretty
{
  "size": 0,
  "aggs": {
    "avg_score": {
      "avg": {
        "field": "score"
      },
      "aggs": {
        "groupbytime": {
          "date_histogram": {
            "field": "timestamp",
            "calendar_interval": "day"
          }
        }
      }
    }
  }
}

timestamp is a typical high-cardinality field. To apply enhanced aggregation to such a field, set parameters as follows:

//Configure an index
"settings" : {
    "index" : {
        "search" : {
            "turbo" : {
                "enabled" : "true" //Enable optimization
            }
        },
        "sort" : { //Specify a sorting key
            "field" : [
                "timestamp"
            ]
        }
    }
}

Low-cardinality and High-cardinality Field Mixing

Where low-cardinality and high-cardinality fields are mixed, first groups and aggregates low-cardinality fields, and then aggregates high-cardinality fields using histograms.

For example, the query statement is as follows:

POST testindex/_search
{
  "size": 0,
  "aggs": {
    "groupby_region": {
      "terms": {
        "field": "region"
      },
      "aggs": {
        "groupby_host": {
          "terms": {
            "field": "host"
          },
          "aggs": {
            "groupby_timestamp": {
              "date_histogram": {
                "field": "timestamp",
                "interval": "day"
              },
              "aggs": {
                "avg_score": {
                  "avg": {
                    "field": "score"
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

To first group the low-cardinality field region, and then the low-cardinality field host, and then cluster the high-cardinality field timestamp using a histogram, set the parameters as follows:

//Configure an index
"settings" : {
    "index" : {
        "search" : {
            "turbo" : {
                "enabled" : "true" //Enable optimization
            }
        },
        "sort" : { //Specify a sorting key
            "field" : [
                "region",
                "host",
                "timestamp",
                "other"
            ]
        },
        "cluster" : {
            "field" : [ //Specify a clustering key
                "region",
                "host"
            ]
        }
    }
}

• The clustering key must be a subset of the sorting key.
• High-cardinality fields must be in the sorting key, and high-cardinality fields must follow the last low-cardinality field.

Performance Testing

Test Result

This test focuses on the query result of dropoff_datetime aggregation, that is, the results of tasks autohisto_agg and date_histogram_agg. The following table compares the test results between when enhanced aggregation is disabled and when it is enabled.

Test Conclusion

Given the same cluster configuration, aggregation performance improves significantly when enhanced aggregation is enabled. Query throughput improves by more than 2.5 times, and latency decreases by more than 60%.