Esta página ainda não está disponível no idioma selecionado. Estamos trabalhando para adicionar mais opções de idiomas. Agradecemos sua compreensão.

Compute
Elastic Cloud Server
Huawei Cloud Flexus
Bare Metal Server
Auto Scaling
Image Management Service
Dedicated Host
FunctionGraph
Cloud Phone Host
Huawei Cloud EulerOS
Networking
Virtual Private Cloud
Elastic IP
Elastic Load Balance
NAT Gateway
Direct Connect
Virtual Private Network
VPC Endpoint
Cloud Connect
Enterprise Router
Enterprise Switch
Global Accelerator
Management & Governance
Cloud Eye
Identity and Access Management
Cloud Trace Service
Resource Formation Service
Tag Management Service
Log Tank Service
Config
OneAccess
Resource Access Manager
Simple Message Notification
Application Performance Management
Application Operations Management
Organizations
Optimization Advisor
IAM Identity Center
Cloud Operations Center
Resource Governance Center
Migration
Server Migration Service
Object Storage Migration Service
Cloud Data Migration
Migration Center
Cloud Ecosystem
KooGallery
Partner Center
User Support
My Account
Billing Center
Cost Center
Resource Center
Enterprise Management
Service Tickets
HUAWEI CLOUD (International) FAQs
ICP Filing
Support Plans
My Credentials
Customer Operation Capabilities
Partner Support Plans
Professional Services
Analytics
MapReduce Service
Data Lake Insight
CloudTable Service
Cloud Search Service
Data Lake Visualization
Data Ingestion Service
GaussDB(DWS)
DataArts Studio
Data Lake Factory
DataArts Lake Formation
IoT
IoT Device Access
Others
Product Pricing Details
System Permissions
Console Quick Start
Common FAQs
Instructions for Associating with a HUAWEI CLOUD Partner
Message Center
Security & Compliance
Security Technologies and Applications
Web Application Firewall
Host Security Service
Cloud Firewall
SecMaster
Anti-DDoS Service
Data Encryption Workshop
Database Security Service
Cloud Bastion Host
Data Security Center
Cloud Certificate Manager
Edge Security
Situation Awareness
Managed Threat Detection
Blockchain
Blockchain Service
Web3 Node Engine Service
Media Services
Media Processing Center
Video On Demand
Live
SparkRTC
MetaStudio
Storage
Object Storage Service
Elastic Volume Service
Cloud Backup and Recovery
Storage Disaster Recovery Service
Scalable File Service Turbo
Scalable File Service
Volume Backup Service
Cloud Server Backup Service
Data Express Service
Dedicated Distributed Storage Service
Containers
Cloud Container Engine
Software Repository for Container
Application Service Mesh
Ubiquitous Cloud Native Service
Cloud Container Instance
Databases
Relational Database Service
Document Database Service
Data Admin Service
Data Replication Service
GeminiDB
GaussDB
Distributed Database Middleware
Database and Application Migration UGO
TaurusDB
Middleware
Distributed Cache Service
API Gateway
Distributed Message Service for Kafka
Distributed Message Service for RabbitMQ
Distributed Message Service for RocketMQ
Cloud Service Engine
Multi-Site High Availability Service
EventGrid
Dedicated Cloud
Dedicated Computing Cluster
Business Applications
Workspace
ROMA Connect
Message & SMS
Domain Name Service
Edge Data Center Management
Meeting
AI
Face Recognition Service
Graph Engine Service
Content Moderation
Image Recognition
Optical Character Recognition
ModelArts
ImageSearch
Conversational Bot Service
Speech Interaction Service
Huawei HiLens
Video Intelligent Analysis Service
Developer Tools
SDK Developer Guide
API Request Signing Guide
Terraform
Koo Command Line Interface
Content Delivery & Edge Computing
Content Delivery Network
Intelligent EdgeFabric
CloudPond
Intelligent EdgeCloud
Solutions
SAP Cloud
High Performance Computing
Developer Services
ServiceStage
CodeArts
CodeArts PerfTest
CodeArts Req
CodeArts Pipeline
CodeArts Build
CodeArts Deploy
CodeArts Artifact
CodeArts TestPlan
CodeArts Check
CodeArts Repo
Cloud Application Engine
MacroVerse aPaaS
KooMessage
KooPhone
KooDrive

Configuring Large Query Isolation for an Elasticsearch Cluster

Updated on 2024-10-26 GMT+08:00

Scenario

You can isolate query requests that consume a large amount of memory or take a long period of time. This way, you ensure service availability for other requests. If the heap memory usage of a node is too high, an interrupt control program will be triggered to terminate a large query based on the policies you configured. You can also configure a global query timeout duration. Long queries will be intercepted.

Large query isolation can effectively solve the following problems and improve the search performance of clusters:
  • A small number of queries occupy large chunks of node heap memory, resulting in frequent Garbage Collection (GC) and even out of memory (OOM) exceptions.
  • Frequent GC causes node disconnections. As a result, queries cannot get response and may fail.
  • The CPU usage is high due to heavy query load, affecting online services.

Constraints

Only Elasticsearch 7.6.2 and Elasticsearch 7.10.2 clusters support large query isolation.

Configuring Large Query Isolation

The large query isolation and global timeout features are disabled by default. If you enable them, the configuration will take effect immediately.

  1. Log in to the CSS management console.
  2. Choose Clusters in the navigation pane. On the Clusters page, locate the target cluster, and click Access Kibana in the Operation column.
  3. In the navigation pane of Kibana on the left, choose Dev Tools. Run the following command to enable large query isolation and global timeout features:
    PUT _cluster/settings
    {
      "persistent": {
        "search.isolator.enabled": true,
        "search.isolator.time.enabled": true
      }
    }

    The two features each has an independent switch. Table 1 describes their parameters.

    Table 1 Parameters for large query isolation and global timeout duration

    Switch

    Parameter

    Description

    search.isolator.enabled

    search.isolator.memory.task.limit

    search.isolator.time.management

    Thresholds of a shard query task. A query task exceeding one of these thresholds is identified as a large query.

    search.isolator.memory.pool.limit

    search.isolator.memory.heap.limit

    search.isolator.count.limit

    Resource usage thresholds for isolation. If the resource usage of a query task exceeds one of these thresholds, the task will be interrupted.

    NOTE:

    search.isolator.memory.heap.limit defines the limit on the heap memory consumed by write, query, and other operations of a node. If this limit is exceeded, large query tasks in the isolation pool will be interrupted.

    search.isolator.strategy

    search.isolator.strategy.ratio

    Policy for selecting query tasks to pause in the isolation pool.

    search.isolator.time.enabled

    search.isolator.time.limit

    Global timeout interval of query tasks.

  4. Configure large query isolation.
    • Run the following commands to configure thresholds for defining a large query task:
      PUT _cluster/settings
      {
        "persistent": {
          "search.isolator.memory.task.limit": "50MB",
          "search.isolator.time.management": "10s"
        }
      }
      Table 2 Parameter description

      Parameter

      Data Type

      Description

      search.isolator.memory.task.limit

      String

      Threshold of the memory requested by a query task to perform aggregation or other operations. If the requested memory exceeds the threshold, the task will be isolated and observed.

      • Value range: 0b to the maximum heap memory of a node
      • Default value: 50MB
      NOTE:

      You can run the following command to query the current heap memory and the maximum heap memory of a cluster:

      GET _cat/nodes?&h=id,ip,port,r,ramPercent,ramCurrent,heapMax,heapCurrent

      search.isolator.time.management

      String

      Threshold of the duration of a query (started when cluster resources are used for query). If the duration of a query exceeds the threshold, it will be isolated and observed.

      • Value range: ≥ 0ms
      • Default value: 10s
    • Configure the resource usage thresholds for triggering the isolation of query tasks.
      PUT _cluster/settings
      {
        "persistent": {
          "search.isolator.memory.pool.limit": "50%",
          "search.isolator.memory.heap.limit": "90%",
          "search.isolator.count.limit": 1000
        }
      }
      Table 3 Parameter description

      Parameter

      Data Type

      Description

      search.isolator.memory.pool.limit

      String

      Threshold of the heap memory percentage of the current node. If the total memory requested by large query tasks in the isolation pool exceeds the threshold, the interrupt control program will be triggered to cancel one of the tasks.

      • Value range: 0.0 to 100.0%
      • Default value: 50%

      search.isolator.memory.heap.limit

      String

      Heap memory threshold of the current node. If the heap memory of the node exceeds the threshold, the interrupt control program will be triggered to cancel a large query task in the isolation pool.

      • Value range: 0.0 to 100.0%
      • Default value: 90%

      search.isolator.count.limit

      Integer

      Threshold of the number of large query tasks in the current node isolation pool. If the number of observed query tasks exceeds the threshold, the interrupt control program will be triggered to stop accepting new large queries. New large query requests will be directly canceled.

      • Value range: 1050000
      • Default value: 1000
      NOTE:

      In addition to search.isolator.memory.pool.limit and search.isolator.count.limit parameters, you can configure search.isolator.memory.task.limit and search.isolator.time.management to control the number of query tasks that enter the isolation pool.

    • Configure a policy for selecting query tasks to pause in the isolation pool.
      PUT _cluster/settings
      {
        "persistent": {
          "search.isolator.strategy": "fair",
          "search.isolator.strategy.ratio": "0.5%"
        }
      }

      Parameter

      Data Type

      Description

      search.isolator.strategy

      String

      Policy for selecting large queries when the interrupt control program is triggered. The selected queries will be interrupted.

      NOTE:

      The large query isolation pool is checked every second until the heap memory is within the safe range.

      Values: fair, mem-first, or time-first
      • mem-first: The query task that uses the most heap memory in the isolation pool is interrupted.
      • time-first: The query task that has been running for the longest time in the isolation pool is interrupted.
      • fair: If the difference between the heap memory of shard queries is smaller than Maximum_heap_memory x search.isolator.strategy.ratio, the query that takes the longest time should be interrupted. Otherwise, the query that uses the most heap memory is interrupted.

      Default value: fair

      search.isolator.strategy.ratio

      String

      Threshold of the fair policy. This parameter takes effect only if search.isolator.strategy is set to fair. If the difference between the memory usage of large query tasks does not exceed the threshold, the query that takes the longest time should be interrupted. If the difference between the memory usage of large query tasks exceeds the threshold, the query that uses the most memory is interrupted.

      • Value range: 0.0 to 100.0%
      • Default value: 1%
  5. Configure the global timeout duration of query tasks.
    Run the following command to set the global timeout of query tasks:
    PUT _cluster/settings
    {
      "persistent": {
        "search.isolator.time.limit": "120s"
      }
    }

    Parameter

    Data Type

    Description

    search.isolator.time.limit

    String

    Global query timeout duration. Any query task that exceeds this duration will be canceled.

    • Value range: ≥ 0ms
    • Default value: 120s

Usamos cookies para aprimorar nosso site e sua experiência. Ao continuar a navegar em nosso site, você aceita nossa política de cookies. Saiba mais

Feedback

Feedback

Feedback

0/500

Selected Content

Submit selected content with the feedback