Monitoring P99 Latency and HTTP Status Codes

How the Feature Works

P99 latency (99th percentile latency) indicates that 99% of search requests have a response time less than or equal to this value within a specified statistical period. Unlike average latency, P99 latency accurately reflects the actual impact of slow queries on overall service quality. Even if the average latency appears normal, a small number of high-latency requests may still cause noticeable delays for many users. When the difference between P99 latency and the median latency (P50) is large, it usually indicates the presence of severe slow queries or resource contention in the cluster, requiring immediate investigation.

HTTP status code monitoring collects statistics on the HTTP status codes returned by Elasticsearch APIs (such as 200, 201, 400, 404, and 429) to help you understand the service status of the cluster. For example, a sudden increase in client errors (4xx) may indicate that upstream applications are sending invalid requests, while server errors (5xx) may indicate that cluster nodes are in an abnormal state. By continuously tracking status code distributions, you can quickly identify application-side issues and take prompt actions.

Constraints

• Cluster version requirements: Only Elasticsearch 7.6.2 and 7.10.2 support P99 latency and HTTP status code monitoring. The commands for querying HTTP status codes are different between the two versions. Use the appropriate command based on your cluster version.
• Data reset: After a cluster restart, all P99 latency statistics (overall, last_one_day, and latest) are cleared.
• Approximate calculation: P99 latency is an approximate value rather than an exact measurement. The accuracy is higher for percentiles closer to the boundary (such as P99 and P1).

Logging In to Kibana

Log in to Kibana and go to the command execution page. Elasticsearch clusters support multiple access methods. This topic uses Kibana as an example to describe the operation procedures.

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch.
3. In the cluster list, find the target cluster, and click Kibana in the Operation column to log in to the Kibana console.
4. In the left navigation pane, choose Dev Tools.

   The left part of the console is the command input box, and the triangle icon in its upper-right corner is the execution button. The right part shows the execution result.

Monitoring P99 Latency

• Query the full percentile latency distribution.

  Obtain the complete percentile latency distribution of the current cluster to quickly determine whether slow queries exist.

  Run the following command:

  GET /search/stats/percentile 

  Example response:

  {
    "overall" : {
      "1.0" : 2.0,
      "5.0" : 2.0,
      "25.0" : 6.5,
      "50.0" : 19.5,
      "75.0" : 111.0,
      "95.0" : 169.0,
      "99.0" : 169.0,
      "max" : 169.0,
      "min" : 2.0
    },
    "last_one_day" : {
      "1.0" : 2.0,
      "5.0" : 2.0,
      "25.0" : 6.5,
      "50.0" : 19.5,
      "75.0" : 111.0,
      "95.0" : 169.0,
      "99.0" : 169.0,
      "max" : 169.0,
      "min" : 2.0
    },
    "latest" : {
      "1.0" : 26.0,
      "5.0" : 26.0,
      "25.0" : 26.0,
      "50.0" : 26.0,
      "75.0" : 26.0,
      "95.0" : 26.0,
      "99.0" : 26.0,
      "max" : 26.0,
      "min" : 26.0
    }
  }

  In this example, P99 equals P50, indicating that there are no excessively slow queries.

  Table 1 Response parameters

  Parameter	Description
  overall	Percentile latency distribution of all search requests since the cluster was started. It is used to create a historical overall performance baseline for the cluster. The unit is milliseconds (ms), indicating the request latency at each percentile.
  last_one_day	Percentile latency distribution in the last 24 hours. It is used for daily inspection and recent performance monitoring. The unit is milliseconds (ms), indicating the request latency at each percentile.
  latest	Percentile latency distribution since the most recent reset operation. It is used to compare and verify performance before and after an O&M operation. The unit is milliseconds (ms), indicating the request latency at each percentile.

• Query the latency in custom percentiles.

  Specify the percentiles to query to reduce irrelevant data and focus on specific performance metrics.

  Run the following command to query latency data for the 1st, 50th, and 90th percentiles:

  GET /search/stats/percentile
  {
    "percents": [1, 50, 90]
  }

  The percents array supports any value between 0 and 100. You can specify the percentiles to query based on your service-level agreement (SLA) requirements. For example, if the SLA requires P95 latency to be less than 500 ms, you can focus on monitoring P95.

  Example response:

  {
    "overall" : {
      "1.0" : 0.0,
      "50.0" : 0.0,
      "90.0" : 0.0,
      "max" : 146.0,
      "min" : 0.0
    },
    "last_one_day" : {
      "1.0" : 0.0,
      "50.0" : 0.0,
      "90.0" : 0.0,
      "max" : 146.0,
      "min" : 0.0
    },
    "latest" : {
      "1.0" : 0.0,
      "50.0" : 0.0,
      "90.0" : 0.0,
      "max" : 146.0,
      "min" : 0.0
    }
  }

• Reset the latest statistics.

  Before and after a cluster change (such as scaling out or parameter adjustment), reset the latest statistics baseline to accurately compare performance differences.

  Run the following command:

  POST /search/stats/reset

  

  The reset operation only clears the latest statistics. The overall and last_one_day statistics are not affected.

  Example response:

  {
    "nodes" : {
      "css-c9c8-ess-esn-1-1" : "ok"
    }
  }

  If the status of each node is ok, the reset is successful.

Monitoring HTTP Status Codes

Obtain cumulative HTTP status code statistics for each cluster node to understand request success rates and abnormal distributions.

• Elasticsearch 7.10.2

  Run the following command:

  GET /_nodes/stats/http 

  Example response:

  {
  ...
    "cluster_name" : "css-2985",
    "nodes" : {
  ...
      "omvR9_W-TsGApraMApREjA" : {
  ...
        "http" : {
          "current_open" : 4,
          "total_opened" : 37,
          "http_code" : {
            "200" : 25,
            "201" : 7,
            "429" : 0,
            "400" : 3,
            "404" : 0,
            "405" : 0
          },
          "http_worker_pending_tasks" : {
            "elasticsearch[css-xxx-ess-esn-1-1][transport_worker][T#1]" : 0,
            "elasticsearch[css-xxx-ess-esn-1-1][transport_worker][T#2]" : 0
          }
        }
      }
    }
  }

  Table 2 Response parameters

  Parameter	Description
  current_open	Number of HTTP connections currently maintained by the node. A continuously high value (for example, greater than 1000) may indicate connection leaks on the client side.
  total_opened	Total cumulative number of HTTP connections established by the node since startup. Comparing this value with current_open helps evaluate connection reuse efficiency.
  http_code	Cumulative number of requests for each HTTP status code since the node was started.
  http_worker_pending_tasks	Number of pending tasks for HTTP worker threads. A continuously high number of pending tasks for each transport_worker thread may indicate request processing bottlenecks on the node.

• Elasticsearch 7.6.2

  Run the following command:

  GET /_nodes/http_stats

  Example response:

  {   
      "_nodes" : {     
        "total" : 1,    
        "successful" : 1,     
        "failed" : 0   },  
       "cluster_name" : "css-8362",   
       "nodes" : {     
        "F9IFdQPARaOJI7oL7HOXtQ" : {       
           "http_code" : {        
              "200" : 114,        
              "201" : 5,        
              "429" : 0,       
              "400" : 7,         
              "404" : 0,        
              "405" : 0      
             }     
           }   
        }
   }

  Table 3 Response parameters

  Parameter	Description
  http_code	Cumulative number of requests for each HTTP status code since the node was started.

Reference for evaluating cluster health in terms of HTTP status code distribution:

• 200/201 account for the vast majority (>95%): Normal.
• A small number of 400 errors (<1%): Acceptable. These usually indicate a small number of client-side syntax errors.
• A small number of 404 errors: Acceptable.
• 429 errors continuously increase: The cluster is overloaded and requests are being throttled. Recommended actions:
  • Check for large queries or full scans.
  • Consider scaling out data nodes.
  • Consider configuring index-level throttling policies.
• 400 errors increase significantly: Client code likely contains DSL syntax issues. Check the client code.
• 5xx errors occur: Indicates internal cluster errors. Check cluster logs and node status immediately.

Joint Monitoring of P99 Latency and HTTP Status Codes

P99 latency monitoring and HTTP status code monitoring can be used together to quickly locate performance issues. Table 4 lists typical combinations of symptoms and possible causes.