Updated on 2025-07-29 GMT+08:00

Querying and Managing Elasticsearch Cluster Logs

CSS provides log query, log backup, and log ingestion, enabling you to easily manage and analyze logs to efficiently locate faults, optimize performance, and enhance system security.

  • Log query: On the log management page of the CSS management console, you can query the latest 10,000 log records by node, log type, and other criteria. A maximum of 100 results are displayed, enabling you to quickly locate issues.
  • Log backup: Cluster logs are periodically synchronized to OBS buckets. You can download them for in-depth analysis at any time. You can configure custom log backup policies by specifying backup schedules and storage locations. The system backs up all critical logs, including run logs, slow query logs, and deprecation logs. They provide comprehensive data for auditing and troubleshooting purposes.
  • Log ingestion: Cluster logs are transmitted in real time to the current cluster or another cluster on the same network. (Relevant version and network compatibility requirements must be met.) You can use Kibana or other visualization tools for log search and analysis. You can also set index prefixes and retention periods to flexibly manage the log lifecycle. Logs can be transferred across clusters, facilitating centralized monitoring in a multi-cluster environment.

Impact on Billing

When log backup is enabled, the generated log backups are stored in OBS buckets, which will result in additional costs. For details, see Object Storage Service Billing.

Prerequisites

The OBS bucket used for storing log backups has been created. The OBS bucket must meet the following requirements:

  • Storage Class: Standard.
  • Region: the same as that of the cluster.

Querying Logs

  1. Log in to the CSS management console.
  2. Choose Clusters in the navigation pane. On the Clusters page, click the cluster whose logs you want to query. The cluster information page is displayed.
  3. In the navigation pane on the left, choose Log Management.
  4. Query logs on the log management page.
  5. Select the node, log type, and log level, and click . The query result is displayed.

    The latest 10,000 logs can be queried, and a maximum of 100 records are displayed.

    When a log file reaches 128 MB, the system automatically compresses and archives it. Only unarchived logs appear on the log query page, while archived logs remain accessible through the log backup function.

Backing Up Logs

CSS cluster logs can be periodically backed up to specified OBS buckets.

  1. Log in to the CSS management console.
  2. Choose Clusters in the navigation pane. On the Clusters page, click the name of the target cluster. The cluster information page is displayed.
  3. Click the Logs tab and toggle on the Log Management switch.
  4. In the Edit Log Backup Configuration dialog box, set the parameters.

    If log backup has been enabled for the cluster, you can also click the edit icon next to Log Backup Configuration to update the settings.

    Table 1 Log backup settings

    Parameter

    Description

    OBS Bucket

    Select an OBS bucket for storing log backups from the drop-down list box.

    If no OBS buckets meet your requirements, click Create Bucket to go to the OBS console and create one. For details, see Creating a Bucket.

    Select an OBS bucket that meets the following requirements:

    • Storage Class: Standard.
    • Region: the same as that of the cluster.

    Backup Path

    Set the log storage location in the OBS bucket.

    The backup path cannot:
    • Contain the following characters: \:*?"<>|'{}
    • Start with a slash (/).
    • Start or end with a period (.).
    • Contain more than two consecutive slashes (/) or periods (.).
    • Exceed 512 characters.

    IAM Agency

    Select an IAM agency to grant the current account the permission to access and use OBS. To store logs to an OBS bucket, you must have the required OBS access permissions.

    • If you are configuring an agency for the first time, click Automatically Create IAM Agency to create css-obs-agency.
    • If there is an IAM agency automatically created earlier, you can click One-click authorization to have the OBS Administrator permissions deleted automatically, and have the following custom policies added automatically instead to implement more refined permissions control.
      "obs:bucket:GetBucketLocation",
      "obs:object:GetObjectVersion",
      "obs:object:GetObject",
      "obs:object:DeleteObject",
      "obs:bucket:HeadBucket",
      "obs:bucket:GetBucketStoragePolicy",
      "obs:object:DeleteObjectVersion",
      "obs:bucket:ListBucketVersions",
      "obs:bucket:ListBucket",
      "obs:object:PutObject"
    • When OBS buckets use SSE-KMS encryption, the IAM agency must be granted KMS permissions. You can click Automatically Create IAM Agency and One-click authorization to have the following custom policies created automatically.
      "kms:cmk:create",
      "kms:dek:create",
      "kms:cmk:get",
      "kms:dek:decrypt",
      "kms:cmk:list"
    • To use Automatically Create IAM Agency and One-click authorization, the following minimum permissions are required:
      "iam:agencies:listAgencies",
      "iam:roles:listRoles",
      "iam:agencies:getAgency",
      "iam:agencies:createAgency",
      "iam:permissions:listRolesForAgency",
      "iam:permissions:grantRoleToAgency",
      "iam:permissions:listRolesForAgencyOnProject",
      "iam:permissions:revokeRoleFromAgency",
      "iam:roles:createRole"
    • To use an IAM agency, the following minimum permissions are required:
      "iam:agencies:listAgencies",
      "iam:agencies:getAgency",
      "iam:permissions:listRolesForAgencyOnProject",
      "iam:permissions:listRolesForAgency"
    WARNING:

    The agency name can contain only letters (case-sensitive), digits, underscores (_), and hyphens (-). Otherwise, the backup will fail.

  5. Back up logs.
    • Automatically backing up logs

      Click the icon on the right of Auto Backup to enable automatic log backup.

      After automatic backup is enabled, set the backup start time in the Configure Auto Backup dialog box. When the scheduled time arrives, the system will back up logs automatically.

      After Automatic Snapshot Creation is enabled, you can click the Edit icon on the right of the parameter to change the backup start time.

    • Manually backing up logs

      On the Log Backup tab page, click Back Up. In the displayed dialog box, click Yes to start backup.

      If Task Status in the log backup list is Successful, the backup is successful.

    If log backup fails, click on the right of the Log Backup tab to check the number of failed tasks and learn the failure causes. A maximum of 10 failed tasks can be displayed. When log backup is disabled or the cluster is deleted, the failure records are also cleared.

  6. View log files.

    Logs are backed up incrementally. After the backup is successful, you can access the target OBS bucket to obtain the full log files. Table 2 lists the log types, where clustername indicates the cluster name.

    Figure 1 OBS bucket address
    Table 2 Log types

    Log Name

    Description

    clustername_deprecation.log

    Deprecation log file

    clustername_index_indexing_slowlog.log

    Slow indexing log file

    clustername_index_search_slowlog.log

    Slow query log file

    clustername.log

    Run log file

    clustername_access.log

    Access log file

Log Ingestion

With log ingestion, you can store the real-time logs of a CSS cluster in itself or a different cluster on the same network, facilitating log search and analysis using Kibana.

To use log ingestion, the cluster must meet the following requirements. If the cluster does not meet these requirements, you are advised to upgrade the cluster first.
  • The cluster version is Elasticsearch 7.10.2, OpenSearch 1.3.6, or OpenSearch 2.11.0.
  • The cluster image version is 24.2.0 or later. You can check the cluster image version in the Version column of the cluster list, as shown in the following figure.
    Figure 2 Checking the cluster version
  1. Log in to the CSS management console.
  2. Choose Clusters in the navigation pane. On the Clusters page, click the name of the target cluster. The cluster information page is displayed.
  3. Click the Logs tab and toggle on the Log Ingestion switch.

    If the Log Ingestion switch is not displayed, the cluster does not support log ingestion.

  4. In the Log Ingestion Configuration dialog box, set relevant parameters.

    If log ingestion has been enabled for the cluster, you can also click the edit icon next to Log Ingestion Configuration to update the settings.

    Table 3 Log ingestion settings

    Parameter

    Description

    Index Prefix Name

    If you set a prefix for the log file indexes, the log index names will use the format index prefix name + log ingestion date. The unit of log ingestion is days.

    An index prefix name is a string of 1 to 128 characters. It can contain only digits, lowercase letters, underscores (_), and hyphens (-).

    Retention Period

    Set the log retention period. Ingested logs will be deleted upon expiration of this period.

    Unit: days

    Value range: 1 to 3650

    Default value: 30 days

    Log Storage Cluster

    Select a cluster to store collected logs. Options include Current cluster and Other clusters.

    • Current cluster: Store collected logs in the current cluster.
    • Other clusters: Store collected logs in another cluster. In this case, you need to further select a target cluster and check network connectivity to this cluster. Both clusters must reside in the same VPC network and use the same version.

    Default: Current cluster.

  5. Click OK to enable cluster log ingestion.

    If Status changes to Running, log ingestion has started.

    Click Access Kibana to log in to the cluster and search for and view logs.

    Click the cluster name in the Log Storage Cluster area to go to the cluster details page.

    Figure 3 Log Ingestion
  6. To disable log ingestion, click the toggle button next to Log Ingestion. In the displayed dialog box, click OK.

    After log ingestion is disabled, the retained logs will not be cleared right away. Instead, they will be deleted upon expiration of their retention period, which is part of the log ingestion settings.

Log Introduction

Table 4 Introduction to different log types

Log Type

Description

Purpose

Run logs

Run logs, or main logs, record the cluster status and key information about write and query operations. For example, write logs record operations such as index creation, index mapping update, and write queue exhaustion; and query logs record query queue status and query exceptions.

Check the status and write and query operations of each cluster node, including inter-node connectivity, full GC, index creation or deletion, and cluster-level query errors.

Slow indexing logs

Slow indexing logs record indexing operations (such as bulk, index, update, and delete) that took a long time to complete, helping you identify performance bottlenecks.

In the case of slow write performance, you can query slow indexing logs to locate the cause.

Slow query logs

Slow query logs record search requests that took a long time to complete. They help you monitor and analyze time-consuming search requests, so you can identify performance bottlenecks, optimize SQL queries, and improve overall system performance.

In the case of slow query performance, you can query slow query logs to locate the cause.

Deprecation logs

Deprecation logs record deprecation warnings. Deprecation warnings are written to this log when you use APIs, configurations, or functions that are marked for removal in future versions.

Check for APIs or features that are about to expire in future versions.

Access logs

Access logs record cluster access requests, such as the request path and source address.

You cannot check access logs on the console. To check them, you need to back them up to an OBS bucket or transfer them to a target cluster first.

If there is a surge in service requests, you can analyze the request sources and paths by checking the access logs.

  • Run log description

    Run logs record the cluster status and key information about write and query operations. For example, the log record below indicates that an index named test was created and afterwards the cluster status changed from YELLOW to GREEN.

    Figure 4 A sample of run logs
    Log content:
    • 1. Log generation time
    • 2. Log level, which can be DEBUG, INFO, WARN, or ERROR
    • 3. Log-generating module
    • 4. Name of the log-generating node
    • 5. Log content
  • Slow indexing log description

    Slow indexing logs record indexing operations that took a long time to complete. For example, the log record below shows an indexing request that lasted longer than the preset threshold. The log contains the index name, duration, and request content.

    Figure 5 A sample of slow indexing logs
    Log content:
    • 1. Log generation time

    • 2. Log level, which can be DEBUG, INFO, WARN, or ERROR
    • 3. Log-generating module
    • 4. Name of the log-generating node
    • 5. Index name and ID
    • 6. Log content In this example, the log recorded the request execution duration, index type, and index request body.
  • Slow query log description

    Slow query logs record search requests that took a long time to complete. For example, the log record below shows a search request that lasted longer than the preset threshold. The log contains the index name, duration, and request content.

    Figure 6 A sample of slow query logs
    Log content:
    • 1. Log generation time
    • 2. Log level, which can be DEBUG, INFO, WARN, or ERROR
    • 3. Log-generating module
    • 4. Name of the log-generating node
    • 5. Index name and shard ID
    • 6. Log content In this example, the log recorded the query duration, number of hits, and query request body.
  • Deprecation log description

    Deprecation logs record deprecation warnings. For example, the log record below indicates that GET /_cat/master has been deprecated and should be replaced with GET /_cat/cluster_manager.

    Figure 7 A sample of deprecation logs
    Log content:
    • 1. Log generation time
    • 2. Log level, which can be DEBUG, INFO, WARN, ERROR, or DEPRECATION
    • 3. Log-generating module
    • 4. Name of the log-generating node
    • 5. Log content
  • Access log description

    Access logs record cluster access requests and source addresses. For example, the log record below has recorded source information for the /_snapshot/my_backup/my_snapshot/_restore?pretty=true operation.

    Figure 8 A sample of access logs
    Log content:
    • 1. Log generation time
    • 2. Name of the log-generating node
    • 3. Name of the log-generating thread
    • 4. Log level, which can be DEBUG, INFO, WARN, or ERROR
    • 5. Log request method
    • 6. Request path
    • 7. Source and destination addresses of the request