Querying and Managing Elasticsearch Cluster Logs

Querying Logs

1. Log in to the CSS management console.
2. Choose Clusters in the navigation pane. On the Clusters page, click the cluster whose logs you want to query. The cluster information page is displayed.
3. In the navigation pane on the left, choose Log Management.
4. Query logs on the log management page.
5. Select the node, log type, and log level you want to query, and then click . The query result is displayed.

   When you search for logs, the latest 10,000 logs are matched. A maximum of 100 logs are displayed.

Backing Up Logs

CSS cluster logs can be periodically backed up to specified OBS buckets.

1. Log in to the CSS management console.
2. Choose Clusters in the navigation pane. On the Clusters page, click the name of the target cluster. The cluster information page is displayed.
3. Click the Logs tab and toggle on the Log Management switch.
4. In the Edit Log Backup Configuration dialog box, set the parameters.

   In the displayed dialog box, OBS Bucket, Backup Path, and IAM Agency are automatically created for log backup. You can change the default values by referring to Table 1.

   If Log Backup has been enabled for the cluster, you can click the Edit icon on the right of Log Backup Configuration and modify the configuration in the displayed Edit Log Backup Configuration dialog box. For details, see Table 1.

   Table 1 Configuring log backup

   Parameter	Description	Remarks
   OBS Bucket	Select an OBS bucket from the drop-down list for storing logs. You can also click Create Bucket on the right to create an OBS bucket.	The OBS bucket and the cluster must be in the same region.
   Backup Path	Storage path of logs in the OBS bucket	The backup path cannot: Contain the following characters: \:*?"<>| Start with a slash (/). Start or end with a period (.). Exceed 1023 characters.
   IAM Agency	IAM agency authorized by the current account for CSS to access or maintain data stored in OBS buckets.	If you are configuring an agency for the first time, click Automatically Create IAM Agency to create css-obs-agency. If there is an IAM agency automatically created earlier, you can click One-click authorization to delete the OBS Administrator permissions, and add the following custom policies instead to implement more refined permissions control. "obs:bucket:GetBucketLocation", "obs:object:GetObjectVersion", "obs:object:GetObject", "obs:object:DeleteObject", "obs:bucket:HeadBucket", "obs:bucket:GetBucketStoragePolicy", "obs:object:DeleteObjectVersion", "obs:bucket:ListBucketVersions", "obs:bucket:ListBucket", "obs:object:PutObject" To use Automatically Create IAM Agency and One-click authorization, the following minimum permissions are needed: "iam:agencies:listAgencies", "iam:roles:listRoles", "iam:agencies:getAgency", "iam:agencies:createAgency", "iam:permissions:listRolesForAgency", "iam:permissions:grantRoleToAgency", "iam:permissions:listRolesForAgencyOnProject", "iam:permissions:revokeRoleFromAgency", "iam:roles:createRole" To use an IAM agency, the following minimum permissions are needed: "iam:agencies:listAgencies", "iam:agencies:getAgency", "iam:permissions:listRolesForAgencyOnProject", "iam:permissions:listRolesForAgency"

5. Back up logs.
   • Automatically backing up logs

     Click the icon on the right of Auto Backup to enable the auto backup function.

     After the automatic backup function is enabled, set the backup start time in the Configure Auto Backup dialog box. When the scheduled time arrives, the system will back up logs automatically.

     After the Automatic Snapshot Creation function is enabled, you can click the Edit icon on the right of the parameter to change the backup start time.

   • Manually backing up logs

     On the Log Backup tab page, click Back Up. In the displayed dialog box, click Yes to start backup.

     If Task Status in the log backup list is Successful, the backup is successful.

   

   If log backup fails, click on the right of the Log Backup tab to view the number of failed tasks and learn the failure causes. A maximum of 10 failed tasks can be displayed. When log backup is disabled or the cluster is deleted, the failure records are also cleared.

6. View log files.

   After logs are successfully backed up, you can click OBS Buckets to go to the bucket list, and find the bucket that stores the log files to view log files.

   Figure 1 Accessing OBS
   

Collecting Logs

With log collection, you can store the real-time logs of a CSS cluster in itself or a different cluster on the same network, facilitating log search and analysis using Kibana.

1. Log in to the CSS management console.
2. Choose Clusters in the navigation pane. On the Clusters page, click the name of the cluster whose logs you want to collect. The cluster information page is displayed.
3. Click the Logs tab and toggle on the Log Ingestion switch.
   

   If the Log Ingestion switch is not displayed, the cluster does not support log ingestion.

4. In the Log Ingestion Configuration dialog box, set relevant parameters.

   Table 2 Log ingestion settings

   Parameter	Description
   Index Prefix Name	If you set a prefix for the log file indexes, the index names will use the format index prefix name + log collection date. The unit of log ingestion is days. An index prefix name is a string of 1 to 128 characters. It can contain only digits, lowercase letters, underscores (_), and hyphens (-).
   Retention Period	Retention period of collected logs, in days. The value ranges from 1 to 3650. Retained logs are deleted upon expiration of this retention period.
   Log Storage Cluster	Select a cluster to store collected logs. Options include Current cluster and Other clusters. Current cluster is selected by default. If you select Other clusters, you need to further select a target cluster and check network connectivity to this cluster. Both clusters must reside in the same VPC network and use the same version.

   If log ingestion is enabled, you can click Edit in the upper-right corner to modify log ingestion settings.

5. Click OK to enable cluster log ingestion.

   If Status changes to Running, log ingestion has started.

   Click Access Kibana to log in to the cluster and search for and view logs.

   Click the cluster name in the Log Storage Cluster area to go to the cluster details page.

   Figure 3 Log Ingestion
   
6. To disable log ingestion, click the toggle button next to Log Ingestion. In the displayed dialog box, click OK.

   After log ingestion is disabled, the retained logs will not be cleared right away. Instead, they will be deleted upon expiration of their retention period, which is part of the log ingestion settings.

Log Files

Deprecation logs, run logs, index slow logs, and search slow logs are backed up for Elasticsearch and OpenSearch clusters.