Compute
Elastic Cloud Server
Huawei Cloud Flexus
Bare Metal Server
Auto Scaling
Image Management Service
Dedicated Host
FunctionGraph
Cloud Phone Host
Huawei Cloud EulerOS
Networking
Virtual Private Cloud
Elastic IP
Elastic Load Balance
NAT Gateway
Direct Connect
Virtual Private Network
VPC Endpoint
Cloud Connect
Enterprise Router
Enterprise Switch
Global Accelerator
Management & Governance
Cloud Eye
Identity and Access Management
Cloud Trace Service
Resource Formation Service
Tag Management Service
Log Tank Service
Config
OneAccess
Resource Access Manager
Simple Message Notification
Application Performance Management
Application Operations Management
Organizations
Optimization Advisor
IAM Identity Center
Cloud Operations Center
Resource Governance Center
Migration
Server Migration Service
Object Storage Migration Service
Cloud Data Migration
Migration Center
Cloud Ecosystem
KooGallery
Partner Center
User Support
My Account
Billing Center
Cost Center
Resource Center
Enterprise Management
Service Tickets
HUAWEI CLOUD (International) FAQs
ICP Filing
Support Plans
My Credentials
Customer Operation Capabilities
Partner Support Plans
Professional Services
Analytics
MapReduce Service
Data Lake Insight
CloudTable Service
Cloud Search Service
Data Lake Visualization
Data Ingestion Service
GaussDB(DWS)
DataArts Studio
Data Lake Factory
DataArts Lake Formation
IoT
IoT Device Access
Others
Product Pricing Details
System Permissions
Console Quick Start
Common FAQs
Instructions for Associating with a HUAWEI CLOUD Partner
Message Center
Security & Compliance
Security Technologies and Applications
Web Application Firewall
Host Security Service
Cloud Firewall
SecMaster
Anti-DDoS Service
Data Encryption Workshop
Database Security Service
Cloud Bastion Host
Data Security Center
Cloud Certificate Manager
Edge Security
Managed Threat Detection
Blockchain
Blockchain Service
Web3 Node Engine Service
Media Services
Media Processing Center
Video On Demand
Live
SparkRTC
MetaStudio
Storage
Object Storage Service
Elastic Volume Service
Cloud Backup and Recovery
Storage Disaster Recovery Service
Scalable File Service Turbo
Scalable File Service
Volume Backup Service
Cloud Server Backup Service
Data Express Service
Dedicated Distributed Storage Service
Containers
Cloud Container Engine
SoftWare Repository for Container
Application Service Mesh
Ubiquitous Cloud Native Service
Cloud Container Instance
Databases
Relational Database Service
Document Database Service
Data Admin Service
Data Replication Service
GeminiDB
GaussDB
Distributed Database Middleware
Database and Application Migration UGO
TaurusDB
Middleware
Distributed Cache Service
API Gateway
Distributed Message Service for Kafka
Distributed Message Service for RabbitMQ
Distributed Message Service for RocketMQ
Cloud Service Engine
Multi-Site High Availability Service
EventGrid
Dedicated Cloud
Dedicated Computing Cluster
Business Applications
Workspace
ROMA Connect
Message & SMS
Domain Name Service
Edge Data Center Management
Meeting
AI
Face Recognition Service
Graph Engine Service
Content Moderation
Image Recognition
Optical Character Recognition
ModelArts
ImageSearch
Conversational Bot Service
Speech Interaction Service
Huawei HiLens
Video Intelligent Analysis Service
Developer Tools
SDK Developer Guide
API Request Signing Guide
Terraform
Koo Command Line Interface
Content Delivery & Edge Computing
Content Delivery Network
Intelligent EdgeFabric
CloudPond
Intelligent EdgeCloud
Solutions
SAP Cloud
High Performance Computing
Developer Services
ServiceStage
CodeArts
CodeArts PerfTest
CodeArts Req
CodeArts Pipeline
CodeArts Build
CodeArts Deploy
CodeArts Artifact
CodeArts TestPlan
CodeArts Check
CodeArts Repo
Cloud Application Engine
MacroVerse aPaaS
KooMessage
KooPhone
KooDrive

Index Settings

Updated on 2023-11-29 GMT+08:00

An index is a storage structure used to query and analyze logs. Different index settings will generate different query and analysis results. Configure the index settings as required.

Log Example

The following is a typical log. The value of the content field is the original log text. Use commas (,) to parse the original log into three fields: level, status, and message.

In the example log, hostName, hostIP, and pathFile are common built-in reserved fields. For details about the built-in fields, see Built-in Reserved Fields.

       { "hostName":"epstest-xx518",
            "hostIP":"192.168.0.31",
            "pathFile":"stdout.log",
            "content":"error,400,I Know XX",
            "level":"error",
            "status":400,
            "message":"I Know XX"
        }

The following figure shows a typical index setting of a log example.

Index Types

The following table lists the index types supported by LTS.

Table 1 Index types

Index Type

Description

Index Whole Text

LTS splits all field values of an entire log into multiple words when this function is enabled.

NOTE:
  • The custom label field uploaded by the user is not included in the full-text index. If you want to search for the custom label field, add the corresponding index field.
  • Reserved fields are not included in full-text indexes. You need to use the Key:Value index to search for fields. For details, see Built-in Reserved Fields.

Index Fields

Query logs by specified field names and values (Key:Value).

NOTE:
  • By default, LTS creates index fields for some built-in reserved fields. For details, see Built-in Reserved Fields.
  • If an index field is configured for a field, the delimiter of the field value is subject to the index field configuration.
  • The quick analysis column in structuring settings has been removed. To use this function, configure index fields and enable quick analysis for the required fields.

Here are two examples:

  • In the log example, the level and status index fields are configured. The level field is of the string type, the field value is error, and a delimiter is configured. The status field is of the long type, and no delimiter needs to be configured. You can use level:error to search for all logs whose level value is error.
  • In the log example, LTS creates indexes for built-in reserved fields such as hostName, hostIP, and pathFile by default.

Precautions

  • Either whole text indexing or index fields must be configured.
  • Index settings (such as adding, editing, and deleting fields and modifying items) take effect only for new log data but not for historical log data. Currently, indexes cannot be recreated for historical logs.
  • After the index function is disabled, the storage space of historical indexes is automatically cleared after the data storage period of the current log stream expires.
  • By default, LTS creates index fields for some built-in reserved fields. For details, see Built-in Reserved Fields.
  • Different index settings will generate different query and analysis results. Configure the index settings as required. Full-text indexes and index fields do not affect each other.

Configuring Whole Text Indexing

  1. Log in to the LTS console and choose Log Management.
  2. In the log group list, click on the left of a log group, and click a log stream to go to the details page.
  3. Click in the upper right corner to go to the Index Settings page.
  4. Index Whole Text is enabled by default.

    NOTE:
    • For automatic configuration, the intersection of the raw logs and built-in fields in the last 15 minutes is obtained by default. LTS automatically combines the intersection of the raw logs and built-in fields, current structured fields, and tag fields to form the table data below the field index.
    • If no raw log is generated within 15 minutes, obtain the hostIP, hostName, pathFile, structured field, and tag field to form the table data below the field index.
    • When Log Structuring is configured for ECS ingestion, the category, hostName, hostId, hostIP, hostIPv6 and pathFile fields are automatically added on the Index Settings page. A field will not be added if the same one already exists.
    • When Log Structuring is configured for CCE ingestion, the category, clusterId, clusterName, nameSpace, podName, containerName, appName, hostName, hostId, hostIP, hostIPv6 and pathFile fields are automatically added to Index Settings page. A Field will not be added if the same one already exists.

  5. Set parameters as described in Table 2.

    Table 2 Whole text indexing parameters

    Parameter

    Description

    Index Whole Text

    If Index Whole Text is enabled, a full-text index is created.

    Case-Sensitive

    Indicates whether letters are case-sensitive during query.

    • If this function is enabled, the query result is case-sensitive. For example, if the example log contains Know, you can query the log only with Know.
    • If this function is disabled, the query result is case-insensitive. For example, if the example log contains Know, you can also query the log with KNOW or know.

    Include Chinese

    Indicates whether to distinguish between Chinese and English during query.

    • After the function is enabled, if the log contains Chinese characters, the Chinese content is split based on unigram segmentation and the English content is split based on delimiters.
      NOTE:

      Unigram segmentation is to split a Chinese string into Chinese characters.

      The advantage of unigram segmentation is efficient word segmentation of massive logs, and other Chinese segmentation methods have great impact on the write speed.

    • After this function is disabled, all content is split based on delimiters.

    For example, assume that the log content is:

    error,400,I Know TodayIsMonday.

    • After this function is disabled, the English content is split based on delimiters. The log is split into error, 400, I, Know, and TodayIsMonday. You can search for the log by error or TodayIsMonday.
    • After this function is enabled, the background analyzer of LTS splits the log into error, 400, I, Know, Today, Is, and Monday. You can search for the log by error or Today.

    Delimiters

    Splits the log content into multiple words based on the specified delimiter. Default delimiters include ,'";=()[]{}@&<>/:\n\t\r and spaces. If the default settings cannot meet your requirements, you can customize delimiters. All ASCII codes can be defined as delimiters.

    If the delimiter is set to null, the field value is regarded as a whole. You can search for the corresponding log only through the complete character string or fuzzy search.

    For example, assume that the log content is:

    error,400,I Know TodayIsMonday.

    • If no delimiter is set, the entire log is regarded as a string error,400,I Know TodayIsMonday. You can search for the log only by the complete string error,400,I Know TodayIsMonday or by fuzzy search error,400,I K*.
    • If the delimiter is set to a comma (,), the raw log is split into: error, 400, and I Know TodayIsMonday. You can find the log by fuzzy search or exact words, for example, error, 400, Kn*, and TodayIs*.
    • If the delimiter is set to a comma (,) and space, the raw log is split into: error, 400, I, Know, TodayIsMonday. You can find the log by fuzzy search or exact words, for example, Know, and TodayIs*.

  6. Click OK.

Configuring Index Fields

When creating a field index, you can add a maximum of 500 fields. A maximum of 100 subfields can be added for JSON fields.

  1. Log in to the LTS console and choose Log Management.
  2. In the log group list, click on the left of a log group, and click a log stream to go to the details page.
  3. Click in the upper right corner to go to the Index Settings page. Click Add Field and enter the field name.
  4. Configure the index field by referring to Table 3.

    NOTE:
    • The preceding indexing parameters take effect only for the current field.
    • Index fields that do not exist in log content are invalid.
    Table 3 Index field parameters

    Parameter

    Description

    Field Name

    Log field name, including level in the example log.

    The field name can contain only letters, digits, and underscores (_), and must start with a letter or underscore (_). The field name cannot contain double underscores (__).

    NOTE:
    • Double underscores (__) are used in built-in reserved fields that are not displayed to users in LTS. Double underscores (__) cannot be used in custom log field names. Otherwise, field index names cannot be configured.
    • By default, LTS creates index fields for some built-in reserved fields. For details, see Built-in Reserved Fields.

    Type

    • Data type of the log field value. The options are string, long, and float.
    • Fields of long and float types do not support Case-Sensitivity, Include Chinese and Delimiters.

    Quick Analysis

    By default, this option is enabled, indicating that this field will be sampled and collected. For details, see Quick Analysis.

    NOTE:
    • The principle of quick analysis is to collect statistics on 100,000 logs that match the search criteria, not all logs.
    • The maximum length of a field for quick analysis is 2000 bytes.
    • The quick analysis field area displays the first 100 records.

    Operation

    Click to delete the target field.

  5. Click OK.

Auto Index Field Configuration

When creating an index field, you can click Auto Config. The log service automatically adds some index fields. You can add or delete fields as required.

  • The log service automatically generates an index field based on the first content in the preview data during collection.
  • The log service selects several common built-in reserved fields (such as hostIP, hostName, and pathFile) and adds them to the index field.

We use cookies to improve our site and your experience. By continuing to browse our site you accept our cookie policy. Find out more

Feedback

Feedback

Feedback

0/500

Selected Content

Submit selected content with the feedback