Elasticsearch Logs
Elasticsearch, an open-source distributed search and analytics engine, is widely for building real-time and scalable search and analytics applications. Its primary functions are to store, search for, and analyze large amount of data. Elasticsearch logs record various information throughout Elasticsearch running. There are primarily two log types:
- cluster-name.log (main log, named after the cluster name): records comprehensive information about all events that occurred during the ElasticSearch running, for example, a query failure or a new node being added to the cluster. These logs are important for understanding the overall running status of Elasticsearch clusters.
- cluster-name_index_search_slowlog.log (slow query log, named after the cluster name): When a query runs slowly (takes more than half a second), Elasticsearch generates a log to record the query. These logs are useful for analyzing and optimizing query performance.
Elasticsearch also provides other log types, such as deprecated logs (recording deprecated functions or configurations) and access logs (recording details about all query requests).
Collection Method
Install ICAgent and configure ECS text log ingestion by referring to Installing ICAgent (Intra-Region Hosts) and Ingesting ECS Text Logs to LTS. Then, ICAgent will collect ECS text logs and report them to LTS.
Elasticsearch Log Examples and Fields
- Main log
{ "log.level": "INFO", "@timestamp": "2023-10-23T15:46:30.123Z", "log.origin.file.name": "Elasticsearch.java", "log.origin.file.line": 123, "message": "version[7.17.2], pid[12345], build[default/tar/1234567/2023-10-01T12:00:00.000Z], OS[Linux/5.10.0-10-amd64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/11.0.14.1/11.0.14.1+1-LTS]", "type": "server", "cluster.name": "my-cluster", "node.name": "node-1", "node.id": "abcdefg123", "ecs.version": "1.8.0" } { "log.level": "WARN", "@timestamp": "2023-10-23T15:47:00.456Z", "log.origin.file.name": "SlowLog.java", "log.origin.file.line": 456, "message": "[index][search_slowlog][index_name] took[5.6s], took_millis[5600], total_hits[1000], types[], stats[], search_type[QUERY_THEN_FETCH], source[...]", "type": "slowlog", "cluster.name": "my-cluster", "node.name": "node-1", "index.name": "index_name" }
Table 1 Main log fields Field
Description
Example
log.level
Log level, which can be INFO, WARN, or ERROR.
INFO
@timestamp
Log timestamp.
2023-10-23T15:46:30.123Z
log.origin.file.name
Name of the source file that generated the log entry.
Elasticsearch.java
log.origin.file.line
Line number of the source file that generated the log entry.
123
message
Actual log message content.
version[7.17.2], pid[12345], build[default/tar/1234567/2023-10-01T12:00:00.000Z], OS[Linux/5.10.0-10-amd64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/11.0.14.1/11.0.14.1+1-LTS]
type
Log entry type, such as server, slowlog, or transport.
server
cluster.name
Elasticsearch cluster name.
my-cluster
node.name
Name of the Elasticsearch node for which the log was generated.
node-1
node.id
ID of the Elasticsearch node for which the log was generated.
abcdefg123
index.name
Name of the index associated with the log entry (available to some log entries).
index_name
ecs.version
Version of Elastic Common Schema (ECS). ECS is a specification used to standardize Elasticsearch log fields.
1.8.0
- Slow query log
{ "type": "slowlog", "timestamp": "2023-10-23T16:15:30,234Z", "level": "WARN", "component": "o.e.i.s.q", "cluster.name": "my-cluster", "node.name": "node-1", "node.id": "abcdefg123", "took": "10.5s", "took_millis": 10500, "source": "{\"query\":{\"match_all\":{}}}\n", "shards": [ { "index": "my_index", "shard": "0", "search_type": "QUERY_THEN_FETCH", "total_hits": 12345, "status": 200, "forced_refresh": false, "request_cache": true, "scroll_size": 0 } ] }
Table 2 Slow query log fields Field
Description
Example
type
Log type.
"slowlog" indicates a slow query log entry.
timestamp
Timestamp of the log entry.
2023-10-23T16:15:30,234Z
level
Log level.
"WARN" indicates that the log entry level is warning.
component
Component for which the log was generated.
"o.e.i.s.q" indicates a specific component or module in Elasticsearch.
cluster.name
Elasticsearch cluster name.
my-cluster
node.name
Name of the node for which the log was generated.
node-1
node.id
ID of the node for which the log was generated.
abcdefg123
took
Total query execution time, in a human-readable format.
10.5s
took_millis
Total query execution time, in milliseconds.
10500
source
Original query statement (in JSON format) that triggered the slow query.
{"query":{"match_all":{}}}\n
shards
Shard information array involved in the slow query.
-
index
Name of the index where the shard is located.
my_index
shard
Shard ID.
0
search_type
Query type.
QUERY_THEN_FETCH
total_hits
Total number of hits returned by the query.
12345
status
HTTP status code of the query.
200
forced_refresh
Whether to forcibly refresh shards.
false
request_cache
Whether to use the request cache.
true
scroll_size
Size of the scrolling query. If not applicable, set it to 0.
0
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot