Help Center> Log Tank Service> FAQs> Log Management> How Do I Select LTS Compared with Self-Built ELK?

Updated on 2024-03-07 GMT+08:00

View PDF

How Do I Select LTS Compared with Self-Built ELK?

This document helps you better understand the main functions and advantages of Huawei Cloud LTS by comparing LTS with self-built ELK.

Background

Many people use ELK Stack (Elasticsearch/Logstash/Kibana) to build an open-source ELK solution for log search. You can find plenty of content and use cases in the community to guide you.

LTS provides a fully managed log analysis platform that covers three scenarios: application O&M, graded protection compliance, and service operation. It enables customers to collect, store, query, process, analyze, and report logs with ease.

Function

LTS outperforms ELK in terms of function and feature completeness and log search and analysis performance. For details, see the following table.

Feature	Subfeature	LTS	ELK	Description
Log Collection	Cloud service log collection	☆☆☆☆☆	None	ELK: You cannot ingest logs from cloud services. LTS: Logs of the cloud service tenant plane are collected to LTS.
	VM and container log collection	☆☆☆☆☆	☆☆☆☆	ELK: Open-source collectors such as Logstash or Filebeat are used to collect logs. LTS: ICAgent is used to collect logs. A wizard page is provided, which is easy to use.
	Multi-language SDK Log Collection	☆☆☆	None	ELK: No LTS: Provides a Java SDK to directly report logs to LTS.
	Host group management (dynamic scaling of hosts)	☆☆☆☆☆	None	ELK: No LTS: Allows you to manage hosts and host groups. You can customize host groups and scale them in or out dynamically.
	Log structuring parsing	☆☆☆☆	☆☆☆☆☆	ELK: Implements structuring parsing of customized logs based on the collector. LTS: Enables structuring parsing logs. You can use regular expressions, JSON, separators, or customized templates to parse logs.
Log Search	Keyword search, fuzz match, and quick analysis	☆☆☆☆☆	☆☆☆☆☆	ELK and LTS: Provide similar keyword search functions.
	Viewing real-time logs	☆☆☆☆☆	None	ELK does not provide the page for viewing real-time logs. LTS provides page for viewing real-time logs.
	Search of tens of billions of logs in seconds	☆☆☆☆☆	☆☆	ELK: Limited by the number of machine resources, it takes a long time to search for massive logs. LTS: With a large number of elastic computing resources of the public cloud, search results can be returned within 3 seconds for tens of billions of logs.
	Iterative search of hundreds of billions of logs	☆☆☆☆☆	None	ELK: Unable to search hundreds of billions of logs directly. And the response times out. LTS: Provides iterative search. Users can directly search for hundreds of billions of logs.
	Log management scale	100 PB level	100 TB level	ELK: It is often time consuming to keep an eye on machine expansion. LTS: Pay-per-use. LTS automatically manages 100 PB level logs regardless of underlying resource consumption.
Log Search	SQL analysis logs	☆☆☆☆☆	☆☆	ELK does not support nested SQL statements in syntax due to poor performance. LTS supports nested SQL statements with high performance.
Log Search	SQL functions	☆☆☆☆☆	☆☆	ELK only supports basic SQL statistics functions. Besides basic SQL functions, LTS offers a rich set of extended functions, such as IP, statistics, chain and parallel comparison, and URL functions, that broaden the range of use cases.
Log Search	Charts	☆☆☆☆	☆☆☆	LTS: provides multiple visualized charts, such as tables, line charts, pie charts, and bar charts.
Log Search	Dashboards	☆☆☆☆☆	☆☆	ELK: There is no ready-to-use dashboard for cloud service logs. LTS: Provides ready-to-use dashboards for common cloud service logs, such as ELB, APIG, DDS, DCS, and CFW.
Log alarms	Keyword and SQL alarms	☆☆☆☆☆	☆	ELK: Log alarm is not available. LTS: Quasi-real-time log keyword and SQL alarms are available.
Log alarms	Alarm notification channels (such as email, SMS, and HTTPS)	☆☆☆☆☆	☆	ELK: Alarms cannot be sent to users through DingTalk, WeChat, or SMS messages. LTS: Interconnects with the Simple Message Notification (SMN) service of Huawei Cloud to notify customers through email, SMS, WeChat, DingTalk, Flying Book, and HTTP.
Log transfer	Transfer to OBS	☆☆☆☆☆	None	ELK: Logs cannot be directly transferred to OBS. LTS: Logs can be transferred to OBS through simple page configuration.
Log Transfer	Transfer to Kafka	☆☆☆☆☆	☆☆	ELK: You need to deploy a program to forward logs to Kafka. LTS: Logs can be transferred to Kafka through simple page configuration.
Log Transfer	Transfer to the data warehouse	☆☆☆☆☆	None	ELK: Logs cannot be directly transferred to the data warehouse. LTS: Logs can be transferred to the data warehouse through simple page configuration.
Log jobs	Scheduled SQL jobs	☆☆☆☆☆	None	ELK does not support scheduled SQL jobs. LTS: You can configure scheduled SQL jobs to process original logs and collect statistics on a small number of logs.
Log jobs	Function jobs	☆☆☆☆☆	None	ELK: Log jobs are not available. LTS: Supports function triggers. You can write custom scripts in the function service to handle logs flexibly.

Cost comparison

Scenario 1:

Suppose you generate 100 GB of raw logs per day (the average log rate is 1.16 MB/s), keep them for 30 days on average, and store them as one primary and one standby. The total size of original logs generated in 30 days is 3000 GB.

Based on the official recommendation of Elasticsearch, the total storage space for raw logs, backup data, and index data is about 2.2 times the size of raw logs in the one primary and one standby mode. Plus, the ES cluster has uneven write and the disk is not fully utilized. So, to store 3000 GB of raw logs, you need disks with at least 3000 GB x 2.2 (storage expansion) x 2 (50% disk redundancy) = 13200 GB.

ES needs at least three ECSs (16 vCPUs, 64 GB memory, and 5 TB) as a typical configuration. Two Kafka replicas can store logs of the past 12 hours.

Category	Subcategory	Monthly Cost	Expense Proportion
Setting up ES	3 ECSs (C6 16 vCPUs \| 64 GB)	3 x 1999 = 5997	47.6%
Setting up ES	Elastic Volume Service (EVS) (high I/O 15 TB)	0.35 x 15 x 1024 = 5376	42.7%
Setting up Kafka	3 ECSs (2 vCPUs \| 4 GB)	3 x 208 = 624	4.9%
Setting up Kafka	EVS (ultra-high I/O 3 x 200 GB)	600	4.7%
-	-	Self-built ELK: 12,597 in total	-

The monthly cost of LTS calculated using Price Calculator is around CNY2102, which is 16.7% of the cost of self-built ELK. This is because self-built ELK has a high initial resource cost in scenarios with few logs, while LTS charges you only for what you use, giving it a big edge.

Scenario 2:

Suppose you generate 1 TB of raw logs per day (the average log rate is 11.6 MB/s), keep them for 7 days on average, and store them as one primary and one standby. The total size of original logs generated in 7 days is 7 TB. Based on the official recommendation of Elasticsearch, the total storage space for raw logs, backup data, and index data is about 2.2 times the size of raw logs in the one primary and one standby mode. Plus, the ES cluster has uneven write and the disk is not fully utilized. So, to store 7 TB of raw logs, you need disks with at least 7 TB x 2.2 (storage expansion) x 2 (50% disk redundancy) = 31 TB.

ES needs at least three ECSs (16 vCPUs, 64 GB memory, and 10 TB) as a typical configuration. Two Kafka replicas can store logs of the past 12 hours.

Category	Subcategory	Monthly Cost	Expense Proportion
Setting up ES	3 ECSs (C6 16 vCPUs \| 64 GB)	3 x 1999 = 5997	31.7%
Setting up ES	EVS (high I/O 31 TB)	0.35 x 31 x 1024 = 11,110	58.7%
Setting up Kafka	3 ECSs (2 vCPUs \| 4 GB)	3 x 208 = 624	3.3%
Setting up Kafka	EVS (ultra-high I/O 3 x 400 GB)	1200	6.3%
-	-	Self-built ELK: 18,931 in total	-

The monthly cost of LTS calculated using Price Calculator is around CNY13,408, which is 71% of the cost of self-built ELK. This is because LTS storage is pay-per-use, while self-built ELK needs a lot of extra disks to keep the clusters running smoothly.

Scenario 3:

Suppose you generate 5 TB of raw logs per day (the average log rate is 58 MB/s), keep them for 30 days on average, and store them as one primary and one standby. The total size of original logs generated in 30 days is 150 TB.

Based on the official recommendation of Elasticsearch, the total storage space for raw logs, backup data, and index data is about 2.2 times the size of raw logs in the one primary and one standby mode. Plus, the ES cluster has uneven write and the disk is not fully utilized. So, to store 150 TB of raw logs, you need disks with at least 150 TB x 2.2 (storage expansion) x 2 (50% disk redundancy) = 660 TB.

ES needs at least 66 ECSs (16 vCPUs, 64 GB memory, and 10 TB) as a typical configuration. Two Kafka replicas can store logs of the past 12 hours.

Category	Subcategory	Monthly Cost	Expense Proportion
Setting up ES	66 ECSs (C6 16 vCPUs \| 64 GB)	66 x 1999 = 131,934	35.3%
Setting up ES	EVS (high I/O 660 TB)	0.35 x 660 x 1024 = 236,544	63.2%
Setting up Kafka	3 ECSs (2 vCPUs \| 4 GB)	3 x 208 = 624	0.2%
Setting up Kafka	EVS (ultra-high I/O 3 x 1700 GB)	5100	1.4%
-	-	Self-built ELK: 374,202 in total	-

The monthly cost of LTS calculated using Price Calculator is around CNY107,655, which is 28.8% of the cost of self-built ELK. This is because LTS storage is pay-per-use, while self-built ELK needs a lot of extra disks to keep the clusters running smoothly.

Summary

LTS beats ELK in functions, performance, and costs. You are advised to use fully managed LTS instead of self-built ELK.

Parent topic: Log Management

Log Management FAQs

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot

more