Help Center/ Log Tank Service/ FAQs/ Consultation/ What Are the Advantages of LTS Compared with Self-built ELK Stack?
Updated on 2024-08-21 GMT+08:00
View PDF
Share

What Are the Advantages of LTS Compared with Self-built ELK Stack?

This section describes the main functions and advantages of Huawei Cloud LTS by comparing it with self-built ELK Stack.

Background

The open-source ELK Stack, comprising Elasticsearch, Logstash, and Kibana, is extensively used for log search, with a variety of content and use cases available within its community.

LTS is a fully managed log analysis platform that covers application O&M, security compliance, and service operations. You can use it to collect, store, query, process, analyze, and report logs with ease. For details, see Infographics.

Functions

LTS outperforms ELK in terms of feature completeness and log search and analysis performance.

Feature

Subfeature

LTS

ELK

Description

Log collection

Cloud service log collection

☆☆☆☆☆

N/A

ELK: does not collect cloud service logs.

LTS: collects all logs of the cloud service tenant plane.

VM and container log collection

☆☆☆☆☆

☆☆☆☆

ELK: uses open-source collectors such as Logstash or Filebeat.

LTS: uses ICAgent to collect logs and provides easy-to-use wizard pages.

Collection via multi-language SDKs

☆☆☆

N/A

ELK: not supported.

LTS: provides a Java SDK to directly report logs to LTS.

Host group management (dynamic scaling of hosts)

☆☆☆☆☆

N/A

ELK: not supported.

LTS: supports host and host group management. You can add custom identifiers to host groups and scale host groups in or out.

Log structuring parsing

☆☆☆☆

☆☆☆☆☆

ELK: enables custom structuring parsing based on the collectors.

LTS: enables structuring parsing with regular expressions, JSON, delimiters, or custom templates.

Log search

Keyword search, fuzz match, and quick analysis

☆☆☆☆☆

☆☆☆☆☆

ELK and LTS: provide similar keyword search functions.

Real-time log viewing

☆☆☆☆☆

N/A

ELK: does not provide the page for viewing real-time logs.

LTS: provides the page for viewing real-time logs.

Search of tens of billions of logs in seconds

☆☆☆☆☆

☆☆

ELK: Limited by the server resources, it takes a long time to search for massive logs.

LTS: With the extensive scalable computing resources of Huawei Cloud, search results can be returned in 3 seconds.

Iterative search of hundreds of billions of logs

☆☆☆☆☆

N/A

ELK: Response timeout occurs when hundreds of billions of logs are searched.

LTS: Iterative search enables search of hundreds of billions of logs.

Log management scale

100 PB level

100 TB level

ELK: It is often time-consuming to keep an eye on server scaling.

LTS: automatically manages 100 PB of logs. You do not need to worry about the underlying resource consumption and will be charged on a pay-per-use basis.

Log search

SQL analysis

☆☆☆☆☆

☆☆

ELK: does not support nested SQL statements in syntax due to poor SQL performance.

LTS: provides high SQL performance and supports nested SQL statements.

Log search

SQL functions

☆☆☆☆☆

☆☆

ELK: supports only basic SQL statistics functions.

LTS: Besides basic SQL functions, LTS offers various extended functions, such as IP, statistics, chain and parallel comparison, and URL functions, to support more scenarios.

Log search

Charts

☆☆☆☆

☆☆☆

LTS: provides various visual charts, such as tables, and line, pie, and bar charts.

Log search

Dashboards

☆☆☆☆☆

☆☆

ELK: There is no ready-to-use dashboard for cloud service logs.

LTS: provides ready-to-use dashboards for common cloud services, such as ELB, APIG, Document Database Service (DDS), DCS, and Cloud Firewall (CFW).

Log alarms

Keyword and SQL alarms

☆☆☆☆☆

☆

ELK: No log alarm function is available.

LTS: Quasi-real-time log keyword and SQL alarms are available.

Alarm notification channels (such as email, SMS, and HTTPS)

☆☆☆☆☆

☆

ELK: does not send alarms to users through DingTalk, WeCom, or SMS messages.

LTS: interconnects with Huawei Cloud Simple Message Notification (SMN) to notify users through channels such as email, SMS, WeCom, DingTalk, Lark, and HTTP.

Log transfer

Transfer to OBS

☆☆☆☆☆

N/A

ELK: cannot transfer logs to OBS directly.

LTS: allows you to transfer logs to OBS with simple page configurations.

Log transfer

Transfer to Kafka

☆☆☆☆☆

☆☆

ELK: requires you to deploy a program.

LTS: allows you to transfer logs to Kafka in real time with simple page configurations.

Log transfer

Transfer to data warehouses

☆☆☆☆☆

N/A

ELK: cannot transfer logs to data warehouses.

LTS: allows you to transfer logs to data warehouses with simple page configurations.

Log jobs

Scheduled SQL jobs

☆☆☆☆☆

N/A

ELK: does not support scheduled SQL jobs.

LTS: allows you to configure scheduled SQL jobs to convert raw logs into summarized results.

Log processing with functions

☆☆☆☆☆

N/A

ELK: does not support log processing.

LTS: provides function triggers. You can write custom scripts in FunctionGraph to process logs flexibly.

Costs

Scenario 1:

Assume that a total of 3,000 GB of raw logs are generated in 30 days (100 GB per day) at an average log rate of 1.16 MB/s, and those logs are retained for an average of 30 days in dual storage (primary and standby).

Elasticsearch recommends that the total storage space for raw logs, backup data, and index data in dual storage mode be about 2.2 times the size of raw logs. Considering the Elasticsearch cluster's uneven write distribution and partial disk utilization, storing 3,000 GB of raw logs needs at least 13,200 GB of disk space, calculated as 3,000 GB x 2.2 (storage expansion) x 2 (allowing for 50% disk redundancy).

Typically at least three ECSs (16 vCPUs, 64 GB memory, and 5 TB capacity) and two Kafka replicas are required for Elasticsearch to store logs of the past 12 hours.

Table 1 Self-built ELK

Category

Subcategory

Monthly Cost (Total: $1,764 USD)

Expense Proportion

Setting up Elasticsearch

3 Elastic Cloud Servers (ECSs) (C6 16 vCPUs | 64 GB)

3 x 1,999 x 0.1401 = $840 USD

47.6%

Elastic Volume Service (EVS) (high I/O 15 TB)

0.35 x 15 x 1,024 x 0.1401= $753 USD

42.7%

Setting up Kafka

3 ECSs (2 vCPUs | 4 GB)

3 x 208 x 0.1401 = $87 USD

4.9%

EVS (ultra-high I/O 3 x 200 GB)

600 x 0.1401 = $84 USD

4.7%

According to the price calculator, the monthly cost of LTS is around $539.89 USD, just 16.7% of the cost of self-built ELK. This significant saving is attributed to LTS's pay-per-use billing mode, contrasting with the high initial resource cost of ELK in scenarios with few logs.

Scenario 2:

Assume that a total of 7 TB of raw logs are generated in 7 days (1 TB per day) at an average log rate of 11.6 MB/s, and those logs are retained for an average of 30 days in dual storage (primary and standby). Elasticsearch recommends that the total storage space for raw logs, backup data, and index data in dual storage mode be about 2.2 times the size of raw logs. Considering the Elasticsearch cluster's uneven write distribution and partial disk utilization, storing 7 TB of raw logs needs at least 31 TB of disk space, calculated as 7 TB x 2.2 (storage expansion) x 2 (allowing for 50% disk redundancy).

Typically at least three ECSs (16 vCPUs, 64 GB memory, and 10 TB capacity) and two Kafka replicas are required for Elasticsearch to store logs of the past 12 hours.

Table 2 Self-built ELK

Category

Subcategory

Monthly Cost (Total: $2,652 USD)

Expense Proportion

Setting up Elasticsearch

3 ECSs (C6 16 vCPUs | 64 GB)

3 x 1,999 x 0.1401 = $840 USD

31.7%

EVS (high I/O 31 TB)

0.35 x 31 x 1,024 x 0.1401= $1,557 USD

58.7%

Setting up Kafka

3 ECSs (2 vCPUs | 4 GB)

3 x 208 x 0.1401 = $87 USD

3.3%

EVS (ultra-high I/O 3 x 400 GB)

1,200 x 0.1401 = $168 USD

6.3%

According to the price calculator, the monthly cost of LTS is around $3,409.92 USD, just 71% of the cost of self-built ELK. This significant saving is attributed to LTS's pay-per-use billing mode, contrasting with the extensive disk requirements for maintaining smooth running in self-built ELK clusters.

Scenario 3:

Assume that a total of 150 TB of raw logs are generated in 30 days (5 GB per day) at an average log rate of 58 MB/s, and those logs are retained for an average of 30 days in dual storage (primary and standby).

Elasticsearch recommends that the total storage space for raw logs, backup data, and index data in dual storage mode be about 2.2 times the size of raw logs. Considering the Elasticsearch cluster's uneven write distribution and partial disk utilization, storing 150 TB of raw logs needs at least 660 TB of disk space, calculated as 150 TB x 2.2 (storage expansion) x 2 (allowing for 50% disk redundancy).

Typically at least 66 ECSs (16 vCPUs, 64 GB memory, and 10 TB capacity) and two Kafka replicas are required for Elasticsearch to store logs of the past 12 hours.

Table 3 Self-built ELK

Category

Subcategory

Monthly Cost (Total: $52,440 USD)

Expense Proportion

Setting up Elasticsearch

66 ECSs (C6 16 vCPUs | 64 GB)

66 x 1,999 x 0.1401 = $18,489 USD

35.3%

EVS (high I/O 660 TB)

0.35 x 660 x 1,024 x 0.1401 = $33,149 USD

63.2%

Setting up Kafka

3 ECSs (2 vCPUs | 4 GB)

3 x 208 x 0.1401 = $87 USD

0.2%

EVS (ultra-high I/O 3 x 1,700 GB)

5,100 x 0.1401 = $715 USD

1.4%

According to the price calculator, the monthly cost of LTS is around $27,648 USD, just 28.8% of the cost of self-built ELK. This significant saving is attributed to LTS's pay-per-use billing mode, contrasting with the extensive disk requirements for maintaining smooth running in self-built ELK clusters.

Summary

LTS beats ELK in functions, performance, and costs. You are advised to use fully managed LTS instead of self-built ELK.

Parent topic: Consultation