Analyzing Application Run Logs in Log4j Format on the LTS Console

Introduction

Log4j is Apache's open-source project used for logging. We can calculate the number and proportion of logs at different levels, or gather statistics on services from run logs.

For example, you can know the transaction volume of an offering on a day from logs such as the following:

2020-12-28_21:10:48.081 [http-nio-8083-exec-6] INFO  discounted shoes - num is :9

Analyzing Application Run Logs in Log4j Format on the LTS Console

1. Log in to the LTS console and choose Log Ingestion in the navigation pane.
2. Click Elastic Cloud Server (ECS) to configure log ingestion.

3. Select a log stream.
   1. Select a log group from the drop-down list of Log Group. If there are no desired log groups, click Create Log Group to create one.
   2. Select a log stream from the drop-down list of Log Stream. If there are no desired log streams, click Create Log Stream to create one.
   3. Click Next: (Optional) Select Host Group.

4. Select host groups.
   1. Select one or more host groups from which you want to collect logs. If there are no desired host groups, click Create above the host group list to create one. For details, see Managing Host Groups.
      

      You can also skip this step, but the collection configuration will not take effect. You are advised to select a host group during the first ingestion configuration. If you skip this step, follow either of the following ways to configure host groups after the ingestion configuration is complete:

      • On the Ingestion Rule tab page, click Edit in the Operation column. On the displayed page, select required host groups.
      • Choose Host Management in the navigation pane, click the Host Groups tab, and make the association, or
   2. Click Next: Collection Configuration.

5. Configure the collection.
   1. Configure the collection parameters. For details, see Configuring Collection.
   2. Click Next: Index Settings. Retain the default settings.
   3. Click Submit. The ingestion is successful.

6. On the log stream details page, click . On the Cloud Structuring Parsing page, select Regular Expressions, select a log event, and extract four fields: Time1, ThreadName, Level, and Message.
7. On the log stream details page, click Log Analysis and run SQL queries. For details about how to visualize query results, see Log Structuring.
   • To query the error type distribution in the last seven days, run the following SQL statement:

      SELECT Level, count(*) as Number group by Level

   • To query the running threads in the last 5 minutes, set the time range to Last 5 minutes and run the following SQL statement:

     SELECT distinct(ThreadName)

   • To query the total transaction volume of a product, run the following SQL statement:

     SELECT sum(cast(regexp_extract(Message, 'num is\s:(?<Total>[\d]+)', 1) as double)) as Total WHERE Message like '%shoes%'