Analyzing Application Run Logs (in Log4j Format)

Introduction

Log4j is Apache's open-source project used for logging. We can calculate the number and proportion of logs at different levels, or gather statistics on services from run logs.

For example, you can know the transaction volume of an offering on a day from logs such as the following:

2020-12-28_21:10:48.081 [http-nio-8083-exec-6] INFO  discounted shoes - num is :9

Procedure

1. Log in to the LTS console and choose Log Ingestion in the navigation pane.
2. Click Ingest Log in the upper right corner.

   

3. On the Select Ingestion Type page, select Host.

   

4. Select a log stream.
   1. Select a log group from the drop-down list of Log Group. If there are no desired log groups, click Create Log Group to create one.
   2. Select a log stream from the drop-down list of Log Stream. If there are no desired log streams, click Create Log Stream to create one.
   3. Click Next: Select Host Group.

      

5. Select host groups.
   1. Select one or more host groups from which you want to collect logs. If there are no desired host groups, click Create above the host group list to create one. For details, see "Managing Host Groups".
      NOTE:

      You can choose not to select a host group in this step, but associate a host group with the ingestion configuration after you finish the procedure here. To do this, either:

      • Choose Host Management in the navigation pane, click the Host Groups tab, and make the association, or
      • Choose Log Ingestion in the navigation pane, click an ingestion configuration, and make the association on the details page.
   2. Click Next: Collection Configuration.

      

6. Configure the collection.
   1. Configure the collection parameters. For details, see "Configuring Collection".
   2. Click Submit.

7. On the log stream details page, choose Log Configuration in the navigation pane and click the Log Structuring tab. On the page displayed, select Regular Expressions, select a log event, and extract four fields: Time1, ThreadName, Level, and Message, as shown in Figure 1.
   Figure 1 Structuring logs with regular expressions
   

8. On the log stream details page, click Visualization and run SQL queries. For details about how to visualize query results, see "Log Structuring".
   • To query the error type distribution in the last seven days, run the following SQL statement:

      SELECT Level, count(*) as Number group by Level
     Figure 2 Error type distribution
     
   • To query the running threads in the last 5 minutes, set the time range to Last 5 minutes and run the following SQL statement:

     SELECT distinct(ThreadName)
     Figure 3 Running threads
     
   • To query the total transaction volume of a product, run the following SQL statement:

     SELECT sum(cast(regexp_extract(Message, 'num is\s:(?<Total>[\d]+)', 1) as double)) as Total WHERE Message like '%shoes%'

     The query looks for fuzzy match, and the following is an example query result.

     Figure 4 Total transaction volume