Configuring Log Collection

Updated on 2024-05-25 GMT+08:00

View PDF

CAE provides log collection. Currently, logs can be collected only to LTS. You can configure the log collection path. In advanced settings, you can configure the log format (single-line (default) or multi-line).

This section uses Kafka as an example to describe how to customize log paths.

NOTICE:

If log files are mounted to system directories such as / and /var/run, components may not work. An empty directory is recommended. If the directory is not empty, ensure that the directory does not contain any file that affects component startup. Otherwise, the files will be replaced, causing component startup exceptions. As a result, the component fails to be created.
The log path must contain the log file name, for example, /var/log/test/error.log. To use a wildcard, you must also specify the extension, for example, /var/log/test/*.log, not /var/log/test/* or /var/log/test/*.*.

The log files must be text files.
By default, standard logs are collected for all components and are stored in the stdout file.
The log path must be different from the cloud storage configuration path.
An application in CAE maps to a log group in LTS. Creating a CAE application creates a log group.

Adding Log Collection

NOTE:

You can configure up to 20 log collection paths.

Log in to CAE.
Choose Component Configurations.
Select the target component from the drop-down list in the upper part of the page.
Click Edit in the Log Collection module.
Click Add Log Path.
Enter the path of the logs to be collected in Log Collection Path. For example: /var/log/springboot.log

Figure 1 Configuring a log path
Click Save.

Configure advanced settings

By default, the system collects and displays the logs printed by the program by line. If a complete log occupies multiple lines and you want to collect and display the entire log, you can set Log Format to enable multi-line logs.

Single-line: The system collects logs by line.

Multi-line: Multiple lines are merged into one line. The system collects logs by configured matching rules. The log lines that do not meet the matching rules will be merged with the lines that last met the matching rules.

If you select Log time, the time matching mode is used. If you select Regular expression, the regular expression matching mode is used.

Time Wildcard: Enter the time wildcard when Log Segmentation is set to Log time.

For example, if the time format of each log is YYYY-MM-DD hh:mm:ss, set the time wildcard to YYYY-MM-DD hh:mm:ss.

Example time wildcard:

YY   - year (19)
YYYY - year (2019)
M    - month (1)      
MM   - month (01)
D    - day (1) 
DD   - day (01)
hh   - hours (23)
mm   - minutes (59)
ss   - seconds (59) 
SSS  - millisecond (999)
hpm     - hours (03PM)
h:mmpm    - hours:minutes (03:04PM) 
h:mm:sspm  - hours:minutes:seconds (03:04:05PM)
hh:mm:ss ZZZZ (16:05:06 +0100)
hh:mm:ss ZZZ  (16:05:06 CET)   
hh:mm:ss ZZ   (16:05:06 +01:00)

Regular expression: If Log Segmentation is set to Regular expression, enter the regular expression based on the format of the beginning of each log.

Example regular expression:

Example 1:

19:41:33.217 [http-nio-8000-exec-1] ERROR o.a.c.c.C.[.[localhost].[/].[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.NullPointerException: Cannot invoke "com.example.springboothello.controller.HelloController.write()" because "helloController" is null] with root cause
java.lang.NullPointerException: Cannot invoke "com.example.springboothello.controller.HelloController.write()" because "helloController" is null
	at com.example.springboothello.controller.HelloController.nullPointException(HelloController.java:234)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)

Regular expression for merging the preceding logs into one line: ^\d{2}:\d{2}:\d{2}.

All lines that do not start with time are merged to the previous line.

Example 2:

Exception in thread "main" java.lang.IllegalStateException: A book has a null property
       at com.example.myproject.Author.getBookIds(Author.java:38)
       at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Caused by: java.lang.NullPointerException
       at com.example.myproject.Book.getId(Book.java:22)
       at com.example.myproject.Author.getBookIds(Author.java:35)
       ... 1 more

Regular expression for merging the preceding logs into one line: ^Exception.

All lines that do not start with Exception are merged to the previous line.

The page provides regular expression verification. You can copy the logs to Log Example, enter a regular expression, and click Verify to check whether the regular expression matches.

Click OK.
Make the configurations take effect.
- If the component has been deployed, click Activate Settings in the upper part of the page. In the dialog box displayed on the right, confirm the configurations and click OK for the configurations to take effect.
- If the component has not been deployed, click Set and Deploy Component in the upper part of the page. In the dialog box displayed on the right, click OK. After the deployment is complete, the configurations take effect.
  After the configurations take effect, you can view component logs in a specified path on the Component Logs page

Modifying a Log Path

Log in to CAE.
Choose Component Configurations.
Select the target component from the drop-down list in the upper part of the page.
Click Edit in the Log Collection module.
Select the target path and click Edit in the Operation column.
Reconfigure the log collection path, for example, /var/log/CAE/logs/*.out.

Figure 2 Modifying a log path
(Optional) Reconfigure the log collection format as required.
Click Save and OK.
Click Activate Settings in the upper part of the page. In the dialog box displayed on the right, confirm the configurations and click OK for the configurations to take effect.
View the logs in the new path.

Deleting a Log Path

Log in to CAE.
Choose Component Configurations.
Select the target component from the drop-down list in the upper part of the page.
Click Edit in the Log Collection module.
Select the target path and click Delete in the Operation column.
In the displayed dialog box, click Yes. After deleting the path, click OK.

Figure 3 Deleting a log path
Click Activate Settings in the upper part of the page. In the dialog box displayed on the right, confirm the configurations and click OK for the configurations to take effect.

NOTE:

Deleted path files no longer collect logs. But you can still view the historical logs of the corresponding log file.