Configuring Log Collection

Constraints

• If log files are mounted to system directories such as / and /var/run, components may not work. An empty directory is recommended. If the directory is not empty, ensure that the directory does not contain any file that affects component startup. Otherwise, the files will be replaced, causing component startup exceptions. As a result, the component fails to be created.
• The log path must contain the log file name, for example, /var/log/test/error.log. To use a wildcard, you must also specify the extension, for example, /var/log/test/*.log, not /var/log/test/* or /var/log/test/*.*.
• You can configure up to 20 paths.
• The log files must be text files.
• By default, standard logs are collected for all components and are stored in the stdout file.
• The log path must be different from the cloud storage configuration path.
• Once the log collection path takes effect, a subdirectory named after the pod where the current component is located is created in the cloud storage subpath.
• The log collection path is read from the cloud storage configuration and must support the write permission. If umask is set, the read and write permissions must be supported. That is, the second digit cannot be 2, 4, or 6.
• A maximum of five levels of subdirectories can be checked.
• There are limitations on specifications for local disk storage. If the storage specifications exceeded, the component will be restarted. For details about the storage specifications, see Specifications.

Adding Log Collection

You can configure up to 20 log collection paths.

1. Log in to CAE.
2. Choose Component Configurations.
3. Select the target component from the drop-down list in the upper part of the page.
4. Click Edit in the Log Collection module.
5. Click Add Log Collection Path.
6. Enter the path of the logs to be collected in Log Collection Path by referring to Table 1. For example: /var/log/springboot.log

   Table 1 Collection path parameters

   Parameter	Description
   Log Collection Type	You can select Cloud Storage or Local Disk from the log collection type drop-down list. Cloud Storage and Local Disk are two different log data sources, which determine the location from which the system obtains the log files to be collected. Cloud Storage: the system collects log files from Huawei Cloud storage services, such as Object Storage Service (OBS) and Scalable File Service (SFS). Local Disk: the system collects log files stored on the server where the CAE service is deployed, or collects log files stored on the physical or virtual hard disk directly connected to the VM.
   Log Collection Path	If the log collection type is Cloud Storage, select the configured cloud storage from the drop-down list and enter the path of the logs to be collected. If the cloud storage is not displayed in the drop-down list, configure the cloud storage. For details, see Configuring Cloud Storage. If the log collection type is Local Disk, enter the path of the logs to be collected. The log path must contain the log file name, for example, /var/log/test/error.log. To use a wildcard, you must also specify the extension, for example, /var/log/test/*.log, not /var/log/test/* or /var/log/test/*.*.

7. Click Save.
8. Configure advanced settings
   By default, the system collects and displays the logs printed by the program by line. If a complete log occupies multiple lines and you want to collect and display the entire log, you can set Log Format to enable multi-line logs.

   • Single-line: The system collects logs by line.
   • Multi-line: Multiple lines are merged into one line. The system collects logs by configured matching rules. The log lines that do not meet the matching rules will be merged with the lines that last met the matching rules.
     • If you select Log time, the time matching mode is used. If you select Regular expression, the regular expression matching mode is used.

       Time Wildcard: Enter the time wildcard when Log Segmentation is set to Log time.

       For example, if the time format of each log is YYYY-MM-DD hh:mm:ss, set the time wildcard to YYYY-MM-DD hh:mm:ss.

       Example time wildcard:

       YY   - year (19)
       YYYY - year (2019)
       M    - month (1)      
       MM   - month (01)
       D    - day (1) 
       DD   - day (01)
       hh   - hours (23)
       mm   - minutes (59)
       ss   - seconds (59) 
       SSS  - millisecond (999)
       hpm     - hours (03PM)
       h:mmpm    - hours:minutes (03:04PM) 
       h:mm:sspm  - hours:minutes:seconds (03:04:05PM)
       hh:mm:ss ZZZZ (16:05:06 +0100)
       hh:mm:ss ZZZ  (16:05:06 CET)   
       hh:mm:ss ZZ   (16:05:06 +01:00)

     • Regular expression: If Log Segmentation is set to Regular expression, enter the regular expression based on the format of the beginning of each log.

       Example regular expression:

       Example 1:

       19:41:33.217 [http-nio-8000-exec-1] ERROR o.a.c.c.C.[.[localhost].[/].[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.NullPointerException: Cannot invoke "com.example.springboothello.controller.HelloController.write()" because "helloController" is null] with root cause
       java.lang.NullPointerException: Cannot invoke "com.example.springboothello.controller.HelloController.write()" because "helloController" is null
       	at com.example.springboothello.controller.HelloController.nullPointException(HelloController.java:234)
       	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
       	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       	at java.base/java.lang.reflect.Method.invoke(Method.java:568)

       Regular expression for merging the preceding logs into one line: ^\d{2}:\d{2}:\d{2}.

       All lines that do not start with time are merged to the previous line.

       Example 2:

       Exception in thread "main" java.lang.IllegalStateException: A book has a null property
              at com.example.myproject.Author.getBookIds(Author.java:38)
              at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
       Caused by: java.lang.NullPointerException
              at com.example.myproject.Book.getId(Book.java:22)
              at com.example.myproject.Author.getBookIds(Author.java:35)
              ... 1 more

       Regular expression for merging the preceding logs into one line: ^Exception.

       All lines that do not start with Exception are merged to the previous line.

       The page provides regular expression verification. You can copy the logs to Log Example, enter a regular expression, and click Verify to check whether the regular expression matches.

9. Click OK.
10. Make the configurations take effect.
    • If the component has been deployed, click Activate Settings in the upper part of the page. In the dialog box displayed on the right, confirm the configurations and click OK for the configurations to take effect.
    • If the component has not been deployed, click Set and Deploy Component in the upper part of the page. In the dialog box displayed on the right, click OK. After the deployment is complete, the configurations take effect.

      After the configurations take effect, you can view component logs in a specified path on the Component Logs page

Modifying a Log Path

1. Log in to CAE.
2. Choose Component Configurations.
3. Select the target component from the drop-down list in the upper part of the page.
4. Click Edit in the Log Collection module.
5. Select the target path and click Edit in the Operation column.
6. Reconfigure the log collection path, for example, /var/log/CAE/logs/*.out.
7. (Optional) Reconfigure the log collection format as required.
8. Click Save and OK.
9. Click Activate Settings in the upper part of the page. In the dialog box displayed on the right, confirm the configurations and click OK for the configurations to take effect.
10. View the logs in the new path.

Deleting a Log Path

1. Log in to CAE.
2. Choose Component Configurations.
3. Select the target component from the drop-down list in the upper part of the page.
4. Click Edit in the Log Collection module.
5. Select the target path and click Delete in the Operation column.
6. In the displayed dialog box, click Yes. After deleting the path, click OK.
7. Click Activate Settings in the upper part of the page. In the dialog box displayed on the right, confirm the configurations and click OK for the configurations to take effect.
   

   Deleted path files no longer collect logs. But you can still view the historical logs of the corresponding log file.