LTS Log4j2 SDK

The LTS Java SDK provides an extension plug-in compatible with log4j 2. You can configure a Huawei Cloud appender in log4j 2 to report generated logs to LTS.

Transport Protocol

HTTPS

Prerequisites

You have registered a Huawei account and enabled Huawei Cloud services.
You have obtained the name of the region where LTS is deployed.
You have obtained the AK/SK of your HUAWEI ID.
You have obtained a project ID of your Huawei Cloud account. For details, see API Credentials.
You have obtained the IDs of the target log group and log stream in LTS.
You have installed the JDK. LTS Java SDK supports JDK 1.8 or later. You can run the java –version command to check the installed JDK version. If JDK is not installed, download the installation package from the Java official website and install it.
The Log4j2 SDK supports cross-cloud and local log reporting. This function is available only in regions CN North-Beijing4, CN East-Shanghai1, CN South-Guangzhou, and CN Southwest-Guiyang1.
When logs are reported to LTS via the Log4j2 SDK, the time difference between the log reporting time you set and the current time must not exceed two days. Otherwise, the reported logs will be deleted by LTS.

Installing the SDK

Add dependencies to the Maven project.

To use Maven to build a project, add the Huawei external open-source repository as a mirror in the settings.xml file. Example:

<mirror>
    <id>huaweicloud</id>
    <mirrorOf>*</mirrorOf>
    <url>https://repo.huaweicloud.com/repository/maven/</url>
</mirror>

Obtain the latest version of the LTS log4j2 SDK dependencies from the Huawei external open-source repository.
To use the LTS log4j2 SDK in a Maven project, you only need to add the corresponding dependencies to the pom.xml file. The Maven project management tool automatically downloads the related JAR packages. Add the following content to <dependencies>.
The latest version (1.1.3) is recommended. You can check the SDK version in step 2.
```
<dependency> 
     <groupId>io.github.huaweicloud</groupId> 
     <artifactId>lts-sdk-common</artifactId> 
     <version>version</version> 
 </dependency> 
 <dependency> 
     <groupId>io.github.huaweicloud</groupId> 
     <artifactId>lts-sdk-log4j2</artifactId> 
     <version>version</version> 
 </dependency>
```

Sample Code

Example of log4j2.xml:

<?xml version="1.0" encoding="UTF-8"?> 
 <Configuration status="WARN"> 
  
     <Appenders> 
         <!-- projectId: project ID of the Huawei Cloud account. -->
         <!-- logGroupId: Huawei Cloud LTS log group ID. -->
         <!-- logStreamId: Huawei Cloud LTS log stream ID. -->
         <!-- accessKeyId: AK of the Huawei Cloud account. Hard-coded or plaintext AK is risky. For security, encrypt your AK. Decrypt it only when necessary. -->
         <!-- accessKeySecret: SK of the Huawei Cloud account. Hard-coded or plaintext SK is risky. For security, encrypt your SK. Decrypt it only when necessary. -->
         <!-- regionName: region where LTS is deployed. -->
         <!-- totalSizeInBytes: maximum size of logs that can be cached by a single appender. --> 
         <!-- maxBlockMs: blocking time when the producer sends logs. -->
         <!-- ioThreadCount: size of the thread pool for executing log sending tasks. -->
         <!-- batchSizeThresholdInBytes: maximum size of logs sent by the producer in a batch. -->
         <!-- batchCountThreshold: maximum number of logs sent by the producer in a batch. -->
         <!-- lingerMs: waiting time for the producer to send logs in a batch. -->
         <!-- retries: number of retries after the producer fails to send logs. -->
         <!-- baseRetryBackoffMs: backoff time for the first retry. -->
         <!-- maxRetryBackoffMs: maximum backoff time for retry. -->
         <!-- giveUpExtraLongSingleLog: If a log exceeds 1 MB, the log is split into two parts: 1 MB is retained and the excess part is discarded. -->
         <!-- enableLocalTest: If it is set to true, logs can be reported across clouds. If it is set to false (default), logs can be reported only from Huawei Cloud ECSs. -->
         <!-- isLogGroupStreamByName: Whether logs are reported using log group and log stream names. -->
         <ltsLog name="log4j2sdkappender" 
                 projectId="xxx" 
                 logGroupId="xxx" 
                 logStreamId="xxx" 
                 accessKeyId="xxx" 
                 accessKeySecret="xxx" 
                 regionName="xxx" 
                 totalSizeInBytes="104857600" 
                 maxBlockMs="0" 
                 ioThreadCount="8" 
                 batchSizeThresholdInBytes="524288" 
                 batchCountThreshold="4096" 
                 lingerMs="2000" 
                 retries="5" 
                 baseRetryBackoffMs="500" 
                 maxRetryBackoffMs="500" 
                 giveUpExtraLongSingleLog="true"
                 enableLocalTest="false"
                 isLogGroupStreamByName="false"
                 >
             <PatternLayout pattern="%d %-5level [%thread] %logger{0}: %msg"/> 
         </ltsLog> 
     </Appenders> 
  
     <Loggers> 
         <Root level="WARN"> 
             <AppenderRef ref="log4j2sdkappender"/> 
         </Root> 
     </Loggers> 
  
 </Configuration>

Logs generated during code compilation:

package com.huawei.log; 
 
import org.apache.logging.log4j.LogManager; 
import org.apache.logging.log4j.Logger; 
 
public class TestLog4j2AppenderInfo { 
 
    private static final Logger LOGGER = LogManager.getLogger(TestLog4j2AppenderInfo.class); 
 
    public static void main(String[] args) { 
        LOGGER.info("test~log4j2~~~info"); 
    } 
 
}

Producer Performance Baseline

If one of the following performance baseline values is exceeded, log reporting may be abnormal.

Recommended ECS configurations:

Instance flavor: General computing-plus c7.xlarge.2
CPU: 4 vCPUs
Memory: 8 GB
Assured bandwidth: 100 Mbit/s
OS: Huawei Cloud EulerOS release 2.0
JVM: OpenJDK 64-Bit Server VM (build 17.0.7+7, mixed mode)

Test procedure (for a single producer):

totalSizeInBytes: 104857600
maxBlockMs: 0
batchSizeThresholdInBytes: 1048576
batchCountThreshold: 40960
lingerMs: 2000
ioThreadCount: specific to test cases

Initial/Maximum heap size of the JVM: 1 GB
Total number of sent logs: 100,000,000
Total size of sent logs: about 50 GB

After the parameters are set based on the baseline values, use a Huawei Cloud ECS as the log reporting environment and report logs via the Huawei Cloud intranet service entry.

Example test results of the SDK reporting performance baseline:

Reported logs: In this test, each log is about 550 bytes. To simulate random data, the test uses random strings as log data.

The following table lists the reference values of the performance baseline.

**Table 1** Performance baseline
I/O Threads	Data Throughput	Data Throughput Rate	CPU Usage
2	8 MB/s	16,000 logs/s	15%
4	15.2 MB/s	30,000 logs/s	29%
6	20.7 MB/s	41,000 logs/s	36%

Summary:

Most of the CPU time is spent on object serialization and compression, and CPU usage increases as throughput rises. However, in a typical daily environment, the average data traffic of a single ECS is 100 KB/s, resulting in negligible CPU consumption.
Increasing the number of I/O threads significantly improves throughput, especially when the number of I/O threads is less than the number of available processors.
Increasing the totalSizeInBytes value has minimal impact on the throughput but can increase CPU usage. You are advised to use the default value.
If the reported logs exceed the capacity of a single producer:
1. To ensure steady SDK log reporting, you are advised to separate logs into multiple log streams and use multiple producers to distribute the traffic.
2. If maxBlockMs is set to 0, the SDK will be in a non-blocking state, which will trigger the protection mechanism to automatically downgrade. As a result, some logs may be discarded.
3. If maxBlockMs is set to a value greater than 0, the SDK will be in a blocked state with a blocking duration of maxBlockMs, potentially blocking the SDK's log sending.

Parameters

Configure the following Appender parameters.

**Table 2** Appender parameters
Parameter	Description	Type	Mandatory	Default Value
projectId	Project ID of the Huawei Cloud account.	String	Yes	-
accessKeyId	AK of the Huawei Cloud account. Hard-coded or plaintext AK is risky. For security, encrypt your AK. Decrypt it only when necessary.	String	Yes	-
accessKeySecret	SK of the Huawei Cloud account. Hard-coded or plaintext SK is risky. For security, encrypt your SK. Decrypt it only when necessary.	String	Yes	-
regionName	Region where LTS is deployed.	String	Yes	-
logGroupId	LTS log group ID.	String	Yes	-
logStreamId	LTS log stream ID.	String	Yes	-
totalSizeInBytes	Maximum size of logs that can be cached in a producer instance.	int	No	104857600 (indicates 100 MB.)
maxBlockMs	Maximum blocking time (in milliseconds) that the caller will wait on the send method when the producer's available space is insufficient. The default value is 60,000 ms. The recommended value is 0 ms. If maxBlockMs is set to a value greater than or equal to 0, the blocking time is enforced. If the memory space is still insufficient after the blocking time expires, an error will be reported and the logs will be discarded. If maxBlockMs is set to -1, the blocking continues until the logs are successfully sent to LTS and the logs will not be discarded.	long	No	60,000 ms
ioThreadCount	Size of the thread pool for executing log sending tasks.	int	No	Number of idle CPUs on the customer's host. The value must be greater than 1.
batchSizeThresholdInBytes	Once the size of logs cached in a ProducerBatch reaches or exceeds the batchSizeThresholdInBytes value, they are sent to LTS.	int	No	524288 (indicates 0.5 MB.)
batchCountThreshold	Once the number of logs cached in a ProducerBatch reaches or exceeds the batchCountThreshold value, they are sent to LTS.	int	No	4096
lingerMs	Duration from the time when a ProducerBatch is created to the time when the logs can be sent.	int	No	2s
retries	Number of allowed retry attempts for a ProducerBatch when the first attempt to send the logs cached in the ProducerBatch fails. The recommended value is 5. If the value of retries is less than or equal to 0, the ProducerBatch will be added to the failure queue after the first sending attempt fails.	int	No	10
baseRetryBackoffMs	Backoff time for the first retry attempt.	long	No	0.1s
maxRetryBackoffMs	Maximum backoff time for retry attempts.	long	No	50s
giveUpExtraLongSingleLog	If a log exceeds 1 MB, the log is split into two parts: 1 MB is retained and the excess part is discarded.	boolean	No	false
enableLocalTest	Whether to enable cross-cloud log reporting. (This function is used only for development and commissioning. The reporting performance is not guaranteed.) If this parameter is set to true, cross-cloud log reporting is enabled and users access LTS through the public network. This function is available only in regions CN North-Beijing4, CN East-Shanghai1, CN South-Guangzhou, and CN Southwest-Guiyang1. If this parameter is set to false, cross-cloud log reporting is disabled and users access LTS through hosts in the current Huawei Cloud region.	boolean	No	false
isLogGroupStreamByName	Whether to report logs based on log group or log stream names. This function is supported only by SDK 1.1.0 or later. If the value is true, logs are reported by log group name and log stream name. Set logGroupId to the log group name and logStreamId to the log stream name.	boolean	No	false
proxyIp	User-defined IP address and port number for log reporting.	String	No	-

Obtaining Parameters

Regions

**Table 3** Regions
Region Name	Region
CN North-Beijing2	cn-north-2
CN North-Beijing4	cn-north-4
CN North-Beijing1	cn-north-1
CN East-Shanghai2	cn-east-2
CN East-Shanghai1	cn-east-3
CN South-Guangzhou	cn-south-1
CN South-Shenzhen	cn-south-2
CN Southwest-Guiyang1	cn-southwest-2
AP-Singapore	ap-southeast-3

To obtain a log group's ID, choose Log Management in the navigation pane of the LTS console and hover over the target log group's name.
To obtain a log stream's ID, click next to the corresponding log group and hover over the target log stream's name.