Importing Log Files from OBS to LTS
You can transfer infrequently accessed logs from LTS to OBS for storage. To quickly search for and analyze these logs stored in OBS, you can import them back to LTS.
This practice describes how to transfer logs to OBS and import them back to LTS. The following example values are for reference only.
Prerequisites
- Logs have been ingested to LTS. For details, see Log Ingestion.
- You have created an OBS bucket. For details, see Creating a Bucket.
You will be billed for the storage you use in OBS. For details, see Huawei Cloud Pricing.
Creating a Task for Transferring Logs to OBS
- Log in to the LTS console.
- Choose Log Transfer in the navigation pane.
- Click Configure Log Transfer in the upper right corner.
- On the displayed page, configure the log transfer parameters. The following example values are for reference only.
Table 1 Transfer parameters Parameter
Description
Example Value
Source Account
Current: Logs of the current account will be transferred.
Current
Enable Transfer
Enabled by default.
Enabled
Transfer Destination
Select a cloud service to which logs are transferred.
OBS
Log Group Name
Select a log group.
lts-xunjian-group
Enterprise Project Name
Select an enterprise project.
default
Log Stream Name
Select a log stream.
Log streams that have been configured with OBS transfer settings cannot be selected.
lts-xunjian-topic
OBS Bucket
Select an OBS bucket.
bj4-xunjian
Key Name
For an encrypted OBS bucket, select a key name. If no keys are available, click Create Key and Authorize to go to the Data Encryption Workshop (DEW) console and create a key.
-
Custom Log Transfer Path
Disabled: Logs will be transferred to the default path. The default path is LogTanks/Region name/%Y/%m/%d/Log group/Log stream/Log file name.
Disabled
Log Prefix
Add a file name prefix to the log files transferred to an OBS bucket.
The prefix must meet the following requirements:
- Can contain up to 64 characters.
- Can contain only letters, digits, hyphens (-), underscores (_), and periods (.).
Example: If you enter LTS-log, the log file name will be LTS-log_Log file name.
LTS-log
Format
The storage format of logs. The value can be Raw Log Format or JSON.
- Example of the raw log format:
- Example of the JSON format:
{"host_name":"ecs-bd70","ip":"192.168.0.54","line_no":249,"message":"Sep 30 14:40:01 ecs-bd70 CRON[4363]: (root) CMD (/opt/oss/servicemgr/ICAgent/bin/manual/mstart.sh > /dev/null 2>&1)\n","path":"/var/log/syslog","time":1569825602303}
JSON
Transfer Interval
The interval for automatically transferring logs to OBS buckets. The value can be 2, 5, or 30 minutes, or 1, 3, 6, or 12 hours.
2 minutes
Time Zone
When logs are transferred to OBS buckets, the time in the transfer directory and file name will use the specified UTC time zone.
(UTC) Coordinated Universal Time
Filter by Tag Fields
During transfer, logs will be filtered by tag fields collected by ICAgent.
- Enabled: Default tag fields include those for hosts (hostIP, hostId, hostName, pathFile, and collectTime) and for Kubernetes (clusterName, clusterId, nameSpace, podName, containerName, and appName). Optional public tag fields are regionName, logStreamName, logGroupName, and projectId.
NOTE:
When Filter by Tag Fields is enabled, Format must be JSON.
- Transfer Tag: After this function is enabled, log stream tags are also transferred.
Enabled
Compressed Format
Non-compression and gzip/zip/snappy compression are supported.
NOTE:This function is available only to whitelisted users. To use it, submit a service ticket.
GZIP
- Click OK. When the log transfer status changes to Normal, the transfer task has been created.
- Wait for 2 minutes. On the Log Transfer page, click the OBS bucket name in the Transfer Destination column to switch to OBS Console. On the Objects page displayed, click the desired folder name to find the transferred log files LTS-log_xx.json.gz.
Click Download in the Operation column of these files to download them to your local PC for viewing.
Creating a Task for Importing Files from OBS to LTS
- Choose Log Ingestion > Ingestion Center in the navigation pane and click OBS (Object Storage Service).
- Select a log group from the Log Group drop-down list. In this practice, select lts-group-test.
If there is no desired log group, click Create Log Group. For details, see Creating a Log Group.
- Select a log stream from the Log stream drop-down list. In this practice, select lts-topic-test. If there is no desired log stream, click Create Log Stream. For details, see Creating a Log Stream.
- Click Next: Configurations.
- On the Configurations page, set parameters. The following example values are for reference only. For details, see .
Table 2 Configuring the collection Parameter Type
Parameter
Description
Example Value
Basic Settings
Collection Configuration Name
Enter a name containing 1 to 64 characters. Only letters, digits, hyphens (-), underscores (_), and periods (.) are allowed. It cannot start with a period or underscore, or end with a period.
test-LTS-log
Task Monitoring
Enabled by default.
It logs task execution statuses to log stream lts-system/lts-obs2lts-statistics, allowing you to view the OBS import task monitoring dashboard and configure alarm rules to promptly detect any import issues.
Enabled
OBS Data Source Configuration
OBS Bucket
Select the OBS bucket from which log files are imported to LTS.
bj4-xunjian
Folder Prefix
Enter the prefix (your_prefix/) or the full path (your_prefix/file.gz) of the OBS files to be imported for better locating. Only files with an original size (before compression) of no larger than 5 GB can be imported.
On the Objects page of OBS Console, locate the file to be imported to LTS (LTS-log_LogTank_037bcc7382bf4bf2bb4068684a3275da_2025-01-20T11-02-00Z_2273089614c402d6.json.gz). Click More > Copy Path in the Operation column of this file. The copied path is LogTanks/cn-north-4/2025/01/20/lts-xunjian-group/lts-xunjian-topic/LTS-log_LogTank_037bcc7382bf4bf2bb4068684a3275da_2025-01-20T11-02-00Z_2273089614c402d6.json.gz. Therefore, the folder prefix is LogTanks/cn-north-4/2025/01/20/lts-xunjian-group/lts-xunjian-topic.
LogTanks/cn-north-4/2025/01/20/lts-xunjian-group/lts-xunjian-topic
Regular Expression for File Filtering
Enter a regular expression for filtering files, so that only files with names matching the regular expression will be imported. If no regular expression is specified, files are not filtered.
In this practice, enter the file name, indicating that only the content of this file is imported.
LTS-log_LogTank_037bcc7382bf4bf2bb4068684a3275da_2025-01-20T11-02-00Z_2273089614c402d6.json.gz
Compressed Format
Auto check, non-compression, and gzip/zip/snappy compression are supported. If you select ZIP, only zip packages containing a single file without any folders are supported.
Auto check
Import Interval
One-off: LTS imports files only once and does not detect new files.
One-off
Data Format Configuration
Log File Code
The log file encoding format can be UTF-8 or GBK.
UTF-8
Extraction Mode
Select an extraction policy based on your log type for LTS to parse logs in OBS files.
When parsing logs exceeding 1 MB in OBS, LTS applies the following rules based on the log type:
- If a single-line log exceeds 1 MB, the excess part of that line will be truncated and discarded.
- If a multi-line block in a multi-line log exceeds 1 MB, the excess part of that block will be truncated and discarded.
- Logs in ORC and JSON formats are parsed as single lines. If a single line exceeds 1 MB, the entire line will be discarded.
JSON
- Click Preview to preview the file content. After confirming that the file content is correct, click Submit.
- The ingestion configuration is created. Go to the Ingestion Management page to view it.
- Click the log stream name lts-topic-test to go to its details page and check whether there is data in it. If data exists, the transferred files have been imported to LTS. Compare the log data in log stream lts-xunjian-topic (selected for transferring LTS logs to OBS) with that in log stream lts-topic-test (selected for importing OBS log files to LTS). The log data should be the same.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot
