Transferring Logs to OBS

Billing

LTS charges you based on the actual volume of logs transferred. For details, see LTS Pricing Details. In addition, you will be billed for your OBS storage usage. For details, see OBS Pricing Details.

Prerequisites

• Logs have been ingested to LTS. For details, see Log Ingestion.
• You have created an OBS bucket.

Configuring Log Transfer

LTS allows you to choose scheduled or one-time transfers.

Configuring a Scheduled Log Transfer Task

• When creating a log transfer task, you must have the following permissions in addition to the LTS permissions: setting a bucket ACL (obs:bucket:PutBucketAcl), listing buckets (obs:bucket:ListAllMyBuckets), obtaining the bucket metadata (obs:bucket:HeadBucket), and obtaining a bucket ACL (obs:bucket:GetBucketAcl), and obtaining bucket encryption configuration(obs:bucket:GetEncryptionConfiguration). For details, see Bucket Actions.
• LTS transfers only logs generated after the transfer task is configured, not historical logs.

1. Log in to the LTS console.
2. Choose Log Transfer in the navigation pane.
3. Click Configure Log Transfer in the upper right corner.
4. On the displayed page, configure the log transfer parameters.
   Figure 1 Scheduled transfer
   

   Table 1 Transfer parameters

   Parameter	Description	Example Value
   Source Account	Current: Logs of the current account will be transferred. Other: Logs of the delegator account will be transferred. Ensure that the delegator has created an agency for log transfer delegation in IAM. For details, see Creating an Agency.	Current
   Agency Name	This parameter is required when Log Source Account is set to Other. Enter the name of the agency created by the delegator in IAM.	-
   Delegator Account Name	This parameter is required when Log Source Account is set to Other. Enter the account name of the delegator.	-
   Enable Transfer	Enabled by default.	Enabled
   Transfer Destination	Select a cloud service to which logs are transferred.	OBS
   Log Group Name	Select a log group.	-
   Log Stream Name	Select a log stream. Log streams that have been configured with OBS transfer settings cannot be selected. To avoid transferring empty files, ensure that raw logs have been uploaded to the selected log stream.	-
   OBS Bucket	Select an OBS bucket. If no OBS buckets are available, click View OBS Bucket to access the OBS console and create an OBS bucket. If encryption has been enabled for the selected OBS bucket, select a key name and select I agree to grant permissions on Key Management Service (KMS) to LTS so LTS can create and use keys to encrypt and decrypt transferred logs. The selected bucket will grant the read and write permissions to LTS. Exercise caution when modifying the bucket policy to prevent log transfer failures. LTS supports OBS buckets of the Standard storage class and restored Archive storage class.	-
   Key Name	For an encrypted OBS bucket, select a key name. If no keys are available, click Create Key and Authorize to go to the Data Encryption Workshop (DEW) console and create a key.	-
   Custom Log Transfer Path	Enabled: Logs will be transferred to a custom path to separate transferred log files of different log streams. The format is /LogTanks/Region name/%Group name/%Stream name/Custom path. The default custom path is lts/%Y/%m/%d, where %Y indicates the year, %m indicates the month, and %d indicates the day. A custom path must meet the following requirements: Must start with /LogTanks/Region name. %Group name indicates the log group, and %Stream name indicates the log stream. Can contain only letters, digits, and the following special characters: &$@;:,=+?-._/ %. The character % can only be followed only by Y (year), m (month), d (day), H (hour), and M (minute). Any number of characters can be added before and after %Y, %m, %d, %H, and %M, and the sequence of these variables can be changed. Can contain 1–128 characters. Example: If you enter LTS-test/%Y/%m/%d/%H/%M, the path is LogTanks/Region name/LTS-test/Y/m/d/H/M/Log file name. If you enter LTS-test/%d/%H/%m/%Y, the path is LogTanks/Region name/LTS-test/d/H/m/Y/Log file name. Disabled: Logs will be transferred to the default path. The default path is LogTanks/Region name/2019/01/01/Log group/Log stream/Log file name.	LTS-test/%GroupName/%StreamName/%Y/%m/%d/%H/%M
   Log Prefix	Add a file name prefix to the log files transferred to an OBS bucket. The prefix must meet the following requirements: Can contain up to 64 characters. Can contain only letters, digits, hyphens (-), underscores (_), and periods (.). Example: If you enter LTS-log, the log file name will be LTS-log_Log file name.	LTS-log
   Format	The storage format of logs. The value can be Raw log format or JSON. Example of the raw log format: (Logs displayed on the LTS console are in the raw format.) Sep 30 07:30:01 ecs-bd70 CRON[3459]: (root) CMD (/opt/oss/servicemgr/ICAgent/bin/manual/mstart.sh > /dev/null 2>&1) Example of the JSON format: {"host_name":"ecs-bd70","ip":"192.168.0.54","line_no":249,"message":"Sep 30 14:40:01 ecs-bd70 CRON[4363]: (root) CMD (/opt/oss/servicemgr/ICAgent/bin/manual/mstart.sh > /dev/null 2>&1)\n","path":"/var/log/syslog","time":1569825602303}	JSON
   Transfer Interval	The interval for automatically transferring logs to OBS buckets. The value can be 2, 5, or 30 minutes, or 1, 3, 6, or 12 hours.	3 hours
   Time Zone	When logs are transferred to OBS buckets, the time in the transfer directory and file name will use the specified UTC time zone.	(UTC) Coordinated Universal Time
   Filter by Tag Fields	During transfer, logs will be filtered by tag fields collected by ICAgent. Disabled: Logs will not be filtered by tag fields. Enabled: The default tag fields include those for hosts (hostIP, hostId, hostName, pathFile, and collectTime) and for Kubernetes (clusterName, clusterId, nameSpace, podName, containerName, and appName). Optional public tag fields are regionName, logStreamName, logGroupName, and projectId. When Filter by Tag Fields is enabled, Format must be JSON. Transfer Tag: After this function is enabled, log stream tags are also transferred.	Enabled
   Compressed Format	Non-compression and gzip/zip/snappy compression are supported. NOTE: This function is available only to whitelisted users. To use it, submit a service ticket.	GZIP

5. Click OK. When the log transfer status changes to Normal, the transfer task has been created.
6. Click the OBS bucket name in the Transfer Destination column to switch to OBS Console and view the transferred log files.

   Transferred logs can be downloaded from OBS to your local PC for viewing. Logs stored in OBS are in raw or JSON format.

   Figure 2 Transferring logs to OBS
   

Configuring a One-off Log Transfer Task

This function is available only in regions CN East-Shanghai1, CN East-Shanghai2, CN North-Beijing4, CN South-Guangzhou, and AP-Singapore. To use it in other regions, submit a service ticket.

1. Click Configure Log Transfer in the upper right corner.
2. On the displayed page, configure the log transfer parameters.
   Figure 3 One-time transfer
   

   Table 2 Transfer parameters

   Parameter	Description	Example Value
   Transfer Mode	One-time: Logs are transferred to OBS for long-term storage in a one-time way.	One-time
   Transfer Destination	Select a cloud service to which logs are transferred.	OBS
   Log Group Name	Select a log group.	-
   Log Stream Name	Select a log stream. Log streams that have been configured with OBS transfer settings cannot be selected. To avoid transferring empty files, ensure that raw logs have been uploaded to the selected log stream.	-
   Filter By	Keyword is selected by default. Enter the keyword to be filtered in the text box.	-
   Log Time Range	There are three types of time range: relative time from now, relative time from last, and specified time. Select a time range as required. From now: queries log data generated in a time range that ends with the current time, such as the previous 1, 5, or 15 minutes. For example, if the current time is 19:20:31 and 1 hour is selected as the relative time from now, the charts on the dashboard display the log data that is generated from 18:20:31 to 19:20:31. From last: queries log data generated in a time range that ends with the current time, such as the previous 1 or 15 minutes. For example, if the current time is 19:20:31 and 1 hour is selected as the relative time from last, the charts on the dashboard display the log data that is generated from 18:00:00 to 19:00:00. Specified: queries log data that is generated in a specified time range.	-
   Total Log Events	Total number of logs in the selected transfer time range. Click Search next to this parameter. LTS automatically calculates and displays the total number of logs within the selected transfer time range. If there is no log, change the transfer time range.	-
   Log Files	Max log events for each transfer: 20 million. Max transfer files: 200.	-
   OBS Bucket	Select an OBS bucket. If no OBS buckets are available, click View OBS Bucket to access the OBS console and create an OBS bucket. Currently, LTS supports only Standard OBS buckets. Data cannot be transferred to an OBS bucket whose storage class is Archive or for which cross-region replication has been configured. If you select an unauthorized OBS bucket, LTS will take 15 minutes to authorize the ACL for the bucket. After you configure a one-off transfer task, if the task fails for the first time, it will be automatically retried 15 minutes later. To prevent log transfer failures, exercise caution when modifying the bucket policy.	-
   Bucket Directory	OBS bucket directory. The value cannot start or end with a slash (/). It can contain only letters, digits, hyphens (-), underscores (_), and periods (.). You can obtain the bucket directory as follows: In the bucket list on OBS Console, click the desired bucket to go to the Objects page. On the Objects page, locate the OBS folder to store LTS logs and click More > Copy Path in the Operation column. (The example is for reference only.) If you want to save logs to the test/yicixing folder, test/yicixing will be copied. The copied path is the bucket directory. Figure 4 Copying a path	test/yicixing
   Transfer File Name	Custom transfer file name. Only letters, digits, hyphens (-), underscores (_), and periods (.) are allowed.	-
   Format	Storage format of logs. The value can be Raw log format, JSON, or CSV. Example of the raw log format: (Logs displayed on the LTS console are in the raw format.) Sep 30 07:30:01 ecs-bd70 CRON[3459]: (root) CMD (/opt/oss/servicemgr/ICAgent/bin/manual/mstart.sh > /dev/null 2>&1) Example of the JSON format: {"host_name":"ecs-bd70","ip":"192.168.0.54","line_no":249,"message":"Sep 30 14:40:01 ecs-bd70 CRON[4363]: (root) CMD (/opt/oss/servicemgr/ICAgent/bin/manual/mstart.sh > /dev/null 2>&1)\n","path":"/var/log/syslog","time":1569825602303} CSV: Log content is displayed in a table.	JSON

3. Click OK. When the log transfer status changes to Normal, the transfer task has been created.

   After a log transfer task is created, the parameters cannot be modified. Ensure that the parameters are correctly set.

4. Click the OBS bucket name in the Transfer Destination column to switch to OBS Console and view the transferred log files.

   Transferred logs can be downloaded from OBS to your local PC for viewing.

   Figure 5 One-time transfer to OBS
   

Modifying a Log Transfer Task

Scheduled transfer tasks can be modified, but one-time transfer tasks cannot.

1. Locate the row that contains the target transfer task and click Modify in the Operation column.
2. Click OK.

Viewing Transfer Details

1. Locate the target log transfer task and click More > Details in the same row to view its details.
2. On the displayed Transfer Details page, you can view the log transfer details.

Deleting a Log Transfer Task

If logs do not need to be transferred, you can delete the transfer task.

Deleting a transfer task will stop log transfer, and the deleted task cannot be restored. Exercise caution when performing this operation.

• After a transfer task is deleted, the logs that have been transferred remain in OBS.
• When you create a transfer task, OBS will grant read and write permissions to LTS for the selected bucket. To delete a transfer task that uses an OBS bucket, perform the following operations:
  • If only one transfer task uses this OBS bucket, delete the bucket access permission granted to specific users on the Permissions > Bucket ACL tab page on the OBS console when you delete the transfer task.
  • If multiple transfer tasks use this OBS bucket, do not delete the bucket access permission. Otherwise, data transfer will fail.

1. Delete a single transfer task:
   1. Click Delete in the Operation column of the target task.
   2. In the displayed dialog box, click OK.
2. Delete transfer tasks in batches:
   1. Select the transfer tasks to be deleted and click Delete above the list.
   2. In the displayed dialog box, click OK.

Viewing Transfer Status

The status of a transfer task can be Normal, Abnormal, or Closed.

• Normal: The log transfer task works properly.
• Abnormal: An error occurred in the log transfer task. The possible causes are as follows:
  • Access control on the OBS bucket is configured incorrectly. Access the OBS console to correct the settings.
  • The key for the encrypted OBS bucket has been deleted or the authorization has been canceled. Ensure that the key is valid.
• Closed: The log transfer task is stopped.

Helpful Links

• If you have any questions when configuring log transfer, see Log Transfer in the FAQ.
• You can create, query, and delete log transfer tasks by calling APIs. For details, see Log Transfer.