Connecting Logstash to OBS

Overview

Logstash collects data from a multitude of sources, transforms it, and then ships it to the storage system. This section describes how to connect Logstash to OBS.

Precautions

Logstash 7.10.2 or later is recommended.

Procedure

The following uses logstash-7.10.2 as an example.

Download logstash-7.10.2-linux-x86_64.tar.gz and decompress it to the /opt/logstash-7.10.2-linux-x86_64 directory.

Check whether the connection is successful.

Example: Use file as the source and OBS as the destination.

Create a configuration file file2obs.conf. Table 1 describes the parameters. For more information, see Logstash Reference.

input {
  file {
    path => "/opt/nginx/logs/access.log"
    start_position => "beginning"
  }
}

output {
  s3 {
    endpoint => "obs endpoint"   # The endpoint should be an HTTP or HTTPS URL
    access_key_id => "ak"
    secret_access_key => "sk"
    bucket => "obs bucket name"
    size_file => 1048576
    time_file => 1
    prefix => "logstash/"
    enable_metric => true
  }
}

**Table 1** Parameters
Parameter	Description
endpoint	An endpoint of OBS. Examples are as follows: https://obs.eu-west-101.myhuaweicloud.eu http://obs.eu-west-101.myhuaweicloud.eu
access_key_id	AK for accessing OBS.
secret_access_key	SK for accessing OBS.
bucket	OBS bucket name.
size_file	Specifies the file size (in bytes). When the size exceeds this parameter value, a new file is created.
time_file	Sets the time (in minutes). When the data write period exceeds this parameter value, a new file is created.
prefix	File storage directory, for example, logstash/. In this case, files will be written to the logstash/ directory of the bucket. The directory cannot start with a slash (/).