Connecting Elasticsearch to Logstash

Scenario

Logstash is an open-source, server-side data processing pipeline that integrates data from multiple sources, converts it, and then stores it to Elasticsearch in scenarios such as log search.

The basic authentication is used for Elasticsearch to connect Logstash, which is insecure. You are advised to use the client tool esClient.sh to encrypt the password.

Prerequisites

You have downloaded and installed Logstash.

1. Download the Logstash installation package logstash-oss-7.10.2-linux-x86_64.tar.gz and upload it to the node where it will be installed, for example, the /opt directory. Check whether an x86 server or a TaiShan server is used, and select an open source community address to download the installation package.
   • x86 server: https://artifacts.elastic.co/downloads/logstash/logstash-oss-7.10.2-linux-x86_64.tar.gz
   • TaiShan server: https://artifacts.elastic.co/downloads/logstash/logstash-oss-7.10.2-linux-aarch64.tar.gz
2. Decompress the installation package and go to the corresponding decompression directory. The following uses the x86 server as an example to describe the installation process.

   cd /opt

   tar -xvf logstash-oss-7.10.2-linux-x86_64.tar.gz

   cd logstash-7.10.2/

Procedure

Using Logstash to write data to Elasticsearch

1. Run the following command to create the data source file log.log. Save the file in the directory, for example, the /opt/log/ directory.
   touch log.log

   2016-07-11T23:56:42.000+00:00 INFO this is a test! 127.0.0.1
   2016-07-11T23:56:42.000+00:00 INFO this is a test! 127.0.0.222

2. Run the following command to create the configuration file safe2Es.conf. Save the file in the directory, for example, the /opt/logstash-7.10.2/ directory.
   touch safe2Es.conf

   input{
           file{
                   path => "/opt/log/log.log"
           }
   }
   filter{
           grok{
                   match=>{ "message" =>"%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log-level} %{DATA:message} %{IP:address}" }
           }
   }
   output{
           elasticsearch{
                   hosts => ["https://ip1:httpport1","https://ip2:httpport2"]
                   index => "filtertest"
                   user => "username"
                   password => "password"                // Enter the encrypted password.
                   ssl => true
                   ssl_certificate_verification => false
                   cacert => "${BIGDATA_HOME}/om-agent/nodeagent/security/cert/subcert/certFile/sslservercet.crt" 
           }
   }

   

   The key configuration parameters are described as follows:

   • input: data source
   • filter: data filtering mode
   • output: data output mode
     • hosts: IP address and port number of the EsNode instance to be connected
     • index: index name
     • user: user who has the specified Elasticsearch operation permissions. For details about how to set Elasticsearch user permissions, see Elasticsearch Authentication Mode.
     • password: user password encrypted using the esClient.sh tool.
     • ssl: If this parameter is set to true, SSL encryption is enabled.
     • ssl_certificate_verification: If this parameter is set to false, Logstash does not verify the Elasticsearch server certificate.
   • In normal mode, you do not need to set user, password, ssl, or ssl_certificate_verification.
   • Before setting password, run the esClient.sh encrypt command to encrypt the password. For details, see 6 in Using the Elasticsearch Client.
   • cacert: path of the sslservercet.crt certificate file on any node in the MRS cluster. If Logstash is installed outside the MRS cluster, you need to set this parameter and copy the certificate to the node where Logstash is installed.

3. Run the following command to start Logstash:

   cd /opt/logstash-7.10.2/

   sh bin/logstash -f safe2Es.conf

   

   • After Logstash starts, add data to the log.log file. By default, Logstash is configured in incremental mode. When detecting changes in data sources, Logstash writes data to Elasticsearch.
   • To start Logstash in the background, run the following command:

     nohup sh bin/logstash -f safe2Es.conf &

   • Before starting Logstash, ensure that the default directory data.path is empty.

4. Run the following command to query and verify the Elasticsearch index data that is written:

   curl -XGET --tlsv1.2 --negotiate -k -u : "https://ip:httpport/filtertest/_search?pretty"