Interconnecting Elasticsearch with Beats (Filebeat)

Scenario

Filebeat is a tool for forwarding log files. After the client is installed on the server, Filebeat monitors the log directory or specified log files, trace file changes, read the files repeatedly, and forwards the information to Elasticsearch.

• The basic authentication is used for Elasticsearch to connect Filebeat, which is insecure. You are advised to use the client tool esClient.sh to encrypt the password.
• The Arm version is not supported.

Prerequisites

Filebeat has been downloaded and installed.

1. Download the Filebeat 7.10.2 installation package.
   1. Download the Filebeat installation package filebeat-oss-7.10.2-linux-x86_64.tar.gz and upload it to the directory of the node to be installed, for example, /opt.

      Open-source community website: https://www.elastic.co/cn/downloads/past-releases/filebeat-oss-7-10-2

   2. Run the following commands to decompress the installation package and go to the directory of that package: The preparations are complete.

      cd /opt

      tar -xvf filebeat-oss-7.10.2-linux-x86_64.tar.gz

      cd filebeat-7.10.2-linux-x86_64/

2. Download the Filebeat 6.6.2 installation package.
   1. Download the Filebeat installation package filebeat-oss-6.6.2-linux-x86_64.tar.gz and upload it to the node to be installed, for example, /opt.

      Open source community: https://www.elastic.co/downloads/past-releases/filebeat-oss-6-6-2

   2. Run the following commands to decompress the installation package and go to the directory of that package:

      cd /opt

      tar -xvf filebeat-oss-6.6.2-linux-x86_64.tar.gz

      cd filebeat-6.6.2-linux-x86_64/

Procedure

Using Filebeat to collect server logs and store data to Elasticsearch

1. Configure the filebeat.yml file. The following is an example:

   filebeat.prospectors:
   # Each - is a prospector. Most options can be set at the prospector level, so
   # you can use different prospectors for various configurations.
   # Below are the prospector specific configurations.
   - type: log
     # Change to true to enable this prospector configuration.
     enabled: true
     # Paths that should be crawled and fetched. Glob based paths.
     paths:
       - /var/log/Bigdata/elasticsearch/esnode1/elasticsearch_cluster.log
    
   setup.template.settings:
     index.number_of_shards: 3                //Sets the number of primary shards of the index.
     index.number_of_replicas: 1              //Sets the number of replica shards of the index.
     index.codec: best_compression
     #_source.enabled: false
    
   # Enable automatic index template generation.
   setup.template.enabled: true
   # Configuration file for generating the index template
   setup.template.fields: fields.yml
   # Name of the generated index template
   setup.template.name: mytemplate
   # Index format of the generated index template
   setup.template.pattern: filebeat*
    
   output.elasticsearch:
     # Array of hosts to connect to.
     hosts: ["ip:httpport"]
     index: filebeat                //Sets the index name.
    
     # Optional protocol and basic auth credentials.
     protocol: "https"
     username: "username"      
     password: "password"                    // Enter the encrypted password.
   ssl.verification_mode: none               //Configures Filebeat not to verify the Elasticsearch server certificate.

   

   The key configuration parameters are described as follows:

   • paths: indicates the monitoring file directory. In this example, it is an Elasticsearch system log file.
   • hosts: indicates the HTTP IP address and port number for connection.
   • index: indicates the index name.
   • protocol: indicates HTTPS in the security mode.
   • username: user who has the specified Elasticsearch operation permissions. For details about how to set Elasticsearch user permissions, see Elasticsearch Authentication Mode.
   • password: user password encrypted using the esClient.sh tool.
   • ssl.verification_mode: If this parameter is set to none, Filebeat does not verify the Elasticsearch server certificate.
   • In normal mode, you do not need to set protocol, username, password, or ssl.verification_mode.
   • Before setting password, run the esClient.sh encrypt command to encrypt the password. For details, see 6 in Using the Elasticsearch Client.

2. Run the following command to start Filebeat:
   • Use Filebeat 7.10.2 to run the following commands:

     cd /opt/filebeat-7.10.2-linux-x86_64/

     ./filebeat -e -c filebeat.yml

   • Use Filebeat 6.6.2 to run the following commands:

     cd /opt/filebeat-6.6.2-linux-x86_64/

     ./filebeat -e -c filebeat.yml

3. Run the following command to query and verify the Elasticsearch index data that is written:

   curl -XGET --tlsv1.2 --negotiate -k -u : "https://ip:httpport/filebeat/_search?pretty"