Interconnecting Elasticsearch with Beats (Metricbeat)

Scenario

Metricbeat periodically collects operating system and server running metrics (including CPU, memory, hard disk, I/O read/write speed, and process). Metricbeat can send the collected indicators and data to a specified output, for example, Elasticsearch, to achieve server monitoring.

• The basic authentication is used for Elasticsearch to connect Metricbeat, which is insecure. You are advised to use the client tool esClient.sh to encrypt the password.
• The Arm version is not supported.

Prerequisites

Metricbeat has been downloaded and installed.

1. Download the Metricbeat 7.10.2 installation package.
   1. Download the Metricbeat installation package metricbeat-oss-7.10.2-linux-x86_64.tar.gz and upload it to the node to be installed, for example, /opt.

      Open-source community website: https://www.elastic.co/cn/downloads/past-releases/metricbeat-oss-7-10-2

   2. Run the following commands to decompress the installation package and go to the directory of that package:

      cd /opt

      tar -xvf metricbeat-oss-7.10.2-linux-x86_64.tar.gz

      cd metricbeat-7.10.2-linux-x86_64/

2. Download the Metricbeat 6.6.2 installation package.
   1. Download the Metricbeat installation package metricbeat-oss-6.6.2-linux-x86_64.tar.gz and upload it to the node to be installed, for example, /opt.

      Open-source community website: https://www.elastic.co/downloads/past-releases/metricbeat-oss-6-6-2.

   2. Run the following commands to decompress the installation package and go to the directory of that package:

      cd /opt

      tar -xvf metricbeat-oss-6.6.2-linux-x86_64.tar.gz

      cd metricbeat-6.6.2-linux-x86_64/

Procedure

1. Configure the metricbeat.yml file. The following is an example:

   metricbeat.config.modules:
     # Glob pattern for configuration loading
     path: ${path.config}/modules.d/*.yml
    
     # Set to true to enable config reloading
     reload.enabled: false
    
   setup.template.settings:
     index.number_of_shards: 3               // Set the number of primary shards of the index.
     index.number_of_replicas: 1              // Set the number of replica shards of the index.
     index.codec: best_compression
     #_source.enabled: false
    
   # Enable automatic index template generation.
   setup.template.enabled: true
   # Configuration file for generating the index template
   setup.template.fields: fields.yml
   # Name of the generated index template
   setup.template.name: metricbeat_template
   # Index format of the generated index template
   setup.template.pattern: metricbeat*
    
   output.elasticsearch:
     # Array of hosts to connect to.
     hosts: ["IP:Httpport"]
     index: metricbeat                           //Set the index name.
    
     # Optional protocol and basic auth credentials.
     protocol: "https"
     username: "usename"
     password: "password"                       // Enter the encrypted password.
     ssl.verification_mode: none               //Disable the Elasticsearch server certificate verification for Merticbeat.

   

   The key configuration parameters are described as follows:

   • hosts: indicates the HTTP IP address and port number for connection.
   • index: indicates the index name.
   • protocol: indicates HTTPS in the security mode.
   • username: user who has the specified Elasticsearch operation permissions. For details about how to set Elasticsearch user permissions, see Elasticsearch Authentication Mode.
   • password: indicates the user password encrypted using the esClient.sh tool.
   • ssl.verification_mode: If this parameter is set to none, Metricbeat does not verify the Elasticsearch server certificate.
   • In normal mode, you do not need to set protocol, username, password, or ssl.verification_mode.
   • Before setting password, run the esClient.sh encrypt command to encrypt the password.

2. Run the following command to start Metricbeat:
   • Use Metricbeat 7.10.2 to run the following commands:

     cd /opt/metricbeat-7.10.2-linux-x86_64/

     ./metricbeat -e -c metricbeat.yml

   • Use Metricbeat 6.6.2 to run the following commands:

     cd /opt/metricbeat-6.6.2-linux-x86_64/

     ./metricbeat -e -c metricbeat.yml

3. Run the following command to query and verify the Elasticsearch index data that is written:

   curl -XGET --tlsv1.2 --negotiate -k -u : "https://ip:httpport/metricbeat/_search?pretty"