Using Prometheus and Grafana to Monitor Servers in Real Time

Background

In modern cloud-native and distributed systems, monitoring the health, performance, and availability of applications and infrastructure is a big challenge. As systems scale, traditional monitoring methods struggle to effectively manage real-time data, generate meaningful alerts, and provide actionable insights. The complexity and dynamic nature of cloud environments demand solutions that can efficiently track metrics, detect exceptions, and visualize data to ensure system reliability and quickly resolve issues.

Prometheus is an open-source system monitoring and alerting framework widely used to monitor and visualize metrics in various cloud environments. Prometheus efficiently collects, stores, and queries time series data, making it ideal for monitoring infrastructure, applications, and services. For more concepts and details about Prometheus, see Prometheus official documents.

Grafana supplements Prometheus with rich visualization capabilities, enabling users to create dynamic dashboards and charts to track performance metrics.

Prometheus and Grafana work together to monitor the health of servers, visualize network traffic, and trigger alerts for key events, providing a complete observability solution for dynamic cloud environments.

This section describes how to configure Prometheus in the Huawei Cloud environment to collect monitoring metrics, use Grafana to visualize the collected data, and integrate with Slack to receive alert notifications.

Resource Planning

Procedure

Prerequisites

• Three ECSs have been purchased and configured based on the resource planning, and the ECSs can communicate with each other.
• You have learned about PromQL expression rules. For details, see the Prometheus official website.

Installing and Configuring Prometheus

To install Prometheus on ECS-Prometheus and start Prometheus, do as follows:

1. Update the software package repository.

   sudo apt update

2. Create a Prometheus user.

   sudo useradd --no-create-home --shell /bin/false prometheus

3. Create a Prometheus directory.

   sudo mkdir /etc/prometheus
   sudo mkdir /var/lib/prometheus

4. Set the directory ownership to ensure that the Prometheus service can read and write data properly.

   sudo chown prometheus:prometheus /var/lib/prometheus

5. Download the Prometheus binary file.

   cd /tmp
   wget https://github.com/prometheus/prometheus/releases/download/v2.54.1/prometheus-2.54.1.linux-amd64.tar.gz
   tar -xvf prometheus-2.54.1.linux-amd64.tar.gz

6. Move the configuration file and set its ownership to the Prometheus user.

   cd prometheus-2.54.1.linux-amd64
   sudo mv console* /etc/prometheus
   sudo mv promtool /usr/local/bin/
   sudo mv prometheus.yml /etc/prometheus
   sudo chown -R prometheus:prometheus /etc/prometheus
   sudo mv prometheus /usr/local/bin/
   sudo chown prometheus:prometheus /usr/local/bin/prometheus

7. Check the Prometheus configuration file to ensure that the file exists and the content is correct.

   sudo vim /etc/prometheus/prometheus.yml

   

8. Create a systemd service file to enable automatic startup and management of the Prometheus service.

   sudo vim /etc/systemd/system/prometheus.service

   Write the following configuration to the prometheus.service file:

   [Unit]
   Description=Prometheus
   Wants=network-online.target
   After=network-online.target
   
   [Service]
   User=prometheus
   Group=prometheus
   Type=simple
   ExecStart=/usr/local/bin/prometheus \
       --config.file /etc/prometheus/prometheus.yml \
       --storage.tsdb.path /var/lib/prometheus/ \
       --web.console.templates=/etc/prometheus/consoles \
       --web.console.libraries=/etc/prometheus/console_libraries
   
   [Install]
   WantedBy=multi-user.target

9. Start the Prometheus service and check its status.

   systemctl daemon-reload
   systemctl start prometheus
   systemctl enable prometheus
   systemctl status prometheus

   

10. Open the following address in your browser to view the Prometheus page:

    http://EIP of ECS-Prometheus:9090/graph

    

    If the page cannot be opened, check whether the involved port (for example, 9090) has been enabled in the security groups.

    

Installing and Configuring Prometheus Alertmanager

To install and configure Alertmanager on ECS-Prometheus, do as follows:

1. Install Alertmanager.

   sudo apt install prometheus-alertmanager
   systemctl status prometheus-alertmanager
   ps -u prometheus

   

   If an error is reported during the installation, perform the following steps to clear the environment and reinstall Alertmanager:

   1. Remove all traces of the package.

      sudo dpkg --remove --force-remove-reinstreq prometheus-alertmanager

      If this command fails, forcibly remove the package.

      sudo dpkg --purge --force-all prometheus-alertmanager

   2. Remove related files.

      sudo rm -rf /var/lib/dpkg/info/prometheus-alertmanager.*

      sudo rm -f /var/lib/dpkg/status

      sudo cp /var/lib/dpkg/status-old /var/lib/dpkg/status # If there is a backup, run this command.

   3. Rebuild the dpkg database.

      sudo dpkg --configure -a

      sudo apt-get update

   4. Repair damaged packages.

      sudo apt-get install -f

   5. Start Prometheus Alertmanager.

      sudo systemctl start prometheus-alertmanager.service

   6. Check status.

      sudo systemctl status prometheus-alertmanager.service

      If active is displayed, the startup is normal. You can run the ps -u prometheus command to check the service status.

      If inactive is displayed and error unmask is reported, run the following commands to unmask the service and then restart the service and check the status:

      sudo systemctl unmask prometheus-alertmanager.service

      sudo systemctl start prometheus-alertmanager.service

      sudo systemctl status prometheus-alertmanager.service

      ps -u prometheus

   

2. Modify the Prometheus configuration file.

   sudo vim /etc/prometheus/prometheus.yml

   Modify the Alertmanager part in the Prometheus configuration file.

   # Alertmanager configuration
   alerting:
     alertmanagers:
       - static_configs:
         - targets: ['localhost:9093']

   

3. Use promtool to check the Prometheus configuration file and restart the Prometheus service.

   promtool check config /etc/prometheus/prometheus.yml

   

   sudo systemctl restart prometheus
   sudo systemctl status prometheus

Installing Grafana

To install Grafana on ECS-Prometheus, do as follows:

1. Install Grafana dependency packages.

   sudo apt install -y wget
   sudo apt install -y software-properties-common
   sudo apt install -y apt-transport-https

   

   

   If an error is reported during the installation, perform the following steps:

   # Repair damaged packages.

   sudo dpkg --configure -a

   # Repair dependencies.

   sudo apt --fix-broken install

   # Remove possible lock files.

   sudo rm -f /var/lib/dpkg/lock

   sudo rm -f /var/lib/apt/lists/lock

   sudo rm -f /var/cache/apt/archives/lock

   # Update the package list.

   sudo apt update

   # Check the needrestart status.

   dpkg -l | grep needrestart

   # Reconfigure needrestart if necessary.

   sudo dpkg-reconfigure needrestart

   # Or temporarily skip needrestart (in a container or an environment that does not require restart).

   echo '$nrconf{restart} = "l";' | sudo tee -a /etc/needrestart/needrestart.conf

   # Reinstall the packages.

   sudo apt install -y wget

   sudo apt install -y software-properties-common

   sudo apt install -y apt-transport-https

2. Add the Grafana GPK key.

   sudo mkdir -p /etc/apt/keyrings/
   wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null

3. Add the Grafana APT repository.

   echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
   sudo apt update

4. Install Grafana.

   sudo apt install grafana

5. Start the Grafana service and check the status.

   sudo grafana-server -v
   sudo systemctl start grafana-server
   sudo systemctl enable grafana-server
   sudo systemctl status grafana-server

   

6. Open the following address in your browser to log in Grafana:

   http://EIP of ECS-Prometheus:3000/login

   

   If the page cannot be opened, check whether the involved port (for example, 3000) has been enabled in the security groups.

   The default username and password are admin and admin, respectively. Reset the password as prompted after your first login.

   

   

   If the login is successful, the following page is displayed.

   

Installing Node Exporter on Target Servers

To install Node Exporter on ECS-target-0001 and ECS-target-0002, respectively, do as follows:

1. Update the software package repository.

   sudo apt update

2. Download the Node Exporter binary file.

   curl -s https://api.github.com/repos/prometheus/node_exporter/releases/latest| grep browser_download_url|grep linux-amd64|cut -d '"' -f 4|wget -qi -

   

   This command execution takes a long time. Please wait patiently.

   tar -xvf node_exporter*.tar.gz
   cd node_exporter*/
   cp node_exporter /usr/local/bin
   node_exporter --version

3. Create a systemd service for Node Exporter.

   sudo useradd -rs /bin/false nodeusr
   sudo vim /etc/systemd/system/node_exporter.service

   Configure the file as follows:

   [Unit]
   Description=Node Exporter
   After=network.target
   
   [Service]
   User=nodeusr
   Group=nodeusr
   Type=simple
   ExecStart=/usr/local/bin/node_exporter
   
   [Install]
   WantedBy=multi-user.target

   

4. Start the Node Exporter service and check the service status.

   systemctl daemon-reload
   systemctl start node_exporter
   systemctl status node_exporter

   

Adding Target Servers to the Prometheus Server

To add ECS-target-0001 and ECS-target-0002 to ECS-Prometheus, do as follows:

1. Configure jobs in the Prometheus configuration file.

   sudo vim /etc/prometheus/prometheus.yml

   Add the following content to the configuration file. Replace the public IP addresses (EIPs) of ECS-target-0001 and ECS-target-0002 with the actual ones.

   - job_name: 'node_exporter_ubuntu'
      scrape_interval: 5s
      static_configs:
        - targets: ['10.x.x.196:9100', '10.x.x.198:9100']

   

2. Use promtool to check the configuration rules and restart Prometheus.

   promtool check config /etc/prometheus/prometheus.yml
   sudo systemctl restart prometheus
   sudo systemctl status prometheus

Configuring Prometheus Alerts

1. Configure Prometheus alerting rules in the YAML file.

   vim /etc/prometheus/rules.yml

   In the alerting rule file, you can define alerting rule groups to organize alerting rules that are logically related, making them easier to manage and maintain. An alerting rule consists of the following:

   • alert: Name of the alerting rule.
   • expr: PromQL expression that determines whether any time series meets the condition. For more information about PromQL expression rules, see the Prometheus official website.
   • for: Evaluation waiting time, which is optional. It indicates that an alert is sent only after the trigger condition is met for a period of time. New alerts generated during the waiting period are in pending state.
   • labels: User-defined labels.
   • annotations: A group of additional information, for example, the description of an alert.

   Example alerting rules are as follows:

   groups:
   - name: instance-down
     interval: 15s
     rules:
     - alert: InstanceDown
       expr: up == 0
       for: 5m
       labels:
         severity: high
       annotations:
         message: "the {{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes"
   - name: cpu-usage
     interval: 15s
     rules:
     - alert: CPUHigh
       expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 50
       for: 2m
       labels:
         severity: warning
       annotations:
         message: "CPU usage is above 25% for the last 5 minutes on instance {{ $labels.instance }} of job {{ $labels.job }}"
   - name: network-traffic
     interval: 15s
     rules:
     - alert: HighNetworkTraffic
       expr: rate(node_network_receive_bytes_total[3m]) > 1000000
       for: 3m
       labels:
         severity: warning
       annotations:
         message: "Incoming network traffic is higher than 1MB/s on instance {{ $labels.instance }} for more than 3 minutes"

   

2. Configure prometheus.yml to include the alerting rules.

   vim /etc/prometheus/prometheus.yml

   rule_files:
     - "rules.yml"

   

3. Use promtool to verify the configuration of the rules and restart Prometheus.

   promtool check rules /etc/prometheus/rules.yml
   promtool check config /etc/prometheus/prometheus.yml
   sudo systemctl restart prometheus
   sudo systemctl status prometheus

   

Configuring Grafana

1. Configure Grafana alert recipients to ensure that alert notifications can be sent to the specified channels, such as Slack or email. In this example, Slack is used.

   

2. Configure the data source and select the Prometheus data source.

   

   Configure the Prometheus address (http://localhost:9090) for Prometheus server URL and retain the default settings for other parameters.

   

   Click Save & test. If the following figure is displayed, the data source is successfully connected. If the connection fails, check whether the Prometheus port is occupied.

   

3. View the alerting rules configured for Prometheus and ensure that the rules meet the expectation.

   

4. Configure the dashboard to customize the monitoring metrics to be queried.

   

   

   • Select Prometheus as the data source.
   • Select code for querying monitoring metrics.
   • Enter the query you want to run. Prometheus supports not only simple metric queries but also arithmetic and operations between metrics.
     For example, to query the CPU usage of a server, run the following command:

     (1 - sum(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance) / sum(rate(node_cpu_seconds_total[5m])) by (instance) ) * 100

   • Select the time range for querying monitoring metrics.
   • Click Run queries.
   • Set the panel title and save the dashboard settings.

Verifying the Result

1. Check whether the target servers are correctly configured in Prometheus.
   

   If target servers are not in the UP state, check whether the ports (for example, 9100) configured in the security groups of the servers are open to the Prometheus server.

   

2. Check whether alerting rules are correctly configured in Prometheus.

   

3. Use PromQL to query metrics in Prometheus.

   

   

4. After a target server is shut down, an alert is triggered. You can view the alert information in Prometheus and Grafana, and receive notifications in Slack.