Using Grafana Loki to Visualize Application Data Logs

Scenarios

Grafana Loki is a log aggregation system designed to provide efficient, scalable, and cost-effective logging for applications. Unlike traditional log management tools, Loki focuses on indexing metadata rather than the entire log content, significantly reducing resource usage and complexity. It seamlessly integrates with Grafana to provide powerful log visualization and querying functions, making it a natural choice for applications that already use Prometheus for metric monitoring. Traditional log management systems typically require full-text indexing of log data, which can be resource-intensive, complex, and expensive. As applications scale, these systems can become costly and inefficient, making log management and querying difficult. Grafana Loki reduces resource usage by indexing only the metadata (labels) of logs rather than their full content, streamlines log management, and integrates seamlessly with existing observability tools like Prometheus and Grafana, enabling unified monitoring and troubleshooting.

In this section, we will:

• Understand how Grafana Loki works.
• Configure Loki to collect various application logs and push the log data to Grafana Loki for further analysis and querying on Huawei Cloud.

Prerequisites

1. EIPs have been bound to the ECSs.
2. The rule listed in Table 1 has been added to the security group that the target ECS belongs to. For details, see Adding a Security Group Rule.

   Table 1 Security group rules

   Direction	Priority	Action	Type	Protocol & Port	Source
   Inbound	1	Allow	IPv4	TCP: 3001	0.0.0.0/0
   Inbound	1	Allow	IPv4	TCP: 3100	0.0.0.0/0
   Inbound	1	Allow	IPv4	TCP: 9093	0.0.0.0/0

Resource Planning

Procedure

To use Grafana Loki to visualize and query application data logs, do as follows:

1. Installing and Configuring Grafana Loki
2. Installing Promtail Agent on a Target Server
3. Creating a Simple Node.js Application on the Other Target Server
4. Configuring the Nginx Log Collection Job
5. Verifying the Result

Installing and Configuring Grafana Loki

1. Remotely log in to ecs-prometheus.
2. Update the repository.

   sudo apt update

3. Install Grafana.

   wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
   sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
   sudo apt update
   sudo apt install grafana

4. Enable Grafana.

   sudo systemctl start grafana-server
   sudo systemctl enable grafana-server

5. Install Grafana Loki.

   curl -s https://api.github.com/repos/grafana/loki/releases/latest | grep browser_download_url |  cut -d '"' -f 4 | grep loki-linux-amd64.zip | wget -i -
   sudo apt install unzip
   unzip loki-linux-amd64.zip
   sudo mv loki-linux-amd64 /usr/local/bin/loki

6. Check the Grafana Loki version.

   loki --version

   Information similar to the following is displayed.

   

7. Create a Grafana Loki configuration file.

   sudo mkdir -p /data/loki
   sudo wget -O /etc/loki-local-config.yaml https://raw.githubusercontent.com/grafana/loki/v3.1.1/cmd/loki/loki-local-config.yaml

8. Modify the Grafana Loki configuration file.

   sudo vim /etc/loki-local-config.yaml

   Configure the file as follows:

   auth_enabled: false
   server:
     http_listen_port: 3100
     grpc_listen_port: 9096
   common:
     instance_addr: 127.0.0.1
     path_prefix: /tmp/loki
     storage:
       filesystem:
         chunks_directory: /tmp/loki/chunks
         rules_directory: /tmp/loki/rules
     replication_factor: 1
     ring:
       kvstore:
         store: inmemory
   query_range:
     results_cache:
       cache:
         embedded_cache:
           enabled: true
           max_size_mb: 100
   schema_config:
     configs:
       - from: 2020-10-24
         store: tsdb
         object_store: filesystem
         schema: v13
         index:
           prefix: index_
           period: 24h
   ruler:
     alertmanager_url: http://localhost:9093 // Replace localhost with the EIP of ecs-target-0001.

9. Create the Loki service.

   sudo vim /etc/systemd/system/loki.service

10. Add the following content to the loki.service file:

    [Unit]
    Description=Loki service
    After=network.target
    
    [Service]
    Type=simple
    User=root
    ExecStart=/usr/local/bin/loki -config.file /etc/loki-local-config.yaml
    
    [Install]
    WantedBy=multi-user.target

11. Press Esc to exit insert mode. Then, enter :wq to save the settings and exit.
12. Reload the systemd configuration for the new service file to take effect.

    sudo systemctl daemon-reload

13. Enable the Loki service and set it to automatically start upon system startup.

    sudo systemctl start loki.service
    sudo systemctl enable loki.service

14. Check the Loki service status and ensure that it is running.

    sudo systemctl status loki.service

    Information similar to the following is displayed.

    

Installing Promtail Agent on a Target Server

1. Remotely log in to ecs-target-0001.
2. Update the repository.

   sudo apt update

3. Download the binary file.

   curl -s https://api.github.com/repos/grafana/loki/releases/latest | grep browser_download_url |  cut -d '"' -f 4 | grep promtail-linux-amd64.zip | wget -i -

4. Install the Promtail agent.

   sudo apt install unzip
   unzip promtail-linux-amd64.zip
   sudo mv promtail-linux-amd64 /usr/local/bin/promtail

5. Check the Promtail version.

   promtail --version

   Information similar to the following is displayed.

   

6. Configure Promtail.

   cd /etc
   wget https://raw.githubusercontent.com/grafana/loki/v3.1.1/clients/cmd/promtail/promtail-local-config.yaml
   sudo vim promtail-local-config.yaml

   Configure the file as follows:

   server:
     http_listen_port: 9080
     grpc_listen_port: 0
   
   positions:
     filename: /data/loki/positions.yaml
   
   clients:
     - url: http://localhost:3100/loki/api/v1/push // Replace localhost with the EIP of ecs-prometheus.
   
   scrape_configs:
   - job_name: system
     static_configs:
     - targets:
         - localhost
       labels:
         job: varlogs
         __path__: /var/log/*log

7. Press Esc to exit insert mode. Then, enter :wq to save the settings and exit.
8. Open the Promtail service file.

   sudo vim /etc/systemd/system/promtail.service

9. Add the following content to the file:

   [Unit]
   Description=Promtail service
   After=network.target
   
   [Service]
   Type=simple
   User=root
   ExecStart=/usr/local/bin/promtail -config.file /etc/promtail-local-config.yaml
   
   [Install]
   WantedBy=multi-user.target

10. Press Esc to exit insert mode. Then, enter :wq to save the settings and exit.
11. Reload the systemd configuration to apply the change.

    sudo systemctl daemon-reload

12. Enable the Promtail service and set it to automatically start upon system startup.

    sudo systemctl start promtail.service
    sudo systemctl enable promtail.service

13. Check the Promtail service status and ensure that it is running.

    sudo systemctl status promtail.service

    Information similar to the following is displayed.

    

Creating a Simple Node.js Application on the Other Target Server

1. Remotely log in to ecs-target-0002.
2. Install Node.js.

   sudo apt update
   sudo apt install nodejs npm -y

3. Edit the Node.js file.

   sudo mkdir app
   sudo vim app/app.js

   Configure the file as follows:

   const http = require('http');
   const fs = require('fs');
   
   let requestCounter = 0;
   
   const server = http.createServer((req, res) => {
       requestCounter++;
       const logMessage = `Request #${requestCounter} received at ${new Date()}\n`;
   
       fs.appendFile('/var/log/nodejsapp/app.log', logMessage, (err) => {
           if (err) {
               console.error('Error writing to log file:', err);
           }
       });
   
       res.statusCode = 200;
       res.setHeader('Content-Type', 'text/plain');
       res.end(`Hello, World!\nTotal requests: ${requestCounter}`);
   });
   
   server.listen(3001, () => {
       console.log('Server running at http://localhost:3001/');
   });

4. Press Esc to exit insert mode. Then, enter :wq to save the settings and exit.
5. Create log directories.

   sudo mkdir -p /var/log/nodejsapp
   sudo touch /var/log/nodejsapp/app.log
   sudo chmod 777 /var/log/nodejsapp/app.log

6. Run the app.js application in the background.

   sudo node app/app.js &

   Information similar to the following is displayed.

   Server running at http://localhost:3001/

7. Use a browser to access http://Server IP address:3001 to access the application.

   

Configuring the Nginx Log Collection Job

1. Remotely log in to ecs-target-0001.
2. Navigate to the target server where Promtail is installed.

   sudo apt install nginx
   sudo systemctl status nginx

3. Configure the Promtail log collection job.

   sudo vim /etc/promtail-local-config.yaml

   Configure the file as follows:

   server:
     http_listen_port: 9080
     grpc_listen_port: 0
   
   positions:
     filename: /tmp/positions.yaml
   
   clients:
     - url: http://localhost:3100/loki/api/v1/push // Replace localhost with the EIP of ecs-prometheus.
   
   scrape_configs:
     - job_name: system
       static_configs:
         - targets:
             - localhost
           labels:
             job: varlogs
             __path__: /var/log/*log

4. Restart the Promtail service.

   systemctl restart promtail.service
   systemctl status promtail.service

Verifying the Result

1. Ensure that the Grafana Loki service is active and running.
   Figure 1 Grafana Loki service
   
2. Ensure that the Promtail service is active and running on the target server.
   Figure 2 Promtail service
   
3. Ensure that the application is running. Refresh the web page multiple times to increase the number of requests.
   Figure 3 Access request 1
   
   Figure 4 Access request 2
   
4. Add a data source connection to Grafana Loki.
   Figure 5 Adding a data source connection
   
5. Check whether application logs have been pushed to Grafana Loki.
   1. Click Explore on the Grafana dashboard.
   2. Set label filters and click Execute to run the log query.

      The logs are stored in Grafana Loki, and the number of received requests indicates how many times the application is accessed from the Internet.

      Figure 6 Application data logs in Grafana Loki
      
      Figure 7 Log details
      

Conclusion

In this section, we learned how to install and configure Loki on a server to collect application logs from target servers in the Huawei Cloud environment. We also learned how to push log data to Grafana Loki and use Grafana's built-in functions to visualize and query application logs in a browser.