Monitor VMs with Prometheus

Introduction

In this article I’m going to describe the process of monitoring your VMs (in my case hosted on ProxMox) with Prometheus.

Download Node Exporter

The first step is to set-up Node Exporter, which is a “Prometheus exporter for hardware and OS metrics exposed by *NIX kernels, written in Go with pluggable metric collectors”, on a VM we would like to monitor.

Get the latest version from the Prometheus downloads page, by copying the link address of the tarball you require

curl -LO <Link Address>

Then unpack the tarball

tar -xvf <file>

Move the binary to /usr/local/bin

sudo mv <unpacked directory>/node_exporter /usr/local/bin/

Create a Custom Node Exporter Service

Create a node_exporter user to run the node exporter service

sudo useradd -rs /bin/false node_exporter

Create a node_exporter service file under systemd

sudo vi /etc/systemd/system/node_exporter.service

Add the following service file content to the service file and save it.

[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Reload the system daemon and star the node exporter service.

sudo systemctl daemon-reload
sudo systemctl start node_exporter

Check the node exporter status to make sure it is running in the active state

sudo systemctl status node_exporter

Enable the node exporter service to the system startup

sudo systemctl enable node_exporter

Now, node exporter would be exporting metrics on port 9100. You can see all the server metrics by visiting your server URL on /metrics.

Set-up Prometheus

On whichever host you want to host Prometheus, create a docker-compose.yml file and add the following contents, substituting <path to directory> to where you would like to keep your prometheus configuration file:

version: "3.3"
services:
  prometheus:
    image: prom/prometheus:v2.34.0
    container_name: prometheus
    networks:
      - reporting
    ports:
      - 9090:9090
    volumes:
      - <path to directory>/prometheus.yml:/etc/prometheus/prometheus.yml

networks:
  reporting:

Then create the prometheus.yml in the location you have specified, and add the following:

global:
  scrape_interval: 10s
scrape_configs:
  - job_name: node
    metrics_path: /metrics
    static_configs:
      - targets: [ '<vm-host>:9100' ]

Now start prometheus with docker-compose up -d.

All being well, you should be able to navigate to http://<host ip>:9090/targets and see a list of targets you have configured.

Setting up Grafana (Optional)

To use Grafana, add the following service to your docker-compose.yml file

grafana:
  image: grafana/grafana:8.4.4-ubuntu
  container_name: grafana
  networks:
    - reporting
  ports:
    - 3000:3000

And start the container with docker-compose up -d.

Configure Prometheus Data Source

Navigate to the Grafana UI at http://<host ip>:3000 and log in with the default username/password admin:admin.

Then go to Configuration > Data sources > Add data source > Prometheus. Specify a name for the data source, the HTTP url of the Prometheus instance (http://prometheus:9090 if you’ve added everything as is detailed in this article) and click on Save & test.

If you receive a “Data source is working” message, everything is working as intended.

Add a dashboard

Navigate to Create (+) > Import, add 13978 to Import via grafana.com and click Load. Select the Prometheus data source that you created before and clickImport.

All being well, you should now have a good starting place to monitor your VMs.