Monitor VMs with Prometheus

Download Node Exporter

The first step is to set-up Node Exporter, which is a "Prometheus exporter for hardware and OS metrics exposed by *NIX kernels, written in Go with pluggable metric collectors", on a VM we would like to monitor.

Get the latest version from the Prometheus downloads page, by copying the link address of the tarball you require

curl -LO <Link Address>

Then unpack the tarball

tar -xvf <file>

Move the binary to /usr/local/bin

sudo mv <unpacked directory>/node_exporter /usr/local/bin/

Create a Custom Node Exporter Service

Create a node_exporter user to run the node exporter service

sudo useradd -rs /bin/false node_exporter

Create a node_exporter service file under systemd

sudo vi /etc/systemd/system/node_exporter.service

Add the following service file content to the service file and save it.

[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Reload the system daemon and star the node exporter service.

sudo systemctl daemon-reload
sudo systemctl start node_exporter

Check the node exporter status to make sure it is running in the active state

sudo systemctl status node_exporter

Enable the node exporter service to the system startup

sudo systemctl enable node_exporter

Now, node exporter would be exporting metrics on port 9100. You can see all the server metrics by visiting your server URL on /metrics.

Set-up Prometheus

On whichever host you want to host Prometheus, create a docker-compose.yml file and add the following contents, substituting <path to directory> to where you would like to keep your prometheus configuration file:

version: '3.3'
services:
  prometheus:
    image: prom/prometheus:v2.34.0
    container_name: prometheus
    networks:
      - reporting
    ports:
      - 9090:9090
    volumes:
      - <path to directory>/prometheus.yml:/etc/prometheus/prometheus.yml

networks:
  reporting:

Then create the prometheus.yml in the location you have specified, and add the following:

global:
  scrape_interval: 10s
scrape_configs:
  - job_name: node
    metrics_path: /metrics
    static_configs:
      - targets: ['<vm-host>:9100']

Now start prometheus with docker-compose up -d.

All being well, you should be able to navigate to http://<host ip>:9090/targets and see a list of targets you have configured.

Setting up Grafana (Optional)

To use Grafana, add the following service to your docker-compose.yml file

grafana:
  image: grafana/grafana:8.4.4-ubuntu
  container_name: grafana
  networks:
    - reporting
  ports:
    - 3000:3000

And start the container with docker-compose up -d.

Configure Prometheus Data Source

Navigate to the Grafana UI at http://<host ip>:3000 and log in with the default username/password admin:admin.

Then go to Configuration > Data sources > Add data source > Prometheus. Specify a name for the data source, the HTTP url of the Prometheus instance (http://prometheus:9090 if you've added everything as is detailed in this article) and click on Save & test.

If you receive a “Data source is working” message, everything is working as intended.

Add a dashboard

Navigate to Create (+) > Import, add 13978 to Import via grafana.com and click Load. Select the Prometheus data source that you created before and clickImport.

All being well, you should now have a good starting place to monitor your VMs.