Installing Prometheus Node Exporter

Posted on Saturday, January 30, 2021


I recently wrote an article where I installed Prometheus on Ubuntu 20.04 http://www.whiteboardcoder.com/2021/01/installing-prometheus-on-ubuntu-2004.html [1]

Now I want to start getting some real data into it to have something to graph! 

 

One good first step is to use Prometheus Node Exporter https://prometheus.io/docs/guides/node-exporter/ [2]
https://github.com/prometheus/node_exporter [3]

 

There are two good reasons to start with node exporter.  One, it will provide tons on Unix information about your server… Memory usage, disk usage, kernel info etc.   Two, you can use it to suck up other data you create with your own cron jobs that create prometheus formatted data and put it in the correct folder.


 

Download and setup

 

Using https://www.digitalocean.com/community/tutorials/how-to-install-prometheus-on-ubuntu-16-04 [4]

 First see what version of ubuntu you are on

 

  > lsb_release -a

 

 

 

On this particular server I have set up a second drive located at /prometheus and that is where I put prometheus data

 

Create new user for node exporter


  > sudo useradd --no-create-home --shell /bin/false node_exporter

 

Head over to https://github.com/prometheus/node_exporter

 


 

Go find the latest (click on latest)


 


Right click on this guy and get the copy the link in my case its https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.darwin-amd64.tar.gz

 

 


  > wget https://github.com/prometheus/node_exporter/releases/download/v0.15.1/node_exporter-0.15.1.linux-amd64.tar.gz

 

Untar it and install


  > tar xvf node_exporter-0.15.1.linux-amd64.tar.gz
  > sudo cp node_exporter-0.15.1.linux-amd64/node_exporter /usr/local/bin
  > sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter



Create a folder to use for extra metrics


  > sudo mkdir -p /prometheus/metrics
  > sudo chown node_exporter:node_exporter /prometheus/metrics

 


SystemD setup

 

Create the systemD file

 

  > sudo vi /lib/systemd/system/node_exporter.service

 

 

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target 

[Service]
Type=simple
User=node_exporter
Group=node_exporter
ExecStart=/usr/local/bin/node_exporter --collector.textfile.directory=/prometheus/metrics
Restart=always
RestartSec=10s
NotifyAccess=all 

[Install]
WantedBy=multi-user.target


 



Enable the service so it will auto start on reboot

 

  > sudo systemctl enable node_exporter

 

 

 

  > sudo systemctl status node_exporter

 


 

Wahoo


Test it out

 

  > sudo systemctl start node_exporter
  > sudo systemctl status node_exporter

 

 


 

Now pull ports and test.

 

  > ssh prometheus -L 9090:localhost:9090 –L 9100:localhost:9100


http://localhost:9100/metrics

 


Wahoo data.

 


Now to scrape it

 

The data is available now but we still need to tell prometheus to scrape the data.

So let’s tweak the prometheus settings.  (Prometheus is running on the same server)

 

 

  > sudo vi /prometheus/pheus/prometheus.yml 

 



Now add another thing to scrape on this same server.

 

 

  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9100']

 

 



Restart Prometheus

 

  > sudo systemctl restart prometheus


Now let’s check prometheus and see if its getting the data.

http://localhost:9090/

 


Open Targets



Looks like its scraping it.

 


 

Go to Graph

Search for

 

node_cpu{mode="system"}

 

 

(click Execute)

 

 

There is data!

 


 

 


Custom Data

If I look at /lib/systemd/system/node_exporter.service

 

  > sudo vi /lib/systemd/system/node_exporter.service

 


 

I have set the variable collector.textfile.directory to look at the folder /prometheus/metrics for “extra” prometheus formatted data to ingest and have node exporter show at :9100/metrics


So let me make a simple python program and create a cron job to run it.

 

 

 

  > sudo mkdir /prometheus/code
  > sudo vi /prometheus/code/prometheus_random.py

 

 

 

 

#!/usr/bin/env python3
#
# Random stuff
#
##########################

import sys
import os
import random

base_dir="/prometheus/metrics/"

##########################
#  Main
##########################
if __name__ == '__main__':

  #Test if base folder exist

  try:
    if not os.path.exists(base_dir):
      print("folder '" + base_dir + "' does not exist")
      sys.exit(1)
    else:
      prom_data  = "# HELP my_test_data_01 just some test data\n"
      prom_data += "# TYPE my_test_data_01 gauge\n"
      prom_data += "my_test_data_01 " + str(random.randint(1,100)) + "\n"   

      prom_data += "# HELP my_test_data_02 just some test data\n"
      prom_data += "# TYPE my_test_data_02 gauge\n"
      prom_data += "my_test_data_02 " + str(random.randint(1,100)) + "\n"

      prom_data += "# HELP my_test_data_03 just some test data\n"
      prom_data += "# TYPE my_test_data_03 gauge\n"
      prom_data += "my_test_data_03 " + str(random.randint(1,100)) + "\n" 

      f = open(base_dir + "/my_data.prom", 'w')
      f.write(prom_data)
      f.close() 

  except Exception as e:
    print("Exception {0}", e)
    sys.exit(1)

 

 

 

 

  > sudo chown node_exporter:node_exporter /prometheus/code/prometheus_random.py
  > sudo chown node_exporter:node_exporter /prometheus/code
  > sudo chmod u+x /prometheus/code/prometheus_random.py

 

 

Let me switch to the node_exporter user and run the python code.

 

  > sudo su node_exporter -s /bin/bash
  > /prometheus/code/prometheus_random.py

 

 

Now check the output

 

  > cat /prometheus/metrics/my_data.prom

 


 

Now let me run a curl to prove that node_exporter is getting this extra data

 

 

  > curl -s localhost:9100/metrics  | egrep my_test_data

 


 

Wahoo now let me set up a curl to make up new random data every 5 minutes
Exit out of the node_exporter user

 

 

  > sudo vi /etc/cron.d/my_fake_data

 

And place the following into it

 

*/5 * * * * node_exporter /prometheus/code/prometheus_random.py

 



Now the file should update every 5 minutes
Now to go look at the data
my_test_data_01

 


 

 

Sweet it is working J

 

 


 

References

 

[1]        Installing Prometheus on Ubuntu 20.04

http://www.whiteboardcoder.com/2021/01/installing-prometheus-on-ubuntu-2004.html
Accessed 1/2020

[2]        MONITORING LINUX HOST METRICS WITH THE NODE EXPORTER

https://prometheus.io/docs/guides/node-exporter/
Accessed 1/2020

[3]        Node Exporeter’s Github page

http://www.whiteboardcoder.com/2021/01/installing-prometheus-on-ubuntu-2004.html
Accessed 1/2020

[4]        How To Install Prometheus on Ubuntu 16.04

https://www.digitalocean.com/community/tutorials/how-to-install-prometheus-on-ubuntu-16-04
Accessed 1/2020

 

 

No comments:

Post a Comment