Installing Prometheus on Ubuntu 20.04

Posted on Wednesday, January 20, 2021


Finally getting around to installing prometheus/grafana on a local home server so I can graph a thing or two J [1]


I used this guide as a reference [2]



Download and setup

First see what version of ubuntu you are on


  > lsb_release -a



On this particular server I have set up a second drive located at /prometheus and that is where I am going to place prometheus data

Go check out the current prometheu download at [3]


Use their nice pop up tool to narrow the downloads to linux

Right click on the file name and copy the link address


Now use wget to download it locally


  >  wget



Once it downloads confirm its sha


  >  sha256sum prometheus-2.24.0.linux-amd64.tar.gz


If that number matches the one on the site you are good


Create new users then install


  > sudo useradd --no-create-home --shell /bin/false prometheus
  > sudo useradd --no-create-home --shell /bin/false node_exporter


Make directories


  > sudo mkdir /etc/prometheus
  > sudo mkdir /var/lib/prometheus
  > sudo chown prometheus:prometheus /etc/prometheus

Let me make a data folder


  > sudo mkdir -p /prometheus/data
  > sudo chown -R prometheus:prometheus /prometheus/data



  > tar xvf prometheus-2.24.0.linux-amd64.tar.gz

Move it to /usr/loca/bin


  > sudo cp prometheus-2.24.0.linux-amd64/prometheus /usr/local/bin/
  > sudo cp prometheus-2.24.0.linux-amd64/promtool /usr/local/bin/



  > sudo chown prometheus:prometheus /usr/local/bin/prometheus
  > sudo chown prometheus:prometheus /usr/local/bin/promtool

Copy console files over


  > sudo cp -r prometheus-2.24.0.linux-amd64/consoles /etc/prometheus
  > sudo cp -r prometheus-2.24.0.linux-amd64/console_libraries /etc/prometheus

  > sudo chown -R prometheus:prometheus /etc/prometheus/consoles
  > sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries



Create prometheus.yml file


  > sudo vi /prometheus/prometheus.yml


And place the following in it.


  scrape_interval:     15s

   - job_name: 'prometheus'
     scrape_interval: 5s
      - targets: ['localhost:9090']
  - job_name: 'node_exporter'
    scrape_interval: 5s
      - targets: ['localhost:9100']





  > sudo chown -R prometheus:prometheus /prometheus/prometheus.yml

Quick test… run manually


  > sudo -u prometheus /usr/local/bin/prometheus \
     --config.file  /prometheus/prometheus.yml \

    --storage.tsdb.path /prometheus/data


Wahoo that works.

Now to set up a SystemD to run it


SystemD setup


Create the systemD file


  > sudo vi /lib/systemd/system/prometheus.service




ExecStart=/usr/local/bin/prometheus \
    --config.file /prometheus/prometheus.yml \
    --storage.tsdb.path /prometheus/data \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries \
    --log.level=info \





  > sudo systemctl status prometheus



Enable the service so it will auto start on reboot


  > sudo systemctl enable prometheus



  > sudo systemctl status prometheus



Test it out 


  > sudo systemctl start prometheus
  > sudo systemctl status prometheus




Now pull ports and test. 


  > ssh prometheus -L 9090:localhost:9090


  Working J




Reboot and check

No last check reboot the box and see if it all comes back up.


  > sudo reboot now


Log back in with port forwarding


  > ssh prometheus -L 9090:localhost:9090


Check status


  > sudo systemctl status prometheus



Log back in


Wahoo it rebooted.



Get some data

In the /prometheus/prometheus.yml I defined some targets to pull from


  > sudo vi /prometheus/prometheus.yml



Here it will scrape http://localhost:9090/metrics  and http://localhost:9100/metrics

OK after a reboot let me log back in and pull both these ports


  > ssh prometheus -L 9090:localhost:9090 -L 9100:localhost:9100


If I open up


I see prometheus data

If I open



I get nothing…
Because I have not yet installed nodeexporter locally [4]
So… for now we won’t get data from that source.

Open up


Open up Targets

The first one "node_exporter" is down and not able to scrape because node exporter is not installed… so that makes sense

But the promeheus one…. Should be working


Look at the time -7 hours and out of bounds….
I think there is a bug

Poking around I think this may have some clues on why [5]


How to fix this?

Just stop and start prometheus…
For some reason on a reboot it will have this issue again


  > sudo systemctl stop prometheus
  > sudo systemctl start prometheus


Now check again


Now its happy J


Now open Graph

Enter in go_goroutines and click Execute



But I get no results…

I believe this is because my server is on UTC and this is using the local time of my server so to fix that… or at least get an answer



Switch it to Graph and up the hours to 12 hours or even 1d and hopefully in a few minutes you will see data J



Fixing the time issue…


On my server I am having what I think are time sync issues.
My server is a virtual ESXi server and when I auto start prometheus on  restart I am having time issues.

Poking around I think what may be happening is a server time issue within the server.   That maybe when it starts up the time is wayyyy of and prometheus starts then the time is fixed and now prometheus is in a bad state.

If I do not enable prometheus to start on a boot but only start it myself by hand I have no problem.  But when I auto start it… problem.

So Maybe a solution would be for the prometheus to wait to start until the time has synced?


Look at that it took 30-60 seconds for my server to sync its time…
So this is on me…

Looking at this for help [6]

Also some notes here [7]

More [8]

  > sudo systemctl enable --now systemd-time-wait-sync.service


OK now edit my prometheus file


  > sudo vi /lib/systemd/system/prometheus.service



Edit this part





Enable the service so it will auto start on reboot


  > sudo systemctl enable prometheus
  > sudo reboot now


Log back in and run this to check


  > sudo journalctl -b \
 -u systemd-timesyncd.service \
 -u systemd-time-wait-sync.service \
 -u \
 -u prometheus.service







[1]        Prometheus  
             Accessed 1/2020

[2]       How To Install Prometheus on Ubuntu 16.04


            Accessed 1/2020

[3]        Prometheus Download page
Accessed 1/2020

[4]        Node Exporter
Accessed 1/2020

[5]        Prometheus raise out of bounds error for all targets after resume the
             linux system from a suspend #8243
Accessed 1/2020


1 comment: