Installing Prometheus on Ubuntu 20.04

Posted on Wednesday, January 20, 2021


 


Finally getting around to installing prometheus/grafana on a local home server so I can graph a thing or two J  https://prometheus.io/ [1]

 

I used this guide as a reference https://www.digitalocean.com/community/tutorials/how-to-install-prometheus-on-ubuntu-16-04 [2]

 


 

Download and setup

First see what version of ubuntu you are on

 

  > lsb_release -a

 

 


On this particular server I have set up a second drive located at /prometheus and that is where I am going to place prometheus data

Go check out the current prometheu download at
https://prometheus.io/download/ [3]

 



Use their nice pop up tool to narrow the downloads to linux

Right click on the file name and copy the link address

 



Now use wget to download it locally

 

  >  wget https://github.com/prometheus/prometheus/releases/download/v2.24.0/prometheus-2.24.0.linux

-amd64.tar.gz

 


Once it downloads confirm its sha

 

  >  sha256sum prometheus-2.24.0.linux-amd64.tar.gz

 



If that number matches the one on the site you are good

 




Create new users then install

 

  > sudo useradd --no-create-home --shell /bin/false prometheus
  > sudo useradd --no-create-home --shell /bin/false node_exporter

 



Make directories

 

  > sudo mkdir /etc/prometheus
  > sudo mkdir /var/lib/prometheus
  > sudo chown prometheus:prometheus /etc/prometheus



Let me make a data folder

 

  > sudo mkdir -p /prometheus/data
  > sudo chown -R prometheus:prometheus /prometheus/data


Unpack

 

  > tar xvf prometheus-2.24.0.linux-amd64.tar.gz


Move it to /usr/loca/bin

 

  > sudo cp prometheus-2.24.0.linux-amd64/prometheus /usr/local/bin/
  > sudo cp prometheus-2.24.0.linux-amd64/promtool /usr/local/bin/


 

 

  > sudo chown prometheus:prometheus /usr/local/bin/prometheus
  > sudo chown prometheus:prometheus /usr/local/bin/promtool



Copy console files over

 

  > sudo cp -r prometheus-2.24.0.linux-amd64/consoles /etc/prometheus
  > sudo cp -r prometheus-2.24.0.linux-amd64/console_libraries /etc/prometheus

  > sudo chown -R prometheus:prometheus /etc/prometheus/consoles
  > sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries


 

 

Create prometheus.yml file

 

  > sudo vi /prometheus/prometheus.yml

 

And place the following in it.

 

global:
  scrape_interval:     15s

 scrape_configs:
   - job_name: 'prometheus'
     scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9100']

 

 


 

 

  > sudo chown -R prometheus:prometheus /prometheus/prometheus.yml


Quick test… run manually

 

  > sudo -u prometheus /usr/local/bin/prometheus \
     --config.file  /prometheus/prometheus.yml \

    --storage.tsdb.path /prometheus/data



 



Wahoo that works.

Now to set up a SystemD to run it


 

SystemD setup

 

Create the systemD file

 

  > sudo vi /lib/systemd/system/prometheus.service

 

 

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
TimeoutStartSec=0
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
ExecStart=/usr/local/bin/prometheus \
    --config.file /prometheus/prometheus.yml \
    --storage.tsdb.path /prometheus/data \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries \
    --log.level=info \
    --storage.tsdb.retention.time=400d

Restart=always
RestartSec=10s
NotifyAccess=all

[Install]
WantedBy=multi-user.target


 





 

  > sudo systemctl status prometheus

 


 

Enable the service so it will auto start on reboot

 

  > sudo systemctl enable prometheus

 


 

  > sudo systemctl status prometheus

 

Wahoo

Test it out 

 

  > sudo systemctl start prometheus
  > sudo systemctl status prometheus

 

 


 

Now pull ports and test. 

 

  > ssh prometheus -L 9090:localhost:9090


http://localhost:9090/



  Working J

 

 

 

Reboot and check

No last check reboot the box and see if it all comes back up.

 

  > sudo reboot now

 


Log back in with port forwarding

 

  > ssh prometheus -L 9090:localhost:9090

 


Check status

 

  > sudo systemctl status prometheus

 

 

Log back in

http://localhost:9090/


Wahoo it rebooted.

 


 

Get some data

In the /prometheus/prometheus.yml I defined some targets to pull from

 

  > sudo vi /prometheus/prometheus.yml

 

 


Here it will scrape http://localhost:9090/metrics  and http://localhost:9100/metrics

OK after a reboot let me log back in and pull both these ports

 

  > ssh prometheus -L 9090:localhost:9090 -L 9100:localhost:9100

 


If I open up

http://192.168.0.101:9090/metrics

 


I see prometheus data

If I open http://192.168.0.101:9100/metrics

 


 

I get nothing…
Because I have not yet installed nodeexporter locally https://github.com/prometheus/node_exporter [4]
So… for now we won’t get data from that source.


Open up http://192.168.0.101:9090/

 

Open up Targets



The first one "node_exporter" is down and not able to scrape because node exporter is not installed… so that makes sense

But the promeheus one…. Should be working

 

Look at the time -7 hours and out of bounds….
I think there is a bug

Poking around I think this may have some clues on why https://github.com/prometheus/prometheus/issues/8243 [5]

 

How to fix this?

Just stop and start prometheus…
For some reason on a reboot it will have this issue again

 

  > sudo systemctl stop prometheus
  > sudo systemctl start prometheus

 


Now check http://192.168.0.101:9090/targets again

 


Now its happy J




 

Now open Graph

Enter in go_goroutines and click Execute

 


 

But I get no results…

I believe this is because my server is on UTC and this is using the local time of my server so to fix that… or at least get an answer

 


 

Switch it to Graph and up the hours to 12 hours or even 1d and hopefully in a few minutes you will see data J

 

 


Fixing the time issue…

 

On my server I am having what I think are time sync issues.
My server is a virtual ESXi server and when I auto start prometheus on  restart I am having time issues.

Poking around I think what may be happening is a server time issue within the server.   That maybe when it starts up the time is wayyyy of and prometheus starts then the time is fixed and now prometheus is in a bad state.

If I do not enable prometheus to start on a boot but only start it myself by hand I have no problem.  But when I auto start it… problem.

So Maybe a solution would be for the prometheus to wait to start until the time has synced?


 

Look at that it took 30-60 seconds for my server to sync its time…
So this is on me…

Looking at this for help https://blog.debiania.in.ua/posts/2020-11-27-howto-delay-a-systemd-service-until-the-clock-is-synchronized.html [6]

Also some notes here https://raspberrypi.stackexchange.com/questions/94635/how-can-i-delay-the-startup-of-systemd-services-until-the-datetime-is-set-no-rt [7]

More

https://github.com/prometheus/prometheus/issues/8243 [8]


  > sudo systemctl enable --now systemd-time-wait-sync.service

 


OK now edit my prometheus file


 

  > sudo vi /lib/systemd/system/prometheus.service

 

 

Edit this part

 

[Unit]
Description=Prometheus
Wants=time-sync.target
After=time-sync.target

 


 


Enable the service so it will auto start on reboot

 

  > sudo systemctl enable prometheus
  > sudo reboot now

 


Log back in and run this to check

 

  > sudo journalctl -b \
 -u systemd-timesyncd.service \
 -u systemd-time-wait-sync.service \
 -u time-sync.target \
 -u prometheus.service

 

 

 










 

References

 

[1]        Prometheus  
              
https://prometheus.io/
             Accessed 1/2020

[2]       How To Install Prometheus on Ubuntu 16.04

            https://www.digitalocean.com/community/tutorials/how-to-install-prometheus-on-ubuntu-16-04

            Accessed 1/2020

[3]        Prometheus Download page

https://prometheus.io/download/
Accessed 1/2020

[4]        Node Exporter

https://github.com/prometheus/node_exporter
Accessed 1/2020

[5]        Prometheus raise out of bounds error for all targets after resume the
             linux system from a suspend #8243

https://github.com/prometheus/prometheus/issues/8243
Accessed 1/2020

 

No comments:

Post a Comment