Finally getting around to installing prometheus/grafana on a local home server so I can graph a thing or two J https://prometheus.io/ [1]

I used this guide as a reference https://www.digitalocean.com/community/tutorials/how-to-install-prometheus-on-ubuntu-16-04 [2]

Download and setup

First see what version of ubuntu you are on

> lsb_release -a

On this particular server I have set up a second drive located at /prometheus and that is where I am going to place prometheus data

Go check out the current prometheu download at
https://prometheus.io/download/ [3]

Use their nice pop up tool to narrow the downloads to linux

Right click on the file name and copy the link address

Now use wget to download it locally

> wget https://github.com/prometheus/prometheus/releases/download/v2.24.0/prometheus-2.24.0.linux

-amd64.tar.gz

Once it downloads confirm its sha

> sha256sum prometheus-2.24.0.linux-amd64.tar.gz

If that number matches the one on the site you are good

Create new users then install

> sudo useradd --no-create-home --shell /bin/false prometheus
> sudo useradd --no-create-home --shell /bin/false node_exporter

Make directories

> sudo mkdir /etc/prometheus
> sudo mkdir /var/lib/prometheus
> sudo chown prometheus:prometheus /etc/prometheus

Let me make a data folder

> sudo mkdir -p /prometheus/data
> sudo chown -R prometheus:prometheus /prometheus/data

Unpack

> tar xvf prometheus-2.24.0.linux-amd64.tar.gz

Move it to /usr/loca/bin

> sudo cp prometheus-2.24.0.linux-amd64/prometheus /usr/local/bin/
> sudo cp prometheus-2.24.0.linux-amd64/promtool /usr/local/bin/

> sudo chown prometheus:prometheus /usr/local/bin/prometheus
> sudo chown prometheus:prometheus /usr/local/bin/promtool

Copy console files over

> sudo cp -r prometheus-2.24.0.linux-amd64/consoles /etc/prometheus
> sudo cp -r prometheus-2.24.0.linux-amd64/console_libraries /etc/prometheus

> sudo chown -R prometheus:prometheus /etc/prometheus/consoles
> sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries

Create prometheus.yml file

> sudo vi /prometheus/prometheus.yml

And place the following in it.

global:
scrape_interval: 15s

scrape_configs:
- job_name: 'prometheus'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']
- job_name: 'node_exporter'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9100']

> sudo chown -R prometheus:prometheus /prometheus/prometheus.yml

Quick test… run manually

> sudo -u prometheus /usr/local/bin/prometheus \
--config.file /prometheus/prometheus.yml \

--storage.tsdb.path /prometheus/data

Wahoo that works.

Now to set up a SystemD to run it

SystemD setup

Create the systemD file

> sudo vi /lib/systemd/system/prometheus.service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
TimeoutStartSec=0
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
ExecStart=/usr/local/bin/prometheus \
--config.file /prometheus/prometheus.yml \
--storage.tsdb.path /prometheus/data \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries \
--log.level=info \
--storage.tsdb.retention.time=400d

Restart=always
RestartSec=10s
NotifyAccess=all

[Install]
WantedBy=multi-user.target

> sudo systemctl status prometheus

Enable the service so it will auto start on reboot

> sudo systemctl enable prometheus

> sudo systemctl status prometheus

Wahoo

Test it out

> sudo systemctl start prometheus
> sudo systemctl status prometheus

Now pull ports and test.

> ssh prometheus -L 9090:localhost:9090

http://localhost:9090/

Working J

Reboot and check

No last check reboot the box and see if it all comes back up.

> sudo reboot now

Log back in with port forwarding

> ssh prometheus -L 9090:localhost:9090

Check status

> sudo systemctl status prometheus

Log back in

http://localhost:9090/

Wahoo it rebooted.

Get some data

In the /prometheus/prometheus.yml I defined some targets to pull from

> sudo vi /prometheus/prometheus.yml

Here it will scrape http://localhost:9090/metrics and http://localhost:9100/metrics

OK after a reboot let me log back in and pull both these ports

> ssh prometheus -L 9090:localhost:9090 -L 9100:localhost:9100

If I open up

http://192.168.0.101:9090/metrics

I see prometheus data

If I open http://192.168.0.101:9100/metrics

I get nothing…
Because I have not yet installed nodeexporter locally https://github.com/prometheus/node_exporter [4]
So… for now we won’t get data from that source.

Open up http://192.168.0.101:9090/

Open up Targets

The first one "node_exporter" is down and not able to scrape because node exporter is not installed… so that makes sense

But the promeheus one…. Should be working

Look at the time -7 hours and out of bounds….
I think there is a bug

Poking around I think this may have some clues on why https://github.com/prometheus/prometheus/issues/8243 [5]

How to fix this?

Just stop and start prometheus…
For some reason on a reboot it will have this issue again

> sudo systemctl stop prometheus
> sudo systemctl start prometheus

Now check http://192.168.0.101:9090/targets again

Now its happy J

Now open Graph

Enter in go_goroutines and click Execute

But I get no results…

I believe this is because my server is on UTC and this is using the local time of my server so to fix that… or at least get an answer

Switch it to Graph and up the hours to 12 hours or even 1d and hopefully in a few minutes you will see data J

Fixing the time issue…

On my server I am having what I think are time sync issues.
My server is a virtual ESXi server and when I auto start prometheus on restart I am having time issues.

Poking around I think what may be happening is a server time issue within the server. That maybe when it starts up the time is wayyyy of and prometheus starts then the time is fixed and now prometheus is in a bad state.

If I do not enable prometheus to start on a boot but only start it myself by hand I have no problem. But when I auto start it… problem.

So Maybe a solution would be for the prometheus to wait to start until the time has synced?