Pfsense ntop getting prometheus data

 

I have pfsense with ntopng installed on it.
I would like to get some of this data from ntop into my prometheus se

so I can make cool graphs of my data usage in Grafana.

 

 

 

 

 

 

Ntopng installed

 

If you do not have ntopng installed you can check out my post on it here http://www.whiteboardcoder.com/2018/12/installing-ntopng-on-pfsense.html [1]

 

 

 

 

Open up ntopng under the Diagnostics menu

This will require a login

After you login open up the /metrics page
In my case its https://192.168.0.1:3000/metrics

 

 

You should see lots of prometheus style data points

Now you can get prometheus to pull data from here using this URL but you would also need to give prometheus the user name and password for your admin.  Probably a bad idea.

Instead let’s make a limited user.

 

 

 

Click on Manage Users

On the far right click Add a User

 

Make a new user but make them Non Privilaged.
Now we can use this user to get the data from.

Update Prometheus scrape settings

 

My prometheus has its setting file at sudo vi /prometheus/prometheus.yml see http://www.whiteboardcoder.com/2021/01/installing-prometheus-on-ubuntu-2004.html [2] if you are curious how I set it up.

 

> sudo vi /prometheus/prometheus.yml

 

And add the following to scrape

 

- job_name: 'pfsense_topng'     scrape_interval: 5s     static_configs:       - targets: ['192.168.0.1:3000']     scheme: https     basic_auth:       username: 'prometheus'       password: 'prometheus'     tls_config:         insecure_skip_verify: true

Save the file and restart prometheus

> sudo systemctl restart prometheus

 

It may be a good idea to confirm that it is not being scraped

Let me log back into my prometheus server but pull port 9090/3000 to local

 

> ssh prometheus -L 9090:localhost:9090 -L 3000:localhost:3000

Open http://localhost:9090/targets

 

I can see that it is pulling info.

 

 

Now what do with the info in Grafana?

 

Let me log into grafana and make a new board

http://localhost:3000/

 

 

 

Now for some fun queries to get data out.
Let me run a curl with password on it.

 

> curl -s --insecure -u prometheus:prometheus https://192.168.0.1:3000/metrics

 

OK now grep it

 

> curl -s --insecure -u prometheus:prometheus https://192.168.0.1:3000/metrics \  | egrep stats.bytes.sent | egrep igb0

 

 

Here are some queries I came up with

 

The total of Bytes sent from interface igb0 (my wan) per hour rate

sum(increase(hosts{ifname="igb0",metric="stats.bytes.sent"} [1h]))

 

The total of Bytes received from interface igb0 (my wan) per hour rate

sum(increase(hosts{ifname="igb0",metric="stats.bytes.rcvd"} [1m]))

 

The total of Bytes sent from interface igb0 (my wan) per hour rate

12*sum(increase(hosts{ifname="igb0",metric="stats.bytes.sent"} [5m]))

 

The total of Bytes received from interface igb0 (my wan) per hour rate

12*sum(increase(hosts{ifname="igb0",metric="stats.bytes.rcvd"} [5m]))

 

 

sum(increase(hosts{ifname="igb0",metric="stats.bytes.rcvd"} [28d]))/(1024*1024*1024) + sum(increase(hosts{ifname="igb0",metric="stats.bytes.sent"} [28d]))/(1024*1024*1024)

 

Total in and out over a 28 day period

 

Who are my offenders on my other networks..

Who received  more than 2GiB/hr rate

 

increase(hosts{ifname="igb1",metric="stats.bytes.rcvd"} [1h]) > 2*1024*1024*1024

 

Who sent more than 2 GiB /hr rate

increase(hosts{ifname="igb1",metric="stats.bytes.sent"} [1h]) > 2*1024*1024*1024

 

After a little fiddling

 

 

 

Of course I have had data coming in for a few weeks now so your mileage may vary until you have more data.

My one gripe so far on this would it would be nice if they added hostname to the data.  I do get the IP address of offenders but then I have to go look it up.

 

 

A word of warning …

Looks like this prometheus export was experimental and has been dropped in ntopng 4.0
https://github.com/ntop/ntopng/issues/3705 

 

 

References

 

[1]        Installing ntopng on pfsense
              http://www.whiteboardcoder.com/2018/12/installing-ntopng-on-pfsense.html
             Accessed 02/2021

[2]        Installing Prometheus on Ubuntu 20.04

http://www.whiteboardcoder.com/2021/01/installing-prometheus-on-ubuntu-2004.html
Accessed 02/2021