Pfsense ntop getting prometheus data

Posted on Sunday, February 14, 2021


 

I have pfsense with ntopng installed on it.
I would like to get some of this data from ntop into my prometheus se

so I can make cool graphs of my data usage in Grafana.

 

 

 

 

 


 

Ntopng installed

 

If you do not have ntopng installed you can check out my post on it here http://www.whiteboardcoder.com/2018/12/installing-ntopng-on-pfsense.html [1]

 

 


 

 

Open up ntopng under the Diagnostics menu



This will require a login

After you login open up the /metrics page
In my case its https://192.168.0.1:3000/metrics

 

 


You should see lots of prometheus style data points


Now you can get prometheus to pull data from here using this URL but you would also need to give prometheus the user name and password for your admin.  Probably a bad idea.

Instead let’s make a limited user.

 

 

 

Click on Manage Users

On the far right click Add a User

 



Make a new user but make them Non Privilaged.
Now we can use this user to get the data from.



Update Prometheus scrape settings

 

My prometheus has its setting file at sudo vi /prometheus/prometheus.yml see http://www.whiteboardcoder.com/2021/01/installing-prometheus-on-ubuntu-2004.html [2] if you are curious how I set it up.

 


  > sudo vi /prometheus/prometheus.yml

 

And add the following to scrape

 


  - job_name: 'pfsense_topng'
    scrape_interval: 5s
    static_configs:
      - targets: ['192.168.0.1:3000']
    scheme: https
    basic_auth:
      username: 'prometheus'
      password: 'prometheus'
    tls_config:
        insecure_skip_verify: true






Save the file and restart prometheus



  > sudo systemctl restart prometheus

 

It may be a good idea to confirm that it is not being scraped

Let me log back into my prometheus server but pull port 9090/3000 to local

 


  > ssh prometheus -L 9090:localhost:9090 -L 3000:localhost:3000


Open
http://localhost:9090/targets

 

I can see that it is pulling info.

 


 

Now what do with the info in Grafana?

 

Let me log into grafana and make a new board

http://localhost:3000/

 



 




 


Now for some fun queries to get data out.
Let me run a curl with password on it.

 


  > curl -s --insecure -u prometheus:prometheus https://192.168.0.1:3000/metrics

 

OK now grep it

 


  > curl -s --insecure -u prometheus:prometheus https://192.168.0.1:3000/metrics \
 | egrep stats.bytes.sent | egrep igb0

 

 

Here are some queries I came up with

 

The total of Bytes sent from interface igb0 (my wan) per hour rate


sum(increase(hosts{ifname="igb0",metric="stats.bytes.sent"} [1h]))

 


The total of Bytes received from interface igb0 (my wan) per hour rate


sum(increase(hosts{ifname="igb0",metric="stats.bytes.rcvd"} [1m]))

 

The total of Bytes sent from interface igb0 (my wan) per hour rate


12*sum(increase(hosts{ifname="igb0",metric="stats.bytes.sent"} [5m]))

 

The total of Bytes received from interface igb0 (my wan) per hour rate


12*sum(increase(hosts{ifname="igb0",metric="stats.bytes.rcvd"} [5m]))

 


 



sum(increase(hosts{ifname="igb0",metric="stats.bytes.rcvd"} [28d]))/(1024*1024*1024) + sum(increase(hosts{ifname="igb0",metric="stats.bytes.sent"} [28d]))/(1024*1024*1024)


 

Total in and out over a 28 day period

 

Who are my offenders on my other networks..


Who received  more than 2GiB/hr rate

 


increase(hosts{ifname="igb1",metric="stats.bytes.rcvd"} [1h]) > 2*1024*1024*1024


 

Who sent more than 2 GiB /hr rate


increase(hosts{ifname="igb1",metric="stats.bytes.sent"} [1h]) > 2*1024*1024*1024

 


After a little fiddling

 

 

 

Of course I have had data coming in for a few weeks now so your mileage may vary until you have more data.


My one gripe so far on this would it would be nice if they added hostname to the data.  I do get the IP address of offenders but then I have to go look it up.

 

 


A word of warning …

Looks like this prometheus export was experimental and has been dropped in ntopng 4.0
https://github.com/ntop/ntopng/issues/3705 


 

 

References

 

[1]        Installing ntopng on pfsense
              
http://www.whiteboardcoder.com/2018/12/installing-ntopng-on-pfsense.html
             Accessed 02/2021

[2]        Installing Prometheus on Ubuntu 20.04

http://www.whiteboardcoder.com/2021/01/installing-prometheus-on-ubuntu-2004.html
Accessed 02/2021

 

 

 

 

71 comments:

  1. Excellent Blog! I would like to thank you for the efforts you have made in writing this post. Gained lots of knowledge.
    Data Analytics Course

    ReplyDelete
  2. I bookmarked your website because this site contains valuable information. I am very satisfied with the quality and the presentation of the articles. Thank you so much for saving great things. I am very grateful for this site.

    Data Science Training in Bangalore

    ReplyDelete
  3. I have voiced some of the posts on your website now, and I really like your blogging style. I added it to my list of favorite blogging sites and will be back soon ...

    Digital Marketing Training in Bangalore

    ReplyDelete
  4. Awesome article. I enjoyed reading your articles. this can be really a good scan for me. wanting forward to reading new articles. maintain the nice work!
    Data Science Courses in Bangalore

    ReplyDelete
  5. I found Habit to be a transparent site, a social hub that is a conglomerate of buyers and sellers willing to offer digital advice online at a decent cost.

    Artificial Intelligence Training in Bangalore

    ReplyDelete
  6. The Extraordinary blog went amazed by the content that they have developed in a very descriptive manner. This type of content surely ensures the participants explore themselves. Hope you deliver the same near the future as well. Gratitude to the blogger for the efforts.

    Machine Learning Course in Bangalore

    ReplyDelete
  7. I need to thank you for this very good read and i have bookmarked to check out new things from your post. Thank you very much for sharing such a useful article and will definitely saved and revisit your site.
    Data Science Course

    ReplyDelete
  8. What an incredible message this is. Truly one of the best posts I have ever seen in my life. Wow, keep it up.
    AI Courses in Bangalore

    ReplyDelete
  9. Wow, happy to see this awesome post. I hope this think help any newbie for their awesome work and by the way thanks for share this awesomeness, i thought this was a pretty interesting read when it comes to this topic. Thank you..
    Artificial Intelligence Course

    ReplyDelete
  10. I am sure it will help many people. Keep up the good work. It's very compelling and I enjoyed browsing the entire blog.
    Business Analytics Course in Bangalore

    ReplyDelete
  11. Thank a lot. You have done excellent job. I enjoyed your blog . Nice efforts
    Data Science Certification in Hyderabad

    ReplyDelete
  12. Your site is truly cool and this is an extraordinary moving article and If it's not too much trouble share more like that. Thank You..
    Digital Marketing Course in Hyderabad

    ReplyDelete
  13. Unfortunately the developers have removed the support for Prometheus https://github.com/ntop/ntopng/issues/3705

    ReplyDelete
  14. I am glad to discover this page. I have to thank you for the time I spent on this especially great reading !! I really liked each part and also bookmarked you for new information on your site.
    Data Science Course Syllabus

    ReplyDelete
  15. I am really enjoying reading your well written articles. I am looking forward to reading new articles. Keep up the good work.
    Data Science Courses in Bangalore

    ReplyDelete
  16. I feel very grateful that I read this. It is very helpful and very informative and I really learned a lot from it.
    Data Analytics Course

    ReplyDelete
  17. Very good message. I stumbled across your blog and wanted to say that I really enjoyed reading your articles.
    AI Courses in Bangalore

    ReplyDelete
  18. I just got to this amazing site not long ago was actually captured with the piece of resources you have got here and big thumbs up for making such wonderful blog page!
    Data Scientist Course

    ReplyDelete
  19. Excellent post to make this blog more wonderful, attractive and cool stuff you have. Thank You.
    Data Science Course in India with Placements

    ReplyDelete
  20. Informative Post. The information you have posted is very useful and sites you have referred was good. Thanks for sharing.
    Data Science Course with Placement

    ReplyDelete
  21. Very interesting post and this is my first time visit here and found so many interesting stuff in your blog especially its discussion. Thanks for the post!
    Digital Marketing Training Institute in Hyderabad

    ReplyDelete
  22. You have done excellent job Thanks a lot and I enjoyed your blog. Great Post.
    Ethical Hacking Course in Hyderabad

    ReplyDelete
  23. Do you want to make sure your digital marketing services is on the right track? Digital World Giant is one of the most renowned agencies in New York. They specialize in Web Development, Digital Marketing and SEO services.

    ReplyDelete
  24. Extraordinary post I should state and a debt of gratitude is in order for the data. Instruction is unquestionably a clingy subject. Be that as it may, is still among the main subjects within recent memory. I value your post and anticipate more.data science training in lucknow

    ReplyDelete
  25. Thank you for sharing this wonderful blog, I read that Post and got it fine and informative. Please share more like that...
    Ethical Hacking Institute in Bangalore

    ReplyDelete
  26. Nice Post. Thank you for helping people get the information they need and great stuff as usual. Keep up the great work!!!
    Artificial Intelligence Courses in Hyderabad

    ReplyDelete
  27. Nice Post thank you very much for sharing such a useful information and will definitely saved and revisit your site and i have bookmarked to check out new things frm your post.
    Data Science Course

    ReplyDelete
  28. A good blog always contains new and exciting information, and reading it I feel like this blog really has all of these qualities that make it a blog.

    Machine Learning Training in Bangalore

    ReplyDelete
  29. Really, this article is truly one of the best in the article. And this one that I found quite fascinating and should be part of my collection. Very good work!.
    Ethical Hacking Course in Bangalore

    ReplyDelete
  30. Very interesting blog. A lot of the blogs I see these days don't provide anything that interests me, but I'm really interested in this one. I just thought I would post and let you know.

    Data Science Course in Ernakulam

    ReplyDelete
  31. Superb Information, I really appreciated with it, This is fine to read and valuable pro potential, I really bookmark it, pro broaden read. Appreciation pro sharing. I like it. Data Analytics Course in Dehradun

    ReplyDelete
  32. I have read your article, it is very informative and useful to me, I admire the valuable information you offer in your articles. Thanks for posting it ...

    Data Science Course in Durgapur

    ReplyDelete
  33. I am excited about this blog. It is an informative subject. It helped me a lot in solving some problems. Your opportunity is so fantastic and the style of work so fast.

    Data Science Training in Ernakulam

    ReplyDelete
  34. Very nice job... Thanks for sharing this amazing and educative blog post!
    Data Science Course in Chandigarh

    ReplyDelete
  35. This is an excellent article. I like this topic. This site has many advantages. I have found a lot of interesting things on this site. It helps me in so many ways. Thanks for posting this again.

    Data Analytics Course in Durgapur

    ReplyDelete
  36. I always search online for articles that can help me. Obviously, there is a lot to know about this. I think you made a few good points about the features as well. Keep up the good work!

    Data Scientist Course in Ernakulam

    ReplyDelete
  37. This blog was really great, never seen a great blog like this before. i think im gonna share this to my friends..

    Data Scientist Training in Bangalore

    ReplyDelete
  38. Very informative Blog! There is so much information here that can help thank you for sharing.
    Data Analytics Training in Bangalore

    ReplyDelete
  39. Thank you for posting this information. I just want to let you know that I just visited your site and find it very interesting and informative. I look forward to reading most of your posts.

    Data Science Course in Patna

    ReplyDelete
  40. Good blog and absolutely exceptional. You can do a lot better, but I still say it's perfect. Keep doing your best.

    Data Analytics Course in Nagpur

    ReplyDelete
  41. It is late to find this act. At least one should be familiar with the fact that such events exist. I agree with your blog and will come back to inspect it further in the future, so keep your performance going.

    Best Data Analytics Courses in Bangalore

    ReplyDelete
  42. I really appreciate this wonderful message you have given us. I assure you that would be beneficial for most people.

    Data Analytics Course in Nagpur

    ReplyDelete
  43. Really impressed! Everything is a very open and very clear clarification of the issues. It contains true facts. Your website is very valuable. Thanks for sharing.

    Best Data Analytics Courses in Bangalore

    ReplyDelete
  44. Wonderful blog post. It's absolute magic on your part! I have never seen a more wonderful article than this. You really made my day today with this. Hope you continue like this!

    Data Scientist Course in Patna

    ReplyDelete
  45. I wanted to leave a little comment to support you and wish you the best of luck. We wish you the best of luck in all of your blogging endeavors.

    Data Analytics Course in Patna

    ReplyDelete
  46. Wonderful illustrated information. Thank you. It will certainly be very useful for my future projects. I would love to see more articles on the same topic!

    Data Science Training in Bangalore

    ReplyDelete
  47. Happy to chat on your blog, I feel like I can't wait to read more reliable posts and think we all want to thank many blog posts to share with us.

    Data Science Training Institutes in Bangalore

    ReplyDelete
  48. Thank you very much for this interesting article. In fact, it is exceptional. You are looking for this type of notice later.

    Data Scientist Course in Nagpur

    ReplyDelete
  49. Happy to chat on your blog, I feel like I can't wait to read more reliable posts and think we all want to thank many blog posts to share with us.

    Data Science in Bangalore

    ReplyDelete
  50. This is just the information I find everywhere. Thank you for your blog, I just subscribed to your blog. It's a good blog.

    Best Data Analytics Courses in Bangalore

    ReplyDelete
  51. I think this is an informative and very useful and knowledgeable blog. therefore, I would like to thank you for your effort.
    Data Science Course in Amritsar

    ReplyDelete
  52. I am overwhelmed by your post with such a beautiful subject. I usually visit their blogs and update myself through the information they contain, but today's blog would be most appreciated. Well done!

    Business Analytics Course in Durgapur

    ReplyDelete
  53. I couldn't leave your website until I told you that I really appreciated the high quality information it presents to your visitors. I will come back frequently to check for new posts.

    Data Scientist Course in Durgapur

    ReplyDelete
  54. Really nice and amazing post. I was looking for this kind of information, Keep posting. Thanks for sharing.
    Data Science Courses in Bangalore

    ReplyDelete
  55. We are really grateful for your blog post. You will find a lot of approaches after visiting your post. Great work Business Analytics Course in Vadodara

    ReplyDelete
  56. I have read good things here. Definitely worth to bookmark again. I am amazed at how much effort you put into creating such an informative website.

    Data Science Training in Bangalore

    ReplyDelete
  57. Thanks for sharing this valuable information, we also provide instagram video download and,
    really appreciate your hard work and good research that help us in for of your good valuable article. Keep updating us with your awesome content.

    ReplyDelete
  58. Very happy to find a good place for many here in the post, the writing is just great, thanks for the post.

    Data Science Course in Ernakulam

    ReplyDelete
  59. Some really good points you wrote here ... Great things ... I think you raised some really interesting points. Keep up the good work.

    Best Data Analytics Courses in Bangalore

    ReplyDelete
  60. It was a wonderful opportunity to visit this type of site and I am happy to hear about it. Thank you very much for giving us the opportunity to have this opportunity.

    Data Analytics Course in Durgapur

    ReplyDelete
  61. I am delighted to discover this page. I must thank you for the time you devoted to this particularly fantastic reading !! I really liked each part very much and also bookmarked you to see new information on your site.

    Business Analytics Course in Durgapur

    ReplyDelete
  62. Good blog and absolutely exceptional. You can do a lot better, but I still say it's perfect. Keep doing your best.

    Data Science Training in Durgapur

    ReplyDelete
  63. Thanks for sharing this great article we appreciate it, we provide instagram reels download freely and unlimited.

    ReplyDelete
  64. I was very happy to find this site. I wanted to thank you for this excellent reading !! I really enjoy every part and have bookmarked you to see the new things you post.

    Data Science Course in Durgapur

    ReplyDelete
  65. Superb Information, I really appreciated with it, This is fine to read and valuable pro potential, I really bookmark it, pro broaden read. Appreciation pro sharing. I like it.Data Analytics Course in Dombivli

    ReplyDelete
  66. 360DigiTMG is the most recommended Data Science course institute in Chennai. Get trained by top professionals from IIT, IIM and, ISB. Enroll now!

    Data Science Course in Jodhpur

    ReplyDelete
  67. Learn the fundamentals of Data Science and master the skills to handle epic amounts of data collected by organizations today. Gain expertise in organizing, analyzing, and transforming data to uncover its hidden power. Drive your career forward with a Data Science course in Bangalore and learn to uncover insights to present critical findings using visualization tools. Not only this, avail the best-in-class content delivered by stellar faculty who use a blended approach of theory as well as practical knowledge to ensure all the concepts are crystal clear.

    Best Data Science Institute in Bangalore

    ReplyDelete
  68. Register for the Data Science certification in Bangalore and gain recognition and credibility in your organization. Learn the techniques to examine large data sets and discover patterns that are valuable to predict market trends. Learn Data Exploration and Visualizations, Neural Networks and Deep Learning, Model Evaluation and Analysis. This course will ensure that you are challenged to go from a beginner with no Data Science experience to someone who can juggle data with ease.

    Data Science Training and Placement in Bangalore

    ReplyDelete
  69. Register for the Data Science certification in Bangalore and gain recognition and credibility in your organization. Learn the techniques to examine large data sets and discover patterns that are valuable to predict market trends. Learn Data Exploration and Visualizations, Neural Networks and Deep Learning, Model Evaluation and Analysis. This course will ensure that you are challenged to go from a beginner with no Data Science experience to someone who can juggle data with ease.

    Data Science Training and Placement in Bangalore

    ReplyDelete