It's time to try another
monitoring solution! I have been using
Sensu for years and I really like it. It
has been an effective tool. But, every
so often you need to get out of your comfort zone and try something new.
In this article I am
going to install Prometheus on an Ubuntu 16.04 server and try to monitor
something J
The official "Hello
world" installation guide from Prometheus can be found here https://prometheus.io/docs/introduction/getting_started/ [1] I will
be using this guide myself.
Installing
Select Linux and amd64
from the pull down.
Right click on 1.5.2 and
select copy address. In my case I got
this URL https://github.com/prometheus/prometheus/releases/download/v1.5.2/prometheus-1.5.2.linux-amd64.tar.gz
Run this command to
download it.
> wget https://github.com/prometheus/prometheus/releases/download/v1.5.2/prometheus-1.5.2.linux-amd64.tar.gz
|
Untar it
> tar -xvf
prometheus-1.5.2.linux-amd64.tar.gz
|
Then cd into the directory and run it.
> cd prometheus-1.5.2.linux-amd64/
|
Quoting from the Prometheus guide
Prometheus collects metrics from
monitored targets by scraping metrics HTTP endpoints on these targets. Since
Prometheus also exposes data in the same manner about itself, it can also
scrape and monitor its own health.
So in short we can use Prometheus to monitor Prometheus, as
a starting point.
Edit the Prometheus.yml file
> vi prometheus.yml
|
The default file looks fine as is.
This scrape_configs: section should get information from
localhost:9090.
Start Prometheus
To start Prometheus run this command from the Prometheus
directory.
> ./prometheus
-config.file=prometheus.yml
|
Now open it up in a URL. Open up http://localhost:9090/ (In my case the machine is running at
192.168.0.120 so I opened up http://192.168.0.120:9090/ )
Ok now what?
In theory it is
collecting data about itself.
Hey look data
From the pull down select prometheys_target_interval_length_seconds
Now click Execute.
This has returned every data point for this variable.
Graph it
If I click Graph I see this…. This graph looks a little confusing at
first. The Y starts at 15 s (which may be fine for detail but I would
rather push it down to 0.
I do not see a way to do it here. But looking around this is not the place to
do it anyway https://prometheus.io/docs/visualization/browser/
[4]
This is primarily useful for ad-hoc
queries and debugging. For graphs, use Grafana or Console templates.
So in other words this is not going to be the place to view
your graphs so don't worry about it looking perfect here.
Looking at this graph is a bit confusing at first. One data type was chosen, yet we are getting
back 5 different data types??
If you look at the legend you will see that there are two
sections the
If I do a quick curl with grep I can confirm that this data
is being scraped from http://localhost:9090/metrics
> curl --silent localhost:9090/metrics \
| grep
"prometheus_target_interval_length_seconds{"
|
I can see that it does produce 5 data points. The difference being the different labels
(see the quantile label)
If I want to filter based on this value I can with a promQL
query like this.
prometheus_target_interval_length_seconds{quantile="0.5"}
|
I then get the one data point
What if I wanted to sum or average these data points (I know
with these data points it does not make sense to do that… but it will with
other data so I just want to know)
sum(prometheus_target_interval_length_seconds)
|
avg(prometheus_target_interval_length_seconds)
|
Getting outside data?
How can I get some outside data into Prometheus? Let
me set up a static web server using python.
Create a folder structure
> mkdir ~/webserver
> cd ~/webserver
|
Create the code
> vi webserver_metrics.py
|
Place the following code in it. (which I tweaked from one I found at https://daanlenaerts.com/blog/2015/06/03/create-a-simple-http-server-with-python-3/
[5])
#!/usr/bin/python3
import
random
from
http.server import BaseHTTPRequestHandler, HTTPServer
#
HTTPRequestHandler class
class
testHTTPServer_RequestHandler(BaseHTTPRequestHandler):
# GET
def do_GET(self):
if self.path == "/metrics":
self.send_response(200)
self.send_header('Content-type','text/plain')
message = "http_random_numbers
" + str(random.randint(1, 10)) + "\n"
self.wfile.write(bytes(message,
"utf8"))
else:
self.send_response(200)
self.send_header('Content-type','text/html')
self.end_headers()
message = "TESTING"
self.wfile.write(bytes(message,
"utf8"))
return
def
run():
print('starting server...')
# Server settings
server_address = ('127.0.0.1', 1234)
httpd = HTTPServer(server_address,
testHTTPServer_RequestHandler)
print('running server...')
httpd.serve_forever()
run()
|
Prometheus's default place to scrape is /metrics.
Make it executable and run it
Make it executable and run it
> ./webserver_metrics.py
|
Curl it a few times to make sure it's producing random
numbers
> curl localhost:1234/metrics
|
Looks good.
This is going to be a very basic exposition using a very
basic text format. For a lot more
details on exposition formats see
Update Prometheus.yml
Stop the running Prometheus and edit the prometheus.yml
file, adding in the ability to scrape data from this new server.
> vi prometheus.yml
|
Add the following to the end
- job_name: 'test_scraper'
static_configs:
- targets: ['localhost:1234']
|
Start Prometheus back up
> ./prometheus
-config.file=prometheus.yml
|
Errors!
When I bring this up I get no errors, but I am not getting
the data point http_random_numbers populated.
It's not there…
Go to Status -> Targets
There I can see its being listed as effectively down. I have malformed
HTTP status code "2"
Editing my webserver
> vi webserver_metrics.py
|
I forgot the end the headers
#!/usr/bin/python3
import
random
from
http.server import BaseHTTPRequestHandler, HTTPServer
#
HTTPRequestHandler class
class
testHTTPServer_RequestHandler(BaseHTTPRequestHandler):
# GET
def do_GET(self):
if self.path == "/metrics":
self.send_response(200)
self.send_header('Content-type','text/plain')
self.end_headers()
message = "http_random_numbers
" + str(random.randint(1, 10)) + "\n"
self.wfile.write(bytes(message,
"utf8"))
else:
self.send_response(200)
self.send_header('Content-type','text/html')
self.end_headers()
message = "TESTING"
self.wfile.write(bytes(message,
"utf8"))
return
def
run():
print('starting server...')
# Server settings
server_address = ('127.0.0.1', 1234)
httpd = HTTPServer(server_address,
testHTTPServer_RequestHandler)
print('running server...')
httpd.serve_forever()
run()
|
Fix that and restart it.
> ./webserver_metrics.py
|
Reload the Prometheus Targets page
And we are listening now!
Now it shows up!
And I can graph it
OK, I think that is enough for a start into Prometheus.
References
[1] Prometheus Getting Started
Guide
[2] Prometheus download page
[3] Vim YAML syntax highlighter
[4] Visualization Browser
[5] Create a simple python
server with Python 3
[6] Exposition Formats
No comments:
Post a Comment