I have the uchiwa dashboard installed to display my locally
running Sensu monitoring system.
Uchiwa's github page it located at https://github.com/sensu/uchiwa
[1].
I am working on a team using a different Dashboard for Sensu. The now deprecated, Sensu-dashboard. https://github.com/sensu/sensu-dashboard
[2]
Our team is, more than likely, going to be updating our
Sensu Set up in the near future. A
thought came up on the team… More and
more teams, outside ours, may want to use Sensu in the future. If/when they do and if they make use of the
same Sensu Server and dashboard we have a problem.
Simply stated if 2+ teams are using the same Sensu-dashboard
no team can use the dashboard easily. If
there are issues on the board, are they your teams or someone else's?
Each team could just spin up its own Sensu setup. That may be the answer. But, I think we may be able to use the uchiwa
dashboard to filter events by team. If
we can this has the advantage of only needing one board to look at.
Looking at my own Uchiwa set up I can see that I have one
DataCenter I can filter on "Sensu".
I only has one Sensu Master set up.
Can I set up a second Sensu master that monitors different Sensu
clients, yet reports to this same dashboard under a different name?
Change Datacenter's Name
I already have uchiwa set up and running at home. My notes on how I did this install and set up
are at http://www.whiteboardcoder.com/2014/10/sensu-getting-started.html
The first thing I want to try to do is see if I can change
the name of the datacenter, currently its "Sensu".
Edit the uchiwa.json
file.
> sudo vi /etc/sensu/uchiwa.json
|
Here is my current
file
{
"sensu": [
{
"name": "Sensu",
"host": "127.0.0.1",
"ssl": false,
"port": 4567,
"user": "",
"pass": "",
"path": "",
"timeout": 5000
}
],
"uchiwa": {
"user": "",
"pass": "",
"port": 3000,
"stats": 10,
"refresh": 10000
}
}
|
I am simply going to update the name to "Team_One"
Save it and restart uchiwa
>
sudo /etc/init.d/uchiwa restart
|
That worked!
Now it's listed as Team_One.
Clicking on it filters the result on any page you are on.
At this point I only have data from TEAM_ONE so it's not
much of a filter.
Setting up another Sensu Master/Client
I am going to set up another Sensu Master/Client for
TEAM_TWO. I set this server up to run on
IP address 192.168.0.0.160.
I am installing this on an Ubuntu 14.04 server.
Before I get too far, sense I
have a fresh install of Ubuntu 14.04, I need to update and upgrade apt-get.
> sudo apt-get update
> sudo apt-get upgrade
|
Installing RabbitMQ
Sensu uses RabbitMQ. I am following the guide at http://www.rabbitmq.com/install-debian.html
[2]
Edit to /etc/apt/sources.list
> sudo vi /etc/apt/sources.list
|
Append this to the end
deb http://www.rabbitmq.com/debian/ testing main
|
Add an ssl key
> wget
http://www.rabbitmq.com/rabbitmq-signing-key-public.asc
> sudo apt-key add
rabbitmq-signing-key-public.asc
> sudo apt-get update
|
Install rabitmq-server
> sudo apt-get install rabbitmq-server
|
The RabbitMQ server
should have started, but if not you can start it with this command.
> sudo service rabbitmq-server start
|
Create and and edit
the /etc/rabbitmq/rabbitmq.config
> sudo vi
/etc/rabbitmq/rabbitmq.config
|
Put the following in
it
[
{rabbit, [
{tcp_listeners,[{"127.0.0.1",5672}]}
]}
].
|
Create a virtual host and user for Sensu. (Change mypassword to whatever password you
want)
> sudo rabbitmqctl add_vhost /sensu
> sudo rabbitmqctl add_user sensu mypassword
> sudo rabbitmqctl set_permissions -p
/sensu sensu ".*" ".*" ".*"
|
Restart rabbitMQ
> sudo service rabbitmq-server restart
|
Enable the web UI interface
(Optional… I don't think you need to do this, but I am going to)
> sudo rabbitmq-plugins enable
rabbitmq_management
|
Restart rabbitMQ
> sudo service rabbitmq-server restart
|
Make sure the web UI is running by running this command
> netstat -teapu | grep 15672
|
Open the web UI in my case 192.168.0.160:15672
Login with guest:guest
Click on Admin
Click Add a user
Make an admin user.
Set a name/password then click on Admin, or enter administrator in the
Tags section. Click Add user.
The user has been added now click on the name.
Click on Set Permission/
Now this user should be set up.
Remove the guest user
To remove the guest user, do the following. (It's a personal goal of mine to make super sure the default logins are removed ASAP)
Open the admin tab.
Click on the guest user.
Scroll to the bottom and click on Delete this user.
Then click Delete.
You get a pop up.
Click OK
Head back to http://localhost:15672/
and login as your new admin.
Install Redis
> sudo apt-get install redis-server
|
Install Sensu
Edit to /etc/apt/sources.list
> sudo vi /etc/apt/sources.list
|
Append this to the end
deb http://repos.sensuapp.org/apt sensu main
|
Add an ssl key
> wget -q
http://repos.sensuapp.org/apt/pubkey.gpg
> sudo apt-key add pubkey.gpg
> sudo apt-get update
|
Install Sensu
> sudo apt-get install sensu
|
Configure the Sensu Master
Create the
rabbitmq.json file.
> sudo vi
/etc/sensu/conf.d/rabbitmq.json
|
Put the following in
it
{
"rabbitmq": {
"host": "localhost",
"port": 5672,
"vhost": "/sensu",
"user": "sensu",
"password": "mypassword"
}
}
|
Change and edit the
redis.json file
> sudo vi /etc/sensu/conf.d/redis.json
|
And place the
following in it.
{
"redis": {
"host": "localhost",
"port": 6379
}
}
|
Create the api.json
file
> sudo vi /etc/sensu/conf.d/api.json
|
And place the
following in it
{
"api": {
"host": "localhost",
"port": 4567
}
}
|
Enable the services
to start automatically
Run the following
commands
> sudo update-rc.d sensu-server
defaults
> sudo update-rc.d sensu-api defaults
|
Reboot the server
> sudo reboot now
|
Open the URL again
to see if it's running
Setting up Uchiwa to see the second Sensu Server
Hopefully this is relatively painless…
On the Uchiwa Server
Edit the uchiwa.json file.
> sudo vi /etc/sensu/uchiwa.json
|
Here is the updated
file, I added the second team
{
"sensu": [
{
"name": "Team_One",
"host": "127.0.0.1",
"ssl": false,
"port": 4567,
"user": "",
"pass": "",
"path": "",
"timeout": 5000
},
{
"name": "Team_Two",
"host": "192.168.0.160",
"ssl": false,
"port": 4567,
"user": "",
"pass": "",
"path": "",
"timeout": 5000
}
],
"uchiwa": {
"user": "",
"pass": "",
"port": 3000,
"stats": 10,
"refresh": 10000
}
}
|
Save it and restart uchiwa
>
sudo /etc/init.d/uchiwa restart
|
Let me see if it worked.
I'm going to open my Uchiwa dashboard at http:/192.168.0.150:3000/
Hey it worked! I now
have a TEAM_TWO.
I currently have some checks going off on for TEAM_ONE, but
none for TEAM_TWO, in fact TEAM_TWO does not even have a Sensu client yet.
If I click on TEAM_TWO to filter the list I get
Nothing,
as expected.
If I
click on TEAM_ONE I get.
A list of all the current TEAM_ONE events.
I am seeing this error pop up.
On the right hand tool bar click on i (Information). Looking at this page I can see the problem,
a little more clearly….
It's just plain old not talking to the second server. Why?
Tracking down the error.
Before I move on… I think this error needs to be fixed. If I look at the uchiwa.log file.
> tail -f /var/log/uchiwa.log
|
I see this error \"Error: connect ECONNREFUSED\"
Yep, its not connecting.
Running
> netstat --listen
|
On both boxes. I
can see my new Sensu box has nothing running on port 4567
Missing…
I think I found the problem….
My /etc/sensu/conf.d/api.json file is wrong, probably a copy and
paste error I did not notice.
OK, not that looks better.
Next, time I will follow my own instructions!
I am going to just
reboot the server.
> sudo reboot now
|
Running netstat again
> netstat --listen
|
Fixed! Although it
was my own dumb fault.
Going back to the Uchiwa Dashboard I can see that it is now
reading from Team_Two. No need to reboot
it, the changes were picked up J
Create a Sensu Client for TEAM_TWO
OK now that I
finally go the second Sensu Master server working so Uchiwa could pick it up I
need to create a Sensu client and a check,
so I have something to view.
Create the
client.json file
> sudo vi
/etc/sensu/conf.d/client.json
|
And place the
following in it
{
"client": {
"name": "team_two-senus-master",
"address": "192.168.0.160",
"subscriptions": [
"ALL" ]
}
}
|
Start the client
> sudo service sensu-client start
|
Click on the Client
tool.
In a few seconds I
saw the team_two-sensu-master client show up.
Now it just needs an actual check.
Install Ruby
> sudo apt-get install ruby ruby-dev
build-essential
|
Install the sensu-plugin gem
> sudo gem install sensu-plugin
|
Create a check
Install the mixlib-cli gem
> sudo gem install mixlib-cli
|
Create the check-file.rb file
> sudo vi
/etc/sensu/plugins/check-file.rb
|
And place the
following in it
#!/usr/bin/env ruby
require 'sensu-plugin/check/cli' class CheckFile < Sensu::Plugin::Check::CLI option :file, :description => "Path to file", :short => '-f FILE', :long => '--file FILE', :required => true def initilize super end def run if(File.exists?(config[:file])) ok("The file '" + config[:file] + "' Exists! :)") else critical("The file '" + config[:file] + "' dose not Exists! :(") end end end |
This codes just
checks to see if given file exists, if it does not issue a critical.
Make it executable
> sudo chmod a+x
/etc/sensu/plugins/check-file.rb
|
Test it (run it
directly)
> cd /etc/sensu/plugins/
> ./check-file.rb -f ~/test.txt
|
I get a Critical
Error since the file ~/test.txt does not exist.
If I create it
> touch ~/test.txt
|
And run it again
> ./check-file.rb -f ~/test.txt
|
This time it finds
the file and returns OK.
Create the check Definition
Create the check
definition, this file will determine how often this check is run amongst many
other things. (I am still trying to
figure this all out)
> sudo vi
/etc/sensu/conf.d/check_file.json
|
And place the
following in it. (change the location of
the file to your own location)
{
"checks": {
"check_file": {
"handlers": [
"default"
],
"command":
"/etc/sensu/plugins/check-file.rb -f /home/patman/test.txt",
"interval": 60,
"occurrences": 3,
"subscribers": [
"ALL"
]
}
}
}
|
Restart the
sensu-server on the client
> sudo service sensu-client restart
|
Restart the Sensu
Master with the following command.
> sudo
service sensu-server restart && sudo service sensu-api restart
|
And the Team_Two
check is live J
It's passing because
the file its looking for ~/test.txt exists.
Let me remove it and see it alerts.
> rm ~/test.txt
|
And there is the alert!
Click on TEAM_TWO. This is the team filter. Now you will only see the alerts that are
coming from the Team_Two Sensu master!
I think Uchiwa is perfect for
multiple teams. Each team can roll and
run its own Sensu Master. You could have
one master Uchiwa Dashboard that has all the alerts, which allows any team to
see all the current alerts. Then you can
simply filter by team to see who's issues are on the board.
Heck if one team really
wanted to roll their own uchiwa board, for whatever reason, the main uchiwa
dashboard could still pull in their data.
I don't think there is any issue with that.
Perfect, thanks Uchiwa team!
One last thing
Enable the services
to start automatically
Run the following
command
> sudo update-rc.d sensu-client defaults
|
Reboot to make sure
it all comes up like it should
> sudo reboot now
|
Open the Uchiwa
dashboard and confirm it all comes back up
In my case that's at
http://192.168.0.150:3000/
Looks good.
References
[1] Uchiwa's github page
Accessed 1/2015
[2] Sensu-dashboard's github page
Accessed 1/2015
Epic Goal: My goal is to just really figure out how to use Sensu.
No comments:
Post a Comment