Sensu Getting Started

Posted on Tuesday, October 21, 2014




This guide will get you through the very first steps in setting up Sensu Code (I want to keep it simple)

I am going to install a Sensu Master and a client on the same machine, a brand new Ubuntu 14.04 install.   For this write up I will not be using SSL keys for RabbitMQ to communicate with (This is fine for testing and may be an OK thing to do in an internal network)





What is Sensu



Sensu is a monitoring tool written in Ruby that uses RabbitMQ as a message broker and Redis for storing data. It is well suited for monitoring cloud environments.



Setup


I am going to start with servers that have a fresh install of Ubuntu 14.04

I have a fresh install of Ubuntu 14.04 on my test servers, I am going to start from there.





Sensu Server


I am first going to set up the Sensu Server, I will set up the client later on.

Before I get too far, sense I have a fresh install of Ubuntu 14.04, I need to update and upgrade apt-get.


    > sudo apt-get update
    > sudo apt-get upgrade





Installing RabbitMQ


Sensu uses RabbitMQ.  I am following the guide at http://www.rabbitmq.com/install-debian.html [2]


Edit to /etc/apt/sources.list


    > sudo vi /etc/apt/sources.list



Append this to the end


deb http://www.rabbitmq.com/debian/ testing main








Add an ssl key


    > wget http://www.rabbitmq.com/rabbitmq-signing-key-public.asc
    > sudo apt-key add rabbitmq-signing-key-public.asc
    > sudo apt-get update



Install rabitmq-server


    > sudo apt-get install rabbitmq-server




The RabbitMQ server should have started, but if not you can start it with this command.


    > sudo service rabbitmq-server start




Create and and edit the /etc/rabbitmq/rabbitmq.config



    > sudo vi /etc/rabbitmq/rabbitmq.config


Put the following in it


[
    {rabbit, [
       {tcp_listeners,[{"127.0.0.1",5672}]}
     ]}
].




Create a virtual host and user for Sensu.  (Change mypassword to whatever password you want)


    > sudo rabbitmqctl add_vhost /sensu
    > sudo rabbitmqctl add_user sensu mypassword
    > sudo rabbitmqctl set_permissions -p /sensu sensu ".*" ".*" ".*"






Restart rabbitMQ


    > sudo service rabbitmq-server restart




Enable the web UI interface (Optional… I don't think you need to do this, but I am going to)


    > sudo rabbitmq-plugins enable rabbitmq_management




Restart rabbitMQ


    > sudo service rabbitmq-server restart


Make sure the web UI is running by running this command



    > netstat -teapu | grep 15672





Open the web UI in my case 192.168.0.150:15672


Login
Username:  guest
Password: guest






I hate leaving a default user/password around so I am going to fix this.





Add admin to RabbitMQ UI





Click Admin






Enter a username and password.  Click on the Admin button, this will add administrator to Tags.  Then click Add user.





Click on the new name




Click Set permission






Now you are set up.


Log out and back in with your new user



To remove the quest user do the following






Click Admin







Click guest







Click Delete this user.






Click Delete





Install Redis




    > sudo apt-get install redis-server








Install Sensu


Edit to /etc/apt/sources.list


    > sudo vi /etc/apt/sources.list



Append this to the end


deb http://repos.sensuapp.org/apt sensu main





Add an ssl key


    > wget -q http://repos.sensuapp.org/apt/pubkey.gpg
    > sudo apt-key add pubkey.gpg
    > sudo apt-get update


Install Sensu


    > sudo apt-get install sensu

Configure the Sensu Master


Create the rabbitmq.json file.


    > sudo vi /etc/sensu/conf.d/rabbitmq.json


Put the following in it


{
  "rabbitmq": {
    "host": "localhost",
    "port": 5672,
    "vhost": "/sensu",
    "user": "sensu",
    "password": "mypassword"
  }
}




Change and edit the redis.json file


    > sudo vi /etc/sensu/conf.d/redis.json


And place the following in it.


{
  "redis": {
    "host": "localhost",
    "port": 6379
  }
}






Create the api.json file


    > sudo vi /etc/sensu/conf.d/api.json


And place the following in it



{
  "api": {
    "host": "localhost",
    "port": 4567
  }
}







Install a Sensu Dashboard


Sensu recommends using the Uchiwa dashboard http://sensuapp.org/docs/latest/install_a_dashboard  [3]

Install Uchiwa


    > sudo apt-get install uchiwa





edit uchiwa.json


    > sudo vi /etc/sensu/uchiwa.json


The default file is fine for my purposes.  Here is what is contained in it.


{
  "sensu": [
    {
      "name": "Sensu",
      "host": "127.0.0.1",
      "ssl": false,
      "port": 4567,
      "user": "",
      "pass": "",
      "path": "",
      "timeout": 5000
    }
  ],
  "uchiwa": {
    "user": "",
    "pass": "",
    "port": 3000,
    "stats": 10,
    "refresh": 10000
  }



Restart the Sensu Master with the following command.


    > sudo service sensu-server restart && sudo service sensu-api restart



Start up uchiwa


    > sudo /etc/init.d/uchiwa restart



Make sure its running.


    > netstat -teapu | grep 3000









Open up the URL  in my case 192.168.0.150:3000/




Its working, although there is not much there yet.
Enable the services to start automatically



Run the following commands


    > sudo update-rc.d sensu-server defaults
    > sudo update-rc.d sensu-api defaults
    > sudo update-rc.d uchiwa defaults


Reboot the server


    > sudo reboot now


Open the URL again to see if it's running






Setting up a Client


I have a Sensu Master server, but no client yet to actually monitor anything.

For this write-up I am going run a Sensu client on the same box as the Sensu Master. 





If you roll over the clients section on Uchiwa you will see there are currently 0 clients


Create the client.json file


    > sudo vi /etc/sensu/conf.d/client.json


And place the following in it


{
  "client": {
    "name": "server",
    "address": "192.168.0.150",
    "subscriptions": [ "ALL" ]
  }
}



Start the client


   > sudo service sensu-client start






In a few seconds it should shows up
Enable the services to start automatically

Run the following command


  > sudo update-rc.d sensu-client defaults




Reboot to make sure it all comes up like it shoould


    > sudo reboot now


Open the Uchiwa dashboard and confirm it all comes back up

In my case that's at http://192.168.0.150:3000/



Success!

Finally Add a Check


If you scroll over the check in the Uchiwa Dashboard




You will see a big fat 0 Check.  I have no checks!





Install Ruby



    > sudo apt-get install ruby ruby-dev build-essential



Install the sensu-plugin gem


    > sudo gem install sensu-plugin






Create a check



Instead I am going to use the recommendation at
https://github.com/sensu/sensu-plugin [4] which says to subclass Sensu::Plugin::Check::CLI

Also I will use the mixlib-cli tool https://github.com/opscode/mixlib-cli [5] to give

Install the mixlib-cli gem


    > sudo gem install mixlib-cli





Create the check-file.rb file


    > sudo vi /etc/sensu/plugins/check-file.rb


And place the following in it


#!/usr/bin/env ruby
require 'sensu-plugin/check/cli'

class CheckFile < Sensu::Plugin::Check::CLI

 
option :file,
        
:description => "Path to file",
        
:short => '-f FILE',
        
:long => '--file FILE',
        
:required => true

  def
initilize
   
super
  end

  def
run
   
if(File.exists?(config[:file]))
      ok(
"The file '" + config[:file] + "' Exists!  :)")
   
else
     
critical("The file '" + config[:file] + "' dose not Exists! :(")
   
end
  end
end


Make it executable


    > sudo chmod a+x /etc/sensu/plugins/check-file.rb



Test it (run it directly)


   >  cd /etc/sensu/plugins/
   > ./check-file.rb -f ~/test.txt





I get a Critical Error since the file ~/test.txt does not exist.  If I create it


    > touch ~/test.txt


And run it again


    > ./check-file.rb -f ~/test.txt




This time it finds the file and returns OK.






Create the check Definition


Create the check definition, this file will determine how often this check is run amongst many other things.  (I am still trying to figure this all out)


    > sudo vi /etc/sensu/conf.d/check_file.json


And place the following in it.  (change the location of the file to your own location)


{
    "checks": {
        "check_file": {
            "handlers": [
                "default"
            ],
            "command": "/etc/sensu/plugins/check-file.rb -f /home/patman/test.txt",
            "interval": 60,
            "occurrences": 3,
            "subscribers": [
                "ALL"
            ]
        }
    }
}







Restart the sensu-server on the client


    > sudo service sensu-client restart






And…. Nothing…

The Sensu Master Server needs to be restarted (and in the case of a remote client server the Sensu Master would need to have the check code as well)






Restart the Sensu Master with the following command.


    > sudo service sensu-server restart && sudo service sensu-api restart









And now we have a check !



At the moment everything looks happy.

If I remove the test.file


    > rm ~/test.txt







Now I have 1 critical event





Click the Events icon





Click anywhere along the line where the event is displayed





I can now see all the details of the event.




If I let it got for X minutes if I look at Uchiwa I can see that I have X Critical occurrences.  Since my check is set to run once every 60 seconds.







Recreate the test.txt file



    > touch ~/test.txt






And the event clears out within 60 seconds.  (the next check marks it as OK).




I think I am going to like Sensu, I really think it is a great tool you almost need in the arsenal in a cloud computer era.




References
[1]        How To Configure Sensu Monitoring, RabbitMQ, and Redis on Ubuntu 14.04
                Accessed 10/2014
[2]        Installing on Debian / Ubuntu
                        http://www.rabbitmq.com/install-debian.html
                Accessed 10/2014
[3]        Install a Sensu Dashboard
                        http://sensuapp.org/docs/latest/install_a_dashboard
                Accessed 10/2014
[4]        github sensu-plugin page
                        https://github.com/sensu/sensu-plugin

                Accessed 10/2014
[5]        github mixlib-cli library
                        https://github.com/opscode/mixlib-cli

                Accessed 10/2014





This post is a part of and epic, the Sensu Epic.

Epic Goal:   My goal is to figure out how to use Sensu to monitor systems.

10 comments:

  1. Hmmm, strange.

    I cant login to rabbitmq login neither using sensu password nor guest:guest.

    ReplyDelete
    Replies
    1. Oh, these are changes in new version of RabbitMQ.

      Delete
  2. > I do not like the example check I see at...

    Why you don't like it?

    ReplyDelete
    Replies
    1. Haha I should have written that down back when I first wrote this article. Looking at it again I think it is a fine example but I prefer a simpler example for the first check. Like just check if a file exists. The example on the page in question was looking to see if a process was running or not. And a specific process at that (Apache). So now they would have to install Apache or figure out how to fake a process... too much work for a simple example

      Delete
    2. Ok, fine.) Did you try Sensu in prod? Do you have any real experience of doing that?

      Delete
    3. Yes, I have been running Sensu in Production for several years now. We have most of our monitoring set up around it. I have it post to several slack chat rooms.

      Delete
  3. I have been having some issues with starting up the sensu server, I followed your steps exactly, but I am having some trouble with the setting up ssl keys set up, I receive a message:
    W: GPG error: http://www.rabbitmq.com testing InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY

    ReplyDelete
  4. Ultimate article , Good explanation ..

    ReplyDelete
  5. How to configure this Sensu in Windows server to monitor the virtualbox clients?

    ReplyDelete