This guide will go over setting up Sensu to talk to
HipChat. The idea here is to create
HipChat rooms to track sensu alerts. In
my current position we use HipChat almost exclusively for this purpose. The basic idea is this; leave HipChat open
and join the rooms where Sensu alerts are sent. Now when an alert goes out to a HipChat room
you get a ping and a notice pops up on your screen. If your system starts to get really
unhealthy you get a lot of pings.
This guide assumes you have Sensu installed and running and
a HipChat account. If you need help
setting up sensu check out these guides I wrote. http://www.whiteboardcoder.com/2014/10/sensu-getting-started.html or http://www.whiteboardcoder.com/2014/10/sensu-setting-up-client.html
If you are unfamiliar with HipChat I wrote an article on how
to set it up here http://www.whiteboardcoder.com/2014/08/hipchat-getting-stared.html
Installing hipchat gem
There is a nice hipchat gem, their github page is at https://github.com/hipchat/hipchat-rb
[1]
To install this gem run
> sudo gem install hipchat
|
To list your installed gems run this command
> gem list
|
Trying to send a message to a room
Before I get too far into it I want to do some testing and
confirm that I can send a message to one of my HipChat rooms just using a
simple ruby script.
To do this I first need an API key from HipChat.
Getting your HipChat API key
Sign into your HipChat account at https://www.hipchat.com/
Click on Group Admin
Click on API
Enter your password again (it has this additional level of
protection)
Select Notification (I only want this API key to be good for
sending messages not adding users or creating new rooms)
Give it a name, I named mine "Sensu Test" and click create.
Your token has been created!
In this case my token is '08eee07bb20a761df770998edead7c' The only reason I am showing the actual API
key is that by the time I post this this API key will be deleted and no longer
work ;)
A simple ruby Script
OK I have my API token now I want to write a simple ruby
script to write to a HipChat room. This
script in no way will use Sensu, but it will use the HipChat gem I installed.
> vi hipchat_test.ruby
|
Here is my code
#!/usr/bin/ruby
require 'hipchat'
api_token = '08eee07bb20a761df770998edead7c'
client =
HipChat::Client.new(api_token)
client['Sensu Test'].send('Test
Bot', 'Hello World')
|
Save it then make the file executable
> chmod u+x
hipchat_test.ruby
|
Run the program
> ./hipchat_test.ruby
|
Success the message showed up.
If you have the API token it looks like all you really need
is the name of a room. The 'Test Bot' is
not a user in my HipChat. So if I
change that part of the code to 'Test Bot 2'
and run it….
#!/usr/bin/ruby
require 'hipchat'
api_token = '08eee07bb20a761df770998edead7c'
client =
HipChat::Client.new(api_token)
client['Sensu Test'].send('Test Bot 2', 'Hello
World')
|
You can also set the color of the message.
#!/usr/bin/ruby
require 'hipchat'
api_token = '08eee07bb20a761df770998edead7c'
client =
HipChat::Client.new(api_token)
client['Sensu
Test'].send('Test Bot 2', 'Hello World', :color => 'red')
|
You can also do 'green', 'yellow', 'purple'
You can get someone's attention using the @ here is some
code that will do that. (you need to add
@username and :message_format => 'text'
#!/usr/bin/ruby
require 'hipchat'
api_token =
'08eee07bb20a761df770998edead7c'
client =
HipChat::Client.new(api_token)
client['Sensu
Test'].send('Test Bot 2',
'@PatrickBailey Hello World', :message_format => 'text')
|
Sensu Handlers
Now that I have figured out how to use the HipChat gem I
need to figure out Sensu Handlers.
The Sensu doc page for this is located at http://sensuapp.org/docs/0.16/handlers
[2]
Looking over this page there are several different types of
handlers, pipe, TCP, UDP, AMQP, and
Sets.
For my near term purposes I think I only really need pipe
and sets.
Pipe handlers execute a script and pass the event in via
STDIN.
Sets are used for grouping handlers. It’s a way to send the same event to several
handlers at the same time. For example
if you want an event to two different Pipe handlers, one which sends a message
to HipChat and one that sends an email, you can use a set handler. I am not going to use a Set handler in this
document, but I thought it worth mentioning.
I found this github repo https://github.com/sensu/sensu-community-plugins/tree/master/handlers/notification
[3] which contains several Sensu Handlers you can just copy and use. The HipChat handler can be found at https://github.com/sensu/sensu-community-plugins/blob/master/handlers/notification/hipchat.rb
[4]
Here is the code (as it was on 11/16/2014)
#!/usr/bin/env ruby
require 'rubygems' if RUBY_VERSION < '1.9.0'
require 'sensu-handler'
require 'hipchat'
require 'timeout'
class HipChatNotif < Sensu::Handler
option
:json_config,
:description => 'Config Name',
:short => '-j JsonConfig',
:long => '--json_config JsonConfig',
:required => false
def
event_name
@event['client']['name'] + '/' + @event['check']['name']
end
def handle
json_config = config[:json_config] || 'hipchat'
server_url = settings[json_config]["server_url"] ||
'https://api.hipchat.com'
apiversion = settings[json_config]["apiversion"] || 'v1'
proxy_url = settings[json_config]["proxy_url"]
hipchatmsg =
HipChat::Client.new(settings[json_config]["apikey"], :api_version
=> apiversion, :http_proxy => proxy_url, :server_url => server_url)
room =
settings[json_config]["room"]
from =
settings[json_config]["from"] || 'Sensu'
message
= @event['check']['notification'] || @event['check']['output']
# If the
playbook attribute exists and is a URL, "[<a
href='url'>playbook</a>]" will be output.
# To
control the link name, set the playbook value to the HTML output you would
like.
if
@event['check']['playbook']
begin
uri
= URI.parse(@event['check']['playbook'])
if
%w( http https ).include?(uri.scheme)
message << " [<a
href='#{@event['check']['playbook']}'>Playbook</a>]"
else
message << "
Playbook:
#{@event['check']['playbook']}"
end
rescue
message << "
Playbook:
#{@event['check']['playbook']}"
end
end
begin
timeout(3) do
if
@event['action'].eql?("resolve")
hipchatmsg[room].send(from, "RESOLVED - [#{event_name}] -
#{message}.", :color => 'green')
else
hipchatmsg[room].send(from, "ALERT - [#{event_name}] -
#{message}.", :color => @event['check']['status'] == 1 ? 'yellow' :
'red', :notify => true)
end
end
rescue
Timeout::Error
puts
"hipchat -- timed out while attempting to message #{room}"
end
end
end
|
Create the HipChat handler
(first create the notification folder
> sudo mkdir -p
/etc/sensu/handlers/notifications
> sudo vi
/etc/sensu/handlers/notifications/hipchat.rb
|
In my case I want the room to be passed in via the command ,
also I am just going to be using HipChat V1 API (I seem to be having issues
using the V2). I am also not currently
using the playbook feature so I will remove that.
#!/usr/bin/env ruby
require 'rubygems' if
RUBY_VERSION < '1.9.0'
require
'sensu-handler'
require 'hipchat'
require 'timeout'
class HipChatNotif
< Sensu::Handler
option :room,
:description => 'HipChat Room
Name to send messages to',
:short => '-r ROOM',
:long => '--room ROOM',
:default => 'Sensu Test'
def event_name
@event['client']['name'] + '/' +
@event['check']['name']
end
def handle
hipchatmsg =
HipChat::Client.new(settings["hipchat"]["apikey"])
room = config[:room]
from =
settings["hipchat"]["from"]
message = @event['check']['notification']
|| @event['check']['output']
begin
timeout(3) do
if
@event['action'].eql?("resolve")
hipchatmsg[room].send(from,
"RESOLVED -
[#{event_name}] - #{message}.", :color => 'green')
else
hipchatmsg[room].send(from,
"ALERT - [#{event_name}] - #{message}.",
:color =>
@event['check']['status'] == 1 ? 'yellow' : 'red', :notify => true)
end
end
rescue Timeout::Error
puts "hipchat -- timed out while
attempting to message #{room}"
end
end
end
|
This is my code.
Save it then make the file executable
> sudo chmod a+x /etc/sensu/handlers/notifications/hipchat.rb
|
It still requires a handler and a "hipchat" json
setting.
Create the hipchat handler
> sudo mkdir -p /etc/sensu/conf.d/handlers
> sudo vi /etc/sensu/conf.d/handlers/hipchat.json
|
{
"handlers": {
"hipchat": {
"command": "/etc/sensu/handlers/notifications/hipchat.rb
-r 'Sensu Test'",
"type": "pipe",
"severities": [
"ok",
"critical",
"unknown"
]
}
},
"hipchat": {
"apikey": "08eee07bb20a761df770998edead7c",
"from": "Sensu Master"
}
}
|
Then I have to add this handler to a check. In my case I had a check called
check_file.json I had created before, so I will edit that.
> sudo vi
/etc/sensu/conf.d/check_file.json
|
{
"checks": {
"check_file": {
"handlers": [
"default", "hipchat"
],
"command":
"/etc/sensu/plugins/check-file.rb -f /home/patman/test.txt",
"interval": 60,
"occurrences": 3,
"subscribers": [
"check-from-sensu-master",
"client-1",
"client-2",
"aws-client"
]
}
}
}
|
All I did was add the "hipchat" handler in the
handlers section.
Restart the Sensu
Master with the following command, and its client
> sudo
service sensu-server restart && sudo service sensu-api restart
&& sudo service sensu-client restart
|
To trigger my check_file check I just need to remove a file
from my home directory.
> rm ~/test.txt
|
After 3 occurrences the handler triggers.
Bringing back the file resolves the issue.
> touch ~/test.txt
|
The HipChat handler successfully posts to my HipChat
room! Both its alert and its own
Resolved message.
Playing with the handler for a bit and looking at the logs I
saw this message.
{"timestamp":"2014-11-16T19:48:21.566544-0700","level":"info","message":"handler
output","handler":{"command":"/etc/sensu/handlers/notifications/hipchat.rb
-r 'Sensu
Test'","type":"pipe","severities":["ok","critical","unknown"],"name":"hipchat"},"output":"only handling every 30
occurrences: sensu-master/check_file\n"}
|
Only handling every 30 occurences.
I found this post https://github.com/sensu/sensu/issues/613
[5] which mentions that if you are using the sensu-handler this is
the expected behavior. The default is to
only trigger once every 30 minutes (after the initial trigger…. The initial
trigger delay does not count).
I did a little test and let it run for a while
It will only run the handler code once in a 30 minute
interval. In this case the alert was
triggered at 6:46 PM, but it had a setting of 3 occurrences in a 60 s interval. As a result the first message to hipchat is
delayed by 3 minutes and occurs at 6:49 PM.
But the next one occurs 30 minutes after the alert at 7:16PM, then every
30 minutes after until its resolved.
If I change my interval to 30 seconds, trigger my alert and
look at my logs.
{"timestamp":"2014-11-16T20:28:05.125612-0700","level":"info","message":"handler
output","handler":{"command":"/etc/sensu/handlers/notifications/hipchat.rb
-r 'Sensu Test'","type":"pipe","severities":["ok","critical","unknown"],"name":"hipchat"},"output":"only handling every 60
occurrences: sensu-master/check_file\n"}
|
So it looks like it’s a 30 minute interval.
That is all well and good, but what if I want to only issue a
message every hour, or only once? How do
I override the default?
I think I found it.
You need to set the refresh variable.
Edit the check file.
> sudo vi
/etc/sensu/conf.d/check_file.json
|
And add a refresh variable.
{
"checks": {
"check_file": {
"handlers": [
"default", "hipchat"
],
"command": "/etc/sensu/plugins/check-file.rb -f
/home/patman/test.txt",
"interval": 60,
"occurrences": 3,
"refresh": 600,
"subscribers": [
"check-from-sensu-master",
"client-1",
"client-2",
"aws-client"
]
}
}
}
|
The normal default is 1800 second (30 minutes). I set mine to 600 seconds (10 minutes) as a
test. And it worked!
I guess if you want to only issue a handler once you could
up the refresh rate to a very high value like 1209600 (14 days)
I changed my refresh rate to 1209600 just to test it. And looking at my logs, it seems to be
working.
{"timestamp":"2014-11-16T21:23:37.112900-0700","level":"info","message":"handler
output","handler":{"command":"/etc/sensu/handlers/notifications/hipchat.rb
-r 'Sensu
Test'","type":"pipe","severities":["ok","critical","unknown"],"name":"hipchat"},"output":"only
handling every 20160
occurrences: sensu-master/check_file\n"}
|
With a one minute interval 20160 comes out to 14 days.
OK that is enough for this long winded tutorial. Hope it helps someone out there get Sensu
talking to HipChat and maybe help figure out Sensu Handlers a little better.
References
[1] HipChat Wrapper github page
Accessed 11/2014
[2] Sensu Handlers doc page
Accessed 11/2014
[3] sensu-community-plugins
Accessed 11/2014
[4] sensu-community-plugins hipchat.rb
Accessed 11/2014
[5] bugs in multiple handlers #613
Accessed 11/2014
Epic Goal: My goal is to figure out how to use Sensu to moni
Epic Goal: My goal is to figure out how to use Sensu to moni
Thank u.. Its a nice post..
ReplyDelete