This guide will get you through signing up for a test
account and setting up a rotating schedule with escalation policies
What is pagerduty ?
Pagerduty
is a great tool for alerting your team of incidents in your system. Its located at http://www.pagerduty.com/ [1]. There services can tie into several
monitoring services, and when triggered can contact the proper on duty person
on your team.
Signing up for a free trial
Currently, 8/2014, pagerduty has a free 30 day trial.
Click on Sign Up Now
Enter your relevant information and click continue.
The sign in screen should open. Enter your password and click Sign in.
And now you are logged in.
If you look on the upper right hand you will see this
I have 30 days left of my free trial.
Surprise I have not had to enter credit card number yet. Way to go pagerduty!
Now let's do something
Let's follow the pager duty suggested steps
1. Set
your contact Info
2. Add
Users
3. Create
an on-call schedule
4. Add
services
1. Set your contact Info
Click ON "Set Your Contact Info"
Hey its smart enough to grab my gravatar photo, which reminds me it needs to be updated to my
newer one.
This is much better.
Not so serious and blank faced, almost angry, look.
To fix the time zone, click on Edit Profile in the upper
right
Select your time zone.
Click Save Changes
Click Add Phone Number
Enter in your phone number set it to mobile (if it's your
cell) and click save
Do the same thing for SMS
If you have an iPhone, iPad, or android device you can
download pagerduty app. The iOS app is
at https://itunes.apple.com/us/app/pagerduty/id594039512?mt=8
[2]
Pagerduty iOS app
Enter the email address you use for pagerduty and click search
Enter your password and click Sign In.
Click OK to allow pagerduty to push notifications
Head back to the pagerduty web page and refresh
Your iOS device is registered
As a test click
"Send Test Notification"
You should get this pop up
Click View
2. Add Users
It's time to add another user
It looks like pagerduty is smart enough to figure out we
passed phase one. Click on Add Users to
start phase two. Remember each user will
cost you $19/mo when you finally go live so think about that before you add
everyone and their brother.
Enter your team members info, name and email. Set them as a User. Then click Add.
Nice, it keeps this prompt open and allows you to keep
adding team members.
J
When you are done click Send Invites
The Users interface opens and you can see the users you have
added and if they have accepted the invite.
Your new users should receive an email with an accept
link.
When your user accepts the invite they are presented with
this screen. They need to enter in a
password and click Sign Up.
The new user needs to click on their email address in the upper
right and select My Profile.
This page will allow them to add their contact methods.
After the user has accepted the invite the resend invite
button should go away.
3. Create an On-Call Schedule
Click Create an On-Call Schedule.
Click Create your first on-call schedule
Give the On-Call Schedule a name and set the desired Time
Zone to use.
Click Add User
A user pull down will appear, Select a user and click Add User
The user has been added.
Add another user and click Done.
Now we need to set up the rotation between who is on call.
For this test I am going to set up daily rotation that runs
from 8:00 AM to 5:00 PM Monday through Saturday (no Sunday).
From the pull down select Daily.
The schedule now appears
Click the checkbox here to restrict the shift times.
This Restrictions dialog will open.
Select "Restrict on-call duty to specific times of the
day"
And set the from and to times
Click Apply
The calendar updates with the restrictions.
But that is not the schedule I want its 8:00 AM to 5:00 PM
every day of the week….
Click edit to change the schedule
Select the second restrict radio button and set to Monday
from 0:800 to 17:00, then click the plus button.
This will add another range.
Set this range for Tuesday from 8:00 AM to Tuesday 5:00 PM/
Keep doing this until you have the 6 days a week set up.
Click Apply
That gives me the schedule I want.
When you are all done click Save Changes
From here I can click on Calendar view to see a nice
Calendar of the on duty schedule.
4. Add Service
Now we need to add a service. Click Add Services.
Click + Add New service
Give the service a name and, for now, select default as the
escalation type. I will go over this
later.
Now you need to select how you are going to integrate with
this tool.
Look at the pull down menu, there are a lot of options.
For now keep it simple. Select Integrate via email.
An email address will be shown (you can change its name here
if you wish)
Click Add service.
The service screen will now display this service (which is
not quite done).
Give it a test run
Send an email out to the integration email of the service.
My iPad was notified via the pagerduty iOS software.
On the pager duty dashboard I can see its been triggered but
not acknowledge.
But now phone call?
What is wrong?
I will acknowledge this alarm and start, figure out how to
fix the phone call and start over.
checkbox the incident and click Acknowledge
On the iOS device I can see that it was acknowledged and by
whom.
From the web site I can see that it was acknowledged. Click Resolve to resolve the issue.
Open up My profile.
Here is my problem. I
need to add a Notification rule.
Currently its only set up to email and push to my iPad.
Click Add Notification Rule
Select your phone and click Save. Now it will call this number 0 minutes after
an incident occurs.
Send another email out to trigger an alarm.
Here is the audio I got back.
It read the email subject line, but not the email contents.
Keep this in mind when creating emails to send.
As a side note pagerduty calls always come from
1-415-349-5750
Looking at the pagerduty site I can see the incident and
that it was acknowledged and who acknowledged it.
Click on Details
You can see all the details sent, including the body of the
e-mail message sent.
Click the Incident number to open its detailed page
Perfect! This is the page
I was interested in.
This has
·
Time the Incident was opened
·
Who is assigned to it
·
What time the incident was acknowledged
·
What time the incident was resolved
·
Any notes added to the Incident (scroll down to
the bottom)
What's this?
… In 2 minutes I will find out.
It called me again. I acknowledged it and did not resolve it, and as a result it called me again. This is the default behavior.
In my case I do not want it to call back after an
acknowledge and I do not want it to ever auto resolve. A solution could take ½ a day and I want to
record that fact.
Setting the default auto resolve and acknowledge call back
Click On Services
From the controls pull down on the service select edit.
Uncheck
Incident
Ack Timeout
And
Incident Auto-Resolution
Save Changes
Run another test
Perfect I acknowledged the call and it's not going to call me back at all.
The missing Escalation Policy
What I think I am
missing at this point is the escalation policy.
Open the Escalation
Policies
Add a new Escalation
Policy.
Here is a new blank
escalation policy
Give the escalation
policy a name.
Click in the
"Alert the following user section and select a schedule. I selected the Emergency! Schedule.
You are free to add
more people to the list or schedules.
You could have it always call you no matter who is on schedule.
I changed the min to
wait until next rule kicks in to 20 minutes.
If there is no acknowledgement from the first alert the next rule will
be applied.
For the second alert
rule I am having it call me.
By checking the box
and entering in 9 I am setting it to repeat this sequence up to 9 times if
there is no acknowledgement.
Click Save.
Go back to services
Edit the service you want to add the escalation policy to.
Select the new
Escalation Policy
Scroll to the bottom
and Save Changes.
Running another test
I sent an email
again. It correctly called the first
person on the duty.
They acknowledged it
and it did not continue onto the next person.
Then I tried to trigger another incident, but I had not resolved the
last one. Instead of triggering another
event the email was absorbed and documented within the prior event.
After this I "resolved"
the issue and sent a new email. This
time I let the first person ignore it and let it go onto the next person. This worked like it should 20 minutes after
the first person ignored the call, the second one received a call and
acknowledged it.
That should do it! There are the basics to get going with pagerduty.com
This post is a part of and epic, the pagerduty and twilio epic.
Epic Goal: Set up a phone number, via twillio, that when called will set off a pagerduty event.
That should do it! There are the basics to get going with pagerduty.com
References
[1] pagerduty web site
Accessed 08/2014
Epic Goal: Set up a phone number, via twillio, that when called will set off a pagerduty event.
No comments:
Post a Comment