Sensu Setting up a Second Client with SSL

Posted on Monday, October 27, 2014





This guide will go over setting up an additional Sensu Client, but one that uses SSL to speak back to the Sensu Master.  I currently have a set up with a Sensu Master, with its own client running on the same machine, and a second machine just running a Sensu Client.  I put up articles on how I set these up at http://www.whiteboardcoder.com/2014/10/sensu-getting-started.html

Currently none of the Sensu Clients I have running talk to the Master in a secure way over SSL.  I am going to set up a third Sensu client on a third machine and have it talk balk to the Sensu Master via SSL.  I am going to see if I can do this and keep the other two Sensu clients still talking without SSL.






Set up SSL certificates on the Sensu Master


Run the following commands to create some SSL certificates to use.


    > cd /tmp
    > wget http://sensuapp.org/docs/0.13/tools/ssl_certs.tar
    >  tar -xvf ssl_certs.tar
    >  cd ssl_certs && ./ssl_certs.sh generate



Copy the newly created certs to the correct Sensu directory


    >  sudo mkdir -p /etc/rabbitmq/ssl
    > sudo cp /tmp/ssl_certs/sensu_ca/cacert.pem /tmp/ssl_certs/server/cert.pem /tmp/ssl_certs/server/key.pem /etc/rabbitmq/ssl




Edit the /etc/rabbitmq/rabbitmq.config


    > sudo vi /etc/rabbitmq/rabbitmq.config


Here is my current rabbitmq.config file


[
    {rabbit, [
     ]} 
].



I updated it to.


[
    {rabbit, [
      {ssl_listeners, [5671]},
      {ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
                             {certfile,"/etc/rabbitmq/ssl/cert.pem"},
                             {keyfile,"/etc/rabbitmq/ssl/key.pem"},
                             {verify,verify_peer},
                             {fail_if_no_peer_cert,true}]}
  ]}
].


Looking at this file, it seems that you either have SSL communication from all the Sensu Clients or you don't.  I don't see where I can have a mixed environment, some SSL and some not.  If I am wrong please send me a note!
Sensu needs the SSL certificates. make a directory and copy them over.


    > sudo mkdir -p /etc/sensu/ssl
    >  sudo cp /tmp/ssl_certs/client/cert.pem /tmp/ssl_certs/client/key.pem /etc/sensu/ssl




edit the rabbitmq.json file.


    > sudo vi /etc/sensu/conf.d/rabbitmq.json


Add the ssl section


{
  "rabbitmq": {
    "ssl": {
      "cert_chain_file": "/etc/sensu/ssl/cert.pem",
      "private_key_file": "/etc/sensu/ssl/key.pem"
    }, 
    "host": "localhost",
    "port": 5672,
    "vhost": "/sensu",
    "user": "sensu",
    "password": "mypassword"
  }
}



Restart RabbitMQ


    > sudo service rabbitmq-server restart



Restart the Sensu Master with the following command.


    > sudo service sensu-server restart && sudo service sensu-api restart



Looking at my Uchiwa Dashboard for Sensu http://192.168.0.150:3000/#/clients 




I see that I have an error.


Looking at the sensu-api.log file


    > sudo tail -f /var/log/sensu/sensu-api.log




Looks like its having issues talking with rabbitmq


I think I found the culprit, rabbitmq.json has the wrong port number, (I changed it from 5671 to 5672)



edit the rabbitmq.json file.


    > sudo vi /etc/sensu/conf.d/rabbitmq.json


Change the port number to 5671


{
  "rabbitmq": {
    "ssl": {
      "cert_chain_file": "/etc/sensu/ssl/cert.pem",
      "private_key_file": "/etc/sensu/ssl/key.pem"
    },
    "host": "localhost",
    "port": 5671,
    "vhost": "/sensu",
    "user": "sensu",
    "password": "pass"
  }
}


Restart the Sensu Master with the following command.


    > sudo service sensu-server restart && sudo service sensu-api restart


That seemed to work.


Checking it out


It looks like its all working now!

If I open my Uchiwa dashboard It looks ok, I even did a few test to trigger alerts and they worked.




Doing a quick check of my rabbitMQ UI





Click on Connections







I can see that the Sensu Master server and its local Sensu Client are talking over ssl, but my second Sensu client at 192.168.0.151 is working just fine and its not talking over ssl.

In fact if I fix the Sensu Client on the Sensu Master server it does not have to talk over ssl either.




edit the rabbitmq.json file.


    > sudo vi /etc/sensu/conf.d/rabbitmq.json


Remove the ssl section and change the port to 5672


{
  "rabbitmq": {
    "host": "localhost",
    "port": 5672,
    "vhost": "/sensu",
    "user": "sensu",
    "password": "mypassword"
  }
}


And restart the Sensu client


    > sudo service sensu-client restart









Looking at the rabbitMQ UI I can see that both Sensu clients are talking without SSL.



Creating another Sensu Client with SSL


Now that I have confirmed that is working I am going to set up Sensu client on another server and have that one use SSL.

This third server is located at 192.168.0.151 in my system and it’s a fresh install of Ubuntu 14.04.


Before I get too far, sense I have a fresh install of Ubuntu 14.04, I need to update and upgrade apt-get.


    > sudo apt-get update
    > sudo apt-get upgrade






Install Sensu


Edit to /etc/apt/sources.list


    > sudo vi /etc/apt/sources.list



Append this to the end


deb http://repos.sensuapp.org/apt sensu main




Add an ssl key


    > wget -q http://repos.sensuapp.org/apt/pubkey.gpg
    > sudo apt-key add pubkey.gpg
    > sudo apt-get update


Install Sensu


    > sudo apt-get install sensu








Create the client.json file


Create the client.json file


    > sudo vi /etc/sensu/conf.d/client.json


And place the following in it


{
  "client": {
    "name": "client-2",
    "address": "192.168.0.152",
    "subscriptions": [ "client-2" ],
    "safe_mode":true
  }
}








Create the rabbitmq.json file


Create the rabbitmq.json file.


    > sudo vi /etc/sensu/conf.d/rabbitmq.json


Put the following in it, host contains the Master Sensu hostname or IP address


{
  "rabbitmq": {
    "ssl": {
      "cert_chain_file": "/etc/sensu/ssl/cert.pem",
      "private_key_file": "/etc/sensu/ssl/key.pem"
    }, 
    "host": "192.168.0.150",
    "port": 5671,
    "vhost": "/sensu",
    "user": "sensu",
    "password": "mypassword"
  }
}





Copy over the SSL certificates from the Sensu Master Server to this server.

From the Sensu Master run something like this


    > scp /etc/sensu/ssl/* 192.168.0.152:


Then from the new client, something like this



    > sudo mkdir /etc/sensu/ssl
    > sudo cp *.pem /etc/sensu/ssl/








Install Ruby



    > sudo apt-get install ruby ruby-dev build-essential



Install the sensu-plugin gem


    > sudo gem install sensu-plugin





Next I need to install the mixlib-cli gem


    > sudo gem install mixlib-cli







Add Checks


I have two current checks that I want to add this new client-2 as a subscriber.  I am going to update the file on the Sensu Master (adding the new subscriber) and then copy them over to the new sensu client. 


From the Master server edit the following check


    > sudo vi /etc/sensu/conf.d/check_file.json


Adding the client-2 as a new subscriber


{
    "checks": {
        "check_file": {
            "handlers": [
                "default"
            ], 
            "command": "/etc/sensu/plugins/check-file.rb -f /home/patman/test.txt",
            "interval": 60,
            "occurrences": 3,
            "subscribers": [
               "check-from-sensu-master",
               "client-1",
               "client-2"
            ]  
        }  
    }  
}



And edit the second check


    > sudo vi /etc/sensu/conf.d/check_second_file.json


Adding the client-2 as a new subscriber


{
    "checks": {
        "check_file_2": {
            "handlers": [
                "default"
            ], 
            "command": "/etc/sensu/plugins/check-file.rb -f /home/patman/test-2.txt",
            "interval": 60,
            "occurrences": 3,
            "subscribers": [
               "client-1" ,
               "client-2"
            ]       
        }  
    }  
}


Restart the Sensu Master Services


    > sudo service sensu-server restart && sudo service sensu-api restart





Now I need to copy these checks and the actual ruby code over from Master to the new client.


  > scp /etc/sensu/conf.d/check_*.json 192.168.0.152:
  > scp /etc/sensu/plugins/check-file.rb 192.168.0.152:


Then from the new Sensu Client move then to the correct place.


  > sudo cp check_*.json /etc/sensu/conf.d/
  > sudo cp check-file.rb /etc/sensu/plugins/



Start up the client Sensu service


  > sudo service sensu-client restart



Enable the services to start automatically

Run the following command


  > sudo update-rc.d sensu-client defaults




Looking at my Uchiwa Sensu Dashboard it seems like its working.  I am getting alerts because I am missing two files.

Let me quickly create them.


  > touch ~/test.txt
  > touch  ~/test-2.txt


That cleaned up my Board J

If I look at my RabbitMQ UI http://192.168.0.150:15672/#/connections




I can see the new server 192.168.0.152 is communicating with it and us use SSL.


That is it for this tutorial.





References
[1]        Sensu Documentation FAQ
                        http://sensuapp.org/docs/0.11/faq  
                Accessed 10/2014 





This post is a part of and epic, the Sensu Epic.


Epic Goal:   My goal is to figure out how to use Sensu to moni

No comments:

Post a Comment