Tuesday, December 26, 2017

AWS Lambda for forwarding a WebHook to Slack

Recently I had to deal with a monitoring service that provided a WebHook to send responses out.  However it was not Slack aware and did not know how to format the data sent in the WebHook so that it would show up in Slack Correctly.

So I thought…
One way to fix this is to have some service sit in-between and capture the outgoing WebHook filter the data and send it on to Slack in a format it understands.




Something like this (I had an Outgoing WebHook in Distill.io).


This in theory would work.  Set up a Server, write a bit of code and run it on the server to accomplish this.  

1.      But that does cause some other problems to crop up…
First I need to rent a server 24x7…. A cloud server is cheap enough… but even at a few $$/ a month it’s a bit overkill for just a few messages every now and then.
2.      I need to write, maintain, debug and deploy the code.
3.      I need to set up some DNS entries for my server to route the traffic
4.      I need to set up SSL certs to encrypt my traffic
5.      If I want a HA system I need to run a couple of servers behind a load balancer in case one should go down.
6.      I need to monitor my system somehow so that when it goes down I know it!

Etc. etc. etc.

All those things may be worth it if you need to relay lots of messages.  But what if you only need to send 100, or 1,000 a month?  What if you only need to send at most 60 per hour.  Those are some pretty low numbers these days.

I think AWS lambda may be an ideal solution for this problem.




What is AWS Lambda?


AWS lambda is a tool that allows you to put a bit of code out there and let AWS handle all the HA, scaling, Monitoring… etc.



But it comes at a cost https://aws.amazon.com/lambda/pricing/ [2]




For my purposes my costs should come up to $0.00 a month… So free J





Now onto setting one up


Log into the AWS console and open up the Lambda tool.




Click on create function!





Choose Author from Scratch





Give it a name.  Chose a runtime you want to use.  I am using Node.js 6.10 for this example.

Then you need to give it permissions (a role).  In this simple case I do not need to give it access to anything special like an S3 bucket or anything so I just chose an existing role lambda_basic_execution.  See http://docs.aws.amazon.com/lambda/latest/dg/intro-permission-model.html [3] for more details.





Click Create Function on the lower right.





Congratulations you have a lambda function… Now what?




Setting up a restful endpoint


I can set up a restful endpoint by setting an API gateway.  (Using this tool will have some costs https://aws.amazon.com/api-gateway/pricing/ [6] But, should be minimal.




Click on API Gateway.






Now click the Configuration required.




I am going to choose the defaults here and for the security just select open for now.




Click Add





Click Save






Now I have a URL

In my case it is.


(don’t worry after I finishing writing this up I will be deleting this lambda function entirely)

Now let me see if I can use it!
I am just going to use a simple curl to see if I can get something to show up in the logs.




   > curl https://8zj3nwhe2m.execute-api.us-east-1.amazonaws.com/prod/webhookRelay




OK a server error…

Looking around I found this page that goes over how to fix this issue http://www.awslessons.com/2017/lambda-api-gateway-internal-server-error/ [4]


The API gateway is expecting a json object with the three following… statusCode, headers, and body.



So let me update my code to the following and test it again.


exports.handler = (event, context, callback) => {
   
    var response = {
        'statusCode': 200,
        'headers': { 'Content-Type': 'application/json' },
        'body':  "Testing it out"
    }
    console.log("LOG:: " + JSON.stringify(response))
   
    callback(null, response);
};


Here is my updated code…



   > curl https://8zj3nwhe2m.execute-api.us-east-1.amazonaws.com/prod/webhookRelay






Wahoo past that hurtle. 

Let me do another quick test to get some JSON response back.



exports.handler = (event, context, callback) => {
   
    var details = {
        "one": "44",
        "two" : {
            "details" :"Some details",
            "another" :"More details"
        }
    }
   
    var response = {
        'statusCode': 200,
        'headers': { 'Content-Type': 'application/json' },
        'body': JSON.stringify(details)
    }
    console.log("LOG:: " + JSON.stringify(response))
   
    callback(null, response);
};




Now let me curl it again put this time pass it to a tool called jq to format it.


   > curl -s https://8zj3nwhe2m.execute-api.us-east-1.amazonaws.com/prod/webhookRelay | jq .




Wahoo worked!





Logging


Let’s figure out how to do some logging of incoming data.  I really need that if I am going to debug how the webhook from Distil.io sends data.

Let me update my code to log what I receive. 



exports.handler = (event, context, callback) => {
   
   
    console.log("MYLOG" + JSON.stringify(event))
   
    var details = {
        "one": "44",
        "two" : {
            "details" :"Some details",
            "another" :"More details"
        }
    }
   
    var response = {
        'statusCode': 200,
        'headers': { 'Content-Type': 'application/json' },
        'body': JSON.stringify(details)
    }
    console.log("LOG:: " + JSON.stringify(response))
   
    callback(null, response);
};



Save it and run a curl test


   > curl -s -X POST https://8zj3nwhe2m.execute-api.us-east-1.amazonaws.com/prod/webhookRelay | jq .




I got back the results I wanted.  Now let me go find the local logs in Lambda.






Click On monitoring





Click on Jump to Logs






Click on the time then click on relative





Choose 3 hours.  That will give us the last 3 hours of logs.





Search for the text string “MYLOG”





You may have to hit refresh





There is my text and then there is the JSON file sent to my code !




OK now that I have that working I can work on my webhook to send data to this Lambda function.




Distill.io


I am just using Distill.io in this example

I already have a page I am monitoring in Distilio as a test.  I am just going to open it up and add another action to it.





Click on Add Action and select “Call Webhook”.


From a



Click on Options




Here is what I see.  It looks like this particular outgoing webhook allows you to tweak it a bit to send the data you want to send.




I am going to leave it as is, but put the URL for my lambda function.

Now to save it and test it !

(In distill.io’s case I need to edit the web page it is monitoring to get it to send a message to lambda)








It worked.



It put the info sent into the “body” section.



OK now that I can see what I am being sent let me get the lambda code to send a simple message to a slack room.





Simple Slack message


OK I just want it to send a simple slack message no matter what data come in.  Here is my simple code that accomplishes that.



var https = require('https');

exports.handler = (event, context, callback) => {
   
    console.log("MYLOG" + JSON.stringify(event))
   
    var post_data = JSON.stringify({
      "text": "TESTING IT OUT 111"
    });
 
    // An object of options to indicate where to post to
    var post_options = {
      host: 'hooks.slack.com',
      port: '443',
      path: '/services/YOURWEBHOOKHERE',
      method: 'POST',
      headers: {
          'Content-Type': 'application/json',
          'Content-Length': Buffer.byteLength(post_data)
      }
    };
   
    // Set up the request
    var post_req = https.request(post_options, function(res) {
      res.setEncoding('utf8');
      res.on('data', function (chunk) {
          console.log('Response: ' + chunk);
      });
    });
   
    // post the data
    post_req.write(post_data);
    post_req.end();
   
   
    var details = {
        "one": "44",
        "two" : {
            "details" :"Some details",
            "another" :"More details"
        }
    }
   
    var response = {
        'statusCode': 200,
        'headers': { 'Content-Type': 'application/json' },
        'body': JSON.stringify(details)
    }
    console.log("LOG:: " + JSON.stringify(response))
   
    callback(null, response);
};

Of course if you wanted it to hit your slack WebHook you would need to update the path to the correct path.


Now a simple curl to test it.



   > curl -s -X POST https://8zj3nwhe2m.execute-api.us-east-1.amazonaws.com/prod/webhookRelay | jq .





Wahoo it worked J

Now let me update my page and see if the Distilio webhook will do the same thing..




OK that worked!


Now to clean it up and make it more complex.


Complex Message


I am using some old notes of mine at http://www.whiteboardcoder.com/2015/03/posting-message-to-slack-via-webhooks.html [5] which show how to use the slack WebHook integration and how to make some fancier formatted messages.

Here is the code I came up with.

var https = require('https');

exports.handler = (event, context, callback) => {
   
    console.log("MYLOG" + JSON.stringify(event))
   
    var post_data = JSON.stringify({
        "username": "Distil.io",
        "icon_emoji":":distillio:",
        "channel": "#distillio",
        "attachments": [
            {
                "color": "danger",
                "fields": [
                    {
                        "title": "Website Changed!",
                        "value": "The website changed",
                        "short": false
                    }
                ]
            }
        ]
    });
 
    // An object of options to indicate where to post to
    var post_options = {
      host: 'hooks.slack.com',
      port: '443',
      path: path: '/services/YOURWEBHOOKHERE',
      method: 'POST',
      headers: {
          'Content-Type': 'application/json',
          'Content-Length': Buffer.byteLength(post_data)
      }
    };
   
    // Set up the request
    var post_req = https.request(post_options, function(res) {
      res.setEncoding('utf8');
      res.on('data', function (chunk) {
          console.log('Response: ' + chunk);
      });
    });
   
    // post the data
    post_req.write(post_data);
    post_req.end();
   
   
    var details = {
        "one": "44",
        "two" : {
            "details" :"Some details",
            "another" :"More details"
        }
    }
   
    var response = {
        'statusCode': 200,
        'headers': { 'Content-Type': 'application/json' },
        'body': JSON.stringify(details)
    }
    console.log("LOG:: " + JSON.stringify(response))
   
    callback(null, response);
};


If I run a simple curl test against this I get




It now uses an emoji as a user icon, has a username of Distil.io, and highlights the message with a red line.


What it does not do yet is grab incoming data and put it into my slack message.



OK after many tries and tweaks got this one working.


var https = require('https');

exports.handler = (event, context, callback) => {
   
    console.log("MYLOG" + JSON.stringify(event))
   
    //Custom Fields sent in from Distilio
    var body = JSON.parse(event.body)
    var name = body.name
    var uri = body.uri
    var text = body.text.substring(0, 120) //Don't want all the data
   
    var post_data = JSON.stringify({
        "username": "Distil.io",
        "icon_emoji":":distillio:",
        "channel": "#distillio",
        "attachments": [
            {
                "color": "danger",
                "fields": [
                    {
                        "title": name + " Website Changed!",
                        "value": "uri: " + uri
                               + "\nbody: " + text,
                        "short": false
                    }
                ]
            }
        ]
    });
 
    // An object of options to indicate where to post to
    var post_options = {
      host: 'hooks.slack.com',
      port: '443',
      path: '/services/YOURWEBHOOKHERE',
      method: 'POST',
      headers: {
          'Content-Type': 'application/json',
          'Content-Length': Buffer.byteLength(post_data)
      }
    };
   
    // Set up the request
    var post_req = https.request(post_options, function(res) {
      res.setEncoding('utf8');
      res.on('data', function (chunk) {
          console.log('Response: ' + chunk);
      });
    });
   
    // post the data
    post_req.write(post_data);
    post_req.end();
   
   
    var details = {
        "status": "OK"
    }
   
    var response = {
        'statusCode': 200,
        'headers': { 'Content-Type': 'application/json' },
        'body': JSON.stringify(details)
    }
    console.log("LOG:: " + JSON.stringify(response))
   
    callback(null, response);
};



I also put this up as a gist for easy copying.





References

[1]        AWS Lambda
            https://aws.amazon.com/lambda/
                Accessed 12/2017
[2]        AWS Lambda Pricing
            https://aws.amazon.com/lambda/pricing/
                Accessed 12/2017
[3]        AWS Lambda intro to permissions.
            http://docs.aws.amazon.com/lambda/latest/dg/intro-permission-model.html
                Accessed 12/2017
[4]        Solving AWS Lambda and API Gateway Internal Server Errors
            http://www.awslessons.com/2017/lambda-api-gateway-internal-server-error/
                Accessed 12/2017
[5]        Posting Message to Slack via Webhooks Integration
            http://www.whiteboardcoder.com/2015/03/posting-message-to-slack-via-webhooks.html
                Accessed 12/2017
[6]        Amazon API Gateway Pricing
            https://aws.amazon.com/api-gateway/pricing/
                Accessed 12/2017


8 comments:

  1. Nice post, I like to read this blog. It is very interesting to read.
    fibonacci in python
    python class inheritance

    ReplyDelete
  2. "MY PROGRAMMER IS TRYING TO CONVINCE ME TO MOVE TO .NET
    FROM PHP. I HAVE ALWAYS DISLIKED THE IDEA BECAUSE OF THE EXPENSES.
    BUT HE'S TRYING NONE THE LESS. I'VE BEEN USING WORD PRESS ON A NUMBER OF WEBSITES
    FOR ABOUT A YEAR AND AM NERVOUS ABOUT SWITCHING TO ANOTHER PLATFORM.
    I HAVE HEARD GREAT THINGS ABOUT BLOG ENGINE IS THERE A
    WAY I CAN TRANSFER ALL MY WORD PRESS POSTS INTO IT? ANY KIND OF HELP WOULD BE GREATLY APPRECIATED!"
    무료야설

    ReplyDelete
  3. You can't imagine simply how much time I had spent for this info! Thanks!립카페

    ReplyDelete
  4. I just could not depart your site prior to suggesting that I really enjoyed the standard info a person provide for your visitors? Is going to be back often to check up on new posts 스포츠마사지

    ReplyDelete
  5. After looking into a few of the blog posts on your site, I seriously appreciate your way of blogging.출장마사지

    ReplyDelete
  6. This is a very good thing! you are providing the content is very unique and very impressed to me. I always follow your blog, please keep blogging. Join our interactive online English tuition classes and unlock your true potential in the English language from the comfort of your home.
    For more info visit English tuition classes

    ReplyDelete