Let's Encrypt HAProxy Round 2

Posted on Saturday, September 10, 2016




I wrote a How to use HAProxy with Let's encrypt a few months ago http://www.whiteboardcoder.com/2016/05/lets-encrypt-haproxy.html .  Now I have recently had a chance to tweak on it again and I figured out a better way to do it.   Also I am going to tackle a little more advanced example.

I am going to do this all in Ubuntu 16.04.







At the end of this I will have this basic setup.


I am going to have
·         A single HAProxy box
o   It will load balance two URLS
o   It will handle SSL certs for those two URLS
o   It will update the SSL certs via cron job using Let's Encrypt





NGINX boxes



The first nginx box will be
·         Listen on port 8080
·         Has a check at /check used by the HAProxy (Check always returns UP unless the file /tmp/check exists)

Install nginx server


  > sudo apt-get install nginx


Edit the /etc/nginx/nginx.conf


  > sudo vi /etc/nginx/nginx.conf


And place the following in it


user www-data;
worker_processes 4;
pid /var/run/nginx.pid;



events {
  worker_connections 1024;
  use epoll;
  multi_accept on;
}


http {

  include /etc/nginx/mime.types;
  index    index.html index.htm;

  default_type application/octet-stream;

  sendfile     on;
  tcp_nopush   on;
  tcp_nodelay on;
  server_names_hash_bucket_size 128;
  keepalive_timeout    70;
  types_hash_max_size 2048;


  gzip on;
  gzip_disable "msie6";

  log_format  main_fmt '$remote_addr - $remote_user [$time_local]  $status '
    '"$request" $body_bytes_sent "$http_referer" '
    '"$http_user_agent" "$http_x_forwarded_for"';


  proxy_buffering    off;
  proxy_set_header   X-Real-IP $remote_addr;
  proxy_set_header   X-Scheme $scheme;
  proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header   Host $http_host;

  access_log /var/log/nginx/access.log main_fmt;

  server {
      listen       8080;      server_name  localhost;

      location / {
          root   /usr/share/nginx/html;
          index  index.html index.htm;
      }

      location /check {
        if (-f /tmp/check) {           return  404;        }         add_header Content-Type text/plain;         return 200 "UP";      }

      error_page   500 502 503 504  /50x.html;
      location = /50x.html {
          root   /usr/share/nginx/html;
      }
  }





I also edited the /usr/share/nginx/html/index.html file per machine so I would know which machine I was hitting.


  > sudo vi /usr/share/nginx/html/index.html


Place the following in it.


<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
    span {
        color: red;
    }
    h1 {
        font-size: 75px;
    }
</style>
</head>
<body>
<center>
<h2>This is the nginx server <span>NGINX-01</span></h2>
</center>
</body>
</html>



Restart nginx


  > sudo service nginx restart



Quick test open http://192.168.0.10:8080/









HAProxy server


Before I get Let's Encrypt working I want to get HAProxy installed and load balancing between the two servers.

Install haproxy


  > sudo apt-get -y install haproxy


Now edit /etc/haproxy/haproxy.cfg


  > sudo vi /etc/haproxy/haproxy.cfg


Let me just load balance the first two servers on port 80


global
  log         127.0.0.1 syslog
  maxconn     1000
  user        haproxy
  group       haproxy
  daemon


defaults
  log  global
  mode  http
  option  httplog
  option  dontlognull
  option  http-server-close
  option  forwardfor except 127.0.0.0/8
  option  redispatch
  option  contstats
  retries  3
  timeout  http-request 10s
  timeout  queue 1m
  timeout  connect 10s
  timeout  client 1m
  timeout  server 1m
  timeout  check 10s


###########################################
#
# HAProxy Stats page
#
###########################################
listen stats
  bind *:9090
  mode  http
  maxconn  10
  stats  enable
  stats  hide-version
  stats  realm Haproxy\ Statistics
  stats  uri /
  stats  auth admin:admin

###########################################
#
# Front end for all
#
###########################################
frontend ALL
  bind   *:80
  mode   http

  # Define hosts
  acl host_foo hdr(host) -i foo.test.10x13.com
  acl host_bar hdr(host) -i bar.test.10x13.com

  # Direct hosts to backend
  use_backend foo if host_foo
  use_backend bar if host_bar

###########################################
#
# Back end for foo
#
###########################################
backend foo
  balance         roundrobin
  option          httpchk GET /check
  http-check      expect rstring ^UP$
  default-server  inter 3s fall 3 rise 2
  server          server1 192.168.0.10:8080 check
  server          server2 192.168.0.11:8080 check

###########################################
#
# Back end for bar
#
###########################################
backend bar
  balance         roundrobin
  option          httpchk GET /check
  http-check      expect rstring ^UP$
  default-server  inter 3s fall 3 rise 2
  server          server1 192.168.0.12:8080 check
  server          server2 192.168.0.13:8080 check



Reload haproxy


  > sudo service haproxy reload







Looks like it's all connected just fine


















Certbot and nginx


When using the letsencrypt command line tool it will try and hit the URL you are requesting the ssl cert for at

<URL>:80/.well-known/acme-challenge/

With this in mind I am going to install nginx on the same box as the haproxy box.  Have the haproxy route that path to the local nginx box.  This will allow me to handle and update all the ssl certs on the haproxy box with ease.

If you want to avoid this error

Errors were encountered while processing:
 nginx-core
 nginx
E: Sub-process /usr/bin/dpkg returned an error code (1)

You first need to stop the nginx service before installing haproxy (I think because nginx wants to grab port 80)


  > sudo service haproxy stop


Install nginx


  > sudo apt-get -y install nginx


Edit the nginx.conf file to listen locally on port 8888

Edit the /etc/nginx/nginx.conf


  > sudo vi /etc/nginx/nginx.conf


And place the following in it


user www-data;
worker_processes 4;
pid /var/run/nginx.pid;



events {
  worker_connections 1024;
  use epoll;
  multi_accept on;
}


http {

  include /etc/nginx/mime.types;
  index    index.html index.htm;

  default_type application/octet-stream;

  sendfile     on;
  tcp_nopush   on;
  tcp_nodelay on;
  server_names_hash_bucket_size 128;
  keepalive_timeout    70;
  types_hash_max_size 2048;


  gzip on;
  gzip_disable "msie6";

  log_format  main_fmt '$remote_addr - $remote_user [$time_local]  $status '
    '"$request" $body_bytes_sent "$http_referer" '
    '"$http_user_agent" "$http_x_forwarded_for"';


  proxy_buffering    off;
  proxy_set_header   X-Real-IP $remote_addr;
  proxy_set_header   X-Scheme $scheme;
  proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header   Host $http_host;

  access_log /var/log/nginx/access.log main_fmt;

  server {
      listen       8888;
      server_name  localhost;

      location /.well-known/acme-challenge/ {
          root   /usr/share/nginx/html/;
      }

      error_page   500 502 503 504  /50x.html;
      location = /50x.html {
          root   /usr/share/nginx/html;
      }

      location / {
         return 404;
      }
  }

}




Restart it


  > sudo service nginx restart


Restart haproxy it


  > sudo service haproxy restart




Tweak haproxy.cfg


So that it will send /.well-known/acme-challenge/ to the local nginx box.


  > sudo vi /etc/haproxy/haproxy.cfg


Add the highlighted portion


global
  log         127.0.0.1 syslog
  maxconn     1000
  user        haproxy
  group       haproxy
  daemon


defaults
  log  global
  mode  http
  option  httplog
  option  dontlognull
  option  http-server-close
  option  forwardfor except 127.0.0.0/8
  option  redispatch
  option  contstats
  retries  3
  timeout  http-request 10s
  timeout  queue 1m
  timeout  connect 10s
  timeout  client 1m
  timeout  server 1m
  timeout  check 10s


###########################################
#
# HAProxy Stats page
#
###########################################
listen stats
  bind *:9090
  mode  http
  maxconn  10
  stats  enable
  stats  hide-version
  stats  realm Haproxy\ Statistics
  stats  uri /
  stats  auth admin:admin

###########################################
#
# Front end for all
#
###########################################
frontend ALL
  bind   *:80
  mode   http

  # Define path for lets encrypt
  acl is_letsencrypt path_beg -i /.well-known/acme-challenge/
  use_backend letsencrypt if is_letsencrypt

  # Define hosts
  acl host_foo hdr(host) -i foo.test.10x13.com
  acl host_bar hdr(host) -i bar.test.10x13.com

  # Direct hosts to backend
  use_backend foo if host_foo
  use_backend bar if host_bar


###########################################
#
# Back end letsencrypt
#
###########################################
backend letsencrypt
  server letsencrypt 127.0.0.1:8888

###########################################
#
# Back end for foo
#
###########################################
backend foo
  balance         roundrobin
  option          httpchk GET /check
  http-check      expect rstring ^UP$
  default-server  inter 3s fall 3 rise 2
  server          server1 192.168.0.10:8080 check
  server          server2 192.168.0.11:8080 check

###########################################
#
# Back end for bar
#
###########################################
backend bar
  balance         roundrobin
  option          httpchk GET /check
  http-check      expect rstring ^UP$
  default-server  inter 3s fall 3 rise 2
  server          server1 192.168.0.12:8080 check
  server          server2 192.168.0.13:8080 check



Reload haproxy


  > sudo service haproxy reload









Install Certbot


Certbot is up on github at https://github.com/certbot/certbot [1]

Good notes on how to install and use certbot can be found at https://certbot.eff.org/ [2]



It even helps walk you through.






I'll choose HAProxy on Ubuntu 16.04







And it shows me the install procedure.
Looks like Ubuntu 16.04 has a simpler install procedure.




  > sudo apt-get install letsencrypt


Make sure your URL is accessible from the outside world then run this command to get a certificate automatically.

Now let me get a test cert  (change the email and domain to your own)  Set the --server to use the staging  server so  you don't hit the weekly cert limit when you are testing see https://letsencrypt.org/docs/staging-environment/ [4]


  > sudo letsencrypt certonly \
        --server https://acme-staging.api.letsencrypt.org/directory  \
        --webroot --webroot-path "/usr/share/nginx/html/" \
        --keep-until-expiring \
        --text \
        -v \
        --email me@example.com \
        --agree-tos \
        -d foo.test.10x13.com


This will create a file in /usr/share/nginx/html//.well-known/acme-challenge/ that letsencrypt uses to confirm you own the domain name




Now you have a cert at /etc/letsencrypt/live/foo.test.10x13.com


  > sudo tree /etc/letsencrypt/live/




This cert was made using the staging server so it won't work.  To get a legit cert run this command.




  > sudo letsencrypt certonly \
        --webroot --webroot-path "/usr/share/nginx/html/" \
        --keep-until-expiring \
        --text \
        -v \
        --email me@example.com \
        --agree-tos \
        -d foo.test.10x13.com



Now that you have a legit cert it needs to be combined (this is needed for haproxy)

Make a directory to stick the combined cert


  > sudo mkdir -p /etc/haproxy/certs/




  > sudo bash -c "cat /etc/letsencrypt/live/foo.test.10x13.com/fullchain.pem /etc/letsencrypt/live/foo.test.10x13.com/privkey.pem > /etc/haproxy/certs/foo.test.10x13.com.pem"












My script



List the domains you want to get SSL certs for and this script will obtainer the cert if you don't have it or if you do have it will see if it needs to be renewed and renew it.   At the end if a cert has been added or renewed it reloads haproxy.


  > sudo vi /usr/local/sbin/le-renew-haproxy



Here is my script


#!/bin/bash
#
#  Let's Encrypt HAProxy script
#
###################################

DOMAINS=(
   "foo.test.10x13.com"
   "bar.test.10x13.com"
)

EMAIL="me@example.com"
WEB_ROOT="/usr/share/nginx/html/"

#When cert is down to this many days
#It is allowed to renew
EXP_LIMIT=30;

#Only reload HAProxy if a cert was created/updated
RELOAD=false

for domain in "${DOMAINS[@]}"
do
  CERT_FILE="/etc/letsencrypt/live/$domain/fullchain.pem"
  KEY_FILE="/etc/letsencrypt/live/$domain/privkey.pem"

  ##################################
  #
  # If no ssl for domain create it
  #
  ##################################
  if [ ! -f $CERT_FILE ]; then
    echo "Creating certificate for domain $domain."
    letsencrypt certonly \
        --webroot --webroot-path $WEB_ROOT \
        --email $EMAIL \
        --agree-tos \
        -d $domain

    ###################################
    #
    # Combine certs for HAProxy and
    # Reload HAProxy
    #
    ###################################
    mkdir -p /etc/haproxy/certs/  #location to place combine cert
    RELOAD=true
    COMBINED_FILE="/etc/haproxy/certs/${domain}.pem"
    echo "Creating $COMBINED_FILE with latest certs..."
    cat /etc/letsencrypt/live/$domain/fullchain.pem \
        /etc/letsencrypt/live/$domain/privkey.pem > $COMBINED_FILE


    RELOAD=true
  else
    ##################################
    #
    # Check How long cert is valid
    #
    ##################################
    EXP=$(date -d "`openssl x509 -in $CERT_FILE -text -noout|grep "Not After"|cut -c 25-`" +%s)
    DATE_NOW=$(date -d "now" +%s)
    DAYS_EXP=$(echo \( $EXP - $DATE_NOW \) / 86400 |bc)

    if [ "$DAYS_EXP" -gt "$EXP_LIMIT" ] ; then
      echo "$domain, no need for renewal ($DAYS_EXP days left)."
    else
      #################################
      #
      # Renew Certifcate
      #
      #################################
      echo "The certificate for $domain is about to expire soon."
      echo "Starting Let's Encrypt renewal script..."
      letsencrypt certonly \
        --webroot --webroot-path $WEB_ROOT \
        --keep-until-expiring \
        --text \
        -v \
        --email $EMAIL \
        --agree-tos \
        -d $domain

      ###################################
      #
      # Combine certs for HAProxy and
      # Reload HAProxy
      #
      ###################################
      mkdir -p /etc/haproxy/certs/  #location to place combine cert
      RELOAD=true
      COMBINED_FILE="/etc/haproxy/certs/${domain}.pem"
      echo "Creating $COMBINED_FILE with latest certs..."
      cat /etc/letsencrypt/live/$domain/fullchain.pem \
          /etc/letsencrypt/live/$domain/privkey.pem > $COMBINED_FILE

      echo "Renewal process finished for domain $domain"
    fi
  fi
done

if [ "$RELOAD" = true ]
then
  echo " ========================= "
  echo " =                       = "
  echo " === Reloading HAProxy === "
  echo " =                       = "
  echo " ========================= "
  service haproxy reload
fi


Make it executable


  > sudo chmod u+x /usr/local/sbin/le-renew-haproxy





To get this script to run I had to install bc


  > sudo apt-get install bc



Now run it


  > sudo /usr/local/sbin/le-renew-haproxy


The first time you run it if you do not already have certs it will create them.




If I run it again it checks the certs to see if they even need to be renewed.










Update HAProxy.cfg to handle SSL




  > sudo vi /etc/haproxy/haproxy.cfg


Add the highlighted portion



global
  log         127.0.0.1 syslog
  maxconn     1000
  user        haproxy
  group       haproxy
  daemon
  tune.ssl.default-dh-param 4096
  ssl-default-bind-options no-sslv3 no-tls-tickets
  ssl-default-bind-ciphers EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH


defaults
  log  global
  mode  http
  option  httplog
  option  dontlognull
  option  http-server-close
  option  forwardfor except 127.0.0.0/8
  option  redispatch
  option  contstats
  retries  3
  timeout  http-request 10s
  timeout  queue 1m
  timeout  connect 10s
  timeout  client 1m
  timeout  server 1m
  timeout  check 10s


###########################################
#
# HAProxy Stats page
#
###########################################
listen stats
  bind *:9090
  mode  http
  maxconn  10
  stats  enable
  stats  hide-version
  stats  realm Haproxy\ Statistics
  stats  uri /
  stats  auth admin:admin

###########################################
#
# Front end for all
#
###########################################
frontend ALL
  bind   *:80
  bind   *:443 ssl crt /etc/haproxy/certs/bar.test.10x13.com.pem crt /etc/haproxy/certs/foo.test.10x13.com.pem
  mode   http

  # Define path for lets encrypt
  acl is_letsencrypt path_beg -i /.well-known/acme-challenge/
  use_backend letsencrypt if is_letsencrypt

  # Define hosts
  acl host_foo hdr(host) -i foo.test.10x13.com
  acl host_bar hdr(host) -i bar.test.10x13.com

  # Direct hosts to backend
  use_backend foo if host_foo
  use_backend bar if host_bar

  # Redirect port 80 to 443
  # But do not redirect letsencrypt since it checks port 80 and not 443
  redirect scheme https code 301 if !{ ssl_fc } !is_letsencrypt

###########################################
#
# Back end letsencrypt
#
###########################################
backend letsencrypt
  server letsencrypt 127.0.0.1:8888

###########################################
#
# Back end for foo
#
###########################################
backend foo
  balance         roundrobin
  option          httpchk GET /check
  http-check      expect rstring ^UP$
  default-server  inter 3s fall 3 rise 2
  server          server1 192.168.0.10:8080 check
  server          server2 192.168.0.11:8080 check

###########################################
#
# Back end for bar
#
###########################################
backend bar
  balance         roundrobin
  option          httpchk GET /check
  http-check      expect rstring ^UP$
  default-server  inter 3s fall 3 rise 2
  server          server1 192.168.0.12:8080 check
  server          server2 192.168.0.13:8080 check



Reload HAproxy


  > sudo service haproxy reload



Now if you open http://foo.test.10x13.com/   it will redirect you to the secure version https://foo.test.10x13.com/.   The same goes for bar.test.10x13.com.

But if you go to the lets encrypt path it will not. http://foo.test.10x13.com/.well-known/acme-challenge/ Does not go to https.




Cron Job


Let me add it this script as a cron job to run every day.  (I am going to run it every day at 11:00 AM local time)



  > sudo vi /etc/crontab


This line should work


  > 10 11   * * *   root    /usr/local/sbin/le-renew-haproxy




Now to just wait 60 days and prove this works J



References


[1]        Certbot's github page
Accessed 08/2016
[2]        Certbot eff page
Accessed 08/2016
[3]        LetsEncrypt renewal gist example
Accessed 08/2016
[4]        LetsEncrypt Staging Server
Accessed 08/2016



6 comments:

  1. Nice article, but what about using letsencrypt with haproxy+Keepalived (or other fail-over solution)?
    Certificates should be synchronised between two servers. After that we will have to renew certificates manually (perhaps also synchronising letsencrypt directory) or we need other solution how to implement auto-renew on fail-over server.

    ReplyDelete
    Replies
    1. I think you would want to keep your certs synced with your backup server or for that matter any situation where you have multiple HAProxy boxes. If you had a situation where you had multiple HAProxy boxes I would designate one HAProxy box the ssl cert renewer and point /.well-known/acme-challenge from other haproxy boxes to that one. Then have the other boxes update their certs once a day from the cert haproxy box.

      ... if the haproxy box that renews the certs should fail... you have 30-90 days before you certs go bad... I think that gives you plenty of time to either get that server back up or turn one of the other ones into the cert haproxy box.

      Delete
  2. Hi Patrick :)
    Your article helped me alot in using Lets Encrypt. I have one question though. What if instead of one domain, i had multiple domains on single nginx box? lets say there were "foo1", "foo2" on box 1 and they were in different root folders. How would you configure nginx box on HaProxy box then, and how would your le-renew scrypt looked like?
    Thank you in advance
    Marijan Kovacic

    ReplyDelete
    Replies
    1. There are a couple of ways to tackle it. Let's say you are limited to one box that would host the nginx and haproxy (a little odd but lets go wit that). I would give port 80 and 443 to the haproxy box. The domain names would hit the haproxy box where it can filter by domain (I used subdomains in this example, but it can handle full domains as well). I would have nginx set up to listen on an odd port per domain foo1 port 1080 foo2 port 1180. Then nginx can handle the fact that the different web sites are located in different folders on the same machine.

      THe only question now is how to route the .well-known/acme-challenge/ ... let me think for a second
      Here is a link to my gist renewal script https://gist.github.com/patmandenver/7500fde43ed032b6fc853af826ea3ab6
      (Swap out your domains and email)

      The way it is set up the local letsencrypt tool will put a file in /usr/share/nginx/html/ (this is defined in the script WEB_ROOT="/usr/share/nginx/html/") and will ask letsencrypt to hit it to confirm you actually control the domain you are claiming to control.

      the script is not multithreaded it only updates one ssl cert at a time. (the first then second then third...and so on)
      so reusing the same folder is fine... thinking...

      Oh then it's easy just have keep the haproxy cfg file almost the same having all .well-known traffic route to port 8888 locally. Then in the nginx.conf file have one server section for handling all letsencrypt traffic regardless of domain, then a server section defined per domain you want to run.

      Sorry I know this was a little longwinded, if it does not make sense ping me again

      Delete
    2. Hi Patrick :)

      I have tried every possible way of redirecting Let's encrypt to box behind HaProxy but there was no success. Your approach works perfectly. In my case i have used already installed Apache web server to listen on port 54321 which is used for KeepAlive between two balancers. Made puppet agent to sync certs between two HaProxy boxes.

      You should add into your tutorial that this works for any number of domains that are served from backend nodes and that let's encrypt does not care if specified webroot is really root folder from where domain is served as long it can reach that path and read challenge. I did not understood that before :)

      Really want to thank you for this :) keep up good work

      Delete
    3. Glad to hear it worked out well for you. and thanks for the notes :)

      Delete