SSL load balancing with HAProxy in VMWare

So this is a new project I’ve recently finished.

Objective
Create a secure high availability (HA) load balancing service spreading user load across two pairs of two servers, providing two different sets of services:

One service requires SSL passthrough, while the other is a websockets connection over SSL, where the use of a proxy demands SSL termination. Securing communications with the web backend for the latter is done by routing the traffic via an OpenVPN tunnel.

The software I’ve chosen for this, is HAProxy 1.5 on FreeBSD 10.1-Release, running in a VSphere 5.5 environment.

Basic server setup

All programs and services are installed via the excellent Ports system, which in itself is a reason to love FreeBSD in a server environment.

The requirement to pass on some SSL traffic untouched and terminate and send on other SSL traffic means I need the two URLs to point at different external IP addresses, and listening on separate addresses in my DMZ after the firewall has taken care of the NAT. The way to be able to listen to several addresses on the same subnet, is to create aliases for the network interface:

/etc/rc.conf:

...
 ifconfig_vmx3f0="inet 172.27.1.15 netmask 255.255.255.0"
 ifconfig_vmx3f0_alias0="inet 172.27.1.20/32"
 defaultrouter="172.27.1.1"
 ...

Note the /32 mask for the alias interface. This is not an error.

HAProxy setup

I’ve set up two separate front-ends for the server. The “integration” part passes on SSL traffic to be terminated by the web server, and the “application” part terminates the SSL connection:


/usr/local/etc/haproxy.conf:

...
 frontend integration_frontend
 bind 172.27.1.15:443
 mode tcp
 option tcplog
 log global
 option logasap
 default_backend integration_pool

frontend application_frontend
 bind 172.27.1.20:80
 redirect scheme https code 301 if !{ ssl_fc }
 bind 172.27.1.20:443 ssl crt /etc/ssl/sitename.pem
 mode http
 log global
 option httplog
 option logasap
 option forwardfor
 default_backend bfx_application_pool
...

In the “application” section, I add an X-Forwarded-For field to the http header after terminating the SSL connection. The background for this, is that otherwise, the originator of the requests for data from the application web servers seems to be the HAProxy server. With the field in place, the application can return data directly to the actual client.

I also listen for the “upgrade” word in the http header, to be able to properly handle web sockets, using access control lists (ACLs).

The difference between option tcplog and option httplog is that the latter examines the http header, while the former doesn’t attempt to go deeper than the actual tcp packet information. The reason for the difference in configuration, of course, is that if I don’t open the SSL packets, I can’t know the contents of the http header nor manipulate it.

The backend setup:

/usr/local/etc/haproxy.conf

...
 backend integration_pool
 log global
 option tcplog
 balance roundrobin
 option ssl-hello-chk

 server webserverdmz01 172.27.1.150:443 check
 server webserverdmz02 172.27.1.160:443 check

backend application_pool
 mode http
 log global
 option httplog
 option http-server-close
 balance roundrobin

stick-table type binary len 32 size 30k expire 30m
 acl clienthello req_ssl_hello_type 1
 acl serverhello rep_ssl_hello_type 2
 tcp-request inspect-delay 5s
 tcp-request content accept if clienthello
 tcp-response content accept if serverhello
 stick on payload_lv(43,1) if clienthello
 stick store-response payload_lv(43,1) if serverhello

 server webserver01 10.0.5.1:80 check
 server webserver02 10.0.6.1:80 check
...

One immediate difference between the “integration” and the “application” sections, is that the integration section naturally tests for the existence of its web servers using SSL.

The sections that begin with “stick-table” in the application section creates what’s called a sticky session, where we attempt to direct a given user to the same web server for every new request they make. This simplifies life a bit for the web application programmers.

Finally, in the application and web-sockets sections, I connect to the respective web servers on port 80, which I can do since I connect via OpenVPN connections directly to the servers.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s