A Simpler Load-Balancing Setup With HAProxy

Fri Feb 05 15:32:16 EST 2021

...where by "simpler" I mean relative to the setup I detailed six years ago.

For a good long time now, I've had a reverse-proxy + load balancer setup that uses nginx for the main front end and HAProxy as an intermediary to do the actual load balancing. The reason I set it up this way was that I was constrained by two limitations:

  • nginx's built-in load balancing didn't do sticky sessions like I needed, which would break server-side-state frameworks like XPages
  • HAProxy didn't do HTTPS

In the intervening half-decade, things have improved. I haven't checked on nginx's load balancing, but HAProxy sprouted splendid HTTPS capabilities. So, for the new servers I've been setting up, I decided to take a swing at it with HAProxy alone.

Disclaimer

Before I go any further, I should point out that this is only a viable solution because I would otherwise use nginx only for being the HTTPS frontend. In other cases, I've used it to host files directly, run CGI scripts, etc., and it'd be best to keep it around if you want to do similar things.

Basic SSL Config

The "global" section of haproxy.cfg contains settings for your TLS ciphers and related parameters, and Mozilla's config generator is your friend here. Today, I ended up with this (slightly tweaked to generate dhparams locally):

global
	#
	# SNIP: a bunch of default stuff
	#
	crt-base /etc/ssl/private

	# See: https://ssl-config.mozilla.org/#server=haproxy&version=2.0.13&config=intermediate&openssl=1.1.1d&guideline=5.6
	ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
	ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
	ssl-default-bind-options prefer-client-ciphers no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

	ssl-default-server-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
	ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
	ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
	
	# sudo openssl dhparam -out /etc/haproxy/dhparams.pem 2048
	ssl-dh-param-file /etc/haproxy/dhparams.pem

Though arcane, that's fairly standard stuff for TLS configuration.

Frontend Config

Years ago, my original config put everything in a listen block, but it's properly split up into frontend and backend now. The frontend block is pretty simple:

frontend frontend1
	bind *:80
	bind *:443 ssl crt star.clientdomain1.com.pem crt star.clientdomain2.com.pem alpn h2,http/1.1
	http-request redirect scheme https unless { ssl_fc }
	default_backend domino

HAProxy's configuration file is almost painfully terse, but at least this part ends up readable enough. I bind to ports 80 and 443 on all IP addresses, and then provide multiple certificate files to be picked based on SNI. Conveniently, HAProxy does a nice job of just picking the right one, and you don't have to explicitly match them up with incoming host names.

One oddity here is the particular format for those ".pem" files. HAProxy expects the actual certificate, its chain, and the private key to all be concatenated together. This is as opposed to nginx, where the chain and private key are two files, or Apache's split into cert+chain+key files. It's also very explicitly not a PKCS file, which is the more-common way to package a key in with the certs: there's no encryption and no password assigned for this.

Additionally, I just put the base names for the files there because they're in /etc/ssl/private, as configured in global.

Back to the rest of the configuration: the http-request line does the work of auto-redirecting from HTTP to HTTPS. Again, very terse, and it's using the ssl_fc configuration token to check if the incoming connection is SSL.

Finally, default_backend domino ties in to the next section.

Backend Config

The backend configuration is the meat of it:

backend domino
	balance roundrobin
	cookie backend insert httponly secure
	option httpchk HEAD /names.nsf?login HTTP/1.0
	http-request add-header $WSRA %[src]
	http-request add-header $WSRH %[src]
	http-request add-header X-ConnectorHeaders-Secret 12345
	
	# "cookie d*" = set and use a cookie to tie to the backend
	# "check" = I don't know, but I assume it checks something
	# "ssl" = Connect to the backend with SSL
	# "verify none" = Don't bother with SSL verification checks
	# "sni ssl_fc_sni" = Use the incoming SNI hint when connecting to the backend
	server domino-1 domino-1.client.com:443 cookie d1 check ssl verify none sni ssl_fc_sni
	server domino-2 domino-2.client.com:443 cookie d2 check ssl verify none sni ssl_fc_sni

The balance roundrobin and cookie ... lines tell HAProxy to cycle through the backends for incoming connections, but to stick the client with a specific backend server based on the value of the backend cookie, if present, and then to set it in the response. That covers our sticky sessions.

The next line, option httpchk HEAD /names.nsf?login HTTP/1.0, tells HAProxy how to check the health of the servers. This should be something very inexpensive that's also a reliable way to tell if the server is working. I went with asking for headers for the default login page - something all Domino servers (with session auth) will have and which doesn't risk running application code like / might.

The next three lines are my beloved Domino connector headers, plus the shared secret from my locking-down DSAPI filter (I mean, it's not the actual shared secret, but that's where it goes). Note that I don't need to include $WSSN to denote the requested Host value, since HAProxy passes that along by default.

Finally, there are the actual backend configuration lines. Because the load balancer is communicating with Domino via SSL, I tell it to do so and to not bother validating the certificates. Additionally, I tell it to pass along the incoming SNI hint to Domino, which, since Domino finally supports SNI, routes the request to the correct web site on the Domino site.

If you were to connect to the Domino servers via HTTP, you could snip off a bit from those lines and add http-request add-header $WSIS True above.

Conclusion

I haven't actually put this into production yet, so the details my change, but I'm thoroughly pleased that I can simplify the configuration a good deal. I've found learning about how to configure HAProxy a little less pleasant than learning about nginx, but part of that is just learning some of the terminology and how to navigate the documentation - it's all there; it's just a little arcane.

New Comment