Setting up nginx in Front of a Domino Server
Thu Sep 18 13:08:46 EDT 2014
- Setting up nginx in Front of a Domino Server
- Adding Load Balancing to the nginx Setup
- Arbitrary Authentication with an nginx Reverse Proxy
- Domino and SSL: Come with Me If You Want to Live
As I've mentioned before and now presented on, I'm a big proponent of using a reverse proxy in front of Domino. There are numerous benefits to be gained, particularly when you expand your infrastructure to include multiple back-end servers. But even in the case of a single server, I've found it very worthwhile to set up, and not overly complicated. This example uses nginx and Domino on Ubuntu Linux, but the ideas and some configuration apply much the same way on other OSes and with other web servers.
Domino
The first step involves a bit of configuation on the Domino server. The first is to move Domino off the main port 80, disable SSL, and, ideally, bind it to a local-only IP address. The port setting is familiar - I picked port 8088 here, but it doesn't matter too much what you pick as long as it doesn't conflict with anything else on your server:
The next step is to bind Domino to a local-only adapter so external clients don't access its HTTP stack directly. In this example, I have a LAN-only adapter whose IP address I named "terminus-local" in /etc/hosts, but I imagine "localhost" would work just fine in this case:
Once that's set, the last stage of configuration is to enable the WebSphere connector headers by setting a notes.ini property:
HTTPEnableConnectorHeaders=1
Enabling these will allow us to send specialized headers from our reverse proxy to Domino to make Domino act as if the request is coming to it directly.
After that, restart Domino (or just HTTP, probably).
nginx
Next, it's on to setting up nginx. On Ubuntu/Debian, it's pretty straightforward:
# apt-get install nginx
The main config file /etc/nginx/nginx.conf should be good as-is. The way the Ubuntu config works, you set up individual web site files inside the /etc/nginx/sites-available directory and then create symlinks to them in the /etc/nginx/sites-enabled directory. Out of convention, I name them like "000-somesite" to keep the priority clear. The first file to create is a site to listen on port 80, which will serve entirely as a redirect to SSL. You don't have to do this - instead, you could bring the content from the next file into this one instead of the redirection line. This is usually a good idea, though. This file is 001-http-redirect:
server {
listen [::]:80;
return https://$host$request_uri;
}
The only really oddball thing here is the "listen" line. Normally, that would just be "listen 80", but adding the brackets and colons allows it to work on IPv4 and IPv6 on all addresses.
The next file is the important one for doing the proxying, as well as SSL. It's 002-domino-ssl:
server {
listen [::]:443;
client_max_body_size 100m;
ssl on;
ssl_certificate /etc/nginx/ssl/ssl-unified-noanchor.pem;
ssl_certificate_key /etc/nginx/ssl/ssl.key;
location / {
proxy_read_timeout 240;
proxy_pass http://localhost:8088;
proxy_redirect off;
proxy_buffering off;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header $WSRA $remote_addr;
proxy_set_header $WSRH $remote_addr;
proxy_set_header $WSSN $host;
proxy_set_header $WSIS True;
}
}
The client_max_body_size line is to allow uploads up to 100MB. One thing to be aware of when using proxies is that they can impose their own limits on request sizes just as Domino does, and nginx's default is relatively low.
nginx's keychain format is almost as simple as just pointing it to your certificate and private key, with one catch: to have intermediate signing certificates (like those from your SSL provider or registrar), you concatenate the certificates into a single file. This tutorial covers it (and this config) nicely.
The core of the reverse proxy comes in with that location / block. In a more-complicated setup, you might have several such blocks to point to different apps, app servers, or local directories, but in this case we're just passing everything directly through to Domino. The first four lines do just that, setting a couple options to account for very-long-loading pages, to point to Domino, and some other options.
The proxy_set_header lines are the payoff for the connector headers we set up in Domino. The first is to pass the correct host name on to Domino so it knows which web site document to use, the second is a fairly standard-outside-of-Domino header for reverse proxies, and then the rest are a set of the available WebSphere (hence "$WS") headers, specifying what Domino should see as the remote address, the remote host name (I don't have nginx configured to do reverse DNS lookups, so it's the same value), the host name again, and whether or not it should act as being over SSL.
Once that's set, create symlinks to these files in the sites-enabled directory from the sites-available directory and restart nginx:
# ln -s ../sites-enabled/001-http-redirect # ln -s ../sites-enabled/002-domino-ssl # service nginx restart
Assuming all went well, you should be all set! This gets you a basic one-server proxy setup. The main advantage is the superior SSL handling - nginx's SSL stack is OpenSSL and thus supports all the modern features you'd expect, including SHA-2 certificates and the ability to serve up multiple distinct SSL certificates from the same IP address (this would be done with additional config files using the server_name parameter after listen). Once you have this basis, it's easy to expand into additional features: multiple back-end servers for load balancing and failover, better error messages when Domino crashes (which is more frequent than nginx crashing), and nifty plugins like GeoIP and mod_pagespeed.
Edit 2015-09-16: In my original post, I left out mentioning what to do about those "sites-enabled" documents if you're not running on Ubuntu. There's nothing inherently special about those directories to nginx, so a differently-configured installation may not pick up on documents added there. To make them work in an installation that doesn't initially use this convention, you can add a line like this to the /etc/nginx/nginx.conf (or equivalent) file, at the end of the http code block:
http {
# ... existing stuff here
include /etc/nginx/sites-enabled/*;
}
Ray Bilyk - Thu Sep 18 14:14:27 EDT 2014
Your presentation was great! I just wished I didn't miss the beginning of it. I hope to see it again soon... hopefully in the January timeframe...
Richard Moy - Thu Sep 18 16:51:26 EDT 2014
Great Job Jesse.
Tinus Riyanto - Fri Sep 19 05:43:51 EDT 2014
I am curios to know if it is possible for nginx to authenticate first all request before passing it to Domino. I assume this would involve setting up some sort of landing page on nginx and a way to "pass" authentication to Domino and reading its result ?
Jesse Gallagher - Fri Sep 19 10:31:25 EDT 2014
Oh crap, how did I never think about that? The answer is yes - the $WSRU header lets you specify a username and Domino will honor it. Actually implementing that in a good way goes beyond my current knowledge, but the gist is that you could have some script on the nginx side to set that header value, and then Domino will run as that user. That is... potentially huge. I could replace my DSAPI filter for custom auth and that could be perfect for things like REST API keys.
Jesper Kiaer - Fri Sep 19 10:51:50 EDT 2014
Thanks!
Pere Martinez - Tue Nov 04 11:15:07 EST 2014
Really very good post. Thank you for that. I am having some problems with Domino Web Services and nginx server, and definitely I am going to try this configuration.
Thank you again!
Pere
Ryan Benech - Tue Nov 11 21:55:21 EST 2014
THANK YOU! This is keeping our domino web server out of the POODLE attack vector. Great instructions and the performance is quite a bit better than native domino. THANKS AGAIN!
Andrew - Sat Nov 15 16:45:47 EST 2014
Thanks for the post! Really interesting one! There is also out-of-the-box possibility to use native IBM HTTP Server (which based on Apache HTTP) together with Domino: when selecting the type of Domino install - just choose "Custom" and select IBM HTTP in components selection table. It will install IBM HTTP together with Domino and such deployment is absolutely native and pre-configured for Domino.
Jesse Gallagher - Sat Nov 15 16:49:54 EST 2014
Yes indeed - if you're on Windows, the IBM HTTP Server route is a very good way to go. On balance, there's probably not THAT much less configuration to it, but it's generally better to go with the "native" one when it fits the bill. If the integration ever gets ported to other OSes, it may be worth doing there too (IHS runs on Linux currently, but without the special Domino integration).
On the other side of the coin, going with nginx or "vanilla" Apache has the advantage of being easier to get community support for, outside of the specifics of connecting it to Domino.
Andrew Grabovetsky - Sun Nov 16 04:00:40 EST 2014
Sure, agree!
...re balancing: currently I considering HA (high availability) deployment of Domino (for critical XPages app) and testing / verifying native deployments, covered by IBM support. One of the targets - to get real load balancing, based on app server load, like in native Cluster setup... But this is topic & comment for another blogpost and when I get first results... ;)
Chris C. - Wed Dec 17 14:08:28 EST 2014
How would this work with Notes replication? Do you have to create an nginx config file that handles port 1352?
Jesse Gallagher - Wed Dec 17 14:12:19 EST 2014
For Notes access, you'd still point to the servers directly. You could theoretically do TCP redirection for the port, though I don't think nginx itself handles that. Personally, I just continue to access the servers directly and rely on the normal Notes knowledge of clustermates for failover.
Phil - Tue Dec 23 09:19:24 EST 2014
Great article, works great for GET requests, but I'm getting (chrome):
>>>
SSL connection error
Hide details
Unable to make a secure connection to the server. This may be a problem with the server or it may be requiring a client authentication certificate that you don't have.
Error code: ERR_SSL_PROTOCOL_ERROR
<<<
On POST requests.
Only departure from above setup was in the redirection file, I used rewrite instead of return, return wasn't working in my env. (jaunty, don't ask :o)
Any help greatly appreciated.
Phil.
Phil - Tue Dec 23 14:16:10 EST 2014
Update, POST is working, but a :80 is being appended to the domain for urls sent back from the server ie. when using @URLOpen, so aaa.com/aaa is being turned into aaa.com:80/aaa
Regards.
Phil.
Phil - Thu Dec 25 05:50:35 EST 2014
Turning:
proxy_set_header Host $host;
into
proxy_set_header Host $host:443;
Worked for me :o)
Got the inspiration from http://serverfault.com/questions/351212/nginx-redirects-to-port-8080-when-accessing-url-without-slash
Cheers.
Phil.
Yama - Tue Feb 10 08:00:14 EST 2015
Hi Jesse,
I install Nginx in front of Domino. I use your manual and it works perfect.
But... I use on that Domino (with Nginx proxy) iNotes. And that is a problem. Because we use Connections 4.5 on other server (RHELL + WS) with mail plugin. And this plugin is not working - error with autentication. When been used only Domino HTTP task, working correctly, i think its some problem with SSO, or Authentication cookies.
Please, do you know how to solve it? I try to find any information about this, but unsuccessful. Thanx
Jesse Gallagher - Tue Feb 10 08:42:39 EST 2015
I haven't dealt with an interaction between WebSphere and Domino specifically, but I'd start troubleshooting by looking into the connector headers - presumably, WebSphere still uses the same $-headers used here for passing through port/host/etc. info, including username information. If the WebSphere plugin is pointing to nginx instead of Domino directly, I wouldn't be surprised if nginx strips the incoming $-headers by default (I believe it did when I tested something like that earlier), so you may need a way to intentionally pass those through. I'm not sure off the top of my head how to do that specifically, but that's a path to look down.
Or, if it's possible, ensure that WebSphere is pointed at Domino directly (the most secure setup would be to have Domino's HTTP bound to a local-only network address and have both nginx and WebSphere point to that, but have it unavailable on the larger network).
Yama - Wed Feb 11 04:02:49 EST 2015
Thanx for response.
I try to find something and i see, in firebug, when i login to connections and than to the iNotes, pages have not same LtpaTokens.
Of course, it mean, when I log in to the Connections, and then I connect to the iNotes, and refresh Connections page, web say, I´m logged out. I think, that is problem.
Yama - Mon Feb 16 10:29:51 EST 2015
I find where was a problem.
1. Dont know why, but I must reconfigure tokens.
2. Problem was in ssl support configuration on Nginx, because I had configured only TLS 1 - 1.2 and mail plugin in connections 4.5 (maybe is problem in websphere) support only SSLv3. When I disable ssl support in Nginx conf, everything work OK.
Chris C. - Fri Feb 20 16:55:46 EST 2015
Thanks! This worked great for me. I didn't need to change anything on the Domino end, I just installed and configured Nginx on a Linux VM. I had our firewall continue to forward ports 1352 and 25 to the Domino server while forwarding ports 80 and 443 to the Nginx server.
Also, if you want to disable SSL 3 on the Nginx side, add this line inside server section of the 002-domino-ssl file: ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
Thanks for your help!
Dalie - Tue Aug 18 05:10:07 EDT 2015
Excellent walkthrough!
Are there any known (additional) settings regarding Traveler that you possibly know of? Currently we encounter the "this message has not been downloaded from the server" within the iOS mail/Verse app. The email headers are properly downloaded but the body is missing.
Regarding XPages we only encountered the same issue regarding "paths" when working with e.g. context.redirectToPage(...) but that was quickly resolved using FacesContext.
Again, thanks, any suggestions are welcome!
Jesse Gallagher - Wed Sep 16 08:52:09 EDT 2015
Hmm, I haven't run into the redirectToPage problem, though I can see how that would be an issue. In a simple case, that can presumably be worked around with the "$WSIS" header from nginx: setting that to "True" will cause Domino to act fully like the request is coming in through SSL. That can lead to trouble because it emulates the behavior too well in an environment where you have multiple SSL-enabled host names; Domino ends up falling back to its legacy behavior of only allowing one SSL web site document. In that situation, then having a hard-coded 301 redirect back to SSL will cover it, though that HTTPS -> HTTP -> HTTPS transition is ungainly.
Jason Norton - Tue Dec 01 21:52:07 EST 2015
So I'm trying to test this configuration out and have the following headers in my configuration file but I'm getting the error when I try and restart nginx.
sudo service nginx restart
nginx: [emerg] unknown directive " proxy_set_header $WSRA " in /etc/nginx/conf.d/lamp.conf:11
nginx: configuration file /etc/nginx/nginx.conf test failed
location / {
proxy_pass http://10.0.4.168:80;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header $WSRA $remote_addr;
proxy_set_header $WSRA $remote_addr;
proxy_set_header $WSSN $host;
proxy_set_header $WSIS True;
Any suggestions on why the Web Sphere headers are not working? Everything works until I add these.
Graham - Mon Feb 01 11:12:29 EST 2016
Thank you for the great post! Question: Have you ever done this same task with IBM IHS server acting as a reverse proxy for Domino?
I need to terminate SSL (SHA2) on a IBM HTTP Server reverse proxy as a short term workaround to mitigate the upcoming deprecation on SHA1 SSL Certificates on Domino.
Best Regards,
Graham
Tihomir Pantovic - Mon Nov 14 16:33:36 EST 2016
Do we really need HTTPEnableConnectorHeaders=1?
First tests look good with HTTPEnableConnectorHeaders=0 (or forced $WSRU="").
Alex Elliott - Tue Jun 27 00:25:38 EDT 2017
A great article covering the configuration of NGINX with Domino.
Taking this a step further an article is available on the AGECOM website showing how you can provide HTTP/2 connectivity to your Domino server using NGINX and running it as a service on the Windows platform:
https://www.agecom.com.au/support/agecomkb.nsf/0/464e2235a8e13f86ca258130003b472f?OpenDocument
gamesellru - Sun May 16 16:51:18 EDT 2021
In it something is. I thank for the help in this question, now I will not commit such error.
David Grossi - Thu Mar 17 04:17:46 EDT 2022
Great post.
Unfortunately, when Domino is behind a https proxy, it's not aware of it. It means that some @Macro used in the web navigation - like @DbCommand("Domino"; "ViewNextPage") - won't work because Domino will generate http(not s) requests which will raise a browser "mixed content" security issue, and block the content.
Those naviagtion @Macro are unfortunately inevitable when browsing complex categorized views, because the "next" page is computed server-side, and there is no way to mimic the behavior properly client-side. In short, we have to rely on Domino calculation of what is the "next" page, but the URL returned has an absolute http(not s) path.
If you have any idea about how to solve this (maybe by a proxy config?), I'll be happy to hear about it. Thanks.