Better Living Through Reverse Proxies

Thu May 30 15:59:02 EDT 2013

  1. Putting Apache in Front of Domino
  2. Better Living Through Reverse Proxies
  3. Domino's Server-Side User Security
  4. A Partially-Successful Venture Into Improving Reverse Proxies With Domino
  5. PSA: Reverse-Proxy Regression in Domino 12.0.1

A while ago, I set up Apache as a reverse proxy in front of Domino. Initially, that was in order to serve a WordPress site on the same server, but since then it has more than proven its worth in other ways, mostly in the areas of fault tolerance and SSL support.

IBM recognizes those benefits, which is why Domino 9 on Windows comes bundled with IHS; however, I am a responsible admin, so I don't run Windows on my main servers. Fortunately, stock Apache does the job nicely... or it does usually. Filled with enthusiasm for this kind of setup, I rolled out a small Linode VM dedicated entirely to running Apache as a proxy (sort of like their NodeBalancers, but manual). Unfortunately, I started running into a problem where sometimes sites fronted by it (such as this one) would not properly include their host information and would instead load the main I Know Some Guys site, which is the default on Domino. I wasn't able to find a fix that actually worked, so I decided to use that as an excuse to switch to a cute little number named Nginx.

So far, my experience with Nginx has been fantastic. The config files are like a cleaned-up version of Apache's and it matches it for each feature I've needed (namely, load balancing, easy config, and multiple SSL certificates). As a nice bonus, I didn't have to do any of the config massaging I had to for Apache in order to get XSP's funky resource-aggregation rules to work. If you have the means, I highly recommend it.

 

My latest foray into proxying also gave me an opportunity to look back into my main bugbears with the setup: Domino tracking the proxy server's IP instead of the original requester's and its lack of knowledge of SSL (which causes it to redirect from an SSL login page to a non-SSL one). Fortunately, it turns out that these problems have been sort-of-solved for years via a notes.ini setting added as part of Domino's terminal WebSphere infection: HTTPEnableConnectorHeaders.

By enabling that on my Domino server, I was able to start providing some of those headers. The remote-host headers ("$WSRA" and "$WSRH") work perfectly: setting that to the incoming host causes Domino to act just like that was the original requester, namely filling that in for the REMOTE_HOST field in classic and facesContext.getExternalContext().getRequest().getRemoteAddr() in XPages.

Unfortunately, I was stymied when I set "$WSIS" to True. Though it does indeed cause Domino to acknowledge that the incoming request is via SSL, it does it TOO well: Domino appears to revert to its behavior of only acknowledging a single SSL site, so it caused requests to essentially ignore the Host (and "$WSSN") headers. So that problem remains unsolved.

 

Still, I feel pretty good about my switch to Nginx and my abuse of the HTTP connector headers and look forward to tinkering some more. For reference, here is the config I use for the standard HTTP proxy ("arcturus" is the short name I gave for the main upstream target):

server {
    listen 80;
	
    location / {
        proxy_pass http://arcturus;
        proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
        proxy_redirect off;
        proxy_buffering off;
        proxy_set_header    Host            $host;
        proxy_set_header    X-Real-IP       $remote_addr;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    $WSRA           $remote_addr;
        proxy_set_header    $WSRH           $remote_addr;
        proxy_set_header    $WSSN           $host;
    }
}
Commenter Photo

Bjørn Cintra - Thu May 30 17:36:00 EDT 2013

We have been running a Domino Web-hotel in a hub-spoke topology with a pair of clustered nginx-servers (using keepalived) as reverse proxies in front for a couple of years now. The hub is used as a hot failover if a spoke panics (happens about once a week).

This setup never fails, and has only been down once (and only due to a physical move, and only because we had to move the switch in front).

Nginx in front of Domino just work.

Commenter Photo

Russell Maher - Thu May 30 20:11:15 EDT 2013

Fantastic information.  I had looked at Nginx a while back several times and just didn't have the bandwidth to give it a go against Domino.  Great to know it can be done and how to wokr with it.

Commenter Photo

Christian Glanz - Wed Oct 29 06:09:22 EDT 2014

I am just doing this right now but on Domino 8.5.3FP6.

The problem I just ran into is that logging in domlog.nsf does not log the actual client IP address but 'localhost' (since my Nginx is running on the same machine as my Domino). The proxy header fields are set and transmitted the way you describe.

Will I need to upgrade to Domino 9.x to get domlog.nsf logging the client IPs by reading them from the request header?

Commenter Photo

Jesse Gallagher - Wed Oct 29 06:19:22 EDT 2014

No need to upgrade (well, you really should, but don't have to for this): that is where HTTPEnableConnectorHeaders comes into play, which should work back to R6. If you enable that ini setting and pass in the headers like above, Domino will use the external address in all situations.

Commenter Photo

Christian Glanz - Wed Oct 29 08:19:21 EDT 2014

Thank you for answering so quickly!

I just tested all of this (headers in the request and the HTTPEnableConnectorHeaders=1 in notes.ini after a Domino restart) on a Domino 9 using Telnet but also it does not seem to work there.
Client IPs in domlog.nsf are logged as 'localhost' because I was using Telnet on the same machine. The request headers seem to have been discarded.
Also 'localhost' is logged in the text log files, instead of the IP I gave in the request headers.

What else could be wrong? Is it really possible to get the client IP given in the request headers by Nginx reverse proxy logged into a document in domlog.nsf? Has somebody done exactly that?

Commenter Photo

Christian Glanz - Thu Oct 30 09:14:54 EDT 2014

Addition to my last post: I also tested log-in via browser with false credentials on my 8.5.3FP6 machine and while doing so I was able to spot my real client's IP address in the Domino Console in Domino Administrator.
Nevertheless in domlog.nsf this only appeared with 'localhost', not my real client's IP.

That is why I am sure the request headers DO work (and my settings are correct) but they DO NOT appear in the web log.

Commenter Photo

Sam Sirry - Fri Feb 03 01:15:24 EST 2017

How come no one mentioned that it's impossible to use the dollar sign in this way in the nginx config file?

New Comment