XPages Renderers and Thread Safety

May 27, 2021, 9:56 AM

Tags: xpages

I got to thinking yesterday about renderers in XPages - the part of the stack that takes the abstract back-end representation and actually turns it into HTML. They've been an interest of mine for a good while and they have interesting characteristics worth revisiting from time to time.

One of those characteristics is their general statelessness: the way they work, there's generally only one instance of the renderer object, and then it's applied to many pages. This is reflected in the Javadoc for the base javax.faces.render.Renderer class (Java EE 5 here because that's what XPages is based on):

Individual `Renderer` instances will be instantiated as requested during the rendering process, and will remain in existence for the remainder of the lifetime of a web application. Because each instance may be invoked from more than one request processing thread simultaneously, they MUST be programmed in a thread-safe manner.

As a quick aside, that all-caps "MUST" is an application of the delightful RFC 2119, which defines common meanings for those types of words in specs. It's worth reading.

Implementation

Anyway, how does this shake out in practice? Well, we can use the Bootstrap theme as provided by IBM back when it was open source as a baseline example of how it's done. If you look at the, for example, ResponsiveAppLayoutRenderer class, you can see a ton of code, but no instance variables. Looking at any number of classes, there are static constants, but no instance variables. In general, that bundle as present on GitHub there is a great example of the craft.

In general, thread safety is tricky, and the easiest way to make a class thread-safe is to not have any instance variables. These renderer examples take that route, and for good reason: since renderers are instantiated once per app, they're potentially shared across all pages in the app, multiple components within a page, and multiple instances of the same page.

Demonstration

Thread safety in general and in Java in particular is a huge topic, and it's something that harries basically all programmers working in a potentially-threaded environment, even when the code you're writing doesn't seem troublesome. It's one thing if you're explicitly writing multithreaded code, divvying up tasks among executors or something, but what would be the trouble here? Well, fortunately, writing a demonstration is fairly quick. To do so, I made a new NSF with a few design elements. First, a theme:

1
2
3
4
5
6
7
8
9
<theme>
    <control>
        <name>ViewRoot</name>
        <property>
            <name>rendererType</name>
            <value>renderer.TestViewRoot</value>
        </property>
    </control>
</theme>

You can name the theme whatever you want, since the name of a theme is intended to be of human interest only. It will only matter that it's then selected in the Xsp Properties file.

Then, a customized faces-config.xml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
<?xml version="1.0" encoding="UTF-8"?>
<faces-config>
  <render-kit>
    <renderer>
      <component-family>javax.faces.ViewRoot</component-family>
      <renderer-type>renderer.TestViewRoot</renderer-type>
      <renderer-class>renderer.TestViewRoot</renderer-class>
    </renderer>
  </render-kit>
  <!--AUTOGEN-START-BUILDER: Automatically generated by HCL Domino Designer. Do not modify.-->
  <!--AUTOGEN-END-BUILDER: End of automatically generated section-->
</faces-config>

This has prepared us to use a custom renderer for the view root, which is the main page itself. Finally, as the last step before the renderer class, I made five XPages, named "home1.xsp" through "home5.xsp". The content is irrelevant, so I made them the simplest possible:

1
2
<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core"/>

Now, to the renderer class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
package renderer;

import javax.faces.component.UIComponent;
import javax.faces.context.FacesContext;
import javax.faces.render.Renderer;

import com.ibm.xsp.component.UIViewRootEx;

public class TestViewRoot extends Renderer {
    String viewId;
    
    @Override
    public void encodeBegin(FacesContext context, UIComponent component) {
        UIViewRootEx viewRoot = (UIViewRootEx)context.getViewRoot();
        viewId = viewRoot.getViewId();
    }
    @Override
    public void encodeEnd(FacesContext context, UIComponent component) {
        UIViewRootEx viewRoot = (UIViewRootEx)context.getViewRoot();
        String endingViewId = viewRoot.getViewId();
        if(!this.viewId.equals(endingViewId)) {
            System.out.println("finished rendering view ID "+ endingViewId + " - started with " + this.viewId);
        }
    }
}

This class gets the view ID (e.g. "/home1") in encodeBegin and again in encodeEnd. When the two are different - essentially, our bug condition - it emits a message to the console. This "viewId" is a stand-in for any sort of expected shared state, to demonstrate that the renderer can make no assumptions that encodeBegin and encodeEnd are called in sequence for the same page or page instance. The latter could be demonstrated by checking viewRoot.getUniqueViewId() instead, which identifies the specific page instance and is thus distinct even across different users on the same page.

Then, I wrote a script that, in a multithreaded fashion, requests home1.xsp through home5.xsp randomly for a little while, and it was only a couple seconds before the messages started appearing:

1
2
3
4
5
[0B4C:000A-14A0] 05/27/2021 09:25:21 AM  HTTP JVM: finished rendering view ID /home4 - started with /home2
[0B4C:0023-1388] 05/27/2021 09:25:21 AM  HTTP JVM: finished rendering view ID /home2 - started with /home1
[0B4C:001E-0C88] 05/27/2021 09:25:21 AM  HTTP JVM: finished rendering view ID /home2 - started with /home1
[0B4C:001B-1784] 05/27/2021 09:25:21 AM  HTTP JVM: finished rendering view ID /home3 - started with /home1
[0B4C:001C-1AE8] 05/27/2021 09:25:22 AM  HTTP JVM: finished rendering view ID /home2 - started with /home5

Knock-On Demo

Okay, so that's bad, but there's a subtle other problem I created here and that's to do with code reuse via subclassing. In general, renderers are subclassed out the wazoo. Just look at the type hierarchy for FacesRendererEx:

FacesRendererEx Type Hierarchy

It goes on for a while like that.

While renderers being subclass-friendly isn't a "MUST"-type rule like thread safety, it's both an associated benefit of that and a general cultural idiom. But imagine if I were to subclass my example above and override just the encodeBegin portion (to represent, say, changing just the output of the page header but leaving the footer the same):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
package renderer;

import javax.faces.component.UIComponent;
import javax.faces.context.FacesContext;

public class TestViewRootEx extends TestViewRoot {
    @Override
    public void encodeBegin(FacesContext context, UIComponent component) {
        // Do something else here
    }
}

If I use that as the renderer, I get a dramatically-new look for my page:

NPE on subclass

The trouble here is that I overrode encodeBegin but didn't call super.encodeBegin - which I naturally wouldn't, since I explicitly don't want whatever the parent class is outputting. However, since encodeEnd assumes that its version of encodeBegin runs and sets this.viewId, I hit a NullPointerException when it tries to access it. Curses.

Conclusion

Anyway, this is all a long-winded way of pointing out that designing for thread safety is tricky business, and it can crop up when you wouldn't otherwise expect it. It's good here that the JSF developers included that "MUST" business in the Javadoc, but, unfortunately, Java-the-language doesn't have a way of actually enforcing it, making room for this sort of thing to creep in.

My Tortured Relationship With libnotes

May 22, 2021, 12:31 PM

Tags: c java

A tremendous amount of my work lately involves wrangling the core Notes library, variously "libnotes.dylib", "libnotes.so", or (for some reason) "nnotes.dll" (edit: see the comments!). I do almost all of my daily work in Liberty servers on the Mac loading this library, my first major use of the Domino Docker image was to use it as an overgrown carrier for this native piece, and in general I spend a lot of time putting this to use.

It ain't easy, though! Unlike a lot of native libraries that you can just kind of load from wherever, libnotes is extremely picky about its loading environment. There are a few main ways that this manifests.

Required-Library References

libnotes isn't standalone, and it doesn't just rely on standard OS stuff like libc. It also relies on other Notes-specific libraries like libxmlproc and libgsk8iccs, and some of those refer back to each other. This all shakes out in normal practice, since they're all next to the Notes/Domino executable and each other, but it makes things finicky when you're running from outside there.

This seems to be the most finicky on macOS, where the references to each library are marked with @executable_path relative paths, meaning relative to the running executable.

I've wrangled with this quite a bit over the years, but the upshot is that, on macOS and Linux, you really want to set some environment variables before you ever load your program. Naturally, since your running program can't do anything about what happened before it was loaded, this means you have to balance the teacups in your environment beforehand.

Non-Library Files

Beyond the dynamic libraries, libnotes also need some program-support executable files, things like string resource files and whatnot. And, very importantly, it needs an active data directory with an ID, notes.ini, and names.nsf at least. The data-directory contents bits are at least a little more forgiving: though there's still a hard requirement that they be present on the filesystem (since libnotes loads them by string path, not as configurable binary streams), you could at least bundle them or copy them around. For example, for my Dockerized app runners, I tend to have some basic versions of those in the repo that get copied into the Docker container's notesdata directory during build.

Working Around It

As I mentioned, the main way I go about dealing with this is by telling whatever is running my program to use applicable environment variables, ideally in a reproducible config-file-based way. This works perfectly with Docker and with tycho-surefire-plugin in Maven, but doesn't work so well for maven-surefire-plugin (the normal unit test runner) or Eclipse's JUnit tools. In Eclipse's case, I can at least fill in the environment variables in the Run Configuration. It hampers me a bit (I can't just right-click a test case and run it individually without first running the whole suite or setting up a configuration specially), but it works.

I gave a shot recently to copying the Mac dylibs and programmatically fiddling with otool and install_name_tool to adjust their dependency paths, and I got somewhere with that, but that somewhere was an in-libnotes fatal panic, so I'm clearly missing something. And besides, even if I got that working, it'd be a bit of a drag.

What Would Be Nice

What would be really nice would be a variant of this that's just more portable - something where I can just System.load a library, call NotesInitExtended to point to my INI and ID, and be good to go. I'm not really sure what this would entail, especially since I'd also want libinotes, libjnotes, and liblsxbe. I do know that I don't have the tools to do it, which fortunately frees me up to idly speculate about things I'd like to have delivered to me.

As long as I'm wishing for stuff, I'll say what would be even cooler would be a WebAssembly library usable with Wasmer, which is a multi-language WebAssembly runtime. The promise there is that you'd have a single compiled library that would run on any OS on any processor that supports WebAssembly, from now until forever. I'm not sure that this would actually be doable at the moment - for one, I don't know if callback parameters work, which would be fairly critical. Still, my birthday is coming up, and that would be a nice present.

OpenNTF May 2021 Webinar Followup

May 20, 2021, 1:33 PM

Tags: openntf

Earlier today, I took part in OpenNTF's May webinar on recent project updates. During that, I gave a quick overview of several of the projects I've worked on in the past year, and I figured it'd be useful to put together some followup notes for future reference.

Project Links

I didn't include any actual links to the projects, which could be useful:

Slides

To begin with, I posted my slides on SlideShare:

OpenNTF Webinar May 2021 - Jesse from Jesse Gallagher

NSF ODP Tooling

I gave a brief overview of the NSF ODP Tooling, but, if anyone is interested in a more in-depth overview, my presentation from CollabSphere last year should cover it:

Import and Export for Designer

My quick mention of this refreshed project focused mainly on importing components, but generating new ones for upload is a pretty-critical part of it too. The tool comes with an option for that, which (to the credit of the original creators) does a splendid job packaging your controls and applying a license. The PDF from the project page goes over that in detail. Once the components are packaged up, you can upload them as a release on OpenNTF and I'll figure out how to add it to the download list.

More Notes on Filesystem and Charset Portability

May 18, 2021, 3:39 PM

Tags: java
  1. Java Hiccups
  2. Bitwise Operators
  3. Java Grab Bag 2
  4. Java Travelogue: The Care and Feeding of Locales
  5. More Notes on Filesystem and Charset Portability

A few months back, I talked about some localization troubles in the NSF ODP Tooling and how it's important to be explicit in your handling of this sort of thing to make sure your code will work in an environment that isn't specifically "Linux or macOS in an en-US environment".

Well, after making a bunch of little tweaks over the last few days, I have two additional tips in this arena! Specifically, my foes this round came from three sources: Windows, my use of a ZIP file filesystem, and the old reliable charset.

Path Separators

The first bit of trouble had to do with how those two things interact. For a long time, I've been in the (commonly-held) habit of using File.separator and File.separatorChar to get the default path separator for the system - that is, \ on Windows and / on most other platforms. Those work well enough - no real trouble there.

However, my problem came from using the Java NIO ZIP filesystem on Windows. Take this bit of code:

1
2
3
4
5
6
7
public static String toJavaClassName(Path path) {
	String name = path.toString();
	if(name.endsWith(".java")) {
		return name.substring(0, name.length()-".java".length()).replace(File.separatorChar, '.');
	}
	/* Other conditions here */
}	

When Path is a path on the local filesystem, that works just fine, taking a path like "com/example/Foo.java" and turning it into "com.example.Foo". It also works splendidly on macOS and Linux in all cases, the two systems I actually use. However, when path represents a path within a ZIP file and you're working on Windows, it fails, returning a "class name" like "com/example/Foo".

This is exactly what happens when compiling an ODP using a remote Domino server running on Windows. For the portability reasons mentioned in my previous post, the client sends a ZIP of the ODP to the server and then the compilation pulls directly out of that ZIP instead of writing it out to the filesystem. The way the ZIP filesystem driver in Java is written, it uses / for its path separator on all platforms, which is consistent with dealing with ZIP files generally. But, when mixed with the native filesystem separator, that line resolved to:

1
return "com/example/Foo".replace('\\', '.');

...and there's the problem. The fix is to change the code to instead get the directory separator from the contextual filesystem in question:

1
2
3
4
5
6
7
public static String toJavaClassName(Path path) {
	String name = path.toString();
	if(name.endsWith(".java")) {
		return name.substring(0, name.length()-".java".length()).replace(path.getFileSystem().getSeparator(), ".");
	}
	/* Other conditions here */
}

A little more verbose, sure, but it has the advantage of functioning consistently in all environments.

This also has significant implications if you use static properties to store filesystem-dependent elements. This came into play in my OnDiskProject class, which contains a bunch of path matchers to find design elements to import from the ODP. Originally, I kept these in a static property that was generated by writing them Unix-style, then running them through a generator to use the platform-native separator character. This had to change, since the actual ODP store may or may not be the platform-native filesystem. This sort of thing is pervasive, and it'll take me a bit to get over my long-standing habit.

Over-Interpreting Character Sets

This one is similar to the charset troubles in my previous post, but ran into subtle trouble in the ODP compiler. Here was the sequence of events:

  1. The ODP Compilers reads the XSP source of a page or custom control using ODPUtil, which read in the string as UTF-8
  2. It then passes that string to the Bazaar's DynamicXPageBean
  3. That method uses StringReader and an IBM Commons ReaderInputStream to read the content
  4. That content is then read in by FacesReader, which uses the default DOM parser to read the XML

In general, that flow worked just fine. However, that's because, in general, I write US-ASCII markup. However, when the page contains, say, Czech diacritics, this goes off the rails. Somewhere in the interpretation and re-interpretation of the file, the UTF-8-iness of it breaks.

Fortunately, this one was a clean one: XML has its own mechanism for declaring its encoding (and it's almost always UTF-8 anyway), so my code doesn't actually need to be responsible for interpreting the bytes of the file before it gets to the DOM parser. So I added a version of the Bazaar method that takes an InputStream directly and modified NSF ODP to use it, with no extra interpretation in between.

Tinkering With Cross-Container Domino Addins

May 16, 2021, 1:35 PM

Tags: docker domino

A good chunk of my work lately involves running distinct processes with a Domino runtime, either run from Domino or standalone for development or CI use. Something that had been percolating in the back of my mind was another step in this: running these "addin-ish" programs in Docker in a separate container from Domino, but participating in that active Domino runtime.

Domino addins in general are really just separate processes and, while they gain some special properties when run via load foo on the console or OSLoadProgram in the C API, that's not a hard requirement to getting a lot of things working.

I figured I could get this working and, armed with basically no knowledge about how this would work, I set out to try it.

Scaffolding

My working project at hand is a webapp run with the standard open-liberty Docker images. Though I'm using that as a starting point, I had to bring in the Notes runtime. Whether you use the official Domino Docker images from Flexnet or build your own, the only true requirement is that it match the version used in the running server, since libnotes does a version check on init. My Dockerfile looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
FROM --platform=linux/amd64 open-liberty:beta

USER root
RUN useradd -u 1000 notes

RUN chown -R notes /opt/ol
RUN chown -R notes /logs

# Bring in the Domino runtime
COPY --from=domino-docker:V1200_03252021prod /opt/hcl/domino/notes/latest/linux /opt/hcl/domino/notes/latest/linux
COPY --from=domino-docker:V1200_03252021prod /local/notesdata /local/notesdata

# Bring in the Liberty app and configuration
COPY --chown=notes:users /target/jnx-example-webapp.war /apps/
COPY --chown=notes:users config/* /config/
COPY --chown=notes:users exec.sh /opt/
RUN chmod +x /opt/exec.sh

USER notes

ENV LD_LIBRARY_PATH "/opt/hcl/domino/notes/latest/linux"
ENV NotesINI "/local/notesdata/notes.ini"
ENV Notes_ExecDirectory "/opt/hcl/domino/notes/latest/linux"
ENV Directory "/local/notesdata"
ENV PATH="${PATH}:/opt/hcl/domino/notes/latest/linux:/opt/hcl/domino/notes/latest/linux/res/C"

EXPOSE 8080 8443

ENTRYPOINT ["/opt/exec.sh"]

I'll get to the "exec.sh" business later, but the pertinent parts now are:

  • Adding a notes user (to avoid permissions trouble with the data dir, if it comes up)
  • Tweaking the Liberty container's ownership to account for this
  • Bringing in the Domino runtime
  • Copying in my WAR file from the project and associated config files (common for Liberty containers)
  • Setting environment variables to tell the app how to init

So far, that's largely the same as how I run standalone Notes-runtime-enabled apps that don't talk to Domino. The only main difference is that, instead of copying in an ID and notes.ini, I instead mount the data volume to this container as I do with the main Domino one.

Shared Memory

The big new hurdle here is getting the separate apps to participate in Domino's shared memory pool. Now, going in, I had a very vague notion of what shared memory is and an even vaguer one of how it works. Certainly, the name is straightforward, and I know it in Domino's case mostly as "the thing that stops Notes from launching after a crash sometimes", but I'd need to figure out some more to get this working. Is it entirely a filesystem thing, as the Notes problem implies? Is it an OS-level thing with true memory? Well, both, apparently.

Fortunately, Docker has this covered: the --ipc flag for docker run. It has two main modes: you can participate in the host's IPC pool (essentially like what a normal, non-contained process does) or join another container specifically. I opted for the latter, which involved changing both the Domino launch arguments.

For Domino, I added --ipc=shareable to the argument list, basically registering it as an available host for other containers to glom on to.

For the separate app, I added --ipc=container:domino, where "domino" is the name of the Domino container.

With those in place, the "addin" process was able to see Domino and do addin-type stuff, like adding a status line and calling AddinLogMessageText to display a message on the server's console.

Great: this proved that it's possible. However, there were still a few show-stopping problems to overcome.

PIDs

From what I gather, Notes keeps track of processes sharing its memory by their reported process IDs. If you have a process that joins the pool and then exits (maybe only if it exits abruptly; I'm not sure) and then tries to rejoin with the same PID, it will fail on init with a complaint that the PID is already registered.

Normally, this isn't a problem, as the OS hands out distinct PIDs all the time. This is trouble with Docker, though: by default, in general, the direct process in a Docker container sees itself as PID 1, and will start as such each time. In the Domino container, its PID 1 is "start.sh", and that's still going, and it's not going to hear otherwise from some other process calling itself the same.

Fortunately, this was a quick fix: Docker's -pid option. Though the documentation for this is uncharacteristically slight, it turns out that the syntax for my needs is the same as the last option. Thus: --pid=container:domino. Once I set that, the running app got a distinct PID from the pool. That was pleasantly simple.

SIGTERM

And now we come to the toughest problem. As it turns out, dealing with SIGTERM - the signal sent by docker stop - is a whole big deal in the Java world. I banged my head at this for a while, with most of the posts I've found being not quite applicable, not working at all for me, or technically working but only in an unsustainable way.

For whatever reason, the Open Liberty Docker image doesn't handle this terribly well - when given a SIGTERM order, it doesn't stop the servlet context before dying, which means the contextDestroyed method in my ServletContextListener (such as this one) doesn't fire.

In many webapp cases, this is fine, but Domino is extremely finicky when it comes to memory-sharing processes needing to exit cleanly. If a process calls NotesInit but doesn't properly call NotesTerm (and close all its Notes-enabled threads), the server panics and dies. This is... not great behavior, but it is what it is, and I needed to figure out how to work with it. Unfortunately, the Liberty Docker container wasn't doing me any favors.

One option is to use Runtime.getRuntime().addShutdownHook(...). This lets you specify a Thread to execute when a SIGTERM is received, and it can work in some cases. It's a little shaky sometimes, though, and it's bad form to riddle otherwise-normal webapps with such things: ideally, even webapps that you intend to run in a container should be written such that they can participate in a normal multi-app environment.

What I ended up settling on was based on this blog post, which (like a number of others) uses a shell script as the main entrypoint. That's a common idiom in general, and Open Liberty's image does it, but its script doesn't account for this, apparently. I tweaked that post's shell script to use the Liberty start/stop commands and ended up with this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
#!/usr/bin/env bash
set -x

term_handler() {
  /opt/ol/helpers/runtime/docker-server.sh /opt/ol/wlp/bin/server stop
  exit 143; # 128 + 15 -- SIGTERM
}

trap 'kill ${!}; term_handler' SIGTERM

/opt/ol/helpers/runtime/docker-server.sh /opt/ol/wlp/bin/server start defaultServer

# echo the Liberty console
tail -f /logs/console.log &

while true
do
  tail -f /dev/null & wait ${!}
done

Now, when I issue a docker stop to the container, the script issues an orderly shutdown of the Liberty instance, which properly calls the contextDestroyed method and allows my code to close down its ExecutorService and call NotesTerm. Better still, Domino keeps running without crashing!

Conclusion

My final docker run scripts ended up being:

Domino

1
2
3
4
5
6
7
8
9
docker run --name domino \
	-d \
	-p 1352:1352 \
	-v notesdata:/local/notesdata \
	-v notesmisc:/local/notesmisc \
	--cap-add=SYS_PTRACE \
	--ipc=shareable \
	--restart=always \
	iksg-domino-12beta3

Webapp

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
docker build . -t example-webapp
docker run --name example-webapp \
	-it \
	--rm \
	-p 8080:8080 \
	-v notesdata:/local/notesdata \
	-v notesmisc:/local/notesmisc \
	--ipc=container:domino \
	--pid=container:domino \
	example-webapp

(Here, the webapp is run to be temporary and tied to the console, hence -it, --rm, and no -d)

One nice thing to note is that there's nothing webapp- or Java-specific here. One of the nice things about Docker is that it removes a lot of the hurdles to running whatever-the-heck type of program you want, so long as there's a Linux Docker image for it. I just happen to default to Java webapps for basically everything nowadays. The above script could be tweaked to work with most anything: the original post had it working with a Node app.

Now, considering that I was starting from nearly scratch here, I certainly can't say whether this is a bulletproof setup or even a reasonable idea in general. Still, it seems to work, and that's good enough for me for now.

Replicating Domino to Azure Data Lake With Darwino

May 10, 2021, 10:06 AM

Tags: darwino

Though Darwino is an app-dev platform, one of the main ongoing uses it's had has been for reporting on Domino data. By default, when replicating with Domino, Darwino uses its table layout in its backing SQL database, making use of the various vendors' native JSON capabilities to have storage capabilities analogous to Domino. Once it's there, even if you don't actually build any apps on top of it, it's immediately useful for querying at scale with various reporting tools: BIRT, Power BI, Crystal Reports, what have you. Add in some views and, as needed, extra indexes and you have an extraordinarily-speedy way to report on the data.

Generic Replication

But one of the neat other uses comes in with that "by default" above. Darwino's replication engine is designed to be thoroughly generic, and that's how it replicates with Domino at all: since an adapter only needs to implement a handful of classes to expose the data in a Darwino-friendly way, the source or target's actual storage mechanism doesn't matter overmuch. Darwino's own storage is just "first among equals" as far as replication is concerned, and the protocol it uses to replicate with Domino is the same as it uses to replicate among multiple Darwino app servers.

From time to time, we get a call to make use of this adaptability to target a different backend. In this case, a customer wanted to be able to push data to Azure Data Lake, which is a large-scale filesystem-like storage service Microsoft offers. The idea is that you get your gobs of data into Data Lake one way or another, and then they have a suite of tools to let you report on and analyze it to your heart's content. It's the sort of thing that lets businesspeople make charts to point to during meetings, so that's nice.

This customer had already been using Azure's SQL Server services for "normal" Darwino replication from Domino, but wanted to avoid doing something like having a script to transform the SQL data into Data-Lake-friendly formats after the fact. So that's where the custom adapter came in.

The Implementation

Since the requirement here is just going one way from Domino to Data Lake, that took a bit of the work off our plate. It wouldn't be particularly conceptually onerous to write a mechanism to go the other way - mostly, it'd be finding an efficient way to identify changed documents - but the "loose" filesystem concept of Data Lake would make identifying changes and dealing with arbitrary user-modified data weird.

The only real requirements for a Darwino replication target are that you can represent the data in JSON in some way and that you are able to identify changes for delta replication. That latter one is technically a soft requirement, since an adapter could in theory re-replicate the entire thing every time, but it's worlds better to be able to do continuous small replications rather than nightly or weekly data dumps. In Darwino's own storage, this is handled by indexed columns to find changes, while with Domino it uses normal NSFSearch-type capabilities to find modifications since a specific date.

Data Lake is a little "metadata light" in this way, since it represents itself primarily as a filesystem, but the lack of need to replicate changes back meant I didn't have to worry about searching around for an efficient call. I settled on a basic layout:

Data Lake layout

Within the directory for NSFs, there are a few entities:

  • darwino.json keeps track of the last time the target was replicated to, so I can pick up on that for delta replication in the future
  • docs houses the documents themselves, named like "(unid).json" and containing the converted JSON content of the Domino documents
  • attachments has subfolders named for the UNIDs of the documents referenced, followed by the attachment and embedded images, names prefixed with the fields they're from

Back on the Domino side, I can set this up in the Sync Admin database the same way I do for traditional Darwino targets, where the Data Lake extension is picked up:

Data Lake in Sync Admin

Once it's set up, I can turn on the replication schedule and let it do its thing in the background and Data Lake will stay in step with the NSFs.

Conclusion

Now, immediately, this is really of interest just to our client who wanted to do it, but I felt like it was a neat-enough trick to warrant an overview. It's also satisfying seeing the layers working together: the side that reads the Domino data needed no changes to work with this entirely-new target, and similarly the core replication engine needed no tweaks even though it's pointing at a fully-custom destination.

Implementing Custom Token-Based Auth on Liberty With Domino

Apr 24, 2021, 12:31 PM

This weekend, I decided to embark on a small personal side project: implementing an RSS sync server I can use with NetNewsWire. It's the delightful sort of side project where the stakes are low and so I feel no pressure to actually complete it (I already have what I want with iCloud-based syncing), but it's a great learning exercise.

Fair warning: this post is essentially a travelogue of not-currently-public code for an incomplete side app of mine, and not necessarily useful as a tutorial. I may make a proper example project out of these ideas one day, but for the moment I'm just excited about how smoothly this process has gone.

The Idea

NetNewsWire syncs with a number of services, and one of them is FreshRSS, a self-hosted sync tool that uses PHP backed by an RDBMS. The implementation doesn't matter, though: what matters is that that means that NNW has the ability to point at any server at an arbitrary URL implementing the same protocol.

As for the protocol itself, it turns out it's just the old Google Reader protocol. Like Rome, Reader rose, transformed the entire RSS ecosystem, and then crumbled, leaving its monuments across the landscape like scars. Many RSS sync services have stuck with that language ever since - it's a bit gangly, but it does the job fine, and it lowers the implementation toll on the clients.

So I figured I could find some adequate documentation and make a little webapp implementing it.

Authentication

My starting point (and all I've done so far) was to get authentication working. These servers mimic the (I assume antiquated) Google ClientLogin endpoint, where you POST "Email" and "Passwd" and get back a token in a weird little properties-ish format:

1
2
3
4
POST /accounts/ClientLogin HTTP/1.1
Content-Type: application/x-www-form-urlencoded

Email=ffooson&Passwd=secretpassword

Followed by:

1
2
3
4
5
6
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8

SID=null
LSID=null
Auth=somename/8e6845e089457af25303abc6f53356eb60bdb5f8

The format of the "Auth" token doesn't matter, I gather. I originally saw it in that "name/token" pattern, but other cases are just a token. That makes sense, since there's no need for the client to parse it - it just needs to send it back. In practice, it shouldn't have any "=" in it, since NNW parses the format expecting only one "=", but otherwise it should be up to you. Specifically, it will send it along in future requests as the Authorization header:

1
2
GET /reader/api/0/stream/items/ids?n=1000&output=json&s=user/-/state/com.google/starred HTTP/1.1
Authorization: GoogleLogin auth=somename/8e6845e089457af25303abc6f53356eb60bdb5f8

This is pretty standard stuff for any number of authentication schemes: often it'll start with "Bearer" instead of "GoogleLogin", but the idea is the same.

Implementing This

So how would one go about implementing this? Well, fortunately, the Jakarta EE spec includes a Security API that allows you to abstract the specifics of how the container authenticates a user, providing custom user identity stores and authentication mechanisms instead of or in addition to the ones provided by the container itself. This is as distinct from a container like Domino, where the HTTP stack handles authentication for all apps, and the only way to extend how that works is by writing a native library with the C-based DSAPI. Possible, but cumbersome.

Identity Store

We'll start with the identity store. Often, a container will be configured with its own concept of what the pool of users is and how they can be authenticated. On Domino, that's generally the names.nsf plus anything configured in a Directory Assistance database. On Liberty or another JEE container, that might be a static user list, an LDAP server, or any number of other options. With the Security API, you can implement your own. I've been ferrying around classes that look like this for a couple of years now:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
/* snip */

import javax.security.enterprise.credential.Credential;
import javax.security.enterprise.credential.UsernamePasswordCredential;
import javax.security.enterprise.identitystore.CredentialValidationResult;
import javax.security.enterprise.identitystore.IdentityStore;

@ApplicationScoped
public class NotesDirectoryIdentityStore implements IdentityStore {
    @Inject AppConfig appConfig;

    @Override public int priority() { return 70; }
    @Override public Set<ValidationType> validationTypes() { return DEFAULT_VALIDATION_TYPES; }

    public CredentialValidationResult validate(UsernamePasswordCredential credential) {
        try {
            try(DominoClient client = DominoClientBuilder.newDominoClient().build()) {
                String dn = client.validateCredentials(appConfig.getAuthServer(), credential.getCaller(), credential.getPasswordAsString());
                return new CredentialValidationResult(null, dn, dn, dn, getGroups(dn));
            }
        } catch (NameNotFoundException e) {
            return CredentialValidationResult.NOT_VALIDATED_RESULT;
        } catch (AuthenticationException | AuthenticationNotSupportedException e) {
            return CredentialValidationResult.INVALID_RESULT;
        }
    }

    @Override
    public Set<String> getCallerGroups(CredentialValidationResult validationResult) {
        String dn = validationResult.getCallerDn();
        return getGroups(dn);
    }

    /* snip */
}

There's a lot going on here. To start with, the Security API goes hand-in-hand with CDI. That @ApplicationScoped annotation on the class means that this IdentityStore is an app-wide bean - Liberty picks up on that and registers it as a provider for authentication. The AppConfig is another CDI bean, this one housing the Domino server I want to authenticate against if not the local runtime (handy for development).

The IdentityStore interface definition does a little magic for identifying how to authenticate. The way it works is that the system uses objects that implement Credential, an extremely-generic interface to represent any sort of credential. When the default implementation is called, it looks through your implementation class for any methods that can handle the specific credential class that came in. You can see above that validate(UsernamePasswordCredential credential) isn't tagged with @Override - that's because it's not implementing an existing method. Instead, the core validate looks for other methods named validate to take the incoming class. UsernamePasswordCredential is one of the few stock ones that comes with the API and is how the container will likely ask for authentication if using e.g. HTTP Basic auth.

Here, I use some Domino API to check the username+password combination against the Domino directory and inform the caller whether the credentials match and, if so, what the user's distinguished name and group memberships are (with some implementation removed for clarity).

Token Authentication

That's all well and good, and will allow a user to log into the app with HTTP Basic authentication with a Domino username and password, but I'd also like the aforementioned GoogleLogin tokens to count as "real" users in the system.

To start doing that, I created a JAX-RS resource for the expected login URL:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
@Path("accounts")
public class AccountsResource {
    @Inject TokenBean tokens;
    @Inject IdentityStore identityStore;

    @PermitAll
    @Path("ClientLogin")
    @POST
    @Consumes(MediaType.APPLICATION_FORM_URLENCODED)
    @Produces(MediaType.TEXT_HTML)
    public String post(@FormParam("Email") @NotEmpty String email, @FormParam("Passwd") String password) {
        CredentialValidationResult result = identityStore.validate(new UsernamePasswordCredential(email, password));
        switch(result.getStatus()) {
        case VALID:
            Token token = tokens.createToken(result.getCallerDn());
            String mangledDn = result.getCallerDn().replace('=', '_').replace('/', '_');
            return MessageFormat.format("SID=null\nLSID=null\nAuth={0}\n", mangledDn + "/" + token.token()); //$NON-NLS-1$ //$NON-NLS-2$
        default:
            // TODO find a better exception
            throw new RuntimeException("Invalid credentials");
        }
    }

}

Here, I make use of the IdentityStore implementation above to check the incoming username/password pair. Since I can @Inject it based on just the interface, the fact that it's authenticating against Domino isn't relevant, and this class can remain blissfully unaware of the actual user directory. All it needs to know is whether the credentials are good. In any event, if they are, it returns the weird little format in the response and the RSS client can then use it in the future.

The TokenBean class there is another custom CDI bean, and its job is to create and look up tokens in the storage NSF. The pertinent part is:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
@ApplicationScoped
public class TokenBean {
    @Inject @AdminUser
    Database adminDatabase;

    public Token createToken(String userName) {
        Token token = new Token(UUID.randomUUID().toString().replace("-", ""), userName); //$NON-NLS-1$ //$NON-NLS-2$
        adminDatabase.createDocument()
            .replaceItemValue("Form", "Token") //$NON-NLS-1$ //$NON-NLS-2$
            .replaceItemValue("Token", token.token()) //$NON-NLS-1$
            .replaceItemValue("User", token.user()) //$NON-NLS-1$
            .save();
        return token;
    }

    /* snip */
}

Nothing too special there: it just creates a random token string value and saves it in a document. The token could be anything; I could have easily gone with the document's UNID, since it's basically the same sort of value.

I'll save the @Inject @AdminUser bit for another day, since we're already far enough into the CDI weeds here. Suffice it to say, it injects a Database object for the backing data DB for the designated admin user - basically, like opening the current DB with sessionAsSigner in XPages. The @AdminUser is a custom annotation in the app to convey this meaning.

Okay, so great, now we have a way for a client to log in with a username and password and get a token to then use in the future. That leaves the next step: having the app accept the token as an equivalent authentication for the user.

Intercepting the incoming request and analyzing the token is done via another Jakarta Security API interface: HttpAuthenticationMechanism. Creating a bean of this type allows you to look at an incoming request, see if it's part of your custom authentication, and handle it any way you want. In mine, I look for the "GoogleLogin" authorization header:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
@ApplicationScoped
public class TokenAuthentication implements HttpAuthenticationMechanism {
    @Inject IdentityStore identityStore;
    
    @Override
    public AuthenticationStatus validateRequest(HttpServletRequest request, HttpServletResponse response,
            HttpMessageContext httpMessageContext) throws AuthenticationException {
        
        String authHeader = request.getHeader("Authorization"); //$NON-NLS-1$
        if(StringUtil.isNotEmpty(authHeader) && authHeader.startsWith(GoogleAccountTokenHandler.AUTH_PREFIX)) {
            CredentialValidationResult result = identityStore.validate(new GoogleAccountTokenHeaderCredential(authHeader));
            switch(result.getStatus()) {
            case VALID:
                httpMessageContext.notifyContainerAboutLogin(result);
                return AuthenticationStatus.SUCCESS;
            default:
                return AuthenticationStatus.SEND_FAILURE;
            }
        }
        
        return AuthenticationStatus.NOT_DONE;
    }

}

Here, I look for the "Authorization" header and, if it starts with "GoogleLogin auth=", then I parse it for the token, create an instance of an app-custom GoogleAccountTokenHeaderCredential object (implementing Credential) and ask the app's IdentityStore to authorize it.

Returning to the IdentityStore implementation, that meant adding another validate override:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
@ApplicationScoped
public class NotesDirectoryIdentityStore implements IdentityStore {
    /* snip */

    public CredentialValidationResult validate(GoogleAccountTokenHeaderCredential credential) {
        try {
            try(DominoClient client = DominoClientBuilder.newDominoClient().build()) {
                String dn = client.validateCredentialsWithToken(appConfig.getAuthServer(), credential.headerValue());
                return new CredentialValidationResult(null, dn, dn, dn, getGroups(dn));
            }
        } catch (NameNotFoundException e) {
            return CredentialValidationResult.NOT_VALIDATED_RESULT;
        } catch (AuthenticationException | AuthenticationNotSupportedException e) {
            return CredentialValidationResult.INVALID_RESULT;
        }
    }
}

This one looks similar to the UsernamePasswordCredential one above, but takes instances of my custom Credential class - automatically picked up by the default implementation. I decided to be a little extra-fancy here: the particular Domino API in question supports custom token-based authentication to look up a distinguished name, and I made use of that here. That takes us one level deeper:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
public class GoogleAccountTokenHandler implements CredentialValidationTokenHandler<String> {
    public static final String AUTH_PREFIX = "GoogleLogin auth="; //$NON-NLS-1$
    
    @Override
    public boolean canProcess(Object token) {
        if(token instanceof String authHeader) {
            return authHeader.startsWith(AUTH_PREFIX);
        }
        return false;
    }

    @Override
    public String getUserDn(String token, String serverName) throws NameNotFoundException, AuthenticationException, AuthenticationNotSupportedException {
        String userTokenPair = token.substring(AUTH_PREFIX.length());
        int slashIndex = userTokenPair.indexOf('/');
        if(slashIndex >= 0) {
            String tokenVal = userTokenPair.substring(slashIndex+1);
            Token authToken = CDI.current().select(TokenBean.class).get().getToken(tokenVal)
                .orElseThrow(() -> new AuthenticationException(MessageFormat.format("Unable to find token \"{0}\"", token)));
            return authToken.user();
        }
        throw new AuthenticationNotSupportedException("Malformed token");
    }

}

This is the Domino-specific one, inspired by the Jakarta Security API. I could also have done this lookup in the previous class, but this way allows me to reuse this same custom authentication in any API use.

Anyway, this class uses another method on TokenBean:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
@ApplicationScoped
public class TokenBean {    
    @Inject @AdminUser
    Database adminDatabase;

    /* snip */

    public Optional<Token> getToken(String tokenValue) {
        return adminDatabase.openCollection("Tokens") //$NON-NLS-1$
            .orElseThrow(() -> new IllegalStateException("Unable to open view \"Tokens\""))
            .query()
            .readColumnValues()
            .selectByKey(tokenValue, true)
            .firstEntry()
            .map(entry -> new Token(entry.get("Token", String.class, ""), entry.get("User", String.class, ""))); //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$ //$NON-NLS-4$
    }
}

There, it looks up the requested token in the "Tokens" view and, if present, returns a record indicating that token and the user it was created for. The latter is then returned by the above Domino-custom GoogleAccountTokenHandler as the authoritative validated user. In turn, the JEE NotesDirectoryIdentityStore considers the credential validation successful and returns it back to the auth mechanism. Finally, the TokenAuthentication up there sees the successful validation and notifies the container about the user that the token mapped to.

Summary

So that turned into something of a long walk at the end there, but the result is really neat: as far as my app is concerned, the "GoogleLogin" tokens - as looked up in an NSF - are just as good as username/password authentication. Anything that calls httpServletRequest.getUserPrincipal() will see the username from the token, and I also use this result to spawn the Domino session object for each request.

Once all these pieces are in place, none of the rest of the app has to have any knowledge of it at all. When I implement the API to return the actual RSS feed entries, I'll be able to just use the current user, knowing that it's guaranteed to be properly handled by the rest of the system beforehand.

Bonus: Java 16

This last bit isn't really related to the above, but I just want to gush a bit about newer techs. My plan is to deploy this app using my Open Liberty Runtime, which means I can use any Open Liberty and Java version I want. Java 16 came out recently, so I figured I'd give that a shot. Though I don't think Liberty is officially supported on it yet, it's worked out just fine for my needs so far.

This lets me use the features that have come into Java in the last few years, a couple of which moved from experimental/incubating into finalized forms in 16 specifically. For example, I can use records, a specialized type of Java class intended for immutable data. Token is a perfect case for this:

1
2
public record Token(String token, String user) {
}

That's the entirety of the class. Because it's a record, it gets a constructor with those two properties, plus accessor methods named after the properties (as used in the examples above). Neat!

Another handy new feature is pattern matching for instanceof. This allows you to simplify the common idiom where you check if an object is a particular type, then cast it to that type afterwards to do something. With this new syntax, you can compress that into the actual test, as seen above:

1
2
3
4
5
6
7
@Override
public boolean canProcess(Object token) {
    if(token instanceof String authHeader) {
        return authHeader.startsWith(AUTH_PREFIX);
    }
    return false;
}

Using this allows me to check the incoming value's type while also immediately creating a variable to treat it as such. It's essentially the same thing you could do before, but cleaner and more explicit now. There's more of this kind of thing on the way, and I'm looking forward to the future additions eagerly.

The Joyful Utility of Optionals in Java

Apr 23, 2021, 11:11 AM

Tags: java
  1. The Cleansing Flame of Null Analysis
  2. Quick Tip: JDK Null Annotations for Eclipse
  3. The Joyful Utility of Optionals in Java

A while back, I talked about how I had embraced nullness annotations in several of my projects. However, that post predated Domino's laggardly move to Java 8 and so didn't discuss one of the tools that came to Java core in that version: java.util.Optional.

The Concept

Java's Optional is a passable implementation of the Option type concept that's been floating around programming circles for a good long time. It's really come to the fore with the proliferation of large-platform languages like Swift and Kotlin that have the concept built in to the syntax.

Java's implementation doesn't go that far - there's no special syntax for them, at least not yet - but the concept remains the same. The idea is that, when you embrace Optional use, your code will no longer return null, with the goal of cutting down on the pernicious NullPointerException. While you may still return an empty value, you will be doing so in a way that allows (and forces) the downstream programmer to check for that case more cleanly and adapt it into their code.

In Practice

For an example of where this sort of thing is well-suited, take a look at this snippet of code, which is likely to be pretty universal in Domino code:

1
2
3
4
5
View users = db.getView("Users");
Document user = users.getDocumentByKey(userName, true);
if(user != null) {
	// ...
}

Most of the time, this will run fine. However, if, say, the "Users" view is unavailable (if it was replaced in a design refresh, or another developer removed it, or it's reader-inaccessible to the current user), you'll end up with a NullPointerException in the second line. When you have the code in front of you, the problem is obvious quickly, but that will require you to crack open the app and look into what's going on before you can even start actually fixing the trouble. That's also the "good" version of the case - if you're using code that separates the #getView from the call to #getDocumentByKey with a bunch of other code, it'll be harder to track down.

Imagine instead if the Domino API used Optional, and returned an Optional<View> in #getView and similar for #getDocumentByKey. That could look more like this:

1
2
3
4
5
View users = db.getView("Users")
	.orElseThrow(() -> new IllegalStateException("Unable to open view \"Users\""));
users.getDocumentByKey(userName, true).ifPresent(user -> {
	// ...
});

The idea is the same, and you'll still get an exception if "Users" is unavailable, but it will be immediately obvious in the error message what it is that you need to fix.

This also forces the programmer to conceptualize that case in a way that they wouldn't necessarily have without the need to "unwrap" the Optional. Maybe it's actually okay if "Users" doesn't exist - in that case, you could just return early and not even run the risk of an exception at all. Or maybe there's a way to recover from that - maybe look up the user another way, or create the view on the fly.

Implementing It

When spreading Optional across an existing codebase or writing a new one around it, I've found that there are some important things to keep in mind.

First, since Optional is implemented as just a class and not special syntax, I've found that the best way to implement it in your code is to go all or nothing: if you decide you want to use Optional, do it everywhere. The trouble if you mix-and-match is that you'll run into some cases where you still do if(foo != null) { ... }; since an empty Optional is non-null, that habit will bite you.

Usually, though, that's not too much trouble: when you start to change your code, you'll run into tons of type-related problems around code like that, so you'll be cued in to change it while you're working anyway. Just make sure to not leave yourself null-returning method traps elsewhere.

Another fun gotcha you'll hit early is that Optional.of(foo) will throw a NullPointerException if foo is null. That's the JDK being (reasonably) pedantic: if you want to wrap a potentially-null value, you have to instead do Optional.ofNullable(foo). While irksome at first, it drives home the point that one of the virtues of Optional is that it forces you, the programmer, to consider the null case much more than you did previously.

Unwrapping

Optional also provides a number of ways to "unwrap" the value or deal with it, and it's useful to know about them for different situations.

The first one is just someOptional.get(), which will return the contained value or immediately throw a NullPointerException if it's null. I've found that this is best for when you're very confident that the value is non-null, either because you already checked with isPresent or if it being null is a sign that the system is so fubar already that there's no virtue in even customizing the exception.

Somewhat safer than that, though, is what I had above: someOptional.orElseThrow(() -> ...), which will either return the wrapped value or throw a customized exception if it's null. This is ideal for either halting execution with a message for how the developer/admin can fix it or for throwing a useful exception declared in your documentation for a downstream programmer to catch.

There's also someOptional.orElse(someOtherValue). For example, take this case where you have a configuration API that returns an Optional<String> for a given lookup key:

1
String emailTarget = getConfig("EmailTarget").orElse("admin@company.com");

That's essentially the "get-or-default" idiom. Now, you can actually do .orElse(null) if you want, though that's often not the best idea. Still, that can be handy if you're adapting existing code that does a null check immediately already, or if you're writing to another existing API that does use null.

Optional also has a #map method, which may seem a bit weird at first, but can be useful on its own, and is particularly well-suited to use with results from the Stream API like #findFirst. It lets you transform a non-null value into something else and thereby change your Optional to then be an Optional of whatever the transformed value type is. For example:

1
2
3
String userName = findLoggedInUser()
	.map(UserObj::getUserName)
	.orElse("Anonymous");

In this case, findLoggedInUser returns a Optional<UserObj>. Then, the #map call gets the username if present and returns an Optional<String>, which is thereby either unwrapped or turned into "Anonymous".

Optional As A Parameter

Up until this point, I've been talking about Optional as a method return value, but what about using it as a parameter? Well, you can, but the consensus is that you probably shouldn't.

At first, I chafed against this advice - after all, Optional is finally an in-JDK way to express an optional or nullable parameter, so why not use it? The arguments about how it's less efficient for the compiler, while true, didn't sway me much - after all, compilers improve, and it's generally better to do something correct than something microscopically faster.

However, the real thing that convinced me was realizing that, if you have Optional as a method parameter, you're still going to have to null-check it anyway, since you don't control who might be calling your code. So, not only will you have to check and unwrap the Optional, you'll still have to have a null guard for the Optional itself, defeating much of the point.

I do think that there's some utility in Optional parameters in your own in-implementation code, not exposed to the outside. It can be useful to indicate that a parameter value is intended to be ignored outright, for example. As one comment on that SO thread mentions, an Optional parameter allows for three states: null, an empty value, or a present value. You could use null to indicate that you don't want that value checked at all, and then have an empty value mean something particular in the code. But that kind of fiddly hair-splitting is exactly why it should be of limited use and even then very-clearly documented for yourself.

If Java ever gets syntax sugar for Optional (say, declaring the parameter as Object? and having calls passing null auto-wrap them into Optional or something), then this could change.

Interaction With Null Analysis

Finally, I'll mention how using Optional interacts with null annotation analysis.

To begin with, if you're using Eclipse, the immediate answer will be "poorly". Because Eclipse doesn't ship with nullness hints for the core JDK, it won't know that Optional.of returns a non-null value, defeating the entire point. For that, you'll want the lastNPE.org nullness annotations, which will provide such hints. I've found that they're still not perfect here, causing Eclipse to frequently complain about someOptional.orElse(null), but the experience becomes good enough.

IntelliJ, for the record, has such hints built-in, so you don't need to worry there.

Once you have that sorted out, though, they go together really well. For example, pairing the two can help you find cases where, in your Optional translation journey, you're checking whether the Optional itself is null: with null-annotated code, the compiler can see that it will never be null as such and will tell you to change it.

So I advise pairing the two: use Optional for return values everywhere, especially if you're making an API for downstream consumption, and then pair them with null annotations to make sure your own implementation code is correct and to provide hints for opting-in users.

Goodbye, Nathan

Apr 12, 2021, 9:26 PM

It would be difficult to overstate Nathan T. Freeman's impact on the Domino community. I can only imagine he was similarly impactful in other parts of his life, but that's how I knew him.

Back when I was first getting involved in Domino, he was one of the main people to follow. I'd read blogs and comments and know that with his name came something very much worth reading. His assertiveness came across clearly, but was immediately followed by the justification.

When I started to get involved in the community, he was the one who brought me into the XPages Skype channel. Over the years, whether it'd be something I did for one of my own projects or something as part of our collaboration on the OpenNTF Domino API, there were plenty of times when I wrote something and thought "man, I bet Nathan will like this". Generally, it was followed by a - quieter - "at least, I hope so".

A couple years back, there was an opening on the OpenNTF board, and I remember that I was talking to Nathan about it at some user-group soirée, and he pitched me on the idea of running for the seat on the premise of "world domination". Though the OpenNTF server-infrastructure plans in question at the time didn't quite rise to that level, you can't fault the ambition.

He aggressively sought to be ahead of the curve, and was more often right than not. He was deep into XPages before most of us, and was out of it similarly early. It's a thoroughly-admirable trait, particularly when paired with the technical acumen to back it up.

I consider myself fortunate to have had the chance to work with him on several occasions, and it's a shame that it was only those few. The community will be lessened for his loss, but we're all a lot better for him having been here.

But there are brass tacks involved here too: his family can use whatever help you can give, and there is a GoFundMe page set up for this purpose. Please, if you can, donate something.

Using Server-Sent Events on Domino

Mar 30, 2021, 8:57 AM

Tags: jakartaee java

Though Domino's HTTP stack infamously doesn't support WebSocket, WebSocket isn't the only game in town when it comes to getting push-type information to HTTP clients. HTML5 also brought with it the less-famous Server-Sent Events standard, which is basically half of WebSocket: it allows the server to push events to the client, but it's still a one-way communication channel.

The Standard

The technique that SSE uses is almost ludicrously simple: the client makes a request and the server replies that it will provide text/event-stream content and keeps the connection open. Then, it starts emitting events delimited by blank lines:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
HTTP/1.1 200 OK
Content-Type: text/event-stream;charset=UTF-8



event: timeline
data: hello

event: timeline
data: hello

Unlike WebSocket, there's no Upgrade header, no two-way communication, and thereby no special requirements on the server. It's so simple that you don't even really need a server-side library to use it, though it still helps.

In Practice

I've found that, though SSE is intentionally far less capable than WebSocket, it actually provides what I want in almost all cases: the client can receive messages instantaneously from the server, while the server can receive messages from the client by traditional means like POST requests. Though this is less efficient and flexible than WebSocket, it suits perfectly the needs of apps like server monitors, chat rooms, and so forth.

Using SSE on Domino

JAX-RS, the Java REST service framework, provides a mechanism for working with server-sent events pretty nicely. Baeldung, as usual, has a splendid tutorial covering the API, and a chunk of what I say here will be essentially rehashing that.

However, though Domino ships with JAX-RS by way of the ExtLib, the library only implements JAX-RS 1.x, which predates SSE support. Fortunately, newer JAX-RS implementations work pretty well on Domino, as long as you bring them in in a compatible way. In my XPages Jakarta EE Support project, I did this by way of RESTEasy, and there did the legwork to make it work in Domino's OSGi environment. For our example today, though, I'm going to skip that and build a small webapp using the com.ibm.pvc.webcontainer.application extension point. In theory, this should also work XPages-side with my project, though I haven't tested that; it might require messing with the Servlet response cache.

The Example

I've uploaded my example to GitHub, so the code is available there. I've aimed to make it pretty simple, though there's always some extra scaffolding to get this stuff working on Domino. The bulk of the "pom.xml" file is devoted to two main things: packaging an app as an OSGi bundle (with RESTEasy embedded) and generating an update site with site.xml to import into Domino.

Server Side

The real work happens in TimeStreamResource, the JAX-RS resource that manages client connections and also, in this case, happens to emit the messages as well.

This resource, when constructed, spawns two threads. The first one monitors a BlockingQueue for new messages and passes them along to the SseBroadcaster:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
try {
    String message;
    while((message = messageQueue.take()) != null) {
        // The producer below may send a message before setSse is called the first time
        if(this.sseBroadcaster != null) {
            this.sseBroadcaster.broadcast(this.sse.newEvent("timeline", message)); //$NON-NLS-1$
        }
    }
} catch(InterruptedException e) {
    // Then we're shutting down
} finally {
    this.sseBroadcaster.close();
}

Here, I'm using the Sse#newEvent convenience method to send a basic text message. In practice, you'll likely want to use the builder you get from Sse#newEventBuilder to construct more-complicated events with IDs and structured data types (usually JSON).

A BlockingQueue implementation (such as LinkedBlockingDeque) is ideal for this task, as it provides a simple API to add objects to the queue and then wait for new ones to arrive.

The second one emits a new message every 10 seconds. This is just for the example's sake, and would normally be actually looking something up or would itself be a listener for events it would like to broadcast.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
try {
    while(true) {
        String eventContent = "- At the tone, the Domino time will be " + OffsetDateTime.now();
        messageQueue.offer(eventContent);

        // Note: any sleeping should be short enough that it doesn't block HTTP restart
        TimeUnit.SECONDS.sleep(10);
    }
} catch(InterruptedException e) {
    // Then we're shutting down
}

Browsers can register as listeners just by issuing a GET request to the API endpoint:

1
2
3
4
5
@GET
@Produces(MediaType.SERVER_SENT_EVENTS)
public void get(@Context SseEventSink sseEventSink) {
    this.sseBroadcaster.register(sseEventSink);
}

That will register them as an available listener when broadcast events are sent out.

Additionally, to simulate something like a chat room, I added a POST endpoint to send new messages beyond the periodic ten-second broadcast:

1
2
3
4
5
6
@POST
@Produces(MediaType.TEXT_PLAIN)
public String sendMessage(String message) throws InterruptedException {
    messageQueue.offer(message);
    return "Received message";
}

That's really what there is to it as far as "business logic" goes. There's some scaffolding in the Servlet implementation to get RestEasy working nicely and manage the ExecutorService and the obligatory "plugin.xml" to register the app with Domino and "web.xml" to account for Domino's old Servlet spec, but that's about it.

Client Side

On the client side, everything you need is built into every modern browser. In fact, the bulk of "index.html" is CSS and basic HTML. The JavaScript involved in blessedly slight:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
function sendMessage() {
    const cmd = document.getElementById("message").value;
    document.getElementById("message").value = "";
    fetch("api/time", {
        method: "POST",
        body: cmd
    });
    return false;
}
function appendLogLine(line) {
    const output = document.getElementById("output");
    output.innerText += line + "\n";
    output.scrollTop = output.scrollHeight;
}
function subscribe() {
    const eventSource = new EventSource("api/time");
    eventSource.addEventListener("timeline",  (event) => {
        appendLogLine(event.data);
    });
    eventSource.onerror = function (err) {
        console.error("EventSource failed:", err);
    };
}

window.addEventListener("load", () => subscribe());

The EventSource object is the core of it and is a standard browser component. You give it a path to watch and then listen for events and errors. fetch is also standard and is a much-nicer API for dealing with HTTP requests. In a real app, things might get a bit more complicated if you want to pass along credentials and the like, but this is really it.

Gotchas

The biggest thing to keep in mind when working with this is that you have to be very careful to not block Domino's HTTP task from restarting. If you don't keep everything in an ExecutorService and account for InterruptedExceptions as I do here, you're highly likely to run into a situation where a thread will keep chugging along indefinitely, leading to the dreaded "waiting for session to finish" loop. The ExecutorService's shutdownNow method helps you manage this - as long as your threads have escape hatches for the InterruptedException they'll receive, you should be good.

I also, admittedly, have not yet tested this at scale. I've tried it out here and there for clients, but haven't pulled the trigger on actually shipping anything with it. It should work fine, since it's using standard JAX-RS stuff, but there's always the chance that, say, the broadcaster registry will fill up with never-ending requests and will eventually bloat up. The stack should handle that properly, but you never know.

Beyond any worries about the web container, it's also just a minefield of potential threading and duplicated-work trouble. For example, when I first wrote the example, I found that messages weren't shared, and then that the time messages could get doubled up. That's because JAX-RS, by default, creates a new instance of the resource class for each request. Moving the declaration from the Application class's getClasses() method (which creates new objects) to getSingletons() (which reuses single objects) fixed the first problem. After that, I found that the setSse method was called multiple times even for the singleton, and so I moved the thread spawning to the constructor to ensure that they're only launched once.

Once you have the threading sorted out, though, this ends up being a pretty-practical path to accomplishing the bulk of what you would normally do with WebSocket, even with an aging HTTP stack like Domino's.