A while back, I talked about the uses of HttpService in Domino. In that post, I talked about how the various HttpService implementations take a look at incoming URLs, see if they're something that should be handled on the Java layer, and then either handle them or pass them back to the legacy NHTTP code to do its thing. My fiddling with the XPages Jakarta EE project in recent days has gotten me thinking about this layer again, and I think it'll be interesting to expand how how this whole layer works (at least as I understand it).

Along with this post, it might be useful to peruse the slide deck for AD105 from LotusSphere 2011. There's a lot of good stuff in there, and basically nothing has changed in the intervening 11 years.

The Stack (Conceptually)

The Domino HTTP stack looks conceptually something like this:

This is, setting aside the specifics, pretty similar to how other app servers of various kinds are laid out. There's some bottom layer that handles the actual network connection, some part just above that that handles interpreting the requests as HTTP (optionally bypassed in some cases), and then an orchestrator that manages the actual apps sitting on the top layer and routes requests as appropriate.

The Stacks (Java-wise)

Before I continue, I think it will be useful to have some stack traces to reference back to, to see what they share in common and where they diverge. These three examples - from an XPage request, an Equinox-registered Servlet, and an OSGi-packaged webapp - all cover the part of the stack from the bottom up until where user code comes into play.

XPages (DesignerFacesServlet is what handles serving an XPage):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

     at com.ibm.xsp.webapp.DesignerFacesServlet.service(DesignerFacesServlet.java:103)
     at com.ibm.designer.runtime.domino.adapter.ComponentModule.invokeServlet(ComponentModule.java:600)
     at com.ibm.domino.xsp.module.nsf.NSFComponentModule.invokeServlet(NSFComponentModule.java:1352)
     at com.ibm.designer.runtime.domino.adapter.ComponentModule$AdapterInvoker.invokeServlet(ComponentModule.java:877)
     at com.ibm.designer.runtime.domino.adapter.ComponentModule$ServletInvoker.doService(ComponentModule.java:820)
     at com.ibm.designer.runtime.domino.adapter.ComponentModule.doService(ComponentModule.java:589)
     at com.ibm.domino.xsp.module.nsf.NSFComponentModule.doService(NSFComponentModule.java:1336)
     at com.ibm.domino.xsp.module.nsf.NSFService.doServiceInternal(NSFService.java:725)
     at com.ibm.domino.xsp.module.nsf.NSFService.doService(NSFService.java:515)
     at com.ibm.designer.runtime.domino.adapter.LCDEnvironment.doService(LCDEnvironment.java:363)
     at com.ibm.designer.runtime.domino.adapter.LCDEnvironment.service(LCDEnvironment.java:319)
     at com.ibm.domino.xsp.bridge.http.engine.XspCmdManager.service(XspCmdManager.java:272)

Equinox Servlet (the org.eclipse.equinox.http.registry.servlets extension point):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

	at com.example.SomeServlet.service(SomeServlet.java:104)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:806)
	at org.eclipse.equinox.http.registry.internal.ServletManager$ServletWrapper.service(ServletManager.java:180)
	at org.eclipse.equinox.http.servlet.internal.ServletRegistration.handleRequest(ServletRegistration.java:90)
	at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:111)
	at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:67)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:806)
	at com.ibm.domino.xsp.adapter.osgi.OSGIModule.invokeServlet(OSGIModule.java:167)
	at com.ibm.domino.xsp.adapter.osgi.OSGIModule.access$0(OSGIModule.java:153)
	at com.ibm.domino.xsp.adapter.osgi.OSGIModule$1.invokeServlet(OSGIModule.java:134)
	at com.ibm.domino.xsp.adapter.osgi.AbstractOSGIModule.invokeServletWithNotesContext(AbstractOSGIModule.java:181)
	at com.ibm.domino.xsp.adapter.osgi.OSGIModule.doService(OSGIModule.java:128)
	at com.ibm.domino.xsp.adapter.osgi.OSGIService.doService(OSGIService.java:418)
	at com.ibm.designer.runtime.domino.adapter.LCDEnvironment.doService(LCDEnvironment.java:363)
	at com.ibm.designer.runtime.domino.adapter.LCDEnvironment.service(LCDEnvironment.java:319)
	at com.ibm.domino.xsp.bridge.http.engine.XspCmdManager.service(XspCmdManager.java:272)

Web Container (the com.ibm.pvc.webcontainer.application extension point):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

	at ccom.example.ExampleServlet.doGet(ExampleServlet.java:18)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:693)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:806)
	at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1661)
	at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:937)
	at com.ibm.pvc.internal.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:85)
	at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:500)
	at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3810)
	at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:276)
	at com.ibm.pvc.internal.webcontainer.VirtualHost.handleRequest(VirtualHost.java:143)
	at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:931)
	at com.ibm.pvc.internal.webcontainer.WebContainerBridge.handleRequest(WebContainerBridge.java:25)
	at com.ibm.domino.osgi.core.webContainer.WebApplicationsTracker.doService(WebApplicationsTracker.java:141)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.ibm.domino.xsp.adapter.osgi.webContainer.OSGIWebContainerModule.invokeWebAppContainerService(OSGIWebContainerModule.java:207)
	at com.ibm.domino.xsp.adapter.osgi.webContainer.OSGIWebContainerModule.doService(OSGIWebContainerModule.java:178)
	at com.ibm.domino.xsp.adapter.osgi.OSGIService.doService(OSGIService.java:418)
	at com.ibm.designer.runtime.domino.adapter.LCDEnvironment.doService(LCDEnvironment.java:363)
	at com.ibm.designer.runtime.domino.adapter.LCDEnvironment.service(LCDEnvironment.java:319)
	at com.ibm.domino.xsp.bridge.http.engine.XspCmdManager.service(XspCmdManager.java:272)

Each one of these has some intriguing lines, but we'll end up focusing on the bottom 5-6 in each.

Anyway, back to the details.

Orchestrator

The bottom three lines of all three stacks are identical, and they show the entrypoint from the C side to Java.

The job of XspCmdManager is to take a bunch of handles and flags given to it by the C side, wrap them into something a little more suitable for polite company, and pass that off immediately to LCDEnvironment. At this level, while the code is clearly focused around getting to the point of handling Servlets, the actual classes don't actually implement javax.servlet parts - they're all a little more abstract than that. XspCmdManager has a few other responsibilities, but it's best thought of as just the glue layer between the native side and the Java stack.

From there, it passes the request along to LCDEnvironment, as the sole implementation of LCDRequestHandler. Things get a little meatier here. This is the part that loads up all of the HttpService implementations we've discussed before. It uses these services to answer calls to isXspUrl (which basically means "should Java handle this?") and then to handle the incoming requests. The first HttpService (sorted by its getPriority() value) that will handle the incoming request gets it, and here's our first point of divergence above. NSFService is the in-NSF XPages-and-stuff handler, taking care of requests with ".nsf" and either "xsp" or a registered extension in the path. OSGIService, for its part, handles both Equinox-registered servlets and Expeditor webapps, albeit in different ways.

ComponentModules

The next layer is interesting, and it's a part I didn't really talk about in the previous post. While an HttpService can just handle an incoming request directly (as the proxy service in the Domino Open Liberty Runtime project does), the idiom in the Domino stack is to use ComponentModule implementations to do it. These correspond conceptually to web apps deployed from WAR files in a standard app server: they're a cordoned-off blob of user code, with its own ClassLoader and notions for how to access resources.

For an NSF, this is NSFComponentModule. These objects are spawned by NSFService as needed when a matching request comes in for an NSF the first time and create a weird sort of webapp out of the NSF contents (with special handling for Single-Copy XPage Design). It doesn't go as far in that direction as the full web container, but it's enough to run and retain the XPages application. This type of module also opts in to the IServletFactory extension system, where contributors can add Servlets to the module either internally or via an OSGi extension. All ComponentModule types have the ability to opt in to this, but NSFComponentModule is the only one that does in practice.

Though both the Equinox Servlet and the PVC web container app go through OSGIService, here is where they split apart into OSGIModule (for standalone Servlets) and OSGIWebContainerModule.

OSGIModule uses some of the parts of the Equinox Servlet Container support to run an individual Servlet. It's the smallest of the three and mostly just creates the ServletRequest et al implementations, does a little wrapping in Equinox garb, and passes them on to your HttpServlet class.

OSGIWebContainerModule is fancier, since its job is to run (most of) a JEE-style web.xml-based app contained in an OSGi bundle. To do this, it uses a chopped-down fork of WebSphere - indeed, many of the classes in the stack still exist in much the same form in Liberty, but here are intermingled with Domino-specific stuff and some Expeditor PvC detritus. This isn't quite as capable as a full Servlet 2.5 container, lacking things like Filters and various listeners, but it gets the job done.

Module Adaptability

Though this system hasn't in practice grown beyond the built-in implementations to my knowledge, it's a neat little structure. There's some unfinished stuff in there in a mashupmaker package that I guess must have been to do with Lotus Mashups (which is apparently a product that existed at some point), but that's about it as far as extending it.

This is an intriguing layer, though, since it's also the one where the "adapter" Servlet objects from above are actually turned into instances of javax.servlet classes, specifically using classes like LCDAdapterHttpServletResponse. At this layer, you have a good amount of Java scaffolding, but being before the full conversion to javax.servlet classes means that a lot of the limitations in Domino's Servlet support only really come in in the upper echelons. Other than network- and HTTP-layer technologies like WebSocket and HTTP/2 (which are handled at the native layer), it'd be entirely possible to plug into this system with more-modern technologies while still participating cleanly in the environment. For example, you could write an HttpService that declares a higher priority than NSFService and use it to treat an NSF as an entirely-different sort of app, intercepting all URLs prefixed with it. I don't know that it would be a good idea to do so, but it's possible, and it's fun to think about.

About a year ago, I wrote a post discussing what I saw as potential future options for all the XPages code we've collectively written over the years. That post was written with the assumption that the future life of an XPages app isn't XPages as it exists today - while that's still a possibility, it's a lot less fun to speculate about.

As it has been for years, this remains on my mind, but my recent addition of JSF to the XPages Jakarta EE project caused it to bubble back to the top of my musings.

Conceptual Tools

Before getting into some speculative specifics, I'd like to first take a step back and assess the concepts we're working with.

In that post, and as I frequently do, I referred back to an old post of mine discussing how an XPage is best thought of as a tree of components, one most-commonly described by way of the XSP markup language. The critical concept there is that there's nothing that inherently ties an XPage to the specific thing that sends HTML to the browser, and all the moreso when you're talking about the XML. While "XPages" as an entity on Domino encompasses an entire stack that starts just barely above the bottom of nHTTP, "an XPage" as such is almost always just XML that describes what you want to happen on the page. There are parts of it that are more specific than others, for sure: <xp:inputText/> has equivalents in a million other languages, but value="#{javascript: ...}" starts getting into some gnarly implementation specifics. Still, the point is that anything that can successfully interpret XSP markup can be considered functionally "XPages" for most uses.

The next handy concept to keep in mind is related, and that's that code is data. This is most apparent with an XML-based language like XSP, but it applies to everything. SSJS code is absolutely data: it's just strings that happen to be parsed out at runtime as ASTs and then executed. For SSJS, there's no explicit guarantee that facesContext refers to an instance of javax.faces.context.FacesContext or one of its com.ibm.xsp subclasses. Heck, there's no specific requirement that even referring to the class by name would reference the same thing. It's just data and can be interepreted as the runtime seems fit.

And that goes further: Java bytecode is just data too. OSGi has a mechanism to alter classes at load time that already exists and works great on Domino. Also pertinently, Eclipse Transformer is a tool that translates code (source or compiled) that references javax.* JEE classes to jakarta.* classes, and is used commonly now by webapp runtimes to support both legacy and new apps with the same codebase. Fancy stuff, that.

The Musing

So back to what I've been pondering. I think I did mostly a good job covering the bases in my original post, but time and experience has given me further perspective.

JSF Driver For XSP

When I first mentioned it, I brushed the concept of writing a driver for JSF off a bit - not fully, but I did give it a shorter shrift than it deserved.

To begin with, this concept exists in JSF: the view declaration language. Now, it's not actually used for this sort of "transplant" thing, mind you: in practice, it's a concept that was used to transition from JSP as the language to write JSF pages to Facelets and not much else. Still, there's no inherent reason why one couldn't write a VDL driver for JSF that would interpret XSP markup and create components based on it. This is especially true because XSP itself doesn't really contain any real curveballs: it's a pretty basic mapping of tags to component names and attributes to bean properties, with the main hiccup being <xp:this.action>-type complex properties.

Components

I've also given some thought to that tag-to-component mapping. One way to do that would be to make an interpreter that sees <xp:inputText/> and, instead of translating it to com.ibm.xsp.component.UIInputText, would instead translate it to a newer stock component. I thereby brushed it off as being properly too complex to be workable, as XPages's nature as a hard fork of JSF meant that things like the java.faces.component.UIComponent#_xspGetStateId method would prove intractable.

I'm less sure of this difficulty now, though. The switch from javax.faces to jakarta.faces means there's room for coexistence on the same classpath, as I now have in the JEE project. This means there's a lot of room for adapting older code unchanged by way of using wrapper objects and proxies.

For the former, what I mean is that, while a JSF 4.x implementation like MyFaces 4 now has no knowledge of what a javax.faces.component.UIInput is, it doesn't have to: all it needs is a subclass of jakarta.faces.component.UIComponent that can fit into a tree and respond to normal JSF calls like processUpdates. The "real" JSF stack in front could pass in incoming form data from the client just the same as it does now, and a wrapped legacy XPages class could handle it exactly as it does now. Things get more complicated than that, but not impossibly so.

This is similar to all the Servlet-object wrapping and unwrapping I do in the JEE support project. These wrappers interpret and route equivalent method calls to the delegates they're wrapping, and so Servlet 5 code doesn't care that it's working with a Servlet 2.4 request, and similarly the Servlet 2.4 code can continue to know nothing at all about Servlet 5. The glue code handles the simple wrapper-based translations between the layers.

So, in this way, the XPages fork of JSF could remain essentially untouched. As long as the runtime does things like inserting an appropriate old-style FacesContext object into javax.faces.context.FacesContext and the like, no one needs to be the wiser.

And this is where Java object proxies could come into play. Where I've above said things like "an instance of javax.faces.context.FacesContext" or "an... object", that doesn't even need to be a real class that you construct with new Foo(). While Java's built-in proxies only work with interfaces, libraries like Javassist can generate proxy objects for true classes as well. One could have a proxy object that checks to see if an incoming method is compatible with the new style and use that one, or otherwise route to a wrapped class, and yet remain class-cast compatible with old code. I'm not sure this would be required if wrapping was done fully, but it's good to know as an option.

Code

Then there's user code. I was going to have a bunch of stuff to say in this section, but it's actually all largely covered by the steps that would be necessary for keeping runtime code compatible, if that were the route to take. If user code calls javax.faces.context.FacesContext#getCurrentInstance, then it'd get a legacy object; if it calls jakarta.faces.context.FacesContext#getCurrentInstance, then it'd get the new-era one. That'd be the same as the work necessary to keep the old parts chugging along.

Now, if the old parts were to change - were the task to be to make it so that the com.ibm.xsp classes are based on JSF 4.x - then there'd be some fiddling to do for user code. But, again, it'd be largely the same idea. You could either translate the code as it's loaded (for Java) or executed (for SSJS) or you could pass wrapper objects around. So all the same ideas apply.

Coexistence

I've also been thinking a lot about the notion of an "XPages 2" or the like existing alongside XPages as it is now. For example, you could imagine a variant of "XPages" that is indeed a JSF view definition language that interprets XSP markup, but which doesn't attempt to guarantee pure compatibility with existing code. Maybe it'd get 80% of the way there - it'd interpret components in largely the same way, but wouldn't guarantee that every little bit of code that dives down into the runtime implementation would work identically, and wouldn't guarantee that every fiddly detail of the XPages lifecycle and its optimizations would execute in the exact same way.

In such a scenario, there wouldn't be any particular need to remove or change the old stuff. This is the scenario that JEE app servers all faced with the jakarta.* move: you can't realistically expect every app to be transformed (automatically or otherwise) to work with the new versions, especially because old apps may target even-older specs that have since had other breaking changes.

In Domino, there are a few ways one could go about accomplishing this. One way would be to do kind of like what I do for JSP and JSF: tell NSFService to handle a given extension and then write a factory to handle it. In this way, you could declare a new extension, say ".xspx" (note: please do not use this extension), and then route incoming requests to a handler that's like normal XPages but not 100% compatible.

Alternatively, you could do something I've pondered for a while, which is to make another HttpService implementation that would check for a flag in the database referenced in incoming .nsf-containing URLs to see if it's set to "new mode" and, if so, interpret the request however which way it would like. Such a service could disregard all old rules for URL handling if it wanted, and could worry much less about the potential of cross-contamination between request types. It'd be more work, but is an intriguing possibility.

Conclusion

So is any of this the right way to go about dealing with our dear old friend XPages? I don't know - maybe. This is all pretty off-the-cuff, and I haven't necessarily thought through all the implications, but I do feel like there's more wiggle room here than I'd originally assumed. It's interesting to think about, at the very least.

When I talked about adding Jakarta NoSQL to the JEE support project, I mentioned how that in a way completed the pitch for this project as a full development mechanism. With data access and MVC+JSP in place, now it provides tools to build REST-based or server-rendered UIs backed cleanly by Domino data. Neat!

But, much like I had previously danced around the question of data access, there was still a spectre lurking in the background, a long-standing part of the Java/Jakarta EE platform: JSF, now named Jakarta Server Faces. This has unsurprisingly been gnawing at the back of my mind, since "upgrade JSF" has been one of the longest-running requests for XPages, starting basically right after JSF 1.2 came out with a smattering of technologies XPages didn't get.

While the notion of un-forking XPages is intriguing, it's its own big pile of work, and not in this project's bailiwick to address. What this project can do, though, is bring current JSF to an NSF alongside XPages. This proposition has a lot going on, so I think it'll be worth discussing both the practical elements and the implications.

How Is JSF Doing Nowadays, Anyway?

To start out with, it will be instructive to look at how JSF has fared since XPages split off from it in the JSF 1.1 era. We'll use the version history from Wikipedia as a starting point:

1. 2004 - JSF 1.0
2. 2004 - JSF 1.1 (minor release)
3. 2006 - JSF 1.2 (feature release, included in Java EE)
4. 2009 - JSF 2.0 (major release with significant improvements)
5. 2010 - JSF 2.1 (minor release)
6. 2013 - JSF 2.2 (feature release)
7. 2017 - JSF 2.3 (feature release)

So... this is a real mixed bag, huh? Since XPages, JSF received a few major feature releases, progressing to 2.3. But, uh, the gaps in the years, though - that's ominous. And 2017 was the last feature update? Hrm.

But wait - when I tweeted about it, the screenshot says "JSF 4.0". What's going on there?

Well, like all active JEE specs, Faces received a major-version bump for semantic-versioning purposes when moving to the jakarta.* namespace. Version 3.0 came out in 2020, but was functionally identical to 2.3 from 2017, just with the API package names changed.

After that point, though, the future is in the Eclipse Foundation's hands, and each Jakarta spec is deciding its fate for future releases. Some specs, like JSP, are focusing on non-breaking releases that focus on just a handful of fixes and clarifications. Faces, though, seems to have be the focus of some pent-up desires for change that are coming to fruition now that there's headroom for it.

Faces 4.0 gets another major-version bump because it involves some breaking changes that go beyond just the package names, including outright removing old deprecated features in favor of better modern options. Beyond that, it gains some outright new features - not the sort of features that would change a developer's life, but useful stuff.

Does this mean that JSF is in the prime of its life? Well, time will tell, and merely having developers settle what was presumably very old business doesn't guarantee a vibrant future, but it's a nice sign. In any event, it's not dead, so it has that going for it. Plus, the repositories for "ecosystem" libraries like PrimeFaces and Tobago are quite active, which is important.

So Is It Good? Is This How We Should Write Apps?

Maybe! I'm not sure. I mean, there's no getting around the general popularity of client-JS-first app dev, and the server side of that is covered well by JAX-RS. That said, current popularity isn't everything when it comes to development: it's all about the problems it solves. For the type of development usually done for Domino, server-side apps fit quite well, and even an exceedingly-adept developer can often get results quicker in an integrated stack of this type.

In any event, I expect I'll give it a shot for a bit and see how it shakes out.

How Does This Even Work? Doesn't It Conflict With XPages?

I'm glad you asked! Part of the problem I'd tangled with with this project from the start was that the aging versions of the various JEE specs - and the XPages JSF fork - are essentially unmovable obstacles. If I want to have a project that enhances Domino in place rather than replacing it in whole or in part, I have to deal with the stuff that's there.

Fortunately, like with pseudo-updating the core Servlet spec, the javax.*-to-jakarta.* namespace move is my savior. Honestly, being a trademark stickler may be the best thing Oracle's ever done from my perspective. Thanks to this switch, all the concerns API-side about distinguishing versions disappeared. While I might know that javax.faces.context.FacesContext and jakarta.faces.context.FacesContext are competing versions of the same spec, Java doesn't know - they're completely unrelated.

However, as with some other components, the API isn't the whole story. While the API packages all changed for Jakarta EE 9, the implementations generally kept their original packages. XPages is based on the original JSF implementation, which is now Mojarra over at Eclipse. This implementation uses the com.sun.faces namespace. While most of the XPages-specific stuff is under com.ibm.xsp, all those Sun-branded classes are still floating around, like com.sun.faces.RIConstants and com.sun.faces.context.FacesContextImpl. Moreover, XPages still uses com.sun.faces prefixes for stored application properties. For example, there's an ApplicationAssociate object that Sun-JSF and XPages use to keep track of app-specific information, and it stashes this in the conceptual NSF application. Using Mojarra, these sorts of classes and properties conflict, and things got hairy, with me trying to stash the com.sun stuff to the side per-request, with only partial success.

Fortunately, Mojarra isn't the only game in town: Apache MyFaces is alive and well, including with an actively-developed 4.0 branch. This implementation does not derive from the original, and doesn't use the com.sun.faces namespace. As an intriguing tidbit, Notes (but not Domino) actually includes MyFaces 1.1; I'm guessing it's another thing in there to support some kind of "Social" foofaraw.

Anyway, MyFaces was my ticket to success. Not only did it remove the problem of needing to walk on eggshells with package and attribute names, but the fact that it's a wholly-different implementation removed some of the other weird problems I was dealing with from the table.

Required Infrastructure Improvements

Unlike some of the more staid specs (like JSP) or cutting-edge ones (like MVC and NoSQL), both JSF implementations lean heavily on Servlet-spec features that showed up after Domino's 2.4 implementation. While they fortunately don't require filters, they do heavily use listeners of many stripes.

So, like when I had to hack together RequestDispatcher support to implement MVC, I had to do similarly here to support attribute listeners and manually kick of lifecycle event notifications.

I realize I'm kind of baking my way into making a full Servlet 5 container on top of Domino's rickety old one. There's way more work to make that an actual thing, but it's intriguing seeing it take shape just as incidental work from implementing other parts.

Next Steps

For the next steps, I'm not quite sure. For what it is, I think it's all working pretty well. However, while JSF on its own provides a lot of functionality, it's stuff like the component libraries from PrimeFaces that makes it a full toolkit. In a normal case, the server itself wouldn't provide component packs like PrimeFaces or Tobago - the app itself would bring them in as Maven/etc. dependencies and they would be included as part of the WAR. This would probably work in an NSF, but the experience of developing in an NSF when you have JARs in the classpath is... not great. Accordingly, I'm pondering including one or both of those in the project. It'd have tradeoffs, but it might make sense, or maybe it'd make sense to have associated projects that package them as distinct libraries to include. We'll see.

Over the last two weeks, Graham Acres and I recorded a video series for OpenNTF about my XPages Jakarta EE Support project, which has seen a flurry of development in the last few months. The 15-part series is up on YouTube:

The project itself saw the release of version 2.3.0 today, which is the first release with the Jakarta NoSQL driver I blogged about recently.

I think the project has turned into a pretty-interesting "platform update" for XPages, and I hope the video series captures that a bit. I'm still mulling over a sort of "thesis statement" about the whole thing, but for now describing the various new capabilities and how they interact will have to suffice.

A few weeks back, I talked about my use of DQL and QRP in writing a JNoSQL driver for Domino. In that, I left the specifics of the JNoSQL side out and focused on the Domino side, but that former part certainly warrants some expansion as well.

Background

As a quick overview, Jakarta NoSQL is an approaching-finalization spec for working with NoSQL databases of various stripes in a Jakarta EE app. This is as opposed to the venerable JPA, which is a long-standing API for working with RDBMSes in JEE.

JNoSQL is the implementation of the Jakarta NoSQL spec, and is also an Eclipse project. As a historical note, the individual components of the implementation used to have Greek-mythological names, which is why older drivers like my Darwino driver or original Domino driver are sprinkled with references to "Diana" and "Artemis". The "JNoSQL" name also pre-dates its reification into a Jakarta spec - normally, spec names and implementations aren't quite so similarly named.

The specification is broken up into two main categories. The README for the implementation describes this well, but the summary is:

1. "Communication" handles interpreting JNoSQL CRUD operations and actually applying them to the database.
2. "Mapping" handles what the app developer interacts with: annotating classes to relate them to the back-end database and querying object repositories.

An individual driver may include code for both sides of this, but only the Communication side is obligatory to implement. A driver would contribute to the Mapping side as well if they want to provide database-specific higher-level concepts. For example, the Darwino driver does this to provide explicit annotations for its full-text search, stored-cursor, and JSQL capabilities. I may do similarly in the Domino driver to expose FT search, view operations, or DQL queries directly.

Jakarta NoSQL handles Key-Value, Column, Graph, and Document data stores, but we only care about the last category for now.

Implementation Overview

Now, on to the actual implementation in question. The handful of classes in the implementation fall into a few categories:

• DatabaseSupplier and SessionSupplier are mechanisms for the environment to inject the Database and Session (as signer) objects for the driver to use. This indirection is to allow the driver to be used outside of the XPages JEE project specifically - in this project, the XPages library bundle provides implementations using the current NotesContext.
• DominoDocumentCollectionManager defines the interface for the Domino-specific manager type. This interface is currently empty beyond what it inherits, but it's where I may provide hooks for DB-specific capabilities.
  • DefaultDominoDocumentCollectionManager, in turn, is the implementation of this and actually defines CRUD operations. More on this down the page for sure.
  • DominoDocumentCollectionManagerFactory is the class that, based on configuration, emits the above for the environment.
• DominoDocumentConfiguration is the class where you can configure the above, and serves as the entrypoint for practical use. It uses Jakarta NoSQL's config and settings concepts, either taking those suppliers from the passed-in configuration or trying to find them from the current CDI environment.
• EntityConverter contains support code for CRUD operations, assisting in mapping between JNoSQL's internal entity representation and Domino documents.
• QueryConverter does similarly for queries, taking JNoSQL's abstract query tree concept and translating it to DQL.
• Speaking of which, DQL is a wholesale ripoff properly-licensed adaptation of Karsten Lehmann's class of the same name, modified slightly to remove Domino-JNA-specific elements and improve java.time handling a bit.

Implementation Details

The core entrypoint for data operations is DefaultDominoDocumentCollectionManager, and JNoSQL specifies a few main operations to implement, basically CRUD plus total count:

• insert and its overrides handle taking an abstract DocumentEntity from JNoSQL and turning it into a new lotus.domino.Document in the target database.
• update does similarly, but with the assumption that the incoming entity represents a modification to an existing document.
• delete takes an incoming abstract query and deletes all documents matching it.
• select takes an incoming abstract query, finds matching documents, converts them to a neutral format, and returns them to JNoSQL. This is what my earlier post was all about.
• count retrieves a count for all documents in a "collection". "Collection" here is a MongoDB-ism and the most-practical Domino equivalent is "documents with a specific Form name".

Entity Conversion

The insert, update, and select methods have as part of their jobs the task of translating between Domino's storage and JNoSQL's intermediate representation, and this happens in EntityConverter.

Now, this point of the code has some... nomenclature-based issues. There's lotus.domino.Document, our legacy representation of a Domino document handle. Then, there's jakarta.nosql.document.Document: this oddly-named interface actually represents a single key-value pair within the conceptual document - roughly, this corresponds to lotus.domino.Item. Finally, there's jakarta.nosql.document.DocumentEntity, which is the higher-level representation of a conceptual document on the JNoSQL side, and this contains many jakarta.nosql.document.Documents. This all works out in practice, but it's important to know about when you look into the implementation code.

The first couple methods in this utility class handle converting query results of different types: QRP result JSON, QRP result views, and generic DocumentCollections. Strictly speaking, I could remove the first one now that it's unused, but there's a non-zero chance that I'll return to it if it ends up being efficient down the line.

Each of those methods will eventually call to toDocuments, which converts a lotus.domino.Document object to an equivalent List of JNoSQL Documents (i.e. the individual key-value pairs). Due to the way JNoSQL works, this method has no way to know what the actual desired fields the higher level will want are, so it attempts to convert all items in the document to more-common Java types. There's much more work to do here, some of it based on just needing to add other types (like improving rich text handling) and some of it based on needing a better Notes API (like proper conversion from Notes times to java.time).

In the other direction, there's the method that converts from a JNoSQL DocumentEntity to a Domino document, which is used by the insert and update methods. This converts some known common incoming types and converts them to Domino item values. Like the earlier methods, this could use some work in translating types, but that's also something that a better Notes API could handle for me.

Query Conversion

The QueryConverter class has a slightly-simpler job: taking JNoSQL's concept of a query and translating it to a document selection.

The jakarta.nosql.document.DocumentQuery type does a bit of double duty: it's used both for arbitrary queries (Foo='Bar'-type stuff) as well as for selecting documents by UNID. The select method covers that, producing a QueryConverterResult object to ferry the important information back to DefaultDominoDocumentCollectionManager.

The core work of QueryConverter is in getCondition, where it performs an AST-to-DQL conversion. JNoSQL has a couple mechanisms for querying entities: explicit Java-based queries, implicit queries based on repository definition, or a SQL-like query language. Regardless of what the higher level does to query, though, it comes to the Communication driver as this tree of objects (technically, the driver can handle the last specially, but by default it arrives parsed).

Fortunately, this sort of work is a common sort of idiom. You start with the top node of the tree, handle it based on its type and, as needed, recurse down into the next node. So if, for example, the top node is an EQUALS, all this converter needs to do is return the DQL representation of "this field equals that value", and so forth for other comparison operators. If it encounters AND, OR, or NOT, then the job changes to making a composite query of that operator plus the results of converting whatever the operator is applied to - which is where the recursion back into the same method comes in.

Future Work

The main immediate work to do here is enhancing the data conversion: handling more outgoing Domino item types and incoming Java object types. A good deal of this can be done as-is, but doing some other parts reliably will be best done by changing out the specific Notes API in use. I used lotus.domino because it's present already, but it's a placeholder for sure.

There are also a bunch of efficiency tweaks I can make: more lazy loading in conversion, optimizing data fetching for specific queries, and logging DQL explain results for developers.

Beyond that, I'll have to consider if it's worth adding extensions to the mapping side. As I mentioned, the Darwino driver has some extensions for its JSQL language and similar concepts, and it's possible that it'd be worth adding similar things for Domino, in particular direct FT searching. That said, DQL does a pretty good job being the all-consuming target, and so translating JNoSQL queries to DQL may suffice to extract what performance Domino can provide.

So we'll see. A lot of this will be based on what I need when I actually put this into real use, since right now it's partly hypothetical. In any event, I'm looking forward to finding places where I can use this instead of explicitly coding to Notes API objects for sure.

Last year, I wrote about how I built images to use Testcontainers to run tests against a Liberty app that uses a Domino runtime. In that situation, I used the Domino Docker image from Flexnet but then pulled out the program files and stock data, mixed with pre-configured server support files from the repository.

Recently, though, I've had a need to have a similar setup, but altered in three ways:

1. It should fully run Domino, not just use the data and program files for the runtime in Liberty
2. It should not require pre-populating a server ID, notes.ini, and names.nsf from outside - that is, it should be self-configuring
3. It should also have an extra component installed, one that must be installed after Domino is configured but before the image is fully built
4. On the next launch, I need a post-install agent to run for final configuration, and the tests need to wait on that

Additionally, it should still be runnable with the same basic tools - the image should be built and the container started and destroyed by automated tests. This stricture comes into play in weird ways.

One-Touch Setup

The second requirement - that the container be self-configuring - is handled adroitly by One-Touch Setup, the feature in Domino 12 and above that lets you specify a configuration JSON file. I used this here in basically the same way as I described in that post: the script sets up and certifies a throwaway domain with a known admin username + password, and also deploys a few NTFs on first proper launch. Since this server intentionally doesn't communicate with the outside world, I don't need to provide any external files like a certificate authority or server ID.

Switching the Baseline

Initially, I continued to use the official image from Flexnet. However, I had run into some trouble with multi-stage builds using this earlier. I lack enough Docker knowledge to be sure, but my suspicion is that it declares /local/notesdata as an expected externally-provided volume. This on its own is fine, but it interacts oddly with automated container building. The trouble comes in when you do something like this:

1
2
3
4
5
6
7

FROM domino-docker:V1201_11222021prod
ENV SetupAutoConfigure "1"
ENV SetupAutoConfigureParams "/local/domino-config.json"
COPY --chown=notes:wheel domino-config.json /local/
RUN /local/start.sh

RUN /some/script/that/uses/notesdata

By the time it hits that second RUN line using automated build mechanisms, /local/notesdata is depopulated. I'm not sure why this is different from a command-line docker build, but it is what it is.

Fortunately, the "community" version of the image builder from https://github.com/IBM/domino-docker doesn't exhibit this behavior. I had been considering switching over already, so this made the decision all the easier.

Breaking Up Setup Into Stages

With this change in hand, I was able to more-properly break up the setup into multiple stages. This is important both for requirement #3 above and because it allows Docker to cache intermediate results. Though I want the server to auto-configure itself when building, I don't need the results of that to be different every run, and thus I can save tons of time in subsequent launches if I handle these caches well.

So my Dockerfile started to look something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

FROM hclcom/domino:12.0.1
# Relay Domino output to the container logs
ENV DOMINO_DOCKER_STDOUT "yes"
ENV SetupAutoConfigure "1"
ENV SetupAutoConfigureParams "/local/DominoAutoConfig.json"

COPY --chown=notes:wheel domino-config.json /local/DominoAutoConfig.json
COPY --chown=notes:wheel notesdata/* /local/notesdatatemp/

RUN /domino_docker_entrypoint.sh

# Copy the app executable and support scripts
USER root
COPY appinstall /local/appinstall
RUN chmod +x /local/appinstall/install.sh && /local/appinstall/install.sh

# Back to notes user for the Domino entrypoint
USER notes

But here I hit another distinction between docker build and the automated mechanisms: that RUN /domino_docker_entrypoint.sh line would execute and get to the point where it emits "Application configuration completed successfully", but then would not actually exit properly. Again, I'm not sure why: the JSON file tells the server to not launch after configuration, and indeed it doesn't, but the script just doesn't relinquish control - but does when built from the command line.

So I rolled up my sleeves and wrote a wrapper script to kill the process when it's known to be done:

1
2
3

#!/usr/bin/env bash
/domino_docker_entrypoint.sh > /tmp/domsetup &
until tail -f /tmp/domsetup | grep -q "Application configuration completed successfully"; do : sleep 1; done

This runs the setup process, redirecting the output to a temp file, in the background. Then, it watches that file for the known ending text and exits when observed. A little janky in multiple ways for multiple reasons, but it works. That allows image build to progress normally to the next step in all environments, caching the results of the initial server setup. I replaced the RUN /domino_docker_entrypoint.sh line above with copying in and executing this script, and all was well.

Post-Install Agent

After the "appinstall" step, I have the peculiar need to then run code in a Notes context to fiddle with some components that aren't configured earlier. For now, I've settled on writing an agent that runs on server start, then signing and enabling it in the domino-config.json file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

{
    "action": "create",
    "filePath": "postinstall.nsf",
    "title": "Post Install",
    "templatePath": "/local/notesdatatemp/postinstall.ntf",
    "signUsingAdminp": true,
    "agents": [
        {
            "name": "PostInstall",
            "action": "sign"
        },
        {
            "name": "PostInstall",
            "action": "enable"
        }
    ]
}

Originally, I had this agent emit the text "postinstall done" when it finished, so that the Testcontainers runtime could look for that to know when it's safe to execute tests. However, this added a good while to the launch stage: at this point, launching the container has to wait on final post-install tasks from Domino, then signing the DB with adminp, then actually executing the agent. This added about a minute to the test pre-run time, and thus was a prime target for further caching.

So I altered the agent to check to see if it actually needs to work and then, if so, shuts down the server when it's done:

1
2
3
4
5
6

Session session = getSession();

if(needToDoWork()) {
    doWork();
    session.sendConsoleCommand(session.getServerName(), "q");
}

Then, I altered my Dockerfile to amend the end bit:

1
2
3
4
5
6
7
8
9

# ...snip
# Back to notes user for the Domino entrypoint
USER notes

# Run the server once to wait for postinstall to execute, then shut down
WORKDIR /local/notesdata
RUN /opt/hcl/domino/bin/server

# Now let the entrypoint launch again, without requiring further configuration

Again: janky, but it works. Now, all of the setup stages are cached on subsequent runs.

Building the Image in Testcontainers

In my original post, I showed using dockerfile-maven-plugin to build the image just before executing the tests. This works, but it complicated the pom.xml a bit and meant that running the tests in an IDE meant first running a Maven build. Not the end of the world, but not ideal.

Fortunately, Testcontainers can also build images. That meant that, rather than building the image in Maven and then re-using the same-named container in Java, I could do it all Java-side. To do this, I created a subclass of GenericContainer to centralize the configuration of the container:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

package example;

import java.io.IOException;
import org.testcontainers.containers.GenericContainer;
import org.testcontainers.containers.wait.strategy.HttpWaitStrategy;
import org.testcontainers.images.builder.ImageFromDockerfile;

public class DominoAppContainer extends GenericContainer<DominoAppContainer> {
    public static class DominoAppImage extends ImageFromDockerfile {
        public DominoAppImage() {
            super("example-app-it-testcontainers:1.0.0", false);
            withFileFromClasspath("Dockerfile", "/docker/Dockerfile");
            withFileFromClasspath("domino-config.json", "/docker/domino-config.json");
            withFileFromClasspath("firstrun.sh", "/docker/firstrun.sh");
            withFileFromClasspath("notesdata/exampledata.ntf", "/docker/notesdata/exampledata.ntf");
            withFileFromClasspath("notesdata/postinstall.ntf", "/docker/notesdata/postinstall.ntf");
            withFileFromClasspath("appinstall/install.sh", "/docker/appinstall/install.sh");
            withFileFromClasspath("appinstall/appinstall.jar", "/docker/appinstall/appinstall.jar");
        }
    }

    public DominoAppContainer() {
        super(new DominoAppImage());

        withImagePullPolicy(imageName -> false);
        withExposedPorts(80);
        waitingFor(
            new HttpWaitStrategy()
            .forPort(80)
            .forStatusCodeMatching(code -> code >= 200 && code < 400)
        );
        withLogConsumer(frame -> {
            switch (frame.getType()) {
                case STDERR:
                    try {
                        System.err.write(frame.getBytes());
                    } catch (IOException e) {
                        e.printStackTrace();
                    }
                    break;
                case STDOUT:
                    try {
                        System.out.write(frame.getBytes());
                    } catch (IOException e) {
                        e.printStackTrace();
                    }
                    break;
                default:
                case END:
                    break;
            }
        });
    }
}

It's a little fiddlier in that now I need to enumerate each classpath resource to copy in, but that also makes it all the more portable. I removed the dockerfile-maven-plugin execution from the pom.xml and switched to this instead. Since I name the image and tell it to not auto-delete on completion, this retained the desired caching behavior.

Conclusion

Overall, this whole process brought the test pre-launch time (after the first run) down from 3-5 minutes to about 20 seconds while also reducing the split between the Maven config and Java. Much more bearable when making small tweaks and re-running the suite, and it makes it a bit more explicable what's going on.

For a good while now, I've been making use of the HTTPEnableConnectorHeaders notes.ini property in Domino to allow my reverse proxies to have Domino "see" the real remote system. Though this feature is coarse-grained and is best paired with some tempering, it's served me well when used on appropriately-configured servers.

Unfortunately, HCL saw fit to remove this feature in 12.0.1, declaring it a security vulnerability. I don't think this made it into the release notes as such, but did eventually get patched into the "Components no longer included in this release" page for V12.

Obviously, the true problem here is that it makes my blog entries retroactively less useful. However, a secondary issue is that it will damage your applications in two main ways if you were making use of these headers:

Authentication

The $WSRU header allowed you to specify a username to act as for the duration of the web request, and this would be honored from the core: all legacy and Servlet components would see that as the true authenticated user. This header is essentially a quick-and-dirty SSO mechanism, and works similarly in that way to an LTPA token.

Fortunately, this header has some workarounds, though proper ones would likely require diving into some C:

1. If you have a simple situation where you were using it to log in as a single known user, you could switch to sending HTTP Basic auth with that user's credentials
2. If the server in front of Domino happens to have been written by IBM, you could potentially spin your own LTPA tokens on that side and have them trusted by Domino
3. You could establish your own exchange between the proxy server and Domino to have Domino generate an SSO token for an arbitrary name for you and then pass that along (note: if you do that, use Domino JNA to do the heavy lifting)
4. You could write a DSAPI filter to implement whatever type of authentication you like

With any of those, Domino will trust the user you provide it to the same extent it previously would trust the $WSRU header when configured to do so.

Note: if you do implement your own authentication, don't just re-use $WSRU. It appears that part (all?) of the change on the Domino side is to hard-strip the $WS* headers from the incoming request.

Remote IP Address, etc.

The other use, which I've used in a good many deployed apps, is to use $WSRA and friends to tell Domino what the true remote IP address is. When you do this, CGI item values like REMOTE_ADDR will reflect the external user's IP address instead of the proxy server's.

Since 9.0.1FP8, there's been a notes.ini property - HTTP_LOG_ACCESS_XFORWARDED_FOR - that will tell Domino to pay a little attention to the de-facto standard X-Forwarded-For header that proxies often use to tip an app server off to the proxy hops a request took to get to its destination. From what I can tell, this support remains limited to just writing to a new field in log entries in domlog.nsf. The in-app CGI variables will still reflect the reverse proxy's address.

Unfortunately, as I found when I was trying to make a DSAPI filter to properly trust that header, these request values are not writable in such filters. It might hypothetically be possible to get some of this with EM triggers juggling fields around for classic use and futzing with the low levels of the XPages stack, but even I wouldn't resort to that.

The upshot is that if, for example, you're storing REMOTE_ADDR in created documents or making use of ServletRequest#getRemoteHost, you'll have to alter your applications to also consider X-Forwarded-For, and the same goes for any other such fields you were using.

As above, if you do this, you can't use $WSRA et al, as I don't think they'll be visible in your app.

As I've been adding new technologies to and talking about the XPages Jakarta EE project, I've kind of danced around a major missing layer: data access.

Technically, the toolchain has provided Domino data access all along, by way of having the same contextual sessions and database as XPages. You could use those to access whatever data you want, and they'd do the job as well as they ever do (c'est-?-dire: poorly). Beyond that, though, there's no equivalent to the (questionable) xp:dominoDocument and xp:dominoView components of XPages, and definitely no pre-provided object-to-database mapper.

The answer is pretty clear: Jakarta NoSQL. This API isn't quite finalized, but it's been usable for a long time: I wrote a Darwino driver for it years ago, and that driver is powering this very blog. I also wrote a Domino driver years ago, but it was very much a proof-of-concept: since it pre-dated DQL, it used formula queries for everything, and thus would scale extremely poorly. It was a nice exercise, but not anything useful.

For XPages JEE, I decided to take another swing at that. The implementation of the driver will warrant a tale on its own, but for now I'd like to focus on the DQL side of it.

DQL

I talked a bit about DQL when it came out, back when it wasn't well-understood, but since then I haven't actually had much occasion to put it to use. For the times I've needed complex Domino data access since then, it's been built on pre-existing operations on top of views. While adding DQL has been something I've considered from time to time, it'd never hit the threshold of being worth it: our needs involve extracting tons of data to bulk send it to service clients, and so views have remained necessary. While we could in theory alter our querying and filtering to select documents and project those selections onto the views, it'd have been a lot of work for partial benefits.

DQL itself has gotten more capable in the intervening years, and just on its own it's a perfect match for JNoSQL needs. Since all JNoSQL operations are sent to the driver as either individual doc IDs or an arbitrary query, something like DQL is required, and it's up to the task now.

It's half of the story, though. What DQL (by way of the DominoQuery object) gives you is a DocumentCollection, effectively just the list of note IDs. You can, as I'd hypothesized about doing, apply that against a view to extract data, but that still requires you to separate out the act of view management from the act of doing queries. If you want to have data sorted or categorized, you would still have to create an equivalent or superset view.

QueryResultsProcessor

So that's where the addition of QueryResultsProcessor comes in. QRP is technically distinct from DQL - you can use it to process arbitrary document collections, for example - but they're certainly a conceptual match. If you're comparing it to a SQL statement, DQL is the "FROM foo" and "WHERE x" parts, while QRP is the "SELECT a,b,c", "ORDER BY y", and "GROUP BY z" parts.

The general way it works is that you:

1. Create a QueryResultsProcessor from a Database instance (as opposed to Session - this distinction becomes important later)
2. Feed it sources of documents: DQL queries or arbitrary document collections
3. Add any desired columns to extract data. These are Domino-style columns, and you can also specify sorting and categorization here, as you would when building a view
4. Since data may come from multiple databases, you can also customize column formulas to account for that
5. Execute the process and retrieve the results, currently either as JSON or as a "view". More on these "views" later

When I first heard about QRPs, I had a concern with step 2: I'd thought that you could only pass a built DocumentCollection to the processor, which would significantly limit the room for Domino to add behind-the-scenes efficiencies. However, my fears were unfounded; the ability to pass in a DominoQuery object and the DQL directly and let the QRP execute it means that HCL is free to do whatever they want to make it fast. That's the sort of thing that makes SQL queries potentially so stupidly efficient: because you're just asking the database for results, the DB is free to optimize the heck out of them. This pairing potentially brings that to Domino, and that's what makes it important.

JSON Output

The executeToJson method is pretty straightforward if a somewhat-peculiar choice. It has no parameters, and returns the results of your query as reasonably-formatted JSON. It's unfortunate that this returns a String and not an InputStream, which adds some inherent inefficiency to dealing with it on the Java side, but that will only really hurt with very-large data sets.

Along with the requested fields, formula results, and aggregations, the document entries include the note ID (oddly in "formula" format) and the database file path, so you can use that to open up the document.

Anyway, this is a workmanlike format and can be potentially just sent to REST clients directly, though it'd be good form to at least strip out the DB paths and note IDs.

View Output

Now here's the spicy one. The executeToView method stores the results in a very-weird type of view. This has a few big advantages over the JSON mechanism:

• The view persists in the database, up to a number of hours you specify programmatically. This allows you to essentially offload some extra caching to the database, which is ideal
• You can use ViewNavigator and other efficient mechanisms to work with the view data, meaning you don't have the "here's a big result blob in memory" problem you have with the JSON format
• Since it's a "view", anything that works with view data can work with it. This is presumably the reason it's implemented this way at all, rather than as some new kind of entity - building on existing mechanisms
• The "anything that works with view data" doesn't just mean things like ViewNavigator: it also means the Notes client and view data sources

Now, these "views" have a lot of weird characteristics. It's useful to see the specifics listed out like that, but they all derive from a core lesson to ingest:

This is not a stored query; it is a cached result.

These views are not auto-updated, nor is there any mechanism I know of to refresh them outside of deleting and re-creating them. They're equivalent in concept to if you took the JSON from the first type and stored it in a document somewhere: it'll only change if you change it. The only way Domino will act on them is to delete them when they're expired.

Anyway, the data in these views is the same data that would go to the JSON format, just stored as Notes collation data instead of a string. It contains columns, potentially categorized and aggregated, for the data you requested, as well as hidden "$DBPath" and "$NoteID" columns at the end. The entry-level note ID (the one from entry.getNoteID()) is arbitrary and intended to not represent an actual document - since, after all, the documents may come from distinct databases. I've found the value of entry.getUniversalID() to be the doc's original UNID, but this is best treated as not a guarantee and so should not be used.

Designer Rights

So here's a fun catch: though any Reader can perform a query, you need Designer access to create a view. This seemed like a problem to me at first, since I'd want the generated results to be from a specific user for reader-field purposes, but it's not really an impediment, at least when you're in an environment like XPages.

Above, I mentioned that the fact that you create a QueryResultsProcessor object from a Database is important, and this is one of the reasons why. Though traditionally you wouldn't mix descendants of session and sessionAsSigner together, there's no actual rule against it. You can re-open your context database with sessionAsSigner, make a QRP object from that, and then feed it a DominoQuery object created from the non-signer database:

1
2
3
4
5
6
7
8

Database database = ExtLibUtil.getCurrentDatabase();
Session sessionAsSigner = ExtLibUtil.getCurrentSessionAsSigner();
Database databaseAsSigner = sessionAsSigner.getDatabase(database.getServer(), database.getFilePath());

DominoQuery dominoQuery = database.createDominoQuery();
QueryResultsProcessor qrp = databaseAsSigner.createQueryResultsProcessor();
qrp.addDominoQuery(dominoQuery, "some DQL", null);
View result = qrp.executeToView("some view name");

Because the QueryResultsProcessor uses the provided DominoQuery object as the "engine" for the DQL search, the query will use the normal user's rights while the processing will use the signer rights.

Naming and Expiring Results

As seen there, you have to name your views. While you could in theory use this mechanism to kind of manage your own views for general use and name them things like "People By First Name" or whatever, you'll likely want to work with them programmatically and name them based on your query input.

In the case of this JNoSQL driver, I compute a predictable-from-input hash-based name from the name of the creating class, the current user, and the sort/skip/limit attributes of the incoming query. You could really do whatever you want here, but having at least some sort of hash like this is likely the way to go.

Now there's the matter of detecting when you need to refresh the data. In some applications, it may suffice to go with the "expire in X hours" parameter when creating the view, though that's extremely coarse and only really useful on its own for specific needs (like a daily report).

The tack I took here was to try to do an efficient check of view creation time compared to the last data modification time from the source database. The Database class only has a "last modified" time in general, but I can't very well use that when my results caches are being added as design elements: a second distinct query would "invalidate" the first even when the data hasn't changed. There might be a proper way to get this in lotus.domino, the NAPI has a wrapper for NSFDbModifiedTimeByName: NotesSession.getLastDataModificationDateByName. That lets you get the last data-mod time in epoch seconds, and you can then compare that to the creation time of the view.

While it's unfortunate that you have to remove the view outright to refresh it instead of doing a delta update like NIF would do, I get it, and it's generally fast enough. Plus, there's enough hand-wavy stuff going on with feeding the DQL query to the QRP that Domino would be free to secretly retain results for a bit and do deltas internally if it so desires.

Storing Result Views

The other interesting aspect of creating a QRP object from a Database and not a Session is that that DB serves as the destination to house the views. While in a single-DB environment it would seem very natural to just store the views in the same place as the data, there's no particular requirement to do so. Moreover, if you're querying multiple databases, you're naturally not going to do this for all docs anyway, so you'll be forced to conceptualize this anyway.

Now, personally, I'm fine with a bunch of temporary machine-named views hanging out in the NSF (especially since the names are wrapped in parentheses to hide them from default UI listings), I can see why it could be annoying. For one, these views sync to an ODP in Designer, which I put in as an Aha idea to change, but might actually rightly be called a bug. Beyond that, while these views won't meaningfully contribute to NIF's workload (since NIF will skip them), they're unsightly and would get in the way if you're trying to tend to the design of your NSF like a garden.

So you might want to have a side database to store these views, and this could also be a way to get around the "needing Designer access" requirement if you're in an environment where you don't have a signer session. In the Notes client, you could store the results in a local NSF; on the server, you could make a "scratch" NSF somewhere to house them, and then add readers to the view design note when doing so to prevent leaking data across users and apps.

Conclusion

Anyway, this is all pretty neat. Reusing view design elements to just be static containers for collation data is weird, but I get the practical reasons why it makes sense. Importantly, this pairing solves some very-real problems with querying and extracting data from Domino. For example, if you do all of your querying via this route, you can use DQL's "EXPLAIN" capability to actually get some insight into database performance for once. You could imagine having an optional mode where you log the EXPLAIN results and execution times for all queries your app is performing, and then manually create "index" views to fix hotspots. It's quite satisfying to finally get that kind of ability in Domino. It'd be neat if that also came to QueryResultsProcessor.

I'm looking forward to expanding the JNoSQL driver further and then either using that directly in client work or adapting the code I use there. I'll definitely add such a logging capability, which will go a long way to put some numbers to the "feels slow" problem that can crop up. Beyond that, barring any show stoppers, I'm thoroughly excited by the prospect of moving away from fetching explicitly-named views in code and switching to an idiom of querying the pool of documents and letting the database make it work for me.

I've just released version 2.2.0 of the XPages Jakarta EE project, and this contains some fun additions.

Part of what makes this project satisfying to work on is the fact that it has a clear maximum scope: there are only so many Jakarta EE and MicroProfile specifications. Moreover, their delineated nature makes for satisfying progress "chunks": setting aside any later tweaks and improvements, each new spec is slotted into place in a gratifying way.

For this release, I focused on adding a number of capabilities from MicroProfile. MicroProfile is an interesting beast. Its initial and overall goal is to create a toolkit geared towards writing microservices, and it does this by taking a subset of Jakarta EE specs and then adding on its own new capabilities. Now, personally, I don't give two hoots about microservices, but the technologies added to MicroProfile aren't really specific to those needs, and most of them really just fill in gaps in the JEE lineup in ways that are just as useful to bloated monoliths as they are to microservices.

From the list, I added in five new ones, in addition to the already-present OpenAPI generator. Each one is extremely powerful and deserves a bit of discussion, I feel.

Config

The MicroProfile Config spec is a way to externalize your app's configuration and then inject configuration values via CDI. For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

@ApplicationScoped
public class ConfigExample {
    @Inject
    @ConfigProperty(name="java.version")
    private String javaVersion;
    
    @Inject
    @ConfigProperty(name="xsp.library.depends")
    private String xspDepends;
    
    /* use the above */
}

When this bean is used, the javaVersion value will be populated with the java.version system property and xspDepends will get the list of libraries used by the app from Xsp Properties. The latter also shows the pluggable nature of the spec: as part of adding it to the project, I added a custom configuration source to read from the Xsp Properties of the app, as well as one to read from notes.ini. One could in theory (and I absolutely will) write a custom source to read configuration from a view or other Notes-type source.

Rest Client

One of the ways that Domino has long been deficient has been in accessing remote REST services. There's some stuff in the ExtLib, I think, and then there are HTTP primitives in LotusScript, but they don't really do much work for you. Java on its own has URLConnection and friends, and that works, but it's basically as low level as LotusScript.

What the Rest Client spec does is build on familiar Jakarta REST annotations (@Path, @GET, etc.) to allow you to declare how your service works and then access it like a normal Java object. For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

@ApplicationScoped
public class RestClientExample {
    public static class JsonExampleObject {
        private String foo;
        
        public String getFoo() {
            return foo;
        }
        public void setFoo(String foo) {
            this.foo = foo;
        }
    }
    
    @Path("someservice")
    public interface JsonExampleService {
        @GET
        @Produces(MediaType.APPLICATION_JSON)
        JsonExampleObject get(@QueryParam("foo") String foo);
    }
    
    public Object get() {
        URI serviceUri = URI.create("http://example.com/api/v1/");
        JsonExampleService service = RestClientBuilder.newBuilder()
            .baseUri(serviceUri)
            .build(JsonExampleService.class);
        JsonExampleObject responseObj = service.get("some value");
        Map<String, Object> result = new LinkedHashMap<>();
        result.put("called", serviceUri);
        result.put("response", responseObj);
        return result;
    }
}

Here, I've defined an imaginary remote resource available at a URL like "http://example.com/api/v1/someservice?foo=some%20value" and created an interface with REST annotations to define its expected behavior. The MP Rest Client takes it from there: it converts provided arguments to the expected data types (query parameters, body as JSON/XML, etc.), makes the HTTP call, parses the response into the expected type, and returns it to you, while you get to use it like any old Java object.

It's extremely convenient.

Fault Tolerance

The Fault Tolerance spec allows you to add annotations to methods to handle and describe failure situations. That's pretty vague, but an example may help:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

@ApplicationScoped
public class FaultToleranceBean {
    @Retry(maxRetries = 2)
    @Fallback(fallbackMethod = "getFailingFallback")
    public String getFailing() {
        throw new RuntimeException("this is expected to fail");
    }
    
    private String getFailingFallback() {
        return "I am the fallback response.";
    }
    
    @Timeout(value=5, unit=ChronoUnit.MILLIS)
    public String getTimeout() throws InterruptedException {
        TimeUnit.MILLISECONDS.sleep(10);
        return "I should have stopped.";
    }
    
    @CircuitBreaker(delay=60000, requestVolumeThreshold=2)
    public String getCircuitBreaker() {
        throw new RuntimeException("I am a circuit-breaking failure - I should stop after two attempts");
    }
}

There are three capabilities in use here:

• @Retry and @Fallback allow you to specify that a method that throws an exception should be automatically re-tried X number of times and then, if it still fails, the caller should instead call a different method. This could make sense when calling a remote service that may be intermittently unavailable.
• @Timeout allows you to specify a maximum amount of time that a method should be allowed to run. If execution exceeds that amount, then it will throw an InterruptedException. This could make sense when performing a task that normally takes a short amount of time, but has a known failure state where it stalls - again, calling a remote service is a natural case for this.
• @CircuitBreaker allows you to put a cap on the number of times per X milliseconds that a method is called when it fails. This could be useful for a situation where a method might fail for a user-generated reason (say, a user attempted to modify a document they can't edit) but also might fail for a systemic reason (say, a DB is corrupt) and where repeated attempts to perform the task might be damaging or otherwise very undesirable.

The cool thing about these annotations is the way they make use of CDI's capabilities. If you @Inject this bean into another class, you'll simply call bean.getFailing(), etc., and the stack will handle actually enforcing the retry and fallback behavior for you. You don't have to write any code to handle these checks beyond the annotations.

Metrics

The Metrics API allows you to annotate your objects to tell the runtime to track statistics about the call, such as invocation count and execution time. For example:

1
2
3
4
5

@GET
@SimplyTimed
public Response hello() {
    /* Perform the work */
}

Once the method is marked as @SimplyTimed, you can retrieve statistics at the /xsp/app/metrics endpoint in your NSF, which will get you a bunch of information, including lines like this:

1
2
3
4

# TYPE application_rest_Sample_hello_total counter
application_rest_Sample_hello_total 2.0
# TYPE application_rest_Sample_hello_elapsedTime_seconds gauge
application_rest_Sample_hello_elapsedTime_seconds 0.0025678

That's OpenMetrics format, which means you could consume this data in visualizer tools readily.

Health

And speaking of data for visualizers, that brings us to the last MP spec I added in this version: Health. This API allows you to write classes that will be used to query the health of your application: whether individual components are up or down, whether the app is started ready to receive requests, and any custom attributes you want to define. Now, admittedly, an app on Domino will usually either be "100% up" or "catastrophically down", so making use of this spec will take a little finesse. Still, it's not too hard to envision using this to emit some dashboard-type information. For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

@ApplicationScoped
@Liveness
public class PassingHealthCheck implements HealthCheck {
    @Override
    public HealthCheckResponse call() {
        HealthCheckResponseBuilder response = HealthCheckResponse.named("I am the liveliness check");
        try {
            Database database = NotesContext.getCurrent().getCurrentDatabase();
            NoteCollection notes = database.createNoteCollection(true);
            notes.buildCollection();
            return response
                .status(true)
                .withData("noteCount", notes.getCount())
                .build();
        } catch(NotesException e) {
            return response
                .status(false)
                .withData("exception", e.text)
                .build();
        }
    }
}

While "count of notes in an NSF" isn't usually too useful, you can imagine replacing that with something more app-specific: open support tickets, pending vacation requests, or the like. You could also combine this with the Rest Client spec to make a coordinating NSF with no business logic that makes calls to your other, more-likely-to-break apps to check their health and report it here.

Once you write these checks, the runtime will automatically pick up on them and make them available at /xsp/app/health and some sub-paths:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

{
    "status": "DOWN",
    "checks": [
        {
            "name": "I am the liveliness check",
            "status": "UP",
            "data": {
                "noteCount": 63
            }
        },
        {
            "name": "I am a failing readiness check",
            "status": "DOWN"
        },
        {
            "name": "started up fine",
            "status": "UP"
        }
    ]
}

Other Improvements

Beyond those major additions, I made some improvements to clean some aspects up and refine some details (such as making REST services emit stack traces as JSON in the response instead of printing to the console).

Feature-wise, the main addition of note is support for the @RolesAllowed annotation on REST services, which allows you to restrict access by Notes-ACL-type name:

1
2
3
4
5

@GET
@RolesAllowed({ "*/O=SomeOrg", "LocalDomainAdmins", "[Admin]" })
public Object get() {
    // ...
}

When a user doesn't match one of those names-list entries, they're given a 401 response. You can also use the pseudo-name "login" to require that the user be at least logged in and not Anonymous.

What's Next?

Well, I'm not sure. All of the above have immediate uses in my work, so I plan to get rid of some of our old workarounds for this sort of thing and bring in the new abilities.

Beyond that, there are two shipped MicroProfile specs remaining: OpenTracing (which I don't know what it is, but maybe it's useful) and JWT Propagation (which would only make sense when paired with a whole JWT thing).

There's also MP GraphQL, which seems to be like MVC in Jakarta EE, where it's a solid spec but not part of the official standard yet. I may take a swing at it just because it may be easy to add, though for my client needs we already have a more-dynamic GraphQL implementation.

Back on the Jakarta side, the spec list contains quite a few that aren't in there, though a lot of those are either essentially not applicable to Domino (like Authentication or WebSocket) or are primarily of interest to legacy applications (like Enterprise Beans or XML services).

I'd also like to do some breaking reorganization. Most of these specs are added as individual XPages libraries, but that's gotten really unwieldy. Moreover, I'm not sure what the situation would be where you'd want to include, say, REST but not CDI. I'll probably look into making it a single "make my dev experience good" library to select. That'll take some work, though, since there's a lot of OSGi-dependency stuff to balance there.

Beyond that, it'll be refinements, bug fixes, and improvements for use in OSGi-based apps. I have a bunch of things tracked and that'll give me plenty to do as I have time.

Yesterday, I had a problem. I was trying to get MicroProfile Config working inside an NSF to add to the XPages Jakarta EE project, and I was severely blocked by odd behavior.

To describe that, I'll lightly cover what MP Config is. It's a CDI extension that allows you to annotate properties on a bean to indicate that they're intended to come from an available configuration source - often a .properties file in the project, but it's a pluggable system. Your bean will look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

package example;

// ...

@ApplicationScoped
public class ConfigExample {
    @Inject
    @ConfigProperty(name="java.version", defaultValue="unknown")
    private String javaVersion;
    
    /* other methods go here */
}

The idea is that you'll then have a properties file or environment variable to fill in the value, allowing you to separate your configuration from the implementation in a consistent way. Here, I'm making use of the fact that a default provider looks up Java system properties, so I could just get it working before investigating adding providers.

Since I'd already added CDI and a CDI-based extension in the form of MVC, I figured this would be easy.

The Problem

The problem I hit, though, was bizarre. CDI would identify the bean above, but would hit this problem:

1
2
3
4
5

org.jboss.weld.exceptions.DeploymentException: WELD-001408: Unsatisfied dependencies for type String with qualifiers @Default
  at injection point [UnbackedAnnotatedField] @Inject private example.ConfigExample.javaVersion
  at example.ConfigExample.javaVersion(ConfigExample.java:0)
WELD-001475: The following beans match by type, but none have matching qualifiers:
  - Producer Method [String] with qualifiers [@Any @ConfigProperty] declared as [[UnbackedAnnotatedMethod] @Dependent @Produces @ConfigProperty protected io.smallrye.config.inject.ConfigProducer.produceStringConfigProperty(InjectionPoint)]

The gist of this is that it noticed that the javaVersion property is supposed to be an injected property, but it had no idea what the source should be. It did know about the MicroProfile provider, which handles @ConfigProperty, but it couldn't put two and two together.

I banged my head against this for a while, and eventually determined that the class as loaded from the NSF is stripped of the @ConfigProperty annotation outright. Other annotations, such as @Inject and even custom annotations, would remain, but not @ConfigProperty. I wrestled with OSGi dependency chains for a while, to no avail.

The Enemy

Eventually, I found the core, and it was an old nemesis of mine. It's this method in com.ibm.xsp.util.ClassLoaderUtil:

This field is called by the ClassLoader used in an NSF to ensure that certain classes, by name prefix, cannot be loaded by code coming from an NSF. The last three lines there make a sort of sense: Domino is supposed to be an app container for XPages apps, and ideally it's not a simple process for an app to break out of its container to muck about in the parent environment. Fair enough. The NAPI line is presumably there because IBM wanted to protect developers from themselves, even though Notes devs had been making unauthenticated calls to C APIs for freaking ever.

It's the first two, and specifically the first, that are the source of my trouble. Those prohibitions are presumably meant to isolate XPages apps from the fact that they live in an OSGi world, with the assumption that anything beginning with org.eclipse. refers to something like org.eclipse.core.runtime, the OSGi system bundle.

And this is the issue. MicroProfile is not in any way related to OSGi, but it sure is an Eclipse project. Accordingly, the class name of @ConfigProperty is org.eclipse.microprofile.config.inject.ConfigProperty, and thus cannot be loaded from an NSF.

Attempted Workarounds

So I considered my options.

One was to fork MP Config and rename the packages. That would work, but it would defeat the portability goals of the XPages JEE project, and would also just be a hassle - I've already had to fork a few specs, and each new one adds to the maintenance burden. That remained an option, but it would be a last resort.

My next idea was to wrap around the ModuleClassLoader class used by NSFComponentModule for class loading purposes. This class is blessedly non-final, and so in theory I could look at the instance in a module and swap it out with a replacement. I tinkered with this a bit, but the trouble became the way it's layered, with a DynamicClassLoader private class within - something harder to subclass. In theory, I could reproduce the behavior of it wholesale, but that would be both fragile (if the implementation ever changes) but also verging on if not outright illegal (it's one thing to be API-compatible, and another to reproduce the internal functionality). After some wrangling, I decided to look elsewhere.

The True Workaround

I realized eventually that I don't really care about ModuleClassLoader as such: it does its job fine, and it's only the response that it gets from ClassLoaderUtil that is the problem. If I could change that, I would be set.

I've used the Javassist project here and there for a long time, ever since its inclusion in ODA for one reason or another. It's a handy toolkit, and notably includes the capability to alter a method implementation on the fly. There's my loophole.

The reason this kind of thing can work is related to how Java handles classes and calls between them. For all intents and purposes, you can consider a method call from one bit of Java to another to be a string-based lookup, saying "find a class named X and a method named Y, and then execute it". The "find" part there is much looser than you might think. It's easy to think of class references like C static linking, but they're really not. When code asks for a class, it asks the context ClassLoader, and that object can do basically whatever the heck it wants to find it, as long as it eventually emits a Class that the runtime can deal with.

Javassist's manipulation makes use of the fact that classes are generally eventually just a bunch of bytes, and you can do whatever you want with a bunch of bytes. Using Javassist, it's fairly simple to, once you have a handle on the class, alter the method. Truncated, that's:

1
2
3
4
5
6

ClassPool pool = /* build a ClassPool that can load the class */;
CtClass cc = /* get the class from the pool */;
cc.defrost();
CtMethod m = cc.getDeclaredMethod("checkProhibitedClassNames");
m.setBody("{ return false; }");
Class<?> result = cc.toClass();

And this works, as far as it goes: I now have a Class version of ClassLoaderUtil that skips the onerous check.

The trouble now was to get this to be actually used by other classes. Generally, once a ClassLoader loads a class, it's difficult to feed it another version unless it's designed to do so: most ClassLoader implementations, including those used here, are designed to read and emit classes by their own rules, not have new data fed into them.

I tried digging through the Eclipse OSGi ModuleClassLoader (distinct from the NSF ModuleClassLoader) for entrypoints and had some initially-promising work with Eclipse's internal ClassLoaderHook type, but eventually determined that this would require more patching than I'd want, if it was possible at all.

I also considered using Java's instrumentation capabilities to intercept class loading, but that would require setting up a special Java agent in the launch parameters, which would be too onerous.

But then I remembered something I had heard about when looking into getting ServiceLoader to work with OSGi: a concept in the OSGi spec called "weaving".

OSGi Weaving

I had noted that this concept existed, but set it aside in large part due to how esoteric it sounds: the term "weaving" makes it sound like it's a way to interact with the threads of fate or something, which is evocative but not something that seems immediately useful.

What it really is, though, is an OSGi-friendly version of the above: when the OSGi runtime goes to load a class from a bundle, it reads the data but then gives any such listeners an opportunity to manipulate the code before it's actually reified into a class. This is how the ServiceLoader mediator does its thing: it looks for ServiceLoader calls during loading and re-"weaves" them to run through OSGi instead.

This was perfect: it provides exactly the hook I want and it does it in a clean, spec-based way, without having to do weird reflection to reassign object properties or anything.

The Implementation

So I went about writing such an implementation. All the pieces are there on Domino, and the mechanism for registering a WeavingHook is something I'd done before in Open Liberty: it's a type of OSGi service that you can register and manage in an Activator class. It's also the sort of thing that would work well with Declarative Services, but Domino doesn't have a DS handler installed and I figured I didn't need to solve that quite yet.

So I wrote a WeavingHook implementation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

public class UtilWeavingHook implements WeavingHook {
    @Override
    public void weave(WovenClass c) {
        if("com.ibm.xsp.util.ClassLoaderUtil".equals(c.getClassName())) {
            ClassPool pool = new ClassPool();
            pool.appendClassPath(new LoaderClassPath(ClassLoader.getSystemClassLoader()));
            CtClass cc;
            try(InputStream is = new ByteArrayInputStream(c.getBytes())) {
                cc = pool.makeClass(is);
            } catch (IOException e) {
                throw new UncheckedIOException(e);
            }
            cc.defrost();
            try {
                CtMethod m = cc.getDeclaredMethod("checkProhibitedClassNames");
                m.setBody("{ return false; }");
                c.setBytes(cc.toBytecode());
            } catch(NotFoundException | CannotCompileException | IOException e) {
                new RuntimeException("Encountered exception when weaving ClassLoaderUtil replacement", e).printStackTrace();
            }
        }
    }
}

This builds on the above Javassist usage to now load the class from the byte array provided by OSGi, transform it, and then write the new version back. Since this happens while OSGi is reading the class to begin with, there's never a time when there's an older, less-permissive version of the class running around, as long as I get my service in early enough.

This service is registered in the Activator without too much fuss:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

public class JakartaActivator implements BundleActivator {
    private final List<ServiceRegistration<?>> regs = new ArrayList<>();

    @Override
    public void start(BundleContext context) throws Exception {
        regs.add(context.registerService(WeavingHook.class.getName(), new UtilWeavingHook(), null));
    }

    @Override
    public void stop(BundleContext context) throws Exception {
        regs.forEach(ServiceRegistration::unregister);
        regs.clear();
    }
}

The final bit to get right was the "get the service in early enough" aspect. The main task was making sure that this bundle was activated before any XPages apps loaded, and that was a job for my old friend IServiceFactory, which is the extension point that's intended to add handlers for incoming URLs but has the desirable attribute of being initialized right at the very start of HTTP loading.

With this in place, I now have a fix automatically applied to that fiendish class on load, and MicroProfile Config (and future MP specs) works like a charm.

Conclusion

This was an arduous one, and I think the FTL victory jingle actually physically played when I got it working. I've hated this restriction for a long time, and I'm glad to finally have a workaround.

It was also enlightening to properly learn about OSGi's weaving capability. As mentioned above, this is what the ServiceLoader bridge does, and I'd tinkered with that at one point, but never got it working. I suspect now that it should be entirely doable to make it work, most likely also involving bringing in an implementation of the Declarative Services OSGi capability. That should be a fun project in its own right.

Moreover, the fact that I now have a system in place to do this weaving on the fly means that I may be able to un-fork some of the specs I had to fork to get working previously, which specifically required altering ServiceLoader calls. Even if I don't get the official service bridge in, perhaps I can use this technique to just alter the parts I need to on the fly, and otherwise use stock implementations from Maven.

But, for now, the way is cleared for further progress, and a bizarre mystery is solved. I call that a good day.