One of my favorite features of the XPages JEE Support project is the inclusion of the MicroProfile Rest Client. This framework makes consuming remote APIs (usually JSON, but not always) much, much better than in traditional XPages or even in most other frameworks.

MicroProfile Rest Client

If you're not familiar with it, the way it works is that you can use Jakarta REST annotations to describe the remote API. For example, say you want to connect to a vendor's service at "https://api.example.com/v1/users", which allows for GET requests that list existing users and POST requests to create new ones. With the MP Rest Client, you'd create an interface like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

package com.example.api;

import java.util.List;

import com.example.api.model.User;
import jakarta.ws.rs.Consumes;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.POST;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;

@Path("v1")
public interface ExampleApi {
	@Path("users")
	@GET
	@Produces(MediaType.APPLICATION_JSON)
	List<User> getUsers();

	@Path("users")
	@POST
	@Consumes(MediaType.APPLICATION_JSON)
	@Produces(MediaType.APPLICATION_JSON)
	User createUser(User newUser);
}

Then, in your Java code, you can use it like this (among other potential ways):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

ExampleApi api = RestClientBuilder.newBuilder()
	.baseUri(URI.create("https://api.example.com"))
	.build(ExamplApi.class);

List<User> existingUsers = api.getUsers();

// ...

User foo = makeSomeUser();
api.createUser(foo);

The MP Rest Client code takes care of all the actual plumbing of turning that interface into a usable REST client, translating between JSON and custom classes (like User there), and other housekeeping work. Things can get much more complicated (authorization, interceptors, etc.), but that will do for now.

Records

What has been a nice addition with Domino 14 and recent releases of the XPages JEE project is that now I can also use Java records with these interfaces. Records were introduced in Java 14 and are meant for a couple of reasons - for our needs, they cut down on boilerplate code, but they also have beneficial implications with multithreading.

In our example above, we might define the User class as:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

package com.example.api.model;

import jakarta.json.bind.annotation.JsonbProperty;

public class User {
	private String username;
	@JsonbProperty("first_name")
	private String firstName;
	@JsonbProperty("last_name")
	private String lastName;

	public String getUsername() {
		return this.username;
	}
	public void setUsername(String username) {
		this.username = username;
	}

	public String getFirstName() {
		return this.firstName;
	}
	public void setFirstName(String firstName) {
		this.firstName = firstName;
	}

	public String getLastName() {
		return this.lastName;
	}
	public void setLastName(String lastName) {
		this.lastName = lastName;
	}
}

This is a standard old Java bean class, and it's fine. Ideally, we'd add more to it - equals(...), hashCode(), and toString() in particular - and this version only has three fields. It'd keep growing dramatically the more we add: more fields, bean validation annotations, and so forth.

For cases like this, particularly where you likely won't be writing constructor code much (unless they add derivation syntax), records are perfect. In this case, we could translate the class to:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

package com.example.api.model;

import jakarta.json.bind.annotation.JsonbProperty;

public record User(
	String username,
	@JsonbProperty("first_name")
	String firstName,
	@JsonbProperty("last_name")
	String lastName
) {
}

This is dramatically smaller and will scale much better, lines-of-code-wise. And, since the important parts of the stack support it, it's nearly a drop-in replacement: JSON conversion knows about it, so it works in the MP Rest Client and in your own REST services, and Expression Language can bind to it, so it's usable in Pages and elsewhere.

Since records are read-only, they're not a universal replacement for bean-format classes (in particular, beans used in XPages forms), but they're extremely convenient when they fit and they're another good tool to have in your toolbelt.

This week and last, I uploaded a couple updates for some of my OpenNTF projects.

XPages Jakarta EE 3.2.0

First up is a new version of the XPages Jakarta EE project. This one is primarily about bug fixing and shoring up some of the 3.x line of features some more. Of particular note, it fixed a problem that crept in that caused the use of Concurrency to break the application, which was a bit of a problem.

WebFinger for Domino 2.0.0

About two years ago, I created the WebFinger for Domino project, primarily as a way to publish profile information in a way that works with Mastodon, though the result would work with ActivityPub services generally.

I'm fond of the goals of open formats like WebFinger and the related IndieWeb world. While Domino's licensing has been hostile to this world of personal sites since basically the 90s, it's still a good conceptual fit, and so I like to glom stuff like this on top of it when I can.

Anyway, I was reminded of this project and started thinking about ways to expand it beyond just its original use. For now, I mostly wanted to dust it off and make it extensible. The result is version 2.0.0, which refactors the code to allow for more stuff in the output. Along those lines, I added the ability to specify PGP public keys and have them included in the list, along with a Servlet to host them on your server. I plan to look around for other things people like to include in their profiles and add them in. I don't actually ever do much with PGP, but it's nice to have it in there.

Back when Notes/Domino 14 came out, I made a post where I described some of the fallout of it. One of the entries was about the upstream removal of the "jvm/lib/ext" directory and the moving of all common extension JARs to the "ndext" directory. The upshot there was that any JARs that you want to add to the filesystem in Designer to match deployment on the server would have to be added to the active JRE in Designer in order to be recognized.

HCL presumably noticed this problem and altered the installation to accommodate it in FP1 and FP2. However, the approach they took is to add all JARs from ndext to the JVM. Thus, a fresh install+upgrade of Notes 14 to FP2 (or 14.5 EAP1) has a JVM that looks like this:

This is a problem in a couple ways, but the most immediate is that it includes the toxic "jsdk.jar" I warned about in the earlier post. This JAR contains primordial Servlet classes from the very first addition of Servlet to Domino, predating XPages, and that version lacks even the convenience methods added in the ancient-but-less-so version in XPages. To demonstrate this, you can write this code:

1
2

HttpServletRequest req = null; /* pretend this is assigned to something */
Map param = req.getParameterMap();

This will work in a clean Designer 14 installation but will break on upgrade to 14 FP2, with Designer complaining that the getParameterMap method does not exist. There are others like this too, but basically any "The method foo() is undefined..." error for Servlet classes is a sign of this.

The fix is to go into your JVM definition (Preferences - "Java" - "Installed JREs" - "jvm (default)" - "Edit...") and remove jsdk.jar. While you're in there, I recommend also removing POI and its related JARs (poi-*, xmlbeans, ooxml-schemas, fr.opensagres.poi.*, commons-*) too, unless you also happen to have deployed them to the server, since they're not normally present on Domino and are thus mostly there to lead you astray. Honestly, almost none of the JARs present in there by default are useful for the XPages JVM definition, since the critical ones are contributed via OSGi plugins. I guess guava.jar is important just because it's going to contaminate the server's JVM too, so you want to account for that. Otherwise, it's probably best to treat it like a 14 install and only add the new JARs you've explicitly added and deployed to the server.

In the OpenNTF Discord (join if you haven't!), Daniel Nashed brought up a task he was facing: in the Notes client, writing pretty-printed JSON. LotusScript has its NotesJSON* classes that can process JSON in their stark way, but the stringify output is meant for machine reading and doesn't include whitespace and line breaks, making it ill-suited for things like configuration files or other things a human might read or edit.

Since the goal is to get it working in the full Notes client and not Nomad, Java is on the table, but Java - for dumb historical reasons - has no proper built-in JSON library. However, as of 12.something HCL shunted IBM Commons down to the global classpath in order to support the "share Java design elements between XPages and agents" feature. Among many other things, IBM Commons includes a JSON library that can suit.

I wrote a post almost a decade ago talking about this library and its limited nature, but it's nonetheless less limited than the LotusScript classes, and it's up to the task. There are a couple ways to go about this, depending on your needs, but for now I'll just cover the basic case here of "I have a string of JSON and want to format it".

To do this, you can make a Script Library of the Java type named, say, "JsonPrettyPrint" and make a new class in the "com.example" package and named "JsonPrettyPrint" with this contents:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

package com.example;

import java.io.IOException;

import com.ibm.commons.util.io.json.JsonException;
import com.ibm.commons.util.io.json.JsonGenerator;
import com.ibm.commons.util.io.json.JsonJavaFactory;
import com.ibm.commons.util.io.json.JsonParser;

public class JsonPrettyPrint {
	public String prettyPrint(String json) throws JsonException, IOException {
		Object jsonObj = JsonParser.fromJson(JsonJavaFactory.instanceEx, json);
		return JsonGenerator.toJson(JsonJavaFactory.instanceEx, jsonObj, false);
	}
}

Then, you can instantiate this object with LS2J and pass it a JSON string, such as one from NotesJSONNavigator:

1
2
3
4
5
6
7
8

UseLSX "*javacon"
Use "JsonPrettyPrint"

Sub Initialize
	Dim jsession As New JavaSession, jsonPrinter As JavaObject
	Set jsonPrinter = jsession.GetClass("com.example.JsonPrettyPrint").CreateObject()
	MsgBox jsonPrinter.prettyPrint(|{"foo":{"bar":"baz"}}|)
End Sub

That'll get you something presentable:

While the stringify-parse-stringify process you'd do if you generated your JSON with the NotesJSON* classes is inefficient, it's not too bad, especially for the size of content you're likely to want to emit here. You could alternatively do more work with JsonJavaObject and friends from IBM Commons directly to save a little overhead, but this path is a good way to do the vast majority of work in "normal" LotusScript and then only dip in to Java for the last step.

As mentioned at the start, the presence of Java means this won't work for Nomad, unfortunately. There may be a way to wrangle your way to this result using the primordial JavaScript runtime present in that, but that may not be worth the trouble unless you really need it. Better would be to vote for the Aha idea to add pretty printing to LS.

Today, I released version 2.15.0 of the XPages Jakarta EE project. As is often the case lately, this version contains bug fixes but also a few notable features:

• You can now specify Servlets in WEB-INF/web.xml (as opposed to just via the @WebServlet annotation). This is helpful for defining a Servlet when the actual implementation is in a JAR or when following non-annotation-based examples
• You can now specify context-param values in WEB-INF/web.xml in the NSF and META-INF/web-fragment.xml in JAR design elements, which will be available to JSP, JSF, JAX-RS, @WebServlet-annotated Servlets, and web.xml-defined Servlets
• Added @BooleanStorage annotation for NoSQL entities to define how boolean values are converted to note items
• Added CRUD operations for calendar events to NoSQL, around a few new methods on Repository. This exposes some of the capabilities of NotesCalendar and can be used for, for example, providing an iCalendar feed based on a mail database. To go with that, XPages JEE also re-exports iCal4J as included in the Domino stack for NSF use, though this API is... not smooth

The first two here are focused around bringing NSFs more in line with "normal" Jakarta EE applications, while the latter are some nice improvements for the NoSQL driver. I hope to put the last one in particular to good use - for example, OpenNTF's site will be able to provide a calendar of webinars and other events that we can manage internally using a normal Notes calendar, and that sounds nice to me.

Next Versions

I still have the 3.x branch of the project chugging along, and I think it'll be ready for a real release before too long. Since it'll be a breaking-changes release thanks to upstream changes, I'm using it as an opportunity to consolidate the sprawl of features and XPages Libraries. Currently, my plan is:

• One for "core", covering most things in the Jakarta EE Core Profile, plus the other utility specs I've implemented: Transactions, Bean Validation (which really should be in Core in my estimation), Concurrency, Servlet, and so forth, plus Data and NoSQL
• One for "UI", covering Jakarta Pages, Jakarta Faces, and MVC - basically, the stuff you could use to replace XPages to make an HTML-generating app in your NSF
• One for MicroProfile, or at least the specs I've implemented so far. I'm a little tempted to wrap this in to Core, since things like OpenAPI are useful almost all the time, but it's a clean-enough separation that it'll be fine

This will require Domino 14, since Jakarta EE 10 requires at least Java 11.

That brings me to some unexpected good news: though Jakarta EE 11 was long planned to use Java 21 as its minimum version (since 21 is the current LTS), it looks like they've switched to making Java 17 the baseline. For me, this is a little sad in an idealistic sense, since it pushes things like Virtual Threads out of the realm of being a core part of JEE, but I'm very happy that I'll be able to use all JEE 11 specs in Domino 14. Even if Domino 15 used Java 21, it'd still be a long while before that came, and we'd lag behind the standard for at least a year. Instead, this puts the project back in line with upstream, and allows me personally to potentially resume committing to Jakarta NoSQL - I'd been out of the loop for a very long time when it moved to 11 and then 17 as its required version.

I don't know right now whether JEE 11 will be the same sort of breaking change for the project (which would mean a 4.x release) or if I'll be able to make it a 3.x one - the specs aren't out yet, so time will tell. The big focus of 11 will be further centralization on CDI instead of EJB, and I'm all for it.

My plan is to get 3.x out for Domino 14, based on JEE 10, as soon as time allowed, and then I'll start looking into bumping to JEE 11 when it releases in the summer.

Notes and Domino 14 are out now and, as I discussed back in June the big deal for me is the move to Java 17. This also came with a refresh of the Eclipse innards, from Neon (circa 2016) to 2021-12 (circa, uh, 2021). The Eclipse update is welcome, but so far it's been less impactful than the Java update - at some point, I'll want to see if some of the current-era Eclipse plugins work here, but that's for the future.

In the mean time, there's a bunch to know, so let's get to it! I've broken this one down into "critical" and "less critical" sections, since this post will likely have a similar life to my target platform one.

Critical To Know

ndext

I mentioned this in June, but an important thing to know about Java 17 is that "jvm/lib/ext" no longer exists. Fortunately, Notes and Domino have long had a secondary location for this sort of thing: "ndext" in the program directory. Any JARs in this folder are available at runtime on both platforms, and so the quick fix is to move anything you had in "jvm/lib/ext" to "ndext".

...but.

When it comes to Notes, there's a distinct difference between "available at runtime" and "on the build path". While Java agents here are fine - they modified the editor to include "ndext" in the build path - there's no such accommodation for XPages. So far, the best workaround I've thought of is to add any such JARs manually to the JRE definition in Eclipse's preferences:

When doing this, there's a huge caveat: do not just add everything from this directory to the JRE. Most of it will be redundant, like the SWT stuff, but some will be actively harmful, in particular "jsdk.jar". That's an even-older Servlet version than comes with XPages, and including that will cause Designer to think that methods added in Servlet 2 aren't present. Only add the JARs you're extending it with.

Ideally, this will be remedied one way or another in the future, but for now we'll have to do this to account for it. The silver lining may be that it's a good impetus to OSGi-ify your dependencies if possible.

Poi Remains

This isn't new, but it's worth mentioning while we're on the topic of "ndext". Since Notes 11, the client ships with an old version of Apache Poi, but this is painfully distributed right in the JVM and not at the OSGi layer. Newer versions of Poi 4 XPages deal with this, but it's important to know that, while Poi is in Notes, it's not in Domino. If you write agents using these classes, or add them to your JRE and use them in XPages, you'll also need to deploy them to Domino.

Java Policy Location

This one's more of a note, and it's reiterating something from June: the location of "java.policy" and (if you add it) "java.pol" changed since Java 8. They used to be in "jvm/lib/security" but they're in "jvm/conf/security" now. They work the same as before, as does putting a file named ".java.policy" in the Domino user's home dir, so the other characteristics haven't changed.

sun.misc.BASE64Decoder

Back in Domino 11, the move from IBM J9 to OpenJ9 meant that some IBM internal forks of Sun internal classes were no longer available. The most notable of these were the BASE64 classes com.ibm.misc.BASE64Encoder and BASE64Decoder. These were very popular to use in the pre-Java-8 days, before java.util.Base64 existed, and so it was worth noting that they were gone then.

Well, sun.misc.Base64Encoder has now met a similar fate - it's probably still in there somewhere, but it's no longer accessible by user code. If you haven't made the switch to java.util.Base64, do so now.

Target Platform Bug

This is another note, but this classic bug that's been with us since 9.0.1FP10 is... fixed, I think! I'd thought at first that it remained, since the Target Platform config still just lists the Eclipse home dir, but installing and using an XPages library seemed to work properly without further change. Good!

Java Compiler Level

This is a fiddly one. Though Designer uses Java 17 by default now, it has the ability to compile down to older versions for past compatibility, such as running on a pre-14 server. Java agents have had their own way of dealing with this for a while, though it seems like maybe they default to Java 8 now, which is good. With XPages, it seems like it's a little less obvious.

From my experience, this is what I've found:

• Existing projects may have a specific compiler setting specified (similar to agents in the above-linked post) and so will remain as Java 1.8
• New projects seem to use the active workspace setting, which will matter
• Upgrades of Designer in place will keep the default compiler level for the workspace at 1.8
• A fresh installation sets the compiler level to 17

This is all... fine. It'd be nice if it was a little more obvious to the developer (like if it was controlled by the xsp.properties setting for minimum version), but this is more or less the Eclipse way of doing things. We'll just have to be aware for a while of these settings and their interactions when developing for pre-14 deployment. If you're targeting an older server, you should make sure that you go to Project Properties for your NSF and set the Java Compiler level to fit:

Less Critical

Alright, that covers the big-ticket stuff, but there are a few more things to know.

Java Deprecation Warning On HTTP Start

This one is documented, though oddly in the "no longer included" page: when you start HTTP on Domino, you'll see a message like this:

WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by lotus.notes.AgentSecurityManager (file:/C:/Domino/ndext/Notes.jar)
WARNING: Please consider reporting this to the maintainers of lotus.notes.AgentSecurityManager
WARNING: System::setSecurityManager will be removed in a future release

This is actually totally fine. As it says, the whole SecurityManager apparatus is gone in future versions, but Notes and Domino still use it for agents (and, unfortunately, XPages). While I long for the day when I never have to think about it again, this is a reasonable-enough compromise for 14 as a "transitional" version in its Java journey. So... you can ignore this and not worry.

"Java Main Sources" and "Java Test Sources" Working Sets

If you use Working Sets in Designer, you may notice two entries not of your creation:

These showed up in Eclipse somewhere along the line, presumably to make it easier for people to select those types of projects without manually managing their working sets. Designer inherits this and HCL didn't remove them, but you're free to delete them if you want.

"AbstractCompiledPage cannot be resolved"

This is another one that came up back in 9.0.1FP10 and it's still here, but it's less of an issue: every once in a while, when building a project, Designer will complain about the AbstractCompiledPage class not being found. In my experience, this only shows up temporarily during a build, but it's worth noting that, if it does stick around for you, a Project - Clean should fix it.

JAX-B and CORBA

After Java 8, a few Java EE components were removed from the normal JRE distribution in favor of developing them in Java EE, then Jakarta EE. JAX-B is one of them (now Jakarta XML Binding) - we don't normally use this directly, but it comes up sometimes, either directly (likely as one of the other old-timey BASE64 workarounds) or as a dependency. It shouldn't matter in Designer, but, if you're writing Domino-targeting code outside Designer, you may need to be aware. One way or another, you should add this as a dependency - in a basic case, you can add "jakarta.xml.bind-api.jar" and "jaxb-impl.jar" from "ndext" to your build path.

Similarly, CORBA support classes ("org.omg" stuff, usually) used to come with the JRE and now don't. To work around this, HCL did what I've done in the past and included "glassfish-corba-omgapi.jar", in their case putting it in "ndext". If you're using Notes.jar with Java > 8 outside Designer, you'll need this one too.

Conclusion

I think that covers it for now. Considering how much changed with Java from 8 to 17, this could have been a lot rougher, though I fear that some of these workarounds will plague us for a long time.

In the mean time, I'd appreciate it if you vote for this Aha idea to keep Domino on a good Java update cadence. It'd be a shame if we sit on 17 right up until the very end of its life, as we did with 6 and 8, in part since these multi-version moves are more painful than single-LTS updates.

Though it's been over a month since CollabSphere 2023, I've somehow now yet gotten around to talking about it here. Time to remedy that!

Webinar and Workshop

I presented a couple sessions at CollabSphere, but the meatiest was my workshop on the XPages Jakarta EE project. A bit before the show, I wrote a post discussing modes of development with it and that formed the structure of the workshop.

It also formed the structure of August's OpenNTF webinar. Fortunately, unlike the workshop, the webinar was recorded, and that is up on OpenNTF's YouTube channel. Though the workshop was longer and had some refinements and audience discussion, they both worked from the same original slide deck and so the webinar did a pretty good job of covering the same material.

As part of the presentations, I created four versions of the same basic to-do tracker app, written in each of the four modes I talked about. I put them up in the project repository with a quick README introducing each of them. With the project installed (they were written to 2.13.0 and above), you should be able to sync the ODPs with NSFs in Designer and poke around yourself. They're all intentionally-similarly structured and basic, with all of their elements in either Java classes, file resources, or stylesheets. They're also all intentionally under-developed, so you're not allowed to make fun of them.

JNX

This one will be a doozy! In fact, it's such a doozy that I'm going to keep kicking the can down the road as far as properly talking about it.

In short, Domino JNX is a more-modern API for Domino access - it was initiated and is used heavily by the Domino REST API (DRAPI), but also stands on its own. It started out essentially as another swing at Karsten Lehmann's Domino JNA and shares a lot of the same ideas and approaches (and code - the committer info in the GitHub repo is deceptive, as I got to paste a whole mountain of existing code from another repo into this one and thus got credit for it).

Now that it's open source, there's some significant work to do as far as documentation, examples, integration, and distribution go. I plan to find time to improve a lot of these aspects, including blogging here, though it's the sort of thing that always ends up below other priorities.

In any event, it's quite neat and has a lot of good capabilities. I plan to write a driver for JNoSQL for it, either to be alongside or to wholly supplant the Notes.jar-based one currently used by the project. The neat thing there is that users of XPages JEE won't have to care much: it'll just get a bit faster in parts but should largely work the same, except maybe with a change to the way you point at other databases.

I've been working on my workshop for this year's CollabSphere, and one of the main decisions I have to make is what I'm going to focus on. The idea of the workshop is to give a bit more brass-tacks information about how to use the project: rather than just a list of features, it'll be about the specific business of building an app using it.

But how does one build an app in it? There's certainly no lack of tools available, but that leads to the opposite problem: what's the right one for your project? What's likely to be the most common path people take?

The Types

As I've been working on it, I've grouped things into four main categories, and I figured it'd be useful to enumerate them here to coordinate my thoughts and provide some general information. There aren't hard lines between these: you can use any mixture of some or all of the parts in an app, and do different mixes in different apps. These are just what I expect to be the main groupings:

• "XPages Plus", using some new capabilities in existing or new apps with XPages-based UIs
• REST services, focusing on providing REST endpoints for JavaScript-based apps or other servers
• MVC and JSP, focusing on clean, lightweight UIs for document-based apps, but less ideal for complex business logic
• JSF, building the same sorts of apps XPages is adept at, but using newer technology

"XPages Plus"

The first route is how the project got started: you keep building XPages apps but sprinkle in a few new capabilities to improve them.

For example, you could replace your managed beans defined in faces-config.xml with CDI beans, allowing you to get the quick benefit of annotation-based definitions and then the bigger benefits of @Inject, producer methods, and interceptors.

You could also start using newer EL features, like the long-desired ability to pass parameters to methods.

This path wouldn't necessarily require a lot of reworking of your app or changing the way you think about XPages development, but would still be something of a minor development refresh and can set you up well for future improvements.

Your data access will likely still be through the traditional xp:dominoDocument and xp:dominoView components, but you could also write beans that access data with lotus.domino or ODA, or switch to using the NoSQL driver.

REST Services

Alternatively, you could decide you want to focus your apps around REST services with either a JavaScript app in, for example, React as the front end, or providing services to remote servers.

With this, you'd largely stop using XPages design elements entirely, instead defining your services in Java classes with JAX-RS annotations. This brings huge advantages over other ways to write REST services on Domino, with the JAX-RS annotations allowing for clear, logical definition of services, their parameters, and their output. Moreover, the ancillary tooling brings things like automatic OpenAPI definitions, which would be annoying to maintain using things like the XPages-side REST controls.

This path is good if you're specifically aiming to build a JavaScript-based app, either because you just like it, because your organization decided to go that route, or if you have a larger team that splits the duties of front-end and back-end developers. It can also naturally blend into the next one.

Your data access here won't be through the XPages components, but you could still use lotus.domino or ODA classes, or switch to the NoSQL driver. That actually goes for the next two, too, so we'll just count that as assumed.

MVC and JSP

I'll admit that part of the reason I want to consider this a top-tier route is because I just personally really like it. I've had a blast writing apps like this blog and the OpenNTF site using this path, with its much-cleaner code and back-to-basics approach to HTML.

Regardless of my personal enjoyment of it, though, this has some nice advantages. The fact that MVC builds on top of JAX-RS means that it melds well with the REST-services approach above. For example, you might primarily write REST services for a JS app, but then do a set of "admin" pages with MVC. Or you might use this as part of the prototype phase: structure your app the same way you will when you expand to a multi-tier team, but start out by doing a quick UI with MVC on top of the same or related endpoints.

With this path, your app will start with Java classes with JAX-RS annotations, and then you'd mix it with JSP files inside WebContent/WEB-INF. One down side to this approach is that Designer doesn't provide much help for writing JSP files. In the tooling, I bind .jsp and .tag files to the HTML editor, so you at least get normal HTML assistance, but that won't help you with specific JSP tags and EL. Fortunately, the set of tools you'll likely use in JSP is comparatively small, so you'll eventually memorize things like <c:forEach items="..." var="...">...</c:forEach> in much the same way that you could eventually write out an <xp:repeat/> in your sleep in XPages.

JSF

This one, technically tricky though it may be, is conceptually straightforward: write the same sort of apps you do with XPages, but do it with modern JSF instead. This makes a lot of sense, since JSF shares XPages's acumen with complicated forms with partial refreshes and changing state data, but has benefited from some development that didn't happen on the XPages side.

It's not a direct replacement: in particular, JSF has no knowledge of Domino data sources, so there's no xp:dominoDocument or xp:dominoView. You'd still need to do your data access via beans, as in the previous two options, likely using either lotus.domino/ODA or the NoSQL driver. Additionally, Designer really doesn't help you here - again, I map .xhtml and .jsf files to the HTML editor, but JSF components have a lot of properties to set, and so you'll be spending a lot of time referencing documentation.

Still, it's clear why this is proving to be a popular path. The development model is the same as in XPages, while the JSF stack (especially including PrimeFaces) brings a lot of amenities that aren't in XPages and are also more portable to other environments.

Conclusion

So, for now, I'm thinking of splitting up the workshop to cover each of these paths a bit. That runs the risk of feeling like too much of a grab bag, but I don't want to give the opposite impression, that the project only allows for some specific path. It's a broad platform update, accommodating many development approaches, and I want to keep that clear. Fortunately, each path has a pretty-clean pitch, and the shared components (CDI, bean validation, the REST client, etc.) build on each other well, so the idea that it's a pool of features that you can swim in is, I think, compelling.

As promised, HCL launched the beta program for Domino 14 the other day. There's some neat stuff in there, but I still mostly care about the JVM update.

I quickly got to downloading the container image for this to see about making sure my projects work on it, particularly the XPages JEE Support project. As expected, I had a few hoops to jump through. I did some groundwork for this a while back, but there was some more to do. Before I get to some specifics, I'll mention that I put up a beta release of 2.13.0 that gets almost everything working, minus JSP.

Now, on to the notes and tips I've found so far.

AccessController and java.policy

One of the changes in Java 17 specifically is that our old friend AccessController and the whole SecurityManager framework are marked as deprecated for removal. Not a minute too soon, in my opinion - that was an old relic of the applet days and was never useful for app servers, and has always been a thorn in the side of XPages developers.

However, though it's deprecated, it's still present and active in Java 17, so we still have to deal it. One important thing to note is that java.policy moved from the "jvm/lib/security" directory to "jvm/conf/security". A while back, I switched to putting .java.policy in the Domino user's home directory and this remains unchanged; I suggest going this route even with older versions of Domino, but editing java.policy in its new home will still work.

Extra JARs and jvm/lib/ext

Another change is that "jvm/lib/ext" is gone, as part of a general desire on Java's part to discourage installing system-wide libraries.

However, since so much Java stuff on Domino is older than dirt, having system-wide custom libraries is still necessary, so the JVM is configured to bring in everything from the "ndext" directory in the Domino program directory in the same way it used to use "jvm/lib/ext". This directory has actually always been treated this way, and so I started also using this for third-party libraries for older versions, and it'll work the same in 14. That said, you should ideally not do this too much if it's at all avoidable.

The Missing Java Compiler

I mentioned above that everything except JSP works on Domino 14, and the reason for this is that V14 as it is now doesn't ship with a Java compiler (JSP, for historical reasons, does something much like XPages in that it translates to Java and compiles, but this happens on the server).

I originally offhandedly referred to this as Domino no longer shipping with a JDK, but I think the real situation was that Domino always used a JRE, but then had tools.jar added in to provide the compiler, so it was something in between. That's why you don't see javac in the jvm/bin directory even on older releases. However, tools.jar there provided a compiler programmatically via javax.tools.ToolProvider.getSystemJavaCompiler() - on a normal JRE, this API call returns null, while on a JDK (or with tools.jar present) it'd return a usable compiler. tools.jar as such doesn't exist anymore, but the same functionality is bundled into the newer-era weird runtime archives for efficiency's sake.

So this is something of a showstopper for me at the moment. JSP uses this system compiler, and so does the XPages Bazaar. The NSF ODP Tooling project uses the Bazaar to do its XSP -> Java -> bytecode compilation of XPages, and so the missing compiler will break server-based compilation (which is often the most reliable compilation currently). And actually, NSF ODP Tooling is kind of double broken, since importing non-raw Java agents via DXL is currently broken on Domino 14 for the same reason.

I'm pondering my options here. Ideally, Domino will regain its compiler - in theory, HCL could switch to a JDK and call it good. I think this is the best route both because it's the most convenient for me but also because it's fairly expected that app servers run on a JDK anyway, hence JSP relying on it in the first place.

If Domino doesn't regain a compiler, I'll have to look into something like including ECJ, Eclipse's redistributable Java compiler. I'd actually looked into using that for NSF ODP Tooling on macOS, since Mac Notes lost its Java compiler a long time ago. The trouble is that its mechanics aren't quite the same as the standard one, and it makes some more assumptions about working with the local filesystem. Still, it'd probably be possible to work around that... it might require a lot more of extracting files to temporary directories, which is always fiddly, but it should be doable.

Still, returning the built-in one would be better, since I'm sure I'm not the only one with code assuming that that exists.

Overall

Despite the compiler bit, I've found things to hold together really well so far. Admittedly, I've so far only been using the container for the integration tests in the XPages JEE project, so I haven't installed Designer or run larger apps on it - it's possible I'll hit limitations and gotchas with other libraries as I go.

For now, though, I'm mostly pleased. I'm quite excited about the possibility of making more-dramatic updates to basically all of my projects. I have a branch of the XPages JEE project where I've bumped almost everything up to their Jakarta EE 10 versions, and there are some big improvements there. Then, of course, I'm interested in all the actual language updates. It's been a very long time since Java 8, and so there's a lot to work with: better switch expression, text blocks, some pattern matching, records, the much-better HTTP client, and so forth. I'll have a lot of code improvement to do, and I'm looking forward to it

At Engage, HCL officially announced Java 17 in Domino 14 (I'm sure they announced other things too, but I have my priorities). This will allow me to do a lot in pretty much all of my projects, but it's particularly pertinent to XPages JEE.

Currently, the project targets generally Jakarta EE 9, which came out in late 2020 and was "just" a switch from javax.* to jakarta.*, with no official new features. However, Jakarta EE 10 came out a year ago - in addition to bringing a raft of new features, it also bumped the minimum Java version to Java 11, pushing it outside of Domino's realm. Accordingly, I've had to hold off on a lot of major- and minor-version bumps in the XPages JEE project as new releases started being compiled for Java 11. Once V14 is out, though, I'll be able to move to the current JEE platform... at least until JEE 11 comes out next year and requires Java 21, anyway.

So I've been working on how I'm going to approach this, and what I'm thinking is that I'll do it in two phases: first, a final 2.x release that provides Java 17/Domino 14 compatibility for existing components, and then a new 3.x breaking-changes release to bring in Jakarta EE 10 components.

The Final 2.x Release

I currently have this penciled in for the next release, 2.12.0, but that may change if I decide I want to get a real 2.12.0 release out before Domino 14 is at least in stable beta form. Let's call it "2.99.0" for now.

The idea here will be that I'll want to make sure all existing code in NSFs continues to work unchanged: upgrade your server to V14, install 2.99.0, and your apps keep working. In theory, this shouldn't be too complex. There's some shimming needed for Weld (the CDI implementation) to account for changes from Project Jigsaw in Java 9 and later, and there might be some stuff around AccessController, but in general I expect it'll just be some tweaks here and there. Time will tell, of course.

Once that's out, I plan to not look back (unless there's demand, I suppose). The switch to Java 17 is a huge deal, and I don't think it'll be worth spending much more time with Java 8 once it's no longer required. The 2.x branch is already, I feel, in a pretty good place, so I'll feel comfortable having a stable final version.

The Breaking 3.0 Release

Then, the plan will be to start down the path of 3.x with breaking changes - not everything, but some. For one, JEE 10 has a handful of backwards-incompatible changes. Those are mostly for legacy true-JEE code, though, and the main ones that XPages JEE code will likely want to be aware of will be the switch of XML namespaces to shorter representations. That will affect JSP and JSF code, but the old URIs (the jcp.org ones) will continue to work, at least for a while.

Most of the breaking changes will probably happen internally. I've talked for a long while now about my desire to do some reorganization of the project. The big one is wrangling the proliferation of Eclipse Features and XPages Libraries. Anyone who has installed the project in Designer is well aware of just how many times you have to click "yes, I want to install the thing I'm installing", and that alone is enough to warrant a reorganization. Beyond that, though, I've had to take care to try to make it so that the individual components don't depend on each other unnecessarily. There's a certain amount of good discipline that provides, but it eventually wears a bit thin.

I'm not quite sure what form the consolidation will take, but it'll probably be something like three features: "core", "extended", and "MicroProfile". "Core" would probably roughly map to the actual Jakarta Core Profile, plus things that I find essentially obligatory like Bean Validation. "Extended" would be all the things like JSP and JSF, the "leaves" on the dependency tree: they depend on core features, but nothing depends on them. Then "MicroProfile" would be, well, MicroProfile features. The only thing still giving me pause is that there's not too much case for not installing all of these all the time anyway - if you don't want to use, say, JSF, you don't have to; additionally, it's not like Domino is a svelte cloud-native mini server meant to be deployed a thousand times in a cluster, so having the extra bundles sitting there isn't really onerous. We'll see. I hem and haw a lot on this, but eventually I'll have to make a decision.

Regardless of what form that takes, I expect that the changes to in-NSF code will be either minimal or none - for users of the project, it'll mostly be a matter of making sure to fully uninstall the old plugins before an upgrade and then tweaking Xsp Properties to select whatever the new form of the XPages Libraries ends up being.

Side Note: Jakarta NoSQL and Data

One interesting aspect of this move will be the path Jakarta NoSQL has been on. Though I've included it in the XPages JEE project for a little while now (and continue to heavily expand on it), it's always been technically a beta release. It's clearly proven itself stable even in its beta form, but it's going through a shift in the run-up to Jakarta EE 11. Specifically, the higher abstraction levels - the Repository interface and friends - are moving to a new project, Jakarta Data. The idea of that project will be that it will be able to sit on top of Jakarta NoSQL and other storage types, namely JPA.

It's going to be very neat, but it's created a bit of a pickle for me. Since it's targetting Jakarta EE 11, that means the release of it and NoSQL are going to require at least Java 21, and there's no word on when Domino will support that.

One option would be to stick with what I have now for the foreseeable future: a mildly-forked version of Jakarta NoSQL 1.0.0-b4. It's a workhorse and has been doing a good job, and it'd mean that app code wouldn't have to change. I'm not crazy about this for obvious reasons: I don't want to have one component stuck way behind while all the other parts get a nice jump forward, even if it works.

The other main option I'm considering is sliding forward to another beta release and landing there until Java 21 support shows up. The current development versions of the Data spec and JNoSQL with its implementation target Java 17, so I'll probably go with whatever the last beta is before the official switch to 21. Though it's tough to predict the future, that will probably end up being API-wise similar enough to the release forms of them that future jumps won't be difficult. We shall see, anyway.

Timeline

Anyway, the timeline for this is a little vague, and will mostly depend on when the Domino 14 betas come out and whether they contain anything show-stopping. My hope is to be able to have something that passes all the test cases ASAP with betas and then to have it continue to be stable through to the actual release.

I'm looking forward to leaving Java 8 behind for good, though, that much is certain.

Last year, I wrote a post describing some early work on a Jakarta NoSQL driver for the Domino REST API (hereafter referred to as "Keep" to avoid ambiguity with the various other Domino REST APIs).

I've since picked back up on the project and similar aspects, and I figured it'd be useful to return to provide some more details.

OpenAPI

For starters, I mentioned in passing my configuration of the delightful openapi-generator tool, but didn't actually detail my configuration. It's changed a little since my first work, since I found where you can specify using the jakarta.* namespace.

I use a config.yaml file like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

additionalProperties:
  library: microprofile
  dateLibrary: java8
  apiPackage: org.openntf.xsp.nosql.communication.driver.keep.client.api
  invokerPackage: org.openntf.xsp.nosql.communication.driver.keep.client
  modelPackage: org.openntf.xsp.nosql.communication.driver.keep.client.model
  useBeanValidation: true
  useRuntimeException: true
  openApiNullable: false
  microprofileRestClientVersion: "3.0"
  useJakartaEe: true

That will generate client interfaces that will mostly compile in a plain Jakarta EE project. The files have some references to an implementation-specific MIME class to work around JAX-RS's historical lack of one, but those imports can be safely deleted.

Keycloak/OIDC in Keep

I also mentioned only in passing that you could configure Keep to trust the Keycloak server's public keys with a link to the documentation. Things on the Keep side have expanded since then, and you can now configure Keep to reference Keycloak using Vert.x's internal OIDC support, and also skip the step of creating special fields in your person docs to house the Notes-format DN. For example, in a Keep JSON config file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

{
	"oidc": {
		"my-keycloak": {
			"active": true,
			"providerUrl": "https://my.keycloak.server/auth/realms/myrealm",
			"clientId": "keep-app",
			"clientSecret": "<my secret>",
			"userIdentifierInLdapFormat": true
		}
	}
}

That will cause Keep to fetch much of the configuration information from the well-known endpoint Keycloak exposes, and also to map names from Keycloak from the LDAP-style format of "cn=Foo Fooson,o=SomeOrg" to Domino-style "CN=Foo Fooson/O=SomeOrg". This is useful even when using Domino as the Keycloak LDAP backend, since Domino does the translation in the other direction first.

Keycloak/OIDC in Jakarta EE

In the original post in the series, talking about configuring app authentication for the AppDev Pack, I talked about Open Liberty's openidConnectClient feature, which lets you configure OIDC at the server level. That's neat, and I remain partial to putting authentication at the server level when it makes sense, but it's no longer the only game in town. The version of Jakarta Security that comes with Jakarta EE 10 supports OIDC inside the app in a neat way, and so I've switched to using that.

To do that, you make a CDI bean that defines your OIDC configuration - this can actually be on a class that does other things as well, but I like putting it in its own place:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

package config;

import jakarta.enterprise.context.ApplicationScoped;
import jakarta.security.enterprise.authentication.mechanism.http.OpenIdAuthenticationMechanismDefinition;
import jakarta.security.enterprise.authentication.mechanism.http.openid.ClaimsDefinition;

@ApplicationScoped
@OpenIdAuthenticationMechanismDefinition(
	clientId="${oidc.clientId}",
	clientSecret="${oidc.clientSecret}",
	redirectURI="${baseURL}/app/",
	providerURI="${oidc.domain}",
	claimsDefinition = @ClaimsDefinition(
		callerGroupsClaim = "groups"
	)
)
public class AppSecurity {
}

There are a couple EL references here. baseURL is provided for "free" by the framework, allowing you to say "wherever the app is hosted" without having to hard-code it. oidc here refers to a bean I made that's annotated with @Named("oidc") and has getters like getClientId() and so forth. You can make a class like that to pull in your OIDC config and secrets from outside, such as a resource file, environment variables, or so forth. providerURI should be the same base URL as Keep uses above.

Once you do that, you can start putting @RolesAllowed annotations on resources you want protected. So far, I've been using @RolesAllowed("users"), since my Keycloak puts all authenticated users in that group, but you could mix it up with "admin" or other meaningful roles per endpoint. For example, inside a JAX-RS class:

1
2
3
4
5
6
7

@Path("superSecure")
@GET
@Produces(MediaType.TEXT_PLAIN)
@RolesAllowed("users")
public String getSuperSecure() {
	return "You're allowed in!";
}

When accessing that endpoint, the app will redirect the user to Keycloak (or your OIDC provider) automatically if they're not already logged in.

Accessing the Token

In my previous posts, I mentioned that I was able to access the OIDC token that the server used by setting accessTokenInLtpaCookie in the Liberty config, and then getting oidc_access_token from the Servlet request object's attributes, and that that only showed up on requests after the first.

The good news is that, with the latest Jakarta Security, there's a standardized way to do this. In a CDI bean, you can inject an OpenIdContext object to get the current user's token:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

package bean;

import jakarta.enterprise.context.RequestScoped;
import jakarta.inject.Inject;
import jakarta.security.enterprise.identitystore.openid.OpenIdContext;

@RequestScoped
public class OidcContextBean {
  
	@Inject
	private OpenIdContext context;
  
	public String getToken() {
		// Note: if you don't restrict everything in your app, do a null check here
		return context.getAccessToken().getToken();
	}
}

There are other methods on that OpenIdContext object, providing access to specific claims and information from the token, which would be useful in other situations. Here, I only really care about the token as a string, since that's what I'll send to Keep.

With that token in hand, you can build a MicroProfile Rest Client using the generated API interfaces. For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

public class SomeClass {
	/* snip */
	@Inject
	private OidcContextBean oidcContext;

	/* snip */

	private DataApi getDataApi() {
		return RestClientBuilder.newBuilder()
			.baseUri("http://your.keep.server:8880/api/v1/")
			.register((ClientRequestFilter) (ctx) -> {
				ctx.getHeaders().add(HttpHeaders.AUTHORIZATION, "Bearer " + oidcContext.getToken()); //$NON-NLS-1$
			})
			.build(DataApi.class);
	}
}

That will cascade the OIDC token used for your app login over to Keep, allowing your app to access data on behalf of the logged-in user smoothly.

I've been kicking the tires on some example apps and fleshing out the Jakarta NoSQL driver using this, and it's been going really smoothly so far. Eventually, my goal will be to make it so that you can take code using the JNoSQL driver for Domino inside an NSF using the XPages JEE project and move it with minimal changes over to a "normal" JEE app using Keep for access. There'll be a bit of rockiness in that the upstream JNoSQL API is changing a bit to adapt to Jakarta Data and will do so in time for JEE to require Java 21, but at least it won't be too painful an analogy.

Yesterday, I put two releases up on OpenNTF, and I figure it'd be worth mentioning them here.

XPages Jakarta EE Support

The first is a new version of the XPages Jakarta EE Support project. As with the last few, this one is mostly iterative, focusing on consolidation and bug fixes, but it added a couple neat features.

The largest of those is the JPA support I blogged about the other week, where you can build on the JDBC support in XPages to add JPA entities. This is probably a limited-need thing, but it'd be pretty cool if put into practice. This will also pay off all the more down the line if I'm able to add in Jakarta Data support in future versions, which expands the Repository idiom currently in the NoSQL build I use to cover both NoSQL and RDBMS databases.

I also added the ability to specify a custom JsonbConfig object via CDI to customize the output of JSON in REST services. That is, if you have a service like this:

1
2
3
4
5

@GET
@Produces(MediaType.APPLICATION_JSON)
public SomeCustomObject get() {
	return findSomeObject();
}

In this case, the REST framework uses JSON-B to turn SomeCustomObject into JSON. The defaults are usually fine, but sometimes (either for personal preference or for migration needs) you'll want to customize it, particularly changing the behavior from using bean getters for properties to instead use object fields directly as Gson does.

I also expanded view support in NoSQL by adding a mechanism for querying views with full-text searches. This is done via the ViewQuery object that you can pass to a repository method. For example, you could have a repository like this:

1
2
3
4

public interface EmployeeRepository extends DominoRepository<Employee, String> {
	@ViewEntries("SomeView")
	Stream<Employee> listFromSomeView(Sorts sorts, ViewQuery query);
}

Then, you could perform a full-text query and retrieve only the matching entries:

1
2
3
4
5

Stream<Employee> result = repo.listFromSomeView(
	Sorts.sorts().asc("lastName"),
	ViewQuery.query()
		.ftSearch("Department = 'HR'", Collections.singleton(FTSearchOption.EXACT))
);

Down the line, I plan to add this capability for whole-DB queries, but (kind of counter-intuitively) that would get a bit fiddlier than doing it for views.

XPages Javadoc Provider

The second one is a new project, the XPages Javadoc Provider. This is a teeny-tiny project, though, not even containing any Java code. This is a plugin for either Designer or normal Eclipse and it provides Javadoc for some standard XPages classes - specifically, those covered in the official Javadoc for Designer and the XPages Extensibility APIs. This covers things like com.ibm.commons and the core stuff from com.ibm.xsp, but doesn't cover things like javax.faces.* or lotus.domino.

The way this works is that it uses Eclipse's Javadoc extension point to tell Designer/Eclipse that it can find Javadoc for a couple bundles via the hosted version, really just linking the IDE to the public HTML. I went this route (as opposed to embedding the Javadoc in the plugin) because the docs don't explicitly say they're redistributable, so I have to treat them as not. Interestingly, the docs are actually still hosted at public.dhe.ibm.com. If HCL publishes them on their site or makes them officially redistributable, I'll be able to update the project, but for now it's relying on nobody at IBM remembering that they're up there.

In any event, it's not a huge deal, but it's actually kind of nice. Being able to have Javadoc for things like XspLibrary removes a bit of the guesswork in using the API and makes the experience feel just a bit better.

Yesterday, I wrote a post detailing my recent switch to Apple Silicon, and in it I noted that the main remaining problem I was butting heads with was running Domino in Docker containers, specifically in the context of working with Java.

While my general solution of connecting to Docker running remotely is good, it's all the better to have this stuff working locally. I gave it another shot today and found that the solution is in the post: disable the JIT.

Java uses the JIT - its just-in-time compiler - to speed up execution dynamically when running a program, and it's part of what makes Java surprisingly fast in practice. It's a whole rabbit hole of performance implications and comparisons, but it will suffice to say "JIT = good". However, as I found when running test cases that load libnotes.dylib directly, J9's JIT doesn't play well with the x64-to-ARM JIT that these Macs use, either Rosetta's or qemu's.

So, long story short, the solution was to modify my testing container to disable the JIT when in such an environment (the referenced commit has since been followed up a few times).

First off, I added a bit in my Domino One-Touch Config file to point to a Java options file in the notes.ini:

1
2
3
4
5
6
7
8

{
	/* ... */
	"notesINI": {
		/* ... */
		"JavaUserOptionsFile": "/local/JavaOptionsFile.txt"
	}
	/* ... */
}

I modified the Dockerfile to bring this in from the build environment:

1
2

# ...
COPY --chown=notes:notes staging/JavaOptionsFile.txt /local/

Finally, I modified the class I use to build the container to optionally populate this file with the flag to disable the JIT:

1
2
3
4
5
6

String arch = DockerClientFactory.instance().getInfo().getArchitecture();
if(!"x86_64".equals(arch)) {
	withFileFromTransferable("staging/JavaOptionsFile.txt", Transferable.of("-Djava.compiler=NONE"));
} else {
	withFileFromTransferable("staging/JavaOptionsFile.txt", Transferable.of(""));
}

(Transferable is a Testcontainers-ism for any available source of a file to put into the build environment. Often, this will be something from the project classpath, but it can also be arbitrary data like this.)

With that in place, the test suite works! Eventually!

The thing to know about this is that it's slow. It's going to be naturally slow due to emulation, and I imagine the lack of Java's JIT doesn't help anything. Running the XPages JEE test suite takes about 520 seconds in this setup, as compared to 180 seconds when run via my remote native VM. Such a dramatic drop in speed is enough that I'm likely to continue using the remote VM for most things, and only use local emulation for things that significantly benefit from filesystem binds. For what it's worth, it seems like multithreading is what really kills it: most tests are slower by roughly 50%, while a big multithread test balloons from 20s to 160s. That's consistent with my experience with local unit tests, where the multithread ones were the ones that crashed the process.

For the record, I found that this currently only works at all with Docker Desktop's default (qemu-based) emulation. When I switch the emulator to Rosetta, I hit a StackOverflowError from the HTTP JVM during init without any useful other information. That's fair enough, since that emulation type is still flagged as Experimental in Docker Desktop anyway.

Anyway, it's good to know that there's a way to make it work. It's still no replacement for a native container, but at least it works at all. Similar techniques should work with other tasks, like setting the above option in the JAVA_OPTS environment variable.

Almost exactly two years ago now, I wrote a post describing using Apple Silicon for work while my iMac was in the shop. That was an interesting experience, but too thorny for me to want to really stick with. Some of that was due to the limitations of the hardware - it was a pretty low-end MacBook Air - but most of that was due to the very-much-in-progress world of Java tooling for ARM Macs.

Since then, I'd been itching to find a machine to replace the iMac Pro and the recently-released Mac mini finally fit the bill. It, now dubbed Tethys, arrived on Friday, and I've been going through the process of re-carving-out my working environment, this time starting clean.

Eclipse

Compared to my first dive into ARM Macs, things have gotten a lot smoother. The biggest one for me is the stability of Java generally and Eclipse specifically. Last time, I was on the bleeding edge of that, and I was fortunate enough to at least help diagnose and report some of the things barring the port from being complete. The experience was very janky but the sheer snappiness of Eclipse on ARM really stuck with me, and it's still there in spades.

And, really, there's not even a lot of special things to know. If you download the aarch64 build of Eclipse, it just works like you'd want it to. I had to make sure to download some x64 builds of Java for when I'm running legacy native code, but that wasn't really different from my current collection of different Java versions anyway.

This is all a huge distinction from before, where I was going to weird lengths to try to run Eclipse via X11 remotely just to get something responsive.

Notes and libnotes

For my needs, Notes works fine. It's a shame that it's not native, but, as long as I run my test suites with an x64 JVM, things work about as well as they do on x64.

One immediate problem I ran into, though, was that some test cases that push multithreading would hard crash with a native error to do with the x64-to-ARM translation and JIT. Running with an OpenJ9-based JVM, I found that specifying -Djava.compiler=NONE in the launch args avoided this trouble. I'm sure that the tests run a bit slower now, but they were already emulated anyway, so it's not noticeable.

Domino and Docker

One of the sticking points with my half-transition years ago was the need to run my Domino Docker containers in emulation. Though HCL will have to port Domino to ARM eventually, they haven't yet released such a thing, and so that situation remains the same.

And, unfortunately, the segfault I encounter during my use remains. I had hoped that changed in qemu would fix it, but it appears to be unchanged. Recent versions of Docker can also take advantage of Apple's support for Rosetta in Linux, but for my needs that just creates a different segfault earlier in the process.

So, on this front, I'm doing the same thing I started doing before: running Docker on a VM on actual x64 hardware and using the DOCKER_HOST environment variable to point to it. Fortunately, I've taken steps in the intervening years to make this more practical. The main thing to work around with such a setup is the inability to use filesystem binds, and so I've changed a lot of my Dockerfiles and Testcontainers setups to instead copy more into the image at build. I still haven't solved every problem there, in particular a huge in-container build I'd like to do that points at a giant pool of dependencies that would be onerous to put into the image, but I'm working on that.

Conclusion

Overall, I'm pleased as punch with this thing. It's zippy all around, I haven't once heard the fan even when really hitting the CPU and GPU, and the state of the software ecosystem is such that I only use a very few x64-compiled processes during my daily work. I'd expected to have more of a checklist to work down to clean them all up, but it's really just the for-now-required bits. Not bad at all.

For a good long time now, XAgents have been one of the common ways to do non-HTML output in an XPages environment - JSON, mostly. I think the technique was codified and the term coined by Stephan Wissel back in 2008 and the idea has been the same since.

Effectively, an XAgent lets you write a Servlet but with a bit more scaffolding. Though XPages has a path to use Servlets officially, that method is more out-of-the-way than XAgents and doesn't (without further hoop jumping) give you some niceties like sessionAsSigner.

However, though they're venerable and sort of convenient, XAgents are lacking in a number of ways. For one, stuffing them inside an XPage is ungainly: even if the XPage just calls out to some Java code, you're polluting your UI-element space with something kind of unrelated, forcing you to name your XPages stuff like "apiEmployees.xsp". More importantly, though, it doesn't provide a lot of affordances for the sort of work you'd actually want to do when writing a REST API. Though XPages has a couple mechanisms for generating JSON, little of that is exposed by the XPage editor environment, and I suspect that many or most XAgents writing JSON commit the cardinal sin of doing so through direct string concatenation, likely often without much escaping. Further, there's no built-in mechanism for picking apart path segments or multi-branched routing.

So I figured today is a good opportunity to talk a bit about one of the XPages Jakarta EE Support project's flagship features: writing REST services that consume and emit JSON. This post covers some of the same ground I've talked about before, but sometimes it's useful to re-contextualize this sort of thing.

The XAgent Way

To level-set the discussion, I'll give a starting example in an XAgent, and we'll keep it simple. The idea here will be that you have Person documents in a database and you want to write an API that will take a UNID and emit the corresponding person's first and last name. This is very similar to the Employees example in the "Code-First REST API" example project from the JEE repo, but without diving into more-complex parts or the Jakarta NoSQL data layer.

In an XAgent, you might do something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core" viewState="nostate" rendered="false">
	<xp:this.afterRenderResponse><![CDATA[#{javascript:
		var resp = facesContext.getExternalContext().getResponse();
		resp.setContentType("application/json");
		
		var writer = facesContext.getResponseWriter();
		
		var doc = database.getDocumentByUNID(param.unid);
		var result = toJson({
			"firstName": doc.getItemValueString("FirstName"),
			"lastName": doc.getItemValueString("LastName")
		});
		writer.write(result);
		
		writer.endDocument();
		facesContext.responseComplete();
	}]]&gt;</xp:this.afterRenderResponse>
</xp:view>

You'd put that in an XPage and call it like:

1
2

$ curl http://your.server/someapp.nsf/apiPeople.xsp?unid=68D1CAA80F65780D8525894D006B1CE7
{"lastName":"Fooson","firstName":"Foo"}

That does the job well enough. However, this will get gangly very fast. If you want to expand on the types of data the Person document contains, add more operations to the API, add other query parameters, or so forth, you'll have to either have a giant blob of code here, spin it off to an SSJS script library, or move it to Java classes. And, all along the way, the dev environment won't be giving you any help with the process - it knows nothing about writing REST APIs, and so it's all "manual". Better than with a classic agent, but not great.

Jakarta Way, Take 1

So let's move this over to Jakarta EE. We'll start by doing basically a concept-for-concept move: just take the above and make a REST class out of it.

Such a class could look something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

package rest;

import com.ibm.xsp.extlib.util.ExtLibUtil;

import jakarta.json.Json;
import jakarta.json.JsonObject;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.core.MediaType;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.PathParam;
import jakarta.ws.rs.Produces;

import lotus.domino.Database;
import lotus.domino.Document;
import lotus.domino.NotesException;

@Path("v1/person")
public class PersonResourceV1 {
	
	@Path("{unid}")
	@GET
	@Produces(MediaType.APPLICATION_JSON)
	public JsonObject get(@PathParam("unid") String unid) throws NotesException {
		Database database = ExtLibUtil.getCurrentDatabase();
		Document doc = database.getDocumentByUNID(unid);
		
		return Json.createObjectBuilder()
			.add("firstName", doc.getItemValueString("FirstName"))
			.add("lastName", doc.getItemValueString("LastName"))
			.build();
	}
}

This one you'd call in a similar way, and get similar results:

1
2

$ curl http://your.server/someapp.nsf/xsp/app/v1/person/68D1CAA80F65780D8525894D006B1CE7
{"firstName":"Foo","lastName":"Fooson"}

There's a bit more boilerplate - it is Java - but you can already see some of the benefits. The URL is a little more REST-like and the code is much more task-focused. Instead of disabling all the normal features of the XPage and manually emitting text, a large amount of the code is explicitly describing what you intend to do in a REST service. The @Path annotations describe the components of the URL, and the use of @Path("{unid}") lets us name one of those parts without having to do a query string. Similarly, the @GET and @Produces(MediaType.APPLICATION_JSON) lines directly say what's going on: you can do a GET request to the URL and get JSON back.

These annotations pay off in a couple ways. First of all, the code will be a lot easier to understand when you come back later. Imagine if your API also handled new documents, deletions, and modifications - with an XAgent, you'd have branching paths and would have to rely on either good naming or commenting to mentally traverse it. With this, you would have other methods with similar annotations - @PUT, @POST, @DELETE - and would see exactly where requests are going to go.

And it's not just for you the programmer (or for the next human replacing you when you retire to a beach somewhere): these annotations mean something to tools that process it as well. One such tool comes with the XPages JEE project: the MicroProfile OpenAPI generator. With this class present, you automatically get a useful OpenAPI spec:

1

$ curl http://your.server/someapp.nsf/xsp/app/openapi.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

---
openapi: 3.0.3
info:
  title: XAgent Comparison
servers:
- url: http://your.server/someapp.nsf/xsp/app
paths:
  /v1/person/{unid}:
    get:
      parameters:
      - name: unid
        in: path
        required: true
        schema:
          type: string
      responses:
        "200":
          description: OK
          content:
            application/json:
              schema:
                type: object

This sort of thing pays off tremendously once you start working with a client JS app, especially with a larger development team.

Jakarta Way, Take 2

But we can do better than this. As it is, the direct translation from the XAgent still left the actual output pretty vague. We at least know it's emitting a JSON object, but that's the extent of it. Moreover, the app code wastes some conceptual time actually building the JSON object, which is okay but unnecessary. Let's bring JSON Binding into the mix. This will increase the size of the code, but will pay off in conceptual cleanliness and in future work.

That could look like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

package rest;

import com.ibm.xsp.extlib.util.ExtLibUtil;

import jakarta.validation.constraints.NotEmpty;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.core.MediaType;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.PathParam;
import jakarta.ws.rs.Produces;

import lotus.domino.Database;
import lotus.domino.Document;
import lotus.domino.NotesException;

@Path("v2/person")
public class PersonResourceV2 {
	
	// Normally, this class would be in a separate file
	public static class Person {
		private String firstName;
		private @NotEmpty String lastName;
		
		public String getFirstName() {
			return firstName;
		}
		public void setFirstName(String firstName) {
			this.firstName = firstName;
		}
		public String getLastName() {
			return lastName;
		}
		public void setLastName(String lastName) {
			this.lastName = lastName;
		}
	}
	
	@Path("{unid}")
	@GET
	@Produces(MediaType.APPLICATION_JSON)
	public Person get(@PathParam("unid") String unid) throws NotesException {
		Database database = ExtLibUtil.getCurrentDatabase();
		Document doc = database.getDocumentByUNID(unid);
		
		Person result = new Person();
		result.setFirstName(doc.getItemValueString("FirstName"));
		result.setLastName(doc.getItemValueString("LastName"));
		return result;
	}
}

Calling this will work the same way as last time, but with a "v2" to indicate our new-and-improved back end:

1
2

$ curl http://your.server/someapp.nsf/xsp/app/v2/person/68D1CAA80F65780D8525894D006B1CE7
{"firstName":"Foo","lastName":"Fooson"}

What does this get us? Well, for one, we're no longer explicitly working with JSON, which is nice. We could change the media type to XML and the output type will automatically adapt, and the same will go for any future formats we write an adapter for. Additionally, this will work better with a more-structured database access layer. I'll leave that part out here, but the aforementioned "Code-First REST API" example shows that.

Some more of the payoff shows up when we check the generated OpenAPI spec. If we make the same call as above, the output will be more descriptive now:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

---
openapi: 3.0.3
info:
  title: XAgent Comparison
servers:
- url: http://your.server/someapp.nsf/xsp/app
paths:
  /v2/person/{unid}:
    get:
      parameters:
      - name: unid
        in: path
        required: true
        schema:
          type: string
      responses:
        "200":
          description: OK
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Person'
components:
  schemas:
    Person:
      required:
      - lastName
      type: object
      properties:
        firstName:
          type: string
        lastName:
          minLength: 1
          type: string

For someone consuming this spec, this is starting to get really good. No longer does the output just say it's a generic JSON object: now it describes it as "Person", shows the two properties that will be output, and declares that lastName will always be at least one character both for input and output. As you add more operations and types, this spec will continue to grow, naturally piggybacking on the annotations and types you write.

Conclusion

Strictly speaking, you can do the same stuff with an XAgent that you can with the XPages JEE project. You could manage the internal complexity and you could manually write the OpenAPI spec, but it would be much, much harder to do, and I think it'd be a safe bet to say that very few XAgent-based APIs have specs to go with them anyway.

Especially as the scope of your app grows, the development and maintenance experience with the JEE approach is night-and-day compared to classical mechanisms like XAgents. If you're still writing APIs that way or similar, I definitely recommend you give this path a try.

Keeping with my productive week off, today I release version 2.9.0 of the XPages Jakarta EE Support project. Similar to the previous release, this one contains new features primarily related to Jakarta NoSQL, but also has some improvements for JSF and a bunch of bug fixes and compatibility improvements.

Jakarta NoSQL

The improvements to the JNoSQL driver come from some needs I came across when moving older lotus.domino/ODA-based code to using JNoSQL repositories. In particular, I added the remaining applicable view entry properties as available fields to map, added better support for reading note IDs, and fetching documents by note ID.

JSF

While JSF support remains limited by not having a proper way to add in third-party component libraries like PrimeFaces, it's still a potentially-compelling tool in an NSF as an alternative to XPages in some cases. Accordingly, I fixed a few bugs I had run into when loading pages after modifying the NSF design. Additionally, I fixed up support for JSF as an MVC view engine. It now properly joins JSP as a mechanism for rendering your output with an MVC structure, and I think there's some real potential there.

Bug Fixes and Compatibility

Most of the other closed issues deal with a few bugs here and there, and in particular involve some improvements for running apps in XPiNC and on a server with Domino Leap also installed. I don't use XPiNC anymore and haven't tried Leap, so I greatly appreciate bug reports specific to these and the assistance in tracking down the trouble.

The Future and Next Steps

I'm pondering now what the next release of the project will focus on. I have no shortage of feature ideas, and there are a few potentially-disruptive changes I'd like to make.

Unfortunately, those changes will be largely confined to improving the support for the specs that are already present and not advancing to new versions. The predicted Java-version wall arrived: Jakarta EE 10 is out and requires Java 11 and above. Since Domino remains mired in Java 8, that means that new versions of the specs and implementations are hard-incompatible until that changes.

On the plus side, there's still a lot of improvements I can make with Jakarta EE 9 as the baseline.

Reorganization

One big one I've been thinking about is a reorganization of the individual libraries that make up the project. The way it's been designed, almost every spec has its own Equinox Feature and XPage Library to go with it. This was fine early on when it was just CDI, EL, and JAX-RS, but it's grown annoying: installing the project in Designer is a seemingly-endless process of approving each plug-in at a time and the list of libraries to check in Xsp Properties is interminable. More critically, being able to selectively turn on and off specs like this doesn't make sense anymore. CDI has grown so important to Jakarta EE in general and this spec in particular that it doesn't make sense to not have it present if you're going to use this project at all. It's a foundational component of so many other parts and is essentially The Way to do Jakarta-based development.

So I'm thinking I'm going to reorganize the projects into fewer features and libraries, which will be a breaking change that will necessitate a bump to 3.0 - fortunately, the numbers line up well for that. I have a few potential options here:

1. Just lump them all into one. You'd have basically one big switch to say "this is a JEE project" and everything would be on. The virtue here is that this is how I already work and is essentially the recommended way to do things. Additionally, as far as I know, while having additional components may slow first load (though not as much as other parts), I don't think they have a significant impact if enabled but unused during runtime.
2. Try to line the specs up with one of the existing Jakarta Profiles. Those profiles are meant to be curated selections of useful specs, and this project has enough to implement what in newer versions is deemed the Core Profile. The trouble with this, though, is that the Core Profile is very much geared to be the shared subset with MicroProfile and similar and is a bit thin for Domino's monolith-focused development style. The Web and Full profiles, on the other hand, require "traditional" APIs like EJB that are not present in this project.
3. Break them apart into my own "core" and "optional" features. For example, it doesn't make sense to use this without JAX-RS, CDI, and Bean Validation enabled, but JSF is entirely independent of the other specs and is among the least likely to be used in practice for now. This would also allow me to establish a running flow where "experimental" features start out as optional add-ins and then eventually make their way to core.

I'm currently waffling between #1 and #3, with a slight lean towards #1. If I can be sure that either everything or nothing is present, I could get rid of some weird hedges and workarounds, like how the JAX-RS implementation doesn't "officially" know about the CDI library yet references CDI classes explicitly by name.

New Application Types

Currently, to use this project, you can either put your code into an NSF and use the automatic behavior of the libraries or you can put your code in OSGi-based webapps or Servlets and then manually manage integration with these specs.

Both of these are limited by their reliance on the many assumptions IBM built in to how these apps should work. In-NSF apps require that all Jakarta code come from a request including "xsp" in the URL or to a file ending in ".jsp", ".jsf", or ".xhtml". If you're writing, say, an MVC-based app, all of your URLs are going to have to start with something like "foo.nsf/xsp/app/...", which is okay but ugly. Additionally, the way these apps are implemented - NSFComponentModule - severely limits my hooks for listening for things like application and session expiration, which hampers CDI's lifecycle handling a bit.

For a good while, I've pondered the notion of adding another ComponentModule type to handle the case where you want to go all-in on Jakarta EE. With this idea, the new module implementation would have full control over incoming requests, allowing URLs without the xsp/app bit in there, and would have better handling of lifecycles. In this way, I could make it so that your could would look more like (or be identical to) a "normal" .war-based webapp, with fewer workarounds for the existing XPages stuff. This would also allow me to do things like lessen the amount of Servlet 2.5-to-5.0 bridging and could assist tremendously in improving JSF support.

Along similar lines, I've been considering doing something similar for OSGi-based webapps, and I've made some progress along those lines in a feature branch. The idea here would be to do something similar to how you can deploy web.xml-based webapps via OSGi now, but with built-in support for Jakarta EE 9 features (with web.xml then being optional). With this setup, you'd be able to write an app that does an Import-Package for the various jakarta.* packages you want and add a bit in your MANIFEST.MF to signal to this project that it should participate. This could either be a variant of the extension point used by the existing OSGi webapp support or using the Web-ContextPath directive from the OSGi spec. One of the goals here would be to make it so that you would be able to write a Jakarta EE 9 app using normal development tools - Eclipse/IntelliJ/VS Code, Maven, etc. - and then just use maven-bundle-plugin to add the OSGi info you need without having any specific dependencies on Domino bits, especially the nightmare of depending on the non-redistributable XPages OSGi artifacts.

Other Options

And, in the mean time, I have a bunch of other tasks I could work on. Slowly converting my client project to Jakarta NoSQL instead of direct ODA use has turned up a whole slew of things that would be useful to add (for example, stampAll support), so I can slowly burn down that feature-request list.

There's also the notion of documentation! While a lot of the behavior of this project is in theory documented by virtue of the upstream specs and the general world of Jakarta blogs, videos, and courses, there's enough to know about the specifics of the interactions with Domino that more documentation is in order. Historically, I've just done this by expanding the README, but it's gotten pretty unwieldy at this point. It would probably make sense to break the specifics and examples out into at least wiki pages, if not a format that can be built into a PDF/etc. and included in the distribution.

So yep, I'll have my hands busy with this thing for a good while more, I figure.

The other day, I talked about some changes/workarounds for Notes/Domino 12.0.2. Today, I made a few updates to some of the open-source projects I maintain, including another update to the generate-domino-update-site Maven plugin.

Domino Update Site Generator

In the 4.2.0 release, I added code to (mostly, as it turns out) account for HCL moving the NAPI implementation JAR down to jvm/lib/ext. In subsequent use, I found that, while that will suffice for building applications that use the OSGi dependencies, it didn't work for launching applications using it as a baseline - namely, the NSF ODP Tooling Maven plugin.

Today, I released a 4.2.1 version that improves this behavior by re-adding dependencies in the implementation bundle.

I also created a project page for it on OpenNTF. Though the project has always been hosted at the OpenNTF org on GitHub, I hadn't created a project page for it due to it just being a standalone Maven plugin. I figured it'd be useful to create an official page there for it, though.

NSF ODP Tooling

Speaking of the NSF ODP Tooling, I also found that local operations once again started crashing on macOS. Due to changes in macOS and the very weird ways that Notes works, local operations on there are a very moving target, and I have to do a lot of work in the project to account for changes to the embedded JVM and whether specific Notes versions work better with HotSpot or OpenJ9 JVMs.

Long story short, I release 3.10.0 today to account for this. Though I've found that the spawned JVM will still sometimes crash, it's after completing its work, so I considered that fine for now.

OpenNTF Domino API

Finally, I came to the OpenNTF Domino API. This project has admittedly been neglected for a little while: I'm the only active maintainer, and the client project I use it in targets Domino 11.0.1, so the 12.x builds have remained in an incomplete state for a while.

With the release of 12.0.2, I decided I should finish the wrappers for the new classes added in 12.x, so I did so and uploaded a build for 12.0.2. This primarily adds those wrappers, but also included a contributed fix and changes the distribution packaging to combine the XSP and non-XSP versions.

Notes and Domino 12.0.2 came out today. Generally, there are some neat features in development and on the server, but there are also a couple things you may run into depending on your workflow and installation type.

Update Site NSF

The update site NSF that ships with Domino uses SWT for some of its GUI elements when importing contents. This still works fine in the 32-bit client, but is broken in the 64-bit client. My guess on that front is that the 64-bit client doesn't come with a 64-bit native SWT JAR, probably because the SWT version used for this likely pre-dates x64's popularity on the desktop.

For now, the workaround is to use a 32-bit client if you're working with the Update Site NSF. Karsten said he's going to patch the OpenNTF version of the NSF to deal with this, so you can also wait for that one.

Domino Update Site Generator

I maintain the generate-domino-update-site Maven plugin that can be used to generate update sites for OSGi development against the Domino stack. These sites are replacements for the IBM Domino Update Site for Build Management, which was released for 9.0.1 and never updated since. Only HCL can make a new distributable version of that, so my tool lets you generate one for yourself from a Notes or Domino installation.

In 12.0.2, HCL shunted the NAPI implementation JAR down to jvm/lib/ext to support the shared JARs between XPages and agents feature. As a side effect, existing versions of my plugin would lack the NAPI classes.

Today, I released version 4.2.0, which fixes this and also contains improvements to let the plugin work on current Java versions and generate sites based on 12+ macOS clients.

Along these lines, I made Aha idea DDXP-I-352 a couple years ago to request that HCL provide such sites themselves or give OpenNTF permission to provide them, so I'd appreciate it if you voted for that.

The Target Platform Bug

This isn't a new thing, but it's worth mentioning here since it comes up frequently: the target platform bug from 9.0.1FP10 remains. As of earlier this year, defect article KB0086688 mentions this, though the status is "Deferred". If this afflicts you, it may help to bring it up with Support and reference that article.

A few years ago, I wrote a post about Java service location, which covered things like META-INF/services and OSGI extensions. Today, I'd like to discuss a similar concept: code in a top-level API that finds a specific implementation. For reasons that will become clear shortly, I'll call this the "FactoryFinder pattern".

Background

Not all Java code uses this kind of thing and, while service loading is related, the overlap isn't complete. Where this does come up a lot is in a framework like Jakarta EE, which is very intentionally split between vendor-neutral specification classes/interfaces (the ones starting with jakarta.*) and specific implementations.

For example, the Jakarta REST (n?e JAX-RS) specification only defines various classes and interfaces within the jakarta.ws.rs package space, but doesn't include any actual implementation. That's left to various vendors. The number of implementations varies by spec, and JAX-RS is particularly prolific on this front. In the XPages Jakarta EE project, we use RESTEasy, whose classes are all in the org.jboss.resteasy package space.

There's a (usually) hard wall between these layers: the spec declares an API that programmers can use, and then the implementation has to allow itself to be called by those class names and obey the specification's rules. When writing JAX-RS resources in an NSF, the fact that it's using RESTEasy does not enter into your experience. That raises the question, though, of how this works. How does the vendor-neutral specification locate the implementation classes to hand off the work? Well, that question has a number of different answers.

Entrypoint Classes and Locating Implementations

In general, each spec accomplishes this using one or more entrypoint classes. For example, JAX-RS uses RuntimeDelegate and its static getInstance() method to locate server implementations and ClientBuilder and its newBuilder() method to load client implementations. Outwardly, these methods just promise that they'll find and provide an implementation, but the actual way that specs do this varies.

One of the most common ways to coordinate this loading is to have a class named FactoryFinder. This idiom and specific name proved very popular over at Sun as they built up the JEE specs:

Despite their identical names, each of these classes is a different implementation, and they have different characteristics. There are routines in common, and each spec uses a subset of these. I'll go over the common ones here, in no particular order other than that I'll start with the ones found in the JAX-RS API first.

ServiceLoader

This one is used in basically every spec up until the latest era. This uses the java.util.ServiceLoader class to find implementations by way of text files in META-INF/services named after the spec class and containing implementation class names. For example, RESTEasy contains a file named META-INF/services/jakarta.ws.rs.ext.RuntimeDelegate that references the class org.jboss.resteasy.core.providerfactory.ResteasyProviderFactoryImpl. That looks like this:

1
2
3
4
5

Iterator<T> iterator = ServiceLoader.load(service, FactoryFinder.getContextClassLoader()).iterator();

if(iterator.hasNext()) {
	return iterator.next();
}

FactoryFinder.getContextClassLoader() there is a utility method that just uses an AccessController block to work with Java policy limitations like we see on Domino all the time.

This is simple enough in the normal case, but can get a little tricky when you add in something like OSGi. By default, ServiceLoader will look in the thread-context class loader, which will usually be where your application code lives. Inside an app container, like an NSF, the implementation class may not actually be visible, though. Accordingly, many of these finders fall back to looking using the class loader of the spec class, which has a higher chance of seeing the implementation. That looks similar:

1
2
3
4
5

Iterator<T> iterator = ServiceLoader.load(service, FactoryFinder.class.getClassLoader()).iterator();

if(iterator.hasNext()) {
	return iterator.next();
}

In the XPages Jakarta EE project, neither of these calls will tend to work by default, since neither the app nor the API bundle won't see the implementation bundle by default. In some cases, I deal with this via the methods below, but in others I will do so by re-packaging the implementation as an OSGi fragment bundle. Fragment bundles attach themselves onto their host's classloader fully, and this allows ServiceLoader to find the implementation.

Configuration Properties

A handful of these specs, JAX-RS included, will also look for the name of an implementation class using an external properties file. The placement of this in the priority order - as a fallback after ServiceLoader - and the classes used in the implementation make me figure that these are quite often relics of earlier habits.

JAX-RS, for its part, will look within the java.home system property, which points to the JVM's installation directory. In there, it looks for a properties file named lib/jaxrs.properties:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

String javah = System.getProperty("java.home");
configFile = javah + File.separator + "lib" + File.separator + "jaxrs.properties";
File f = new File(configFile);
if (f.exists()) {
	Properties props = new Properties();
	inputStream = new FileInputStream(f);
	props.load(inputStream);
	String factoryClassName = props.getProperty(factoryId);
	return newInstance(factoryClassName, classLoader);
}

This tries to use the thread-context class loader only, so it wouldn't work for a complex app server situation. Likely, it's meant for either an older type of application or a standalone special-purpose JAR.

System Properties

Similar to reading a designated properties file, these specs will often then fall back to looking for a Java system property of a given name. These properties may be dynamically set at runtime or may be set during the JVM launch. Often, this property will be the name of the interface/abstract class being looked up, like so:

1
2
3
4

String systemProp = System.getProperty(factoryId);
if (systemProp != null) {
	return newInstance(systemProp, classLoader);
}

This one can actually come in handy sometimes - though not ideal, I've used similar cases where I set the name of an implementation or delegation class in a property before initializing the spec. It's best to avoid that when possible, but I'm often glad it's there.

OSGi Escape

Next up is one that JAX-RS doesn't use, but shows up periodically. Though Jakarta EE isn't based around OSGi, a good number of the implementations historically have used (and still use) it, and OSGi always sits in a "not standard, but too popular to consistently ignore" limbo.

To account for this, there's a similarly semi-standard library called the OSGi resource locator. This library provides a class named org.glassfish.hk2.osgiresourcelocator.ServiceLoader that does its own search and loading for ServiceLoader-compatible META-INF/services files within OSGi bundles in the current platform. The idea is that, if you have an OSGi-based platform that you want to work with this type of loading, you will provide the Resource Locator class and let any loaders written to use it fall back to it.

Because this class is not normally present even when actually in OSGi, APIs that make use of it have to be careful and indirect about trying to load it at all. We'll use JAX-B as our example here. They'll generally try to load the bridge class reflectively, which avoids having OSGi-wrapping tools like bnd create a potentially-undesired dependency on the presence of the bridge. Then, they'll reflectively ask it to load service implementations. That tends to look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

// Use reflection to avoid having any dependency on ServiceLoader class
Class serviceClass = Class.forName(factoryId);
Class target = Class.forName(OSGI_SERVICE_LOADER_CLASS_NAME);
Method m = target.getMethod(OSGI_SERVICE_LOADER_METHOD_NAME, Class.class);
Iterator iter = ((Iterable) m.invoke(null, serviceClass)).iterator();
if (iter.hasNext()) {
	Object next = iter.next();
	logger.fine("Found implementation using OSGi facility; returning object [" +
		next.getClass().getName() + "].");
	return next;
} else {
	return null;
}

That's also generally wrapped in a big try/catch block to avoid gumming up the works if any pieces are missing.

The XPages Jakarta EE project actually contains a reimplementation of this that avoids some hurdle or other that I found with the stock version. I avoided doing something like that for a while, but it ended up being the most practical way to get some of these specs working.

Default Implementation

Back outside the realm of OSGi, a handful of these specifications will also include a hard-coded default provider class name. These are generally the classes from what used to be dubbed reference implementations and which are largely components of GlassFish by virtue of that being Sun's version.

For example, the JSON-P API has a final fallback of trying to look for org.glassfish.json.JsonProviderImpl by name:

1
2

Class<?> clazz = Class.forName(DEFAULT_PROVIDER);
return (JsonProvider) clazz.getConstructor().newInstance();

Though these implementations generally also declare themselves via ServiceLoader files, this is presumably useful in historical or edge cases where there's still a decent chance that the RI will be available. This does have an unfortunate effect on error messages, though, where the case of "I can't find any implementation at all" ends up being reported as e.g. "Provider org.glassfish.json.JsonProviderImpl not found". That's not really a problem with the approach as such, though, but rather just the way it shakes out in practice.

Manually-Set Implementation or Locator

The final mechanism I'm going to discuss is sort of a final escape hatch. Sometimes, the provider class will have a method that lets you set an arbitrary implementation yourself, without having the API do any of these lookups at all. Some, like MicroProfile Config and CDI even go one step further and provide a method that configures not just a specific implementation but rather an implementation locator. These APIs are my friends and I love them.

This mechanism works well for my needs in the XPages Jakarta EE project, where either it's easier to just set one implementation for the whole server or, like with CDI, there's complex logic that requires inspecting the active Servlet request to see what NSF I'm in.

APIs of this style will usually have a method named like setInstance or setProvider on either their core entrypoint class or on the provider locator. For example, MicroProfile Config provides the former on its ConfigProviderResolver class:

1
2
3

public static void setInstance(ConfigProviderResolver resolver) {
	instance = resolver;
}

instance here is a static property. Once it's set - either by this method or by a dynamic lookup - the main instance() method will use it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

if (instance == null) {
	synchronized (ConfigProviderResolver.class) {
		if (instance != null) {
			return instance;
		}
		instance = loadSpi(ConfigProviderResolver.class.getClassLoader());
	}
}

return instance;

The XPages JEE project makes use of this method at HTTP start, setting a provider resolver that includes some stock config sources as well as some classes that know how to read properties from the Notes environment and from the xsp.properties file.

Though this mechanism seems like the crudest out of the bunch, I'm extremely happy whenever it's there.

Conclusion

That was a lot! And there's not really a lesson to be learned here, but rather more that it's often useful to know about all these different mechanisms. When working in the XPages JEE project, I've had to use almost all of them at one time or another, and I've had to familiarize myself with which APIs use which and adapt them individually. For some, I've altered the implementation to be a fragment bundle; for others, I've created my own fragment to provide services and implementations; and so forth. It's a bit of a shame that there's no grand unified system for this, but at least it can be interesting to see the messy path that these specs have taken as Java technologies and the ecosystem evolved.

In my last post, I talked about how I implemented a partial Jakarta NoSQL driver using the AppDev Pack as a back end instead of the Notes.jar classes used by the primary implementation. Though the limitations in the ADP mean that it lacks a number of useful features compared to the primary one, it was still an interesting experiment and has the nice side effect of working with essentially any Java app server and Java version 8 or above.

Beyond the Proton API calls, the driver brought up the interesting topic of handling authentication. Proton has three ways of working in this regard:

• Anonymous, which is what you might expect based on how that works elsewhere in Domino. This is easy but not particularly useful except in specific circumstances.
• Client certificate authentication, where you create a TLS keychain for a given user and associate it with a Directory user (e.g. CN=My Proton App/O=MyOrg), and then your app performs all operations as that user. This is basically like if you ran a remote app with NRPC using a client Notes ID.
• Act-as-User, which builds on the above authentication by configuring an OAuth broker service that can hand out OIDC tokens on behalf of named users. This is sort of like server-to-server communication with the "Trusted Servers" config field in the server doc, but different in key ways.

Client Certificate Authentication

When doing app development, the middle route makes sense as your starting point, since most of your actual work will likely be the same regardless of whether you later then add on act-as-user support. For that, you'll follow the guide to set up your TLS keychain and then feed those files to the com.hcl.domino.db.model.Server object:

1
2
3
4
5
6
7

Path base = Paths.get(BASE_PATH);
		
File ca = base.resolve("rootcrt.pem").toFile();
File cert = base.resolve("clientcrt.pem").toFile();
File key = base.resolve("clientkey.pem").toFile();

Server server = new Server("ceres.frostillic.us", 3003, ca, cert, key, null, null, Executors.newSingleThreadExecutor());

The use of java.io.File classes here is a bit of a shame, but not the end of the world. In practice, you'd likely store your keychain somewhere on the filesystem anyway and then feed the BASE_PATH property to your app via an environment variable. Otherwise, if they were pulled from some other source, you could use Files.createTempFile to store them on the filesystem while your app is running. Those nulls are for the passphrases for the certificates, so they might be populated for you. For the last parameter, making a new executor is fine, but you might want to hand it a ManagedExecutorService.

You can initialize this connection basically anywhere - I put it in a ServletRequestListener to init and term the object per-request, but I think it would actually be fine to do it in a ServletContextListener and keep it app-wide. I did it out of habit from Notes.jar and its heavy requirements on threads and the way Session objects are different per-user, but that's not how Proton connections work.

This form of authenticate is a prerequisite for Act-as-User below, but it might suffice for your needs anyway. For example, if you have a "utility" app, like a bot that looks up data and posts messages to Slack or something, you can call it good here.

Act-as-User

But though the above may suffice sometimes, the integration of user identity with data access is one of the hallmarks of Domino, so Domino apps that wouldn't need Act-as-User support are few and far between.

Act-as-User is a bit daunting to set up, though. In traditional server-to-server communication in Domino, it suffices to just add a server's name or group to the "Trusted Servers" list in the server doc of the server being accessed. Then, it will trust any old name that the app-housing server sends along. Generally, this will be a user that was already authenticated with Domino, like using an XPages-supplied Session object to access a remote server, but it doesn't have to be.

Act-as-User, though, uses OpenID Connect with an authentication server to do the actual authentication, and then the Proton task is told to accept those tokens as legal for acting on behalf of a given user. While you could in theory write your own OIDC server that dispenses tokens for any user name willy-nilly, in practice you'll almost definitely use an existing implementation. In the default case, that implementation will in turn almost definitely be IAM, which is an OAuth broker service with the AppDev Pack that stores its configuration data in an NSF but and reads users from Domino (or elsewhere) via LDAP.

IAM, though, isn't special in this regard. It's packaged with the ADP, sure, but the way it deals with tokens is entirely standards-based. That means that any compatible implementation can fill this role, and, since I'd heard great things about Keycloak, I figured I'd give that a shot. With some gracious assistance from Heiko Voigt, I was able to get this working - I don't want to steal his future thunder by going into too much detail, but honestly the main hurdles for me were just around learning how Keycloak works. Once you have the concepts down, you basically plug in the Keycloak client details in for the IAM ones in the same configuration.

With that set up, you can feed your user token from the web app into your Proton API calls, and then your actions will be running as your user in the same way as if you were authenticated in an on-server XPages or other app. The way this manifests in the Java API is a little weird, but it works well enough: almost all Proton calls have a varargs portion at the end of their method signature that takes OptionalArg instances. One such type is OptionalAccessToken, which takes your auth token as a String. I have a method that will stitch in an access token when present. That gets passed in when making calls, such as to read documents:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

OptionalItemNames itemNamesArg = new OptionalItemNames(itemNames);
OptionalStart startArg = new OptionalStart((int)skip);
OptionalCount countArg = new OptionalCount(limit < 1 ? Integer.MAX_VALUE : (int)limit);
			
List<Document> docs = database.readDocuments(
	dql,
	composeArgs(
		itemNamesArg,
		startArg,
		countArg
	)
).get();

App Authentication

Okay, so that's what you do when you have a setup and a token, but that leaves the process of the user actually acquiring the token. From the user's perspective, this will generally take the form of doing an "OAuth dance" where, when the user tries to access a protected resource, they're sent over to Keycloak to authenticate, which then sends them back to the app with token in hand.

There are a lot of ways one might accomplish this, varying language-to-language, framework-to-framework, and server-to-server. You will be shocked to learn that I'm using Open Liberty for my app here, and that comes with built-in support for OIDC.

Before I go further, I should put forth a big caveat: I'm really muddling through with this one for the time being. The setup I have only kind of works, and is clearly not the ideal one, but it was enough to make the connection happen. I'm not sure if the right path long-term is to keep using this built-in feature or to switch to either a different built-in option or another library entirely. So... absolutely do not take anything here as advice in the correct way to do this.

Anyway, with that out of the way, you can configure your Liberty server to talk to your Keycloak server (or IAM, probably, but I didn't do that):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

<openidConnectClient id="client01"
    clientId="liberty-tester"
    clientSecret="some-client-secret"
    discoveryEndpointUrl="https://some.keycloak.server/auth/realms/master/.well-known/openid-configuration"
    signatureAlgorithm="RS256"
    sslRef="httpSsl"
    accessTokenInLtpaCookie="true"
    userIdentifier="preferred_username"
    groupIdentifier="groups">
</openidConnectClient>
<ssl id="httpSsl" trustDefaultCerts="true" keyStoreRef="myKeyStore" trustStoreRef="OIDCTrustStore" />
<keyStore id="myKeyStore" password="super-secure-password1" type="PKCS12" location="${server.config.dir}/BasicKeyStore.p12" />
<keyStore id="OIDCTrustStore" password="super-secure-password2" type="PKCS12" location="${server.config.dir}/OIDCTrustStore.p12" />

The keyStores there contain appropriate certificate chains for the TLS connection to your Keycloak server, while the clientId and clientSecret match what you configure/generate on Keyclaok for this new client app.

What got me able to actually use this token for downstream access was the accessTokenInLtpaCookie property. If you set that, then your HttpServletRequest objects after the initial one will have an oidc_access_token property on them containing your token in the format that Proton needs. So that's where the ContextDatabaseSupplier in the previous post got it:

1
2
3
4
5
6
7

@Produces
public AccessTokenSupplier getAccessToken() {
	return () -> {
		HttpServletRequest request = CDI.current().select(HttpServletRequest.class).get();
		return (String)request.getAttribute("oidc_access_token");
	};
}

This is also one of the parts that makes me think I'm not quite doing this ideally. It's weird that the token shows up in only requests after the first, though that wouldn't be an impediment in a lot of app types. It's also very unfortunate to have the app use a server-specific property like that.

Fortunately, Jakarta Security 3.0 sprouted official OIDC support, though the build of Open Liberty I was using didn't quite have all the pieces in place for that - reasonable, considering Jakarta EE 10 only officially came out yesterday and this was weeks ago. It looks like that may provide the token in a contextual object, so I'll have to give that a shot once support settles in.

With my setup in place, janky as it may be, I'm able to access a resource (e.g. a JAX-RS endpoint) marked with @RolesAllowed("uma_authorization") and the server will automatically kick me over to Keycloak and then accept the token when I get back. Then, I can pick that up from the request attributes and use it for Domino data access. Keycloak is getting its user directory from Domino via LDAP in the same way as IAM usually would, but, like IAM with AD, it could be configured to use different user directories. I don't know that I'll want to do that, but it's good to know.

Conclusion

Like the original driver itself, this was mostly an educational exercise for me. I don't currently have any requirements to use the AppDev Pack or OIDC/Keycloak, but I'd wanted to dip my toes in both for a while now, and I'm pleased that I came out successful. I imagine that I'll have an occasion to implement something like this eventually. It may not be the same specific parts, but the core concepts are common, like in Keep's JWT and OAuth support. It's a neat setup, and it's definitely worth doing something similar if you have some experimentation time on your hands.

Though the bulk of the work I've been doing for the XPages Jakarta EE project is to bring JEE technologies to Domino, the NoSQL driver has been designed to lead a double life: it works in an XPages context, but it's written to not have any XPages dependencies. One reason for this is that I want it to be usable if you use, for example, the Open Liberty runtime project to side-car apps on a Domino server but still use Notes.jar for data access.

Another reason for its organization, though, is that I intended for the driver to be portable across implementations. The driver itself is split into a main bundle and a ".lsxbe" implementation bundle. My original thought was to make that ready for a JNX or Domino JNA implementation, but it's pretty flexible.

Proton

And that brings me to the topic of this post: the AppDev Pack. I'd actually not used the AppDev Pack at all until I started in on this little project. While I like the idea, it at first only had a JS client, and then even with a Java client lacked some important capabilities I'd need for most of what I do. That's still the case, unfortunately, but my general annoyance with the problems of developing on top of a local Notes runtime tipped the scales to get me to investigate it.

In essence, the core part of the AppDevPack - the Proton addin - is kind of like a modern take on the CORBA driver for Notes.jar, in that it provides a remote way to make API calls on the server that are "low-level-ish" and ideally higher-performance than REST APIs. Since it's remote, it imposes no requirements on the environment of the app making the calls, and could in theory work with any language that has a gRPC library (though in practice HCL has only ever shipped JS and Java clients without providing the gRPC spec to generate others).

The API that Proton provides is heavily geared around batch operations, focusing on using DQL for querying. This suits my needs well, as Jakarta NoSQL essentially assumes you'll have something like DQL available to do arbitrary document queries. It's also geared around requesting specific items from documents to return, which further suits me well - for efficiency purposes, I wrote code into the LSXBE driver to only read desired items anyway, so that adapted naturally.

The Driver

So, over the long weekend here, I set out into writing a driver suitable for use in standalone JEE applications to access Domino via Proton. The goal here is to make it so that code written to target the LSXBE driver will work the same way with this Proton-based one, with the only differences being limitations in what Proton provides and a switch to providing a lotus.domino.Database context to providing a com.hcl.domino.db.model.Database one.

And, though the limitation list there is a bit lengthy, I accomplished the main part of my goal there. Since it shares a lot of code in common with the LSXBE driver, the implementation here is pretty small. The bulk of the code happens in ProtonDocumentCollectionManager, which is the implementation of all the raw JNoSQL primitives - insert, update, select, etc. - and then ProtonEntityConverter, which is the utility class that translates between Proton's concept of Document and JNoSQL's concept of DocumentEntity.

The Proton API is... very weird from a Java perspective. Some of it bears a little similarity to the NIO Files API if you squint, but otherwise it's just an odd duck. Fortunately, a consumer of this driver doesn't have to care about that beyond the initial connection to the server. The specifics of actually connecting to a server and opening a database are the same as with normal Proton use - you do that and then provide a CDI bean to hand off the Database object to the driver:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

@RequestScoped
public class ContextDatabaseSupplier {
	// This type of supplier is required
	@Produces
	public DatabaseSupplier get() {
		// Here, the `Database` object is set in a `ServletRequestListener`, though it likely could be app-wide
		return () -> {
			HttpServletRequest request = CDI.current().select(HttpServletRequest.class).get();
			return (Database)request.getAttribute(DatabaseContextListener.ATTR_DB);
		};
	}
	
	// This supplier is optional - leaving it out or returning null will cause the driver to skip performing Act-As-User
	@Produces
	public AccessTokenSupplier getAccessToken() {
		return () -> {
			HttpServletRequest request = CDI.current().select(HttpServletRequest.class).get();
			return (String)request.getAttribute("oidc_access_token");
		};
	}
}

Next Steps

Now, admittedly, I'm not sure how much more time I'm going to put into this driver. Though there are a few enhancements that could be made - attachments and rich text, namely - I'm probably not going to use this myself in the near future. The immediate cases I have where I'd want to use this would end up involving things like views that aren't currently accessible over Proton, and so it's be really just speculative development. Still, I wanted to make this to kick the tires a bit and see what's possible, and it's got my gears turning.

There'll also be a bit more I want to go over in the next post. In the above code block, I mention Act-As-User tokens; the driver will use those when provided, and the driver itself doesn't care how you acquire them, but it's an interesting topic on its own. During my testing, I was able to get a Liberty-hosted app performing OIDC authentication for its own login and then using that token for access to Proton, and I think that that will deserve some expansion on its own.

In the mean time, if you're curious, the driver is available up on GitHub:

https://github.com/OpenNTF/jnosql-driver-proton

The artifact is published to OpenNTF's Maven repository, though you'll need to follow the instructions in the README there to add the non-redistributable Proton client library to your local Maven repo.

Over the weekend, I got a wild hair to try something that had been percolating in my mind recently: get Passkeys, Apple's term for WebAuthn/FIDO technologies, working with the new Safari against a Domino server. And, though some aspects were pretty non-obvious (it turns out there's a LOT of binary-data stuff in JavaScript now), I got it working thanks to a few tools:

1. JavaSapi, the "yeah, I know it's not supported" DSAPI Java peer
2. webauthn.guide, a handy resource
3. java-webauthn-server, a Java implementation of the WebAuthn server components

Definition

First off: what is all this? All the terms - Passkey, FIDO, WebAuthn - for our purposes here deal with a mechanism for doing public/private key authentication with a remote server. In a sense, it's essentially a web version of what you do with Notes IDs or SSH keys, where the only actual user-provided authentication (a password, biometric input, or the like) happens locally, and then the local machine uses its private key combined with the server's knowledge of the public key to prove its identity.

WebAuthn accomplishes this with the navigator.credentials object in the browser, which handles dealing with the local security storage. You pass objects between this store and the server, first to register your local key and then subsequently to log in with it. There are a lot of details I'm fuzzing over here, in large part because I don't know the specifics myself, but that will cover it.

While browsers have technically had similar cryptographic authentication for a very long time by way of client SSL certificates, this set of technologies makes great strides across the board - enough to make it actually practical to deploy for normal use. With Safari, anything with Touch ID or Face ID can participate in this, while on other platforms and browsers you can use either a security key or similar cryptographic storage. Since I have a little MacBook Air with Touch ID, I went with that.

The Flow

Before getting too much into the specifics, I'll talk about the flow. The general idea with WebAuthn is that the user gets to a point where they're creating a new account (where password doesn't yet matter) or logged in via another mechanism, and then have their browser generate the keypair. In this case, I logged in using the normal Domino login form.

Once in, I have a button on the page that will request that the browser create the keypair. This first makes a request to the server for a challenge object appropriate for the user, then creates the keypair locally, and then POSTs the results back to the server for verification. To the user, that looks like:

That keypair will remain in the local device's storage - in this case, Keychain, synced via iCloud in upcoming versions.

Then, I have another button that performs a challenge. In practice, this challenge would be configured on the login page, but it's on the same page for this proof-of-concept. The button causes the browser to request from the server a list of acceptable public keys to go with a given username, and then prompts the user to verify that they want to use a matching one:

The implementation details of what to do on success and failure are kind of up to you. Here, I ended up storing active authentication sessions on a server in a Map keyed the SessionID cookie value to user name, but all options are open.

Dance Implementation

As I mentioned above, my main tool on the server side was java-webauthn-server, which handles a lot of the details of the processs. For this experiment, I cribbed their InMemoryRegistrationStorage class, but a real implementation would presumably store this information in an NSF.

For the client side, there's a npm module to handle the specifics, but I was doing this all on a single HTML page and so I just borrowed from it in pieces (particularly the Base64/ByteArray stuff).

On the server, I created an OSGi plugin that contained a com.ibm.pvc.webcontainer.application webapp as well as the JavaSapi service: since they're both in the same OSGi bundle, that meant I could share the same classes and memory space, without having to worry about coordination between two very-distinct parts (as would be the case with DSAPI).

The webapp part itself actually does more of the work, and does so by way of four Servlets: one for the "create keys" options, one for registering a created key, one for the "I want to start a login" pre-flight, and finally one for actually handling the final authentication. As an implementation note, getting this working involved removing guava.jar from the classpath temporarily, as the library in question made heavy use of a version slightly newer than what Domino ships with.

Key Creation

The first Servlet assumes that the user is logged in and then provides a JSON object to the client describing what sort of keypair it should create:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

Session session = ContextInfo.getUserSession();
			
PublicKeyCredentialCreationOptions request = WebauthnManager.instance.getRelyingParty()
	.startRegistration(
		StartRegistrationOptions.builder()
			// Creates or retrieves an in-memory object with an associated random "handle" value
		    .user(WebauthnManager.instance.locateUser(session))
		    .build()
	);

// Store in the HTTP session for later verification. Could also be done via cookie or other pairing
req.getSession(true).setAttribute(WebauthnManager.REQUEST_KEY, request);
	
String json = request.toCredentialsCreateJson();
resp.setStatus(200);
resp.setHeader("Content-Type", "application/json"); //$NON-NLS-1$ //$NON-NLS-2$
resp.getOutputStream().write(String.valueOf(json).getBytes());

The client side retrieves these options, parses some Base64'd parts to binary arrays (this is what the npm module would do), and then sends that back to the server to create the registration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

fetch("/webauthn/creationOptions", { "credentials": "same-origin" })
	.then(res => res.json())
	.then(json => {
		json.publicKey.challenge = base64urlToBuffer(json.publicKey.challenge, c => c.charCodeAt(0));
		json.publicKey.user.id = base64urlToBuffer(json.publicKey.user.id, c => c.charCodeAt(0));
		if(json.publicKey.excludeCredentials) {
			for(var i = 0; i < json.publicKey.excludeCredentials.length; i++) {
				var cred = json.publicKey.excludeCredentials[i];
				cred.id = base64urlToBuffer(cred.id);
			}
		}
		navigator.credentials.create(json)
			.then(credential => {
				// Create a JSON-friendly payload to send to the server
				const payload = {
					type: credential.type,
					id: credential.id,
					response: {
						attestationObject: bufferToBase64url(credential.response.attestationObject),
						clientDataJSON: bufferToBase64url(credential.response.clientDataJSON)
					},
					clientExtensionResults: credential.getClientExtensionResults()
				}
				fetch("/webauthn/create", {
					method: "POST",
					body: JSON.stringify(payload)
				})
			})
	})

The code for the second call on the server parses out the POST'd JSON and stores the registration in the in-memory storage (which would properly be an NSF):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

String json = StreamUtil.readString(req.getReader());
PublicKeyCredential<AuthenticatorAttestationResponse, ClientRegistrationExtensionOutputs> pkc = PublicKeyCredential
	.parseRegistrationResponseJson(json);

// Retrieve the request we had stored in the session earlier
PublicKeyCredentialCreationOptions request = (PublicKeyCredentialCreationOptions) req.getSession(true)
	.getAttribute(WebauthnManager.REQUEST_KEY);

// Perform registration, which verifies that the incoming JSON matches the initiated request
RelyingParty rp = WebauthnManager.instance.getRelyingParty();
RegistrationResult result = rp
	.finishRegistration(FinishRegistrationOptions.builder().request(request).response(pkc).build());

// Gather the registration information to store in the server's credential repository
DominoRegistrationStorage repo = WebauthnManager.instance.getRepository();
CredentialRegistration reg = new CredentialRegistration();
reg.setAttestationMetadata(Optional.ofNullable(pkc.getResponse().getAttestation()));
reg.setUserIdentity(request.getUser());
reg.setRegistrationTime(Instant.now());
RegisteredCredential credential = RegisteredCredential.builder()
	.credentialId(result.getKeyId().getId())
	.userHandle(request.getUser().getId())
	.publicKeyCose(result.getPublicKeyCose())
	.signatureCount(result.getSignatureCount())
	.build();
reg.setCredential(credential);
reg.setTransports(pkc.getResponse().getTransports());

Session session = ContextInfo.getUserSession();
repo.addRegistrationByUsername(session.getEffectiveUserName(), reg);

Login/Assertion

Once the client has a keypair and the server knows about the public key, then the client can ask the server for what it would need if one were to log in as a given name, and then uses that information to make a second call. The dance on the client side looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

var un = /* fetch the username from somewhere, such as a login form */
fetch("/webauthn/assertionRequest?un=" + encodeURIComponent(un))
	.then(res => res.json())
	.then(json => {
		json.publicKey.challenge = base64urlToBuffer(json.publicKey.challenge, c => c.charCodeAt(0));
		if(json.publicKey.allowCredentials) {
			for(var i = 0; i < json.publicKey.allowCredentials.length; i++) {
				var cred = json.publicKey.allowCredentials[i];
				cred.id = base64urlToBuffer(cred.id);
			}
		}
		navigator.credentials.get(json)
			.then(credential => {
				const payload = {
					type: credential.type,
					id: credential.id,
					response: {
						authenticatorData: bufferToBase64url(credential.response.authenticatorData),
						clientDataJSON: bufferToBase64url(credential.response.clientDataJSON),
						signature: bufferToBase64url(credential.response.signature),
						userHandle: bufferToBase64url(credential.response.userHandle)
					},
					clientExtensionResults: credential.getClientExtensionResults()
				}
				fetch("/webauthn/assertion", {
					method: "POST",
					body: JSON.stringify(payload),
					credentials: "same-origin"
				})
			})
	})

That's pretty similar to the middle code block above, really, and contains the same sort of ferrying to and from transport-friendly JSON objects and native credential objects.

On the server side, the first Servlet - which looks up the available public keys for a user name - is comparatively simple:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

String userName = req.getParameter("un"); //$NON-NLS-1$
AssertionRequest request = WebauthnManager.instance.getRelyingParty()
	.startAssertion(
		StartAssertionOptions.builder()
			.username(userName)
			.build()
	);

// Stash the current assertion request
req.getSession(true).setAttribute(WebauthnManager.ASSERTION_REQUEST_KEY, request);

String json = request.toCredentialsGetJson();
resp.setStatus(200);
resp.setHeader("Content-Type", "application/json"); //$NON-NLS-1$ //$NON-NLS-2$
resp.getOutputStream().write(String.valueOf(json).getBytes());

The final Servlet handles parsing out the incoming assertion (login) and stashing it in memory as associated with the "SessionID" cookie. That value could be anything that the browser will send with its requests, but "SessionID" works here.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

String json = StreamUtil.readString(req.getReader());
PublicKeyCredential<AuthenticatorAssertionResponse, ClientAssertionExtensionOutputs> pkc =
	    PublicKeyCredential.parseAssertionResponseJson(json);

// Retrieve the request we had stored in the session earlier
AssertionRequest request = (AssertionRequest) req.getSession(true).getAttribute(WebauthnManager.ASSERTION_REQUEST_KEY);

// Perform verification, which will ensure that the signed value matches the public key and challenge
RelyingParty rp = WebauthnManager.instance.getRelyingParty();
AssertionResult result = rp.finishAssertion(FinishAssertionOptions.builder()
	.request(request)
	.response(pkc)
	.build());

if(result.isSuccess()) {
	// Keep track of logins
	WebauthnManager.instance.getRepository().updateSignatureCount(result);
	
	// Find the session cookie, which they will have by now
	String sessionId = Arrays.stream(req.getCookies())
		.filter(c -> "SessionID".equalsIgnoreCase(c.getName()))
		.map(Cookie::getValue)
		.findFirst()
		.get();
	WebauthnManager.instance.registerAuthenticatedUser(sessionId, result.getUsername());
}

Trusting the Key

At this point, there's a dance between the client and server that results in the client being able to perform secure, password-less authentication with the server and the server knowing about the association, and so now the remaining job is just getting the server to actually trust this assertion. That's where JavaSapi comes in.

Above, I used the "SessionID" cookie as a mechanism to store an in-memory association between a browser cookie (which is independent of authentication) to a trusted user. Then, I made a JavaSapi service that looks for this and tries to find an authenticated user in its authenticate method:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

@Override
public int authenticate(IJavaSapiHttpContextAdapter context) {
	Cookie[] cookies = context.getRequest().getCookies();
	if(cookies != null) {
		Optional<Cookie> cookie = Arrays.stream(context.getRequest().getCookies())
			.filter(c -> "SessionID".equalsIgnoreCase(c.getName())) //$NON-NLS-1$
			.findFirst();
		if(cookie.isPresent()) {
			String sessionId = cookie.get().getValue();
			Optional<String> user = WebauthnManager.instance.getAuthenticatedUser(sessionId);
			if(user.isPresent()) {
				context.getRequest().setAuthenticatedUserName(user.get(), "WebAuthn"); //$NON-NLS-1$
				return HTEXTENSION_REQUEST_AUTHENTICATED;
			}
		}
	}
	
	return HTEXTENSION_EVENT_DECLINED;
}

And that's all there is to it on the JavaSapi side. Because it shares the same active memory space as the webapp doing the dance, it can use the same WebauthnManager instance to read the in-memory association. You could in theory do this another way with DSAPI - storing the values in an NSF or some other mechanism that can be shared - but this is much, much simpler to do.

Conclusion

This was a neat little project, and it was a good way to learn a bit about some of the native browser objects and data types that I haven't had occasion to work with before. I think this is also something that should be in the product; if you agree, go vote for the ideas from a few years ago.

In what may be the last in this series for a bit, I'll talk about the current approach I'm taking for the UI for the new OpenNTF web site. This post will also tread ground I've covered before, when talking about the Jakarta MVC framework and JSP, but it never hurts to reinforce the pertinent aspects.

MVC

The entrypoint for the UI is Jakarta MVC, which is a framework that sits on top of JAX-RS. Unlike JSF or XPages, it leaves most app-structure duties to other components. This is due both to its young age (JSF predates and often gave rise to several things we've discussed so far) and its intent. It's "action-based", where you define an endpoint that takes an incoming HTTP request and produces a response, and generally won't have any server-side UI state. This is as opposed to JSF/XPages, where the core concept is the page you're working with and the page state generally exists across multiple requests.

Your starting point with MVC is a JAX-RS REST service marked with @Controller:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

package webapp.controller;

import java.text.MessageFormat;

import bean.EncoderBean;
import jakarta.inject.Inject;
import jakarta.mvc.Controller;
import jakarta.mvc.Models;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.NotFoundException;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.PathParam;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;
import model.home.Page;

@Path("/pages")
public class PagesController {
    
    @Inject
    Models models;
    
    @Inject
    Page.Repository pageRepository;
    
    @Inject
    EncoderBean encoderBean;

    @Path("{pageId}")
    @GET
    @Produces(MediaType.TEXT_HTML)
    @Controller
    public String get(@PathParam("pageId") String pageId) {
        String key = encoderBean.cleanPageId(pageId);
        Page page = pageRepository.findBySubject(key)
            .orElseThrow(() -> new NotFoundException(MessageFormat.format("Unable to find page for ID: {0}", key)));
        models.put("page", page); //$NON-NLS-1$
        return "page.jsp"; //$NON-NLS-1$
    }
}

In the NSF, this will respond to requests like /foo.nsf/xsp/app/pages/Some_Page_Name. Most of what is going on here is the same sort of thing we saw with normal REST services: the @Path, @GET, @Produces, and @PathParam are all normal JAX-RS, while @Inject uses the same CDI scaffolding I talked about in the last post.

MVC adds two things here: @Inject Models models and @Controller.

The Models object is conceptually a Map that houses variables that you can populate to be accessible via EL on the rendered page. You can think of this like viewScope or requestScope in XPages and is populated in something like the beforePageLoad phase. Here, I use the Models object to store the Page object I look up with JNoSQL.

The @Controller annotation marks a method or a class as participating in the MVC lifecycle. When placed on a class, it applies to all methods on the class, while placing it on a method specifically allows you to mix MVC and "normal" REST resources in the same class. Doing that would be useful if you want to, for example, provide HTML responses to browsers and JSON responses to API clients at the same resource URL.

When a resource method is marked for MVC use, it can return a string that represents either a page to render or a redirection in the form "redirect:some/resource". Here, it's hard-coded to use "page.jsp", but in another situation it could programmatically switch between different pages based on the content of the request or state of the app.

While this looks fairly clean on its own, it's important to bear in mind both the strengths and weaknesses of this approach. I think it will work here, as it does for my blog, because the OpenNTF site isn't heavy on interactive forms. When dealing with forms in MVC, you'll have to have another endpoint to listen for @POST (or other verbs with a shim), process that request from scratch, and return a new page. For example, from the XPages JEE example app:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

@Path("create")
@POST
@Consumes(MediaType.APPLICATION_FORM_URLENCODED)
@Controller
public String createPerson(
        @FormParam("firstName") @NotEmpty String firstName,
        @FormParam("lastName") String lastName,
        @FormParam("birthday") String birthday,
        @FormParam("favoriteTime") String favoriteTime,
        @FormParam("added") String added,
        @FormParam("customProperty") String customProperty
) {
    Person person = new Person();
    composePerson(person, firstName, lastName, birthday, favoriteTime, added, customProperty);
    
    personRepository.save(person);
    return "redirect:nosql/list";
}

That's already fiddlier than the XPages version, where you'd bind fields right to bean/document properties, and it gets potentially more complicated from there. In general, the more form-based your app is, the better a fit XPages/JSF is.

JSP

While MVC isn't intrinsically tied to JSP (it ships with several view engine hooks and you can write your own), JSP has the advantage of being built in to all Java webapp servers and is very well fit to purpose. When writing JSPs for MVC, the default location is to put them in WEB-INF/views, which is beneath WebContent in an NSF project:

The "tags" there are the general equivalent of XPages Custom Controls, and their presence in WEB-INF/tags is convention. An example page (the one used above) will tend to look something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

<%@page contentType="text/html" pageEncoding="UTF-8" trimDirectiveWhitespaces="true" session="false" %>
<%@taglib prefix="t" tagdir="/WEB-INF/tags" %>
<%@taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core" %>
<%@taglib prefix="fn" uri="http://java.sun.com/jsp/jstl/functions" %>
<t:layout>
    <turbo-frame id="page-content-${page.linkId}">
        <div>
            ${page.html}
        </div>
        
        <c:if test="${not empty page.childPageIds}">
            <div class="tab-container">
                <c:forEach items="${page.cleanChildPageIds}" var="pageId" varStatus="pageLoop">
                    <input type="radio" id="tab${pageLoop.index}" name="tab-group" ${pageLoop.index == 0 ? 'checked="checked"' : ''} />
                    <label for="tab${pageLoop.index}">${fn:escapeXml(encoder.cleanPageId(pageId))}</label>
                </c:forEach>
                    
                <div class="tabs">
                    <c:forEach items="${page.cleanChildPageIds}" var="pageId">
                        <turbo-frame id="page-content-${pageId}" src="xsp/app/pages/${encoder.urlEncode(pageId)}" class="tab" loading="lazy">
                        </turbo-frame>
                    </c:forEach>
                </div>
            </div>
        </c:if>
    </turbo-frame>
</t:layout>

There are, by shared lineage and concept, a lot of similarities with an XPage here. The first four lines of preamble boilerplate are pretty similar to the kind of stuff you'd see in an <xp:view/> element to set up your namespaces and page options. The tag prefixing is the same idea, where <t:layout/> refers to the "layout" custom tag in the NSF and <c:forEach/> refers to a core control tag that ships with the standard tag library, JSTL. The <turbo-frame/> business isn't JSP - I'll deal with that later.

The bits of EL here - all wrapped in ${...} - are from Expression Language 4.0, which is the current version of XPages's aging EL. On this page, the expressions are able to resolve variables that we explicitly put in the Models object, such as page, as well as CDI beans with the @Named annotation, such as encoderBean. There are also a number of implicit objects like request, but they're not used here.

In general, this is safely thought of as an XPage where you make everything load-time-bound and set viewState="nostate". The same sorts of concepts are all there, but there's no concept of a persistent component that you interact with. Any links, buttons, and scripts will all go to the server as a fresh request, not modifying an existing page. You can work with application and session scopes, but there's no "view" scope.

Hotwired Turbo

Though this app doesn't have much need for a lot of XPages's capabilities, I do like a few components even for a mostly "read-only" app. In particular, the <xe:djContentPane/> and <xe:djTabContainer/> controls have the delightful capability of deferring evaluation of their contents to later requests. This is a powerful way to speed up initial page load and, in the case of the tab container, skip needing to render parts of the page the user never uses.

For this and a couple other uses, I'm a fan of Hotwired Turbo, which is a library that grew out of 37 Signals's Rails-based development. The goal of Turbo and the other Hotwired components is to keep the benefits of server-based HTML rendering while mixing in a lot of the niceties of JS-run apps. There are two things that Turbo is doing so far in this app.

The first capability is dubbed "Turbo Drive", and it's sort of a freebie: you enable it for your app, tell it what is considered the app's base URL, and then it will turn any in-app links into "partial refresh" links: it downloads the page in the background and replaces just the changed part on the page. Though this is technically doing more work than a normal browser navigation, it ends up being faster for the user interface. And, since it also updates the URL to match the destination page and doesn't require manual modification of links, it's a drop-in upgrade that will also degrade gracefully if JavaScript isn't enabled.

The second capability is <turbo-frame/> up there, and it takes a bit more buy-in to the JS framework in your app design. The way I'm using Turbo Frames here is to support the page structure of OpenNTF, which is geared around a "primary" page as well as zero or more referenced pages that show up in tabs. Here, I'm buying in to Turbo Frames by surrounding the whole page in a <turbo-frame/> element with an id using the page's key, and then I reference each "sub-page" in a tab with that same ID. When loading the frame, Turbo makes a call to the src page, finds the element with the matching id value, and drops it in place inside the main document. The loading="lazy" parameter means that it defers loading until the frame is visible in the browser, which is handy when using the HTML/CSS-based tabs I have here.

I've been using this library for a while now, and I've been quite pleased. Though it was created for use with Rails, the design is independent of the server implementation, and the idioms fit perfectly with this sort of Java app too.

Conclusion

I think that wraps it up for now. As things progress, I may have more to add to this series, but my hope is that the app doesn't have to get much more complicated than the sort of stuff seen in this series. There are certainly big parts to tackle (like creating and managing projects), but I plan to do that by composing these elements. I remain delighted with this mode of NSF-based app development, and look forward to writing more clean, semi-declarative code in this vein.

The design for the OpenNTF home page has been with us for a little while now and has served us pretty well. It looks good and covers the bases it needs to. However, it's getting a little long in the tooth and, more importantly, doesn't cover some capabilities that we're thinking of adding.

While we could potentially expand the current one, this provides a good opportunity for a clean start. I had actually started taking a swing at this a year and a half ago, taking the tack that I'd make a webapp and deploy it using the Domino Open Liberty Runtime. While that approach would put all technologies on the table, it'd certainly be weirder to future maintainers than an app inside an NSF (at least for now).

So I decided in the past few weeks to pick the project back up and move it into an NSF via the XPages Jakarta EE Support project. I can't say for sure whether I'll actually complete the project, but it'll regardless be a good exercise and has proven to be an excellent way to find needed features to implement.

I figure it'll also be useful to keep something of a travelogue here as I go, making posts periodically about what I've implemented recently.

The UI Toolkit

The original form of this project used MVC and JSP for the UI layer. Now that I was working in an NSF, I could readily use XPages, but for now I've decided to stick with the MVC approach. While it will make me have to solve some problems I wouldn't necessarily have to solve otherwise (like file uploads), it remains an extremely-pleasant way to write applications. I am also not constrained to this: since the vast majority of the logic is in Java beans and controller classes, switching the UI front-end would not be onerous. Also, I could theoretically mix JSP, JSF, XPages, and static HTML together in the app if I end up so inclined.

In the original app (as in this blog), I made use of WebJars to bring in JavaScript dependencies, namely Hotwire Turbo to speed up in-site navigation and use Turbo Frames. Since the NSF app in Designer doesn't have the Maven dependency mechanism the original app did, I just ended up copying the contents of the JAR into WebContent. That gave me a new itch to scratch, though: I'd love to be able to have META-INF/resources files in classpath JARs picked up by the runtime and made available, lowering the number of design elements present in the NSF.

The Data Backend

The primary benefit of this project so far has been forcing me to flesh out the Jakarta NoSQL driver in the JEE support project. I had kind of known hypothetically what features would be useful, but the best way to do this kind of thing is often to work with the tool until you hit a specific problem, and then solve that. So far, it's forced me to:

• Implement the view support in my previous post
• Add attachment support for documents, since we'll need to upload and download project releases
• Improve handling of rich text and MIME, though this also has more room to grow
• Switched the returned Streams from the driver to be lazy loading, meaning that not all documents/entries have to be read if the calling code stops reading the results partway through
• Added the ability to use custom property types with readers/writers defined in the NSF

Together, these improvements have let me have almost no lotus.domino code in the app. The only parts left are a bean for formatting Notes-style names (which I may want to make a framework service anyway) and a bean for providing access to the various associated databases used by the app. Not too shabby! The app is still tied to Domino by way of using the Domino-specific extensions to JNoSQL, but the programming model is significantly better and the amount of app code was reduced dramatically.

Next Steps

There's a bunch of work to be done. The bulk of it is just implementing things that the current XPages app does: actually uploading projects, all the stuff like discussion lists, and so forth. I'll also want to move the server-side component of the small "IP Tools" suite I use for IP management stuff in here. Currently, that's implemented as Wink-based JAX-RS resources inside an OSGi bundle, but it'll make sense to move it here to keep things consolidated and to make use of the much-better platform capabilities.

As I mentioned above, I can't guarantee that I'll actually finish this project - it's all side work, after all - but it's been useful so far, and it's a further demonstration of how thoroughly pleasant the programming model of the JEE support project is.

In the spirit of not leaving well enough alone, I decided the other day to tinker a bit more with JavaSapi, the DSAPI peer tucked away undocumented in Domino. While I still maintain that this is too far from supported for even me to put into production, I think it's valuable to demonstrate the sort of thing that this capability - if made official - would make easy to implement.

JWT

I've talked about JWT a bit before, and it was in a similar context: I wanted to be able to access a third-party API that used JWT to handle authorization, so I wrote a basic library that could work with LS2J. While JWT isn't inherently tied to authorization like this, it's certainly where it's found a tremendous amount of purchase.

JWT has a couple neat characteristics, and the ones that come in handy most frequently are a) that you can enumerate specific "claims" in the token to restrict what the token allows the user to do and b) if you use a symmetric signature key, you can generate legal tokens on the client side without the server having to generate them. "b" there is optional, but makes JWT a handy way to do a quick shared secret between servers to allow for trusted authentication.

It's a larger topic than that, for sure, but that's the quick and dirty of it.

Mixing It With An NSF

Normally on Domino, you're either authenticated for the whole server or you're not. That's usually fine - if you want to have a restricted account, you can specifically grant it access to only a few NSFs. However, it's good to be able to go more fine-grained, restricting even powerful accounts to only do certain things in some contexts.

So I had the notion to take the JWT capability and mix it with JavaSapi to allow you to do just that. The idea is this:

1. You make a file resource (hidden from the web) named "jwt.txt" that contains your per-NSF secret.
2. A remote client makes a request with an Authorization header in the form of Bearer Some.JWT.Here
3. The JavaSapi interceptor sees this, checks the target NSF, loads the secret, verifies it against the token, and authorizes the user if it's legal

As it turns out, this turned out to be actually not that difficult in practice at all.

The main core of the code is:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

public int authenticate(IJavaSapiHttpContextAdapter context) {
    IJavaSapiHttpRequestAdapter req = context.getRequest();

    // In the form of "/foo.nsf/bar"
    String uri = req.getRequestURI();
    String secret = getJwtSecret(uri);
    if(StringUtil.isNotEmpty(secret)) {
        try {
            String auth = req.getHeader("Authorization"); //$NON-NLS-1$
            if(StringUtil.isNotEmpty(auth) && auth.startsWith("Bearer ")) { //$NON-NLS-1$
                String token = auth.substring("Bearer ".length()); //$NON-NLS-1$
                Optional<String> user = decodeAuthenticationToken(token, secret);
                if(user.isPresent()) {
                    req.setAuthenticatedUserName(user.get(), "JWT"); //$NON-NLS-1$
                    return HTEXTENSION_REQUEST_AUTHENTICATED;
                }
            }
        } catch(Throwable t) {
            t.printStackTrace();
        }
    }

    return HTEXTENSION_EVENT_DECLINED;
}

To read the JWT secret, I used IBM's NAPI:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

private String getJwtSecret(String uri) {
    int nsfIndex = uri.toLowerCase().indexOf(".nsf"); //$NON-NLS-1$
    if(nsfIndex > -1) {
        String nsfPath = uri.substring(1, nsfIndex+4);
        
        try {
            NotesSession session = new NotesSession();
            try {
                if(session.databaseExists(nsfPath)) {
                    // TODO cache lookups and check mod time
                    NotesDatabase database = session.getDatabase(nsfPath);
                    database.open();
                    NotesNote note = FileAccess.getFileByPath(database, SECRET_NAME);
                    if(note != null) {
                        return FileAccess.readFileContentAsString(note);
                    }
                }
            } finally {
                session.recycle();
            }
        } catch(Exception e) {
            e.printStackTrace();
        }
    }
    return null;
}

And then, for the actual JWT handling, I use the auth0 java-jwt library:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

public static Optional<String> decodeAuthenticationToken(final String token, final String secret) {
	if(token == null || token.isEmpty()) {
		return Optional.empty();
	}
	
	try {
		Algorithm algorithm = Algorithm.HMAC256(secret);
		JWTVerifier verifier = JWT.require(algorithm)
		        .withIssuer(ISSUER)
		        .build();
		DecodedJWT jwt = verifier.verify(token);
		Claim claim = jwt.getClaim(CLAIM_USER);
		if(claim != null) {
			return Optional.of(claim.asString());
		} else {
			return Optional.empty();
		}
	} catch (IllegalArgumentException | UnsupportedEncodingException e) {
		throw new RuntimeException(e);
	}
}

And, with that in place, it works:

That text is coming from a LotusScript agent - as I mentioned in my original JavaSapi post, this authentication is trusted the same way DSAPI authentication is, and so all elements, classic or XPages, will treat the name as canon.

Because the token is based on the secret specifically from the NSF, using the same token against a different NSF (with no JWT secret or a different one) won't authenticate the user:

If we want to be fancy, we can call this scoped access.

This is the sort of thing that makes me want JavaSapi to be officially supported. Custom authentication and request filtering are much, much harder on Domino than on many other app servers, and JavaSapi dramatically reduces the friction.

Earlier today, I published version 2.5.0 of the XPages Jakarta EE Support project. It's mostly a consolidation and bug-fix release, but there are few interesting features and notes about the implementation. Plus, as teased in the post title up there, there's a looming problem for the project.

New Features

There are two main new features in this version.

First, I added some configurable CORS support for REST services. Fortunately for me, RestEasy comes with a CORS filter by default, and it just needs to be enabled. I wired it up using MicroProfile Config to read some values out of xsp.properties:

1
2
3
4
5
6
7
8

rest.cors.enable=true                   # required for CORS
rest.cors.allowCredentials=true         # defaults to true
rest.cors.allowedMethods=GET,HEAD       # defaults to all
rest.cors.allowedHeaders=Some-Header    # defaults to all
rest.cors.exposedHeaders=Some-Header    # optional
rest.cors.maxAge=600                    # optional
# allowedOrigins is required, and can be "*"
rest.cors.allowedOrigins=http://foo.com,http://bar.com

I also added support for using the long-standing @WebServlet annotation. Though REST services will generally do what you want, sometimes it's handy to use the lower-level Servlet capability, and now you can do so inline:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

@WebServlet(urlPatterns = { "/someservlet", "/someservlet/*", "*.hello" })
public class ExampleServlet extends HttpServlet {
	private static final long serialVersionUID = 1L;
	
	@Inject
	ApplicationGuy applicationGuy;

	@Override
	protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
		resp.setContentType("text/plain");
		resp.getWriter().println("Hello from ExampleServlet. context=" + req.getContextPath() + ", path=" + req.getServletPath() + ", pathInfo=" + req.getPathInfo());
		resp.getWriter().println("ApplicationGuy: " + applicationGuy.getMessage());
		resp.getWriter().flush();
	}
}

Consolidation

There were a couple specs where I had previously either copied the source into the repository (CDI, Mail) or had maintained a local branch fork (NoSQL). Those were always uncomfortable concessions to reality, but I decided to look further into ways to handle that.

For NoSQL, part of it was what I talked about in my last post: using Eclipse Transformer to make use of javax.* compiled binaries and source converted to jakarta.* automatically. But beyond that, it had the same problem that I had forked Mail for. Namely, it hits the same trouble that lots of non-OSGi code does in an OSGi context, where it uses ServiceLoader in a non-extensible way. Though I have an open PR to make use of the pseudo-standard "HK2" ServiceLoader provider, waiting for that would mean continuing the local-build trouble.

Instead, for all of these cases I made use of OSGi's Weaving capability to re-write those parts of the class files on the fly. While this is a bit unfortunate, it works well in practice. The only real down side for now is having to be a bit more careful when bumping the versions in the future, but this type of code changes very rarely.

The Looming Wall

While this has been going swimmingly, I've started to hit some real impediments with Domino's Java version. The next release of Jakarta EE, version 10, requires Java 11 as a minimum. This is similar to the move Equinox (Domino's OSGi framework of choice) made just under two years ago, and which has itself bitten me with a blockage to upgrading Tycho to version 2.0 and above. Java 11 is about four years old now, and is no longer even the latest LTS release, so this all makes sense.

I've known this was coming for a while, but incompatible versions of JEE specs and implementations started to trickle in over the past year, leading to me leaving notes for myself about maximum versions. JEE 10 itself is fairly imminent now, so I'll be capped at the ones released with JEE 9 a while ago.

So I've been pondering my options here.

In one sense, I solved this problem years ago. The Domino Open Liberty Runtime project has had the ability to download any version of open-source Java that you want, and I expanded it last year to let you pick from several common flavors. Liberty maintains a breathless pace of advancement, adding official support for Java 18 the month after it came out. If one wants to run JEE apps on Domino, that's the most complete way. However, though it does its job technologically well, it's not exactly a natural fit for Domino developers in its current state.

But I've been considering anew a notion I had years ago, which is to write an extension for Liberty so that it reads class files and resources out of an NSF directly. In some early investigation a bit ago, this started to appear quite doable. In theory, I could write an adapter that would take an incoming request for "foo.nsf" and then read files out of the NSF in the same way XPages does, but instead feeding them to Liberty's runtime. Doing this would essentially implement all remaining JEE and MicroProfile specs in one fell swoop on top of the "any Java version" support, but would add the fault-prone attribute of running a separate process and proxying requests to it. In practice, that setup has proven itself good, but it's certainly more complicated than the "single process on port 80" deal that Domino's HTTP is now.

That route also wouldn't inherently support XPages, which would be something of an impediment to the XPages JEE project's original remit. That's something I've also pondered, and in theory I could make an auto-vivifying version of the XPages Runtime project that grabs all the pertinent XPages bundles from the current server and patches them into the Liberty server as an extension feature, similar to how all the built-in Liberty features work. This could be done, but I'll admit that I balk a bit at the prospect. Though I run XPages outside Domino constantly, it's with full knowledge of the tradeoffs and special considerations. Getting a normal NSF-based XPages app to run in this way would take some additional work.

Anyway, those options could work, but none of them are great. The true fix would naturally be for HCL to move to a newer Java version in Domino's HTTP stack, but I don't control that, so I'll content myself with considering what to do in the mean time. Admittedly, pondering this sort of thing is enjoyable in its own right. Also fortunately, even without tackling this, there's still plenty of stuff in the pile for me to tackle as the fancy strikes me.

Setting code aside, the backbone of the XPages Jakarta EE Support project is its dependency pool. In it, I use my fork of the p2-maven-plugin to wrangle all the spec and implementation dependencies. Aside from just collecting them, this file does a ton of work to create and reconfigure their OSGi bundle rules to get everything working on Domino.

There have been limitations, though, and some of them have to do with the Jakarta NoSQL project. Though there are side branches of that project using the jakarta.* namespace, the main master branch is still on javax.* for a couple Jakarta depenencies. Historically, I've dealt with this by running a build locally and deploying it to OpenNTF's Maven server. However, this adds a bit of randomness to the mix: if a snapshot build of NoSQL goes out to the main repository that happens to be newer, then building the dependency repository locally might pick up on that instead, since it's named the same thing.

Transformer

Fortunately, IBM wrote the solution for me: Eclipse Transformer. This Transformer is a rules engine to translate files (Java and related resources, namely) based on configuration - and, while it's generic, it's really designed for the transition from javax.* to jakarta.* namespaces.

It allows you to do these transformations at runtime or (as I'll be doing here) ahead of time, even if you don't have access to the original source. Though I do have access to the source, it's more useful at the moment to act like I don't.

I'd known about the tool and have seen how it's used heavily by both app servers and implementation vendors to be able to support both old- and new-style uses, and so I've kept it in mind for in case the need ever came up. It's a perfect fit for this.

p2-maven-plugin

I considered a couple ways to handle this, but realized the cleanest for now would be to integrate it into the dependency pool generator that I already have, since it fits right in with the OSGi transformations I'm doing.

So I went on over to the p2-maven-plugin fork and got to work. When defining Maven artifacts to bring in, the format looks like this:

1
2
3
4

<artfiact>
    <id>jakarta.servlet:jakarta.servlet-api:4.0.4</id>
    <source>true</source>
</artfiact>

Now, Servlet already has a jakarta.* version, but it'll be useful here as an example that avoids the other transformations I'm doing.

My addition is to add a transform configuration option here, with jakarta as the only value for now:

1
2
3
4
5

<artfiact>
    <id>jakarta.servlet:jakarta.servlet-api:4.0.4</id>
    <source>true</source>
    <transform>jakarta</transform>
</artfiact>

...and that'll be it! When that is specified, the code will now run the artifact and its source JAR transparently through Transformer and the version you get in your p2 repository will reflect the transition. And, well, it works perfectly in my case. The resultant NoSQL spec and dependencies are functionally equivalent to the ones in the jakarta.* source branch, but without having to actually change the source files yet. Neat.

Implementation

Though it took a bit to track down the best way to do it, it turned out that Transformer is quite easy to embed into a Java app like the Maven plugin. The majority of the code ends up being effectively Java boilerplate to provide the default values for Jakarta transformation. Truncated, it looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

String inputFileName = t.getAbsolutePath(); // the artifact in ~/.m2/repository
File dest = File.createTempFile(t.getName(), ".jar"); //$NON-NLS-1$
String outputFileName = dest.getAbsolutePath();

Map<String, String> optionDefaults = JakartaTransform.getOptionDefaults();
Function<String, URL> ruleLoader = JakartaTransform.getRuleLoader();
TransformOptions options = /* build TransformOptions object that reads the above variables */

Transformer transformer = new Transformer(logger, options);
ResultCode result = transformer.run();
switch(result) {
case ARGS_ERROR_RC:
case FILE_TYPE_ERROR_RC:
case RULES_ERROR_RC:
case TRANSFORM_ERROR_RC:
	throw new IllegalStateException("Received unexpected result from transformer: " + result);
case SUCCESS_RC:
default:
	return dest;
}

There are plenty of options to specify, but that's really about it. Once given the Jakarta defaults, it will do the right thing in the normal case, both for the compiled class files as well as the source JAR.

I'm not sure if I'll need it in other cases (NoSQL will move over in the main branch eventually), but it's sure handy here and should be useful in a pinch. From time to time, I've run across dependencies that would be useful to include but use old JEE specs, and this could do the trick in those cases too.

Earlier this morning, Serdar Basegmez and Karsten Lehmann had a chat on Twitter regarding the desire for OAuth on Domino and their recollections of a not-quite-shipped technology from a decade ago going by the name "JSAPI".

Seeing this chat go by reminded me of some stuff I saw when I was researching the Domino HTTP Java entrypoint last year. Specifically, these guys, which have been sitting there since at least 9.0.1:

I'd made note of them at the time, since there's a lot of tantalizing stuff in there, but had put them back on the shelf when I found that they seemed to be essentially inert at runtime. For all of IBM's engineering virtues (and there are many), they were never very good at cleaning up their half-implemented experiments when it came time to ship, and I figured this was more of the same.

What This Is

Well, first and foremost, it is essentially a non-published experiment: I see no reference to these classes or how to enable them anywhere, and so everything within these packages should be considered essentially radioactive. While they're proving to be quite functional in practice, it's entirely possible - even likely - that the bridge to this side of thing is full of memory leaks and potential severe bugs. Journey at your own risk and absolutely don't put this in production. I mean that even more in this case than my usual wink-and-nod "not for production" coyness.

Anyway, this is the stuff Serdar and Karsten were talking about, named "JavaSapi" in practice. It's a Java equivalent to DSAPI, the API you can hook into with native libraries to perform low-level alterations to requests. DSAPI is neat, but it's onerous to use: you have to compile down to a native library, target each architecture you plan to run on, deploy that to each server, and enable it in the web site config. There's a reason not a lot of people use it.

Our new friend JavaSapi here provides the same sorts of capabilities (rewriting URLs, intercepting requests, allowing for arbitrary user authentication (more on this later), and so forth) but in a friendlier environment. It's not just that it's Java, either: JavaSapi runs in the full OSGi environment provided by HTTP, which means it swims in the same pool as XPages and all of your custom libraries. That has implications.

How To Use It

By default, it's just a bunch of classes sitting there, but the hook down to the core level (in libhttpstack.so) remains, and it can be enabled like so:

set config HTTP_ENABLE_JAVASAPI=1

(strings is a useful tool)

Once that's enabled, you should start seeing a line like this on HTTP start:

[01C0:0002-1ADC] 05/19/2022 03:37:17 PM  HTTP Server: JavaSapi Initialized

Now, there's a notable limitation here: the JavaSapi environment isn't intended to be arbitrarily extensible, and it's hard-coded to only know about one service by default. That service is interesting - it's an OAuth 2 provider of undetermined capability - but it's not the subject of this post. The good news is that Java is quite malleable, so it's not too difficult to shim in your own handlers by writing to the services instance variable of the shared JavaSapiEnvironment instance (which you might have to construct if it's not present).

Once you have that hook, it's just a matter of writing a JavaSapiService instance. This abstract class provides fairly-pleasant hooks for the triggers that DSAPI has, and nicely wraps requests and responses in Servlet-alike objects.

Unlike Servlet objects, though, you can set a bunch of stuff on these objects, subject to the same timing and pre-filtering rules you'd have in DSAPI. For example, in the #rawRequest method, you can add or overwrite headers from the incoming request before they get to any other code:

1
2
3
4
5

public int rawRequest(IJavaSapiHttpContextAdapter context) {
    context.getRequest().setRequestHeader("Host", "foo-bar.galaxia");
        
    return HTEXTENSION_EVENT_HANDLED;
}

If you want to, you can also handle the entire request outright:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

public int rawRequest(IJavaSapiHttpContextAdapter context) {
    if(context.getRequest().getRequestURI().contains("foobar")) {
        context.getResponse().setStatus(299);
        context.getResponse().setHeader("Content-Type", "text/foobar");
        try {
            context.getResponse().getOutputStream().print("hi.");
        } catch (IOException e) {
            e.printStackTrace();
        }
        return HTEXTENSION_REQUEST_PROCESSED;
    }
    
    return HTEXTENSION_EVENT_HANDLED;
}

You probably won't want to, since we're not lacking for options when it comes to responding to web requests in Java, but it's nice to know you can.

You can even respond to tell http foo commands:

1
2
3
4
5
6
7
8
9

public int processConsoleCommand(String[] argv, int argc) {
    if(argc > 0) {
        if("foo".equals(argv[0])) { //$NON-NLS-1$
            System.out.println(getClass().getSimpleName() + " was told " + Arrays.toString(argv));
            return HTEXTENSION_SUCCESS;
        }
    }
    return HTEXTENSION_EVENT_DECLINED;
}

So that's neat.

The fun one, as it usually is, is the #authenticate method. One of the main reasons one might use DSAPI in the first place is to provide your own authentication mechanism. I did it years and years ago, Oracle did it for their middleware, and HCL themselves did it recently for the AppDev Pack's OAuth implementation.

So you can do the same here, like this super-secure implementation:

1
2
3
4

public int authenticate(IJavaSapiHttpContextAdapter context) {
    context.getRequest().setAuthenticatedUserName("CN=Hello From " + getClass().getName(), getClass().getSimpleName());
    return HTEXTENSION_REQUEST_AUTHENTICATED;
}

The cool thing is that this has the same characteristics as DSAPI: if you declare the request authenticated here, it will be fully trusted by the rest of HTTP. That means not just Java - all the classic stuff will trust it too:

Conclusion

Again: this stuff is even further from supported than the usual components I muck around in, and you shouldn't trust any of it to work more than you can actively observe. The point here isn't that you should actually use this, but more that it's interesting what things you can find floating around the Domino stack.

Were this to be supported, though, it'd be phenomenally useful. One of Domino's stickiest limitations as an app server is the difficulty of extending its authentication schemes. It's always been possible to do so, but DSAPI is usually prohibitively difficult unless you either have a bunch of time on your hands or a strong financial incentive to use it. With something like this, you could toss Apache Shiro in there as a canonical source of user authentication, or maybe add in Soteria - the Jakarta Security implementation - to get per-app authentication.

There's also that OAuth 2 thing floating around in there, which does have a usable extension point, but I think it's fair to assume that it's unfinished.

This is all fun to tinker with, though, and sometimes that's good enough.

I've spent a lot of time over the last while tinkering with the XPages Jakarta EE Support project in particular and Jakarta technologies in general, and I figured it'd be worth discussing a bit why I like this stack and why I think it's worth putting work into.

There are a couple facets to this, I think. Why is it good on its own? Why is it good as a complement or replacement for XPages? And why is it good compared to the other roads offered for Domino developers?

Quick Aside: Spring and Others

Before I get much further, I should mention early on that this isn't so much about Jakarta as opposed to technologies like Spring. Spring is good! It's similar in concept, both because it started from a JEE-aligned mindset and now because Jakarta and MicroProfile have been adopting a lot of the best concepts. It's kind of a "D&D and Pathfinder" situation. While there are some philosophical differences, and Jakarta is (now) run by an open-source organization as opposed to an individual company, the distinction for our purposes isn't important.

This also goes for some other technologies that could potentially be slotted in for server-based app dev, like Vert.x. Vert.x, for its part, often serves different purposes, and so that discussion is also separate.

Technical Reasons

Going into all the specific things that I think are good about JEE technologies would be quite an ordeal, so I'll stick to summarizing some overarching themes that I appreciate.

Presumably as a sign of my own ever-increasing age, I appreciate the staid nature of many aspects of it. While some of that comes from the near-stagnation the stack suffered from towards the end of Oracle's sole stewardship of it, it's good that things like Servlet have remained consistent in important ways since the very beginning. Some aspects have come and some will soon go, but the main aspects have remained pleasantly consistent because they were designed to be simple and largely adaptable. Servlet has its limitations, but they're limitations that don't generally show up for normal use.

I also quite appreciate how annotation-based most of the specs are. This was a good way of moving away from the original "pile of XML" configuration process of the early versions of Java EE while still retaining introspection abilities. What I mean by that is the ability of programs (like a server or an IDE) to look at a Jakarta app and glean important information without having to actually execute the code. As a point of comparison, take this hypothetical version of a REST server, where you declare endpoints programmatically:

1
2
3
4

public void initServer(ServerConfig config) {
    config.addHandler("/foo", new FooHandler());
    config.addHandler("/foo/bar", new BarHandler());
}

...and then compare that to the annotation-based way of doing it:

1
2
3
4
5
6
7
8

@Path("/foo")
public class FooHandler {
	/* snip */

	@GET
	@Path("bar")
	public Object getBar() { /* ... */ }
}

Both could be functionally the same at runtime, but the latter allows tools to inspect the classes statically to provide summaries and capabilities in the UI in a way that would be technically possible but much more difficult otherwise. This is certainly not unique to Jakarta, but it's an important feature of it nonetheless.

Moreover, I think that the stack is morphing itself nicely into a cleaner, modern form. It's been a rocky process, but a lot of the individual specs are either adapting themselves onto CDI or using it as the baseline. As much as I sang the praises of Servlet in the earlier paragraph, you can write a thoroughly-capable app using CDI and JAX-RS without ever caring about much else beyond a data layer.

This adaptability is also paying off with newer-era work like Quarkus. Quarkus is an intriguing project that combines slices of Jakarta, MicroProfile, and others with the native-compilation capabilities of GraalVM to provide a toolchain that lets you write quite-efficient compiled apps, targeted primarily for Kubernetes deployments where the startup and response time of a single node is very important. This is really solving a lot of problems I don't have, but it's interesting to watch, and to see how these goals feed back into Jakarta with things like CDI Lite.

Jakarta As An XPages Extension Or Successor

XPages was (and is) a fork of a subset of Java EE, with the split happening somewhere just before 2006's Java EE 5. It's a small subset, but you can look down that list of technologies and see a few that remain to this day: Servlet 2.4/2.5, JSF 1.1/1.2 by way of the XPages fork, JavaMail, and a few other miscellaneous packages. DAS and the Extension Library brought in JAX-RS 1.1, so you can add a dash of 2009's Java EE 6 to the pot.

The XPages Jakarta EE Support project started as a mechanism for bringing in a newer JAX-RS version, followed by CDI to replace managed beans and EL 3 to replace XPages's primordial Expression Language support - essentially, as a slowly-growing platform update. In its current form, it brings in a wide slate of technologies, but the fact that it was starting as an extension of an existing ancient Java EE fork made it possible to do this gradually, piece by piece. Really, up until the move to the jakarta.* namespace, it was a process of just glomming compatible parts onto the existing Servlet baseline.

Even after that switch, the historical alignment with the older parts of the stack makes building on comparatively straightforward. That applies both to me as the person doing the adapting the spec implementations and (hopefully) to a developer actually using them. While XPages predated the annotation-heavy push in JEE as well as CDI entirely, a lot of the core concepts are in common, and I expect that it'd be an easier transition from XPages alone to Jakarta EE than, say, classic Domino web dev to XPages was. It certainly was for me, anyway.

Jakarta As A Cultural Match

This topic covers both my general appreciation for a thoroughly-open-source platform and why I specifically like it in relation to other roads open to Domino developers.

Java EE had for a long time been kind of open source: though Sun and then Oracle held the reins, the specs grew to be free to implement, and over time there flourished a slew of compatible servers, many of which are now or have always been fully open-source.

I like this for a lot of reasons. For one, it's just good as a programmer to have source access. Normally, you can just go by the spec, but having full access to the server's source lets you debug thorny problems when you hit an edge case. While closed-source software certainly has its place, there's just a layer of "all else being equal, source access is better".

Beyond being able to see and debug the source, it's valuable that the platform is open source and the implementations I use are as well, and the whole thing is guided by the extremely-established Eclipse Foundation. While a company handing something over to an open-source organization can sometimes just be a way to usher it to a plausibly-deniable death, the activity around Jakarta EE shows that isn't the case here. While Oracle and IBM still tend to naturally top the charts, it has a diverse pool of contributors, and its fate isn't tied to the interests of a single company. As with closed-source software, sometimes being shepherded by a single company has advantages, but it leaves you more exposed to the winds of their financial incentives.

This all contributes to a platform where I can be comfortable writing a bunch of code with the knowledge that, while it may not be good forever, the path will be at least clear. While there's something of an industry for modernizing old Java EE applications (one our old friend Niklas Heidloff is involved in), it's a task shared by a lot of companies and, indeed, a lot of the work is "replace old vendor-specific code with standards-based code". While nothing can truly prevent you from having a pile of obsolete code other than not writing it in the first place, following a path like this that's shared with a broad slice of the industry is a good way to mitigate the trouble.

And I think that paragraph is what a lot of it comes down to for me. As much as I can be, I want to be out of the business of writing code that doesn't have a built-in "plan B". If Domino magically stopped existing tomorrow, code written in this way wouldn't necessarily directly work elsewhere, but it'd be a much shorter journey than for the Notes-client and classic Domino web code I wrote in my early career. And, really, some of it would directly work elsewhere. The stuff that's just describing a REST entrypoint or a page layout? The stuff that describes the interaction between those and an abstracted data layer? That code doesn't care what your server is, and there's a whole ecosystem of servers ready to do the job. That, there, is what makes this worthwhile for me.

Yesterday, I had a problem. I was trying to get MicroProfile Config working inside an NSF to add to the XPages Jakarta EE project, and I was severely blocked by odd behavior.

To describe that, I'll lightly cover what MP Config is. It's a CDI extension that allows you to annotate properties on a bean to indicate that they're intended to come from an available configuration source - often a .properties file in the project, but it's a pluggable system. Your bean will look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

package example;

// ...

@ApplicationScoped
public class ConfigExample {
    @Inject
    @ConfigProperty(name="java.version", defaultValue="unknown")
    private String javaVersion;
    
    /* other methods go here */
}

The idea is that you'll then have a properties file or environment variable to fill in the value, allowing you to separate your configuration from the implementation in a consistent way. Here, I'm making use of the fact that a default provider looks up Java system properties, so I could just get it working before investigating adding providers.

Since I'd already added CDI and a CDI-based extension in the form of MVC, I figured this would be easy.

The Problem

The problem I hit, though, was bizarre. CDI would identify the bean above, but would hit this problem:

1
2
3
4
5

org.jboss.weld.exceptions.DeploymentException: WELD-001408: Unsatisfied dependencies for type String with qualifiers @Default
  at injection point [UnbackedAnnotatedField] @Inject private example.ConfigExample.javaVersion
  at example.ConfigExample.javaVersion(ConfigExample.java:0)
WELD-001475: The following beans match by type, but none have matching qualifiers:
  - Producer Method [String] with qualifiers [@Any @ConfigProperty] declared as [[UnbackedAnnotatedMethod] @Dependent @Produces @ConfigProperty protected io.smallrye.config.inject.ConfigProducer.produceStringConfigProperty(InjectionPoint)]

The gist of this is that it noticed that the javaVersion property is supposed to be an injected property, but it had no idea what the source should be. It did know about the MicroProfile provider, which handles @ConfigProperty, but it couldn't put two and two together.

I banged my head against this for a while, and eventually determined that the class as loaded from the NSF is stripped of the @ConfigProperty annotation outright. Other annotations, such as @Inject and even custom annotations, would remain, but not @ConfigProperty. I wrestled with OSGi dependency chains for a while, to no avail.

The Enemy

Eventually, I found the core, and it was an old nemesis of mine. It's this method in com.ibm.xsp.util.ClassLoaderUtil:

This field is called by the ClassLoader used in an NSF to ensure that certain classes, by name prefix, cannot be loaded by code coming from an NSF. The last three lines there make a sort of sense: Domino is supposed to be an app container for XPages apps, and ideally it's not a simple process for an app to break out of its container to muck about in the parent environment. Fair enough. The NAPI line is presumably there because IBM wanted to protect developers from themselves, even though Notes devs had been making unauthenticated calls to C APIs for freaking ever.

It's the first two, and specifically the first, that are the source of my trouble. Those prohibitions are presumably meant to isolate XPages apps from the fact that they live in an OSGi world, with the assumption that anything beginning with org.eclipse. refers to something like org.eclipse.core.runtime, the OSGi system bundle.

And this is the issue. MicroProfile is not in any way related to OSGi, but it sure is an Eclipse project. Accordingly, the class name of @ConfigProperty is org.eclipse.microprofile.config.inject.ConfigProperty, and thus cannot be loaded from an NSF.

Attempted Workarounds

So I considered my options.

One was to fork MP Config and rename the packages. That would work, but it would defeat the portability goals of the XPages JEE project, and would also just be a hassle - I've already had to fork a few specs, and each new one adds to the maintenance burden. That remained an option, but it would be a last resort.

My next idea was to wrap around the ModuleClassLoader class used by NSFComponentModule for class loading purposes. This class is blessedly non-final, and so in theory I could look at the instance in a module and swap it out with a replacement. I tinkered with this a bit, but the trouble became the way it's layered, with a DynamicClassLoader private class within - something harder to subclass. In theory, I could reproduce the behavior of it wholesale, but that would be both fragile (if the implementation ever changes) but also verging on if not outright illegal (it's one thing to be API-compatible, and another to reproduce the internal functionality). After some wrangling, I decided to look elsewhere.

The True Workaround

I realized eventually that I don't really care about ModuleClassLoader as such: it does its job fine, and it's only the response that it gets from ClassLoaderUtil that is the problem. If I could change that, I would be set.

I've used the Javassist project here and there for a long time, ever since its inclusion in ODA for one reason or another. It's a handy toolkit, and notably includes the capability to alter a method implementation on the fly. There's my loophole.

The reason this kind of thing can work is related to how Java handles classes and calls between them. For all intents and purposes, you can consider a method call from one bit of Java to another to be a string-based lookup, saying "find a class named X and a method named Y, and then execute it". The "find" part there is much looser than you might think. It's easy to think of class references like C static linking, but they're really not. When code asks for a class, it asks the context ClassLoader, and that object can do basically whatever the heck it wants to find it, as long as it eventually emits a Class that the runtime can deal with.

Javassist's manipulation makes use of the fact that classes are generally eventually just a bunch of bytes, and you can do whatever you want with a bunch of bytes. Using Javassist, it's fairly simple to, once you have a handle on the class, alter the method. Truncated, that's:

1
2
3
4
5
6

ClassPool pool = /* build a ClassPool that can load the class */;
CtClass cc = /* get the class from the pool */;
cc.defrost();
CtMethod m = cc.getDeclaredMethod("checkProhibitedClassNames");
m.setBody("{ return false; }");
Class<?> result = cc.toClass();

And this works, as far as it goes: I now have a Class version of ClassLoaderUtil that skips the onerous check.

The trouble now was to get this to be actually used by other classes. Generally, once a ClassLoader loads a class, it's difficult to feed it another version unless it's designed to do so: most ClassLoader implementations, including those used here, are designed to read and emit classes by their own rules, not have new data fed into them.

I tried digging through the Eclipse OSGi ModuleClassLoader (distinct from the NSF ModuleClassLoader) for entrypoints and had some initially-promising work with Eclipse's internal ClassLoaderHook type, but eventually determined that this would require more patching than I'd want, if it was possible at all.

I also considered using Java's instrumentation capabilities to intercept class loading, but that would require setting up a special Java agent in the launch parameters, which would be too onerous.

But then I remembered something I had heard about when looking into getting ServiceLoader to work with OSGi: a concept in the OSGi spec called "weaving".

OSGi Weaving

I had noted that this concept existed, but set it aside in large part due to how esoteric it sounds: the term "weaving" makes it sound like it's a way to interact with the threads of fate or something, which is evocative but not something that seems immediately useful.

What it really is, though, is an OSGi-friendly version of the above: when the OSGi runtime goes to load a class from a bundle, it reads the data but then gives any such listeners an opportunity to manipulate the code before it's actually reified into a class. This is how the ServiceLoader mediator does its thing: it looks for ServiceLoader calls during loading and re-"weaves" them to run through OSGi instead.

This was perfect: it provides exactly the hook I want and it does it in a clean, spec-based way, without having to do weird reflection to reassign object properties or anything.

The Implementation

So I went about writing such an implementation. All the pieces are there on Domino, and the mechanism for registering a WeavingHook is something I'd done before in Open Liberty: it's a type of OSGi service that you can register and manage in an Activator class. It's also the sort of thing that would work well with Declarative Services, but Domino doesn't have a DS handler installed and I figured I didn't need to solve that quite yet.

So I wrote a WeavingHook implementation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

public class UtilWeavingHook implements WeavingHook {
    @Override
    public void weave(WovenClass c) {
        if("com.ibm.xsp.util.ClassLoaderUtil".equals(c.getClassName())) {
            ClassPool pool = new ClassPool();
            pool.appendClassPath(new LoaderClassPath(ClassLoader.getSystemClassLoader()));
            CtClass cc;
            try(InputStream is = new ByteArrayInputStream(c.getBytes())) {
                cc = pool.makeClass(is);
            } catch (IOException e) {
                throw new UncheckedIOException(e);
            }
            cc.defrost();
            try {
                CtMethod m = cc.getDeclaredMethod("checkProhibitedClassNames");
                m.setBody("{ return false; }");
                c.setBytes(cc.toBytecode());
            } catch(NotFoundException | CannotCompileException | IOException e) {
                new RuntimeException("Encountered exception when weaving ClassLoaderUtil replacement", e).printStackTrace();
            }
        }
    }
}

This builds on the above Javassist usage to now load the class from the byte array provided by OSGi, transform it, and then write the new version back. Since this happens while OSGi is reading the class to begin with, there's never a time when there's an older, less-permissive version of the class running around, as long as I get my service in early enough.

This service is registered in the Activator without too much fuss:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

public class JakartaActivator implements BundleActivator {
    private final List<ServiceRegistration<?>> regs = new ArrayList<>();

    @Override
    public void start(BundleContext context) throws Exception {
        regs.add(context.registerService(WeavingHook.class.getName(), new UtilWeavingHook(), null));
    }

    @Override
    public void stop(BundleContext context) throws Exception {
        regs.forEach(ServiceRegistration::unregister);
        regs.clear();
    }
}

The final bit to get right was the "get the service in early enough" aspect. The main task was making sure that this bundle was activated before any XPages apps loaded, and that was a job for my old friend IServiceFactory, which is the extension point that's intended to add handlers for incoming URLs but has the desirable attribute of being initialized right at the very start of HTTP loading.

With this in place, I now have a fix automatically applied to that fiendish class on load, and MicroProfile Config (and future MP specs) works like a charm.

Conclusion

This was an arduous one, and I think the FTL victory jingle actually physically played when I got it working. I've hated this restriction for a long time, and I'm glad to finally have a workaround.

It was also enlightening to properly learn about OSGi's weaving capability. As mentioned above, this is what the ServiceLoader bridge does, and I'd tinkered with that at one point, but never got it working. I suspect now that it should be entirely doable to make it work, most likely also involving bringing in an implementation of the Declarative Services OSGi capability. That should be a fun project in its own right.

Moreover, the fact that I now have a system in place to do this weaving on the fly means that I may be able to un-fork some of the specs I had to fork to get working previously, which specifically required altering ServiceLoader calls. Even if I don't get the official service bridge in, perhaps I can use this technique to just alter the parts I need to on the fly, and otherwise use stock implementations from Maven.

But, for now, the way is cleared for further progress, and a bizarre mystery is solved. I call that a good day.

Over the weekend, I wrapped up the transition to jakarta.* for my XPages JEE Support project and uploaded it to OpenNTF.

With that in the bag, I decided to investigate adding some other things that I had been itching to get working for a while now: JSP and MVC.

JSP? Isn't That, Like, A Billion Years Old?

Okay, first: shut up.

Expanding on that point, it is indeed pretty old - arriving in 1999 - and its early form was pretty bad. It was designed as an answer to things like PHP and ASP and bore all those scars: it used actual Java syntax on the page to control output, looping, conditionals, and the like. It even had special directives to import Java classes for the page! All that stuff is still in there, too, which isn't great.

However, JSP used judiciously - focusing on JSTL tags for control/looping and EL references to CDI beans for data access - is a splendid little thing, and it has the advantage that it remains part of the JEE spec.

Domino flirted with JSP for a long time. It's what Garnet was all about and was part of how OpenNTF got off the ground. IBM did eventually ship the custom tags, and they ship with Domino to this day, sitting in the data/domino/java directory, gathering dust. Domino also inherited JSP from WebSphere as part of XPages... kind of. It has hooks for using JSP files in Expeditor-container webapps, but the implementation is conspicuously missing - present only in Notes, presumably for some sort of Social nonsense reason.

For better or for worse, none of that matters now anyway: it's all crusty and old and, critically, uses javax.*. I had to go a different route.

JSP Implementation

From what I gather, there's basically only one real open-source JSP implementation: Jasper, which is a part of Tomcat. Basically everyone just uses that, and that works well enough. There are various re-bundlings of it to remove the Tomcat dependencies, and I went with the GlassFish one, since it was pretty clean.

Diving into it, there were a few things that were potential and actual problems.

First, JSP files aren't evaluated directly. Instead, they're compiled into Servlet class implementations, either on the fly or ahead of time. This process is basically the same as how XPages work: the JSP is translated into a Java file, which is then compiled into a class, which is then reused by the runtime for subsequent requests. Jasper has a dependency on Eclipse JDT, which worried me: when I looked into this in the past, I found that JDT (at least how it was used for JSP) makes a lot of assumptions about working with the actual filesystem. I lucked out here, though: Jasper actually uses the JavaCompiler API, which is more flexible. The JDT dependency seems like either a vestige of an older version or a fallback option.

However, despite the fact that JavaCompiler can work purely in memory, Jasper does do a lot of filesystem-bound work when it comes to loading tag libraries, such as JSTL. I ended up having to deploy a bunch of stuff to the filesystem. Ideally, I'll find a better way around this.

Hooking It Up To Domino

Having a JSP interpreter is one thing, but having it respond to URLs like "http://example.com/foo.nsf/bar.jsp" is another, especially if that should also participate in the XPages class space of the NSF.

I originally considered an HttpService implementation that would accept incoming *.jsp URLs. This could work, but it would be less than ideal: the HttpService, while working in the XPages OSGi layer, wouldn't know about the internal layout of the NSF. I'd have to either reinvent it or wrangle my way over to the active NSFService (the one that runs XPages), find or load the NSF's module, and root around in there. Possible, but not ideal.

Fortunately, I lucked out tremendously: the NSFService class has an addHandledExtensions static method that I can just call to tell it that incoming ".jsp" requests should go to the XPages runtime. This looks like it was added for more Social-nonsense reasons, but I'm happy it's there regardless. Better still, when the runtime finds a URL it was told to handle, it queries IServletFactory implementations like those you can use in an NSF for custom servlets. I already had one in place for JAX-RS, so I made another one (refactored since that commit) for JSP.

Once that was in place (plus some other fiddly details), I got to what I wanted: writing JSPs inside an NSF and having them share the XPages class space:

Next Up: MVC

Once I had JSP in place (and after some troublesome fiddling with JSF), I decided to take a swing at adding my beloved MVC to the stack.

This had its own complications, this time for the inverse problem as JSP. While Jasper is a creature of the early 2000s and uses older, less-flexibile Java APIs that I had to write around, MVC is the opposite. It's a pure child of the modern, CDI-based world and thus does everything through CDI and ServiceLoaders. However, though I've had CDI support in the project for a long time, actually tying together anything to do with CDI or ServiceLoaders in OSGi is eternally difficult, especially on Domino.

Service Loading

I had to wrangle for this for a while, but I eventually came up with a functional-but-odd workaround: I made use of my own custom ServiceParticipant extension capability that lets me have an object perform pre/post behavior around each JAX-RS request in order to futz with the ClassLoader. I had trouble where the NSF ClassLoader didn't find classes from the MVC implementation, though it should have, so I ended up overriding the ClassLoader to first look explicitly there. It's not pretty, but it works and at least it doesn't require filesystem stuff.

Servlets and Request Dispatchers

Another aspect of being a more-modern child than Jasper is that Krazo makes ready use of Servlet capabilities that have been there for a while but which don't exist on Domino.

For example, Krazo uses a ServletContainerInitializer instance to do pre-research in the app to find classes that should get MVC behavior. Without this scan, MVC won't be applied. This is a Servlet 3.0 feature dating to 2009 and Domino doesn't support it - or any kind of annotation-based classpath scanning, for that matter.

Fortunately, I didn't really need to fully support this concept - I really just needed to make sure this ran whenever the JAX-RS support was being loaded for an NSF. So I made it possible to contribute these via an extension point and added my own scanning implementation to gather the applicable types. Essentially, a backport of this feature to apply in an NSF. With that in place, I was able to register the initializer and have it do its work.

My next hurdle was to do with the way Krazo delegates to JSPs. Specifically, it queries the ServletContext (essentially, the app container) for Servlet registrations that can handle the desired extensions (".jsp" and ".jspx" here) and routes to that using a RequestDispatcher. Well, Domino supports none of this. Trying to get a RequestDispatcher is hard-coded to throw an exception saying "Domino doesn't support this" and the bit about getting ServletRegistrations was new in 3.0. Originally, I stubbed these out, but I decided to give a swing at backporting this as well.

While an NSF doesn't have "Servlet registrations" as such, it does have a list of the aforementioned IServletFactory instances, so I decided to write my own. I wrote a getRequestDispatcher implementation that queries the current module's Servlet factories for a match and, when found, return a basic implementation. Then, I wrote a custom subtype of IServletFactory to provide additional information and made use of that to emulate the Servlet 3+ behavior, at least well enough to let Krazo do what it needs.

Seeing It Together

Once I figured out all these hurdles, I got to what I wanted: I can make a JAX-RS service in an NSF that acts as an MVC controller:

Neat! There are still some rough edges to clean, but it's great to see in action.

Conclusion and Next Steps

So why is this good? Well, there's a certain amount of box-checking going on: the more JEE specs I can get going, the better.

But beyond that, this is helping to crystallize some of my thinking about what Domino (web) developers are even supposed to freaking do nowadays. This remains an extremely-vexing problem, but I know the answer isn't XPages as it exists now. Maybe the answer is to move XPages to a better container or maybe it's to add a better container to Domino (or both of those, I guess). This is another option, one that preserves the "just fire up Designer and edit some code" niceties of the XPages experience while gaining better, more modern capabilities. I could see writing an app with this, doing all my work in CDI beans and using JSP as the front end - pure open-source solutions with active developers - all inside the NSF. Is it the real best answer? I don't know. Maybe. It's something, though, and specifically something worth trying.

I think it's been a little while since I talked about the XPages Jakarta EE Support project of mine. The goal of that is sort of the inverse of the XPages Runtime project: rather than bringing XPages to a proper modern app server, the JEE Support project brings a handful of current Jakarta EE specs to XPages. It started out a few years ago as a sort of proof-of-concept, but I've since been using it for client work to do things like use newer Jakarta REST (n?e JAX-RS), CDI, and newer EL in XPages and OSGi bundles.

The Specification Move

Originally, I targeted a set of specifications from Java/Jakarta EE 8. Some of these were new to Domino outright, while some (such as JAX-RS) existed in the XPages stack already but in very old forms. I implemented those and for a good while just used the project as a source of parts for client work, tweaking it here and there as needed.

However, the long-prophesized package-name switch from javax.* to jakarta.* came to fruition in Jakarta EE 9, released a bit over a year ago. In the intervening year, most implementations of the specs made the switch, and the versions I was using started to show their age (for example, I was using RESTEasy 3, which was already old when I adopted it, and it's going to 6 now). Beyond just the philosophical sadness of my project being behind, I started to grow specific needs to upgrade components: we switched to JSON-B a while ago, but then some new bug fixes in Yasson were coming only to post-jakarta.* builds.

The Initial Work

I first gave a shot to this in August, initially planning to move only JSON-P and JSON-B over to the new namespace. However, I quickly hit the limits of that, since a lot of these specs are interdependent. JAX-RS using JSON-P and JSON-B to emit JSON content, Yasson has some ties to CDI, and so forth. I realized that it was going to have to be all-or-nothing.

So I rolled up my sleeves and assessed the task ahead of me. At a basic level, there was the job of updating my dependencies, which immediately had some good aspects and bad aspects:

• Previously, I was using a hodgepodge of spec packages like the JBoss bundling of JAX-RS in order to get something that would work and be license-friendly. Now that it was all over at Eclipse, I could switch to the nice, clean official versions and have no license worries.
• I also used to have all sorts of OSGi rule overrides to account for Domino-specific oddities like ancient versions of various specs being supplied by the default classpath or other, conflicting bundles, all with no versioning. Once I was looking for e.g. jakarta.annotation instead of javax.annotation, I was no longer bound to that particular nightmare.
• Not all of my dependencies were ready. When I first started, RESTEasy (my JAX-RS provider of choice) had not yet uploaded a JEE-9-compatible version. My main choices were to try using Eclipse Transformer, which would add a whole new layer to the task, or to switch to another provider.

Then there's the elephant in the room: the freaking Servlet API, which much of this depends on. Since the Servlet API is the job of the web container, I can't realistically upgrade it. Fortunately, that's only half true: I can't give it new capabilities (like Web Sockets), but I can wrap the old stuff with the new. And, like the other specs, the switch of the package name was a tremendous blessing, allowing me to deploy the official Servlet 5 API unchanged. Then, I did the tedious work of writing a slew of adapter classes, half wrapping a javax.servlet component and pretending it's jakarta.servlet and half going the other direction. Since the methods are either direct analogs, optional features, or can be emulated, this was actually much easier than I thought it would be. And there: Servlet 5 on Domino! Kind of!

The Showstopper

However, I soon hit what seemed to be a show-stopper: a LinkageError problem when using CDI that didn't show up previously. My search for the topic found only one hit: an issue in Open Liberty referencing almost exactly the same problem. My heart sank when I read that their fix was to upgrade the Equinox runtime - something that's outside my powers on Domino (probably).

So, disheartened, I set it aside for a couple months. I figured there was a small chance that Weld (the CDI implementation at the heart of the trouble) would put out an update that fixed it - after all, an older version worked.

Resuming Work

After setting it aside, it kept eating away at the back of my mind, and two things kept pushing me to go back to it:

• I'll need to do it eventually. I (and my client projects) can't just be stuck at the old style forever.
• I really didn't want to admit defeat and switch back to Gson for JSON processing.

So I went back to it. My initial hope - that a new version of Weld would magically fix the problem - proved to not have come to fruition. Still, though, I wasn't sure that it was the exact same problem Liberty encountered. For one, my use of CDI studiously avoids actually telling it about OSGi, since I've had little luck making use of that with Domino's OSGi stack. That was enough cause to make me think I could work around it.

And work around it I did! The trouble turned out to be, unsurprisingly, a bit esoteric, but boiled down to the runtime re-registering proxy classes for the same core components. My guess is that, somewhere along the line, Weld changed some sort of internal cache in a way that would break when using a bunch of ephemeral per-NSF containers as I do. I implemented my own (since it's an intended extension point) and added a bit of a cache, and I was back to the races.

As a convenient blessing, RESTEasy released 6.0.0.Beta1 just days before I got back to it, a major release targeted at JEE 9. That meant that I could save a ton of work by not having to re-work everything for another JAX-RS implementation. I had been looking into Jersey, which I'm sure would have done the job, but it's fiddly work trying to put all these pieces together on Domino, and I was all the happier to not have to re-do it all.

JavaMail

But then I hit a new problem: the javax.mail API, now jakarta.mail. The first part of this is easy enough: bring in the new spec bundle and everything will point to it. Great! I hit an immediate problem, and one I had been dreading dealing with. Though the spec changed package names, the implementation didn't. That brought me face-to-face again with an old nemesis of mine, sitting there in Domino's classpath, corrupting it:

The way the Mail API works is that there's a file, called "mailcap", that lists implementations for common data types, like:

1
2
3
4

text/plain;;		x-java-content-handler=com.sun.mail.handlers.text_plain
text/html;;		x-java-content-handler=com.sun.mail.handlers.text_html
text/xml;;		x-java-content-handler=com.sun.mail.handlers.text_xml
multipart/*;;		x-java-content-handler=com.sun.mail.handlers.multipart_mixed; x-java-fallback-entry=true

So, while all the entrypoint classes are jakarta.mail.* now, the implementations remain com.sun.mail.*, all with the same class names. And, since this little jerk of a JAR is sitting in the system classpath, it has a way of showing up all the time, complaining that com.sun.mail.handlers.text_plain is incompatible with jakarta.activation.DataContentHandler.

This is extremely fiddly to deal with, potentially involving writing a special ClassLoader implementation that blocks calls down to the lower-level JAR. While maybe possible, I'm not sure it'd be possible in a way that would be practical for normal use in an app.

And so, with a heavy heart, I forked the thing and added an "org.openntf" in front of all the package names. And that... works! It works just fine. It means that I'm on the hook for manually integrating any upstream changes, but at least it works without having to worry about any conflicts.

That wasn't the end of my trouble with this spec, though. The spec package itself, in jakarta.mail.Session uses ServiceLoader to look for services, and it uses it in the form that looks them up with the current thread's ClassLoader. Because I'm working in OSGi, that ClassLoader - the XPage app's loader - won't know about the implementation classes directly, and this call fails. And, while there's a whole sub-spec in OSGi for dealing with this, I've never had success actually getting it working in Domino.

So I forked that freaking thing too and modified the calls to use its own ClassLoader, which could find the implementation by way of it being a fragment bundle attached to it.

And, with that, finally, I had Jakarta Mail properly hooked up and working without having to jump through too many hoops. I'd still prefer to not have forked the source, but it was the best of a bad lot of choices.

The Final Tally

That brings the specs updated/added in this project to:

• Expression Language 4.0
• Contexts and Dependency Injection 3.0
  • Annotations 2.0
  • Interceptors 2.0
  • Dependency Injection 2.0
• RESTful Web Services (JAX-RS) 3.0
• Bean Validation 3.0
• JSON Processing 2.0
• JSON Binding 2.0
• XML Binding 3.0
• Mail 2.1
  • Activation 2.1

Not too shabby, if I say so myself. Technically, Servlet 5.0 is in there, but it doesn't actually bring any newer-than-2.4 powers to the Servlet container, so it's really just infrastructural details.

Now I'll just have the work of updating my client project and finally getting to use whatever that Yasson bug fix was that prompted this in the first place.

In my last post, I sang the praises of InputStream and OutputStream: two classes from Java 1 that, while not perfect, remain tremendously useful and used everywhere.

Then, a tweet by John Curtis got me thinking about the opposite cases: APIs from the early Java days that are still with us, are still used relatively frequently, but which are best avoided or used very sparingly.

There are a handful of APIs from the early days that may or may not still exist, but which aren't regularly encountered in most of our work: the Applet API, for example, was only recently actually removed, and it was clear for a long time that it wasn't something to use. Some other APIs are more insidious, though. They're right there alongside newer counterparts, and they're not marked as @Deprecated, so you just have to kind of magically know why you shouldn't use them.

Old Collections

One of these troublesome holdovers is a "freebie" for Domino developers: java.util.Vector. This is paired with other "first revision" collection classes like Hashtable, classes that predate the Collections Framework in 1.2 and which were retrofitted into it.

These classes aren't incorrect as such: they do what they're supposed to do and function as working implementations of List and Map. The trouble comes in that they're sub-optimal compared to other options. In particular, they're very-heavily synchronized in a way that hurts performance in the normal cases and isn't even really ideal in the complex multi-threaded case.

Unfortunately, since these classes aren't deprecated, an IDE would only warn you about it if it's using some stylistic validation above normal compilation. Such classes are identified best by looking for a warning paragraph like this at the bottom of their Javadoc:

java.util.Date

The java.util.Date class has a simple concept: represent a point in time. However, it's a neverending font of limitations and caveats:

• It's essentially a wrapper for a Unix timestamp in milliseconds precision, and doesn't get more precise
• It's not immutable even though it'd make sense to be. Effective Java includes repeated examples of why this is bad
• Though it's called "Date", it's always a single timestamp, and can't represent a day in the abstract
• In Java 1, it also was responsible for parsing date strings, and this functionality remains (though at least deprecated)
• As mentioned in the prompting tweet, the DateFormat classes that go with this are not thread-safe, even though one could reasonably assume they would be based on their job
• There's no concept of time zone, though the string representation would lead one to think there might be
• The related Calendar class is a little more structured, but in a weird way and having a lot of the same limitations

Nonetheless, Date is the obvious go-to for date/time-related operations due to its age and alluring name. And, in fact, it wasn't even until Java 8 that there was a first-party better option. That's when Java basically adopted Joda Time outright and brought it into Java as the java.time package. This system has what's required: the notion of dates and times as separate entities, time zones both as named entities (like "America/New_York") and just as offsets (like UTC-5:00), and full immutability and thread-safety, and tons else.

Unfortunately, it will be a long time for old habits to die and longer for older code to fade away, so we're stuck with Date for a while, even if only to always call #toInstant on it.

java.io.File

The java.io.File class is kind of similar to Date: it was created in Java 1 as a basic way to work with files on the filesystem. It still does that, and (as far as I know) it's not as outright bad as the above, but it's limited and non-optimal.

In Java 7, the NIO Path API was added, which replaces File in a more-generic and -adaptable way. Whereas File refers specifically to the filesystem, the Path API is adaptable to whatever you'd like while sharing the same semantics. It can also participate in the NIO ecosystem properly.

Much like how Date has a #toInstant method, File has a #toPath method to work with the transition. I make a habit of doing this almost all the time when I'm working with existing code that still uses File. And there is... a lot of this code. Even APIs that can take Path arguments will potentially turn them into Files internally to keep working with their older implementation.

There are also a bunch of related APIs where the replacements exist but aren't quite as straightforward. ZipFile is a perfect example of this: it (and its child class JarFile) has constructors that take either a File or a String representing a file path, and that's alarming. However, the ZIP File System Provider that works with Paths is neat, but it's not as clear of a replacement for ZipFile as Path is for File. That's actually one of the reasons I use ZipInputStream even in a case where ZipFile would also work.

Conclusion

I'm sure there are other similar traps around, but those are the main ones I can think of off the top of my head. It's a bit of a shame that Sun/Oracle have been so historically reticent to mark classes wholesale as deprecated. While IDEs and and toolchains have gotten better at providing "stylistic" recommendations like this, it's been slow going, and it's not universal. The best thing you can do for now is to just know about the newer alternatives and use them enough that the old kinds immediately read as "code smell" when you come across them.

When working on version 3.0 of the Domino Open Liberty Runtime, I had occasion to do something I've done in other situations, but it occurred to me that it'd make a good post on its own. Specifically, part of one of the new features involved creating archive data on the fly, purely in-memory, and that's something that comes in handy quite a bit.

Background: The Task

The task at hand in that project involved the way the runtime will deploy custom extension features for Liberty when creating the server. There are a few of these, all centered around adding integration with Domino in one way or another. For the previously-existing Liberty features, this was done in three parts:

• The actual Liberty extension code, which is a Java project that produces a Liberty-compatible OSGi bundle.
• A "subsystem" module, which is a code-less Maven project that uses esa-maven-plugin to embed the above bundle and generate a "SUBSYSTEM.MF" file to describe it. This ESA/subsystem bit is a mechanism for distributing packaged features from the OSGi spec.
• A "deployment" module, which is a small Java project that provides an extension for the Domino-side runtime to house and deploy the above ESA file.

For 3.0, I wanted to make a feature that would provide Notes.jar and the NAPI to application. Since those files are proprietary and non-distributable, I couldn't include them in the actual runtime distribution and would instead have to look them up from Domino's environment at runtime. Additionally, since all I wanted to do was provide the existing API and not add any new code, there was no particular need to make a code project like the first one above.

More Background: Java Streams

The way these extensions are registered to Domino is by classes that provide some metadata about the feature and then a method called getEsaData() that returns an InputStream. Though InputStreams aren't the only way to represent arbitrary binary blobs like this, they're used everywhere by virtue of them arriving with Java 1, and they're extremely adaptable.

Basically, the idea of an InputStream is that it's just a mechanism to read a sequence of bytes from somewhere. In Domino terms, they're like NotesStream, but good.

Their utility comes from their simplicity and adaptability. Because the abstract class only deals with reading bytes and a few operations for skipping around, they can be used for all sorts of things. The prototypical use is for reading a file. For example:

1
2
3
4

Path someFile = Paths.get("/foo/bar/baz.txt");
try(InputStream is = Files.newInputStream(someFile)) {
  // read file data from the stream
}

They're not limited to that, though: the JDK comes with all sorts of InputStream variants like ByteArrayInputStream, which lets you read from a byte[] in memory.

In addition to being arbitrary as to where the byes are coming from, streams are also very composable. Many types of streams either must or may wrap an existing stream to alter it in some way. One of the more-common cases where you'd do this is when reading ZIP file data. Taking something similar to above:

1
2
3
4
5
6
7
8

Path someZipFile = Paths.get("/foo/bar/baz.zip");
try(
  InputStream is = Files.newInputStream(someFile);
  ZipInputStream zis = new ZipInputStream(is, StandardCharsets.UTF_8)
) {
  ZipEntry entry = zis.getFirstEntry();
  // work with the ZIP entries
}

The thing to note here is that, while this happens to be coming from a ZIP file on disk, it doesn't have to be: that first is could just as easily be a stream coming from HttpURLConnection or a ByteArrayInputStream.

Along with InputStream, Java also has OutputStream. Luckily, OutputStream is similarly simply designed, and has uses that are a direct mirror for everything above: there exist ByteArrayOutputStream, ZipOutputStream, and all sorts of others.

Putting It Together

Back to the original goal, my task was to create a class that would provide an InputStream containing ESA data - that is to say, a ZIP file - to the runtime, which could then deploy it as a Liberty feature. The previous extensions did this by embedding the ESA in their JAR and then returning an InputStream to that. Now, though, I wanted to do it all dynamically.

Now, I talked a big game above about how streams didn't have to have anything to do with files, and it could all be done in memory. That's still all true, but technically here I ended up using files for caching purposes. The above is still good to know, though!

So anyway, my goal was to deliver an InputStream to the runtime that represented an ESA that looks like this:

Of those entries, "corba.jar" is the CORBA API from Maven Central to make Notes.jar work on Java 9+, while "Notes.jar" comes from jvm/lib/ext and "lwpd.commons.jar" and "lwpd.domino.napi.jar" come from the OSGi framework in the running Domino server. The remaining entries - the two "MF" files and the embedded JAR - are composed on the fly.

The starting point here is that I identify a cache location within my working directory based on the current Domino build number, and I name that path out. Then, I open it up as a ZIP to fill with contents, like above:

1
2
3
4
5
6

try(
  OutputStream os = Files.newOutputStream(out, StandardOpenOption.CREATE);
 ZipOutputStream zos = new ZipOutputStream(os, StandardCharsets.UTF_8)
) {
  // work happens here
}

SUBSYSTEM.MF

The next part is to build the "SUBSYSTEM.MF" file. As implied by the extension, this file has the same syntax as "MANIFEST.MF" files, and so I can use the java.util.zip.Manifest class to handle encoding and formatting. I start out by loading a template from the current bundle's resources:

1
2
3
4

Manifest subsystem;
try(InputStream is = getClass().getResourceAsStream("/subsystem-template.mf")) { //$NON-NLS-1$
  subsystem = new Manifest(is);
}

There, I'm using the constructor from Manifest that reads from an existing stream. Often, that would be reading a "MANIFEST.MF" from an existing JAR, but it'll work with any stream.

Then, I fill it in with some details, with lines like:

1
2
3
4
5

Attributes attrs = subsystem.getMainAttributes();
attrs.putValue("Subsystem-Name", getShortName()); //$NON-NLS-1$
String featureName = getShortName() + "-" + getFeatureVersion(); //$NON-NLS-1$
attrs.putValue("IBM-ShortName", featureName); //$NON-NLS-1$
// etc.

Finally, I create an entry in the ZipOutputStream and write the contents. The way ZipOutputStream works is that its "stream-iness" counts towards whatever the most-recently-added entry is.

1
2

zos.putNextEntry(new ZipEntry("OSGI-INF/SUBSYSTEM.MF")); //$NON-NLS-1$
subsystem.write(zos);

Embedded Bundle

Alright, so far, so good. Up until now, this is the "normal" case for working with ZIP files, where you make a new entry and pour in some text data. What's neat, though, is that the encapsulation capabilities of these streams can be stacked, which is what comes up next.

Specifically, I wanted to put a ZIP file (the .jar) within this surrounding ZIP (the .esa). The way this is done is by just composing the same tools we've been working with again. Here, esa is what zos was above: the outermost package ZIP contents. I just renamed it in this method for clarity inside the code itself.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

// This will be a shell bundle that in turn embeds the API JARs
esa.putNextEntry(new ZipEntry(BUNDLE_NAME + "_" + dominoVersion + ".0.0.jar")); //$NON-NLS-1$ //$NON-NLS-2$
		
// Build the embedded JAR contents
try(ZipOutputStream zos = new ZipOutputStream(esa, StandardCharsets.UTF_8)) {
  Manifest manifest;
  try(InputStream is = getClass().getResourceAsStream("/manifest-template.mf")) { //$NON-NLS-1$
    manifest = new Manifest(is);
  }
  
  // Finish manifest
  // More work here
}

So there, I'm doing basically the same thing as I did originally, to make a ZipOutputStream. Since the ZipOutputStream really doesn't care what the stream is writing to is, it works just as well when writing to a file as when writing to another ZIP stream - the cascading streams handle their own encoding and it works out in the end.

Once I write the manifest, I can make use of the Files utility class to embed each of the JARs from the filesystem:

1
2
3
4

for(Path jar : embeds) {
  zos.putNextEntry(new ZipEntry(jar.getFileName().toString()));
  Files.copy(jar, zos);
}

Finally, I download the CORBA JAR on the fly, so for that one I use a utility function to download from the remote URL:

1
2
3
4
5

OpenLibertyUtil.download(new URL(URL_CORBA), is -> {
  zos.putNextEntry(new ZipEntry("corba.jar")); //$NON-NLS-1$
  IOUtils.copy(is, zos);
  return null;
});

Here, I use IOUtils from Apache Commons IO because it's not copying from a filesystem path, but the idea is basically the same, and exactly the same as far as the destination ZIP is concerned.

The Final Result

Once this is all written to the cached file on the filesystem, the final result is just to return a stream from it:

1

return Files.newInputStream(out);

Since the job of this extension class is only to return an InputStream, the consuming code doesn't care that the extension did all this work, as opposed to the other ones that just return a stream of an embedded resource: everything else is the same.

So, all in all, this isn't a groundbreaking new technique, but that's the point: the way these lower-level JDK components work, you get a tremendous amount of flexibility from just a few common parts.

I mentioned in my last post that I've been tinkering with a modern structure for OpenNTF's web site as a side project. In that, I talked about how I've been going with Jakarta MVC for the front end, but ran into an odd problem with the latest versions in Open Liberty, and that was the impetus to tinkering with ERB.

Well, I decided to go back and take a swing at trying to make JSP work in this case, since it's (still) a good engine for this purpose, and it could be a fun experiment. I was indeed able to do it, and I think the path I took is worth chronicling here.

Context

In previous projects, such as this blog, I've used older versions of the software stack involved - basically, Jakarta 8, which is before the "big bang" switch from the javax.* to jakarta.* package namespace. Since this is a clean new app, I really want to lean to the newest versions across the board, so I pegged my plans to that.

Though Jakarta EE 9 and 9.1 (the Java 11 official version) have been out for a bit, the switchover comes with the sort of turbulence one would expect. Until just this past week, Open Liberty supported JEE 9 only in beta releases - these have historically been plenty stable for me, but it's always asking for trouble. Even with that non-beta version out, I found myself still on the beta track: I'm addicted to using MicroProfile Config, and MicroProfile's move to JEE 9 support is still itself in the RC stage.

So, okay, betas it is.

The Problem

Once I set everything up on JEE 9 and MP 5, I hit this exception when trying to render a JSP via an MVC Controller object:

java.lang.RuntimeException: SRV.8.2: RequestWrapper objects must extend ServletRequestWrapper or HttpServletRequestWrapper
  at com.ibm.wsspi.webcontainer.util.ServletUtil.unwrapRequest(ServletUtil.java:89)
  at [internal classes]
  at org.eclipse.krazo.engine.ServletViewEngine.forwardRequest(ServletViewEngine.java:135)
  at org.eclipse.krazo.engine.JspViewEngine.processView(JspViewEngine.java:58)
  at org.eclipse.krazo.core.ViewableWriter.writeTo(ViewableWriter.java:159)
  at org.eclipse.krazo.core.ViewableWriter.writeTo(ViewableWriter.java:1)
  at org.jboss.resteasy.core.interception.jaxrs.ServerWriterInterceptorContext.lambda$writeTo$1(ServerWriterInterceptorContext.java:79)
  ... 4 more

The short of it is that Krazo (the MVC implementation) passes JSP rendering along to the app container, rather than doing its own JSP work, which makes perfect sense. This, however, hits trouble within Liberty's ServletUtil class, which attempts to "unwrap" the incoming HttpServletRequest object to find the core Liberty-specific object to use extended methods on.

Normally, this sort of thing would work fine: every app server has its own variant of HttpServletRequest for its own uses, and it's perfectly reasonable to do this kind of unwrapping. However, for some reason, this was going awry.

The specific code from Krazo that calls down into Liberty code does do new HttpServletRequestWrapper(request), but that's also legal: the unwrapRequest method is intended specifically to unwrap spec-standard HttpServletRequestWrapper objects like that. So that's not our culprit, and I had to dig deeper.

Investigation

To start my investigation, I knew I'd need to work with the Krazo layer. Fortunately, though many of the moving parts here are baked into the Liberty server, Krazo is not - I include it as a Maven dependency in my app. So I cloned the Krazo source, added it to my workspace, and set my dependency on the SNAPSHOT version, allowing me to do my work inside Krazo's classes.

Context

So what was going on? At first, I thought that maybe something had snuck in a javax.* class somewhere - old code that wasn't fully migrated to JEE 9. That would certainly cause the trouble: javax.servlet.ServletRequestWrapper and jakarta.servlet.ServletRequestWrapper are, to the JVM's eyes, entirely-unrelated classes with no compatibility whatsoever. And, indeed, looking at the source of ServletUtil could give one that impression right away, since the code uses javax.*.

That's not the trouble, however. Though the class is written to javax.*, I gather that it's run through Eclipse Transporter during packaging of the app server, and the actual class that's involved uses jakarta.*. Okay, that's good to know and makes sense, but it also doesn't get us any closer to the root problem.

For my next step, I wanted to figure out what, specifically, it was looking for. The unwrapRequest method takes a Class<?> parameter to find the needed request type, but the stack trace above hid the path it took to get there. By attaching a debugger to the server, I gleaned that it was being called by the unwrapRequest variant above it that looks specifically for a com.ibm.wsspi.webcontainer.servlet.IExtendedRequest.

Okay, so I have the name of the interface it's looking for - I can work with this. My next step was to try to get a programmatic handle on it. The basic approach, when you don't have the class as part of your project, it to look it up via:

1

Class<?> requestClass = Class.forName("com.ibm.wsspi.webcontainer.servlet.IExtendedRequest");

That doesn't work here, though: though the app server definitely has that class, it (properly) shields the running application from accessing the class directly, so that the app runtime isn't contaminated by the surrounding server code.

What I needed to do next was find a ClassLoader that does know about it, and the best way to do that is to find a class provided from outside the running app and ask that. Fortunately, the incoming request is exactly that. So:

1

Class<?> requestClass = Class.forName("com.ibm.wsspi.webcontainer.servlet.IExtendedRequest", false, request.getClass().getClassLoader());

What that does is ask whatever ClassLoader the request object comes from - that is to say, the container's loader - to find the class. And it worked! Now I could test to verify whether the core request matches the type needed:

1
2

Class<?> requestClass = Class.forName("com.ibm.wsspi.webcontainer.servlet.IExtendedRequest", false, request.getClass().getClassLoader());
System.out.println("does it match? " + requestClass.isInstance(request));

As expected, that resolved to false. In this case, that's good, since it'd have been a much-worse problem if it hadn't. But what is the request object, anyway? Well:

1
2

System.out.println(request.getClass());
// output: jdk.proxy15.$Proxy68

Right, okay, that makes sense: all sorts of stuff uses proxy objects in Java, not the least of which being the CDI environment running the whole show.

Cracking Open Proxies

So I had some good information at this point: the HttpServletRequest that Krazo is handed is a proxy object, but Liberty has a hard requirement that its dispatcher is given an instance of IExtendedRequest, which this is not. That means that something in the stack is taking the original Liberty request object and making a proxy for it - fair enough, but inconvenient for me.

My next thought was that maybe I could track down the type of proxy object it is and, with that knowledge, get the underlying delegate request. That's a common-enough pattern: have an instance property in your proxy class that contains the delegate, and (if I'm lucky) have it accessible via a getter. Java's java.lang.reflect.Proxy class has a static method for determining the object that actually handles called methods:

1
2

System.out.println(Proxy.getInvocationHandler(request));
// output: org.jboss.resteasy.core.ContextParameterInjector$GenericDelegatingProxy@fc04422d

This was starting to come together all the more. Liberty recently switched from Apache CXF to RESTEasy for its JAX-RS implementation, and that could explain why this is trouble now when it wasn't before. More importantly for my immediate needs, that also gave me a lead to track down the proxy class.

However, though it was easy enough to find, my heart sank a bit at what I found: rather than having an easy instance property to get the real request, it uses an object from its container class and in turn asks that for the request. Maybe I'd be able to get to that via reflection, but the prospect of figuring out how to work with nested class contexts caused me to try to look around elsewhere instead.

CDI

Another potential answer came to me in a flash: CDI! Access to the CDI environment is standardized, and maybe I could fetch the original request from there. It'd be extremely likely that it'd just hand me back a similar proxy, but it'd be worth a shot. So here we go:

1
2
3

HttpServletRequest cdiReq = CDI.current().select(HttpServletRequest.class).get();
System.out.println(cdiReq);
// output: com.ibm.ws.webcontainer40.srt.SRTServletRequest40@1a22712c

Oh! Good! That's one of Liberty's internal types! Is it the object I need, though? Well... no. Crap. requestClass.isInstance(cdiReq) is false, so this didn't really get me very far. That's a shame, too, since that solution could have involved no implementation-specific code at all.

Internal Liberty Classes

My next thought was that I should try to find another way to get around to finding the true request object. I looked back through the debug stack to find where it was originally calling unwrapRequest to get a bit more context:

1
2
3

WebContainerRequestState reqState = WebContainerRequestState.getInstance(false);

wasReq = (IExtendedRequest) ServletUtil.unwrapRequest(request);

Okay, so what's this with WebContainerRequestState? That sure smells like an object that's meant to be a request-wide object to get to all sorts of state. If I were to write such a class, I'd use it to stash the incoming request as well as any other incidental data that I wouldn't want to ferret away in a way that might leak into the app. I was a little wary that WebAppRequestDispatcher didn't use it to get the IExtendedRequest, but maybe I'd luck out.

And boy, did I! Looking down the source of the file, I found my mark: public IExtendedRequest getCurrentThreadsIExtendedRequest().

The (Provisional) Solution

Now, I had all the tools I needed. Towards the bottom of Krazo's ServletViewEngine class, I conjured up this reflective incantation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

try {
  Class<?> requestStateClass = Class.forName("com.ibm.wsspi.webcontainer.WebContainerRequestState", false, request.getClass().getClassLoader());
  Method getInstance = requestStateClass.getDeclaredMethod("getInstance", boolean.class);
  Object requestState = getInstance.invoke(null, false);
  Method getCurrentThreadsIExtendedRequest = requestStateClass.getDeclaredMethod("getCurrentThreadsIExtendedRequest");
  request = (HttpServletRequest)getCurrentThreadsIExtendedRequest.invoke(requestState);
} catch (Throwable e1) {
  // Not on WAS
}
rd.forward(new HttpServletRequestWrapper(request), new HttpServletResponseWrapper(response));

So what I'm doing here is breaking into the container's ClassLoader in order to get a handle on WebContainerRequestState. From there, I'm able to call the method to get the current instance, and then in turn call the method to get the IExtendedRequest. I overwrite the request variable we're working with, and then pass that along to the dispatcher. If any of that were to fail, I just throw up my hands, assume I'm not on Liberty, and continue on as before.

And... it works! It actually works! The pages now render properly, with all the niceness of modern JSP at my fingertips! It was fun to toy with the idea of ERB, but I like this better for an otherwise-pure-Java app for sure.

Next Steps

So I have a solution that works for me, but it's so ugly and implementation-specific that I can't exactly be comfortable with it. Still, the trouble comes from an implementation-specific source, so that may be required. Maybe I'll have to leave it like this.

More responsibly, though, what I should do is narrow this down into a reproducible case without all the other moving parts in the app to make sure that it's actually a bug/incompatibility, and thus something that I can report. This is all open-source software, after all, and it'd do nobody any good for me to let a potential actual problem linger. I'll just have to properly identify where the true culprit is first. Is it because Krazo uses RequestDispatcher in a somewhat-unusual way? Is it because RESTEasy is too aggressive about wrapping requests with no proper way to get to the delegate? Is it that Liberty should handle this case better internally? Or maybe it's just some side effect of the other things I have going on. Research is warranted.

In the mean time, that was a fun one. I don't know that I'll have a need for this specific solution again, but it was good to find, and it's always good to get some troubleshooting practice like this for sure.

One of the very-long-term side projects I have going on is a rewrite of OpenNTF's site. While the one we have has served us well, we have a lot of ideas about how we want to present projects differently and similar changes to make, so this is as good a time as any to go whole-hog.

The specific hog in question involves an opportunity to use modern Jakarta EE technologies by way of my Domino Open Liberty Runtime project, as I do with my blog here. And that means I can, also like this blog, use the delightful Jakarta MVC Spec.

However, when moving to JEE 9.1, I ran into some trouble with the current Open Liberty beta and its handling of JSP as the view template engine. At some point, I plan to investigate to see if the bug is on my side or in Liberty's (it is beta, in this case), but in the mean time it gave my brain an opportunity to wander: in theory, I could use ERB (a Ruby-based templating engine) for this purpose. I started to look around, and it turns out the basics of such a thing aren't difficult at all, and I figure it's worth documenting this revelation.

MVC ViewEngines

The way the MVC spec works, you have a JAX-RS endpoint that returns a string or is annotated with a view-template name, and that's how the framework determines what page to use to render the request. For example:

1
2
3
4
5
6
7
8
9

@Path("home")
@GET
@Produces(MediaType.TEXT_HTML)
public String get() {
  models.put("recentReleases", projectReleases.getRecentReleases(30)); //$NON-NLS-1$
  models.put("blogEntries", blogEntries.getEntries(5)); //$NON-NLS-1$

  return "home.jsp"; //$NON-NLS-1$
}

Here, the controller method does a little work to load required model data for the page and then hands it off to the view engine, identified here by returning "home.jsp", which is in turn loaded from WEB-INF/views/home.jsp in the app.

In the framework, it looks through instances of ViewEngine to find one that can handle the named page. The default spec implementation ships with a few of these, and JspViewEngine is the one that handles view names ending with .jsp or .jspx. The contract for ViewEngine is pretty simple:

1
2
3
4

public interface ViewEngine {
  boolean supports(String view);
  void processView(ViewEngineContext context) throws ViewEngineException;
}

So basically, one method to check whether the engine can handle a given view name and another one to actually handle it if it returned true earlier.

Writing A Custom Engine

With this in mind, I set about writing a basic ErbViewEngine to see how practical it'd be. I added JRuby to my dependencies and then made my basic class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

@ApplicationScoped
@Priority(ViewEngine.PRIORITY_APPLICATION)
public class ErbViewEngine extends ViewEngineBase {

  @Inject
  private ServletContext servletContext;

  @Override
  public boolean supports(String view) {
    return String.valueOf(view).endsWith(".erb"); //$NON-NLS-1$
  }

  @Override
  public void processView(ViewEngineContext context) throws ViewEngineException {
    // ...
  }
}

At the top, you see how a custom ViewEngine is registered: it's done by way of making your class a CDI bean in the application scope, and then it's good form to mark it with a @Priority of the application level stored in the interface. Extending ViewEngineBase gets you a handful of utility classes, so you don't have to, for example, hard-code WEB-INF/views into your lookup code. The bit with ServletContext is there because it becomes useful in implementation below - it's not part of the contractual requirement.

And that's basically the gist of hooking up your custom engine. The devil is in the implementation details, for sure, but that processView is an empty canvas for your work, and you're not responsible for the other fiddly details that may be involved.

First-Pass ERB Implementation

Though the above covers the main concept of this post, I figure it won't hurt to discuss the provisional implementation I have a bit more. There are a couple ways to use JRuby in a Java app, but the way I'm most familiar with is using JSR 223, which is a generic way to access scripting languages in Java. With it, you can populate contextual objects and settings and then execute a script in the target language. The Krazo MVC implementation actually comes with a generic Jsr223ViewEngine that lets you use any such language by extension.

In my case, the task at hand is to read in the ERB template, load up the Ruby interpreter, and then pass it a small script that uses the in-Ruby ERB class to render the final page. This basic implementation looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

@Override
public void processView(ViewEngineContext context) throws ViewEngineException {
  Charset charset = resolveCharsetAndSetContentType(context);

  String res = resolveView(context);
  String template;
  try {
    // From Apache Commons IO
    template = IOUtils.resourceToString(res, StandardCharsets.UTF_8);
  } catch (IOException e) {
    throw new ViewEngineException("Unable to read view", e);
  }

  ScriptEngineManager scriptEngineManager = new ScriptEngineManager();
  ScriptEngine scriptEngine = scriptEngineManager.getEngineByExtension("rb"); //$NON-NLS-1$
  Object responseObject;
  try {
    Bindings bindings = scriptEngine.createBindings();
    bindings.put("models", context.getModels().asMap());
    bindings.put("__template", template);
    responseObject = scriptEngine.eval("require 'erb'\nERB.new(__template).result(binding)", bindings);
  } catch (ScriptException e) {
    throw new ViewEngineException("Unable to execute script", e);
  }

  try (Writer writer = new OutputStreamWriter(context.getOutputStream(), charset)) {
    writer.write(String.valueOf(responseObject));
  } catch (IOException e) {
    throw new ViewEngineException("Unable to write response", e);
  }
}

The resolveCharsetAndSetContentType and resolveView methods come from ViewEngineBase and do basically what their names imply. The rest of the code here reads in the ERB template file and passes it to the script engine. This is extremely similar to the generic JSR 223 implementation, but diverges in that the actual Ruby code is always the same, since it exists just to evaluate the template.

If I continue down this path, I'll put in some time to make this more streamable and to provide access to CDI beans, but it did the job to prove that it's quite doable.

All in all, I found this exactly as pleasant and straightforward as it should be.

Yesterday, I talked about how I dove into Testcontainers for my app-testing needs. Today, I decided to use this to close another bit of long-open business: automated browser testing. I've been very much a dilettante when it comes to that, but we have a handful of browser-ish tests just to make sure the login page, the main page, and some utility pages load up and include expected content, and those can serve as a foundation for much more.

Background

In general, when you think "automated browser testing", that means Selenium. As a toolkit, Selenium has hooks for the browsers you want and has essentially universal support, working smoothly in Java with JUnit. However, the actual act of loading a real browser is miserable, mostly on account of needing you to install the browser and point to it programmatically, which is doable but is another potential system-specific configuration that I'd much, much rather avoid in my automated builds.

Accordingly, and because my needs have been simple, I've used HtmlUnit, which is a portable Java browser-like library that does the yeoman's work of letting you perform basic Selenium tests without having to configure actual native OS installations. It's neat, imposes basically no strictures on your workflow, and I recommend it for lots of uses. Still, it's not the same as real browsers, and I had to do things like disable JavaScript processing to avoid it tripping up on some funky JS that full-grown browsers can deal with.

Enter Webdriver Containers

So, now that I had Testcontainers configured to run the web app, my eye turned to Webdriver Containers, an ancillary capability of Testcontainers that lets you run these full-fledged browsers via their Docker images, and even has cool abilities like letting you record the screen interactions over VNC. Portability and full production representation? Sign me up.

The initial setup was pretty easy, just adding some dependencies for the Selenium remote driver (replacing my HtmlUnit driver) and the Testcontainers Selenium module:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

<dependency>
    <groupId>org.seleniumhq.selenium</groupId>
    <artifactId>selenium-remote-driver</artifactId>
    <version>3.141.59</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>selenium</artifactId>
    <version>1.15.3</version>
    <scope>test</scope>
</dependency>

Programmatic Container Setup

After that, my next task was to configure the containers. I'll skip over some of my troubleshooting and just describe where I ended up. Basically, since both the webapp and browsers are in Docker containers, I had to coordinate how they communicate with each other. There seem to be a few ways to do this, but the route I went was to build a Docker network in my container orchestration class, bind all of the containers to it, and then reference the app via a network alias.

With that addition and some containers for Chrome and Firefox, the class looks more like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

public enum AppTestContainers {
    instance;
    
    public final Network network = Network.builder()
        .driver("bridge") //$NON-NLS-1$
        .build();
    public final GenericContainer<?> webapp;
    public final BrowserWebDriverContainer<?> chrome;
    public final BrowserWebDriverContainer<?> firefox;
    
    @SuppressWarnings("resource")
    private AppTestContainers() {
        webapp = new GenericContainer<>(DockerImageName.parse("client-webapp-test:1.0.0-SNAPSHOT")) //$NON-NLS-1$
                .withExposedPorts(8080)
                .withNetwork(network)
                .withNetworkAliases("client-webapp-test"); //$NON-NLS-1$
        
        chrome = new BrowserWebDriverContainer<>()
            .withCapabilities(new ChromeOptions())
            .withNetwork(network);
        firefox = new BrowserWebDriverContainer<>()
            .withCapabilities(new FirefoxOptions())
            .withNetwork(network);

        webapp.start();
        chrome.start();
        firefox.start();
        
        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            webapp.close();
            chrome.close();
            firefox.close();
            network.close();
        }));
    }
}

Now that they're all on the same Docker network, the browser containers are able to refer to the webapp like "http://client-webapp-test:8080".

Adding Parameterized Tests

The handful of UI tests I'd set up previously had lines like WebDriver driver = new HtmlUnitDriver(BrowserVersion.FIREFOX, true) to create their WebDriver instance, but now I want to run the tests with both real Firefox and real Chrome. Since I want to test that the app works consistently, I'll want the same tests across browsers - and that's a call for parameterized tests in JUnit.

The way parameterized tests work in JUnit is that you declare a test as being parameterized, and then feed it your parameters via one of a number of mechanisms - "all values of an enum", "this array of strings", and a handful of others. The one to use here is to make a class implementing ArgumentsProvider and configure that:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

import java.util.stream.Stream;

import org.junit.jupiter.api.extension.ExtensionContext;
import org.junit.jupiter.params.provider.Arguments;
import org.junit.jupiter.params.provider.ArgumentsProvider;
import org.testcontainers.containers.BrowserWebDriverContainer;

public class BrowserArgumentsProvider implements ArgumentsProvider {
    @Override
    public Stream<? extends Arguments> provideArguments(ExtensionContext context) throws Exception {
        return Stream.of(
            AppTestContainers.instance.chrome,
            AppTestContainers.instance.firefox
        )
        .map(BrowserWebDriverContainer::getWebDriver)
        .map(Arguments::of);
    }
}

This class will take my configured browser containers, get the WebDriver instance for each, and provide that as parameters to a test method. In turn, the test method looks like this:

1
2
3
4
5
6
7
8

@ParameterizedTest
@ArgumentsSource(BrowserArgumentsProvider.class)
public void testDefaultLoginPage(WebDriver driver) {
    driver.get(getContainerRootUrl());
    assertEquals("Expected App Title", driver.getTitle());

    // Other tests follow
}

Now, JUnit will run the test twice, once for each browser, and I can add any other configurations I want smoothly.

Minor Gotcha: Container vs. Non-Container URLs

Though some of my tests were using Selenium already, most of them just use the JAX-RS REST client from the testing JVM directly, which is not containerized in this setup. That meant that I had to start worrying about the distinction between the URLs - the containers can't access "localhost:(some random port)", while the JUnit JVM can't access "client-webapp-test:8080".

For the most part, that's not too tough: I added some more utility methods named to suit and changed the UI tests to use those. However, there was one tricky bit: one of the UI tests uses Selenium to fetch the page and process the HTML, but then uses the JAX-RS client to make sure that a bunch of references on the page resolve to non-404 resources properly. Stuff like this:

1
2
3
4
5

driver.findElements(By.xpath("//link[@rel='stylesheet']"))
    .stream()
    .map(link -> link.getAttribute("href"))
    .map(href -> rootUri.resolve(href))
    .forEach(uri -> checkUrlWorks(uri, jaxRsClient));

(It's highly likely that there's a better way to do this in Selenium, but hey, it's still a useful example.)

The trouble with the above was that the URLs coming out of Selenium included the full container URL, not the host-accessible one.

Fortunately, that's not too tricky - it's really just string substitution, since the host and container URLs are known at runtime and won't conflict with anything. So I added a "decontainerize" method and run my URLs through it in the stream:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

public URI decontainerize(URI uri) {
    String url = uri.toString();
    if(url.startsWith(getContainerRootUrl())) {
        return URI.create(getRootUrl() + url.substring(getContainerRootUrl().length()));
    } else {
        return uri;
    }
}

// later

driver.findElements(By.xpath("//link[@rel='stylesheet']"))
    .stream()
    .map(link -> link.getAttribute("href"))
    .map(href -> rootUri.resolve(href))
    .map(this::decontainerize)
    .forEach(uri -> checkUrlWorks(uri, jaxRsClient));

With that, all the results came back green again.

Overall, this was a little fiddly, but mostly in a way that helped me learn a little bit more about how this sort of thing works, and now I'm prepped to do real, portable full test suites. Neat!

Though my previous post was also about proxies, I've had this topic percolating for a while, and it's related mostly by name and very-loose concept. Specifically, I'd like to talk about dynamic proxy classes in Java, which is a mechanism to allow you to create "fake" classes that programmatically intercept method calls.

This is something that I didn't properly realize was possible in Java until diving into CDI (though its mechanism is slightly different). In retrospect (and by "@since" annotation), it's obvious that this has been present for a long time, since Java 1.3, but outside of my realm of experience.

Definition, the Roundabout Way

So, to begin with, I'll have to define what I even mean by this, or at least an example of how it works in practice.

Normally, you just have classes and objects based on them - class Foo gets instantiated via new Foo() and there's a pretty clear direct relationship, "Object-Oriented Programming 101" sort of stuff. Let's add a little indirection by way of our old friend Interfaces. Say you have this setup:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

interface Person {
    String getName();
}

class PersonImpl implements Person {
    @Override
    public String getName() {
        return "Joe Schmoe";
    }
}

// ...

Person foo = new PersonImpl();
System.out.println("Hello from " + foo.getName());

This is essentially the same kind of thing that you're doing in the original concept - instantiating an object that you can call methods on - but you're taking a step back. You know here that you're just calling new PersonImpl(), but that's less of a hard requirement: you could instead do Person foo = lookupPerson("Joe Schmoe") and that method could return any implementation of the interface it likes.

So let's do just that, and here's where we see proxies:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

class PersonProxy implements InvocationHandler {
    public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
        if("getName".equals(method.getName())) {
            return "Proxy-Eyed Joe";
        }
        throw new UnsupportedOperationException("I don't know how to handle " + method);
    }
}

// ...

public Person lookupPerson(String name) {
    return (Person)Proxy.newProxyInstance(Thread.currentThread().getContextClassLoader(), new Class<?>[] { Person.class }, new PersonProxy());
}

For the code calling lookupPerson, this will function largely identically to if you used a normal class, but it's got this weirdness going on beneath. When code calls getPerson (or any other method), rather than calling a directly-implemented method like in the normal class, it instead calls invoke on this InvocationHandler, which can then handle it any way it'd like. If you've used more-dyanmic languages, this may look similar to method_missing in Ruby or invocation forwarding in Objective-C.

Non-Thorough Mention of Other Proxy Types

Before continuing into why you'd want to use this, I think it's important to note that the Proxy object in question above is java.lang.reflect.Proxy, which is a built-in mechanism that ships with the JVM. However, it's both limited and not the only game in town. The main way it's limited is that you can only proxy to interfaces with it, not normal classes. If Person above were class Person instead of interface Person, then the stock Proxy class would be out of luck.

There are other implementations, though - the ones that spring to mind are cglib and Javassist, but I believe there are others. These differ in their implementation (often doing things like bytecode manipulation), capabilities (these generally allow you to proxy to full classes), and performance characteristics. The concepts are largely the same, though, so from now on I'll use "proxying" to refer to the concept generally and not solely to the specific capability that comes with Java.

So What's the Use?

Okay, so you can make a proxy object that allows for programmatic handling of method calls. How would this actually be useful in practice?

In my work, I've had a couple cases where I implement proxies for specific behaviors, and they're also extremely common in Jakarta EE and Spring development. Some of these uses can get pretty arcane in concept or implementation, but others will hopefully be simpler to demonstrate.

Example 1: Counting Method Calls

For this case, say you want to count how many times a method is called in practice (and also say that YourKit doesn't exist for some reason). Taking our example InvocationHandler above, we can expand it to keep a running total of calls:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

class PersonProxy implements InvocationHandler {
    private final Map<Method, AtomicLong> counter = new HashMap<>();

    public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
        this.counter.computeIfAbsent(method, key -> new AtomicLong()).incrementAndGet();

        if("getName".equals(method.getName())) {
            return "Proxy-Eyed Joe";
        }
        throw new UnsupportedOperationException("I don't know how to handle " + method);
    }
}

(I use AtomicLong here because it's a convenient number holder, but it's also incidentally a step in the direction of making this thread-safe)

Now, as any method is called on your object, you'll get a count of invocations. You can imagine elsewhere having an admin console that lists totals for each object, giving you an idea of where the performance-sensitive parts of your code likely are.

This is, in fact, what MicroProfile Metrics does, albeit in a more-flexible and -complete way than this example. That spec uses annotations to define what you want tracked and how, and then the CDI-based proxy objects can keep track of counts and execution times.

Example 2: Performance Improvements

For this case, imagine you have a model framework in Domino where you have classes representing back-end documents. The loading of these documents can get pretty expensive, especially if you implemented it in a traditional way where your code loads up all values for the front-end Java class from the document at once. However, say you also have a mechanism to look up these documents in bulk via views, where values from the document might be already indexed and readable faster without having to crack the whole thing open. You might end up with a proxy class like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

class PersonModelProxy implements InvocationHandler {
    private final ViewEntry viewEntry;
    private PersonImpl realDoc;

    public PersonModelProxy(ViewEntry viewEntry) {
        this.viewEntry = viewEntry;
    }

    public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
        if("getId".equals(method.getName())) {
            return viewEntry.getUniversalID();
        }

        PersonModel obj = someExpensiveObjectLoad();
        return method.invoke(obj, args);
    }
}

PersonModel person = fetchModelWithProxy();
String unid = person.getId(); // Fast!
String firstName = person.getFirstName(); // Slow, but could be made fast

Here, because the UNID will already be present in the view entry, we can just return that immediately rather than loading the full backend document. If you expand this to apply to other properties that can come from view columns, you can do efficient batch lookups and tables while not having to have a separate "read from view instead of doc" mechanism on the front end.

This is something I'm doing (using Javassist's proxies) with a client project to put view entries over existing model objects that had accrued over the years. This allows us to keep the same logic while massively speeding up operations that don't actually need the document to be loaded.

Example 3: Repetitive but Predictable Code

If you have behavior that can be reliably derived from some non-algorithmic source (say, from annotations, app configuration, or so forth), proxies can help you write adaptive code once that avoids the need to write tons of boilerplate methods or classes.

As an example, I'll expand a bit on the first notion above, which is profiling. A long time ago, the erstwhile XPages team released the XPages Toolbox, which provides a reasonably-fine-grained view into what takes up the time during an XPages request. It also provides a way for your code to opt into this profiling, but the idiom is extremely repetitive. You can see it in the Extension Library, and it tends to look something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

public class SomeBean {
    private static final ProfilerType profilerType = new ProfilerType("SomeBean");

    public void doFoo() {
        if(Profiler.isEnabled()) {
            ProfilerAggregator agg = Profiler.startProfileBlock(profilerType, "doFoo");
            long ts = Profile.getCurrentTime();
            try {
                _realDoFoo();
            } finally {
                Profiler.endProfileBlock(agg, ts);
            }
        } else {
            _realDoFoo();
        }
    }

    private void _realDoFoo() {
        // Actual code goes here
    }
}

That's certainly explicable, but imagine writing that for all methods, or even for a large chunk of methods you want to optimize. That could be done a little more cleanly now with Java 8+ features, but it'd still be a drag.

If we go back to the idea handled by MicroProfile Metrics, it would instead look more like:

1
2
3
4
5
6
7

@Profiler(name="SomeBean")
public class SomeBean {
    @Timed(name="doFoo")
    public void doFoo() {
        // Actual code goes here
    }
}

That's a little nicer! In practice, this is really done with CDI, which provides its own nice layer around proxies, but you could see implementing it with IBM's profiler and a proxy like so (forgive the fragility of the code):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

class ProfilerProxy implements InvocationHandler {
    private final Map<Class<?>, ProfilerType> profilers = new HashMap<>();

    public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
        ProfilerType profilerType = profilers.computeIfAbsent(proxy.getClass(), c -> new ProfilerType(c.getAnnotation(Profiler.class).name()));

        if(Profiler.isEnabled()) {
            String metricName = method.getAnnotation(Timed.class).name();
            ProfilerAggregator agg = Profiler.startProfilerBlock(profilerType, metricName);
            long ts = Profiler.getCurrentTime();
            try {
                method.invoke(obj, args);
            } finally {
                Profiler.endProfilerBlock(agg, ts);
            }
        } else {
            method.invoke(obj, args);
        }
    }
}