Shortly after we moved to our current house, we got the usual deluge of welcome letters: ads for local business, insurance scams, that sort of thing. Among them was the newsletter for the local Friends of the Library group. "I like books," I figured, so I sent a donation their way. That led to an invitation to attend one of their board meetings, which it turns out was a (successful) scheme to get me to join their board.

What struck me immediately was just how much libraries - and Friends groups that support them - do. I went in with just the basic view that they were the place where you went to borrow books, but quickly learned about all the ancillary services: book discussions, art nights, chess and D&D clubs, computer access for those without, gardening programs, museum passes, tech education, and so on. They also generally act as one of the few places one can go to idly socialize without also being expected to pay money. They're often community centers in their own right, providing a friendly place for anyone, in particular marginalized or less-well-off patrons. Also, just being a member often gets you access to various e-book-lending and video streaming services for free.

Thus, despite being an inveterate introvert, I'm still involved with the local Friends and (thanks to the way our system works) on the system's board as well. It's quite likely that you have a similar group for your local library system, and I can heartily recommend you get involved. The people in it are almost definitely delightful and it's a great way to get involved in a light and flexible way.

Libraries and their Friends are also, of course, always in need of donations. A nice attribute of that is that they're usually small enough that it's easy to see the path of "more money" to "better programs". If you can - and especially if you're in a place where libraries are currently assailed by ruinous powers - find and donate to your local library. If you're lucky enough to be in a place where your system is well-funded, my local system and Friends group can always put donations to exceptional use.

Tomorrow, January 23rd, will kick off a new collaboration between OpenNTF and HCL: the Open Mic series. The idea of these is to replicate a bit of the "Ask the Developers" feeling from the Lotusphere days, with each session featuring developers, product managers, and other applicable HCL personnel for the topic at hand.

Like webinars, these will focus on a specific topic each month and, also like webinars, these topics will vary greatly, with some being developer-focused, others being admin-focused, and others potentially being more user-focused.

The first one is tomorrow at 11 AM US Eastern time, and you can register for it on GotoWebinar. We have a good slate of these lined up, and our plan is to run them monthly.

Though one day I'd really like to sit down and work on expanding and categorizing the documentation for the XPages JEE project, in the mean time I can at least put together some scattered info in the form of blog posts, webinars, and example apps. Add this post to the pile! Some of it will be a rehash of previous posts, but it doesn't hurt to see it rephrased.

One of the main tentpoles of a pure XPages JEE app (I also wish I had a catchier name for it) is the data-access layer, which uses Jakarta Data and NoSQL to have an abstraction layer over the specifics of Domino data. We as Domino developers are conditioned by privation to do our data access in a fairly primitive way: though the lotus.domino classes technically sit a couple layers of abstraction over the base, they're still stuck at the level of dealing with documents, views, items, and other constructs specifically. If you want to read a the "first name" value from a document representing a person, you always have to do doc.getItemValueString("FirstName") - then the same for last name, job title, and so forth. If you're reading from a view, you have to have an alternate version of your code that loops through that - using the same loop style you've written a million times in more- and less-efficient ways - and reads from column values. In a simple case, fine, but this ends up being a lot of boilerplate code.

Anyway, without further ado, I'll get in to how it should be done.

How It Should Be Done

A basic class for representing a Person would look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

package model;

import java.util.stream.Stream;

import org.openntf.xsp.jakarta.nosql.mapping.extension.DominoRepository;

import jakarta.nosql.Column;
import jakarta.nosql.Entity;
import jakarta.nosql.Id;

@Entity
public class Person {
	public interface Repository extends DominoRepository<Person, String> {
		Stream<Person> findAll();
		
		Stream<Person> findByLastName(String lastName);
	}
	
	@Id
	private String documentId;
	@Column
	private String firstName;
	@Column
	private String lastName;
	
	/* snip: getDocumentId(), setDocumentId(), getFirstName(), etc.) */
}

There's a bit of a nod to being Domino-specific there, which we'll cover later, but there's no business where you have to do doc.getUniversalID(), doc.getItemValueString("FirstName"), and all that. You don't even need to implement any code that does the document lookups - the framework provides that for you. Thanks to some CDI magic, the you can use the repository as-is in a bean or REST service:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

package rest;

import java.util.Comparator;
import java.util.List;

import jakarta.inject.Inject;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.QueryParam;
import jakarta.ws.rs.core.MediaType;
import model.Person;

@Path("person")
public class PersonResource {
	
	@Inject
	private Person.Repository personRepository;
	
	@Path("byLastName")
	@Produces(MediaType.APPLICATION_JSON)
	public List<Person> getByLastName(@QueryParam("lastName") String lastName) {
		return personRepository.findByLastName(lastName)
			.sorted(Comparator.comparing(Person::getFirstName))
			.toList();
	}
}

Under the hood, the JNoSQL driver translates the call to personRepository.findByLastName(lastName) to a DQL query like Form = "Person" and LastName = "Fooson", reads the "FirstName", "LastName", and UNID from each document, and then returns Person objects in a stream.

Adding A Bit Of Finesse

In that example, I have the bit where the REST service sorts the users it found by first name - that's fine, but it's really the database's job. Fortunately, that's easy to add. First, we can change our repository to add a jakarta.data.Sort method:

1
2
3
4

public interface Repository extends DominoRepository<Person, String> {
	// ...	
	Stream<Person> findByLastName(String lastName, Sort<Person> sortOrder);
}

Then, we can simplify the REST service:

1
2
3
4
5

@Path("byLastName")
@Produces(MediaType.APPLICATION_JSON)
public List<Person> getByLastName(@QueryParam("lastName") String lastName) {
	return personRepository.findByLastName(lastName, Sort.asc("firstName")).toList();
}

The result will be the same (or actually probably more what you want, since this will be case-insensitive), but now the DB is doing the work. Specifically, the work it's doing is that this will cause the driver to make a temporary NSF that will house a QueryResultsProcessor view that lists those documents sorted by the FirstName item. It will also be mildly intelligent about this: if the data documents in the DB haven't changed, it'll re-use the existing index, which means that subsequent calls to the same method with the same parameters (and user) will be very fast.

Domino Optimizations

Using that default DQL+QRP behavior is sufficiently performant in a lot of cases (moreso than you might think, too), but it's still potentially prone to the usual pitfalls we have with Domino data access. If you have more documents or complex selections, you'll want to fall back to views or folders.

Say you want to juice your quarterly numbers so you can buy another house, and so you want to get rid of a bunch of employees and you're triaging them via a folder. First, let's add a column to show their salary to our list of fields in the Person class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

@Entity
public class Person {
	// ...
	
	@Id
	private String documentId;
	@Column
	private String firstName;
	@Column
	private String lastName;
	@Column
	private int salary;

	// snip
}

Then, we can add another method to the repository class for this:

1
2
3
4
5

public interface Repository extends DominoRepository<Person, String> {
	// ...
	@ViewEntries("Vacation-House Triage")
	Stream<Person> findDoomedEmployees(Sort<Person> sortOrder);
}

The @ViewEntries annotation there will override the default DQL-and-QRP behavior, and instead open and traverse the view or folder with a ViewNavigator. That also means altering the behavior of the Sort parameter: if you pass Sort.desc("salary"), instead of creating a sorted column in the temporary QRP view, now it will call view.resortView(...) to make use of the "Click on column header to sort" functionality in the folder's design, making the lookup as fast as possible.

Creating And Modifying Documents

So far, these examples have only been to read existing data, but other parts of CRUD similarly work with high-level objects.

If we want to create a new user, there's no more work to do in the repository - we can just add an endpoint to our REST service:

1
2
3
4
5
6

@POST
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.APPLICATION_JSON)
public Person createPerson(Person newPerson) {
	return personRepository.save(newPerson);
}

The save method is provided by the base repository interface, so that's all that's needed. The framework will take a value like:

1
2
3
4

{
	"firstName": "Foo",
	"lastName": "Fooson"
}

...convert the JSON to a Java object, write the values into a document with the "Person" form value, retrieve a new version of the object with the UNID filled in, and return:

1
2
3
4
5

{
	"documentId": "020322A6F79CB02C85258C140058A1FF",
	"firstName": "Foo",
	"lastName": "Fooson"
}

Similar to the Domino-specific support for views, there's another version of the save method that you can call to compute with form: personRepository.save(newPerson, true).

To update an existing document, you do basically the same thing, but you put the UNID into the object (in practice, the method above should strip the UNID just in case), which will be in a document retrieved from. For example:

1
2
3
4
5
6
7
8

@Path("{documentId}")
@PUT
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.APPLICATION_JSON)
public Person updatePerson(@PathParam("documentId") String documentId, Person newPerson) {
	newPerson.setDocumentId(documentId);
	return personRepository.save(newPerson);
}

You might want to check if the document by UNID already exists for business-logic purposes, but the underlying behavior is mostly the same anyway.

What We Didn't Have To Do

The primary thing that I want to emphasize is how much work the app developer doesn't have to do. Anything you can do with the JEE framework is doable with the traditional XPages toolkit, but, by the same token, it'd also all be doable with assembly language - you definitely shouldn't do that, though.

This takes care of a tremendous amount of busywork: opening and processing documents, traversing view efficiently, efficient DQL and QRP use, converting to and from JSON, and building REST endpoints. I didn't get into other normal problems that could be solved here: model validation via annotations, sending 404s via meaningful exceptions, modifying data-reading behavior via custom reader classes (critical with weird old Notes data), calling external REST services by interface, and so forth.

Whenever I work with Domino apps not designed with this sort of thing, it's a depressing experience. It's the same mess of varyingly-performant view reading, manually mapping data to intermediate objects, manually writing or reading JSON objects, manually doing HTTP calls for REST services, and all of that over and over and over. It's annoying to write, annoying to read, easy to get wrong, and extremely difficult for a new developer (even an experienced one) to maintain. If you haven't tried the framework and you're writing web apps for Domino, I highly recommend giving it a shot.

As part of finishing my holiday gift shopping, I published version 3.3.0 of the XPages Jakarta EE project today.

This release contains a number of bug fixes to do with asynchronous and scheduled tasks based on some edge-case and intermittent trouble I ran into while developing some apps with it. Additionally, it has some consistency fixes for the Jakarta NoSQL support - in particular, it improves mapping of object properties to columns, matching the item names case-insensitively and matching special fields like FIELD_CDATE to matching columns with formulas like @Created.

Additionally, I took the occasion to bump some dependencies. While Jakarta EE 11 was pushed to next year, MicroProile 7.0 was released a few months back. This brings some version bumps to specs included in this project, including Rest Client, Open API, and Fault Tolerance. While the changes aren't dramatic, there are some nice refinements in there. I was also able to drop the Apache HttpClient in favor of an implementation that uses URLConnection, and it's always nice to lessen the number of dependencies.

However, there will be some more work to do in future versions when it comes to MicroProfile. The Metrics spec was dropped from MP 7.0 in favor of Telemetry, which I glean hews more closely to common practices in other tools. In XPages JEE 3.3.0, Metrics remains and I have yet to add Telemetry. It may end up being a breaking change, but I'm not sure that alone would warrant a major-version bump for this project, in large part because I don't know how much use the existing Metrics implementation gets here. In any event, my plan is to at least add Telemetry to 3.4.0 or so.

After that, while waiting for JEE 11 for some large updates, I'm a little tempted to dive into the "Better NSF Webapps" idea from my post the other day. That's one I've wanted to do for a while, and it would be really nice to get rid of the "xsp/app" part of URLs for full-Jakarta apps. I kind of doubt that that feature, even if I start working on it, would make it into the next version in a proper way, but it'd at least be interesting to take a swing at.

TL;DR: If you use XPages, don't use Domino 14.0 FP3. To fix the unrelated mail-routing bug, use the IFs for 14.0 FP2 or any earlier version. For the curious, the below bugs are tracked in HCL as SPR #RKRYDC2MHV.

(Updated 2024-12-17 below)

Fix Pack 3 for Domino 14.0 came out last week and, in addition to the usual spate of fixes you'd expect from an FP, it brought a potentially-significant and -breaking change to the way XPages handles style attributes for components. This is presumably in the interest of supporting the pathological strictures of Content-Security-Policy, but it applies whether or not you have taken any other steps to implement CSP.

The specific change is that the XPages renderers will now take the style attribute, externalize it to a class, and then make a <style> block for it. Take this XSP markup:

1

<xp:text style="color: red" value="I should be red"/>

While the above XSP was rendered as <span style="color: red">I should be red</span> on all previous versions, the rendered HTML (with unnecessary bits removed) now looks like this:

1
2
3
4
5
6
7

<span class="xc1">I should be red</span>

<style>
.xc1{
color: red
}
</style>

In this example, the results are equivalent: the text is red. However, this breaks in more-complicated examples. For example, say you have this in your XPage:

1
2
3
4

<xp:text style="color: red" styleClass="foo" value="I should be red"/>
<style>
	body .foo { color: green }
</style>

(The <style> block could just as easily be an external stylesheet for this purpose.)

In this case, the resultant HTML is:

1
2
3
4
5
6
7
8

<span class="xc1 foo">I should be red</span>
<style>body .foo { color: green }</style>
<style>

.xc1{
color: red
}
</style>

...and the text is green, though it should not be. Inline style attributes have a level of specificity that isn't quite equivalent to any other selector construct ("!important" rules included), so this change leads to insidious problems as things get more complicated. While inline styles have always been something of a faux pas, they're extremely common, all the more so for us because Designer encourages it and the official XPages tutorial tells you to use them.

Unfortunately, these changes are not opt-in and, because the problems are not technically errors, you won't see any console or error logs about it, and just looking at a page may not make visual changes (if there are any) immediately obvious. When using afflicted versions, the only real way to see if you need to change anything for now is to check each application manually to see if it breaks.

In the mean time, if you're upgrading 14.0 servers to account for the recent mailing problem and have XPages applications in production, I recommend sticking with Fix Pack 2, which got an Interim Fix 2 release to fix mail. If you're in the 14.5 Early Access program, it would probably also be helpful for you to chime in with your own experiences in the thread I made in the forum.

Update: Discussions in the OpenNTF Discord have highlighted several side effects or related changes that break xe:dialog elements and add extra <span>s around <xp:text>s that previously didn't have them, breaking more apps. Accordingly, my advice to not use 14.0 FP3 is only stronger now.

Lately, the XPages Jakarta EE project is in a very good place: the move to Jakarta EE 10 cleaned up a lot of the codebase, there aren't currently any more looming brick walls, and the app development using it I've been doing has remained exceedingly productive. The product has a long list of issues to work on - a few of them are difficult-to-reproduce bugs, but most are small-to-medium-sized features. There are a handful of things, though, that would be big projects on their own that I'd love to find the time to do, but don't currently have enough of an impetus to devote the time to.

In no particular order:

Jakarta Batch (Or Scheduled Jobs Generally)

The Jakarta Batch spec is a way to define data-processing tasks in a standardized way - think agents but more explicit. It's a bit of a staid spec - it came from IBM and sure looks like it was specifically designed to be a very mainframe-y way of doing things - but that may be fine.

The advantages of this over agents would be that it would run in the OSGi class space and could use app code fully and that the definition language is pretty flexible. Additionally, though the XPages JEE project already has programmatically-defined scheduled tasks by way of Jakarta Concurrency, those tasks are pretty opaque from outside, while Batch jobs could theoretically be described usefully in a server-wide way for administrative purposes.

Jakarta Messaging

The Jakarta Messaging spec defines a consistent way to work with message queues and pub/sub systems, which helps when making an app as part of a larger system. There are a bunch of pretty enterprise-y implementations, but this looks like it could be a pretty good fit for local use on a Domino server, potentially with Domino's own message queues. Being able to have apps send messages to each other (or to tasks outside the HTTP JVM) could have a lot of uses, and having it baked in to the framework would make it worth considering much more than it's currently used.

Code Generation For NoSQL

When using NoSQL entity classes, it's not strictly necessary to have an actual Form design element, but it's a very common case that you'd have one - either because you make a quick-and-dirty Notes UI first or you're building a JEE app on top of an existing Notes app. Accordingly, it'd be neat to have an option in Designer to automatically generate Java classes for existing forms, saving some tedium of manually defining each property.

There are a couple things that would make doing this sort of a PITA, though. For one, it'd involve writing a UI plugin for Designer, which sounds like it'd just be a miserable process. I could probably work primarily or entirely with the Eclipse VFS and project APIs instead of Designer-specific classes, but still. Beyond that, there'd be the matter of trying to correctly describe a form. Text and number fields would be easy enough, but things would get tricky quickly. Should a multi-option field be presented as an Enum if the options are compatible? Should it try to guess boolean-storage fields? What system-type fields - like SaveOptions - should be included in the model as opposed to handled via compute-with-form? Should the display options for date/time fields map exactly to java.time classes? Not impossible, especially since "good enough" would be significantly better than nothing, but still a deep well of potential work.

Quality-of-Life Extension Projects

For the most part, the XPages JEE project focuses on implementing the specs as they are, with only a few things that are Domino-specific. Over time, I've been building up ideas for little features to add: a bean to handle Domino name formatting, supporting alternative HTML templating tools like Thymeleaf, packaged libraries for common tasks like Markdown formatting and RSS feeds, and so forth. These wouldn't really fit in the core project because I don't want it to bloat out of scope, but I could see them being their own additions or a general "extension library" for it.

Better OSGi Webapps

The JEE project has a couple capabilities in the direction of being used in OSGi-based webapps, but constantly bumps into ancient limitations with it. Something I've considered doing is making an alternative extension point to basically do this but better, presenting a Servlet 6 environment for OSGi-wrapped webapps, avoiding the need for wrappers and translation layers between old and new.

This would be a big undertaking, though. A chunk of it is handled by the existing HttpService/ComponentModule system, so it wouldn't be like writing a Java app server from scratch, but there'd be a lot to do as far as managing application lifecycles and so forth. Still, having control over the ComponentModule level would let me handle things that are finicky or impractical currently, like annotation-based Servlets, ServletContainerInitializers, and various listeners that just aren't implemented in the existing stack.

To go along with this, I'd want to look into what I can do with the Jakarta Data/NoSQL support to make it practical to not use Domino-specific interfaces all the time. The idea there would be that you could write an app that uses NoSQL with one of the other supported databases but then, when running on Domino, would store in an NSF instead. This would make it possible to develop entirely outside of Domino (and thus not have to worry about Designer) in a way that's basically the same. There'd be some trouble there in that it's pretty easy to hit a point with Domino as a data store where you need to give hints for views or data storage to make it practical, so it wouldn't always be doable, but it's worth considering.

Better NSF Webapps

Which leads to the last big potential feature: doing a custom ComponentModule but for NSFs. The idea here would be that you would, one way or another, register your NSF as a webapp container, and then it would be handled by a new ComponentModule type that eschews the XPages and legacy parts in favor of exerting full control over events, listeners, and URLs. This would allow apps to skip the "xsp/app" stuff in the URL, make it easier to do things like Filters, and have proper hooks for all app lifecycle listeners.

Like OSGi webapps, this would be a real 80/20 sort of thing, where some of the early steps would be fairly straightforward but would quickly get into the weeds (for example, having to write a custom resource provider for stylesheets, files, etc.). Still, I keep running into limitations of the current container, and this would potentially be a way out. It's probably the one I'd want to do most, but would also be the most work. We'll see.

I may get to some or all of these on my own time anyway, and any of them may end up cropping up as a real client need to bump them up the priority list. It's also just kind of nice having a good stable of "rainy day" projects to tickle the mind.

One of my favorite features of the XPages JEE Support project is the inclusion of the MicroProfile Rest Client. This framework makes consuming remote APIs (usually JSON, but not always) much, much better than in traditional XPages or even in most other frameworks.

MicroProfile Rest Client

If you're not familiar with it, the way it works is that you can use Jakarta REST annotations to describe the remote API. For example, say you want to connect to a vendor's service at "https://api.example.com/v1/users", which allows for GET requests that list existing users and POST requests to create new ones. With the MP Rest Client, you'd create an interface like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

package com.example.api;

import java.util.List;

import com.example.api.model.User;
import jakarta.ws.rs.Consumes;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.POST;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;

@Path("v1")
public interface ExampleApi {
	@Path("users")
	@GET
	@Produces(MediaType.APPLICATION_JSON)
	List<User> getUsers();

	@Path("users")
	@POST
	@Consumes(MediaType.APPLICATION_JSON)
	@Produces(MediaType.APPLICATION_JSON)
	User createUser(User newUser);
}

Then, in your Java code, you can use it like this (among other potential ways):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

ExampleApi api = RestClientBuilder.newBuilder()
	.baseUri(URI.create("https://api.example.com"))
	.build(ExamplApi.class);

List<User> existingUsers = api.getUsers();

// ...

User foo = makeSomeUser();
api.createUser(foo);

The MP Rest Client code takes care of all the actual plumbing of turning that interface into a usable REST client, translating between JSON and custom classes (like User there), and other housekeeping work. Things can get much more complicated (authorization, interceptors, etc.), but that will do for now.

Records

What has been a nice addition with Domino 14 and recent releases of the XPages JEE project is that now I can also use Java records with these interfaces. Records were introduced in Java 14 and are meant for a couple of reasons - for our needs, they cut down on boilerplate code, but they also have beneficial implications with multithreading.

In our example above, we might define the User class as:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

package com.example.api.model;

import jakarta.json.bind.annotation.JsonbProperty;

public class User {
	private String username;
	@JsonbProperty("first_name")
	private String firstName;
	@JsonbProperty("last_name")
	private String lastName;

	public String getUsername() {
		return this.username;
	}
	public void setUsername(String username) {
		this.username = username;
	}

	public String getFirstName() {
		return this.firstName;
	}
	public void setFirstName(String firstName) {
		this.firstName = firstName;
	}

	public String getLastName() {
		return this.lastName;
	}
	public void setLastName(String lastName) {
		this.lastName = lastName;
	}
}

This is a standard old Java bean class, and it's fine. Ideally, we'd add more to it - equals(...), hashCode(), and toString() in particular - and this version only has three fields. It'd keep growing dramatically the more we add: more fields, bean validation annotations, and so forth.

For cases like this, particularly where you likely won't be writing constructor code much (unless they add derivation syntax), records are perfect. In this case, we could translate the class to:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

package com.example.api.model;

import jakarta.json.bind.annotation.JsonbProperty;

public record User(
	String username,
	@JsonbProperty("first_name")
	String firstName,
	@JsonbProperty("last_name")
	String lastName
) {
}

This is dramatically smaller and will scale much better, lines-of-code-wise. And, since the important parts of the stack support it, it's nearly a drop-in replacement: JSON conversion knows about it, so it works in the MP Rest Client and in your own REST services, and Expression Language can bind to it, so it's usable in Pages and elsewhere.

Since records are read-only, they're not a universal replacement for bean-format classes (in particular, beans used in XPages forms), but they're extremely convenient when they fit and they're another good tool to have in your toolbelt.

This week and last, I uploaded a couple updates for some of my OpenNTF projects.

XPages Jakarta EE 3.2.0

First up is a new version of the XPages Jakarta EE project. This one is primarily about bug fixing and shoring up some of the 3.x line of features some more. Of particular note, it fixed a problem that crept in that caused the use of Concurrency to break the application, which was a bit of a problem.

WebFinger for Domino 2.0.0

About two years ago, I created the WebFinger for Domino project, primarily as a way to publish profile information in a way that works with Mastodon, though the result would work with ActivityPub services generally.

I'm fond of the goals of open formats like WebFinger and the related IndieWeb world. While Domino's licensing has been hostile to this world of personal sites since basically the 90s, it's still a good conceptual fit, and so I like to glom stuff like this on top of it when I can.

Anyway, I was reminded of this project and started thinking about ways to expand it beyond just its original use. For now, I mostly wanted to dust it off and make it extensible. The result is version 2.0.0, which refactors the code to allow for more stuff in the output. Along those lines, I added the ability to specify PGP public keys and have them included in the list, along with a Servlet to host them on your server. I plan to look around for other things people like to include in their profiles and add them in. I don't actually ever do much with PGP, but it's nice to have it in there.

The most common way that I expect people use Domino's CertMgr/certstore.nsf is to use Let's Encrypt with the default HTTP-based validation. This is very common in other products too and usually works great, but there are cases when it's not what you want. I hit two recently:

• My Domino servers are behind Traefik reverse proxies to blend them with other Docker-based services, and by default the HTTP challenge doesn't like when there's already an HTTP->HTTPS redirect in place
• I also have dev servers at home that aren't publicly-visible at all, and so can't participate in the HTTP flow

The first hasn't been trouble until recently, since the reverse proxy is fine, but now I want to have a proper certificate for other services like DRAPI. For the second, I've had a semi-manual process: my pfSense-based router uses its Acme Certificate plugin to do the dns-01 challenge (since it knows about my DNS provider) and then, every three months, I would export that certificate and put it into certstore.nsf.

Re-Enter CertMgr

Domino's CertMgr can handle those DNS challenges just fine, though, and the HCL-TECH-SOFTWARE/domino-cert-manager project on GitHub contains configuration documents for several common providers/protocols.

For historical reasons (namely: I didn't like Network Solutions in 2000), I use joker.com as my registrar, and they're not in the default list. Indeed, it seems like their support for this process is very much a "oh geez, everyone's asking us for this, so let's hack something together" sort of thing. Fortunately, the configuration docs are adaptable with formula (and other methods) - I'll spare you the troubleshooting details and get to the specifics.

DNS Provider Configuration

In certstore.nsf, the "DNS Configuration" view lets you create configuration documents for custom providers. Before I go further, I'll mention that I put the DXL of mine in OpenNTF's Snippets collection - certstore.nsf has a generic "Import DXL" action that lets you point to a file and import it, made for exactly this sort of situation.

Anyway, the meat of the config document happens on the "Operations" tab. This tab has a bunch of options for various lookup/query actions that different providers may need (say, for pre-request authorization flows), but we won't be using most of those here.

Operations

Our type here is "HTTP Request" - there are options to shell out to a command or run an agent if you need even more flexibility, but that type should handle most cases.

The "Status formula" field controls what Domino considers the success/failure state of the request. It contains a formula that will be run in the context of a consistent document used across each phase. If your provider responds with JSON, the JSON will be broken down into JSONPath-ish item names, as you can see in the HCL-provided examples. You can then use that to determine success or failure. Joker replies in a sparse human-readable text format, but does set the HTTP status code nicely, so I set this to ret_AddStatus.

The "DNS provider delay" field indicates how long the challenge check will wait after the "Add" operation, and that latency will depend on your specific provider. I did 20 seconds to be safe, and it's proven fine.

During development, setting "HTTP request tracing" to "Enabled" is helpful for learning how things go, and then "Write trace on error" is likely the best choice once things look good.

HTTP Lookup Request

For Joker, you can leave this whole section blank - its "API" doesn't support this and it's optional, so ignore it.

HTTP Add Request

This section is broken up into two parts, "Query request" and "Request". Set/leave the "Query request type" to "None" or blank, since it's not useful here.

Now we're back into the meat of the configuration. For "Request type", set it to "POST".

"URL formula" should be cfg_URL, which represents the URL configured above. Other providers may have URL permutations for different operations, but Joker has only the one.

Joker is very picky about the Content-Type header, so set the "Header formula" field to "Content-Type: application/x-www-form-urlencoded", which will include that constant string in the upload.

Things get a bit more complicated when it gets to the "Post data formula". For this, we want to match the example from Joker's example, but we also need to do a bit of processing based on the specific name you're asking for. Some DNS providers may want you to send a DNS key value like _acme-challenge.foo.example.com, while others (like Joker) want just _acme-challenge.foo. So we do a check here:

txtName := @If(@Ends(param_DnsTxtName; "."+cfg_DnsZone); @Left(param_DnsTxtName; "."+cfg_DnsZone); param_DnsTxtName);

"username=" + @UrlEncode("Domino"; cfg_UserName) + "&password=" + @UrlEncode("Domino"; cfg_Password) + "&zone=" + @UrlEncode("Domino"; cfg_DnsZone) + "&label=" + @UrlEncode("Domino";txtName) + "&type=TXT&value=" + @UrlEncode("Domino"; param_DnsTxtValue)

In my experience, this covers both single-host certificates and wildcard certificates.

HTTP Delete Request

This is for the cleanup step, so your DNS isn't littered with a bunch of useless TXT challenge records.

As before, make sure "Query request type" is "None" or blank.

Similarly, "Request type", "URL formula", and "Header formula" should all be the same as in the "Add" section.

Finally, the "Post data formula" is almost the same, but sets the value to nothing:

txtName := @If(@Ends(param_DnsTxtName; "."+cfg_DnsZone); @Left(param_DnsTxtName; "."+cfg_DnsZone); param_DnsTxtName);

"username=" + @UrlEncode("Domino"; cfg_UserName) + "&password=" + @UrlEncode("Domino"; cfg_Password) + "&zone=" + @UrlEncode("Domino"; cfg_DnsZone) + "&label=" + @UrlEncode("Domino";txtName) + "&type=TXT&value="

Putting It To Use

Once you have your generic provider configured, you can create a new Account document in the "DNS Providers" view.

In this document, set your "Registered domain" to your, uh, registered domain - in my case, "frostillic.us". This remains the case even if you want to register wildcard certificates for subdomains, like if I wanted "*.foo.frostillic.us". CertMgr will use this to match your request, and matches subdomain wildcards too.

There's a lot of room for special tokens and keys here, but Joker only needs three fields:

"DNS zone" is again your domain name.

"User name" is the user name you get when you go to your DNS configuration and enable Dynamic DNS - it's not your normal account name. This is a good separation, and a lot of other providers will likely have similar "don't use your normal account" stuff.

Similarly, "Password" is the Dynamic-DNS-specific password.

TLS Credentials

Your last step is to actually put in the certificate request. This stage is actually pretty much identical to the normal process, with the extra capability that you can now make use of wildcard certificates.

On this step, you can fill in your host name, servers with access, and then your ACME account. Even more than with the normal process, it's safest to start with "LetsEncryptStaging" instead of "LetsEncryptProduction" to avoid them temporarily banning you if you make too many requests.

With a custom provider, I recommend opening up a server console for your CertMgr server before you hit "Submit Request", since then you can see its progress as it goes. You can get potentially more info if you launch CertMgr as load certmgr -d for debug output. Anyway, with that open, you can click "Submit Request" and let it rip.

As it goes, you'll see a couple lines reading "CertMgr: Error parsing JSON Result" and so forth. This is normal and non-fatal - it comes from the default behavior of trying to parse the response as JSON and failing, but it should still put the unparsed response in the document. What you want is something at the end starting "CertMgr: Successfully processed ACME request", and for the document in certstore.nsf to get its nice little lock icon. If it fails, check the error message in the cert document as well as the document in the "DNS Trace Logs" view - that will contain logs of each step, and all of the contextual information written into the doc used by your formulas.

Wrapping Up

This process is, unfortunately, necessarily complicated - since each DNS provider does their own thing, there's not a one-config-fits-all option. But the nice thing is that, once it's configured, it should be good for a long while. You'll be able to generate certificates for non-public servers and wildcard at will, and that makes a lot of things a lot more flexible.

Back when Notes/Domino 14 came out, I made a post where I described some of the fallout of it. One of the entries was about the upstream removal of the "jvm/lib/ext" directory and the moving of all common extension JARs to the "ndext" directory. The upshot there was that any JARs that you want to add to the filesystem in Designer to match deployment on the server would have to be added to the active JRE in Designer in order to be recognized.

HCL presumably noticed this problem and altered the installation to accommodate it in FP1 and FP2. However, the approach they took is to add all JARs from ndext to the JVM. Thus, a fresh install+upgrade of Notes 14 to FP2 (or 14.5 EAP1) has a JVM that looks like this:

This is a problem in a couple ways, but the most immediate is that it includes the toxic "jsdk.jar" I warned about in the earlier post. This JAR contains primordial Servlet classes from the very first addition of Servlet to Domino, predating XPages, and that version lacks even the convenience methods added in the ancient-but-less-so version in XPages. To demonstrate this, you can write this code:

1
2

HttpServletRequest req = null; /* pretend this is assigned to something */
Map param = req.getParameterMap();

This will work in a clean Designer 14 installation but will break on upgrade to 14 FP2, with Designer complaining that the getParameterMap method does not exist. There are others like this too, but basically any "The method foo() is undefined..." error for Servlet classes is a sign of this.

The fix is to go into your JVM definition (Preferences - "Java" - "Installed JREs" - "jvm (default)" - "Edit...") and remove jsdk.jar. While you're in there, I recommend also removing POI and its related JARs (poi-*, xmlbeans, ooxml-schemas, fr.opensagres.poi.*, commons-*) too, unless you also happen to have deployed them to the server, since they're not normally present on Domino and are thus mostly there to lead you astray. Honestly, almost none of the JARs present in there by default are useful for the XPages JVM definition, since the critical ones are contributed via OSGi plugins. I guess guava.jar is important just because it's going to contaminate the server's JVM too, so you want to account for that. Otherwise, it's probably best to treat it like a 14 install and only add the new JARs you've explicitly added and deployed to the server.