A few years ago, I made a little project that hosts an SFTP server that stores documents in an NSF. I've used it here and there since then - as in the original post, I stashed some company docs in it to have them nicely synced among our Domino servers, and I've also had cases where clients use it to, for example, provide a way for their vendors to upload files in a standard way.

The other week, I decided to dive back into it to add some capabilities I'd wanted for a while, and the result is version 2.0.0. This version is a significant revamp that adds quite a bit.

Multiple Mounts

The first limitation I wanted to improve was the fact that the first version was restricted to a single NSF. That works fine in the basic case, but I wanted to start doing things like storing server config backups in there, and wouldn't necessarily want them in the same NSF as, say, company contracts, credentials, and secrets.

The way I went about this was to make it so that the new configuration NSF has a "Mounts" view that lets you specify a path in a conceptual top level of the filesystem that would then point to a target NSF. This allows the admin to do things like have separate ACLs for different mounts - since the client will act as an authenticated Domino user, these will be properly obeyed, and a user won't be able to access documents they don't have rights to.

Additionally, I could configure it so that not all mounts are present on all servers, which will come into play particularly with the next feature.

New Filesystem Types

Once I had a composite top-level filesystem, I realized that it wouldn't be terribly difficult to allow more filesystem types than the original NSF file store I made. That filesystem is built using the NIO File System Provider framework added in Java 7, and that system is designed to be pretty extensible. By default, Java comes with a few providers: the normal local filesystem as well as one that can treat a ZIP or JAR as a contained filesystem itself. These are accessed in a generic way, where you specify a URI and a map of "env" properties that are provider-specific.

For example, the ZIP filesystem takes a URI in the form of "jar:file:/some/path/to/file.zip" and an environment map configuring whether the filesystem should be created if it doesn't already exist in memory and what encoding to use for filenames (very important if you have Unicode characters in there).

So I added ways to configure a mount to the local server filesystem (similar to what the Mindoo FTP Server does) and then a generic configuration for any installed provider. It's probably uncommon that you will have a custom File System Provider implementation in your Java classpath, but hey, maybe you do, and I want to allow that to work.

I also added an extension point to the project itself that allows adding new providers via plugin.xml files in OSGi, and I can think of a couple other projects that may use this, like the NSF ODP Tooling.

WebContent Filesystem

Beyond adding the JVM-provided systems, I wrote another new filesystem type, one that provides access to the conceptual "WebContent" directory presented in Package Explorer in Designer:

The idea here is that this could be used to deploy, say, a JavaScript client application to an NSF without the developer or build server having to know anything about Domino. Pretty much everything can work with SFTP, so this makes accessing those files a lot easier. This is similar to the WebDAV capabilities Domino has had for a very long time, but with a different protocol.

Server Keypairs

In the first version, the app would generate and store the server's SSH keypair on the filesystem, in the data directory. This is fine, but part of the point of this whole project is that I like to get away from non-replicating stuff, and so I moved these keys to the configuration NSF. Now, on first connection, the server will look for a keypair document in the NSF and, if it doesn't exist, will generate a new one and store it there. Since I've been working with encrypted fields a lot for client work lately, I also realized that this was a good use for it: the public key is a normal text item (so you could distribute and verify it as needed), but the private key is encrypted with the generating server's ID file. Since only the server itself ever needs to know its private key, this works swimmingly.

JNX

This isn't a new app feature per se, but this was a good situation for me to put JNX to work in an open-source project. I had originally written this using the lotus.domino classes for most work and the IBM NAPI for things like generating sessions for a given username, but switching to JNX let me ditch both of those.

Admittedly, this is a case where switching to JNX didn't grant me significant new capabilities, but it DID let me do a couple things better. Some things are distinct feature improvements, like improving password authentication (previously, I was doing a compare of hashes "manually", which is fragile), while others are just making the code smoother, like no longer having to do the read-convert-recycle dance with DateTimes in LSXBE. It's just pleasant, and let me find a few places where the JNX API could be improved.

Future Additions

When I pick this project back up, there are certainly a couple things I'd like to add.

One would be to look into rsync support: rsync is tremendously useful for things like synchronizing filesystem-bound configs, but it's its own protocol tunneled over SSH, and so just having SFTP isn't enough. The underlying Apache Mina SSHD project is a general SSH server and not just SFTP, so it may be possible to do it by intercepting the commands sent over to initialize rsync, but it will be non-trivial. There's a library in Java that provides an rsync server, but it's GPL-licensed, and so I have to keep away for license-safety's sake.

Beyond that, it's mostly that I'd like to implement more filesystem types. Presenting data as a filesystem can be a very powerful tool: you could imagine providing access to documents in a DB as DXL or YAML, or listing files from a Document Library NSF, or (as I'd like to do some day) having the NSF ODP Tooling project replicate the ODP layout over SFTP.

For now, I'm looking forward to putting it to more use as a coordinating point. If I keep messing around with apps on TrueNAS, it'll give me a good feeling of security to have more info stashed in Domino and less prone to destruction if one server happens to blow up.

Yesterday, I put two releases up on OpenNTF, and I figure it'd be worth mentioning them here.

XPages Jakarta EE Support

The first is a new version of the XPages Jakarta EE Support project. As with the last few, this one is mostly iterative, focusing on consolidation and bug fixes, but it added a couple neat features.

The largest of those is the JPA support I blogged about the other week, where you can build on the JDBC support in XPages to add JPA entities. This is probably a limited-need thing, but it'd be pretty cool if put into practice. This will also pay off all the more down the line if I'm able to add in Jakarta Data support in future versions, which expands the Repository idiom currently in the NoSQL build I use to cover both NoSQL and RDBMS databases.

I also added the ability to specify a custom JsonbConfig object via CDI to customize the output of JSON in REST services. That is, if you have a service like this:

1
2
3
4
5

@GET
@Produces(MediaType.APPLICATION_JSON)
public SomeCustomObject get() {
	return findSomeObject();
}

In this case, the REST framework uses JSON-B to turn SomeCustomObject into JSON. The defaults are usually fine, but sometimes (either for personal preference or for migration needs) you'll want to customize it, particularly changing the behavior from using bean getters for properties to instead use object fields directly as Gson does.

I also expanded view support in NoSQL by adding a mechanism for querying views with full-text searches. This is done via the ViewQuery object that you can pass to a repository method. For example, you could have a repository like this:

1
2
3
4

public interface EmployeeRepository extends DominoRepository<Employee, String> {
	@ViewEntries("SomeView")
	Stream<Employee> listFromSomeView(Sorts sorts, ViewQuery query);
}

Then, you could perform a full-text query and retrieve only the matching entries:

1
2
3
4
5

Stream<Employee> result = repo.listFromSomeView(
	Sorts.sorts().asc("lastName"),
	ViewQuery.query()
		.ftSearch("Department = 'HR'", Collections.singleton(FTSearchOption.EXACT))
);

Down the line, I plan to add this capability for whole-DB queries, but (kind of counter-intuitively) that would get a bit fiddlier than doing it for views.

XPages Javadoc Provider

The second one is a new project, the XPages Javadoc Provider. This is a teeny-tiny project, though, not even containing any Java code. This is a plugin for either Designer or normal Eclipse and it provides Javadoc for some standard XPages classes - specifically, those covered in the official Javadoc for Designer and the XPages Extensibility APIs. This covers things like com.ibm.commons and the core stuff from com.ibm.xsp, but doesn't cover things like javax.faces.* or lotus.domino.

The way this works is that it uses Eclipse's Javadoc extension point to tell Designer/Eclipse that it can find Javadoc for a couple bundles via the hosted version, really just linking the IDE to the public HTML. I went this route (as opposed to embedding the Javadoc in the plugin) because the docs don't explicitly say they're redistributable, so I have to treat them as not. Interestingly, the docs are actually still hosted at public.dhe.ibm.com. If HCL publishes them on their site or makes them officially redistributable, I'll be able to update the project, but for now it's relying on nobody at IBM remembering that they're up there.

In any event, it's not a huge deal, but it's actually kind of nice. Being able to have Javadoc for things like XspLibrary removes a bit of the guesswork in using the API and makes the experience feel just a bit better.

One of my main goals in the slow-moving OpenNTF home-page revamp project I'm doing (which I recently moved to a public repo, by the way) is to, like on this blog, keep things extremely simple. There's almost no JavaScript - just Hotwire Turbo so far - and the UI is done with very-low-key JSP pages and tags.

The Library

This also extends to the server side, where I use the XPages Jakarta EE project as the baseline but otherwise don't yet have any other dependencies to distribute. I have had a few Java dependencies kept as JARs within the project, though, and they're always a bit annoying. The other day, when Designer died again trying to sync one of them to the ODP, I decided to try to remove PrettyTime in favor of something slimmer.

PrettyTime is a handy little library that does the job of turning a moment in time into something human-friendly relative to the current time. OpenNTF uses this for, for example, the list of recent releases, where it'll say things like "19 hours ago" or "5 months ago". Technically the same information as if it showed the raw date, but it's a little nicer. It's also svelte, with the JAR coming in at about 160k. Still, nice as it is, it's a binary blob and a third-party dependency, so it was on the chopping block.

The Strategy

My antipathy for using JavaScript isn't so much about an objection to the language itself but to the sort of bloated monstosities it gives rise to. Using it in the "old" way - progressive enhancement - is great. Same goes for Web Components: I think they're the right tool a lot less frequently than a lot of JavaScript UI toolkits do, but they have their place, and this is one such place.

What I want is a way to send a "raw" ISO time value to the client and have it actually display something nice. Conveniently, just the other week, Stephan Wissel tipped me off to the existence of the Intl object in JavaScript, which handles the fiddly business of time formating, pluralization, and other locale-related needs on behalf of the user. Using this, I could write code that translates an ISO date into something friendly without also then being on the hook for coming up with translations for any future non-English languages that the OpenNTF app may support.

The Component

In the original form, when I was using PrettyTime, the code in JSP to emit relative times tended to look like this:

1

<c:out value="${temporalBean.timeAgo(release.releaseDate)}"/>

I probably could have turned that into a JSP tag like <t:timeAgo value="${release.releaseDate}"/>, but it was already svelte enough in the normal case. Either way, this already looks very close to what it would be with a web component, just happening server-side instead of on the client.

As it happens, Web Components aren't actually terribly difficult to write. A lot of JS toolkits optimize for the complex case - Shadow DOM and all that - but something like this can be readily written by hand. It can start with some normal JavaScript (since I'm writing this in Designer, I made this a File resource, since then the JS editor doesn't die on modern syntax):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

class TimeAgo extends HTMLElement {
	constructor() {
		super();
	}

	connectedCallback() {
		/* Do the formatting */
	}
}

customElements.define("time-ago", TimeAgo);

Put that in a .js file included in the page and then you can use <time-ago/> at will! It won't do anything yet, but it'll be legal.

While the Intl library will help with this task, it doesn't inherently have all the same functionality as PrettyTime. Fortunately, there's a pretty reasonable idiom that basically everybody who has sought to solve this problem has ended up on. Plugging that into our component, we can get this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

class TimeAgo extends HTMLElement {
	static units = {
		year: 24 * 60 * 60 * 1000 * 365,
		month: 24 * 60 * 60 * 1000 * 365 / 12,
		day: 24 * 60 * 60 * 1000,
		hour: 60 * 60 * 1000,
		minute: 60 * 1000,
		second: 1000
	};
	static relativeFormat = new Intl.RelativeTimeFormat(document.documentElement.lang);
	static dateTimeFormat = new Intl.DateTimeFormat(document.documentElement.lang, { dateStyle: 'short', timeStyle: 'short' });
	static dateFormat = new Intl.DateTimeFormat(document.documentElement.lang, { dateStyle: 'medium' });

	constructor() {
		super();
	}

	connectedCallback() {
		let date = new Date(this.getAttribute("value"));

		this.innerText = this._relativize(date);
		if (this.getAttribute("value").indexOf("T") > -1) {
			this.title = TimeAgo.dateTimeFormat.format(date);
		} else {
			this.title = TimeAgo.dateFormat.format(date)
		}
	}

	_relativize(d1) {
		var elapsed = d1 - new Date();

		for (var u in TimeAgo.units) {
			if (Math.abs(elapsed) > TimeAgo.units[u] || u == 'second') {
				return TimeAgo.relativeFormat.format(Math.round(elapsed / TimeAgo.units[u]), u)
			}
		}
		return TimeAgo.dateTimeFormat.format(d1);
	}
}

customElements.define("time-ago", TimeAgo);

The document.documentElement.lang bit there means that it'll hew to being the professed language of the page - that could also use the default language of the user, but it'd probably be disconcerting if only the times were in one's native language.

With that in place, I could replace the server-side version with the Web Component:

1

<time-ago value="${release.releaseDate}" />

When that loads on the page, the browser will display basically the same thing as before, but the client side is doing the lifting here.

It's not perfect yet, though. Ideally, I'd want this to extend a standard element like <time/>, so you'd be able to write like <time is="time-ago" datetime="${release.releaseDate}">${release.releaseDate}</span> - that way, if the browser doesn't have JavaScript enabled or something goes awry, there'd at least be an ISO date visible on the page. That'd be the real progressive-enhancement path. When I tried that route, I wasn't able to properly remove the original text from the DOM like I'd like, but that's presumably something that I could do with some more investigation.

In the mean time, I'm pretty pleased with the switch. One fewer binary blob in the NSF, a bit of new knowledge of a browser technology, and the app's code remains clean and meaningful. I'll take it.

Keeping with my productive week off, today I release version 2.9.0 of the XPages Jakarta EE Support project. Similar to the previous release, this one contains new features primarily related to Jakarta NoSQL, but also has some improvements for JSF and a bunch of bug fixes and compatibility improvements.

Jakarta NoSQL

The improvements to the JNoSQL driver come from some needs I came across when moving older lotus.domino/ODA-based code to using JNoSQL repositories. In particular, I added the remaining applicable view entry properties as available fields to map, added better support for reading note IDs, and fetching documents by note ID.

JSF

While JSF support remains limited by not having a proper way to add in third-party component libraries like PrimeFaces, it's still a potentially-compelling tool in an NSF as an alternative to XPages in some cases. Accordingly, I fixed a few bugs I had run into when loading pages after modifying the NSF design. Additionally, I fixed up support for JSF as an MVC view engine. It now properly joins JSP as a mechanism for rendering your output with an MVC structure, and I think there's some real potential there.

Bug Fixes and Compatibility

Most of the other closed issues deal with a few bugs here and there, and in particular involve some improvements for running apps in XPiNC and on a server with Domino Leap also installed. I don't use XPiNC anymore and haven't tried Leap, so I greatly appreciate bug reports specific to these and the assistance in tracking down the trouble.

The Future and Next Steps

I'm pondering now what the next release of the project will focus on. I have no shortage of feature ideas, and there are a few potentially-disruptive changes I'd like to make.

Unfortunately, those changes will be largely confined to improving the support for the specs that are already present and not advancing to new versions. The predicted Java-version wall arrived: Jakarta EE 10 is out and requires Java 11 and above. Since Domino remains mired in Java 8, that means that new versions of the specs and implementations are hard-incompatible until that changes.

On the plus side, there's still a lot of improvements I can make with Jakarta EE 9 as the baseline.

Reorganization

One big one I've been thinking about is a reorganization of the individual libraries that make up the project. The way it's been designed, almost every spec has its own Equinox Feature and XPage Library to go with it. This was fine early on when it was just CDI, EL, and JAX-RS, but it's grown annoying: installing the project in Designer is a seemingly-endless process of approving each plug-in at a time and the list of libraries to check in Xsp Properties is interminable. More critically, being able to selectively turn on and off specs like this doesn't make sense anymore. CDI has grown so important to Jakarta EE in general and this spec in particular that it doesn't make sense to not have it present if you're going to use this project at all. It's a foundational component of so many other parts and is essentially The Way to do Jakarta-based development.

So I'm thinking I'm going to reorganize the projects into fewer features and libraries, which will be a breaking change that will necessitate a bump to 3.0 - fortunately, the numbers line up well for that. I have a few potential options here:

1. Just lump them all into one. You'd have basically one big switch to say "this is a JEE project" and everything would be on. The virtue here is that this is how I already work and is essentially the recommended way to do things. Additionally, as far as I know, while having additional components may slow first load (though not as much as other parts), I don't think they have a significant impact if enabled but unused during runtime.
2. Try to line the specs up with one of the existing Jakarta Profiles. Those profiles are meant to be curated selections of useful specs, and this project has enough to implement what in newer versions is deemed the Core Profile. The trouble with this, though, is that the Core Profile is very much geared to be the shared subset with MicroProfile and similar and is a bit thin for Domino's monolith-focused development style. The Web and Full profiles, on the other hand, require "traditional" APIs like EJB that are not present in this project.
3. Break them apart into my own "core" and "optional" features. For example, it doesn't make sense to use this without JAX-RS, CDI, and Bean Validation enabled, but JSF is entirely independent of the other specs and is among the least likely to be used in practice for now. This would also allow me to establish a running flow where "experimental" features start out as optional add-ins and then eventually make their way to core.

I'm currently waffling between #1 and #3, with a slight lean towards #1. If I can be sure that either everything or nothing is present, I could get rid of some weird hedges and workarounds, like how the JAX-RS implementation doesn't "officially" know about the CDI library yet references CDI classes explicitly by name.

New Application Types

Currently, to use this project, you can either put your code into an NSF and use the automatic behavior of the libraries or you can put your code in OSGi-based webapps or Servlets and then manually manage integration with these specs.

Both of these are limited by their reliance on the many assumptions IBM built in to how these apps should work. In-NSF apps require that all Jakarta code come from a request including "xsp" in the URL or to a file ending in ".jsp", ".jsf", or ".xhtml". If you're writing, say, an MVC-based app, all of your URLs are going to have to start with something like "foo.nsf/xsp/app/...", which is okay but ugly. Additionally, the way these apps are implemented - NSFComponentModule - severely limits my hooks for listening for things like application and session expiration, which hampers CDI's lifecycle handling a bit.

For a good while, I've pondered the notion of adding another ComponentModule type to handle the case where you want to go all-in on Jakarta EE. With this idea, the new module implementation would have full control over incoming requests, allowing URLs without the xsp/app bit in there, and would have better handling of lifecycles. In this way, I could make it so that your could would look more like (or be identical to) a "normal" .war-based webapp, with fewer workarounds for the existing XPages stuff. This would also allow me to do things like lessen the amount of Servlet 2.5-to-5.0 bridging and could assist tremendously in improving JSF support.

Along similar lines, I've been considering doing something similar for OSGi-based webapps, and I've made some progress along those lines in a feature branch. The idea here would be to do something similar to how you can deploy web.xml-based webapps via OSGi now, but with built-in support for Jakarta EE 9 features (with web.xml then being optional). With this setup, you'd be able to write an app that does an Import-Package for the various jakarta.* packages you want and add a bit in your MANIFEST.MF to signal to this project that it should participate. This could either be a variant of the extension point used by the existing OSGi webapp support or using the Web-ContextPath directive from the OSGi spec. One of the goals here would be to make it so that you would be able to write a Jakarta EE 9 app using normal development tools - Eclipse/IntelliJ/VS Code, Maven, etc. - and then just use maven-bundle-plugin to add the OSGi info you need without having any specific dependencies on Domino bits, especially the nightmare of depending on the non-redistributable XPages OSGi artifacts.

Other Options

And, in the mean time, I have a bunch of other tasks I could work on. Slowly converting my client project to Jakarta NoSQL instead of direct ODA use has turned up a whole slew of things that would be useful to add (for example, stampAll support), so I can slowly burn down that feature-request list.

There's also the notion of documentation! While a lot of the behavior of this project is in theory documented by virtue of the upstream specs and the general world of Jakarta blogs, videos, and courses, there's enough to know about the specifics of the interactions with Domino that more documentation is in order. Historically, I've just done this by expanding the README, but it's gotten pretty unwieldy at this point. It would probably make sense to break the specifics and examples out into at least wiki pages, if not a format that can be built into a PDF/etc. and included in the distribution.

So yep, I'll have my hands busy with this thing for a good while more, I figure.

The other day, I talked about some changes/workarounds for Notes/Domino 12.0.2. Today, I made a few updates to some of the open-source projects I maintain, including another update to the generate-domino-update-site Maven plugin.

Domino Update Site Generator

In the 4.2.0 release, I added code to (mostly, as it turns out) account for HCL moving the NAPI implementation JAR down to jvm/lib/ext. In subsequent use, I found that, while that will suffice for building applications that use the OSGi dependencies, it didn't work for launching applications using it as a baseline - namely, the NSF ODP Tooling Maven plugin.

Today, I released a 4.2.1 version that improves this behavior by re-adding dependencies in the implementation bundle.

I also created a project page for it on OpenNTF. Though the project has always been hosted at the OpenNTF org on GitHub, I hadn't created a project page for it due to it just being a standalone Maven plugin. I figured it'd be useful to create an official page there for it, though.

NSF ODP Tooling

Speaking of the NSF ODP Tooling, I also found that local operations once again started crashing on macOS. Due to changes in macOS and the very weird ways that Notes works, local operations on there are a very moving target, and I have to do a lot of work in the project to account for changes to the embedded JVM and whether specific Notes versions work better with HotSpot or OpenJ9 JVMs.

Long story short, I release 3.10.0 today to account for this. Though I've found that the spawned JVM will still sometimes crash, it's after completing its work, so I considered that fine for now.

OpenNTF Domino API

Finally, I came to the OpenNTF Domino API. This project has admittedly been neglected for a little while: I'm the only active maintainer, and the client project I use it in targets Domino 11.0.1, so the 12.x builds have remained in an incomplete state for a while.

With the release of 12.0.2, I decided I should finish the wrappers for the new classes added in 12.x, so I did so and uploaded a build for 12.0.2. This primarily adds those wrappers, but also included a contributed fix and changes the distribution packaging to combine the XSP and non-XSP versions.

Earlier today, I presented for OpenNTF's webinar series on the topic of the XPages Jakarta EE Support project. I felt like it went pretty well, and the video and slides are up now.

Video (great thumbnail choice on this one, YouTube):

Slides (kindly ignore the mangling that happened in the Keynote -> PPT -> Slideshare transition):

This Thursday (two days from now), I'll be presenting for OpenNTF's webinar series on the topic of the XPages Jakarta EE Support project. From our summary:

The XPages Jakarta EE Support project on OpenNTF adds an array of modern capabilities to NSF-based Java development. These improvements can be used for wholly-new applications or added incrementally to existing ones.

In this webinar Jesse Gallagher will demonstrate how to use this project to perform common tasks in better ways, such as creating and consuming REST services, writing managed beans with CDI, and using new EL features in XPages. Though these examples will largely use Java, they do not require any knowledge of OSGi or extension library development, nor any tools other than Designer.

This webinar will take place on August 18, 2022 at 11:00AM (New York Time) to 12:30PM.

Register for this webinar at: https://register.gotowebinar.com/register/6878765070462193675

My intent for this is to show the most-common components used with some examples of how I'm using them in practice. I hope it will also be an opportunity for anyone who (reasonably) balks at the opaque monolith to ask questions and get a better idea for whether it'd be helpful for them.

This month contains a few presentations and sessions of note for me, and I realized I should compile a list.

To start out with, I'll be the second presenter at next week's OpenNTF webinar, which will be about the Domino One-Touch Setup capability from V12 onward. The first leg of the presentation will feature Roberto Boccadoro covering its use in the standard case of easing the lives of administrators, while my section will cover more developer-centric needs, particularly my use of it as a tool for repeatable integration-test suites of Domino code. This webinar will take place next week, on May 19th, and you can register for it at https://attendee.gotowebinar.com/register/2937214894267353356.

While I won't personally be attending Engage this year, it's shaping up to be a good conference and OpenNTF and Jakarta EE will be well represented (in chronological order):

• Ro05. Happy Birthday, OpenNTF! Now What? - on Tuesday, Serdar Basegmez (and possibly other OpenNTF directors) will run our traditional community roundtable, where you can hear about what OpenNTF has been up to and provide ideas about where you'd like us to go.
• De20. Domino apps CI with Docker - also on Tuesday, Martin Pradny will discuss the use of the NSF ODP Tooling project within a Docker container to automate NSF building within a CI infrastructure in a smooth and reliable way.
• De17. Domino + Jakarta EE = AppDev In Heaven - on Wednesday morning, Daniele Vistalli will cover the XPages Jakarta EE Support project, going over the myriad Jakarta and MicroProfile specifications that it brings to Domino development and showing how you can bring them to bear to fix a lot of long-standing Domino dev limitations.

If you're attending Engage, I definitely suggest you check out all of those.

For the last few weeks, I had a minor flurry of work in a couple of the open-source projects I maintain, and I figured this would be as good a time as any to give an overview of my active work in these projects and how they relate.

Overview

I had a few minor contributions and picked-up projects through the year, but most of my currently-public work went towards four main projects:

• Domino Open Liberty Runtime
• NSF ODP Tooling
• XPages Jakarta EE Support
• XPages Runtime (to a lesser extent for now)

I do find it interesting to consider how these relate. Some aspects are easy: they're all Domino-related for sure, and they all at one time or another have played a significant infrastructural role in my client work. Beyond that, though, they form a nebulous message: though I don't know for sure what to do with all the XSP markup we have, I know it can't be the status quo and I'm fairly confident that Jakarta EE is the best route forward.

Domino Open Liberty Runtime

This project allows you to run instances of Open Liberty as a spawned process from Domino, which in turn means both that you can readily(-ish) access Domino data and also that you can deploy these apps in an NSF-based way to your servers, without having to have particular mastery of Liberty administration as such.

The big-ticket news this year was my addition of a Domino-hosted reverse proxy and arbitrary JVM selection. With these additions, the project ended up being a particularly-compelling way to glom modern apps on to Domino without even necessarily worrying about pointing to a different port. I also added in the standalone proxy to both the apps and Domino - which would gain you Web Sockets and HTTP/2 - which is another nice way to get better app toolkits without having to bother an admin.

XPages Jakarta EE Support

This one saw a burst of activity in just this past month. For a while, it had sat receiving only minor tweaks: I use it for EL, CDI, and JAX-RS in my client project, and the changes I made were generally just to add features or fix bugs needed there.

This month saw the big switch from Java/Jakarta EE 8 (javax.* packages) to Jakarta EE 9 (jakarta.* packages). This was a very-interesting prospect: though it on paper just involved switching class names around, it necessitated adding some Servlet 5 shims around Domino's irreponsibly-old Servlet 2.4/2.5 hybrid layer. While this didn't bring full Servlet 5 features, it does mean I'm suddenly much less bound by the strictures of the older version: a lot of Servlet-based software casually depends on at least 3 even for just convenience methods (like getting a ServletContext from a ServletRequest).

I also took the opportunity to go back and add some features I've long wanted - JSP and MVC - to the NSF side. These have less immediate call in my client work (which primarily involves additions on the OSGi servlet layer and less in the NSF), but suddenly created a surprisingly-compelling update to in-NSF development. It's stymied by, naturally, a lack of support in Designer, but the idea of writing something that approaches a true modern Jakarta app inside an NSF is intriguing indeed.

NSF ODP Tooling

The NSF ODP Tooling has proven to be my workhorse. The ODP-to-NSF compilation alone has saved me countless hours from the previous laborious task of syncing two dozen NSFs with their ODPs and the fault-prone process of trying to get clean NTF copies of them for each build. Now, the former is done with a single script I can run in the background and the latter happens automatically every single push to our Git repository. Delicious.

It also provides an invaluable part of my normal development process for this client. Alongside the next project, it lets me do my XPages development outside of Designer, meaning I only need to schlep my way back to that IDE to look at legacy elements in context or to troubleshoot something with the Notes or Domino OSGi view of the world.

The work in this project this year has primarily been around edge cases, bug fixes, and scrambling through the rocky shoals of the ever-changing macOS Notes client. It's been a tough time here and there: certain parts of the NSF that I use less frequently have their own edge-case needs (like SSJS sort of existing in two places and the CD storage being surprisingly difficult to work with. I also had some fun combat with filesystems and charsets, which was fortunately even-more enlightening than it was annoying.

XPages Runtime

The XPages Runtime project admittedly had a slow year, but it's nonetheless a critical component in my CI/CD workflow, and gets periodic fixes for trouble I run into. The good news there is that it generally does what it promises: I run XPages outside of Domino constantly with this thing. Though it still requires more coordination on the app side than I'd like, it's gradually approaching a state where it feels like a peer to other server-side toolkits that one can bring into a WAR file, and that's nice.

It will likely have some work coming up in the near future, though: if I'm to move my client's app over to the jakarta.* namespace, that will require at least some level of cooperation with this project. While I can't change the source of XPages to accept these coordinates itself, it should be doable to do much like what I did with the XPages Jakarta EE Support project and use my shims to translate back and forth between old and new classes. The main difference here will be that the surrounding container will speak the new form natively, but that should be fine.

I expect a certain amount of annoying trouble with things like XPages-internal expectations about JAX-B and JavaMail, and it's certainly possible that such dependencies will end up proving to be debilitating, but I'm optimistic. If I'm successful, it'll be one more way that I'm crafting a whole workflow where modern technologies are the primary target and XPages can remain a component in the lineup.

Miscellaneous Grab Bag

Beyond those big ones, I had a handful of other contributions here and there. I'm sure there were a few others, but I'll close on two that I found pleasing.

The other week, I got a Pull Request merged into the Eclipse Krazo project - while not a huge deal, it does always give me a little thrill when my code goes into a project where I'm not the primary or sole contributor.

I also adopted POI4XPages, which was for more-practical reasons. I've used POI4XPages for a couple clients for a while, but it was certainly showing its age (sitting at 3.x since 2017). Moreover, Notes 11's corruption of its classpath with POI 4.x made working with it annoying beyond just being out-of-date and lacking some breaking changes in the mean time. Since I had moved one of my clients to POI 5.0 a bit ago, I decided to break that code out and adapt it into POI4XPages. Then, of course, along came Log4Shell and I scrambled out three subsequent patch versions just to update log4j. So it goes.

When I last talked about my Domino Open Liberty Runtime project, I mentioned the various main tasks I'd been doing in gearing up for a 3.0 release.

Well, it's been a while since then, but so it goes with open-source projects sometimes. Fortunately, I've had some time here and there to dust it off some more and take care of a lot of lingering tasks I had assigned to the 3.0 release, enough to make it to the finish line. Some of these I found interesting to note, and so I'll note them here.

Java Runtime Shifts

One of the neat tricks the project does is to auto-download a specified Java runtime for you: you can pick your version (e.g. 8, 11, or 17) and your flavor (HotSpot, OpenJ9, or GraalVM CD) and the tooling will do the work of downloading it on your behalf.

Originally, this used just AdoptOpenJDK, the previous go-to home of open-source Java builds. When I added GraalVM, I generalized the code to handle those downloads as well. Like AdoptOpenJDK's binaries, they're hosted on GitHub, but under a different organization and using a different naming scheme.

That refactoring paid off now: AdoptOpenJDK moved to the Eclipse Foundation and became the Adoptium project: much the same idea, but a new home. During that move, though, IBM stopped pushing OpenJ9 builds to that organization and instead now host them themselves, under the name Semeru. I assume this was all due to some wrangling over the use of "JDK" and other political scuffling, but either way I had to deal with it.

Fortunately, though the Semeru home page is standard IBM fare, the binaries themselves are still hosted on GitHub, and the organization still mirrors the original AdoptOpenJDK/Adoptium style. So I made another class for that, but the logic is largely the same.

Notes API Feature

Since it's extremely likely that apps deployed adjacent to Domino this way will want to access Domino data, I originally set up deployment to copy the server's Notes.jar into the global "extra JARs" directory for deployed Liberty instances. This would allow apps to make use of lotus.domino classes without having to package Notes.jar inside the WAR.

This works well enough, but it always felt a little wrong: I don't like polluting the server's classpath that way, and it got all the worse because I needed to also bring in an open-source copy of CORBA classes for cases where the JVM in use is newer than 8.

So I decided to take another swing at it, this time packaging it up as a feature to be loaded as desired in Liberty. The project already has a few of these, generally consisting of the Liberty-specific OSGi bundle, the ESA feature file to define it in Liberty, and then an extension on Domino to deploy that to new servers.

In this case, I'd want to do it a little differently: the other ones included custom code on my part, but this one will just want to grab JARs from the Domino installation and make them available to apps. So, for this one, I didn't need the first two components - instead, I'd build the feature JARs on the fly. I went about doing that, the the result is a class that creates a notesApi-1.0 feature automatically. It makes use of the way streams work in Java to automatically build a feature container based on the current Domino build version (in case the API changes), creating the necessary manifests programmatically and then locating Notes.jar, the NAPI, and their CORBA and IBM Commons dependencies.

This new route is nicer all around: it's explicitly opt-in, which I like, it now also provides the NAPI, and it hides the non-API dependencies from the apps.

Generic Tooling

For a while now, I've been toying with the idea of making this less Open-Liberty-specific and adding the ability to deploy different kinds of apps. After all, all Liberty is here is a forked process, and that could be anything: a different app server, a single JAR file, or something not Java-related at all. Though the tooling will still only support Open Liberty in 3.0, I've been gradually reorganizing the structure to make such a change practical to implement in, say, 4.0.

Rather than the core runtime assuming it's using Liberty, now it uses generic interfaces to represent any kind of server, and then that further drills down into servers that use a JVM, and from there down to a Liberty server.

This sort of genericizing has made the runtime much more service-based internally - it's more indirect and it takes more discipline, but I feel good about it. I picture it now as a central core coordinator (still named OpenLibertyRuntime internally) receiving notifications of changes in the admin NSF and then sending out appropriate messages to the various deployment services. The fact that there's only one such deployment service currently doesn't diminish the splendidness of my mental model.

Reverse Proxy

The reverse proxy implementations that I'd mentioned before haven't really changed much since I introduced them, but that's, I think, a good sign: they do what they're supposed to and don't necessarily need dramatic changes. The Domino-side reverse proxy - the one that lets your Liberty apps look like they're running inside Domino's normal HTTP server - has risen in my estimation since I made it, since it makes for a very-reasonable story. It's essentially like if you use the Equinox extension points to deploy a servlet or web application, except you trade the ability to run within a database context for a much-more-modern development environment.

Release

The full list of closed issues for this release is available on GitHub, and I've uploaded the distribution to OpenNTF.

Earlier today, I took part in OpenNTF's May webinar on recent project updates. During that, I gave a quick overview of several of the projects I've worked on in the past year, and I figured it'd be useful to put together some followup notes for future reference.

Project Links

I didn't include any actual links to the projects, which could be useful:

• OpenNTF Domino API
• NSF ODP Tooling
• XPages Bazaar
• NSF Router
• Import and Export for Designer

Slides

To begin with, I posted my slides on SlideShare:

NSF ODP Tooling

I gave a brief overview of the NSF ODP Tooling, but, if anyone is interested in a more in-depth overview, my presentation from CollabSphere last year should cover it:

Import and Export for Designer

My quick mention of this refreshed project focused mainly on importing components, but generating new ones for upload is a pretty-critical part of it too. The tool comes with an option for that, which (to the credit of the original creators) does a splendid job packaging your controls and applying a license. The PDF from the project page goes over that in detail. Once the components are packaged up, you can upload them as a release on OpenNTF and I'll figure out how to add it to the download list.

In my last post, I mentioned that I decided to write the File Store app as a Jakarta EE application running in a web server alongside Domino, instead of as an app running on the server itself. In the comments, Fredrik Norling asked the natural question of whether it'd have been difficult to run this on the server, which in turn implies a lot of questions about deployment, toolkits, and other aspects.

Why Not

My immediate answer is that it would certainly be doable to run this on Domino, but the targeted nature of the app meant that I had some leeway in how I structured it. There are a couple reasons why I went this route, but most of them just boil down to not having to deal with all the gotchas, big and small, of doing development on top of Domino.

As a minor example, I wanted to add some configuration parameters to the app, and for this I used MicroProfile Config. MicroProfile Config is a small spec that standardizes the process of doing key-value-based configuration, allowing me to just say that I want a named value and let the runtime pick it up from the environment, system properties, or a config file as necessary. It wouldn't be difficult to write a configuration checker, but why bother reinventing the wheel when it's right there?

Same goes for having the SFTP server load at startup and be running consistently. If I did this on Domino, I could either put it in an NSF-based XPages app with an ApplicationListener and depend on the preload notes.ini parameter, or I could go my usual route and use an HttpService that manages the lifecycle. Neither route is difficult either, but they're both weirder and more fiddly than using servlet context listeners in a normal JEE app.

And then there's just the death by a thousand cuts: needing to use Tycho to build, having to deal with Eclipse Target Platforms, making sure anything using reflection is wrapped in an AccessController block to avoid Java policy issues, the nightmare of dependency management, the requirement to use either Designer or the okay-but-still-cumbersome Domino HTTP restart development cycle, and on and on. With a JEE app, all of those problems disappear into thin air.

The Main Hurdle

All that said, there's still the hurdle of actually implementing Notes native API access outside of Domino. At its core, what I'm doing is the same as what has been possible for years and years, initializing a Notes/Domino runtime in a secondary process. The setup for this varies platform by platform but generally involves either setting a couple environment variables for your process or (as is the case with the Domino Open Liberty Runtime) spawning the process from Domino itself.

Beyond setting up your process's environment, there's also the matter of initializing each thread of your app. On Domino, all threads are inherently NotesThreads, but outside of that there's some specific management to be done. You can call NotesThread.sinitThread() and NotesThread.stermThread() manually or run your code in specifically-spawned NotesThreads. I largely do the latter, making use of an ExecutorService to handle maintaining a thread pool for me. I then added some supporting code on top of that to let me run blocks of code as an arbitrary named user while retaining a cached set of sessions. That part wouldn't really be necessary if you're writing an app that doesn't need to act as different user names, but it's handy for something like this, where incoming connections must run as a user to enforce security fields properly.

Philosophical Advantages

Beyond my desire to avoid hassles and get to use modern Jakarta EE goodies, I think there are also some more philosophical advantages to writing applications this way, and specifically advantages that line up with some of HCL's stated long-term goals as well. Domino has long been a monolith, and that has largely served it well, but keeping everything from the DB all the way up through to the app-dev stack in the same bag means you're often constrained in your toolkits and deployment choices. By moving things just outside of the main Domino tower, you're freed up to use different languages and techniques that don't have to be integrated and maintained in the core. This could be a much larger jump than I'm doing here, and that's just what HCL has been pushing with the AppDev Pack and the associated Node.js domino-db module.

I think it's beneficial to picture Domino more as a dominant central core with ancillary servers and apps running just adjacent to it - not a full-on Microservices architecture, but just a little decentralized to keep areas of concern nice and separate. Done well, this setup is a lot more flexible and fault-tolerant, while still being fairly straightforward and performant. It's also a perfect match for a project like this that's geared towards implementing a new protocol - it doesn't even have to worry about HTTP SSO or reverse proxies yet.

So I think that this is where things are heading anyway, and it's just a nice cherry on top that it also happens to be a much (much, much) more pleasant way to write Java apps than OSGi plugins.

When I Know Some Guys kicked off, we bought a couple of Transporter devices to handle our file-syncing needs without having to rely on Dropbox or another hosted service. Unfortunately, Nexsan killed off Transporters a couple years back, and, though they still kind of work, it's been a back-burnered project for us to find a good replacement.

Ideally, we'd find something that would handle syncing data from our various locations transparently while also allowing for normal file access through some common protocol. Aside from the various hosted commercial services, there are various software packages you can run locally, like OwnCloud and NextCloud. I even got a Raspberry Pi with a USB hard drive to tinker with those, though I never got around to actually doing so.

Yesterday, though, I realized that we already have a fleet of privately-owned servers that replicate seamlessly in the form of our Domino domain. They also, conveniently, have nice capabilities for blob storage, shared user authentication, and fine-grained access control. What they didn't have, though, was any good form of file protocol. I'm pretty sure that Domino still has WebDAV built-in, but that's just for design elements. Years ago, Stephan Wissel created a project that works with file attachments, but that didn't cover all the bases I wanted and I didn't want to adopt the code base to extend it myself. There's also Karsten Lehmann's Mindoo FTP Server from around the same time, but that was non-SSH FTP and targeted at the local filesystem.

So that meant it was time for a new project!

The Plan

I initially looked at WebDAV, since it's commonly supported, but it's also very long in the tooth, and that has led to all of the projects implementing that being pretty old and cumbersome as well.

Then, I found the Apache Mina project, which implements a number of server protocols, SSH included, and is actively maintained. Looking into how its SFTP support works, I found that it's shockingly simple and well-designed. All of the filesystem access is based on the Java NIO packages added in Java 7, which is a pluggable system for making arbitrary filesystems.

Using SFTP and SCP means that it'll work with common tools like Transmit and - critically - rsync. That means that, even in the absence of an custom app like Dropbox, mobile access and syncing with a local filesystem come "for free".

The Project

So out of this was born a new project, NSF File Server. Thanks to how good Mina is, I was able to get a NIO filesystem implementation and SFTP+SCP server up and running in very little time:

In its current form, there aren't a lot of tricks: the files are stored as attachments to normal documents in a "filestore.nsf" database with two views, which allow for directory-contents and individual-file lookup while also being pretty self-explanatory to a Notes client. I have some ideas about other ways to structure this, but there's an advantage to having it be pretty basic:

Similarly simple are the authentication mechanisms, which allow for both password and public-key authentication based on the HTTPPassword and sshPublicKey fields in a person document, respectively (and maybe via LDAP in directory assistance? I never remember the mechanics of @NameLookup).

The App

Because this is a scratch-our-itch project and I'm personally tired of dealing with Domino's OSGi environment, the app itself is implemented as a WAR file, expected to be deployed to a modern Jakarta EE server like my precious Open Liberty. Conveniently, I have just the project for that as well, making deploying NSF-accessing WARs to Domino a bit more reasonable.

For now, the app is faceless: the only "web app" bits are some listeners that initialize the Notes environment and then spawn the SSH server. I plan to add at least a basic web UI, though.

Future Plans

My immediate plan is to kick the tires on this enough to get it to a point where it can serve as its original goal of a syncing SFTP server. I do have other potential ideas in mind for the future, though, if I feel so inspired. Most of my current logged issues are optional enhancements like POSIX attribute support, more-efficient handling with the C API, and better security handling.

It's also a good foundation for any number of other interfaces. A normal web UI is the natural next step, but it could easily provide, for example, S3 API compatibility.

For now, though I haven't gotten around to uploading a build to OpenNTF yet, feel free to poke around the code and let me know if any ideas strike your fancy.

The primary focus of my recent work with the OpenNTF Domino API has been the implementation of a fleshed-out design API. I've done a lot of work in my Domino-programming career manipulating design elements, usually via DXL (such as with my Forms 'n' Views app), and my aim is to codify all of that into the API itself, eventually growing it to cover almost every design element that Designer does (we'll see about Navigators, Composite Apps, and the like).

One of the big hurdles in recent years when manipulating design elements has been dealing with XPages-related elements - XPages themselves, Java design elements, Jars, faces-config.xml, etc.. Though XPages by their nature have a very nice XML representation, that doesn't carry over to DXL - all of those elements are dumped out in the "raw" data format that DXL uses when it doesn't have a better idea of how to handle what you're trying to export. And more specifically, though the XSP markup and the compiled Java code are present in the result, attempts to actually manipulate that were stymied by a "checksum" block of bits at the start of the data - you couldn't just drop the Java bytecode into the doc and expect it to work.

Fortunately, it turns out that those bits aren't a weird "checksum" at all: the data is just stored as Composite Data, the internal format of rich text in Domino. Specifically, all of those elements share the same internal representation as File Resources and Style Sheets, being composed of a CDFILEHEADER and list of CDFILESEGMENTs. When DXL uses the "raw" format, it exports that data as an array of Base64-encoded bytes directly. After discovering this, I wrote the start of a CD implementation in the API and got it to work: the API is now able to read/write the attached data of any one of those file types. Now, there are a few steps that would have to be implemented between this and "write a new XPage with a few lines of code", but this is a step in that direction (and you could theoretically modify your faces-config.xml, xsp.properties, etc. on the fly now, though I haven't tested much).

More immediately, it's also a huge step in the direction of interesting read-focused uses. One of these uses in particular has caught my fancy: loading and using XSP-related Java classes in other contexts. To go along with my work, I wrote a DatabaseClassLoader that gives you access to all of those class types from anywhere using the API (including Jar design elements). The possibilities for this have made me giddy: re-using data-model objects in external scheduled tasks, sharing classes with Java agents (...with the caveat that you have to use reflection or a scripting language), "loading" other XPage apps from inside another, treating NSFs as replicating, access-controlled Jars, and so on. I have my own notions for some practical uses I may want to explore, and I'm very interested in seeing what kind of things this lets others do as well.