For a couple of years now, first IBM and then HCL have worked on and adapted community work to get Domino running in Docker. I've observed this for a while, but haven't had a particular need: while it's nice and all to be able to spin up a Domino server in Docker, it's primarily an "admin" thing. I have my suite of development Domino servers in VMs, and they're chugging along fine.

However, a thought has always gnawed at the back of my mind: a big pitch of Docker is that it makes not just deployment consistent, but also development, taking away a chunk of the hassle of setting up all sorts of associated tools around development. It's never been difficult, per se, to install a Postgres server, but it's all the better to be able to just say that your app expects to have one around and let the tooling handle the specifics for you. Domino isn't quite as Docker-friendly as Postgres or other tools, but the work done to get the official image going with 11.0.1 brought it closer to practicality. This weekend, I figured I'd give it a shot.

The Problem

It's worth taking a moment to explain why it'd be worth bothering with this sort of setup at all. The core trouble is that running an app with a Notes runtime is extremely annoying. You have to make sure that you're pointing at the right libraries, they're all in the right place to be available in their internal dependency tree, you have to set a bunch of environment variables, and you have to make sure that you provide specialized contextual info, like an ID file. You actually have the easiest time on Windows, though it's still a bit of a hurdle. Linux and macOS have their own impediments, though, some of which can be showstoppers for certain tasks. They're impediments worth overcoming to avoid having to use Windows, but they're impediments nonetheless.

The Setup

But back to Docker.

For a little while now, the Eclipse Marketplace has had a prominent spot for Codewind, an IBM-led Eclipse Foundation project to improve the experience of development with Docker containers. The project supplies plugins for Eclipse, IntelliJ, and VS Code / Eclipse Che, but I still spend most of my time in Eclipse, so I went with the former.

To begin with, I started with the default "Open Liberty" project you get when you create a new project with the tooling. As I looked at it, I realized with a bit of relief that there's not too much special about the project itself: it's a normal Maven project with war packaging that brings in some common dependencies. There's no Maven build step that expects Docker at all. The specialized behavior comes (unsurprisingly, if you use Docker already) in the Dockerfile, which goes through the process of building the app, extracting the important build results into a container based on the open-liberty runtime image, bringing in support files from the project, and launching Liberty. Nothing crazy, and the vast majority of the code more shows off MicroProfile features than anything about Docker specifically.

Bringing in Domino

The Docker image that HCL provides is a fully-fledged server, but I don't really care about that: all I really need is the sweet, sweet libnotes.so and associated support libraries. Still, the easiest way to accomplish that is to just copy in the whole /opt/hcl/domino/notes/11000100/linux directory. It's a little wasteful, and I plan to find just what's needed later, but it works to do that.

Once you have that, you need to do the "user side" of it: the ID file and configuration. With a fully-installed Domino server, the data directory balloons in side rapidly, but you don't actually need the vast majority of it if you just want to use the runtime. In fact, all you really need is an ID file, a notes.ini, and a names.nsf - and the latter two can even be massively trimmed down. They do need to be custom for your environment, unfortunately, but at least it's much easier to provide just a few files than spin up and maintain a whole server or run the Notes client locally.

Then, after you've extracted the juicy innards of the Domino image and provided your local resources, you can call NotesInitExtended pointing to your data directory (/local/notesdata in the HCL Docker image convention) and the notes.ini, and voila: you have a running app that can make local and remote Notes native API calls.

Example Project

I uploaded a tiny project to demonstrate this to GitHub: https://github.com/jesse-gallagher/domino-docker-war-example. All it does is provide one JAX-RS resource that emits the server ID, but that shows the Notes API working. In this case, I used the Darwino Domino NAPI (which I really need to refresh from upstream), but Domino JNA would also work. Notes.jar would too, but I think you'll need one of those projects to do the NotesInitExtended call with arguments.

The Dockerfile for the project goes through the steps enumerated above, based on how the original example image does it, and was tweaked to bring in the Domino runtime and support files. I stripped the Liberty-specific stuff out of the pom.xml - I think that the original route the example did of packaging up the whole server and then pulling it apart in Docker image creation has its uses, but isn't needed here.

Much like the pom.xml, the code itself is slim and doesn't explicitly refer to Docker at all. I have a ServletContextListener to init and term the Notes runtime, as well as a Filter implementation to init/term the request thread, but otherwise it just calls the Notes API with no fuss.

Larger Projects

I haven't yet tried this with larger projects, but there's no reason it shouldn't work. The build-deploy-run cycle takes a bit more time with Docker than with just a Liberty server embedded in Eclipse normally, but the consistency may be worth it. I've gotten used to running a killall -KILL java whenever an errant process gloms on to my Notes ID file and causes the server to stop being able to init the runtime, but I'd be glad to be done with that forever. And, for my largest project - the one with the hundreds of XPages and CCs - I don't see why that wouldn't work here too.

Normal Domino Projects

Another route that I've considered for Domino in Docker is to use it to deploy NSFs and OSGi projects. This would involve using the Domino image for its intended purpose of running a full server, but configuring the INI to just serve HTTP, and having the Dockerfile place the built OSGi plugins and NSFs in their right places. This would certainly be much faster than the build-deploy-run cycle of replacing NSF designs and deploying the plugins to an Update Site NSF, though there would be a few hurdles to get over. Not impossible, though.

I figure I'll kick the tires on this some more this week - maybe try deploying the aforementioned giant XPages .war project to it - to see if it will fit into my workflow. There's a chance that the increased deployment times won't be worth it, and I won't really gain the "consistent with production" advantages of Docker when the way I'm developing the app is already a wildly-unsupported configuration. It might be worth it if I try the remote mode of Codewind, though: I have some Liberty servers that Jenkins deploys to, but it'd be even-better to be able to show my running app to co-developers to work on something immediately, instead of waiting for the full build. It's worth some investigation, anyway.

When I was getting familiar with modern Java server development, one of the biggest conceptual stumbling blocks by far was CDI. Part of the trouble was that I kind of jumped in the deep end, by way of JNoSQL's examples. JNoSQL is a CDI citizen through and through, and so the docs would just toss out things like how you "create a repository" by just making an interface with no implementation.

Moreover, CDI has a bit of the "Maven" problem, where, once you do the work of getting familiar with it, the parts that are completely baffling to newcomers become more and more difficult to remember as being unusual.

Fortunately, like how coming to Maven by way of Tycho OSGi projects is "hard mode", coming to CDI by way of a toolkit that uses auto-created proxy objects is a more difficult path than necessary. Even better, XPages developers have a clean segue into it: managed beans.

JSF Managed Beans

XPages inherited the original JSF concept of managed beans, where you put definitions for your beans in faces-config.xml like so:

1
2
3
4
5
6
7
8
9

<managed-bean>
	<managed-bean-name>someBean</managed-bean-name>
	<managed-bean-class>com.example.SomeBeanClass</managed-bean-class>
	<managed-bean-scope>application</managed-bean-scope>
	<managed-property>
		<property-name>database</property-name>
		<value>#{database}</value>
	</managed-property>
</managed-bean>

Though the syntax isn't Faces-specific, the fact that it is defined in faces-config.xml demonstrates what a JSF-ism it is. Newer versions of JSF (not XPages) let you declare your beans inline in the class, skipping the XML part:

1
2
3
4
5
6
7
8

package com.example;
// ...
@ManagedBean(name="someBean")
@ApplicationScoped
public class SomeBeanClass {
	@ManagedProperty(value="#{database}")
	private Database someProp;
}

These annotations were initially within the javax.faces package, highlighting that, while they're a new developer convenience, it's still basically the same JSF-specific thing.

While all this was going on (and before it, really), the Enterprise JavaBeans (EJB) spec was chugging along, serving some similar concepts but it really is kind of its own, all-consuming beast. I won't talk about it much here, in large part because I've never used it, but it has an important part in this history, especially when we get to the "dependency injection" parts.

Move to CDI

Since it turns out that managed beans are a terrifically-useful concept beyond just JSF, Java EE siphoned concepts from JSF and EJB to make the obtusely named Contexts and Dependency Injection spec, or CDI. CDI is paired with some associated specs like Common Annotations and Inject to make a new bean system. With a switch to CDI, the bean above can be tweaked to something like:

1
2
3
4
5
6
7
8

package com.example;
// ...
@Named(name="someBean")
@ApplicationScoped
public class SomeBeanClass {
	@Inject @Named("database")
	private Database someProp;
}

Not wildly different - some same-named annotations in a different package, and some semantic switches, but the same basic idea. The difference here is that this is entirely divorced from JSF, and indeed from web apps in general. CDI specifically has a mode that works outside of a JEE/Servlet container and could work in e.g. a command-line program.

Newer versions of JSF (and other UI engines) deprecated their own version of this to allow for CDI to be the consistent pool of variable resolution and creation for the UI and for the business logic.

The Conceptual Leap

One of the things blocking me from properly grasping CDI at first was that @Inject annotation on a property. If it's just some Java object, how would that property ever be set? Certainly, CDI couldn't be so magical that I could just do new SomeBeanClass() and have someProp populated, right? Well, yes, that's right. No matter how gussied up your class definition is with CDI annotations, constructing an instance with new will pay no attention to any of it.

What got me over the hurdle is realizing that, in a modern web app in particular, almost everything you do runs through CDI. JSP request? That can resolve CDI. JAX-RS resource? That's managed by CDI. Filters? CDI. And, because those objects are all being instantiated by CDI, the CDI runtime can do whatever the heck it wants with them. That's why the managed property in the original example is so critical: it's the same idea, just managed by the JSF runtime instead of CDI.

That's how you can get to a class like the controller that manages the posts in this blog. It's annotated with all sorts of stuff: the JAX-RS @Path, the MVC spec @Controller, the CDI @RequestScoped, and, importantly, the @Inject'ed properties. Because the JAX-RS environment instantiates its resource classes through CDI in a JEE container, those will be populated from various sources. HttpServletRequest comes from the servlet environment itself, CommentRepository comes from JNoSQL as based on an interface in my non-JEE project (more on that in a bit), and UserInfoBean is a by-the-numbers managed bean in the CDI style.

There's certainly more indirect "magic" going on here than in the faces-config.xml starting point, but it's a clear line from there to here.

The Weird Stuff

CDI covers more ground, though, and this is the sort of thing that tripped me up when I saw the JNoSQL examples. Among CDI's toolset is the creation of "proxy" objects, which are dynamic objects that intercept normal method calls with new behavior. This is a language-level Java feature that I didn't even know this was a thing in this way, but it's been there since 1.3.

Dynamic scripting languages do this sort of thing as their bread and butter. In Ruby, you can define method_missing to be called when code calls a method that wasn't already defined, and that can respond however you'd like. Years ago, I used this to let you do doc.foo to get a document item value, for example. In Java, you get a mildly-less-loosey-goosey version of this kind of behavior with a proxy's InvocationHandler.

CDI does this extensively, even when you might think it's not. With CDI, all instances are dynamic proxy objects, which allows it to not only inject field values, but also add wrapper code around method calls. This allows tools like MicroProfile Metrics to do things like count invocations, measure timings, and so forth without requiring explicit code beyond the annotations.

And then there are the whole-cloth new objects, like the JNoSQL repositories. To take one of the examples from jnosql.org, here's a full definition of a JNoSQL repository as far as the app developer is concerned:

1
2
3
4
5
6

public interface PersonRepository extends Repository<Person, Long> {

  List<Person> findByName(String name);

  Stream<Person> findByPhones(String phone);
}

Without knowledge of CDI, this is absolute madness. How could it possibly work? There's no code! The trick to it is that CDI ends up creating a dynamic proxy implementation of the interface, which is in turn backed by an InvocationHandler instance. That instance receives the incoming method call as a string and array of parameters, parses the method to look for a concept it handles, and either generates a result or throws an exception. Once you see the capabilities the stack has, the process to get from a JAX-RS class using @Inject PersonRepository foo to having that actually work makes more sense:

• The JAX-RS servlet receives a request for the resource
• It asks the CDI environment to create a new instance of the resource class
• CDI runs through the fields and methods of the class to look for annotations it can handle, where it finds @Inject
• It looks through its contributed extensions and finds JNoSQL's ServiceLoader-provided extension
• One of the beans from that extension can handle creating Repository instances
• That bean creates a proxy object, which handles method calls via invoke

Still pretty weird, but at least there's a path to understanding.

The Overall Importance

The more I use modern JEE, the more I see CDI as the backbone of the whole development experience. It's even to the point where it feels unsafe to not have it present, managing objects, like everything is held together by shoestring. And its importance is further driven home by just how many specs depend on it. In addition to many existing technologies either switching to or otherwise supporting it, like JSF above, pretty much any new Jakarta EE or MicroProfile technology at least has it as the primary mechanism of interaction. Its importance can't be overstated, and it's worth taking some time either building an app with it or at least seeing some tutorials of it in action.

One of the specific problems that we encountered with my aforementioned client app first when expanding it to include REST services and then later to be portable outside an NSF entirely is dealing with varying mechanisms for interacting with the surrounding environment.

The Problem to Solve

The immediate way this distinction comes up when adding JAX-RS services or other OSGi servlets is trying to get a handle on the current Domino user session or context database. In an XPages app (including in code called in a plugin-based library), you can just do:

1

Session s = ExtLibUtil.getCurrentSession();

However, this will return null if called while processing an OSGi servlet. Instead, servlet code should call:

1

Session s = ContextInfo.getUserSession();

Same idea - they both return a session based on the current authenticated user from the HTTP stack - but they have different backing implementations. So my first pass was to coordinate these inside an AppUtil class in a method like this:

1
2
3
4
5
6
7

public static Session getSession() {
	if(FacesContext.getCurrentInstance() != null) {
		return ExtLibUtil.getCurrentSession();
	} else {
		return ContextInfo.getUserSession();
	}
}

This worked pretty well, until I added Tycho-based compile-time unit tests, which is an OSGi environment where neither of those paths would return a session. So I had to add a fallback that would just eventually spawn a new NotesFactory.createSession() if it couldn't find another one.

It's one thing for a getSession() method to balloon in logic, but Notes runtime access isn't the only problem like this. Take the case of validating model objects as part of the "save" process. In an XPages environment, validation errors should be reported as FacesMessages on the view root or, ideally, attached directly to the form control that represents the invalid field. In a REST service, though, the ConstraintViolationException should bubble right up to the top and be returned as an appropriately-formatted JSON object with a corresponding HTTP status code. Originally, we handled this similarly: we moved the FacesMessage stuff out of the model objects and into the AppUtil class and handled it with an if tree.

The RuntimeEnvironment Class

Eventually, though, there was enough customizable behavior that these branching methods in one class got out of hand, and that's even before getting into cases where a class (like FacesContext) may not even be available at runtime at all. So I implemented a RuntimeEnvironment class as a service. It started out like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

public interface RuntimeEnvironment {
	static final List<RuntimeEnvironment> knownEnvironments = AppUtil.findExtensions(RuntimeEnvironment.class).stream()
			.sorted((a, b) -> Integer.compare(b.getWeight(), a.getWeight()))
			.collect(Collectors.toList());

	public static RuntimeEnvironment current() {
		return knownEnvironments.stream()
			.filter(RuntimeEnvironment::isCurrent)
			.findFirst()
			.orElseGet(UnknownEnvironment::new);
	}

	boolean isCurrent();
	int getWeight();
}

The AppUtil.findExtensions method is a simplified wrapper around the IBM Commons ExtensionManager call to find services in a type-safe way.

This allows me to define a series of RuntimeEnvironment implementations that may or may not be included in a given packaging of the app, while the isCurrent() and getWeight() methods allow me to distinguish between multiple valid environments to find the most specific. To get an idea of what I mean, here is the current suite of environment implementations:

These run a wide gamut. XPagesEnvironment and OSGiServletEnvironment are the big ones that kicked it off, but TychoEnvironment is there to handle compile-time tests, while NotesEnvironment lets the same code work in some utilities we launch from within the Notes client - and SWTRuntimeEnvironment allows those same tools to run outside of OSGi.

Once I broke ground on these classes, the number of situations where they're useful began to become obvious. Take, for example, resolving a variable. The on-Domino XPages implementation looks like what you'd expect:

1
2
3
4
5

@Override
public <T> @Nullable T resolveVariable(final String varName) {
	FacesContext context = FacesContext.getCurrentInstance();
	return (T) context.getApplication().getVariableResolver().resolveVariable(context, varName);
}

In the OSGiServletEnvironment and JakartaRuntimeEnvironment cases, though, I can use CDI instead:

1
2
3
4
5

@Override
public <T> @Nullable T resolveVariable(final String varName) {
	Instance<Object> instance = CDI.current().select(NamedLiteral.of(varName));
	return instance.isResolvable() ? (T)instance.get() : null;
}

It gets down to little things, too, like how the POST destination for form-based login can be /names.nsf?Login on Domino but /j_security_check on other webapp servers.

Seeing It Elsewhere

This sort of idiom is by no means anything I came up with. You can see it pretty frequently - in fact, I highlighted the way the IBM Commons stack does a very similar thing when running XPages outside Domino:

This serves essentially the same purpose, being filled with mechanisms for getting output streams, finding resource locations, and retrieving named objects.

Regular Use

Should you implement something like this for most apps? Probably not, no - for most even-moderately-complex XPages applications, having some if tests in a central util to distinguish between XPages and OSGi servlets should be enough. I think it's a useful instructional example, though, and it sure was critical in getting this massive thing working outside Domino. As we make our apps more portable, this is the sort of technique we should keep in mind.

Lately, one of my client projects has been picking up the pace on the years-long effort of taking a giant XPages app, making the business logic portable, and incrementally cutting down on the "XPage-iness" of it all. I expect that this will be a recurring source of blog posts, and this one is about distinguishing between "XPages the UI toolkit" and "XPages the web app framework".

XPages in an NSF

Coming from a Domino perspective, there is no distinction between the two, and that's largely because of the path we took to get here. Other than existing inside the same NSF, the distinction between "classic" Notes apps (web or client) and XPages couldn't be more stark. Legacy design elements are only shared in ways that are completely divorced from their original UI presentation (for the better), and the runtimes - as much as legacy elements can be said to have a "runtime" - are entirely distinct.

An XPages app can be thought of as conceptually a "normal" Java WAR-based webapp housed inside an NSF, and it has a lot of the trappings: classes in WEB-INF/classes, libraries in WEB-INF/lib, and an OSGi-style WebContent folder for miscellaneous files. It's not technically a normal webapp - there's no "web.xml" and the XPages outer "LCD" runtime is actually more like one giant webapp that acts like many - but it's close.

Critically, though, the Domino HTTP router only routes requests for ".xsp" files or "/xsp/" folders to your app's XPages environment, and this is the biggest technical and conceptual impediment. You can't (within an NSF) intercept just any incoming request and process it as you would in a normal webapp. You can kind of shim your way into it with servletFactory, but it's a fiddly process and limited to "/xsp/..." URLs.

Additionally, a running XPages app only exists in a very constrained way between requests. While the JSF-level "Application" and the Servlet-level session exist, you don't work with a per-app ServletContext the way you do in a Servlet webapp. You can hook in with ApplicationListeners and similar constructs, but they're still based on the lifecycle of the XPages app, which comes into existence only on the first request and dies (usually) half an hour after the last.

These, plus the specifics of Domino data access, combine to make the "XAgent" - an abomination of a concept - the catchall replacement for specialized rendering, batch processing, and even scheduled tasks.

XPages the View Engine

These are all accidents of history, though. They stem from the firm requirement that existing Domino HTTP behavior remain intact even with its brain transplant, as well as the "soft" requirement that XPages in an NSF pretend to be "forms with repeats and partial refresh".

At its core, XPages is "just" a web view engine: its only job is to accept a request from an HTTP client and return some HTML. The concepts it uses to accomplish this - components, renderers, managed beans, themes - are all incidental to the main task. This is the "V" part of MVC. Admittedly, even without the NSF compromises, XPages bleeds beyond its assigned third of the triad, and it inherited this from JSF. JSF is also billed as MVC, but it completely subsumes the "Controller" part and partially eats the "Model" part with its bean management.

Still, though, even a domineering framework like JSF slots in as just one component of a normal webapp, rather than being the whole thing as XPages is in an NSF. For example, take the app behind this blog, which partially looks like this:

It uses JSP as its view template engine, but is it a "JSP app"? Not really. The fact that it uses MVC 1.0 is more important to understanding it, but that's really an extension to JAX-RS. You could make a strong case that it's a "JAX-RS app", especially when you expand the "Services" section in Eclipse:

That covers more of it, but still leaves parts out. It has application-wide beans by way of CDI, entirely-UI-free scheduled tasks kicked off from a ServletContextListener, and core business logic and model objects that are kept in a module that doesn't even know about the Servlet API.

It's layered, but the layers are explicable and the distinctions create a tremendous amount of flexibility. I could, if I wanted, change to ThymeLeaf for the front end with essentially no friction, JSF or Vaadin with only mildly more, or to a client JS REST UI by chopping off the top two layers outright.

Okay, So?

This description isn't a call to action - there's nothing inherently wrong about an XPages app in an NSF, especially a small-to-medium one - but this will be an important part of the conceptual groundwork in the months to come. To figure out what to do with all these piles of XSP markup and framework-specific business logic we have, we'll have to do a lot of deconstruction.

In my last post, I casually mentioned the concept of ClassLoaders a couple times, and I think that they deserve their own post. ClassLoaders are exactly the kind of thing where, once you do Java long enough, you start to take for granted, but which aren't necessarily immediately obvious for people not as immersed.

The Basics

The core job of a ClassLoader is what it says on the tin: it loads classes. Say you have this bit of code using a class from the core Java library:

1

long now = System.currentTimeMillis();

This uses two types: long, which is a built-in primitive type and not a class at all, and java.lang.System. long doesn't have to come from anywhere, but java.lang.System does, and that's the job of a ClassLoader. In this case, the Java VM will ask the contextual ClassLoader for a class by that name, and the ClassLoader will (at least in Java 8 - things got weird later) look into the core library and find a file named "java/lang/System.class" within "rt.jar", parse its binary contents into an executable class, and hand it back to the VM.

ClassLoaders are also the source of two problem reports you've likely seen: ClassNotFoundException and NoClassDefFoundError. These two basically mean the same thing: the running app tried to load a class by name, but it wasn't found - they just differ in context (the former generally when a class is asked for dynamically, the latter when it's referenced as part of compiled code). This sort of thing can occur when you write code using a class that's present in your development environment but is not present when run later - among XPages developers, this happens quite a bit when people drop some JARs into jvm/lib/ext in their Designer installation but don't do the same on Domino.

Resource Loading

In addition to finding classes, ClassLoaders have a few other tasks, the main one of which of interest to us is loading resources. In my previous post, I talked about how ServiceLoader looks for service files by a given name, like META-INF/services/com.sprockets.data.FizzBuzzConverter. It does this by checking with the current ClassLoader and calling cl.getResources("META-INF/services/com.sprockets.data.FizzBuzzConverter"), which will return a listing of resources from JARs (and JAR-like sources, like an NSF) that it knows about matching that name. In that way, multiple JARs can declare services with the same name without conflicting.

ClassLoader Trees

Though conceptually your running program has "a ClassLoader", in reality it's almost definitely a chained series of ClassLoaders, rooted in the core system ClassLoader and then drilling down more specifically to your app's code. For example, take an application running in Apache Tomcat. In that case, Tomcat's documentation describes four basic tiers:

• The core JVM ("bootstrap") ClassLoader that comes with any running Java program. As Tomcat's docs note, this implementation may vary
• The central ("system") ClassLoader that contains the "just above the metal" classes, such as those you may add in the "CLASSPATH" environment variable
• The Tomcat-specific ("common") ClassLoader, containing classes shared among all running applications. For example, javax.servlet.Servlet would be found here
• Your app's ClassLoader, containing classes you write as well as any third-party JARs you bundled into your WAR file in WEB-INF/lib

When your code executes and requests a new class, the runtime will check first with your app's local ClassLoader and return what it finds there if present - if the class isn't present there, then that ClassLoader will delegate up to its parent, and so forth until it either finds a class or hits the root and throws a NoClassDefFoundError.

The way that each app has its own ClassLoader is also how you can have multiple apps on the same server that can each know about common core classes, but don't step on each others' toes with their own custom classes. Though javax.servlet.Servlet is the same class for two running apps, one app could have an internal class named "com.example.SomeBusinessLogic" and it wouldn't be visible by other running apps.

Dynamic ClassLoaders

Though the normal case of ClassLoaders is that sort of "do I have this class? If not, ask my parent" chain, the fact that a ClassLoader is itself a custom Java class means that its behavior can be pretty arbitrary. This is present in a normal web app ClassLoader: it knows to look in the WEB-INF/classes path within the WAR file instead of the normal behavior of checking from the root of a JAR, and it knows how to look in WEB-INF/lib for additional JARs to search.

In an XPages application, the active ClassLoader is roughly similar to Tomcat's app ClassLoader example, but with a couple additional capabilities. The main one is that the NSF's ClassLoader - an instance of com.ibm.domino.xsp.module.nsf.ModuleClassLoader - has knowledge of how to treat an NSF as if it were a WAR file. In Designer's "Package Explorer" pane, you get a view of the NSF that makes it look basically like a normal WAR, where classes go in WEB-INF/classes and JARs go in WEB-INF/lib. However, it's still really a nebulous pool of notes floating around, and so the ModuleClassLoader does design-collection lookups for file resources of various types and loads the class bytecode or resource data from there.

It also, in a move presumably designed to inconvenience me personally, has explicit restrictions on what classes it can load: even though it knows about, for example, org.eclipse or com.ibm.domino.napi classes, it has a check to explicitly bar loading these. That's why, even if you configure Designer to see those classes and compile XPages code that references them, they won't be available at runtime.

OSGi ClassLoaders

OSGi ClassLoaders are a particular kind of dynamic ClassLoader. In addition to the normal hierarchical view of the world, they take on special responsibilities for ensuring that your OSGi module (which an XPages app kind of is) sees classes from other bundles based on its dependency rules, but not necessarily their resources. For example, take rules like this in an OSGi bundle's META-INF/MANIFEST.MF:

1
2

Require-Bundle: com.ibm.xsp.core
Import-Package: com.ibm.commons.util

These simple lines hide some beguiling complexity. With this definition, a running class in your bundle will be able to see:

• All classes at the system level, such as java.lang.System
• All classes contained within and exported by "com.ibm.xsp.core", such as com.ibm.xsp.FacesExceptionEx and com.ibm.xsp.url.UrlHandler
  • There's also special behavior going on here, because those classes are contained within an embedded JAR in the bundle, referenced as Bundle-ClassPath: lwpd.xsp.core.jar - this is an OSGi-ism
  • Though this bundle lists all of its packages in its Export-Package header, this is not a requirement: it's common for an OSGi bundle to have classes internally that are not accessible from outside
• All classes exported by its bundle dependency that it marks as visibility:=reexport: "com.ibm.pvc.servlet" and "com.ibm.designer.lib.jsf"
  • This is why you can have a dependency on just "com.ibm.xsp.core" and access javax.faces.context.FacesContext even though it's not in the core XSP bundle
  • This is also transitive, though neither of those re-exported dependencies themselves re-export any dependencies
• The classes from the "com.ibm.commons" bundle in the "com.ibm.commons.util" package. This means that com.ibm.commons.util.StringUtil is visible, but com.ibm.commons.extension.ExtensionManager is not, despite both being within the same bundle JAR

There are also tons of weird visibility and dependency details as well in OSGi, but that's the gist of it. Note that I specifically mentioned that the resources aren't visible. Though the Require-Bundle: com.ibm.xsp.core line makes all classes exported from the XSP core visible to your code, calling ServiceLoader.load(com.ibm.xsp.acf.HtmlFilteringFactory.class) will not find the DefaultHtmlFilteringFactory implementation declared in there, even though it's done in a ServiceLoader-compatible way. This is why IBM Commons papers over that difference with its "plugin.xml" extension declarations. OSGi actually contains a Service Loader Mediator specification to bridge this gap, but Domino doesn't include an implementation of that part.

Fragment Bundles

There's one special case with OSGi bundles that's worth highlighting: fragments. Normally, each bundle effectively has its own ClassLoader space, walled off from all others by OSGi's broker. However, if you declare your bundle as having a Fragment-Host of another active bundle, your code acts as if it's within the parent, gaining access to not just all of the parent bundle's classes, but also its non-class resources. Moreover, this works in the reverse: the parent also gains access to the fragment's classes as resources, though it generally won't "know" about them at the time of development.

This is a technique that's come in handy for me many times, in particular in cases like the XPages Jakarta EE Support project, where API bundles will use ServiceLoader to find their implementations. In those cases, one of the ways I get it to work in OSGi is to create a fragment bundle out of the implementation, meaning that the bundles remain distinct but now the API can find the META-INF/services files and classes it needs to operate.

This has a good number of other uses, too, such as providing platform-specific native code to an otherwise-platform-independent core bundle. The Notes.jar wrapper used in XPages land uses this type of technique. Though, to my knowledge, Notes.jar doesn't contain any actual native code, it's still delivered in two pieces:

1. The "com.ibm.notes.java.api" bundle, which lists all of the exported packages but holds no code itself
2. The "com.ibm.notes.java.api.win32.linux" bundle, which contains the actual Notes.jar and declares Fragment-Host: com.ibm.notes.java.api
   • I'm not sure why this is the case, but maybe Notes.jar is different on System i or something

If you have a bundle that needs access to lotus.domino classes, you then can either do Require-Bundle: com.ibm.notes.java.api or Import-Package: lotus.domino and it'll be resolved out of the fragment. There's also an Eclipse-ism in here: the first bundle has Eclipse-ExtensibleAPI: true, which is a tip-off to the IDE that it should specifically allow fragments to contribute available classes to the development environment. This is generally required when developing with Eclipse's plug-in tooling (shared with Designer), but it's not actually enforced one way or the other by the runtime.

Wrapping It Up

This is all definitely in the category of "you don't normally need to worry about it, but it's very helpful to know", like the previous ServiceLoader topic. Until you're implementing some low-level stuff, you're not likely to interact with the ClassLoader directly, especially to a level beyond finding Thread.currentThread().getContextClassLoader() or Foo.class.getClassLoader(). Knowing about it can help make clear what's going on in situations where a class shows up in development but not at runtime, or when the XPages ClassLoader tries to get to fancy and throws up on itself.

The concept of "services" in Java is fairly critical, but, especially with the XPages stack we've grown used to, the term covers quite a few different technologies.

Definition

Before I continue on, I want to make clear what I mean by "service" in this context. It's unrelated to REST services or even remote access of any kind; instead, it's about how an app can find implementations of some kind of class or interface within its runtime.

A very-common type of this sort of thing is a data adapter or converter. Say you have your own object FizzBuzz that you use within your app, one that represents data storable in multiple ways. One way to handle converting from various types to FizzBuzz would be a giant if tree, like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

public FizzBuzz convert(Object input) {
	if(input instanceof String) {
		// ...
	} else if(input instanceof JsonObject) {
		// ...
	} else if(input instanceof org.w3c.dom.Document) {
		// ...
	} else {
		throw new IllegalArgumentException("Cannot convert to FizzBuzz: " + input);
	}
}

That'd work well enough, especially for a small app. You can imagine, though, how this might get out of hand in an even moderately-complicated case, with the if tree turning into a tangled mess. Moreover, this doesn't allow for any extensibility without directly modifying the convert method - any new type will have to go into this, making management of a large team more cumbersome and completely cutting off the possibility of third-party additions.

So, to keep things scalable, it'd make sense to create an interface that would specify a generic way to convert some type of object to a FizzBuzz:

1
2
3
4
5
6

package com.sprockets.data;

public interface FizzBuzzConverter {
	boolean canConvert(Object o);
	FizzBuzz convert(Object o);
}

Then the code that actually needs to convert would look more like this:

1
2
3
4
5
6
7
8

public FizzBuzz convert(Object input) {
	Stream<FizzBuzzConverter> converters = moreOnHowToFetchLater();
	return converters
		.filter(converter -> converter.canConvert(input))
		.findFirst()
		.map(converter -> converter.convert(input))
		.orElseThrow(() -> new IllegalArgumentException("Cannot convert to FizzBuzz: " + input));
}

In a small case like this, that's not necessarily going to be a big deal, but it doesn't take too long for it to become desirable to break it apart. Take the case of JAX-RS providers, which do exactly this kind of entity conversion when processing HTTP requests. Everything over HTTP comes in as plain text (more or less), but programmers want to be able to accept an int input parameter, or to automatically convert their custom business-logic object to JSON. Without a separation like this, the code to handle all known types would be impossible to manage all in one place, and there'd be no way to handle custom types that didn't exist when the code was written.

Types

There are quite a few distinct types of services that I've run across, and I'll list them here in roughly the likelihood that a programmer coming from an XPages background will encounter them.

ServiceLoader Services

This is the most-common kind of service you're likely to encounter in a Java application, and you can generally identify it by its use of the META-INF/services directory inside a JAR. java.util.ServiceLoader itself was added to Java in 1.6 but was designed to codify habits that become common beforehand.

The way this works is designed to be simple: you create a plain-text file within META-INF/services named after the service class you're implementing, and then put the names of your implementing classes within it, one on each line. So, in our above example, you'd create a file named META-INF/services/com.sprockets.data.FizzBuzzConverter and fill it with something like:

com.sprockets.data.impl.StringFizzBuzzConverter
com.sprockets.data.impl.JsonObjectFizzBuzzConverter
com.sprockets.data.impl.DomDocumentFizzBuzzConverter

Code that calls ServiceLoader.load(FizzBuzzConverter.class) will find all of those files within the current ClassLoader space (more fun with that down the line) and instantiate the named classes, returning an Iterator to loop through them.

IBM Commons Services

Within the XPages stack, the bulk of service interactions are managed by the IBM Commons ExtensionManager class, which is a generic way to ask for a service type by String name.

In the normal case, this acts as a slightly-old-timey variant of the now-standard ServiceLoader mechanism, likely by dint of preceding the standard's introduction. Like ServiceLoader, it looks for files with the name you pass it in the META-INF/services directory in your app and adds instances of all the names it finds within.

What makes it important (and what gives it longevity in non-XPages OSGi apps on Domino) is that it also bridges into the Equinox OSGi service infrastructure when available and looks for services registered there by the com.ibm.commons.Extension name. The reason this is important is that, in an OSGi context, one bundle can't by default see the files in another bundle in its ClassLoader, which means that services registered via META-INF/services in one won't be picked up by a ServiceLoader call in another.

Since XPages's life spanned a pre-OSGi era and the 8.5.2 "Extensibility API" era, it bears the signifiers of both, smoothly papered over by IBM Commons:

The Equinox loader looks in both places, in fact, which is why you can declare XPages services within an application using META-INF/services as well as within an OSGi bundle's plugin.xml file.

Equinox plugin.xml Extensions

I mentioned above that IBM Commons bridges the difference between ServiceLoader and Equinox, but now I'd better go into a little more detail about the latter.

"Equinox" refers to the particular OSGi implementation that underlies both Eclipse-the-IDE (and thus Notes) and Domino's web stack. While Equinox is the fully-fledged reference implementation of OSGi, plugin.xml is specific to it and I believe pre-dates Eclipse's migration to OSGi (which we still see reflected in 9.0.1FP10+'s plugin trouble).

plugin.xml used to house a lot of information that was moved over to META-INF/MANIFEST.MF, but its primary remaining function is to declare services for the Equinox environment. Eclipse itself uses this extensively, and it remains the primary way to extend the IDE's capabilities.

One important thing to note here is that plugin.xml's extensions aren't limited to just providing a service class implementation. While many do that, it's also used heavily to provide configuration information without executable classes at all.

Multi-type "FactoryFinder" style

This type of service locator is similar to the IBM Commons ExtensionManager, but is usually confined to an individual domain, like a specific Jakarta EE spec. The way this idiom works is that there's a central coordinating class, usually named FactoryFinder, whose job it is to locate implementations of services from one or more sources, and often using a known fallback implementation.

I encountered one of these when diving deep into the XPages stack. javax.faces.FactoryFinder is responsible for finding implementations of very-low-level entities, like the services that spit out JSF applications at the start of initialization, or those that create FacesContext objects.

These will often have specialized behavior. For example, the standard SOAP API looks through a system property, then an external "jaxm.properties" file, then ServiceLoader, then an older META-INF/services name, then OSGi, and finally falls back to a default class name.

Java 9 Modules

I have to admit that I haven't actually used this, but it's too important to skip. Java 9 and above include a module system that is sort of like an OSGi bundle in that it lets you declare what your module exports and other characteristics about its interactions with the outside world.

Along with this support came a new way to declare services. Since this is also baked in to Java itself, it gets the advantage of also working with ServiceLoader. In this case, instead of writing a text file in META-INF/services, you declare the type of service you're providing and the class implementing it in the module definition. This not only unifies the service with other module information, but it also makes it more type-safe and programmatically clear. It's neat-looking.

OSGi

I already mentioned that Equinox can use the "plugin.xml" file to do cross-bundle services in OSGi, but I also mentioned that it's specific to that one implementation and not actually part of the OSGi spec.

Instead, OSGi has a couple (for some reason) standard mechanisms for providing and consuming services. I encountered these mechanisms in practice when I created a UserRegistry implementation for Open Liberty.

In my first version, I declared my services programmatically in the bundle's activator (which is a class that you can write to run when your bundle is loaded/unloaded). In that way, you can dynamically tell the runtime that your bundle provides any number of services.

In my second revision, I changed to using what's dubbed Declarative Services. These do basically the same thing, but are defined for the runtime in a combination of the META-INF/MANIFEST.MF file and some service-definition files in the bundle - essentially, like a re-thought version of plugin.xml.

Summary

Okay! So, what's the upshot? Well, in my work, I use the first two all the time: inside a non-Domino app, META-INF/services is king; when working with Domino, IBM Commons ExtensionManager handles everything I need.

As far as implementing your own services, it's definitely a critical concept to keep in your pocket (I can only assume it's somewhere in Design Patterns). You could certainly go crazy with it and make a real mess of incomprehensible indirection, but it's probably useful more often than you'd think at first. Give it a shot next time you find yourself writing a big if tree with complicated branches.