This post is a drum I've been banging for a long time, from nagging the dev team in the IBM days through to formally requesting it in HCL's Ideas Portal. That idea there has been "Likely to implement" for a little while now, which is heartening, and either way I figured it'd be useful to have a proper blog post explaining the trouble and what a useful better version would be.

The Core Trouble

The main thing I'm talking about here is the act of having a third-party or (particularly) open-source project that depends on Domino artifacts - namely, Notes.jar, the NAPI, and the XPages UI components. I have more than a few such projects, so it's something I deal with pretty much daily.

When you're dealing with an XPages app in an NSF, this isn't really an issue: all the parts you need are there and are part of the classpath. You just reference lotus.domino.Database or com.ibm.xsp.extlib.util.ExtLibUtil and don't even give it a second thought. When you have a project outside of an NSF or Designer, though, you start to have to worry about this.

OSGi Projects

For OSGi-based projects, this means that you need to have a Target Platform that points to the XPages artifacts and then either have a variant of that that includes a packages Notes.jar or also include Notes.jar in your classpath another way. In Eclipse, this might be accomplished by adding Notes.jar to your active JVM and referencing a Notes or Domino installation's OSGi directories - this is something the XPages SDK helps with.

The immediate trouble this involves is if you want to build this project outside of Eclipse - most commonly now with Maven. This is where the IBM Domino Update Site for Build Management came in, which is a cleanly-packaged p2 update site of the XPages artifacts and Notes.jar, suitable for use with Maven+Tycho and any other tool (like Eclipse) that gets its dependencies out of a p2 repository. Unfortunately, it hasn't been updated since its initial release, and contains just the original 9.0.1 versions.

To aid with creating updated versions of that, I created the generate-domino-update-site tool a while back. Since no one outside HCL can legally share update sites themselves, the tool is the next-best thing: point it at Notes or Domino and it'll make one for you in a consistent way.

With either of those routes, though, there's still a gotcha: you still need to have each developer set up the update site for themselves, and it's only consistent across projects because the community settled on the notes-platform Maven property as a URI pointing to the update site. This is as opposed to something like Eclipse-the-IDE's repositories, which (as a virtue of being open-source) are publicly available and can be referenced freely.

Overall, it's a drag having to bring-your-own-site, but at least the use of notes-platform as a pseudo-standard smooths it out.

Non-OSGi Projects

Things get stickier with non-OSGi projects, though. With OSGi projects, the dependency mechanism lines up with the way the artifacts are delivered from the vendor: they all have OSGi metadata (or have a ready-made hook for it, like Notes.jar) and so just making a p2 site out of them makes them ready to go. They don't, though, have Maven metadata, and so referencing them that way takes extra processing.

I've gone about this two ways to date:

• The aforementioned update site project also has a mechanism for "Mavenizing" update sites. You point the tool at an existing p2 site (like one created by the first step), pick a groupId for it, and it'll install the files into your local repository.
• The P2 Maven Resolver plugin, which cuts out that middle step and uses a p2 repository as a source of Maven dependencies directly. This route is more "clever", but some tools get a little shaky with it.

Either way, the experience is okay but not perfect. There are some oddities to do with the different dependency mechanisms between OSGi and Maven, but overall it gets the job done.

The core trouble with it is that it's even less consistent across developers/projects than the Tycho notes-platform idiom. I've personally gone through a couple iterations of the Mavenized layout, with different inter-dependency schemes and groupIds. That leads to drift and incompatibility among projects. For example, I use the xpages-runtime project for client work to do my lingering XPages development, and there's some friction in keeping the dependency schemes between that and the client project in line, even though I'm the only developer.

What I'd Like

What I'd really like would be an official HCL-provided or -sanctioned repository for p2 and Maven use for these artifacts. I've pitched the idea of OpenNTF hosting this, since I already have the tools and servers on hand, though we'd have to come up with a way to agree about who is legally allowed to access it. All the better would be consistently-updated HCL-hosted repositories, where they could link access to one of the various HCL accounts we tend to have.

The best route would be to publish it on a repository that doesn't require authentication. While I'm making wishes, attaching Javadoc would be a classy touch too.

Anyway, that's the gist of it. It's one of the two main thorns in my side when doing Domino-targeted development (the other being initializing the runtime itself in the process), and it'd save me a whole lot of heartache if it had a proper solution.

A few years back, I more-or-less found the religion of null analysis, and I've stuck with it with at least my larger projects.

One of the sticking points all along, though, has been Eclipse's lack of knowledge about what code not annotated with nullness annotations does, with the biggest blind spot being the JDK itself. For example, take this bit of code:

1

BigDecimal foo = BigDecimal.valueOf(10).add(1);

That will never throw a NullPointerException, but, since BigDecimal#valueOf isn't annotated at all, Eclipse doesn't know that for sure, and so it may flag it as a potential problem. To deal with this, Eclipse has the concept of external annotations, where you can associate a specially-formatted file with a set of classes and Eclipse will act as if those classes had nullness annotations already.

Core JDK Annotations

Unfortunately - and as opposed to things like IntelliJ - Eclipse for some reason doesn't ship with this knowledge out of the box. For a while, I just dealt with it, throwing in technically-unnecessary checks around things like Optional#get that are guaranteed to return non-null. The other day, though, I decided to look into it more and found lastNPE.org, which is a community-driven project to provide such external annotations.

Better still, they also provide an Eclipse plugin (404 expected on that link - Eclipse knows what to do with it) to apply rules from your project's Maven configuration to the IDE. This not only covers applying external annotations, but also synchronizing compiler configurations.

Sidebar: The Eclipse Compiler

By default, a Java project is compiled with javac, the stock Java compiler. Eclipse maintains its own compiler, varyingly called ECJ or (as shorthand) JDT. Eclipse's compiler is, unsurprisingly, well-geared towards IDE use, and part of that is that it can flag and process a great deal of semantic and stylistic issues that the stock compiler doesn't care about. This included null annotations.

Maven Configuration

With this information in hand, I went to configure my project's Maven build. The first step was to change it to use Eclipse's compiler, since I had recently switched the project away from being Tycho-based (which uses ECJ by default). This can be done by configuring maven-compiler-plugin:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

<build>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-compiler-plugin</artifactId>
      <version>3.8.1</version>
      <configuration>
        <failOnWarning>false</failOnWarning>
        <compilerId>jdt</compilerId>
        <compilerArguments>
          <properties>${project.basedir}/../../config/org.eclipse.jdt.core.prefs</properties>
        </compilerArguments>
      </configuration>
      <dependencies>
        <dependency>
          <groupId>org.eclipse.tycho</groupId>
          <artifactId>tycho-compiler-jdt</artifactId>
          <version>1.7.0</version>
        </dependency>
      </dependencies>
    </plugin>
  <plugins>
<build>

I use an inline dependency on Tycho's tycho-compiler-jdt to provide the compiler. I stuck with version 1.7.0 for now because Tycho 2.0+ uses the newer core runtime that requires Java 11, which this project does not yet for platform lag reasons. I also find it useful to set <failOnWarning>false</failOnWarning> here because ECJ throws many more (legitimate) warnings than javac. Long-term, it's cleanest to keep this enabled.

I also configured Eclipse's compiler settings like I wanted for one of the project's modules, then copied the settings file to a common location. That's where the compilerArguments bit comes from.

Then, I went through the available libraries from lastNPE.org, found the ones that match the libraries we use, and added them as dependencies in my root project:

<properties>
  <lastnpe-version>2.2.1</lastnpe-version>
</properties>

<dependencies>
  <dependency>
    <groupId>org.lastnpe.eea</groupId>
    <artifactId>jdk-eea</artifactId>
    <version>${lastnpe-version}</version>
    <scope>provided</scope>
  </dependency>
  <!-- and so on -->
</dependencies>

Once I updated the project configurations, Eclipse churned for a while, and then I got to work cleaning up the giant pile of new errors and warnings it brought up. As usual with null checks, this was a mix of "oh, nice catch" and "okay, sure, technically, but come on". For example, it flags System.out.println as a potential NPE because System.out is assignable - this is true, but realistically my app's code is going to be the least of my concerns when System.out is set to null.

In any event, I was pleased as punch to find this. Now, I have a way to not only properly check nullness with core classes and common libraries, but it's a way that's shared among the whole project team automatically and enforced at compile time.

One of the main goals of OSGi is proper runtime dependency management, not only allowing a bundle to declare what its dependencies are to ensure that they're there, but even to select one of multiple available versions. For example, you might have one bundle that expects Guava 15 but not higher and another that expects 18 or above, and both can be loaded successfully if you have multiple Guava versions installed. The goal is that you're supposed to be able to just throw a whole bunch of bundles into a pot and the system will figure it out.

Before I get into why this system is hobbled on Domino, I'll add a little more background.

Mechanisms

For our purposes here, bundles have two main ways to declare what they are (there are more than this, but they're not relevant right now):

• The bundle's symbolic name combined with its bundle version. For example, the core bundle for ODA is named "org.openntf.domino" and has a version like "11.0.1.202006091416".
• The bundle's exported packages, which are Java class packages and might also individually have versions. Using ODA again as an example, it exports a slew of packages, such as org.openntf.domino and org.openntf.domino.nsfdata, though it doesn't specify versions for any of these.

The versions of the bundle and the exported packages don't need to be the same, nor do all the exported packages need to have the same version. This shows up a lot in "spec bundles", where a vendor will wrap a standard spec API in their own bundle for various reasons. For example, Apache Geronimo has a bundle called "org.apache.geronimo.specs.geronimo-jaxrs_2.1_spec", which provides the JAX-RS 2.1 spec. The bundle itself has a version of "1.1.0", while the exported packages are all version "2.1".

To require a bundle by name, you use the Require-Bundle header, like Require-Bundle: org.openntf.domino;bundle-version="11.0.0", which requires specifically the ODA bundle, version 11 or above. To require a package, you use Import-Package, like Import-Package: javax.ws.rs;version="2.1.0". The two methods have some implications when it comes to how the ClassLoaders work, which I touched on a bit earlier this year.

Culture Differences

The package-based mechanism is generally preferred nowadays over the bundle-based one, largely for the kind of flexibility and division-of-responsibilities it provides. For example, if I want version 2.1.0 of javax.ws.rs, my code shouldn't care at all whether it comes from Geronimo's bundle, JBoss's, or anywhere else - it's all the same thing (in theory). This is extremely common for projects created with bnd and tools based on it, which can generated imported and exported packages automatically based on your code and some small configuration. For example, the bundles that make up Open Liberty use bnd config files that have some loose configuration, which then is processed out to full listings of packages with version information. This is also often paired with OSGi's "Capability" system, but that's one of the "not important for now" things.

Domino - and I believe this is inherited from Eclipse, which does the same thing - is largely based on bundle requirements. For example, the org.eclipse.jdt.ui bundle (part of the Java development tools in Eclipse and Designer) requires a bevy of bundles by name version and doesn't tag any of its exported packages with versions. This is similarly reinforced in the tools. Eclipse, unsurprisingly, uses Tycho to bring the "Eclipse PDE" style to the table, where the MANIFEST.MF file is more hand-crafted, and the tools don't do as much for you automatically. You can go full Import-Package and attach versions to your exported packages with Eclipse, but the tooling doesn't encourage it.

Conflicts in Practice

That brings us back to how this all contributes to making working with OSGi a little extra annoying on Domino. This is something that comes up constantly in my XPages Jakarta EE Support project: I want to bring in an implementation component for a JEE spec, but it will either already have or will have generated for it OSGi rules that lean towards the "package-style" of doing things, versions and all. Because there are common packages (like, say, javax.activation) that are supplied by bundles present in the Domino runtime, I want to use those. However, since the packages from Domino don't have versions specified, I need to re-wrap the bundles to import the packages without versions. Thus begins this ongoing nightmare.

Another approach I could take would be to bring my own, nicely-OSGi-ified versions of the afflicted packages, and options abound. However, that leads to a sneaky other trouble: because some of the Domino bundles are multi-spec monsters - with com.ibm.designer.lib.javamail being the absolute worst culprit - some other bundle on the system might casually import javax.activation and javax.mail. If I have this other spec implementation floating around, then it could get matched to my bundle for the former and IBM's for the latter... and crash right into "exposed to a package via two dependency chains" problem. That wouldn't necessarily be an issue if everything used versions on packages, but leaving them off means it's kind of up to the container to match bundles to each other, and it's entirely happy creating impossible conflicts.

Ways Around It

When working through this sort of trouble, I've found a few ways around the things I've run into. The first is what I mentioned above: using p2-maven-plugin to re-wrap bundles with instructions that make them more Domino-friendly. This involves a few tricks:

• Excluding specific dependencies from the final distribution, when they would otherwise bring in specs that are already present by default or in my implementation
• Adding explicit Require-Bundle lines in order to "hard bind" the bundles where I need them
• Using "visibility:=reexport" in the above, which means that bundles that depend on the current one will also depend on the re-exported dependency by name.
• Adding bundle-symbolic-name rules to package imports, which is like a weird corruption of package imports that says "only accept this package from a bundle named X"

Aside from reworking existing bundles, I have a few times ended up creating fragment bundles that glom onto one bundle to tie it to another one after the fact, usually for the components that bridge different JEE specs.

The Wildcard: The System Bundle

In general, with OSGi, you can expect the packages you need to be provided in bundles, but it also allows for the core JDK to be implicitly available - that is, things like java.lang don't need to be imported, and are always present. That's fine, since it's generally well-enough-defined what makes up the JDK and what you'd need to instead bring in.

However, the Domino core classpath doesn't contain just the JDK, but also anything in jvm/lib/ext and (as a curveball) ndext from the Domino directory. Above, I specifically pointed out the javax.activation package, and that's because it suffers the most from both the javamail bundle as well as this. If you run tell http osgi packages javax.activation on Domino's console, you should see that bundles get this from two places: some use com.ibm.designer.lib.javamail, while others use the system bundle, org.eclipse.osgi. That "system bundle" concept is not only the thing in charge, but it also passively provides access to classes found in the lower-level classpath. In a fully OSGi environment, that system classpath will be pretty clean, but Domino's isn't that.

The good news here is that it isn't usually a big problem. If you import packages by a non-zero version, they'll never match from the system bundle this way. Still, it's always there, lurking, and the problems increase if you start adding more JAR files to jvm/lib/ext to avoid policy or amgr-memory issues. We've seen this periodically with ODA, where we used to support the use case of putting the core files there and using just a shim at the XSP layer, before it became too much of a hassle to manage. I ran into it again more recently when Guava showed up there: because I hadn't specified a version, it was able to match from the system bundle, and ended up with classes available at compile time but missing at runtime.

The Upshot

The upshot here is that there's no simple advice for dealing with this. Cultural and implementation factors make bringing third-party code into Domino unusually difficult, but it can generally be dealt with via various patching mechanisms. The XPages JEE project's dependency module ended up turning into a trove of such workarounds, so perhaps it can be useful if you ever run into this sort of thing yourself.