Showing posts for tag "open-source"

Recent Open-Source Project Updates

Fri Sep 06 14:25:34 EDT 2024

I've released a spate of open-source project updates recently, and I figured it'd be good to round up what's new. Most of them are utilitarian in nature - mostly fixes for things that crop up with Domino 14 and Java > 8 - but the first one is larger.

XPages Jakarta EE

Today, I released version 3.1.0 of the XPages JEE project. This is mostly about fixing up some edge-case and sporadic bugs that cropped up in 3.0, but also includes some performance updates and contributions from new contributors. Additionally, it should work on the newly-launched Domino 14.5 EAP1. The use of Java 21 in that version of Domino won't necessarily affect XPages JEE in a while, since JEE 11 targets Java 17, but there's some neat stuff in there for general use.

p2-layout-resolver

The p2-layout-resolver is a plugin that allows the use of p2 (Eclipse-style) repositories as Maven dependencies in non-Tycho projects. I use this in a lot of cases where I move a project from Tycho to maven-bundle-plugin for simplicity in dependency management.

Version 1.9.0 includes a very-useful contribution that fixes dependencies in cases where a bundle has a Bundle-ClassPath entry that references an embedded JAR that doesn't exist. In the Domino world, this cropped up in Domino 14, so it's useful if you're building anything that targets that version of the runtime or above.

p2-maven-plugin

For various Domino-related needs, I maintain a fork of the p2-maven-plugin, which is useful for its additions of things like generating site.xml files (still important for importing into an NSF update site, after all these years) and the <transform>jakarta</transform> option to run JARs through Eclipse Transformer when bundling them, allowing use of pre-Jakarta JEE artifacts in a smooth way.

The 3.1.x versions focused on fixing problems when running on Java > 8 (namely no longer using IBM Commons XML) and improving handling of some other hiccups.

NSF File Server 2.0

Sun Apr 07 13:59:18 EDT 2024

A few years ago, I made a little project that hosts an SFTP server that stores documents in an NSF. I've used it here and there since then - as in the original post, I stashed some company docs in it to have them nicely synced among our Domino servers, and I've also had cases where clients use it to, for example, provide a way for their vendors to upload files in a standard way.

The other week, I decided to dive back into it to add some capabilities I'd wanted for a while, and the result is version 2.0.0. This version is a significant revamp that adds quite a bit.

Multiple Mounts

The first limitation I wanted to improve was the fact that the first version was restricted to a single NSF. That works fine in the basic case, but I wanted to start doing things like storing server config backups in there, and wouldn't necessarily want them in the same NSF as, say, company contracts, credentials, and secrets.

The way I went about this was to make it so that the new configuration NSF has a "Mounts" view that lets you specify a path in a conceptual top level of the filesystem that would then point to a target NSF. This allows the admin to do things like have separate ACLs for different mounts - since the client will act as an authenticated Domino user, these will be properly obeyed, and a user won't be able to access documents they don't have rights to.

Additionally, I could configure it so that not all mounts are present on all servers, which will come into play particularly with the next feature.

Screenshot of the Mounts view in the file server config NSF

New Filesystem Types

Once I had a composite top-level filesystem, I realized that it wouldn't be terribly difficult to allow more filesystem types than the original NSF file store I made. That filesystem is built using the NIO File System Provider framework added in Java 7, and that system is designed to be pretty extensible. By default, Java comes with a few providers: the normal local filesystem as well as one that can treat a ZIP or JAR as a contained filesystem itself. These are accessed in a generic way, where you specify a URI and a map of "env" properties that are provider-specific.

For example, the ZIP filesystem takes a URI in the form of "jar:file:/some/path/to/file.zip" and an environment map configuring whether the filesystem should be created if it doesn't already exist in memory and what encoding to use for filenames (very important if you have Unicode characters in there).

So I added ways to configure a mount to the local server filesystem (similar to what the Mindoo FTP Server does) and then a generic configuration for any installed provider. It's probably uncommon that you will have a custom File System Provider implementation in your Java classpath, but hey, maybe you do, and I want to allow that to work.

I also added an extension point to the project itself that allows adding new providers via plugin.xml files in OSGi, and I can think of a couple other projects that may use this, like the NSF ODP Tooling.

WebContent Filesystem

Beyond adding the JVM-provided systems, I wrote another new filesystem type, one that provides access to the conceptual "WebContent" directory presented in Package Explorer in Designer:

Screenshot showing Designer and Transmit looking at the same WebContent in an NSF

The idea here is that this could be used to deploy, say, a JavaScript client application to an NSF without the developer or build server having to know anything about Domino. Pretty much everything can work with SFTP, so this makes accessing those files a lot easier. This is similar to the WebDAV capabilities Domino has had for a very long time, but with a different protocol.

Server Keypairs

In the first version, the app would generate and store the server's SSH keypair on the filesystem, in the data directory. This is fine, but part of the point of this whole project is that I like to get away from non-replicating stuff, and so I moved these keys to the configuration NSF. Now, on first connection, the server will look for a keypair document in the NSF and, if it doesn't exist, will generate a new one and store it there. Since I've been working with encrypted fields a lot for client work lately, I also realized that this was a good use for it: the public key is a normal text item (so you could distribute and verify it as needed), but the private key is encrypted with the generating server's ID file. Since only the server itself ever needs to know its private key, this works swimmingly.

JNX

This isn't a new app feature per se, but this was a good situation for me to put JNX to work in an open-source project. I had originally written this using the lotus.domino classes for most work and the IBM NAPI for things like generating sessions for a given username, but switching to JNX let me ditch both of those.

Admittedly, this is a case where switching to JNX didn't grant me significant new capabilities, but it DID let me do a couple things better. Some things are distinct feature improvements, like improving password authentication (previously, I was doing a compare of hashes "manually", which is fragile), while others are just making the code smoother, like no longer having to do the read-convert-recycle dance with DateTimes in LSXBE. It's just pleasant, and let me find a few places where the JNX API could be improved.

Future Additions

When I pick this project back up, there are certainly a couple things I'd like to add.

One would be to look into rsync support: rsync is tremendously useful for things like synchronizing filesystem-bound configs, but it's its own protocol tunneled over SSH, and so just having SFTP isn't enough. The underlying Apache Mina SSHD project is a general SSH server and not just SFTP, so it may be possible to do it by intercepting the commands sent over to initialize rsync, but it will be non-trivial. There's a library in Java that provides an rsync server, but it's GPL-licensed, and so I have to keep away for license-safety's sake.

Beyond that, it's mostly that I'd like to implement more filesystem types. Presenting data as a filesystem can be a very powerful tool: you could imagine providing access to documents in a DB as DXL or YAML, or listing files from a Document Library NSF, or (as I'd like to do some day) having the NSF ODP Tooling project replicate the ODP layout over SFTP.

For now, I'm looking forward to putting it to more use as a coordinating point. If I keep messing around with apps on TrueNAS, it'll give me a good feeling of security to have more info stashed in Domino and less prone to destruction if one server happens to blow up.

My 2022 Year In Review

Tue Jan 03 14:46:32 EST 2023

Tags: open-source

When I remember, I like to do a bit of a year-in-review post, which generally revolves around my open-source work. For that purpose, I realized that my GitHub commit graph can set the stage nicely:

My GitHub commits in 2022

Apparently, April and October were busy months for the client work I do that isn't on github.com.

Projects

XPages JEE Support

My biggest project for the year was definitely the XPages Jakarta EE Support project. At the end of last year, I made the move to Jakarta EE 9 as the baseline, with its associated package-name switch from javax.* to jakarta.*. This move really let the project flourish. Since I'm no longer regularly fighting with the ancient versions of JEE specs that ship as part of the XPages stack, I've been able to add a bunch of new features, such as JSF, Concurrency, Transactions, and improved Servlets. I've also been able to make huge strides on the Jakarta NoSQL support, adding the necessary evil of view-based operations and provisional drivers for the AppDev Pack and Keep.

Though that project is starting to get very constrained by Jakarta EE 10's move to Java 11 as a base requirement, there's still a lot of work I can do with the EE 9 specs. I have some larger ideas in mind and I have a stack of issues in the list of various sizes to look at. This project is the baseline for one of my larger client projects, and it's my current best idea for what Java development with Domino should look like.

NSF ODP

Next up would probably be my workhorse, the NSF ODP Tooling. Though I put out a good few releases this year, they've primarily been about compatibility with new Eclipse and macOS releases. macOS compilation in particular is a real bugbear: the OS's tightening restrictions and Notes's varying ways of adapting to them have made it a real moving target.

I've had it in my mind for a while now to add a specific Docker-based compilation option. It's always been possible to run the compilation inside Docker by building your own container and using the tooling within it, but it'd be useful to have that as a standardized thing controlled by the Maven plugin. That way, you'd still do the normal Maven commands for your environment and, if configured, it would build an image internally based on the Domino container image and do the build in there. That should be more reliable, particularly for macOS and Windows, but I'll have some fiddly details to iron out. Should be doable, though - it's on my hopeful list for this year.

Domino Open Liberty Runtime

The Open Liberty Runtime project is in a weird spot. It's been progressing in fits and spurts, and it's gradually moving towards a good future. My general notion for it is to make it an easier way to run tasks attached to a Domino server. So far, those "tasks" have been specifically Liberty running webapps, but the core concept is just that it's managing executables that can cooperate with the Domino server. I'd like to make this more generic, smoothing the process of running other Java-based tasks with an arbitrary JVM and potentially managing non-Java executables as well. There have been some rocky days in there dealing with the changing landscape for how open-source JVM builds are located and deployed, but the growth of the Adoptium Marketplace and its stable API looks to have settled things down for me.

Other Stuff

Beyond specific projects, I can think of a few notable public things for me this past year.

Local Library Work

Since shortly after I moved to my current house, I've been involved with the Friends of the Library group for the local branch of our library system. This past year, I also joined the overall board for the system, as its bylaws reserve a seat for a member of each of the Friends groups. This has been interesting, since, while it's still a comparatively-low-key thing, it's a bit more structured of a group than other organizations I've been involved with. It's also been a good way to keep a bit more informed about local politics without actually getting into local politics.

Also, we just officially went fine-free today, which is neat. That's one of those things where you originally think that fines are essentially required, but statistics show that they're effectively just a tax on poor kids and systems are better off dropping basic fines. I'm pretty pleased with the move.

Mastodon

I remain thoroughly pleased with my mostly-move to Mastodon. I still check Twitter, but I've been cutting down on follows over there, mostly from service accounts and those who have also made the switch. It feels like a loose tooth now, and is mostly waiting for replacement by a good Mac client. I got in the Ivory beta a while back, and slotted that right in place where Tweetbot used to be on my phone.

With any luck, Twitter will continue to be afflicted by catastrophe after catastrophe and more people will make the jump. If you haven't yet, I certainly suggest you give it a shot. I like Action Retro's video for a primer.

GitHub Sponsors

A few months ago, I set up a GitHub Sponsors profile. I figure this may be a good way to handle some cases where something I'm doing doesn't rise to the level of a whole contract or to essentially "vote" in favor of me doing the sort of open-source work I do. Admittedly, I expect I'll keep doing that regardless, but the contributions I've received so far are very gratifying.

Resolutions

I haven't been the sort of person to do New Year's Resolutions as such, but I still figure it's a nice time of year to make general plans and goals for the year. For example, I found that my gaming habits got into a bit of a rut last year - lots of Stardew Valley and Terraria, and fewer games that I haven't already played to death. I've got a growing stack of acclaimed games to play, and I'd like to work down the list, even when the games are older than dirt. Same goes for reading: considering I'm on the board of the library system, I sure don't do a lot of book reading lately. I should fix that!

Beyond that, we'll see. There are some neat things going on at OpenNTF that could set the tone for the year - in the short term, we're planning on running some "repair caf?s" this month as a way to do another kind of community interaction. I wouldn't mind doing that sort of thing more often; I've done a few "let's talk about Java" calls in the past and always enjoy them. I hope those take off enough for us to do them regularly.

In any event: Happy New Year!

New Small Project: WebFinger For Domino

Tue Dec 06 10:47:28 EST 2022

As part of moving more of my attention over to ActivityPub/Mastodon, I've seen a couple posts like this one describing how you can configure your primary domain to contain information that can lead searches to find your Mastodon account.

Most often, this is useful for people who have their own blog but use one of the larger existing Mastodon instances for their social activity. I'm not quite in that circumstance, but I still wouldn't mind it if you could look up my name as "@jesse@frostillic.us" and have it resolve to my actual account of "@jesse@pub.frostillic.us". Not a huge thing, but also seemingly not difficult to solve.

So, this morning, I set out to solve it. Before I get to the project I made, I'll make this clear: the reasonable way to do this would be to put a static JSON file up and have the "query" URL point to it. I didn't do it the reasonable way, but I didn't do it entirely unreasonably either.

How It's Done

The main mechanic in this sort of redirect lookup is WebFinger, a historically-but-disturbingly-named RFC from 2013. The idea of that is to be a HTTP-and-JSON-based equivalent for the sort of thing one used to use finger for. As is often the case, it's generalized beyond the specific need here, and it'd be interesting to implement more of its options, but for now the useful part is to be able to lookup a "resource" like acct:jesse@frostillic.us to get account info.

Mastodon provides this and related services for users in its directory, but that wouldn't help for trying to alias a user that doesn't exist in there. Since I'm already using Domino for my user store, I decided to implement a simple version of this to look up this info from Domino itself.

WebFinger For Domino

Thus was born the WebFinger For Domino project, which I expect to be a minor project with few releases. This project includes a single Servlet, listening at "/.well-known/webfinger", and it only supports lookups of account resources in the format that Mastodon uses. If you visit, for example, "https://frostillic.us/.well-known/webfinger?resource=acct:jesse@frostillic.us", you'll get output like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
{
  "aliases": [
    "https://frostillic.us"
  ],
  "subject": "acct:jesse@frostillic.us",
  "links": [
    {
      "rel": "http://webfinger.net/rel/profile-page",
      "href": "https://pub.frostillic.us/@jesse"
    },
    {
      "rel": "self",
      "href": "https://pub.frostillic.us/users/jesse",
      "type": "application/activity+json"
    },
    {
      "template": "https://pub.frostillic.us/authorize_interaction?uri={uri}",
      "rel": "http://ostatus.org/schema/1.0/subscribe"
    }
  ]
}

If you try to look up a user that doesn't exist or isn't explicitly configured to participate in this, you'll get a 404 Not Found response.

As when I added the "MastodonUsername" field to begin with, I implemented this by adding some new fields to my Person document:

Screenshot of new fields in my names.nsf Person document for WebFinger use

These fields are included directly or used to compose the URLs above. The specifics are documented in the project README.

It seems to hold together well enough, in that I was able to successfully search for my user by "@jesse@frostillic.us", and that's all I could ask. If you're using Mastodon and have your own Domino-backed domain, give the project a look.

XPages Jakarta EE 2.9.0 and Next Steps

Tue Nov 22 12:53:21 EST 2022

  1. Updating The XPages JEE Support Project To Jakarta EE 9, A Travelogue
  2. JSP and MVC Support in the XPages JEE Project
  3. Migrating a Large XPages App to Jakarta EE 9
  4. XPages Jakarta EE Support 2.2.0
  5. DQL, QueryResultsProcessor, and JNoSQL
  6. Implementing a Basic JNoSQL Driver for Domino
  7. Video Series On The XPages Jakarta EE Project
  8. JSF in the XPages Jakarta EE Support Project
  9. So Why Jakarta?
  10. XPages Jakarta EE 2.5.0 And The Looming Java-Version Wall
  11. Adding Concurrency to the XPages Jakarta EE Support Project
  12. Adding Transactions to the XPages Jakarta EE Support Project
  13. XPages Jakarta EE 2.9.0 and Next Steps
  14. XPages JEE 2.11.0 and the Javadoc Provider
  15. The Loose Roadmap for XPages Jakarta EE Support
  16. XPages JEE 2.12.0: JNoSQL Views and PrimeFaces Support
  17. XPages JEE 2.13.0
  18. XPages JEE 2.14.0
  19. XPages JEE 2.15.0 and Plans for JEE 10 and 11

Keeping with my productive week off, today I release version 2.9.0 of the XPages Jakarta EE Support project. Similar to the previous release, this one contains new features primarily related to Jakarta NoSQL, but also has some improvements for JSF and a bunch of bug fixes and compatibility improvements.

Jakarta NoSQL

The improvements to the JNoSQL driver come from some needs I came across when moving older lotus.domino/ODA-based code to using JNoSQL repositories. In particular, I added the remaining applicable view entry properties as available fields to map, added better support for reading note IDs, and fetching documents by note ID.

JSF

While JSF support remains limited by not having a proper way to add in third-party component libraries like PrimeFaces, it's still a potentially-compelling tool in an NSF as an alternative to XPages in some cases. Accordingly, I fixed a few bugs I had run into when loading pages after modifying the NSF design. Additionally, I fixed up support for JSF as an MVC view engine. It now properly joins JSP as a mechanism for rendering your output with an MVC structure, and I think there's some real potential there.

Bug Fixes and Compatibility

Most of the other closed issues deal with a few bugs here and there, and in particular involve some improvements for running apps in XPiNC and on a server with Domino Leap also installed. I don't use XPiNC anymore and haven't tried Leap, so I greatly appreciate bug reports specific to these and the assistance in tracking down the trouble.

The Future and Next Steps

I'm pondering now what the next release of the project will focus on. I have no shortage of feature ideas, and there are a few potentially-disruptive changes I'd like to make.

Unfortunately, those changes will be largely confined to improving the support for the specs that are already present and not advancing to new versions. The predicted Java-version wall arrived: Jakarta EE 10 is out and requires Java 11 and above. Since Domino remains mired in Java 8, that means that new versions of the specs and implementations are hard-incompatible until that changes.

On the plus side, there's still a lot of improvements I can make with Jakarta EE 9 as the baseline.

Reorganization

One big one I've been thinking about is a reorganization of the individual libraries that make up the project. The way it's been designed, almost every spec has its own Equinox Feature and XPage Library to go with it. This was fine early on when it was just CDI, EL, and JAX-RS, but it's grown annoying: installing the project in Designer is a seemingly-endless process of approving each plug-in at a time and the list of libraries to check in Xsp Properties is interminable. More critically, being able to selectively turn on and off specs like this doesn't make sense anymore. CDI has grown so important to Jakarta EE in general and this spec in particular that it doesn't make sense to not have it present if you're going to use this project at all. It's a foundational component of so many other parts and is essentially The Way to do Jakarta-based development.

So I'm thinking I'm going to reorganize the projects into fewer features and libraries, which will be a breaking change that will necessitate a bump to 3.0 - fortunately, the numbers line up well for that. I have a few potential options here:

  1. Just lump them all into one. You'd have basically one big switch to say "this is a JEE project" and everything would be on. The virtue here is that this is how I already work and is essentially the recommended way to do things. Additionally, as far as I know, while having additional components may slow first load (though not as much as other parts), I don't think they have a significant impact if enabled but unused during runtime.
  2. Try to line the specs up with one of the existing Jakarta Profiles. Those profiles are meant to be curated selections of useful specs, and this project has enough to implement what in newer versions is deemed the Core Profile. The trouble with this, though, is that the Core Profile is very much geared to be the shared subset with MicroProfile and similar and is a bit thin for Domino's monolith-focused development style. The Web and Full profiles, on the other hand, require "traditional" APIs like EJB that are not present in this project.
  3. Break them apart into my own "core" and "optional" features. For example, it doesn't make sense to use this without JAX-RS, CDI, and Bean Validation enabled, but JSF is entirely independent of the other specs and is among the least likely to be used in practice for now. This would also allow me to establish a running flow where "experimental" features start out as optional add-ins and then eventually make their way to core.

I'm currently waffling between #1 and #3, with a slight lean towards #1. If I can be sure that either everything or nothing is present, I could get rid of some weird hedges and workarounds, like how the JAX-RS implementation doesn't "officially" know about the CDI library yet references CDI classes explicitly by name.

New Application Types

Currently, to use this project, you can either put your code into an NSF and use the automatic behavior of the libraries or you can put your code in OSGi-based webapps or Servlets and then manually manage integration with these specs.

Both of these are limited by their reliance on the many assumptions IBM built in to how these apps should work. In-NSF apps require that all Jakarta code come from a request including "xsp" in the URL or to a file ending in ".jsp", ".jsf", or ".xhtml". If you're writing, say, an MVC-based app, all of your URLs are going to have to start with something like "foo.nsf/xsp/app/...", which is okay but ugly. Additionally, the way these apps are implemented - NSFComponentModule - severely limits my hooks for listening for things like application and session expiration, which hampers CDI's lifecycle handling a bit.

For a good while, I've pondered the notion of adding another ComponentModule type to handle the case where you want to go all-in on Jakarta EE. With this idea, the new module implementation would have full control over incoming requests, allowing URLs without the xsp/app bit in there, and would have better handling of lifecycles. In this way, I could make it so that your could would look more like (or be identical to) a "normal" .war-based webapp, with fewer workarounds for the existing XPages stuff. This would also allow me to do things like lessen the amount of Servlet 2.5-to-5.0 bridging and could assist tremendously in improving JSF support.

Along similar lines, I've been considering doing something similar for OSGi-based webapps, and I've made some progress along those lines in a feature branch. The idea here would be to do something similar to how you can deploy web.xml-based webapps via OSGi now, but with built-in support for Jakarta EE 9 features (with web.xml then being optional). With this setup, you'd be able to write an app that does an Import-Package for the various jakarta.* packages you want and add a bit in your MANIFEST.MF to signal to this project that it should participate. This could either be a variant of the extension point used by the existing OSGi webapp support or using the Web-ContextPath directive from the OSGi spec. One of the goals here would be to make it so that you would be able to write a Jakarta EE 9 app using normal development tools - Eclipse/IntelliJ/VS Code, Maven, etc. - and then just use maven-bundle-plugin to add the OSGi info you need without having any specific dependencies on Domino bits, especially the nightmare of depending on the non-redistributable XPages OSGi artifacts.

Other Options

And, in the mean time, I have a bunch of other tasks I could work on. Slowly converting my client project to Jakarta NoSQL instead of direct ODA use has turned up a whole slew of things that would be useful to add (for example, stampAll support), so I can slowly burn down that feature-request list.

There's also the notion of documentation! While a lot of the behavior of this project is in theory documented by virtue of the upstream specs and the general world of Jakarta blogs, videos, and courses, there's enough to know about the specifics of the interactions with Domino that more documentation is in order. Historically, I've just done this by expanding the README, but it's gotten pretty unwieldy at this point. It would probably make sense to break the specifics and examples out into at least wiki pages, if not a format that can be built into a PDF/etc. and included in the distribution.

So yep, I'll have my hands busy with this thing for a good while more, I figure.

More Open-Source Updates for Notes/Domino 12.0.2

Mon Nov 21 13:27:51 EST 2022

The other day, I talked about some changes/workarounds for Notes/Domino 12.0.2. Today, I made a few updates to some of the open-source projects I maintain, including another update to the generate-domino-update-site Maven plugin.

Domino Update Site Generator

In the 4.2.0 release, I added code to (mostly, as it turns out) account for HCL moving the NAPI implementation JAR down to jvm/lib/ext. In subsequent use, I found that, while that will suffice for building applications that use the OSGi dependencies, it didn't work for launching applications using it as a baseline - namely, the NSF ODP Tooling Maven plugin.

Today, I released a 4.2.1 version that improves this behavior by re-adding dependencies in the implementation bundle.

I also created a project page for it on OpenNTF. Though the project has always been hosted at the OpenNTF org on GitHub, I hadn't created a project page for it due to it just being a standalone Maven plugin. I figured it'd be useful to create an official page there for it, though.

NSF ODP Tooling

Speaking of the NSF ODP Tooling, I also found that local operations once again started crashing on macOS. Due to changes in macOS and the very weird ways that Notes works, local operations on there are a very moving target, and I have to do a lot of work in the project to account for changes to the embedded JVM and whether specific Notes versions work better with HotSpot or OpenJ9 JVMs.

Long story short, I release 3.10.0 today to account for this. Though I've found that the spawned JVM will still sometimes crash, it's after completing its work, so I considered that fine for now.

OpenNTF Domino API

Finally, I came to the OpenNTF Domino API. This project has admittedly been neglected for a little while: I'm the only active maintainer, and the client project I use it in targets Domino 11.0.1, so the 12.x builds have remained in an incomplete state for a while.

With the release of 12.0.2, I decided I should finish the wrappers for the new classes added in 12.x, so I did so and uploaded a build for 12.0.2. This primarily adds those wrappers, but also included a contributed fix and changes the distribution packaging to combine the XSP and non-XSP versions.

My 2021 Open-Source Year

Fri Dec 31 16:34:27 EST 2021

For the last few weeks, I had a minor flurry of work in a couple of the open-source projects I maintain, and I figured this would be as good a time as any to give an overview of my active work in these projects and how they relate.

Overview

I had a few minor contributions and picked-up projects through the year, but most of my currently-public work went towards four main projects:

I do find it interesting to consider how these relate. Some aspects are easy: they're all Domino-related for sure, and they all at one time or another have played a significant infrastructural role in my client work. Beyond that, though, they form a nebulous message: though I don't know for sure what to do with all the XSP markup we have, I know it can't be the status quo and I'm fairly confident that Jakarta EE is the best route forward.

Domino Open Liberty Runtime

This project allows you to run instances of Open Liberty as a spawned process from Domino, which in turn means both that you can readily(-ish) access Domino data and also that you can deploy these apps in an NSF-based way to your servers, without having to have particular mastery of Liberty administration as such.

The big-ticket news this year was my addition of a Domino-hosted reverse proxy and arbitrary JVM selection. With these additions, the project ended up being a particularly-compelling way to glom modern apps on to Domino without even necessarily worrying about pointing to a different port. I also added in the standalone proxy to both the apps and Domino - which would gain you Web Sockets and HTTP/2 - which is another nice way to get better app toolkits without having to bother an admin.

XPages Jakarta EE Support

This one saw a burst of activity in just this past month. For a while, it had sat receiving only minor tweaks: I use it for EL, CDI, and JAX-RS in my client project, and the changes I made were generally just to add features or fix bugs needed there.

This month saw the big switch from Java/Jakarta EE 8 (javax.* packages) to Jakarta EE 9 (jakarta.* packages). This was a very-interesting prospect: though it on paper just involved switching class names around, it necessitated adding some Servlet 5 shims around Domino's irreponsibly-old Servlet 2.4/2.5 hybrid layer. While this didn't bring full Servlet 5 features, it does mean I'm suddenly much less bound by the strictures of the older version: a lot of Servlet-based software casually depends on at least 3 even for just convenience methods (like getting a ServletContext from a ServletRequest).

I also took the opportunity to go back and add some features I've long wanted - JSP and MVC - to the NSF side. These have less immediate call in my client work (which primarily involves additions on the OSGi servlet layer and less in the NSF), but suddenly created a surprisingly-compelling update to in-NSF development. It's stymied by, naturally, a lack of support in Designer, but the idea of writing something that approaches a true modern Jakarta app inside an NSF is intriguing indeed.

NSF ODP Tooling

The NSF ODP Tooling has proven to be my workhorse. The ODP-to-NSF compilation alone has saved me countless hours from the previous laborious task of syncing two dozen NSFs with their ODPs and the fault-prone process of trying to get clean NTF copies of them for each build. Now, the former is done with a single script I can run in the background and the latter happens automatically every single push to our Git repository. Delicious.

It also provides an invaluable part of my normal development process for this client. Alongside the next project, it lets me do my XPages development outside of Designer, meaning I only need to schlep my way back to that IDE to look at legacy elements in context or to troubleshoot something with the Notes or Domino OSGi view of the world.

The work in this project this year has primarily been around edge cases, bug fixes, and scrambling through the rocky shoals of the ever-changing macOS Notes client. It's been a tough time here and there: certain parts of the NSF that I use less frequently have their own edge-case needs (like SSJS sort of existing in two places and the CD storage being surprisingly difficult to work with. I also had some fun combat with filesystems and charsets, which was fortunately even-more enlightening than it was annoying.

XPages Runtime

The XPages Runtime project admittedly had a slow year, but it's nonetheless a critical component in my CI/CD workflow, and gets periodic fixes for trouble I run into. The good news there is that it generally does what it promises: I run XPages outside of Domino constantly with this thing. Though it still requires more coordination on the app side than I'd like, it's gradually approaching a state where it feels like a peer to other server-side toolkits that one can bring into a WAR file, and that's nice.

It will likely have some work coming up in the near future, though: if I'm to move my client's app over to the jakarta.* namespace, that will require at least some level of cooperation with this project. While I can't change the source of XPages to accept these coordinates itself, it should be doable to do much like what I did with the XPages Jakarta EE Support project and use my shims to translate back and forth between old and new classes. The main difference here will be that the surrounding container will speak the new form natively, but that should be fine.

I expect a certain amount of annoying trouble with things like XPages-internal expectations about JAX-B and JavaMail, and it's certainly possible that such dependencies will end up proving to be debilitating, but I'm optimistic. If I'm successful, it'll be one more way that I'm crafting a whole workflow where modern technologies are the primary target and XPages can remain a component in the lineup.

Miscellaneous Grab Bag

Beyond those big ones, I had a handful of other contributions here and there. I'm sure there were a few others, but I'll close on two that I found pleasing.

The other week, I got a Pull Request merged into the Eclipse Krazo project - while not a huge deal, it does always give me a little thrill when my code goes into a project where I'm not the primary or sole contributor.

I also adopted POI4XPages, which was for more-practical reasons. I've used POI4XPages for a couple clients for a while, but it was certainly showing its age (sitting at 3.x since 2017). Moreover, Notes 11's corruption of its classpath with POI 4.x made working with it annoying beyond just being out-of-date and lacking some breaking changes in the mean time. Since I had moved one of my clients to POI 5.0 a bit ago, I decided to break that code out and adapt it into POI4XPages. Then, of course, along came Log4Shell and I scrambled out three subsequent patch versions just to update log4j. So it goes.

My Active Open-Source Projects

Fri May 08 11:01:42 EDT 2020

Over the years, I've spawned a number of open source projects, both in my personal GitHub account and in OpenNTF's, but it'd be fair to say that not all of them are actively updated or see common use.

Nowadays, I have a set of tools that I actively develop (either solo or with a team) and which make up critical parts of my development infrastructure, and I figured it'd be useful to give an overview of them.

NSF ODP Tooling

This is my current favorite project by virtue of how much time it saves me every day and for its future potential. I wrote a series on this project a while ago, so I won't go over all the details of it here. The gist of it, though, is that this project lets me have a Maven tree for one of my big client projects that includes an array of OSGi bundles and have the Maven install project build all of those, assemble an update site with them and a bevy of dependencies, compile over a dozen NSFs (most with complicated Java code), and end up with a distribution ZIP containing importable update sites and deployable NTFs, all from my Mac with no Designer involved.

I have visions of this project forming the central infrastructure for a post-Designer world, and that's shaping up in a couple ways so far. One of those ways is the DXL and XPages LSP contributor component that allows for pretty-solid editing of, uh, DXL and XPages in tools that use the XML Language Server, such as Eclipse and Visual Studio Code. And that plays in to the other project I use daily, the XPages JEE Runtime.

XPages JEE Runtime

This is the project that started as a frenzied descent into madness and which I eventually hammered into shape enough to run real apps (with a side path where I also got XPages running on Android and iOS).

Now, this is the main way I do development on that client app. I have an Open Liberty server set up in Eclipse and a webapp variant of the XPages app that points to the same XPages, Custom Controls, and Java code from the NSF's ODP representation, and I have some hooks to direct all database references to the DB running in my dev VM. Since it's not a 100% perfect representation of the Domino environment, I still need to periodically sync it back to the NSF and test how it runs in there (and with the OSGi environment that I'm not using in the webapp), but I'm experienced enough at this point to generally know the potential pitfalls.

There's also a dark part of me that keeps being tempted to actually use this for production at some point, since it works so well now, and pushes aside so many hassles of loading and deploying on Domino itself. That would play in to the next project, the one that's hosting this very blog right now.

Domino Open Liberty Runtime

This is my project where I set up a sidecar Open Liberty instance alongside Domino, which allows for using native local NSF access while also having a full, modern Jakarta EE server with all the bells and whistles.

Though this project is a bit more staid than some of the others, I've gone in and made some interesting improvements lately. One was my journey into RunJava the other month, which I still think is a little too cute to put into production, but which actually should do the job just fine.

The other improvement, though, has some more immediate benefits. I added the ability to specify and auto-download AdoptOpenJDK Java runtimes to use instead of Domino's provided JVM. These runtimes still gain the same benefit of running with local Domino NSF access, but aren't constrained by Domino's once-again-long-in-the-tooth JVM. So you can, for example, specify that you'd rather bring in Java 14 and the runtime will auto-download it for you and launch Liberty using that. I haven't quite rolled that one out to this blog server yet, but it's on the docket. I'd love to bring in Java records, for example, and now there's nothing stopping me from doing so.

XPages Jakarta EE Support

I didn't have a good segue for this one.

This is a project I started a couple years ago initially as a way to expand on Martin Pradny's original plugin to make writing JAX-RS resources inside an NSF easy. It's grown into my project to essentially try to bring the XPages runtime up to code, at least in the parts that I want to use for work. Though it's constrained by the hard limit of the ancient Servlet API Domino's container provides, I've been able to bring in some important updates for EL and JAX-RS, and also allow for using CDI for managed beans and JAX-RS resources.

CDI is actually a whole huge topic that I have some draft posts for. As far as Java development is concerned, CDI is Important with a capital "I".

ODA

There's not a lot of fanfare with the OpenNTF Domino API, but that's largely intentional: as an improvement on the normal lsxbe API, it does its job and doesn't currently need any radical changes. I'm mostly including it here because, though it doesn't change much, it's periodically updated to cover the sprinkling of new Java methods HCL adds with each release.

generate-domino-update-site

While I don't use this project as such daily, I sure do benefit from its output. This is the Maven plugin that generates new update sites, which is required for up-to-date OSGi development for Domino in lieu of IBM/HCL ever updating their own release.

Other than being something I run every new Domino release, I've also made some improvements recently. Some of those just related to improving behavior in edge cases, but a nice one I added the other week was downloading of source components from Eclipse Neon. Though the source for the XPages runtime and the whole Expeditor scaffolding remain unavailable, I am able to look up and download the source for the unmodified Eclipse components, and this results in a more-pleasant development experience in Eclipse.


I have a few other projects that I use periodically, such as the NSF File Server, but those are the big-ticket ones.

Winter Project #2: Maven P2 Repository Resolver

Sat Dec 28 14:12:26 EST 2019

  1. Converting Tycho Projects to maven-bundle-plugin, Initial Phase
  2. Winter Project #2: Maven P2 Repository Resolver
  3. OpenNTF Fork of p2-maven-plugin
  4. The Intricate Work of OSGi Dependencies on Domino

The second project I took on this past week was related to the first, and also relates to my ongoing struggles with Tycho.

While working on the NSF ODP Tooling, I figured that it could be a good candidate to move away from Tycho and to maven-bundle-plugin or bnd directly. Since I've been Mavenizing the Domino OSGi bundles for a good while now, and the tooling doesn't have any OSGi-dependent tests in it, it seemed like it could go smoothly. Unlike historical precedent, though, my enemy in this endeavor wasn't Domino, but rather Eclipse.

Repository Layouts

One of the big things that makes working with OSGi bundles - at least specifically ones in the Eclipse style - difficult with Maven is that they're generally provided using a repository layout called "p2". This is the evolution of the "site.xml" Update Site style and shares a lot of characteristics. In fact, a p2 repository will often have a "site.xml" file alongside its "artifacts.xml" and "contents.xml" (often Jar'd up) to provide backwards compatibility. It's how we package up XPages plugins and how once upon a time IBM provided [the XPages artifacts for Tycho use](https://openntf.org/main.nsf/project.xsp?r=project/IBM Domino Update Site for Build Management). As a live example, Eclipse 2019-12 is distributed via such a repository.

Maven has its own repository layout, variously called "Maven", "Maven2", "m2", or just "default". This serves a similar purpose, but is structured differently - whereas p2 just has the repo and its "features" and "plugins" directory (and, potentially, composite repositories) - Maven's repository system is organized like a conceptual folder tree based on translating a Group ID (like, say, "org.openntf.maven") into a successive series of subdirectories (like "org/openntf/maven"), followed by a directory for an Artifact ID, which in turn contains directories for each version, and finally within there are any of the actual files that make up a given named "artifact". As a live example, Maven Central is browsable in this manner.

Translating Between Them

Though the two layouts share a common core job - hosting Jar files (mainly) - they diverge enough in how the tools expect the metadata to be laid out that they're difficult to mesh. Tools like bnd can often work with whatever, and even Tycho can try to find OSGi bundles via Maven dependencies, but it's not smooth.

Over time, a pseudo-standard of adding p2 repositories to Maven has emerged, but it's only actually used as a marker to pass along to true Eclipse tools. The most common in our sphere is this construct, seen in projects like ODA:

1
2
3
4
5
<repository>
	<id>notes</id>
	<layout>p2</layout>
	<url>${notes-platform}</url>
</repository>

There, you reference the XPages update site somewhere, and then Tycho can use that to resolve dependencies for things like Require-Bundle: com.ibm.xsp.core. However, it's used only for Tycho's specific OSGi needs. You can't bring in the Tycho plugins and then have a non-Tycho module in your tree declare a dependency like that. Tycho has an implementation class for this, but it's intentionally stubbed out.

My Needs

The reason why tools that can work with both are so heavily slanted to the specific task of generating a true OSGi environment is that that's usually what you want. If you're designing, say, an Eclipse plugin or Eclipse-derived product, you want all of your tooling to know about OSGi from top to bottom, and that's where Tycho excels. It makes sure that all of your dependencies are correct and everything is OSGi-friendly.

This is as opposed to something like maven-bundle-plugin, which is most typically used to put an OSGi coat of paint on a project that isn't primarily geared towards OSGi.

I kind of want a middle ground, though. A project like the NSF ODP Tooling has grown into a sprawling hydra, with heads for Maven, Eclipse, Domino, and now Visual Studio Code. While that works, putting Tycho at the front of it ends up feeling needlessly proscriptive, and I'd love to toss it aside. However, I was blocked in my desire by a small thing: though Eclipse publishes their core bundles on Maven nowadays, Wild Web Developer is currently p2-only.

The Project

So I set out to solve this problem for myself and learn something in the process. As indicated by the <layout>p2</layout> option on the repository above, Maven's repository system is intended to be extensible. Unfortunately, it seems like it hasn't been extended particularly often in practice, and most of what I could find about it was that it's possible to do, but only via references to people saying that one could.

Fortunately, it turns out that it's actually not too difficult to implement after all, and I did just that.

What this Maven plugin does is allow you to specify p2 repositories in any old Maven project, using the ID of the repository you add as the Group ID of dependencies and then the Symbolic Name as the artifact ID. Now, with the plugin added to the project, I'm able to reference the p2-only Wild Web Developer artifact I need:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
<repositories>
	<repository>
		<id>org.eclipse.wildwebdeveloper</id>
		<url>https://download.eclipse.org/wildwebdeveloper/releases/0.8.0/</url>
		<layout>p2</layout>
	</repository>
</repositories>

<dependencies>
	<dependency>
		<groupId>org.eclipse.wildwebdeveloper</groupId>
		<artifactId>org.eclipse.wildwebdeveloper.xml</artifactId>
		<version>[0.5.0,)</version>
	</dependency>
</dependencies>

And, just like that, the bundle and its explicit dependencies show up in my Maven Dependencies group in Eclipse:

p2 Maven Dependencies in Eclipse

As a side bonus, this obviates the need for the mavenizeBundles half of the generate-domino-update-site project, since now I can just reference the generated site directly and get the dependencies, including with better behavior for embedded Jars than I had there:

com.ibm.xsp.core Dependency in Eclipse

The Tiny Details

I think that this plugin is about where it needs to be to suit my needs, but there is still an array of [tiny details I've yet to contend with](https://github.com/OpenNTF/p2-layout-provider/issues?q=is:open is:issue). It'll never be quite a perfect match for an arbitrary OSGi bundle (though some also contain useful Maven metadata), and so there will always be rough edges with something like this, but I think that it will solve a lot of headaches I'd otherwise have to deal with down the line.

If you think it'd be useful for your projects, take a look and let me know if you run into any trouble.

Writing Domino JEE Apps Outside Domino

Fri Nov 08 13:47:13 EST 2019

In my last post, I mentioned that I decided to write the File Store app as a Jakarta EE application running in a web server alongside Domino, instead of as an app running on the server itself. In the comments, Fredrik Norling asked the natural question of whether it'd have been difficult to run this on the server, which in turn implies a lot of questions about deployment, toolkits, and other aspects.

Why Not

My immediate answer is that it would certainly be doable to run this on Domino, but the targeted nature of the app meant that I had some leeway in how I structured it. There are a couple reasons why I went this route, but most of them just boil down to not having to deal with all the gotchas, big and small, of doing development on top of Domino.

As a minor example, I wanted to add some configuration parameters to the app, and for this I used MicroProfile Config. MicroProfile Config is a small spec that standardizes the process of doing key-value-based configuration, allowing me to just say that I want a named value and let the runtime pick it up from the environment, system properties, or a config file as necessary. It wouldn't be difficult to write a configuration checker, but why bother reinventing the wheel when it's right there?

Same goes for having the SFTP server load at startup and be running consistently. If I did this on Domino, I could either put it in an NSF-based XPages app with an ApplicationListener and depend on the preload notes.ini parameter, or I could go my usual route and use an HttpService that manages the lifecycle. Neither route is difficult either, but they're both weirder and more fiddly than using servlet context listeners in a normal JEE app.

And then there's just the death by a thousand cuts: needing to use Tycho to build, having to deal with Eclipse Target Platforms, making sure anything using reflection is wrapped in an AccessController block to avoid Java policy issues, the nightmare of dependency management, the requirement to use either Designer or the okay-but-still-cumbersome Domino HTTP restart development cycle, and on and on. With a JEE app, all of those problems disappear into thin air.

The Main Hurdle

All that said, there's still the hurdle of actually implementing Notes native API access outside of Domino. At its core, what I'm doing is the same as what has been possible for years and years, initializing a Notes/Domino runtime in a secondary process. The setup for this varies platform by platform but generally involves either setting a couple environment variables for your process or (as is the case with the Domino Open Liberty Runtime) spawning the process from Domino itself.

Beyond setting up your process's environment, there's also the matter of initializing each thread of your app. On Domino, all threads are inherently NotesThreads, but outside of that there's some specific management to be done. You can call NotesThread.sinitThread() and NotesThread.stermThread() manually or run your code in specifically-spawned NotesThreads. I largely do the latter, making use of an ExecutorService to handle maintaining a thread pool for me. I then added some supporting code on top of that to let me run blocks of code as an arbitrary named user while retaining a cached set of sessions. That part wouldn't really be necessary if you're writing an app that doesn't need to act as different user names, but it's handy for something like this, where incoming connections must run as a user to enforce security fields properly.

Philosophical Advantages

Beyond my desire to avoid hassles and get to use modern Jakarta EE goodies, I think there are also some more philosophical advantages to writing applications this way, and specifically advantages that line up with some of HCL's stated long-term goals as well. Domino has long been a monolith, and that has largely served it well, but keeping everything from the DB all the way up through to the app-dev stack in the same bag means you're often constrained in your toolkits and deployment choices. By moving things just outside of the main Domino tower, you're freed up to use different languages and techniques that don't have to be integrated and maintained in the core. This could be a much larger jump than I'm doing here, and that's just what HCL has been pushing with the AppDev Pack and the associated Node.js domino-db module.

I think it's beneficial to picture Domino more as a dominant central core with ancillary servers and apps running just adjacent to it - not a full-on Microservices architecture, but just a little decentralized to keep areas of concern nice and separate. Done well, this setup is a lot more flexible and fault-tolerant, while still being fairly straightforward and performant. It's also a perfect match for a project like this that's geared towards implementing a new protocol - it doesn't even have to worry about HTTP SSO or reverse proxies yet.

So I think that this is where things are heading anyway, and it's just a nice cherry on top that it also happens to be a much (much, much) more pleasant way to write Java apps than OSGi plugins.

Another Side Project: NSF SFTP File Store

Tue Nov 05 16:12:58 EST 2019

When I Know Some Guys kicked off, we bought a couple of Transporter devices to handle our file-syncing needs without having to rely on Dropbox or another hosted service. Unfortunately, Nexsan killed off Transporters a couple years back, and, though they still kind of work, it's been a back-burnered project for us to find a good replacement.

Ideally, we'd find something that would handle syncing data from our various locations transparently while also allowing for normal file access through some common protocol. Aside from the various hosted commercial services, there are various software packages you can run locally, like OwnCloud and NextCloud. I even got a Raspberry Pi with a USB hard drive to tinker with those, though I never got around to actually doing so.

Yesterday, though, I realized that we already have a fleet of privately-owned servers that replicate seamlessly in the form of our Domino domain. They also, conveniently, have nice capabilities for blob storage, shared user authentication, and fine-grained access control. What they didn't have, though, was any good form of file protocol. I'm pretty sure that Domino still has WebDAV built-in, but that's just for design elements. Years ago, Stephan Wissel created a project that works with file attachments, but that didn't cover all the bases I wanted and I didn't want to adopt the code base to extend it myself. There's also Karsten Lehmann's Mindoo FTP Server from around the same time, but that was non-SSH FTP and targeted at the local filesystem.

So that meant it was time for a new project!

The Plan

I initially looked at WebDAV, since it's commonly supported, but it's also very long in the tooth, and that has led to all of the projects implementing that being pretty old and cumbersome as well.

Then, I found the Apache Mina project, which implements a number of server protocols, SSH included, and is actively maintained. Looking into how its SFTP support works, I found that it's shockingly simple and well-designed. All of the filesystem access is based on the Java NIO packages added in Java 7, which is a pluggable system for making arbitrary filesystems.

Using SFTP and SCP means that it'll work with common tools like Transmit and - critically - rsync. That means that, even in the absence of an custom app like Dropbox, mobile access and syncing with a local filesystem come "for free".

The Project

So out of this was born a new project, NSF File Server. Thanks to how good Mina is, I was able to get a NIO filesystem implementation and SFTP+SCP server up and running in very little time:

Screen shot of the SFTP server in Transmit

In its current form, there aren't a lot of tricks: the files are stored as attachments to normal documents in a "filestore.nsf" database with two views, which allow for directory-contents and individual-file lookup while also being pretty self-explanatory to a Notes client. I have some ideas about other ways to structure this, but there's an advantage to having it be pretty basic:

Screen shot of the File Store NSF

Similarly simple are the authentication mechanisms, which allow for both password and public-key authentication based on the HTTPPassword and sshPublicKey fields in a person document, respectively (and maybe via LDAP in directory assistance? I never remember the mechanics of @NameLookup).

The App

Because this is a scratch-our-itch project and I'm personally tired of dealing with Domino's OSGi environment, the app itself is implemented as a WAR file, expected to be deployed to a modern Jakarta EE server like my precious Open Liberty. Conveniently, I have just the project for that as well, making deploying NSF-accessing WARs to Domino a bit more reasonable.

For now, the app is faceless: the only "web app" bits are some listeners that initialize the Notes environment and then spawn the SSH server. I plan to add at least a basic web UI, though.

Future Plans

My immediate plan is to kick the tires on this enough to get it to a point where it can serve as its original goal of a syncing SFTP server. I do have other potential ideas in mind for the future, though, if I feel so inspired. Most of my current logged issues are optional enhancements like POSIX attribute support, more-efficient handling with the C API, and better security handling.

It's also a good foundation for any number of other interfaces. A normal web UI is the natural next step, but it could easily provide, for example, S3 API compatibility.

For now, though I haven't gotten around to uploading a build to OpenNTF yet, feel free to poke around the code and let me know if any ideas strike your fancy.

Anatomy of a Clean Open-Source Project

Mon Mar 18 11:54:47 EDT 2019

Tags: open-source

Over the years, initially thanks to Peter Tanner's diligent work as the OpenNTF IP Manager and now my own occupation of that seat, I've learned to really appreciate the virtues of dotting your "i"s and crossing your "t"s when it comes to making an open-source project legally clean and clear.

It's definitely something I underrated early on, though - caring about the specific differences between licenses and, in particular, maintaining things like per-file license/copyright headers felt like annoying busywork. For a project that only you will ever use, it technically is, but the hope of open source is that you'll get other people using your work and, ideally, contributing back in turn, and that's when it's important to make sure you have everything sorted out.

Why Bother?

Well, for one, you or your users could theoretically be sued or otherwise legally entangled if you don't keep track of this stuff. Admittedly, it's fairly unlikely, but the consequence of, for example, unknowingly including GPL software in your proprietary product is potentially significant.

It's for that sort of reason that it's important to make sure everything is clean before some large corporations will risk even looking at your project. IBM is particularly good about this because they were significantly burned in the past, and came out of it with extremely-strict view, and that rubbed off on OpenNTF both culturally and with their gracious technical assistance along the way.

And, since large consumers require this sort of vetting, it's also important to know how to do it if you want to contribute code to an open-source organization like OpenNTF, Eclipse, or Apache.

It's also surprisingly satisfying once you get into the swing of it, I've found.

The Example Project

Since I've been spending a lot of time recently with the NSF ODP Tooling project, we'll look at its GitHub repository.

Common Files

There are a couple common features that tend to show up, and which both people and tooling (like GitHub's license identifier) look for:

  • The LICENSE file, which is the most critical. This contains the text of the license you're using, as well as one of the declarations of the copyright year (though, admittedly, it's easy to forget to include that part). This is what declares the effective license for the code in the repository that you own the copyright to, and should be included right from the start if possible.

  • The NOTICE file, which is vital if you're including any code from any sources not covered under the main copyright. This file should list all of the third-party code you have included in the repository, its license type, and, if possible, where to acquire it. If your project's distributable form includes additional third-party code not included in the repository (such as Maven or npm dependencies), these should be enumerated here as well

    • Writing this file has an important side effect in that it forces you to account for the licenses of your dependencies. More than once, I've run into a situation where I found that a common dependency had an incompatible license (such as the pure GPL). In some cases, this has meant abandoning the dependency outright, while in others it has meant finding a better-licensed alternative. Eclipse Orbit exists in large part for this purpose.
  • A legal directory containing any additional license/redistribution information not covered by the NOTICE. This can also sometimes take the form of files like NOTICE-Weld in the root of a project, and is useful for mass-including copyright/notice information from third parties in their original form.

In addition to including these files in the project repository, you should also make sure to include them in any binary distributions you make. In my projects, this takes the shape of inclusions in a Maven Assembly Plugin packaging file.

File Headers

I originally chafed against the idea of per-file copyright/license headers. They're not strictly necessary when the files are included in the original repository, they're redundant, you end up with massive commits touching hundreds of files just to change a year, and they can dwarf the size of the actual code they're copyrighting.

However, I've really come around to the practice of including them, and the main reason is that it makes the files easier for others to copy and use legally. It's one thing when someone finds their way to the root of your repository or distribution package, but it's another when they find an individual class by doing a web search or hitting F3 in Eclipse. In those cases, they can find their way up to the license (assuming your source package includes it), but it's much easier if it's just declared right up front.

It's also easier to clearly distinguish the third-party code you're including. When each file has its copyright information clearly noted, you can easily tell the difference between a sui-generis project file and an included third-party file without having to parse through the NOTICE every time.

And, fortunately, it doesn't have to be a huge hassle to maintain. In each of my Maven projects, I include a license plugin configuration to declare copyright information, any special data types, and which files to not include. Then, whenever I add new files or make a change after the turn of a year, I can run mvn license:format and it'll keep everything tidy for me.

pom.xml Configuration

Maven (and it's not alone in this) provides a lot of pom.xml-level elements to declare all sorts of metadata about your project, like its SCM repository, issue tracker, and, critically, license and developers. I like to declare the inception year, the license, and the <developers> block:

	<inceptionYear>2018</inceptionYear>

	<licenses>
		<license>
			<name>The Apache Software License, Version 2.0</name>
			<url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
		</license>
	</licenses>

	<developers>
		<developer>
			<name>Jesse Gallagher</name>
			<email>jesse@frostillic.us</email>
		</developer>
	</developers>

I use that <inceptionYear/> value as part of the license-header file to keep track of the copyright range, at least when there's contiguous multi-year development.

OSGi Stuff

Since most of my projects are still OSGi, I've been aiming to improve my licensing setup there too. The main place where this comes into play is in feature.xml files, which have required elements to specify the copyright and license. It's not terribly unusual for these to end up as their "[Enter Copyright Description here.]" defaults, but it's important to fill these in. They're included in the "accept the licenses" dialog when installing into Eclipse/Designer, and are available in the "Installed software" descriptions in the UI.

But What License to Use to Begin With?

I'll finish off this post with what is actually the most important part of the process, but which can usually be answered simply. There are a lot of open-source licenses out there, and you could theoretically make up your own, but for our purposes the choice tends to follow some basic rules:

  • If you're contributing to an established OS organization, use theirs - for example, if you're contributing to Eclipse, use the EPL.

  • If you want your code to be mixed other OS projects and (potentially) proprietary ones, pick Apache or something like it.

    • At OpenNTF, we have a preference for Apache over other similar licenses, because it's well-established and makes copyright handling clearer than the equivalents, something that is critical for large companies. Let past lawyers do your legwork on this one.
  • If you want to require that users of your code keep the code open source, consider the GPL.

    • Be extremely wary of this, however: the GPL is intentionally "infectious" and limits how the code can be used. Various projects carve out little exceptions to the GPL to allow use in otherwise-non-GPL products, but it's still something of a minefield.
    • The GPL is one of the approved licenses for OpenNTF, but we kind of discourage it except in cases where a project is GPL because it's derived from previously-GPL'd code.
  • If you don't want to be bothered too much by copyright and just want the code out there, consider Public Domain. In practice, it's usually best for you to retain copyright, but explicitly declaring Public Domain is certainly an effective way of allowing any use.

For projects in our community, the quick answer is "use Apache". It's permissive, covers copyright, and is known and trusted by pretty much everyone.

More Work Than It's Worth?

Both the earlier parts of this post and Betteridge's Law contribute to making it clear that my answer is "no, it's not more work than it's worth", but I can certainly see why it'd feel that way. The first couple times I submitted projects to OpenNTF and got a "here's some stuff to fix" email from Peter Tanner, part of me definitely chafed at the whole thing. That can be particularly the case for Notes-based contributions - sometimes, you just want to plunk an NTF on the project page and be done with it, and Notes certainly doesn't have a "wrap this NSF copy in a ZIP with LICENSE and NOTICE files" checkbox.

However, as I learned more about the legal importance of having licenses correct and got more practice at doing this stuff from the start, I started to appreciate the whole process. It also turned out to be really helpful to sort this stuff out on smaller projects before working on larger ones, especially ones with established teams and procedures.

In all, it's worth it both to allow you to contribute to larger projects and, regardless of project size, it's worth it for anyone consuming your code.