In this probably-final entry in the series, I'd like to muse a bit about possible future improvements and additions to the compiler and the NSF ODP Tooling generally. For the most part, the big-ticket future additions seek to answer one question:

Could this be used to replace Designer?

The quick answer is "yes, it could", but that would take a lot of work. There are a couple things inherent in the task and specific to my implementation that both help and hinder this kind of thing.

Notes Runtime

The biggest stumbling block is the hard requirement on a Notes or Domino runtime initialized for the current process. Being able to use C API calls is required by both my code and some of the underlying XPages bits, and that means initializing the runtime. The good news here is that it doesn't require any specific Notes-based program - it can be run with either the libraries that come with Notes or Domino, and it doesn't require Designer at all. That loosens things up a bit, but still means that one of the supported platforms is obligatory at some step of the process.

Even on a supported platform, though, it's not just as easy as calling an init function - the process's environment needs to be set up specifically to know about the Notes program and data directories, and this varies platform-by-platform. This means that it wouldn't be straightforward to have, for example, an Eclipse plugin that initializes the process, since it would depend on initialized environment variables and loading paths implicitly referenced by lower layers, and over which the programmer doesn't have much control after the fact.

The good news here is that the tooling is already designed to support remote work for compilation and export, both truly remote and with the local Equinox runners. For a true IDE experience, the communication between the IDE and compiler would have to be more complex than the "tell the compiler what to do and hear messages back" simple mechanism it has now, but it'd still be a natural evolution.

OSGi Runtime

The requirements posed by compiling complicated XPages applications presents a similar dependency as above, but on an Equinox environment. Though it's possible to fake the basics of OSGi for known plugins, that wouldn't work for arbitrary third-party libraries.

For integration in Eclipse, this wouldn't necessarily mean any new work - Eclipse is already the premier Equinox product, and so it supports what XPages compilation needs innately. However, Eclipse wouldn't be the only target; any work done here should work with other IDEs like IntelliJ, but also continue to work IDE-free via a Maven or Gradle environment.

So this ends up being another strong argument for retaining the "separate process" model that already exists.

Incremental Compilation

Beyond retaining the runtime requirements, the big thing would be a switch to supporting incremental compilation. Currently, the compiler is designed to do everything in one pass: you point it at an ODP and it spits out a freshly-created NSF. This allows it to build up and tear down its environment cleanly, initializing the OSGi plugins for any XPages libraries at the start and doing similarly for any custom classpath jars to be included in the Java runtime.

What supporting incremental compilation would require to be at all speedy and efficient is having a persistent compilation environment. Instead of everything happening sequentially, the IDE would init the compiler process and then send it requests as files need compilation. This has implications for both local and server-based compilation.

Local complication would need to change less: mostly, it would require picking an IPC mechanism and having the launched Equinox process remain alive until it's no longer needed.

Server-based compilation would be similar in implementation, probably using something like HTTP long polling to be able to run in the Domino HTTP container. The trouble would be that a straightforward implementation of this would mean that the Domino server would pretty much have to be dedicated to a single IDE. There's already a potential conflict scenario with two developers doing compilation at the same time: since the XPages compiler needs to install and uninstall OSGi bundles, they could step on each others' toes if any of them overlap. Keeping the compiler environment resident on the server would mean it would have to be effectively locked out to one connection for long periods of time. Assuming HCL continues the Community Edition licensing model, this will be legal to do, but it's still cumbersome.

This could lead into something I've been mulling over: running a Domino compiler server in Docker. This would loosen a lot of the runtime requirements and mean that the encapsulated Domino server would be both dedicated to the purpose and consistent from the perspective of the compiler. Domino's setup requirements initially made it an awkward fit for Docker, but it looks like things have progressed along nicely.

This would all tie perfectly into the Language Server Protocol, which is an IDE-agnostic way to do basically this: have a little running process that knows about the nitty-gritty of the language, and then tell the IDE only what it needs to auto-complete and other features.

Live NSFs

Currently, the compiler starts with an ODP and emits a clean NSF with each build, and this is absolutely, 100% the correct way that it should work. However, Notes being what it is, it'd be expected that a Designer replacement would be able to work with a live NSF, so you could just crack one open, change a view, and be done. The second part of this process is in there, since the compiler uses the normal Notes APIs to store in an NSF as it is. It's the first part that would have to be new, allowing the tooling to selectively look into an NSF.

The exporter already does this, but, like the compiler, goes in one pass. What would potentially make sense would be to do essentially what Designer does: implement a VFS layer to represent an NSF in an equivalent way to the on-disk project. It's more easily said than done, but would be particularly straightforward for Eclipse, for the same reasons that it was straightforward for IBM to do it for Designer.

The secondary question here would be if it would be better to do continue to use DXL as the sole transport mechanism (so editing a view in a live NSF would export it to DXL and then re-import on save) or to instead try to represent things differently. Though DXL is less efficient, particularly for large notes, I think it'd make sense to stick with it - there would be tremendous work involved in trying to make it smarter, and that would be a breeding ground for bugs that just wouldn't exist with DXL.

IDE Features

Getting an NSF to compile dynamically is one thing, but the other part of this kind of project would be making the experience of working with design elements pleasant. In Designer, we have the benefit of having purpose-built editors for each design element type, but these aren't portable even if licensing allowed: the legacy ones all are just wrappers around C++-based "native" UIs, while the newer-era ones are based on Designer's bizarre internal RPC system.

I've done some work along these lines, initially to add autocomplete for custom controls and known core+ExtLib controls to .xsp files. Since that earlier work, I also added a contributor that tells Eclipse to use the DXL schema file for DXL files. While this doesn't give a proper GUI editor, it does provide enough information for Eclipse to pick up on the allowed elements and properties:

While I don't think it'd be worth trying to fully reproduce the various WYSIWYG editors Designer gives you (particularly the view editor, which is laughably bad for data-centric use), I think it'd be worth adding some editors along the lines of my old Forms 'n' Views project. Having some basic editors with a strong focus on the resulting data structure would be perfect for XPages support use and even mostly useful for legacy use.

Time

The core trouble with getting to all these goals, though, is time. For the main compilation and export work, I could justify spending a good amount of time because it eventually more than paid off in less time fighting with Designer to create consistent builds. For this other stuff, though, it's more dependent on whether my hatred for using Designer is enough to tilt the scales. Sometimes, it almost gets there, but I do also need to be able to pay my mortgage, so that puts a bit of a limit on things. It sure would be nice to leave Designer in the dust for good, though.