A year ago, I started a project to compile NSFs - and particularly large XPages projects - independently of Designer. Since the initial releases, the project has grown, gaining an ODP exporter, extra Eclipse UI integration, and the ability to run without installing components on a remote server. It's become an integral part of my workflow with several projects, where I include the NSFs as part of a large Maven build alongside OSGi plugins and a final distribution.

Building this tooling required learning a lot about the internals of XPages, the specifics of how various design elements are stored and handled in an NSF, and miscellaneous bits about Equinox and Maven. Since there's a good amount of arcane knowledge embedded in the project, I think it'll be helpful to take some time to dive deep into what's going on, starting with XPages.

XSP to Java to Bytecode

The first challenge for me to overcome was how to go from XPages XML source to the Java class files (like those seen in the "Local" source folder in Designer) and finally to compiled Java bytecode. Much like in Designer's process, the middle part is just incidental: only the XSP source and the bytecode are actually stored in the NSF.

The official XSP -> Java compiler exists only in Designer, and so one route would be to try to get those plugins working on Domino. I think that'd work, but it'd be a huge hassle and, fortunately, an unnecessary one. The XPages Bazaar project contains essentially a clean re-implementation of the glue code required to coax the runtime into emitting what I needed. I used the Bazaar as an incubator for the early versions of the compiler, and tweaked the core code with additions and fixes to work with various needs I ran into.

OSGi Bundles

On its own, the Bazaar did the job well of taking an XPage and compiling it with whatever the surrounding environment had. However, to compile a full XPages app as part of a Maven build, I'd need the ability to dynamically load XPages libraries and dependencies on the fly.

To do this, I added the option to include an update site directory, and then I have the compiler stream through all the plugins and initialize them. Fortunately, the OSGi environment makes this pretty easy. The BundleContext object that's available from each OSGi bundle has an installBundle method you can use by pointing at a bundle file URL, and then you can call start() on that result to actually initialize it. I had to do a little extra work to account for bundles that shouldn't be started (source-only bundles and "fragments", which are additions onto normal plugins) and the like, but it's not too complicated.

XSP Libraries

Just installing the OSGi plugins isn't enough to get the XPages runtime to know about any libraries that may be included, however. This is done by finding all of the library extension contributors, sorting them for compatibility, wrapping them in a com.ibm.xsp.library.LibraryWrapper for some reason, wrapping that in a com.ibm.xsp.registry.config.SimpleRegistryProvider, and then adding the results of that to an in-memory com.ibm.xsp.registry.FacesSharableRegistry instance.

This is one of those cases where the actual code involved in the end isn't terribly long, but the amount of delving into the framework to figure out the needed parts was immense. In essence, each XPages application is represented as a FacesProject implementation object, which retains a registry of the libraries it knows about. In a normal running server, this happens automatically during initialization: the runtime opens the NSF, figures out which libraries it needs, finds them from the ones it knows about, and constructs the app's contained world. For the compiler, I ended up having to do something of an ad-hoc version of this as it goes, finding all the parts that need to be kicked to notice the available libraries and have them around to map something like <abc:someCustomComponent/> to an instance of com.abc.xsp.CustomComponent.

Custom Controls

Custom controls are a similar story, but they use a specialized variant of the "real" library system called LibraryFragment. The way these work, before the actual XSP -> Java compilation step, the compiler reads their definitions and adds them to the in-memory FacesProject. It's important to define them all in this way before actually trying to interpret their source (or any XPages), because this allows for the interpreter to understand a reference to one CC from another. Fortunately, the process is separated enough that the definitions can all happen before any of the Java classes actually exist. Otherwise, I would have had to either try to make a dependency graph of which CC references which other, or just keep trying to compile them repeatedly until it got the right order by brute force. The latter process will actually show up in a future blog post.

Java Source Files

Compiling individual Java source files - both those that show up in the Code/Java part of an NSF (or a custom source folder) as well as the translated XSP source - is handled by the Bazaar, in a class called JavaSourceClassLoader. This class wraps around the in-memory Java compilation capabilities that were added to the JDK in Java 1.6. In particular, this class, paired with SourceFileManager, provides knowledge to resolve classes from OSGi bundles in the active environment, including specialized knowledge of dependency management based on re-exported dependencies, embedded jars, and other OSGi-isms that play a big part in XPages libraries.

In essence, these classes provide a similar compilation environment to what Designer does when it creates the Eclipse project for compiling an NSF. They build up an environment with knowledge of all the dependencies that, in Designer, show up in "Plug-in Dependencies", and then the compiler feeds it all of the Java source files in the project. With all of this environment provided, the underlying Java compiler is able to do its job of converting them to bytecode en masse, which is then passed back to the compiler for insertion into the NSF.

Other Big Components

In the next blogs posts in this series, I'll go over some of the other big hurdles I had to overcome to get everything working properly: namely, figuring out all of the specialized behavior necessary to flag imported/created design notes properly and then the arcane incantations necessary to get this OSGi environment working outside of the Domino server.

Apple announced quite a bit at WWDC last week, and some of the stealth favorites for programmers have turned out to be SwiftUI and Combine. My macOS and iOS development is incidental at most, but I like to pay attention to this stuff, and I think that these frameworks in particular are notable for how they reflect attributes of other languages and frameworks that I do use.

SwiftUI

Before beginning, I should point out that the best source of information about SwiftUI at the moment is Apple's WWDC video archive, in particular "Introducing SwiftUI" and "SwiftUI Essentials".

SwiftUI is generally summed up as a "declarative UI framework", with "declarative" here primarily setting it apart from imperatively-defined UI. Truth be told, a mix of both has been common in both Cocoa development and elsewhere for a very long time - the definition of UI layout in .nib files comes from the old NeXT days, for example. The things that set this apart are some technical improvements and a switch of focus from your program creating and managing a UI to instead your program defining what it wants the UI to be and letting the framework handle it.

Declaring the UI

The best way to think of the way this works is that, instead of actively calling methods and setting properties on a window, some buttons, etc. on the screen, what your code does in SwiftUI is instead create a work order for what it wants the current state of the UI to be and hands that to the framework to figure out what needs to happen to get there. That "what needs to happen" can range from initially creating the layout to figuring the right way to find the difference between the new states, removing old elements, adding new ones, and animating transitions between them.

The most immediate analogue for this in common use today is React, which has essentially the same model. In (presumably) both SwiftUI and React, your job as a programmer is to have a method on a component object that is called frequently to emit what it thinks its contents should be right now. So if you have, for example, an array of objects that's displayed as an unordered list, you'll have a render() method that looks like:

render() {
    return (
      <ul>
        {this.props.items.map(item => (
          <li key={item.id}>{item.text}</li>
        ))}
      </ul>
    );
  }

In SwiftUI, a cut-down version would look like:

var body: some View {
  List(model.items) {
    Text(item.text)
  }
}

And, as I mentioned, the core concepts aren't new. In core XPages, we'd write something very similar:

<xp:repeat value="#{items}" var="item">
	<xp:this.facets>
    	<xp:text xp:key="header" contentType="HTML" value="&lt;ul&gt;"/>
    	<xp:text xp:key="footer" contentType="HTML" value="&lt;/ul&gt;"/>
  </xp:this.facets>
  
  <li><xp:text value="#{item.text}"/></li>
</xp:repeat>

However, the last one differs in a number of ways, not the least of which is that the UI-generation part doesn't retain a concept of shifting state. You, the programmer, may know that the items list may grow or shrink by a value, but, in the XPages model, the browser just starts with one block of HTML and replaces it with another on a refresh. Internally on both server and browser, I'm sure there's some retained memory for efficiency's sake, but a removed list item won't, for example, know to gracefully fade out and let the remaining ones shift into its place.

SwiftUI drives this distinction home by preferring UI component definitions to be "structs". Java doesn't currently have an analogue to structs, but they come from C and they're effectively a "pure data" type. The distinction between classes and structs gets a little murky in Swift, but the core idea is that structs are meant to show pure data in a given state and are copied when passed around. That relates here because your SwiftUI component's job is not to be the UI, but rather to emit in-memory blueprints for what it wants given the current state of the data. When data changes, the framework asks the component for a new representation, compares the old one and new one in memory, and then makes changes to the UI representation as appropriate.

It's kind of an odd distinction to write out, but I found that there was a point when I was learning React when it "clicked" in my head.

Separation of UI Definition and Output

Beyond the simplicity of declaring the UI, one of the big things that the SwiftUI presentations drive home is how one UI definition can be used across all of Apple's platforms. A list is a list is a list, and the fact that it looks one way on a watch and another way on a TV isn't inherently important.

Seeing this made me feel pretty good, since it reminded me of a post a wrote years ago that dealt with the component/renderer split in XPages at a conceptual level. Though renderers in XPages don't go so far as spitting out an AppKit application at runtime, the concept of an abstractly-defined UI that is then rendered differently based on the target is very important.

What remains to be seen with SwiftUI is how clean this distinction can remain. In my XPage example, I used some of the ExtLib's semantic form components, but real XPage applications are almost always mucked up with inline CSS classes and styles, client and server JavaScript, meaningless HTML tags like <div>, and so forth. It's one thing to say "oh, this just means a checkbox on macOS and a toggle on iOS", but another to handle a complex, crafted user interface that may be radically different in different contexts.

DSLs

SwiftUI is written in the full Swift language, but it's best described as a domain-specific language written in Swift. The term "DSL" originally meant a whole language dedicated to a small task, but nowadays usually refers instead to a style of writing an API that, when paired with a general-purpose language, acts like it's a language dedicated to the task. This is usually a feature of "scripting"-type languages, where the syntax is clean and flexible enough to not interfere.

In the Java world, the go-to language for this has long been Groovy. Darwino uses Groovy for the database adapter DSL, and SmartNSF does as well for defining services. The most common use nowadays is probably Gradle, the Maven-competing build system. It uses Groovy's closures and parentheses-less method calls to make a build script that looks more like a configuration file than a script:

plugins {
    id 'java'
    id 'application'
}

repositories {
    jcenter() 
}

dependencies {
    implementation 'com.google.guava:guava:26.0-jre' 

    testImplementation 'junit:junit:4.12' 
}

mainClassName = 'demo.App' 

The key reason why these DSLs are useful (other than not having to write a new parser) is that you have the full abilities of the underlying language at your disposal. In SwiftUI, you can do your "hide-when" logic by using the normal old if structure, rather than having a specialized "when should this show up?" property. That also means that you can bring in whatever other logic you have, third-party libraries, and so forth, without having to have specialized support in the API.

Combine

Combine, in addition to having an ominous name, is a less-flashy addition than SwiftUI, but is nonetheless important and also shows the integration of growing themes in programming elsewhere, specifically reactive programming. It's one of those topics where you can very easily fall off a conceptual cliff, and I found even Apple's introductory session to make it sound more daunting than it is by focusing on the specific Swift protocols in practice.

I've found that the best way to learn something like this is to set aside the "asynchronous" aspect at first to focus on the "data flow" part. For this, we're in luck, since this is what Java 8 streams are. Streams are all about starting with some source of data - often just a List implementation, but it's intentionally arbitrary - and changing, filtering, sorting, and otherwise manipulating it to get a result. So a prototypical example from Java can be:

Stream.of(10, 1, -3, 5)
	.filter(i -> i > 0)                  // [10, 1, 5]
	.map(i -> i * 2)                     // [20, 2, 10]
	.sorted()                            // [2, 10, 20]
	.map(String::valueOf)                // ["2", "10", "20"]
	.collect(Collectors.joining(", "));  // "2, 10, 20"

Most of the time in practice, the actual implementation will pretty much match the sequential reading of this path and so will be roughly similar to if you wrote it out with for loops, if statements, and so forth. However, because your code is describing what you want done rather than how to do it, there's room here for short-circuits, multithreading, and optimizations for different collection types.

If you look at a snippet of an example from Combine, you can see near-identical syntax in Swift, with the same idiomatic indentation:

let p4 = p1
	.merge(with: p2)
	.append(5) // add 5 to the end of the sequence
	.allSatisfy { $0 >= 1 } // check if all values are bigger than 0
	.count() // how many values: 1

Importantly, the same style of syntax applies when the incoming data is asynchronous and when the final output is similarly out-of-band. This is what all of Apple's Combine examples start with, which is why I think it can seem a bit daunting. But I think the only real switch to make in your head is to slot in the term "Publisher" for the starting provider of your data (originally an array here, but it could be a keyboard or network resource) and "Subscriber" for the code that deals with it.

Admittedly, things can get more complicated there when you need to add in error handling, value binding, and other practical considerations, but the simplicity of the core concept remains.

Overall

For Domino needs specifically and web development generally, SwiftUI and Combine don't themselves matter much. Still, I think it's useful to take times like this to see larger trends and, when you can, bask in the satisfaction of having seen the concepts elsewhere.