This past weekend was a nice one for releases to a couple of the projects I work on: the OpenNTF Domino API and Darwino.

The ODA release is something of a "consolidation" release over 2.0.0: it fixes a few of the bugs that cropped up since then, adds some important lower-level tweaks, and brings the graph REST API into the mainline release. One note with the REST API is that, due to making use of a recently-added extension to the core code, it requires a recent release of the Extension Library, 9.0.1_17 or higher.

The Darwino release is something of a preparatory release, containing some significant improvements since the initial 1.0 earlier this year. A lot of the features will warrant a proper announcement post, but the ones I find most intriguing (or worked on the most) are significantly-improved Domino replication, a new scheduling framework, and some nice cloud-deployment improvements, such as Watson integration tools and proper support for Microsoft Azure. This also sets the ground for a number of features in the next major release, which exist in initial form now, but need some final polish.

Should I have time (it continued to be a very-busy summer), both of these will warrant some more discussion. But, in the mean time, give the new versions a shot!

Today, I put two long-overdue releases up on OpenNTF.

First and by-far-foremost is version 2.0.0 of the OpenNTF Domino API. The major version reflects not so much a major new architectural change over the 1.5.x release candidates as it does the fact that those releases were conservatively named and presaged a Java-style "1.x forever" future. Various development builds and release candidates have been used in production by the API team and others for a while now, and so this represents a mature release of changes such as the Maven conversion, revamped auto-recycling, and graph API.

Alongside it, I bumped my own framework up to version 1.1.0 to reflect improved stability and a clean dependency on ODA 2.0.0. I also improved its packaging and created a distributable along the lines of the new ODA version.

So: enjoy!

Over the weekend, I took a bit of time to apply some of my hard-won recent Maven knowledge to a project I wish I had more time to work with lately: the ODA. The development branches have been Maven-ized for half a year or so, but primarily just to the point of getting the compile to work. Now that I know more about it, I was able to go in and make great strides towards several important goals.

As a preliminary note: don't take my current implementations as gospel. There are parts that will no doubt change; for example, there are some intermittent timing issues currently with the final assembly. But the changes I did make have borne some early fruit.

Source Bundles

Over the releases, it's proven surprisingly fiddly to get parameter names, inline Javadoc, and attached source to work in Designer, leaving some builds no better off than the legacy API in those regards. The apparently-consistent fix for this is the use of "source" plugins: OSGi plugins that go alongside the normal one that just contain the source of each class. Those aren't too bad to generate manually from Eclipse, but the point of Maven is getting away from that sort of manual stuff.

Fortunately, Tycho (the OSGi toolkit for Maven) includes a plugin that allows you to generate these source bundles alongside the normal ones, by including this in the list of plugins executed during the build:

<plugin>
	<groupId>org.eclipse.tycho</groupId>
	<artifactId>tycho-source-plugin</artifactId>
	<version>${tycho-version}</version>
	<executions>
		<execution>
			<id>plugin-source</id>
			<goals>
				<goal>plugin-source</goal>
			</goals>
		</execution>
	</executions>
</plugin>

Once you have that (which I added to the top-level project, so it cascades down), you can then add the plugins to the OSGi feature with the same name as the base plugin plus ".source". Eclipse will give a warning that the plugins don't exist (since they exist only during a Maven build), but you can ignore that.

Javadoc

Javadoc generation is an area where I suspect I'll make the most changes down the line, but I managed to wrangle it into a spot that mostly works for now.

Not every project in the tree needs Javadoc (for example, we don't need to include docs for third-party modules necessarily), but it's still useful to specify configuration. So I took the already-existing basic config in the parent pom and moved it to pluginManagement for the children:

<pluginManagement>
	<plugins>
		<plugin>
			<!-- javadoc configuration -->
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-javadoc-plugin</artifactId>
			<version>2.9</version>
			<configuration>
				<failOnError>false</failOnError>
				<excludePackageNames>com.sun.*:com.ibm.commons.*:com.ibm.sbt.core.*:com.ibm.sbt.plugin.*:com.ibm.sbt.jslibrray.*:com.ibm.sbt.proxy.*:com.ibm.sbt.security.*:*.util.*:com.ibm.sbt.portlet.*:com.ibm.sbt.playground.*:demo.*:acme.*</excludePackageNames>
			</configuration>
		</plugin>
	</plugins>
</pluginManagement>

Then, I added specific plugin references in the applicable child projects:

<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-javadoc-plugin</artifactId>
	<executions>
		<execution>
			<id>generate-javadoc</id>
			<phase>package</phase>
			<goals>
				<goal>jar</goal>
			</goals>
		</execution>
	</executions>
</plugin>

With those, the build can generate Javadoc appropriate for consumption in the final assembly down the line.

Assembly

The final coordinating piece is referred to as the "assembly". The job of the Maven Assembly Plugin is to take your project components and output - built Jars, source files, documentation, etc. - and assembly them into an appropriate final format, usually a ZIP file.

The route I took is to add a distribution project to the tree whose sole job it is to wait until the other components are done and then assemble the results. The pom for this project primarily consists of telling Maven to run the assembly plugin to create an appropriately-named ZIP file using what's called an "assembly descriptor": an XML file that actually provides the instructions. There are a couple stock descriptors, but for something like this it's useful to write your own. It's quite a file (and also liable to change as I figure out the best practices), but is broken down into a couple logical segments.

First off, we have a rule telling it to include all files from the "src/main/resources" folder in the current (assembly) projet:

<fileSets>
	<fileSet>
		<directory>src/main/resources</directory>
		<includes>
			<include>**/*</include>
		</includes>
		<outputDirectory>/</outputDirectory>
	</fileSet>
</fileSets>

This folder contains a README description of the result as well as the miscellaneous presentations and demo files the ODA has collected over time.

Next, in addition to the source bundles mentioned earlier, I want to include ZIP files of the important project sources in the distribution, for easy access (technically wasteful, but not by too much):

<moduleSet>
	<useAllReactorProjects>true</useAllReactorProjects>
	<includes>
		<include>org.openntf.domino:org.openntf.domino</include>
		<include>org.openntf.domino:org.openntf.domino.xsp</include>
		<include>org.openntf.domino:org.openntf.formula</include>
		<include>org.openntf.domino:org.openntf.junit4xpages</include>
	</includes>
	
	<binaries>
		<attachmentClassifier>src</attachmentClassifier>
		<outputDirectory>/source/</outputDirectory>
		<unpack>false</unpack>
		<outputFileNameMapping>${module.artifactId}.${module.extension}</outputFileNameMapping>
	</binaries>
</moduleSet>

I use the "binaries" tag here instead of "sources" because I want to include the ZIP forms (hence unpack=false) - this is one part that may change, but it works for now.

Next, I gather the Javadocs generated earlier, but these I do want to unpack:

<moduleSet>
	<useAllReactorProjects>true</useAllReactorProjects>
	<includes>
		<include>org.openntf.domino:org.openntf.domino</include>
		<include>org.openntf.domino:org.openntf.domino.xsp</include>
		<include>org.openntf.domino:org.openntf.formula</include>
	</includes>
	
	<binaries>
		<attachmentClassifier>javadoc</attachmentClassifier>
		<outputDirectory>/apidocs/${module.artifactId}</outputDirectory>
		<unpack>true</unpack>
	</binaries>
</moduleSet>

This results in an "apidocs" folder containing the Javadoc HTML for each of those three projects in subfolders.

Finally, I want to include the built and ZIP'd Update Site for use in Designer and Domino:

<moduleSet>
	<useAllReactorProjects>true</useAllReactorProjects>
	<includes>
		<include>org.openntf.domino:org.openntf.domino.updatesite</include>
	</includes>
	
	<binaries>
		<attachmentClassifier>assembly</attachmentClassifier>
		<outputDirectory>/</outputDirectory>
		<unpack>false</unpack>
		<includeDependencies>false</includeDependencies>
		<outputFileNameMapping>UpdateSite.zip</outputFileNameMapping>
	</binaries>
	
	<sources>
		<outputDirectory>/</outputDirectory>
		<includeModuleDirectory>false</includeModuleDirectory>
		<includes>
			<include>LICENSE</include>
			<include>NOTICE</include>
		</includes>
	</sources>
</moduleSet>

While grabbing the Update Site, I also copy the all-important LICENSE and NOTICE files from this current project - these may be best moved to the resources folder above.

The result of all this is a nicely-packed ZIP containing everything a user should need to get started with the API:

Next Steps

So, as I mentioned, this work isn't complete, in large part because I'm still learning the ropes. I suspect that the way I'm gathering the sources in the assembly and generating and gathering the Javadoc are not quite right - and this shows in the way that slightly-different host configurations (like on a Bamboo build server or when doing a multi-threaded build) fail during packaging.

Additionally, it's somewhat wasteful to include the source plugins even for server distributions; I won't really lose sleep over it, but it'd still be ideal to continue the recent policy of providing ExtLib-style distinct Update Sites. I'm not sure if this will require creating multiple feature and update-site projects or if it can be accomplished with build profiles.

Finally, I would love to be able to get rid of the source-form third-party dependencies like Guava and Javolution. One of the main benefits of Maven is that you can very-easily consume dependencies by listing them in the config, but Tycho and Eclipse throw a wrench into that: when you configure a project to use Tycho, then Eclipse stops referencing the Maven dependencies. Moreover, even though I believe all of the dependencies we use contain OSGi metadata, which would satisfy a Tycho command-line build, both Eclipse and the requirement that we build an old-style (non-p2) Update Site prevent us from doing that simply. It's possible that the best route will be to have Maven download and copy in the Jar files of the dependencies, but even that has its own suite of issues.

But, in any event, it's satisfying seeing this come together - and nice for me personally to build on the work Nathan, Paul, and Roland-and-co. have been doing lately. Maven is a monster and still suffers from severe "how the F does this stuff work?" problems, but it does feel good to put it to work.

...yet here we are. It's been a very interesting week in ODA circles. It started with Daniele Vistalli being curious if it was possible to run the OpenNTF Domino API on WebSphere, specifically the surprisingly-friendly Liberty profile. And not just curious: it turned out he had the chops to make it happen. Before too long, with the help of the ODA team, he wrote an extension to the WebSphere server that spins up native (lotus.domino.local) Notes RPC connections using Notes.jar and thus provides a functioning entrypoint to using ODA.

The conversation and possibilities spun out from there. It's too early to talk about too many specifics, but it's very exciting. After the initial bulk of the work, I was able to pitch in a bit yesterday and help figure out how to make this happen (click to view the full size):

That's a JSF app being built and deployed to a WebSphere Liberty server managed in Eclipse on the local machine, iterating over the contents of a DbDirectory pointed at a remote server. And it all happens to be running natively on the Mac - no VM involved.

So, like I said: it's been an interesting week. There are more possibilities than just this case, but even this alone holds tremendous promise: connecting to NSF data with the true, full API (brought into the modern day by ODA) but with the full array of J2EE support available.

Happy New Year!

As is characteristic of his blog, Sven Hasselbach recently posted two interesting posts on using the NAPI classes in the XPages runtime to manipulate files in the WebContent folder. If you haven't read the posts, I suggest you do so now, because it's knowledge that is very good to have. The NAPI classes are chock full of cheating sorcery.

But the point of this post here is a bit of me-too-ism. Which is to say: the OpenNTF Domino API has you covered on this front. The API's Design package (org.openntf.domino.design.*) contains classes that, among other things, let you retrieve, create, and modify both "normal" and these extra file resources.

The starting point is the getDesign() method on an ODA database object. Using this DatabaseDesign object, you can get access to files. For example:

Database database = FrameworkUtils.getDatabase();

DatabaseDesign design = database.getDesign();

for(FileResource res : design.getFileResources()) {
	// do stuff
}

FileResource newResource = design.createFileResource();
newResource.setName("some file.txt");
newResource.setFileData("some content".getBytes());
newResource.save();

FileResource existing = design.getAnyFileResource("existing-file.txt");
String existingContent = new String(existing.getFileData());
existing.setFileData((existingContent + " new stuff").getBytes());
existing.save();

Created file resources show up in the File Resources section currently (you can call setHideFromDesignList(true), which removes them from there, but puts them at the root of the VFS... that works, but it'd be better for them to show up in WebContent (putting "WebContent/" in the name LOOKS like it works, but doesn't)). The getAnyFileResource method will search for any "file" type element with the given name - normal file resources, WebContent files, XPages, and Java source/class files. There are better classes for manipulating the latter two, but they show up because that's how they're implemented.

One note: the Design API uses DXL, not the NAPI classes, which means it gains freedom from an XSP dependency at the cost of memory efficiency. To do its thing, it must export the file as DXL, which will be larger in memory than the file size, and then convert that BASE64 representaiton to bytes, meaning the memory cost is more than double the size of the file. For most files, that's fine, but be wary of dealing with, say, 200MB video files. That sort of thing totally works (I tested), but only if you have the memory to spare and your JVM configured to use it. Maybe one day the API will have direct access to the notes.

Okay, a second note: it's important to make sure that the max Internet access level is Designer or above for this to work, at least with the normal session object.

If you haven't taken a look at the Design API before, it may be worth at least glancing over the DatabaseDesign interface to get a feel for what it currently allows. Maybe attention on it will coax me into finding free time to finish all the other aspects I plan to get to.

In the latest release of the OpenNTF Domino API, the installation process has changed a bit, which is most notable for Designer. The reason for this is due to the weird requirements in Designer for properly getting source and documentation working.

When downloading the file, instead of the previous Eclipse update sites, there are two Update Site NSFs: one for Designer and one for Domino. There are a couple ways you can use these:

• If you're already using Update Sites for Designer or Domino, you can use the "Import Database..." action in your existing DB to import the appropriate NSF from the distribution.
• For Domino, if you're only using the API as far as OSGi bundles go, you can copy the Update Site NSF up to the server and use the OSGI_HTTP_DYNAMIC_BUNDLES INI parameter to point to it.
• If you'd like to install in Designer from the NSF directly, you can drop it in your data directory, open it in Notes, and go to the "Show URLs..." action on the menu:
  
  
  
  That will display URLs for HTTP and NRPC - the latter is the better one. You can use that to add an update site in the normal File → Application → Install... dialog.

There's an important note about the Designer install: due to the restructuring of the plugins since the last release, it's probably safest to remove any existing installation first. You can do this via File → Application → Application Management:

Find any "OpenNTF Domino" features and uninstall them each in turn:

After that, proceed with installing the API normally from the provided NSF Update Site or your own.