Showing posts for tag "xpages"

PSA: XPages Breaking Changes in 14.0 FP3

Mon Dec 16 14:08:27 EST 2024

Tags: xpages
  1. AbstractCompiledPage, Missing Plugins, and MANIFEST.MF in FP10 and V10
  2. Domino 11's Java Switch Fallout
  3. fontconfig, Java, and Domino 11
  4. Notes/Domino 12.0.2 Fallout
  5. Notes/Domino 14 Fallout
  6. PSA: ndext JARs on Designer 14 FP1 and FP2
  7. PSA: XPages Breaking Changes in 14.0 FP3

TL;DR: If you use XPages, don't use Domino 14.0 FP3. To fix the unrelated mail-routing bug, use the IFs for 14.0 FP2 or any earlier version. For the curious, the below bugs are tracked in HCL as SPR #RKRYDC2MHV.

(Updated 2024-12-17 below)

Fix Pack 3 for Domino 14.0 came out last week and, in addition to the usual spate of fixes you'd expect from an FP, it brought a potentially-significant and -breaking change to the way XPages handles style attributes for components. This is presumably in the interest of supporting the pathological strictures of Content-Security-Policy, but it applies whether or not you have taken any other steps to implement CSP.

The specific change is that the XPages renderers will now take the style attribute, externalize it to a class, and then make a <style> block for it. Take this XSP markup:

1
<xp:text style="color: red" value="I should be red"/>

While the above XSP was rendered as <span style="color: red">I should be red</span> on all previous versions, the rendered HTML (with unnecessary bits removed) now looks like this:

1
2
3
4
5
6
7
<span class="xc1">I should be red</span>

<style>
.xc1{
color: red
}
</style>

In this example, the results are equivalent: the text is red. However, this breaks in more-complicated examples. For example, say you have this in your XPage:

1
2
3
4
<xp:text style="color: red" styleClass="foo" value="I should be red"/>
<style>
	body .foo { color: green }
</style>

(The <style> block could just as easily be an external stylesheet for this purpose.)

In this case, the resultant HTML is:

1
2
3
4
5
6
7
8
<span class="xc1 foo">I should be red</span>
<style>body .foo { color: green }</style>
<style>

.xc1{
color: red
}
</style>

...and the text is green, though it should not be. Inline style attributes have a level of specificity that isn't quite equivalent to any other selector construct ("!important" rules included), so this change leads to insidious problems as things get more complicated. While inline styles have always been something of a faux pas, they're extremely common, all the more so for us because Designer encourages it and the official XPages tutorial tells you to use them.

Unfortunately, these changes are not opt-in and, because the problems are not technically errors, you won't see any console or error logs about it, and just looking at a page may not make visual changes (if there are any) immediately obvious. When using afflicted versions, the only real way to see if you need to change anything for now is to check each application manually to see if it breaks.

In the mean time, if you're upgrading 14.0 servers to account for the recent mailing problem and have XPages applications in production, I recommend sticking with Fix Pack 2, which got an Interim Fix 2 release to fix mail. If you're in the 14.5 Early Access program, it would probably also be helpful for you to chime in with your own experiences in the thread I made in the forum.

Update: Discussions in the OpenNTF Discord have highlighted several side effects or related changes that break xe:dialog elements and add extra <span>s around <xp:text>s that previously didn't have them, breaking more apps. Accordingly, my advice to not use 14.0 FP3 is only stronger now.

PSA: ndext JARs on Designer 14 FP1 and FP2

Thu Sep 12 11:02:18 EDT 2024

Tags: java xpages
  1. AbstractCompiledPage, Missing Plugins, and MANIFEST.MF in FP10 and V10
  2. Domino 11's Java Switch Fallout
  3. fontconfig, Java, and Domino 11
  4. Notes/Domino 12.0.2 Fallout
  5. Notes/Domino 14 Fallout
  6. PSA: ndext JARs on Designer 14 FP1 and FP2
  7. PSA: XPages Breaking Changes in 14.0 FP3

Back when Notes/Domino 14 came out, I made a post where I described some of the fallout of it. One of the entries was about the upstream removal of the "jvm/lib/ext" directory and the moving of all common extension JARs to the "ndext" directory. The upshot there was that any JARs that you want to add to the filesystem in Designer to match deployment on the server would have to be added to the active JRE in Designer in order to be recognized.

HCL presumably noticed this problem and altered the installation to accommodate it in FP1 and FP2. However, the approach they took is to add all JARs from ndext to the JVM. Thus, a fresh install+upgrade of Notes 14 to FP2 (or 14.5 EAP1) has a JVM that looks like this:

Screenshot of the 'Edit JRE' screen in Designer 14 FP2

This is a problem in a couple ways, but the most immediate is that it includes the toxic "jsdk.jar" I warned about in the earlier post. This JAR contains primordial Servlet classes from the very first addition of Servlet to Domino, predating XPages, and that version lacks even the convenience methods added in the ancient-but-less-so version in XPages. To demonstrate this, you can write this code:

1
2
HttpServletRequest req = null; /* pretend this is assigned to something */
Map param = req.getParameterMap();

This will work in a clean Designer 14 installation but will break on upgrade to 14 FP2, with Designer complaining that the getParameterMap method does not exist. There are others like this too, but basically any "The method foo() is undefined..." error for Servlet classes is a sign of this.

The fix is to go into your JVM definition (Preferences - "Java" - "Installed JREs" - "jvm (default)" - "Edit...") and remove jsdk.jar. While you're in there, I recommend also removing POI and its related JARs (poi-*, xmlbeans, ooxml-schemas, fr.opensagres.poi.*, commons-*) too, unless you also happen to have deployed them to the server, since they're not normally present on Domino and are thus mostly there to lead you astray. Honestly, almost none of the JARs present in there by default are useful for the XPages JVM definition, since the critical ones are contributed via OSGi plugins. I guess guava.jar is important just because it's going to contaminate the server's JVM too, so you want to account for that. Otherwise, it's probably best to treat it like a 14 install and only add the new JARs you've explicitly added and deployed to the server.

XPages JEE 3.0

Sun Jun 09 14:45:14 EDT 2024

Today, I uploaded the release version of 3.0.0 of the XPages Jakarta EE Support project. It's been proving stable in my use since the last beta, and so I think this is as good a time as any to release it properly.

Changes

The big-ticket change remains the move to Jakarta EE 10 as the baseline, which brings a handful of new features as well as a new Java version requirement. That means that this release also requires at least Domino 14. Domino 12.x served us well, but its time has passed.

Jakarta EE 10, for its part, is mostly about solving a lot of old business in the JEE community: it continues the gradual deprecation of EJB in favor of CDI, it removes some old stuff like applet requirements, and then also brings in a couple "scratch an itch" features.

Of particular note is the addition of the EntityPart type for REST services. Though it's a small feature, it's a real "finally" one, in that there hadn't been a proper way to deal with multipart/form-data MIME body parts individually, and so each implementation of Jakarta REST would bring in its own, or you'd have to fall back to taking an InputStream and parsing the MIME body yourself. Now, you can do so in a spec-based way:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
@POST
@Consumes(MediaType.MULTIPART_FORM_DATA)
public String post(@FormParam("part") EntityPart part) throws IOException {
	String mediaType = part.getMediaType();
	String name = part.getName();
	String fileName = part.getFileName();
	MultivaluedMap<String, String> headers = part.getHeaders();
	byte[] data = part.getContent().readAllBytes();
		
	// ...
}

There's also the split of Jakarta NoSQL into that spec plus the new Jakarta Data. In this release of XPages JEE, I mostly aimed to keep the same level of functionality while accounting for the renaming of packages and types, but I'll be interested in building on this in the future.

Finally, there's the project-specific change of condensing the many, many XPages libraries just down to Core, UI, and MicroProfile. That didn't impact functionality as such, but it sure is nice only having three (or six, with the source features) features to say "yes, install this" to when updating it in Designer and only three to check in Xsp Properties. It also allowed me to delete a lot of weird shim and conditional code, and it'll make maintenance of it much easier in the future without having to worry about every permutation of what libraries you have enabled in an NSF.

The Future

Speaking of which, that brings me to some of the next things in the docket. I imagine that the immediate work will be cleaning up any loose ends from the move. For example, Jakarta Concurrency 3.0 brought a bunch of new features, but I haven't actually checked to see if they work or if I need to do more adapting.

Additionally, Jakarta Data is intended to go beyond just NoSQL, and can also layer on top of Jakarta Persistence (née JPA, the API for working with relational DBs) and arbitrary services. I don't know yet if there's a usable implementation beyond the one in Open Liberty, so that may have to wait, but it'll be interesting to tinker with.

There are also a bunch of features I'd like to get cracking on now that this hurdle is done. For example, I'd like to move the NoSQL driver to use JNX, which would let me do a couple things that the Notes.jar classes just can't. Along with that, I'd like to add an option to publish MP Metrics to the Domino statistics store, which adopting JNX would allow easily.

Fortunately, I don't expect that there will be any other breaking-change discontinuities in the future. Jakarta EE 11 has some deprecations and removals, but it's mostly similar to JEE 10 in that it's about classes and idioms that are much, much older than any code built using this project. That should give this 3.x series a good, long period to be a comfortable baseline, even though the next major version of JEE.

XPages JEE 3.0 Beta 4

Wed May 22 13:48:19 EDT 2024

Earlier today, I uploaded beta 4 of XPages JEE 3.0 to GitHub. I've been taking a slow approach to this release due to its "breaking changes" nature, but I think it's just about ready for release.

Domino 14

Like previous betas, this release requires Domino 14 (and Notes 14 for development), since it moves to a baseline of Jakarta EE 10, which in turn requires Java 11. Doing this let me get rid of some extra shim code that was needed to support both Domino 14 and previous versions, and also let me move to some newer language constructs. If you're interested in the sorts of things that the new versions of Java brought, check out the OpenNTF webinar from April, where I talked about just that.

Library Reorganization

Beyond the Java version requirement, the big breaking change I made was to finally shrink the number of XPages libraries and p2 features in the project. As the project grew, I kept adding new distinct XPages libraries, for the principle of keeping each spec distinct, as they often technically are. A few things made me want to fix this, though:

  • Checking all the boxes in the Xsp Properties editor for each library was annoying
  • Checking "Yes, install this plug-in" for every single component, plus its source version, when installing in Designer was very annoying
  • I had to do weird tricks to add features that touched multiple specs. For example, the project tree had a bunch of cross-spec fragments like "jaxrs.cdi" and "json.cdi" to contribute parts for when CDI was present but not break things when it wasn't. This added an extra layer of indirection and maintenance hassle
  • The specs themselves have been converging, particularly in the sense that more and more they assume the "backbone" of CDI is present. For example, Faces removed its original @ManagedBean and related support in favor of going all-in on CDI. Jakarta REST is moving towards the same
  • It was hard to think of realistic scenarios where it would be important to split up the specs like this, using, say, REST but not CDI or Validation

Now, there are just three: "org.openntf.xsp.jakartaee.core", "org.openntf.xsp.jakartaee.ui", and "org.openntf.xsp.microprofile". I was tempted to roll MicroProfile into "core", but they're conceptually (and administratively) distinct enough that it was worth separating them. With this change, it's not only less annoying to install, but it lets me make a lot more assumptions about what is present across specs, simplifying a lot of little things.

Deep-Dive Sidebar: Class Loading

One interesting aspect I ran into when making this change was that I had to readjust my mental model for how class loading is done from an NSF-based application and the libraries it uses. The way it mostly works conceptually aligns with what you see in Designer:

  • Select a library to depend on
  • The XspLibrary has a "getPluginId" method, which then Designer uses to add the OSGi bundle to the classpath
  • Any Require-Bundle dependencies in that plugin marked as "visibility:=reexport" are also included on the classpath

So, in this way, you'd previously select the "org.openntf.xsp.cdi" library, which would then add a dependency on the bundle of the same name, which would in turn re-export the things the NSF should see, such as the CDI API classes.

When I consolidated the libraries, I did it in the straightforward way: I made new "*.library" bundles for them and then added the existing spec-specific bundles as re-exported dependencies. As far as Designer was concerned, all was well, and there was just another little layer in between.

However, that's not quite the whole story when it comes to the runtime on the server. Though Designer presents the NSF as a pseudo-OSGi bundle using the Plug-in Development Environment, Domino doesn't do the same thing. What Domino does is use a class called ModuleClassLoader (not to be confused with Equinox's ModuleClassLoader, which is entirely different and IS an OSGi loader) to handle loading classes from the NSF and its dependencies. The way it gets to its dependencies isn't really a "true" OSGi way, though: it keeps track of a collection of ClassLoader objects as extraDepends, which it consults each in turn as needed. Those ClassLoader objects, at least in post-8.5.2-era Domino, are the internal class loaders from the library OSGi bundles. This is cheating, and I imagine it was made for pragmatic transitional reasons when OSGi came into the picture.

The old layout conceptually looks like this:

Diagram of NSF to old library dependency

At first blush, this seems like a "six of one, half a dozen of the other" sort of situation, but it's not quite. What this setup does that normal OSGi doesn't is that it exposes META-INF/services files inside the direct dependencies to the application's ClassLoader, whereas these are normally encapsulated in OSGi. The effect was that a bunch of things that used to work started to fail - REST couldn't find all its output-writing classes, Validation couldn't find its implementation, and so forth. This is because they would all internally ask the thread-context ClassLoader (i.e. the NSF's loader) for resources within META-INF/services, and the extraDepends list used to be able to find them. Now that there was a layer of indirection, this no longer worked: the extraDepends loaders could see their own stuff but would not traverse the OSGi barrier to peek inside their further dependencies for these. Conceptually, now we have this:

Diagram of NSF to new library dependency

A direct ClassLoader dependency allows reading of resources, but a true OSGi-type dependency does not. So the result is that I had to "promote" a bunch of META-INF/services files from the now-downstream plugins into the "*.library" ones. It all makes sense once you see how the gears are moving, but it sure threw me for a loop for a while.

Bundle and Package Renaming

Okay, now back to the changes.

Since I was already breaking things anyway, I decided this was a good opportunity to fix the names of the bundles and packages in the project's source. For example, some names were antiquated: what was once "JSF" is "Jakarta Faces", but my bundle was "org.openntf.xsp.jsf". Additionally, I was inconsistent in my hierarchy: while Transaction was in "org.openntf.xsp.jakarta.transaction", others (like Faces there) skipped the "jakarta" level of the hierarchy. These don't normally matter to developers consuming the library, but they annoyed me. Now, all of the bundles and their contained packages are within either "org.openntf.xsp.jakarta", "org.openntf.xsp.jakartaee" (for platform-wide capabilities), or "org.openntf.xsp.microprofile".

Along with this will be a couple potential breaking changes for app-level code, such as moving org.openntf.xsp.beanvalidation.XPagesValidationUtil to org.openntf.xsp.jakarta.validation.XPagesValidationUtil, but there won't be TOO many due to this change.

Jakarta Data and NoSQL Changes

This one isn't from my latest round of changes and has been the case since early in the 3.x stream, but it's worth mentioning again here. The Repository concept from Jakarta NoSQL moved from that spec to the new "Jakarta Data" spec, and so related packages changed from jakarta.nosql.mapping to jakarta.data. Additionally, since the NoSQL spec shrunk to accommodate, things like @Column changed from jakarta.nosql.mapping.Column to jakarta.nosql.Column. It makes sense as NoSQL has been an evolving spec all along, but I suspect that this will be the biggest app-code-breaking change it experiences for a good while.

Release and Future Versions

My next steps are to put this through its paces now that all the issues are closed. Though I've ported everything to the JEE 10 versions, I haven't tested to make sure that most of the new features work. While JEE was largely a "cleanup" release, there are a bunch of new features, particularly in Faces, which is in turn always the jankiest part of the stack on Domino.

Post-3.0, I expect that my focus will start to shift to Jakarta EE 11. For a time, I was going to be SOL with it: though Domino 14 bumped Java to 17, JEE 11 was slated to target Java 21 at a minimum. In the mean time, however, that target shifted down to 17, putting it back on the table for Domino. JEE 11 was originally slated for Q1 of this year, but it slipped to some time around summer. That fits reasonably well with my cadence here. JEE 11 is technically also a breaking release, but I suspect that it won't break features that XPages JEE users use, at least not after this hurdle here.

XPages JEE 2.15.0 and Plans for JEE 10 and 11

Fri Feb 16 15:30:40 EST 2024

  1. Updating The XPages JEE Support Project To Jakarta EE 9, A Travelogue
  2. JSP and MVC Support in the XPages JEE Project
  3. Migrating a Large XPages App to Jakarta EE 9
  4. XPages Jakarta EE Support 2.2.0
  5. DQL, QueryResultsProcessor, and JNoSQL
  6. Implementing a Basic JNoSQL Driver for Domino
  7. Video Series On The XPages Jakarta EE Project
  8. JSF in the XPages Jakarta EE Support Project
  9. So Why Jakarta?
  10. XPages Jakarta EE 2.5.0 And The Looming Java-Version Wall
  11. Adding Concurrency to the XPages Jakarta EE Support Project
  12. Adding Transactions to the XPages Jakarta EE Support Project
  13. XPages Jakarta EE 2.9.0 and Next Steps
  14. XPages JEE 2.11.0 and the Javadoc Provider
  15. The Loose Roadmap for XPages Jakarta EE Support
  16. XPages JEE 2.12.0: JNoSQL Views and PrimeFaces Support
  17. XPages JEE 2.13.0
  18. XPages JEE 2.14.0
  19. XPages JEE 2.15.0 and Plans for JEE 10 and 11

Today, I released version 2.15.0 of the XPages Jakarta EE project. As is often the case lately, this version contains bug fixes but also a few notable features:

  • You can now specify Servlets in WEB-INF/web.xml (as opposed to just via the @WebServlet annotation). This is helpful for defining a Servlet when the actual implementation is in a JAR or when following non-annotation-based examples
  • You can now specify context-param values in WEB-INF/web.xml in the NSF and META-INF/web-fragment.xml in JAR design elements, which will be available to JSP, JSF, JAX-RS, @WebServlet-annotated Servlets, and web.xml-defined Servlets
  • Added @BooleanStorage annotation for NoSQL entities to define how boolean values are converted to note items
  • Added CRUD operations for calendar events to NoSQL, around a few new methods on Repository. This exposes some of the capabilities of NotesCalendar and can be used for, for example, providing an iCalendar feed based on a mail database. To go with that, XPages JEE also re-exports iCal4J as included in the Domino stack for NSF use, though this API is... not smooth

The first two here are focused around bringing NSFs more in line with "normal" Jakarta EE applications, while the latter are some nice improvements for the NoSQL driver. I hope to put the last one in particular to good use - for example, OpenNTF's site will be able to provide a calendar of webinars and other events that we can manage internally using a normal Notes calendar, and that sounds nice to me.

Next Versions

I still have the 3.x branch of the project chugging along, and I think it'll be ready for a real release before too long. Since it'll be a breaking-changes release thanks to upstream changes, I'm using it as an opportunity to consolidate the sprawl of features and XPages Libraries. Currently, my plan is:

  • One for "core", covering most things in the Jakarta EE Core Profile, plus the other utility specs I've implemented: Transactions, Bean Validation (which really should be in Core in my estimation), Concurrency, Servlet, and so forth, plus Data and NoSQL
  • One for "UI", covering Jakarta Pages, Jakarta Faces, and MVC - basically, the stuff you could use to replace XPages to make an HTML-generating app in your NSF
  • One for MicroProfile, or at least the specs I've implemented so far. I'm a little tempted to wrap this in to Core, since things like OpenAPI are useful almost all the time, but it's a clean-enough separation that it'll be fine

This will require Domino 14, since Jakarta EE 10 requires at least Java 11.

That brings me to some unexpected good news: though Jakarta EE 11 was long planned to use Java 21 as its minimum version (since 21 is the current LTS), it looks like they've switched to making Java 17 the baseline. For me, this is a little sad in an idealistic sense, since it pushes things like Virtual Threads out of the realm of being a core part of JEE, but I'm very happy that I'll be able to use all JEE 11 specs in Domino 14. Even if Domino 15 used Java 21, it'd still be a long while before that came, and we'd lag behind the standard for at least a year. Instead, this puts the project back in line with upstream, and allows me personally to potentially resume committing to Jakarta NoSQL - I'd been out of the loop for a very long time when it moved to 11 and then 17 as its required version.

I don't know right now whether JEE 11 will be the same sort of breaking change for the project (which would mean a 4.x release) or if I'll be able to make it a 3.x one - the specs aren't out yet, so time will tell. The big focus of 11 will be further centralization on CDI instead of EJB, and I'm all for it.

My plan is to get 3.x out for Domino 14, based on JEE 10, as soon as time allowed, and then I'll start looking into bumping to JEE 11 when it releases in the summer.

XPages JEE 2.14.0

Fri Oct 27 11:47:02 EDT 2023

  1. Updating The XPages JEE Support Project To Jakarta EE 9, A Travelogue
  2. JSP and MVC Support in the XPages JEE Project
  3. Migrating a Large XPages App to Jakarta EE 9
  4. XPages Jakarta EE Support 2.2.0
  5. DQL, QueryResultsProcessor, and JNoSQL
  6. Implementing a Basic JNoSQL Driver for Domino
  7. Video Series On The XPages Jakarta EE Project
  8. JSF in the XPages Jakarta EE Support Project
  9. So Why Jakarta?
  10. XPages Jakarta EE 2.5.0 And The Looming Java-Version Wall
  11. Adding Concurrency to the XPages Jakarta EE Support Project
  12. Adding Transactions to the XPages Jakarta EE Support Project
  13. XPages Jakarta EE 2.9.0 and Next Steps
  14. XPages JEE 2.11.0 and the Javadoc Provider
  15. The Loose Roadmap for XPages Jakarta EE Support
  16. XPages JEE 2.12.0: JNoSQL Views and PrimeFaces Support
  17. XPages JEE 2.13.0
  18. XPages JEE 2.14.0
  19. XPages JEE 2.15.0 and Plans for JEE 10 and 11

Today, I released version 2.14.0 of the XPages Jakarta EE Support project. As with the last few releases, this is primarily about bug fixes and compatibility as I prepare for the big switch in 3.0, but there are some notable, if small, feature additions.

To begin with, I improved handling of reading JSON in NoSQL entities when reading from a view. This applies to the @ItemStorage(type=ItemStorage.Type.JSON) annotation on entity properties, which causes the value to be loaded and stored as JSON, useful for storing custom class types in a document. Now, such values can be read from view entries - previously, this processing was skipped for those. Of note when using this: normally, storing as JSON will automatically set the item's summary flag to false, to avoid overflowing the summary limit. However, you can add @ItemFlags(summary=true) to the property to override this behavior so that the values can show up in views.

Additionally, I added the ability to use JAXRSClassContributors inside the NSF. These were originally an internal mechanism for the project to dynamically add REST endpoints and extensions, like those used by MVC and OpenAPI. Now, though, I've made it so that such classes can be registered via a file named "META-INF/services/org.openntf.xsp.jaxrs.JAXRSClassContributor" in the NSF, and also added the ability to specify configuration properties. The latter is important because, though all xsp.properties values were already inserted into the JAX-RS configuration, there was no way to provide non-string values. This came up in the context of MVC, which has a CSRF configuration property that must be an enum value.

For the final feature, I improved support for JAX-RS's status-indicating exceptions, such as NotSupportedException and BadRequestException. Previously, the project supported translating NotFoundException to a 404, but now it will also translate these other standard exceptions to their corresponding HTTP statuses.

The release is otherwise rounded out by a number of bug fixes to fix problems encountered in the wild. Additionally, I added a workaround for some classpath pollution in the latest Domino 14 beta - I hope that the trouble will be gone for GA, but this project should handle it either way.

XPages JEE 2.13.0

Fri Jul 21 11:51:52 EDT 2023

  1. Updating The XPages JEE Support Project To Jakarta EE 9, A Travelogue
  2. JSP and MVC Support in the XPages JEE Project
  3. Migrating a Large XPages App to Jakarta EE 9
  4. XPages Jakarta EE Support 2.2.0
  5. DQL, QueryResultsProcessor, and JNoSQL
  6. Implementing a Basic JNoSQL Driver for Domino
  7. Video Series On The XPages Jakarta EE Project
  8. JSF in the XPages Jakarta EE Support Project
  9. So Why Jakarta?
  10. XPages Jakarta EE 2.5.0 And The Looming Java-Version Wall
  11. Adding Concurrency to the XPages Jakarta EE Support Project
  12. Adding Transactions to the XPages Jakarta EE Support Project
  13. XPages Jakarta EE 2.9.0 and Next Steps
  14. XPages JEE 2.11.0 and the Javadoc Provider
  15. The Loose Roadmap for XPages Jakarta EE Support
  16. XPages JEE 2.12.0: JNoSQL Views and PrimeFaces Support
  17. XPages JEE 2.13.0
  18. XPages JEE 2.14.0
  19. XPages JEE 2.15.0 and Plans for JEE 10 and 11

Today, I released version 2.13.0 of the XPages Jakarta EE Support project. Though there's not a single big banner feature, this one brings a number of good enhancements in a bunch of areas.

Domino 14

The first thing it brings is compatibility with Domino 14 EAP1. The goal here is to just bring the same features to that version - it doesn't bump the individual components to their Jakarta EE 10 versions yet, since that will come with breaking changes and prevent use on 12.0.2 and before.

There remains a caveat here, which is that EAP1 doesn't include a Java compiler, and so JSP doesn't work unless you shim in parts of a JDK into a Domino installation. If you're not using JSP, though, you should be able to run your apps on 14 using this new build.

JSF

It turns out that Faces support is a popular feature, which makes sense: it's the most direct analogue to writing XPages, while bringing in a lot of new features. While Faces has always been tricky to keep working, this build includes some fixes for stability and usability. I'd still consider this route to be the least-proven way to do UIs with this project, but it's shaping up really nicely.

JavaSapi

Speaking of experimental features, this release comes with a new feature that builds on the JavaSapi bridge I added a bit ago: you can now specify extensions within an NSF that will participate in JavaSapi pre-processing of requests.

To do this, you can make a file named META-INF/services/org.openntf.xsp.jakartaee.jasapi.JavaSapiExtension in your NSF's Java classpath (e.g. the Code/Java folder) and have it name a JavaSapiExtension class. For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
package javasapi;

import org.openntf.xsp.jakartaee.jasapi.JavaSapiContext;
import org.openntf.xsp.jakartaee.jasapi.JavaSapiExtension;

public class TestJavaSapiExtension implements JavaSapiExtension {
	@Override
	public Result rawRequest(JavaSapiContext context) {
		// Add a custom header to all responses
		context.getResponse().setHeader("X-InNSFCustomHeader", "Hello from NSF");
		return Result.SUCCESS;
	}
	
	@Override
	public Result authenticate(JavaSapiContext context) {
		// Custom authentication mechanism. If you use this, do it more securely!
		String overrideName = context.getRequest().getHeader("X-OverrideName");
		if(overrideName != null && !overrideName.isEmpty()) {
			context.getRequest().setAuthenticatedUserName(overrideName, "Basic");
			return Result.REQUEST_AUTHENTICATED;
		}
		return JavaSapiExtension.super.authenticate(context);
	}
}

As with any time I do anything with JavaSapi, I can't stress enough how unsupported this is. It's not even officially a feature of Domino, and I've found it fairly easy to crash the server by doing the wrong thing here. On the other hand, it's neat and fun, so... feel free to tinker with it.

NoSQL

Finally, I added some methods to DominoRepository instances to access profile and named notes:

1
2
3
4
SomeEntity profile = repository.findProfileDocument("SomeProfile", "Your Username")
	.orElseThrow(() -> new NotFoundException("Could not find profile for user"));
SomeEntity named = repository.findNamedDocument("Some Name", "Your Username")
	.orElseThrow(() -> new NotFoundException("Could not find named doc for user"));

I made them return Optional for safety's sake - I think they'll in general create the documents if they don't exist, but I wanted to leave room in the API for a future ability to only return them if they haven't previously been explicitly created.

Anyway, that's one more step in making the driver useful as a general-purpose Domino access mechanism. My goal is to make it so that you'd only need lotus.domino, ODA, or another Domino-specific API in specific edge cases. I can already do almost everything I need to, and now I'm just working down the list of less-critical features.

Next Steps

As I've been working on 2.13.0, I've also been working on the 3.0 branch, including an early beta last month. That's the branch that breaks pre-Domino-14 compatibility and bumps most components up to their Jakarta EE 10 versions. Since I can't realistically have a proper release of that until Domino 14 is out, my plan is to keep tinkering with the side branch and releasing betas from time to time.

In the mean time, I wouldn't be surprised if there's a 2.14.0. There are some tweaks and efficiency improvements I want to make particularly for JSF, so I expect I'll have enough on my plate before Domino 14's release to get another current-line release out.

XAgents to Jakarta REST Services

Sun Feb 05 15:16:48 EST 2023

  1. Code-First REST APIs With XPages Jakarta EE Support
  2. Code-First REST APIs Followup: OpenAPI
  3. XAgents to Jakarta REST Services

For a good long time now, XAgents have been one of the common ways to do non-HTML output in an XPages environment - JSON, mostly. I think the technique was codified and the term coined by Stephan Wissel back in 2008 and the idea has been the same since.

Effectively, an XAgent lets you write a Servlet but with a bit more scaffolding. Though XPages has a path to use Servlets officially, that method is more out-of-the-way than XAgents and doesn't (without further hoop jumping) give you some niceties like sessionAsSigner.

However, though they're venerable and sort of convenient, XAgents are lacking in a number of ways. For one, stuffing them inside an XPage is ungainly: even if the XPage just calls out to some Java code, you're polluting your UI-element space with something kind of unrelated, forcing you to name your XPages stuff like "apiEmployees.xsp". More importantly, though, it doesn't provide a lot of affordances for the sort of work you'd actually want to do when writing a REST API. Though XPages has a couple mechanisms for generating JSON, little of that is exposed by the XPage editor environment, and I suspect that many or most XAgents writing JSON commit the cardinal sin of doing so through direct string concatenation, likely often without much escaping. Further, there's no built-in mechanism for picking apart path segments or multi-branched routing.

So I figured today is a good opportunity to talk a bit about one of the XPages Jakarta EE Support project's flagship features: writing REST services that consume and emit JSON. This post covers some of the same ground I've talked about before, but sometimes it's useful to re-contextualize this sort of thing.

The XAgent Way

To level-set the discussion, I'll give a starting example in an XAgent, and we'll keep it simple. The idea here will be that you have Person documents in a database and you want to write an API that will take a UNID and emit the corresponding person's first and last name. This is very similar to the Employees example in the "Code-First REST API" example project from the JEE repo, but without diving into more-complex parts or the Jakarta NoSQL data layer.

In an XAgent, you might do something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core" viewState="nostate" rendered="false">
	<xp:this.afterRenderResponse><![CDATA[#{javascript:
		var resp = facesContext.getExternalContext().getResponse();
		resp.setContentType("application/json");
		
		var writer = facesContext.getResponseWriter();
		
		var doc = database.getDocumentByUNID(param.unid);
		var result = toJson({
			"firstName": doc.getItemValueString("FirstName"),
			"lastName": doc.getItemValueString("LastName")
		});
		writer.write(result);
		
		writer.endDocument();
		facesContext.responseComplete();
	}]]&gt;</xp:this.afterRenderResponse>
</xp:view>

You'd put that in an XPage and call it like:

1
2
$ curl http://your.server/someapp.nsf/apiPeople.xsp?unid=68D1CAA80F65780D8525894D006B1CE7
{"lastName":"Fooson","firstName":"Foo"}

That does the job well enough. However, this will get gangly very fast. If you want to expand on the types of data the Person document contains, add more operations to the API, add other query parameters, or so forth, you'll have to either have a giant blob of code here, spin it off to an SSJS script library, or move it to Java classes. And, all along the way, the dev environment won't be giving you any help with the process - it knows nothing about writing REST APIs, and so it's all "manual". Better than with a classic agent, but not great.

Jakarta Way, Take 1

So let's move this over to Jakarta EE. We'll start by doing basically a concept-for-concept move: just take the above and make a REST class out of it.

Such a class could look something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
package rest;

import com.ibm.xsp.extlib.util.ExtLibUtil;

import jakarta.json.Json;
import jakarta.json.JsonObject;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.core.MediaType;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.PathParam;
import jakarta.ws.rs.Produces;

import lotus.domino.Database;
import lotus.domino.Document;
import lotus.domino.NotesException;

@Path("v1/person")
public class PersonResourceV1 {
	
	@Path("{unid}")
	@GET
	@Produces(MediaType.APPLICATION_JSON)
	public JsonObject get(@PathParam("unid") String unid) throws NotesException {
		Database database = ExtLibUtil.getCurrentDatabase();
		Document doc = database.getDocumentByUNID(unid);
		
		return Json.createObjectBuilder()
			.add("firstName", doc.getItemValueString("FirstName"))
			.add("lastName", doc.getItemValueString("LastName"))
			.build();
	}
}

This one you'd call in a similar way, and get similar results:

1
2
$ curl http://your.server/someapp.nsf/xsp/app/v1/person/68D1CAA80F65780D8525894D006B1CE7
{"firstName":"Foo","lastName":"Fooson"}

There's a bit more boilerplate - it is Java - but you can already see some of the benefits. The URL is a little more REST-like and the code is much more task-focused. Instead of disabling all the normal features of the XPage and manually emitting text, a large amount of the code is explicitly describing what you intend to do in a REST service. The @Path annotations describe the components of the URL, and the use of @Path("{unid}") lets us name one of those parts without having to do a query string. Similarly, the @GET and @Produces(MediaType.APPLICATION_JSON) lines directly say what's going on: you can do a GET request to the URL and get JSON back.

These annotations pay off in a couple ways. First of all, the code will be a lot easier to understand when you come back later. Imagine if your API also handled new documents, deletions, and modifications - with an XAgent, you'd have branching paths and would have to rely on either good naming or commenting to mentally traverse it. With this, you would have other methods with similar annotations - @PUT, @POST, @DELETE - and would see exactly where requests are going to go.

And it's not just for you the programmer (or for the next human replacing you when you retire to a beach somewhere): these annotations mean something to tools that process it as well. One such tool comes with the XPages JEE project: the MicroProfile OpenAPI generator. With this class present, you automatically get a useful OpenAPI spec:

1
$ curl http://your.server/someapp.nsf/xsp/app/openapi.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
---
openapi: 3.0.3
info:
  title: XAgent Comparison
servers:
- url: http://your.server/someapp.nsf/xsp/app
paths:
  /v1/person/{unid}:
    get:
      parameters:
      - name: unid
        in: path
        required: true
        schema:
          type: string
      responses:
        "200":
          description: OK
          content:
            application/json:
              schema:
                type: object

This sort of thing pays off tremendously once you start working with a client JS app, especially with a larger development team.

Jakarta Way, Take 2

But we can do better than this. As it is, the direct translation from the XAgent still left the actual output pretty vague. We at least know it's emitting a JSON object, but that's the extent of it. Moreover, the app code wastes some conceptual time actually building the JSON object, which is okay but unnecessary. Let's bring JSON Binding into the mix. This will increase the size of the code, but will pay off in conceptual cleanliness and in future work.

That could look like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
package rest;

import com.ibm.xsp.extlib.util.ExtLibUtil;

import jakarta.validation.constraints.NotEmpty;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.core.MediaType;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.PathParam;
import jakarta.ws.rs.Produces;

import lotus.domino.Database;
import lotus.domino.Document;
import lotus.domino.NotesException;

@Path("v2/person")
public class PersonResourceV2 {
	
	// Normally, this class would be in a separate file
	public static class Person {
		private String firstName;
		private @NotEmpty String lastName;
		
		public String getFirstName() {
			return firstName;
		}
		public void setFirstName(String firstName) {
			this.firstName = firstName;
		}
		public String getLastName() {
			return lastName;
		}
		public void setLastName(String lastName) {
			this.lastName = lastName;
		}
	}
	
	@Path("{unid}")
	@GET
	@Produces(MediaType.APPLICATION_JSON)
	public Person get(@PathParam("unid") String unid) throws NotesException {
		Database database = ExtLibUtil.getCurrentDatabase();
		Document doc = database.getDocumentByUNID(unid);
		
		Person result = new Person();
		result.setFirstName(doc.getItemValueString("FirstName"));
		result.setLastName(doc.getItemValueString("LastName"));
		return result;
	}
}

Calling this will work the same way as last time, but with a "v2" to indicate our new-and-improved back end:

1
2
$ curl http://your.server/someapp.nsf/xsp/app/v2/person/68D1CAA80F65780D8525894D006B1CE7
{"firstName":"Foo","lastName":"Fooson"}

What does this get us? Well, for one, we're no longer explicitly working with JSON, which is nice. We could change the media type to XML and the output type will automatically adapt, and the same will go for any future formats we write an adapter for. Additionally, this will work better with a more-structured database access layer. I'll leave that part out here, but the aforementioned "Code-First REST API" example shows that.

Some more of the payoff shows up when we check the generated OpenAPI spec. If we make the same call as above, the output will be more descriptive now:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
---
openapi: 3.0.3
info:
  title: XAgent Comparison
servers:
- url: http://your.server/someapp.nsf/xsp/app
paths:
  /v2/person/{unid}:
    get:
      parameters:
      - name: unid
        in: path
        required: true
        schema:
          type: string
      responses:
        "200":
          description: OK
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Person'
components:
  schemas:
    Person:
      required:
      - lastName
      type: object
      properties:
        firstName:
          type: string
        lastName:
          minLength: 1
          type: string

For someone consuming this spec, this is starting to get really good. No longer does the output just say it's a generic JSON object: now it describes it as "Person", shows the two properties that will be output, and declares that lastName will always be at least one character both for input and output. As you add more operations and types, this spec will continue to grow, naturally piggybacking on the annotations and types you write.

Conclusion

Strictly speaking, you can do the same stuff with an XAgent that you can with the XPages JEE project. You could manage the internal complexity and you could manually write the OpenAPI spec, but it would be much, much harder to do, and I think it'd be a safe bet to say that very few XAgent-based APIs have specs to go with them anyway.

Especially as the scope of your app grows, the development and maintenance experience with the JEE approach is night-and-day compared to classical mechanisms like XAgents. If you're still writing APIs that way or similar, I definitely recommend you give this path a try.

XPages Jakarta EE 2.5.0 And The Looming Java-Version Wall

Wed May 25 14:41:52 EDT 2022

  1. Updating The XPages JEE Support Project To Jakarta EE 9, A Travelogue
  2. JSP and MVC Support in the XPages JEE Project
  3. Migrating a Large XPages App to Jakarta EE 9
  4. XPages Jakarta EE Support 2.2.0
  5. DQL, QueryResultsProcessor, and JNoSQL
  6. Implementing a Basic JNoSQL Driver for Domino
  7. Video Series On The XPages Jakarta EE Project
  8. JSF in the XPages Jakarta EE Support Project
  9. So Why Jakarta?
  10. XPages Jakarta EE 2.5.0 And The Looming Java-Version Wall
  11. Adding Concurrency to the XPages Jakarta EE Support Project
  12. Adding Transactions to the XPages Jakarta EE Support Project
  13. XPages Jakarta EE 2.9.0 and Next Steps
  14. XPages JEE 2.11.0 and the Javadoc Provider
  15. The Loose Roadmap for XPages Jakarta EE Support
  16. XPages JEE 2.12.0: JNoSQL Views and PrimeFaces Support
  17. XPages JEE 2.13.0
  18. XPages JEE 2.14.0
  19. XPages JEE 2.15.0 and Plans for JEE 10 and 11

Earlier today, I published version 2.5.0 of the XPages Jakarta EE Support project. It's mostly a consolidation and bug-fix release, but there are few interesting features and notes about the implementation. Plus, as teased in the post title up there, there's a looming problem for the project.

New Features

There are two main new features in this version.

First, I added some configurable CORS support for REST services. Fortunately for me, RestEasy comes with a CORS filter by default, and it just needs to be enabled. I wired it up using MicroProfile Config to read some values out of xsp.properties:

1
2
3
4
5
6
7
8
rest.cors.enable=true                   # required for CORS
rest.cors.allowCredentials=true         # defaults to true
rest.cors.allowedMethods=GET,HEAD       # defaults to all
rest.cors.allowedHeaders=Some-Header    # defaults to all
rest.cors.exposedHeaders=Some-Header    # optional
rest.cors.maxAge=600                    # optional
# allowedOrigins is required, and can be "*"
rest.cors.allowedOrigins=http://foo.com,http://bar.com

I also added support for using the long-standing @WebServlet annotation. Though REST services will generally do what you want, sometimes it's handy to use the lower-level Servlet capability, and now you can do so inline:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
@WebServlet(urlPatterns = { "/someservlet", "/someservlet/*", "*.hello" })
public class ExampleServlet extends HttpServlet {
	private static final long serialVersionUID = 1L;
	
	@Inject
	ApplicationGuy applicationGuy;

	@Override
	protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
		resp.setContentType("text/plain");
		resp.getWriter().println("Hello from ExampleServlet. context=" + req.getContextPath() + ", path=" + req.getServletPath() + ", pathInfo=" + req.getPathInfo());
		resp.getWriter().println("ApplicationGuy: " + applicationGuy.getMessage());
		resp.getWriter().flush();
	}
}

Consolidation

There were a couple specs where I had previously either copied the source into the repository (CDI, Mail) or had maintained a local branch fork (NoSQL). Those were always uncomfortable concessions to reality, but I decided to look further into ways to handle that.

For NoSQL, part of it was what I talked about in my last post: using Eclipse Transformer to make use of javax.* compiled binaries and source converted to jakarta.* automatically. But beyond that, it had the same problem that I had forked Mail for. Namely, it hits the same trouble that lots of non-OSGi code does in an OSGi context, where it uses ServiceLoader in a non-extensible way. Though I have an open PR to make use of the pseudo-standard "HK2" ServiceLoader provider, waiting for that would mean continuing the local-build trouble.

Instead, for all of these cases I made use of OSGi's Weaving capability to re-write those parts of the class files on the fly. While this is a bit unfortunate, it works well in practice. The only real down side for now is having to be a bit more careful when bumping the versions in the future, but this type of code changes very rarely.

The Looming Wall

While this has been going swimmingly, I've started to hit some real impediments with Domino's Java version. The next release of Jakarta EE, version 10, requires Java 11 as a minimum. This is similar to the move Equinox (Domino's OSGi framework of choice) made just under two years ago, and which has itself bitten me with a blockage to upgrading Tycho to version 2.0 and above. Java 11 is about four years old now, and is no longer even the latest LTS release, so this all makes sense.

I've known this was coming for a while, but incompatible versions of JEE specs and implementations started to trickle in over the past year, leading to me leaving notes for myself about maximum versions. JEE 10 itself is fairly imminent now, so I'll be capped at the ones released with JEE 9 a while ago.

So I've been pondering my options here.

In one sense, I solved this problem years ago. The Domino Open Liberty Runtime project has had the ability to download any version of open-source Java that you want, and I expanded it last year to let you pick from several common flavors. Liberty maintains a breathless pace of advancement, adding official support for Java 18 the month after it came out. If one wants to run JEE apps on Domino, that's the most complete way. However, though it does its job technologically well, it's not exactly a natural fit for Domino developers in its current state.

But I've been considering anew a notion I had years ago, which is to write an extension for Liberty so that it reads class files and resources out of an NSF directly. In some early investigation a bit ago, this started to appear quite doable. In theory, I could write an adapter that would take an incoming request for "foo.nsf" and then read files out of the NSF in the same way XPages does, but instead feeding them to Liberty's runtime. Doing this would essentially implement all remaining JEE and MicroProfile specs in one fell swoop on top of the "any Java version" support, but would add the fault-prone attribute of running a separate process and proxying requests to it. In practice, that setup has proven itself good, but it's certainly more complicated than the "single process on port 80" deal that Domino's HTTP is now.

That route also wouldn't inherently support XPages, which would be something of an impediment to the XPages JEE project's original remit. That's something I've also pondered, and in theory I could make an auto-vivifying version of the XPages Runtime project that grabs all the pertinent XPages bundles from the current server and patches them into the Liberty server as an extension feature, similar to how all the built-in Liberty features work. This could be done, but I'll admit that I balk a bit at the prospect. Though I run XPages outside Domino constantly, it's with full knowledge of the tradeoffs and special considerations. Getting a normal NSF-based XPages app to run in this way would take some additional work.

Anyway, those options could work, but none of them are great. The true fix would naturally be for HCL to move to a newer Java version in Domino's HTTP stack, but I don't control that, so I'll content myself with considering what to do in the mean time. Admittedly, pondering this sort of thing is enjoyable in its own right. Also fortunately, even without tackling this, there's still plenty of stuff in the pile for me to tackle as the fancy strikes me.

Structure of the Domino Web App Container

Fri Feb 25 15:01:33 EST 2022

A while back, I talked about the uses of HttpService in Domino. In that post, I talked about how the various HttpService implementations take a look at incoming URLs, see if they're something that should be handled on the Java layer, and then either handle them or pass them back to the legacy NHTTP code to do its thing. My fiddling with the XPages Jakarta EE project in recent days has gotten me thinking about this layer again, and I think it'll be interesting to expand how how this whole layer works (at least as I understand it).

Along with this post, it might be useful to peruse the slide deck for AD105 from LotusSphere 2011. There's a lot of good stuff in there, and basically nothing has changed in the intervening 11 years.

The Stack (Conceptually)

The Domino HTTP stack looks conceptually something like this:

Diagram of the Domino HTTP stack

This is, setting aside the specifics, pretty similar to how other app servers of various kinds are laid out. There's some bottom layer that handles the actual network connection, some part just above that that handles interpreting the requests as HTTP (optionally bypassed in some cases), and then an orchestrator that manages the actual apps sitting on the top layer and routes requests as appropriate.

The Stacks (Java-wise)

Before I continue, I think it will be useful to have some stack traces to reference back to, to see what they share in common and where they diverge. These three examples - from an XPage request, an Equinox-registered Servlet, and an OSGi-packaged webapp - all cover the part of the stack from the bottom up until where user code comes into play.

XPages (DesignerFacesServlet is what handles serving an XPage):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
     at com.ibm.xsp.webapp.DesignerFacesServlet.service(DesignerFacesServlet.java:103)
     at com.ibm.designer.runtime.domino.adapter.ComponentModule.invokeServlet(ComponentModule.java:600)
     at com.ibm.domino.xsp.module.nsf.NSFComponentModule.invokeServlet(NSFComponentModule.java:1352)
     at com.ibm.designer.runtime.domino.adapter.ComponentModule$AdapterInvoker.invokeServlet(ComponentModule.java:877)
     at com.ibm.designer.runtime.domino.adapter.ComponentModule$ServletInvoker.doService(ComponentModule.java:820)
     at com.ibm.designer.runtime.domino.adapter.ComponentModule.doService(ComponentModule.java:589)
     at com.ibm.domino.xsp.module.nsf.NSFComponentModule.doService(NSFComponentModule.java:1336)
     at com.ibm.domino.xsp.module.nsf.NSFService.doServiceInternal(NSFService.java:725)
     at com.ibm.domino.xsp.module.nsf.NSFService.doService(NSFService.java:515)
     at com.ibm.designer.runtime.domino.adapter.LCDEnvironment.doService(LCDEnvironment.java:363)
     at com.ibm.designer.runtime.domino.adapter.LCDEnvironment.service(LCDEnvironment.java:319)
     at com.ibm.domino.xsp.bridge.http.engine.XspCmdManager.service(XspCmdManager.java:272)


Equinox Servlet (the org.eclipse.equinox.http.registry.servlets extension point):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
	at com.example.SomeServlet.service(SomeServlet.java:104)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:806)
	at org.eclipse.equinox.http.registry.internal.ServletManager$ServletWrapper.service(ServletManager.java:180)
	at org.eclipse.equinox.http.servlet.internal.ServletRegistration.handleRequest(ServletRegistration.java:90)
	at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:111)
	at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:67)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:806)
	at com.ibm.domino.xsp.adapter.osgi.OSGIModule.invokeServlet(OSGIModule.java:167)
	at com.ibm.domino.xsp.adapter.osgi.OSGIModule.access$0(OSGIModule.java:153)
	at com.ibm.domino.xsp.adapter.osgi.OSGIModule$1.invokeServlet(OSGIModule.java:134)
	at com.ibm.domino.xsp.adapter.osgi.AbstractOSGIModule.invokeServletWithNotesContext(AbstractOSGIModule.java:181)
	at com.ibm.domino.xsp.adapter.osgi.OSGIModule.doService(OSGIModule.java:128)
	at com.ibm.domino.xsp.adapter.osgi.OSGIService.doService(OSGIService.java:418)
	at com.ibm.designer.runtime.domino.adapter.LCDEnvironment.doService(LCDEnvironment.java:363)
	at com.ibm.designer.runtime.domino.adapter.LCDEnvironment.service(LCDEnvironment.java:319)
	at com.ibm.domino.xsp.bridge.http.engine.XspCmdManager.service(XspCmdManager.java:272)


Web Container (the com.ibm.pvc.webcontainer.application extension point):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
	at ccom.example.ExampleServlet.doGet(ExampleServlet.java:18)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:693)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:806)
	at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1661)
	at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:937)
	at com.ibm.pvc.internal.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:85)
	at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:500)
	at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3810)
	at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:276)
	at com.ibm.pvc.internal.webcontainer.VirtualHost.handleRequest(VirtualHost.java:143)
	at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:931)
	at com.ibm.pvc.internal.webcontainer.WebContainerBridge.handleRequest(WebContainerBridge.java:25)
	at com.ibm.domino.osgi.core.webContainer.WebApplicationsTracker.doService(WebApplicationsTracker.java:141)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.ibm.domino.xsp.adapter.osgi.webContainer.OSGIWebContainerModule.invokeWebAppContainerService(OSGIWebContainerModule.java:207)
	at com.ibm.domino.xsp.adapter.osgi.webContainer.OSGIWebContainerModule.doService(OSGIWebContainerModule.java:178)
	at com.ibm.domino.xsp.adapter.osgi.OSGIService.doService(OSGIService.java:418)
	at com.ibm.designer.runtime.domino.adapter.LCDEnvironment.doService(LCDEnvironment.java:363)
	at com.ibm.designer.runtime.domino.adapter.LCDEnvironment.service(LCDEnvironment.java:319)
	at com.ibm.domino.xsp.bridge.http.engine.XspCmdManager.service(XspCmdManager.java:272)

Each one of these has some intriguing lines, but we'll end up focusing on the bottom 5-6 in each.

Anyway, back to the details.

Orchestrator

The bottom three lines of all three stacks are identical, and they show the entrypoint from the C side to Java.

The job of XspCmdManager is to take a bunch of handles and flags given to it by the C side, wrap them into something a little more suitable for polite company, and pass that off immediately to LCDEnvironment. At this level, while the code is clearly focused around getting to the point of handling Servlets, the actual classes don't actually implement javax.servlet parts - they're all a little more abstract than that. XspCmdManager has a few other responsibilities, but it's best thought of as just the glue layer between the native side and the Java stack.

From there, it passes the request along to LCDEnvironment, as the sole implementation of LCDRequestHandler. Things get a little meatier here. This is the part that loads up all of the HttpService implementations we've discussed before. It uses these services to answer calls to isXspUrl (which basically means "should Java handle this?") and then to handle the incoming requests. The first HttpService (sorted by its getPriority() value) that will handle the incoming request gets it, and here's our first point of divergence above. NSFService is the in-NSF XPages-and-stuff handler, taking care of requests with ".nsf" and either "xsp" or a registered extension in the path. OSGIService, for its part, handles both Equinox-registered servlets and Expeditor webapps, albeit in different ways.

ComponentModules

The next layer is interesting, and it's a part I didn't really talk about in the previous post. While an HttpService can just handle an incoming request directly (as the proxy service in the Domino Open Liberty Runtime project does), the idiom in the Domino stack is to use ComponentModule implementations to do it. These correspond conceptually to web apps deployed from WAR files in a standard app server: they're a cordoned-off blob of user code, with its own ClassLoader and notions for how to access resources.

For an NSF, this is NSFComponentModule. These objects are spawned by NSFService as needed when a matching request comes in for an NSF the first time and create a weird sort of webapp out of the NSF contents (with special handling for Single-Copy XPage Design). It doesn't go as far in that direction as the full web container, but it's enough to run and retain the XPages application. This type of module also opts in to the IServletFactory extension system, where contributors can add Servlets to the module either internally or via an OSGi extension. All ComponentModule types have the ability to opt in to this, but NSFComponentModule is the only one that does in practice.

Though both the Equinox Servlet and the PVC web container app go through OSGIService, here is where they split apart into OSGIModule (for standalone Servlets) and OSGIWebContainerModule.

OSGIModule uses some of the parts of the Equinox Servlet Container support to run an individual Servlet. It's the smallest of the three and mostly just creates the ServletRequest et al implementations, does a little wrapping in Equinox garb, and passes them on to your HttpServlet class.

OSGIWebContainerModule is fancier, since its job is to run (most of) a JEE-style web.xml-based app contained in an OSGi bundle. To do this, it uses a chopped-down fork of WebSphere - indeed, many of the classes in the stack still exist in much the same form in Liberty, but here are intermingled with Domino-specific stuff and some Expeditor PvC detritus. This isn't quite as capable as a full Servlet 2.5 container, lacking things like Filters and various listeners, but it gets the job done.

Module Adaptability

Though this system hasn't in practice grown beyond the built-in implementations to my knowledge, it's a neat little structure. There's some unfinished stuff in there in a mashupmaker package that I guess must have been to do with Lotus Mashups (which is apparently a product that existed at some point), but that's about it as far as extending it.

This is an intriguing layer, though, since it's also the one where the "adapter" Servlet objects from above are actually turned into instances of javax.servlet classes, specifically using classes like LCDAdapterHttpServletResponse. At this layer, you have a good amount of Java scaffolding, but being before the full conversion to javax.servlet classes means that a lot of the limitations in Domino's Servlet support only really come in in the upper echelons. Other than network- and HTTP-layer technologies like WebSocket and HTTP/2 (which are handled at the native layer), it'd be entirely possible to plug into this system with more-modern technologies while still participating cleanly in the environment. For example, you could write an HttpService that declares a higher priority than NSFService and use it to treat an NSF as an entirely-different sort of app, intercepting all URLs prefixed with it. I don't know that it would be a good idea to do so, but it's possible, and it's fun to think about.

What To Do With All This XSP Markup? Redux

Thu Feb 17 14:53:01 EST 2022

Tags: jsf xpages

About a year ago, I wrote a post discussing what I saw as potential future options for all the XPages code we've collectively written over the years. That post was written with the assumption that the future life of an XPages app isn't XPages as it exists today - while that's still a possibility, it's a lot less fun to speculate about.

As it has been for years, this remains on my mind, but my recent addition of JSF to the XPages Jakarta EE project caused it to bubble back to the top of my musings.

Conceptual Tools

Before getting into some speculative specifics, I'd like to first take a step back and assess the concepts we're working with.

In that post, and as I frequently do, I referred back to an old post of mine discussing how an XPage is best thought of as a tree of components, one most-commonly described by way of the XSP markup language. The critical concept there is that there's nothing that inherently ties an XPage to the specific thing that sends HTML to the browser, and all the moreso when you're talking about the XML. While "XPages" as an entity on Domino encompasses an entire stack that starts just barely above the bottom of nHTTP, "an XPage" as such is almost always just XML that describes what you want to happen on the page. There are parts of it that are more specific than others, for sure: <xp:inputText/> has equivalents in a million other languages, but value="#{javascript: ...}" starts getting into some gnarly implementation specifics. Still, the point is that anything that can successfully interpret XSP markup can be considered functionally "XPages" for most uses.

The next handy concept to keep in mind is related, and that's that code is data. This is most apparent with an XML-based language like XSP, but it applies to everything. SSJS code is absolutely data: it's just strings that happen to be parsed out at runtime as ASTs and then executed. For SSJS, there's no explicit guarantee that facesContext refers to an instance of javax.faces.context.FacesContext or one of its com.ibm.xsp subclasses. Heck, there's no specific requirement that even referring to the class by name would reference the same thing. It's just data and can be interepreted as the runtime seems fit.

And that goes further: Java bytecode is just data too. OSGi has a mechanism to alter classes at load time that already exists and works great on Domino. Also pertinently, Eclipse Transformer is a tool that translates code (source or compiled) that references javax.* JEE classes to jakarta.* classes, and is used commonly now by webapp runtimes to support both legacy and new apps with the same codebase. Fancy stuff, that.

The Musing

So back to what I've been pondering. I think I did mostly a good job covering the bases in my original post, but time and experience has given me further perspective.

JSF Driver For XSP

When I first mentioned it, I brushed the concept of writing a driver for JSF off a bit - not fully, but I did give it a shorter shrift than it deserved.

To begin with, this concept exists in JSF: the view declaration language. Now, it's not actually used for this sort of "transplant" thing, mind you: in practice, it's a concept that was used to transition from JSP as the language to write JSF pages to Facelets and not much else. Still, there's no inherent reason why one couldn't write a VDL driver for JSF that would interpret XSP markup and create components based on it. This is especially true because XSP itself doesn't really contain any real curveballs: it's a pretty basic mapping of tags to component names and attributes to bean properties, with the main hiccup being <xp:this.action>-type complex properties.

Components

I've also given some thought to that tag-to-component mapping. One way to do that would be to make an interpreter that sees <xp:inputText/> and, instead of translating it to com.ibm.xsp.component.UIInputText, would instead translate it to a newer stock component. I thereby brushed it off as being properly too complex to be workable, as XPages's nature as a hard fork of JSF meant that things like the java.faces.component.UIComponent#_xspGetStateId method would prove intractable.

I'm less sure of this difficulty now, though. The switch from javax.faces to jakarta.faces means there's room for coexistence on the same classpath, as I now have in the JEE project. This means there's a lot of room for adapting older code unchanged by way of using wrapper objects and proxies.

For the former, what I mean is that, while a JSF 4.x implementation like MyFaces 4 now has no knowledge of what a javax.faces.component.UIInput is, it doesn't have to: all it needs is a subclass of jakarta.faces.component.UIComponent that can fit into a tree and respond to normal JSF calls like processUpdates. The "real" JSF stack in front could pass in incoming form data from the client just the same as it does now, and a wrapped legacy XPages class could handle it exactly as it does now. Things get more complicated than that, but not impossibly so.

This is similar to all the Servlet-object wrapping and unwrapping I do in the JEE support project. These wrappers interpret and route equivalent method calls to the delegates they're wrapping, and so Servlet 5 code doesn't care that it's working with a Servlet 2.4 request, and similarly the Servlet 2.4 code can continue to know nothing at all about Servlet 5. The glue code handles the simple wrapper-based translations between the layers.

So, in this way, the XPages fork of JSF could remain essentially untouched. As long as the runtime does things like inserting an appropriate old-style FacesContext object into javax.faces.context.FacesContext and the like, no one needs to be the wiser.

And this is where Java object proxies could come into play. Where I've above said things like "an instance of javax.faces.context.FacesContext" or "an... object", that doesn't even need to be a real class that you construct with new Foo(). While Java's built-in proxies only work with interfaces, libraries like Javassist can generate proxy objects for true classes as well. One could have a proxy object that checks to see if an incoming method is compatible with the new style and use that one, or otherwise route to a wrapped class, and yet remain class-cast compatible with old code. I'm not sure this would be required if wrapping was done fully, but it's good to know as an option.

Code

Then there's user code. I was going to have a bunch of stuff to say in this section, but it's actually all largely covered by the steps that would be necessary for keeping runtime code compatible, if that were the route to take. If user code calls javax.faces.context.FacesContext#getCurrentInstance, then it'd get a legacy object; if it calls jakarta.faces.context.FacesContext#getCurrentInstance, then it'd get the new-era one. That'd be the same as the work necessary to keep the old parts chugging along.

Now, if the old parts were to change - were the task to be to make it so that the com.ibm.xsp classes are based on JSF 4.x - then there'd be some fiddling to do for user code. But, again, it'd be largely the same idea. You could either translate the code as it's loaded (for Java) or executed (for SSJS) or you could pass wrapper objects around. So all the same ideas apply.

Coexistence

I've also been thinking a lot about the notion of an "XPages 2" or the like existing alongside XPages as it is now. For example, you could imagine a variant of "XPages" that is indeed a JSF view definition language that interprets XSP markup, but which doesn't attempt to guarantee pure compatibility with existing code. Maybe it'd get 80% of the way there - it'd interpret components in largely the same way, but wouldn't guarantee that every little bit of code that dives down into the runtime implementation would work identically, and wouldn't guarantee that every fiddly detail of the XPages lifecycle and its optimizations would execute in the exact same way.

In such a scenario, there wouldn't be any particular need to remove or change the old stuff. This is the scenario that JEE app servers all faced with the jakarta.* move: you can't realistically expect every app to be transformed (automatically or otherwise) to work with the new versions, especially because old apps may target even-older specs that have since had other breaking changes.

In Domino, there are a few ways one could go about accomplishing this. One way would be to do kind of like what I do for JSP and JSF: tell NSFService to handle a given extension and then write a factory to handle it. In this way, you could declare a new extension, say ".xspx" (note: please do not use this extension), and then route incoming requests to a handler that's like normal XPages but not 100% compatible.

Alternatively, you could do something I've pondered for a while, which is to make another HttpService implementation that would check for a flag in the database referenced in incoming .nsf-containing URLs to see if it's set to "new mode" and, if so, interpret the request however which way it would like. Such a service could disregard all old rules for URL handling if it wanted, and could worry much less about the potential of cross-contamination between request types. It'd be more work, but is an intriguing possibility.

Conclusion

So is any of this the right way to go about dealing with our dear old friend XPages? I don't know - maybe. This is all pretty off-the-cuff, and I haven't necessarily thought through all the implications, but I do feel like there's more wiggle room here than I'd originally assumed. It's interesting to think about, at the very least.

Video Series On The XPages Jakarta EE Project

Mon Feb 07 15:54:15 EST 2022

  1. Updating The XPages JEE Support Project To Jakarta EE 9, A Travelogue
  2. JSP and MVC Support in the XPages JEE Project
  3. Migrating a Large XPages App to Jakarta EE 9
  4. XPages Jakarta EE Support 2.2.0
  5. DQL, QueryResultsProcessor, and JNoSQL
  6. Implementing a Basic JNoSQL Driver for Domino
  7. Video Series On The XPages Jakarta EE Project
  8. JSF in the XPages Jakarta EE Support Project
  9. So Why Jakarta?
  10. XPages Jakarta EE 2.5.0 And The Looming Java-Version Wall
  11. Adding Concurrency to the XPages Jakarta EE Support Project
  12. Adding Transactions to the XPages Jakarta EE Support Project
  13. XPages Jakarta EE 2.9.0 and Next Steps
  14. XPages JEE 2.11.0 and the Javadoc Provider
  15. The Loose Roadmap for XPages Jakarta EE Support
  16. XPages JEE 2.12.0: JNoSQL Views and PrimeFaces Support
  17. XPages JEE 2.13.0
  18. XPages JEE 2.14.0
  19. XPages JEE 2.15.0 and Plans for JEE 10 and 11

Over the last two weeks, Graham Acres and I recorded a video series for OpenNTF about my XPages Jakarta EE Support project, which has seen a flurry of development in the last few months. The 15-part series is up on YouTube:

The project itself saw the release of version 2.3.0 today, which is the first release with the Jakarta NoSQL driver I blogged about recently.

I think the project has turned into a pretty-interesting "platform update" for XPages, and I hope the video series captures that a bit. I'm still mulling over a sort of "thesis statement" about the whole thing, but for now describing the various new capabilities and how they interact will have to suffice.

Intercepting Class Loading in OSGi, A Travelogue

Mon Jan 10 10:36:08 EST 2022

Tags: java osgi xpages

Yesterday, I had a problem. I was trying to get MicroProfile Config working inside an NSF to add to the XPages Jakarta EE project, and I was severely blocked by odd behavior.

To describe that, I'll lightly cover what MP Config is. It's a CDI extension that allows you to annotate properties on a bean to indicate that they're intended to come from an available configuration source - often a .properties file in the project, but it's a pluggable system. Your bean will look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
package example;

// ...

@ApplicationScoped
public class ConfigExample {
    @Inject
    @ConfigProperty(name="java.version", defaultValue="unknown")
    private String javaVersion;
    
    /* other methods go here */
}

The idea is that you'll then have a properties file or environment variable to fill in the value, allowing you to separate your configuration from the implementation in a consistent way. Here, I'm making use of the fact that a default provider looks up Java system properties, so I could just get it working before investigating adding providers.

Since I'd already added CDI and a CDI-based extension in the form of MVC, I figured this would be easy.

The Problem

The problem I hit, though, was bizarre. CDI would identify the bean above, but would hit this problem:

1
2
3
4
5
org.jboss.weld.exceptions.DeploymentException: WELD-001408: Unsatisfied dependencies for type String with qualifiers @Default
  at injection point [UnbackedAnnotatedField] @Inject private example.ConfigExample.javaVersion
  at example.ConfigExample.javaVersion(ConfigExample.java:0)
WELD-001475: The following beans match by type, but none have matching qualifiers:
  - Producer Method [String] with qualifiers [@Any @ConfigProperty] declared as [[UnbackedAnnotatedMethod] @Dependent @Produces @ConfigProperty protected io.smallrye.config.inject.ConfigProducer.produceStringConfigProperty(InjectionPoint)]

The gist of this is that it noticed that the javaVersion property is supposed to be an injected property, but it had no idea what the source should be. It did know about the MicroProfile provider, which handles @ConfigProperty, but it couldn't put two and two together.

I banged my head against this for a while, and eventually determined that the class as loaded from the NSF is stripped of the @ConfigProperty annotation outright. Other annotations, such as @Inject and even custom annotations, would remain, but not @ConfigProperty. I wrestled with OSGi dependency chains for a while, to no avail.

The Enemy

Eventually, I found the core, and it was an old nemesis of mine. It's this method in com.ibm.xsp.util.ClassLoaderUtil:

ClassLoaderUtil.checkProhibitedClassNames

This field is called by the ClassLoader used in an NSF to ensure that certain classes, by name prefix, cannot be loaded by code coming from an NSF. The last three lines there make a sort of sense: Domino is supposed to be an app container for XPages apps, and ideally it's not a simple process for an app to break out of its container to muck about in the parent environment. Fair enough. The NAPI line is presumably there because IBM wanted to protect developers from themselves, even though Notes devs had been making unauthenticated calls to C APIs for freaking ever.

It's the first two, and specifically the first, that are the source of my trouble. Those prohibitions are presumably meant to isolate XPages apps from the fact that they live in an OSGi world, with the assumption that anything beginning with org.eclipse. refers to something like org.eclipse.core.runtime, the OSGi system bundle.

And this is the issue. MicroProfile is not in any way related to OSGi, but it sure is an Eclipse project. Accordingly, the class name of @ConfigProperty is org.eclipse.microprofile.config.inject.ConfigProperty, and thus cannot be loaded from an NSF.

Attempted Workarounds

So I considered my options.

One was to fork MP Config and rename the packages. That would work, but it would defeat the portability goals of the XPages JEE project, and would also just be a hassle - I've already had to fork a few specs, and each new one adds to the maintenance burden. That remained an option, but it would be a last resort.

My next idea was to wrap around the ModuleClassLoader class used by NSFComponentModule for class loading purposes. This class is blessedly non-final, and so in theory I could look at the instance in a module and swap it out with a replacement. I tinkered with this a bit, but the trouble became the way it's layered, with a DynamicClassLoader private class within - something harder to subclass. In theory, I could reproduce the behavior of it wholesale, but that would be both fragile (if the implementation ever changes) but also verging on if not outright illegal (it's one thing to be API-compatible, and another to reproduce the internal functionality). After some wrangling, I decided to look elsewhere.

The True Workaround

I realized eventually that I don't really care about ModuleClassLoader as such: it does its job fine, and it's only the response that it gets from ClassLoaderUtil that is the problem. If I could change that, I would be set.

I've used the Javassist project here and there for a long time, ever since its inclusion in ODA for one reason or another. It's a handy toolkit, and notably includes the capability to alter a method implementation on the fly. There's my loophole.

The reason this kind of thing can work is related to how Java handles classes and calls between them. For all intents and purposes, you can consider a method call from one bit of Java to another to be a string-based lookup, saying "find a class named X and a method named Y, and then execute it". The "find" part there is much looser than you might think. It's easy to think of class references like C static linking, but they're really not. When code asks for a class, it asks the context ClassLoader, and that object can do basically whatever the heck it wants to find it, as long as it eventually emits a Class that the runtime can deal with.

Javassist's manipulation makes use of the fact that classes are generally eventually just a bunch of bytes, and you can do whatever you want with a bunch of bytes. Using Javassist, it's fairly simple to, once you have a handle on the class, alter the method. Truncated, that's:

1
2
3
4
5
6
ClassPool pool = /* build a ClassPool that can load the class */;
CtClass cc = /* get the class from the pool */;
cc.defrost();
CtMethod m = cc.getDeclaredMethod("checkProhibitedClassNames");
m.setBody("{ return false; }");
Class<?> result = cc.toClass();

And this works, as far as it goes: I now have a Class version of ClassLoaderUtil that skips the onerous check.

The trouble now was to get this to be actually used by other classes. Generally, once a ClassLoader loads a class, it's difficult to feed it another version unless it's designed to do so: most ClassLoader implementations, including those used here, are designed to read and emit classes by their own rules, not have new data fed into them.

I tried digging through the Eclipse OSGi ModuleClassLoader (distinct from the NSF ModuleClassLoader) for entrypoints and had some initially-promising work with Eclipse's internal ClassLoaderHook type, but eventually determined that this would require more patching than I'd want, if it was possible at all.

I also considered using Java's instrumentation capabilities to intercept class loading, but that would require setting up a special Java agent in the launch parameters, which would be too onerous.

But then I remembered something I had heard about when looking into getting ServiceLoader to work with OSGi: a concept in the OSGi spec called "weaving".

OSGi Weaving

I had noted that this concept existed, but set it aside in large part due to how esoteric it sounds: the term "weaving" makes it sound like it's a way to interact with the threads of fate or something, which is evocative but not something that seems immediately useful.

What it really is, though, is an OSGi-friendly version of the above: when the OSGi runtime goes to load a class from a bundle, it reads the data but then gives any such listeners an opportunity to manipulate the code before it's actually reified into a class. This is how the ServiceLoader mediator does its thing: it looks for ServiceLoader calls during loading and re-"weaves" them to run through OSGi instead.

This was perfect: it provides exactly the hook I want and it does it in a clean, spec-based way, without having to do weird reflection to reassign object properties or anything.

The Implementation

So I went about writing such an implementation. All the pieces are there on Domino, and the mechanism for registering a WeavingHook is something I'd done before in Open Liberty: it's a type of OSGi service that you can register and manage in an Activator class. It's also the sort of thing that would work well with Declarative Services, but Domino doesn't have a DS handler installed and I figured I didn't need to solve that quite yet.

So I wrote a WeavingHook implementation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
public class UtilWeavingHook implements WeavingHook {
    @Override
    public void weave(WovenClass c) {
        if("com.ibm.xsp.util.ClassLoaderUtil".equals(c.getClassName())) {
            ClassPool pool = new ClassPool();
            pool.appendClassPath(new LoaderClassPath(ClassLoader.getSystemClassLoader()));
            CtClass cc;
            try(InputStream is = new ByteArrayInputStream(c.getBytes())) {
                cc = pool.makeClass(is);
            } catch (IOException e) {
                throw new UncheckedIOException(e);
            }
            cc.defrost();
            try {
                CtMethod m = cc.getDeclaredMethod("checkProhibitedClassNames");
                m.setBody("{ return false; }");
                c.setBytes(cc.toBytecode());
            } catch(NotFoundException | CannotCompileException | IOException e) {
                new RuntimeException("Encountered exception when weaving ClassLoaderUtil replacement", e).printStackTrace();
            }
        }
    }
}

This builds on the above Javassist usage to now load the class from the byte array provided by OSGi, transform it, and then write the new version back. Since this happens while OSGi is reading the class to begin with, there's never a time when there's an older, less-permissive version of the class running around, as long as I get my service in early enough.

This service is registered in the Activator without too much fuss:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
public class JakartaActivator implements BundleActivator {
    private final List<ServiceRegistration<?>> regs = new ArrayList<>();

    @Override
    public void start(BundleContext context) throws Exception {
        regs.add(context.registerService(WeavingHook.class.getName(), new UtilWeavingHook(), null));
    }

    @Override
    public void stop(BundleContext context) throws Exception {
        regs.forEach(ServiceRegistration::unregister);
        regs.clear();
    }
}

The final bit to get right was the "get the service in early enough" aspect. The main task was making sure that this bundle was activated before any XPages apps loaded, and that was a job for my old friend IServiceFactory, which is the extension point that's intended to add handlers for incoming URLs but has the desirable attribute of being initialized right at the very start of HTTP loading.

With this in place, I now have a fix automatically applied to that fiendish class on load, and MicroProfile Config (and future MP specs) works like a charm.

Conclusion

This was an arduous one, and I think the FTL victory jingle actually physically played when I got it working. I've hated this restriction for a long time, and I'm glad to finally have a workaround.

It was also enlightening to properly learn about OSGi's weaving capability. As mentioned above, this is what the ServiceLoader bridge does, and I'd tinkered with that at one point, but never got it working. I suspect now that it should be entirely doable to make it work, most likely also involving bringing in an implementation of the Declarative Services OSGi capability. That should be a fun project in its own right.

Moreover, the fact that I now have a system in place to do this weaving on the fly means that I may be able to un-fork some of the specs I had to fork to get working previously, which specifically required altering ServiceLoader calls. Even if I don't get the official service bridge in, perhaps I can use this technique to just alter the parts I need to on the fly, and otherwise use stock implementations from Maven.

But, for now, the way is cleared for further progress, and a bizarre mystery is solved. I call that a good day.

Migrating a Large XPages App to Jakarta EE 9

Thu Jan 06 19:44:57 EST 2022

  1. Updating The XPages JEE Support Project To Jakarta EE 9, A Travelogue
  2. JSP and MVC Support in the XPages JEE Project
  3. Migrating a Large XPages App to Jakarta EE 9
  4. XPages Jakarta EE Support 2.2.0
  5. DQL, QueryResultsProcessor, and JNoSQL
  6. Implementing a Basic JNoSQL Driver for Domino
  7. Video Series On The XPages Jakarta EE Project
  8. JSF in the XPages Jakarta EE Support Project
  9. So Why Jakarta?
  10. XPages Jakarta EE 2.5.0 And The Looming Java-Version Wall
  11. Adding Concurrency to the XPages Jakarta EE Support Project
  12. Adding Transactions to the XPages Jakarta EE Support Project
  13. XPages Jakarta EE 2.9.0 and Next Steps
  14. XPages JEE 2.11.0 and the Javadoc Provider
  15. The Loose Roadmap for XPages Jakarta EE Support
  16. XPages JEE 2.12.0: JNoSQL Views and PrimeFaces Support
  17. XPages JEE 2.13.0
  18. XPages JEE 2.14.0
  19. XPages JEE 2.15.0 and Plans for JEE 10 and 11

Last month, I moved my XPages Jakarta EE project to JEE 9, which involved the large hurdle of switching package names from javax.* to jakarta.*. That was all well and good for the project and opened the door to further improvements, but it's one thing to do it in a support library and another to move an actual large project over.

So I set my sights on my workhorse client project, the one with the sprawling OSGi bundle set and complicated XPages project, which I have running regularly on both Domino and Open Liberty. Over the last few days, I did the porting work and came out successful, and I think it ended up being another tale worth telling.

There are two main topics when it comes to this project: why and how.

Why

Now, why are we gonna do this? I mean, the app is running fine as it is, and the immediate goal of the switch is to keep it functionally the same. Why is it worth going through this hassle for an active project?

There are a few reasons.

The first and biggest is that the jakarta.* switch isn't going to go backwards. There is never going to be a feature update for any of the javax.* versions of the JEE specs, and so staying on that version is stagnation. While we can do what we want to do today, that won't hold true tomorrow unless we make the move. Since that's inevitable, it meant that every new line of javax.* code is technical debt, and the sooner you can stop creating that, the better.

The second reason is that, while the Jakarta EE spec move from 8 to 9 retained the same functionality, we actually gained technical improvements in the switch.

One technical gain was very immediate: the version of RESTEasy I had put into the XPages JEE project previously was a couple major versions old now, and this let me bump it up to the latest.

Another technical gain had to do with the nightmare that is dealing with OSGi dependencies on Domino. Due to the lack of proper versioning of standard specs in the XPages stack, the increasingly-corrupt classpath, and the bundling of specs with utility libraries, I've always had to lose a lot of time dealing with manually tweaking OSGi imports and dependencies. While this move doesn't eliminate all of it, it removes a lot. Terror packages like javax.mail and javax.activation can now be left behind in favor of jakarta.mail and jakarta.activation without worry of conflicting imports from the XPages libraries and the ndext classpath directory. The move to JEE 9 resulted in a significant reduction in such code and configuration.

And finally, it's going to help bring in new features we haven't introduced yet but will benefit from. For example, I have my eye on the MicroProfile REST client, which makes consuming REST services in a clear and type-safe way a dream. While it's possible that I'd be able to add that in earlier versions, the switch to jakarta.* will remove huge tasks from my plate entirely.

How

So now that I'd convinced myself that it's a good idea, the only remaining problem was actually doing it. Fortunately, I already solved most of it in the XPages JEE project itself. Moreover, the way I'd solved things there allowed me to remove extra dependencies that kind of "double-solved" the problem in this specific app, things like making sure that javax.inject was compatible between that project and the other third-party dependencies we use.

One of the big things that was different between the XPages JEE project on its own and this app was the way this runs across multiple platforms.

Historically, the fact that Liberty was running Servlet 4 and Domino was running Servlet 2.4/2.5 didn't come much into play. The newer versions of the javax.servlet classes were entirely compatible with the old ones: XPages could consume a Servlet 4 HttpServletRequest without issue and would just ignore the new methods added. Now, though, I had to do some shimming in two directions.

Domino, since I don't control the lowest layers, would remain a Servlet 2.5-ish container natively, while for Liberty I would move to a true Servlet 5 container. Since both the XPages markup and app code must remain the same, that meant a double emulation setup:

Diagram of the Domino and Liberty Stacks

In these diagrams, "JEE 9 App" represents the app's use of Jakarta standards other than XPages: CDI, JAX-RS, JSON-B, and so forth.

The first part of this work took place in the XPages Jakarta EE project. There, I created wrapper versions of all pertinent javax.servlet/jakarta.servlet classes going both from "old to new" and "new to old". Most of these classes are just direct delegations, but some parts involve either throwing exceptions for trying to use new features on an old stack, emulating new behavior on top of the old, or quietly not supporting some capabilities. These classes get me most of the way. When I need to move in one direction or the other, I call the appropriate method from ServletUtil and it takes care of the differences. This project also handled a lot of fiddly details to do with things like the JavaMail to Jakarta Mail switch, so I could just bring that in too and not worry about it.

That handled some distinctions. The next big one was to use this to make sure XPages can survive in a Servlet 5 world. The good news here is that it was simpler than I thought it would be. XPages only has a few actual entrypoints - there's a Servlet to handle global resources (the URLs involving /xsp/.ibm* and the like), another to handle actual XPages *.xsp requests, and maybe one or two others I'm not thinking of. As long as you can handle those URLs and send a legacy javax.servlet object to the closed-source code, the stack will take it from there: things like externalContext.getRequest() will return the wrapped object you passed in in the first place, and don't try to do any weird magic to fetch the request object from the container or anything.

Previously, I had Servlets that extended the stock ones like DesignerFacesServlet directly, but this had to change. Fortunately, it's a simple matter of delegation. Instead of subclassing the original ones, now I create an instance internally and pass along incoming requests to that, appropriately dumbing down the Servlet 5 objects to 2.5-level ones. I was able to do this with fewer functional changes than one might think and put it into new versions of the XPages Runtime project.

Beyond those big-ticket items, there were a few cases where I had to take care to appropriately handle knowing when my code was going to receive a javax.servlet object or a jakarta.servlet one, but otherwise there wasn't much to do beyond updating dependencies and a find-and-replace on class imports. None of the XSP markup changed, very little of the in-NSF code changed, and the only large in-app code changes I made were to remove workarounds that were no longer needed.

Conclusion

All in all, I'd say this went better than I'd expected. There's naturally still room for trouble (it's not in production yet, for one), but overall I feel that this bore out my intent in making the move in the first place.

It's also an interesting case study in the way my XPages Jakarta EE and Open Liberty Runtime projects conceptually interact. They both approach the same ideal from different directions: write open-standards-based applications that make use of Domino data. With this move to JEE 9, the addition of new specs to the XPages JEE project, and the shedding of (some) old limitations, they're converging all the more. More code can be directly shared between the two app types, and the code that isn't shared unmodified can at least be written all the more similarly. JAX-RS is the in-common way to do REST services; CDI is the in-common way to do managed beans; OpenAPI annotations are an in-common way to document services. These are technologies that have wide support with multiple implementations and, crucially, are open standards. XPages was one step towards a non-proprietary stack, and this is several more.

XPages Renderers and Thread Safety

Thu May 27 09:56:13 EDT 2021

Tags: xpages

I got to thinking yesterday about renderers in XPages - the part of the stack that takes the abstract back-end representation and actually turns it into HTML. They've been an interest of mine for a good while and they have interesting characteristics worth revisiting from time to time.

One of those characteristics is their general statelessness: the way they work, there's generally only one instance of the renderer object, and then it's applied to many pages. This is reflected in the Javadoc for the base javax.faces.render.Renderer class (Java EE 5 here because that's what XPages is based on):

Individual `Renderer` instances will be instantiated as requested during the rendering process, and will remain in existence for the remainder of the lifetime of a web application. Because each instance may be invoked from more than one request processing thread simultaneously, they MUST be programmed in a thread-safe manner.

As a quick aside, that all-caps "MUST" is an application of the delightful RFC 2119, which defines common meanings for those types of words in specs. It's worth reading.

Implementation

Anyway, how does this shake out in practice? Well, we can use the Bootstrap theme as provided by IBM back when it was open source as a baseline example of how it's done. If you look at the, for example, ResponsiveAppLayoutRenderer class, you can see a ton of code, but no instance variables. Looking at any number of classes, there are static constants, but no instance variables. In general, that bundle as present on GitHub there is a great example of the craft.

In general, thread safety is tricky, and the easiest way to make a class thread-safe is to not have any instance variables. These renderer examples take that route, and for good reason: since renderers are instantiated once per app, they're potentially shared across all pages in the app, multiple components within a page, and multiple instances of the same page.

Demonstration

Thread safety in general and in Java in particular is a huge topic, and it's something that harries basically all programmers working in a potentially-threaded environment, even when the code you're writing doesn't seem troublesome. It's one thing if you're explicitly writing multithreaded code, divvying up tasks among executors or something, but what would be the trouble here? Well, fortunately, writing a demonstration is fairly quick. To do so, I made a new NSF with a few design elements. First, a theme:

1
2
3
4
5
6
7
8
9
<theme>
    <control>
        <name>ViewRoot</name>
        <property>
            <name>rendererType</name>
            <value>renderer.TestViewRoot</value>
        </property>
    </control>
</theme>

You can name the theme whatever you want, since the name of a theme is intended to be of human interest only. It will only matter that it's then selected in the Xsp Properties file.

Then, a customized faces-config.xml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
<?xml version="1.0" encoding="UTF-8"?>
<faces-config>
  <render-kit>
    <renderer>
      <component-family>javax.faces.ViewRoot</component-family>
      <renderer-type>renderer.TestViewRoot</renderer-type>
      <renderer-class>renderer.TestViewRoot</renderer-class>
    </renderer>
  </render-kit>
  <!--AUTOGEN-START-BUILDER: Automatically generated by HCL Domino Designer. Do not modify.-->
  <!--AUTOGEN-END-BUILDER: End of automatically generated section-->
</faces-config>

This has prepared us to use a custom renderer for the view root, which is the main page itself. Finally, as the last step before the renderer class, I made five XPages, named "home1.xsp" through "home5.xsp". The content is irrelevant, so I made them the simplest possible:

1
2
<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core"/>

Now, to the renderer class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
package renderer;

import javax.faces.component.UIComponent;
import javax.faces.context.FacesContext;
import javax.faces.render.Renderer;

import com.ibm.xsp.component.UIViewRootEx;

public class TestViewRoot extends Renderer {
    String viewId;
    
    @Override
    public void encodeBegin(FacesContext context, UIComponent component) {
        UIViewRootEx viewRoot = (UIViewRootEx)context.getViewRoot();
        viewId = viewRoot.getViewId();
    }
    @Override
    public void encodeEnd(FacesContext context, UIComponent component) {
        UIViewRootEx viewRoot = (UIViewRootEx)context.getViewRoot();
        String endingViewId = viewRoot.getViewId();
        if(!this.viewId.equals(endingViewId)) {
            System.out.println("finished rendering view ID "+ endingViewId + " - started with " + this.viewId);
        }
    }
}

This class gets the view ID (e.g. "/home1") in encodeBegin and again in encodeEnd. When the two are different - essentially, our bug condition - it emits a message to the console. This "viewId" is a stand-in for any sort of expected shared state, to demonstrate that the renderer can make no assumptions that encodeBegin and encodeEnd are called in sequence for the same page or page instance. The latter could be demonstrated by checking viewRoot.getUniqueViewId() instead, which identifies the specific page instance and is thus distinct even across different users on the same page.

Then, I wrote a script that, in a multithreaded fashion, requests home1.xsp through home5.xsp randomly for a little while, and it was only a couple seconds before the messages started appearing:

1
2
3
4
5
[0B4C:000A-14A0] 05/27/2021 09:25:21 AM  HTTP JVM: finished rendering view ID /home4 - started with /home2
[0B4C:0023-1388] 05/27/2021 09:25:21 AM  HTTP JVM: finished rendering view ID /home2 - started with /home1
[0B4C:001E-0C88] 05/27/2021 09:25:21 AM  HTTP JVM: finished rendering view ID /home2 - started with /home1
[0B4C:001B-1784] 05/27/2021 09:25:21 AM  HTTP JVM: finished rendering view ID /home3 - started with /home1
[0B4C:001C-1AE8] 05/27/2021 09:25:22 AM  HTTP JVM: finished rendering view ID /home2 - started with /home5

Knock-On Demo

Okay, so that's bad, but there's a subtle other problem I created here and that's to do with code reuse via subclassing. In general, renderers are subclassed out the wazoo. Just look at the type hierarchy for FacesRendererEx:

FacesRendererEx Type Hierarchy

It goes on for a while like that.

While renderers being subclass-friendly isn't a "MUST"-type rule like thread safety, it's both an associated benefit of that and a general cultural idiom. But imagine if I were to subclass my example above and override just the encodeBegin portion (to represent, say, changing just the output of the page header but leaving the footer the same):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
package renderer;

import javax.faces.component.UIComponent;
import javax.faces.context.FacesContext;

public class TestViewRootEx extends TestViewRoot {
    @Override
    public void encodeBegin(FacesContext context, UIComponent component) {
        // Do something else here
    }
}

If I use that as the renderer, I get a dramatically-new look for my page:

NPE on subclass

The trouble here is that I overrode encodeBegin but didn't call super.encodeBegin - which I naturally wouldn't, since I explicitly don't want whatever the parent class is outputting. However, since encodeEnd assumes that its version of encodeBegin runs and sets this.viewId, I hit a NullPointerException when it tries to access it. Curses.

Conclusion

Anyway, this is all a long-winded way of pointing out that designing for thread safety is tricky business, and it can crop up when you wouldn't otherwise expect it. It's good here that the JSF developers included that "MUST" business in the Javadoc, but, unfortunately, Java-the-language doesn't have a way of actually enforcing it, making room for this sort of thing to creep in.

What To Do With All This XSP Markup?

Mon Mar 29 14:51:29 EDT 2021

Tags: xpages

In some previous posts, I've started talking about some steps one can take to make a complicated XPages app more platform-independent. There's a lot to be done there, refactoring code to bridge differences between runtime environments and to lessen dependencies on XPages-specifics things, but there's a huge elephant in the room: all that XSP markup.

Even if you have a cleanly-structured application where all of your logic is in Java and all of that code doesn't make expectations about the UI, there's still bound to be a big pile of XPages XML markup around, and that's not going anywhere. That's the best case, too: most XPages apps, even Java-based ones, are riddled with all sorts of expectations about the UI, from FacesContext to ExtLibUtil to the DominoDocument model layer.

This is a sticky problem, made all the moreso by the fact that, although XPages is a fork of JSF underneath, the XSP layer is its own special language and isn't really how stock JSF pages were ever written.

There's no really-great answer, but I've never been one to shy away from writing a list of possibilities. These range from actual things one can do right now to hypothetical speculation about what one could build to deal with it. This all starts from the assumption that you want to do something to lessen or remove your XPages dependency. You can instead choose, I suppose, to keep chugging along with it.

Practical Steps

Throw It All Out

This scenario is pretty straightforward: dump your XPages code and never look back. While this could take the form of dumping the stack entirely, I think in practice it will generally take the form of first refactoring your logic (if you haven't already) and then exposing it with REST services. Then, you let your XPages app chug along as-is while you build a new app in whatever else you want, and then swap over when your new app is complete enough.

Throw It All Out, But Slowly

This is similar to above, but you rebuild your app piecemeal, either in place or by sending users to a different app for some parts. This is a very-practical route for large, sprawling applications, and it's what we're doing with one of my clients.

The way it's specifically taking form there is that, when it comes time to write a new module or rewrite an existing one, we build that individual component as an Angular app using REST services and served from an OSGi bundle, and then host it in an <iframe> inside the XPage. So the app continues on as it is, but every once in a while a big chunk of it is deleted and replaced. The use of an <iframe> means that the JS app doesn't have to worry about clashes with the surrounding JavaScript libraries included on the XPage, but gets to share the authentication session. Over time, the XPages app will become essentially a master of ceremonies for the individual modules, and then one day we'll probably swap out that shell too.

Run It In A Webapp

This is a path that would really best be combined with something else, and, admittedly, is essentially specific to me personally. In this case, you use the xpages-runtime project to run your XPages inside a normal WAR container on a good server, and then use that as your base of operations for rebuilding.

My instinct with this project is always to say "well, it's really just an experimental thing", but I use it as my primary means of XPages development and as part of my client's CI chain to host testing builds deployed by Jenkins. There are some minor down sides involved in that you have to really know the innards of the stack inside and out if something goes wrong, and it's also absolutely unsupported by anyone. So... your mileage may vary.

That all said, it makes transforming your XPages app into a modern Java app a dream. You get the full Maven experience for dependencies, and you can use newer technologies without the hassle inherent in trying to cram them onto Domino. And, practicality-wise, it'd really just take a small amount of "abetted but not supported" tweaks on HCL's side to make it less me-specific.

Hypothetical Projects

These hypothetical approaches are naturally on a much-larger scale, and aren't really the sort of thing that one would do to solve their dilemma for an individual project. Really, they'd be HCL-led product decisions, and I'm spitballing even more than usual here.

Transform It To JSF

So I mentioned earlier that JSF markup isn't the same as XSP. The immediate difference between the two is the starting conceit: where an XPage is a fully-composed entity starting at xp:view, JSF syntax evolved from JSP and takes an "embedded in XHTML" approach, like this "hello world":

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" 
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml"
      xmlns:f="http://java.sun.com/jsf/core"      
      xmlns:h="http://java.sun.com/jsf/html">
    
    <h:head>
        <title>JSF 2.0 Hello World</title>
    </h:head>
    <h:body>
    	<h2>JSF 2.0 Hello World Example - hello.xhtml</h2>
    	<h:form>
    	   <h:inputText value="#{helloBean.name}"></h:inputText>
    	   <h:commandButton value="Welcome Me" action="welcome"></h:commandButton>
    	</h:form>
    </h:body>
</html>

If you're starting from an equivalent XPage, it wouldn't be too difficult to get here, and you might even be able to do it with XSLT. Take the xp:view pageTitle and move it to the <title> element, swap out xp:inputText for h:inputText, and so forth, and you're good to go.

That is... not what your average XPage looks like, though, and it doesn't take long for the notion of a clean transformation to crash and burn. SSJS aside, there are all sorts of gotchas: themes, custom controls, xp:eventHandler, any component outside of the core, on-page data sources, and so forth. You'd constantly hit things that are either too different in JSF or don't have equivalents at all.

Though I'm not a JSF master, I expect that it'd be essentially impractical to transform the source fully in this way. That said, you could use it as a starting point: auto-convert what you can and leave commented-out versions of the rest as TODOs for the developer.

Write A Driver For JSF

The other route would be to essentially re-implement XSP on top of JSF. All the XSP is there to do is to describe the XPage as a tree of components, and something could certainly interpret the XML into components slightly more easily than a source translation.

Still, though, this would essentially be equivalent in effort to the "update JSF" requests that the community has been making for years. That's easy to say, but much harder to actually do. Additionally, it'd be more implementation work than the above: while components like h:inputText and xp:inputText share a common ancestor, they're not perfectly compatible, and so there'd have to be a parallel component tree in the JSF runtime.

A Mix of Both

By this, I mean that you could take the "transform the XML to normal JSF" approach as above, converting compatible components over to their stock equivalents, but then re-basing the truly-XPages-specific parts into jakarta.faces classes and including them as a component package so that they'd coexist. This is essentially the "dominoFaces" idea.

While I'm skeptical of the value that this would provide to the larger world, it would be a practical hybrid approach, limiting the amount of code that would break to the stuff that really gets into the weeds of XPages-specific assumptions.

And maybe this is how I'd do it if I was tasked with the job. This would run into more-explicable edge cases than trying to transform the source and wouldn't implicitly encourage writing more pure XSP markup like the second option would.

Transform It To Something Else

Of course, JSF isn't the only game in town, so one could hypothetically try to convert these apps to something else entirely. I'm a little skeptical of the options here, admittedly. An approach that would try to split it to be more client-side than XPages is now would essentially require running the stack on the server anyway to handle all the server-side bindings, so I'm not sure what you'd gain. Moving it to a non-JSF server-side framework would avoid some of that trouble, but I'm not sure what you'd gain that would be worth the nightmare of edge cases.

Still, I want to give the option a mention, since it wouldn't be impossible to do something very clever and functional in this way. I just have my doubts about how worth it it would be. In my mind, moving back to mainline JSF on a good app container would be simpler to do while also leaving the door fully open for working with other tools alongside it much more easily than Domino has offered to date.

The Rest of the Work

This is all musing about the task of dealing with XSP markup specifically, and presupposes that you're willing to at least rewrite a bunch of logic as REST services or (more enjoyably) move to a non-Domino app server. While I have my various projects to make this sort of thing easier, I recognize that (for some reason) there's a big difference between "Jesse said this is possible" and "my company is investing heavily into doing this". Just getting a viable, supportable deployment environment that isn't another dead end would be a project of its own.

One big chunk of the work outside of the XSP markup itself and its relation to JSF is the way that "XPages" as such really represents a whole application stack, not just a UI framework. While there is a slice of it that remains essentially a distinct UI kit, there's a tremendous amount of stuff that lives nebulously in the realm between a root web server and the application framework. The HttpService stuff that I've talked about recently is one such part, sitting below the "web container" portion but being (at this point) an XPages-specific thing. Not all of that would need to come along for the ride, but some of it would, or at least some apps would have to account for it going missing.

Anyway, it's admittedly all a big ball of wax, and no option is really perfect. Still, I think it's important to consider and, ideally, execute on something.

XPages: Dealing With "Cookie name X is a reserved token"

Wed Feb 03 10:49:54 EST 2021

Tags: xpages

The other day, John Dalsgaard asked a question in the XPages Slack Community to do with an exception that a client was seeing when going to any XPage:

java.lang.IllegalArgumentException: Cookie name ""categories":"[\"performance\",\"unclassified\",\"targeting\",\"functionality\"]"" is a reserved token
	at javax.servlet.http.Cookie.<init>(Cookie.java:144)
	at com.ibm.domino.xsp.bridge.http.servlet.XspCmdHttpServletRequest.parseCookieString(XspCmdHttpServletRequest.java:349)
	at com.ibm.domino.xsp.bridge.http.servlet.XspCmdHttpServletRequest.getCookies(XspCmdHttpServletRequest.java:283)
	at com.ibm.domino.xsp.bridge.http.servlet.XspCmdHttpServletRequest.readSessionId(XspCmdHttpServletRequest.java:185)
	at com.ibm.domino.xsp.bridge.http.servlet.XspCmdHttpServletRequest.<init>(XspCmdHttpServletRequest.java:156)
	at com.ibm.domino.xsp.bridge.http.engine.XspCmdManager.service(XspCmdManager.java:256)

As the uncharacteristically-short stack trace implies, this happens long before any actual XPage code in an NSF. What's going on here is that something - possibly a too-clever-for-its-own good script - set a cookie using a JSON value so that it can store structured data. However, this is kind of an illegal thing to do: by the spec, commas are reserved in the Set-Cookie header and, by virtue of the shared cookie-octet part of the spec, are also illegal in the client-sent Cookie header.

Who Is Wrong Here?

And actually, as I type, I'm starting to blame XPages less for this: commas in HTTP headers indicate multiple wholly-distinct values. For example, take an Accept header, like:

text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

The commas there indicate distinct values according to the HTTP spec itself, while the semicolons are just an idiom used by the Accept header.

The Cookie header doesn't make use of this meaning of the comma, instead relying entirely on semicolons for some reason. Still, HTTP-wise, it seems that a server should treat this:

Cookie: foo=[bar,baz];othercookie=hi

...as equivalent to this conceptual version:

Cookie: foo=[bar
Cookie: baz];othercookie=hi

... which then should break, as "baz];othercookie" is wildly illegal in the rules for tokens because "]" and ";" show up in the separators list.

Long story short, unencoded JSON is extremely likely to run afoul of all sorts of rules here, and ideally the browser wouldn't send a header like that in the first place.

The Workaround

The XPages developers were aware of this, but made the fix an opt-in thing at the server filesystem level. To avoid this specific trouble, go to the "xsp" directory in your Domino program directory (not the data directory), create a file named "bootstrap.properties", and set its contents to:

1
xsp.commas.not.delimiters.in.cookie=true

To my knowledge, the only "documentation" that exists for this is an incidental mention in the XPages Portable Command Guide, where the property being false by default shows up in the sample output from running tell http xsp show settings on the console. Fortunately, once you know that it exists, the name is pretty self-documenting, and it does just what it says on the tin.

As with other server configuration options, I think this should be configurable at the NSF level, and should at the very least be something configurable in the data directory. Doing anything in the program directory only gives me the willies. The stack should also give a better error earlier, rather than relying on the Servlet Cookie class to balk at the malformed name.

In any event, if you have a case where you're using a library or same-domain-server app that sets a header like this, this property should help.

Writing the XSP Transpiler Maven Plugin

Thu Jul 09 10:33:44 EDT 2020

Tags: maven xpages

When I was first getting my XPages webapp support project into workable shape, I was faced with the immediate problem of translating XSP source into a usable form. Though the XPages core contains both the code for translating XSP source to Java and the loader that executes the compiled Java classes, they're best thought of as two disjoint components in a larger toolchain. Designer uses the translator to create Java source, which it then compiles into .class files like any other Java source. At runtime, Java uses the CompiledPageDriver implementation of the FacesPageDriver to look for these compiled classes based on translating page names like Foo.xsp to class names like xsp.Foo, loading them with the active classloader, and calling their methods to emit the UIComponent tree.

The fact that XSP is transformed to Java and then bytecode is incidental, though: the FacesPageDriver interface only requires outputting some object that can build page trees. I've tinkered a bit with building on the Bazaar's existing dynamic-interpretation code to go directly from XSP to the tree of UIComponents, but there are a lot of fiddly details. Onerous as it may be, the translation+compilation process covers all of the edge cases that may show up.

The translation process requires a classpath populated with both the XPages core code and any libraries you have, since libraries are defined as dynamic Java classes and not, for example, statically-readable XML configuration files (there are XML files in there, but they're only identified by the Java class). Designer deals with this by making you install XPages libraries into your runtime: the classes have to be present in the Eclipse environment for Designer to be able to identify and load them. That works, but it's onerous and not practical for my uses.

Runtime Compilation

The tack I took initially with the webapp support was to write a FacesPageDriver implementation that translates XSP to Java and then compiles those classes on the fly. This has the distinct advantage of having the entire running app going, so all libraries and control definitions are available. There's overhead on first load for each page, especially for complicated ones, but subsequent loads are as speedy as the precompiled route.

Incidentally, this is basically how JSPs work in normal app servers: the JSP source is included in the .war file, and then it's translated into a Servlet implementation Java class and compiled on the fly.

Maven Compilation

Still, I really wanted to avoid having the app have to translate and compile on the fly. While it works, it's wasteful and adds noticeably to the initial load time of a freshly-deployed instance.

My goal was to do this compilation process during Maven compilation - independent of any particular IDE. The trouble there is that there's still a hard requirement on having the actual app class environment available so that library classes can be resolved. It's not enough to just solve the problem of including XPages artifacts as Maven dependencies, since that wouldn't account for using e.g. ODA in an app.

My original tack for this was to do what I do in the NSF ODP Tooling: create an Equinox environment containing the app and its dependencies, and then execute the transpilation in there. I event went so far as to implement it, though it's essentially an undocumented feature of 3.0 and above. This didn't sit quite right with me, though. For one, it's kind of outside of the Tooling's bailiwick: while it certainly does XSP compilation as part of the overall NSF assembly, it's really a distinct activity. Moreover, though, loading a whole Equinox environment is fiddly and unnecessarily requires a Notes runtime to be configured along with it.

So I took a second pass at it in the xpages-runtime project, and this has been working out well. I realized that I didn't need to have all of the app's classes available to the Maven plugin, nor did I need to spawn a whole second process. I could instead construct something of a jail ClassLoader to house the process. I build a ClassLoader based on the project's dependency tree (which inherently includes the required XSP core classes), copy in a transpiler implementation, and execute the process reflectively. This means that the whole thing can happen in-process and without a special Notes runtime, just like a normal Maven plugin.

Better still, I use the BuildContext apparatus to identify changes, and Eclipse's m2e hooks into this. In this way, I can do essentially incremental compilation that fires off whenever you modify a .xsp file inside Eclipse, essentially giving the same kind of experience that you get in Designer (with less crashing). In both Maven and Eclipse, the actual Java ? bytecode compilation happens with the normal compiler: I just drop class files in the right place, tell the project about the generated source folder, and let it do its thing.

All in all, I'm pretty pleased with how this turned out. It's still primarily useful for the type of development workflow I've set up personally, but it's definitely had a noticeable impact on the modify-deploy-run cycle. For a moribund stack that I'm actively working away from, I've built myself a pretty-respectable toolshed.

The RuntimeEnvironment Idiom

Thu Jun 18 09:16:48 EDT 2020

Tags: java xpages
  1. XPages: The UI Toolkit and the App Framework
  2. The RuntimeEnvironment Idiom
  3. NSF ODP Tooling 3.1.0: Dynamically Including Web Resources

One of the specific problems that we encountered with my aforementioned client app first when expanding it to include REST services and then later to be portable outside an NSF entirely is dealing with varying mechanisms for interacting with the surrounding environment.

The Problem to Solve

The immediate way this distinction comes up when adding JAX-RS services or other OSGi servlets is trying to get a handle on the current Domino user session or context database. In an XPages app (including in code called in a plugin-based library), you can just do:

1
Session s = ExtLibUtil.getCurrentSession();

However, this will return null if called while processing an OSGi servlet. Instead, servlet code should call:

1
Session s = ContextInfo.getUserSession();

Same idea - they both return a session based on the current authenticated user from the HTTP stack - but they have different backing implementations. So my first pass was to coordinate these inside an AppUtil class in a method like this:

1
2
3
4
5
6
7
public static Session getSession() {
	if(FacesContext.getCurrentInstance() != null) {
		return ExtLibUtil.getCurrentSession();
	} else {
		return ContextInfo.getUserSession();
	}
}

This worked pretty well, until I added Tycho-based compile-time unit tests, which is an OSGi environment where neither of those paths would return a session. So I had to add a fallback that would just eventually spawn a new NotesFactory.createSession() if it couldn't find another one.

It's one thing for a getSession() method to balloon in logic, but Notes runtime access isn't the only problem like this. Take the case of validating model objects as part of the "save" process. In an XPages environment, validation errors should be reported as FacesMessages on the view root or, ideally, attached directly to the form control that represents the invalid field. In a REST service, though, the ConstraintViolationException should bubble right up to the top and be returned as an appropriately-formatted JSON object with a corresponding HTTP status code. Originally, we handled this similarly: we moved the FacesMessage stuff out of the model objects and into the AppUtil class and handled it with an if tree.

The RuntimeEnvironment Class

Eventually, though, there was enough customizable behavior that these branching methods in one class got out of hand, and that's even before getting into cases where a class (like FacesContext) may not even be available at runtime at all. So I implemented a RuntimeEnvironment class as a service. It started out like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
public interface RuntimeEnvironment {
	static final List<RuntimeEnvironment> knownEnvironments = AppUtil.findExtensions(RuntimeEnvironment.class).stream()
			.sorted((a, b) -> Integer.compare(b.getWeight(), a.getWeight()))
			.collect(Collectors.toList());

	public static RuntimeEnvironment current() {
		return knownEnvironments.stream()
			.filter(RuntimeEnvironment::isCurrent)
			.findFirst()
			.orElseGet(UnknownEnvironment::new);
	}

	boolean isCurrent();
	int getWeight();
}

The AppUtil.findExtensions method is a simplified wrapper around the IBM Commons ExtensionManager call to find services in a type-safe way.

This allows me to define a series of RuntimeEnvironment implementations that may or may not be included in a given packaging of the app, while the isCurrent() and getWeight() methods allow me to distinguish between multiple valid environments to find the most specific. To get an idea of what I mean, here is the current suite of environment implementations:

RuntimeEnvironment type hierarchy

These run a wide gamut. XPagesEnvironment and OSGiServletEnvironment are the big ones that kicked it off, but TychoEnvironment is there to handle compile-time tests, while NotesEnvironment lets the same code work in some utilities we launch from within the Notes client - and SWTRuntimeEnvironment allows those same tools to run outside of OSGi.

Once I broke ground on these classes, the number of situations where they're useful began to become obvious. Take, for example, resolving a variable. The on-Domino XPages implementation looks like what you'd expect:

1
2
3
4
5
@Override
public <T> @Nullable T resolveVariable(final String varName) {
	FacesContext context = FacesContext.getCurrentInstance();
	return (T) context.getApplication().getVariableResolver().resolveVariable(context, varName);
}

In the OSGiServletEnvironment and JakartaRuntimeEnvironment cases, though, I can use CDI instead:

1
2
3
4
5
@Override
public <T> @Nullable T resolveVariable(final String varName) {
	Instance<Object> instance = CDI.current().select(NamedLiteral.of(varName));
	return instance.isResolvable() ? (T)instance.get() : null;
}

It gets down to little things, too, like how the POST destination for form-based login can be /names.nsf?Login on Domino but /j_security_check on other webapp servers.

Seeing It Elsewhere

This sort of idiom is by no means anything I came up with. You can see it pretty frequently - in fact, I highlighted the way the IBM Commons stack does a very similar thing when running XPages outside Domino:

IBM Commons Platform hierarchy

This serves essentially the same purpose, being filled with mechanisms for getting output streams, finding resource locations, and retrieving named objects.

Regular Use

Should you implement something like this for most apps? Probably not, no - for most even-moderately-complex XPages applications, having some if tests in a central util to distinguish between XPages and OSGi servlets should be enough. I think it's a useful instructional example, though, and it sure was critical in getting this massive thing working outside Domino. As we make our apps more portable, this is the sort of technique we should keep in mind.

XPages: The UI Toolkit and the App Framework

Wed Jun 17 21:19:40 EDT 2020

Tags: java xpages
  1. XPages: The UI Toolkit and the App Framework
  2. The RuntimeEnvironment Idiom
  3. NSF ODP Tooling 3.1.0: Dynamically Including Web Resources

Lately, one of my client projects has been picking up the pace on the years-long effort of taking a giant XPages app, making the business logic portable, and incrementally cutting down on the "XPage-iness" of it all. I expect that this will be a recurring source of blog posts, and this one is about distinguishing between "XPages the UI toolkit" and "XPages the web app framework".

XPages in an NSF

Coming from a Domino perspective, there is no distinction between the two, and that's largely because of the path we took to get here. Other than existing inside the same NSF, the distinction between "classic" Notes apps (web or client) and XPages couldn't be more stark. Legacy design elements are only shared in ways that are completely divorced from their original UI presentation (for the better), and the runtimes - as much as legacy elements can be said to have a "runtime" - are entirely distinct.

An XPages app can be thought of as conceptually a "normal" Java WAR-based webapp housed inside an NSF, and it has a lot of the trappings: classes in WEB-INF/classes, libraries in WEB-INF/lib, and an OSGi-style WebContent folder for miscellaneous files. It's not technically a normal webapp - there's no "web.xml" and the XPages outer "LCD" runtime is actually more like one giant webapp that acts like many - but it's close.

Critically, though, the Domino HTTP router only routes requests for ".xsp" files or "/xsp/" folders to your app's XPages environment, and this is the biggest technical and conceptual impediment. You can't (within an NSF) intercept just any incoming request and process it as you would in a normal webapp. You can kind of shim your way into it with servletFactory, but it's a fiddly process and limited to "/xsp/..." URLs.

Additionally, a running XPages app only exists in a very constrained way between requests. While the JSF-level "Application" and the Servlet-level session exist, you don't work with a per-app ServletContext the way you do in a Servlet webapp. You can hook in with ApplicationListeners and similar constructs, but they're still based on the lifecycle of the XPages app, which comes into existence only on the first request and dies (usually) half an hour after the last.

These, plus the specifics of Domino data access, combine to make the "XAgent" - an abomination of a concept - the catchall replacement for specialized rendering, batch processing, and even scheduled tasks.

XPages the View Engine

These are all accidents of history, though. They stem from the firm requirement that existing Domino HTTP behavior remain intact even with its brain transplant, as well as the "soft" requirement that XPages in an NSF pretend to be "forms with repeats and partial refresh".

At its core, XPages is "just" a web view engine: its only job is to accept a request from an HTTP client and return some HTML. The concepts it uses to accomplish this - components, renderers, managed beans, themes - are all incidental to the main task. This is the "V" part of MVC. Admittedly, even without the NSF compromises, XPages bleeds beyond its assigned third of the triad, and it inherited this from JSF. JSF is also billed as MVC, but it completely subsumes the "Controller" part and partially eats the "Model" part with its bean management.

Still, though, even a domineering framework like JSF slots in as just one component of a normal webapp, rather than being the whole thing as XPages is in an NSF. For example, take the app behind this blog, which partially looks like this:

Java and JSP resources in the blog

It uses JSP as its view template engine, but is it a "JSP app"? Not really. The fact that it uses MVC 1.0 is more important to understanding it, but that's really an extension to JAX-RS. You could make a strong case that it's a "JAX-RS app", especially when you expand the "Services" section in Eclipse:

REST services in the blog

That covers more of it, but still leaves parts out. It has application-wide beans by way of CDI, entirely-UI-free scheduled tasks kicked off from a ServletContextListener, and core business logic and model objects that are kept in a module that doesn't even know about the Servlet API.

It's layered, but the layers are explicable and the distinctions create a tremendous amount of flexibility. I could, if I wanted, change to ThymeLeaf for the front end with essentially no friction, JSF or Vaadin with only mildly more, or to a client JS REST UI by chopping off the top two layers outright.

Okay, So?

This description isn't a call to action - there's nothing inherently wrong about an XPages app in an NSF, especially a small-to-medium one - but this will be an important part of the conceptual groundwork in the months to come. To figure out what to do with all these piles of XSP markup and framework-specific business logic we have, we'll have to do a lot of deconstruction.

Java ClassLoaders

Fri Jun 05 10:47:55 EDT 2020

Tags: java osgi xpages
  1. Java Services (Not the RESTful Kind)
  2. Java ClassLoaders
  3. Managed Beans to CDI
  4. The Myriad Idioms For Finding Implementations In Java

In my last post, I casually mentioned the concept of ClassLoaders a couple times, and I think that they deserve their own post. ClassLoaders are exactly the kind of thing where, once you do Java long enough, you start to take for granted, but which aren't necessarily immediately obvious for people not as immersed.

The Basics

The core job of a ClassLoader is what it says on the tin: it loads classes. Say you have this bit of code using a class from the core Java library:

1
long now = System.currentTimeMillis();

This uses two types: long, which is a built-in primitive type and not a class at all, and java.lang.System. long doesn't have to come from anywhere, but java.lang.System does, and that's the job of a ClassLoader. In this case, the Java VM will ask the contextual ClassLoader for a class by that name, and the ClassLoader will (at least in Java 8 - things got weird later) look into the core library and find a file named "java/lang/System.class" within "rt.jar", parse its binary contents into an executable class, and hand it back to the VM.

ClassLoaders are also the source of two problem reports you've likely seen: ClassNotFoundException and NoClassDefFoundError. These two basically mean the same thing: the running app tried to load a class by name, but it wasn't found - they just differ in context (the former generally when a class is asked for dynamically, the latter when it's referenced as part of compiled code). This sort of thing can occur when you write code using a class that's present in your development environment but is not present when run later - among XPages developers, this happens quite a bit when people drop some JARs into jvm/lib/ext in their Designer installation but don't do the same on Domino.

Resource Loading

In addition to finding classes, ClassLoaders have a few other tasks, the main one of which of interest to us is loading resources. In my previous post, I talked about how ServiceLoader looks for service files by a given name, like META-INF/services/com.sprockets.data.FizzBuzzConverter. It does this by checking with the current ClassLoader and calling cl.getResources("META-INF/services/com.sprockets.data.FizzBuzzConverter"), which will return a listing of resources from JARs (and JAR-like sources, like an NSF) that it knows about matching that name. In that way, multiple JARs can declare services with the same name without conflicting.

ClassLoader Trees

Though conceptually your running program has "a ClassLoader", in reality it's almost definitely a chained series of ClassLoaders, rooted in the core system ClassLoader and then drilling down more specifically to your app's code. For example, take an application running in Apache Tomcat. In that case, Tomcat's documentation describes four basic tiers:

  • The core JVM ("bootstrap") ClassLoader that comes with any running Java program. As Tomcat's docs note, this implementation may vary
  • The central ("system") ClassLoader that contains the "just above the metal" classes, such as those you may add in the "CLASSPATH" environment variable
  • The Tomcat-specific ("common") ClassLoader, containing classes shared among all running applications. For example, javax.servlet.Servlet would be found here
  • Your app's ClassLoader, containing classes you write as well as any third-party JARs you bundled into your WAR file in WEB-INF/lib

When your code executes and requests a new class, the runtime will check first with your app's local ClassLoader and return what it finds there if present - if the class isn't present there, then that ClassLoader will delegate up to its parent, and so forth until it either finds a class or hits the root and throws a NoClassDefFoundError.

The way that each app has its own ClassLoader is also how you can have multiple apps on the same server that can each know about common core classes, but don't step on each others' toes with their own custom classes. Though javax.servlet.Servlet is the same class for two running apps, one app could have an internal class named "com.example.SomeBusinessLogic" and it wouldn't be visible by other running apps.

Dynamic ClassLoaders

Though the normal case of ClassLoaders is that sort of "do I have this class? If not, ask my parent" chain, the fact that a ClassLoader is itself a custom Java class means that its behavior can be pretty arbitrary. This is present in a normal web app ClassLoader: it knows to look in the WEB-INF/classes path within the WAR file instead of the normal behavior of checking from the root of a JAR, and it knows how to look in WEB-INF/lib for additional JARs to search.

In an XPages application, the active ClassLoader is roughly similar to Tomcat's app ClassLoader example, but with a couple additional capabilities. The main one is that the NSF's ClassLoader - an instance of com.ibm.domino.xsp.module.nsf.ModuleClassLoader - has knowledge of how to treat an NSF as if it were a WAR file. In Designer's "Package Explorer" pane, you get a view of the NSF that makes it look basically like a normal WAR, where classes go in WEB-INF/classes and JARs go in WEB-INF/lib. However, it's still really a nebulous pool of notes floating around, and so the ModuleClassLoader does design-collection lookups for file resources of various types and loads the class bytecode or resource data from there.

It also, in a move presumably designed to inconvenience me personally, has explicit restrictions on what classes it can load: even though it knows about, for example, org.eclipse or com.ibm.domino.napi classes, it has a check to explicitly bar loading these. That's why, even if you configure Designer to see those classes and compile XPages code that references them, they won't be available at runtime.

OSGi ClassLoaders

OSGi ClassLoaders are a particular kind of dynamic ClassLoader. In addition to the normal hierarchical view of the world, they take on special responsibilities for ensuring that your OSGi module (which an XPages app kind of is) sees classes from other bundles based on its dependency rules, but not necessarily their resources. For example, take rules like this in an OSGi bundle's META-INF/MANIFEST.MF:

1
2
Require-Bundle: com.ibm.xsp.core
Import-Package: com.ibm.commons.util

These simple lines hide some beguiling complexity. With this definition, a running class in your bundle will be able to see:

  • All classes at the system level, such as java.lang.System
  • All classes contained within and exported by "com.ibm.xsp.core", such as com.ibm.xsp.FacesExceptionEx and com.ibm.xsp.url.UrlHandler
    • There's also special behavior going on here, because those classes are contained within an embedded JAR in the bundle, referenced as Bundle-ClassPath: lwpd.xsp.core.jar - this is an OSGi-ism
    • Though this bundle lists all of its packages in its Export-Package header, this is not a requirement: it's common for an OSGi bundle to have classes internally that are not accessible from outside
  • All classes exported by its bundle dependency that it marks as visibility:=reexport: "com.ibm.pvc.servlet" and "com.ibm.designer.lib.jsf"
    • This is why you can have a dependency on just "com.ibm.xsp.core" and access javax.faces.context.FacesContext even though it's not in the core XSP bundle
    • This is also transitive, though neither of those re-exported dependencies themselves re-export any dependencies
  • The classes from the "com.ibm.commons" bundle in the "com.ibm.commons.util" package. This means that com.ibm.commons.util.StringUtil is visible, but com.ibm.commons.extension.ExtensionManager is not, despite both being within the same bundle JAR

There are also tons of weird visibility and dependency details as well in OSGi, but that's the gist of it. Note that I specifically mentioned that the resources aren't visible. Though the Require-Bundle: com.ibm.xsp.core line makes all classes exported from the XSP core visible to your code, calling ServiceLoader.load(com.ibm.xsp.acf.HtmlFilteringFactory.class) will not find the DefaultHtmlFilteringFactory implementation declared in there, even though it's done in a ServiceLoader-compatible way. This is why IBM Commons papers over that difference with its "plugin.xml" extension declarations. OSGi actually contains a Service Loader Mediator specification to bridge this gap, but Domino doesn't include an implementation of that part.

Fragment Bundles

There's one special case with OSGi bundles that's worth highlighting: fragments. Normally, each bundle effectively has its own ClassLoader space, walled off from all others by OSGi's broker. However, if you declare your bundle as having a Fragment-Host of another active bundle, your code acts as if it's within the parent, gaining access to not just all of the parent bundle's classes, but also its non-class resources. Moreover, this works in the reverse: the parent also gains access to the fragment's classes as resources, though it generally won't "know" about them at the time of development.

This is a technique that's come in handy for me many times, in particular in cases like the XPages Jakarta EE Support project, where API bundles will use ServiceLoader to find their implementations. In those cases, one of the ways I get it to work in OSGi is to create a fragment bundle out of the implementation, meaning that the bundles remain distinct but now the API can find the META-INF/services files and classes it needs to operate.

This has a good number of other uses, too, such as providing platform-specific native code to an otherwise-platform-independent core bundle. The Notes.jar wrapper used in XPages land uses this type of technique. Though, to my knowledge, Notes.jar doesn't contain any actual native code, it's still delivered in two pieces:

  1. The "com.ibm.notes.java.api" bundle, which lists all of the exported packages but holds no code itself
  2. The "com.ibm.notes.java.api.win32.linux" bundle, which contains the actual Notes.jar and declares Fragment-Host: com.ibm.notes.java.api
    • I'm not sure why this is the case, but maybe Notes.jar is different on System i or something

If you have a bundle that needs access to lotus.domino classes, you then can either do Require-Bundle: com.ibm.notes.java.api or Import-Package: lotus.domino and it'll be resolved out of the fragment. There's also an Eclipse-ism in here: the first bundle has Eclipse-ExtensibleAPI: true, which is a tip-off to the IDE that it should specifically allow fragments to contribute available classes to the development environment. This is generally required when developing with Eclipse's plug-in tooling (shared with Designer), but it's not actually enforced one way or the other by the runtime.

Wrapping It Up

This is all definitely in the category of "you don't normally need to worry about it, but it's very helpful to know", like the previous ServiceLoader topic. Until you're implementing some low-level stuff, you're not likely to interact with the ClassLoader directly, especially to a level beyond finding Thread.currentThread().getContextClassLoader() or Foo.class.getClassLoader(). Knowing about it can help make clear what's going on in situations where a class shows up in development but not at runtime, or when the XPages ClassLoader tries to get to fancy and throws up on itself.

Winter Project #2.5: XML Schemas for XPages

Thu Jan 02 10:59:58 EST 2020

Tags: xml xpages

After I did my initial port of my XSP completion assistant to LSP4XML, I got to thinking about improvements I could make to it. One of these was the notion of creating XML Schemas for a project's XPages, similar to how the DXL contributor just passes along the schema files that ship with Notes and Domino.

Doing the same with XSP isn't nearly so easy, and comes with a good number of gotchas that make it not only impossible to fully validate an XPage with schemas, but also difficult-to-impractical to generate such schemas on the fly outside of Designer. But first, two asides!

Aside #1: Mechanisms of Content Assistance

At its core, doing content assistance in a text editor is essentially a matter of the editor saying to the assistance plugin "I have a file here, with this content, at this location, and the user's cursor is in this spot. What should I suggest as the next part to type? And, once they've typed it, can you tell me if it is valid?".

In the simplest case, this could be something like a dictionary of words when writing prose. A content assistance plugin for English-the-language could merely consist of a list of known words, and it would take a prompt from the editor of "ca" and suggest "car", "cat", and so forth. There's an upper limit to what that sort of thing can and should do, and just doing that would probably suffice.

Programming languages are more complicated, and often the best route for autocomplete is to basically also be a whole compiler with full knowledge of the language and the structure of not only the current file, but also of other files in the project and all the dependencies. That way, an IDE can suggest types, method names, parameters, and all sorts of complicated external notions.

XML/HTML content assistance is often somewhere in between there. In both the case of my original XSP implementation and the LSP4XML port, the route I took was to let the in-between layer handle knowledge of raw XML mechanics like tags and attributes, but then basically provide a dictionary of known words when asked. So, when the editor said "the user typed xp:v", my code searched through all of its known components and responded with "maybe xp:view or xp:viewPanel".

I decided over the last couple days to take another route, based on the fact that XML is designed to be a toolkit for making fully-described and -validated markup.

Aside #2: XML Validation

XML itself is something of a meta-language: it has its rules, but it's intended to be a format used to describe other, more-specific grammars. On its own, it has the notion of whether or not a document is well-formed: this means specifically that it follows all the syntax rules of XML, like the proper use of brackets, attribute quotes, element hierarchy, and all that. Well-formedness is comparatively easy to enforce, and basically any XML editor does, but it doesn't say anything about whether a given XML document is a valid example of its kind.

That job is left up to a secondary definition, usually done via either a Document Type Definition file or an XML Schema, though there are more ways than that. These are the things that say, for example, that in XHTML the root element must be <html>, and that element contains zero or one <head> element and one <body> element, and so forth.

With the aid of a document schema, an XML processor (such as a structured text editor) can verify first that a document is well-formed XML and second that all of the elements that it can match up with the schema are valid. Moreover, it can itself maintain the list of potential elements, attributes, and values, and display them in a clean and fast way without the content-assistance plugin having to worry about parsing and substring matching.

That "...elements that it can match up..." caveat comes in to play because XML allows mixing grammars in a single file and the processor may or may not require that all of those grammars be strictly defined. What identifies these grammars in a file is the use of an XML "namespace", which is a URI that may or may not actually go anywhere. For example, the MathML namespace is "http://www.w3.org/1998/Math/MathML" and the SVG one is "http://www.w3.org/2000/svg". Both of those are URLs that resolve to pages, but that's just because the W3C is being nice; the only requirement is that they are unique URIs. In an individual document, these namespaces may be matched up to prefixes, and one can be defined as the base namespace for elements that don't have prefixes.

XSP as an XML Grammar

Which brings us to XPages. XSP-the-markup-language is XML-based and so every XSP document must be well-formed. Designer won't even give you the time of day if, say, you leave off the closing </xp:view> tag at the end of a file. Additionally, XSP has many of the trappings of a fully-validated XML language. It uses namespaces as its prime identifier to identify tags and you can't just write any old tag in one of those namespaces or give an existing tag some random new attribute. Moreover, many attributes and elements have special rules about their content: you can't put text inside <xp:this.data/>, and <xp:text disableOutputTag="foo"/> is illegal. These are all specialities of XML Schema.

XSP does not, though, have any schema. Most of the validation happens at build time (including of XML well-formedness, apparently), and some is even delayed until runtime (like that invalid boolean above). There are some good reasons for this. First and foremost is the fact that XSP really describes Java objects that are themselves contributed programmatically, by way of XSP library classes. Once your path to validity runs through arbitrary Java code, it means that you don't have the option to statically compare the file to a schema - you have to run that code inside your environment. Additionally, the XPages namespace ("http://www.ibm.com/xsp/core") isn't even defined in a single place, and has controls declared across several plugins. Same goes for the ExtLib's "http://www.ibm.com/xsp/coreex" namespace, and then there's the Custom Control namespace, "http://www.ibm.com/xsp/custom", which can't even be determined at a global level and has to be synthesized repeatedly on a project-by-project basis. And then, beyond all of that, XSP has rules that just can't be expressed in XML Schema at all, so Designer would have to have a secondary validator anyway, dampening the benefits of codifying a schema.

But Could It Work, Though?

Still, I figured that, if I could craft a schema that's good enough for basic use, I could get some extra completion and suggestion assistance while hopefully marking everything as vague enough to not run into trouble with the flexible nature of XSP.

My biggest ally here is that, while the core and ExtLib namespaces are technically open at any point, it would be such bad form for, for example, a company-specific library to declare its components as part of it that I can ignore that possibility entirely. In effect, for a given Domino release, those namespaces are sealed and fully describable ahead of time.

So I set out to see if I could make XML schemas to contribute to LSP4XML to give it some more knowledge of what it's working with. Like when I generated the JSON used by the original content assistance, the tack I took was to write a servlet that runs in an XPages context and emits the files I want based on a stock runtime. My initial version of this was too clever: XML Schema allows for subclassing, inheritance, and references, and I originally set out to have the schemas match the structure of the underlying component trees. I got pretty close on this, but ended up spending all my time fighting namespace collisions, and in particular the really-subtle one where the roots of the tree are actually defined in the "http://www.ibm.com/xsp/jsf/core" namespace, which is an IBM fork of JSF's "http://java.sun.com/jsf/core". As a side note, there are no concrete component definitions there, so you can't get any use out of it in an XSP file; it's just an interesting implementation detail.

The Current Results

When I took a step back and gave it another whack, I ended up coming up with something that pretty much works. I'm still not sure if it'll actually be the way I keep going, though. For one, there are currently a bevy of open bugs, some of which may end up being showstoppers. Beyond that, though, since XSP still requires a secondary processor to check validity, it may end up being the most reasonable route to just go back to offering completion ideas and maybe some post-processing to check.

Still, this is one of those types of projects that's worth it just as a learning experience. I hadn't even really looked at XML Schemas since around when they came out, and this sure was a good way to get a crash course. And, in the mean time, I think that the generated schema files are pretty-interesting artifacts.

Winter Project #1: XPages LSP4XML Extension

Fri Dec 27 16:27:14 EST 2019

Tags: nsfodp xml xpages

The last couple weeks of the year are always a good time to work on some side projects or small utilities to scratch an itch, and this year definitely ended up that way, seeing me work on a couple interesting things over this Christmas week.

The Project

The first of these is an enhancement to the NSF ODP Tooling project. Among the various components that I've put in there over time is an Eclipse content assistance plugin that provides some autocomplete capabilities when working with XPages and Custom Controls within an ODP. Specifically, it knows about the stock and ExtLib controls that ship with Domino, as well as any Custom Controls within the same project, and it allows Eclipse to provide completion suggestions. The code in there is actually hairier than you might think, and that's because it's building on top of Eclipse's generic text completion system, with only a little assistance from an existing implementation I cribbed. Still, it does the job.

The Problem

However, most of my XPages development nowadays takes place first outside Domino and then only ends up back on Domino when I want to make sure it works.

The trouble there is that the content assistance plugin I wrote is tied to both the ODP nature (an Eclipse-ism meaning that it knows a given project is associated with a set of capabilities) and the specific layout of an on-disk project.

The Quick Route

I originally set out to just loosen up that association a bit - take the existing plugin and allow it to work with any .xsp file and to try to find custom controls in a more webapp-type layout of a project.

However, I figured this was a good option to look beyond that. Though the plugin does indeed do what I want, the trouble is that it's thoroughly tied to Eclipse specifically. All of the classes use Eclipse core and XML tooling classes extensively and none of that would be portable to any other IDE. I figured this would be a perfect time to jump into the world of the Language Server Protocol.

Aside: What is LSP?

The Language Server Protocol is a standard for a way to provide IDE-type support in a way that's not dependent on any specific IDE. It grew out of Visual Studio Code and is gradually seeping its way across the whole development landscape.

It allows for the specifics of handling a language - checking validity, resolving classes and other entities, identifying keywords, and so forth - to be separated from the IDE used for editing it. Using LSP, if an IDE wants to support editing, for example, JavaScript, its creator no longer needs to create and maintain a tool to handle all of the intricacies and changing rules of the language - instead, it can bring in the LSP implementation for it and then focus just on the specifics of what the IDE does differently from others.

Additionally, this decoupling means that the LSP implementation doesn't even have to be written in the same language as the IDE using it, and are often just written in the same language that they're implementing. If you look across the list of implementations, you can see that the Swift server is written in Swift, the Ruby one in Ruby, and the Java one is actually Eclipse's Java tooling extracted from the IDE.

Wild Web Developer

Since Eclipse long predates LSP, it's historically had its own implementations for any languages it supports. Though it's had enough clout to support a lot of languages, some of them have trailed behind. Like, really far behind. Prime among these have been its web-language-related editors, which do an okay-enough job editing basic HTML, CSS, and JavaScript, but pretty much missed the boat on newer features, transpiled languages like TypeScript, and project structures like NPM. While there have long been plugins for Angular and TypeScript, they never fully kept up and the whole thing ended up falling far, far behind other IDEs like VS Code.

Enter Wild Web Developer, the whimsically-named project to bring the fruits of the LSP development to bolster Eclipse's web-tech support. Though it's named for its web language implementations, what it really is is a combination of two things: a generic text editor backed by a small array of LSP implementations and a syntax-coloring system derived from TextMate, which itself became a pseudo-standard for syntax coloring.

LSP4XML

That brings us to the piece that ties together LSPs and my immediate desires: LSP4XML, the most-popular XML Language Server implementation, which is used by both VS Code and Eclipse, and just so happens to be written in Java and is designed to be extended.

Since LSP4XML is so smoothly extensible and Wild Web Developer just added a way to contribute these extensions in Eclipse, that meant I could accomplish what I want without having to worry about writing a whole LSP implementation just to support XSP and DXL.

XSP Completion Participant

Contributing to an LSP4XML server involves creating an extension class that then registers the individual capabilities you want to provide.

In this case, I contributed an ICompletionParticipant implementation. ICompletionParticipant has a delightfully-straightforward API, and all you have to do is provide tag, content, and attribute suggestions based on the context the user's cursor is in.

With this simpler API, I was able to significantly refactor down my earlier implementation, making it much more readable and focused.

DXL Schema Contributor

The other piece of XML completion that I added to the NSF ODP Tooling was to provide the (blessedly-redistributable) DXL schemas that ship with Domino to Eclipse. Unlike the XSP completion assistant, this plugin is entirely code-free, consisting solely of the schema file itself and an extension contribution in plugin.xml. The reason this works is that each DXL file declares its XML namespace at the top of the file, and so I can tell Eclipse to look for the schema file I'm providing when editing DXL.

LSP4XML also provides a way to provide schema files, but it's a little more complicated, involving a resolver class implementation. The idea is the same, though, mapping the namespace to the DTD file.

In both cases, the standardized and descriptive nature of XML schemas means that merely providing them to the IDE allows for all sorts of code assistance, even down to the level of suggesting and validating attribute values. It's pretty great.

Side Benefits

Making this switch to LSP4XML accomplished my original goal: by changing the XSP handling in the NSF ODP Tooling for Eclipse, I switched over to the Wild Web Developer editor and got editing in .xsp files anywhere (and a bit snappier to boot, since it's inherently heavily multithreaded).

But, like I mentioned, Language Servers are used across IDEs, most notably Visual Studio Code. Thanks to VS Code's equivalent LSP4XML extension mechanism, I was able to contribute the same extensions used for Eclipse there, and get the same type of results. That's a far cry from being able to get all of the NSF ODP Tooling capabilities outside of Eclipse, but it's a big start.

The Next Version

Currently, these additions are just in the develop branch of the tooling and haven't made their way to a proper release yet, but they've proven themselves so far in my use. My hope is to make a few more improvements, get the VS Code extension into shape, and make it part of a "3.0" release of the Tooling.

CollabSphere 2019 Slides

Sat Nov 02 09:04:19 EDT 2019

I've uploaded my slides for my session at CollabSphere: "Preparing Your XPages Applications for a Changing Future":

CollabSphere 2019 - Preparing Your XPages Applications for a Changing Future

While working on the presentation, I realized that pretty much all of the slides in there warrant blog posts of their own, so, as I have the time and wherewithal, I plan to expand on the topics a good deal in the weeks to come.

Additionally, Howard Greenberg uploaded the slides from our shared presentation ("shared" in the sense that he did 90% of the work), "Let's Calendar That":

Dev112 let's calendar that from Howard Greenberg

Concept Proven: A Complex Production XPages App Running Outside Domino

Mon Oct 07 13:13:43 EDT 2019

Tags: javaee xpages
  1. Letting Madness Take Hold: XPages Outside Domino
  2. XPages on Android
  3. Concept Proven: A Complex Production XPages App Running Outside Domino

Around the start of the year, I took a little time to see if I could get XPages as it exists today running outside of a Domino server, specifically inside Open Liberty. I met with a good deal of success, finding that I could run stripped-down apps "freely" inside a normal WAR web app, and I could run full-fledged NSF-hosted apps using the LCDEnvironment container-within-a-container technique that normal XPages uses.

Since then, the prospects had been percolating in my mind, including coming back around a few months later to get XPages running on iOS and Android by way of Darwino. At the end of that post, I mused a bit about what a proper setup would look like, which would be essentially taking XPages, custom controls, and supporting resources and putting them into a Maven-structured webapp.

This past week, I decided to put WoW Classic aside for a bit and put some evening-and-weekend time into seeing if I could my get client's huge, 500-XPages-and-CC, plugin-backed, JAX-RS-heavy app working in this setup. And, like before:

Short Answer

Yep!

Long Answer

Building on what I had before, there were some medium- and large-scale hurdles I had to overcome:

  • Hook up the remaining Servlets and listeners at the right times
  • Figure out what to do about transpiling XSP source
  • Add and adapt the remaining required IBM Commons Extensions
  • Tweak the OSGi bridge for more bundle-like behavior and remove Equinox dependencies in our code
  • Make the stack have a Notes runtime, but not think it's too much like Domino
  • Adapt Java EE standards use to ensure it'd work with different implementations

A lot of this work ended up being finding the right little bit to tweak or add - an environment variable here, a META-INF/services file there - but a couple of the topics warrant expansion.

App Legwork

The core of what made this possible was a combination of coincidental and intentional work I had been doing in the main app for a good while. The first big part was that I had started bringing in components of my separate XPages Jakarta EE Support project, in particular bean validation and JAX-RS 2.1. We had previously used an older version of Hibernate Validator, and we were using the ExtLib-provided Wink for a good while to build out our REST services. Moving to RESTEasy's JAX-RS 2.1 implementation not only let us use the newer features it provided, but also let us set aside the Wink-isms we had been using in a couple places in favor of what became standard after Wink went moribund.

I also stripped out as many Equinox-isms as I could that I had made assumptions about over the years, such as removing home-brewed Equinox extension points in favor of portable IBM Commons extensions and reducing our use of Require-Bundle in favor of Import-Package, which pointed out all the areas where we depended on a specific implementation of a standard as opposed to just the cross-server spec.

Notes Runtime

One of the pleasant things I found out in my original experiments was that, while a lot of the newer and more-complex components of the XPages stack end up having some assumptions about the presence of a Notes or Domino environment, there's no dependency on nHTTP specifically. As long as the stack can call into the lotus.domino and NAPI classes, it's happy.

Really, the only thing I had to tiptoe around was making sure that I didn't include the LCDEnvironment stuff and its associated platform assumptions. At one point, I did investigate whether I could use that and make the running web app just another "module" like an NSF - there's even partial support classes like FileModule for this kind of thing in a "mashupmaker" package - but I ran into too many assumptions about the environment and class hierarchies that way. It's a bit of a shame, since that would have allowed transparently referencing NSF-housed apps in the same server, but that was only a "nice to have" anyway. Plus, it would have had the down side of keeping everything within the LCD wrappers, which do things like report the environment as Servlet 2.5 no matter how they're running (which is, horrifically, an upgrade over the underlying 2.4 on Domino).

XSP Source

My original experiment and my bootstrapping here involved copying the generated Java xsp.* files from Designer into the src/main/java build path of the Maven project, which are then picked up by the CompiledPageDriver used by the XPages runtime. This works, but it's not exactly developer-friendly. To make it in any way practical, I'd need to be able to continue working with the XML source like in Designer.

The trouble here is conceptually similar to the impediment to incremental compilation in the ODP Compiler: due to the way XSP Libraries work, they require not only the presence of the full classpath to know what components are available, but also an active running Java application. They can't just be derived statically - this is why you have to install library plugins into Designer-the-application.

However, I had a distinct advantage here: though I didn't have a running app at editing/compile time, I sure would have a running app while the app is, uh, running. And that is a problem I did solve in the ODP Compiler, thanks to the Bazaar's hooks into the XSP interpreter inside the XPages stack. I realized that I could implement a FacesPageDriver that received requests for pages and compiled them on the fly. The interface is very simple: it contains only one method, which takes a FacesContext and a page name (like "/foo.xsp") and returns a FacesPageDispatcher, which is the obliquely-named interface implemented by the translated xsp.* Java classes.

Really, this could be anything; it's not actually tied to the Java translation and compilation process at all, and could be something like a live translator of XSP into spitting out UIComponent objects on the fly, which is something I considered. Though the use of compiled classes is an implementation detail, I ended up deciding to still piggyback on that, since a dynamic interpreter would have extra legwork to do to make sure the behavior was the same, whereas using the translator would guarantee the same results as if they were compiled the normal way. In practice, the main differences between my code here and the code in the ODP Compiler is that I could use the existing component registry from the running app and that I ended up compiling the classes as Groovy source instead of Java. I did the latter because the Groovy in-memory compiler didn't require the same kind of classpath crawling that the Java compiler does, and which had ended up being a major performance sink in practice. Groovy is almost a strict superset of Java syntax, so it only took a little bit of tweaking to the generated source to make it work - tweaking that could be done simply since the problem domain is so small.

Liberty-Specific Gotcha

Though I'm still smitten with Open Liberty as my Java host of choice, I did pull my hair out a bit over some of the side effects of it. Specifically, I ran into trouble when having the JSP feature enabled (which is on by default), where it would load up com.sun.faces.config.ConfigureListener much earlier in the process than it's supposed to be. It's something to do with an optimization for when you have real JSF enabled as well, but, since IBM didn't rename the core packages, Liberty picked up on the presence of it and kicked it off on server start, instead of during app load when it's expected.

Fortunately, Liberty's architecture is such that, if you don't choose to enable a feature, it's entirely absent at runtime, so there was a clear workaround. It's a bit of a shame, since going from XSP ? JSP is a legitimate potential path, but it's not the end of the world. Additionally, there's nothing Liberty-specific in my approach here. I haven't tried it, but the same app should work in Tomcat (if you bring in some standards implementations) or in another JEE server like Glassfish.

The Results

After wrangling with this stuff enough, though, the results were ideal: the full app, with all its dynamic content, runtime component additions, use of home.xsp/foo/bar path info shenanigans (its own hurdle in the Servlet world), dependency on ODA and half a dozen other XPages Libraries, and its plugin-served resources works just as it does on Domino. And, though some of the runtime setup was a bit hairy, the app itself is pretty svelte and straightforward. All the Java code is where it's expected to be, resources in src/main/webapp show up like you'd want, and it all just acts like it should.

The Future

So now I have an odd conundrum. I prefaced my post in January by saying that I didn't plan to do anything with the experiment, but now I'm looking over a precipice where I could theoretically plop the WAR file into the Domino Open Liberty Runtime, add a redirection rule to point to it, and call it a day. I certainly don't plan to, since there could be any number of weird problems with this and the app still points back to normal classic Domino web resources in some places, but now there's a dark voice whispering into my mind. "No more Designer," it says. "No more Tycho, or writing OSGi plugins just to get a third-party library to work nicely. No more Java policy nonsense. No more Servlet 2, or missing Javadoc, or the server not loading pages because the wrong logging JAR went into jvm/lib/ext. You could be free!" The dark voice makes a good case, I'll give it that. We'll see.

Medium-Term Ways to Improve XPages

Wed Aug 28 13:18:14 EDT 2019

Tags: xpages
  1. What Makes XPages "Not Modern"?
  2. Medium-Term Ways to Improve XPages

Following up on my jeremiad the other day, I've been thinking about some of the short- to medium-term ways that XPages could be improved. I'm not fully sold on the idea that it should be massively improved in-place due to some of the systemic decisions and the general goal of future-proofing, but some improvements that HCL could make in the near future would make everyday XPages development better and would help people like me in our tasks of bringing in other tools.

The big caveat here is that I'm labeling these as small-scale tasks in theory. Since I don't know about all the workings of the lowest layers, in particular the way the XPages stack interacts with traditional nHTTP, this could easily fall into "this should be easy!" territory for reasons I can't know.

The other caveat is that I'm largely going to leave out bringing in new runtimes as a stated goal: no integrating Open Liberty, no cramming in a Node runtime, or so forth. The most expedient fix for some of this may actually be to do something like that, depending on how the "WebSphere" parts of Domino work to begin with, but that would be an implementation detail for the purposes of this list.

With those out of the way, let's proceed!

Servlet API

This has been a bugbear for a long time, and being able to bump up the Servlet API version would be huge for both its immediate features and for compatibility with newer standards and third-party libraries. I'm not sure what the source of the limitation is - it could be primarily to do with the deprecated web app part of Equinox Domino uses or it could be due to nHTTP-side C code.

Remove or Don't Export Apache Commons Logging From the JSF Libs Plugin

This may seem overly specific at first, but it's been the source of an outsized amount of hair-pulling for me and others.

I'm referring here to the "com.ibm.designer.lib.jsf" plugin that's a central part of the XPages stack. In addition to the obligatory "jsf-api" and "jsf-impl" JARs, it also contains very-old versions Apache Commons BeanUtils, Collections, Digester, and Logging. The trouble is that the Logging packages are made available outside this bundle and then these are automatically on the classpath of every XPages application and plugin. In a void, that's not terrible, but logging packages are a source of tremendous heartache when developing XPages and Java agents, and in particular they can often end up in jvm/lib/ext and cause ClassLoader trouble by mixing with the version in this plugin.

Ideally, it'd be removed outright or taken out of the exported packages and instead included just where it's needed. Next best would be to include newer versions as proper OSGi bundles with package-version information (more on that next).

Add version information to exported packages

What I'm referring to here is the OSGi mechanism for assigning versions not just to a bundle as a whole, but also to individual packages. As an example, here's a trimmed-down and cleaned-up version of the export list from the aforementioned JSF libs plugin:

Export-Package: javax.faces,
 org.apache.commons.logging,
 org.apache.commons.logging.impl

With version information (including a made-up "1.3" version to differentiate XPages JSF), it'd be more like:

Export-Package: javax.faces;version="1.3.0",
 org.apache.commons.logging;version="1.0.3",
 org.apache.commons.logging.impl;version="1.0.3"

Doing this across the board would make the XPages runtime much more OSGi-friendly.

Provide Public or OpenNTF-Permission-Gated p2 and Maven Access (With Javadoc!)

I've been on this soap box for a long while, and it comes up pretty frequently: because the XPages runtime exists solely as a component of Notes and Domino, you have to jump through hoops to depend on it as part of an external project. The "Update Site for Build Management" was a good step, in that it became one place to get a nicely-formatted p2 repository, but that was never updated past 9.0.1 and even then was just a ZIP download, not an HTTP-accessible p2 site like you'd get with things like Eclipse. Additionally, there's never been a Mavenized version of these artifacts, meaning anyone wanting to use XPages JARs without PDE or Tycho has to reinvent the wheel.

I made tools to automate these processes, but it'd be up to HCL to give the legal approval to make the results of that available generally (or to authenticated OpenNTF users who agreed to an EULA).

Shared-Source the Stack

Open-sourcing XPages would likely be a large task just because of the legal side of it, but perhaps just giving read-only access to licensed customers would be an easier step to take. There's so much about the inner workings of XPages that are effectively a black box, and it's very difficult to figure out how to accomplish a lot of advanced tasks. Having source access would make this sort of thing much, much smoother.

Support WABs or Similar

Alongside XPages came the ability to deploy servlets and JEE web apps via OSGi, but these use the deprecated Equinox extension points and an IBM-Expeditor-specific point, respectively. From what I gather, the more-normal way to do this in OSGi is via Web Application Bundles. That wouldn't necessarily change the capabilities of development for Domino dramatically, but it'd make it a smaller step to go from "normal web app" to deploying on Domino, and wouldn't necessarily involve a full revamp of the Expeditor layers.


That's it for this list, at least for now. I may end up thinking of enough for another one of these posts, but I wouldn't want to load HCL up with too much homework right after the summer break.

What Makes XPages "Not Modern"?

Fri Aug 23 13:53:17 EDT 2019

Tags: xpages
  1. What Makes XPages "Not Modern"?
  2. Medium-Term Ways to Improve XPages

A big part of figuring out how to move past XPages is discussing what makes it no longer modern in the first place. Some of these reasons will be gimmes - out-of-date or outright-missing features - while some will be less about what XPages is strictly capable of and instead more about how XPages development and deployment works. I'm going to leave out some of the ones that are just from recent lack of development attention, such as Bootstrap 4 support remaining in alpha state.

Admittedly, I suspect that this post will come across as an enumeration of complaints, but I don't really mean it that way. XPages served us well for a while, but it's one thing to say "ah, XPages, it's a bit old and creaky" and another to think about just how pervasive that is.

Technical Limitations

The most straightforward reasons are the Java technologies that are included in the XPages stack, but are significantly out of date. This includes big-ticket core standards like Servlet 2.4 (faux 2.5), the forked JSF 1.x, JavaMail 1.3, pre-Unified EL, and JAX-RS 1.1; lesser-known standards like Activation 1.1; and even third-party libraries that show up in the fixed XPages classpath like Apache BeanUtils 1.6, Apache Collections 2.1, Apache Logging 1.0, and Guava 19 (a recent addition).

Some of these can be worked around relatively simply (including a newer JAR in your app/plugin or adding an OSGi bundle), but others are either more pernicious (like Logging and Guava, causing runtime ClassLoader trouble) or are outright showstoppers. I've done a lot of work to bring newer and missing Jakarta EE standards to XPages, but the ancient Servlet version is a hard limit on a lot of things. In addition to its own limitations like the lack of Web Sockets, third-party libraries and implementations often expect Servlet 3 as a bare minimum, and nowadays make use of Servlet 4.

Self-Imposed Hurdles

Beyond the technical limitations that largely come from outdated implementations, XPages is afflicted by what I'll deem "self-imposed hurdles": difficulties that spring from choices that Lotus/IBM/HCL actively made or chose to not make that resulted in things being a bit more difficult than in other environments. These decisions were often even good decisions in the aggregate, but they have tradeoffs.

The first big one is the use of the NSF container. Overall, this is a positive: not only did it create a more unified development experience with classic Notes elements, but the NSF itself brings attributes like external-to-the-webapp ACLs and seamless replication, which are extra addons at best with other frameworks. However, because the NSF is essentially a proprietary binary blob, it means that it can't readily participate with other tooling. You can't crack one open with a ZIP file tool, inside-JVM reflection/classpath tools can stumble over it, and it can't inherently participate in source control or automated builds. As above, some of that can be worked around with immense effort, but even then not nearly as smoothly as elsewhere.

Along the lines of the NSF, due to Designer's lineage and (presumably) the state of technology at the time that XPages was integrated, the normal XPages development process has no participation in the larger unified Java development world that Maven largely ushered in. Even besides not fitting into the build system, this introduces just a bit more friction in normal development. Yes, an XPages app can use many third-party Java libraries by just adding them to the Code/Jars section, but they don't benefit at all from version management or automated fetching of source and Javadoc. It may sound like a small thing, but it makes a world of difference once you're used to it.

That plays in to a general Domino/Designer issue that stretches back to the introduction of Java to the platform: the libraries include no parameter info or Javadoc. Designer has its help sidebar for Notes.jar classes and the Javadoc exists for a subset of the XPages stack as of 9.0.1, but they're not integrated in the IDE, and large swaths of the platform don't have any Javadoc at all. This leads to stumbling blocks where you'll seek out documentation for a class that nominally exists in XPages, but you have to be wary of the "Since" note for every method (and even then you may be thrown for a loop, like UIViewRoot#getViewMap() that's "since" 2.0 but does exist in XPages).

Then there's a big sticking point that's created a weird cargo cult within the community: the default Java policy. While the file itself does indeed come from stock Java (including the adorable "properies" typo), the way it interacts with the SecurityManager and ClassLoader setup in XPages has an outsized negative impact. XPages apps butt heads with this policy constantly, in particular when using reflection - which is something that third-party libraries do often as a matter of course. It's had the effect of making many people shy away from using third-party libraries or code that uses normal Java features like this out of a partially-justified desire to not tweak the settings.

The last one in this category is the use of OSGi generally and Equinox specifically. I can't knock the original choice too hard: OSGi is a logical pick for making a large, pluggable system and IBM institutionally had tremendous experience with it by way of Eclipse and WebSphere. However, while it's a solid choice for system architecture, it's often very awkward for app architecture. And, hypothetically, XPages is designed to avoid this. Like the various Java EE servers that are implemented with OSGi but run plain-old WAR-based apps, a basic XPages app isn't meant to care about the OSGi layer at all. However, a combination of the above security-policy troubles, Designer's limitations as an IDE, explicit blocks on what classes an NSF can access, and the severely-limited EE services provided by the XPages runtime means that OSGi plug-in development came immediately to the fore. And developing plug-ins for XPages is hard. It involves all sorts of ceremony and incantations just to get started, and maintaining even a well-structured plug-in project is much more difficult than a normal Maven or Gradle project. Some of this is OSGi, some of it is Equinox specifically, and some of it is unique to XPages, but it all combines to make it all a daunting process even for people who aren't put off by Java-the-language. So, much like with the security policy, you have a large subset of the XPages community that for either developer or admin reasons don't develop XPages libraries and often don't bring in any outside of that HCL ships in the box.

Development Model

And, finally, I'd like to cover the development model created by XPages even when everything is firing on all cylinders.

The first one is the lack of MVC (or similar) structure. XPages was introduced as bringing MVC to Domino development, in large part thanks to the underlying JSF nature, but this was something of a dirty trick. While it's theoretically possible to make a cleanly-structured XPages app, the framework provides no useful model layer, and Designer is almost actively hostile to this kind of separation. And, because the framework doesn't give any guidance and few of us in the community came in as seasoned web developers, the discussion about XPages and MVC is largely about each person's individual version of "MVC-style" development, and Lord knows I'm as guilty as anyone. This, like the lack of dependency management, is something that you don't fully appreciate until you work with a system with real structure and clear guidance on how to do it.

The second point in this section is one that I actually debated putting on this list, and that's XPages's use of server-side state. Depending on who you talk to, this alone is something that makes XPages dead in the water, an ancient relic of a forgotten era. Personally, I'm not so universally down on the notion: I feel that the worries about scalability are usually based on hypotheticals, and being able to make complicated dynamic apps maintained by the server cuts down on the work of writing and securing REST APIs for absolutely everything. Still, it's pretty clear that the server-state model has fallen out of favor, and that's worth at least considering when determining "not modern".

So What About Workarounds?

A running theme through all of this has been "X is behind the times or weird, but can be overcome with a bunch of work". And this shows up a ton in practice: a lot of XPages developers have gone to Herculean lengths to bring in other technologies or carve out a way to do structured programming, and it sometimes works. Others eschew XPages UI components entirely and build JavaScript apps backed by REST services written with the ExtLib components, JAX-RS, or other mechanisms. Especially since the Java 8 switch, you can do a lot to drag XPages forward.

But that doesn't make it "modern" really. This is all work fighting against the platform, work that doesn't need to be done with other tooling. In other environments, if you want to use the latest version of a spec, you make sure your container is up-to-date and then you just use it. If you want to bring in a dependency, you declare it in your "pom.xml" and it shows up, source and all. And then there are the things that other environments provide for you, like OpenAPI documentation, and concepts that don't even exist in XPages, like fault tolerance. Maybe some of those things could be brought to XPages too, but, again, it'd be a struggle every step of the way. There'd always be some weird thing to do with the Servlet version, or Tycho, or ClassLoaders, or loggers, or some of the many other little asterisks that accompany the platform.

The Work To Move Forward

My plan for future entries in this series is to discuss some of the specific steps that I've been taking with my largest active client project to prepare it for the future. That future is as-yet-undefined - it may stay within the Domino web container, it may not - but the good part is that a lot of the work is in common regardless of where we take it, and I think that it will prove useful to others facing similar situations.

Upcoming Webinar with Cameron Gregor: How We Are Using XPages

Sat Aug 17 10:05:54 EDT 2019

Tags: xpages

I ended my XPages post the other day with a request for people who are working on large XPages applications to hit me up on Twitter to tell me about them. Shortly thereafter, the estimable Cameron Gregor did just that. Moreover, he had the suggestion of turning the discussion into an open webinar, so that others can join.

He made a post on his site with the details and a handy time-zone table to account for our respective locations, and the summary is:

  • Sydney Time: 7:00 AM on August 27th, 2019
  • US Eastern: 5:00 PM on August 26th
  • GMT: 10:00 PM on August 26th
  • Zoom Link

I'm pretty curious to take a look myself, and I hope you'll join us in just over a week!

How Do You Solve a Problem Like XPages? Redux

Tue Aug 13 17:33:00 EDT 2019

Tags: xpages

The better part of a year ago, I mused about what to do about the XPages problem. For better or for worse, I'm still musing about it to this day, without a good answer. The nice part for me is that it's not actually my job to come up with a real plan to "solve" XPages in a grand scale, but I do have my own set of clients I'm responsible for, and I've been personally invested in it for so long that I'd like to be involved in bringing it to a safe landing.

Moving Away

And I do think that that "landing" almost definitely has to be a path from XPages to something non-XPages. Hypothetically, a path forward would be for HCL to staff up on a new XPages team and improve the platform. Even if they did, though, I don't think it'd be wise for customers to rely on that - not even because I'd doubt the intent, but rather because any single-vendor, closed-source web stack without a large developer community to buffer it is an unreliable foundation.

If it'd be unsafe to rely on a revitalized platform, I think it's certainly unsafe to rely on one that's clearly in maintenance mode. The nature of web development is such that a stationary platform may as well be moving backwards, and not just because it will miss out on Web Workers or other new technology. Sooner or later, Safari or Chrome will remove a capability for security purposes and either Domino or the Dojo version XPages uses will be caught flat-footed. We've already been dealing with different definitions of "define" and various security improvements tripping us up for pretty much the entire life of XPages, and that kind of thing certainly isn't going to go away. Heck, how long to you figure user agents are going to remain reliable? It's unfortunate, but the fact that XPages came out of that "Web 2.0" era means a given page is less likely to function properly in five years than a JavaScript-free page made in 1995 that's still going strong.

Candidates for a Path

So I do think it's important to have a path, but it's not yet defined. A couple candidates spring to mind for me, but each one has one major drawback or another:

Hoist XPages Back to Jakarta EE

By this I mean taking XPages more-or-less as-is and running it on a normal JEE server. This certainly works, and access to the source would let me make it work better, but it kind of kicks the can down the road. XPages itself would still be moribund, and just running it on, say, a Domino-connected Open Liberty runtime wouldn't magically make it modern.

This would, though, provide some breathing room to manipulate an app in a better environment and transform it gradually. A JEE app can use a number of technologies that an XPages app can't currently, and so this would be a way to migrate the code without ever having a hard cutoff.

Bring to XSP Components to JSF 2.x

This is essentially the "upgrade JSF" request that has followed XPages since just after its birth, but with the slightly-lower goal of leaving XPages-the-stack where it is but making a copy of the components and infrastructure that can be brought into a normal JSF runtime like any other set of components. This would possibly be the hairiest of all options, since it wouldn't really be worth it unless things work near-100%, and there are so many little edge cases that it's harrowing to think about. Take the _xsp* methods grafted onto XPages's javax.faces.component.UIComponent alone, or whatever weird ways the XPages Ajax model differs from JSF's.

Still, it'd be doable if desired, and it'd provide a reasonable path to progressively "melt" the XSP components down until they're very-thin wrappers over normal JSF stuff, until they don't really diverge at all. With infinite resources, this would probably be the nicest route.

Try Transforming XSP Markup

By this I mean two similar possibilities: making an XSP-to-Java "compiler" that emits stock JSF components, or one-pass transforming XSP XML into JSF-compatible XML, though I think only the latter would be worth pursuing. While this could potentially rival the complexity of the JSF "update" route, I think that this would allow more room for things to break. For example, if you made an XML transformer, it could target a subset of controls but emit standard comments with TODOs to cover the parts it doesn't handle. That wouldn't be perfect, but you'd end up with real-world-compatible code without having some sort of intermediary translation layer like keeping the old components would essentially be.

This would probably have to involve porting over the SSJS EL extension and thus retaining support for various uncomfortable XPages- and Domino-isms, but them's the breaks, I suppose.

Focus on REST APIs Only

This is the route where we would basically wash our hands of traditional XPages applications (minus bug fixes) and instead target writing only REST services, whether it be in the NSF, in plugins, or in normal JEE apps. This has the advantage of being easiest for HCL, since it more or less works (though the ExtLib's use of Wink holds back the JAX-RS version significantly), but still would mostly be a "rewrite all your applications" route. The only salvaged code would be anything that's already cleanly separated in Java or SSJS, and I suspect that that's not the bulk of it.

Progressing Without a Defined Path

In the mean time, aside from personally becoming acquainted with other technology, I think it behooves all of us with actively-maintained XPages apps to step up the progress on making them portable. I've been doing this heavily for one of my client projects, and I'll have more specifics to say about that in the future. Some parts are straightforward and have remained good advice for a long time: don't use SSJS, do adopt Jakarta EE technologies, and adopt automated builds (including for your non-XPages NSFs).

The specifics get a lot more complicated, unfortunately. Since we've been swimming in the same XPages pond for a decade, even mostly-clean Java code is likely to be infested with XPages-isms, both out of habit and out of necessity. For example, hooking up file uploads to a Java bean requires using an XPages-specific class, which barely transfers to OSGi-based servlets, let alone any other environment. And there are tons of little things, like using XSPContext to get URL parameters. It's going to be a messy process, but I think it will be necessary for any apps that you plan to keep using for more than a year or so.

I'll probably end up turning this into a series, where I'll discuss the various hurdles I've overcome in taking a complicated XPages apps and gradually laying the groundwork for a different UI technology.

And, in the mean time, if you're working with large active XPages applications, hit me up on Twitter and let me know how you've gone about making them. I realized earlier that, while I certainly have detailed knowledge of how people could write XPages applications, I don't have a good bead on how many actual XPages apps of each stripe exist and what the prevailing methods still are.

How the ODP Compiler Works, Part 2

Mon Jul 01 11:36:57 EDT 2019

Tags: nsfodp xpages
  1. Next Project: ODP Compiler
  2. NSF ODP Tooling 1.0
  3. NSF ODP Tooling Example Project
  4. NSF ODP Tooling 1.2
  5. How the ODP Compiler Works, Part 1
  6. How the ODP Compiler Works, Part 2
  7. How the ODP Compiler Works, Part 3
  8. How the ODP Compiler Works, Part 4
  9. How the ODP Compiler Works, Part 5
  10. How the ODP Compiler Works, Part 6
  11. How the ODP Compiler Works, Part 7

In yesterday's post, I briefly touched on how the XPages runtime sees its environment by way of a FacesProject and related components. Today, I'd like to expand on that a bit, since it's useful to understand the various layers of what makes up an "XPages app" at compilation and runtimes.

Designer and Domino largely take two paths to try to arrive at the same location in how they view an NSF. The way Designer works is more complicated and opaque than Domino, with extra layers of VFS and an internal RPC mechanism(!) for editors, but there is at least some shared code from the XSP runtime. Beyond that, it does almost the same thing to determine the project's dependency classpath, while the internal NSF classpath is entirely distinct, using Eclipse's project structure to builds towards the different structure Domino will use.

Libraries

The notion of an XSP Library is one of the main parts of directly-shared code between the server and Designer. The way an XSP Library works is that you create a class that implements com.ibm.xsp.library.XspLibrary and then declare that as an IBM Commons extension contribution (more on that later) for the com.ibm.xsp.Library service type.

The fact that this is live code sitting in a plugin has a significant implication. Namely, anything that interprets it has to actually load the class and its dependencies. This is as opposed to just a static configuration file, which could be read without executing any custom code. For the server, the distinction doesn't matter too much, since you'll want to load all your class files anyway. For Designer, this is where we get the requirement to install libraries into Designer itself, rather than just adding plugins to the Target Platform. This is also an area that's a breeding ground for IDE bugs, since Designer needs the plugin available both internally and in the Target Platform, but they're not inherently tied together.

Though the XspLibrary implementation class is executable code, its main purpose is to point the runtime to various bits of static configuration information: the unique identifier for the library (e.g. com.ibm.xsp.extlibx.bazaar.library), lists of *.xsp-config and *-faces-config.xml files to define XSP and JSF contributions, and a list of other library IDs that this one depends on.

I believe that Designer and Domino use these bits of information slightly differently - I'm not sure that Domino cares too much about the *.xsp-config files, for example - but there's a lot of overlap here.

Configuration Files

The two main types of static configuration files used by libraries serve distinct purposes.

The *-faces-config.xml files (not required to be so named, but it's a good convention) are layered under the faces-config.xml file contained in your NSF. They define managed beans, converters, PhaseListeners, and other JSF-isms. These files come directly from the underlying JSF implementation and share the same syntax, at least until the JSF-1.2-era forking of XPages.

The *.xsp-config files look similar - they also use the <faces-config/> root element - but I believe that these are largely an XSP-specific detail. It looks like JSF 1.2 also uses the same <faces-config-extension/> tag, but to a different end - perhaps this evolution started the same way but then diverged there. In any event, these files are where Designer (and the XSP compilation process in general) looks for custom-defined components and their accessible properties. There's an interesting point to note there: though defined components are effectively beans with properties, Designer doesn't introspect the object to get its property names and types, but instead relies entirely on the definitions found in these files. It will still eventually use the component class when it goes to compile the translated XSP Java files, so they still need to be correct, but it's certainly a spot where it's easy to make a typo or mismatched property type.

I think that the latter files aren't used by the server, since their purpose is to provide the XSP source Java translator with mappings for components' XML elements to the Java classes. However, the core XPages runtime classes on the server still retain knowledge of this configuration, which is how the Bazaar and ODP Compiler do their thing. The com.ibm.xsp.registry package and sub-packages are filled with a mix of parser classes and in-memory representations, like com.ibm.xsp.registry.parse.ConfigParserImpl and com.ibm.xsp.registry.LibraryFragmentImpl.

Non-Library Contributions

Though not related to libraries, it's useful to know about a handful of XPages-specific class contributions that can come into play at runtime. These use the IBM Commons extension mechanism, like libraries themselves, but contribute to a good many different parts of the runtime and application flow. Some of these can be defined inside an NSF, while some are only recognized when defined in plugins - there's a good rundown of these on the ODA wiki. It's pretty rare to see these in the wild, but you may see an application here or there that uses these contributions, via in-NSF files like META-INF/services/com.ibm.xsp.core.events.ApplicationListener.

OSGi and Dependencies

In the early days, XPages was not OSGi-based. That came in in the 8.5.2 era (I believe - I wasn't aware enough in the 8.5.0/8.5.1 era to know the specifics) with the "extensibility API". For the most part, this lineage remains, and the XPages runtime itself isn't too dependent on OSGi, even when it comes to library contributions. Little bits have crept in here and there - the getPluginId() method in XspLibrary and the getOSGiBundle() method in ExtLibLoaderExtension, for example - but it's still largely incidental.

IBM Commons Extensions

If you've done both XPages plugin and Eclipse-the-IDE plugin development, you may have noticed that, while Eclipse plugins usually contribute to customized extension points with complicated schemas, XPages contributions all look like this:

	<extension point="com.ibm.commons.Extension">
		<service type="com.ibm.xsp.Library" class="com.example.SomeXPagesLibrary" />
	</extension>

There are still some places in Domino where you use different extension points, such as when you register a servlet with the Equinox OSGi runtime directly, but for the most part it's just this one point. This is because this extension point is designed to paper over the differences between OSGi extensions and the vanilla-Java-style ClassLoader#getResources mechanism. The type of the service you provide lines up with the META-INF/services/some.extension.type files you can use in your NSF and which still remain inside the embedded jars in the core XPages plugins.

The reason why this OSGi extension point exists is that OSGi intentionally creates separations between the individual plugins that make up your app runtime. In a "normal" web app, all of your dependency jars end up in the WEB-INF/lib sub-directory and are effectively all poured together to make a single class-loading environment. The ClassLoader#getResources route will look through all of the jars in the classpath for these META-INF/services files, but OSGi puts walls between them, and instead provides its own extension mechanism (among others, but this is the one Domino uses).

Dependency Resolution

Both Domino and Designer view the NSF like an OSGi plugin, but go about resolving the dependencies slightly differently. Fortunately, this is a case where the differences seldom crop up in practice - I've only seen some minor differences in how they honor the Export-Package directive in the bundle manifest and how fragment bundles are included.

When Designer is building an XPages app, it references the xsp.properties file to determine which XSP Libraries to include, and then uses their getPluginId() method to determine which OSGi bundle that matches up to (I think). It adds that plugin to the list of dependencies in plugin.xml and (since 9.0.1 FP10) META-INF/MANIFEST.MF. The Eclipse side of Designer then uses that to compose the Plug-in Dependencies list from those bundle IDs and any of their dependencies that are marked as re-exported. I think that Domino only cares about the generated plugin.xml/MANIFEST.MF files - I don't think that it does the resolution based on the library class, though I might be wrong about that.

ODPCompiler's Version

Currently, the ODP Compiler hews closer to the "Domino-style" route. For resolving the active class path, it trusts that the plugin.xml that exists in the ODP is correct and resolves dependencies from there. In the future, it may make sense to have the compiler generate the plugin.xml file itself, in which case it will also have to resolve the plugins based on the library classes. That wouldn't be too difficult, but for now it relies on the exported ODP.

Layer Cake

Looking at the whole XPages architecture, something that strikes me is how much it's simultaneously a giant stack of parts - config parsers, resolvers, runtime bootstrappers, and so forth - but also a pretty straightforward server-side web stack from a Java EE perspective. I've been diving deep into XPages in various ways for a long time now - building complex apps, writing library plugins, and even yanking the runtime out of Domino - yet writing the compiler led to this whole distinct set of capabilities. But a lot of this is essentially "just" ahead-of-time work, with Designer and the ODP Compiler's jobs being a lot of world-resolution followed by placing compiled pieces into the right places in the NSF.

By the time it gets to the NSF, it actually ends up as a pretty normal-style web app - XPages are just Java classes floating around, non-OSGi dependency jars are in WEB-INF/lib, and the faces-config.xml controls rendering in the same way as in JSF. A lot of that, though, will come up in later posts, where I go into the gotchas involved in taking these compilation results and other ODP resources and actually getting them into an NSF.

How the ODP Compiler Works, Part 1

Sun Jun 30 13:54:40 EDT 2019

Tags: nsfodp xpages
  1. Next Project: ODP Compiler
  2. NSF ODP Tooling 1.0
  3. NSF ODP Tooling Example Project
  4. NSF ODP Tooling 1.2
  5. How the ODP Compiler Works, Part 1
  6. How the ODP Compiler Works, Part 2
  7. How the ODP Compiler Works, Part 3
  8. How the ODP Compiler Works, Part 4
  9. How the ODP Compiler Works, Part 5
  10. How the ODP Compiler Works, Part 6
  11. How the ODP Compiler Works, Part 7

A year ago, I started a project to compile NSFs - and particularly large XPages projects - independently of Designer. Since the initial releases, the project has grown, gaining an ODP exporter, extra Eclipse UI integration, and the ability to run without installing components on a remote server. It's become an integral part of my workflow with several projects, where I include the NSFs as part of a large Maven build alongside OSGi plugins and a final distribution.

Building this tooling required learning a lot about the internals of XPages, the specifics of how various design elements are stored and handled in an NSF, and miscellaneous bits about Equinox and Maven. Since there's a good amount of arcane knowledge embedded in the project, I think it'll be helpful to take some time to dive deep into what's going on, starting with XPages.

XSP to Java to Bytecode

The first challenge for me to overcome was how to go from XPages XML source to the Java class files (like those seen in the "Local" source folder in Designer) and finally to compiled Java bytecode. Much like in Designer's process, the middle part is just incidental: only the XSP source and the bytecode are actually stored in the NSF.

The official XSP -> Java compiler exists only in Designer, and so one route would be to try to get those plugins working on Domino. I think that'd work, but it'd be a huge hassle and, fortunately, an unnecessary one. The XPages Bazaar project contains essentially a clean re-implementation of the glue code required to coax the runtime into emitting what I needed. I used the Bazaar as an incubator for the early versions of the compiler, and tweaked the core code with additions and fixes to work with various needs I ran into.

OSGi Bundles

On its own, the Bazaar did the job well of taking an XPage and compiling it with whatever the surrounding environment had. However, to compile a full XPages app as part of a Maven build, I'd need the ability to dynamically load XPages libraries and dependencies on the fly.

To do this, I added the option to include an update site directory, and then I have the compiler stream through all the plugins and initialize them. Fortunately, the OSGi environment makes this pretty easy. The BundleContext object that's available from each OSGi bundle has an installBundle method you can use by pointing at a bundle file URL, and then you can call start() on that result to actually initialize it. I had to do a little extra work to account for bundles that shouldn't be started (source-only bundles and "fragments", which are additions onto normal plugins) and the like, but it's not too complicated.

XSP Libraries

Just installing the OSGi plugins isn't enough to get the XPages runtime to know about any libraries that may be included, however. This is done by finding all of the library extension contributors, sorting them for compatibility, wrapping them in a com.ibm.xsp.library.LibraryWrapper for some reason, wrapping that in a com.ibm.xsp.registry.config.SimpleRegistryProvider, and then adding the results of that to an in-memory com.ibm.xsp.registry.FacesSharableRegistry instance.

This is one of those cases where the actual code involved in the end isn't terribly long, but the amount of delving into the framework to figure out the needed parts was immense. In essence, each XPages application is represented as a FacesProject implementation object, which retains a registry of the libraries it knows about. In a normal running server, this happens automatically during initialization: the runtime opens the NSF, figures out which libraries it needs, finds them from the ones it knows about, and constructs the app's contained world. For the compiler, I ended up having to do something of an ad-hoc version of this as it goes, finding all the parts that need to be kicked to notice the available libraries and have them around to map something like <abc:someCustomComponent/> to an instance of com.abc.xsp.CustomComponent.

Custom Controls

Custom controls are a similar story, but they use a specialized variant of the "real" library system called LibraryFragment. The way these work, before the actual XSP -> Java compilation step, the compiler reads their definitions and adds them to the in-memory FacesProject. It's important to define them all in this way before actually trying to interpret their source (or any XPages), because this allows for the interpreter to understand a reference to one CC from another. Fortunately, the process is separated enough that the definitions can all happen before any of the Java classes actually exist. Otherwise, I would have had to either try to make a dependency graph of which CC references which other, or just keep trying to compile them repeatedly until it got the right order by brute force. The latter process will actually show up in a future blog post.

Java Source Files

Compiling individual Java source files - both those that show up in the Code/Java part of an NSF (or a custom source folder) as well as the translated XSP source - is handled by the Bazaar, in a class called JavaSourceClassLoader. This class wraps around the in-memory Java compilation capabilities that were added to the JDK in Java 1.6. In particular, this class, paired with SourceFileManager, provides knowledge to resolve classes from OSGi bundles in the active environment, including specialized knowledge of dependency management based on re-exported dependencies, embedded jars, and other OSGi-isms that play a big part in XPages libraries.

In essence, these classes provide a similar compilation environment to what Designer does when it creates the Eclipse project for compiling an NSF. They build up an environment with knowledge of all the dependencies that, in Designer, show up in "Plug-in Dependencies", and then the compiler feeds it all of the Java source files in the project. With all of this environment provided, the underlying Java compiler is able to do its job of converting them to bytecode en masse, which is then passed back to the compiler for insertion into the NSF.

Other Big Components

In the next blogs posts in this series, I'll go over some of the other big hurdles I had to overcome to get everything working properly: namely, figuring out all of the specialized behavior necessary to flag imported/created design notes properly and then the arcane incantations necessary to get this OSGi environment working outside of the Domino server.

My Slides From Engage 2019 - De04. Java With Domino After XPages

Mon May 20 09:24:03 EDT 2019

Tags: xpages engage

Engage 2019 has come and gone, and I had an excellent time. I also quite enjoyed presenting my "group therapy" session on some options that XPages developers have for the future. In a lot of ways, it was similar to my presentation at CollabSphere last year, mixed with the various new developments I've talked about on here since then:

Engage 2019 - De04. Java with Domino After XPages

XPages on Android

Mon Apr 15 16:30:50 EDT 2019

  1. Letting Madness Take Hold: XPages Outside Domino
  2. XPages on Android
  3. Concept Proven: A Complex Production XPages App Running Outside Domino

Around the start of the year, I had a bit of a dalliance with the idea of running XPages outside Domino. The upshot of that project is that it is indeed possible to do so, but there'd be some work to do to make it practical.

This month, we revisited the idea with a healthy dose of Darwino to provide some undergirding technology, with the goal of being able to run a plain old XPages app on mobile devices backed by Darwino DBs and replicating back to Domino. There was a lot of fiddling involved, but it works:

Android is the natural first target, but iOS is about 70% there, with the scaffolding loading up to the point where it loads a page but currently with a bit of trouble when it comes to resolving data classes and executing renderers.

What Specifically Is Going On?

We set out a few required parameters to call it a successful proof-of-concept:

  • It has to use the actual XPages framework - that is to say, the jar files shipped with Notes/Domino
  • The XPages themselves have to be shared among Domino and the Darwino app, in the form of their Java "intermediate" source (compiling from .xsp source is possible but is a whole other thing)
  • It has to use xp:dominoView and xp:dominoDocument data sources
  • It has to load the Extension Library
  • It has to use ancillary elements from the app: managed beans, themes, CSS, images, SSJS libraries

This checks all of those boxes, and it's pretty satisfying to see in action.

The Stumbling Blocks

As I discussed in my post about the original project, there are aspects of XPages that are meant to make this sort of thing possible, with the core parts abstracting out different platforms and runtime environments. Over the years, though, assumptions about Domino and OSGi crept in, with newer additions and the Extension Library taking a bit less care to be environment-neutral.

Moreover, XPages is not open source and I don't have any particular special access to it, making this whole thing essentially, in the video-game sense, hard mode. There are a handful of classes that needed to be outright swapped out, like the annoyingly-final NotesContext class, but there was much less of that than I'd thought.

After that, getting the data to point to Darwino was pleasantly straightforward. Other than a handful of areas where the NAPI comes in, the Domino data sources largely adhere to the rules of the lotus.domino interfaces and don't make assumptions about the implementing classes (this is essentially how ODA shims in as well).

What This Could Be Useful For

With the right fleshing out, this could be a real way to run existing XPages apps on mobile devices. That could be pretty useful on its own, but I don't think that carrying forward XPages apps as-is is the right idea. The side effect of this, though, is that you have a functioning XPages app in a normal old .war file project, structured with Maven or Gradle, and ready to be molded into newer frameworks using whatever tooling you'd like. No Designer, no OSGi, no Servlet 2.4/2.5, just a clean basis running your existing logic and ready to be improved.

If the mountain of existing XPages code is going to have a future, I think it should be something like this.

Java With Domino After XPages

Thu Mar 14 15:10:15 EDT 2019

IBM and HCL held a webcast today to detail some plans for Notes/Domino V11. There were some interesting tidbits elaborating on things like the pub/sub support, and it'll be worth tracking down a recording of the event when it's available.

What's important for this series, though, is that this event served as the long-promised "roadmap" announcement for XPages. The roadmap is, in effect, option three: HCL plans to look into ways to reuse some existing XPages code, but in general you should be aiming to write your UIs in something else, either consuming REST services from an XPages container or accessing Domino data via another route (like the domino-db Node.js module and hypothetical Java gRPC client).

So we know the end of the path: not XPages. However, it's not like we're all just going to throw away our existing apps, so there's work to do determining how we're going to get there. The options remain pretty much what they were after CollabSphere last year, albeit now with the doubt removed. The first two options - returning to LotusScript or going to Node - have their advantages and disadvantages, and you could make a reasonable case for either. Personally, I'm not interested in going down those roads, though, and I think it's better for any app of reasonable complexity to dive into Java. Other members of the community and I have developed tools over the years to make it easier, and now's the time to take some of these steps if you haven't already.

Do Not Use Server JavaScript

Sever JavaScript was always something of a trap for app architecture. There's nothing inherently wrong with having a scripting language on your UI pages, and it certainly helped bridge some gaps, but the way it and Designer intertwined encouraged developers to create non-portable messes. If you're still writing SSJS, stop immediately.

Learn Proper Java

Java has been around for a long time, and the way to right "good" Java code has changed over time and varies greatly by your environment. Some aspects, though, apply generally, and it's useful to stay up-to-date on current practices. I don't know a better resource for this than Effective Java, which has been updated for Java 7-9 since I last read it.

Speaking of which, you should learn about Java 8 streams and lambdas - they're great. Julian Robichaux did a presentation on this topic back at Connect 2017, and the slide deck is very elucidating.

Adopt Standard Java Technologies

Last year, I created a project to bring some modern JEE technologies to XPages. These are some of the same technologies I've been talking about in my "XPages to Java EE" series and, while that project can't bring the full JEE development experience to XPages, using those tools will help you write code that, in some cases, could be directly dropped into a Java EE app with no modifications at all. There's a big asterisk when it comes to actually accessing Domino data, but that's a solvable problem as well (with some more development).

In particular, you should start writing JAX-RS services. Not only is JAX-RS an excellent and very-capable spec, but REST services are portable to absolutely any front end.

Adopt Automated Builds

Maven has been something of a bugaboo for XPages developers for a while, but doesn't have to be. Node development (server- or client-side) revolves around npm and various build plugins, and Maven is much the same thing. One of the biggest improvements I've made lately to all of my active XPages apps is to wrap the on-disk project for them inside a Maven artifact, using the NSF ODP Tooling. That project allows you to automatically build your NSFs alongside other parts of the project (such as OSGi plugins) without having Designer involved.

Check the example project in that repo, and stay tuned for a 2.0 release (probably) imminently.

Learn Other Toolkits

If you're just starting the process of figuring out what to do after XPages, it doesn't particularly matter which other toolkit you learn, as long as it's reasonably modern. If you take some time to learn how to make, say, a React app but end up going with something else down the line, the lessons you learn will apply very closely. A particularly-comfortable option could be to learn JSF, which has a common ancestry with XPages but has up-to-date capabilities.

Whatever it is, though, just learn some other toolkit.

Follow Channels and Accounts for Other Tech

Over the last couple months, I've started following a lot of Jakarta-related blogs and Twitter luminaries. This applies elsewhere - even if you're not using other toolkits yet, it's very helpful to start immersing yourself in the news and culture.

Don't Stay Still

The primary thing to take to heart is the importance of doing something. Unless you're planning to change careers or retire in the short term, you'll have to make one decision or another. XPages is not going to get meaningfully better, and even existing apps will get worse with time as browsers and technology change.

Other environments, though, are already leagues ahead and are constantly improving. Dive in; the water's fine!

Letting Madness Take Hold: XPages Outside Domino

Mon Jan 07 10:59:00 EST 2019

Tags: xpages
  1. Letting Madness Take Hold: XPages Outside Domino
  2. XPages on Android
  3. Concept Proven: A Complex Production XPages App Running Outside Domino

(Opening caveat: unlike some of my other recent dalliances, I don't plan to actually do anything with this one, and it's more of a meandering exploration of the XPages platform)

Since I've been on a real Open Liberty kick lately, over the weekend I decided to go another step further and test something I'd been wondering for a while: whether it'd be possible to run the current form XPages outside of the Domino HTTP stack.

I say "the current form" because XPages's history is long and winding, and led a fruitful life for a long time before being glommed onto Domino at all. If you poke around the core, you can see it bears all the scars of its life: references to WebSphere Portal abound, half of the plugins that make up the runtime are just thin OSGi wrappers around plain old Jars, and all of the "Domino" bits are clearly labeled as "adapters".

Still, it's been over a decade since the stack was intended to run anywhere outside Domino, and that's a lot of time for ingrained assumptions about nHTTP specifically to creep in. Still, I was curious if it was possible to load it up outside of Domino and without OSGi.

Short Answer

Yep!

Long Answer

There are a couple things that contribute to making this setup practical, and they each bear some expansion.

Platforms and Execution Contexts

At a couple levels, the runtime breaks things up into generic concepts of "Platforms" and "ExecutionContexts" to handle some specifics about context directories, class loaders, and other bits. For example, if you get a type hierarchy on com.ibm.commons.Platform in Designer, you'll get a pretty immediate idea of what's going on:

OSGi/Services Bridge

Anyone who has written an XPages Library plugin is familiar with the concept of an OSGi extension: you declare your extension (for our purposes, and usefully, com.ibm.commons.Extension) in plugin.xml and then the environment picks up on it by the code looking for such extensions. The core Java runtime has a similar mechanism - ServiceLoader - that looks for files with the name of the extension type in the META-INF/services directory in your classpath. The result of both is the same: individual Jars/plugins can declare services and some other part of the app can pick up on them without knowing the specifics.

XPages uses IBM Commons's generic "Extension" type to paper over the differences between these, and the runtime will look for both or either depending on where it's working. And here's another part that conveniently still retains the vestiges of its youth: if you look inside the com.ibm.xsp.core plugin (since it's just a ZIP file), you can see these extensions declared both in the top-level plugin.xml and as individual files inside the embedded Jar:

So, if you load in these inner Jars as normal Maven dependencies in a .war file, the services will still tie together in much the same way, at least for the core runtime. Things get less convenient the newer the code is, though: the Extension Library, for example, primarily uses plugin.xml for its services, and so either an adapter runtime would have to look for this or you'd have to re-declare them in the "normal" way.

Light OSGi Use

Speaking of OSGi, that's one of the big potential stumbling blocks. XPages nowadays expects to run inside an Equinox container, and so a lot of code (say, the Dojo plugins) make assumptions about the loading of Activator classes and other things. These need some patching. Fortunately, the actual use of OSGi in most of these cases is extremely light: mostly, it's about instantiating these activators and then getting bundle class loaders. For basic needs, these can just be shimmed in: find the (blessedly public) static instance property in the applicable classes and put in small BundleContext Bundle adapters that just return the context class loader. I'm sure there are bits that run deeper than that, and long-term it'd probably be more practical to just fire up Equinox, but this works for now.

FacesServlet

The core work of rendering an XPage runs through the class FacesServlet and more specifically DesignerFacesServlet (as a side note, I've gathered that seeing "Designer" in these classes refers most likely to "Lotus Component Designer", since those parts of the stack enter in before the Domino dependencies). In a modern JEE context, this'll take a little bit of wrapping, since it implements Servlet but doesn't extend HttpServlet, but not too much. For the most part, once you have your platform set up above, you can make a standard @WebServlet-annoted class and delegate the HttpServletRequest and HttpServletResponse objects to one of these, and it'll pick up on any compiled xsp.PageName classes in your .war:

@Override
public void init(ServletConfig config) throws ServletException {
  this.delegate = new DesignerFacesServlet();
  delegate.init(config);
}

@Override
protected void service(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {	
    delegate.service(new LibertyServletRequestWrapper(req), resp);
}

 

DesignerGlobalResourceServlet

Alongside the core Faces servlet, the DesignerGlobalResourceServlet does the work of, well, serving up global resources. This one's simpler than the Faces servlet, since it is indeed a fully-fledged HttpServlet. You could just declare this in your web.xml, but I like extending these classes in case I want to fiddle with them later:

@WebServlet(urlPatterns="/xsp/.ibmxspres/*")
public class LibertyGlobalFacesResourceServlet extends DesignerGlobalResourceServlet {
	private static final long serialVersionUID = 1L;
}

The NSF Part

Up until now, what I was able to create was a way to run XPages inside a normal web app without any real connection to Domino (other than pulling in the binary plugins). Actually running an existing XPage out of an NSF requires a little more bootstrapping, and unfortunately confines the page a bit.

Specifically, the most expedient route I found to accomplishing this was to fire up an LCDEnvironment object and ferry requests for NSF-hosted apps to this. With the presence of an active Notes runtime (via NotesThread.sinitThread() and bringing in Notes.jar and the NAPI plugin), LCDEnvironment#initialize will do a lot of legwork in assembling its own little world inside your application. It will look for com.ibm.designer.runtime.domino.adapter.HttpService declarations and bring them in, including the vitally-important NSFService.

The nice part of this is that it does a ton of work, handling not just XPages requests, but also in-NSF resource requests. The down side is that the NSFService does its work by heavily wrapping the environment, down to providing a servlet context that declares itself as 2.4 even in a 4.0 runtime. Still, a bit of code in the service method gets it working nicely:

String contextPath = StringUtil.toString(req.getContextPath());
String path = req.getRequestURI().substring(contextPath.length());
RequestContext requestContext = new RequestContext(contextPath, path);
HttpSessionAdapter sessionAdapter = new ServletHttpSessionAdapter(req.getSession());
HttpServletRequestAdapter requestAdapter = new LibertyServletRequestWrapper(req);
HttpServletResponseAdapter responseAdapter = new ServletHttpServletResponseAdapter(resp);
lcdEnvironment.service(requestContext, sessionAdapter, requestAdapter, responseAdapter);

Those adapter/wrapper classes really just delegate the calls, but they're needed because that's what LCDEnvironment expects. I imagine those interfaces exist to create a consistent environment in lots of situations without even tying to the stock HttpServlet* classes. In general, the XPages stack loves adapters.

So, Is This A Good Way To Run XPages?

Nnnnnnnnnope! I mean, not really. Particularly in the first route, where you load up an XPage without any knowledge of the NSF part, you have some intriguing paths to interact with the surrounding Servlet 4.0 environment directly from the page. However, I don't know why you would do this. You could hypothetically create some components or hooks to allow use of, say, Web Sockets, but you'd be better off just using current JSF for that, since the work is already done. The in-NSF XPages runtime adds some extra barriers to that, too... I'm sure it'd be possible to provide a path to it, but, again, you'd be better off using existing tech.

Additionally, this isn't a way to bring XPages "home" to JSF. The JSF API and implementation that XPages uses isn't merely old, but hacked to pieces: if you look at javax.faces.component.UIComponent, you can see it's riddled with "_xsp" methods, indicating a thoroughly-unclean layering. You wouldn't be able to shim in the current JSF without forking it into a distinct project.

Still, it's nice to know it's possible, and it sure was a fun project to tinker with. I do admit that the notion of building an XPages app using MVC 1.0 with XPages as the view technology is a little tantalizing, but it's certainly not worth traveling down that road on the back of a chopped-up hacky rework of the platform. And the only reason it's tantalizing is that I'm still more comfortable with XPages than any of the other front-end stacks, and that doesn't itself make it a good fit. Fun to think about, though.

How Do You Solve a Problem Like XPages?

Fri Nov 02 12:08:08 EDT 2018

Tags: xpages gloom

(Fair warning: this is a meandering one and I'm basically a wet blanket the whole way through)

Last week, HCL held the third of their Twitter-based developer Q&As, with this one focusing on XPages and Designer. The majority of the questions (including, admittedly, all of mine) were along the lines of either "can we get some improvements in the Java/XPages stack?" or "is XPages still supported?". The answer to the latter from HCL, as it would have to be, is that XPages is still alive and "fully supported".

I don't doubt at all that XPages is supported in the sense that it has been for the last couple years: if you as a customer encounter a bug in the platform, support will take your call and will most likely either have a workaround or will get a fix in. This will no doubt happen naturally as time marches on, primarily when a new version of a browser breaks something in the version of Dojo that XPages uses (currently, I believe, a couple notches down the list as 1.9.7). So, that's good, and is better than the worst-case scenario.

It's not great, though. Version 10 had essentially no changes for XPages - that makes sense with its "stanch the bleeding" market goal, but it continues the history of very little progress. Outside of the forced-for-security-reasons bump to Java 8 in 9.0.1FP8 (which is admittedly nice), the last major addition to XPages was the Bluemix tooling via the Extension Library, which, as far as I can tell, only exists because of the way funding politics works inside IBM. Before that, I'd mark it as the promotion of Bootstrap renderers from Bootstrap4XPages to the main ExtLib just shy of four years ago. Before that, it was... I guess adding the ExtLib to the main product in 9.0, which sort of counts. Before that, it was pretty much the introduction of the extension points and ExtLib in the 8.5.2 era. And sessionAsSigner, I suppose.

Not to belabor the point too much further, XPages developers are in an uncomfortable spot. For a decade or so now, XPages was the clear "this is what Domino developers should be doing" choice, especially in the face of many Domino shops wanting to at the very least get rid of the Notes client. However, though it was modern enough when it was introduced, the stack has missed the boat on a lot of evolution, both in simple terms of its Java EE common ancestor improving on its own and in larger terms of changes in the web development world.

Had XPages continued under real active development, it could have gradually improved to fit more comfortably in a world of transpiled JavaScript and CSS pre-processors, strict focus on REST APIs, and reactive and streaming APIs.

But...

But even in that "active development" alternate universe, the path would have been awkward. Though XPages has the capacity to be well-structured, Designer and IBM provided no help on this: no model framework, no internal routing, not even an indication that writing Java code was possible in an XPages app until several versions in. The "MVC" aspects of its JSF components were beaten down to match the expectations of an NSF container and of LotusScript developers. There's very good reason for that - very few Domino shops were likely to send developers to computer-science boot camps to learn about proper Java EE structure, and the only way XPages was going to work at all was if it started out as "forms but with partial refresh".

And, after that first unstructured version, it largely fell prey to the usual problems of enterprise software: IBM isn't in the business of doing things their customers aren't asking for, and the community members asking for, say, a faces-config.xml editor weren't backing those requests up with big licensing checks. I get that, too, I really do. If you spend too much time hypothesizing about and implementing what customers might want, you run the risk of throwing your money down a bottomless hole while your actual customers suffer and leave. So IBM generally swings hard in the other direction, and it's understandable if unfortunate in cases like this, especially when what your customers are strictly asking for is stasis.

So What Now?

Our current situation now is that HCL plans to have a roadmap in Q1 2019, so I suppose we'll wait and see what they say again. I've been mulling over what I think should be the way forward for XPages and its users, and I don't see any clear good solution.

The option that immediately springs to mind is "add more features to it": HTTP/2 and WebSockets, newer renderers, an IDE that encourages good development practices, better way to bring in third-party libraries, cleaner JavaScript, and so forth. But what makes this questionable for me is the sheer amount of work, especially since they'd have to start by digging out of an immense amount of technical debt. And this would be very specialized work indeed - the XPages stack is complicated. I'd wager that just the part of the stack that handles ferrying attachments between the browser and a document is more complex than most entire Domino applications by a good margin. While some of the improvements would be handled via the core Domino server team (part of the HTTP upgrades, namely), HCL would likely have to acquire a team of Java developers and have them learn a giant stack of OSGi, JSF, Domino APIs, a couple decades of legacy decisions before even getting going. Possible? Certainly. It's just kind of a hard sell.

Another possibility would be open-sourcing the stack and either maintaining it as a project themselves, giving it to OpenNTF, or handing it to an organization like Eclipse, which now holds the reins of Java/Jakarta EE generally. I think that open-sourcing it would have immediate benefits to XPages developers regardless of what else they do with it, but I'm skeptical of how much of a life it would have if it was converted to a community-run project. As it stands, I can only think of a handful of people who a) are aware of XPages and b) would be capable of contributing to its core code. That's not a problem if you are employing people to work on it full-time, but I don't think it has a large enough base to exist on a "side project" basis. Attracting new blood would be an uphill battle: even projects like Andmore that have a clear purpose for existing and a contingent of people desperate to keep their workflow can wither on the vine immediately. Outside of Domino developers, XPages would be viewed as "JSF, except old and restricted to a platform you thought died in the 90s".

The other main thing to do with the stack that doesn't involve killing it, I think, would be pushing it to a state that focuses on REST APIs instead of handling the UI itself, which is something that some XPages devs have been doing already, either by switching to plugins serving up JAX-RS services or via in-NSF controls. This is something that IBM kind-of-sort-of said they were aiming for a couple years ago when they put forward SmartNSF as a good option, but the effective demise of the Extension Library cut off its path into the core, at least for now. Overall, I think that this would make sense. The experience of writing REST services inside an NSF (or in an OSGi plugin) is significantly worse than JAX-RS in a normal Java EE or Spring app, but it provides a clear path for existing NSFs and code to continue being used - more or less - without having to set up a second app server. It may not be the preferred solution for Java in Domino, but it would have the advantage of improving the platform without having to worry anymore about how, say, all the core renderers use tables for layout.

For Developers

For XPages developers, my long-proffered advice remains the same: learn things that aren't XPages. Whether that means diving head-first into Java EE or Spring, focusing on client-JS development, learning Node, or any other option, you'll likely be well-served by it.

It's possible that you could continue to do XPages work or go back to the Notes client indefinitely - XPages will get patches if nothing else, and Nomad breathes undeserved life into LotusScript - but there's no guarantee there. There's no guarantee anywhere else, I suppose, but staying too tied to Domino-specific technology keeps you at immediate risk of a from-above directive to switch away.

In short: do not expect a cavalry to ride to your aid.

SNTT: Designer Target Platform

Thu Sep 27 16:38:46 EDT 2018

When working on an XPages project, my development environment is generally set up like so:

  1. Eclipse running in the Mac environment editing Maven-structured plugins and ODPs
  2. Domino running in a local Windows VM, set to run plugins out of the Mac Eclipse workspace
  3. Designer running in the same Windows VM to compile the ODP and work with legacy elements
  4. Firefox DE running in the Mac environment

This has proven to be a fairly comfortable setup, particularly ever since Serdar added the ability to use a remote workspace to the XPages SDK. When I make a change in the plugin, I only need to restart HTTP on Domino and all is well.

But there's been one major annoyance: if I change a method or class that I use in in-NSF Java, I have to install the updated plugins and restart Designer, which is a big distruption to my flow. Even if I was using Eclipse on Windows with the old method of running Designer out of Eclipse (if that even still works), it'd still involve a restart.

Today, though, I realized that I've been doing it the dumb way all along. The reason is that I've been conflating the two aspects of an XPages Library plugin when it comes to Designer.

The first aspect is to tell Designer-the-IDE that an XPages Library named, say, org.example.XspLibrary is available for use in XPages applications, and that it's associated with the plugin org.example.plugin. This is provided by the plugin being installed into the running Designer environment, and absolutely needs a restart when it changes. Designer uses this information to compose the OSGi pseudo-project that makes up the NSF in the workspace (by adding it to the plugin.xml historically and the MANIFEST.MF in 9.0.1 FP10+).

The second aspect is that the plugin is then available in the Target Platform. The Target Platform is an OSGi-ism basically meaning the OSGi runtime's view of the world - in Eclipse, it means the plugins that Eclipse knows about and which can be referenced by projects, such as the NSF. This doesn't have to be related at all to the running platform - and, in fact, it's common in OSGi development to have an entirely-distinct target platform to properly represent a server or other runtime environment.

The reason that these aspects are comingled is that, by default, Designer is configured to have a default Target Platform that's based on its runtime environment and any installed plugins. We, as XPages developers, thus generally don't have to think too much about it. "Install the plugin in Designer" is the single step that handles registering the library and making its classes available.

However, the platform is just a setting in the preferences, and it usually looks something like this:

Default Designer Target Platform

Since it's just a setting, and one that is commonly modified in Eclipse, there's nothing stopping us from modifying it. That's where I realized I was doing it the inefficient way all along. So I modified the active platform to point to the target/site folder of my update site Maven project:

Modified Designer Target Platform

With that change, Designer will see both the installed version as well as the latest results of the Maven build. So now, I can do a Maven build and then, in this dialog, click "Reload..." to get Designer to notice the changes. Once I do, voilà - the new methods/classes show up and I don't have to restart and lose my workspace state.

Compiling and Testing XPages Plugins With Java 9+

Fri Apr 13 15:38:50 EDT 2018

Tags: java xpages

Thanks to 9.0.1 FP8, we've been able to use Java 8 on Domino for a while, and FP10 makes that support a bit more official at the OSGi level. However, Java 8 is no longer the latest Java runtime, and so anyone writing XPages plugins and compiling/testing them via Maven will likely run into a situation where the compiling JRE is 9 or above. There are a couple changes in these runtimes that add some wrinkles to the process, so I took up the task of working around the problems I hit, and I created a Git repository to contain the code and tips I used to do so:

https://github.com/OpenNTF/org.openntf.domino.java9compat

The README in the repo contains a couple bits of Maven configuration that can be used to successfully compile XPages plugins in Java 9 or 10, and the included project contains a patch fragment for the com.ibm.notes.java.api plugin that serves up the Notes.jar to work around the fact that CORBA has been removed from the standard JRE.

For my needs, those changes made me able to compile a reasonably-complex XPages application, but there may be other edge cases I haven't hit yet. As I do, I'll add information and code there.

Next Project: ODP Compiler

Mon Mar 05 17:32:06 EST 2018

Tags: xpages java
  1. Next Project: ODP Compiler
  2. NSF ODP Tooling 1.0
  3. NSF ODP Tooling Example Project
  4. NSF ODP Tooling 1.2
  5. How the ODP Compiler Works, Part 1
  6. How the ODP Compiler Works, Part 2
  7. How the ODP Compiler Works, Part 3
  8. How the ODP Compiler Works, Part 4
  9. How the ODP Compiler Works, Part 5
  10. How the ODP Compiler Works, Part 6
  11. How the ODP Compiler Works, Part 7

One of the larger thorns in my side with my Domino development lately has been trying to automate builds of on-disk projects into NSFs via Jenkins. In theory, the process is pretty straightforward. It even works sometimes! However, particularly once you add in the necessity to deploy OSGi plugins to Designer first and want to run it from Jenkins, things get extraordinarily flaky: Designer may not launch properly from a behind-the-scenes Jenkins runner, the plugin installation may mysteriously fail, and so forth - and the error reporting is difficult at best.

So it's been on my mind for a good while to find a way to get from an ODP to an NSF without involving Designer, and I decided over the last couple days to really take a swing at it. It's not a small task, though; the process involves a number of difficult steps:

  • Install and activate provided OSGi plugins
  • Create an XPages registry that knows about the XPages libraries installed on the server, including those just contributed
  • Translate XPages and Custom Controls into Java source, with intra-file knowledge of the just-added CCs
  • Create a Java classpath that matches the plug-in dependencies that Designer derives from the dependant XPages Libraries
  • Compile the resultant Java source and any Java classes in the NSF into bytecode
  • Recompose the composite data form of the file data for these elements and many file resources into their DXL ".metadata" files for import
  • Create an NSF and import all of this
  • Compile any LotusScript in source-based libraries from the ODP
  • Uninstall any hot-loaded OSGi plugins

Those steps even leave out some fiddly details, like components defined via .xsp-config files in the NSF or XSP-associated .properties files, not to mention any steps I haven't encountered yet. It's a lot of work!

My first hope was to be able to hook into the process that Designer uses, perhaps grabbing a couple pertinent OSGi plugins and going from there. However, from what I can tell, all the involved plugins are intricately tied into many layers of Designer-the-IDE and so are no small matter to use on their own without also including the entire stack. So that left me to cobble together an equivalent process out of parts.

Fortunately, a couple projects have already provided a solid foundation for this. First and foremost is the XPages Bazaar. This is a project that Philippe Riand created a number of years ago, meant to be a workshop for really experimental components in a form less contrained than the ExtLib became. Since he left IBM, it's sat unmaintained, but I figured it'd be a perfect incubator for this project, so I tossed it up on GitHub, recomposed its Maven structure, and cleaned it up a bit for FP10 use. The reason why it makes such a perfect shell is a pair of its features: an XSP interpreter and an on-the-fly Java compiler. The former hooks into the mysterious guts of the XSP runtime to allow for translation of XSP to Java and the latter wraps the official Java compiler API with some OSGi knowledge to compile that along into bytecode.

Even starting with this, the first couple steps still required a lot of digging around. I learned how to install and activate OSGi bundles, how XPages Registries work internally, and made some tweaks to work around problems I encountered. I also encountered the joy of a bizarre javac bug to do with annotations in enum constructors, which my target project used.

Once I had the XPages-side components compiled, the next step was to start composing the NSF. The ODP format for XPages elements and other "file resource"-type entities is to put the code in its "normal" form in a file and then a subset of the DXL in a ".metadata" file next to it. The trouble here is that, even for entities where the file data is stored in the note unprocessed, the storage format isn't a strict binary blob of the file data: it's a composite data stream of file header and segment structures. I thought of two main ways I could go about getting these file resources into the NSF: via IBM's NAPI and by building the structures into the DXL files before import. The NAPI has a convenient FileAccess class for this purpose (presumably used by Designer), but my attempts to use it met primarily with server crashes. I'm sure it's possible to go this route, but I'd already "solved" the DXL problem years ago, for ODA's Design API. So, at least for now, I took the tack of writing out the binary structure manually, pouring it into the DXL as Base64, and importing that. It's a little inefficient, but it works.

Overall, I've made a lot of progress so far, but there's still a lot to be done: not all file types have their data put into the right places, LotusScript isn't properly compiled, Agents don't do anything at all yet, and XPages+CCs aren't actually imported into the NSF. Still, it's in a spot where I'm confident that it can one day work, whichis more than I could have said a week ago. If you'd like, browse around the code and pitch in if it's an itch you'd like to scratch as well.

New Small Project: generate-domino-update-site

Wed Jan 31 21:28:20 EST 2018

Tags: java xpages

For a good while now, the Domino Update Site for Build Management has been an essential tool for anyone setting up a local OSGi/Tycho development environment. However, it's really withered on the vine - the latest official release matches stock Domino 9.0.1, while recent fix packs have brought a number of improvements and new classes/methods. Since the binary code is entirely IBM's to distribute or not at their leisure, we can't update the release itself. Today's release of FP10, though, pushed me over the edge to the point where I at least wrote a tool to help the local creation of updated repositories.

The result of my work is generate-domino-update-site, a small CLI and programmatic script that you can point to a Domino installation, a destination directory, and an Eclipse program root to have it create (effectively) an updated version of the update site. The result contains a bit more than the official one did, but that should hopefully not hurt anything. Beyond just copying files into a directory, the script does the dirty work of re-packing unpacked bundles and features, vivifying the Notes.jar wrapper bundles, and creating p2 metadata (hence the Eclipse dependency).

To use the tool, you can clone the repository and run the jar as described in the readme. I may also get around to uploading it as a compiled result to a proper OpenNTF project page.

Side-Project Monday Evening

Tue Jun 27 09:49:32 EDT 2017

Tags: xpages

Yesterday, in one of my various Slack chats, the topic of JShell - the Java 9 REPL - came up in the context of how useful it would be for XPages development. Being able to open up a "shell" into a running XPages application could be really useful in a lot of ways - and I think that the XPages Debug Toolbar has an SSJS-evaluate feature that would do something like this.

Still, it got me looking around a bit, and I ran across Groovysh Server, which is a project that combines Apache's SSH server with Groovy's REPL to make an interactive remote-login shell running in the context of a given JRE. It even comes with a Spring Framework binding, showing its utility for this sort of thing.

So I decided to see how easy it would be to adapt this into an XPages context, and the answer is "pretty easy". I created a new project called XPages Groovy Shell to do just this. It's an XSP Library that you can enable in an application to, when it's loaded (i.e. when someone visits it via the web), spawn an SSH server on the specified port to allow logins and evaluation of Groovy code using the app's ClassLoader.

Now, I don't expect this to be a real project necessarily - I have a lot of non-tinkering work on my plate - but it can serve as an interesting proof of concept. Still, as it is, it's not too far from being expanded to having some proper user authentication, and, with some mechanism to "break into" the Faces environment to work with existing bean instances, it could be really something. As it stands, take a look - it's not a lot of code, and the concepts could be useful elsewhere too.

That Java Thing, Part 17: My Current XPages Plug-in Dev Environment

Sun Feb 26 11:23:22 EST 2017

Tags: java xpages
  1. That Java Thing, Part 1: The Java Problem in the Community
  2. That Java Thing, Part 2: Intro to OSGi
  3. That Java Thing, Part 3: Eclipse Prep
  4. That Java Thing, Part 4: Creating the Plugin
  5. That Java Thing, Part 5: Expanding the Plugin
  6. That Java Thing, Part 6: Creating the Feature and Update Site
  7. That Java Thing, Part 7: Adding a Managed Bean to the Plugin
  8. That Java Thing, Part 8: Source Bundles
  9. That Java Thing, Part 9: Expanding the Plugin - Jars
  10. That Java Thing, Part 10: Expanding the Plugin - Serving Resources
  11. That Java Thing, Interlude: Effective Java
  12. That Java Thing, Part 11: Diagnostics
  13. That Java Thing, Part 12: Expanding the Plugin - JAX-RS
  14. That Java Thing, Part 13: Introduction to Maven
  15. That Java Thing, Part 14: Maven Environment Setup
  16. That Java Thing, Part 15: Converting the Projects
  17. That Java Thing, Part 16: Maven Fallout
  18. That Java Thing, Part 17: My Current XPages Plug-in Dev Environment

It's been a while since I started this series on Java development, but I've been meaning for a bit now to crack it back open to discuss my current development setup for plug-ins, since it's changed a bit.

The biggest change is that, thanks to Serdar's work on the latest XPages SDK release, I now have Domino running plug-ins from my OS X Eclipse workspace. Previously, I switched between either running on the Mac and doing manual builds or slumming it in Eclipse in Windows. Having just the main Eclipse environment on the Mac is a surprising boost in developer happiness.

The other main change I've made is to rationalize my target platform configuration a bit. In the early parts of this series, I talked about adding the Update Site for Build Management to the active Target Platform and going from there. I still basically do this, but I'm a little more deliberate about it now. Instead of adding to the running platform, I now tend to create another platform just to avoid the temptation to use plug-ins that are from the surrounding modern Eclipse environment (this only really applies in my workspaces where I don't also have actual-Eclipse plug-in projects).

The fullest form of this occurs in one of my projects that has a private-only repo, which allows me to stash the artifacts I can't distribute publicly. In that case, I have a number of library dependencies beyond just the core XPages site, and I took the approach of writing a target platform definition file and storing it in the root project, with relative references to the packaged dependencies. With this route, I or another developer can just open the platform file and set it as the target platform - that will tell Eclipse about everything it needs. To do this, I right-clicked on the project, chose "New" → "Other..." and then "Target Definition" under "Plug-in Development":

Target Definition

Within that file, I used Eclipse variable references to point to the packaged dependencies. In this repo, there is a folder named "osgi-deps" next to the root Maven project, so I wanted to tell Eclipse to start at the root project, go up one level, and then delve down into there for each folder. I added "directory" type entries for each one:

Target Definition Entries

The reference syntax is ${workspace_loc:some-project-name}../osgi-deps/Whatever. workspace_loc resolves the absolute filesystem path of the named project within the workspace - since I don't know where the workspace will be, but I DO know the name of the project, this gets me a useful starting point. Each of those entries points to the root of a p2-format update site for the project. This setup will tell Eclipse everything it needs.

Unfortunately, this is a spot where Maven (or, more specifically, Tycho) adds a couple caveats: not only does Tycho not allow the use of "directory" type entries in a target platform file like this (meaning it can't be simply re-used), but it also expects repositories it points to to have p2 metadata and not just "plugins" and "features" folders or even a site.xml. So there's a bit of conversion involved. The good news is that Eclipse comes with a tool that will upgrade old-style update sites to p2 in-place; the bad news is that it's completely non-obvious. I have a script that I run to convert each new release of the Extension Library to this format, and I adapt it for each dependency I add:

java -jar
	/Applications/Eclipse/Eclipse.app/Contents/Eclipse/plugins/org.eclipse.equinox.launcher_1.3.100.v20150511-1540.jar
	-application org.eclipse.equinox.p2.publisher.UpdateSitePublisher
	-metadataRepository file:///full/path/to/osgi-deps/ExtLib
	-artifactRepository file:///full/path/to/osgi-deps/ExtLib
	-source /full/path/to/osgi-deps/ExtLib/
	-compress -publishArtifacts

Running this for each directory will create the artifacts.jar and content.jar files Tycho needs to read the directories as repositories. The next step is to add these repositories to the root project pom so they can be resolved at build time. To start with, I create a <properties> entry in the pom to contain the base path for each folder:

<osgi-deps-path>${project.baseUri}../../../osgi-deps</osgi-deps-path>

There may be a better way to do this, but the extra "../.." in there is because this property is re-resolved for each project, and so "project.baseUri" becomes relative to each plugin, not the root project. Following the sort of best practice approach to Tycho layouts, the sub-modules in this project are in "bundles", "features", "releng", and "tests" folders, so the path needs to hop up an extra layer. With that, I add <repositories> entries for each in the same root pom:

<repositories>
    <repository>
        <id>notes</id>
        <layout>p2</layout>
        <url>${osgi-deps-path}/XPages</url>
    </repository>
    <repository>
        <id>oda</id>
        <layout>p2</layout>
        <url>${osgi-deps-path}/ODA</url>
    </repository>
    <repository>
        <id>extlib</id>
        <layout>p2</layout>
        <url>${osgi-deps-path}/ExtLib</url>
    </repository>
	<repository>
		<id>junit-xsp</id>
		<layout>p2</layout>
		<url>${osgi-deps-path}/org.openntf.junit.xsp.updatesite</url>
	</repository>
	<repository>
		<id>bazaar</id>
		<layout>p2</layout>
		<url>${osgi-deps-path}/XPagesBazaar</url>
	</repository>
	<repository>
		<id>eclipse-platform</id>
		<url>http://download.eclipse.org/releases/neon/</url>
		<layout>p2</layout>
	</repository>
</repositories>

The last entry is only needed if you have extra build-time dependencies to resolve - I use it to resolve JUnit 4.x, which for Eclipse I just tossed unstructured into a "plugins" folder in the "Misc" folder, without p2 metadata.

Though parts of this are annoyingly fiddly, it falls under the category of "worth it in the end" - after some initial trial and error, my target platform is more consistent and easier to share among multiple developers and automated build servers.

Quick Post: Maven-izing the XSP Repo

Sat Sep 17 06:58:10 EDT 2016

Tags: maven xpages

This post follows in my tradition of extremely-narrow-use-case guides, but perhaps this will come in handy in some situations nonetheless.

Specifically, a while back, I wrote a script that "Maven-izes" the XPages artifacts, as provided by IBM's Update Site for Build Management. This may seem a bit counter-intuitive at first, since the entire point of that download is to be able to compile using Maven, but there's a catch to it: the repository is still in Eclipse ("P2") format, which requires that you use Tycho in your project. That's fine enough in most cases - since Domino-targetted projects are generally purely OSGi, it makes sense to have the full OSGi stack that Tycho provides. However, in a case where Domino is only one of many supported platforms, the restrictions that Tycho imposes on your project can be burdensome.

So, for those uses, I write a JRuby script that reads through the P2 site as downloaded and extracted from OpenNTF and generates best-it-can Maven artifacts out of each plugin. It tries to maintain the plugin names, some metadata (vendor, version, etc.), and dependency hierarchy, and the results seem pretty reliable, at least for the purpose of getting a non-Tycho bundle with XSP references to compile. This isn't necessarily a route you'd want to take in all cases (since you don't get the benefits of normal OSGi resolution and services in your compilation), but may make sense sometimes. In any event, if it's helpful, here you go:

https://github.com/jesse-gallagher/Miscellany/blob/master/UpdateSiteConversion/convert.rb

Quick XPages Utility: Keep Alive and Alert

Tue Aug 30 15:17:40 EDT 2016

Tags: xpages

For one of my projects recently, I put together a small custom control that I sure wish I had thought to do years ago: a "keep alive and alert" control. For a long time now, the Extension Library has had a "keepAlive" control, which keeps a page session (and, usually, authentication) alive while the user has the browser window open, avoiding the otherwise-common issue of someone sitting on a page "too long" and having it break underfoot. However, that doesn't cover the edge cases: the user putting their computer to sleep and waking it with the same page open, the server rebooting, a server cluster failover, expiring SSO token, or so forth. In those cases, the problem will fall more or less "silently" to the JavaScript console, and the page will just be mysteriously unresponsive.

This control has a similar starting point, where it will ping the server periodically (every 10 seconds in this case), but will also display a Bootstrap modal alert when things go awry. It's not too picky about the cause of the problem: since usually the only practical solution is to reload the page, it just says that and leaves it there. This could also be redone to be more efficient like the ExtLib one (which would avoid the page recomputation inherent in the partial refresh), and it may not cover the case of authorization expiring in an app that allows anonymous access, but it should do the job nicely in the normal case:

<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core">
	<xp:div id="keepAliveAndAlert" style="display: none"></xp:div>
	<xp:scriptBlock><xp:this.value><![CDATA[
		window.__keepAliveAndAlert = setInterval(function() {
			XSP.partialRefreshGet("#{id:keepAliveAndAlert}", {
				onComplete: function() {
					// Good!
				},
				onError: function() {
					jQuery(dojo.byId("#{id:keepAliveAndAlertError}")).modal({
						backdrop: 'static'
					});
					
					clearInterval(window.__keepAliveAndAlert);
					window.__keepAliveAndAlert = null;
				}
			});
		}, 10 * 1000);
	]]></xp:this.value></xp:scriptBlock>
	<xp:div style="display: none" id="keepAliveAndAlertError" styleClass="modal fade error-modal" role="dialog">
		<div class='modal-dialog'><div class='modal-content'>
			<div class='modal-header'><h4 class='modal-title'>Page Session Expired</h4></div>
			<div class='modal-body'>
				<p>Your page session has expired.</p>
				<p>Please&#160;<a href="javascript:location.reload()">reload</a>&#160;the page to continue working.</p>
			</div>
		</div></div>
	</xp:div>
</xp:view>

Change Is In The Air

Fri Aug 26 17:41:50 EDT 2016

Tags: xpages domino

During last week’s MWLUG, there was a clear sense that things are a little different this year. Dave Navarre dubbed the technical implications “platform agnosticism”, while I geared my presentation towards the feeling that change is in the air.

This is not totally new. Red Pill Now cast aside the XPages UI layer and most of the assumptions of Domino development to move to a new level; PSC's presentations have long developed a polyglot tone, and this ramped up this year; and people like Paul Withers have been growing with tools like Vaadin.

It's not too important to dive into the specifics of market forces or the shifting sands of technology, and the platform defensiveness that we tend to wear as a cozy blanket doesn't serve anyone properly. Our beloved Domino app-dev platform has grown pretty long in the tooth and it doesn’t seem like it’s in for a revitalization.

The situation is, fortunately, an opportunity.

One of the things I hoped to convey in my presentation is that, though the prospect of learning some of the ever-changing array of modern development tools is daunting, it is also exhilarating and profitable both professionally and personally. As insane as the tangled web of server frameworks, JS optimizers, language transpilers, automation tools, dependency managers, and so forth may seem, particularly compared to the simple days of Notes client development, there is a great deal of good news.

The popular tools are awash in documentation, with clear examples for doing basically whatever you will want to do. There's also a lot of conceptual overlap and familiarity: if your tool of choice loses favor, it won't be a start from square one to learn the next. And it's not required to learn every single thing that comes along. If you build yourself a Java web stack using, say, Spring that does the job, it's not also necessary to learn every single new client-side JS app framework that comes along.

And, in the mean time, there's a lot of great work left to do with Domino. There are XPages applications in use and development, and these will go a very long way. Domino remains a very capable platform, and the path through XPages can be a very natural lead-in to other technologies, especially if you focus on the aspects that carry on: Java, data separation, REST services, and so forth.

For my part, I, too, still have great work to do on Domino and XPages, but I'm also expanding beyond it with eyes open. As I mention frequently, I believe that Darwino is the best path for a number of reasons. When I have the opportunity, I plan to start getting into the meat-and-potatoes reasons why and examples of how to actually use the thing. Time permitting, I hope to have a series at some point for converting my blog over to a Darwino+JEE application, and I'll share my process of picking my tools and replicating with its current NSF form as I go. If all goes well, it should serve as an example of taking an existing XPages app and transforming it into something new.

This is an opportunity, and it's an exciting time.


Postscript: This is the optimistic take, granted, and some people’s situations are a bit more dire. Admins, I imagine, are in a strange spot (I hope you’ve been brushing up on ancillary tools!), and a lot of companies are doing a lot of hand-wringing about the future for app dev and maintenance as well, particularly those with a heavy Notes-client dependency. My point is that it’s not necessary to get too mired in the doom and gloom.

MWLUG 2016 Slides

Sat Aug 20 21:07:29 EDT 2016

Tags: mwlug xpages

I just returned from this year's MWLUG, held in surprisingly-rainy Austin, Texas. As every year, MWLUG is an outstanding event, particularly for a development crowd (though I think admins get tons of material too). I'm definitely looking forward to next year's in DC, and not merely because that's a pretty quick drive for me.

I'll have some more to write later to follow up on the themes of this year's event, but in the mean time I've uploaded the slides from my presentation:

AD106 - Expand Your Apps And Skills To The Wider World

This was a fun one to present, and should lead into a lot of good blog material in the future.

Old App Idea: App Manager

Tue May 24 10:12:26 EDT 2016

Tags: xpages

Years back, I took a small whack at an idea that had been percolating in my head: a "app manager" application that would assist with the XPage-specific portions of running a Domino server. It would cover a lot of ground that Administrator really doesn't touch, such as inspecting NSFs to see which contain XPage artifacts, highlighting potential problems with them, and assisting with app-design backup and deployment.

It never really got too far, since there are always a hundred other things to work on, but I always liked the idea. I decided to toss what code I had laid down up on GitHub:

https://github.com/jesse-gallagher/app-manager

I don't imagine I'll really have time to build on that unless I'm suddenly really struck with inspiration. But either way, the code is there for anyone interested in taking a look.

A Bit of Code Archaeology

Thu Mar 10 09:08:47 EST 2016

Tags: xpages

Yesterday, I decided to toss the source of my first real XPages app up on GitHub:

https://github.com/jesse-gallagher/Raidomatic

It's my WoW guild's web site, which had some forums as well as a raid-management tool and loot tracker. I'm guessing that those tools won't be particularly useful for your average XPages app, but they were interesting things to build, and were a great exercise in figuring out the platform. Since it is quite old, there are also plenty of terrible decisions in there, such as my mass recycler from before I knew that Domino objects are already all recycled at the end of the request, but hey, that's growth.

In any event, it's there for the curious, or in case any of the code ends up being useful for future Googlers.

That Java Thing, Part 12: Expanding the Plugin - JAX-RS

Thu Dec 03 15:21:28 EST 2015

Tags: java xpages
  1. That Java Thing, Part 1: The Java Problem in the Community
  2. That Java Thing, Part 2: Intro to OSGi
  3. That Java Thing, Part 3: Eclipse Prep
  4. That Java Thing, Part 4: Creating the Plugin
  5. That Java Thing, Part 5: Expanding the Plugin
  6. That Java Thing, Part 6: Creating the Feature and Update Site
  7. That Java Thing, Part 7: Adding a Managed Bean to the Plugin
  8. That Java Thing, Part 8: Source Bundles
  9. That Java Thing, Part 9: Expanding the Plugin - Jars
  10. That Java Thing, Part 10: Expanding the Plugin - Serving Resources
  11. That Java Thing, Interlude: Effective Java
  12. That Java Thing, Part 11: Diagnostics
  13. That Java Thing, Part 12: Expanding the Plugin - JAX-RS
  14. That Java Thing, Part 13: Introduction to Maven
  15. That Java Thing, Part 14: Maven Environment Setup
  16. That Java Thing, Part 15: Converting the Projects
  17. That Java Thing, Part 16: Maven Fallout
  18. That Java Thing, Part 17: My Current XPages Plug-in Dev Environment

A couple of months back, Toby Samples wrote a series on using Wink in Domino. I'm not going to cover all of that ground here, but it's still worth dipping into the topic a bink, as writing REST services in an OSGi plugin is a great way to not only add capabilities to your XPages apps, but to also start making your data (and programming skills) more accessible from other platforms.

There are two main terms to know here: "JAX-RS" and "Wink".

JAX-RS stands for "Java API for RESTful Web Services". If you're observant, you may notice that there's no "X" in that phrase. This is because the "JAX" part is sort of a legacy hanger-on from the common "JAX" prefix used in JAX-WS and the other Java XML APIs... even though JAX-RS doesn't necessarily involve any XML. In any event, JAX-RS is a specification for a way to write RESTful web services in Java with (mostly) inline configuration using annotations.

Apache Wink is a specific implementation of this standard. For the most part, with these Java standards, the actual implementation shouldn't matter too much, but there are always edge cases. In any event, Wink has historically been the preferred method of doing this on Domino by virtue of the fact that the ExtLib's DAS is built on it and so it ships with Domino, along with some IBM support code.

Setting up JAX-RS services is pretty similar to the addition of the XSP Library and theme provider: we'll need to create the Java classes and add some configuration to the plugin.xml. In this case, we'll do it in reverse, starting with the configuration first before diving into the servlet class.

There are actually two ways to proceed here. In the past, I've added servlets directly to the plugin.xml file. This works fine, but Toby's series took a different and more-interesting route: instead of having the plugin provide a servlet alone, it provides a web application which in turn contains a servlet. Doing it this way brings the configuration more in line with what you'd see in a non-OSGi Java web application.

To start with, we'll need to add dependencies on four more plugins. Open the META-INF/MANIFEST.MF file, go to the "Dependencies" tab, and add "org.apache.wink", "com.ibm.domino.osgi.core", "org.eclipse.equinox.http.registry", and "com.ibm.wink":

One word of caution: when adding "org.eclipse.equinox.http.registry", make sure to add version "1.0.100". Eclipse will likely have a newer version available as well, but that won't be there on Domino. Setting the version requirement too high will cause the plugin to fail to load.

While on this screen, add "lotus.domino" in the "Imported Packages" section of the page. OSGi has two mechanisms for specifying dependencies - until this point, we've used exclusively the "Required Plug-ins" route, which attaches a plugin by name as a dependency. The "Imported Packages" route is a little different: it says "I want this Java package available, but I don't care what plug-in provides it". Outside of the Domino world, the Packages route is very common, but for our needs we usually use Plug-ins to avoid some common nitty-gritty issues with the XPages stack.

The plugin.xml addition is pretty similar to the services we added before, but with some different tags. We're going to add another extension point to the list, along with a couple lines of configuration. With a minor cleanup to the previous lines, the result should look like this:

<?xml version="1.0" encoding="UTF-8"?>
<?eclipse version="3.4"?>
<plugin>
	<extension point="com.ibm.commons.Extension">
		<service class="com.example.xsp.ExampleLibrary" type="com.ibm.xsp.Library"/>
	</extension>
	<extension point="com.ibm.commons.Extension">
		<service class="com.example.xsp.theme.ExampleStyleKitFactory" type="com.ibm.xsp.stylekit.StyleKitFactory"/>
	</extension>
	<extension point="org.eclipse.equinox.http.registry.servlets">
		<servlet alias="/examplerest" class="com.example.xsp.servlet.ExampleServlet">
			<init-param name="applicationConfigLocation" value="/WEB-INF/exampleapplication"/>
			<init-param name="propertiesLocation" value="/WEB-INF/exampleservlet.properties"/>
			<init-param name="DisableHttpMethodCheck" value="true"/>
		</servlet>
	</extension>
</plugin>

The code inside of the extension point is sort of a compressed version of the kind of servlet configuration you see in JEE's web.xml file.

This is actually an area where Toby's series and these instructions diverge. In his series, he used a different extension point - com.ibm.pvc.webcontainer.application - to provide his servlet as part of a full web application. That is a very interesting route as well, and opens the door to broader Java application development. Either one will get the job done, but the "just the servlet" route saves a bit of cognitive overhead for now.

Now, to create the files mentioned in the configuration. In the src/main/resources folder, create a folder named WEB-INF and add the exampleapplication and exampleservlet.properties files, both plain text:

The exampleservlet.properties file can be left blank for now. It's there to provide Wink-specific initialization properties that are useful as you get further into it, but are not needed now.

The exampleapplication file lists the classes that will provide the individual REST resources, one per line. For now, we'll just have one:

com.example.xsp.servlet.HelloWorldResource

With the configuration set up, it's time to create two Java classes. First, create a new class named "ExampleServlet" in the "com.example.xsp.servlet" package and have it extend "com.ibm.domino.services.AbstractRestServlet":

This class doesn't actually need to do anything other than exist for our purposes, but it can be a useful point for hooking in behavior that occurs for each servlet request. The actual code will go in the "HelloWorldResource" class mentioned in the application file. So create that class in the "com.example.xsp.servlet" package. Unlike the previous class, this one doesn't need to extend anything:

This one contains the core of our business logic (such as it is):

package com.example.xsp.servlet;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import javax.ws.rs.QueryParam;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.UriInfo;

import lotus.domino.NotesException;
import lotus.domino.Session;
import lotus.domino.Database;

import com.ibm.commons.util.StringUtil;
import com.ibm.commons.util.io.json.JsonJavaObject;
import com.ibm.domino.osgi.core.context.ContextInfo;

@Path("/helloworld/{id}")
public class HelloWorldResource {
	
	@GET
	@Produces(MediaType.APPLICATION_JSON)
    public Response get(
    		@Context UriInfo context, @PathParam("id") String id, @QueryParam("echo") String echo
    		) throws NotesException {
    	JsonJavaObject json = new JsonJavaObject();
    	
    	json.put("payload", "Hello world");
    	Session session = ContextInfo.getUserSession();
    	if (session != null) {
    		json.put("userName", session.getEffectiveUserName());
    	}
    	Database database = ContextInfo.getUserDatabase();
    	if(database != null) {
    		json.put("databaseFilePath", database.getFilePath());
    	}
    	if(StringUtil.isNotEmpty(echo)) {
    		json.put("echo", echo);
    	}
    	if(StringUtil.isNotEmpty(id)) {
    		json.put("id", id);
    	}
    	
    	return Response.ok(json.toString()).build();
    }
}

There are quite a few things going on here! Let's start with the JAX-RS annotations. JAX-RS, like other "new-era" Java APIs, using annotations heavily in order to put configuration inline and cut down on the number and size of the previous giant blobs of XML. They look a bit unusual if you haven't encountered Java annotations before, but their intent here is hopefully clear.

At the top, the @Path("/helloworld/{id}") annotation indicates that this class is hooked into "/helloworld" within our servlet, and moreover also expects a path parameter that it will refer to as "id". Then, the @GET annotation indicates that our get method will respond to that HTTP method (the name of the method is arbitrary - it could be "foobar" and still work). The @Produces(MediaType.APPLICATION_JSON) annotation further indicates that the method will supply JSON. There are other ways to accomplish this if the method may return different content types, but we know it's always JSON here.

Inside the method signature, there's a bit more magic going on. We're never going to call this method in any code ourselves, but instead Wink is going to inspect it and its parameter list in order to provide what it requests. By annotating the first parameter with @Context, we're telling Wink that we want to have that populated with the context information for the servlet - this is similar to ExternalContext from XSP development. This object isn't actually used in the example (and could be removed), but I've found that it's handy to have it consistently there. The other two parameters use @PathParam("id") and @QueryParam("echo") to extract the "id" parameter we specified earlier as well as look for "echo=..." in the query string.

The method itself uses IBM's JSON library to produce a simple JSON object containing the information we passed in, as well as some Domino-specific stuff, using the class ContextInfo. This class is sort of like the equivalent of ExtLibUtil for Equinox servlets, at least when it comes to getting access to the current session and database. ExtLibUtil itself, though available, doesn't function properly in a servlet, and so we use this instead. The current session is straightforward enough and works like you'd expect, but the current database is interesting. When you register a servlet via this method, it becomes available not only via "yourserver.com/examplerest", but also within each NSF, like "yourserver.com/foo.nsf/examplerest". By default, that will just provide the same results, but, if your code uses ContextInfo.getCurrentDatabase(), you can get the database the URL is requesting. This is how DAS does its thing, and it allows you to write a servlet that can operate in a contextually-appropriate way in each database.

So phew! Now that this is all written, you can build your update site and deploy it to the server. After restarting HTTP, you should be able to go to a URL like "http://yourserver.com/foo.nsf/examplerest/helloworld/someid?echo=hey", and see a result like this:

{
	"echo": "Hey",
	"userName": "CN=Jesse Gallagher/O=IKSG",
	"id": "someid",
	"databaseFilePath": "foo.nsf",
	"payload": "Hello world"
}

To finish, it's time to commit the changes.

Next up, it will be time to start taking the plunge into Maven. Don't worry; it'll be fun!

That Java Thing, Part 11: Diagnostics

Tue Dec 01 08:43:45 EST 2015

Tags: java xpages
  1. That Java Thing, Part 1: The Java Problem in the Community
  2. That Java Thing, Part 2: Intro to OSGi
  3. That Java Thing, Part 3: Eclipse Prep
  4. That Java Thing, Part 4: Creating the Plugin
  5. That Java Thing, Part 5: Expanding the Plugin
  6. That Java Thing, Part 6: Creating the Feature and Update Site
  7. That Java Thing, Part 7: Adding a Managed Bean to the Plugin
  8. That Java Thing, Part 8: Source Bundles
  9. That Java Thing, Part 9: Expanding the Plugin - Jars
  10. That Java Thing, Part 10: Expanding the Plugin - Serving Resources
  11. That Java Thing, Interlude: Effective Java
  12. That Java Thing, Part 11: Diagnostics
  13. That Java Thing, Part 12: Expanding the Plugin - JAX-RS
  14. That Java Thing, Part 13: Introduction to Maven
  15. That Java Thing, Part 14: Maven Environment Setup
  16. That Java Thing, Part 15: Converting the Projects
  17. That Java Thing, Part 16: Maven Fallout
  18. That Java Thing, Part 17: My Current XPages Plug-in Dev Environment

Though my surprisingly-packed schedule the last few weeks caused a hiatus, it's time to retun to this series with a quick description of some of the diagnostic tools available to you when doing plugin development (outside of connecting the debugger, which I may do eventually).

The primary tool in your "what the heck is going on?" toolbox should be the XPages Log File Reader. This app does a wonderful job providing a web UI for the important diagnostic files you'll likely need to see during plugin development (or normal XPages development as well). Even if you have control over the server and could see the files on the filesystem, having them in one UI is invaluable. By looking through the available tabs and pages, you can usually track down some error message related to your problem. So, if you don't have this installed currently, make a point of adding it.

Your other best friend will be the oddly-named XPages Portable Command Guide. Its purpose isn't as immediately clear as Mastering XPages, but it contains valuable nuggets of wisdom. In particular, for the purposes of developing plugins, it describes how to use the OSGi console commands.

By running tell http osgi followed by an appropriate command on the Domino server console, you can find out a bunch of important diagnostic state information about installed plugins. Things are a little obtuse in there, but eventually you learn enough to glean what you'll need to know. There are three primary commands I use: ss, diag, and bundle.

The ss command allows you to see a status summary of plugins matching a given name prefix. So, for example, if you run it for "com.example", you should get a listing like this:

[1140:00D0-15DC] 12/01/2015 07:44:05 AM  Remote console command issued by Jesse Gallagher/IKSG: tell http osgi ss com.example
[0FA8:0002-0C54] 12/01/2015 07:44:05 AM  Framework is launched.
[0FA8:0002-0C54] 12/01/2015 07:44:05 AM  id State       Bundle
[0FA8:0002-0C54] 12/01/2015 07:44:05 AM  79 RESOLVED    com.example.xsp.plugin.source_1.0.0.201511121147
[0FA8:0002-0C54] 12/01/2015 07:44:05 AM  83 ACTIVE      com.example.xsp.plugin_1.0.0.201511121147

The "State" column's values are states from the OSGi lifecycle, and it's worthwhile to at least read the list on that page. The gist of it is that "INSTALLED" is bad (though one step better than not being listed at all), while "RESOLVED" means "working but not yet activated", "<<LAZY>> means it is waiting to be used (like an XSP library), and "ACTIVE" means all is well (probably). This can be a very useful tool for seeing whether or not a plugin is properly on the server at all, and then whether there is an issue.

The diag command is the next level to drill down into when you want to get information about a misbehaving plugin. If you pass the full name of a plugin (not just the prefix), it will tell you if there are any missing dependencies. Running it on the example plugin, you should get something like this:

[1140:00D0-0D3C] 12/01/2015 08:18:41 AM  Remote console command issued by Jesse Gallagher/IKSG: tell http osgi diag com.example.xsp.plugin
[0FA8:0002-0C54] 12/01/2015 08:18:41 AM  initial@osginsf:osgi-dev.nsf/D3A3F506C30EFAB585257EFB005C40D1/com.example.xsp.plugin_1.0.0.201511121147.jar [83]
[0FA8:0002-0C54] 12/01/2015 08:18:41 AM    No unresolved constraints.

If, however, the plugin has a dependency that is unfulfilled (like this arbitrary one I added for this purpose), you'll get a different message:

[1140:00D0-140C] 12/01/2015 08:26:36 AM  Remote console command issued by Jesse Gallagher/IKSG: tell http osgi diag com.example.xsp.plugin
[0D50:0002-112C] 12/01/2015 08:26:36 AM  initial@osginsf:osgi-dev.nsf/EE0B673EB07C7AC985257F0E0049B171/com.example.xsp.plugin_1.0.0.201512010823.jar [12]
[0D50:0002-112C] 12/01/2015 08:26:36 AM    Direct constraints which are unresolved:
[0D50:0002-112C] 12/01/2015 08:26:36 AM      Missing required bundle org.eclipse.egit.mylyn.ui_4.0.3.

A slight variant of that situation is when there is an optional bundle that is missing, and this is usually fine. For example, it's common to see XPages plugins that have dependencies on "com.ibm.notes.java.api", which is the nicely-packaged version of the lotus.domino classes, but which is not actually on the server (since it uses Notes.jar directly). In that situation, the message is similar, but not a problem:

[1140:00D0-0B70] 12/01/2015 08:29:57 AM  Remote console command issued by Jesse Gallagher/IKSG: tell http osgi diag com.example.xsp.plugin
[0E18:0002-15A4] 12/01/2015 08:29:57 AM  initial@osginsf:osgi-dev.nsf/FB1CC2377927985185257F0E0049F6E8/com.example.xsp.plugin_1.0.0.201512010827.jar [12]
[0E18:0002-15A4] 12/01/2015 08:29:57 AM    Direct constraints which are unresolved:
[0E18:0002-15A4] 12/01/2015 08:29:57 AM      Missing optionally required bundle com.ibm.notes.java.api_9.0.1.

The final command is much more verbose and usually the least useful: bundle. What this does is to spit out a feed of pertinent information about the plugin. The first section of it looks like this:

[1140:00D0-0AF0] 12/01/2015 08:32:10 AM  Remote console command issued by Jesse Gallagher/IKSG: tell http osgi bundle com.example.xsp.plugin
[0E18:0002-15A4] 12/01/2015 08:32:11 AM  initial@osginsf:osgi-dev.nsf/FB1CC2377927985185257F0E0049F6E8/com.example.xsp.plugin_1.0.0.201512010827.jar [12]
[0E18:0002-15A4] 12/01/2015 08:32:11 AM    Id=12, Status=<<LAZY>>    Data Root=C:\Program Files\IBM\Domino\data\domino\workspace\.config\org.eclipse.osgi\bundles\12\data
[0E18:0002-15A4] 12/01/2015 08:32:11 AM    No registered services.
[0E18:0002-15A4] 12/01/2015 08:32:11 AM    No services in use.
[0E18:0002-15A4] 12/01/2015 08:32:11 AM    Exported packages
[0E18:0002-15A4] 12/01/2015 08:32:11 AM      com.example.xsp.beans; version="0.0.0"[exported]
[0E18:0002-15A4] 12/01/2015 08:32:11 AM      org.apache.commons.lang3; version="0.0.0"[exported]
[0E18:0002-15A4] 12/01/2015 08:32:11 AM      org.apache.commons.lang3.builder; version="0.0.0"[exported]
[0E18:0002-15A4] 12/01/2015 08:32:11 AM      org.apache.commons.lang3.concurrent; version="0.0.0"[exported]
[0E18:0002-15A4] 12/01/2015 08:32:11 AM      org.apache.commons.lang3.event; version="0.0.0"[exported]
[0E18:0002-15A4] 12/01/2015 08:32:11 AM      org.apache.commons.lang3.exception; version="0.0.0"[exported]
[0E18:0002-15A4] 12/01/2015 08:32:11 AM      org.apache.commons.lang3.math; version="0.0.0"[exported]
[0E18:0002-15A4] 12/01/2015 08:32:11 AM      org.apache.commons.lang3.mutable; version="0.0.0"[exported]
[0E18:0002-15A4] 12/01/2015 08:32:11 AM      org.apache.commons.lang3.reflect; version="0.0.0"[exported]
[0E18:0002-15A4] 12/01/2015 08:32:11 AM      org.apache.commons.lang3.text; version="0.0.0"[exported]
[0E18:0002-15A4] 12/01/2015 08:32:11 AM      org.apache.commons.lang3.text.translate; version="0.0.0"[exported]
[0E18:0002-15A4] 12/01/2015 08:32:11 AM      org.apache.commons.lang3.time; version="0.0.0"[exported]
[0E18:0002-15A4] 12/01/2015 08:32:11 AM      org.apache.commons.lang3.tuple; version="0.0.0"[exported]
[0E18:0002-15A4] 12/01/2015 08:32:11 AM    Imported packages
[0E18:0002-15A4] 12/01/2015 08:32:11 AM      org.osgi.framework; version="1.4.0"<System Bundle [0]>
[0E18:0002-15A4] 12/01/2015 08:32:11 AM      com.ibm.designer.runtime; version="0.0.0"<update@../../shared/eclipse/plugins/com.ibm.xsp.core_9.0.1.20150605-1000/ [256]>

The "Exported packages" part is usually the most useful: it lets you ensure that the packages you think are being exported are indeed being exported. This won't tell you for sure that all of the classes in those packages are present and working, but it's a start.

The "Imported packages" section, which can run for hundreds of lines, is very verbose, but is potentially useful if you want to track down why code in your plugin that interacts with the outside is misbehaving. Perhaps it hasn't found the right dependent package at runtime, or perhaps the version of the other plugin it's using is not what you expect.

One word of warning about the bundle command: it can very easily display more information than Administrator's remote console will show you, so it's useful to look at the server console directly or the logs for everything. Fortunately, the last couple of lines relate to permission setup, which is not usually useful for this need.


These commands so far have applied to the Domino server, but they also apply to Notes/Designer. To see the OSGi console, though, you need to launch Notes specially, with "-RPARAMS -console" in the command line. I keep a second shortcut on my Start menu around for this purpose. When you launch it, there will be quite a bit of noise in there, since Eclipse is a very chatty thing, but it can be invaluable in a pinch. When using this console, since it's the OSGi console directly and not routed through a server task, you can drop the tell http osgi prefix to the commands and just do things like ss com.example directly.

The client also has useful diagnostic logs, though there's no handy XPages application to go along with it (unless it happens to run in the local web preview, I guess). Instead, if you go to Help → Support in Designer, you have a couple options to view log and trace information. If you're working on a plugin and don't see it in the Xsp Properties page list, this is the place to check.


In the next couple of posts, we'll add a little more code to the plugin, diving into using JAX-RS to serve web services, before marching towards Maven-ization.

That Java Thing, Part 10: Expanding the Plugin - Serving Resources

Thu Nov 12 12:02:24 EST 2015

Tags: java xpages
  1. That Java Thing, Part 1: The Java Problem in the Community
  2. That Java Thing, Part 2: Intro to OSGi
  3. That Java Thing, Part 3: Eclipse Prep
  4. That Java Thing, Part 4: Creating the Plugin
  5. That Java Thing, Part 5: Expanding the Plugin
  6. That Java Thing, Part 6: Creating the Feature and Update Site
  7. That Java Thing, Part 7: Adding a Managed Bean to the Plugin
  8. That Java Thing, Part 8: Source Bundles
  9. That Java Thing, Part 9: Expanding the Plugin - Jars
  10. That Java Thing, Part 10: Expanding the Plugin - Serving Resources
  11. That Java Thing, Interlude: Effective Java
  12. That Java Thing, Part 11: Diagnostics
  13. That Java Thing, Part 12: Expanding the Plugin - JAX-RS
  14. That Java Thing, Part 13: Introduction to Maven
  15. That Java Thing, Part 14: Maven Environment Setup
  16. That Java Thing, Part 15: Converting the Projects
  17. That Java Thing, Part 16: Maven Fallout
  18. That Java Thing, Part 17: My Current XPages Plug-in Dev Environment

After sharing code, one of the handiest uses of a plugin is sharing web resources - stylesheets, JavaScript files, and so forth. This process is similar to the last couple steps in that, though it is not very complicated on the whole, it's pretty non-obvious how to get around to doing it.

To start with, we'll create some resources to serve up. Expand the src/main/resources folder in your project (it will be slightly more useful to use the "normal" folder version and not the source folder with the brown package icon, due to Eclipse's UI) and go to New → Other, then pick "Folder" within "General". Name the folder "web":

The create within it a folder named "example", and within that folders named "css" and "js":

Then, populate it with some files - I created a basic stylesheet named "style.css" and a JavaScript file name "script.js" and placed them within the css and js folders, respectively. It doesn't matter what they contain, as long as it's something you can test later; you could also drag in any files you have from the filesystem.

As a side note, as I did this, creating folders within the hierarchy, Eclipse got a bit wonky about refreshing the list, presumably because of the combined source/normal folder nature of src/main/resources. This inconvenience is a concession to the way m2e, the Maven integrator for Eclipse, will act later - it makes this folder a source folder automatically, so we may as well get used to it.

Next, create a theme file within src/main/resources directly (which is to say, not in the web folder) and name it "example.theme". When you create it, Eclipse will likely have trouble opening up an editor: it will try to use one from the OS, which will almost definitely be invalid. Instead, right-click the file and choose Open With → Text Editor:

Set its contents to:

<theme extends="webstandard" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="platform:/plugin/com.ibm.designer.domino.stylekits/schema/stylekit.xsd" >
	<resources>
		<styleSheet href="/.ibmxspres/.extlib/example/css/style.css"/>
		<script clientSide="true" src="/.ibmxspres/.extlib/example/js/script.js"/>
	</resources>
</theme>

Those paths look so weird because we'll be piggybacking on the ExtLib's resource-provider system.

Now, back to Java to create the classes that will provide these resources to the platform. Because it's less terrifying, we'll start with the StyleKitFactory, which provides the theme to the server. Right-click on src/main/java and create a new Java class:

  • Set its package to "com.example.xsp.theme"
  • Set its name to "ExampleStyleKitFactory"
  • Add two interfaces: com.ibm.xsp.stylekit.StyleKitFactory and com.ibm.xsp.stylekit.StyleKitListFactory. The former will provide the theme to the server, while the latter will tell Designer about the name it can use

Set the class contents to this:

package com.example.xsp.theme;

import java.io.InputStream;
import java.util.Arrays;
import java.util.List;

import com.ibm.xsp.stylekit.StyleKitFactory;
import com.ibm.xsp.stylekit.StyleKitListFactory;

public class ExampleStyleKitFactory implements StyleKitFactory, StyleKitListFactory {

	private static final String[] THEMES = {
		"example" //$NON-NLS-1$
	};
	private static final List<String> THEMES_LIST = Arrays.asList(THEMES);
	
	@Override
	public String[] getThemeIds() {
		return THEMES;
	}

	@Override
	public InputStream getThemeAsStream(String themeId, int scope) {
		if (scope == StyleKitFactory.STYLEKIT_GLOBAL) {
			if (THEMES_LIST.contains(themeId)) {
				return getThemeFromBundle(themeId + ".theme"); //$NON-NLS-1$
			}
		}
		return null;
	}

	@Override
	public InputStream getThemeFragmentAsStream(String themeId, int scope) {
		return null;
	}
	
	private InputStream getThemeFromBundle(final String fileName) {
		ClassLoader cl = getClass().getClassLoader();
		return cl.getResourceAsStream(fileName);
	}
}

There's quite a bit going on here! Some of it is due to the nature of the task and some of it comes from my own built-up habits that come in handy as the class grows. Towards the top of the class, THEMES contains an array of theme names known by the plugin - in this case, just the one, but it's good to have this standardized. THEMES_LIST contains a List wrapper around that array for programmatic convenience later.

The getThemeIds method is from StyleKitListFactory and is used by Designer to generate its list of available themes in the GUI. Implementing this interface isn't required, but it's a cross-the-Ts sort of thing.

The remaining methods implement StyleKitFactory, which provides the server with the theme data itself. getThemeAsStream is called by the runtime when it's searching for a theme requested by the app, so it contains a couple checks to make sure that the request is indeed intended for a plugin-based (global) theme with a name that this plugin knows about. It then uses a small utility method, getThemeFromBundle, to get the theme as an InputStream from the plugin's internal filesystem. Technically, this could be anything - it could construct the theme on the fly, fetch it from a URL, or get it from anywhere else, as long as it returns an InputStream, but this version is the most common.

getThemeFragmentAsStream is an interesting beast. The Extension Library uses it to hook in extra theme info for its Bootstrap themes on the fly. This is presumably useful for ad-hoc theme hierarchies and avoiding the theme-inheritance cap, but we don't have any need for it here.

Finally, the "$NON-NLS-1$" comment business is because I've developed a habit of enabling Eclipse's translated-strings warnings - the comments denote to the IDE that the hard-coded strings on those lines are not intended to be translated. You can ignore those if you so desire.

Moving on, now it's time to set up our resource provider, which will serve up the web resources. Before that, we'll take a minor detour back to the Activator class to fix up a method signature. Add "public" to the getContext method:

public static BundleContext getContext() {
	return context;
}

Now, we're going to implement the resource-loading class, modeled after the one from Bootstrap4XPages. Create a new class:

  • Set its package to "com.example.xsp.minifier"
  • Set its name to "ExampleLoader"
  • Set its superclass to com.ibm.xsp.extlib.minifier.ExtLibLoaderExtension

This class will have three methods of importance to us: getOSGiBundle (which is obligatory), loadCSSShortcuts, and getResourceURL:

package com.example.xsp.minifier;

import java.net.URL;

import javax.servlet.http.HttpServletRequest;

import org.osgi.framework.Bundle;

import com.example.xsp.Activator;
import com.ibm.commons.util.DoubleMap;
import com.ibm.xsp.extlib.minifier.ExtLibLoaderExtension;
import com.ibm.xsp.extlib.util.ExtLibUtil;

public class ExampleLoader extends ExtLibLoaderExtension {
	
	private static final String[] LIBRARY_RESOURCE_NAMESPACES = {
		"example" //$NON-NLS-1$
	};

	@Override
	public Bundle getOSGiBundle() {
		return Activator.getContext().getBundle();
	}

	@Override
	public void loadCSSShortcuts(DoubleMap<String, String> aliases, DoubleMap<String, String> prefixes) {
		if(prefixes != null) {
			for(int i = 0; i < LIBRARY_RESOURCE_NAMESPACES.length; i++) {
				String namespace = LIBRARY_RESOURCE_NAMESPACES[i];
				prefixes.put("9T0a" + i, "/.ibmxspres/.extlib/" + namespace); //$NON-NLS-1$ //$NON-NLS-2$
			}
		}
	}
	
	@Override
	public URL getResourceURL(HttpServletRequest request, String name) {
		for(String namespace : LIBRARY_RESOURCE_NAMESPACES) {
			if(name.startsWith(namespace)) {
				String path = "/web/" + name; //$NON-NLS-1$
				return ExtLibUtil.getResourceURL(getOSGiBundle(), path);
			}
		}
		return null;
	}
}

Once again, there's a lot going on, but you can see some general similarities to the theme provider. The getOSGiBundle method makes use of the Activator.getContext method we just modified, while the other two methods use the same sort of "internal array of Strings" pattern as before to make it easy to add more entries to the list down the line.

Things are a little strange in the loadCSSShortcuts method. The goal of this method is to provide the shorthand codes used in the minified URLs for resources, and so each plugin should make sure to provide unique ones. However, I don't know of any coordinated enforcer class that does this "making sure" for us, so you kind of just have to make up a unique string of characters that should be unique. Here, I picked "9T0a" as the base, just because they usually look like that.

The getResourceURL method is a little simpler, and is the equivalent of the earlier getThemeAsStream - given the incoming request for a resource name, it checks to see if it falls within its bailiwick and, if so, uses a method to provide a URL to the server environment.

These methods, like the ones in the theme provider, can be generally used as a "just drop them in the plugin" sort of thing without modification. There are other potential tweaks you can make, but it's probably best to keep it simple.

There's two more steps to getting these classes working. The theme provider needs to be registered in the plugin.xml and the resource loader needs to be added in the Activator. First, for the theme provider. Add another extension point entry to the plugin.xml:

Your plugin.xml source should look something like:

<?xml version="1.0" encoding="UTF-8"?>
<?eclipse version="3.4"?>
<plugin>
   <extension
         point="com.ibm.commons.Extension">
      <service
            class="com.example.xsp.ExampleLibrary"
            type="com.ibm.xsp.Library">
      </service>
   </extension>
   <extension
         point="com.ibm.commons.Extension">
      <service
            class="com.example.xsp.theme.ExampleStyleKitFactory"
            type="com.ibm.xsp.stylekit.StyleKitFactory">
      </service>
   </extension>

</plugin>

For the loader, open the Activator class and add a new line to the start method to add the resource loader to the ExtLib's bank. The class should now look like:

package com.example.xsp;

import java.util.logging.Level;
import java.util.logging.Logger;

import org.osgi.framework.BundleActivator;
import org.osgi.framework.BundleContext;

import com.example.xsp.minifier.ExampleLoader;
import com.ibm.xsp.extlib.minifier.ExtLibLoaderExtension;

public class Activator implements BundleActivator {

	private static BundleContext context;
	
	public static final Logger log = Logger.getLogger(Activator.class.getPackage().getName());
	static {
		log.setLevel(Level.FINEST);
	}

	public static BundleContext getContext() {
		return context;
	}

	/*
	 * (non-Javadoc)
	 * @see org.osgi.framework.BundleActivator#start(org.osgi.framework.BundleContext)
	 */
	public void start(BundleContext bundleContext) throws Exception {
		Activator.context = bundleContext;
		
		if(log.isLoggable(Level.INFO)) {
			log.info("Starting Example XPages Library");
		}
		
		ExtLibLoaderExtension.getExtensions().add(new ExampleLoader());
	}

	/*
	 * (non-Javadoc)
	 * @see org.osgi.framework.BundleActivator#stop(org.osgi.framework.BundleContext)
	 */
	public void stop(BundleContext bundleContext) throws Exception {
		Activator.context = null;
	}
}

(Despite appearances, the "start" method there is not actually commented out - its blue appearance is a bug in the JavaScript syntax highlighter used in this blog)

Now to see if all this work has paid off: build the update site and install it in Designer and Domino. When you relaunch Designer and open the "Xsp Properties" page of the NSF you're working with, you should see the "example" theme in the list of options (as long as you have a recent ExtLib release installed in Designer - the capability was added post-9.0.1):

Additionally, when you open the app on the web, you should see your CSS and JS files included. One final note about these resources: though the URLs in the theme start with "/.ibmxspres/.extlib/example", URLs references from non-XSP files (say, referencing font files from with CSS) should start instead with "/xsp/.ibmxspres/.extlib/example". The XPages runtime adds that "/xsp" when generating URLs for pages, but it doesn't do any such processing for static references.

So... that was more complicated than it seemed at the start! Still, once it's in place, it's all pretty much "set it and forget it" - in the future, you can add files and themes more easily, just adjusting the static arrays in the appropriate classes as necessary.

Finally, commit the changes and take a relaxing breath.

That Java Thing, Part 9: Expanding the Plugin - Jars

Wed Nov 11 07:50:14 EST 2015

Tags: java xpages
  1. That Java Thing, Part 1: The Java Problem in the Community
  2. That Java Thing, Part 2: Intro to OSGi
  3. That Java Thing, Part 3: Eclipse Prep
  4. That Java Thing, Part 4: Creating the Plugin
  5. That Java Thing, Part 5: Expanding the Plugin
  6. That Java Thing, Part 6: Creating the Feature and Update Site
  7. That Java Thing, Part 7: Adding a Managed Bean to the Plugin
  8. That Java Thing, Part 8: Source Bundles
  9. That Java Thing, Part 9: Expanding the Plugin - Jars
  10. That Java Thing, Part 10: Expanding the Plugin - Serving Resources
  11. That Java Thing, Interlude: Effective Java
  12. That Java Thing, Part 11: Diagnostics
  13. That Java Thing, Part 12: Expanding the Plugin - JAX-RS
  14. That Java Thing, Part 13: Introduction to Maven
  15. That Java Thing, Part 14: Maven Environment Setup
  16. That Java Thing, Part 15: Converting the Projects
  17. That Java Thing, Part 16: Maven Fallout
  18. That Java Thing, Part 17: My Current XPages Plug-in Dev Environment

So it appears that I once again forgot to commit my changes. Well, consider this a cautionary tale, but we can still salvage the situation a bit by committing the previous changes before embarking on an unrelated modification - it's that mixing of different changes that can cause trouble in a source control repository.

For today's post, we'll add a third-party Jar to our plugin in order to use it internally and provide it to external applications (and we'll cover why those are distinct things).

The Jar we'll use is Apache Commons Lang, because it's a nice simple case without complicated dependencies. Embedding it inside a plugin is actually not the ideal deployment strategy for this, since it already contains OSGi metadata, but this setup is often the most expedient.

To start, download the latest binary release from their site and extract the Zip. Now, in Eclipse, right-click the "com.example.xsp.plugin" project, choose New → Folder, and name it "lib". Then, right-click that new folder and choose "Import...":

Then, choose "File System" from within "General":

On the next pane, browse to the extracted folder and, within Eclipse's file-picker list, choose the binary Jar:

Next, we have to make sure that this Jar file gets included in the final build and is available to classes within the Jar. To do that, open the META-INF/MANIFEST.MF file and go to the "Runtime" tab. In the "Classpath" section, click "Add...", and find the newly-added file:

This does two things: it adds it to the plugin's internal classpath (specified in the MANIFEST.MF file) and, as a convenience in Eclipse, adds the file to build.properties (which controls what is included in the build). If you go to the "build.properties" tab, it should look something like this:

source.. = src/main/java/,\
           src/main/resources/
output.. = target/classes/
bin.includes = META-INF/,\
               .,\
               plugin.xml,\
               lib/commons-lang3-3.4.jar

You should also see the Jar within your project list - being at the top level with this "jar of bits" icon means that Eclipse knows that it is included in the project's classpath:

At this point, the Jar's classes are available for use inside the plugin itself, but not exposed to the outside world. So we could add a method like this to the ExampleBean class and then call it from an XPage:

/**
 * @return the Java home directory path according to Apache Commons Lang
 */
public String getJavaHome() {
	return org.apache.commons.lang3.SystemUtils.getJavaHome().getAbsolutePath();
}

As an aside, sometimes Designer doesn't quite obey the normal rules of OSGi visibility, but Domino does, so you can fall into a trap with packages that aren't marked as visible but are still accessible in Designer.

Sometimes we do want to make these classes visible to downstream users, such as when the point of including the Jar is to distribute a set of foundational libraries used across apps. To do this, go back to the MANIFEST.MF file's "Runtime" tab. Now, click the "Add..." in the "Exported Packages" section. You should see all of the Apache commons packages - select them and click OK:

Now, when you build and install this plugin, the classes will be directly available in your XPages applications. This can be a good way to take a handful of Jars you may have sitting around in your jvm/lib/ext folder and distribute them in a better way. It's not the best way, since it misses some OSGi advantages like source and Javadoc for the embedded classes, but it can be much simpler than doing it "right".

Now, commit the changes for today:

In the next post, we'll add some web resources to the plugin and make them available in a theme and able to participate in resource aggregation.

That Java Thing, Part 8: Source Bundles

Tue Nov 10 07:46:28 EST 2015

Tags: java xpages
  1. That Java Thing, Part 1: The Java Problem in the Community
  2. That Java Thing, Part 2: Intro to OSGi
  3. That Java Thing, Part 3: Eclipse Prep
  4. That Java Thing, Part 4: Creating the Plugin
  5. That Java Thing, Part 5: Expanding the Plugin
  6. That Java Thing, Part 6: Creating the Feature and Update Site
  7. That Java Thing, Part 7: Adding a Managed Bean to the Plugin
  8. That Java Thing, Part 8: Source Bundles
  9. That Java Thing, Part 9: Expanding the Plugin - Jars
  10. That Java Thing, Part 10: Expanding the Plugin - Serving Resources
  11. That Java Thing, Interlude: Effective Java
  12. That Java Thing, Part 11: Diagnostics
  13. That Java Thing, Part 12: Expanding the Plugin - JAX-RS
  14. That Java Thing, Part 13: Introduction to Maven
  15. That Java Thing, Part 14: Maven Environment Setup
  16. That Java Thing, Part 15: Converting the Projects
  17. That Java Thing, Part 16: Maven Fallout
  18. That Java Thing, Part 17: My Current XPages Plug-in Dev Environment

Before anything else today, Eric McCormick reminding that yesterday's post missed the final step: committing the changes. So, let's take care of that right now. On my side, since my Windows VM is also being accessed from the Mac side, it's worthwhile to add a .gitignore file to the root of the repo to keep out all the .DS_Store nonsense. GitHub's Java gitignore is a good start, though skipping the part about ignoring Jar files. In your text editor of choice, create a new file named ".gitignore" in the root directory of the Git repository, with this content:

._*
Thumbs.db
.DS_Store

*.class

# Mobile Tools for Java (J2ME)
.mtj.tmp/

# Package Files #
#*.jar
*.war
*.ear

# virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml
hs_err_pid*

Then commit the files:

Now, on to today's topic: adding a source bundle. This is a small topic, but will be very useful as you work down the line. As it stands right now, even though you have access to the source code of your plugin, Designer doesn't, and that means that you don't have nice parameter names, inline Javadoc, or viewable source when working with your classes in an actual XPages application. Since XSP libraries are almost always open-source or internal-use-only, including a source bundle is the fastest way to achieve all of these.

The way to do this is pretty non-obvious, but not complex once you know what to do. The work will be done with a new feature project, so go to File → New → Other and then "Feature Project" within "Plug-in Development". This should be set up very similarly to the original feature:

  • Set the project name to "com.example.xsp.source.feature"
  • Override the location with a folder inside the Git repository
  • Pick a variant on your original feature name, so "Example XSP Library Source Feature"

Unlike before, however, we don't want to select any plugins on the next pane - instead, just hit "Finish". This feature is going to use some Eclipse trickery to automatically generate a "source" version of the existing feature. Once the project is created, open its feature.xml file and go to the "feature.xml" tab. On there, add a bit of XML to manually specify the non-existent source plugin (I've left out the "description", "copyright", and "license" blocks, which are required even if just stubs):

<?xml version="1.0" encoding="UTF-8"?>
<feature
      id="com.example.xsp.source.feature"
      label="Example XSP Library Source Feature"
      version="1.0.0.qualifier">

   <!-- *snip* -->

	<includes
         id="com.example.xsp.plugin.source"
         version="0.0.0"/>
</feature>

Then, go to the "build.properties" tab and add a second line to tell Eclipse's builder to generate the source plugin:

bin.includes = feature.xml
generate.feature@com.example.xsp.plugin.source = com.example.xsp.feature

Next, open the site.xml file in the update site project and add the new feature to the "Managing the Site" section:

Now, save and "Build All". Then, go back to Designer and install the latest version from the update site, which should now have a second feature listed:

When you install that and relaunch Notes/Designer, you'll be able to access the source of your plugin. To see this in action, open your application in Designer and then go to the "Package Explorer" view (you may have to add it via Window → Show Eclipse Views → Package Explorer). Find your NSF's project and expand the "Plug-in Dependencies" category. Towards the bottom, you should see the example plugin Jar file - expand that and double-click on one of the classes:

If all went well, you should see the source to the original class, instead of Eclipse's incomprehensible bytecode dump. Additionally, now, when you access these classes from Java code inside your NSF, you'll get improved inline help, with correct parameter names and Javadoc (if you actually write the Javadoc).

The next step will be another small but useful topic: adding some third-party Jars to your plugin, either to use internally or to expose to your XPages applications.

That Java Thing, Part 7: Adding a Managed Bean to the Plugin

Mon Nov 09 06:37:27 EST 2015

Tags: java xpages
  1. That Java Thing, Part 1: The Java Problem in the Community
  2. That Java Thing, Part 2: Intro to OSGi
  3. That Java Thing, Part 3: Eclipse Prep
  4. That Java Thing, Part 4: Creating the Plugin
  5. That Java Thing, Part 5: Expanding the Plugin
  6. That Java Thing, Part 6: Creating the Feature and Update Site
  7. That Java Thing, Part 7: Adding a Managed Bean to the Plugin
  8. That Java Thing, Part 8: Source Bundles
  9. That Java Thing, Part 9: Expanding the Plugin - Jars
  10. That Java Thing, Part 10: Expanding the Plugin - Serving Resources
  11. That Java Thing, Interlude: Effective Java
  12. That Java Thing, Part 11: Diagnostics
  13. That Java Thing, Part 12: Expanding the Plugin - JAX-RS
  14. That Java Thing, Part 13: Introduction to Maven
  15. That Java Thing, Part 14: Maven Environment Setup
  16. That Java Thing, Part 15: Converting the Projects
  17. That Java Thing, Part 16: Maven Fallout
  18. That Java Thing, Part 17: My Current XPages Plug-in Dev Environment

For today's step, we'll deal more with the meat of the task: putting your own code in the plugin. There are a lot of options for what a plugin can provide, ranging from just storing classes to be accessed from elsewhere to being full-on OSGi web applications. Adding a managed bean certainly falls on the simpler side of this spectrum, but it's also one of the most common uses. These steps should also be very familiar if you've created a managed bean in an XPages NSF before.

Before we get started, it will be useful later to access Extension Library code from this plugin, so there's a quick detour. In order to allow the plugin to "see" ExtLib code, the appropriate ExtLib plugin needs to be added as a dependency. Go to the MANIFEST.MF file's "Dependencies" tab and add com.ibm.xsp.extlib to the list:

Now, create a new class with an appropriate name by right-clicking on the src/main/java folder and selecting New → Class as before. Set its package to "com.example.xsp.beans" and its name to "ExampleBean". You don't have to name your bean classes with "Bean", but sometimes it makes sense. Make sure to add the java.io.Serializable interface via the "Add..." button in that section.

When you create the class, Eclipse will put a squiggly underline beneath the class name, indicating that it has a warning for you. If you hover your cursor over it (or hit Ctrl-1 while the cursor is on the line), you'll get more information and an option to fix it:

What this message is talking about has to do with serialization - the process of storing the Java object outside of memory and retrieving it later - and is something that would come up more in applications that intentionally store data long-term. For a thorough explanation of this and many other things, I can't recommend Effective Java enough. For our purposes as XPages developers, though, it's fine to pick the "default" option - we want more classes Serializable than other Java apps, but we care less about their long-term storage.

Next, add a basic "getter" method that returns a String we can see later, as well as a static method to retrieve the bean from the XPages environment from Java:

package com.example.xsp.beans;

import java.io.Serializable;

import javax.faces.context.FacesContext;

import com.ibm.xsp.extlib.util.ExtLibUtil;

public class ExampleBean implements Serializable {
	private static final long serialVersionUID = 1L;
	
	public static ExampleBean get() {
		return (ExampleBean)ExtLibUtil.resolveVariable(FacesContext.getCurrentInstance(), "exampleBean");
	}

	public String getFoo() {
		return "hello from " + getClass().getSimpleName();
	}
}

Now, in order to make this a managed bean, well have to add a local faces-config file. These files have the same format as the one in the NSF. There's another nod towards Maven here: we'll put the XML file in a separate "resources" folder in the project, which is where Maven expects to find this sort of thing. Right-click the project and choose New → Source Folder:

Name it "src/main/resources":

Right-click on this newly-created source folder and choose New → Other, and within the dialog choose "File" from the "General" section:

On the next screen, name the file "beans.xml" and click Finish:

Most likely, Eclipse will open the file on the "Design" tab, which is pretty useless for this - it's better to switch to the "Source" tab. In this file, fill in the basic faces-config structure and an entry for the managed bean:

<?xml version="1.0" encoding="UTF-8"?>
<faces-config>
	<managed-bean>
		<managed-bean-name>exampleBean</managed-bean-name>
		<managed-bean-class>com.example.xsp.beans.ExampleBean</managed-bean-class>
		<managed-bean-scope>view</managed-bean-scope>
	</managed-bean>
</faces-config>

Now that we have this file, we have to tell the XPages runtime to actually use it as a faces-config file. To do that, go back to the ExampleLibrary class and implement the getFacesConfigFiles method, which lists the XML files from the plugin to contribute to the environment:

@Override
public String[] getFacesConfigFiles() {
	return new String[] {
			"beans.xml"
	};
}

The "@Override" bit above the method is indicating to the compiler that you are intentionally overriding a method declared in the superclass or interface - this is to help you ensure that you keep the method names correct for any code calling them.

Now, there are two minor remaining items to take care of before we deploy: exposing the class to Java code elsewhere and making sure the newly-created source folder is included in the build.

To do the former, open up the MANIFEST.MF file and go to the "Runtime" tab. Here, we can list the Java packages that should be available to code within the application (or other plugins). Since we'll want to have code that accesses the bean directly, add the "com.example.xsp.beans" package to the "Exported Packages" list:

This is important to do for any new package that is going to be accessed externally.

Next, go to the "build.properties" tab of the MANIFEST.MF editor (or open the build.properties file directly) and add the src/main/resources folder to the list of source folders:

source.. = src/main/java/,\
           src/main/resources/
output.. = target/classes
bin.includes = META-INF/,\
               .,\
               plugin.xml

The syntax of this file takes a little getting used to - the backslashes on many lines indicate that the next line should also be part of the same definition, but is split up for legibility.

With that, we're all set on this side. Head over to the site.xml file in the update site project and click "Build All" to build a new version of the plugin. Once that's done, go to the Update Site NSF for the server and import the new build, and then restart task http on the server console. Similarly, use the File → Application → Install routine in Designer to install the new version of the plugin there, and relaunch Notes.

Now that the bean is installed, you can test it out to make sure it's working by creating a new XPage in an app on the server with this content on the "Source" tab:

<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core">
	<xp:text value="#{exampleBean.foo}"/>
</xp:view>

When you launch the page in a browser, it should say "hello from ExampleBean". To try out the static get() method we created, change the XPage source to:

<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core">
	<xp:text><xp:this.value><![CDATA[#{javascript:
		"the bean says: " + com.example.xsp.beans.ExampleBean.get().getFoo()
	}]]></xp:this.value></xp:text>
</xp:view>

When you view that, you should see basically the same thing, prefixed with "the bean says: ". Whenever I make a change to test something that should have the same visual result, I like to toss in a small variation to make sure that my changes did, in fact, take effect.

For a lot of cases, this step may be all you need - many XPage libraries consist primarily as storehouses for Java classes and a few managed beans. The next couple steps are going to deal with some variations on this, plus some nice-to-haves.

That Java Thing, Part 6: Creating the Feature and Update Site

Sun Nov 08 10:45:08 EST 2015

Tags: java xpages
  1. That Java Thing, Part 1: The Java Problem in the Community
  2. That Java Thing, Part 2: Intro to OSGi
  3. That Java Thing, Part 3: Eclipse Prep
  4. That Java Thing, Part 4: Creating the Plugin
  5. That Java Thing, Part 5: Expanding the Plugin
  6. That Java Thing, Part 6: Creating the Feature and Update Site
  7. That Java Thing, Part 7: Adding a Managed Bean to the Plugin
  8. That Java Thing, Part 8: Source Bundles
  9. That Java Thing, Part 9: Expanding the Plugin - Jars
  10. That Java Thing, Part 10: Expanding the Plugin - Serving Resources
  11. That Java Thing, Interlude: Effective Java
  12. That Java Thing, Part 11: Diagnostics
  13. That Java Thing, Part 12: Expanding the Plugin - JAX-RS
  14. That Java Thing, Part 13: Introduction to Maven
  15. That Java Thing, Part 14: Maven Environment Setup
  16. That Java Thing, Part 15: Converting the Projects
  17. That Java Thing, Part 16: Maven Fallout
  18. That Java Thing, Part 17: My Current XPages Plug-in Dev Environment

The last post covered turning our nascent plugin into a proper XPages library, so now it's time to fill in the remaining pieces to get it installed.

To do that, we'll need a feature and an update site. The feature will point to our plugin - in a more-complicated setup, this could point to several related plugins, but we'll just need the one. The update site will do similarly, referencing the feature in a way that Eclipse-type platforms know about.

Go to File → New and choose "Feature Project" within "Plug-in Development":

Hit "Next" and fill out the feature details:

  • Set the project name to "com.example.xsp.feature"
  • Override the location with a folder inside the Git repository (remembering to include the feature name in the path, to avoid placing it in the top-level folder)
  • Pick a feature name, such as "Example XSP Library Feature"

Click "Next", find and check "com.example.xsp.plugin" in the list of plugins to include in the feature, and click "Finish":

That's about it for the feature. Next up: the update site. As before, go to File → New and choose "Update Site Project" within "Plug-in Development":

There's not much configuration to do for this one: just name it "com.example.xsp.update" and override its location to be within your Git repository:

Once the update site is created, click "Add Feature.." in the site.xml editor and add the new "com.example.xsp.feature" project:

One last optional step is to set up a category in the update site, which will help organization on the server. To do that, click "New Category" and give it a name and ID. Then, drag the feature in the list onto the category to add it to the group:

Now the update site definition is set up, and the next step is to click "Build All" to have Eclipse package your plugin inside the update site. That will create a number of files within your update site project, which are the actual artifacts we need:

Next is to install the library into Designer. Launch Notes, then Designer (the sequence is important to avoid a bug with NSF update sites, so I stick to this habit), and open Preferences. In there, go to the "Domino Designer" section and check "Enable Eclipse plug-in install":

Hit "OK" to close the preferences and then go to File → Application → Install. In the dialog that pops up (sometimes there's a lengthy delay before it does for some reason), select "Search for new features to install" and click "Next". On the next pane, choose "Add Folder Location..." and browse to the update site project:

The next dialog will prompt for a name, and for now the default will do. Click "OK" in this dialog and "Finish" in the main one. If all goes well, that should bring up another dialog after a couple of seconds, which will allow you to select the feature:

Click "Next" and then "Finish". As it installs, you'll get a prompt to install the unsigned plugin, so select "Install this plug-in" and click "OK". Designer will do its thing and then display a small toast window in the bottom right asking if you want to restart now. It at least used to be the case that clicking this didn't do a proper restart (presumably due to the Notes stuff), so I still avoid clicking it. Instead, close Notes and Designer, and then relaunch them (Notes first again).

Now, to see that everything is working, open an existing application in Designer (or make a new one), go to Application Configuration → Xsp Properties → Page Generation tab, and look for "com.example.xsp.library" in the "XPage Libraries" section:

If it's not there, something went wrong. Unfortunately, debugging this sort of thing can get hairy, so it'd probably be best to ask me or someone else knowledgable about plugins for assistance for now.

Assuming it did work, though: great! Next up is the installation on the server. This is a whole tutorial on its own, and fortunately IBM has done the job for me. The upshot of those instructions is:

  • Create a database on the server using the "Eclipse Update Site" template (updatesite.ntf) and clean up the ACL to something appropriate. To see the template, make sure to click "Show advanced templates".
  • Import com.example.xsp.update by using the "Import Local Update Site" option and pointing to its site.xml file
  • Set the notes.ini property "OSGI_HTTP_DYNAMIC_BUNDLES" on the server to point to the file name of that update site
  • Restart HTTP or the entire server

Once HTTP or the server is restarted, you can test to see if it's working by creating an application on the server that has the library checked in the Xsp Properties page as above, and then visiting an XPage in it with a browser. If it's working, it will load normally; otherwise, it will complain about the application relying on a missing library.

So that covers the build and installation cycle! There's one last change to make in the projects before we commit them to Git. The update site project produces tons of Jar files - two for every "Build All" click - and there's no need to check them into the repository. This is a job for a .gitignore file, which we'll put in the update site project because we don't want to ignore intentional Jars later - .gitignore files cascade hierarchically. Right-click on the update site project in Eclipse and choose New → File:

Name the file ".gitignore" (with the starting dot) and click "Finish". The file contents should be a single line:

*.jar

Now, go to commit the changes to the repository, and you should be able to see the projects and Git ignore file we created, but not the Jars:

Now that we have the library built and deployed to the server, the next step will be to actually make it useful, by adding some Java code and accessing it from the XPages application.

That Java Thing, Part 5: Expanding the Plugin

Fri Nov 06 08:48:55 EST 2015

Tags: java xpages
  1. That Java Thing, Part 1: The Java Problem in the Community
  2. That Java Thing, Part 2: Intro to OSGi
  3. That Java Thing, Part 3: Eclipse Prep
  4. That Java Thing, Part 4: Creating the Plugin
  5. That Java Thing, Part 5: Expanding the Plugin
  6. That Java Thing, Part 6: Creating the Feature and Update Site
  7. That Java Thing, Part 7: Adding a Managed Bean to the Plugin
  8. That Java Thing, Part 8: Source Bundles
  9. That Java Thing, Part 9: Expanding the Plugin - Jars
  10. That Java Thing, Part 10: Expanding the Plugin - Serving Resources
  11. That Java Thing, Interlude: Effective Java
  12. That Java Thing, Part 11: Diagnostics
  13. That Java Thing, Part 12: Expanding the Plugin - JAX-RS
  14. That Java Thing, Part 13: Introduction to Maven
  15. That Java Thing, Part 14: Maven Environment Setup
  16. That Java Thing, Part 15: Converting the Projects
  17. That Java Thing, Part 16: Maven Fallout
  18. That Java Thing, Part 17: My Current XPages Plug-in Dev Environment

In the last post, we created an empty plugin project for our XPages library. Today, we'll flesh that plugin out a bit and add the actual XSP hooks.

The first step is to start filling in some details in the plugin's Manifest file, which is the core description of what the plugin does and what it requires in its environment. To begin with, open the MANIFEST.MF file in the META-INF folder of your project and check the "This plug-in is a singleton" checkbox:

During one of the steps later, Eclipse would have yelled at us to do that anyway. This checkbox means that the plugin expects to be the only one of its kind active on the server. This is because it will contribute to the XSP Library extension point, and it should prevent duplicate contributions of the same library from different versions.

Next, go to the Dependencies tab of the Manifest editor, click "Add..." in the "Required Plug-ins" section on the left, and add "com.ibm.xsp.core":

Do the same with "com.ibm.commons":

Now, save and close the Manifest. The next step is to fix up a minor problem I forgot about in the last post: the base package name. By default, the base package is named after the plugin project, which means it now contains a redundant "plugin" part. That's not a problem per se, but it's not needed, and this will be a good introduction to Eclipse refactoring.

Expand the "src/main/java" (or "src" if you left it as the default) package folder within the project, then right-click on "com.example.xsp.plugin" and choose "Refactor" → "Rename...". Change the name to "com.example.xsp", make sure "Update references" is checked, check "Update fully qualified names in non-Java text files", and click "Preview". This should list a couple changes - Eclipse looks for references to the class in both Java and non-Java files to try to cover all of the bases. In a larger project, there may still be lingering references elsewhere, but in this case it does everything for us. Click "OK".

It's possible that there may be a small detour at this point. On my machine, I noticed a strange problem in Eclipse after making this change: it set the output folder of the project to a nonsense directory within the source. To make sure this isn't happening, double-click on build.properties and go to the "build.properties" tab. Make sure the "output.." line reads output.. = target/classes (or output.. = bin if you left the defaults). This is the file that controls many of the compilation settings for the plugin, and this specifically specifies the location for the compiled Java classes.

Now, time to expand the Activator a bit. Expand "com.example.xsp" and open the "Activator.java" file. The Activator class is an optional but there-by-default class that provides a couple hooks during the lifecycle of the plugin. For XSP libraries, there's usually not too much going on here, but it can serve as a useful coordinating and debugging point.

One such potential use is a centralized logger configuration. At least for now, we'll just use the basic JVM logging classes, since "proper" OSGi logging is a whole thing. With the addition of a logging property and initialization message, the Activator looks like this:

package com.example.xsp;

import java.util.logging.Level;
import java.util.logging.Logger;

import org.osgi.framework.BundleActivator;
import org.osgi.framework.BundleContext;

public class Activator implements BundleActivator {

	private static BundleContext context;
	
	public static final Logger log = Logger.getLogger(Activator.class.getPackage().getName());
	static {
		log.setLevel(Level.FINEST);
	}

	static BundleContext getContext() {
		return context;
	}

	/*
	 * (non-Javadoc)
	 * @see org.osgi.framework.BundleActivator#start(org.osgi.framework.BundleContext)
	 */
	public void start(BundleContext bundleContext) throws Exception {
		Activator.context = bundleContext;
		
		if(log.isLoggable(Level.INFO)) {
			log.info("Starting Example XPages Library");
		}
	}

	/*
	 * (non-Javadoc)
	 * @see org.osgi.framework.BundleActivator#stop(org.osgi.framework.BundleContext)
	 */
	public void stop(BundleContext bundleContext) throws Exception {
		Activator.context = null;
	}

}

The static block that sets the log level is a good trick to know about: that block executes once, when the class is first loaded. It's distinct from a contructor, which turns each time a new object of the class is created. It may help to think of the static portions of a class (the properties, methods, and this block) as a sort of special singleton version of the class created by the runtime. This sort of thing is non-obvious to new Java developers, but it starts to make sense after you do it for a while.

With the Activator in place, the next step is the actual XPages library class, which provides the specific details about the plugin's XSP interactions. Right-click on the "com.example.xsp" package and choose "New" → "Class" (if "Class" doesn't appear in that list, choose "Other" and then find "Class" within the "Java" folder). In the "New Java Class" dialog, click "Browse..." in the "Superclass" line around the middle, and look for the class AbstractXspLibrary:

Set the class name to "Example Library" and leave everything else as the defaults:

When creating the class, it will fill in a single required method: getLibraryId(). This method's job is to return a string that should be unique across XPages libraries, and which will show up as the library's identifier in Designer. By convention, this is related to the Java reverse-DNS name you're using, plus a suffix like ".library":

@Override
public String getLibraryId() {
	return Activator.class.getPackage().getName() + ".library";
}

There are a number of other methods, though, that are worth overriding in a normal plugin, related to the plugin setup, the other XSP libraries it depends on, and its versioning. This is a reasonable baseline for a modern XPages library that will use the Extension Library:

package com.example.xsp;

import java.util.logging.Level;
import java.util.logging.Logger;

import com.ibm.xsp.library.AbstractXspLibrary;

public class ExampleLibrary extends AbstractXspLibrary {
	
	private static final Logger log = Activator.log;
	
	static {
		if(log.isLoggable(Level.FINE)) {
			log.fine(ExampleLibrary.class.getName() + " loaded");
		}
	}

	@Override
	public String getLibraryId() {
		return Activator.class.getPackage().getName() + ".library";
	}

	@Override
	public String getPluginId() {
		return Activator.getContext().getBundle().getSymbolicName();
	}
	
	@Override
	public String getTagVersion() {
		return "1.0.0";
	}
	
	@Override
	public String[] getDependencies() {
		return new String[] {
				"com.ibm.xsp.core.library",
				"com.ibm.xsp.extsn.library",
				"com.ibm.xsp.domino.library",
				"com.ibm.xsp.designer.library",
				"com.ibm.xsp.extlib.library"
		};
	}
	
	@Override
	public boolean isGlobalScope() {
		return false;
	}
}

With that class in place, there's one final step for making this plugin a proper XSP library: the extension point. Extension points are how the XPages runtime knows which plugins provide libraries like this. Open the MANIFEST.MF file again and click on "Extensions" on the right side of the first page:

Eclipse will ask you if you want to show the hidden Extensions panel, which you do. On that tab, click "Add..." and choose "com.ibm.commons.Extension":

Once added, there will be two fields on the right side. In "type", enter "com.ibm.xsp.Library". In "class", click "Browse" and search for the name of the library created earlier, "com.example.xsp.ExampleLibrary". Once this is added, you should be able to click on the "plugin.xml" tab in the editor and see something like this:

<?xml version="1.0" encoding="UTF-8"?>
<?eclipse version="3.4"?>
<plugin>
   <extension
         point="com.ibm.commons.Extension">
      <service
            class="com.example.xsp.ExampleLibrary"
            type="com.ibm.xsp.Library">
      </service>
   </extension>

</plugin>

There will also be a file named "plugin.xml" in your project.

After all that, we have a functional basis for our plugin. There are definitely a lot of things to remember in this process, but they start to make a sort of sense the more you do it.

With that set, commit your changes, where you can see the moved Activator file and the addition of the Library stuff:

In the next post, we'll create the feature and update-site projects needed to actually install this plugin in Designer and Domino.

That Java Thing, Part 4: Creating the Plugin

Thu Nov 05 09:38:56 EST 2015

Tags: java xpages
  1. That Java Thing, Part 1: The Java Problem in the Community
  2. That Java Thing, Part 2: Intro to OSGi
  3. That Java Thing, Part 3: Eclipse Prep
  4. That Java Thing, Part 4: Creating the Plugin
  5. That Java Thing, Part 5: Expanding the Plugin
  6. That Java Thing, Part 6: Creating the Feature and Update Site
  7. That Java Thing, Part 7: Adding a Managed Bean to the Plugin
  8. That Java Thing, Part 8: Source Bundles
  9. That Java Thing, Part 9: Expanding the Plugin - Jars
  10. That Java Thing, Part 10: Expanding the Plugin - Serving Resources
  11. That Java Thing, Interlude: Effective Java
  12. That Java Thing, Part 11: Diagnostics
  13. That Java Thing, Part 12: Expanding the Plugin - JAX-RS
  14. That Java Thing, Part 13: Introduction to Maven
  15. That Java Thing, Part 14: Maven Environment Setup
  16. That Java Thing, Part 15: Converting the Projects
  17. That Java Thing, Part 16: Maven Fallout
  18. That Java Thing, Part 17: My Current XPages Plug-in Dev Environment

To make a basic XPages library, we'll need to create the trio of OSGi projects: the plugin, the feature, and the update site. For a long time, the XSP Starter Kit has been a great go-to starting point for this sort of thing. It definitely covers almost all of the potential ground, but it can be a bit overkill when you just want to put some classes in a shared place. So, for this exercise, we'll start from scratch.

But before we do that, we should create a local Git repository first. This isn't required, but it's a good idea, and Git repositories are so "cheap", technically, that there's no reason not to. There are a number of ways to do it - you can create and manage them via the command line, via a dedicated tool like SourceTree, or via the embedded Git client in Eclipse. We'll do the last one here.

To work with Git repositories, first add the Git Repositories view (Eclipse's "view" refers to the panes you see in the window) to your Eclipse UI by going to Windows → Show View → Other...:

Once you add it, you can click on "Create a new local Git repository" in the pane - I suggest creating a folder beneath the "git" folder in your home directory, for organizational purposes:

Now, on to creating an actual project. To do that, go to File → New → Project... and find "Plug-in Project" inside "Plug-in Development":

In the form that shows up after you hit Next, fill in some project details:

  • Set the project name to "com.example.xsp.plugin"
  • Override the location with a folder inside the Git repository you created earlier (make sure to include the plugin name in the path, rather than using the top level of the Git repo)
  • Change the source folder to "src/main/java" and output folder to "target/classes". These are nods towards Maven that aren't strictly required, but are a "may as well" thing.
  • Set the target platform to "an OSGi framework" → "Equinox"

On the next page, the only change needed is to set the Execution Environment to "JavaSE-1.6" (at least until Domino gets a newer JVM):

Then, click "Finish". It will ask you if you want to switch to the "Plug-in Development" perspective - "perspectives" are an Eclipse term for groupings+layouts of views for different purposes. You can choose either Yes or No, since the other perspective is very similar to the default J2EE perspective. If you choose "Yes", you'll have to re-add the Git Repositories view as above.

Once you've created the project, it's time to check it in to Git. Right-click on the Git repository in the Git Repositories view and choose "Commit...". The first time you do this, it will prompt you for a name and email address, which are pretty arbitrary, but it's good to keep them consistent across your Git presences. On the commit page, provide a useful message and click the "Select All" button (the middle one in the top right of the "Files" section) to include all of the newly-created files, and hit "Commit":

Now that the plugin's skeleton is in place, the next post will cover some details about it and how to turn it into an XSP Library.

That Java Thing, Part 3: Eclipse Prep

Wed Nov 04 04:26:58 EST 2015

Tags: java xpages
  1. That Java Thing, Part 1: The Java Problem in the Community
  2. That Java Thing, Part 2: Intro to OSGi
  3. That Java Thing, Part 3: Eclipse Prep
  4. That Java Thing, Part 4: Creating the Plugin
  5. That Java Thing, Part 5: Expanding the Plugin
  6. That Java Thing, Part 6: Creating the Feature and Update Site
  7. That Java Thing, Part 7: Adding a Managed Bean to the Plugin
  8. That Java Thing, Part 8: Source Bundles
  9. That Java Thing, Part 9: Expanding the Plugin - Jars
  10. That Java Thing, Part 10: Expanding the Plugin - Serving Resources
  11. That Java Thing, Interlude: Effective Java
  12. That Java Thing, Part 11: Diagnostics
  13. That Java Thing, Part 12: Expanding the Plugin - JAX-RS
  14. That Java Thing, Part 13: Introduction to Maven
  15. That Java Thing, Part 14: Maven Environment Setup
  16. That Java Thing, Part 15: Converting the Projects
  17. That Java Thing, Part 16: Maven Fallout
  18. That Java Thing, Part 17: My Current XPages Plug-in Dev Environment

Before you dive into OSGi development, you'll need a proper development environment, and that usually means Eclipse. Domino Designer, being Eclipse-based, can technically do the job and has the advantage of some of the setup being done, but it's so out-of-date at this point that it's not worth it (believe me, I tried originally). Newer Eclipse versions have improved noticeably, so it's best to grab a recent version. Currently, that means Mars, or Eclipse 4.5.1.

Before Eclipse, you'll need to install Java if you haven't already. To do that, go to Oracle's site and download the latest release for your platform. Once you install it and find out how many devices run Java, you can move on to installing Eclipse.

Head to the Eclipse download page and download the "Eclipse IDE for Java EE Developers" for your platform. They've recently added an Installer at the top of the page, and that works as well, but adds some new wrinkles that haven't made it into common knowledge yet. For now, you may want to stick with the "...or download an Eclipse Package" section. The "bitness" (32 or 64) doesn't matter too much - you should generally match the bitness of your OS.

Once you have Eclipse installed and running ("installation" generally involves dragging the extracted ZIP file content somewhere), the next step is to teach Eclipse about the XPages artifacts. If you're running on Windows and have a local Domino server, the XPages SDK may be your friend, though it hasn't been fully tested for Mars. If you are running on a non-Windows OS or otherwise don't want to go the full SDK route, there's a quick route that lacks the automation and debugger integration.

Setting up the environment used to be something of a hassle, but it became much simpler ever since IBM released the Update Site for Build Management. Primarily made for Maven-development use (sort of), this Update Site also serves as a quick stop to pick up all of the dependencies you need for XPages development in one place.

To make use of the Update Site, download the file from OpenNTF, extract it, and then extract the UpdateSite.zip file contained therein. You should end up with a folder like this:

In Eclipse, go to the preferences ("Window" → "Preferences" on Windows or "Eclipse" → "Preferences" on a Mac), then go to "Plug-in Development" → "Target Platform":

Edit the existing platform and, in the "Locations" tab of the ensuing dialog, click "Add..." → "Directory" → "Next", browse to that extracted UpdateSite folder, and click "Next" to verify that it finds a bunch of plugins, and then "Finish". Now, you'll be able to reference XPages platform plugins in your projects.

At this point, Eclipse is set up for the essentials, but it may still be a good idea to set up an appropriate JVM for Domino development to make sure you have a consistent baseline and to avoid problems where newer Java releases added classes methods not avilable in Java 6. Fortunately, on Windows (and Linux?), a Notes installation comes with a JVM that does the job nicely. To add it, go back to preferences, then "Java" → "Installed JREs" → "Add...". Choose "Standard VM" → "Next", browse to the "jvm" folder in your Notes (or Domino) installation directory, name it something appropriate, and then click Finish:

On the Mac, you can get a similar Java release from Apple, which nowadays will end up in /Library/Java/JavaVirtualMachines (older installers put it in /System/Library/Java/JavaVirtualMachines).

Now that Eclipse is set up, the next step will be to break ground on making an actual XPages plugin.

That Java Thing, Part 2: Intro to OSGi

Tue Nov 03 06:44:39 EST 2015

Tags: java xpages
  1. That Java Thing, Part 1: The Java Problem in the Community
  2. That Java Thing, Part 2: Intro to OSGi
  3. That Java Thing, Part 3: Eclipse Prep
  4. That Java Thing, Part 4: Creating the Plugin
  5. That Java Thing, Part 5: Expanding the Plugin
  6. That Java Thing, Part 6: Creating the Feature and Update Site
  7. That Java Thing, Part 7: Adding a Managed Bean to the Plugin
  8. That Java Thing, Part 8: Source Bundles
  9. That Java Thing, Part 9: Expanding the Plugin - Jars
  10. That Java Thing, Part 10: Expanding the Plugin - Serving Resources
  11. That Java Thing, Interlude: Effective Java
  12. That Java Thing, Part 11: Diagnostics
  13. That Java Thing, Part 12: Expanding the Plugin - JAX-RS
  14. That Java Thing, Part 13: Introduction to Maven
  15. That Java Thing, Part 14: Maven Environment Setup
  16. That Java Thing, Part 15: Converting the Projects
  17. That Java Thing, Part 16: Maven Fallout
  18. That Java Thing, Part 17: My Current XPages Plug-in Dev Environment

OSGi once stood for "Open Services Gateway initiative", but that name slid from "impossibly vague" to "entirely obsolete" rather quickly. For our needs, OSGi is a mechanism for bringing sanity to the "big pile of Jars" that you might otherwise have in a large Java system. It provides a standardized way to describe the name of a library, its version, its dependencies, its capabilities, and its interactions with other libraries. In this way, rather than just "commons-io-2.1.jar", you can conceptually deal with "I require org.apache.commons.io, version 2.0 or above" and the environment can suss out what that means based on what is installed.

As Domino developers, we care about a subset of the OSGi concepts, primarily those related to creating and loading plugins. There is a handful of vital terms involved, some of which I've already been tossing around:

  • JAR: a Java Archive file is a ZIP file aimed at Java, generally containing classes and other resources (text files, images, and so forth). There are variants like "War" (Web Application Archive) and "Ear" (Enterprise Archive), but they're all still ZIP files.
  • Bundle: A "bundle" is, in effect, OSGi's word for a JAR. It expands on a contained file called META-INF/MANIFEST.MF (so named because Java likes to be difficult sometimes) to define the description of the bundle: its name, version, and so forth.
  • Plug-in: This is basically the same thing as a bundle. Presumably, it technically refers to a specialized kind of bundle, but we can use the terms interchangeably. It also technically has that hyphen in it, but I often write it "plugin" anyway.
  • Feature: This excessively-vague term is OSGi's way of grouping plugins together into a single conceptual installable unit. For example, the main feature included in the Extension Library references ten distinct plugins.
  • Update Site: Update sites are to features what features are to plugins: a way to organize these features into a collection, either to install them all at once or to provide a pool of available software to install. Eclipse-the-IDE uses them extensively for the latter case. As Domino developers, we primarily use them to distribute libraries and install them into the server's Update Site NSF.
  • Target Platform: A target platform refers to the set of plugins available in a given environment, which is used to determine whether a given plugin has the dependencies it will need to run. In our case, the target platform is usually the XPages runtime environment, containing OSGi base plugins, Java servlet support plugins, and the XPages runtime plugins themselves (among others). Plugin development environments use these heavily.
  • Extension Point: An extension point is a way for a plugin to either declare that it can either provide or consume a given service. For example, you might have a plugin that says "I can make use of any plugin that provides a color" and then another that says "I can provide the color blue". By using extension points, the first plugin can find the other, even though it may have been developed entirely independently.
  • Java EE: Java Enterprise Edition (sometimes written J2EE thanks to Java's weird relationship with versions) is basically the standard "web server stuff" for Domino (though it covers other aspects). XPages is sort of like a Java EE environment, but the experience is distinct. Notably, a Java EE server is a general platform on which other web-dev toolkits - JSP, JSF, Vaadin, JAX-RS, and so forth - can run. Java EE isn't inherently related to OSGi, but the two eventually bleed into each other with Domino and WebSphere.

OSGi in general goes beyond that, and some other aspects may be useful down the line (such as interacting with the OSGi console), but for now it's enough to think of it as a set of decorations for your projects. With some of the basic terms in line, next I will move on to setting up an Eclipse environment to start development.

That Java Thing, Part 1: The Java Problem in the Community

Mon Nov 02 13:20:23 EST 2015

Tags: java xpages
  1. That Java Thing, Part 1: The Java Problem in the Community
  2. That Java Thing, Part 2: Intro to OSGi
  3. That Java Thing, Part 3: Eclipse Prep
  4. That Java Thing, Part 4: Creating the Plugin
  5. That Java Thing, Part 5: Expanding the Plugin
  6. That Java Thing, Part 6: Creating the Feature and Update Site
  7. That Java Thing, Part 7: Adding a Managed Bean to the Plugin
  8. That Java Thing, Part 8: Source Bundles
  9. That Java Thing, Part 9: Expanding the Plugin - Jars
  10. That Java Thing, Part 10: Expanding the Plugin - Serving Resources
  11. That Java Thing, Interlude: Effective Java
  12. That Java Thing, Part 11: Diagnostics
  13. That Java Thing, Part 12: Expanding the Plugin - JAX-RS
  14. That Java Thing, Part 13: Introduction to Maven
  15. That Java Thing, Part 14: Maven Environment Setup
  16. That Java Thing, Part 15: Converting the Projects
  17. That Java Thing, Part 16: Maven Fallout
  18. That Java Thing, Part 17: My Current XPages Plug-in Dev Environment

Java has been a bugbear for the Domino community for a long time. Though Notes and Domino have presented a top-to-bottom (mostly) Java environment for years, the monumental inertia of the corporate development community, the platform's tortured history of insufficiently-fleshed-out Java hooks, and IBM's "pay no attention to that man behind the curtain" pitch with regard to the language created an environment where Java is often something for "other developers".

XPages represented a tremendous leap forward for development on the platform, ditching the completely-unique bizarre forms-based web-dev experience in favor of something much closer to modern web development. At the time, the unstructured, SSJS-heavy façade presented by Designer made a sort of sense: the switch to XPages was already a huge change for Domino developers, and making XPage development look sort of like classic Notes/Domino development was a spoonful of sugar with the medicine.

But the critical flaw of XPages development has never been fixed: there's no smooth path from "hello, world" to a well-structured complex application. SSJS, as implemented in XPages, is unsuitable to the task, while Designer's presentation of the Java layer ranges from "non-obvious" to "hostile". Still, IBM did a good job of presenting the next major tier for XPages developers: writing Java first in the NSF and then in OSGi plugins. The problem has always been in figuring out how to get there.

This hurdle started out as an inconvenience just for those who wanted to figure out how the machine worked, but has grown into a significant problem for all Domino developers. While the XPages stack has remained relatively stagnant, the rest of the development world has raced forward, with new technologies giving rise to new frameworks and transforming the old. This is no discredit to the XPages team: they have consistently put in tremendous work across the board, but it's quite an industry to keep up with.

So we're in a tough spot now. The XPages platform isn't going away any time soon, but it's important for us as developers to progress. One option is to abandon ship entirely: pack up and move to some unrelated platform, leaving Domino behind entirely. But most of us, out of personal affection, professional acumen, or (primarily) business requirements, don't want to do that. Unfortunately, the path to improvement with Domino has only gotten more complicated with time. It began with learning about Java, OSGi, and Eclipse proper, but has since expanded to include a whole rogues gallery of other technologies: Maven, Tycho, m2e, Jenkins, Git, Wink/JAX-RS, jQuery, Bootstrap, Angular, and on and on.

There's no time like the present, though. I've walked this path, and I want to help everyone else walk it too. To kick that off, this series is going to cover the process of creating a basic XPages plugin, making it a little more complex, converting it to Maven, and building it with Jenkins.

To kick things off, the next post will provide an introduction to the concept of OSGi and the parts of it that we need to know for Domino development. Things may seem weird along the way, but trust me: it's worthwhile.

XPages Devs: Enable "Refresh entire application when design changes"

Mon Sep 14 11:48:46 EDT 2015

Tags: xpages java

When developing an XPages application of beyond-minimal complexity, you're likely to run into a problem where your app starts saying that a class is incompatible with itself in one way or another. The exception usually traces down to something like "foo.SomeClass is incompatible with foo.SomeClass" or "cannot assign instance of foo.SomeClass to field X..." where the field is that same class. This has cropped up since time immemorial.

It's actually, though, something that IBM sort-of fixed in 8.5.3 by adding an xsp.properties option of xsp.application.forcefullrefresh=true and then, in 9.0, a GUI option in Xsp Properties:

Basically, this checkbox amounts to "don't break my app periodically". From what I gather, the default behavior is in the interest of being clever with classloaders, but can lead to creeping problems in complicated apps, to the point where changing seemingly-innocuous things like the ACL breaks the app until you restart HTTP or "kick" the app by modifying certain design elements (namely, Java classes or faces-config.xml). Since that behavior is never desired, there is no reason to not check this box, and I enable it on every new NSF I create.

Adding Components to an XPage Programmatically

Sun Jul 19 09:16:35 EDT 2015

Tags: xpages java

One of my favorite aspects of working with apps using my framework is the component binding capability. This lets me just write the main structure of the page and let the controller do the grunt work of creating fields with validators and converters. There's a lot of magic behind the scenes to make it happen, but the core concept of dynamic component creation is relatively straightforward.

An XPage is a tree of components, and those components are all Java objects on the back end, which can be manipulated and added or removed programmatically. To demonstrate, I'll start with this basic XPage:

<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core" beforePageLoad="#{controller.beforePageLoad}" afterPageLoad="#{controller.afterPageLoad}">
	<xp:div id="container">
	</xp:div>
</xp:view>

Now, I'll add a basic form table using the afterPageLoad method in the controller class:

package controller;

import javax.faces.component.UIComponent;
import javax.faces.context.FacesContext;

import com.ibm.xsp.component.UIViewRootEx2;
import com.ibm.xsp.component.xp.XspInputText;
import com.ibm.xsp.extlib.component.data.UIFormLayoutRow;
import com.ibm.xsp.extlib.component.data.UIFormTable;
import com.ibm.xsp.extlib.util.ExtLibUtil;
import com.ibm.xsp.util.FacesUtil;

import frostillicus.xsp.controller.BasicXPageController;

public class home extends BasicXPageController {
	private static final long serialVersionUID = 1L;

	@SuppressWarnings("unchecked")
	@Override
	public void afterPageLoad() throws Exception {
		super.afterPageLoad();

		UIViewRootEx2 view = (UIViewRootEx2)ExtLibUtil.resolveVariable(FacesContext.getCurrentInstance(), "view");
		UIComponent container = FacesUtil.findChildComponent(view, "container");

		UIFormTable formTable = new UIFormTable();
		formTable.setFormTitle("Some Form");
		formTable.setStyle("margin: 2em; width: 20em");
		container.getChildren().add(formTable);
		formTable.setParent(container);

		UIFormLayoutRow formRow = new UIFormLayoutRow();
		formRow.setLabel("Name");
		formTable.getChildren().add(formRow);
		formRow.setParent(formTable);

		XspInputText inputText = new XspInputText();
		formRow.getChildren().add(inputText);
		inputText.setParent(formRow);
	}
}

There are a few concepts to get a handle on here, but fortunately they're not as esoteric as other aspects of back-end XPages development.

To start out with, there's the question of how you're supposed to know what classes the components are. The best way to find this out is to create a basic XPage containing the control with an ID and then go to the Package Explorer view, then the "Local" source folder, and find the class file for your page in the "xsp" package. In there, you can see the code that actually generates the XPage (which is doing basically the same thing as we're doing here). Look for the ID you gave the control on the page and you can find the class behind it.

Next is the job of finding the parent control you're going to attach the new components to. In this case, I used the FacesUtil class to search for the component by its ID, because I know it's the only one on the page with that base ID. This tack will usually do the trick for you, but there are other ways to find it, such as binding or XspQuery.

Finally, there's the need to both add the new child to the parent's children list and to set the parent in the child itself. This is a bit of housekeeping boilerplate, but it has to be done.

Once you have this code running, you get a basic form (using the Bootstrap3.2.0 theme in this case):

Some Form

This can get much more in-depth: anything you can do declaratively in the XPage XML, you can do programmatically in Java. You can also manipulate existing components: change their properties, rearrange them, or remove them from the tree entirely. I've found knowledge of this to be both useful as a part of the toolbox as well as a way to really clear up the mental model of what's going on on the page.

Quick Tip: Override Form-XPage Mapping in xsp.properties

Thu Jun 11 14:17:19 EDT 2015

Tags: xpages

This is sort of an esoteric feature, but I just ran across it and it fits nicely into an XPages bag of tricks.

Specifically, as it turns out, there's a way to override the default "which XPage should be used for this document?" behavior in some cases. The most-commonly-known behavior is the "Display XPage Instead" set on the form (or the default form if-and-only-if the document has an empty Form field). This wins in all cases, and is the only way to get an XPage to open for a document when using non-XPage-specific URLs, such as traditional-and-clean-ish "db.nsf/viewname/key" URLs.

There's also a secondary behavior that I knew about before, which is that, if it can't find an explicit XPage for the form, it looks for an XPage that matches the form name. So, for example, "SomeForm" becomes "SomeForm.xsp" (case-sensitive, except the first letter). I used this once upon a time to be able to have XPages for documents stored in remote databases without changing the design of those databases (you can also specify an $XPageAlt field on the remote form without the XPage existing in that DB, by the way).

What I learned today is that there's a way to short-circuit that fallback and specify an alternative via xsp.properties. Specifically, you can add a line that looks like this:

xsp.domino.form.xpage.testform=OtherForm

Note that there are a few notes here:

  • This only works for "$$OpenDominoDocument.xsp" URLs, because the XSP runtime is not consulted for "view/key" URLs unless the form explicitly names an XPage.
  • This does not override an explicitly-named XPage in the form.
  • This does work when no form with the specified name actually exists in the database.
  • The form name must be lowercase in the property.
  • The form name must also be properly escaped as per the rules for properties files; importantly, this means that spaces in the form name must be preceded by a backslash.
  • This does work to specify an XPage for the default form when no form is specified in the document.

Most of the time, this trick won't be needed, since either specifying an XPage name in the URL or designating it in the form note will suffice. However, if you're in a case where you have an app that points to data in remote databases and you don't want to modify the design there but still want to use "$$OpenDominoDocument" URLs, this is an option.

Quick Tip: Wrapping a lotus.domino.Document in a DominoDocument

Tue Mar 03 15:13:13 EST 2015

Tags: xpages java

In the XPages environment, there are two kinds of documents: the standard Document class you get from normal Domino API operations and DominoDocument (styled NotesXspDocument in SSJS). Many times, these can be treated roughly equivalently: DominoDocument has some (but not all) of the same methods that you're used to from Document, such as getItemValueString and replaceItemValue, and you can always get the inner Document object with getDocument(boolean applyChanges).

What's less common, though, is going the other direction: taking a Document object from some source (say, database.getDocumentByUNID) and wrapping it in a DominoDocument. This can be very useful, though, as the API on the latter is generally nicer, containing improved support for dealing with attachments, rich text and MIME, and even little-used stuff like getting JSON from an item directly. Take a gander at the API: com.ibm.xsp.model.domino.wrapped.DominoDocument.

It's non-obvious, though, how to make use of it from the Java side. For that, there are a couple wrap static methods at the bottom of the list. The methods are overflowing with parameters, but they align with many of the attributes used when specifying an <xp:dominoDocument/> data source and are mostly fine to leave as blanks. So if you have a given Document, you can wrap it like so:

Document notesDoc = database.getDocumentByUNID(someUNID);
DominoDocument domDoc = DominoDocument.wrap(
	database.getFilePath(), // databaseName
	notesDoc,               // Document
	null,                   // computeWithForm
	null,                   // concurrencyMode
	false,                  // allowDeletedDocs
	null,                   // saveLinksAs
	null                    // webQuerySaveAgent
);

Once you do that, you can make use of the improved API. You can also serialize these objects, though I believe you have to call restoreWrappedDocument on it after deserialization to use it again. I do this relatively frequently when I'm in a non-ODA project and want some nicer methods and better data-type support.

Factories in XPages

Wed Nov 12 18:27:14 EST 2014

Tags: java xpages

In my last post, I intimated that I wanted to write a post covering the concept of factories in XPages, or at least the specific kind in question. This is that post.

The term "factory" covers a number of meanings, and objects named this way crop up all over the place (ODA has one, for example). The kind I care about today are those defined in an XspContributor object in an XPages plugin. These factories are generally (exclusively, perhaps) used for the purpose of generating adapters: assistant objects that allow the framework to perform operations on object types that may have no knowledge of the framework at all.

The way this generally takes form is this: when the framework needs to perform a specialized task, it asks the application (that is to say, the Application object that controls the XPages app) for a list of factories it knows about that conform to a given interface:

FacesContext facesContext = FacesContext.getCurrentInstance();
ApplicationEx app = (ApplicationEx)facesContext.getApplication();
List<ComponentMapAdapterFactory> factories = app.findServices(COMPONENT_MAP_SERVICE_NAME);

Once it has this list, it loops through all of them and passes the object it's trying to adapt. If the factory is written to understand that type of object, it will return an adapter object; if not, it will return null:

ComponentMapAdapter adapter = null;
for(ComponentMapAdapterFactory fac : factories) {
	adapter = fac.createAdapter(object_);
	if(adapter != null) {
		break;
	}
}

The framework can then use that adapter to perform the action on the object, without either the framework or the object knowing anything about the other:

for(String propertyName : adapter.getPropertyNames()) {
	// ...
}

XSP internally uses this for a great many things. One use that I've run into (and butted heads with) is to create adapter objects for dealing with file attachments. The DominoDocument class has a bit of information about attachments, particularly in its AttachmentValueHolder inner class, but doesn't on its own handle the full job of dealing with file upload and download controls. During the processes of getting a list of files for a given field from the document and handling the "delete document" method, the XPages framework looks up appropriate factories to handle the data type.

The reason for this indirection is so that this sort of operation can work with arbitrary data: in an ideal world, the XPages framework doesn't care at all that anything it does is related to Domino, and similarly the Domino data model objects don't care at all that they're embedded in the XSP framework. By allowing the model-object author (IBM, in this case) to say "hey, when you want to get a list of file attachments for an object of this type, I can help", it decouples the operation cleanly (it's broken in this case, but the theory applies). When the framework needs to process an object, it loops through the adapter factory classes, asks each one if it can handle the object in question, and takes the adapter from the first one that returns non-null.

At first blush, this setup seems overly contrived. After all, isn't this what interfaces are for? Well, in many cases, yes, interfaces would do the job. However, sometimes it makes sense to add this extra layer of indirection. Say, for example, that you're adapting between two Java classes you don't control: you can't modify the framework to support a class you want, and you can't modify the class to conform to an interface the framework understands. In that case, an adapter factory is a perfect shim.

But in fact, I've even found it useful to adopt this structure when I control all of the code. When I was designing the model/component adapter in the frostillic.us Framework, I made the conscious decision to not tie the two sides (the controller and the model) together tightly. Instead, I wrote a pair of interfaces in the controller package: ComponentMapAdapterFactory and ComponentMapAdapter. This way, when the controller gets the order to create an input field for an object's "firstName" property, it loops through the list of ComponentMapAdapterFactorys to find one that fits. Over in the model package, I have an appropriate factory and adapter to handle my model framework.

I could have combined these two more tightly, but I enjoy the cleanliness this brings. I may not stick with my same model framework forever, and similarly the expectations of the controller class may change; because of this separation, it's clear where tweaks will need to be made. It also gives me the freedom to use the same component-building code with model objects I don't control, such as the adapter I wrote that pulls its configuration from Notes forms.


These types of factories are not something you're likely to run into during normal XPages development, but it may be useful to bear their existence in mind. The XPages framework is a great morass of moving parts, and so being able to chart the inner workings in your mental map one bit at a time can go a long way to mastering the platform.

Property Resolution in XPages EL

Tue Nov 11 11:24:21 EST 2014

Tags: xpages java

Reading Devin Olson's recent series on EL processing put me in the mood to refresh and fill out my knowledge of how that stuff actually happens.

A while back, I made a small foray of my own into explaining how property resolution in XPages EL works, one which I followed up with a mea culpa explaining that I had left out a few additional supported types. As happens frequently, that still didn't cover the full story. Before getting to what I mean by that, I'll step back to an overview of XPages EL in general.

Components of EL Processing

To my knowledge, there are three main conceptual pieces to the EL-resolution process. I'll use the EL #{foo.bar.baz[1]} as a common example.

  • The EL parser itself. This is what reads the EL above and determines that you want the 1-indexed property of the baz property of the bar property of the foo object. I don't know if there's a realistic way to override or extend this stock-EL behavior.
    • This does, though contain an extensible side path: BindingFactory. This lets you create your own processors for value and method bindings based on the EL prefix, in the same vein as #{javascript: ... }.
  • The VariableResolver. This is a relatively-common bit of XPages extensibility and for good reason: they're quite useful. The variable resolver is what is used by EL (and SSJS, and others) to determine what object is referenced by foo in the example.
  • The PropertyResolver. This is the companion to the VariableResolver and is what handles the rest of the dereferencing in the example. EL asks the app's property resolver to find the bar property of foo, then the baz property of that, and then the 1 indexed property of that. This is the main topic of conversation today.

Setting An Application's PropertyResolver

There are two main ways I know of to make use of property resolvers, and the first is analogous to the VariableResolver: you write a single object and specify it in your faces-config file, like so:

<application>
	<property-resolver>config.PropResolver</property-resolver>
</application>

Then the skeletal implementation of such an object looks like this:

package config;

import javax.faces.el.EvaluationException;
import javax.faces.el.PropertyNotFoundException;
import javax.faces.el.PropertyResolver;

public class PropResolver extends PropertyResolver {
	private final PropertyResolver delegate_;

	public PropResolver(PropertyResolver delegate) {
		delegate_ = delegate;
	}

	@Override
	public Class<?> getType(Object obj, Object property) throws EvaluationException, PropertyNotFoundException {
		return delegate_.getType(obj, property);
	}

	@Override
	public Class<?> getType(Object obj, int index) throws EvaluationException, PropertyNotFoundException {
		return delegate_.getType(obj, index);
	}

	@Override
	public Object getValue(Object obj, Object property) throws EvaluationException, PropertyNotFoundException {
		return delegate_.getValue(obj, property);
	}

	@Override
	public Object getValue(Object obj, int index) throws EvaluationException, PropertyNotFoundException {
		return delegate_.getValue(obj, index);
	}

	@Override
	public boolean isReadOnly(Object obj, Object property) throws EvaluationException, PropertyNotFoundException {
		return delegate_.isReadOnly(obj, property);
	}

	@Override
	public boolean isReadOnly(Object obj, int index) throws EvaluationException, PropertyNotFoundException {
		return delegate_.isReadOnly(obj, index);
	}

	@Override
	public void setValue(Object obj, Object property, Object value) throws EvaluationException, PropertyNotFoundException {
		delegate_.setValue(obj, property, value);
	}

	@Override
	public void setValue(Object obj, int index, Object value) throws EvaluationException, PropertyNotFoundException {
		delegate_.setValue(obj, index, value);
	}
}

If you've done much with DataObjects, you'll likely immediately recognize those methods: I imagine that DataObject was created as a simplest-possible implementation of a PropertyResolver-friendly interface.

So how might you use this? Well, most of the time, you probably shouldn't - in my experience, the standard property resolver is sufficient and using DataObject in custom objects is easy enough that it's the best path. Still, you could use this to patch the behavior that is driving Devin to madness or to paint over other persistent annoyances. For example, DominoViewEntry contains hard-coded properties for accessing the entry's Note ID, but not its cluster-friendly Universal ID. To fix this, you could override the non-indexed getValue method like so:

public Object getValue(Object obj, Object property) throws EvaluationException, PropertyNotFoundException {
	if (obj instanceof DominoViewEntry && "documentId".equals(property)) {
		return ((DominoViewEntry) obj).getUniversalID();
	}
	return delegate_.getValue(obj, property);
}

Now, anywhere where you have a DominoViewEntry, you can use #{viewEntry.documentId} to get the UNID. You could do the same for DominoDocument as well, if you were so inclined. You'll just have to plan to never have a column or field named "documentId" (much like you currently have to avoid "openPageURL", "columnIndentLevel", "childCount", "noteID", "selected", "responseLevel", "responseCount", and "id").

Property Resolver Factories

The other way to use PropertyResolvers is to register and use a PropertyResolverFactory. Unlike the faces-config approach, these do not override (all of) the default behavior, but are instead looked up by IBM's PropertyResolver implementation at a point during its attempts at property resolution. Specifically, that point is after support for ResourceBundle, ViewRowData, and DataObject and before delegation to Sun's stock resolver (which handles Maps, Lists, arrays, and generic POJOs).

If you get a type hierarchy, you can see that IBM uses this route for lotus.domino.Document, com.ibm.commons.util.io.json.JsonObject, and com.ibm.jscript.types.FBSObject (SSJS object) support. So the idea of this route is that you'd have your own custom object type which doesn't implement any of the aforementioned interfaces and for which you want to provide EL support beyond the normal getter/setter support. Normally, this is not something worth doing, but I could see it being useful if you have a third-party class you want to work in, such as a non-Domino/JDBC data source.

The method for actually using one of these is... counterintuitive, but is something you may have run into in plugin development. The first step is simple enough: implement the factory:

package config;

import javax.faces.el.PropertyResolver;
import com.ibm.xsp.el.PropertyResolverFactory;

public class PropResolverFactory implements PropertyResolverFactory {
	public PropertyResolver getPropertyResolver(Object obj) {
		return null;
	}
}

That getPropertyResolver method's job is to check to see if the object is one of the types it supports and, if it is, return a PropertyResolver object (the same kind as above) that will allow the primary resolver to get the property.

Actually registering the factory is weirder. It must be done via a plugin (or by code that manually registers it in the application when needed), and the best way to see what I mean is to take a look at an example: the OpenntfDominoXspContributor class used by the OpenNTF Domino API. The contributor is registered in the plugin.xml and returns an array of arrays (because Java doesn't have tuples or map literals) representing a unique name for your factory plus the implementing class.

This concept of factories actually probably warrants its own blog post down the line. For the time being, the upshot is that this approach is appropriate if you're adding your own data type via a plugin (and which wouldn't be better-suited to implement DataObject), so it's a rare use case indeed.

Using "Verboten" Property Names in Custom Controls

Sun Nov 02 09:46:28 EST 2014

Tags: xpages

In an attempt to save you from yourself, Designer prevents you from naming your custom control properties after SSJS keywords such as "do" and "for". This is presumably because a construct like compositeData.for would throw both a syntax error in SSJS and the developer into a tizzy. However, sometimes you want to use one of those names - they're not illegal in EL, after all, and even SSJS could still use compositeData['for'] or compositeData.get("for") to access the value.

Fortunately, this is possible: if you go to the Package Explorer view in Designer and open up the "CustomControls" folder of your NSF, you'll see each custom control as a pair of files: an ".xsp" file representing the control markup and an ".xsp-config" file representing the metadata specified in the properties pane, including the custom properties. Assuming you attempted to type "for" for the property name and were stuck with "fo", you'll see a block like this:

<property>
	<property-name>fo</property-name>
	<property-class>string</property-class>
</property>

Change that "fo" to "for" and save and all is well. You'll be able to use the property just like you'd expect with a normal property, with the caveat above about how to access it if you use SSJS. I wouldn't make a habit of using certain keywords, such as "class", but "for" is perfectly fine and allows your controls to match stock controls such as xp:pager.

This came up for me in one of the controls I like to keep around when dealing with custom renderers: a rendererInfo control to display some relevant information. Since I keep forgetting where I last used such a control, I figure I should post it here partially for my own future reference.

<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core">
	<table>
		<tr>
			<th>Client ID</th>
			<td><xp:text><xp:this.value><![CDATA[#{javascript:
				var comp = getComponent(compositeData['for']);
				return comp == null ? 'null' : comp.getClientId(facesContext);
			}]]></xp:this.value></xp:text></td>
		</tr>
		<tr>
			<th>Theme Family</th>
			<td><xp:text><xp:this.value><![CDATA[#{javascript:
				var comp = getComponent(compositeData['for']);
				return comp == null ? 'null' : comp.getStyleKitFamily();
			}]]></xp:this.value></xp:text></td>
		</tr>
		<tr>
			<th>Component Family</th>
			<td><xp:text><xp:this.value><![CDATA[#{javascript:
				var comp = getComponent(compositeData['for']);
				return comp == null ? 'null' : comp.getFamily();
			}]]></xp:this.value></xp:text></td>
		</tr>
		<tr>
			<th>Renderer Type</th>
			<td><xp:text><xp:this.value><![CDATA[#{javascript:
				var comp = getComponent(compositeData['for']);
				return comp == null ? 'null' : comp.getRendererType();
			}]]></xp:this.value></xp:text></td>
		</tr>
		<tr>
			<th>Renderer Class</th>
			<td><xp:text><xp:this.value><![CDATA[#{javascript:
				var comp = getComponent(compositeData['for']);
				var renderer = comp == null ? null : comp.getRenderer(facesContext);
				return renderer != null ? renderer.getWrapped().getClass().getName() : 'N/A'
			}]]></xp:this.value></xp:text></td>
		</tr>
	</table>
</xp:view>

Quick Tip: A View-Filtering Search Box

Tue Sep 16 21:31:16 EDT 2014

Tags: xpages

One of the problems that crops up in some situations in XPages is the one described here: executing Ajax queries in too rapid a succession can cause the browser to cap them out until a full refresh. Depending on how you're encountering the problem, there may be a built-in solution: XSP controls that execute Ajax requests often have a throttling or latency parameter, and the same applies for "manual" JS widgets like Select2 (called "quietMillis" there).

Another such situation is the topic of this code snippet: a "filter view" control that allows the user to type and executes partial refreshes for each keypress or clearing of the field. To solve this, I found a block of code I wrote years ago, but should do the trick nicely. It uses setTimeout and clearTimeout to do this sort of delayed search and throttling. As I recall, it worked pretty well, though you'd want to increase the 500ms latency if the request usually takes longer, I suppose (or improve your page speed).

The code is from a Custom Control with styleClass, style, and refreshId properties and stores its value in viewScope.searchQuery.

<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core">
	<xp:div styleClass="lotusSearch #{compositeData.styleClass}" style="#{compositeData.style}" id="searchBox">
		<xp:inputText id="searchQuery" styleClass="lotusText" value="#{viewScope.searchQuery}" type="search">
			<xp:this.attrs><xp:attr name="placeholder" value="Search"/></xp:this.attrs>
			<xp:this.onkeypress><![CDATA[
				if(event.keyCode == 13) {
					if(window.__searchTimeout) { clearTimeout(window.__searchTimeout) }
					XSP.partialRefreshPost("#{javascript:getComponent(compositeData.refreshId).getClientId(facesContext)}", {})
					return false
				}
			]]></xp:this.onkeypress>
			<xp:eventHandler event="search" submit="false">
				<xp:this.script><![CDATA[
					// search is fired when the "X" in WebKit is clicked to clear the box
					if(window.__searchTimeout) { clearTimeout(window.__searchTimeout) }
					window.__searchTimeout = setTimeout(function() {
						XSP.partialRefreshPost("#{javascript:getComponent(compositeData.refreshId).getClientId(facesContext)}", {
							execMode: "partial",
							execId: "#{id:searchBox}"
						})
					}, 500)
				]]></xp:this.script>
			</xp:eventHandler>
			<xp:this.onkeyup><![CDATA[
				// Keypress doesn't fire for deletion
				if(event.keyCode != 13) {
					// Let's try some trickery to auto-search a bit after input
					if(window.__searchTimeout) { clearTimeout(window.__searchTimeout) }
					window.__searchTimeout = setTimeout(function() {
						XSP.partialRefreshPost("#{javascript:getComponent(compositeData.refreshId).getClientId(facesContext)}", {
							execMode: "partial",
							execId: "#{id:searchBox}"
						})
					}, 500)
				}
			]]></xp:this.onkeyup>
		</xp:inputText>
	</xp:div>
</xp:view>

Quick Tip: facetName-less Callbacks in XPages

Wed Aug 13 19:49:59 EDT 2014

Tags: xpages

When you're setting up a Custom Control, you likely know by now that you can set up callback areas to add content to a specified place inside your CC content when it's rendered. They typically look something like this:

<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core">
	<h1>Content Above</h1>
	
	<div>
		<xp:callback facetName="content" id="callback1"/>
	</div>
	
	<h1>Content Below</h1>
</xp:view>

You then drop content in there like so in your XPage:

<xc:someControl>
	<xp:this.facets>
		<xp:text xp:key="content" value="foo"/>
	</xp:this.facets>
</xc:someControl>

That works fine. However, you can benefit greatly by learning that both properties of the callback are optional.

Leaving off the id isn't particularly interesting, but there's a small benefit: removing it means that the callback doesn't generate a <div/> element in the resulant HTML.

Leaving off the facetName, however, is very valuable indeed. When you do that, the callback becomes the target for any non-facet child controls of the CC, in JSF parlance. To wit:

<xc:someControl>
	<p>foo bar</p>
</xc:someControl>

Much cleaner. And you can continue to use other callbacks with names in the control, much like you normally would:

<xc:someControl>
	<xp:this.facets>
		<xp:text xp:key="header" value="Some header text"/>
	</xp:this.facets>
	
	<p>foo bar</p>
</xc:someControl>

The syntax ends up mirroring standard controls like <xp:repeat/>, where the primary content area has no special name, but there are still facets available like "header" and "footer". It also saves you a valuable indent level or two, which can go a very long way to keeping your code readable.


A footnote about my in-practice use of this:

The primary way I use this technique is my "layout" control. Like with most people's apps, that control contains the main page structure (which I used to do as applicationLayout, then switched to Bootstrap-friendly HTML, and then switched back to applicationLayout with custom renderers). Regardless of the implementation, the layouts always have a main content callback and then one or two "column" callbacks. The XSP for the actual control changes a bit per app, but the Design Definition (which you should use) is consistent:

<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core">

	<xp:table style="width:100.0%">
		<xp:tr>
			<xp:td colspan="3" style="background-color:rgb(225,225,225)">Header</xp:td>

		</xp:tr>
		<xp:tr>
			<xp:td style="width: 200px"><xp:callback facetName="LeftColumn" id="callback1"></xp:callback></xp:td>
			<xp:td>
				<p><xp:callback id="callback2"/></p>
			</xp:td>
			<xp:td style="width: 200px"><xp:callback facetName="RightColumn" id="callback3"></xp:callback></xp:td>
		</xp:tr>
	</xp:table>
</xp:view>

(Note: never write code that looks like that when it's not for a Design Definition)

With that, I get a nice, clean, performant table in the WYSIWYG editor with drop targets for each control. I don't use the WYSYWIG editor (and neither should you), but it helps to have that when I open the page initially and it cuts down on crashes. In any event, using it on a page is much like above:

<xc:layout navigationPath="/Home">
	<xp:this.facets>
		<xc:linksbar xp:key="LeftColumn" />
	</xp:this.facets>
	
	Stuff goes here
</xc:layout>

Value and Method Bindings in XPages

Mon Jul 21 17:45:14 EDT 2014

Tags: xpages

When you use any binding in XPages - anything with "#{...}" or "${...}", regardless of whether it's EL, SSJS, or other - you're creating what is called in JSF-land either a ValueBinding or a MethodBinding, and the distinction is important but can be subtle, particularly when dealing with SSJS.

Value Bindings

The simplest case of value bindings are sprinkled throughout even the most basic XPage app, created when you bind a control to a document's field:

<xp:inputText value="#{doc.Title}"/>

The meaning is clear: "this text box connects to the 'Title' property of the 'doc' object". Though simple in concept and syntax, there's a lot of cleverness built into this, both in how many types of objects it can deal with to figure out what "Title" means and also in that it supports both read and write operations with the same syntax. This ability applies to your own Java beans as well. In the absence of any supported interface, EL will map something like #{bean.foo} to both bean.getFoo() and bean.setFoo(...) as appropriate.

The same does not apply to SSJS, however. You may have tried something like this once, only to find your text box became read-only:

<xp:inputText value="#{javascript:doc.getItemValueString('Title')}"/>

The reason for this is that, though you are still creating a value binding, SSJS is only capable of producing read-only bindings. This is because the XPage has no idea that the code inside the brackets is getting the value of a Notes document item; anything in between #{javascript: and } may as well be static as far as the page is concerned. It knows that the code can produce a value, but that's it. Custom binding languages like this CAN allow value writing - the little-used XPath binding does - but SSJS does not, for good reason.

Method Bindings

So that's a value binding - what about method bindings? Well, you're writing these all the time, too. Here's an example of one:

<xp:this.beforePageLoad><![CDATA[#{javascript:
	print("hello")
}]]></xp:this.beforePageLoad>

Method bindings written in SSJS are effectively the same as SSJS value bindings: they just execute the code contained in them and optionally return a value (for example, in button actions).

EL is also capable of creating method bindings, but it behaves slightly differently than it does with value bindings. Take the same binding as before, #{bean.foo}, except now it's attached to the action property of an event handler instead of the value property of a text box. In this context, it is a method binding, and EL no longer cares about supported interfaces or bean conventions. Instead of converting the call into getValue(...) on a DataObject or getFoo()/setFoo(...) on a normal object, EL looks for a method named foo() explicitly. There's no translation and there's no "read mode" or "write mode".

You can see this at play in the stub home page in XPages Scaffolding. The beforePageLoad and afterPageLoad events are method bindings, and so they are bound to methods on the controller with the exact names specified.

There's is a weird gotcha, but it's rare: some method bindings pass parameters along. If you look at the additional event methods on the basic controller class, you see that they expect a PhaseEvent object to be passed in. If you write these same methods in SSJS, you don't have to care about that - in this case, it appears that the page just throws the argument away. However, if you bind an event with arguments to a method without, you'll get an exception like java.lang.IllegalArgumentException: wrong number of arguments. And unfortunately, even adding the parameter doesn't solve it on its own... you have to manually bind the event for some reason. The good news is that those are the only cases I know of where that happens - everything else expects an argument-less method.

When you're writing in primarily SSJS, the distinction between the two only matters for write-mode value bindings, but the more you deal with Java in XPages, the more it pays to know a little about how the XPages framework perceives the binding types.

Building an App with the frostillic.us Framework, Part 4

Thu Jul 17 15:44:16 EDT 2014

Tags: xpages
  1. Building an App with the frostillic.us Framework, Part 1
  2. Building an App with the frostillic.us Framework, Part 2
  3. Building an App with the frostillic.us Framework, Part 3
  4. Building an App with the frostillic.us Framework, Part 4
  5. Building an App with the frostillic.us Framework, Part 5
  6. Building an App with the frostillic.us Framework, Part 6
  7. Building an App with the frostillic.us Framework, Part 7
  1. Define the data model
  2. Create the view and add it to an XPage
  3. Create the editing page
  4. Add validation and translation to the model
  5. Add notification to the model
  6. Add sorting to the view
  7. Basic servlet
  8. REST with Angular.js

Add validation and translation to the model

Now that we have a page to edit the model objects, it's time to spruce them up a bit by adding some type information, an additional field, and some basic translation. When we first set up the Note model, it didn't contain any information about the fields it expects, instead leaving everything up to the pass-through defaults. Now, we'll codify the two fields we added last time, add validation to the title, and add a date/time field:

package model;

import java.util.Date;
import javax.persistence.Table;
import org.hibernate.validator.constraints.NotEmpty;
import frostillicus.xsp.model.domino.AbstractDominoModel;

@Table(name="Note")
public class Note extends AbstractDominoModel {
	private static final long serialVersionUID = 1L;

	@NotEmpty String title;
	Date posted;
	String body;

	// *snip*
}

Model object fields work in the absence of explicit getters and setters - you CAN specify those for additional complexity, but they are not required. You can see here the addition of a validator care of Hibernate Validator. The @NotEmpty annotation works similarly to "not empty" in EL: it requires that the String not be null and also that it not be blank. We also added a date/time field named "Posted" (date- and time-only fields can be added via the java.sql.Date and java.sql.Time classes, respectively) - let's add a form row back on our note.xsp page:

<xe:formRow binding="#{controller.components[note].Title}"/>
<xe:formRow binding="#{controller.components[note].Posted}"/>
<xe:formRow binding="#{controller.components[note].Body}">
	<xe:djTextarea/>
</xe:formRow>

I didn't add any specific information to flag "Title" as required or "Posted" as a date/time field on the XPage, but the controller's component binding picks both up:

The controller will add any validation annotations to the control, but it has special support for @NotEmpty and @NotNull to also add a required validator and property explicitly. Additionally, since the "Posted" field stores both a date and time, the controller builds a text box with a date/time converter and helper, in the same fashion as setting the data type in the GUI.

Now, though the "Posted" field name matches Domino's idiom from the mail template, it's sort of a weird name to present to a user, so we'll use this as an introduction to translation and creating human-friendly names. The way labeling works in Framework component binding, it looks for a resource bundle named in the form "model_translation.properties". Because it builds on the XSP stack's standard resource-bundle code, it also allows localized versions in the format "model_translation_fr.properties", picking the most appropriate one for the browser. The format of the keys to match the fields is the full class name, then a dot, then the name of the field. So here's a basic "model_translation.properties" File Resource:

model.Note.Posted=Created
model.Note.Body=Note Body

By creating this file, the controller automatically picks up on the new human-readable names:

And this goes for other locales as well. Here's a poorly-translated "model_translation_fr.properties" file:

model.Note.Title=Titre
model.Note.Posted=Créé
model.Note.Body=Corps

And if I switch the browser to use French as its top preferred language (the "Save" button is still the purview of normal XPages translation):

By moving these concerns - data type, validation, and translation - into the data model, it not only significantly lowers the amount of code on the XPage itself, but also makes it much easier to find the canonical rules. Types and validation are always in the model class - not strewn across half a dozen XPages and custom controls - and translations are always in the model translation file and tied directly to the fields they're translating - not hidden in XPage-specific properties files keyed by ID paths.

In the next part, we'll add a bit of UI friendliness to the note creation/editing process using a common messages control and a flashScope object.

Building an App with the frostillic.us Framework, Part 3

Thu Jul 17 14:19:39 EDT 2014

Tags: xpages
  1. Building an App with the frostillic.us Framework, Part 1
  2. Building an App with the frostillic.us Framework, Part 2
  3. Building an App with the frostillic.us Framework, Part 3
  4. Building an App with the frostillic.us Framework, Part 4
  5. Building an App with the frostillic.us Framework, Part 5
  6. Building an App with the frostillic.us Framework, Part 6
  7. Building an App with the frostillic.us Framework, Part 7
  1. Define the data model
  2. Create the view and add it to an XPage
  3. Create the editing page
  4. Add validation and translation to the model
  5. Add notification to the model
  6. Add sorting to the view
  7. Basic servlet
  8. REST with Angular.js

On the scale of "bog-standard" to "weird", the next phase of writing a frostillic.us Framework app sits in the middle: the basic steps of creating a model-editing page are pretty standard, but you can use unique aspects of the Framework to save a bunch of coding and add some nice dynamic features.

Create the editing page

The first step to adding the ability to create and edit notes is to adjust the main note-list page to add a button to actually create one. In my standard layout custom control, I have a facet area named "ActionBar" to add buttons to the top of the page:

<xc:layout navigationPath="/Home">
	<xp:this.facets>
		<xc:linksbar xp:key="LeftColumn" />
		<xp:div xp:key="ActionBar">
			<xp:link themeId="Button.Command" value="/note.xsp" text="New Note"/>
		</xp:div>
	</xp:this.facets>
	
	...
</xc:layout>

I like to use normal links for going to the editing page rather than an "openPage" simple action to avoid the extra round trip. By giving it the "themeId" of Button.Command, it ends up with styling that looks more like a button than a link.

Now that that's over with, we move on to the actual editing page. So we create a page named "note.xsp" and add in our basic surrounding XSP and the data source:

<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core" xmlns:xe="http://www.ibm.com/xsp/coreex" xmlns:xc="http://www.ibm.com/xsp/custom" xmlns:ff="http://frostillic.us/framework"
	beforePageLoad="#{controller.beforePageLoad}" afterPageLoad="#{controller.afterPageLoad}" pageTitle="Note">

	<xp:this.data>
		<ff:modelObjectData var="note" managerName="Notes" key="${empty param.id ? 'new' : param.id}"/>
	</xp:this.data>
	
</xp:view>

You can see a couple Framework-isms here: the "ff" namespace for the model data sources, the "beforePageLoad" and "afterPageLoad" events bound to the implied controller class, and the model data source itself. That data source takes the place of the traditional xp:dominoDocument element when using Framework model objects. It has two important properties: "managerName", which is the managed bean name of the manager class to fetch the object from, and "key", which tells the manager to either fetch an existing object by ID (for Domino-backed model objects, the document UNID) or create a new one. For this, I use the ternary operator in EL to check to see if there was an ID passed in via URL - if so, use that ID; otherwise, tell the manager to create a new model. Think of this as the difference between visiting a xp:dominoDocument-using page with action=editDocument&documentId=XXXX and with action=newDocument.

Now, on to the actual "form" portion. For traditional label/field sets, I'm a fan of using the ExtLib's xe:formTable and associated controls, since it's very focused on declaratively describing the form, and then I can use custom renderers to modify the look as I see fit. So let's toss one of those on the page:

<xe:formTable formTitle="Note">
	<xe:this.facets>
		<xp:div xp:key="footer">
			<xp:button id="save" value="Save">
				<xp:eventHandler event="onclick" submit="true" refreshMode="complete" immediate="false" save="true"/>
			</xp:button>
		</xp:div>
	</xe:this.facets>

	<xe:formRow binding="#{controller.components[note].Title}"/>
	<xe:formRow binding="#{controller.components[note].Body}">
		<xe:djTextarea/>
	</xe:formRow>
</xe:formTable>

So far, so good, right? That is, except for that "binding" business. That's the fruit of my recent dabbling in the potential uses of that property. Think of that as telling the back-end framework code that the two rows represent, respectively, the "Title" and "Body" fields of the given note object specifically, and asking it to fill in the messy details. As a result, the framework looks up an appropriate label for the field (by default, just the name of the field in the model, but it also looks for a human-friendly and translated version, which I'll get to in the next entry), creates an input component if one hasn't been provided, and attaches any appropriate validators, converters, and date/time helpers.

There are two types of use here: for the "Title" field, just a plain text box will do, which is what the controller does in the absence of other direction; for the "Body" field, however, I want to use Dojo's auto-expanding textarea control, and so placing one inside the form row will cause the controller to pick up on it and use the existing control rather than adding a new one. We haven't added any extra validation or type information to the model yet, so the result is basic:

There's one last bit of housekeeping to add: sending the user back to the main home page after saving a note. I use navigation rules for this purpose, and this syntax matches what you get if you set the "Next page (success or cancel)" option in the XPage's properties pane:

<xp:this.navigationRules>
	<xp:navigationRule outcome="xsp-success" viewId="/home.xsp"/>
</xp:this.navigationRules>

And that's it! Other than a few Framework-isms in the areas of the data source and the binding code, the result is pretty similar to a page you'd get using traditional methods. In the next entry, I'll spruce things up a bit: adding in some validation, a different field type, and a dash of translation.

Building an App with the frostillic.us Framework, Part 2

Fri Jul 11 18:01:48 EDT 2014

Tags: xpages
  1. Building an App with the frostillic.us Framework, Part 1
  2. Building an App with the frostillic.us Framework, Part 2
  3. Building an App with the frostillic.us Framework, Part 3
  4. Building an App with the frostillic.us Framework, Part 4
  5. Building an App with the frostillic.us Framework, Part 5
  6. Building an App with the frostillic.us Framework, Part 6
  7. Building an App with the frostillic.us Framework, Part 7
  1. Define the data model
  2. Create the view and add it to an XPage
  3. Create the editing page
  4. Add validation and translation to the model
  5. Add notification to the model
  6. Add sorting to the view
  7. Basic servlet
  8. REST with Angular.js

The first stage of building a frostillic.us-Framework-based app was a bit unusual - focusing on creating a model object in Java - but the next phase is about as bog-standard as it gets:

Create the view and add it to an XPage

The view we'll need is a basic one showing all Notes. We'll name it "Notes\All" to match the expected prefix in the model and add our two expected columns: Title and Body. The model classes work with sortable columns, so we'll make Title sortable:

Once we have the view, we can refer to it in EL in an XPage as #{Notes.all}. For demo purposes, we'll drop it on the page using a standard xp:viewPanel - not the fanciest presentation, but it'll get the job done:

<xp:viewPanel value="#{Notes.all}" pageName="/note.xsp">
	<xp:this.facets>
		<xp:pager xp:key="headerPager" id="pager1" partialRefresh="true" layout="Previous Group Next" />
	</xp:this.facets>
	
	<xp:viewColumn columnName="Title" displayAs="link">
		<xp:viewColumnHeader value="Title" sortable="true"/>
	</xp:viewColumn>
	<xp:viewColumn columnName="Body">
		<xp:viewColumnHeader value="Body"/>
	</xp:viewColumn>
</xp:viewPanel>

Here, I'm using value="..." to specify the data source. There is also a normal data source for model collections, but it's rarely needed - the EL syntax is much more concise and clear, and it works well with relations (e.g. #{note.responses}). I also specify a pageName="..." for the generated links, since $$OpenDominoDocument URLs won't help here. I may add something to assist with that down the line, but for now specifying the page will do.

Other than those, though: pretty standard stuff. That's one of the goals of the framework: when it makes sense to do things the standard way, I want to do that - no need to reinvent every wheel. So the lists extend TabularDataModel in addition to implementing List and the model objects themselves implement ViewRowData in addition to DataObject.

The next topic - creating an editing page - continues this trend of working with the standard components while cutting out as much hassle as possible.

Building an App with the frostillic.us Framework, Part 1

Wed Jul 09 10:18:00 EDT 2014

Tags: xpages
  1. Building an App with the frostillic.us Framework, Part 1
  2. Building an App with the frostillic.us Framework, Part 2
  3. Building an App with the frostillic.us Framework, Part 3
  4. Building an App with the frostillic.us Framework, Part 4
  5. Building an App with the frostillic.us Framework, Part 5
  6. Building an App with the frostillic.us Framework, Part 6
  7. Building an App with the frostillic.us Framework, Part 7

Now that the framework I've been building is settling into a real project, I figured one of the best ways for me to explain how it works and to double-check my own assumptions is to make a tutorial on how to actually construct an app using it. This will be a very simple one to start with: a note-taking app with just a couple fields and only one business object to speak of.

The overall conceit of the framework is that it uses standard/ExtLib components and idioms where appropriate but declares its own new idioms when the standard ones are deficient. This is done in a way, though, that is opt-in at each step. Though this app is positively stewing in the "Jesse" way of doing things, each component - controllers, models, servlets, etc. - can operate independently and alongside other methods of XPage development.

Probably unsurprisingly, writing a fully framework-based app is heavy on Java and uses some aspects that may be strange at first (annotations, namely), but I hope that the tutorial will explain why I go this route, and that it will demonstrate how using Java in this way can save a tremendous amount of programming and eliminate swaths of potential bugs.

I have a tentative plan for the series, and I'll adjust this list and add links as appropriate:

  1. Define the data model
  2. Create the view and add it to an XPage
  3. Create the editing page
  4. Add validation and translation to the model
  5. Add notification to the model
  6. Add sorting to the view
  7. Basic servlet
  8. REST with Angular.js

Without further ado, on to part 1:

Define the Data Model

Technically, I should start with part 0: set up the framework. As of this writing, I haven't packaged it up on OpenNTF yet, but I've created a ZIP file containing a current build, the NTF I use to create these apps, as well as the OpenNTF API from my branch - the framework relies on some features not yet released in an API milestone release:

frostillicus.framework-20140709.zip

Requirements-wise, you'll probably need 9.0.1 (the ExtLib that ships with it should suffice) and I haven't tested it on a server without AllPermission granted to all Java.

Now that that's out of the way, on to the actual code. To start with, we'll create the class that represents our core data model - the Note (in the note-taking sense) - and its Manager class. Domino-based model classes in the framework extend the frostillicus.xsp.model.domino.AbstractDominoModel class, while the Managers extend frostillicus.xsp.model.domino.AbstractDominoManager and are kept in the model package:

package model;

import javax.persistence.Table;

import frostillicus.xsp.bean.ApplicationScoped;
import frostillicus.xsp.bean.ManagedBean;
import frostillicus.xsp.model.domino.AbstractDominoModel;
import frostillicus.xsp.model.domino.AbstractDominoManager;

@Table(name="Note")
public class Note extends AbstractDominoModel {



	@ManagedBean(name="Notes")
	@ApplicationScoped
	public static class Manager extends AbstractDominoManager<Note> {

		@Override
		protected String getViewPrefix() {
			return "Notes\\";
		}
	}
}

So! What's going on here? Quite a few things, and just as important are the things that are not going on, such as code to declare all of the fields and each pertinent view's capabilities. Nonetheless, it's still a whole pile of Java concepts - this is the part that's the most unusual from a standard-XPages-dev perspective.

  • Right at the start of the class, you see @Table(name="Note"). This is part of my overall attempt to borrow aspects of the Java Persistence API and Hibernate when appropriate. Normally, @Table refers to a SQL table, but I'm using it here to declare the form name to use when creating documents. This may change in future versions.
  • There are no getters/setters (or other methods) in the Note class! These objects implement DataObject and so the standard way of accessing data is via getValue(...) and setValue(...). There are ways to add special behavior - complex getters, validation, and types - but the default behavior is to mimic the Domino way of just storing anything with any name.
  • The Manager class is stored as what's called an "inner class" in Java. It's a way of storing multiple related classes inside a single .java file. The full name of the class is a combination of the two names - so if you import the Task class in another file, the manager is named Task.Manager.
  • I'm using @ManagedBean annotations. These are something I tossed together a while ago based on how managed beans can be done in newer versions of JSF. In essence, these annotations let you specify the name and scope of your beans in-line with the code, without adding them to faces-config.xml. The result acts just the same way.
  • The only method specified in the Manager class is to provide a standard prefix for accessing views. I'll demonstrate using that later, but this is specified here to encourage the idiom of naming views like "Notes\All" or "Notes\By Day". It's not required, but following it saves some code elsewhere.
  • The Manager is also the place where you can specify an external database to house the data, but I'm not doing that here.

It's a strange bit of code, but my hope is that each line is clear once you understand the concept. My aim is to, as much as possible with Java, make the code in the model classes focus on specifying what the object does and where it's found, not on the messy business of actually retrieving and storing it (while also avoiding the configuration headaches of full JPA/Hibernate).

In the next post in the series, I'll cover adding a view for these documents to the DB and to an XPage.

Things I Rarely Use: sessionScope

Thu Jul 03 20:20:06 EDT 2014

Tags: xpages
  1. Things I Rarely Use: sessionScope

As this post's title implies, I'm considering making this a series on XPages anti-patterns, but no promises there.

In any event, this is a topic that's been stewing in my brain for a little while: my antipathy towards the session scope in XPages. Now, don't get me wrong: other than the "not reset on logout" thing that may be fixed by now, I have no technical qualms with sessionScope; it does what it says on the tin. However, I've often found that many people use it very frequently, whereas I have found fewer and fewer uses over time where it is appropriate.

To set the stage, I use the various scopes in roughly this order by descending frequency:

  1. viewScope
  2. applicationScope
  3. flashScope
  4. requestScope
  5. sessionScope

The main things that I use sessionScope for are things that truly make sense only for the current browser session, such as the current date range to view in a log-viewing app. Other than that, I generally don't use it for:

Caches

Though it's not wrong, per se, to use the session for this, I've found it better overall to use the applicationScope, either directly (applicationScope.put("someCachedValue", whatever)) or by putting a Map keyed by username in there. The latter gets the same user-specific cache benefits of sessionScope (and more reliable, too, due to the potential for switched authentication) while also having the benefit of keeping the cache if the user logs in from another device. This is particularly potent with Anonymous. This is not a hard-and-fast rule, though - you may decide otherwise for cache-size or other reasons.

Primary Navigation or Context

Unlike the previous one, this is a hard-and-fast rule: do not use sessionScope for important page context. The worst would be something like having an "open document" button that puts the desired document UNID (or, worse, note ID) in sessionScope and then navigates to the page. Never do this! Though XPage URLs are a continuing problem, they're still the correct place for target-document information. The rule of thumb is that you should be able to copy the URL any time, paste it into another browser, and be in basically the same place.

Secondary Context

By this I mean things like the active linksbar category for the current page. I've seen things like having a navigation bar link that sends the user to a certain page while also setting a sessionScope variable to indicate the active menu bar. This is a huge problem for a number of reasons: it's a maintenance nightmare (having to code every link to do this), it's just asking for bugs (links setting the wrong or no value), and it breaks completely when the user bookmarks the page or comes back after session expiration. It's technically better than the previous crime, but only barely. The correct place for this information is handled somewhere in the page structure, though the specifics get murky. I generally take a page from the Extension Library example DB and use a "navigationPath" properly on my layout control to define a slash-delimited hierarchy of navigation context.

Page-to-Page Context

I'm thinking of things like a task-tracking system where you're looking at a Client document and want to add a Task to them, providing the Client document's ID for context. This is another area where the URL is the correct choice: ending up with a URL like "/task.xsp?clientId=whatever" makes the intent ("create a new task for client with ID 'whatever'") clear and stable across visits.


Overall, I think of sessionScope as the Petyr Baelish of XPage features: there are some cases where you have to deal with it, but you should generally consider it extremely unreliable and untrustworthy.

Coaxing Enums Into XPage Combo Boxes

Wed May 28 19:07:12 EDT 2014

Tags: java xpages

I've been reworking Forms 'n' Views over the last couple days to use it as a development companion for the OpenNTF Design API, and one of the minor hurdles I ran across was presenting a getter/setter pair that expects a Java Enum to an XPages control (a combo box in this case, but it would be the same with other similar controls). I had assumed at first that it would simply not work - that the runtime would entirely balk at the conversion. However, it turned out that XPages does half the job for you: for displaying existing values, it will properly coax between the string form and the Enum constant. However, it breaks when actually going to set the value.

This whetted my appetite enough that I decided to try to see if I could write a small converter to finish the job in a properly generic way, and it turns out I could. To cut to the chase, I wrote this class:

package converter;

import javax.faces.component.UIComponent;
import javax.faces.context.FacesContext;
import javax.faces.convert.Converter;
import javax.faces.el.ValueBinding;

public class EnumBindingConverter implements Converter {

	@SuppressWarnings("unchecked")
	public Object getAsObject(final FacesContext facesContext, final UIComponent component, final String value) {
		ValueBinding binding = component.getValueBinding("value");
		Class<? extends Enum> enumType = binding.getType(facesContext);

		return Enum.valueOf(enumType, value);
	}

	public String getAsString(final FacesContext facesContext, final UIComponent component, final Object value) {
		return String.valueOf(value);
	}

}

The getAsString method is your standard "I don't care about this" near-passthrough, but the getAsObject portion does the work necessary. Because I'm binding to a Java bean with getters/setters expecting the appropriate Enum type (this would presumably also work with a DataObject with a thoroughly-implemented getType method), the ValueBinding class is able to look up and provide back the expected Enum class. I then use the standard Enum.valueOf method to let Java do the work of actually finding the appropriate Enum constant.

Using this, I'm able to write a control like this:

<xp:comboBox value="#{columnNode.sortOrder}">
	<xp:this.converter><xp:converter converterId="enumBindingConverter"/></xp:this.converter>
	<xp:selectItem itemLabel="None" itemValue="${javascript:org.openntf.domino.design.DesignColumn.SortOrder.NONE}" />
	<xp:selectItem itemLabel="Ascending" itemValue="${javascript:org.openntf.domino.design.DesignColumn.SortOrder.ASCENDING}"/>
	<xp:selectItem itemLabel="Descending" itemValue="${javascript:org.openntf.domino.design.DesignColumn.SortOrder.DESCENDING}"/>
</xp:comboBox>

And it works! That allows you to bind to a getter/setter pair that expects an Enum without having to write in a shim to convert it to a String value beyond the generic converter.

You probably noticed the same thing I did, though: that's ugly as sin. Not only that, but it's a lot of redundant code when what you usually want to do is provide a selection of all available constants for the given Enum. To save myself a bit of future hassle and prettify my code a bit (at the expense of UI, but who cares about that? (I kid)). I wrote this DataObject bean to accept an Enum class name and return a list suitable for use in a control:

package bean;

import java.io.Serializable;
import java.util.*;

import javax.faces.context.FacesContext;
import javax.faces.model.SelectItem;

import org.openntf.domino.utils.Strings;

import com.ibm.xsp.model.DataObject;

import frostillicus.bean.*;

@ManagedBean(name="enumItems")
@ApplicationScoped
public class EnumSelectItems implements Serializable, DataObject {
	private static final long serialVersionUID = 1L;

	public Class<?> getType(final Object key) {
		return List.class;
	}

	@SuppressWarnings("unchecked")
	public List<SelectItem> getValue(final Object key) {
		if(key == null) {
			throw new NullPointerException("key cannot be null.");
		}

		String className = String.valueOf(key);

		try {
			Class<? extends Enum> enumClass = (Class<? extends Enum>)FacesContext.getCurrentInstance().getContextClassLoader().loadClass(className);

			Enum[] vals = enumClass.getEnumConstants();
			List<SelectItem> result = new ArrayList<SelectItem>(vals.length);
			for(Enum val : vals) {
				SelectItem item = new SelectItem();
				item.setLabel(Strings.toProperCase(val.name()));
				item.setValue(val);
				result.add(item);
			}

			return result;
		} catch(Throwable t) {
			throw new RuntimeException(t);
		}
	}

	public boolean isReadOnly(final Object key) {
		return true;
	}

	public void setValue(final Object key, final Object value) { }
}

It uses the @ManagedBean implementation that I cobbled together a while back, but you could just as easily use the normal faces-config route. The way you use this is to pass in the class name of the Enum you want in a selectItems component:

<xp:comboBox value="#{columnNode.sortOrder}" onchange="fnv_mark_dirty('#{id:viewPane}')">
	<xp:this.converter><xp:converter converterId="enumBindingConverter"/></xp:this.converter>
	<xp:selectItems value="${enumItems['org.openntf.domino.design.DesignColumn$SortOrder']}"/>
</xp:comboBox>

Though it only saves a few line in this case, it'd pay off more with larger Enums. It has the down side of being pretty naive about its conversions: it just does a basic first-character proper-casing of the Enum constant name, but there's no reason you couldn't add better human-friendly-ifying to it.

I'm not sure if I'll be able to satisfy all of my Java preferences when it comes to multi-value Enums, though. Java's generic type erasure means that runtime code wouldn't have a way to inspect the getters/setters to figure out the appropriate Enum type. Perhaps I can inspect the select items attached to the control to find the appropriate class.

Book Review: Mastering XPages, Second Edition

Sun May 11 18:07:55 EDT 2014

Tags: review xpages

For a review of Mastering Xpages, I could copy almost wholesale the introduction from my review of XPages Extension Library: the low count of XPages books makes each one an essential purchase. As before, fortunately the book rises above that obligatory baseline and is absolutely worth your time. That's the upshot of this review: if you're a Domino programmer, buy and read this book.

Size

One of the first things you notice about the book is its size, clocking in at over a thousand pages. This is a function of the complexity of the platform and the breadth of its intended audiences. Due to its role as "the" XPages book, it contains content meant for people ranging from legacy Domino devs, experienced XPages devs, people entirely new to the platform, and JSF programmers new to Domino.

Fortunately, the structure of the book makes this manageable. Particularly once you get to Part IV, it is essentially a collection of self-contained topic references. Though it would be a good idea to read through the entire thing, there are large sections that you can hold off on until you have a need, such as XPiNC, mobile development, and internationalization.

Changes Since Last Time

The original edition of Mastering XPages was written for Dominp 8.5.2, before the Extension Library and before knowledge of real XPages development had really started infusing the community outside of a couple teams. The additions since the previous version focus largely on the "mainstreaming" of several Extension Library controls and concepts (without going into the exhaustive coverage of the Extension Library book proper), further coverage of Java's role in XPages development, and sort of a "best of" of IBM's releases in the interim: the XPages Toolbox, improved debugging, Tony McGuckin's Masterclass series on performance and scalability, and even the Rich Text Editor Evolution control. These provide a much more well-rounded view of what modern XPages development actually consists of, beyond the comparatively-primitive (and buggy) options in a stock 8.5.2 installation.

Java

I was very pleased to see heavy coverage of Java development in XPages, starting off early (and possibly intimidatingly) with a chapter on XPages' JSF roots and the Java classes/interfaces that make up its true form. To suit the various target audiences, this coverage runs the gamut from creating managed beans, to logging/performance tracking, to creating controls and renderers whole-cloth.

However, I'll admit that I wished there was another level represented: writing your business logic in Java, probably via managed beans. The IBM-advocated way is to use SSJS for business logic and use Java when you have a particularly JSF-related need, though, so the lack of this topic is a result of that.

Lack of Opinion

My primary quibble with the book is basically the same as my problem with the platform as a whole: it's not opinionated. XPages, as a platform, is massive, and trying to understand every moving part is likely impractical for anyone whose job isn't focused on either writing or manipulating the entire thing. As a Java-based framework, it carries enterprise Java's tendency for massive class hierarchies, factories, layers of configuration, and several types of indirection. As a system grafted on top of Domino, it bears the scars of merging with a platform and development ideology often at odds with JSF's origins. And as a modern web framework, it requires involvement in the dizzying array of web technologies and idioms.

The result of this is that it's often largely unclear what the best way to accomplish a task is. It's outside of the book's bailiwick to establish this One True Way (to my knowledge, no such Way exists for XPages), but the state of affairs leads to some potentially-confusing results.

On a micro scale, there's the example of resources in themes. Themes provide two syntaxes for including JavaScript and CSS files: the "legacy" way (<resource> elements) and the good way (the <resources> element and its XSP-alike children). The book mentions both, and even explicitly describes the latter as the one to use, but it, like Designer help and the default comments in a new theme file, still uses the older style for the actual examples. It would be better to mention the existence of the old style for in case you run into it, but otherwise ignore its existence.

On a macro scale, what the book lacks is a guide to building a full application. The book is rife with downlodable examples, and those are helpful, but I think it could be all the better to demonstrate building one of the canonical "tutorial"-type apps, such as a to-do list or store. An example of using each main topic in the same from-scratch app (CRUD basics with Domino documents, building a layout using Application Layout, mobile-enablement with the mobile controls, adding internationalization, and finally performance monitoring and tweaking) could provide a valuable guide to how to build a solid, performant XPages app. The use of the discussion app does a bit of this, but most of that app is already written, and it's not a consistent thread through the book.

Conclusion

Overall, gripes aside, my original opinion remains: this book is a valuable resource for any Domino developer. Its value goes beyond the initial reading as well, as it will be a powerful go-to reference tome for major subjects (especially if you get the ebook version and can just search). For my part, I expect the chapter on building components to be particularly useful as I delve further into the topic. If you haven't purchased this yet (or coaxed your company into purchasing it for you), remedy that immediately.

Using a ViewHandler to Serve a Different XPage

Sat May 03 10:55:26 EDT 2014

Tags: xpages

I've been using a ViewHandler class for a while now for a specific purpose: injecting controller objects automatically. However, I hadn't really dug into them further than that. I should have, though: I spent a little time poking around this morning and found that they can help solve two minor annoyances I've had for a while.

For the purposes of this post, the pertinent one of those is the ability to serve a different XPage than the one requested. In normal cases, this isn't a particularly useful ability, though I can think of some pie-in-the-sky ideas like loading the compiled XPage classes from a plugin or Jar (like a homebrewed single copy design). Fortunately, I have an immediate use in mind: first-run config.

If you've used a package like WordPress, you've probably run into its first-run behavior, where it runs you through a wizard to set up the app configuration, database paths, and the like. Normally, this isn't important in a Notes app, but it's needed if you want to ship an empty DB and have people set it up for use. In my own case, I've taken to using a keyed document to store the app configuration, and this document doesn't exist when I reconstitute it from soruce. There are a couple ways I could handle this; for example, I could put code at the start of every page (or in each page's controller) to check for a completed config and, if not present, send a redirection header. Putting it in the ViewHandler, though, means each individual page doesn't have to care.

Here's the snippet of applicable code from my new ViewHandler:

public UIViewRoot createView(final FacesContext context, final String pageName) {
	if(!AppConfig.get().isComplete()) {
		return super.createView(context, "/appconfig");
	}

	return super.createView(context, pageName);
}

Now I have my config bean do a self-check to make sure all the fields required for normal app operation are present and, if not, the code forces the config page to load in place of the requested one. It has the side benefit of not actually causing a redirection, and in that way it's similar to Domino's standard login and error forms.

While this isn't the sort of feature that's going to completely change the way I write XPages apps (most likely), it's good to know about one more ability.

An XPage As A Tree: Implications

Wed Apr 16 17:41:28 EDT 2014

Tags: xpages

A few posts ago, I talked about how the skeletal nature of an XPage is really a very abstract representation of a UI. At its core, it isn't conceptually tied to the web - and, indeed, the probable bulk of the work done by JSF is to push against the normal rules of the web.

The goal of XPages/JSF is to abstract away the messy business of writing web pages, much like a cross-platform mobile framework attempts to abstract away the chore of writing to iOS or Android directly.

In practice, I've found this to be both a brilliant and awful idea. I'm not alone in this: if you look around, you'll find proponents of methods of web programming that don't go against the grain so much, and they have excellent points. However, the "abstraction" route isn't without its benefits. So, off the top of my head, I can think of a number of pros and cons of the XPages "component" conceit. In the spirit of suspiciously defying convention, I'll start with the cons.

Cons

  • The abstraction leaks like a sieve. Case in point: the style and styleClass properties. In an ideal world, those have no place in an abstracted framework - in XPages, it should be the job of themes and renderers to sweat those details. But in reality, they're vital.
  • It's harder to write a clean page. Because you're working at a high level, you're relinquishing control of much of the structure of your page to the components' renderers, which will likely spit out all sorts of oddball divs, spans, and the like. Theoretically, you can fix this, but:
  • Writing renderer classes is a colossal PITA. Java has its share of virtues, but writing code to spit out other code is not among them. Writing a renderer for even a basic control is an exercise in verbosity and fighting Java's inferior literal strings. This means:
  • It's harder to keep up with the times. Because you're largely dependent on the implementor of the renderer for the control's HTML/JS, you're likely to run into situations where the HTML is atrocious (see: radio and checkbox groups) or runs counter to the structure you want for your page (see: using pagers on a Bootstrap-ified site). That's not to mention significantly-different programming models like AngularJS.
  • They're a poor match for the realities of HTML/HTTP. The whole conceit of JSF - that you have this tree of Java objects sitting on the server that you're interacting with as if it's a desktop app - requires tremendous code and conceptual overhead to glom onto a stateless network protocol like HTTP.

Pros

  • The abstraction level is often just about right. When you want to create an entity on your page that is conceptually common but has no representation in HTML - say, a widget container or tabbed table - the component model shines. The code you end up writing is extraordinarily task-focused: you're creating a widgetContainer, not a bunch of divs with class names that happen to match the CSS/JS framework of the day. Along those lines:
  • It's easier to keep up with the times. I am aware that the inverse of this is in the "cons" list, but they're both true. Though it can be harder to adapt your components to modern standards, if you do the job well, the component model makes it easier to bring older apps forward. If you app consists entirely of standardized components like applicationLayout, formTable, and the like, a set of new renderers can adapt these conceptually-common controls to a new framework that wasn't in use when you wrote them originally.
  • You can write your own. Since, in spite of the difficulty, JSF actually succeeds in presenting an idealized Java world, it's a minor conceptual leap (though a large code-writing one) to go from using common components to building ones that match your needs. This is where Custom Controls shine, and do the job well either as-is or a starting point for building more "baked in" Java components.
  • HTML is no prize pig either. As much as it's easy to extol the virtues of going with the grain of HTML/HTTP/etc., it's still a huge pain to do so. The closer you get to the "metal", the more you have to deal with non-semantic markup elements, browser bugs, HTML entity escaping, and so forth. It's a fool's errand to do it by hand, so the task is just picking a set of abstractions that strike your fancy.

So what's the best route? Beats me! To a large extent, it doesn't matter, because no web framework lasts forever. For my part, I find the XPages/JSF abstraction to be a fine tool, and I plan to continue refining my ability to write to its strengths.

How I Added File-Download Support To My Model Framework

Sat Apr 12 13:33:24 EDT 2014

Tags: xpages java

Unlike the last few posts, this one isn't as much a tip or tutorial as it is an attempt to share my suffering by detailing the steps I took recently to add file-download and -upload support to my current model framework.

The goal of my framework overall is to retain the benefits of normal Domino document data sources (arbitrary field access, ease of use in simple cases, and compatibility with standard XPage idoms) while adding on useful new abilities (getters/setters, relationships with other objects, abstraction between the XPage and the actual location of the data). Most of this work is easy - by implementing DataObject, my objects worked well with EL bindings and most of the work was focused on hooking up Domino Document objects in a similar way to IBM's DominoDocument class.

However, some of the more esoteric controls are not so easy, and the file controls were chief among them. As I tinkered, I found that you can't directly point a xp:fileDownload control to a normal data source and get it to work well. You can get it to list a single file, but I wasn't able to get it to recognize more than one through traditional means.

In my tinkering, I discovered that there are two main paths for the control: the traditional one and a more-complicated one, which makes a lot of assumptions about the nature and availability of the referenced data source. Importantly, unlike normal bindings (which take your expression, say #{doc.body}, have the EL processor resolve it, and deal with the final resolved entity), the control actually treats the EL expression as a String, splits it in two on the "." character, and looks up the variable on the left side plus ".DATASOURCE". This "doc.DATASOURCE" idiom matches the behavior of data-source controls on an XPage, which make themselves or an associated object available by that name. If the control finds an object by that name and if it implements DataSource (a common interface used by these controls), it assumes that the source contains a DataContainer object that also implements DocumentDataContainer specifically.

Phew, okay, so far so good. It makes some inconvenient sense that the control would expect its referenced value to conform to the norms of the built-in data sources, so I wrote controls to do just that (previously, I got by using normal EL references from managed beans). However, I still ended up getting ClassCastExceptions at runtime: it turned out that the control also uses a nebulous factory object to attempt to convert the received value into a data type it expects, specifically DominoDocument. Unfortunately, unlike the interfaces I implemented already, subclassing DominoDocument was not something I was willing to do. So I was back to digging.

What I found was that this attempted conversion was happening due to an object implementing com.ibm.xsp.model.DataModelFactory, which has the job of taking an intermediate value used in the control (a com.ibm.xsp.model.FileDownloadValue, which contains a single-entry HashMap containing the object and requested field name) and returning the attachment data as an instance of DataModel. In addition to writing the code to do this, there's another wrinkle: the way the factory is found. The way I found to access this is thoroughly awkward: if I use the deprecated method getFactoryLookup() on the application instance, I can then retrieve the default factory by a key ("com.ibm.xsp.DOMINO_DATAMODEL_FACTORY"), wrap it in a new object that knows about my model objects, and replace it in the lookup.

Once I did all that, I was off to the races! Sort of! There remained another problem: the "delete attachment" icon on the control. I discovered that the way this icon works is that it composes an action that tracks down an adapter that knows how to communicate with the associated model object and sends it a deleteAttachments(...) call. As before, the way it "tracks down" the adapter is to look up another factory ("com.ibm.xsp.DESIGNER_DOMINO_ADAPTER_FACTORY") and ask it to provide an appropriate adapter. So again I wrote my own variant of this to intercept calls dealing with model objects and carefully placed it, Indiana-Jones-style, in place of the default factory.

The net result is that it works! I can point a file-download control at a model object (as long as that object was specifically instantiated in a data source control and is referenced directly in "object.field" format) and have it list all of the attachments in the field, along with a functioning delete action if desired. The file-upload control was, fortunately, much easier: when I realized I could swap out most of the guts of my model object with a back-end DominoDocument (now that the OpenNTF API takes care of data conversions for me), I was able to just proxy the upload calls to it and let it do the heavy lifting.

Now, there's only one nagging problem remaining: inline image uploads in rich text controls. But still, even if I never get that part working, the file manipulation will be enough. And, importantly, I've paved the way for me to be able to do use the same controls with model objects that aren't associated with Domino at all, should I desire to do so down the line.

Loading Resources via Themes

Thu Apr 10 13:36:51 EDT 2014

Tags: themes xpages

In today's TLCC XPages webinar, Marky Roden had a great presentation discussing resources and design definitions to help speed up Designer. If you didn't catch it live, it's worth tracking it down when it's available.

I'd like to build on what he was talking about a bit by explaining how to use themes to load resources in a way that also prevents Designer slowdown while having the secondary benefit of being clean design (with some caveats that I'll get into).

If you're not familiar with themes, they're the oddball XML design element introduced alongside XPages, and they serve a number of purposes. One of them, which Tim Tripcony explained well is that you can use them to set properties of your XPages controls - so if you want to, say, apply the CSS class of "btn" to all buttons, you can use a Theme to do that instead of writing it into every <xp:button/> control. They can also be used, though, to load resources in much the same way as you do on an XPage or Custom Control directly. The default theme you get includes some comments that point in this direction, but they use the horrible multi-line syntax. It turns out that you can use a shorter syntax that mirrors the normal XSP markup. Say you have a resource section like this:

<xp:this.resources>
	<xp:styleSheet href="/style.css"/>
	<xp:styleSheet href="/fixes.css"/>
	<xp:styleSheet href="/ieonly.css" rendered="#{context.userAgent.IE}"/>

	<xp:dojoModule name="dojo.behavior"/>
	<xp:script src="/behavior.js" clientSide="true"/>
	
	<xp:metaData name="viewport" content="width=device-width, initial-scale=1.0"/>
	<xp:linkResource href="#{facesContext.externalContext.requestContextPath}/feed.xml" rel="alternate" type="application/rss.xml" title="RSS Feed"/>
</xp:this.resources>

To convert that to theme syntax, you just chop off all the "xp:" and "this." bits:

<theme extends="webstandard">
	<resources>
		<styleSheet href="/style.css"/>
		<styleSheet href="/fixes.css"/>
		<styleSheet href="/ieonly.css" rendered="#{context.userAgent.IE}"/>

		<dojoModule name="dojo.behavior"/>
		<script src="/behavior.js" clientSide="true"/>
		
		<metaData name="viewport" content="width=device-width, initial-scale=1.0"/>
		<linkResource href="#{facesContext.externalContext.requestContextPath}/feed.xml" rel="alternate" type="application/rss.xml" title="RSS Feed"/>
	</resources>
</theme>

Once you apply that to your app in the Xsp Properties, it will be as if you included the resources on the XPage directly, but you avoid the Designer slowdown problem and the XSP load time itself will be (imperceptably) faster. As an important bonus, it separates the concerns nicely: the theme's job is to handle describing the client-only stuff, and accordingly you get to delete some code from your XPages. One less thing to worry about during normal development.

Now, as I mentioned, there are some caveats, and themes are not appropriate in all cases.

  • Most importantly, the resources loaded by themes are not available during programmatic page load. This is not normally a big deal, since the page loading doesn't care about CSS files or client JavaScript anyway, but it DOES matter for two types particularly: SSJS libraries and property bundles. If you have code that needs to access them during initial processing (either in ${}-bound properties or in places like before/afterPageLoad events), you should include them on the XPage directly.
  • There's no good UI for modifying themes. You kind of have to just know how they work. I believe there's a thorough treatment in Mastering XPages, but other than that you're on your own until you learn.
  • Since Designer doesn't load them, it also doesn't help you when specifying style classes. If you're in the habit of using the "Style" properties pane to pick classes from linked stylesheets, this will break that.

But even with those caveats in mind, I'm a huge fan of themes (for this and other uses) and use them constantly. The code cleanliness, performance improvements, and structural advantages are well worth the caveats.

An XPage As A Tree

Tue Apr 01 23:05:41 EDT 2014

Tags: xpages plato

I'd like to take a blog post or two to discuss an aspect of XPages that isn't directly applicable to normal XPage development, but which can be a conceptual shift that causes a lot of things to fall into place: the tree nature of an XPage. This has nothing to do with Java, SSJS, beans, or JSF phases. It barely even has much at all to do with web pages.

You've likely heard that XPages aren't really XML - that they're actually Java, created from the "simpler" XML you write. This is true, and it drives home the important point that an NSF is now best viewed as a Java "enterprise" application. However, this isn't actually the best way to look at it; the best way is that an XPage is a tree of objects.

If you've had a Computer Science education, you'll know the tree as one of the foundational data structures used in almost all programming. If you haven't and you're not familiar with trees, it will pay tremendously to learn a bit about it - most sources I've found are either painfully mathy (like Wikipedia's article) or tied to the instruction of a specific language (often a Lisp dialect), but this page of a Python tutorial looks promising. This tree nature helps explain why XML is the tool of choice for writing XPages: XML is a tree-description language.

So an XPage is a tree. Specifically, it's a tree of user-interface objects, each of which contains methods, properties, events, and zero or more child objects. Let's take an example XPage, using ExtLib controls because they are more descriptive:

<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core"
	xmlns:xe="http://www.ibm.com/xsp/coreex"
	pageTitle="Test Page">

	<xe:formTable formTitle="Some Form">
	
		<xe:formRow label="Field One">
			<xp:inputText/>
		</xe:formRow>
		
		<xe:formRow label="Field Two">
			<xp:radioGroup>
				<xp:selectItem itemLabel="Foo"/>
				<xp:selectItem itemLabel="Bar"/>
			</xp:radioGroup>
		</xe:formRow>
		
	</xe:formTable>
	
	<xp:button value="Click Me"/>
	
</xp:view>

Ignore your default impulse to think "ah, I know what that looks like!". At this point, it doesn't "look like" anything, other than:

  • UIViewRootEx2 (pageTitle="Test Page")
    1. UIFormTable (formTitle="Some Form")
      1. UIFormLayoutRow (label="Field One")
        1. XspInputText
      2. UIFormLayoutRow (label="Field Two")
        1. XspSelectOneRadio
          1. UISelectItemEx (itemLabel="Foo")
          2. UISelectItemEx (itemLabel="Bar")
    2. XspCommandButton (value="Click Me")

The fact that it will eventually be rendered as a OneUI HTML form table won't come into play until renderers are involved (which will be a likely followup post). For now, it's best to think of this as a hierarchical description of a user interface in the abstract. At this point, the only thing standing between the XML code above and looking like this is a nontrivial amount of work:

Okay, the XML code and the related objects don't say anything about the appearance of the resultant interface, so what do they say? They define the state and behavior of the UI and how it responds to user interaction. Again, this will almost definitely consist of an HTML page and the user clicking in the browser to send POST requests to the server, but this is still entirely incidental to the XPage. Conceptually, it's all about a handful of UI controls in a hierarchy, with properties, methods, and events, each looking after itself and, as requested, producing its children. The text box's job is to know that it can contain a string value along with potential hooks for validators, converters, and event handlers - that's it. Nothing about the specifics of HTML or JavaScript.

This also helps conceptualize a specific user's page state. When a user starts interaction with a page, what happens is that the server tracks down the page root class that corresponds with the requested page and asks it to create itself and any applicable children. So you end up with any number of these object trees floating around, each corresponding to an individual interaction session (in our case, a loaded page in a browser tab, but you could also think of it as a loaded process of an application). Interacting with the page consists of calling methods on these objects - setValue, click (which is weird), etc. - and causing them to respond and mutate appropriately.

So where does the HTML come in? That's a topic for another post, but the fact that it hasn't entered yet is the point here: XPages are trees of active objects describing a Platonic ideal interface and the framework has a secondary effect of producing HTML.

My Current Model Framework, Part 2: An Example

Fri Feb 21 15:27:19 EST 2014

Tags: xpages mvc
  1. My Current Model Framework, Part 1
  2. My Current Model Framework, Part 2: An Example

To go along with the updated release of my Scaffolding project yesterday, I've created an example database created with that template and containing a basic use of my model framework:

model-framework-example.zip

This demonstrates a few things about the framework:

  • Setting up the two classes that go with each object type - the object itself (e.g. model.Department) and the "manager" class that handles fetching objects and collections (e.g. model.DepartmentManager) and adding the collections to faces-config.xml.
  • Tying the manager class to the views in a database using views with a common prefix. In this case, they point to the current database, though I recommend storing the data in another DB.
  • Adding relations between objects. A Person object is tied to a given Department by way of the getDepartment method, while a Department lists its people via the getPeople method.
  • Using the collections on an XPage with xp:viewPanel controls, including showing "columns" that are not in the view - the objects automatically fetch from the document when the column isn't present (which is inefficient but useful sometimes).
  • Attaching a model object to an XPage using a xp:dataContext and using a controller class to save it.

I have a few ideas in mind to improve day-to-day use of the model framework (for example, I may set up proper data sources for individual objects so they can tie into the normal save events), so they may change over time, but this is how I develop most apps today.

Data separation with an asterisk

Thu Jan 09 09:50:59 EST 2014

Tags: xpages

A little while ago, I converted over to the practice of separating your XPage app from the NSF that contains the data, even when it's going to be a one-to-one mapping. This confers a number of advantages in the areas of easier testing, easier development, programming discipline, and security. One of my favorite aspects is the ability to do this on your data DB:

Don't allow URL open

This alone adds tons of security: no more worrying about ?ReadViewEntries, hiding your views from the web, errant forms allowing too much access, and so forth. Once you have that checked and you funnel your access through a separate app, the surface of vulnerability is much, much smaller.

But there's a hitch.

I discovered this the other day, and it's only a problem in mixed web/Notes environments: when you have that option enabled, you can't see native rich text content from a document in your XPage. I say "native" to refer to the traditional Composite Data format of rich text; MIME content is fine. The reason for this appears to go down to the C-API level, where setting that flag seems to entirely bar the HTML conversion functions from working with that database.

My guess is that that was deemed the most expedient way to implement the feature at the time, but it's problematic now. I can think of a couple ways to deal with it:

  • Don't use native rich text. If you're writing whole-cloth XPages apps, you don't have to worry about this at all, since MIME/HTML content is still fine. If you have a mixed app and you're fine with some ugly-looking text in the Notes client, you can turn on the field option to store the data as MIME even in the client.
  • Store the app on separate servers. To maintain the security benefits, you could store the data database on a different server, one not publicly accessible via HTTP.
  • Disable that option and use web rules instead. You should probably be able to achieve a lot of the security benefits by substituting all HTTP requests for the data database with blank pages, though that requires coordination of the server config with every DB like this, which is a drag. Also don't forget __$replicaid.nsf URLs.

Fortunately, the fact that it only affects conversion from native rich text to MIME means the impact is limited, but it's still a caveat to data separation.

A Quick, Dirty, and Inefficient @ManagedBean Implementation in XPages

Tue Dec 17 09:24:31 EST 2013

Tags: xpages java

By now, most of us are pretty familiar with the process for adding managed beans to an XPages app: go to faces-config.xml and add a <managed-bean>...</managed-bean> block for each bean. However, JSF 2 added another method for declaring managed beans: inline annotations in the Java class. This wasn't one of the features backported to the XPages runtime, but it turns out it's not bad to add something along these lines, and I decided to give it a shot while working on the OpenNTF API.

The first thing that's required is a version of the annotations that ship with JSF 2. Fortunately, annotations in Java, though ugly to define, are very simple, and each one involves only a few lines of code and can be added to your NSF directly:

javax.faces.bean.ManagedBean

package javax.faces.bean;

import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;

@Retention(value=RetentionPolicy.RUNTIME)
public @interface ManagedBean {
	String name();
}

javax.faces.bean.ApplicationScoped

package javax.faces.bean;

import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;

@Retention(value=RetentionPolicy.RUNTIME)
public @interface ApplicationScoped { }

javax.faces.bean.SessionScoped

package javax.faces.bean;

import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;

@Retention(value=RetentionPolicy.RUNTIME)
public @interface SessionScoped { }

javax.faces.bean.ViewScoped

package javax.faces.bean;

import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;

@Retention(value=RetentionPolicy.RUNTIME)
public @interface ViewScoped { }

javax.faces.bean.RequestScoped

package javax.faces.bean;

import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;

@Retention(value=RetentionPolicy.RUNTIME)
public @interface RequestScoped { }

javax.faces.bean.NoneScoped

package javax.faces.bean;

import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;

@Retention(value=RetentionPolicy.RUNTIME)
public @interface NoneScoped { }

The more-complicated code comes into play when it's time to actually make use of these annotations. There are a couple ways this could be handled, and the route I took was via a variable resolver. Note that this code requires my current branch of the OpenNTF API, though a mild variant would work with the latest M release:

resolver.AnnotatedBeanResolver

package resolver;

import java.io.Serializable;
import java.util.*;
import javax.faces.context.FacesContext;
import javax.faces.el.EvaluationException;
import javax.faces.el.VariableResolver;
import javax.faces.bean.*;

import org.openntf.domino.*;
import org.openntf.domino.design.*;

import com.ibm.commons.util.StringUtil;

public class AnnotatedBeanResolver extends VariableResolver {

	private final VariableResolver delegate_;

	public AnnotatedBeanResolver(final VariableResolver resolver) {
		delegate_ = resolver;
	}

	@SuppressWarnings("unchecked")
	@Override
	public Object resolveVariable(final FacesContext facesContext, final String name) throws EvaluationException {
		Object existing = delegate_.resolveVariable(facesContext, name);
		if(existing != null) {
			return existing;
		}

		try {
			// If the main resolver couldn't find it, check our annotated managed beans
			Map<String, Object> applicationScope = (Map<String, Object>)delegate_.resolveVariable(facesContext, "applicationScope");
			if(!applicationScope.containsKey("$$annotatedManagedBeanMap")) {
				Map<String, BeanInfo> beanMap = new HashMap<String, BeanInfo>();

				Database database = (Database)delegate_.resolveVariable(facesContext, "database");
				DatabaseDesign design = database.getDesign();
				for(String className : design.getJavaResourceClassNames()) {
					Class<?> loadedClass = Class.forName(className);
					ManagedBean beanAnnotation = loadedClass.getAnnotation(ManagedBean.class);
					if(beanAnnotation != null) {
						BeanInfo info = new BeanInfo();
						info.className = loadedClass.getCanonicalName();
						if(loadedClass.isAnnotationPresent(ApplicationScoped.class)) {
							info.scope = "application";
						} else if(loadedClass.isAnnotationPresent(SessionScoped.class)) {
							info.scope = "session";
						} else if(loadedClass.isAnnotationPresent(ViewScoped.class)) {
							info.scope = "view";
						} else if(loadedClass.isAnnotationPresent(RequestScoped.class)) {
							info.scope = "request";
						} else {
							info.scope = "none";
						}

						if(!StringUtil.isEmpty(beanAnnotation.name())) {
							beanMap.put(beanAnnotation.name(), info);
						} else {
							beanMap.put(loadedClass.getSimpleName(), info);
						}
					}
				}

				applicationScope.put("$$annotatedManagedBeanMap", beanMap);
			}

			// Now that we know we have a built map, look for the requested name
			Map<String, BeanInfo> beanMap = (Map<String, BeanInfo>)applicationScope.get("$$annotatedManagedBeanMap");
			if(beanMap.containsKey(name)) {
				BeanInfo info = beanMap.get(name);
				Class<?> loadedClass = Class.forName(info.className);
				// Check its scope
				if("none".equals(info.scope)) {
					return loadedClass.newInstance();
				} else {
					Map<String, Object> scope = (Map<String, Object>)delegate_.resolveVariable(facesContext, info.scope + "Scope");
					if(!scope.containsKey(name)) {
						scope.put(name, loadedClass.newInstance());
					}
					return scope.get(name);
				}
			}
		} catch(Exception e) {
			e.printStackTrace();
			throw new RuntimeException(e);
		}

		return null;
	}

	private static class BeanInfo implements Serializable {
		private static final long serialVersionUID = 1L;

		String className;
		String scope;
	}
}

faces-config.xml

<faces-config>
	...
	<application>
		<variable-resolver>resolver.AnnotatedBeanResolver</variable-resolver>
	</application>
</faces-config>

This is not efficient! What happens is that, whenever a variable is requested that the main resolver can't find (or is null), the code goes through each Java class defined in the NSF (not in plugins, the filesystem, or in attached Jars) and checks if it contains the @ManagedBean definition. Fortunately, the worst of this is only incurred the first time (according to the duration of applicationScope), but that one-time process is still something that would get linearly slower as the number of Java resources in your app increases.

Now, there are still other features of the actual JSF 2 implementation that this doesn't cover, particularly @ManagedProperty, but it can still be useful on its own. In practice, it lets you ditch the faces-config declaration and write your beans like this:

package bean;

import javax.faces.bean.*;
import java.io.Serializable;

@ManagedBean(name="someAnnotatedBean")
@ApplicationScoped
public class AnnotatedBean implements Serializable {
	private static final long serialVersionUID = 1L;

	// ...
}

I plan to give this a shot in the next thing I write to see how it works out. And regardless of future use, it's a nice example of the kind of thing that gets easier with the OpenNTF API.

The Curse of TMTOWTDI

Tue Nov 26 15:41:24 EST 2013

Tags: xpages

TMTOWTDI is a Perl-ism, standing for "there's more than one way to do it". It is often a blessing, particularly in the context of Perl, in that it results in multiple synonymous ways to accomplish the same task, each of which fits best in a different context.

However, as you might expect, this is a double-edged sword that can lead to messy code and a difficult learning curve. It's this negative context that bites the XPages platform in a number of ways.

One way is relatively benign but still a pernicious issue: the Java base of XPages contains a plethora of may-or-may-not-be-important choices to make, such as Vector vs. ArrayList vs. LinkedList. Though it doesn't really matter in small cases which you pick, you still have to make that choice, and some of them (Vector) end up hurting you as the scale increases. That's not something you have to pay too much attention to in most other high-level languages, where there's a standard Array type that does its job well.

The larger issue is the massive divergence of ways that even normal XPages apps can be structured. Do you use one main XPage and then put everything either in other XPages brought in with "include page" or in custom controls (with or without Dynamic Content), or do you do a lot of top-level XPages the user visits directly? If the latter, do you still break out chunks of individual-page functionality to custom controls for readability? How about data - do you store your data in the same NSF as the code, in separate NSFs, or in a non-Domino database? Do you use stock controls and make a OneUI app, toss all that OneUI/Dojo stuff aside and build on Bootstrap/Foundation/etc. and jQuery, or take a hybrid approach? For back-end code, do you use inline SSJS, primarily SSJS libraries, or Java? If you base it on Java, how? Controller classes, object data sources, phase listeners, variable resolvers, core business-logic classes called by SSJS? If you put your data access in Java classes, do you build them to work alongside stock XSP Domino data sources for performance, or build on the Domino API?

The list goes on.

There are likely as many divergent approaches to XPages development as there are XPages developers. While this is good in the sense of feeling out the contours of the platform's capabilities, it makes it difficult to get started and to collaborate with other projects.

As you might expect, I have my own opinions on each question, but some of them are still fluid. This is a natural by-product of a still-relatively-young platform with a strange history, no clear single voice, and a primary target market of ancient systems with massive inertia. In time, I hope that this settles down to a good general-purpose set of techniques (and I hope to influence that). Each application I make feels more coherent than the last, and the differences between them are gradually diminishing. The more consistent the methods (assuming they're good), the better for the platform and the developers on it.

Building My App On Custom Renderers

Sun Nov 24 20:41:15 EST 2013

Tags: xpages java
  1. I Am Terribly Excited About Custom Renderers
  2. Building My App On Custom Renderers

A little while ago, I mentioned how I have become smitten with custom renderers - Java classes that take XPage components like application layouts and widgets and render them in custom HTML. Though I've had a lot of other things to work on in between then and today, my ardor for the subject has not dimmed. So today, I mostly completed the process for my task-tracking app.

At a high level, the app is a fairly typical web app built using the popular-for-good-reason Ace - Responsive Admin Template theme from WrapBootstrap. I'm a huge fan of using these pre-built themes - in addition to handling the visual design, they tend to come packaged with a set of specialized widgets and jQuery plugins that are made to work well as a cohesive whole. However, the down side is that you have to write your code specifically targeting the framework, even moreso than stock Bootstrap (and for this reason, I couldn't directly use Bootstrap4XPages). This makes it a perfect candidate for renderer-ification, as most of the components I'm using - the layout, widget boxes, view panels/data tables, form layouts - have direct analogs in either the stock XPage runtime or in the OneUI-based components in the Extension Library.

So what I have done is to take the basic structure of the Bootstrap4XPages project (and more than a smidge of its code) and used it to build custom renderers for the components I need, packaged into an OSGi plugin:

Project Explorer

It's something of a mess of code, and parts of it have been a pain to deal with, but for the most part it's been a process of looking at the original renderer source in the ExtLib and Bootstrap4XPages and then either copying and tweaking, subclassing and overriding, or implementing from scratch. I've run into some bothersome limitations in the stock objects as well, particularly when it comes to areas where I need to apply CSS classes to elements not represented in the components, like the FontAwesome-based icon classes for the nav bar or specific classes for widget title bars and the body area. Since these components also lack convenient "attr" properties, I've resorted to ugly hacks: I've co-opted the "imageAlt" property of the tree nodes to trigger creation of a Bootstrap-style "i" tag with a class for the navigator, and for the widgets I just made the styleClass apply only to the header and added an ugly exception to look for "no-padding" classes and apply them to the body instead. It's not a pretty process sometimes.

But the results are pretty interesting. By using standard components, I have an app that can swap readily between very different themes (with some rough edges for areas where code is still Ace-specific):

Ace

OneUI v2.1

OneUI v3.0.2

All of this does, of course, raise an important question. Namely: why bother? The original way I implemented it - by writing the Bootstrap layout HTML by hand, tweaking individual classes here and there, and forgoing the OneUI-based components - worked just fine, and it took less work. And really, I'm not terribly likely to suddenly decide that I want to switch this app from Ace to OneUI or stock Bootstrap. There are a couple important benefits to this approach that apply both to me personally and to others who may consider doing the same thing:

  • Practice. This seems odd to put first, but it's vitally important. Writing these renderers has improved my knowledge of the platform a great deal. It's given me a reason to delve further into the Extension Library source, to improve my OSGi-plugin-writing knowledge, and to familiarize myself with an easily-overlooked part of the JSF stack. Like many things with the plumbing of XPages, it's more of a PITA than it absolutely needs to be, but, if you do it, you'll come out a much better developer.
  • Focus. Now that I have almost all of the code that's specific to Bootstrap or Ace out of my XPage app, the remaining code has a sharpened focus on the task at hand. Now I use standard widget, pager, and view controls just like in any run-of-the-mill app and hook them up the same way, without having to worry about how it's going to look. Time spent on the problem at hand is time spent well.
  • Maintainability. During my transition from hardcoded HTML in the XSP source to custom renderers, I also jumped from Ace 1.0 to 1.2, which included an upgrade to Bootstrap 3. For this task, that meant I had to learn about a number of the differences between Bootstrap 2 and 3, as well as tweaking lots of little CSS classes and HTML structures across every page of the app. Next time, though, when Ace version 2 running with Bootstrap 4 comes out, I only need to change the theme, entirely outside of the app, and everything else will come along for the ride (mostly).
  • Less Code in the NSF. This is a corollary to the last bullet point, but is important as its own concept. By moving the theme and supporting files (like the CSS and JS assets themselves) out to a plugin, there's less cruft to carry around in the NSF itself and, more importantly, less stuff to keep up-to-date when used in common across multiple apps. This is not unique to this technique (you can move the resource files out without doing renderers, and you can bundle common custom controls that way too), but it's a valuable side effect.
  • Interoperability. This doesn't matter so much to me currently, since I'm the only one working on this app, but it will matter later, and it would matter immediately to any larger company. If each of your apps, regardless of the final appearance, is built using the standard XPage and ExtLib controls, that means you don't need to familiarize every developer working on it with the CSS framework of your choice, whether it be Bootstrap, your company's home-grown HTML, or something else entirely. You can have one person writing the themes and have the rest of the team go about building their apps by the book (or, rather, books).
  • Flexibility. As demonstrated by my screenshots above, this technique brings tremendous flexibility. In this specific case, I knew what theme I wanted to use when I started writing the app, but in the future I won't need to. Whenever I'm struck by inspiration for a new app, I can start writing code immediately (dressing it up in Ace or OneUI) and then decide on and buy another theme later without the hassle of going back into my functional XPages to replace and tweak a bunch of CSS classes.

It's a pity that there's such overhead in the initial work of writing custom renderers, but it will get gradually easier over time, now that Bootstrap4XPages is available to reference. And still, hassle and all, I believe this to be the best way forward for organized XPage development, and represents one of the platform's distinct advantages over others (well, the others that aren't JSF). I strongly encourage everyone to look at least a little into the idea. And I also, as always, encourage you to contact me at will either to pick my brain or commission my company to help renderer-ify or otherwise improve your apps.

My Current Model Framework, Part 1

Sun Nov 17 12:19:18 EST 2013

Tags: xpages mvc
  1. My Current Model Framework, Part 1
  2. My Current Model Framework, Part 2: An Example

As I do from time to time, I've recently been taking another stab at a standard model framework for my XPage apps. My latest one has been proving its worth in a couple apps I've been writing lately, and I'd like to go over the general goals and advantages of the way it works.

The framework is focused on a couple main ideas:

  • Low Overhead. The Java language requires a certain amount of overhead to get anything done, but I want to minimize that. The task of defining a new model object consists of two classes: one for the object itself and one for the collection manager (e.g. connecting to Domino views), and each is geared towards only writing the code required to describe the necessities.
  • Embracing XSP. The framework is intended for writing XPages apps, and so I want to make sure it works smoothly with EL and standard controls like inputs, repeats, and view panels.
  • Embracing document databases. Since I use almost entirely a document database for storage, I want to make sure to take advantage of that, and that primarily means flexibility in data modeling. Accordingly, data objects allow arbitrary field access, with any strictures from your model class layered on top of that.
  • Embracing Domino. Since the document database I use is Domino, I want to make sure its peculiar features are retained as well: reader/author fields, arbitrary object storage via MIME, (ideally) RT and attachments, full-text search, categorized views, and so forth.
  • Conceptually simple. Though there's some ugly code involved in parts like the Domino view wrapper, the conceptual layout of the framework is kept very simple with few moving parts, making it easy to recognize the function of each aspect at a glance, whether when building it, when looking at another's code, or when returning to your own code months down the line.

For this post, I'll give an example of a model class, while later I'll go into the collection managers and some real-use examples. This is what a basic model class looks like:

package model;

import java.util.Arrays;
import java.util.Collection;
import java.util.List;

import org.openntf.domino.*;

import frostillicus.model.AbstractDominoModel;
import frostillicus.model.DominoColumnInfo;

public class Request extends AbstractDominoModel {
	private static final long serialVersionUID = 589766180414699322L;

	public Request(final Database database) {
		super(database);
		setValue("Form", "Request");
	}

	public Request(final ViewEntry entry, final List<DominoColumnInfo> columnInfo) {
		super(entry, columnInfo);
	}

	public Request(final Document doc) {
		super(doc);
	}

	@Override
	protected Collection<String> nonSummaryFields() {
		return Arrays.asList(new String[] { "Body" });
	}

}

There's more Java overhead than I'd like, but that's the name of the game. Fortunately, each method has a role and can be used as hooks for differing behavior.

The first constructor is the one used to create a new document of the appropriate type in the provided database, so it sets the form field appropriately - it would also set any other appropriate defaults. The second constructor is used when traversing a view - model objects pull values from view entries when available, but also transparently pass along requests to the document when the requested field isn't in the view. The final constructor wraps an existing document.

The "nonSummaryFields" method is one of the hooks available to define the Domino data model, in this case providing a list of fields that should be flagged as non-summary when written, to help work around Domino limits. I also have hooks, not needed here, for authors/readers/names fields and form-style query/post events.

Conspicuously absent are any field definitions. By default, model objects act much like XPage DominoDocuments, passing setValue and getValue calls on to the underlying document more or less directly. However, the framework allows for hooks here by writing getters and setters in the standard format, and their presence changes the behavior of getValue and setValue. When just a getter is present, the field becomes read-only; when a setter is present, the field can be written, but now does data-type validation. The getters and setters allow for arbitrary validation and additional behavior for specific fields (say, changing related fields when one changes) without having to write out giant blocks of "getFoo() { return foo; }" and "setFoo(String foo) { this.foo = foo; }", and this has been a huge win for me. Even though Eclipse helps with the initial creation, the code still has to exist, and it takes a cognitive toll.

I also use these overriding methods to create relations between models through their collection managers. For example:

public Client getClient() {
	String clientId = (String)getValue("ClientID");
	if(clientId == null || clientId.isEmpty()) {
		return null;
	}
	return (Client)JSFUtil.getClientManager().getValue(clientId);
}

This allows for XPage El bindings like #{task.client.name} - no extra panels with data sources, no inline lookup code.

Next time, I will go into the collection-manager side, which provides fairly high-performance access to views while remaining simple and flexible.

I Am Terribly Excited About Custom Renderers

Tue Nov 12 19:56:45 EST 2013

Tags: xpages java
  1. I Am Terribly Excited About Custom Renderers
  2. Building My App On Custom Renderers

It's much to my chagrin that it took me until this past weekend to properly dive into custom renderers. I had seen them before, primarily when working with mypic, but always avoided building my own, primarily because of what a hassle they are to write. However, I'm now convinced that the hassle is worth it, at least for some primary uses.

If you're not familiar with custom renderers, they're sort of an odd beast, but they're the kind of thing where the concept eventually "clicks" in your mind at some point. In XPages, a component (such as a link, a panel, or an application layout) consists of two main aspects: the structural code that defines the properties of the component and the renderer that handles actually outputting the HTML for the browser. What adding your own custom renderers allows is for you to continue to use the same XSP markup to define the control, yet have very different results.

The best example of this (and the source of most of my education) is the recent Bootstrap4XPages project. It takes a standard OneUI applicationLayout-based app and styles it using Bootstrap markup instead of OneUI, even though the XSP markup is the same. In the normal case, this just means that you can reuse some of the basics when building a Bootstrap-targeted app. And that's good enough on its own: it saves you tons of work.

But the more important aspect is that it lets you focus much more on solving the task at hand and much less on writing to the layout framework.

This is a huge advantage for XPages (and any framework with this separation). As low-key as Bootstrap is (and other modern CSS frameworks are), you still have to structure your markup and sprinkle class names around to match its expectations. With a really fleshed-out set of custom renderers, though, you could potentially not care at all about the resultant UI framework when writing your app and instead focus on using stock or ExtLib controls to solve the problem directly, letting the theme+renderers do the work.

In reality, there'll be a bit of leaking (for example, using Bootstrap-friendly FontAwesome classes instead of images doesn't really fit with standard IBM controls), but I aim to minimize that as much as possible in my future projects. My goal is to use applicationLayout and standard controls for as much as I can, and instead writing a set of custom renderers for each primary theme I want to apply.

I'll have more to write about this later, but for now, the TL;DR version is: custom renderers have the potential to dramatically improve the way XPages apps are written, as long as you're willing to do the initial Java legwork.

URLs in XPages

Tue Oct 01 20:28:59 EDT 2013

Tags: urls xpages

The topic of URLs came up today in a chat and in the most recent NotesIn9, and I think this is an area that deserves some particular attention. When designing any web app, it's easy to fall into some traps with URLs, and writing XPages code juggles things around a bit - some things are much easier than usual, some are insidiously harder.

At a base level, there are four main types of URLs you'll use when writing a web app, distinguished by how they begin (don't quote me on the names, or even whether or not they're technically called "URLs"):

  • Full-protocol URLs beginning with a protocol followed by a colon, such as https://frostillic.us/f.nsf/Home.xsp
  • Protocol-relative URLs beginning with a double-forward-slash, such as //frostillic.us/f.nsf/home.xsp (these are fairly rare)
  • Server-relative URLs beginning with a forward-slash, such as /f.nsf/home.xsp
  • Page-relative URLs beginning with basically anything else, such as home.xsp

As a general rule, page-relative URLs are only a good idea if you are absolutely certain of the structure of your app relative to the URL the browser is using. The tricky part here is that it's very easy to write such a URL that works when you test it (say, pointing from one XPage to another with a URL like <xp:link value="otherXPage.xsp" text="Click me"/>), but fails in other situations. For example, see how such a URL would work when visiting each of these URLs, which all point to the same XPage:

Yes, the last one is legal - you can get to the stuff after the ".xsp" via facesContext.getExternalContext().getRequest().getPathInfo().

Fortunately, the XPage environment provides a nice solution to this problem: when using XSP elements, it converts the third type (server-relative URLs) from being relative to the server to be relative to the app. This is almost always1 what you want. So if I write a link as <xp:link value="/otherXPage.xsp" text="Click me"/>, it will work correctly in all of those situations.

Note the opening caveat: you have to be using XSP elements. Fortunately, this covers all WYSIWYG use and pretty much everything you're going to do elsewhere. It's rare that you actually want a stock-HTML <a/> instead of an <xp:link/>, and this includes resource URLs in an XPage, custom control, or theme. So, as a rule, you should almost always preface your in-app links with a forward slash to piggyback on this feature.

There are two cases that need special treatment: when you want to link out of your app from an XPage element and when you want to link into your app from a non-XPage element. Fortunately, there are two bits of non-obvious code that will do these jobs easily and reliably:

  • To break out of the app-relative "jail" in an XPage element, prefix your URL with "/.ibmxspres/domino/". For an explanation as to what this stands for and a list of similar prefix URLs, read Mastering XPages. For now, just know that that prefix means "get me out of here". So to link to, say, the directory, you could write a link like <xp:link value="/.ibmxspres/domino/names.nsf" text="Domain's Directory"/>.
  • To reliably reference the current app from a non-XPage link, use this bizarre EL formulation: ${facesContext.externalContext.requestContextPath}. What that gets you is basically like @WebDbName prefixed with a slash. So, for example, if you wanted to reference a video attached as a file resource, you could do something like <video src="${facesContext.externalContext.requestContextPath}/someVideo.mp4" />.

Keep in mind that the special XPage behavior applies only to when you're writing XSP markup (or otherwise pulling URLs into XPage elements); it doesn't apply to, for example, rich text or other HTML brought into the page.

And on a final, self-aggrandizing note, the "mentoring" offer in that NotesIn9 applies to me and my company. Feel free to contact us or get in touch with me directly if you're ever in need.

1. Except when writing CSS files. In CSS, URLs are relative to the location of the CSS file. This is usually just fine, since you, not the browser, control the URLs you use to refrence the CSS file. It gets tricky when you use resource aggregation, though.

Couchbase Followup and a Github Repository

Tue Jun 18 15:27:33 EDT 2013

Since my initial tinkering with Couchbase, I've been idly thinking about improvements to the data sources and potential uses I may have for it.

As for the latter, I think I may give a shot to implementing IKSG's new project-tracking system in it. I'll lose out on reader and author fields, but otherwise it'd be a pretty direct translation between NSF or Couchbase data stores.

For the former, I decided to drop my development code in a Github repository and start tracking my to-dos there:

https://github.com/jesse-gallagher/Couchbase-Data-for-XPages

The test database stored in that repository is the same as the one from my previous post with two main exceptions:

  1. I added a new parameter to XSPCouchbaseConnection to specify whether or not it should, when acting as a DataObject, treat the bucket as a document store. Since "documents" in Couchbase are just keys with JSON strings as their values, it's a tricky matter to detect intent... thus, I decided I may as well just make it a config option.
  2. I added a second example connection in faces-config.xml: clusterScope. Since the Java API automatically serializes and deserializes objects to store in the bucket, it makes for a perfect backing store for a clustered XPage app's persistent clusterScope.

I can't say how much time I'll have to dedicate to the improvements I want to make in the short term, but I think even the current form should be enough for me to build apps upon. I'm very interested to see if I can come up with tricks to use that are more difficult or are very inefficient in an NSF.

Couchbase Data Sources for XPages

Sun Jun 16 23:25:51 EDT 2013

The other day, I attended a Couchbase Developer Day in Philly. If you're not familiar with Couchbase, it's one of the young crop of NoSQL databases, and more specifically a document database and a conceptual cousin to Domino by way of Damien Katz. It's this similarity that makes it so interesting to me - Couchbase's still-living predecessor CouchDB started life as Notes built from the ground up for the web and the heritage is clear. Couchbase itself ends up being like if you melted down the cores of NSF/NIF and poured the result on top of a speedy key-value store.

That all makes Couchbase a perfect candidate for some XPage tinkering. Ignoring the foundational key-value aspects of Couchbase, the JSON document storage and views in 2.0 make for perfect analogues to the Domino data sources in XPages. To wit:

<xp:this.data>
    <cb:couchbaseView var="beersView" connectionName="couchbasePelias"
        designDoc="beer" viewName="by_name" />
    <cb:couchbaseDocument var="beer" connectionName="couchbasePelias"
        documentId="${param.documentId}"/>
</xp:this.data>

I've set up a demo app on one of my test servers (don't be surprised if that link is periodically unavailable). It demonstrates one of the views in the beer sample database that comes with Couchbase (it's like fakenames.nsf, but more intoxicating), with each row providing a link to edit the document in the DB, though editing is disabled for Anonymous users. The app also contains pages to show the applicable Java classes and support elements. The sources are nowhere near complete works - the view source lacks any knowledge of keys, grouping, etc., while the document source doesn't allow for the creation of new documents, nor is there any representation of non-document values.

Pretty much every concept you work with in Couchbase has a corresponding concept in Domino, but with a bit of slant, like a different dialect (for example, they call it a Royale with Cheese). A table may suit this explanation best:

 

Domino Couchbase Notes
Server Node + Cluster Couchbase has a strong focus on splitting up the job of housing databases into many servers for reliability and scalability reasons. Where Domino's clustering is RAID 1, Couchbase's is more like RAID 5.
Replication XDCR Unlike Domino's replication, Couchbase uses a set of rules to determine which of two conflicting documents lives and which dies unceremoniously.
Database Bucket The Couchbase name is a bit more evocative.
View Design Doc + View Unlike Domino1, Couchbase stores multiple views per design document, and it appears to be a sort of arbitrary art to decide how you want to split your views up among multiple design documents.

 

I'm tempted to find an excuse to build something using Couchbase as a back-end, just to see how its performance stacks up. It seems like my years of built-up NSF habits would serve me well, as many of the programming models and tradeoffs are similar, except that Couchbase views are probably consistently fast without having to cheat.

If I come back to the data sources, I'll probably flesh them out to support creating new documents, properly handle dynamically-computed properties, provide some control over view parameters (maybe including the geoboxing stuff built in), support Couchbase's document-save-conflict detection, and add a data source for basic key/value pairs. And if I do, that could provide a good excuse for me to package them up as a proper OSGi plugin, which I've so far avoided bothering with. We shall see.

1. This is a lie. The Couchbase way is actually very similar to Domino: Domino view design notes contain one or more collations, which correspond to the main index and any click-to-sort column directions. However, each of Couchbase's views inside its design document can produce wildly different data, not just resorts of the same selection, and so the fact that they're similar technically isn't very useful.

More On "Controller" Classes

Mon Jan 07 19:39:09 EST 2013

Tags: xpages mvc java
  1. "Controller" Classes Have Been Helping Me Greatly
  2. More On "Controller" Classes

Since my last post on the matter , I've been using this "controller" class organization method in a couple other projects (including a refresh of the back-end of this blog), and it's proven to be a pretty great way to go about XPages development.

As I mentioned before, the "controller" term comes from the rough equivalent in Rails. Rails is thoroughly MVC based, so a lot of your programming involves creating the UI of a page in HTML with a small sprinkling of Ruby (the "view"), backed by a class that is conceptually tied to it and stores all of the real business logic associated with that specific page (the "controller"). XPages don't have quite this same assumption built in, but the notion of pairing an XPage with a single backing Java class is a solid one. Alternatively, you can think of the "controller" class as being like a stylesheet or client JavaScript file: the HTML page handles the design, and only contains references to functionality defined elsewhere.

What this means in practice is that I've been pushing to eliminate all traces of non-EL bindings in my XSP markup in favor of writing the code in the associated Java class - this includes not only standard page events like beforeRenderResponse , but also anywhere else that Server JavaScript (or Ruby) would normally appear, like value and method bindings. Here's a simple example, from an XPage named Test and its backing class:

Test.xsp
<p><xp:button id="clickMe" value="Click Me">
	<xp:eventHandler event="onclick" submit="true" refreshMode="partial" refreshId="output"
		action="#{pageController.refreshOutput}"/>
</xp:button></p>
	
<p><xp:text id="output" value="#{pageController.output}"/></p>
controller/Test.java
package controller;

import frostillicus.controller.BasicXPageController;
import java.util.Date;

public class Test extends BasicXPageController {
	private static final long serialVersionUID = 1L;

	private String output = "default";
	public String getOutput() { return this.output; }

	public void refreshOutput() {
		this.output = new Date().toString();
	}
}

In a simple case like this, it doesn't buy you much, but imagine a more complicated case, say code to handle post-save document processing and notifications, or complicated code to generate the value for a container control. The separation between the two pays significant dividends when you stop having to worry scrolling through reams of code in the XSP markup when you're working on the page layout, while having a known location for the Java code associated with a page makes it easier to track down functionality. Plus, Server JavaScript is only mildly more expressive than Java, so even the business logic itself stays just about as clean (moving from Ruby to Java, on the other hand, imposes a grotesque LOC penalty).

It sounds strange, but the most important benefit I've derived from this hasn't been from performance or cleanliness (though those are great), but instead the discipline it provides. Now, there's no question where business logic associated with a page should go: in a class that implements XPageController with the same name as the page and put in the "controller" package. If I want pages to share functionality, that's handled either via a common method stored in another class or, as appropriate, via class inheritance. No inline code, no Server JavaScript script libraries. If something is going to be calculated, it's done in Java.

There is one area that's given me a bit of trouble: custom controls. For example, the linksbar on this blog requires computation to generate, but it's not associated with any specific page. For now, I put that in the top-level controller class, but that feels a bit wrong. The same solution also doesn't apply to the code that handles rendering each individual post in the list, which may be on the Home page, the Month page, or individually on the Post page, in which case it also has edit/save/delete actions associated with it. For now, I created a "helper" class that I instantiate (yes, with JavaScript, which is depressing) for each instance. I think the "right" way to do it with the architecture is to create my controls entirely in Java, with renderers and all. That would allow me to handle the properties passed in cleanly, but the code involved with creating those things is ugly as sin. Still, I'll give that a shot later to see if it's worth the tradeoff.

"Controller" Classes Have Been Helping Me Greatly

Wed Dec 26 16:54:16 EST 2012

Tags: xpages mvc java
  1. "Controller" Classes Have Been Helping Me Greatly
  2. More On "Controller" Classes

I mentioned a while ago that I've been using "controller"-type classes paired with specific XPages to make my code cleaner. They're not really controllers since they don't actually handle any server direction or page loading, but they do still hook into page events in a way somewhat similar to Rails controller classes. The basic idea is that each XPage gets an object to "back" it - I tie page events like beforePageLoad and afterRenderResponse to methods on the class that implements a standard interface:

package frostillicus.controller;

import java.io.Serializable;
import javax.faces.event.PhaseEvent;

public interface XPageController extends Serializable {
	public void beforePageLoad() throws Exception;
	public void afterPageLoad() throws Exception;

	public void afterRestoreView(PhaseEvent event) throws Exception;

	public void beforeRenderResponse(PhaseEvent event) throws Exception;
	public void afterRenderResponse(PhaseEvent event) throws Exception;
}

I have a basic stub class to implement that as well as an abstract class for "document-based" pages:

package frostillicus.controller;

import iksg.JSFUtil;

import javax.faces.context.FacesContext;

import com.ibm.xsp.extlib.util.ExtLibUtil;
import com.ibm.xsp.model.domino.wrapped.DominoDocument;

public class BasicDocumentController extends BasicXPageController implements DocumentController {
	private static final long serialVersionUID = 1L;

	public void queryNewDocument() throws Exception { }
	public void postNewDocument() throws Exception { }
	public void queryOpenDocument() throws Exception { }
	public void postOpenDocument() throws Exception { }
	public void querySaveDocument() throws Exception { }
	public void postSaveDocument() throws Exception { }

	public String save() throws Exception {
		DominoDocument doc = this.getDoc();
		boolean isNewNote = doc.isNewNote();
		if(doc.save()) {
			JSFUtil.addMessage("confirmation", doc.getValue("Form") + " " + (isNewNote ? "created" : "updated") + " successfully.");
			return "xsp-success";
		} else {
			JSFUtil.addMessage("error", "Save failed");
			return "xsp-failure";
		}
	}
	public String cancel() throws Exception {
		return "xsp-cancel";
	}
	public String delete() throws Exception {
		DominoDocument doc = this.getDoc();
		String formName = (String)doc.getValue("Form");
		doc.getDocument(true).remove(true);
		JSFUtil.addMessage("confirmation", formName + " deleted.");
		return "xsp-success";
	}

	public String getDocumentId() {
		try {
			return this.getDoc().getDocument().getUniversalID();
		} catch(Exception e) { return ""; }
	}

	public boolean isEditable() { return this.getDoc().isEditable(); }

	protected DominoDocument getDoc() {
		return (DominoDocument)ExtLibUtil.resolveVariable(FacesContext.getCurrentInstance(), "doc");
	}
}

I took it all one step further in the direction "convention over configuration" as well: I created a ViewHandler that looks for a class in the "controller" package with the same name as the current page's Java class (e.g. "/Some_Page.xsp" → "controller.Some_Page") - if it finds one, it instantiates it; otherwise, it uses the basic stub implementation. Once it has the class created, it plunks it into the viewScope and creates some MethodBindings to tie beforeRenderResponse, afterRenderResponse, and afterRestoreView to the object without having to have that code in the XPage (it proved necessary to still include code in the XPage for the before/after page-load and document-related events).

So on its own, the setup I have above doesn't necessarily buy you much. You save a bit of repetitive code for standard CRUD pages when using the document-controller class, but that's about it. The real value for me so far has been a clarification of what goes where. Previously, if I wanted to run some code on page load, or attached to a button, or to set a viewScope value, or so forth, it could go anywhere: in a page event, in a button action, in a dataContext, in a this.value property for a xp:repeat, or any number of other places. Now, if I want to evaluate something, it's going to happen in one place: the controller class. So if I have a bit of information that needs recalculating (say, the total cost of a shopping cart), I make a getTotalCost() method on the controller and set the value in the XPage to pageController.totalCost. Similarly, if I need to set some special values on a document on load or save, I have a clear, standard way to do it:

package controller;

import com.ibm.xsp.model.domino.wrapped.DominoDocument;
import frostillicus.controller.BasicDocumentController;

public class Projects_Contact extends BasicDocumentController {
	private static final long serialVersionUID = 1L;

	@Override
	public String save() throws Exception {
		DominoDocument doc = this.getDoc();
		doc.setValue("FullName", ("CN=" + doc.getValue("FirstName") + " " + doc.getValue("LastName")).trim() + "/O=IKSGClients");

		return super.save();
	}

	@Override
	public void postNewDocument() throws Exception {
		super.postNewDocument();

		DominoDocument doc = this.getDoc();
		doc.setValue("Type", "Person");
		doc.setValue("MailSystem", "5");
	}
}

It sounds like a small thing - after all, who cares if you have some SSJS in the postNewDocument event on an XPage? And, besides, isn't that where it's supposed to go? Well, sure, when you only have a small amount of code, doing it inline works fine. However, as I've been running into for a while, the flexibility of XPages makes them particularly vulnerable to becoming tangled blobs of repeated and messy code. By creating a clean, strict system from the start, I've made it so that the separation is never muddied: the XPage is about how things appear and the controller class is about how those things get to the XPage in the first place.

We'll see how it holds up as I use it more, but so far this "controller"-class method strikes a good balance between code cleanliness without getting too crazy on the backing framework (as opposed so some of the "model" systems I've tried making).

The Ruby Builder for XPages

Wed Nov 07 13:50:14 EST 2012

Tags: xpages ruby

After I got Ruby in XPages to the point where it's generally working enough to power this blog, I set my sights on an even-more-important goal: being able to write backing Java classes in Ruby. While replacing SSJS is quite handy, my general use of inline scripting like that has declined significantly in favor of Java classes.

Fortunately, JRuby has a language cross-"compiler" and some hooks to write Java-compatible Ruby classes. Unfortunately, I was repeatedly stymied by a couple things:

  • Having the conversion happen automatically
  • Having the resultant "compiled" classes use the right class loader, allowing them to access other classes in the app
  • Implementing Java Interfaces

The first one was the most work but also the first to be solved: I wrote an Eclipse builder. The latter two were tough until I realized yesterday that the answers were sitting under my nose the whole time. For the classloader, I was able to switch the compiled output from using the global JRuby runtime to a runtime stored in the application scope and set to use the loader from facesContext.getContextClassLoader(). For the interfaces, it turned out that JRuby already had another annotation for declaring implemented interfaces, unsurprisingly called "java_implements".

So the upshot of this is: now I can write classes in Ruby and have Designer automatically convert them to Java and then compile those classes, ready to be used like any "normal" Java class in an XPage app. For demonstration purposes, I whipped up a useless-in-reality class to implement DataObject:

require "java"

java_package "frostillicus"

class TestDataObject
	java_implements "com.ibm.xsp.model.DataObject"
	
	
	def initialize
		@values = {}
	end
	
	java_signature "Object getValue(Object key)"
	def [](key)
		@values[key] or "#{key.to_s} not found"
	end
	
	java_signature "void setValue(Object key, Object value)"
	def []=(key, value)
		@values[key] = value
	end
	
	java_signature "Class<?> getType(Object key)"
	def get_type(key)
		java.lang.Class.class
	end
	
	java_signature "boolean isReadOnly(Object key)"
	def read_only?(key)
		false
	end
end

Once you have that file, it gets automatically converted to Java and compiled (once you have your build path set up correctly), and then it's available for use in other Java classes, in SSJS (or Ruby-in-XPages), and as a managed bean.

As you might expect with something like this, particularly for a first draft, there are a number of caveats:

  • The implementation is ugly as sin. There's a part that actually contains Java code that writes Ruby code that writes Java. It's very much a product of a series of "I wonder if this would work..." tests.
  • I haven't tested it to look for weird conflicts or leaks, though I suspect that my use of application-specific runtimes will help head off the worst of those potential problems.
  • You can't write classes that extend other classes. While I think you can do this within JRuby itself, the resultant objects are children of RubyObject, and there's no multiple inheritance. Interfaces will work in many cases, but that still limits the applicability.
  • Since the building happens in Designer, you need to have the feature installed client-side, which is not a problem with Ruby-in-XPages.
  • All those Java annotations really harsh the buzz of writing Ruby code. In that example above, the Ruby class isn't really any cleaner than a pure-Java equivalent. However, for larger, more complex classes, the expressiveness benefits of Ruby  would start to show.
  • There's basically no IDE help to be had. I haven't written anything into the builder to highlight syntax errors yet and non-syntax errors (like implementing an Interface but not all the methods) only show up with build errors on the resultant Java class. Plus, there's no autocomplete, and you only get syntax highlighting if you specifically install Ruby syntax support (I think I got it from the Eclipse Ganymede update site). Some of those may be fixable with minor effort, but the really big stuff would go far beyond the amount of time I have to dedicate to this kind of thing.

For now, I tossed the first draft up on GitHub for the curious. It will take more work to be be considered anywhere near finished, but getting it this far means I can start using it in personal/demo apps and really start finding the rough edges in practical use.

Self-Aggrandizement for Fun and Profit

Mon Nov 05 21:38:36 EST 2012

As I alluded to in my previous post, I decided to set up a "portfolio" site to house a list of projects I've made or collaborated on and my recently-updated resume:

http://portfolio.frostillic.us

The site itself is intentionally quite simple: it's meant to house clear content, while further details are available on linked pages. Nonetheless, despite its simplicity, it's allowing me to further refine a couple things I've been tinkering with lately:

  • Bootstrap. Collaboration Today came out of the gate with a very clean implementation of a responsive Bootstrap design, and so I've been following suit by using Bootstrap for my non-data-driven sites (OneUI still feels like a better choice for complex applications in XPages). This portfolio provides a perfect testing ground for thoroughly incorporating the standard Bootstrap elements and making sure everything looks good and works properly on mobile devices.
  • Bootstrap as a Theme. This goes along with including Bootstrap generally, but I figured I'd take this occasion to start making a clean Theme to handle including the appropriate files and applying useful styles that I come across. Currently, this mostly means button styles, but I could see this expanding and potentially including renderers so that you could apply Bootstrap in much the same way that you do OneUI and standard and ExtLib components (like navigators) would adapt accordingly.
  • Cleaner URLs. I mentioned this in the aforementioned previous post, but I want to look for ways to keep XPage URLs as clean as possible. For this database, I'm taking a cautious route: using Domino web rules and giving the database a little knowledge about them, rather than exploring the possibility of running the whole site from a servlet. For a simple database, this is working well, with a basic from/to map and a simple manager bean.
  • Aggressive Open-Sourcing. Most non-work apps I've written lately have found their way to GitHub, but I decided I wanted to further drive the point home on the portfolio site. Towards this end, I included a "View page on GitHub" link in the footer of each page that links to the current master branch version of the XPage.
  • Page Controllers. I mentioned this a bit before, but I'm including my "page controller" idea in the portfolio, at least for editing documents. I created an AbstractDocumentController class that is meant to be a parent class for controllers used on pages with a single document data source. It just provides a couple convenience methods and hooks to override in type-specific cases or use directly in the basic case.
  • Another Simple View/Document Wrapper. This one isn't really anything new, but it's another example of how frequently it's useful to write a model class and manager instead of dealing entirely with the views and documents directly on the XPage with SSJS. The models make it clear what the important data is and the resultant XPage is pleasantly clean.

All in all, this has proven to be a good exercise in writing a focused, simple XPages app with an eye towards extreme cleanliness.

Taking a Swing at the URL Problem

Sun Nov 04 19:20:03 EST 2012

Tags: xpages urls

In the new portfolio web site I'm setting up for myself, I've decided to see what I can do about Domino's, and XPages' specifically, tendency towards ungainly URLs. An XPage URL, particularly an auto-generated one from, say, a view panel, can quickly become filled with undesirable elements - namely, ".nsf", ".xsp", "$$OpenDocument", "action=", and "documentId=". They all make sense and serve important purposes for the server, and to a certain extent URLs other than the main one don't matter, but I want a clean address bar, dang it.

The approach I'm trying now involves creating a managed bean named "url" that implements Map<String, String>. Its get() method takes a normal URL of the kind that you would usually write for a page and returns a cleaned-up version if it can find it, and the original URL otherwise. So a link to, say, the contact page looks like this:

<xp:link text="Contact" value="${url['/Contact.xsp']}"/>

I also set up a "link" custom control to act as a drop-in replacement so I don't have to do that EL bit everywhere. The code itself generates an internal map based on a view of simple From/To "Alias" documents in the DB, though I'm considering having it peek into names.nsf to find any applicable Web Rules more dynamically. The net effect is that I can use that bit of EL or the custom control and hand it a "normal"-style database-relative XPage URL and it'll clean it up if it can or, worst case, pass it through directly.

I can imagine some improvements, beyond the switch to looking at names.nsf instead. For one, I should make it handle "partial" replacements, so that I could map, say "/whatever.nsf/Posts/postid" to just "/posts/postid". I could go beyond that, too, making it do regular expressions to allow it to translate arbitrarily-complex routes beyond what just checking Web Rules could do.

I'm not certain that this is necessarily the best way (the REALLY best way would be to hook into whatever code handles all URL generation in the app), but it has a nice simplicity to it and I figure it's worth a shot.

A Couple Things I've Been Trying Out Lately

Tue Oct 16 09:22:18 EDT 2012

Tags: xpages

I'm always trying to figure out new tricks and (groan) patterns for my XPages development, and I've had a couple trends and experiments lately that I think are worth mentioning.

First off, I've been doing a lot of source control stuff lately, but that's a topic for another post, currently in crummy-draft form.

Beyond that, I think I'll just start a list:

  1. The joys of ExtLibUtil. Historically, I've had a general "JSFUtil" class (with the name and original code copied from here), but I've stopped copying that around from DB to DB and instead have started only copying the methods I really need, instead looking first at the ExtLibUtil class. If you, like me, were slow to hear about this bundle of magic, I suggest you take a look.
  2. Using the DominoDocument class in a bean. This one I'm not sure of, but I figured I'd give it a shot. DominoDocument is one of the classes that the XPages environment uses to wrap a lotus.domino.Document object and make it more or less persist across serialization. I decided to try using this (and maybe the View equivalent later) directly on the theory that it might be a bit easier to use and maybe even faster. So far, it's been a bit of a hassle, but I do enjoy having the various DataObject methods available for passthrough for my bean for EL use.
  3. Dynamic form generation. This is along the lines of something I was discussing in a chat the other day - the notion of having some more default "scaffolding" for XPages, where it could pick up more of the existing database elements without having to actually write the form XPage (or drag the fields over). For a simple data-driven app, having an XPage that generates type-appropriate fields (including keyword values from drop-downs and the like) could save a lot of time up front.
  4. "Controller" classes to back XPages. I've started more often making classes meant to have a bit of knowledge about a specific XPage (say, Document.xsp paired with DocumentController.java) and putting almost all code in there. So, instead of having a bit of SSJS on the save button to handle saving the document, putting a confirmation in the flashScope, and handling a redirection (ideally by returning a string), I do something like <xp:eventHandler ... action="#{documentController.save}"/> and write all the code in Java. This is still slightly awkward, but it's another way to move code out of the XPage itself, aiming to go EL-only as much as possible.

Most little tweaks like these aren't huge on their own, but they all combine to make significant quality-of-life improvements. It's a great feeling to see my XSP markup getting leaner and easier to read day by day.

Forms 'n' Views

Mon Sep 10 19:33:03 EDT 2012

One of my (many) for-fun projects lately has been a design-element editor named "Forms 'n' Views". Though it's not really release-quality, it's been coming together enough to toss it up on GitHub and make a post about it. Basically, it's meant to serve a couple purposes:

  • Help mitigate the "Windows problem" - when all you want to do is make a quick change to a legacy design element (say, a stylesheet), but doing so would mean firing up a Windows VM, launching Designer, and opening the DB. For quick edits, the amount of time that takes wildly eclipses (heh) the time spent actually coding.
  • Experiment with a data-focused way to edit forms and views. The legacy editors are geared towards a Notes client target, dedicating most of their space to aspects that have no meaning when the design element is intended just for backing an XPage.
  • Eventually, provide a way to grant "sudo"-style access to other users to modify certain design elements without granting full designer/manager access, setting up WebDAV, or exposing a poor web designer to Designer.
  • Let me experiment with a different setup for an XPages app - Forms 'n' Views consists of a Dojo BorderContainer- and TabContainer-based UI on a single XPage, which loads up tabs containing custom controls on the fly. None of it is unexplored territory, but it's good to write different kinds of apps sometimes.
  • Let me try out some more backing-class architectures. Each editor is backed by a DOM representation of the exported DXL, which is manipulated through wrapper classes and then re-imported back directly. Things like getting lists of columns are done via XPath queries without storing the data redundantly.

In its current incarnation, it's taken on the pale complexion of Dojo's "claro" theme - I've been going back and forth between that and the Notes-like "soria" theme:

Forms 'n' Views

It should do for now, but I'll likely fiddle with it more to make it look better.

In any event, this has been a fun project so far, and hopefully it'll shape up into a useful tool.

Building XPages servlets with FacesContext access

Thu Sep 06 10:05:00 EDT 2012

I have a confession to make: I'm not crazy about XAgents. Don't get me wrong - they do everything they're supposed to and do it well. However, it's always kind of bothered me that you take a visual design element like an XPage and turn off all the higher levels to get back down to the core servlet. Plus, it muddies the list of XPages in the DB - some are actual XPages, some are just wrappers for scripts. So my objection is essentially pedantry.

However, the fact that my objection is wildly exaggerated has never stopped me from sinking a lot of time into finding the "right" answer before, and it hasn't stopped me now. When what I want to build is, say, an Excel exporter for a particular type of document in a reporting DB, what I really want is a basic servlet that exists only in the database and has access to the surrounding XSP environment and custom classes. Fortunately, between a post by Sven Hasselbach, the referenced Chinese-language developerWorks article, the XSP Starter Kit, and a bit about facesContext.release() on a JSF-focused article and reinforced by the FacesContextServlet class from the ExtLib, I made it work.

The developerWorks article covers the bulk of the work - creating the Factory, setting up the file in META-INF, etc.. After that, you can set up your servlets by extending (using the Starter Kit method) DesignerFacesServlet. It turns out that the important thing to remember is to close out your context when you're done - I ran into a lot of trouble from not doing this, which would break XPages visited after the servlet. As an example, here's the test class I ended up building while figuring out how to make it not blow up my application:

import java.io.*;
import java.util.*;
import com.ibm.commons.util.StringUtil;
import com.ibm.xsp.webapp.DesignerFacesServlet;
import javax.faces.context.FacesContext;
import javax.servlet.*;
import javax.servlet.http.*;

public class TestServlet extends DesignerFacesServlet implements Serializable {
	private static final long serialVersionUID = -1152176824225969420L;

	@SuppressWarnings("unchecked")
	@Override
	public void service(ServletRequest servletRequest, ServletResponse servletResponse) throws ServletException, IOException {
		// Set up handy environment variables
		HttpServletRequest req = (HttpServletRequest)servletRequest;
		HttpServletResponse res = (HttpServletResponse)servletResponse;
		ServletOutputStream out = res.getOutputStream();
		FacesContext facesContext = this.getFacesContext(req, res);

		try {
			res.setContentType("text/plain");

			out.println("start");

			// The sessionScope is available via the ExternalContext. Resolving the variable
			//	would work as well
			Map<Object, Object> sessionScope = facesContext.getExternalContext().getSessionMap();
			sessionScope.put("counter", sessionScope.containsKey("counter") ? (Integer)sessionScope.get("counter") + 1 : 1);
			out.println("Counter: " + sessionScope.get("counter"));

			// A query string map is available via the request. This method, as opposed to
			// 	getting the "param" variable, returns arrays of strings, allowing things like
			//	"?foo=bar&foo=baz" properly
			Map<String, String[]> param = req.getParameterMap();
			for(String key : param.keySet()) {
				out.println(key + " => " + StringUtil.concatStrings(param.get(key), ';', false));
			}

			out.println("done");

		} catch(Exception e) {
			e.printStackTrace(new PrintStream(out));
		} finally {
			out.close();

			// It shouldn't be null if things are going well, but a check never hurt
			if(facesContext != null) {
				facesContext.responseComplete();
				facesContext.release();
			}
		}
	}
}

I set up my ServletFactory to use this one for "/test", and so it's available via a URL like "/database.nsf/xsp/test?foo=bar". As with an XPage, you can also chain more path bits on after the servlet name, like "/database.nsf/xsp/test/some/other/stuff" and get to that via req.getPathInfo() - though with the caveat that, unlike with an XPage, the servlet path is included in the path info, so it would return "/xsp/test/some/other/stuff".

So long as I don't run into any other app-exploding problems, I plan to go this route for non-UI requests like exports and actions. When I have a use for it, I'll also go down the related path of writing custom services, for which the ExtLib provides an extensive foundation.

A Custom Control for dojox.widget.Toaster

Wed Sep 05 09:27:00 EDT 2012

Tags: xpages dojo

Update: Looks like Chris Toohey beat me to this by a couple months: http://www.dominoguru.com/pages/04092012113950.html

Marky Roden did a presentation at MWLUG that included, among other things, a demonstration of Pines Notify, a jQuery plugin that provides Growl-style notifications - something that could come in tremendously handy in a lot of situations.

I wanted to use something like this, but I decided to check to see if there's an equivalent in Dojo already, so I don't have to start including jQuery as well. Fortunately, there is... ish. dojox.widget.Toaster serves a similar purpose, though it takes its design cues from the little "toaster" notifications common on Windows. However, their default appearance is the worst thing in the entire world:

GAH MY EYES

Fortunately, that can be cleaned up with a little CSS to be a crummy version of Growl:

A little better

With a bit more work and attention to detail, it could end up pretty classy. I think it's still not as featureful as Pines Notify (no "clickthrough" options, its handling of multiple messages with different timeouts is not nuanced, etc.), but it's serviceable.

The API to use this widget is actually pretty straightforward: you run a bit of code at page load to create a toaster object to listen to a specific channel name (e.g. "/save/success") and then publish messages whenever you want something to appear. The code for sending a message is pretty simple, and there are two forms, depending on whether or not you want to include additional parameters:

dojo.publish("/test/channel", ["This is message one, lasting five seconds"])
dojo.publish("/test/channel", [{ message: "This is message two, lasting 10 seconds", duration: 10000 }])

I decided to wrap the instantiation and required resources up into a custom control and make another demo DB:

http://frostillic.us/tests/djToaster.nsf

The "toaster.css" page is the CSS I wrote to override the standard behavior. The margins are intended for the top-right positioning, but could be changed readily to equivalents for other corners.

The control exposes five properties: messageTopic (the channel name, required), defaultType ("message", "error", etc.), duration (in milliseconds), positionDirection (the position and slide direction, defaulting to up from the bottom right), and separator (the text that appears between two simultaneous messages).

A Couple Updates to My Domino-One-Offs Repository

Tue Sep 04 18:25:00 EDT 2012

Tags: xpages java

I realized earlier that I let a couple of the classes in Domino-One-Offs stagnate relative to the versions I currently use. Since originally posting my convenience XML library and, more usefully, the DynamicViewCustomizer, I've tweaked both in my various projects to add more features I ended up needing.

The DynamicViewCustomizer has gone further down the path of Notes-client fidelity at the expense of a bit more overhead, doing things like setting column widths (except where the column is set to extend to the window width, either specifically or by virtue of being the last column, as appropriate). It's still not perfect, as I ran into a bit of trouble with, I think, HTML in category columns (and you can see commented-out remnants of various other changes), but it's potentially useful.

For my "man, I hate dealing with all these stupid orthogonal Java XML libraries" XML library, I've added a number of functions to support my Forms 'n' Views app (coming Soon™), namely better attribute handling, a simple NodeList wrapper (that actually acts like a List), and a basic getXml() method for when you just want a string version.

Finally, I tossed in the current version of my SortableMapView class, meant to gobble up Views or ViewEntryCollections and make them sortable by all columns. That one uses Lombok, but I think you could just remove the bits about making it List-compatible and still be fine.

An Extended Document Data Source To Support MIMEBean

Wed Aug 22 20:02:00 EDT 2012

Tags: mime xpages java

Ever since I personally came up with the idea of storing serialized Java objects in Notes documents via MIME, I've been trying to use the technique all over the place, since it's basically the best thing since sliced bread.

My latest notion was that the pattern would be all the cooler if the serialized Java objects could be referenced directly in EL, so you could do something like #{doc.LineItems[3].cost}, pulling directly from a more-or-less normal Domino document. I decided to give implementing this a shot, since it would also have the side benefit of introducing me to writing JSF components/complex types.

The path I took sure feels like the long route (lots of manual doubling up of property definitions and proxying methods), but it seems to work: I have a "mimeDominoDocument" data source that acts just like a normal document source except it transparently serializes and deserializes Java objects. It's not exactly a perfect implementation (the setValue() method has a rather naive method of guessing which objects should be serialized and which shouldn't, and I didn't handle the case of setting a value to an object and then using, say, getItemValueString()), but it should be fine for demo and basic purposes.

I set up an example database with pages for the appropriate code here: /tests/mime.nsf.

Starting With Java Classes First

Mon Aug 20 21:06:00 EDT 2012

Tags: xpages java

Every time I make a new XPages app, I start by using the normal xp:dominoView and xp:dominoDocument data sources, on the theory that building with the standard set of components will keep things clean while I build on top of that. However, with each successive app, the time before I get annoyed at how messy the code has to be and start writing Java wrapper and management classes gets shorter and shorter.

For example, in my Forms 'n' Views mini-Designer app, I have the sidebar list of databases, which is just pointing to a view and showing those entries. That works pretty well, since the values I need are in the view itself. However, now I want to show the database icons, but also want to check to make sure the DBs are on the current server, allow the current user to access them, and are accessible via HTTP (I'm just pointing the image to /whatever.nsf/$Icon). Doing that with just the view entries will get messy, requiring a bunch of Server JavaScript to be crammed into my xp:image object. It'd work, but it'll only be a matter of time before I want to add more smarts, such as if I make it so that you can grant enhanced access to certain design elements to a certain user. The best way to handle this will be to make a "DatabaseEntry" class and give it methods to determine the icon and design-element visibility.

I don't think I really saved myself any time by delaying this - it's pretty rare that I DON'T have this sort of calculation eventually, so I'm just adding a bit of time doing things the "dirty" way first before cleaning them up. I'm not really familiar with "vanilla" JSF, but I get the impression that it doesn't really let you put as much business logic into the view portion as XPages do (I think JSF tends to stick to EL normally). Certainly, when you write a Rails app, you start with the models and controllers cleanly before putting the data on the page. The XPage architecture doesn't make that as easy, with its lack of reasonably-modifiable controllers, but adding Java classes and using collection classes in xp:dataContexts or as managed beans work great.

I've toyed with the idea of expanding the classes I use in my forums app to work smoothly with any view/document, picking up the columns by name and storing them in Map representing the entry/document. The problem with that is that the data source and needs tend to be just different enough per app that it's not generic enough (for example, in this one, I'm getting databases for the "add DB" picker via DbDirectory, not a view). In the mean time, I think I'll just start with the access classes first and see what I do most commonly.

Mixing Dojo's BorderContainer with OneUI

Tue Jul 31 21:18:00 EDT 2012

Tags: xpages dojo

Chris Toohey's post earlier today reminded me of a technique I've been using in some of my apps lately. Since most of what I've been doing has been for internal and admin/reporting-type apps, I've been using OneUI extensively - it's straightforward with the Extension Library, looks pretty good, and the consistency of UI works for that type of application.

However, I ran into a problem with large views: putting a very horizontally-large table into the content area of OneUI breaks it horribly, and there's no good fix. With a lot of tweaking, I made it so it at least scrolls and the header extends the full width, but it was still problematic, particularly when I wanted to add a right sidebar.

The solution I ended up going with was to graft the OneUI look onto a Dojo BorderContainer skeleton, since a BorderContainer takes care of overflowing content nicely. The end result works reasonably well:

/tests/oneui-border.nsf

The layout's code is not pretty, but it doesn't have to be. It provides to its children a couple content facets: the primary content area, LeftColumn, RightColumn, and ActionBar, which is pretty much like a Notes client action bar (my project that uses this takes a lot of cues from an existing Notes app). Additionally, it only shows those extra facets when there's something present, so the sidebars and action bar are entirely hidden when not needed. There are a couple caveats, though:

  • It doesn't provide bindings for every property, as Chris Toohey's does. It certainly could - I just haven't had a need for it.
  • Similarly, I don't have a need for the footer area, so that isn't included. However, it easily could be.
  • It needs some stylesheet patching, and the current stylesheet I use has some flaws in non-Safari browsers (I'm still working on it).
  • Since most of the OneUI structure is written out as HTML, it's not as convenient to tweak as the ExtLib's layout control.

Still, it's been working well for my purposes, and perhaps it'll be useful on its own or as a starting point for other projects.

Faking Sortable View Data

Sun Jul 22 15:28:00 EDT 2012

Tags: xpages data

There's a point in most decent-sized XPages apps I write where I switch from using standard data sources like xp:dominoView to writing Java classes that implement List, usually due to some multi-source data merging I have to do that would be extremely cumbersome or slow to do otherwise. This works well, but one problem is that view data sources have special powers that plain-Jane Lists lack, such as easy sortable column headers.

Having encountered TabularDataModel a bit ago when making an activity stream out of several databases' views, I decided to see how easy it would be to take a fairly standard set of Java data objects - say, a List of Maps, which is conceptually similar to a non-categorized view - and put it into a xp:viewPanel and have all sortable columns. In short: weird, but overall easy.

The TabularDataModel abstract class is very easy to implement: you only need to implement methods for the total number of rows and one to get the current row's data (it uses an internal index accessible via .getRowIndex()). On its own, that's not interesting, since it doesn't get you anything a normal List wouldn't. They key is when you override a couple more methods:

  • .isColumnSortable(String columnName): the xp:viewPanel calls this with a column programmatic name to determine whether it's sortable at all - I made mine return true in all cases.
  • .getResortType(String columnName): once the panel knows a column is sortable, it checks to see what sorting directions it supports. In my case, I return 3, indicating it's sortable both ascending and descending.
  • .getResortState(String columnName): returns the current sorted state of the specified column. Presumably, the column name would be useful if you did custom cascading sorting, which I didn't for now - I just have it return the current sort direction of whatever column is sorted.
  • .getResortColumn(): returns the name of the currently-sorted column.
  • .setResortOrder(String columnName, String sortOrder): this is the interesting one, where the view panel tells the model that it wants to sort a specified column in a specified direction (or "toggle").

When the resort method is called, my class sorts its internal List object by the given column/key name in the given direction via a pretty simple comparator. Once it's all put together, you have the data from the view in a standard format and sortable by all columns.

As usual, there are caveats. First and foremost, there's a reason why you can't normally sort by just any column in a Domino view: Notes was made in 1989 and has barely been improved since then sorting large amounts of data arbitrarily is an expensive operation. It's only practical in this sort of situation because the amount of data is pretty small (only 200 entries) and then initial loading is relatively quick. If you had hundreds of thousands of entries in the view, this would be painfully slow to load and rather greedy with server memory. Additionally, I chose to punt on multi-value columns: I just turn anything that isn't directly comparable (like Vectors) into Strings. Nonetheless, this kind of thing may be practical on its own and is definitely a good exercise in extending your data-manipulation capabilities.

For development, I created a quick database with a couple documents with somewhat randomized data: a date/time value offset by a random amount of hours or days, a random number from -512 to 512, and the document's UNID and Note ID, listed in a view with no sortable columns. The class I wrote just scrapes through the view data with a ViewNavigator and puts each of the entries into a Map. The demo page shows two standard xp:viewPanels (one showing the view directly, the other showing my SortableMapView object) and, on the second tab, the Java code for the three objects I used.

A Prototype In-App Messaging System for XPages

Mon Jul 09 14:35:00 EDT 2012

Tags: xpages java

Sven Hasselbach's post about ApplicationListeners the other day and the notion of post-MVC methods of app architecture got me thinking about the notion of generic messaging/events inside an XPages app. The first example that comes to mind is a project-tracking database where there may be events like "new task created" or "delivery ready for review" - business logic stuff where the fact that it's a document being modified is an implementation detail. In my actual project-tracking database at work, I implemented this by creating "Notification Stub" documents with information about the event and having a scheduled agent process them. That's a fine Notes-y way to do it (and has its own advantages, like having another cluster member send the emails), but I want a way for the running XPages app to notice and react to these things without the immediate code knowing about every single concern.

I did a quick search and didn't find any such capability in basic JSF or XPages, but I realized it wouldn't be terribly difficult to implement it myself: have an application-scoped bean (maybe one per scope, if it's useful) with a methods to dispatch events and declare listeners. Then, other beans could attach themselves as listeners and the normal code would only be responsible for sending out its messages, to be received by zero or more unknown objects:

public interface XPagesEventDispatcher {
	public void dispatchEvent(XPagesEvent event);
	public void addListener(XPagesEventListener listener);
}

The actual event objects would be similarly simple: just a name and an arbitrary array of objects:

public interface XPagesEvent {
	public String getEventName();
	public Object[] getEventPayload();
}

Finally, listeners are the simplest of all, being responsible only for providing a method to handle an XPagesEvent (EventListener is the standard "tagging" interface for this kind of thing):

public interface XPagesEventListener extends EventListener {
	public void receiveEvent(XPagesEvent event);
}

However, after creating my gloriously clean set of interfaces, I ran into the ugly marsh of reality: application-scoped beans only exist after first use. That works fine for the dispatcher itself, but not for listener beans, the whole point of which is to not be referenced in code elsewhere. After a bit of searching, I came up with a solution that may work. When defining a managed bean, you can set properties, and those properties can be Lists. So, rather than creating two beans, I define just the dispatcher bean, but provide it a list of class names of listener beans to create at startup:

<managed-bean>
	<managed-bean-name>applicationDispatcher</managed-bean-name>
	<managed-bean-class>frostillicus.event.SimpleEventDispatcher</managed-bean-class>
	<managed-bean-scope>application</managed-bean-scope>
	<managed-property>
		<property-name>listenerClasses</property-name>
		<list-entries>
			<value-class>java.lang.String</value-class>
			<value>frostillicus.Messenger</value>
		</list-entries>
	</managed-property>
</managed-bean>

I gave my dispatcher implementation class a convenience methods to take a string and 0 or more objects and dispatch the event, so it could be used in code like:

applicationDispatcher.dispatch("home page loaded")
applicationDispatcher.dispatch("delivery submitted", [delivery.getUniversalID()])

So far, this setup is working pretty well. I plan to toy with it a bit and then try it out in proper use. Provided it works well (and it doesn't turn out that this capability already exists and I just missed it), I'll turn it into a project on GitHub/OpenNTF. As long as it keeps working, I'm excited about the prospects, especially when combined with XPages Threads and Jobs for non-blocking event handling.

Basic Org Charts With xe:navigator and xe:beanTreeNode

Fri Jul 06 09:30:00 EDT 2012

The topic of generating org charts on an XPage came up recently, and I decided to try my hand at making a pretty basic one using a technique similar to my re-use of Outline design elements from the other week. Not only are the generated hierarchies potentially useful, but it ended up providing a much cleaner example of how to recursively generate an Extension-Library tree.

To get started, I mocked up some basic data: I created a standard Domino directory database and created users with unique ShortName values and the ShortName of their immediate boss in the Manager field, with anyone at the top level having a blank field. Then, because I'm too lazy to look through the existing views to see if one does it, I created a "Managers" view that simply shows Person documents categorized by their Manager field (with a bit of computation to convert self-referential documents to top-level). You can see the live result in the test database, along with applicable code excerpts:

http://frostillic.us/namestest.nsf

The algorithm itself provides a nice exercise in recursion:

  1. Get a list of all top-level managers
  2. For each of those names:
    1. Find their child count to determine whether or not they're a leaf
    2. Create an outline entry with their name
    3. For each child entry, repeat this process

This could potentially be modified to query an LDAP server instead of reading a view directly, which would make it more flexible (and complicated), and the output could be modified or re-used for other visual representations, as needed.

The absolute most important thing when doing recursion, though, is to make sure you listen to music with an appropriately-nerdy title, such as:

Book Review: XPages Extension Library

Mon Jul 02 19:10:00 EDT 2012

Tags: xpages review

Unlike many development environments, there are only a handful of books about XPages. Combined with the historical lack of documentation, that makes all three - Mastering XPages, XPages Portable Command Guide, and XPages Extension Library - essential. In that sense, the quality of the book is a less important purchasing-decision factor than its mere existence.

Fortunately, XPages Extension Library doesn't skimp on quality or breadth of coverage. The sections of the book track the natural progression of someone new to the Extension Library: how to acquire and install it, how to use its upgraded application-development features like the application layout and slicker Dojo controls, how to do the entirely-new things in the ExtLib like REST and JDBC, and finally a much-needed guide to starting "proper" Java development in XPages.

Since reading through the book in sequence, I have cracked it back open both as an indexed reference and for several of its step-by-step guides.

Part 2, which covers the various application UI components, has proven to be very useful when I have a specific UI task in mind, such as using a particular Dojo layout element or implementing some of the fancy, OneUI-friendly elements like the dataView. The organization of the chapters and code examples provide a much-needed sense of order to the bevy of new controls that you find in Designer's sidebar after installing the Library. And even just reading through the chapters in order is bound to give you a wealth of new UI ideas for your day-to-day use.

Several chapters provide invaluable guides for accomplishing complex and potentially daunting tasks, such as the ideal OSGi-based installation of the library to your servers, setting up a OneUI layout for your app (replete with an explanation of facets and callbacks, which are useful in XPages development generally), and packaging JDBC drivers for OSGi deployment. The last task in particular is almost comically opaque process for anyone thoroughly versed in the peculiar Eclipse way of doing things, and the book provides a clear step-by-step guide on negotiating that thicket.

As anyone who follows this blog can probably surmise, I think that pretty much everyone who programs with XPages would be well-served by familiarizing themselves with Java, and so I'm pleased that the book contains a smooth introduction to how to and why you would write Java classes in your app. This chapter does a fine job explaining how to use already-existing classes and packages, how to set up your Designer environment to make Java development easier, what a "bean" is (it's simpler than you'd think), and how to create and make use of managed beans. It's a great XPages-focused guide to starting down what should be a very fruitful road learning Java and the XPages Java environment.

The last word on this book is simple: if you do XPages development, you should purchase and read this book. XPages development with the Extension Library is much more productive than without, and XPages Extension Library provides an easy-to-read explanation to get your head around its myriad capabilities.

Re-using Classic Domino Outlines, Rough Draft

Tue Jun 19 17:34:00 EDT 2012

Tags: xpages

My current big project at work involves, among many other things, viewing individual project databases through a centralized "portal". These all use the same base template, but can be individually customized, usually with new or changed views. Additionally, I'm going to be granting web users varying roles that correspond to existing or planned access roles in the target database. The result is that I'm spending a good amount of time trying to dynamically adapt some classic Notes elements to show up on XPages (hence my DynamicViewCustomizer, which I've since updated and should post).

In addition to views, I wanted to use the existing Outline design elements in the databases, since they do a fine job and there's no reason to re-invent the wheel if it's not necessary. I looked around a bit and didn't see a way to readily re-use them (though I can't rule out the possibility that such a capability exists somewhere), so I decided I'd roll up my sleeves and write my own adapter.

The route I took was to use the xe:beanTreeNode ExtLib control inside the xe:navigator used in most OneUI apps. Though what I conceptually wanted to do was to feed it a tree structure made up of JavaScript arrays and nodes, that's not how it works: you pass it the name of either an already-created managed bean or the full class name of a bean-compatible (read: zero-argument constructor) implementing ITreeNode. Presumably, the right way to do this would be to create a managed bean and pass in configuration parameters via managed-property values. I didn't do that for the time being, though, instead baking a bit of environmental awareness into the Java code - so sue me. I also print stack traces to the server console, which you shouldn't do.

In any event, what my code does is to open the consistently-named outline design element (in this case, "TempOut" for "template outline"), reads through its entries, and recursively builds the node tree. There are a couple environmental assumptions in there, primarily in the constructor, in order to work with the surrounding app, but the core of it should be pretty transportable. One big note is that I haven't gotten around to adding in the "special" outline entries to show the remaining views and folders, nor do I handle "action" entries. Still, it may be a useful reference for using bean tree nodes.

package mcl;

import com.ibm.xsp.extlib.tree.ITreeNode;
import com.ibm.xsp.extlib.tree.impl.*;
import lotus.domino.*;

import javax.faces.context.FacesContext;
import java.net.URLEncoder;

public class ProjectOutline extends BasicNodeList {
	private static final long serialVersionUID = 2035066391725854740L;
	
	private String contextQueryString;
	private String dbURLPrefix;
	private String viewName;

	public ProjectOutline() throws NotesException {
		try {
			ProjectInfo projectDatabase = (ProjectInfo)JSFUtil.getVariableValue("projectDatabase");
			if(projectDatabase.isValidDB()) {
				Outline tempOut = projectDatabase.getOutline("TempOut");
			
				dbURLPrefix = "/__" + projectDatabase.getReplicaID() + ".nsf/";
				contextQueryString = String.valueOf(JSFUtil.getVariableValue("contextQueryString"));
				viewName = (String)FacesContext.getCurrentInstance().getExternalContext().getRequestParameterMap().get("viewName");
				
				OutlineEntry entry = tempOut.getFirst();
				while(entry != null) {
					entry = processNode(projectDatabase, tempOut, entry, null);
				}
			} else {
				addChild(new BasicLeafTreeNode());
			}
		} catch(NullPointerException npe) {
			addChild(new BasicLeafTreeNode());
		} catch(Exception e) {
			e.printStackTrace();
		}
	}
	
	
	private OutlineEntry processNode(ProjectInfo projectDatabase, Outline outline, OutlineEntry entry, BasicContainerTreeNode root) throws NotesException {
		int level = entry.getLevel();
		
		switch(entry.getType()) {
		case OutlineEntry.OUTLINE_OTHER_UNKNOWN_TYPE:
			// Must be a container
			
			BasicContainerTreeNode containerNode = createSectionNode(entry);
			if(root == null) {
				addChild(containerNode);
			} else {
				root.addChild(containerNode);
			}
			// Look for its children
			OutlineEntry nextEntry = outline.getNext(entry);
			while(nextEntry.getLevel() > level) {
				nextEntry = processNode(projectDatabase, outline, nextEntry, containerNode);
			}
			
			return nextEntry;
		case OutlineEntry.OUTLINE_TYPE_NAMEDELEMENT:
			View view = projectDatabase.getView(entry.getNamedElement());
			if(view != null) {
				ITreeNode leafNode = createViewNode(entry);
				if(root == null) {
					addChild(leafNode);
				} else {
					root.addChild(leafNode);
				}
				view.recycle();
			}
			
			return outline.getNext(entry);
		}
		
		return outline.getNext(entry);
	}
	
	private BasicContainerTreeNode createSectionNode(OutlineEntry entry) throws NotesException {
		BasicContainerTreeNode node = new BasicContainerTreeNode();
		node.setLabel(entry.getLabel());
		node.setImage(dbURLPrefix + urlEncode(entry.getImagesText()) + "?Open&ImgIndex=1");
		
		return node;
	}
	private ITreeNode createViewNode(OutlineEntry entry) throws NotesException {
		BasicLeafTreeNode node = new BasicLeafTreeNode();
		node.setLabel(entry.getLabel());
		node.setImage(dbURLPrefix + urlEncode(entry.getImagesText()) + "?Open&ImgIndex=1");
		node.setHref("/Project_View.xsp?" + contextQueryString + "&viewName=" + urlEncode(entry.getNamedElement()));
		
		node.setSelected(entry.getNamedElement().equals(viewName));
		node.setRendered(resolveHideFormula(entry));
		
		return node;
	}

	private String urlEncode(String value) {
		try {
			return URLEncoder.encode(value, "UTF-8");
		} catch(Exception e) { return value; }
	}
	private boolean resolveHideFormula(OutlineEntry entry) throws NotesException {
		String hideFormula = entry.getHideFormula();
		if(hideFormula != null && hideFormula.length() > 0) {
			Session session = JSFUtil.getSession();
			Database database = entry.getParent().getParentDatabase();
			Document contextDoc = database.createDocument();
			// @UserAccess gave me trouble, so I just did a simple string replacement, since I know that's the only way I used it
			hideFormula = hideFormula.replace("@UserAccess(@DbName; [AccessLevel])", "\"" + String.valueOf(database.getCurrentAccessLevel()) + "\"");
			double result = (Double)session.evaluate(hideFormula, contextDoc).get(0);
			contextDoc.recycle();
			return result != 1;
		}
		return true;
	}
}

XPages MVC: Experiment II, Part 4

Mon Jun 04 19:57:00 EDT 2012

Tags: xpages mvc
  1. XPages MVC: Experiment I
  2. XPages MVC: Experiment II, Part 1
  3. XPages MVC: Experiment II, Part 2
  4. XPages MVC: Experiment II, Part 3
  5. XPages MVC: Experiment II, Part 4

To finish up my series on the infrastructure of my guild forums app, I'd like to mention a couple of the down sides I see with its current implementation, which I'd generally want to fix or avoid if re-implementing it today.

Roll-Your-Own

One of the strengths of this kind of MVC setup is that it works to separate the front-end code from the data source. It would be (relatively) easy for me to replace the model and collection classes with versions that use a SQL database or non-Notes document storage should I so choose. That's a double-edged sword, though: because I'm not using any of the built-in data sources and controls with Domino knowledge, I had to do everything myself. This means a loss of both some nice UI features - like the rich text editor's ability to upload images inline - and the XPage data sources' persistence and caching features.

The collection code deals with View and ViewEntryCollection classes directly, but they can't be serialized, so I had to write my own methods to detect when the object is no longer valid (say, when doing a partial refresh) and re-fetch the collection. This was good in the sense that I learned more about what is and is not efficient in Domino. For example, getNthEntry(...) on a ViewEntryCollection grabbed via getAllEntriesByKey(...) is fast. Conversely, while retrieving data in a view is usually significantly faster than getting the same data from a document, there's a point where the view index size is large enough that, provided you're fetching only a few documents at a time, it's better to use the document. With a lot of work (and a LOT of collection caching), I ended up with something that's quite fast... but since I wrote all the code myself as part of a side project, it was also pretty bug-prone for the first couple weeks after deployment.

Maybe I'd be able to find ways to piggyback more on the built-in functionality if I re-did it, but, as it stands, it's kind of hairy.

Inconsistent Separation

This isn't a TERRIBLE problem, but I'm kind of annoyed with some of the inconsistent choices I made about where DB-specific code goes. For example, collection managers have very little Domino-specific code... except when they need to know about sorting and searching. Similarly, almost all of the code in the model objects deals with pure Java objects and the clean collections API... except the save() methods, which are big blobs of Domino API work. That makes some sense, but the model classes don't handle creation from the database - that's in the collection classes. Like I said, it's not the end of the world, and it's consistent within its own madness, but there are some weird aspects and internal leaks I wouldn't mind cleaning up.

Lack of Data Sources

This is sort of the flip side to my first problem. All of my collection managers are just objects and the collections are just Lists. This is good in the sense that these things work well in Server JavaScript and with all of the built-in UI elements, but it'd be really nice to write my own custom data sources so I can explicitly declare what data I want on the page - using a xp:dataContext or bit of JavaScript in a value property just feels "dirty", like I'm not embracing it completely. However, this problem isn't as much an architectural one as it is a lack of education - I haven't bothered to learn to write my own data sources yet (even though I expect it's more or less straightforward, for Java), so I could remedy that easily enough.

No Proper Controller

This is just another manifestation of the root cause that has me looking into all this MVC stuff to begin with. Even though my collections and models are better than dealing with raw Domino objects, there's still too much of a tie between the UI and the back-end representation, as well as the requisite dependence on the Domino HTTP stack to handle routing requests. Of course, I'm still working on the correct solution to this.

 

Overall, my structure as written has been serving me well. New data elements are pretty easy to set up - I wouldn't mind not having to write three classes per, but hey, it's Java - and working with them is a breeze. Though it took a while to get everything working, now that it is, I can make tons of UI changes without worrying much about the actual data representation. I can change the way data is stored or add on-load or on-save computation beyond the capabilities of Formula language without changing anything in the XPages themselves. So in those senses, my forum back-end code is a huge step up from doing it directly xp:dominoDocuments, but it still doesn't feel completely "right".

Ruby for XPages Programmers, Part 1

Wed May 30 22:54:00 EDT 2012

Tags: ruby xpages
  1. Ruby for XPages Programmers, Part 1

Since I'm always gabbing on about Ruby, I figured it'd be useful to give an overview of the language and, in future posts, why it's worth using for XPages scripting. I don't plan to write an exhaustive tutorial for the language (they're aplenty on the web), but I'll go over the basic concepts and some of how it compares to EL, Server JavaScript, and Java.

Ruby is a dynamic, object-oriented language with functional-programming features and syntax designed for programmer-friendliness. In terms of other languages, it's sort of like a combination of Perl's general syntax, Smalltalk's object model, and (some of) Lisp's functional and metaprogramming abilities. Additionally, Ruby has its own unique features and conventions, such as optional parentheses, property-style getters and setters, and yield statements.

Maybe the best way to provide a gentle introduction to Ruby is to show a relatively simple but contrived code example. Here is a "Point" class meant to represent a point in two-dimensional space where the X coordinate must always be positive:

class Point < Object                                           # 1
    attr_accessor :y                                           # 2
    attr_reader :x
	
    def initialize(x=0, y=0)                                   # 3
        @y = y                                                 # 4
        self.x = x                                             # 5
    end
	
    def x=(x)                                                  # 6
        raise Exception.new("x must be >= 0") if x < 0         # 7
        @x = x
    end
	
    def to_s; "[#{@x}, #{@y}]"; end                            # 8
end

There are a lot of Ruby-isms going on, but hopefully the bulk of the code will be readable without knowing the language. Starting from the top, with a focus on the differences from Java:

  1. Classes are created with just class, with no public modifier, and must begin with a capital letter, which is the language-enforced convention for class names and constants. By unenforced convention, classes and constants are CamelCase. The "<" is equivalent to Java's extends. As in Java, extending Object is not needed, but is done here for demonstration purposes. Finally, # is Ruby's single-line comment delimiter, like // in Java or ' in LotusScript.
  2. attr_accessor and attr_reader are built-in Ruby class methods that generate getters+setters and getters only, respectively. The ":"s in :x and :y indicate that those are Symbols. Symbols are interesting beasts, but in this case think of them as meaning "the things named 'x' and 'y'".
  3. The initialize method is equivalent to Java's constructors and LotusScript's New. This one takes two parameters, but provides default values in case one or both are left out. While Ruby objects are strongly typed in that they all have a class, Ruby variables are typeless and not statically checked.
  4. The "@" symbol is used to denote instance variables, which, like all Ruby variables, don't need to be declared before they're used the first time. In this case, the line is equivalent to "this.y = y;" in Java.
  5. "self.x = x" is used here to indicate that the first "x" is calling the current object's x= method as opposed to setting the local variable "x" to itself. This line is equivalent to "this.setX(x);" in Java.
  6. Ruby, like C++, allows operator overloading, including this special type to define setters. This allows setting of the object's "x" parameter with code like "some_point.x = 3".
  7. This line is jam-packed!
    • Ruby's exceptions are essentially like Java's, except they use raise and begin/rescue instead of throw and try/catch.
    • Objects in Ruby are constructed by calling the new method on the class itself, rather than having new be a language keyword.
    • Ruby allows for conditional and looping statements at the end of single lines as syntactic sugar.
    • Ruby allows parentheses to be dropped when the resultant code is unambiguous.
  8. This line is pretty packed as well:
    • to_s is Ruby's equivalent to Java's toString().
    • Ruby, like JavaScript, uses optional semicolons. Here, they're used to pack the method definition into one line.
    • When a method takes no parameters, the () after the name is optional, both in definition and in use.
    • Ruby strings allow for code interpolation via #{...}. It's like value bindings in XPages or the equivalent feature in PHP or Perl - it allows Ruby code to be embedded in a string instead of +-based concatenation.
    • Like Server JavaScript and Formula Language, the last line in a Ruby method (except in initialize and assignment methods) is an implied return.
    • By convention, Ruby uses all-lowercase underscore_separated_words for variable and method names.

That ended up being a bit more complicated than I had expected, but hopefully it's a reasonable start (if not, there are definitely better Ruby tutorials on the web). In the next post in this thread, I'll describe some other unique or unusual Ruby features and concepts, and later I'll go into some more applicable examples to explain why Ruby is worth using in XPages generally.

XPages MVC: Experiment II, Part 3

Mon May 28 20:00:00 EDT 2012

Tags: xpages mvc
  1. XPages MVC: Experiment I
  2. XPages MVC: Experiment II, Part 1
  3. XPages MVC: Experiment II, Part 2
  4. XPages MVC: Experiment II, Part 3
  5. XPages MVC: Experiment II, Part 4

Continuing on from my last post, I'd like to go over a couple specifics about how I handle fetching appropriate collections of objects from a view and a couple areas where I saved myself some programming hassle.

As I mentioned before, both the "manager" and "collection" classes inherit from abstract classes that handle a lot of the dirty work. The AbstractCollectionManager class is by far the smaller of the two, containing mostly convenience methods and a couple overloaded methods for generating the collection objects. The central one is pretty straightforward:

protected AbstractDominoList<E> createCollection( String sortBy, Object restrictTo, String searchQuery, int preCache, boolean singleEntry) { AbstractDominoList<E> collection = this.createCollection(); collection.setSortBy(sortBy); collection.setRestrictTo(restrictTo); collection.setSearchQuery(searchQuery); collection.setSingleEntry(singleEntry); collection.preCache(preCache); return collection; }

The first three properties actually determine the nature of the collection, while the last two set some optimization bits when the code knows ahead of time about how many elements it expects (such as doing a by-id lookup or a "recent news" list that will only ever display 5 entries). The createCollection call at the start is overridden method in each class that returns a basic collection object to avoid Java visibility issues.

The AbstractDominoList class is much longer; I won't go into too much detail, but a couple parts are pertinent. As I've mentioned, it implements the List interface (via extending AbstractList), which means it just needs to provide get and size methods. The way these work is by checking to see if it houses a valid cached ViewEntryCollection and, if it does, translating the method call to get the indexed entry as an object or the size. If the collection hasn't been fetched yet or if it's no longer valid (say, if it was recycled), it uses the view-location information from the implementing class and any parameters on the object to construct another one.

First, it fetches and configures the view properly:

Database database = this.getDatabase(); View view = database.getView(this.getViewName()); view.refresh(); String sortColumn = this.sortBy == null ? null : this.sortBy.toLowerCase().endsWith("-desc") ? this.sortBy.substring(0, this.sortBy.length()-5) : this.sortBy; boolean sortAscending = this.sortBy == null || !this.sortBy.toLowerCase().endsWith("-desc"); if(this.searchQuery != null) { view.FTSearchSorted(this.searchQuery, 0, sortColumn, sortAscending, false, false, false); } else if(this.sortBy != null && !this.sortBy.equals(this.getDefaultSort())) { view.resortView(sortColumn, sortAscending); } else { view.resortView(); }

Then, it uses code similar to this to store the collection in its cache (I say "similar to" because there are additional branches and methods, but this is the idea):

if(this.restrictTo == null) { ViewEntryCollection collection = view.getAllEntries(); this.cachedEntryCollection = new DominoEntryCollectionWrapper(collection, view); } else { ViewEntryCollection collection = view.getAllEntriesByKey(this.restrictTo, true); this.cachedEntryCollection = new DominoEntryCollectionWrapper(collection, view); }

The other part of the get method is translating the ViewEntry into an object, which is done via a method similar to this monster:

protected E createObjectFromViewEntry(ViewEntry entry) throws Exception { Class currentClass = this.getClass(); Class modelClass = Class.forName(JSFUtil.strLeftBack(currentClass.getName(), ".") + "." + JSFUtil.strLeftBack(currentClass.getSimpleName(), "List")); // Create an instance of our model object E object = (E)modelClass.newInstance(); object.setUniversalId(entry.getUniversalID()); object.setDocExists(true); // Loop through the declared columns and extract the values from the Entry Method[] methods = modelClass.getMethods(); String[] columns = this.getColumnFields(); Vector values = entry.getColumnValues(); for(Method method : methods) { for(int i = 0; i < columns.length; i++) { if(values != null && i < values.size() && columns[i].length() > 0 && method.getName().equals("set" + columns[i].substring(0, 1).toUpperCase() + columns[i].substring(1))) { String fieldType = method.getGenericParameterTypes()[0].toString(); if(fieldType.equals("int")) { if(values.get(i) instanceof Double) { method.invoke(object, ((Double)values.get(i)).intValue()); } } else if(fieldType.equals("class java.lang.String")) { method.invoke(object, values.get(i).toString()); } else if(fieldType.equals("java.util.List<java.lang.String>")) { method.invoke(object, JSFUtil.toStringList(values.get(i))); } else if(fieldType.equals("class java.util.Date")) { if(values.get(i) instanceof DateTime) { method.invoke(object, ((DateTime)values.get(i)).toJavaDate()); } } else if(fieldType.equals("boolean")) { if(values.get(i) instanceof Double) { method.invoke(object, ((Double)values.get(i)).intValue() == 1); } else { method.invoke(object, values.get(i).toString().equals("Yes")); } } else if(fieldType.equals("double")) { if(values.get(i) instanceof Double) { method.invoke(object, (Double)values.get(i)); } } else if(fieldType.equals("java.util.List<java.lang.Integer>")) { method.invoke(object, JSFUtil.toIntegerList(values.get(i))); } else { System.out.println("!! unknown type " + fieldType); } } } } return object; }

Yes, I am aware that it's O(m*n). Yes, I am aware I could move values != null outside the loop. Yes, I am aware I do the same string manipulation on each column name many times. Shut up.

Anyway, that code finds the model class and loops through its methods, looking for set* methods for each column (the column list being specified in the implementation class). When it finds one that matches, it checks the data type of the property, and invokes the method with an appropriately-cast or -converted column value. The result is that the implementation class only needs to provide the code needed for fetching any data that isn't in the view, such as rich text. At one point, I think I had it so that it read the view column titles and used those as field names, but I stopped doing that for some reason... maybe it was too slow, even for this method.

In any event, next time, I'll go over some of the disadvantages that I've found with this method that don't involve big-O notation.

XPages MVC: Experiment II, Part 2

Thu May 24 20:43:00 EDT 2012

Tags: xpages mvc
  1. XPages MVC: Experiment I
  2. XPages MVC: Experiment II, Part 1
  3. XPages MVC: Experiment II, Part 2
  4. XPages MVC: Experiment II, Part 3
  5. XPages MVC: Experiment II, Part 4

Continuing on from my last post, I'd like to go over the collection and model structures I used in my guild-forums app.

I set up the data in the Notes DB in a very SQL-ish way (in part because I considered not using Domino initially). Each Form has a single equivalent view (for the most part), which is uncategorized and sorted by default by an ID column. There is one column for each column that I want to sort by and, in the case of simple documents, one for each applicable field, containing only the field name; additionally, I created $-named "utility" columns for when I wanted to sort by a given bit of computed data or have a second secondary sort column for a given column. For the most part, the data is normalized, but I had to give in to reality in a few situations, for things like sorting Topics by the their latest Post dates.

For collection classes in Java, each collection class file contains one public class for the "manager" and one list implementation class, both of which extend from Abstract versions, so the actual implementations may have very little code. For example, here is the "Groups" Java file (minus some Serialization details):

public class Groups extends AbstractCollectionManager<Group> { protected AbstractDominoList<Group> createCollection() { return new GroupList(); } } class GroupList extends AbstractDominoList<Group> { public static final String BY_ID = "$GroupID"; public static final String BY_NAME = "$ListName"; public static final String BY_CATEGORY = "ListCategory"; public static final String DEFAULT_SORT = GroupList.BY_ID; protected String[] getColumnFields() { return new String[] { "id", "name", "category", "description" }; } protected String getViewName() { return "Player Groups"; } protected String getDatabasePath() { return "wownames.nsf"; } protected Group createObjectFromViewEntry(DominoEntryWrapper entry) throws Exception { Group group = super.createObjectFromViewEntry(entry); Document groupDoc = entry.getDocument(); group.setMemberNames((List>String<)groupDoc.getItemValue("Members")); return group; } }

The actual heavy lifting is done by the parent class, letting the specific classes be composed almost entirely of descriptions of where to find the view, what columns map to model fields, and any extra code that is involved with that object mapping. Other classes are larger, but follow the same pattern. For example, here's a snippet from the manager for Posts:

public class Posts extends AbstractCollectionManager<Post> { // [-snip-] public List<Post> getPostsForTag(String tag) throws Exception { return this.search("[Tags]=" + tag); } public List<Post> getPostsForUser(String user) throws Exception { return this.getCollection(PostList.BY_CREATED_BY, user); } public List<Post> getScreenshotPosts() throws Exception { return this.search("[ScreenshotURL] is present", PostList.BY_DATETIME_DESC); } }

The parent abstract class provides utility methods like getCollection and search that take appropriate arguments for sorting, searching (I love you, FTSearchSorted), and keys.

As for the individual model objects, they start out as your typical bean: a bunch of fields with simple data types and straightforward getters and setters. Additionally, they contain "relational" methods to fetch associated other objects, such as the Forum that contains the Topic, or a list of the Posts within it. Finally, the model class is responsible for actually saving back to the Domino database, as well as taking care of any business rules, like making sure that the Topic document is updated with a new Post's date/time.

The result is that a new object type is easy (-ish) to set up, with all the code being written contributing specifically towards the task at hand. And, since the nuts and bolts of XPages are Lists and Maps, working with these objects is smooth, taking advantage of Expression Language's cleanliness and Server JavaScript's bean-notation support in the UI and Java's clunky-but-still-useful collections features.

Next time, I'll go over some of the specifics of how I deal with fetching collections and individual records, use reflection to save me some coding hassle, and other tidbits from the giant messes of code that make up my abstract classes. After that, I think I'll make a fourth post to detail some down sides to my approach as implemented.

XPages MVC: Experiment II, Part 1

Wed May 23 15:38:00 EDT 2012

Tags: xpages mvc
  1. XPages MVC: Experiment I
  2. XPages MVC: Experiment II, Part 1
  3. XPages MVC: Experiment II, Part 2
  4. XPages MVC: Experiment II, Part 3
  5. XPages MVC: Experiment II, Part 4

As I've mentioned a couple times, my largest XPages app to date is the site I did for my guild's forums. The forum part itself isn't particularly amazing, considering it's almost harder to NOT write a forum in Domino than it is to write one, but it gave me a chance to try my hand at abstracting data access away from the XPage.

I put "Part 1" in the title because I figure it will be best to break the topic up into at least three parts: the overall idea and the result, the structure of the model and collection classes, and some nitty-gritty bits about how I get the data out of the back-end classes.

My first draft of this site was straightforward, from an XPages perspective: I mapped XPages to the corresponding Forms and used a bit of Server JavaScript here and there to pull in bits of support information. This worked pretty well at first, but quickly ran into code and speed scalability issues. Just looking at a "Topic" page, the following data needs to be looked up from outside the Topic document itself:

  • The name and ID of the parent Forum document
  • The name and ID of the parent Forum's parent Category document
  • Whether or not the topic is marked as a "favorite" for the current user, which is stored in a prefs doc for the user
  • The collection of Post documents for the Topic
  • For each Post document:
    • Summary and rich text data from the document
    • The display name, title, and signature of the posting user, which are stored in their Person document
    • The number of posts from the posting user that are visible to the current user, updated live (heh)
    • Whether or not the current user has rights to edit the Post document
  • Any surrounding page elements, such as the count of unread topics from the front page, the "who's online" list, etc.

As you can imagine, writing each one of those as inline SSJS turned into a hairy, slow mess. And that's just for the Topic page! That's leaving out the other pages, such as the forum index, which has to show a tree hierarchy of Categories and Forums with up-to-date Topic and Post counts.

I tried wrapping these things up into SSJS functions in a script library, which helped a LITTLE, but it still sucked. The clear solution was to wrap these all in proper model objects and, after a number of revisions, I came up with a pretty good solution.

Conceptually, the back-end structure consists of:

  • Collection Managers. These are analogous to Views, in that there's one per document type and they communicate with a similarly-named uncategorized View with many sortable columns. I write methods in these managers to allow for returning an appropriate collection for the circumstance (e.g. getPostsForForumId(...) and getPostsForTag(...)), as well as generic methods for getting arbitrary collections for a given key, sort column, and FT search query.
  • Collections. These are analogous (and often wrap) ViewEntryCollections. They implement the List interface to allow for use in xp:repeats and handle efficiently getting the needed model objects and re-fetching the View and Collection if they're recycled away.
  • Model Objects. These are what you'd think: Post, Topic, etc.. These provide getters/setters for the fields of the document, getters for associated collections (e.g. topic.getPosts()), and handle saving the document back to the DB.

The net result is that pages went from having giant blocks of lookup code (or SSJS function calls that pointed to giant block functions) to nice little EL expressions like #{category.forums} and #{forum.latestPost.createdBy}. As importantly, this allowed for a lot of caching to make these relational operations speedy when dealing with live data in a Domino DB.

Next time, I'll go into some specifics about how I structured the collection and model classes.

XPages MVC: Experiment I

Tue May 22 17:28:00 EDT 2012

Tags: xpages mvc
  1. XPages MVC: Experiment I
  2. XPages MVC: Experiment II, Part 1
  3. XPages MVC: Experiment II, Part 2
  4. XPages MVC: Experiment II, Part 3
  5. XPages MVC: Experiment II, Part 4

Now that I've had a bit of time, I've started trying out some ideas for new ways to do XPage development. Specifically, I'm trying out the "XPages as the Controller" setup I pondered last time. The general goal is this:

  • There would be one XPage for each zone of concern (I don't know the right terminology): Posts.xsp, Users.xsp, etc.
  • The XPage itself would have almost nothing on it - it would exist as a trigger for the server to load an appropriate Controller class, which would in turn handle the output, usually by picking a custom control by name.
  • The custom control will handle all the appearance logic... and ONLY the appearance logic. The Controller will feed in any variables and data sources that are needed, and then the custom control will just reference those values entirely via EL.
  • The Controller will marshal will either create Domino View data sources directly or will use some wrapper Model class, in order to allow the XSP environment to handle efficiency and serialization.

So far, I've made some interesting progress. I originally set out to use a ViewHandler class to find the action and feed in the variables, but I ran into trouble getting to the viewScope or an equivalent in time for on-page-load value bindings to access them - either I was too early (before createView) or too late (after createView). What I really wanted was the beforePageLoad event, so I did just that: I created a view-scoped managed bean called "controller" that has some code to check what XPage is being loaded and to find a Controller class of the equivalent name (e.g. "Posts.xsp" → "xsp.controller.Posts"). Then, I do beforePageLoad="#{controller.control}" (I apologize for the names - I may change it to "router"), and it works nicely. The controller manager thing loads the class, asks it to figure out the current action ("list", "tag", etc.), and then tries to call a method by that name via reflection.

It sounds a bit weird, and the implementation is sort of half-baked, but the result is that you end up with a class with a structure like this (with actual implementation stuff removed):

public class Posts extends AbstractController { public void list() { ... } public void show() { ... } public void tag() { ... } }

Each of those methods corresponds to an action that can either be implied (like "list" being the default for Posts.xsp, established via other code) or derived from the URL, like "/Posts.xsp/show/1" to show the post with ID "1". The code in the method would fetch the appropriate object or collection and attach it to the XPage (as a dynamically-added data source, for example). Then, the XPage would load up an appropriate custom control for the action (for example, "pages_list" or "pages_show"), which would know what values to look for in the view scope.

I'm going to keep working on this, but I can think of a couple potential muddy areas:

  • I'm not sure how real data-backed custom controls (e.g. a list of posts by month in the sidebar) work into this. You can add beforePageLoad code in a custom control, but the view root is still the main page you're loading, so it's not as clean.
  • I'm not sure if it's worth trying to shuttle data source actions through the controller (like "save" for a document) or best to reference the methods in the model from the XPage directly. Since the environment is already "impure", I don't know how much bending-over-backwards is warranted.
  • The whole thing may be an exercise in cramming Rails-isms where they don't fit, like when someone familiar with SQL first starts programming for Domino. Just because something works doesn't mean it's correct for the platform.

I'm sure there will be more sticking points, too. Overall, though, this feels mostly good, or at least conceptually better than the other development paradigms I've tried with XPages. As the post title indicates, though, I expect this to be experiment number 1 of many.

More Musing About Controllers

Sun May 13 08:45:00 EDT 2012

Tags: xpages mvc

I've been thinking more about this MVC thing, thanks to re-learning Rails. I'm fairly convinced that moving as much code as possible out of the XPage and into wrapper objects is a big improvement (convinced enough that it feels silly that wasn't doing it already), but it still feels not quite right.

Two parts of the MVC trinity are straightforward in modern Domino development: Forms and wrapper objects are the Model and XPages are the View. The Controller is where things get muddy. Off the top of my head, routing and controlling are handled by:

  • Domino's built-in URL handling
  • "Display XPage Instead" on Forms and Views
  • Web rules in the Directory
  • navigationRules on the XPage itself
  • The XPage's names themselves, as used in URLs
  • Anything fancy you do with document mode switching and multi-page XPages

It's kind of a mess, and very few of these are really programmable. You can do a lot with web rules, but the application itself doesn't know about them, so you end up with code that's coupled with a specific server configuration - very unsafe. Domino's built-in URLs work well enough for basic operations - show a view, open a document, edit a document - but don't understand more complex concepts, like a shopping cart "Checkout" (yes, I just re-did the depot tutorial from the Rails book).

Honestly, it might be enough to move the action code out of the XPage and into the wrapper classes. That's a big step in readability, maintainability, flexibility, and error prevention, and most sites get along just fine with a lot less. Still, it doesn't feel right.

I don't really have a better option, though. This morning, I've been kicking the idea around that you could treat the XPage itself as the Controller, sacrificing URL and code cleanliness for some power. In this setup, a main XPage would correspond to an aspect of your application (say, "AdminSite") or management tools for a data type (say, "Posts"). You would construct your URLs by chaining path info past the ".xsp" part and then use the main page (or, ideally, a shared library across all pages) to figure out what the real request is. You'd end up with an extra level to your URLs, like "Posts.xsp/search?q=foo". It's horrifyingly ugly, especially right after "whatever.nsf", but it makes a sort of sense: ":application/:controller/:action?:parameters". The alternative now is to have multiple XPages that each deal with very similar things and, if you have a large app, name them in a structured way anyway, like "Posts_List.xsp" and "Posts_Search.xsp?q=foo".

You'd have to put the individual action code in custom controls with loaded parameters to match the action - this would add yet another type of element to the custom controls list ("subforms", actual custom controls, and now the View component of each action), but it may be a necessary evil. The Extension Library has something like this, though not exact, in its "Core_DynamicPage" demo.

The whole idea seems a bit half-baked at the moment but worth investigation, so I'll have to tinker with it a bit when I have some time.

Separation of Concerns

Wed May 09 19:53:00 EDT 2012

Tags: xpages

A while back, I wondered about the right way to write XPages. Things have changed a bit since then - Domino has gotten a bit better, the Extension Library exists and is great, and I'm a bit more adept with the environment. The forum app, which I should probably write a post explaining one day, came together kind of like how I mentioned there - Java classes to wrap all of the Domino access, which dramatically reduced the amount of code in the XPages themselves.

Still, I'm not sure I'm doing it right most of the time.

I've never done extensive work in a proper MVC environment, but I've gotten a bit of a taste for it on the occasions that I've tinkered with Rails. I feel like the components are there in XPages, but none of the official documentation encourages using any of them, nor does Designer encourage programming that way. You can do it in bits, though - for event handlers, you can point the action to a Java method via EL (like "#{someObject.doSomething}", if I recall correctly), and the "Next page" things in the XPage's properties panel use this underlying xp:navigationRule architecture, which is probably a mix of View and Controller, but it's an interesting idea.

I may have been on to something with the forum app, but I had to create a very aggressive caching system to make it at all practical. The end result was good - very little code in the XPages themselves - but there was more "framework" necessary than I think I'm comfortable with. Still, that discomfort isn't as bad as seeing a lot of business logic in an XPage (yes, I am aware that my main side project is a way to make it all the easier to do this).

I feel like the "right" way to do it would be to really restrict the XPage itself to be a page-layout engine, concerned only with declaring widgets and connecting them to actions on data. However, you can't get very far with just the components you're given, since you're eventually going to want to, say, set a couple values on a document programmatically. You can sort of stumble along with some of the built-in actions, but that's not necessarily any cleaner... just more XML-y.

Maybe the correct thing to do would be to make it easier to expand on Controllers or Models (my MVC knowledge gets shaky here) in code in Designer so that you could, for example, define a publish action for a blog post document that handles setting the published date, changing the status, and saving it. That way, the XPage would be referencing the actual job to be done - publishing the post - while the back-end code would handle the implementation details of replaceItemValue and so forth. Maybe it'd be best to treat Forms like an object definition - a combination of instance members (fields) and actions to be performed. They're already on their way to being mostly structural support for XPages apps.

I may try to structure my next XPage like this, perhaps writing wrapper objects that take Documents or DominoDocuments in their constructor and providing all the actions that I'd want to perform on them. I could set that up as a dataContext perhaps, and then use its methods instead of adding inline code on the XPage. It's worth some experimenting, I think.

Swapping Between JRuby Embed Methods

Tue May 08 10:23:00 EDT 2012

Tags: jruby xpages

When going about making Ruby in XPages work, I had to figure out which of the three JRuby embed methods to use: Core, JSR 223, and BSF. For my purposes, it seems like the need for BSF was obviated by JSR 223 (there are some differences, but I'm not sure they matter to me), so it's really a choice between the first two. I've tried both, and I keep swapping between them (the next beta, if things stay stable, will likely switch back to Core) and between various configurations inside them, due to various problems I've run into on a couple points:

  • Performance. This one is pretty straightforward - I want pages to load as fast as possible, ideally with no discernible difference between a JavaScript-only XPage and a Ruby-laden one.
  • Classloading. Ruby should be able to access the same class library as Server JavaScript, such as the javax.* classes and any custom classes in the NSF. If I just go the default route for each embedding method, Ruby can't access those classes, but setting the classloader may or may not work with a given configuration.
  • Clean Termination. This one's tough to tell. The "Core" route has a .terminate() method, but I'm not sure if this is strictly necessary or if it's just a good idea to do it when you're going to be spinning up a lot of runtimes, but otherwise it's fine to let the garbage collector do its thing. This should give me an excuse to try out some of that heap dump stuff.
  • Independence. By this I mean that, ideally, each page load and its associated script libraries will be completely independent of future calls - for example, a class declared in a script library should only be available when that library is loaded for the page and not otherwise. This conflicts with performance heavily - the fastest way to do it is to just have a single runtime, either for the JVM (via a singleton) or for the application, but this could lead to lingering methods and classes.

On all these metrics, it seems like Core is more flexible, but JSR 223 is the "right" way to do embedded scripting, being as it is the overarching framework for scripting languages in Java now.

Regardless of which of the two I choose, I have four different types of runtimes to choose from: threadsafe, concurrent, single-thread, and singleton. Those are more or less in order of "most independent" towards "fastest". However, it's not as simple as just choosing how fast or safe I want the code to be - I've also run into weird problems with some setups regarding the classloader and OutOfMemoryErrors. When I use the singleton and single-thread routes, everything is speedy, but class loading works unpredictably - it seems to SOMETIMES use the JSF runtime classloader, sometimes not, and sometimes it's different between my dev and production servers. I'm not entirely sure what causes the memory problems, but I think I've seen them on the concurrent and threadsafe routes.

For now, I have it set to use Core with the single-thread context and calling .terminate() at the end of each request. It SEEMS to be stable, but the home page in particular takes painfully long to load. I'll have to fiddle with it some more before I can safely release another beta.

A Quick-and-Dirty "$$ViewTemplateDefault" (-ish) for XPages

Thu May 03 20:16:00 EDT 2012

Tags: xpages

8.5.3 brought with it the very-handy "Display XPage Instead" property for views. It's great! That way, you can keep more of your existing URLs in old apps or just generally use cleaner ones in new apps - XPages are awesome, but ".xsp" in the URL is not.

In a full-blown app, you're probably going to point each view to its own XPage containing the hand-crafted fancified version. Sometimes, though, you just want to toss a xe:dynamicViewPanel on a page and that's good enough. However, unlike with the equivalent property for Forms, you can't just put a xp:dominoView data source on the page and have it pick up the view the XPage is replacing. With a Form replacement, the server translates a URL like "/view/document" to "/page.xsp?documentId=whatever&action=openDocument" for you; for a View, however, all you get is "/page.xsp". This stymied me when I just wanted to make a generic "View.xsp" to use as scaffolding for new views until I make a real page for each one.

Fortunately, though context.url (context.getUrl() in SSJS) shows the XPage path, you can use a longer property facesContext.externalContext.request.requestURI to get the original request's path info (the part after the server name - the other components are also available in that object). You can use that to fetch what's to the right of the database name (you could use ".nsf/", but facesContext.externalContext.requestContextPath + "/" would probably be technically safer in edge cases, assuming you can use ".nsf" in a folder name on a Domino server) to get the view name (extra line breaks for display purposes):

<xp:dominoView var="dbView">
	<xp:this.viewName><![CDATA[${javascript:
		java.net.URLDecoder.decode(@Right(
			facesContext.externalContext.request.requestURI,
			facesContext.externalContext.requestContextPath
			+ "/"), "UTF-8")
	}]]></xp:this.viewName>
</xp:dominoView>

Not beautiful, but it works. With that and a xe:dynamicViewPanel, you can get a sort of $$ViewTemplateDefault replacement, though you still have to specify it for each view.

Arbitrary Scripting Languages in XPages

Thu Apr 19 20:12:00 EDT 2012

I think I've settled on JSR 223, the generic "Scripting in Java" specification, as the likely best way to embed Ruby. It seems like the "correct" way to do it and generally the cleanest. I don't like the notion that the way to customize the runtime is by setting system properties, so I'm still a little wary, but it'll do for now, in any event.

The side benefit of JSR 223 (and this would be true of BSF as well) is that it supports a crapload of languages, and it does so in a very unified and generic way. Accordingly, I modified my ViewHandler and created a GenericBindingFactory to browse through the list of non-Ruby available languages and create EL bindings for each.

The upshot is that, when this code is active, any JSR-223-compliant language in your server's classpath will become available for "#{whatever: ... }" bindings just by virtue of its presence. Note, though, that that doesn't necessarily mean it will be a good experience. For one, the page's variables (like param, facesContext, view, and so forth) aren't just available - you'd have to make something like the method_missing method I wrote for Ruby to automatically resolve them via FacesContext.getCurrentInstance(). Furthermore, some languages are problematic: for some reason, Jython throws NullPointerException errors for even the most basic formulas on almost every page load, while PHP still requires <?php ?> tags and spits anything else out to the server console.

I know what question is on the tip of your tongue, though, so I won't keep you waiting: yes, you can use your home-grown string-concatenation operators in Scheme! You can breathe a sigh of relief now:

<xp:text><xp:this.value><![CDATA[#{scheme: (set! + (let ((original+ +)) (lambda args (if (and (not (null? args)) (string? (car args))) (apply string-append args) (apply original+ args))))) (+ "Hello " "from " "Scheme") }]]></xp:this.value></xp:text>

Yes, that works, and no, there's no reason to do it. If you really like Lisp, though, Scheme and Clojure can now be in your XPages toolbox.

Once I'm able to post to OpenNTF, I'll include this in my first Ruby-in-XPages release, though I may leave the applicable code commented out by default. Who knows what horrors this could unleash?

An Action Plan for Ruby-in-XPages

Fri Apr 13 11:55:00 EDT 2012

Tags: xpages ruby

So far, my naive implementation of Ruby bindings seems to be working out well and I've been coming up with a list of the main things I want to tackle next:

  1. Figure out if I'm embedding it correctly. I suspect that working this out will go a long way to solving some of the later items in this list, but it's also the best starting place anyway.

    For now, I'm using the "Core" method of embedding JRuby, which is the most straightforward way to do it: create a runtime, execute your scripts, and terminate it. It's also not so simple that it's not flexible: it has a couple different configurations for dealing with threads and variable context, so it's entirely possible that it may, in fact, be the correct way for me to do it.

    However, there are also a couple other methods: JSR 233, BSF, and directly via the JRuby API. I don't know enough yet about the XPages Java stack to know if either of the former two are at all applicable, so they may or may not be worth investigating. Direct use of the API is clearly the most flexible way to go, but I'm not certain that I NEED to get that arcane. ScriptingContainer may end up working fine. There's also this: http://jruby.org/apidocs/org/jruby/embed/osgi/OSGiScriptingContainer.html.

  2. Serialization. The ScriptingContainer itself isn't Serializable, but that doesn't necessarily mean that other parts aren't or that general state can't be retained. Ideally, each XPage's Ruby environment would survive Serialization/restoreState intact, closures and all.

  3. Variable lifetime. I want to figure out how best to handle variable scopes. Conceptually, I think it'd make the most sense for basic local variables ("foo = 'bar'") to be available for the duration of the page. But is that really the right way? As it stands right now, variables are local to the value binding where they're declared (as if they're variables in a function), but defined methods are available elsewhere. Here's what I mean:

    <!-- This produces an error because the second xp:text can't find foo -->
    <xp:text value="#{ruby: foo = 'bar'; foo}"/>
    <xp:text value="#{ruby: foo}"/>

    <!-- This produces "barbar" because the "foo" method persists to the second binding -->
    <xp:text value="#{ruby: def foo; 'bar'; end; foo}"/>
    <xp:text value="#{ruby: foo}"/>

    That might actually be the right way to do it. However, there are other aspects to it. For example, setting a constant (in Ruby, a constant is a variable with an initial capital letter) currently persists between pages, presumably lasting for the duration of the application. THAT certainly doesn't sound right to me. Furthermore, what about other languages? If I make it so that local variables are available between Ruby bindings, should I try to export those variables to the overall page context so that pure-EL and JavaScript can get to them? I wonder if that'd be possible with methods, too.

  4. Spruce up the Domino API a bit. Ruby lets you "re-open" classes at runtime, adding whatever additional methods you want. In JRuby, this power works for Java classes, too. Allow me to demonstrate ("post" is a xp:dominoDocument):

    module Java module LotusDominoLocal class Document def [](field) self.get_item_value(field) end end end end post.document["Form"]

    I think you can imagine how this could improve your programming life.

  5. Pre-compiled JRuby classes. This isn't strictly part of "Ruby-in-XPages", since it'd be done via an Eclipse builder, but I think it'd be worth looking into. JRuby can be used to generate Java classes that can be compiled ahead of time with jrubyc. The generated classes would still require the JRuby runtime and would thus be presumably slower than the equivalent "pure" Java version, but they'd be faster than interpreted or JIT-compiled Ruby that's re-generated on every page load. Ruby programmers write everything in Ruby, and I want to write my back-end logic (and agents) in Ruby too.

    I found an old Builder class I made to accomplish this task, but it's very filesystem-bound, which I'd rather avoid. Still, I'm glad I dug it up to serve as a jumping-off point.

Best of all, I think this is all eminently practical. I'm excited. Are you excited?

Making A Project Out Of The Blog and Ruby

Tue Apr 10 19:47:00 EDT 2012

Tags: xpages

Now that I have the blog in Domino and acting as an accelerating incubator for Ruby-in-XPages (man, I need a better name for that), I decided I may as well put 8.5.3's filesystem synching to good use and put the whole thing up on GitHub with everything else:

https://github.com/jesse-gallagher/frostillic.us-Blog

I've also started keeping a todo list in the README file there with my notes and future plans for the blog, the Ruby runtime, and a tentative "Misc" section.

As for the interpreter itself, it's been coming along nicely. I've got it so it only needs to create a ScriptingContainer once per request. It's not perfect, and ideally it'd only be one per Application or server - the LocalScriptingContext.CONCURRENT mode seems tailor made for this. As it is, I'm pleased that Domino can serve the HTML for the blog's home page in under 300ms, considering most scripting computation is done in an "alien" language. It's a simple app and it's not Basecamp 2 speed, but it's a start.

I have script libraries (of a sort) working. If you drop Ruby files into /WebContent/WEB-INF/ruby, you can reference them with familiar syntax:

<xp:script clientSide="false" src="/testlib.rb" type="text/x-ruby" />

Unfortunately, it doesn't yet pull in script libraries from themes (I think) and I don't know if it's available on beforePageLoad, but, again, it's a step in the right direction. Once I can get a hold of the AD103 slides from LS2012 or find similar information elsewhere, I'll look at surfacing them as a design element.

Ruby-in-XPages is Inching Towards Practicality

Wed Apr 04 21:20:18 EDT 2012

Tags: ruby xpages

I've made some more encouraging progress today in my quest to be able to actually use Ruby when doing XPages development. I implemented method binding (as opposed to just value binding) and found a way to shim in the interpreter so that Ruby can be used for early events like beforePageLoad. The revised code is up on my GitHub profile:

https://github.com/jesse-gallagher/Domino-One-Offs

That last part was kind of tough - I found structures for custom Application objects in the XSP Starter Kit, but I couldn't for the life of me get the faces-config.xml directives to work, like the runtime was ignoring the /faces-config/factory/application-factory node entirely. I set that aside for now and instead picked up on ViewHandlers, which can fire at the start of each page request, which works for this purpose. I wrote a custom handler that adds the RubyBindingFactory if it's not present and then otherwise hands control off to the standard Domino variant.

Another big hurdle I encountered was variable resolution. When making my proof-of-concept servlet, I inserted a couple standard variables, but that wouldn't cut it here - beyond the usual suspects like session and view, I had to make sure that Ruby could find local xp:repeat variables and other non-globals. After trying to find a way to either get a list of available variables in the current Faces context or write a proper variable resolver to attach to the Ruby runtime, I decided to just go the Ruby route for now: method_missing. When instantiating a Ruby runtime, I add facesContext as a global variable and run a bit of hard-coded Ruby to create a method_missing method to act as a variable resolver. Good enough!

So what's next? A couple things I can think of:

  1. Efficiency and doing it the right way. The whole thing is still a bit too hacky for my tastes. It may end up being the best way to do it, but I haven't given up on refining it. Additionally, I currently create a new Ruby runtime for each value and method binding, which is overkill - I think there should be one per page.
  2. Designer. Though Designer rarely outright complains about Ruby code (it doesn't like Ruby's #{} string-interpolation syntax for understandable reasons), I'd love to get it to stop throwing warnings around, but that's not a high priority.
  3. Edge cases. xp:repeat controls appear to do their own syntax-check pass that doesn't like Ruby, so I'd like to figure out how to work around that.
  4. Ruby "Script Libraries". If nothing else, I'd like to add the ability to bring in .rb file resources in a similar way to you reference Server JavaScript. Presumably, I could do this by checking, at initial page load, for controls with certain properties and then fetching the script content like I did for servlets.

I also remembered another potential related use for JRuby: writing Java classes in JRuby and having Designer translate them to Java design elements. I did a little bit with this a while ago, adding a custom builder that would fire for .rb files, use JRuby's API to translate it to Java code, and then store it. If I recall correctly, Designer's built-in builders even automatically took the Java and compiled it. If that would actually work smoothly, it should result in code performance somewhere in between pure Java and Server JavaScript.

I'll turn Domino into a developer-friendly environment if it kills me!

In Between My Project and XPages

Thu Mar 15 16:11:56 EDT 2012

Tags: domino xpages

Despite my grousing about the state of programming for Domino in general and Designer in particular, I'm still mostly a fan of XPages. I use it for my guild's web site and pretty much every new project at work. However, I haven't been able to crack migrating my main work database template over.

Without getting too much into it, the point of the template is to create one database per project to act as a project web site listing online events with arbitrary registration forms and exit evaluations (among other things). Except for custom changes, everything is done via the normal Notes client, not Designer - pages exist as documents with a Body rich text field and associated data and register forms/exit evals, crucially, contain a set of response documents describing their fields. It's important to keep everything as visual as possible, because the users (the people at my company that set up these web sites) are not programmers.

I scrupulously try to avoid modifying the design of the database, so I go out of my way to do custom work via the existing mechanisms as much as possible, and I make pretty good use of rich-text-isms like embedded views, tabbed/computed tables, attachments, buttons, and computed text. In the case of the generated forms, fields can be placed either automatically (with a surrounding template of HTML per-field) or directly into the rich text body via <%FieldName%> placeholders (creating an experience like form design in Designer). Additionally, I have "WebQueryOpen" and "WebQuerySave" fields in the documents for formulas that I pass on into @Eval() in the equivalent places in the actual forms; most of the time, I use these for running agents.

Computed Text

Of the various potential show-stoppers, I think computed text fares the best. For one, it mostly worked - the computed formulas are indeed evaluated and the result is put into place correctly. However, they don't have the same environment you get with the "classic" design elements. The big one that I ran into immediately was @UrlQueryString(...) - it appears that the rich text renderer doesn't inform the rich text about its web environment completely. Prior to 8.5.3, "Display XPage instead" pages didn't know about the real URL, so they couldn't get query parameters at all, but 8.5.3 appears to pass that information along properly. So that means it MIGHT be fixable, if I find a way to properly set fields in the document before the rich text is rendered, so I can set QUERY_STRING - if I can do that, either @UrlQueryString(...) will work or I can manually parse the string as needed.

Embedded Views

They work! ...ish. It looks like icon columns get their URLs a bit messed up, but I can't think of a time when I actually used them, particularly in a situation where I couldn't just write out the URL myself.

Tabbed/Computed Tables

From a cursory glance, I think I'd be SOL when it comes to these. Tabbed tables render flattened out and computed tables don't seem to work. The former can be improved on easily with Dojo, but the latter would mean replacing server-side business logic with client-side JavaScript, which would lead to headaches.

Attachments

These show up as "(See attached file: foo.jpg)". So... not functional. I might be able to fix it by using a filter on the text to replace the HTML with a link to the document, but I don't know if I could get the attachment image properly. Attachment images aren't amazing, but sometimes they do the job.

Buttons

Nope.

WebQueryOpen/WebQuerySave

In some cases, I could run the formulas through session.evaluate(), though I'm not even sure queryOpenDocument/postOpenDocument let me hook into the right spots (sometimes I set fields on document open). I don't think that would run agents, though, so I'd be stuck trying to parse out the text to look for @Command([ToolsRunMacro]; "...").

User-generated Forms

This is the toughest one. I've given this thought a number of times, and I can't think of a great solution. I can't just re-use the subforms I have already, I don't think I can generate and import an XPage via DXL (though I haven't given that a significant shot), and I can't just do a <xp:repeat/> to create all the fields, since some are in the body area. The main routes I can think of to try are a) arranging the fields on the page after the fact via JavaScript and b) generating the page components on the fly with Java or Server JavaScript. I'll probably give the latter a shot eventually, but I expect it to be a bag of hurt.

 

It's a lot of hurdles! I'd love to switch over to XPages, particularly since, for everything that's more difficult than using classic elements, there are 10 things that are way easier. It'd just be quite an investment of time to merely get up to par with the functionality I already have, if it's even possible in all cases.

This Dynamic View Customizer Is Getting Into Shape

Mon Feb 13 10:08:12 EST 2012

Since last week, I've made two nice improvements to my dynamic view customizer:

  1. I added some support for twistie images when the referenced DB is on the same server. The code assumes that the referenced images are image wells with at least two entries, but I can't imagine why that wouldn't be the case in practice.
  2. I vastly improved my handling of color columns. Previously, I had been resorting to hacky methods like hidden <div>s read by JavaScript or surrounding <div>s styled to take up the whole cell, but those were terrible and easy to break. Now, though, I'm doing it right: the code adds a value binding for the column's style attribute to create the CSS for each cell, which is ideal.
  3. Empty categories now are translated to "(Not Categorized)", as in the client.

Now that my code is presentable (albeit oddly structured and uncommented), I figured I may as well toss it up on GitHub:

https://github.com/jesse-gallagher/Domino-One-Offs/blob/master/mcl/reports/DynamicViewCustomizer.java

To note if you want to use this in your own project: it references the "mcl.JSFUtil" object, which started as the mindoo object of the same name and has since turned into my bin for common functions. The methods used here are getSession(), which just gets the value of the JSF "session" variable, and the xmlEncode() and specialTextDecode() functions from my string utils. Additionally, it references "com.raidomatic.xml.*", which are quick wrapper classes I made to ease basic XML access, and which are also available on GitHub:

https://github.com/jesse-gallagher/Domino-One-Offs/tree/master/com/raidomatic/xml

Enhancing xe:dynamicViewPanel For My Own Purposes

Thu Feb 09 08:45:30 EST 2012

I think I have my view rendering problem licked. To recap, I've been working on a way to show views in XPages that met a couple requirements:

  1. Entirely dynamic. Since this will be for a combined reporting site that will show views from customized project databases of wildly varying needs, I couldn't make any assumptions about view layout, categorization, or content. It should pull as much display information from the view design as possible.
  2. Fast. Some of these views have thousands of rows, so it should be as fast as possible to load from the database and render for the browser.
  3. Pagination. Though I don't really like pagination for the average case (who wants to look at a 40-row view 30 rows at a time?), sending thousands of table rows to a browser causes a miserable experience even when that browser isn't IE.
  4. Full-text searching and other Domino goodies. I don't want it to be a pain to just be able to search through a view like you would in the Notes client.
  5. Support for "second tier" Domino view features. I use special text, column hide-when formulas, "show values in this column as links", Notes-style [<span>pass-through-HTML<span>], and color columns constantly.

Of these, 1 and 5 are the most important.

I originally wrote my own code to generate a big HTML blob for the view, which was perfect on points 1 and 5 (and looked great, since I converted categories to OneUI sections), but wasn't as hot on the other points. The Extension Library's xe:dynamicViewPanel control hits points 1 through 4 with aplomb, but misses the crucial point 5. For one crazy moment, I toyed with the idea of trying to get the output of the "legacy" HTML view renderer (which is perfect on all points) via client-side JavaScript or some other hack, but decided against it as being too rickety.

I considered changing my custom HTML code to instead create a bunch of Java objects to make pagination easier, but performance would be an issue - it's tough to beat built-in controls for raw speed. As anyone who has looked at the code for the mail template knows, IBM/Lotus cheats like crazy, and often the only thing you can do is to just go with what they've done and beat it into shape. Thus, the answer came to me: customizer beans for the ExtLib's dynamic view panel.

The dynamic view panel has a "customizerBean" property that takes a string class name of a bean to create to hook into a couple overridable methods and events:

  • ViewFactory getViewFactory() - this lets you override how the panel pulls in all of its information about the view.
  • boolean createColumns(FacesContext context, UIDynamicViewPanel panel, ViewFactory f) - this lets you override how the panel uses the gathered view information to generate the list of data table columns (or simply run code before that happens, if it returns false)
  • IControl createColumn(FacesContext context, UIDynamicViewPanel panel, int index, ColumnDef colDef) - similarly, this lets you change the way each individual column is created based on the ColumnDef (an object generated by your ViewFactory)
  • afterCreateColumn(FacesContext context, int index, ColumnDef colDef, IControl column) - this is an event fired after each column is generated, allowing you to customize the generated column without having to build it from scratch
  • afterCreateColumns(FacesContext context, UIDynamicPanel panel) - similarly, this lets you run code after the column creation is done

These events provided just the hooks I needed to get the dynamic panel to do what I wanted.

Special Text

To add special text support, I added an implementation of afterCreateColumn(...) that replaced the column's default ViewColumnConverter with my own subclass that overrides getValueAsString(...). With that, I can walk up the current component's ancestors to find the panel, find the individual ViewEntry's variable name from there, and use that to process the special text properly.

Column Hide-When Formulas

These are a bit trickier. The getViewFactory() hook lets me provide my own subclass of DefaultViewFactory, but there's a reason that the dynamic view panel doesn't support these formulas by default: they're not exposed by the NotesViewColumn class. The two ways I can think of to get at this information are the C API (or otherwise reading the design information from the view note directly) and DXL. The former is no doubt significantly faster, but DXL is much, much easier and is good enough for my needs at the moment. So I export the view as DXL and process its column nodes and their children. Then, for each with a hide-when formula, I evaluate the formula in the context of a new document in the project's database and use the result to set the column-hidden flag.

"Show values in this column as links"

This goes hand-in-hand with the hide-when formulas. The column nodes in the DXL have a showaslinks attribute that I can look for to set this flag.

Pass-through-HTML

Much like special text, once I have the custom converter attached to the column, I can break apart and reassemble text values on [< and >] and XML-encode the non-HTML bits. While I'm attaching the converter, I set the column's content type to "html", since I'll be handling the encoding.

Color Columns

These are... tricky, and I'm not yet proud of my fix. There are two problems: getting each cell to know about its color information from previous cells and then actually applying that color information. For the former problem, I dealt with it by adding an "activeColorColumn" property to the ColumnDef object and setting it to the programmatic name of the most recent color column before each column. The latter is the ugly part. You can't just set the column's style, since that will apply to the entire column and make a fine mess of everything. What you really want to do is assign it to the current <td> element, but I don't know a good way to do that. In the mean time, I've hacked around it by making a hidden <div> in each cell with the style tucked into a data-cell-style attribute. Then, I have some client JavaScript look for these divs and apply their styling to the parent table cell. It's not pretty, but it works for now.

If I get that last bit of code cleaned up to a point where I'm comfortable letting people see it, I'll post my example code. In the mean time, I'll count this whole thing as a win for only writing the code you need to and letting the platform take care of the rest.

More Idle Complaining About Designer

Sun Jan 22 18:35:08 EST 2012

Tags: xpages

As more and more of my Domino development has been with XPages, my databases have filled up with XPages, Custom Controls, and Java design elements. However, the more I use these, the more of a pain Designer becomes. It gets into its head on a regular basis the notion that it needs to recompile every single such design element - when I open the app in Designer (whether or not it was "greyed out" in the sidebar from before), whenever it "forgets" that I have it set to Automatically Recompile and I have to turn that back on (which is frequent), whenever classes just stop loading properly and I have to Clean the project, and sometimes just for fun.

In a basic app, this isn't so bad... it's pretty annoying, but at least it doesn't take too long. In my forums app, though, which contains dozens of pages, controls, and Java classes, it can take about 10 minutes when I'm not in the office. Ten minutes! Just to open the app! That's even ignoring the side effect that the app is unavailable on the web while this is happening, throwing various server errors until all the classes are back.

I've tried numerous things: switching the Java classes to and from the 8.5.3-standard Code/Java folder to a "classic" WEB-INF/src folder (the 8.5.3 folder seems to make it worse), turning off the "Automatically recompile dependent XPages" option relating to custom controls in the prefs, and creating fresh Windows VMs with entirely-clean installations of Designer. Theoretically, I could turn off automatic compilation in the "Project" menu, but then I'd have to do it manually, and I can only imagine I'd consistently run into times where it "needs" a full recompile anyway.

It's bad enough that it takes this long when I want to get hacking away at the code, but it's all the worse when I just want to go in and fix a CSS or JavaScript file, neither of which have anything to do with the big pile of Java classes Designer is so obsessed with. Nonetheless, if I'm going to get at them, I have to let Designer do its thing. There are potential workarounds - write a web-based editor for those design elements, store them in other databases, or store them as data documents - but those sacrifice either the convenience of a proper editor or of having all of the assets of my app in one place.

Other than my job, there are only a couple thing tying me to XPages for my for-fun projects: the fact that I now have a pretty huge foundation of knowledge in Domino development, the fact that XPages actually make a lot of fancy things very convenient, especially with the illustrious Extension Library, and Reader fields. The last one of these is surprisingly difficult to reproduce elsewhere. They're so elegant and bulletproof, much better than filtering results, writing giant nested SQL queries, or throwing up my hands and sticking with a basic "access level" type of restriction. And all of those would still mean I'd have to worry about ensuring that I don't accidentally select the wrong records anywhere I write a query, whereas Reader fields mean that the documents effectively don't exist, so I CAN'T screw it up.

So that leaves my main options as:

  1. Suck it up and keep using XPages
  2. Suck it up and give up Reader fields in exchange for a programming environment suitable for human use
  3. Stop being lazy and write access classes in Ruby or PHP that use the DAS API, hoping that performance is good enough
  4. Close my eyes and wish real hard that a bug fix patch will come out and magically fix all of the problems in Designer

Number 4 would be ideal, but I think number 1 is what I'll most likely do. Still, being able to use Ruby instead of Java would be such a breath of fresh air, and turning Domino into a "mere" database server via DAS both is appealing and would be mostly new ground. Given that the database-open operation I started halfway through this blog post is in its 16th minute and still going, maybe I really should get off my duff and try that out.

Edit: 32 minutes! Nice!

That Counts as Progress

Tue Nov 22 16:28:21 EST 2011

Tags: domino xpages

A while back, I described the problem I'm having in my guild-forums XPages app, which is that it very easily gets its environment out of whack, to the point where changing any design or data note from outside the XPages environment caused an "X is incompatible with X" ClastCastException. This improved gradually over time. At some point in 8.5.2, I started being able to modify data documents again without the problem. When 8.5.3 came out, it improved again: I can now replicate over changes from my development server to the production one without having to bounce HTTP on the production one. At that point, it became much less of a hassle - sure, I still had to re-save a Java class file every time I modified an XPage, but that was only in development, so I could deal.

However, I think I've found the proper solution. In 8.5.3 (I think), IBM added a custom editor for the xsp.properties file (which you can reach using the "Package Explorer" Eclipse view, in whatever.nsf/WebContent/WEB-INF). Many of the options are duplicates of what you see in the normal "Application Properties" editor (since they edit the same properties), but there are some nifty additional goodies. I won't go into them all, but I urge you to take a look. The important one here is in the "Timeouts" section of the "General" tab: "Refresh entire application when design changes" . That sounded perfect and, lo and behold, it seems to do what I want: once I turned that on, re-saving an XPage stopped causing the ClassCastException. I haven't given it a proper testing, so it might not be everything I'm looking for, but I'm thrilled at the initial results. Having to re-save a Java file for each change wasn't a HUGE problem, but it was a niggling one, and not having to jump through the hoop helps make the whole environment seem less rickety.

That's Weird

Wed Nov 09 09:35:21 EST 2011

Tags: xpages domino

Yesterday, I started working on a small sidebar widget app using an XPage, after finding out that XPages can now (as of 8.5.3) be used in the Notes sidebar in the same way that Forms could before. It's quite a simple page, very Twitter-like: one text input field and then a list of posts. However, even though it's very simple, I ran into two annoying bugs quickly.

The first of them is a Schrödinbug. I set up some code in the "onClientLoad" event to start a setInterval to do a partialRefreshGet on the list of posts, so it would keep itself up to date:

<xp:eventHandler event="onClientLoad" submit="false"> <xp:this.script><![CDATA[ setInterval(function() { XSP.partialRefreshGet("#{id:messagesPanel}") }, 2000) ]]></xp:this.script> </xp:eventHandler>

Simple, right? But I started getting errors in the status bar, along the lines of somethingorother not being defined. That's... odd. So I swapped over to Safari, figuring it'd be some XPiNC-specific thing, but I started getting similar (albeit more specific) errors there, along the lines of:

TypeError: 'undefined' is not an object (evaluating '_166.formId')

Huh. So I tracked it down and the problem was in the code called by partialRefreshGet, where "_166" is the name given to the second (optional) parameter. It makes sense when seeing it - though the code is supposed to fail over when it doesn't work, I can see why the browser would throw up its hands when you try to get a property of an un-provided parameter. But this has worked before! Just looking around on the web, you run into plenty of examples that leave out the second parameter. But sure enough, when I changed the line to XSP.partialRefreshGet("#{id:messagesPanel}", {}), it started working great. I can only think that this is either some 8.5.3-specific bug or some weird thing I managed to do in my code... but there's not even really enough code to mess up.

The second bug is still annoying, and it's somewhere in between a bug and a security feature of Firefox/Gecko. Basically, while you can call the .click() method on a button in client-side JavaScript in Gecko, it doesn't behave exactly like clicking on the button. Specifically, I have a CSS-hidden button that executes the actual action of creating the message document from the text you type in, and I want it to happen when you hit enter. The button itself works great - if I have it show up, I can type, click, and it executes the action and partial-refreshes the list of posts. However, if I hit enter, which uses the .click() method, I get an error about not being able to refresh that part of the page, but then it forces a full-page refresh and still works. So it's KIND OF clicking it, but not quite.

I'm not sure what to do about this one. I looked up a couple things online about trying to emulate the actual click event, but with no better results. I could do a REST service on the page, but I don't want to have to roll out the Extension Library to everyone in the company. Maybe I'll look into sending along the data in a XSP.partialRefreshPost call and eliminate the button entirely. We shall see.

Trying To Escape From Designer

Tue Nov 08 11:14:30 EST 2011

Though I've grown to more or less enjoy writing Domino applications, I always feel like this is in spite of the tools, namely Designer. In a lot of ways, Designer has improved significantly over the last couple versions: as long as you ignore the speed, the Eclipse-ified Java and LotusScript editors are miles ahead of the antiquated previous ones, and it's handy to be able to switch to the Java perspective. However, so much else makes it a drag:

  • It's a Windows app. I use a Mac, so there's simply a big hurdle to using Designer. Parallels has smoothed the process greatly - running Notes in Coherence mode means I can use my preferred OS while still getting work done. However, it means that it's a big to-do whenever I want to make any tiny change in the code. For day-to-day work stuff, I can get a lot done in the Mac Notes client, since I put in a lot of work to make managing client web sites doable without going to the design side, and the Mac client still lets you edit views and agents. However, there's no XPages editor, so I can't use that for serious work.
  • It has a mind of its own. For some reason, a new clean install of Designer I made the other day got it into its head that opening any database should involve recompiling every XPage and Java class. Sometimes, it turns off "Build Automatically" for no reason, and then turning that back on also requires a full recompilation. Sometimes, it just holds up all user actions while it does... something for five minutes. What's it doing? Beats me.
  • How many times have I seen this window? A billion?
    Removing a database from the project list has about a 30% chance of causing a crash and quitting Designer has about a 50% chance. Sometimes, saving a form will do it. Sometimes, walking away from the computer to get a drink is enough. And every time it crashes, I have to dismiss the dialogs and check the task manager to get rid of any residual processes, such as an instance of nsd pegging a processor, then relaunch Notes and get back to the environment I had, which is not a speedy process.
  • I'm not that crazy about Eclipse. DDE is better than previous Designers, yes, but Eclipse is still a giant beast with the same sense of style and simplicity as the monstrous Java language that spawned it. On the plus side, the blue look that IBM came up with is actually rather attractive compared to the standard Eclipse UI, but little quality-of-life things are a drag. For example, how do you specify how many spaces a tab should take up visually? It's in a couple places in the preferences and you have to set them all, including buried inside a weird sub-preferences dialog for messing with your Java formatter. And how about changing your code syntax coloring? You have to go to a dozen places, one for each syntax type. Compare to a text editor like TextMate, where you can swap between packaged color schemes with a drop-down.

That's enough for the rant. So what is there to do about it? I give this problem a thought from time to time, and I don't think there's really a great option, but there are some places to start. The real key to any alternative scheme is DXL - using that, you can (more or less) view and modify design elements freely. I've toyed with this notion before - maybe a web UI that lets me pick the DB and design element I want so I can tweak the DXL manually for when I want to make a quick change but don't already have Windows or Designer open. It would mostly work, but it would take a lot of work to make it practical.

There's a big sticking point, too: XPages. If you export an XPage to DXL, you can see that the exporter basically punts on it - it's exported as a generic "note" type with Base64-encoded binary fields. I haven't run the field data through a decoder, so maybe one of them is the XML "source" of the page, and maybe it could be made to work, but that just raises more questions. Would that require updating the other binary fields in some way? Would importing it with a DXL importer cause it to generate and compile the Java representations? What about generic Java classes? Would those work with DXL and would they be compiled?

I'm starting to get an idea of what would be ideal and almost practical, though. One could write a WebDAV server (possibly with a complex servlet, a small web server to the side, or other trickery) that represents the design elements as editable files in a folder structure similar to the Java perspective view of the database. Traditional files and image resources could be edited as-is (which I think the built-in WebDAV server does), but design elements could be represented just as DXL and then re-imported when modified. Even if it doesn't support XPages, such a scheme might have a lot of promise and wouldn't be reliant on Designer or any other IDE.

If I ever get brave enough to delve into WebDAV or frustrated enough with Designer, I might just look into it myself.

Displaying mixed-type Rich Text in an XPages repeat control

Thu Mar 03 17:07:40 EST 2011

Tags: xpages

I'm not sure if "mixed-type" is the right term for what I mean, but it should do.

A couple months ago, I ran into a problem where I wanted to display the posts in a forum topic on an XPage, which isn't a terribly complicated thing to do. However, I ran into a bit of trouble in how, specifically, to go from the <xp:dominoView/>  data source to proper rendering of the rich text for each post, in part because some posts were MIME data and some were the native Notes rich text format. The rich text naturally wasn't in a column value, so I couldn't just refer to the column by name using the var name int he <xp:repeat/>.

I wrangled with this for a little while, writing JavaScript code to check if it's in MIME or native format, but I kept running into problems with how to convert the rich text data on the fly. I didn't want to just lose all the rich text formatting by using NotesItem.getUnformatetdText() or an equivalent, but I couldn't just return a NotesItem object to the <xp:text/> control. I knew that XPages know how to convert it, since they do it just fine if you're working with a NotesXspDocument, so I wanted some way to access that functionality.

<xp:panel/> came to my rescue. Within the <xp:repeat/>, I created an <xp:panel/> with its own data section containing an <xp:dominoDocument/> with its documentId property set to the UNID of the view entry I was currently working with. Then, I could just create an <xp:text value="#{doc.Body}"/> and let the renderer take care of all the details.

For all I know, that was an extraordinarily inefficient way to do it, server-wise, but it may be the cleanest way to do it from the programmer's perspective.

Dirty and inefficient (but programmer-friendly) SQL queries in XPages

Thu Feb 24 18:39:39 EST 2011

Tags: sql xpages

If you spend enough time working with XPages, you're eventually going to want to access some SQL data, either because of an integration project or because you just want to. It's pretty well-trodden ground, but, in one of my recent projects, I came up with an approach I rather like.

Now, before I get into it, fair warning: this is not scalable, efficient, or good programming practice. This is for when you just want to do a quick query and get back data in a usable fashion. It doesn't handle paging properly and it can easily stomp all over separation of concerns. But if your needs are simple, it may do the job perfectly.

Basically, the problem I had was this: I want to do a couple quick SQL "select" queries on an XPage, but I got tired of writing out similar "stmt = conn.prepareStatement(...); stmt.setString(...)" stuff over and over. Ideally, the solution would be to abstract it all away, but it was a small page deserving of a small fix, so I wrote myself a function:

SQLTools = {}
SQLTools.fetchQuery = function(query, args) {
args = args == null ? [] : args

var stmt = SQL.getConnection().prepareStatement(query)
for(var i = 0; i < args.length; i++) {
    stmt.setObject(i+1, args[i])
}
var rs = stmt.executeQuery()

var cols = []
var meta = rs.getMetaData()
for(var i = 0; i < meta.getColumnCount(); i++) {
    cols.push({
        name: meta.getColumnName(i+1),
        type: meta.getColumnType(i+1)
    })
}

var result = []
while(rs.next()) {
    var row = {}
    for(var i = 0; i < cols.length; i++) {
        row[cols[i].name] = rs.getObject(cols[i].name)

    }
    result.push(row)
}
return result

}

The actual job of connecting to the SQL DB with the right driver is handled by a Java bean called SQL - the aforementioned link covers that part pretty well. What makes this code different is how easy it makes a quick query with some interleaved parameters and how useful the resultant value is for standard XPage controls. For example, you could do a search like this...

SQLTools.fetchQuery("select firstname,lastname,degree from people where age > ? and lastname like ?", [30, "%son"])

...and you'd get back an array containing column-keyed hashes. Your XPage doesn't have to care about ResultSets or ".getString()"s or anything. You could end up with relatively clean code like:

<xp:repeat var="person">
<xp:this.value><![CDATA[#{javascript:
SQLTools.fetchQuery("select firstname,lastname,degree from people where age > ? and lastname like ?", [30, "%son"])
}]]></xp:this.value>

<xp:div><xp:text value="#{person.firstname}"/> <xp:text value="#{person.lastname}"/>, <xp:text value="#{person.degree}"/></xp:div>
</xp:repeat>

That's probably not a final product for displaying the information, but it gets the point across - it's a pretty clean traversal from "I want to query the database" to having XPages-friendly Server JavaScript objects.

Learning the "Right" Way to Write XPages

Sun Nov 28 13:52:27 EST 2010

Tags: xpages

XPages are a tough nut to crack. Thanks to IBM's apparent disinterest in people using their software (which is a big topic in itself), it's an uphill battle to figure out what to do with them. For simple applications, they're overkill - loading a Java framework stack that seems to go on for miles has some noticeable performance penalties. But for larger applications, how are you supposed to write them? The main tutorials I've run across usually involve replicating basic form/view functionality or porting an example Notes application over. That's great, but I already know how to make apps with forms and views.

A couple sources, such as the excellent Mindoo blog and the well-produced Notes in 9 give hints at the "other" way to do XPages development, the way involving pure Java classes and scoped beans. From what I've gathered in my development, this is the "Right" way to do it, but, unfortunately, those blog posts and videos are about the extent of what I've found. I imagine that I could learn more if I hired consultants or attended LUGs in various parts of the world, but those seem like pretty ridiculous solutions to the problem.

Since XPages are an application of JSF, I could run off in that direction, but there are severe problems with that plan, not the least of which is that the XPage tag library is very distinct from other JSF applications and so reading any of those books would be analogous to learning MFC by reading a book about Qt. Same language and same general concepts, sure, but it's not the same environment.

This ambiguity manifests in practical problems. I'm writing a forum at the moment, and I'm wondering what the best way to do it is. On a topic page, for example, I have a topic ID and I want to display a paginated view of the posts in that topic. The straightforward and functional way to do it is to use a <xp:dominoView/> data source with a categoryFilter and feed that to a <xp:repeat/> control. That works and initially is pretty clean - showing data from a view is easy. But then I want to show the author's preferred alias instead of their username, so I put in a custom control with code to look that up. Oh, and some code to determine and show the author's current avatar image. And some code to determine if the current user can edit or delete the post. And some code to look for previous versions of the post to show an edit history. And some code to add or remove the topic from the list of the user's favorite topics. And so on and so on.

None of that code is particularly crazy, and I've moved a lot of it into custom controls to keep it nice and clean, but, at this point, I've got all kinds of business logic performing actions and converting <xp:dominoView>s and <xp:dominoDocument/>s into topic lists and posts. The actual XPage itself is supposed to be the View part of a MVC setup, right? Instead, now I have this View/Controller hybrid with a dash of Model when I have to make a secondary data source to get the rich text of the post. That can't be right. Instead, I should probably write some Java classes called Topic, Post, and Author and get all that crap out of the XPage.

And therein lies the problem. I started clean with a fancy new framework, but I've been largely left to my own devices when it comes to figuring out how to use it. Plus, I've not even totally convinced that Java-ifying everything is actually the right way. Once I step away from the pre-provided Data Sources, I'm suddenly on the hook for handling efficient pagination and cacheing myself, plus "smaller" things like getting the rich text data out of the documents and to the web browser. Should I try to read the MIME entity in Java? Should I punt and provide a .getDocument() method on the Post to let the <xp:dominoDocument/> source do the work? It's not clear. Should I read in all of the pertinent data from the document as soon as I instantiate a Post, or should I leave the Document object sitting around and only pull it in on-demand? I guess I'll have to try both and find out if one is much slower.

Eventually, I expect to find a smooth way of accomplishing everything I want to do and all this will be like second nature. But that "eventually" is quite a while, and a far cry from more opinionated software like Rails (which is written in a better language, to boot). Maybe Mastering XPages will help, but it seems I'll have to content myself with muddling through until at least the end of January.

Turns out Partial Execute is handy

Sun Aug 22 22:53:00 EDT 2010

My main "spare time" project involves working on a lineup editor for raids in WoW. Basically, each player can have one or more characters, each of which can perform one or two roles (out of "Tank", "Healer", or "DPS"), and the UI should allow the person setting it up to pick which character that player will bring and which role they will perform. The setup is a table, with each row looking like:

The checkbox indicates whether or not the player is going on the run, then there's the player name, the currently-selected character (the icon denotes the character's current specialization), then how they originally signed up and their desired role, then three checkboxes for the role, and then their skill/gear rank.

Originally, I had the lineup set up as a Server-Side JavaScript array with ad-hoc objects with fields for each column. This worked out alright at first, but not only stepped all over MVC separation, since the controls had to "know" about the nature of the data they were working with, but made the necessary event-handler code really tangled.

What clinched the deal was the side effects of switching a character or role - not every character can fill every role, so switching a character or role could easily switch the other. Case in point - if I switch my lineup line there to my hunter Elegabal, the role has to change as well:

With a JavaScript array and event handlers, every change meant code attached to the control itself that looked up the current character in the database, the roles they can fill, if they're eligible for the selected role, then either what roles they CAN fill (if a character was picked) or what character is available to fill the selected role. Not impossible, but VERY hairy.

So I switched over to Java objects on the back-end. While the Java code itself is less expressive than JavaScript, the organization is significantly cleaner and more logical. The Lineup is a class that descends from Vector to provide the overall table, and each element is a LineupLine object with getters and setters for each important element. That way, rather than each role radio button having to have code to sanity-check the character, they're just bound to "#{line.role}" and the LineupLine.setRole(String) method handles all the particulars.

Early on, though, I started running into some oddities - I'd pick a different role, the table would refresh, and then my role selection would go back to its previous state. I added in some System.out.print() lines in the Java objects to make sure setRole() was being called and, sure enough, it was - I could see that it was called with the right new value, yet it kept switching back to the old one. Upon further investigation, I found the problem: while the new role was being set, the server was also getting a setCharacter() call with the existing character value. So the role would be switched, then the character would be set, and, since that character couldn't do the new role, it'd naturally switch the character back.

The solution I found was to enable Partial Execution on the input controls. I can only assume that this means that, rather than sending the server the entire contents of the row as the new data to process, it sends only the control's (i.e. the drop-down's or the radio button's) data, which limits the method calls to only the one that actually changed. Presumably, this has the side effect of being slightly faster, since less data is sent to the server to be processed.