Today, I put two long-overdue releases up on OpenNTF.

First and by-far-foremost is version 2.0.0 of the OpenNTF Domino API. The major version reflects not so much a major new architectural change over the 1.5.x release candidates as it does the fact that those releases were conservatively named and presaged a Java-style "1.x forever" future. Various development builds and release candidates have been used in production by the API team and others for a while now, and so this represents a mature release of changes such as the Maven conversion, revamped auto-recycling, and graph API.

Alongside it, I bumped my own framework up to version 1.1.0 to reflect improved stability and a clean dependency on ODA 2.0.0. I also improved its packaging and created a distributable along the lines of the new ODA version.

So: enjoy!

Now that I have a few apps under my belt, I've been getting a better idea of the plusses and minuses of my current development techniques - the frostillic.us Framework combined with stock controls + renderers. This post is basically a mostly-unordered list of my overall thoughts on the current state.

• Component binding is absolutely the way to go. This pays off in a number of ways, but just knowing that the component is pointed unambiguously at a model property - and thus getting its field type and validators from there - feels right.
• Similarly, externalizing all strings for translation via a bean or via component binding is definitely the way to go. The "standard" way of adding translation promises the ability to not have to think about it until you're ready, but the result is more of a drag.
• On the other hand, having to manually write out translation lines for every model property and enum value (e.g. model.Task$TaskStatus.INPROGRESS=In Progress) is a huge PITA. Eventually, it may be worth writing a tool to look for model objects in a DB and present a UI for specifying translations for each property and enum value.
• It feels like there's still too much domain knowledge required for using Framework objects. Though I try to stick with standard Java and XSP idioms as much as possible, you still have to "just know" and remember classes like BasicXPageController, AbstractDominoModel (and that you should make a AbstractDominoManager inside it), and AbstractXSPServlet. This may be largely unavoidable - Java isn't big on implied and generated code without work. But there's enough specialized knowledge that even I've forgotten stuff like how I added support for the @Table annotation for models to set the form.
  • Designer plugins could help with this, providing ways to create each class type with pre-made Java templates and potentially adding views of each class type. I don't know if I want to bother doing that, though.
• @ManagedBean is awesome and makes the faces-config.xml method seem archaic (even in light of my recent dabbling with the editor). The down side I can think of is that you don't have a good overview of what the beans in your app are, but that doesn't really come up as a need in reality.
• The Framework and renderers work great in XPiNC. Good to know, I suppose. They are probably actually a huge boost, speed-wise, over putting a lot of code and resources in the NSF - they dramatically cut down on the network transactions required to render a page.
• Sticking with stock+ExtLib controls is mostly great. Combined with component binding, my XPages are svelte, I have comparatively few custom controls, and I haven't had to go through the laborious process of writing controls in Java.
• On the other hand:
  • Trying to write a Bootstrap app with standard components leaks like a sieve. Though there are many times when the controls match up perfectly - xe:formTable, xe:forumView, etc. - the stock controls have no concept of Bootstrap's column layout, so I've had to write custom controls for that.
  • Some ExtLib controls are surprisingly rigid, lacking attrs properties or having weird interaction models like the onItemClick event with context.submittedValue on trees. I guess I could add them myself, but I don't want to get into the business of maintaining a forked ExtLib.
  • Trying to adapt standard code to Bootstrap+jQuery can be a huge PITA. For example, Select2 (and Chosen) don't trigger onchange event handlers written in XSP. While there are ways to work around it, they involve writing odd code that makes big assumptions about the final rendering, and avoiding that is the whole point. I have a "fix" that sort of works, but it's not ideal - it has the side effect of triggering a too-much-recursion exception on the console. There's a bunch of this sort of thing to deal with.
• My model framework has been serving me extremely well, but some aspects feel weirder over time (like how it blurs the distinction between getting a collection vs. individual model by key). I'm still considering switching to Hibernate OGM, but adapting it would likely be a mountain of work and I'm not 100% sold on its model. Still, the idea of moving to a "real" framework is appealing.
• Using enums for fixed-value-set model properties is great.
• I don't yet have a good solution for multi-lingual data. Maybe I could use a convention like "fieldname$fr". It hasn't actually cropped up, though, so it's a theoretical issue.
• I should standardize the way I do configuration and come up with a standard "keywords" mechanism.
• Similarly, I should codify the bean-backed table into a control, since I use this all the time and end up with similar code all over the place.
• I should add specific support for using model objects as properties on other objects - both referenced by ID and potentially beans stored via MIME in the documents.
• I need to add a way to specify error messages in the model translation file. Currently it's not much better than this.
• I should really add built-in Excel exporting for collections.
• I'm going to be happy I did the REST services down the line.

Overall, it mostly feels right, and working on "normal" apps feels archaic and brittle by comparison.

Well, it's been much longer than planned, and this topic isn't actually particularly groundbreaking, but the series returns!

1. Define the data model
2. Create the view and add it to an XPage
3. Create the editing page
4. Add validation and translation to the model
5. Add notification to the model
6. Add sorting to the view
7. Basic servlet
8. REST with Angular.js

One of the edge features of the Framework is that it assists in writing DesignerFacesServlet servlets - which are sort of like XAgents but written directly as Java classes, without an XPage component.

Before I explain how they work in the Framework, there's a caveat: these servlets do not have (reliable) sessionAsSigner access. The reason for this is that IBM's mechanism for determining the signer doesn't cover the case of just having a Java class. That said, it does have access to the rest of the XPages environment, including the same instances of managed beans available to XPages.

With that unpleasantness aside, here's an example servlet:

package servlet;

import javax.faces.context.FacesContext;
import javax.servlet.ServletOutputStream;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import frostillicus.xsp.servlet.AbstractXSPServlet;

public class ExampleServlet extends AbstractXSPServlet {
	@Override
	protected void doService(HttpServletRequest req, HttpServletResponse res, FacesContext facesContext, ServletOutputStream out) throws Exception {
		out.println("hello");
	}
}

Once you create that class (in the package "servlet"), it is available as "/foo.nsf/xsp/exampleServlet". As with an XPage, you can add arbitrary stuff after the servlet name with a "/" and in the query string. Unlike in an XPage, the servlet name is not removed from the path info. So, for example, this method:

protected void doService(HttpServletRequest req, HttpServletResponse res, FacesContext facesContext, ServletOutputStream out) throws Exception {
	Map<String, String> param = (Map<String, String>) ExtLibUtil.resolveVariable(facesContext, "param");
	out.println("param: " + param);
	out.println("pathInfo: " + req.getPathInfo());
}

...with this URL fragment:

foo.nsf/xsp/exampleServlet/foo?bar=baz

...results in this in the browser:

param: {bar=baz}
pathInfo: /xsp/exampleServlet/foo

By default, the result is served as text/plain, but you can change that as usual, with res.setContentType(...).

For most apps, a servlet like this isn't necessary. And for apps that do have a use for servlets, the XAgent and ExtLib-control routes may be more useful. Nonetheless, I've found a number of uses for these, and I appreciate that I don't have a bunch of extra non-UI XPages cluttering up the list.

One of the overarching goals for my model framework is to get the job of specifying business logic out of the XPage. So far, most of my work in the area revolved around building in the ability to override getters/setters, establish relationships, and so forth. However, a big problem remains: the XPage still needs to be told things about the data that are really the model's job, not the XSP markup's. Namely, validation.

A little while ago, I "solved" this by adopting Hibernate Validator and building it into the framework. This allows very declarative specification of requirements and also let me build the validation into the model's save event. If a field is marked as required in the model, no matter what the XPage says, the object cannot be saved if the value is empty. So: mission accomplished, technically!

But it's not a great user experience: though I made it so that you get proper Faces messages about what the problem was, the error wasn't tied to the control that caused it, nor could the page do any client-side validation to avoid server round-trips. The solution I found for this and several other problems is, as happens very frequenty, a technical path Tim Tripcony started down a few months ago. Specifically, it's the esoteric "binding" property of each control on the XPage, which is a way of telling the control to set itself as the value of a given binding - say, on the page controller - to provide easy access to it without having to crawl the component tree.

I realized that I could use this with a specially-crafted set of objects to tell the controller and model framework exactly what is going on:

<xp:inputText binding="#{controller.components[task].Summary}"/>

That looks weird at first, but the meaning becomes clear: tell the controller that that text box represents the "Summary" property of the "task" model object. Once that binding happens, the controller can take care of a lot of work that would otherwise have to be done manually:

• Set the value="#{...}" binding
• If the model says the value is required, set required="true", allowing for client-side validation and an Aria attribute to be set
• If the model property specifies another known validation type (say, that it's an email address), add in an appropriate server- and client-side validator. For unknown types, add in a generic validator that will respond to any requirement
• If the property is a Collection type, specify a multipleSeparator property
• If the property is an enumeration and the control is a multi-value control (say, a xp:comboBox or xp:radioGroup), automatically add xp:selectItems for each entry
• Add appropriate converters and other assistant components, such as a date/time picker for Date fields

This could be taken much further, to the extent that there could potentially be a single generic xp:input-type control (or a placeholder CC) and then the controller would construct a control based on the model and client needs - this is a drum NTF has banged before. It could also apply to objects other than my model framework's, but that would take an appropriate adapter for each.

Not only does the XSP code get much trimmer, but the validation and type-specification code in Java is clear and to the point:

public class Task extends AbstractDominoModel {

	@NotEmpty String summary;
	@NotNull TimeFrame timeFrame;
	TaskType type;
	
	public static enum TimeFrame { Normal, Rush, Urgent }
	public static enum TaskType { Normal, Question }

	// ...
}

This feels like a nice step forward in the direction of putting concerns in their right places while also reducing the total amount of code. I'd call that the right kind of victory for app development.

The code for this lives in the framework project inside the XPages Scaffolding repository and the work of the component manipulation is in this class:

https://github.com/jesse-gallagher/XPages-Scaffolding/blob/6d7a9dc59ca070e74a5bd157eb49500b61717e57/frostillicus.framework.plugin/src/frostillicus/xsp/controller/ComponentMap.java

For a while now, I've maintained a project called XPages Scaffolding on OpenNTF and GitHub. This project is the NTF that I use when creating a new XPages application and acts as a storehouse for my "bag of tricks", things like my model framework, controller classes, and various goodies I've picked up over the years, like a JSFUtil class and a flashScope. For a while, I've figured that, once this stuff is in proper shape, I'd extract it into an OSGi plugin and get it out of the NSF. Last week, I decided to take the plunge.

Something surprised me about the process: not only did I gain the benefits of a cleaner NSF and a nicer environment for editing the code, but it really solidified my focus for what I want it to be and gave it a real sense of purpose. The newly-dubbed Framework is geared towards turning an NSF app into two main things: a data-access API available via Java and REST and an XPages front-end that is first-among-equals for accessing that API and follows defined structural conventions.

The Model Framework

The model framework is the newest part of the project and the REST API in particular is only days old, but it's rapidly becoming the most important aspect and is likely to be the spiritual core. By providing a way to define models in a way that is simple, consistent, and (theoretically) storage-agnostic, a large part of the work of creating a new application is made significantly easier to write and understand. The models provide numerous characteristics built-in that would previously have been implemented as a series of script libraries, XAgents, and UI components:

• Collection access. The Manager classes let the XPage care only about getting a collection of "my Tasks", rather than "the 'Tasks by Assignee' view filtered to just the category of @UserName()".
• Property control and side effect handling. Model objects allow handling side effects where they belong, so setting, for example, the Client field to a new value needing to reset Project to blank is done there, not on every XPage implementing those fields.
• Validation. I'm currently trying out integrating Hibernate Validator into the framework to allow for declarative constraints on a model's properties. It shouldn't have to be the XPage's job (solely) to know how to validate an email address or ensure that a username is at least N characters long. By declaring it at the model level, it frees up code elsewhere while also allowing to constraints to function when accessing models programmatically.
• REST API. This is the new golden child that has brought this clarity of purpose to the framework. Having "for free" REST access to the model objects in an app removes a ton of potential work from the programmer and also opens up tremendous opportunity for integration. Previously, allowing access to an app's data meant rewriting all of the business logic, either in the foreign app (accessing the data via DAS or legacy Domino APIs) or in XAgents designed to serve up JSON. Now, when you write the business rules for your XPages app, you're also writing a method for external access. Things like a JS-based or mobile app move from "possible with lots of work" to "may as well". It also means you don't have to choose between having either an XPages app or an interoperable API.

The XPages Components

The XPages side of the framework is still something of a grab bag, filled with the aforementioned small utilities, controllers, my dying-on-the-vine messaging framework, support classes for writing in-NSF servlets, and a number of useful converters. In the process of plugin-ifying these, I've made some refinements, such as turning the converters into real JSF components, redubbing "JSFUtil" as "FrameworkUtils" and making it work in a non-Faces OSGi environment, and integrating my @ManagedBean implementation.

The transition has also given me the freedom to be a bit more brash about changes, making backwards-incompatible tweaks that I've been meaning to do for years, making the different parts of the framework a bit more integrated, and a number of other things I have in mind to try out.

But though this side is less integrated than the model side, it still represents a consistent way to write XPages applications: model objects for data access, controllers for primary logic, servlets instead of XAgents when appropriate, SSJS relegated to glue code in Custom Controls, and a consistent, simple naming scheme for Java classes favoring simple "local" package names like "controller" or "model" over fully-qualified ones like "com.somecompany.appdev.domino.xpages.apps.someapp.controllers" when appropriate.

---

All in all, I'm pretty excited about the changes I've been making. For as long as I've been working with XPages, I've been searching for the "right" way to write them, and I feel like I'm finally approaching an answer I can feel comfortable with.

And not only does my XSP code feel better and less cluttered, but it makes the platform as a whole feel better. It's always been the case that server-based app-dev lets you use whatever environment you want, though modern web development has modified that situation somewhat. It's still true that your framework doesn't necessarily matter, but it should be able to adhere to common contracts to function in an interoperable world. Much like other things I want the platform to do, the model framework in particular brings Domino into a role where the NSF-based app is the central dispatch point for a multi-platform environment. Don't want to use NSF for data storage? Plug in a different back end. Don't want to use XPages for the UI? Use the REST API and get the same transparent data access.

I think there's good potential with this setup.

Recently, I've been refactoring out my framework code from XPages Scaffolding into an OSGi plugin. Part of this is just to help with code reuse and versioning: having a whole bunch of framework classes in each NSF is a drag, both from a perspective of having a bunch of extra stuff in the NSF and for having to worry about keeping them up to date (a template would work, but eh...). But another part is that it lets me work on some new features I've been itching to try.

One of those is adding REST services for model objects. Having access to your data via REST+JSON is pretty important, whether you're using Angular.JS, iOS, or miscellaneous. The Data service in the ExtLib does a great job of exposing document and view data directly, but it runs into the same problem that the standard data sources in XPages do: it doesn't have any provisions for your specific domain models. On the XPage side, I've solved this for myself by making a model framework that has done a fine job so far of covering my needs for accessing and collecting data in a pretty friendly way, and now I'm working on building REST APIs "for free".

Using the DAS as an inspiration (both in a vague sense and in a "using the code" sense), I'm working on Wink services to provide access to the model objects defined in each DB in a way that uses all of the same getters/setters with almost no changes to the code. The only change required (at least so far) is an optional annotation to define which fields should be included when accessing the object via REST. I'm aiming to match the data formats used by DAS as much as makes sense, since they're well-thought-out for representing Domino-friendly data, though the URLs will be pretty different due to the nature of my model manager objects.

This has been going well so far, and I'm looking forward to having it fully fleshed out. It should allow me to continue defining my model objects the same way I have been for use in XPages/Java, but then also be able to access the data just as well from other front-end technologies - potentially having NSFs that contain just a set of model definitions to act as a headless API app for use with other clients. And hey, if I get around to adding non-Domino backends to the model framework, the NSF would turn into a platform-agnostic gateway for data access.