Though most of my paying work involves working with clients' creaky old databases, greenfield projects will always have my heart. For these and for my side projects, I've been moving towards an ideal Domino-app strategy that seems to serve me well.

The Setup

My apps consist of at least two NSFs: one for the app itself and at least one for data. The only non-design notes in the app NSF are for app configuration, and even then I keep it limited. Everything else goes in the "data" databases, which have designs consisting only of views, forms (rarely), and any agents needed for migration between versions.

This has several advantages I've talked about before. Importantly, other than the "Don't allow URL open" problem, which can be avoided entirely, I've encountered no down sides to this approach. The amount of flexibility it provides and the discipline it imposes pay massive dividends while not significantly increasing hassle. I recommend everyone do it. I try to keep an eye towards future scaling/separation needs as well: I tend to give my app configuration options where each document category are stored (say, Tasks/Projects vs. Clients vs. Update Notifications), even if I set all of them to the same location.

Scaffolding

As you might expect, I use the latest version of my XPages Scaffolding project for the app NSF. Now that my model framework supports attachments, there's little preventing me from using it extensively for data access in future projects.

Once those classes settle down a bit more, I expect that I will package them up as plugins so I can have a consistent "frostillic.us Framework" to attach a project to, allowing me to roll out bug fixes to major+minor versions (e.g. frostillicus.framework_1.2) without using classic-style design inheritance.

One of the potential future bonuses of using my model framework combined with inherent data separation is that I will be able to write adapters for my framework to connect to other data sources without the app having to care much (beyond the config document for pointing to the other data store). As long as I do the legwork to write the model classes, the bulk of the app won't have to care whether it's talking to Domino, JDBC, CouchDB, or a CSV in the data directory.

Source Control

I use Git early and often. As soon as I start work on a project, I create a repository on Bitbucket and then commit the app and data NSFs to it immediately. This has a couple crucial implications:

1. I try to focus my changes into conceptual units for commit purposes. Though it's exceedingly rare that I actually need to reverse a commit, the better I am about only committing related changes (i.e. not commits like "implemented a wave of client requests"), the easier it is to do so if needed.
2. I develop using either a local Domino server or NSFs on a shared server that is dedicated to my current machine. If, for example, I want to work on the same project on both my desktop and laptop, that means I will have two copies of each NSF even if they're on the same machine. This just saves a lot of hassle while simultaneously reinforcing the source-control-focused dev model.

Prototyping

I haven't had a chance to put this into real use yet, but, now that I'm a convert to renderers, I plan to get into the habit of doing my initial prototyping using standard ExtLib components with OneUI as the theme. I'll still have to keep an eye towards the eventual goal (for example: avoiding the Dojo-exclusive components or ones I don't really want to write renderers for if I'm not planning to actually deploy with OneUI), but this will make it much quicker to start laying down the common components of the app immediately without having to worry about specifically implementing Bootstrap-or-other-framework classes and HTML structure.

Because renderers are more difficult to write than custom controls with the HTML hard-coded, this will increase the overall development time, but the flexibility and off-loading of responsibility to later in the dev cycle will be worth it. Additionally, each render kit is easier to write than the last, particularly if I can reuse/extend previous work.

Deployment

As mentioned, the use of source control imposes a strict structure on development from multiple machines. This discipline contributes to dealing with production versions of the database. It's always been the case that it's a terrible idea to do work in a production version of a database, but Notes always made it so easy to get away with it. Fortunately, XPages' PITA compilation requirements and gotchas make this a terrible idea (who wants their app to go down because you opened it in Designer but forgot to install a required XSP library first?). This is all the worse when source control is involved: a bug in the SCM plugin or a missing network/VM share can easily destroy every design element in the target database. If it happens to be one dev DB among many, who cares? If it's your production DB, now you have a serious problem.

When I'm ready to deploy my changes, I do it via NTFs: I create NTF copies of any applicable dev NSFs and use them to replace the design of the production databases. Because these apps tend to be single instances, I don't bother with template names and design inheritance. In the case where there are many copies of the data databases (say, one core app and then one data DB per client), I'd probably use inheritance and replace the data DBs' NTF's design instead and immediately run the design task.

Since I have NTFs of each deployment sitting around, it makes it easy to roll a production DB back to a previous state. Though I don't currently, I plan to start tagging releases in Git (or just switch to GitFlow, which I've yet to use) to make this clearer as well.

Another improvement I'm considering making is to drastically limit my own rights to the production DB, to a max of Editor access. This way, there's an extra layer standing in between me and the terrible notion of opening the live DB in Designer. I figure I'll either use a dedicated deployment ID or use Full Access Administration mode to make deployments.

I've been feeling pretty good lately about this emerging scheme. A lot of it aligns with standard practices for non-Domino development (data/app separation, heavy source control) while still maintaining the benefits of Domino and XPages. And once this process is firing on all cylinders (namely, once I have skill in renderers to cover all of my regular needs), it should yield pretty impressive results when it comes to putting out fleshed-out, scalable apps quickly without making my development life down the line harder.

A few posts ago, I talked about how the skeletal nature of an XPage is really a very abstract representation of a UI. At its core, it isn't conceptually tied to the web - and, indeed, the probable bulk of the work done by JSF is to push against the normal rules of the web.

The goal of XPages/JSF is to abstract away the messy business of writing web pages, much like a cross-platform mobile framework attempts to abstract away the chore of writing to iOS or Android directly.

In practice, I've found this to be both a brilliant and awful idea. I'm not alone in this: if you look around, you'll find proponents of methods of web programming that don't go against the grain so much, and they have excellent points. However, the "abstraction" route isn't without its benefits. So, off the top of my head, I can think of a number of pros and cons of the XPages "component" conceit. In the spirit of suspiciously defying convention, I'll start with the cons.

Cons

• The abstraction leaks like a sieve. Case in point: the style and styleClass properties. In an ideal world, those have no place in an abstracted framework - in XPages, it should be the job of themes and renderers to sweat those details. But in reality, they're vital.
• It's harder to write a clean page. Because you're working at a high level, you're relinquishing control of much of the structure of your page to the components' renderers, which will likely spit out all sorts of oddball divs, spans, and the like. Theoretically, you can fix this, but:
• Writing renderer classes is a colossal PITA. Java has its share of virtues, but writing code to spit out other code is not among them. Writing a renderer for even a basic control is an exercise in verbosity and fighting Java's inferior literal strings. This means:
• It's harder to keep up with the times. Because you're largely dependent on the implementor of the renderer for the control's HTML/JS, you're likely to run into situations where the HTML is atrocious (see: radio and checkbox groups) or runs counter to the structure you want for your page (see: using pagers on a Bootstrap-ified site). That's not to mention significantly-different programming models like AngularJS.
• They're a poor match for the realities of HTML/HTTP. The whole conceit of JSF - that you have this tree of Java objects sitting on the server that you're interacting with as if it's a desktop app - requires tremendous code and conceptual overhead to glom onto a stateless network protocol like HTTP.

Pros

• The abstraction level is often just about right. When you want to create an entity on your page that is conceptually common but has no representation in HTML - say, a widget container or tabbed table - the component model shines. The code you end up writing is extraordinarily task-focused: you're creating a widgetContainer, not a bunch of divs with class names that happen to match the CSS/JS framework of the day. Along those lines:
• It's easier to keep up with the times. I am aware that the inverse of this is in the "cons" list, but they're both true. Though it can be harder to adapt your components to modern standards, if you do the job well, the component model makes it easier to bring older apps forward. If you app consists entirely of standardized components like applicationLayout, formTable, and the like, a set of new renderers can adapt these conceptually-common controls to a new framework that wasn't in use when you wrote them originally.
• You can write your own. Since, in spite of the difficulty, JSF actually succeeds in presenting an idealized Java world, it's a minor conceptual leap (though a large code-writing one) to go from using common components to building ones that match your needs. This is where Custom Controls shine, and do the job well either as-is or a starting point for building more "baked in" Java components.
• HTML is no prize pig either. As much as it's easy to extol the virtues of going with the grain of HTML/HTTP/etc., it's still a huge pain to do so. The closer you get to the "metal", the more you have to deal with non-semantic markup elements, browser bugs, HTML entity escaping, and so forth. It's a fool's errand to do it by hand, so the task is just picking a set of abstractions that strike your fancy.

So what's the best route? Beats me! To a large extent, it doesn't matter, because no web framework lasts forever. For my part, I find the XPages/JSF abstraction to be a fine tool, and I plan to continue refining my ability to write to its strengths.

Unlike the last few posts, this one isn't as much a tip or tutorial as it is an attempt to share my suffering by detailing the steps I took recently to add file-download and -upload support to my current model framework.

The goal of my framework overall is to retain the benefits of normal Domino document data sources (arbitrary field access, ease of use in simple cases, and compatibility with standard XPage idoms) while adding on useful new abilities (getters/setters, relationships with other objects, abstraction between the XPage and the actual location of the data). Most of this work is easy - by implementing DataObject, my objects worked well with EL bindings and most of the work was focused on hooking up Domino Document objects in a similar way to IBM's DominoDocument class.

However, some of the more esoteric controls are not so easy, and the file controls were chief among them. As I tinkered, I found that you can't directly point a xp:fileDownload control to a normal data source and get it to work well. You can get it to list a single file, but I wasn't able to get it to recognize more than one through traditional means.

In my tinkering, I discovered that there are two main paths for the control: the traditional one and a more-complicated one, which makes a lot of assumptions about the nature and availability of the referenced data source. Importantly, unlike normal bindings (which take your expression, say #{doc.body}, have the EL processor resolve it, and deal with the final resolved entity), the control actually treats the EL expression as a String, splits it in two on the "." character, and looks up the variable on the left side plus ".DATASOURCE". This "doc.DATASOURCE" idiom matches the behavior of data-source controls on an XPage, which make themselves or an associated object available by that name. If the control finds an object by that name and if it implements DataSource (a common interface used by these controls), it assumes that the source contains a DataContainer object that also implements DocumentDataContainer specifically.

Phew, okay, so far so good. It makes some inconvenient sense that the control would expect its referenced value to conform to the norms of the built-in data sources, so I wrote controls to do just that (previously, I got by using normal EL references from managed beans). However, I still ended up getting ClassCastExceptions at runtime: it turned out that the control also uses a nebulous factory object to attempt to convert the received value into a data type it expects, specifically DominoDocument. Unfortunately, unlike the interfaces I implemented already, subclassing DominoDocument was not something I was willing to do. So I was back to digging.

What I found was that this attempted conversion was happening due to an object implementing com.ibm.xsp.model.DataModelFactory, which has the job of taking an intermediate value used in the control (a com.ibm.xsp.model.FileDownloadValue, which contains a single-entry HashMap containing the object and requested field name) and returning the attachment data as an instance of DataModel. In addition to writing the code to do this, there's another wrinkle: the way the factory is found. The way I found to access this is thoroughly awkward: if I use the deprecated method getFactoryLookup() on the application instance, I can then retrieve the default factory by a key ("com.ibm.xsp.DOMINO_DATAMODEL_FACTORY"), wrap it in a new object that knows about my model objects, and replace it in the lookup.

Once I did all that, I was off to the races! Sort of! There remained another problem: the "delete attachment" icon on the control. I discovered that the way this icon works is that it composes an action that tracks down an adapter that knows how to communicate with the associated model object and sends it a deleteAttachments(...) call. As before, the way it "tracks down" the adapter is to look up another factory ("com.ibm.xsp.DESIGNER_DOMINO_ADAPTER_FACTORY") and ask it to provide an appropriate adapter. So again I wrote my own variant of this to intercept calls dealing with model objects and carefully placed it, Indiana-Jones-style, in place of the default factory.

The net result is that it works! I can point a file-download control at a model object (as long as that object was specifically instantiated in a data source control and is referenced directly in "object.field" format) and have it list all of the attachments in the field, along with a functioning delete action if desired. The file-upload control was, fortunately, much easier: when I realized I could swap out most of the guts of my model object with a back-end DominoDocument (now that the OpenNTF API takes care of data conversions for me), I was able to just proxy the upload calls to it and let it do the heavy lifting.

Now, there's only one nagging problem remaining: inline image uploads in rich text controls. But still, even if I never get that part working, the file manipulation will be enough. And, importantly, I've paved the way for me to be able to do use the same controls with model objects that aren't associated with Domino at all, should I desire to do so down the line.

In today's TLCC XPages webinar, Marky Roden had a great presentation discussing resources and design definitions to help speed up Designer. If you didn't catch it live, it's worth tracking it down when it's available.

I'd like to build on what he was talking about a bit by explaining how to use themes to load resources in a way that also prevents Designer slowdown while having the secondary benefit of being clean design (with some caveats that I'll get into).

If you're not familiar with themes, they're the oddball XML design element introduced alongside XPages, and they serve a number of purposes. One of them, which Tim Tripcony explained well is that you can use them to set properties of your XPages controls - so if you want to, say, apply the CSS class of "btn" to all buttons, you can use a Theme to do that instead of writing it into every <xp:button/> control. They can also be used, though, to load resources in much the same way as you do on an XPage or Custom Control directly. The default theme you get includes some comments that point in this direction, but they use the horrible multi-line syntax. It turns out that you can use a shorter syntax that mirrors the normal XSP markup. Say you have a resource section like this:

<xp:this.resources>
	<xp:styleSheet href="/style.css"/>
	<xp:styleSheet href="/fixes.css"/>
	<xp:styleSheet href="/ieonly.css" rendered="#{context.userAgent.IE}"/>

	<xp:dojoModule name="dojo.behavior"/>
	<xp:script src="/behavior.js" clientSide="true"/>
	
	<xp:metaData name="viewport" content="width=device-width, initial-scale=1.0"/>
	<xp:linkResource href="#{facesContext.externalContext.requestContextPath}/feed.xml" rel="alternate" type="application/rss.xml" title="RSS Feed"/>
</xp:this.resources>

To convert that to theme syntax, you just chop off all the "xp:" and "this." bits:

<theme extends="webstandard">
	<resources>
		<styleSheet href="/style.css"/>
		<styleSheet href="/fixes.css"/>
		<styleSheet href="/ieonly.css" rendered="#{context.userAgent.IE}"/>

		<dojoModule name="dojo.behavior"/>
		<script src="/behavior.js" clientSide="true"/>
		
		<metaData name="viewport" content="width=device-width, initial-scale=1.0"/>
		<linkResource href="#{facesContext.externalContext.requestContextPath}/feed.xml" rel="alternate" type="application/rss.xml" title="RSS Feed"/>
	</resources>
</theme>

Once you apply that to your app in the Xsp Properties, it will be as if you included the resources on the XPage directly, but you avoid the Designer slowdown problem and the XSP load time itself will be (imperceptably) faster. As an important bonus, it separates the concerns nicely: the theme's job is to handle describing the client-only stuff, and accordingly you get to delete some code from your XPages. One less thing to worry about during normal development.

Now, as I mentioned, there are some caveats, and themes are not appropriate in all cases.

• Most importantly, the resources loaded by themes are not available during programmatic page load. This is not normally a big deal, since the page loading doesn't care about CSS files or client JavaScript anyway, but it DOES matter for two types particularly: SSJS libraries and property bundles. If you have code that needs to access them during initial processing (either in ${}-bound properties or in places like before/afterPageLoad events), you should include them on the XPage directly.
• There's no good UI for modifying themes. You kind of have to just know how they work. I believe there's a thorough treatment in Mastering XPages, but other than that you're on your own until you learn.
• Since Designer doesn't load them, it also doesn't help you when specifying style classes. If you're in the habit of using the "Style" properties pane to pick classes from linked stylesheets, this will break that.

But even with those caveats in mind, I'm a huge fan of themes (for this and other uses) and use them constantly. The code cleanliness, performance improvements, and structural advantages are well worth the caveats.

While I've been formulating the followup to my last post, I've also been thinking about how wonderful the Java Collections framework is. I've been singing its praises for a long time (as have others), but what's stood out to me lately is how well it encapsulates many of the most important foundational concepts of Java.

One of the reasons Java appears so intimidating and foreign to Notes developers is that the way many are introduced to it - as an auxiliary for XPages - involves learning a torrent of unrelated concepts simultaneously: strict variable typing, object equality and comparison, class casting, managed beans, recycling (though this is just as important in SSJS), confusing recycling with memory management, static methods, inheritance, interfaces, exceptions, abstract classes, database access, serialization, concurrency, and so forth. If you focus on the Collections framework specifically, though, you can learn many of the important basics in a clear way. And as a bonus, these are critical concepts shared with other, better OO languages.

Interface vs. Implementation

One of my favorite lecture topics is the value of separating interface and implementation. Unfortunately, most of the time, the values are unclear - if you're only going to make one class, is it valuable to have an interface? - and it seems like pedantry. The Collection classes, however, bring this value into sharp focus. The core interfaces correspond to elemental data structures of Computer Science and tell you everything you need to know to use them, e.g.

• List: a collection of objects in the order stored by the user, accessed by index.
• Map: a collection of objects accessed by a key of any type specified by the user.
• Set: a collection of unique objects in either arbitrary or, for SortedSets), automatically-sorted order.

These interfaces describe everything you need to know about the object and do so so well that there's rarely any need to specify the actual class at all outside of creation. The differences between the actual classes are, quite literally, implementation details: some focus on single-thread performance (like ArrayList), some serve specialized cases (like EnumMap or IdentityHashMap), some implement multiple useful interfaces (like LinkedList), and some are geared towards multi-threaded use (like the java.util.concurrent objects).

Job-Based Polymorphism

Not only does the Collection framework provide an example of a cleanly-designed interface/implementation separation, but it also provides examples of why this pays off. One of the handiest tricks is the way most of the objects (other than Map) participate in mutual-translation schemes. By this I mean that you can take one Collection and pass it to another (either in the constructor or via addAll) and build a second collection of the same objects, but obeying the rules of the new collection.

For example, say you have a List of usernames that you retrieved from a call to view.getColumnValues(...) and now you want to sort and unique-ify them (basically like @Sort(@Unique(...))). Well, there's a collection type for that: the SortedSet, of which TreeSet is the common implementation. Rather than manually building a unique, sorted List (even using helper methods), you can accomplish this with TreeSet's constructor. You can then add in names from other sources and maintain the same unique-and-sorted guarantee without any additional code:

SortedSet<String> names = new TreeSet<String>(view.getColumnValues(1));
names.addAll(doc.getItemValue("SomeNamesField"));
names.addAll(someNameMap.values());

Because those methods only care that the incoming values are in a Collection, the fact that some may be in Lists, in Sets, or in some other entirely-unknown type that implements Collection is completely irrelevant. As it should be.*

Reusability

This benefit is so inherent to the framework that I almost forgot to include it. Largely by the nature of the task being performed but also by virtue of proper coding, the collection objects are so reusable that they feel like part of the language, even though they're just objects like any you write. They're built to be generic and used in ways far outside of the original ken of their creators. Both the implementations and interfaces can be spread far and wide, such as with my database model collection class that also acts as a List.

Takeaway

Not all of these benefits are going to be useful in all of your code, and it will be very infrequent that you have a reason to write your own framework of this type. By learning the contours of this framework, though, you can advance your knowledge of Java and object-oriented programming generally by leaps and bounds. It will help you write cleaner code and to identify similar concepts used elsewhere.

* For example, did you know that viewScope is not a HashMap but actually a class called "javax.faces.component.UIViewRoot$ViewMap"? That's how much the implementation doesn't matter.

I'd like to take a blog post or two to discuss an aspect of XPages that isn't directly applicable to normal XPage development, but which can be a conceptual shift that causes a lot of things to fall into place: the tree nature of an XPage. This has nothing to do with Java, SSJS, beans, or JSF phases. It barely even has much at all to do with web pages.

You've likely heard that XPages aren't really XML - that they're actually Java, created from the "simpler" XML you write. This is true, and it drives home the important point that an NSF is now best viewed as a Java "enterprise" application. However, this isn't actually the best way to look at it; the best way is that an XPage is a tree of objects.

If you've had a Computer Science education, you'll know the tree as one of the foundational data structures used in almost all programming. If you haven't and you're not familiar with trees, it will pay tremendously to learn a bit about it - most sources I've found are either painfully mathy (like Wikipedia's article) or tied to the instruction of a specific language (often a Lisp dialect), but this page of a Python tutorial looks promising. This tree nature helps explain why XML is the tool of choice for writing XPages: XML is a tree-description language.

So an XPage is a tree. Specifically, it's a tree of user-interface objects, each of which contains methods, properties, events, and zero or more child objects. Let's take an example XPage, using ExtLib controls because they are more descriptive:

<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core"
	xmlns:xe="http://www.ibm.com/xsp/coreex"
	pageTitle="Test Page">

	<xe:formTable formTitle="Some Form">
	
		<xe:formRow label="Field One">
			<xp:inputText/>
		</xe:formRow>
		
		<xe:formRow label="Field Two">
			<xp:radioGroup>
				<xp:selectItem itemLabel="Foo"/>
				<xp:selectItem itemLabel="Bar"/>
			</xp:radioGroup>
		</xe:formRow>
		
	</xe:formTable>
	
	<xp:button value="Click Me"/>
	
</xp:view>

Ignore your default impulse to think "ah, I know what that looks like!". At this point, it doesn't "look like" anything, other than:

• UIViewRootEx2 (pageTitle="Test Page")
  1. UIFormTable (formTitle="Some Form")
     1. UIFormLayoutRow (label="Field One")
        1. XspInputText
     2. UIFormLayoutRow (label="Field Two")
        1. XspSelectOneRadio
           1. UISelectItemEx (itemLabel="Foo")
           2. UISelectItemEx (itemLabel="Bar")
  2. XspCommandButton (value="Click Me")

The fact that it will eventually be rendered as a OneUI HTML form table won't come into play until renderers are involved (which will be a likely followup post). For now, it's best to think of this as a hierarchical description of a user interface in the abstract. At this point, the only thing standing between the XML code above and looking like this is a nontrivial amount of work:

Okay, the XML code and the related objects don't say anything about the appearance of the resultant interface, so what do they say? They define the state and behavior of the UI and how it responds to user interaction. Again, this will almost definitely consist of an HTML page and the user clicking in the browser to send POST requests to the server, but this is still entirely incidental to the XPage. Conceptually, it's all about a handful of UI controls in a hierarchy, with properties, methods, and events, each looking after itself and, as requested, producing its children. The text box's job is to know that it can contain a string value along with potential hooks for validators, converters, and event handlers - that's it. Nothing about the specifics of HTML or JavaScript.

This also helps conceptualize a specific user's page state. When a user starts interaction with a page, what happens is that the server tracks down the page root class that corresponds with the requested page and asks it to create itself and any applicable children. So you end up with any number of these object trees floating around, each corresponding to an individual interaction session (in our case, a loaded page in a browser tab, but you could also think of it as a loaded process of an application). Interacting with the page consists of calling methods on these objects - setValue, click (which is weird), etc. - and causing them to respond and mutate appropriately.

So where does the HTML come in? That's a topic for another post, but the fact that it hasn't entered yet is the point here: XPages are trees of active objects describing a Platonic ideal interface and the framework has a secondary effect of producing HTML.