I'm always looking to improve my programming/development/administration methods, and in the past couple months I've made a few more changes. To my chagrin, these changes are mostly things that I should have been doing for a while now and I've been sloppy to do them differently, but hey - fixing a problem is still improvement!

Serious About Source Control

I've been using source control for a while now, and I hope everyone else has as well. However, I've been lax in a couple ways, which bit me a while ago. For example, I had a couple actively-used projects tied to source control in Designer - a few NSFs, a few NTFs. That was fine NORMALLY, but then this sequence of events happened:

• I run Designer in a Parallels and have my repositories stored Mac-side, pointed to via a drive-letter mapping to my home folder
• I Remote-Desktop'ed into my Windows VM from another machine
• Remote Desktop remapped the drive letters to share local resources
• Designer, which was running at the time, could no longer find the repository folders
• Apparently, Designer/eGit/whatever decided that the rational thing to do was to "sync" the NSFs with the now-missing repositories... which is to say, delete all design elements from the NSFs

Imagine my surprise when a previously-fine DB I hadn't intentionally touched in a while was suddenly naked, consisting of a single view that somehow survived the purge. Fortunately, the nature of the problem meant that the design was still stored in the repository and thus recoverable, but that's still not a problem you want to have.

So I've been shifting my development process another few steps closer to the ideal. The way I see it, that ideal is:

• Never open a production DB in Designer
• Do all development in entirely-distinct NSFs created either from scratch (for a new project) or via the "Associate With New NSF" action from the source-controled on-disk project. These NSFs should be worked on by a single person and should either be in an individual-user dev folder on the server or, better, on a local development server
• Create deployment "builds" as NTFs, either as design-only copies of a source-associated NSF or, again, from the "New NSF" action from an on-disk project (but it should then be disassociated immediately)
• "Deploy" the "builds" using Replace Design on the production NSF/NTF from the Notes client or, better, Administrator
• Ideally, the "build" NSFs should closely match a tag or branch in the source repository, but I can't be bothered to do that for my needs yet

That process goes the same for "classic" and XPages apps. Since source control became viable for Domino devs around the same time as XPages hit the scene, the two are often conceptually tied together, but in my experience the classic binary-DXL handling is plenty good enough to use it in projects without a line of Java or XSP.

Separate Application and Data

The standard way to develop a Notes app - classic or XPages - is to bundle the design and data together. This has historically had a lot of benefits and still makes sense in some cases (say, customizing individual mail files for different uses). However, particularly in the case of a standalone "app" (a help-desk application, a project-tracking system, etc.), this tight joining makes little sense and would be considered a bizarre aberration by anyone using most other systems.

Fortunately, as Domino developers, even if we split up the design and data, we still have it easier than most, since we don't have to worry about drivers and authentication credentials for the back-end DB (unless we want to). In the simplest case, which I've adopted, you have your "app" NSF and a "data" NSF next to it. The "app" NSF contains no forms or views, which the "data" NSF contains no XPages, Java, or other "executable" entities. I've derived a couple benefits from this already:

• The app and data NSFs can have different ACLs
• The data NSF can be set to disallow URL open. This means you don't have to worry about, say, clever users trying ?ReadViewEntries to peek into more than they should see
• You can easily switch between "dev", "production", and "test" data NSFs, just like normal developers do. You don't have to worry about copying data around between different dev versions of your app
• Multiple apps can point to the same data with impunity
• The data NSF can exist on another server... or cluster of servers, which you could switch between programmatically
• Data sharding becomes easier. Depending on the type of app, each user/project/department/whatever could have a different backing NSF fronted by the same single app

Admittedly, this makes some things moderately more complicated. Any Domino data sources will need to have a databaseName property specified, which will need to get its value from somewhere (worst case, you could hard-code it to start with, then "upgrade" to a managed bean). You also won't be able to use "app.nsf/view/key"-style URLs, since the app won't have any views to search by - you'll have to use query parameters or cobble together your own good-looking URLs with the assistance of Web Rules. You might not want to make your structure too document-focused anyway.

These changes have been bringing my development closer to classic desktop-app development, where these are less "techniques" and more requirements due to the nature of development, but they've proven to be valuable. Domino's ease of getting started and fast-and-loose development methods have distinct advantages, but it's always important to question which ones are better than other environments and which are traps.

A while ago, I set up Apache as a reverse proxy in front of Domino. Initially, that was in order to serve a WordPress site on the same server, but since then it has more than proven its worth in other ways, mostly in the areas of fault tolerance and SSL support.

IBM recognizes those benefits, which is why Domino 9 on Windows comes bundled with IHS; however, I am a responsible admin, so I don't run Windows on my main servers. Fortunately, stock Apache does the job nicely... or it does usually. Filled with enthusiasm for this kind of setup, I rolled out a small Linode VM dedicated entirely to running Apache as a proxy (sort of like their NodeBalancers, but manual). Unfortunately, I started running into a problem where sometimes sites fronted by it (such as this one) would not properly include their host information and would instead load the main I Know Some Guys site, which is the default on Domino. I wasn't able to find a fix that actually worked, so I decided to use that as an excuse to switch to a cute little number named Nginx.

So far, my experience with Nginx has been fantastic. The config files are like a cleaned-up version of Apache's and it matches it for each feature I've needed (namely, load balancing, easy config, and multiple SSL certificates). As a nice bonus, I didn't have to do any of the config massaging I had to for Apache in order to get XSP's funky resource-aggregation rules to work. If you have the means, I highly recommend it.

My latest foray into proxying also gave me an opportunity to look back into my main bugbears with the setup: Domino tracking the proxy server's IP instead of the original requester's and its lack of knowledge of SSL (which causes it to redirect from an SSL login page to a non-SSL one). Fortunately, it turns out that these problems have been sort-of-solved for years via a notes.ini setting added as part of Domino's terminal WebSphere infection: HTTPEnableConnectorHeaders.

By enabling that on my Domino server, I was able to start providing some of those headers. The remote-host headers ("$WSRA" and "$WSRH") work perfectly: setting that to the incoming host causes Domino to act just like that was the original requester, namely filling that in for the REMOTE_HOST field in classic and facesContext.getExternalContext().getRequest().getRemoteAddr() in XPages.

Unfortunately, I was stymied when I set "$WSIS" to True. Though it does indeed cause Domino to acknowledge that the incoming request is via SSL, it does it TOO well: Domino appears to revert to its behavior of only acknowledging a single SSL site, so it caused requests to essentially ignore the Host (and "$WSSN") headers. So that problem remains unsolved.

Still, I feel pretty good about my switch to Nginx and my abuse of the HTTP connector headers and look forward to tinkering some more. For reference, here is the config I use for the standard HTTP proxy ("arcturus" is the short name I gave for the main upstream target):

server {
    listen 80;
	
    location / {
        proxy_pass http://arcturus;
        proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
        proxy_redirect off;
        proxy_buffering off;
        proxy_set_header    Host            $host;
        proxy_set_header    X-Real-IP       $remote_addr;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    $WSRA           $remote_addr;
        proxy_set_header    $WSRH           $remote_addr;
        proxy_set_header    $WSSN           $host;
    }
}

Among the new features in Domino 9 is this little guy, found on the Launch pane of a database's properties when you set it to open an XPage:

If you've ever used an XPiNC application before, you'll know this is a godsend, promising the vast performance benefits of running an app on a server combined with the "the users are stuck using the Notes client" benefits of XPiNC.

I turned this on for a new app on one of my client's servers (let's say the server name is ClientName-2/ClientName) and took it out for a spin. However, I immediately noticed something was amiss: the status bar declared that it was unable to find the server "www.clientname-2.com". With a bit of searching, I found I wasn't alone: Notes assumes that the Domino server is available as its common name in local DNS for some reason, and fails in various ways depending on your network's DNS failover behavior and whether or not www.clientname-2.com is a real web site.

So, okay, as the last comment on that linked question indicates, you can "fix" this behavior by manually adding entries to the "XPages Performance" preferences pane (I have two because it's on two clustered servers, you see):

(Though it's blurred here, I note that the "Server" column stores only the common name of the server, not its full hierarchy. Let's hope all your common names are distinct!)

Alright, now that that is sorted out, Notes should use the http://whatever.com URL you typed in rather than its own ludicrously-naïve guess. Open up the app again and bam:

Oh. Okay. Here's where things got fun. At this point, the Notes client is essentially using an embedded web browser to point to http://whatever.com/admin/vr.nsf/Page.xsp and carrying with it no authentication information. The documentation cheekily suggests that this means that your "configuration is not correctly set up" and basically leaves it at that.

I'll spare you the intervening days of frustration and cut to the chase: I needed to create an Account document in the client, set up SSO on the server (I had been using normal session auth), and disable loading Internet configurations from Server\Internet Sites documents (seriously).

Because I wasn't about to have my users use the Preferences -> Accounts pane if I could avoid it, I discovered that you can apply Accounts via policies. I went into the "Accounts" view of the "Configuration" tab of Administrator and created an account like so:

(That bit about "PreferredUsernameField" was because I stole the instructions from something about the social stuff in 9. I don't know if it's required, but I don't dare remove it now.)

Once I had that account document created, I went to the active Desktop policy document, then the "Accounts" tab, clicked "Update Links", and chose "All Supported" (because I only had the one anyway):

So at this point, I had it so that the Notes client was properly picking up the Account document, but it still wasn't signing in via SSO properly. That's when I had flashbacks to setting up a Sametime server, I felt a dread in the pit of my stomach, and I disabled Internet Sites for the server. Once I generated a new SSO configuration (with no "Organization" specified, so it was choosable in the server document) and restarted HTTP, it started to work. Hooray-ish!

Fortunately, in my case, I'm lucky: there's a spare server available that doesn't need to handle normal web requests (at least barring a catastrophe), so I can afford to disable Internet Sites configuration for it. However, long-term, I would be delighted to be wrong in my diagnosis - so if anybody knows a way to get this working while still supporting Internet Sites, please let me know. I would also love to know if there's a way to either avoid using those "run these apps on the server" entries in the client prefs or distribute those via policy as well.

David Leedy has run into some trouble with NotesIn9.com being down and he asked if I could help him get the word out about it, which I'm more than happy to do:

Hi - Just wanted to drop a note out there about my NotesIn9.com website.  Currently it's redirecting to someplace else for some unknown reason.  I assume it's been php hacked but I don't know.

Over the past couple weeks, I've been trying out some new things to make my code a bit cleaner and I've been having a good time of it, so I figured I'd write up a quick summary.

Composed Method

First and foremost, I've been trying out the Composed Method of programming. This is not a new idea - indeed, it's a pretty direct application of general refactoring - but the slight perspective change from "factor out reused code" to starting out with clean, small blocks has been a pleasant switch. The gist of the idea is that, rather than writing a giant, LotusScript-agent-style block of code first and then breaking out individual components after the fact, you think in terms of small, discrete operations that you can name. The result is that your primary "control" methods end up very descriptive and easy to follow, consisting primarily of ordered phrases specifically saying what's going on. I encourage you to search around and familiarize yourself with the technique.

import static

Java's import static statement is a method-level analog to the normal import statement: it lets you make static methods of classes available for easy calling without having to specify the surrounding class name. Most of the examples you'll see (like those on the linked docs) focus on the Math class, which makes sense, but it applies extremely well to XPages programming. Pretty much every app should have a JSFUtil class floating around, and most should also use the ExtLibUtil class for its convenience methods like getCurrentSession:

import static com.ibm.xsp.extlib.util.ExtLibUtil.getSessionScope;
import static com.ibm.xsp.extlib.util.ExtLibUtil.getCurrentSession;

// ...

getSessionScope().put("currentUser", getCurrentSession().getEffectiveUserName());

Night and day? Not really, but it's a little more convenient and easier to read.

Now, as the linked docs above indicate, import static comes with a warning to use it cautiously. Part of this is Java's cultural aversion to concision, but it's also a good idea to not go too crazy. When the methods you're importing are clear in intent and unlikely to overlap with local ones (like those in ExtLibUtil), it makes sense, particularly because Designer/Eclipse allows you to hover over the method or hit F3 and see where it's defined.

createDocument

Since creating a document in the current database with a known set of fields is such a common action, I've taken a page from one of our extended methods in org.openntf.domino and have started using a createDocument method that takes arbitrary pairs of keys and value and runs them through replaceItemValue, along with a helper method to allow for storing additional data types (since I'm still using the legacy API for these projects):

protected static Document createDocument(final Object... params) throws Exception {
	Document doc = ((Database)resolveVariable("database")).createDocument();

	for(int i = 0; i < params.length; i += 2) {
		if(params[i] != null) {
			String key = params[i].toString();
			if(!key.isEmpty()) {
				doc.replaceItemValue(key, toDominoFriendly(params[i+1]));
			}
		}
	}

	return doc;
}
protected static Object toDominoFriendly(final Object value) throws Exception {
	if(value == null) {
		return "";
	} else if(value instanceof List) {
		Vector<Object> result = new Vector<Object>(((List<?>)value).size());
		for(Object listObj : (List<?>)value) {
			result.add(toDominoFriendly(listObj));
		}
		return result;
	} else if(value instanceof Date) {
		return ((Session)resolveVariable("session")).createDateTime((Date)value);
	} else if(value instanceof Calendar) {
		return ((Session)resolveVariable("session")).createDateTime((Calendar)value);
	}
	return value;
}

In use, it looks like this:

Document request = createDocument(
		"Form", "Request",
		"Principal", getCurrentSession().getEffectiveUserName(),
		"StartDate", getNewRequestStartDate(),
		"EndDate", getNewRequestEndDate(),
		"Type", getNewRequestType()
);

The result of these changes is that I'm writing less code and the intent of every line is much more clear. It also happens to be more fun, which never hurts.

I wrote a post over at the Social Biz UG site covering a number of traps and conceptual hurdles I frequently see people running into when it comes to Java:

https://www.socialbizug.org/blogs/gidgerby/entry/java_traps_and_misconceptions?lang=en_us

One of the side effects of working on the OpenNTF Domino API is that I saw every method in the interfaces, including ones that were either new to me or that I had forgotten about a long time ago. One of these is the "parseXML" method found on Items, RichTextItems, and EmbeddedObjects. This was added back in 5.0.3, I assume for some reason related to the mail template, like everything else added back then. Basically, it takes either the contents of a text item, the text of a rich text item, or the contents of an attached XML document and converts it to an org.w3c.dom.Document object (the usual Java standard for dealing with XML docs).

That's actually kind of cool on its own in some edge cases (along with the accompanying transformXML method), but you can also combine it with ANOTHER little-utilized feature: XPath support in XPages. So say you have an XML document like this attached to a Notes doc (or stored in an Item):

<stuff>
	<thing>
		<foo>bar</foo>
	</thing>
	<thing>
		<foo>baz</foo>
	</thing>
</stuff>

Once you have that, you can write code like this*:

<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core">
	<xp:this.data>
		<xp:dominoDocument var="doc" formName="Doc" action="openDocument">
			<xp:this.postOpenDocument><![CDATA[#{javascript:
				requestScope.put("xmlDoc", doc.getDocument().getFirstItem("Attachment").getEmbeddedObjects()[0].parseXML(false))
			}]]></xp:this.postOpenDocument>
		</xp:dominoDocument>
	</xp:this.data>
	
	<xp:repeat value="#{xpath:xmlDoc:stuff/thing}" var="thing">
		<p><xp:text value="#{xpath:thing:foo}"/></p>
	</xp:repeat>

</xp:view>

I don't really have any immediate use to do this kind of thing, but sometimes it's just fun to explore what the platform lets you do. It's one more thing I'll keep in the back of my mind as a potential technique if the right situation arises.

* Never write code like this.

Yesterday, we released milestone 1 of our improved Domino API. This is our first tagged release meant for proper testing - all of the classes are implemented, many of the banner features are in there, and we've been using it in various real-world situations. I switched a couple of my side projects over - my portfolio site, the code for this blog (though I haven't deployed the template yet), and a couple personal game-related apps. That kind of testing is going to be crucial in getting us to a real ready-for-production release, and it's pretty exciting that we're already this far.

I encourage you to go check out the release on OpenNTF, browse and follow the code on GitHub, and don't hesitate to pitch in. We welcome contributors of all types - outright code contributions, real-world testing, or just feature requests. One of the overarching goals of the API is to solve as many of the little annoyances that we've all dealt with for years, so definitely let us know, either individually or via the Issues section of the GitHub project.

We're hard at work on some of the next steps: more documentation, streamlining the process of upgrading from the legacy to the new API in various scenarios (agents, XPage apps, DOTS, etc.), and all of the issues slated for M2. Watch that space!

As Nathan and Tim posted earlier today, a number of us have been working recently on a pretty exciting new project: the org.openntf.domino API. This is a drop-in replacement/extension for the existing lotus.domino Java API that improves its stability and feature set in numerous ways. The way I see it, there are a couple main areas of improvement:

Plain Old Bug Fixes

doc.hasItem(null) crashes a Domino server. That's no good! We've fixed that, and we're going through and fixing other (usually less severe) bugs in the existing API.

Modern Java

Java has progressed a long way since Domino 4.x, but the Java API hasn't much. Our API is chock full of Iterators, proper documentation, generic type definitions, preference for interfaces over concrete classes, and other trappings of a clean, modern API. Additionally, Nathan put together some voodoo programming to take care of the recycle thing. You read that right: recycle = handled.

New Features

In addition to cleaning up the existing functionality, we're adding features that will be useful every day. My favorite of those (since I'm implementing it myself) is baked-in MIMEBean-and-more support in get/replaceItemValue:

Set<String> stringSet = new HashSet<String>();
stringSet.add("foo");
stringSet.add("bar");
doc.replaceItemValue("Set", stringSet);

doc.replaceItemValue("Docs", database.getAllDocuments());

NoteCollection notes = database.createNoteCollection(true);
notes.buildCollection();
doc.replaceItemValue("Notes", notes);

doc.replaceItemValue("ExternalizableObject", new MimeType("text", "plain"));

List<String> notAVector = new ArrayList<String>();
notAVector.add("hey");
doc.replaceItemValue("TextList", notAVector);

doc.replaceItemValue("LongArray", new long[] { 1L, 2L, Integer.MAX_VALUE + 1L });

That's all legal now! We have more on the docket for either the initial release or future versions, too: TinkerPop Blueprints support, easy conversion of applicable entities to XML and JSON, fetching remote documents from DXL or JSON files, proper Logging support, and more.

As we've been developing the new API, we've already found it painful to go back to the old one - it doesn't take long to get spoiled. And fortunately, the transition process will be smooth: you can start by just replacing one entry point for your code (say, changing "JavaAgent" to "org.openntf.domino.JavaAgent" in an agent), then advance to swapping out your "import lotus.domino.*" line for "import org.openntf.domino.*" line, and then start using the new methods. All the while, your existing code will continue to work as well or better than before.

We hope to have a solid release available around the beginning of next month, and in the mean time I highly encourage you to check out the code. Download it, give it a shot, let us know how it works for you and if you have anything you'd like us to add.

Since I've been neck-deep in LotusScript and Java for the past couple weeks, I decided to take a bit of a sanity break today and play around with Ruby. Specifically, I wrote a skeletal wrapper for the Domino Data Service in the ExtLib and the first steps of a Rails app using it a bit. I don't expect this to actually be useful down the line, or even necessarily to get any more work put into it, but it was a fun diversion.

The API takes the same general shape as the normal Domino API, except you start with a database, which can take connection and credential parameters (though the credentials don't actually work for some reason). So to access the testing database I created, I do this:

db = DHTTP::Database.new(
  :server => "api.frostillic.us",
  :path => "tests/http.nsf"
)

In the Database class, I implemented a way to get all views or to get a view by name, while views let you get entries, which in turn contain their values and a way to get documents, which provide the usual item access. For example:

db.views.each do |view|
    puts "Found view: #{view.title}"
end

view = db.get_view("TestView")
view.entries.each do |entry|
	puts "Column values: #{entry.column_values}"
	puts "Doc body: #{entry.document.Body}"
end

It's not particularly amazing, nor is it particularly efficient, but it does the job: it supports the service's paging for view entries and converts Date/Times to Ruby Times for document fields, so it could be used for light read-only use.

Then I set up an extremely bare-bones Rails app that uses the objects in the most direct and ugly way possible, but, again, it works:

class TesterController < ApplicationController
  def index
    db = DHTTP::Database.new(
      :server => "api.frostillic.us",
      :path => "tests/http.nsf"
    )
    @views = db.views
  end
  
  def view
    @title = params[:title]
    db = DHTTP::Database.new(:server => "api.frostillic.us", :path => "tests/http.nsf")
    @view = db.get_view(@title)
  end
end

The two pages just spit out lists of either the views or the view entries:

<ul>
<% @views.each do |view| %>
    <li><%= link_to(view.title, :action => 'view', :title => view.title) %></li>
<% end %>
</ul>

...and...

<ul>
<% @view.entries.each do |entry| %>
    <li><%= entry.column_values.inspect %></li>
<% end %>
</ul>

I think the Domino REST API could be really useful. When I looked into it a while ago, I ran into trouble wherein the view entry counts didn't reflect entries hidden via reader fields, which is probably still the case, but other than that I imagine it could be put to real use. It brings it more in line with the other modern NoSQL databases, which generally use HTTP/JSON-based APIs as well. Combine that with Apache for load balancing and failover and suddenly you have a really compelling modern database back-end for other platforms.

If you're interested in looking at the code, I tossed the API up on GitHub: https://github.com/jesse-gallagher/Domino-HTTP-API-for-Ruby