The other day, I attended Tony McGuckin's Connect session on XPages scalability (a companion to his earlier Masterclass on performance) and I was inspired to try out a thought I had: could it work to use Couchbase to further push the bounds of XPages scalability?

If you're not familiar with Couchbase, it's a product similar to CouchDB, which is in turn essentially Domino-the-DB-server re-done for modern needs. Last summer, I wrote some data sources to use Couchbase data in an XPages app in much the same way that you use Domino documents and views.

In addition to Couchbase's nature as a document database, it has another layer from its lineage as a continuation of Membase: its basis is a key-value store, suitable for caching or storing data in a very fast and scalable way. This capability is potentially a perfect match for the needs of XPages state holding: when an XPage is stored in between requests, it's really just taking a Java object and storing it by key either in memory or on disk. By taking Couchbase's key-value capability and using it for this, you end up with something that may (pending testing) really crank up the scalability.

So I wrote a class to do it, and so far, so good: it stores and retrieves the XPages properly and even automatically obeys session timeout to clean up unneeded entries. When I get home, I plan to convert it into a plugin, but for now I've uploaded the Java file:

CouchbaseStateManager.java

To use it, install the Couchbase Java Client (I dropped it in jvm/lib/ext, since adding them as Jars in the NSF was trouble), copy that class into your NSF in the "frostillicus" package, and modify your faces-config.xml and xsp.properties files similar to:

faces-config.xml

<faces-config>
  <application>
    <state-manager>frostillicus.CouchbaseStateManager</state-manager>
  </application>
  ...
</faces-config>

xsp.properties

frostillicus.couchbase.statemanager.uris=http://10.211.55.2:8091/pools
frostillicus.couchbase.statemanager.bucket=default
frostillicus.couchbase.statemanager.username=
frostillicus.couchbase.statemanager.password=

Once you have it up and running, you'll start seeing serialized views being stored in your Couchbase bucket (which is basically like an NSF) keyed by replica ID and the unique XSP view ID:

I'm looking forward to trying this out in a real setup, where I have the XPage running on one server and pointed at a cluster of multiple Couchbase servers elsewhere. This should really play to the strengths of both: it would use JSF's capabilities to take a little work off of the plate of the XPage server while letting Couchbase do its job of spreading the data across its cluster, caching in memory, and so forth. Barring unfixable performance pitfalls, this could push XPages performance and scalability even further.

TL;DR: "Build Automatically" and "Refresh automatically" are unrelated, but work together when both enabled when using SourceTree for source control.

I just had a conversation on Twitter where I went a little crazy talking about my hopefully-correct view of how the various sync processes in Designer work when you're dealing with source control.

The original impetus was a post by Mark Leusink about the fact that you have to build your NSF project in order to sync it with an on-disk-project (ODP) set up via Team Development. From there, that got into the topic of the "Refresh automatically" Eclipse setting that is helpful when using source control tools outside of Designer (e.g. SourceTree instead of eGit). The problem is that this stack of builders and settings is hard to get a hold of conceptually.

There are three "entities" to know about when it comes to how these things work together, along with a couple processes that can get your source code from one to the other. I tossed together this terrible image to visualize them:

I'll work from right to left, starting with the NSF. That part is nothing new, but the thing to know is how the NSF "knows" how to sync to an ODP. It's done via something called an Eclipse Builder. A Builder is a task you can attach to a project in Eclipse to do essentially arbitrary things when the build process happens for a project (or an individual part of a project). NSFs have a ton of them:

The one in question is the unnamed one at the top (don't ask me why it's nameless): that's the Builder that exports your changes from the NSF to the ODP. There's a companion Builder, also unnamed, in the ODP, that does the reverse when you make a change to the files there (normally you don't, but source control does). Because this sync is implemented with a Builder, it doesn't run automatically when you have "Build Automatically" disabled. Instead, it will run when you manually do a "Build All" or "Clean", or when you choose "Team Development" → "Sync with On-Disk Project" from the application list.

So the Builder handles all of the syncing between the ODP and the NSF. Where does "Refresh automatically" (which is very different from "Build Automatically") come in, and why has it only become a thing recently with the cultural shift away from eGit and towards SourceTree?

The culprit is the way Eclipse/DDE deals with the filesystem. Because OSes have traditionally been bad at notifying programs of when files change on disk, DDE doesn't actually know when a file is changed on disk. This is expected for something like an NSF (say, if someone else adds a view to an NSF you have open, it's no surprise that you have to refresh to see it in Designer (also, don't edit the same NSF from two places like that)), but counterintuitive with local files. The view of the ODP you see in Designer's Package Explorer is analogous to running an "ls" or "dir" command in the terminal and then adding a new file: the output from your command won't change until you run it again. What "Refresh automatically" does is attempt to keep its file listing up-to-date. Later versions of Eclipse modified this option to be more explicit and technical:

The upshot is that turning this on will (hopefully) keep the ODP in step with the filesystem. The chart I made is misleading here: the ODP and filesystem are technically the same storage area, but the ODP you see in DDE is really a listing of what Designer saw the last time it looked.

This is useful when using a tool like SourceTree to handle your source-control operations. Because DDE doesn't normally "know" about filesystem changes, you could update your local source repository from the server with tons of changes, but Designer wouldn't know until you kicked it. With "Refresh automatically", it should notice those changes on its own, update the ODP in Package Explorer, and, if "Build Automatically" is on, sync those changes into your NSF.

The reason this matters now and didn't/doesn't when using eGit is that, because eGit is an Eclipse plugin and does its filesystem access through Eclipse, Designer already knows when a source-control-related operation changes files on the filesystem: since it made the change, it doesn't need to refresh. So if you're using an Eclipse plugin for source control and not editing files on the filesystem manually, you can remain in blissful ignorance about "Refresh automatically."

A little while ago, I converted over to the practice of separating your XPage app from the NSF that contains the data, even when it's going to be a one-to-one mapping. This confers a number of advantages in the areas of easier testing, easier development, programming discipline, and security. One of my favorite aspects is the ability to do this on your data DB:

This alone adds tons of security: no more worrying about ?ReadViewEntries, hiding your views from the web, errant forms allowing too much access, and so forth. Once you have that checked and you funnel your access through a separate app, the surface of vulnerability is much, much smaller.

But there's a hitch.

I discovered this the other day, and it's only a problem in mixed web/Notes environments: when you have that option enabled, you can't see native rich text content from a document in your XPage. I say "native" to refer to the traditional Composite Data format of rich text; MIME content is fine. The reason for this appears to go down to the C-API level, where setting that flag seems to entirely bar the HTML conversion functions from working with that database.

My guess is that that was deemed the most expedient way to implement the feature at the time, but it's problematic now. I can think of a couple ways to deal with it:

• Don't use native rich text. If you're writing whole-cloth XPages apps, you don't have to worry about this at all, since MIME/HTML content is still fine. If you have a mixed app and you're fine with some ugly-looking text in the Notes client, you can turn on the field option to store the data as MIME even in the client.
• Store the app on separate servers. To maintain the security benefits, you could store the data database on a different server, one not publicly accessible via HTTP.
• Disable that option and use web rules instead. You should probably be able to achieve a lot of the security benefits by substituting all HTTP requests for the data database with blank pages, though that requires coordination of the server config with every DB like this, which is a drag. Also don't forget __$replicaid.nsf URLs.

Fortunately, the fact that it only affects conversion from native rich text to MIME means the impact is limited, but it's still a caveat to data separation.

In spite of its heinous, brutal titular attack on English-language rules of plural punctuation*, I urge you to go read David Leedy's recent post about Paul Hannan. I can't add too much other than my agreement. Paul always does an excellent job filtering through our grousing on Twitter and it'll be a shame to not meet him in person at my first Lotusphere.

* I kid.†

† Not really.