In my last post, I ended up recommending that Domino developers and administrators take some time, if at all possible, to dive into new tools. In some cases, that may be in order to stay indefinitely, but, even otherwise, learning a new environment would have tremendous benefit to your existing Domino work.

Since I'm not a full-time administrator myself, my advice is admittedly a bit thin. It mostly involves my oft-repeated suggestion to try out a reverse-proxy setup for HTTP. That could parlay into use with mail as well, but even just becoming more familiar with a multi-tiered setup and the essentially-standard OpenSSL tools for certificate manipulation would pay dividends.

For developers, there are a great many potential paths to walk, and I think it's extremely useful to at least take one step down each. The use of that reminds me of the "Programming Languages" course I took in college. It involved rapid-fire coverage of C, C++, Scheme, and Prolog - four distinct languages with pretty distinct programming styles (the latter two moreso than the former two), meant to introduce students to the different notions of programming. I don't use Lisp or (lord knows) Prolog in my day-to-day programming life, but having learned a little bit about functional and logic programming at the time has paid dividends over the years.

There are a couple broad topics I can think of that are worth investigating, including ones that we as a community have already been working on.

Source Control

For the most part, the importance of source control has basically sunk in as a "we should do this" sort of thing in the Domino community. However, the inconvenience associated with maintaining an on-disk project means we're not all the best at doing this religiously. And, indeed, we're often hampered by both old habits and Domino's licensing terms that prevent us from doing this properly: there's a strong strain of "multiple people working on the same NSF" in Domino's history, and that doesn't mix well with source control (or XPages, for that matter).

So I recommend not just knowing about source control, but also trying (as much as licensing allows) to use this as an opportunity to switch to a more "traditional" model for app development. I like to think of this type of development as equivalent to working on a desktop app: it would be madness to have multiple people working in the same Xcode/VS/etc. workspace on the same binary build, and then particularly trying to have an end user also use that resultant binary. Instead, it's better to have each developer work separately on distinct non-replica NSFs, ideally on local dev servers, and then deploy the changes as an NTF.

Java EE WARs

Java EE in general is a very worthwhile target for interest, especially because it encompasses so many other possibilities. But I'd like to call attention in particular to the WAR file model, which is roughly equivalent to an NSF for app deployment.

When I first learned web programming, with PHP, it followed the "just toss a bunch of files on disk and visit their paths in a web browser" model, which basically worked, but taught some very bad habits. Domino improves on this a bit, with everything in its place in an NSF, but still suffers from a bit of the same malady: the "filesystem" of design elements is what is exposed to the browser directly, URL-wise. In a more fully-app-based system, though, there's a lot more flexibility about how the incoming URLs map to the underlying code and resources, and Java EE's encapsulated nature plays on this well.

With a WAR file, you have the "app container" advantage of an NSF but also gain the "everything after the app name is mine" benefit of a controlled URL router, and that shift helps conceptualize apps as something of a "mount point" onto an app server. So you may have a blog app that you load up as "/blog" on your server, and then that app can control the traffic beneath it, but is similarly a bit walled off from everything above it. It's the sort of "feel" that settle in with time, and then it feels weird to lack that flexibility.

This model also fits particularly well into app platforms like BlueMix or Heroku.

Testing

This one's difficult for Domino developers! No matter the tooling used for an NSF, being able to dive into automated testing is difficult. You can coax Maven into running some tests, though that will lack an XPages environment. You can test on the XPages side with JUnit4Xpages, though that won't be part of an automated build without more hoops. There's no real equivalent to the type of fully-fleshed-out testing tools available for other platforms now.

So that's lead to unit/integration/whatever testing not even being on the horizon for most Domino developers. That's particularly a shame because it's already something of a chore to get into the testing habit even when the tools do everything they can to help you. It's essentially like brushing your teeth when you're a kid: you do it because you're told to, but it's not something you'd ever do of your own accord.

It's worth it, though! It will probably require working on a fully non-Domino project, but going through the steps of building a fleshed-out system with automated tests along the way is a valuable experience. The goal is to get to the point where it feels weird to not have tests, much in the same way as lacking source control - an untested change should feel like it doesn't "count". It's a big world, too: beyond just small unit tests, there's tooling to run a web app in a test environment and run user interactions, or do the same on mobile-device simulators.

Build Servers

This dovetails with testing, in that one of the primary benefits of having a build server like Jenkins or Bamboo is that they can act as a neutral, clean environment to build the whole app and run the tests, informing you of any errors or failures. This is another thing that's made much more difficult by Domino, though it's possible.

Beyond testing, having a build server reaps benefits with project organization and delivery pipelines. Jenkins's recent updates have focused on that sort of thing, where the tool will help you manage pulling down the latest code from SCM, building it, and then deploying the results to dev or live servers, as desired. Build servers won't necessarily change the way you develop, but they will prove to be a faithful companion, with more uses the more you get comfortable with them.

Unix

Get comfortable with the command line. Other than Microsoft's tooling, which has a very schizophrenic relationship with the CLI, basically all modern development environments swim in the Unix world. Not everything has to be done via CLI - certainly, it's much more comfortable to use SourceTree than to do all git commands manually - but it's hard to find a tutorial that doesn't have at least a few tossed-off terminal commands. Mac- and Linux-based developers are at a distinct advantage with this, but Windows has Cygwin and the ported Linux userland in the next Windows 10 release.

Along this line, it's tremendously useful to have a bit of Linux server admin knowledge. I recommend either spinning up a Linux VM locally (Ubuntu has served me well) or using Linode. Linode in particular is so cheap and convenient that it's very much worth tossing them a couple bucks to run a server for a little while with full confidence that you can melt it into the ground without consequence. The delightful iOS-focused podcast Under the Radar had an episode on this topic a while back that's worth a listen.

You should probably also learn a little about vi.

MVC

Back in the realm of pure programming knowledge, it's extremely valuable to learn about MVC and related concepts. XPages sort of brought this to the Domino world, but it suffered from lack of wholehearted adoption and its legacy baggage made it very difficult to really go full MVC even if you wanted to.

The good news is that essentially every other active framework encourages or enforces this approach. When diving into one, it can be a bit jarring at first - there's a lot of "ugh, can't I just put the code here?" going on - but it falls into place very quickly.

So where to start?

Picking a place to start is a daunting prospect, given the ever-shifting sands of the modern development world.

One of my favorite tutorials for a web framework is Michael Hartl's Rails tutorial, which is kindly available for free online (with options to purchase other forms). It not only provides an up-to-date introduction to Rails development, but also covers a lot of the other ground mentioned above: the command line, Heroku, Git, and test-driven development.

In another direction, I recommend taking a look at Meteor, which is something of an all-in approach to JavaScript-based development. It does an interesting job blurring the lines between client and server and makes a very compelling case for "JavaScript everywhere".

Beyond that, really just learning any development toolchain will help. There's not too much need to sweat picking "the right one" at each step - any knowledge gained in one will help in others, either directly or by giving you some additional perspective. The most important thing, especially now, is learning something.

Though most of my paying work involves working with clients' creaky old databases, greenfield projects will always have my heart. For these and for my side projects, I've been moving towards an ideal Domino-app strategy that seems to serve me well.

The Setup

My apps consist of at least two NSFs: one for the app itself and at least one for data. The only non-design notes in the app NSF are for app configuration, and even then I keep it limited. Everything else goes in the "data" databases, which have designs consisting only of views, forms (rarely), and any agents needed for migration between versions.

This has several advantages I've talked about before. Importantly, other than the "Don't allow URL open" problem, which can be avoided entirely, I've encountered no down sides to this approach. The amount of flexibility it provides and the discipline it imposes pay massive dividends while not significantly increasing hassle. I recommend everyone do it. I try to keep an eye towards future scaling/separation needs as well: I tend to give my app configuration options where each document category are stored (say, Tasks/Projects vs. Clients vs. Update Notifications), even if I set all of them to the same location.

Scaffolding

As you might expect, I use the latest version of my XPages Scaffolding project for the app NSF. Now that my model framework supports attachments, there's little preventing me from using it extensively for data access in future projects.

Once those classes settle down a bit more, I expect that I will package them up as plugins so I can have a consistent "frostillic.us Framework" to attach a project to, allowing me to roll out bug fixes to major+minor versions (e.g. frostillicus.framework_1.2) without using classic-style design inheritance.

One of the potential future bonuses of using my model framework combined with inherent data separation is that I will be able to write adapters for my framework to connect to other data sources without the app having to care much (beyond the config document for pointing to the other data store). As long as I do the legwork to write the model classes, the bulk of the app won't have to care whether it's talking to Domino, JDBC, CouchDB, or a CSV in the data directory.

Source Control

I use Git early and often. As soon as I start work on a project, I create a repository on Bitbucket and then commit the app and data NSFs to it immediately. This has a couple crucial implications:

1. I try to focus my changes into conceptual units for commit purposes. Though it's exceedingly rare that I actually need to reverse a commit, the better I am about only committing related changes (i.e. not commits like "implemented a wave of client requests"), the easier it is to do so if needed.
2. I develop using either a local Domino server or NSFs on a shared server that is dedicated to my current machine. If, for example, I want to work on the same project on both my desktop and laptop, that means I will have two copies of each NSF even if they're on the same machine. This just saves a lot of hassle while simultaneously reinforcing the source-control-focused dev model.

Prototyping

I haven't had a chance to put this into real use yet, but, now that I'm a convert to renderers, I plan to get into the habit of doing my initial prototyping using standard ExtLib components with OneUI as the theme. I'll still have to keep an eye towards the eventual goal (for example: avoiding the Dojo-exclusive components or ones I don't really want to write renderers for if I'm not planning to actually deploy with OneUI), but this will make it much quicker to start laying down the common components of the app immediately without having to worry about specifically implementing Bootstrap-or-other-framework classes and HTML structure.

Because renderers are more difficult to write than custom controls with the HTML hard-coded, this will increase the overall development time, but the flexibility and off-loading of responsibility to later in the dev cycle will be worth it. Additionally, each render kit is easier to write than the last, particularly if I can reuse/extend previous work.

Deployment

As mentioned, the use of source control imposes a strict structure on development from multiple machines. This discipline contributes to dealing with production versions of the database. It's always been the case that it's a terrible idea to do work in a production version of a database, but Notes always made it so easy to get away with it. Fortunately, XPages' PITA compilation requirements and gotchas make this a terrible idea (who wants their app to go down because you opened it in Designer but forgot to install a required XSP library first?). This is all the worse when source control is involved: a bug in the SCM plugin or a missing network/VM share can easily destroy every design element in the target database. If it happens to be one dev DB among many, who cares? If it's your production DB, now you have a serious problem.

When I'm ready to deploy my changes, I do it via NTFs: I create NTF copies of any applicable dev NSFs and use them to replace the design of the production databases. Because these apps tend to be single instances, I don't bother with template names and design inheritance. In the case where there are many copies of the data databases (say, one core app and then one data DB per client), I'd probably use inheritance and replace the data DBs' NTF's design instead and immediately run the design task.

Since I have NTFs of each deployment sitting around, it makes it easy to roll a production DB back to a previous state. Though I don't currently, I plan to start tagging releases in Git (or just switch to GitFlow, which I've yet to use) to make this clearer as well.

Another improvement I'm considering making is to drastically limit my own rights to the production DB, to a max of Editor access. This way, there's an extra layer standing in between me and the terrible notion of opening the live DB in Designer. I figure I'll either use a dedicated deployment ID or use Full Access Administration mode to make deployments.

 

---

 

I've been feeling pretty good lately about this emerging scheme. A lot of it aligns with standard practices for non-Domino development (data/app separation, heavy source control) while still maintaining the benefits of Domino and XPages. And once this process is firing on all cylinders (namely, once I have skill in renderers to cover all of my regular needs), it should yield pretty impressive results when it comes to putting out fleshed-out, scalable apps quickly without making my development life down the line harder.

Yesterday, I suggested that everyone enable finalizing parameters in their code cleanups and, better, to do the same in their Java save actions; later, I set Eclipse loose on the org.openntf.domino API to implement it there. Adding final keywords to method parameters (and elsewhere where possible) is one of the habits I picked up from reading Effective Java a bit ago, but it's kind of a subtle thing and the benefits aren't immediately obvious.

If you're not at all familiar with the final keyword, it has a couple meanings in Java - in the context of variable and parameter declarations, it means that you can't reassign the value of the variable after it's been assigned the first time. So, for example, this is illegal and will generate a compile error:

final int i = 1;
i = 2;

In that example, it seems like a "why the heck would I do that?" sort of thing, but it's not too much of a stretch to picture a block of code involving something like this:

void doSomething(Document doc) {
	System.out.println("doc is " + doc.getUniversalID());
	
	// a bunch of code
	doc = database.getDocumentByID("FFFF0002");
	// a bunch more code
	
	System.out.println("doc is " + doc.getUniversalID());
}

That will WORK and it won't damage the original doc that was passed in, but it'll get confusing, particularly the more lines of code or eyeballs involved. Functioning or not, it's poor code style, and changing the method signature to doSomething(final Document doc) means that the compiler will enforce good style.

In addition to being good style, final has some technical benefits. First off, it might be faster. Secondly, if you ever use anonymous classes, final variables are a necessity for accessing information in the surrounding block. If you get into the habit of finalizing your variables, you'll already be ready by the time you have a need for anonymous classes.

final is a component of a larger and more important topic: immutability. I should probably write a lengthy post on immutability in the future, but for now there's an important thing to remember: final does not make your objects immutable. It only prevents assignment - it doesn't prevent calling methods on an object that change the object's value (in fact, Java lacks any language construct or cultural convention for immutability). So even if your Document variable is declared final, you (or any code you pass it to) can still call doc.replaceItemValue("Form", "Ha ha"); with impunity.

I'm always looking to improve my programming/development/administration methods, and in the past couple months I've made a few more changes. To my chagrin, these changes are mostly things that I should have been doing for a while now and I've been sloppy to do them differently, but hey - fixing a problem is still improvement!

Serious About Source Control

I've been using source control for a while now, and I hope everyone else has as well. However, I've been lax in a couple ways, which bit me a while ago. For example, I had a couple actively-used projects tied to source control in Designer - a few NSFs, a few NTFs. That was fine NORMALLY, but then this sequence of events happened:

• I run Designer in a Parallels and have my repositories stored Mac-side, pointed to via a drive-letter mapping to my home folder
• I Remote-Desktop'ed into my Windows VM from another machine
• Remote Desktop remapped the drive letters to share local resources
• Designer, which was running at the time, could no longer find the repository folders
• Apparently, Designer/eGit/whatever decided that the rational thing to do was to "sync" the NSFs with the now-missing repositories... which is to say, delete all design elements from the NSFs

Imagine my surprise when a previously-fine DB I hadn't intentionally touched in a while was suddenly naked, consisting of a single view that somehow survived the purge. Fortunately, the nature of the problem meant that the design was still stored in the repository and thus recoverable, but that's still not a problem you want to have.

So I've been shifting my development process another few steps closer to the ideal. The way I see it, that ideal is:

• Never open a production DB in Designer
• Do all development in entirely-distinct NSFs created either from scratch (for a new project) or via the "Associate With New NSF" action from the source-controled on-disk project. These NSFs should be worked on by a single person and should either be in an individual-user dev folder on the server or, better, on a local development server
• Create deployment "builds" as NTFs, either as design-only copies of a source-associated NSF or, again, from the "New NSF" action from an on-disk project (but it should then be disassociated immediately)
• "Deploy" the "builds" using Replace Design on the production NSF/NTF from the Notes client or, better, Administrator
• Ideally, the "build" NSFs should closely match a tag or branch in the source repository, but I can't be bothered to do that for my needs yet

That process goes the same for "classic" and XPages apps. Since source control became viable for Domino devs around the same time as XPages hit the scene, the two are often conceptually tied together, but in my experience the classic binary-DXL handling is plenty good enough to use it in projects without a line of Java or XSP.

Separate Application and Data

The standard way to develop a Notes app - classic or XPages - is to bundle the design and data together. This has historically had a lot of benefits and still makes sense in some cases (say, customizing individual mail files for different uses). However, particularly in the case of a standalone "app" (a help-desk application, a project-tracking system, etc.), this tight joining makes little sense and would be considered a bizarre aberration by anyone using most other systems.

Fortunately, as Domino developers, even if we split up the design and data, we still have it easier than most, since we don't have to worry about drivers and authentication credentials for the back-end DB (unless we want to). In the simplest case, which I've adopted, you have your "app" NSF and a "data" NSF next to it. The "app" NSF contains no forms or views, which the "data" NSF contains no XPages, Java, or other "executable" entities. I've derived a couple benefits from this already:

• The app and data NSFs can have different ACLs
• The data NSF can be set to disallow URL open. This means you don't have to worry about, say, clever users trying ?ReadViewEntries to peek into more than they should see
• You can easily switch between "dev", "production", and "test" data NSFs, just like normal developers do. You don't have to worry about copying data around between different dev versions of your app
• Multiple apps can point to the same data with impunity
• The data NSF can exist on another server... or cluster of servers, which you could switch between programmatically
• Data sharding becomes easier. Depending on the type of app, each user/project/department/whatever could have a different backing NSF fronted by the same single app

Admittedly, this makes some things moderately more complicated. Any Domino data sources will need to have a databaseName property specified, which will need to get its value from somewhere (worst case, you could hard-code it to start with, then "upgrade" to a managed bean). You also won't be able to use "app.nsf/view/key"-style URLs, since the app won't have any views to search by - you'll have to use query parameters or cobble together your own good-looking URLs with the assistance of Web Rules. You might not want to make your structure too document-focused anyway.

 

These changes have been bringing my development closer to classic desktop-app development, where these are less "techniques" and more requirements due to the nature of development, but they've proven to be valuable. Domino's ease of getting started and fast-and-loose development methods have distinct advantages, but it's always important to question which ones are better than other environments and which are traps.