My current work project involves displaying arbitrary views from various databases into a combined reporting site (written in XPages). This has presented me with two hurdles: pulling in the data completely and accurately and then figuring out a good way to format it.

The former problem is one of the few areas where it seems like "classic" Domino development has an edge: views render rapidly and, as long as you've configured the server to display lots of rows or add in pager controls, completely. And with the recent "enhanced HTML" property, they generally get tagged with some useful CSS classes. The default XPages xp:viewPanel control, by contrast, needs the column definitions at design time, which makes it tough to work with. You can sort of roll your own with nested xp:repeat controls, but you quickly run into problems with categories and other Notes-y features. Fortunately, the Extension Library, as usual, comes to the rescue: the  xe:dynamicViewPanel does pretty much what you want: you give it a view and it renders it out nicely. Using it, you don't have to worry about handling categories, you can add in standard pagers, and you can include all the standard filtering and searching options on the view. It's not perfect (yet), though: it doesn't handle fixed-value columns (like a formula of "hi"), use color-column information, follow column hide-when formulas (though it supports the static "hide this column" checkbox), do Notes-style bracketed pass-through-HTML, process special text, or honor "show values in this column as links." These are all understandable limitations, being as they are either rarely encountered or technically difficult. Unfortunately, I've run into all of them rapidly, so I don't know if I can use xe:dynamicViewPanel as-is. The best thing to do would be for me to roll up my sleeves, implement these features in it, and submit my changes, and maybe I'll do that down the line, but there'd be a lot of learning overhead before that point. What I've taken to in the mean time is a Java class to write out the view contents as HTML. I add the color-column info via inline style="..." attributes, look up the hide-whens by exporting the DXL and then running them through session.evaluate (read: not fast), and split and recombine the strings to handle pass-through-HTML. Unfortunately, I lose speed (because I'm presumably not as good at this as IBM's programmers) and the flexibility to easily use searching, filtering, and paging. The first two will be relatively easy to implement, but paging will be a problem, since the XPage just gets back a big string blob. I could return Java data structures, but then I'd run into problems with categories. I could write classes to dynamically add XSP elements to the page, but at that point I'd be best off cracking open the ExtLib source code anyway.

The other problem is that, regardless of how I render the view, it can be tough to actually display all the data well on a web page. For normal views, with a handful of columns, it's fine to just plop it right on the page and be done with it. However, it doesn't take much to make the table look like crap: headers that are way too long, large strings in the row data, or just having lots of rows all very quickly stretch or break the design. I think I'll try to handle the headers by adding some CSS text-overflow rules to keep them to a fixed width, which will also mean adding in a title="" attribute so the user can hover over them to see the full value. Still, not a big problem. Large data cell values are a bit tougher. Theoretically, I could do the same thing: have the data overflow and add a tooltip. That would work fine if it's a rare occurrence, but what if the whole point of the view is to be able to easily look at that data. I may try to pull out the row-height property from the view and use that to provide a max-height to each cell via inline CSS, so that way I could solve the problem on a view-by-view basis. I could consider doing the same with width as well, rather than going the default route of letting the content width determine the column width. Having a lot of rows can be partially solved with pagination (unless I'm going the "dump out a blob of HTML" route), but is that really a good user experience if the goal is to see all of the data? Paging on the web is pretty awkward, with the number of rows never exactly matching the height of the viewport, worrying about scrolling back to the top when going to the next page, and properly dealing with categories. Via CSS, I've made it so that the primary content area has its own scrollbars via absolute positioning and overflow: auto and that works pretty well for the most part, but I still run into trouble with views containing thousands of rows: with a lot of data, it simply takes a long time to download the HTML and render the table. There are a couple options provided by Dojo and the Extension Library that provide fixed-size grids to store data that I've tinkered around a bit, and maybe I'll settle on one of them, but in the mean time every solution has some serious problem that I quickly run into with my example data.

My search for a useful way to use Domino as a database back-end for a Ruby front-end has continued and, more specifically, has continued to be difficult.

I gave a shot to the Java CORBA API. Initially, that went well: after grabbing NCSO.jar from my Designer installation, enabling DIIOP on the server, and setting up JRuby, I was able to connect to the server using the host name, canonical user name, and password. Great! However, once I exported my app's Java model classes (modified to use NotesFactory instead of the JSF runtime), I immediately ran into a problem: NotesView.resortView() is not implemented in CORBA. Guh. Theoretically, I could rework the app to use a different view for every index, but it's a canary in the coal mine: no doubt plenty of other new things (FTSearchSorted, for example) have been left unimplemented as well. Plus, this is Domino, so "new" probably refers to anything added after, say, R5.

So that's out, at least for now. That leaves me with the C API. Oh, the C API... so much promise, such a PITA to use. I haven't used C itself much since college, and my experience with the C API is limited to progress bars in the Notes client and, more usefully, my authentication filter for my server. However, I'll be programming in Ruby, not C directly, so it won't be so bad, right?

There are three main ways to work with C libraries in Ruby, I've learned: writing a Ruby module, DL, and FFI. In the first case, I'd basically have to write my API in C and then just use those objects from Ruby. That would be the best performance, but it's just asking for a nightmare. DL is programmed in pure Ruby, but it's so sketchy that just loading the library causes your program to emit a warning telling you you probably shouldn't be using it. FFI, though, is pretty great. Much like DL (or declaring external functions in LotusScript), you declare which functions you want to import, along with their signatures, and then it more or less just works.

That "more or less" is, naturally, tricky. I ran into tremendous trouble just getting it to load up libnotes.so, but most of that was due to using a 64-bit OS and a 32-bit Domino installation. I may make a post about it some time in case anyone else is crazy enough to try the same thing, but the upshot is that I ended up compiling my own 32-bit Ruby environment, which started to work.

Since then, I've been wrestling with the inherent traits and idioms of C (using string buffer arguments to return values instead of just returning a "string", for example) and the specifics of Notes (needing to be pointed to the executable directory as well as needing an ID file, an INI file, and a names.nsf to find the servers). After thrashing around for a while, though, I have it working to the point where I can point the initialization routine to an arbitrary INI file and it can pick up on everything it needs from there (I'm using a password-less ID file - I don't know if you can automate passwords).

Now that I have my foot in the door, it's "just" a matter of learning the entire API. I'm looking forward to that part, though - I've demonstrated that I can open a database on a server and get information out of it, so it seems like I have the right environment. If all goes well, I should be able to create a proper Ruby API for Domino, and a very fast one at that.

For reference, here's the script I've written. Don't expect clean code, reusability, comments, or proper methodology - this file is the result of random mauling during the course of development and just barely works to demonstrate the idea. It DOES demonstrate it, though, and works to ping a (hard-coded) server and get some info about its names database.

test.rb

As most developers probably do, I have a grab bag of "utility" functions/methods I use across various projects, and I figured I'd post a few of the handier ones.

The first is a basic XML-encoding function that just takes a string and returns something suitable for putting into an XML or HTML file. I've been too lazy to figure out if the standard Java library has an equivalent that doesn't involve creating an actual XML document in memory, so I just took the one I use for LotusScript and ported it over. It passes standard letter and number characters through as-is and then just encodes all others as Unicode entities. The string size increases accordingly, but it's presumably fast and it gets the job done, even if you have oddball characters.

The next set are just quick-and-dirty Java equivalents to the StrLeft, StrRight, etc. functions from LotusScript. They do more or less the same thing as in LS and make it so you don't have to worry about string indexes all the time.

The last is potentially the most useful, since it's the most esoteric: a special-text decoder. I've had a couple occasions where I want to spit out the contents of a view for the web, but also want to preserve things like child counts on categories. Fortunately, special text is stored in the view column in a workable format: a special non-alphanumeric character, a single letter indicating which function it corresponds to, a number representing the parameter count for the function, and then a delimited list of those parameters with character counts. Using that, I wrote a routine to process all of the special text functions I could think of other than @IsExpandable (since that's meant for the Notes UI). It even reproduces the weirdo behavior you get when you pass a multi-character string to @DocLevel.

StringUtils.java

In the interests of getting crap done, as soon as I was finished with my previous post, I fired up TextMate and a couple Terminal windows to start writing a Ruby wrapper for the DAS API.

It started out great! Wanting to avoid the minor hassle I ran into before with Ruby's built-in Net::HTTP library, I did a quick search for Ruby HTTP/REST libraries and picked one that worked well, named HTTParty. Before long, I had the rudimentary elements of a Notes API working, with classes to represent the server connection, NotesDatabase, and NotesView.

I was about to start working on the NotesViewEntry class when I decided I'll need a way to page through the view, since the call to the view "collection" (wisely) doesn't return all of the data at once. The API developers cleverly store this information in an HTTP header:

Content-Range: items 0-9/35

Perfect! But, um... I was getting the list of available forums as Anonymous, which can only see two, not all 35. Sure enough, the results only contained the two Reader-visible entries (phew), but that left a showstopping problem: I have fairly frequent need for the count of entries, and I don't relish the notion of having to fetch all of the data to get that. Crap.

So DAS is out for now. I remembered, though, about ?ReadViewEntries, an older and thus clearly more mature data-access API. I could probably do everything view-related I need with that and then use DAS to get any non-view document data. So I hit the view with ?ReadViewEntries&OutputFormat=JSON (the results are the same with the XML version) and, lo and behold, the top-level object fields (with data snipped) revealed the same lack of Reader knowledge:

{ "@timestamp": "20120123T034916,85Z", "@toplevelentries": "35" }

Crap!

My quest to treat Domino like a standalone database is off to a rocky start. I can think of a few remaining options:

1. COM: I don't deploy on Windows, so... that's out.
2. C API: seems like overkill and just asking for all sorts of new bugs.
3. Web Services: MIGHT work, but I can't imagine the performance would be anywhere near acceptable.
4. Some sort of crazy thing with servlets: seems too crazy.
5. JRuby and the Domino Java CORBA API.

I think the last is the most plausible. My forum app already has Java classes in between the XPages front-end and the Domino database, so I could possibly take those wholesale and just change the methods I use to get the session and current database to instead initialize a CORBA session. I'm a bit nervous about speed and running across some of the various "not implemented in CORBA" landmines hidden across the API, but it might just work. If it does, my next task will be deciding on a web server and a Ruby web framework that's not terribly SQL-centric.

As more and more of my Domino development has been with XPages, my databases have filled up with XPages, Custom Controls, and Java design elements. However, the more I use these, the more of a pain Designer becomes. It gets into its head on a regular basis the notion that it needs to recompile every single such design element - when I open the app in Designer (whether or not it was "greyed out" in the sidebar from before), whenever it "forgets" that I have it set to Automatically Recompile and I have to turn that back on (which is frequent), whenever classes just stop loading properly and I have to Clean the project, and sometimes just for fun.

In a basic app, this isn't so bad... it's pretty annoying, but at least it doesn't take too long. In my forums app, though, which contains dozens of pages, controls, and Java classes, it can take about 10 minutes when I'm not in the office. Ten minutes! Just to open the app! That's even ignoring the side effect that the app is unavailable on the web while this is happening, throwing various server errors until all the classes are back.

I've tried numerous things: switching the Java classes to and from the 8.5.3-standard Code/Java folder to a "classic" WEB-INF/src folder (the 8.5.3 folder seems to make it worse), turning off the "Automatically recompile dependent XPages" option relating to custom controls in the prefs, and creating fresh Windows VMs with entirely-clean installations of Designer. Theoretically, I could turn off automatic compilation in the "Project" menu, but then I'd have to do it manually, and I can only imagine I'd consistently run into times where it "needs" a full recompile anyway.

It's bad enough that it takes this long when I want to get hacking away at the code, but it's all the worse when I just want to go in and fix a CSS or JavaScript file, neither of which have anything to do with the big pile of Java classes Designer is so obsessed with. Nonetheless, if I'm going to get at them, I have to let Designer do its thing. There are potential workarounds - write a web-based editor for those design elements, store them in other databases, or store them as data documents - but those sacrifice either the convenience of a proper editor or of having all of the assets of my app in one place.

Other than my job, there are only a couple thing tying me to XPages for my for-fun projects: the fact that I now have a pretty huge foundation of knowledge in Domino development, the fact that XPages actually make a lot of fancy things very convenient, especially with the illustrious Extension Library, and Reader fields. The last one of these is surprisingly difficult to reproduce elsewhere. They're so elegant and bulletproof, much better than filtering results, writing giant nested SQL queries, or throwing up my hands and sticking with a basic "access level" type of restriction. And all of those would still mean I'd have to worry about ensuring that I don't accidentally select the wrong records anywhere I write a query, whereas Reader fields mean that the documents effectively don't exist, so I CAN'T screw it up.

So that leaves my main options as:

1. Suck it up and keep using XPages
2. Suck it up and give up Reader fields in exchange for a programming environment suitable for human use
3. Stop being lazy and write access classes in Ruby or PHP that use the DAS API, hoping that performance is good enough
4. Close my eyes and wish real hard that a bug fix patch will come out and magically fix all of the problems in Designer

Number 4 would be ideal, but I think number 1 is what I'll most likely do. Still, being able to use Ruby instead of Java would be such a breath of fresh air, and turning Domino into a "mere" database server via DAS both is appealing and would be mostly new ground. Given that the database-open operation I started halfway through this blog post is in its 16th minute and still going, maybe I really should get off my duff and try that out.

Edit: 32 minutes! Nice!

I've had a chance to start on my Keychain project from last week, enough to put together a thoroughly rough and unmaintainable script to do the uploading. It has all kinds of horrible properties: it doesn't do the Keychain dump itself (instead reading from a hard-coded file containing a keychain dump from the "security" tool), it doesn't abstract away any of the Domino DAS access, it doesn't check for existing versions of the items, it doesn't handle field data types properly, and it even writes to the filesystem. But hey, it's a start:

keychain-upload.rb (requires the json rubygem and uses auth information from ~/.netrc)

In spite of all its faults, it DOES push the data up to the database, and that's something. It stores the keychain file name in a field named "keychain", the "class" in one called "entry_class", the data in "data", and the rest of the arbitrary fields from the item in fields prefixed with "attr_". Once it was in the database, I set up a view called "Passwords" with the following formula:

SELECT Form="Item" & entry_class="genp" & attr_type != "\"note\""

The extra quotes around the type field's value are an artifact of the non-existent type handling in the original script - it just stores the values exactly as they appear in the dump file, rather than converting them to null, numbers, or strings.

It's not pretty, but I now have a way to view my Keychain entries on the web. I'll probably give a shot to doing this the "right" way via a Cocoa app and structured classes down the line, but for now it'll already be useful to me.

With Apple's transition to iCloud, they're getting rid of Keychain sync. This is too bad, since that was one of the MobileMe features I actually used, loved, and never had problems with. Fortunately, all is not lost: worse comes to worse, I can pick up a copy of 1Password, which has the advantage over MobileMe of being cross-platform.

But I'm a programmer, right? Why buy something - especially for $50+ - when I can just write something myself? My needs at the moment are fairly simple - I only REALLY want a way to back up my Keychain and view it on the web, for when I'm in Windows or on my phone and want to check a password. I only use the one Mac currently, so I don't need proper sync, and I don't need to pull new passwords back down into the Keychain. Maybe down the line, but one step at a time.

The Keychain itself takes the form of a file housing a collection of records with a body of essentially arbitrary data (usually a string, but it can be different for certificates or notes), with a composite key consisting of, I think, the type, server, account, and maybe some other fields. The only thing that would make this a better test case for Domino 8.5.3's shiny new Data Access Services would be a 32-digit hexadecimal unique identifier, but it's pretty darn close as it is.

The main problem I have to getting starting is actually getting at the data. I think there are two main ways: the C-based security/Keychain API or using the "security" command-line tool, which is a more-or-less friendly wrapper for much of the same functionality. Being the lazy type that I am that only deals with C when I have to, I'm starting with the "security" tool. It has a handy "dump-keychain" command, so I can type something like "security dump-keychain login.keychain" and get a formatted list of all entries in that keychain, minus the "data" field. I can add the "-d" switch after "dump-keychain" to include that field, but that comes with a big caveat: you have to click an "Allow" button for every single item. Given that I have about 1,100 items, I'd rather avoid this fate. I've tried running it through sudo, but to no avail. I've tried fiddling with the "authorize" command in the tool, but also with no luck - I'm not sure if I'm just doing it wrong or if it's related to something else, which may or may not be what I want. I could use the "-r" switch instead of "-d" to get the encrypted data, which would satisfy my backup desire, but not my "check it from the web" desire. Absolute worst case, I could put on a movie or podcast and click "Always Allow" a thousand times.

However, once I get past that hurdle - whether it be via the C API or a lot of clicking - the rest should be easy. I should be able to create a basic database, grant DAS access to it, create a view sorted by the various components of the items' composite keys, and do a basic "view lookup, then update or create the doc" routine, pushing up the field values pretty much as-is. That should act as a perfect basic-case project for trying DAS out while also solving an actual problem I have.

The other day, I set out to put some work into my guild's forum site. Immediately, though, I noticed that the main forum page was taking significantly longer to load on my development server than on my production server. Specifically, it was taking about five minutes to load, as opposed to about a second. That seemed... problematic. Looking at the profiler, I saw that the page was somehow making 38 million calls to Document.UniversalID, which was slightly out of proportion to the two forums listed on the page.

After a lot of trial and error to find out what the problem was, I narrowed down the culprit to the list of recently-updated topics. Each of my model elements is represented by a class implementing the List interface, which is really like a wrapper around the view. Thanks to some debugging output on the console (with apologies to my bloating log), I found that the list, which is only supposed to get the first 15 entries in the view, was repeatedly processing EVERY entry. Spitting out a stack trace revealed that the XSP list processor was converting the entire List object to a string via its own method of using the List's iterator. Huh.

I investigated a bit, looking at the resultant Java code from my custom control, renaming the property that passes in the List, and looking at the name.xsp-config file that accompanies it, all to no avail. Next, I tried re-creating the custom control, but that has the same behavior. For the time being, I've found that copying the code from the control in place of where I use it in the XPage works for some reason.

I have no idea why doing that worked, since in theory it should be performing the same thing, and I have no idea why all my other uses of similar controls--such as the one for recent posts--aren't freaking out. This will certainly bear some investigation, since until I figure it out I'll have to live in fear of the runtime stringify-ing my giant Lists at will.

I was listening to the latest episode of Build and Analyze on the way home from work yesterday and, as I am wont to do, I started yelling at my iPhone when they started talking about Google Reader and the difficulty of syncing. Admittedly, at the end, they got to the fact that, even if you could do it technically, it'd be tough to make money off of providing an RSS sync server. That part is fair enough, but I still can't let the technical difficulties stand, and I've been thinking more about how it would be done in Domino.

In the basic form, the problem in question is pretty much exactly what NSF and the Notes/Domino relationship is designed to do: seamless replication, deletion stubs, unread marks, and so forth. In fact, RSS syncing is a better fit for the model than mail, since mail required adding all kinds of extra (but useful) functionality, while RSS syncing would just be data and an agent to fetch the feeds periodically.

The way I figure it, there would only be a couple technical hurdles, both related to scaling: storing large volumes of data and fetching new feed content periodically.

Storing large volumes of data might not be too bad. There are a couple ways you could do it. One would be to store the user's list of subscribed feeds and "read" stubs in one database per user, and then store the feeds and feed content in another database (or databases), and do all data access via agents or web services that would pull the data from each distinct location. Another way could be to store the feeds and entries in each user's database, keeping the feed content as attached HTML documents and letting DAOS handle efficient storage. The latter route would let you take advantage of the Domino Data Service and read marks (which DAS conveniently supports).

Updating the feeds would be rough for a single server, but the job could be farmed out to many clusters in a server. You could write agents that would determine the server they're on and, based on, say, its name, pick a slice of the feeds to check, so if you have 10 servers, each would update 10% of the feeds.

I'm sure there'd be other roadblocks during actual implementation (it IS Domino, after all), but I think that'd be basically all you'd need on the server side. The client would be a little tougher, since you couldn't just use NSF and selective replication, but that wouldn't be terribly difficult to handle.

It's too bad it's likely not profitable, between licensing, hosting, and bandwidth costs - it'd be a fun project to try out.

A while back, I described the problem I'm having in my guild-forums XPages app, which is that it very easily gets its environment out of whack, to the point where changing any design or data note from outside the XPages environment caused an "X is incompatible with X" ClastCastException. This improved gradually over time. At some point in 8.5.2, I started being able to modify data documents again without the problem. When 8.5.3 came out, it improved again: I can now replicate over changes from my development server to the production one without having to bounce HTTP on the production one. At that point, it became much less of a hassle - sure, I still had to re-save a Java class file every time I modified an XPage, but that was only in development, so I could deal.

However, I think I've found the proper solution. In 8.5.3 (I think), IBM added a custom editor for the xsp.properties file (which you can reach using the "Package Explorer" Eclipse view, in whatever.nsf/WebContent/WEB-INF). Many of the options are duplicates of what you see in the normal "Application Properties" editor (since they edit the same properties), but there are some nifty additional goodies. I won't go into them all, but I urge you to take a look. The important one here is in the "Timeouts" section of the "General" tab: "Refresh entire application when design changes" . That sounded perfect and, lo and behold, it seems to do what I want: once I turned that on, re-saving an XPage stopped causing the ClassCastException. I haven't given it a proper testing, so it might not be everything I'm looking for, but I'm thrilled at the initial results. Having to re-save a Java file for each change wasn't a HUGE problem, but it was a niggling one, and not having to jump through the hoop helps make the whole environment seem less rickety.