Well, this is a nice quality-of-life improvement: the plugin Eclipse Color Themes, which (so far) works just fine in Designer (8.5.3). Having to go through every single editor type and manually pick each color for each code element every time I wanted to change a color theme was always a huge annoyance, particularly compared to other editors. Fortunately, that plugin handles it pretty well, though it unsurprisingly doesn't support LotusScript, so that's still manual. I installed it without problem, found its settings in General\Appearance\Color Themes, installed a port of my preferred theme, and saved myself tons of hassle.

From my perspective, there are three main problems with Domino: the limits, the client, and the server. Now, that's a lot of stuff... most of the product, in fact. However, the facts that I'm still programming for it and that my company's 16GB project-tracking database is as snappy as it was when it was empty attest to the core quality of the product. Off the top of my head, I can think of a number of things that make Domino salvageable:

• Reader fields. These are hard to beat and hard to find elsewhere. While you can do everything without them that you can do with them, you'd likely spend a lot of time worrying about edge cases or limiting yourself to coarse security. If a user doesn't have Reader access to a document, it doesn't exist (well, except for some shadows in view indexes).
• Solid, straightforward directory integration. Going along with reader fields, it's very helpful that the directory system is part of the overall product and works well enough with LDAP that you don't have to worry too much about it. User names, group memberships, and roles all make sense.
• Full-text searching. It works and it's quick.
• View indexing, more or less. As long as you don't rock the boat too much and don't think about what you used to do with SQL tables and views, Domino does a fine job of maintaining views for you.
• File attachments. They work well and can even be full-text searched if they're the right type.
• ID files. I used to hate these, and they're often more hassle than they're worth (as my C API programming is demonstrating), but they have some distinct advantages. If you squint, you can see them as really long password that may themselves have another level of password. They're also essentially the same concept as SSH keys. They also tie into handy encryption and certificate abilities that I don't usually have much use for.
• Potential capability as a blob store. I saw a great post a while ago talking about the ability to just cram arbitrary data into a Domino document, and I think this is a very valuable line of thinking. If your data fits into the model of "a couple queryable bits and then a block of arbitrary data" or the "upside-down" method of memory-storage-first from the post, such a setup could work extraordinarily well.
• Replication and clustering. Sync is not hard... unless you let your deletion stubs expire.
• Agents, more or less. The language choices and lack of immediacy make agents kind of a pain sometimes, but it's still handy to have code directly associated with the database with schedule and event triggers.
• Read marks. It's nice to have the option to let the server handle this for you when it fits your needs.

That's on top of the various benefits you get from document databases generally, like multi-value fields and the lack of schemas. Note, though, that none of these are (directly) related to Domino's chops as a mail platform, web server, or GUI app environment, its choice in programming languages, the value of its standard array of templates (though they can be handy), or pretty much any virtue extolled by its marketing materials. The guy writing on the invisible whiteboard on IBM's page doesn't care about reader fields or blob storage.

Though most of my Domino work is for the web and XPages are a better way to do that than the legacy elements, it's still a drag. Java as a language is a hassle (the platform has its charms), Server JavaScript isn't a full replacement in the way that Groovy and JRuby can be, and working on top of a giant Jenga tower of Java classes and abstractions can be hazardous to your health.

I really just want to treat Domino like MongoDB, CouchDB/Couchbase, and the rest: an environment-independent document database. This is probably not worth the effort, particularly since IBM seems passively hostile to the notion, but I find it to be a compelling idea. It doesn't have to fit into, say, Rails, but grinding Domino down to its solid NoSQL core could open up a lot of possibilities.

There are a couple things about writing my Ruby wrapper for the C API that make it particularly fun, mostly related to getting to add abilities that I desperately wish were there in the normal APIs.

1. Ruby-style (forall) looping. Anyone who has iterated over a NotesDocumentCollection knows the drill: set a variable to the first element, start a while loop, and make sure to set the variable to the next one at the end. Writing it one time isn't so bad. Writing it hundreds of times, though? It gets to be a drag. Getting to write docs.each { |doc| ... } is a breath of fresh air.
2. Easier design-elements-as-notes access. The normal API lets you get a NotesDocument version of a NotesView via its UniversalID property, but for everything else you need a NotesNoteCollection, which is a hassle. Since all design elements are Documents anyway, I've just made their wrapper objects subclasses of Document (though I may change that to just a #document method if it gets hairy) and I've put a #get_design_note method on Database that lets you find a design note by name and flag class (NIFFindDesignNoteExt).
3. HTML and DXL everywhere. I use DXL fairly constantly (mostly for design elements), and for the most part it's a hassle. Not only do you have to create a NotesDXLExporter, but you also have to get the Document version of the design note you're dealing with and run through its process. Not impossible, but sort of a hassle. Now, though, I just put a #to_dxl method on everything - this is like the .generateXML() method in Java, but consistently applied. Similarly, I can't count the number of times when it would have been handy to get an HTML representation of some element, even if it was just the dated stuff that the legacy renderer puts out. Since that's all there in the C API, I just put a #to_html method on everything and a #get_item_html method on Document for very easy access to web-friendly versions of MIME and Rich Text items.

It's just a shame that I probably won't be able to use this: what I'd really want to do would be to use it like a database driver on a Ruby-driven web site, but the fact that it's so tied to ID files makes that tough. Still, it's great exercise to write it, and maybe I'll cave and set up some sort of multi-process hydra beast to suit my needs.

As I had mentioned before, I've been tinkering about with the Domino C API, specifically with Ruby. Although I'm not sure I'll actually have a use for it (from what I can tell, the C API is very tied to ID files and threads, which would make a multi-user web server thing cumbersome), I've decided to go for it and write a wrapper for the API generally based on the Java/LotusScript API. This is serving a number of purposes:

1. It gets me off my (metaphorical) duff and in front of a text editor during my off hours.
2. It lets me work with Ruby in a real capacity. And moreover, it'll expose me to how to write a Ruby library.
3. It's introducing me to proper version control, something I absolutely need to do better.
4. It will eventually force me to write structured documentation, which I haven't had to do in the past.
5. It should fill a hole currently only partially filled with OLE-based libraries, though the ID-file thing will make it awkward.
6. It brings me back to the world of C structures and pointers, albeit cushioned by FFI.
7. By the time I'm done, I'll know Domino inside and out.

So far, I have some of the basics working: creating a "Session" by passing in the program directory and the path to an INI file, fetching databases, traversing and full-text-searching views, reading view entries, and reading basic and simple MIME data from documents. Obviously, there's plenty more work to do, and I don't know if I'll bother implementing the more esoteric classes like NotesRegistration, but it's off to a fun start so far:

https://github.com/jesse-gallagher/Domino-API-for-Ruby

Point 7 on that list has proven surprisingly interesting. It's fun to see the bits that apparently never fully made it onto the product (like number ranges), the little optimized paths where the documentation goes out of its way to point out that it's particularly efficient, and all the data structure sizes that create the various limitations that make Domino programming such an... experience.

Since last week, I've made two nice improvements to my dynamic view customizer:

1. I added some support for twistie images when the referenced DB is on the same server. The code assumes that the referenced images are image wells with at least two entries, but I can't imagine why that wouldn't be the case in practice.
2. I vastly improved my handling of color columns. Previously, I had been resorting to hacky methods like hidden <div>s read by JavaScript or surrounding <div>s styled to take up the whole cell, but those were terrible and easy to break. Now, though, I'm doing it right: the code adds a value binding for the column's style attribute to create the CSS for each cell, which is ideal.
3. Empty categories now are translated to "(Not Categorized)", as in the client.

Now that my code is presentable (albeit oddly structured and uncommented), I figured I may as well toss it up on GitHub:

https://github.com/jesse-gallagher/Domino-One-Offs/blob/master/mcl/reports/DynamicViewCustomizer.java

To note if you want to use this in your own project: it references the "mcl.JSFUtil" object, which started as the mindoo object of the same name and has since turned into my bin for common functions. The methods used here are getSession(), which just gets the value of the JSF "session" variable, and the xmlEncode() and specialTextDecode() functions from my string utils. Additionally, it references "com.raidomatic.xml.*", which are quick wrapper classes I made to ease basic XML access, and which are also available on GitHub:

https://github.com/jesse-gallagher/Domino-One-Offs/tree/master/com/raidomatic/xml

I think I have my view rendering problem licked. To recap, I've been working on a way to show views in XPages that met a couple requirements:

1. Entirely dynamic. Since this will be for a combined reporting site that will show views from customized project databases of wildly varying needs, I couldn't make any assumptions about view layout, categorization, or content. It should pull as much display information from the view design as possible.
2. Fast. Some of these views have thousands of rows, so it should be as fast as possible to load from the database and render for the browser.
3. Pagination. Though I don't really like pagination for the average case (who wants to look at a 40-row view 30 rows at a time?), sending thousands of table rows to a browser causes a miserable experience even when that browser isn't IE.
4. Full-text searching and other Domino goodies. I don't want it to be a pain to just be able to search through a view like you would in the Notes client.
5. Support for "second tier" Domino view features. I use special text, column hide-when formulas, "show values in this column as links", Notes-style [<span>pass-through-HTML<span>], and color columns constantly.

Of these, 1 and 5 are the most important.

I originally wrote my own code to generate a big HTML blob for the view, which was perfect on points 1 and 5 (and looked great, since I converted categories to OneUI sections), but wasn't as hot on the other points. The Extension Library's xe:dynamicViewPanel control hits points 1 through 4 with aplomb, but misses the crucial point 5. For one crazy moment, I toyed with the idea of trying to get the output of the "legacy" HTML view renderer (which is perfect on all points) via client-side JavaScript or some other hack, but decided against it as being too rickety.

I considered changing my custom HTML code to instead create a bunch of Java objects to make pagination easier, but performance would be an issue - it's tough to beat built-in controls for raw speed. As anyone who has looked at the code for the mail template knows, IBM/Lotus cheats like crazy, and often the only thing you can do is to just go with what they've done and beat it into shape. Thus, the answer came to me: customizer beans for the ExtLib's dynamic view panel.

The dynamic view panel has a "customizerBean" property that takes a string class name of a bean to create to hook into a couple overridable methods and events:

• ViewFactory getViewFactory() - this lets you override how the panel pulls in all of its information about the view.
• boolean createColumns(FacesContext context, UIDynamicViewPanel panel, ViewFactory f) - this lets you override how the panel uses the gathered view information to generate the list of data table columns (or simply run code before that happens, if it returns false)
• IControl createColumn(FacesContext context, UIDynamicViewPanel panel, int index, ColumnDef colDef) - similarly, this lets you change the way each individual column is created based on the ColumnDef (an object generated by your ViewFactory)
• afterCreateColumn(FacesContext context, int index, ColumnDef colDef, IControl column) - this is an event fired after each column is generated, allowing you to customize the generated column without having to build it from scratch
• afterCreateColumns(FacesContext context, UIDynamicPanel panel) - similarly, this lets you run code after the column creation is done

These events provided just the hooks I needed to get the dynamic panel to do what I wanted.

Special Text

To add special text support, I added an implementation of afterCreateColumn(...) that replaced the column's default ViewColumnConverter with my own subclass that overrides getValueAsString(...). With that, I can walk up the current component's ancestors to find the panel, find the individual ViewEntry's variable name from there, and use that to process the special text properly.

Column Hide-When Formulas

These are a bit trickier. The getViewFactory() hook lets me provide my own subclass of DefaultViewFactory, but there's a reason that the dynamic view panel doesn't support these formulas by default: they're not exposed by the NotesViewColumn class. The two ways I can think of to get at this information are the C API (or otherwise reading the design information from the view note directly) and DXL. The former is no doubt significantly faster, but DXL is much, much easier and is good enough for my needs at the moment. So I export the view as DXL and process its column nodes and their children. Then, for each with a hide-when formula, I evaluate the formula in the context of a new document in the project's database and use the result to set the column-hidden flag.

"Show values in this column as links"

This goes hand-in-hand with the hide-when formulas. The column nodes in the DXL have a showaslinks attribute that I can look for to set this flag.

Pass-through-HTML

Much like special text, once I have the custom converter attached to the column, I can break apart and reassemble text values on [< and >] and XML-encode the non-HTML bits. While I'm attaching the converter, I set the column's content type to "html", since I'll be handling the encoding.

Color Columns

These are... tricky, and I'm not yet proud of my fix. There are two problems: getting each cell to know about its color information from previous cells and then actually applying that color information. For the former problem, I dealt with it by adding an "activeColorColumn" property to the ColumnDef object and setting it to the programmatic name of the most recent color column before each column. The latter is the ugly part. You can't just set the column's style, since that will apply to the entire column and make a fine mess of everything. What you really want to do is assign it to the current <td> element, but I don't know a good way to do that. In the mean time, I've hacked around it by making a hidden <div> in each cell with the style tucked into a data-cell-style attribute. Then, I have some client JavaScript look for these divs and apply their styling to the parent table cell. It's not pretty, but it works for now.

If I get that last bit of code cleaned up to a point where I'm comfortable letting people see it, I'll post my example code. In the mean time, I'll count this whole thing as a win for only writing the code you need to and letting the platform take care of the rest.

My current work project involves displaying arbitrary views from various databases into a combined reporting site (written in XPages). This has presented me with two hurdles: pulling in the data completely and accurately and then figuring out a good way to format it.

The former problem is one of the few areas where it seems like "classic" Domino development has an edge: views render rapidly and, as long as you've configured the server to display lots of rows or add in pager controls, completely. And with the recent "enhanced HTML" property, they generally get tagged with some useful CSS classes. The default XPages xp:viewPanel control, by contrast, needs the column definitions at design time, which makes it tough to work with. You can sort of roll your own with nested xp:repeat controls, but you quickly run into problems with categories and other Notes-y features. Fortunately, the Extension Library, as usual, comes to the rescue: the  xe:dynamicViewPanel does pretty much what you want: you give it a view and it renders it out nicely. Using it, you don't have to worry about handling categories, you can add in standard pagers, and you can include all the standard filtering and searching options on the view. It's not perfect (yet), though: it doesn't handle fixed-value columns (like a formula of "hi"), use color-column information, follow column hide-when formulas (though it supports the static "hide this column" checkbox), do Notes-style bracketed pass-through-HTML, process special text, or honor "show values in this column as links." These are all understandable limitations, being as they are either rarely encountered or technically difficult. Unfortunately, I've run into all of them rapidly, so I don't know if I can use xe:dynamicViewPanel as-is. The best thing to do would be for me to roll up my sleeves, implement these features in it, and submit my changes, and maybe I'll do that down the line, but there'd be a lot of learning overhead before that point. What I've taken to in the mean time is a Java class to write out the view contents as HTML. I add the color-column info via inline style="..." attributes, look up the hide-whens by exporting the DXL and then running them through session.evaluate (read: not fast), and split and recombine the strings to handle pass-through-HTML. Unfortunately, I lose speed (because I'm presumably not as good at this as IBM's programmers) and the flexibility to easily use searching, filtering, and paging. The first two will be relatively easy to implement, but paging will be a problem, since the XPage just gets back a big string blob. I could return Java data structures, but then I'd run into problems with categories. I could write classes to dynamically add XSP elements to the page, but at that point I'd be best off cracking open the ExtLib source code anyway.

The other problem is that, regardless of how I render the view, it can be tough to actually display all the data well on a web page. For normal views, with a handful of columns, it's fine to just plop it right on the page and be done with it. However, it doesn't take much to make the table look like crap: headers that are way too long, large strings in the row data, or just having lots of rows all very quickly stretch or break the design. I think I'll try to handle the headers by adding some CSS text-overflow rules to keep them to a fixed width, which will also mean adding in a title="" attribute so the user can hover over them to see the full value. Still, not a big problem. Large data cell values are a bit tougher. Theoretically, I could do the same thing: have the data overflow and add a tooltip. That would work fine if it's a rare occurrence, but what if the whole point of the view is to be able to easily look at that data. I may try to pull out the row-height property from the view and use that to provide a max-height to each cell via inline CSS, so that way I could solve the problem on a view-by-view basis. I could consider doing the same with width as well, rather than going the default route of letting the content width determine the column width. Having a lot of rows can be partially solved with pagination (unless I'm going the "dump out a blob of HTML" route), but is that really a good user experience if the goal is to see all of the data? Paging on the web is pretty awkward, with the number of rows never exactly matching the height of the viewport, worrying about scrolling back to the top when going to the next page, and properly dealing with categories. Via CSS, I've made it so that the primary content area has its own scrollbars via absolute positioning and overflow: auto and that works pretty well for the most part, but I still run into trouble with views containing thousands of rows: with a lot of data, it simply takes a long time to download the HTML and render the table. There are a couple options provided by Dojo and the Extension Library that provide fixed-size grids to store data that I've tinkered around a bit, and maybe I'll settle on one of them, but in the mean time every solution has some serious problem that I quickly run into with my example data.

My search for a useful way to use Domino as a database back-end for a Ruby front-end has continued and, more specifically, has continued to be difficult.

I gave a shot to the Java CORBA API. Initially, that went well: after grabbing NCSO.jar from my Designer installation, enabling DIIOP on the server, and setting up JRuby, I was able to connect to the server using the host name, canonical user name, and password. Great! However, once I exported my app's Java model classes (modified to use NotesFactory instead of the JSF runtime), I immediately ran into a problem: NotesView.resortView() is not implemented in CORBA. Guh. Theoretically, I could rework the app to use a different view for every index, but it's a canary in the coal mine: no doubt plenty of other new things (FTSearchSorted, for example) have been left unimplemented as well. Plus, this is Domino, so "new" probably refers to anything added after, say, R5.

So that's out, at least for now. That leaves me with the C API. Oh, the C API... so much promise, such a PITA to use. I haven't used C itself much since college, and my experience with the C API is limited to progress bars in the Notes client and, more usefully, my authentication filter for my server. However, I'll be programming in Ruby, not C directly, so it won't be so bad, right?

There are three main ways to work with C libraries in Ruby, I've learned: writing a Ruby module, DL, and FFI. In the first case, I'd basically have to write my API in C and then just use those objects from Ruby. That would be the best performance, but it's just asking for a nightmare. DL is programmed in pure Ruby, but it's so sketchy that just loading the library causes your program to emit a warning telling you you probably shouldn't be using it. FFI, though, is pretty great. Much like DL (or declaring external functions in LotusScript), you declare which functions you want to import, along with their signatures, and then it more or less just works.

That "more or less" is, naturally, tricky. I ran into tremendous trouble just getting it to load up libnotes.so, but most of that was due to using a 64-bit OS and a 32-bit Domino installation. I may make a post about it some time in case anyone else is crazy enough to try the same thing, but the upshot is that I ended up compiling my own 32-bit Ruby environment, which started to work.

Since then, I've been wrestling with the inherent traits and idioms of C (using string buffer arguments to return values instead of just returning a "string", for example) and the specifics of Notes (needing to be pointed to the executable directory as well as needing an ID file, an INI file, and a names.nsf to find the servers). After thrashing around for a while, though, I have it working to the point where I can point the initialization routine to an arbitrary INI file and it can pick up on everything it needs from there (I'm using a password-less ID file - I don't know if you can automate passwords).

Now that I have my foot in the door, it's "just" a matter of learning the entire API. I'm looking forward to that part, though - I've demonstrated that I can open a database on a server and get information out of it, so it seems like I have the right environment. If all goes well, I should be able to create a proper Ruby API for Domino, and a very fast one at that.

For reference, here's the script I've written. Don't expect clean code, reusability, comments, or proper methodology - this file is the result of random mauling during the course of development and just barely works to demonstrate the idea. It DOES demonstrate it, though, and works to ping a (hard-coded) server and get some info about its names database.

test.rb

As most developers probably do, I have a grab bag of "utility" functions/methods I use across various projects, and I figured I'd post a few of the handier ones.

The first is a basic XML-encoding function that just takes a string and returns something suitable for putting into an XML or HTML file. I've been too lazy to figure out if the standard Java library has an equivalent that doesn't involve creating an actual XML document in memory, so I just took the one I use for LotusScript and ported it over. It passes standard letter and number characters through as-is and then just encodes all others as Unicode entities. The string size increases accordingly, but it's presumably fast and it gets the job done, even if you have oddball characters.

The next set are just quick-and-dirty Java equivalents to the StrLeft, StrRight, etc. functions from LotusScript. They do more or less the same thing as in LS and make it so you don't have to worry about string indexes all the time.

The last is potentially the most useful, since it's the most esoteric: a special-text decoder. I've had a couple occasions where I want to spit out the contents of a view for the web, but also want to preserve things like child counts on categories. Fortunately, special text is stored in the view column in a workable format: a special non-alphanumeric character, a single letter indicating which function it corresponds to, a number representing the parameter count for the function, and then a delimited list of those parameters with character counts. Using that, I wrote a routine to process all of the special text functions I could think of other than @IsExpandable (since that's meant for the Notes UI). It even reproduces the weirdo behavior you get when you pass a multi-character string to @DocLevel.

StringUtils.java

In the interests of getting crap done, as soon as I was finished with my previous post, I fired up TextMate and a couple Terminal windows to start writing a Ruby wrapper for the DAS API.

It started out great! Wanting to avoid the minor hassle I ran into before with Ruby's built-in Net::HTTP library, I did a quick search for Ruby HTTP/REST libraries and picked one that worked well, named HTTParty. Before long, I had the rudimentary elements of a Notes API working, with classes to represent the server connection, NotesDatabase, and NotesView.

I was about to start working on the NotesViewEntry class when I decided I'll need a way to page through the view, since the call to the view "collection" (wisely) doesn't return all of the data at once. The API developers cleverly store this information in an HTTP header:

Content-Range: items 0-9/35

Perfect! But, um... I was getting the list of available forums as Anonymous, which can only see two, not all 35. Sure enough, the results only contained the two Reader-visible entries (phew), but that left a showstopping problem: I have fairly frequent need for the count of entries, and I don't relish the notion of having to fetch all of the data to get that. Crap.

So DAS is out for now. I remembered, though, about ?ReadViewEntries, an older and thus clearly more mature data-access API. I could probably do everything view-related I need with that and then use DAS to get any non-view document data. So I hit the view with ?ReadViewEntries&OutputFormat=JSON (the results are the same with the XML version) and, lo and behold, the top-level object fields (with data snipped) revealed the same lack of Reader knowledge:

{ "@timestamp": "20120123T034916,85Z", "@toplevelentries": "35" }

Crap!

My quest to treat Domino like a standalone database is off to a rocky start. I can think of a few remaining options:

1. COM: I don't deploy on Windows, so... that's out.
2. C API: seems like overkill and just asking for all sorts of new bugs.
3. Web Services: MIGHT work, but I can't imagine the performance would be anywhere near acceptable.
4. Some sort of crazy thing with servlets: seems too crazy.
5. JRuby and the Domino Java CORBA API.

I think the last is the most plausible. My forum app already has Java classes in between the XPages front-end and the Domino database, so I could possibly take those wholesale and just change the methods I use to get the session and current database to instead initialize a CORBA session. I'm a bit nervous about speed and running across some of the various "not implemented in CORBA" landmines hidden across the API, but it might just work. If it does, my next task will be deciding on a web server and a Ruby web framework that's not terribly SQL-centric.