(Updated 2021-08-01)

The last week or so, I've been working on reading Notes design elements without my longtime unreliable friend DXL. This is a task I've done in parts a few times before, but now I've set out to finish the job.

For the most part, the task is arcane but explicable: the structures are generally documented well enough in the C API docs, and most non-string items are usually only a few elements long. Views - likely because they're as old as Notes itself - are not one of these easy parts.

Background

The $ViewFormat item in a view design note contains, well, the format of the view: the overall settings (like the background colors) as well as the formats for the individual columns. Because views have gotten more complicated over the years, this item has grown. And because these types of structures have to grow in a way that's generally compatible with older code that might read it, the format is essentially an array of progressively-newer structures that alternatingly describe new traits of the view and the columns.

Some of this is described in the C API documentation. In there, you'll find "readview.c", which shows a basic example of how to read a view. Unfortunately, this example stops at the VIEW_TABLE_FORMAT2 structure, which, as we'll see shortly, is not far in. From what I can gather by looking at the "FirstVer" values in the C API documentation NSF, this file demonstrates only what existed before Notes R4.

So that example is a bust, but fortunately "viewfmt.h" has some more information. Near the top of that file is an overview of the $ViewFormat item structure in general, and it's more helpful. It's not fully helpful, though: it stops with things added in R6, and doesn't cover parts added in 6.5 and beyond. Additionally, it doesn't explain the storage of the R6-era variable data - the hide-when formulas and twistie image references - and those are a bit weird.

The mitigating factor for the documentation is that, while it doesn't explain the order of the remaining data, it does at least include definitions for the post-R6 structures, such as VIEW_COLUMN_FORMAT5 and whatnot. That is... it mostly includes that: they're fully present in "viewfmt.h", but some are missing in the NSF doc DB, and the NSF lacks some important flags. Still, it's something.

My final task remained: I had a few hundred bytes left and, armed just with some available structures and Karsten Lehmann's earlier attempt, I had to figure out how views work. And I think I did it! I haven't tested all edge cases, and in particular this will be different for calendar-type views, but this holds together. I'll update this as I find more hiccups.

Implementation Notes

I've been accessing this data as read by using the pointer returned by OSLockObject on an item value's BLOCKID. I'm also running on an amd64 Mac. Those two aspects may affect the need for ODSReadMemory - things may be different if you're on a different architecture, or perhaps the way I read it already brought it into host format. In my use, I found that ODSReadMemory was harmful in all cases, and that it was best to read the memory directly. Additionally, this is a view created using Designer V12 and read using the V12 libnotes.dylib, so anything newer won't be reflected here.

The rules around ODSReadMemory elude me at the moment. The way I'm reading this data involves the value BLOCKID as fetched by NSFItemQueryEx, which indicates that the value will be in canonical format for types including this. In theory, that should mean that ODSReadMemory is required for these structures, but it ends up harmful.

The Structure

To start out with, we have the VIEW_FORMAT_HEADER structure, which defines the format version (always 1) and whether it's a normal "table"-type view or a calendar view.

This header structure is, in table views, actually part of the VIEW_TABLE_FORMAT structure that is the true beginning (and is also where things will diverge for calendar views). This one's pretty straightforward, with no variable data, and contains the critical tidbit of the Columns value for the total number of columns, a number that we will reference again and again. Once read, advance your pointer sizeof(VIEW_TABLE_FORMAT) bytes.

Following this will be one VIEW_COLUMN_FORMAT structure for each column based on the count in the format. For each, read the structure and advance your pointer sizeof(VIEW_COLUMN_FORMAT) bytes for each. The first WORD of each of these will be equal to VIEW_COLUMN_FORMAT_SIGNATURE.

Next up is the variable data for those columns: the item name (string), the title (string), the formula (compiled formula data), and the constant value data (documented as reserved). These should be read by iterating over the above-read VIEW_COLUMN_FORMAT structures and reading byte arrays with length based on ItemNameSize, TitleSize, FormulaSize, and ConstantValueSize.

Next is VIEW_TABLE_FORMAT2, which is the last bit read by the "readview.c" example. Read sizeof(VIEW_TABLE_FORMAT2) bytes and increment your pointer.

(At this point in time, you might be inclined to ask "hey, how come VIEW_COLUMN_FORMAT has a handy signature to identify it but VIEW_TABLE_FORMAT2 doesn't?". Well, to that I say that you sure are asking a lot of nosy questions, pal, and maybe you should mind your own business. Anyway, this will be a reocurring feature.)

This is followed by one VIEW_COLUMN_FORMAT2 structure for every column in the view. Though this includes variable-data sizes, we don't read them yet - just read sizeof(VIEW_COLUMN_FORMAT2) * n in total here. These will each have VIEW_COLUMN_FORMAT_SIGNATURE2 in the first WORD.

Next is VIEW_TABLE_FORMAT3, which is a blessedly fixed-size structure, so you can just read sizeof(VIEW_TABLE_FORMAT3) bytes and move on.

Now here's where we come to the variable data from VIEW_COLUMN_FORMAT2, and where things start getting weird. To read this data, iterate over each FORMAT2 structure for your columns and check the wHideWhenFormulaSize and wTwistieResourceSize properties. If the hide-when size is above zero, read the hide-when formula first as a compiled formula. Then, read the twistie resource as CDRESOURCE composite-data structure if its size is non-zero. Fortunately, the lengths for both of these are defined in VIEW_COLUMN_FORMAT2, so the total amount to read will be wHideWhenFormulaSize plus wTwistieResourceSize from that structure.

Next, we have VIEW_TABLE_FORMAT4, which is a simple little structure. Of note here is that the RepeatType property - which I gather is how to repeat the background image of the view - references "viewprop.h", but that is not in the documentation. Oh well. Pleasantly, though, this structure includes a Length property as its first WORD, so read that many bytes (even though it should always be 8).

After this, our friend CDRESOURCE shows up again, this time as a single entity to represent the background image of the view. Since there's nothing that tells you the length of this before hand, you should glean the length from the CD record header, which is a WSIG - so the second WORD. Read that, then backtrack and read that many bytes.

Now, we have 0-to-n VIEW_COLUMN_FORMAT3 structures - one for each column that includes date formatting (from that "Style" dropdown on the fourth tab in Designer). To read these, look over your array of VIEW_COLUMN_FORMAT2 structures and, for each that has ExtDate in its Flags3 field, read one of these. Each column's structure is immediately followed by variable data for the different formatting strings, so read strings based on DTDsep1Len, DTDsep2Len, DTDsep3Len, and DTTsepLen from the structure, and then move on to the next applicable column.

These are followed by 0-to-n VIEW_COLUMN_FORMAT4 structures. These are the same idea as above, but for number formatting. Look for VIEW_COLUMN_FORMAT2 entries with NumberFormat in their Flags3 and read a VIEW_COLUMN_FORMAT4. Each of these is followed immediately by strings based on DecimalSymLength, MilliSepSymLength, NegativeSymLength, and CurrencySymLength, so read those too before moving to the next applicable column.

We've got one more of these in this sequence, and that's VIEW_COLUMN_FORMAT5: formatting for names columns. Iterate through your VIEW_COLUMN_FORMAT2s and look for NamesFormat in Flags3 and read these structures. Each one is followed immediately by the programmatic name of the column containing the distinguished person name for Sametime purposes, as defined by wDistNameColLen. Fortunately, this structure also includes a dwLength (which is a WORD despite the prefix) that contains the total length of the structure and its associated variable data, so you can read that many bytes in total here. It also, though, contains a nasty trick: you might be inclined to think that this structure also contains the name of the shared column used here, what on account of wSharedColumnAliasLen being in the structure and the documentation saying that's what it means. You'd be wrong, though - that value will be 0 and you'll get nothing from it.

This next part is a doozy. The fact that VIEW_COLUMN_FORMAT5 purports to contain the name of the shared column but in fact doesn't raises the question of where that name actually is. Well, it's right up next - you just need to read the string! But hrm: the length value in the previous structure was 0 and, besides, there won't even be one of those for non-names columns. Is it null-terminated? No: your next bytes will likely be something like 0x0700, and 0x07 is not a column name, that's for sure. What we have here is a P-string, prefixed by a WORD containing the number of characters. These strings represent both shared column titles and any titles for columns marked as "Do not display title in column header", with the latter first. So, loop through your VIEW_COLUMN_FORMAT2s again, looking for HideColumnTitle in Flags3, and, for each match, read a WORD length and then that many characters to glean the title. Then, do the same for columns with IsSharedColumn in Flags3 for your shared-column aliases.

The final structure in play that I know of is VIEW_COLUMN_FORMAT6, which fortunately is handled in basically the same way as formats 3-5. Look through VIEW_COLUMN_FORMAT2 again, this time checking Flags3 for ExtendedViewColFmt6 (I appreciate the functional name), and read a VIEW_COLUMN_FORMAT6 for each. This one again has some variable data immediately after each structure and also has a Length property, so read in that property's value in total bytes. This structure isn't in the doc NSF, but it is in "viewfmt.h", and it contains R8-era composite-app stuff.

Finally, it seems like there's a trailing 0. It makes me nervous that I end up with a single byte, but it seems to be consistent, and so my guess is that it's here to meet a WORD boundary, which is extremely common in the Notes API.

Incidentally, I don't know where VIEW_TABLE_FORMAT5 comes in, if at all. It's in the NSF and "viewfmt.h", but looks like just a duplicate of VIEW_TABLE_FORMAT4. It's possible that it sometimes shows up right after 4 when some specific thing is set, though it has no signature component and would be difficult to detect if so.

The Structure (Condensed)

If you came here to just get a quick overview of the format, I'm afraid I've pulled a "recipe blog post" on you and wrote it all in prose before the important part. In any event, here's the above but condensed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

VIEW_TABLE_FORMAT
VIEW_COLUMN_FORMAT * column count
(variable data for each column)     // Order: (item name, title, compiled formula, constant value), repeat per column
VIEW_TABLE_FORMAT2
VIEW_COLUMN_FORMAT2 * column count
VIEW_TABLE_FORMAT3
(hide-whens and twistie resources)  // Order: (compiled formula, CDRESOURCE), repeat per column where VIEW_COLUMN_FORMAT2 has wHideWhenFormulaSize > 0 or wTwistieResourceSize > 0
VIEW_TABLE_FORMAT4
CDRESOURCE                          // Represents the background image for the view
VIEW_COLUMN_FORMAT3 * x1            // One per column where Flags3 has ExtDate. Order: (VIEW_COLUMN_FORMAT3, var data), repeat per column
VIEW_COLUMN_FORMAT4 * x2            // One per column where Flags3 has NumberFormat. Order: (VIEW_COLUMN_FORMAT4, var data), repeat per column
VIEW_COLUMN_FORMAT5 * x3            // One per column where Flags3 has NamesFormat. Order: (VIEW_COLUMN_FORMAT5, var data), repeat per column
(hidden column titles)              // One per column where Flags3 has HideColumnTitle. Stored as WORD-prefixed P-strings
(shared column names)               // One per column where Flags3 has IsSharedColumn. Stored as WORD-prefixed P-strings
VIEW_COLUMN_FORMAT6 * x4            // One per column where Flags3 has ExtendedViewColFmt6. Order (VIEW_COLUMN_FORMAT6, var data), repeat per column

Conclusion

Hoo boy, this was a fun one. Anyway, assuming this holds up under further testing, my hope is that this post will be useful for the next person to come around to try to decode $ViewFormat. If that's you: good luck!

As is characteristic of his blog, Sven Hasselbach recently posted two interesting posts on using the NAPI classes in the XPages runtime to manipulate files in the WebContent folder. If you haven't read the posts, I suggest you do so now, because it's knowledge that is very good to have. The NAPI classes are chock full of cheating sorcery.

But the point of this post here is a bit of me-too-ism. Which is to say: the OpenNTF Domino API has you covered on this front. The API's Design package (org.openntf.domino.design.*) contains classes that, among other things, let you retrieve, create, and modify both "normal" and these extra file resources.

The starting point is the getDesign() method on an ODA database object. Using this DatabaseDesign object, you can get access to files. For example:

Database database = FrameworkUtils.getDatabase();

DatabaseDesign design = database.getDesign();

for(FileResource res : design.getFileResources()) {
	// do stuff
}

FileResource newResource = design.createFileResource();
newResource.setName("some file.txt");
newResource.setFileData("some content".getBytes());
newResource.save();

FileResource existing = design.getAnyFileResource("existing-file.txt");
String existingContent = new String(existing.getFileData());
existing.setFileData((existingContent + " new stuff").getBytes());
existing.save();

Created file resources show up in the File Resources section currently (you can call setHideFromDesignList(true), which removes them from there, but puts them at the root of the VFS... that works, but it'd be better for them to show up in WebContent (putting "WebContent/" in the name LOOKS like it works, but doesn't)). The getAnyFileResource method will search for any "file" type element with the given name - normal file resources, WebContent files, XPages, and Java source/class files. There are better classes for manipulating the latter two, but they show up because that's how they're implemented.

One note: the Design API uses DXL, not the NAPI classes, which means it gains freedom from an XSP dependency at the cost of memory efficiency. To do its thing, it must export the file as DXL, which will be larger in memory than the file size, and then convert that BASE64 representaiton to bytes, meaning the memory cost is more than double the size of the file. For most files, that's fine, but be wary of dealing with, say, 200MB video files. That sort of thing totally works (I tested), but only if you have the memory to spare and your JVM configured to use it. Maybe one day the API will have direct access to the notes.

Okay, a second note: it's important to make sure that the max Internet access level is Designer or above for this to work, at least with the normal session object.

If you haven't taken a look at the Design API before, it may be worth at least glancing over the DatabaseDesign interface to get a feel for what it currently allows. Maybe attention on it will coax me into finding free time to finish all the other aspects I plan to get to.