Yesterday, I talked about how I dove into Testcontainers for my app-testing needs. Today, I decided to use this to close another bit of long-open business: automated browser testing. I've been very much a dilettante when it comes to that, but we have a handful of browser-ish tests just to make sure the login page, the main page, and some utility pages load up and include expected content, and those can serve as a foundation for much more.

Background

In general, when you think "automated browser testing", that means Selenium. As a toolkit, Selenium has hooks for the browsers you want and has essentially universal support, working smoothly in Java with JUnit. However, the actual act of loading a real browser is miserable, mostly on account of needing you to install the browser and point to it programmatically, which is doable but is another potential system-specific configuration that I'd much, much rather avoid in my automated builds.

Accordingly, and because my needs have been simple, I've used HtmlUnit, which is a portable Java browser-like library that does the yeoman's work of letting you perform basic Selenium tests without having to configure actual native OS installations. It's neat, imposes basically no strictures on your workflow, and I recommend it for lots of uses. Still, it's not the same as real browsers, and I had to do things like disable JavaScript processing to avoid it tripping up on some funky JS that full-grown browsers can deal with.

Enter Webdriver Containers

So, now that I had Testcontainers configured to run the web app, my eye turned to Webdriver Containers, an ancillary capability of Testcontainers that lets you run these full-fledged browsers via their Docker images, and even has cool abilities like letting you record the screen interactions over VNC. Portability and full production representation? Sign me up.

The initial setup was pretty easy, just adding some dependencies for the Selenium remote driver (replacing my HtmlUnit driver) and the Testcontainers Selenium module:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

<dependency>
    <groupId>org.seleniumhq.selenium</groupId>
    <artifactId>selenium-remote-driver</artifactId>
    <version>3.141.59</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>selenium</artifactId>
    <version>1.15.3</version>
    <scope>test</scope>
</dependency>

Programmatic Container Setup

After that, my next task was to configure the containers. I'll skip over some of my troubleshooting and just describe where I ended up. Basically, since both the webapp and browsers are in Docker containers, I had to coordinate how they communicate with each other. There seem to be a few ways to do this, but the route I went was to build a Docker network in my container orchestration class, bind all of the containers to it, and then reference the app via a network alias.

With that addition and some containers for Chrome and Firefox, the class looks more like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

public enum AppTestContainers {
    instance;
    
    public final Network network = Network.builder()
        .driver("bridge") //$NON-NLS-1$
        .build();
    public final GenericContainer<?> webapp;
    public final BrowserWebDriverContainer<?> chrome;
    public final BrowserWebDriverContainer<?> firefox;
    
    @SuppressWarnings("resource")
    private AppTestContainers() {
        webapp = new GenericContainer<>(DockerImageName.parse("client-webapp-test:1.0.0-SNAPSHOT")) //$NON-NLS-1$
                .withExposedPorts(8080)
                .withNetwork(network)
                .withNetworkAliases("client-webapp-test"); //$NON-NLS-1$
        
        chrome = new BrowserWebDriverContainer<>()
            .withCapabilities(new ChromeOptions())
            .withNetwork(network);
        firefox = new BrowserWebDriverContainer<>()
            .withCapabilities(new FirefoxOptions())
            .withNetwork(network);

        webapp.start();
        chrome.start();
        firefox.start();
        
        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            webapp.close();
            chrome.close();
            firefox.close();
            network.close();
        }));
    }
}

Now that they're all on the same Docker network, the browser containers are able to refer to the webapp like "http://client-webapp-test:8080".

Adding Parameterized Tests

The handful of UI tests I'd set up previously had lines like WebDriver driver = new HtmlUnitDriver(BrowserVersion.FIREFOX, true) to create their WebDriver instance, but now I want to run the tests with both real Firefox and real Chrome. Since I want to test that the app works consistently, I'll want the same tests across browsers - and that's a call for parameterized tests in JUnit.

The way parameterized tests work in JUnit is that you declare a test as being parameterized, and then feed it your parameters via one of a number of mechanisms - "all values of an enum", "this array of strings", and a handful of others. The one to use here is to make a class implementing ArgumentsProvider and configure that:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

import java.util.stream.Stream;

import org.junit.jupiter.api.extension.ExtensionContext;
import org.junit.jupiter.params.provider.Arguments;
import org.junit.jupiter.params.provider.ArgumentsProvider;
import org.testcontainers.containers.BrowserWebDriverContainer;

public class BrowserArgumentsProvider implements ArgumentsProvider {
    @Override
    public Stream<? extends Arguments> provideArguments(ExtensionContext context) throws Exception {
        return Stream.of(
            AppTestContainers.instance.chrome,
            AppTestContainers.instance.firefox
        )
        .map(BrowserWebDriverContainer::getWebDriver)
        .map(Arguments::of);
    }
}

This class will take my configured browser containers, get the WebDriver instance for each, and provide that as parameters to a test method. In turn, the test method looks like this:

1
2
3
4
5
6
7
8

@ParameterizedTest
@ArgumentsSource(BrowserArgumentsProvider.class)
public void testDefaultLoginPage(WebDriver driver) {
    driver.get(getContainerRootUrl());
    assertEquals("Expected App Title", driver.getTitle());

    // Other tests follow
}

Now, JUnit will run the test twice, once for each browser, and I can add any other configurations I want smoothly.

Minor Gotcha: Container vs. Non-Container URLs

Though some of my tests were using Selenium already, most of them just use the JAX-RS REST client from the testing JVM directly, which is not containerized in this setup. That meant that I had to start worrying about the distinction between the URLs - the containers can't access "localhost:(some random port)", while the JUnit JVM can't access "client-webapp-test:8080".

For the most part, that's not too tough: I added some more utility methods named to suit and changed the UI tests to use those. However, there was one tricky bit: one of the UI tests uses Selenium to fetch the page and process the HTML, but then uses the JAX-RS client to make sure that a bunch of references on the page resolve to non-404 resources properly. Stuff like this:

1
2
3
4
5

driver.findElements(By.xpath("//link[@rel='stylesheet']"))
    .stream()
    .map(link -> link.getAttribute("href"))
    .map(href -> rootUri.resolve(href))
    .forEach(uri -> checkUrlWorks(uri, jaxRsClient));

(It's highly likely that there's a better way to do this in Selenium, but hey, it's still a useful example.)

The trouble with the above was that the URLs coming out of Selenium included the full container URL, not the host-accessible one.

Fortunately, that's not too tricky - it's really just string substitution, since the host and container URLs are known at runtime and won't conflict with anything. So I added a "decontainerize" method and run my URLs through it in the stream:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

public URI decontainerize(URI uri) {
    String url = uri.toString();
    if(url.startsWith(getContainerRootUrl())) {
        return URI.create(getRootUrl() + url.substring(getContainerRootUrl().length()));
    } else {
        return uri;
    }
}

// later

driver.findElements(By.xpath("//link[@rel='stylesheet']"))
    .stream()
    .map(link -> link.getAttribute("href"))
    .map(href -> rootUri.resolve(href))
    .map(this::decontainerize)
    .forEach(uri -> checkUrlWorks(uri, jaxRsClient));

With that, all the results came back green again.

Overall, this was a little fiddly, but mostly in a way that helped me learn a little bit more about how this sort of thing works, and now I'm prepped to do real, portable full test suites. Neat!

(Fair warning: this post is not about testing, say, a normal XPages app via Testcontainers. One could get there on this path, but this has a lot of prerequisites that are almost specific to me alone.)

For a while now, I've seen the Testcontainers project hanging around in my periphery. The idea of the project is that it uses Docker to allow you to programmatically load services needed by your automated test suites, rather than having to have the servers running separately. This is a clean match for something like a WAR-based Java webapp that uses, say, Postgres as its backend database: with this, you can spin up a Postgres image from the public repository, fill it with test data, run the suite, and tear it down cleanly.

However, this is generally not a proper match for Domino. Since the code you're testing almost always directly uses Domino API calls (from Notes.jar or another source) and that means having a local Notes runtime initialized in the test code, it's no help to have a separate container somewhere. So, instead, I've been left watching from afar, seeing all the kids having fun in a playground I never got to go to.

The Change

This situation has shifted a bit for my needs, though, thanks to secondary effects of changes I've made in one of my client projects. This is the one where I do all the bells and whistles of my tinkering over the years: XPages outside Domino, building a bunch of NSFs with Jenkins, and so forth.

For a while, I had been building test suites run using tycho-surefure-plugin, but somewhat recently moved the project to maven-bundle-plugin to reap the benefits of that. One drawback, though, was that the test suites became much more difficult to run, in large part due to the restrictions on environment propagation in macOS.

Initially, I just let them wither, but eventually I started to rebuild the test suites. The app had REST services for a while, but they've grown in prominence since we've started gradually replacing XPages-based components with Angular apps. And REST services, fortunately, are best tested at a remove.

First Pass: liberty-maven-plugin

The first way I started writing test suites for the REST services was by using liberty-maven-plugin, which is a general Swiss army knife for working with Liberty during Maven builds, but has particular support for starting a server before tests and terminating it after them. So I set up a config that boots up a Liberty server that can then initialize using a configured Notes runtime, and I started writing tests against it using the Jakarta REST client API and a bit of HtmlUnit.

To its credit, this setup did its job swimmingly. It still has the down side that you have to balance teacups to get a Notes or Domino runtime configured, but, once you do, it'll work nicely.

Next Pass: Testcontainers

Still, it'd be all the better to avoid the need to have a local Notes or Domino setup to run these tests. There's still going to be some weirdness due to things like having to have the non-public Domino Docker image pre-loaded and having an ID file and notes.ini somewhere, but that can be overcome. Plus, I've already overcome those for the CI servers I have set up with each build: I have some dev IDs in the repository and, for each build, Jenkins constructs a Docker image housing the webapp and starts a container using a technique similar to what I described a few months back to run a Liberty app with Domino stuff brought in for support.

So I decided to try adapting that to work with Testcontainers. Instead of my Maven config constructing and launching a Liberty server, I would instead build a Docker image that would then be loaded in Java with the Testcontainers library. In the case of the CI server scripts, I used Bash to copy files into a scratch directory to avoid having to include the whole repo in the Docker build context (prohibitive on the Mac particularly), and so I sought to mirror that in Maven as well.

Building the App Image in Maven

To accomplish this goal, I used maven-resources-plugin to copy the app and support files to a scratch directory, and then com.spotify:dockerfile-maven-plugin to build the Docker image:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71

<!-- snip -->
    <!-- Copy Docker support resources into scratch space -->
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-resources-plugin</artifactId>
        <version>3.2.0</version>
        <executions>
            <execution>
                <?m2e ignore?>
                <id>prepare-docker-scratch</id>
                <goals>
                    <goal>copy-resources</goal>
                </goals>
                <phase>pre-integration-test</phase>
                <configuration>
                    <outputDirectory>${project.build.directory}/dockerscratch</outputDirectory>
                    <resources>
                        <!-- Dockerfile to build -->
                        <resource>
                            <directory>${project.basedir}</directory>
                            <includes>
                                <include>testcontainer.Dockerfile</include>
                            </includes>
                        </resource>
                        <!-- The just-built WAR -->
                        <resource>
                            <directory>${project.build.directory}</directory>
                            <includes>
                                <include>client-webapp.war</include>
                            </includes>
                        </resource>
                        <!-- Support files from the main repo Docker config -->
                        <resource>
                            <directory>${project.basedir}/../../../docker/support</directory>
                            <includes>
                                <!-- Contains Liberty server.xml, etc. -->
                                <include>liberty/client-app-a/**</include>
                                <!-- Contains a Domino server.id, names.nsf, and notes.ini -->
                                <include>notesdata-ciserver/**</include>
                            </includes>
                        </resource>
                    </resources>
                </configuration>
            </execution>
        </executions>
    </plugin>
    <!-- Build a Docker image to be used by Testcontainers -->
    <plugin>
        <groupId>com.spotify</groupId>
        <artifactId>dockerfile-maven-plugin</artifactId>
        <version>1.4.13</version>
        <executions>
            <execution>
                <?m2e ignore?>
                <id>build-webapp-image</id>
                <goals>
                    <goal>build</goal>
                </goals>
                <phase>pre-integration-test</phase>
                <configuration>
                    <repository>client-webapp-test</repository>
                    <tag>${project.version}</tag>
                    <dockerfile>${project.build.directory}/dockerscratch/testcontainer.Dockerfile</dockerfile>
                    <contextDirectory>${project.build.directory}/dockerscratch</contextDirectory>
                    <!-- Don't attempt to pull Domino images -->
                    <pullNewerImage>false</pullNewerImage>
                </configuration>
            </execution>
        </executions>
    </plugin>
<!-- snip -->

The Dockerfile itself is basically what I had in the afore-linked post, minus the special ENTRYPOINT stuff.

Of note in this config is <pullNewerImage>false</pullNewerImage> in the dockerfile-maven-plugin configuration. Without that set, the plugin would attempt to look for a Domino image on the public Dockerhub and then fail because it's unavailable. With that behavior disabled, it will just use the one locally loaded.

Configuring the Tests

Now that I had that configured, it was time to adjust the tests to suit. Previously, I had been using system properties passed from the Maven environment into the test runner to identity the Liberty server, but now the container initialization will happen in code. Since this app is pretty heavyweight, I didn't want to do what most of the Testcontainers examples show, which is to let the Testcontainers JUnit hooks spawn and terminate containers for each test. Instead, I set up a centralized class to launch the container once:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

package it.com.example;

import org.testcontainers.containers.GenericContainer;
import org.testcontainers.utility.DockerImageName;

public enum AppTestContainers {
    instance;
    
    public final GenericContainer<?> webapp;
    
    @SuppressWarnings("resource")
    private AppTestContainers() {
        webapp = new GenericContainer<>(DockerImageName.parse("client-webapp-test:1.0.0-SNAPSHOT")) //$NON-NLS-1$
                .withExposedPorts(8080);
        webapp.start();
    }
}

With this setup, there will only be one instance of the container launched for the whole test suite, and then Testcontainers will shut it down for me at the end. I can also use the normal mechanisms from the Testcontainers docs to get the actual name and port it ended up mapped to:

1
2
3
4
5
6

    public String getServicesBaseUrl() {
        String host = AppTestContainers.instance.webapp.getHost();
        int port = AppTestContainers.instance.webapp.getFirstMappedPort();
        String context = "clientapp";
        return AppPathUtil.concat("http://" + host + ":" + port, context, ServicesUtil.DEFAULT_JAXRS_ROOT);
    }

Once I did that, all the tests that had previously been running against a liberty-maven-plugin-run server now worked against the Docker container, and I no longer have any dependency on the local environment actually having Notes or Domino fully installed. Neat!

A Catch: Running on my Jenkins Server

Since the whole point of Docker is to make things reproducible across environments, I was flush with confidence when I checked these changes in and pushed them up to the repo. I watched with bated breath as Jenkins picked up the change and started to build. My heart sank, though, when it got to the integration test suite and it failed with a bunch of:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

Jul 19, 2021 11:02:10 AM org.testcontainers.utility.ResourceReaper lambda$null$1
WARNING: Can not connect to Ryuk at localhost:49158
java.net.ConnectException: Connection refused (Connection refused)
    at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
    at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
    at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
    at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.base/java.net.Socket.connect(Socket.java:609)
    at org.testcontainers.utility.ResourceReaper.lambda$null$1(ResourceReaper.java:163)
    at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
    at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:159)
    at java.base/java.lang.Thread.run(Thread.java:829)

What the heck? Well, I had noticed in my prep that "Ryuk" is the name of something Testcontainers uses in its orchestration work, and is what allowed me to spawn the container manually above without explicitly terminating it. I looked around for a while and saw that a lot of people had reported similar trouble over the years, but usually it was due to some quirk in a specific version of Docker on Windows or macOS, which was not the case here. I did, though, find that Bitbucket Pipelines tripped over this at one point, and it seemed to be due to their switch of using safer user namespaces. Though it sounds like newer versions of Testcontainers fixed that, I figured it's pretty likely that I was hitting a variant of it, as I do indeed use namespace remapping.

So I tweaked my failsafe-maven-plugin configuration to set the TESTCONTAINERS_RYUK_DISABLED environment variable to false and, to be safe, added a shutdown hook at the end of my AppTestContainers init method:

1
2
3

Runtime.getRuntime().addShutdownHook(new Thread(() -> {
    webapp.close();
}));

Now, Testcontainers doesn't use its Ryuk container, but the actual app container loads up just fine and is destroyed at the end of the suite. Perfect! If all continues to go well, this will mean that it'll be one step easier for other devs to run the test suites regardless of their local setup, which is always a thorn in the side of Domino-realm testing.

Closing: What About Testing Domino Apps?

I mentioned in my disclaimer at the start that this is specifically about testing apps that use a Domino runtime, not apps on Domino. Still, I bet you could do this to test a Domino app that you deploy as an NSF and/or OSGi plugins, and I may do that myself down the line so that the test suite even-more-closely matches what is actually running in production. You could adjust the maven-resources-plugin config above (or use maven-dependency-plugin) to bring in NSFs built earlier in the build with NSF ODP Tooling as well as OSGi update sites and then have your Dockerfile copy those into the Domino data directory and the workspace/applications/eclipse directory. Similarly, if you had a Domino addin that you launch as a task and which then itself listens on a port, you could do the same there.

It's still not as convenient as being able to just easily run Domino API tests without all the scaffolding, and it implies a lot of structure that makes these more firmly "integration" than "unit" tests, but that's still a powerful capability to have.

(Updated 2021-08-01)

The last week or so, I've been working on reading Notes design elements without my longtime unreliable friend DXL. This is a task I've done in parts a few times before, but now I've set out to finish the job.

For the most part, the task is arcane but explicable: the structures are generally documented well enough in the C API docs, and most non-string items are usually only a few elements long. Views - likely because they're as old as Notes itself - are not one of these easy parts.

Background

The $ViewFormat item in a view design note contains, well, the format of the view: the overall settings (like the background colors) as well as the formats for the individual columns. Because views have gotten more complicated over the years, this item has grown. And because these types of structures have to grow in a way that's generally compatible with older code that might read it, the format is essentially an array of progressively-newer structures that alternatingly describe new traits of the view and the columns.

Some of this is described in the C API documentation. In there, you'll find "readview.c", which shows a basic example of how to read a view. Unfortunately, this example stops at the VIEW_TABLE_FORMAT2 structure, which, as we'll see shortly, is not far in. From what I can gather by looking at the "FirstVer" values in the C API documentation NSF, this file demonstrates only what existed before Notes R4.

So that example is a bust, but fortunately "viewfmt.h" has some more information. Near the top of that file is an overview of the $ViewFormat item structure in general, and it's more helpful. It's not fully helpful, though: it stops with things added in R6, and doesn't cover parts added in 6.5 and beyond. Additionally, it doesn't explain the storage of the R6-era variable data - the hide-when formulas and twistie image references - and those are a bit weird.

The mitigating factor for the documentation is that, while it doesn't explain the order of the remaining data, it does at least include definitions for the post-R6 structures, such as VIEW_COLUMN_FORMAT5 and whatnot. That is... it mostly includes that: they're fully present in "viewfmt.h", but some are missing in the NSF doc DB, and the NSF lacks some important flags. Still, it's something.

My final task remained: I had a few hundred bytes left and, armed just with some available structures and Karsten Lehmann's earlier attempt, I had to figure out how views work. And I think I did it! I haven't tested all edge cases, and in particular this will be different for calendar-type views, but this holds together. I'll update this as I find more hiccups.

Implementation Notes

I've been accessing this data as read by using the pointer returned by OSLockObject on an item value's BLOCKID. I'm also running on an amd64 Mac. Those two aspects may affect the need for ODSReadMemory - things may be different if you're on a different architecture, or perhaps the way I read it already brought it into host format. In my use, I found that ODSReadMemory was harmful in all cases, and that it was best to read the memory directly. Additionally, this is a view created using Designer V12 and read using the V12 libnotes.dylib, so anything newer won't be reflected here.

The rules around ODSReadMemory elude me at the moment. The way I'm reading this data involves the value BLOCKID as fetched by NSFItemQueryEx, which indicates that the value will be in canonical format for types including this. In theory, that should mean that ODSReadMemory is required for these structures, but it ends up harmful.

The Structure

To start out with, we have the VIEW_FORMAT_HEADER structure, which defines the format version (always 1) and whether it's a normal "table"-type view or a calendar view.

This header structure is, in table views, actually part of the VIEW_TABLE_FORMAT structure that is the true beginning (and is also where things will diverge for calendar views). This one's pretty straightforward, with no variable data, and contains the critical tidbit of the Columns value for the total number of columns, a number that we will reference again and again. Once read, advance your pointer sizeof(VIEW_TABLE_FORMAT) bytes.

Following this will be one VIEW_COLUMN_FORMAT structure for each column based on the count in the format. For each, read the structure and advance your pointer sizeof(VIEW_COLUMN_FORMAT) bytes for each. The first WORD of each of these will be equal to VIEW_COLUMN_FORMAT_SIGNATURE.

Next up is the variable data for those columns: the item name (string), the title (string), the formula (compiled formula data), and the constant value data (documented as reserved). These should be read by iterating over the above-read VIEW_COLUMN_FORMAT structures and reading byte arrays with length based on ItemNameSize, TitleSize, FormulaSize, and ConstantValueSize.

Next is VIEW_TABLE_FORMAT2, which is the last bit read by the "readview.c" example. Read sizeof(VIEW_TABLE_FORMAT2) bytes and increment your pointer.

(At this point in time, you might be inclined to ask "hey, how come VIEW_COLUMN_FORMAT has a handy signature to identify it but VIEW_TABLE_FORMAT2 doesn't?". Well, to that I say that you sure are asking a lot of nosy questions, pal, and maybe you should mind your own business. Anyway, this will be a reocurring feature.)

This is followed by one VIEW_COLUMN_FORMAT2 structure for every column in the view. Though this includes variable-data sizes, we don't read them yet - just read sizeof(VIEW_COLUMN_FORMAT2) * n in total here. These will each have VIEW_COLUMN_FORMAT_SIGNATURE2 in the first WORD.

Next is VIEW_TABLE_FORMAT3, which is a blessedly fixed-size structure, so you can just read sizeof(VIEW_TABLE_FORMAT3) bytes and move on.

Now here's where we come to the variable data from VIEW_COLUMN_FORMAT2, and where things start getting weird. To read this data, iterate over each FORMAT2 structure for your columns and check the wHideWhenFormulaSize and wTwistieResourceSize properties. If the hide-when size is above zero, read the hide-when formula first as a compiled formula. Then, read the twistie resource as CDRESOURCE composite-data structure if its size is non-zero. Fortunately, the lengths for both of these are defined in VIEW_COLUMN_FORMAT2, so the total amount to read will be wHideWhenFormulaSize plus wTwistieResourceSize from that structure.

Next, we have VIEW_TABLE_FORMAT4, which is a simple little structure. Of note here is that the RepeatType property - which I gather is how to repeat the background image of the view - references "viewprop.h", but that is not in the documentation. Oh well. Pleasantly, though, this structure includes a Length property as its first WORD, so read that many bytes (even though it should always be 8).

After this, our friend CDRESOURCE shows up again, this time as a single entity to represent the background image of the view. Since there's nothing that tells you the length of this before hand, you should glean the length from the CD record header, which is a WSIG - so the second WORD. Read that, then backtrack and read that many bytes.

Now, we have 0-to-n VIEW_COLUMN_FORMAT3 structures - one for each column that includes date formatting (from that "Style" dropdown on the fourth tab in Designer). To read these, look over your array of VIEW_COLUMN_FORMAT2 structures and, for each that has ExtDate in its Flags3 field, read one of these. Each column's structure is immediately followed by variable data for the different formatting strings, so read strings based on DTDsep1Len, DTDsep2Len, DTDsep3Len, and DTTsepLen from the structure, and then move on to the next applicable column.

These are followed by 0-to-n VIEW_COLUMN_FORMAT4 structures. These are the same idea as above, but for number formatting. Look for VIEW_COLUMN_FORMAT2 entries with NumberFormat in their Flags3 and read a VIEW_COLUMN_FORMAT4. Each of these is followed immediately by strings based on DecimalSymLength, MilliSepSymLength, NegativeSymLength, and CurrencySymLength, so read those too before moving to the next applicable column.

We've got one more of these in this sequence, and that's VIEW_COLUMN_FORMAT5: formatting for names columns. Iterate through your VIEW_COLUMN_FORMAT2s and look for NamesFormat in Flags3 and read these structures. Each one is followed immediately by the programmatic name of the column containing the distinguished person name for Sametime purposes, as defined by wDistNameColLen. Fortunately, this structure also includes a dwLength (which is a WORD despite the prefix) that contains the total length of the structure and its associated variable data, so you can read that many bytes in total here. It also, though, contains a nasty trick: you might be inclined to think that this structure also contains the name of the shared column used here, what on account of wSharedColumnAliasLen being in the structure and the documentation saying that's what it means. You'd be wrong, though - that value will be 0 and you'll get nothing from it.

This next part is a doozy. The fact that VIEW_COLUMN_FORMAT5 purports to contain the name of the shared column but in fact doesn't raises the question of where that name actually is. Well, it's right up next - you just need to read the string! But hrm: the length value in the previous structure was 0 and, besides, there won't even be one of those for non-names columns. Is it null-terminated? No: your next bytes will likely be something like 0x0700, and 0x07 is not a column name, that's for sure. What we have here is a P-string, prefixed by a WORD containing the number of characters. These strings represent both shared column titles and any titles for columns marked as "Do not display title in column header", with the latter first. So, loop through your VIEW_COLUMN_FORMAT2s again, looking for HideColumnTitle in Flags3, and, for each match, read a WORD length and then that many characters to glean the title. Then, do the same for columns with IsSharedColumn in Flags3 for your shared-column aliases.

The final structure in play that I know of is VIEW_COLUMN_FORMAT6, which fortunately is handled in basically the same way as formats 3-5. Look through VIEW_COLUMN_FORMAT2 again, this time checking Flags3 for ExtendedViewColFmt6 (I appreciate the functional name), and read a VIEW_COLUMN_FORMAT6 for each. This one again has some variable data immediately after each structure and also has a Length property, so read in that property's value in total bytes. This structure isn't in the doc NSF, but it is in "viewfmt.h", and it contains R8-era composite-app stuff.

Finally, it seems like there's a trailing 0. It makes me nervous that I end up with a single byte, but it seems to be consistent, and so my guess is that it's here to meet a WORD boundary, which is extremely common in the Notes API.

Incidentally, I don't know where VIEW_TABLE_FORMAT5 comes in, if at all. It's in the NSF and "viewfmt.h", but looks like just a duplicate of VIEW_TABLE_FORMAT4. It's possible that it sometimes shows up right after 4 when some specific thing is set, though it has no signature component and would be difficult to detect if so.

The Structure (Condensed)

If you came here to just get a quick overview of the format, I'm afraid I've pulled a "recipe blog post" on you and wrote it all in prose before the important part. In any event, here's the above but condensed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

VIEW_TABLE_FORMAT
VIEW_COLUMN_FORMAT * column count
(variable data for each column)     // Order: (item name, title, compiled formula, constant value), repeat per column
VIEW_TABLE_FORMAT2
VIEW_COLUMN_FORMAT2 * column count
VIEW_TABLE_FORMAT3
(hide-whens and twistie resources)  // Order: (compiled formula, CDRESOURCE), repeat per column where VIEW_COLUMN_FORMAT2 has wHideWhenFormulaSize > 0 or wTwistieResourceSize > 0
VIEW_TABLE_FORMAT4
CDRESOURCE                          // Represents the background image for the view
VIEW_COLUMN_FORMAT3 * x1            // One per column where Flags3 has ExtDate. Order: (VIEW_COLUMN_FORMAT3, var data), repeat per column
VIEW_COLUMN_FORMAT4 * x2            // One per column where Flags3 has NumberFormat. Order: (VIEW_COLUMN_FORMAT4, var data), repeat per column
VIEW_COLUMN_FORMAT5 * x3            // One per column where Flags3 has NamesFormat. Order: (VIEW_COLUMN_FORMAT5, var data), repeat per column
(hidden column titles)              // One per column where Flags3 has HideColumnTitle. Stored as WORD-prefixed P-strings
(shared column names)               // One per column where Flags3 has IsSharedColumn. Stored as WORD-prefixed P-strings
VIEW_COLUMN_FORMAT6 * x4            // One per column where Flags3 has ExtendedViewColFmt6. Order (VIEW_COLUMN_FORMAT6, var data), repeat per column

Conclusion

Hoo boy, this was a fun one. Anyway, assuming this holds up under further testing, my hope is that this post will be useful for the next person to come around to try to decode $ViewFormat. If that's you: good luck!

Though my previous post was also about proxies, I've had this topic percolating for a while, and it's related mostly by name and very-loose concept. Specifically, I'd like to talk about dynamic proxy classes in Java, which is a mechanism to allow you to create "fake" classes that programmatically intercept method calls.

This is something that I didn't properly realize was possible in Java until diving into CDI (though its mechanism is slightly different). In retrospect (and by "@since" annotation), it's obvious that this has been present for a long time, since Java 1.3, but outside of my realm of experience.

Definition, the Roundabout Way

So, to begin with, I'll have to define what I even mean by this, or at least an example of how it works in practice.

Normally, you just have classes and objects based on them - class Foo gets instantiated via new Foo() and there's a pretty clear direct relationship, "Object-Oriented Programming 101" sort of stuff. Let's add a little indirection by way of our old friend Interfaces. Say you have this setup:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

interface Person {
    String getName();
}

class PersonImpl implements Person {
    @Override
    public String getName() {
        return "Joe Schmoe";
    }
}

// ...

Person foo = new PersonImpl();
System.out.println("Hello from " + foo.getName());

This is essentially the same kind of thing that you're doing in the original concept - instantiating an object that you can call methods on - but you're taking a step back. You know here that you're just calling new PersonImpl(), but that's less of a hard requirement: you could instead do Person foo = lookupPerson("Joe Schmoe") and that method could return any implementation of the interface it likes.

So let's do just that, and here's where we see proxies:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

class PersonProxy implements InvocationHandler {
    public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
        if("getName".equals(method.getName())) {
            return "Proxy-Eyed Joe";
        }
        throw new UnsupportedOperationException("I don't know how to handle " + method);
    }
}

// ...

public Person lookupPerson(String name) {
    return (Person)Proxy.newProxyInstance(Thread.currentThread().getContextClassLoader(), new Class<?>[] { Person.class }, new PersonProxy());
}

For the code calling lookupPerson, this will function largely identically to if you used a normal class, but it's got this weirdness going on beneath. When code calls getPerson (or any other method), rather than calling a directly-implemented method like in the normal class, it instead calls invoke on this InvocationHandler, which can then handle it any way it'd like. If you've used more-dyanmic languages, this may look similar to method_missing in Ruby or invocation forwarding in Objective-C.

Non-Thorough Mention of Other Proxy Types

Before continuing into why you'd want to use this, I think it's important to note that the Proxy object in question above is java.lang.reflect.Proxy, which is a built-in mechanism that ships with the JVM. However, it's both limited and not the only game in town. The main way it's limited is that you can only proxy to interfaces with it, not normal classes. If Person above were class Person instead of interface Person, then the stock Proxy class would be out of luck.

There are other implementations, though - the ones that spring to mind are cglib and Javassist, but I believe there are others. These differ in their implementation (often doing things like bytecode manipulation), capabilities (these generally allow you to proxy to full classes), and performance characteristics. The concepts are largely the same, though, so from now on I'll use "proxying" to refer to the concept generally and not solely to the specific capability that comes with Java.

So What's the Use?

Okay, so you can make a proxy object that allows for programmatic handling of method calls. How would this actually be useful in practice?

In my work, I've had a couple cases where I implement proxies for specific behaviors, and they're also extremely common in Jakarta EE and Spring development. Some of these uses can get pretty arcane in concept or implementation, but others will hopefully be simpler to demonstrate.

Example 1: Counting Method Calls

For this case, say you want to count how many times a method is called in practice (and also say that YourKit doesn't exist for some reason). Taking our example InvocationHandler above, we can expand it to keep a running total of calls:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

class PersonProxy implements InvocationHandler {
    private final Map<Method, AtomicLong> counter = new HashMap<>();

    public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
        this.counter.computeIfAbsent(method, key -> new AtomicLong()).incrementAndGet();

        if("getName".equals(method.getName())) {
            return "Proxy-Eyed Joe";
        }
        throw new UnsupportedOperationException("I don't know how to handle " + method);
    }
}

(I use AtomicLong here because it's a convenient number holder, but it's also incidentally a step in the direction of making this thread-safe)

Now, as any method is called on your object, you'll get a count of invocations. You can imagine elsewhere having an admin console that lists totals for each object, giving you an idea of where the performance-sensitive parts of your code likely are.

This is, in fact, what MicroProfile Metrics does, albeit in a more-flexible and -complete way than this example. That spec uses annotations to define what you want tracked and how, and then the CDI-based proxy objects can keep track of counts and execution times.

Example 2: Performance Improvements

For this case, imagine you have a model framework in Domino where you have classes representing back-end documents. The loading of these documents can get pretty expensive, especially if you implemented it in a traditional way where your code loads up all values for the front-end Java class from the document at once. However, say you also have a mechanism to look up these documents in bulk via views, where values from the document might be already indexed and readable faster without having to crack the whole thing open. You might end up with a proxy class like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

class PersonModelProxy implements InvocationHandler {
    private final ViewEntry viewEntry;
    private PersonImpl realDoc;

    public PersonModelProxy(ViewEntry viewEntry) {
        this.viewEntry = viewEntry;
    }

    public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
        if("getId".equals(method.getName())) {
            return viewEntry.getUniversalID();
        }

        PersonModel obj = someExpensiveObjectLoad();
        return method.invoke(obj, args);
    }
}

PersonModel person = fetchModelWithProxy();
String unid = person.getId(); // Fast!
String firstName = person.getFirstName(); // Slow, but could be made fast

Here, because the UNID will already be present in the view entry, we can just return that immediately rather than loading the full backend document. If you expand this to apply to other properties that can come from view columns, you can do efficient batch lookups and tables while not having to have a separate "read from view instead of doc" mechanism on the front end.

This is something I'm doing (using Javassist's proxies) with a client project to put view entries over existing model objects that had accrued over the years. This allows us to keep the same logic while massively speeding up operations that don't actually need the document to be loaded.

Example 3: Repetitive but Predictable Code

If you have behavior that can be reliably derived from some non-algorithmic source (say, from annotations, app configuration, or so forth), proxies can help you write adaptive code once that avoids the need to write tons of boilerplate methods or classes.

As an example, I'll expand a bit on the first notion above, which is profiling. A long time ago, the erstwhile XPages team released the XPages Toolbox, which provides a reasonably-fine-grained view into what takes up the time during an XPages request. It also provides a way for your code to opt into this profiling, but the idiom is extremely repetitive. You can see it in the Extension Library, and it tends to look something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

public class SomeBean {
    private static final ProfilerType profilerType = new ProfilerType("SomeBean");

    public void doFoo() {
        if(Profiler.isEnabled()) {
            ProfilerAggregator agg = Profiler.startProfileBlock(profilerType, "doFoo");
            long ts = Profile.getCurrentTime();
            try {
                _realDoFoo();
            } finally {
                Profiler.endProfileBlock(agg, ts);
            }
        } else {
            _realDoFoo();
        }
    }

    private void _realDoFoo() {
        // Actual code goes here
    }
}

That's certainly explicable, but imagine writing that for all methods, or even for a large chunk of methods you want to optimize. That could be done a little more cleanly now with Java 8+ features, but it'd still be a drag.

If we go back to the idea handled by MicroProfile Metrics, it would instead look more like:

1
2
3
4
5
6
7

@Profiler(name="SomeBean")
public class SomeBean {
    @Timed(name="doFoo")
    public void doFoo() {
        // Actual code goes here
    }
}

That's a little nicer! In practice, this is really done with CDI, which provides its own nice layer around proxies, but you could see implementing it with IBM's profiler and a proxy like so (forgive the fragility of the code):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

class ProfilerProxy implements InvocationHandler {
    private final Map<Class<?>, ProfilerType> profilers = new HashMap<>();

    public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
        ProfilerType profilerType = profilers.computeIfAbsent(proxy.getClass(), c -> new ProfilerType(c.getAnnotation(Profiler.class).name()));

        if(Profiler.isEnabled()) {
            String metricName = method.getAnnotation(Timed.class).name();
            ProfilerAggregator agg = Profiler.startProfilerBlock(profilerType, metricName);
            long ts = Profiler.getCurrentTime();
            try {
                method.invoke(obj, args);
            } finally {
                Profiler.endProfilerBlock(agg, ts);
            }
        } else {
            method.invoke(obj, args);
        }
    }
}

Now you only write the boilerplate once and it'll work on anything, or at least anything you're proxying.

Summary

In general Proxies are the sort of thing where it's somewhat rare that you'll have occasion to write one "directly" yourself, but they're immediately useful in the few times when that matters. They're also vital to know about for our brave new CDI world, since they explain so much of how newer Java standards do what they do - I didn't even go into things like how JNoSQL determines desired behavior just from method name, for example. This one's a deep rabbit hole, but it's extremely useful to know it's even possible, even before you get to putting it to use.

Codicil: Performance

Using these proxy objects quickly brings a curious thought to one's mind: what kind of overhead is there? Do I have to worry about performance degradation if there's this extra layer of indirection?

The long answer is that it's complicated. The short answer, though, is that you will almost definitely not have to worry. Certainly, there's going to be some inherent overhead just because more stuff is happening, but in general that extra work is dwarfed by the actual business of your business logic. If you have a method that computes a complicated value, or fetches something from a database, or (lord help your profiler) makes a remote network call, the overhead of the proxy is going to be several orders of magnitude smaller than the action being performed.

That said, it doesn't necessarily hurt to keep the idea of this impact in mind. While very little in a web application will be harmed in any perceptible way by proxying, there's always a chance that you'll run into a situation where a simple method is called thousands and thousands of times more often than more-complex ones, and that's where you may want to either move it to a non-proxied object or comb through the latest performance metrics for different proxy libraries. As with most optimization, though, that's something to do after you've found that there's a performance problem, not (usually) before.

A while back, I added built-in reverse proxy capabilities to the Open Liberty Runtime, first as a standalone process on its own port and then within Domino's normal HTTP stack as an HttpService extension. I realized now that I didn't go much into the specifics, and I think those are interesting.

Commonalities

Though the two modes are very different in their final implementation, they share a common conceptual basis, with shared representations of what it is that's supposed to be reversed proxied to. The specifics are abstracted enough such that the core that any proxy implementation needs is an instance of ReverseProxyConfig, as provided by a ReverseProxyConfigProvider service implementation. In practice, this comes from a class that reads app configuration from a view in the admin NSF:

That lets it determine which port to target, which connector headers to use, and which Domino servers host it. Importantly, the response-doc lines there are the context paths that the apps listen on.

Each individual app attached running on an app server here is given a context root, such as "exampleapp" or "jessehome". When a request comes in, the proxy checks to see if it starts with one of the context roots and then either routes the request to the backing app server or passes it along to Domino (with the mechanism for the latter differing between proxies). This allows all of the apps and Domino to have a unified URL space:

On its own, that's basically what a reverse proxy always does, but this one has the advantage of being auto-configured based on the configuration in the admin NSF, and also deployed alongside or in Domino.

In-HTTP Reverse Proxy

The in-HTTP reverse proxy is implemented as an HttpService extension, which is a just-below-XPages-layer extension available on Domino generally. As I mentioned in my blog post about this the other month, the way it works is that the Domino HTTP server checks with these services to see if any of them want to lay claim to the incoming URL. That claim can be made by any aspect of the URL string - so, for example, the XPages service looks for ".xsp" and related URLs, plus any registered servlets and webapps. In my case, that fits perfectly with what I need here.

The proxy itself consists of one class, which is adapted from an open-source reverse proxy servlet. The proxy checks its target list for the incoming request. If one of those matches, it will then proxy along the request to the given backend. The specifics of implementing the reverse proxy in a servlet-type service are fiddly, but of note is the addition of the standard and WS-specific proxy headers, based on the configuration in the NSF.

This implementation lucks out in the default case: for it to pass unmatched requests to Domino, all it has to do is declare that it doesn't handle the URL, and the rest of the system will handle it from there.

Benefits

The big benefit to this approach is that it ends up being basically transparent to a normal Domino administration. Since it's running as an HttpService, it shares Domino's HTTP and HTTPS ports, and also uses the same TLS configuration. Though the runtinme spawns Libery servers in the background, those can be on high-numbered ports and not visible, so it ends up looking like the apps are running on Domino itself.

Drawbacks

The big drawback stems from the benefits above: since it shares Domino's HTTP ports, it also shares its limitations. Request matching and routing happens at the C level in Domino, so there's no possibility of supporting HTTP/2 or, I believe, WebSocket connections. In a lot of cases, that's not really a big deal, but it diminishes the benefits of using a superior app server.

Separate-Port Reverse Proxy

The standalone reverse proxy listens on its own HTTP and HTTPS ports and makes use of the Undertow project, which is an infrastructural piece of the Wildfly Java app server. In this sort of setup, you'd likely shift Domino's HTTP port to something downstream, like 8080, and configure this proxy to listen on port 80 and 443 itself, taking over main HTTP duties.

Though this listens on a separate port, it still runs from within the JVM of the HTTP service on Domino - it's just spawned in a separate thread as a listener. Since Undertow is designed to be this sort of in-server plumbing, it's configured with a pleasant-enough API, where you can add a number of "handler" types for incoming requests. Of particular note here is the PathHandler class, which lets you assign behavior based on prefix paths, and that's exactly what I do.

For each configured app, my code registers a prefix-based proxying handler, which lets Undertow do the job above of matching incoming URLs to their backend app.

I then set up a catchall proxy at /, which handles passing requests along to the port configured in the Domino server's server doc, along with built-in support for my reverse proxy secret DSAPI filter (which has been proving itself nicely in practice).

Benefits

The big benefit to this approach is that Undertow is much more capable than Domino's HTTP server when it comes to modern web technologies. Your apps can use HTTP/2 and WebSocket without issue, and I can be confident that it will also be updated for future changes down the line. It's also tried-and-true code: while my adapted proxy servlet seems fine in practice, I know for sure that Undertow is up for the task.

It also has a slew of other capabilities and a pluggable architecture that I don't use here, but it opens the door to other potential uses.

Drawbacks

The big drawback is the separate port, which increases the cognitive load for administration and is just "weird" for someone used to Domino's HTTP stack being the only entrypoint. And even if you're okay with doing that (which you should be - it works fine), it's non-obvious when you're trying to get a bead on how the server is configured. If you're tracking down what's actually listening on port 80, you can see that Domino is configured for some other port using normal Domino-admin knowledge, but there's nothing that would tell you to look in libertyadmin.nsf for a configuration for another reverse proxy on a separate thread. That's not wrong, and it's better than if I required you to mangle your names.nsf design, but it's opaque.

Conclusion

I suppose the main conclusion I'll hope you draw is that this stuff is in there and it's neat. It's also the sort of thing that would be adaptable to plenty of other uses: in addition to the Domino Open Liberty project itself being structured such that one could add non-Liberty/Java app runtimes, neither of these proxy techniques are inherently dependent on that project at all.