About a year and a half ago, I wrote a post talking about adding JaCoCo code coverage to the integration-test suite of the XPages JEE project.

Today, I happened to notice that, though unheralded, Domino 14.5 FP1 bumped the JVM from Semeru 21.0.6 to 21.0.8. Normally, these little patch-level bumps are pretty irrelevant, but this one was a big deal: Java Flight Recorder support came to Semeru 21 in 21.0.7, so now we have it.

JFR Basics

Java Flight Recorder, in turn, is a built-in capability to monitor and profile your Java applications, similar to things like YourKit. The prime advantage that JFR brings is that it's very low-overhead, enough that it's possible to just run passively in all but particularly-performance-sensitive situations. Even if you don't keep it on all the time, though, it's possible to turn it on and off at will for a running application, whether or not you configured profiling ahead of time. That makes it very intriguing for production environments where you would be disinclined to install, say, the YourKit agent ahead of time.

With HotSpot JVMs, you can automatically enable JFR via a JVM argument, but Semeru doesn't currently implement that. Instead, Semeru supports just the manual way (which is also in HotSpot), using the jcmd tool, which comes with the JDK.

Working with JFR in Domino

The way jcmd works is that you give it the PID of a running compatible Java application and execute one of a number of commands. The tool predates JFR, but I hadn't had a need to know about it until now.

Domino doesn't ship with jcmd, but the one in a download of a Semeru JDK will work. I don't know if it has to be the same patch version or not, but I got the matching one for my tests. Then, you can run jcmd -l to get a list of running JVMs, like:

C:\java\jdk-21.0.8+9\bin>jcmd -l
10904
8884 <no information available>

Those are... well, they're just the PIDs of Java apps. Here, I happen to know that Domino's nhttp.exe is 10904 because it's the one that doesn't say "no information available", but in practice you'll want to do something proper to get your desired PID. In a Domino container, pgrep http will likely give you the number you want.

PID in hand, you can run execute a "JFR.start" command to start recording:

C:\java\jdk-21.0.8+9\bin>jcmd 10904 JFR.start filename=C:\out.jfr
Start JFR recording to C:\out.jfr

I recommend specifying a filename like that, since otherwise it'll always be the same stock one based on a JFR profile. If you're running automated tools, you'll want to avoid collisions.

Once you do whatever it is you want to profile, you can stop the recording:

C:\java\jdk-21.0.8+9\bin>jcmd 10904 JFR.stop
Stop JFR recording, and dump all Java threads to C:\out.jfr

That's about it, at least for the normal case. That file can be read by a number of tools. I use the aforementioned YourKit, since I have and like it, but you can also use the jfr command-line tool, the official JDK Mission Control app, IntelliJ, and likely other things.

Hooking It Into The Test Suite

Doing manual sampling like this is very useful on its own, and I do this sort of thing a lot when getting into the weeds of performance tuning. The low overhead, though, means it's a perfect addition to a full test suite as a "may as well" sort of thing.

Getting this going took a couple steps, but it wasn't too bad once I figured it out.

The first hurdle is the lack of jcmd in the Domino container. Fortunately, this is a straightforward fix: just grab it from the Semeru Docker image during build.

1

COPY --from=ibm-semeru-runtimes:open-21-jdk /opt/java/openjdk/bin/jcmd /opt/hcl/domino/notes/latest/linux/jvm/bin

Next, I want to kick off JFR before any tests run. Initially, I put this in the containerIsStarted method of the Testcontainer class, but the timing didn't work out there. Instead, it ended up in the PostInstallFactory Java class I have to do post-HTTP-init tasks:

1
2
3
4
5
6
7

Path jcmd = Paths.get("/opt/hcl/domino/notes/latest/linux/jvm/bin/jcmd");
if(Files.isExecutable(jcmd)) {
	long pid = ProcessHandle.current().pid();
	new ProcessBuilder(jcmd.toString(), Long.toString(pid), "JFR.start", "filename=/tmp/flight.jfr")
		.start()
		.waitFor();
}

This command will fail on Domino before 14.5FP1, but that's fine.

Finally, when the test runtime sees it's on a high-even version, it grabs the file on container exit:

1
2
3
4
5
6
7
8

if(useJfr()) {  // Does some checks based on container label
	// Find the PID for http to pass to jcmd
	String pid = this.execInContainer("pgrep", "http").getStdout();
	if(pid != null) {
		this.execInContainer("/opt/hcl/domino/notes/latest/linux/jvm/bin/jcmd", pid, "JFR.dump");
		this.copyFileFromContainer("/tmp/flight.jfr", target.resolve("flight-" + System.currentTimeMillis() + ".jfr").toString());
	}
}

With that in place, running the test suite will drop - alongside the log files and JaCoCo report - the JFR results in the "target" directory. Et voilà:

The information isn't as fine-grained as full profiling may get, but it'll give you a good comparison of relative expense of various calls, and that's often all you really need.

This is very neat and should be a handy addition. I'm looking forward to seeing what other good uses I can get out of JFR now that we have access to it.