Implementing a Basic JNoSQL Driver for Domino
Jan 25, 2022, 1:36 PM
- Updating The XPages JEE Support Project To Jakarta EE 9, A Travelogue
- JSP and MVC Support in the XPages JEE Project
- Migrating a Large XPages App to Jakarta EE 9
- XPages Jakarta EE Support 2.2.0
- DQL, QueryResultsProcessor, and JNoSQL
- Implementing a Basic JNoSQL Driver for Domino
- Video Series On The XPages Jakarta EE Project
- JSF in the XPages Jakarta EE Support Project
- So Why Jakarta?
- XPages Jakarta EE 2.5.0 And The Looming Java-Version Wall
- Adding Concurrency to the XPages Jakarta EE Support Project
- Adding Transactions to the XPages Jakarta EE Support Project
- XPages Jakarta EE 2.9.0 and Next Steps
- XPages JEE 2.11.0 and the Javadoc Provider
- The Loose Roadmap for XPages Jakarta EE Support
- XPages JEE 2.12.0: JNoSQL Views and PrimeFaces Support
- XPages JEE 2.13.0
- XPages JEE 2.14.0
- XPages JEE 2.15.0 and Plans for JEE 10 and 11
A few weeks back, I talked about my use of DQL and QRP in writing a JNoSQL driver for Domino. In that, I left the specifics of the JNoSQL side out and focused on the Domino side, but that former part certainly warrants some expansion as well.
Background
As a quick overview, Jakarta NoSQL is an approaching-finalization spec for working with NoSQL databases of various stripes in a Jakarta EE app. This is as opposed to the venerable JPA, which is a long-standing API for working with RDBMSes in JEE.
JNoSQL is the implementation of the Jakarta NoSQL spec, and is also an Eclipse project. As a historical note, the individual components of the implementation used to have Greek-mythological names, which is why older drivers like my Darwino driver or original Domino driver are sprinkled with references to "Diana" and "Artemis". The "JNoSQL" name also pre-dates its reification into a Jakarta spec - normally, spec names and implementations aren't quite so similarly named.
The specification is broken up into two main categories. The README for the implementation describes this well, but the summary is:
- "Communication" handles interpreting JNoSQL CRUD operations and actually applying them to the database.
- "Mapping" handles what the app developer interacts with: annotating classes to relate them to the back-end database and querying object repositories.
An individual driver may include code for both sides of this, but only the Communication side is obligatory to implement. A driver would contribute to the Mapping side as well if they want to provide database-specific higher-level concepts. For example, the Darwino driver does this to provide explicit annotations for its full-text search, stored-cursor, and JSQL capabilities. I may do similarly in the Domino driver to expose FT search, view operations, or DQL queries directly.
Jakarta NoSQL handles Key-Value, Column, Graph, and Document data stores, but we only care about the last category for now.
Implementation Overview
Now, on to the actual implementation in question. The handful of classes in the implementation fall into a few categories:
DatabaseSupplier
andSessionSupplier
are mechanisms for the environment to inject theDatabase
andSession
(as signer) objects for the driver to use. This indirection is to allow the driver to be used outside of the XPages JEE project specifically - in this project, the XPages library bundle provides implementations using the currentNotesContext
.DominoDocumentCollectionManager
defines the interface for the Domino-specific manager type. This interface is currently empty beyond what it inherits, but it's where I may provide hooks for DB-specific capabilities.DefaultDominoDocumentCollectionManager
, in turn, is the implementation of this and actually defines CRUD operations. More on this down the page for sure.DominoDocumentCollectionManagerFactory
is the class that, based on configuration, emits the above for the environment.
DominoDocumentConfiguration
is the class where you can configure the above, and serves as the entrypoint for practical use. It uses Jakarta NoSQL's config and settings concepts, either taking those suppliers from the passed-in configuration or trying to find them from the current CDI environment.EntityConverter
contains support code for CRUD operations, assisting in mapping between JNoSQL's internal entity representation and Domino documents.QueryConverter
does similarly for queries, taking JNoSQL's abstract query tree concept and translating it to DQL.- Speaking of which,
DQL
is awholesale ripoffproperly-licensed adaptation of Karsten Lehmann's class of the same name, modified slightly to remove Domino-JNA-specific elements and improvejava.time
handling a bit.
Implementation Details
The core entrypoint for data operations is DefaultDominoDocumentCollectionManager
, and JNoSQL specifies a few main operations to implement, basically CRUD plus total count:
insert
and its overrides handle taking an abstractDocumentEntity
from JNoSQL and turning it into a newlotus.domino.Document
in the target database.update
does similarly, but with the assumption that the incoming entity represents a modification to an existing document.delete
takes an incoming abstract query and deletes all documents matching it.select
takes an incoming abstract query, finds matching documents, converts them to a neutral format, and returns them to JNoSQL. This is what my earlier post was all about.count
retrieves a count for all documents in a "collection". "Collection" here is a MongoDB-ism and the most-practical Domino equivalent is "documents with a specific Form name".
Entity Conversion
The insert
, update
, and select
methods have as part of their jobs the task of translating between Domino's storage and JNoSQL's intermediate representation, and this happens in EntityConverter
.
Now, this point of the code has some... nomenclature-based issues. There's lotus.domino.Document
, our legacy representation of a Domino document handle. Then, there's jakarta.nosql.document.Document
: this oddly-named interface actually represents a single key-value pair within the conceptual document - roughly, this corresponds to lotus.domino.Item
. Finally, there's jakarta.nosql.document.DocumentEntity
, which is the higher-level representation of a conceptual document on the JNoSQL side, and this contains many jakarta.nosql.document.Document
s. This all works out in practice, but it's important to know about when you look into the implementation code.
The first couple methods in this utility class handle converting query results of different types: QRP result JSON, QRP result views, and generic DocumentCollection
s. Strictly speaking, I could remove the first one now that it's unused, but there's a non-zero chance that I'll return to it if it ends up being efficient down the line.
Each of those methods will eventually call to toDocuments
, which converts a lotus.domino.Document
object to an equivalent List
of JNoSQL Document
s (i.e. the individual key-value pairs). Due to the way JNoSQL works, this method has no way to know what the actual desired fields the higher level will want are, so it attempts to convert all items in the document to more-common Java types. There's much more work to do here, some of it based on just needing to add other types (like improving rich text handling) and some of it based on needing a better Notes API (like proper conversion from Notes times to java.time
).
In the other direction, there's the method that converts from a JNoSQL DocumentEntity
to a Domino document, which is used by the insert
and update
methods. This converts some known common incoming types and converts them to Domino item values. Like the earlier methods, this could use some work in translating types, but that's also something that a better Notes API could handle for me.
Query Conversion
The QueryConverter
class has a slightly-simpler job: taking JNoSQL's concept of a query and translating it to a document selection.
The jakarta.nosql.document.DocumentQuery
type does a bit of double duty: it's used both for arbitrary queries (Foo='Bar'
-type stuff) as well as for selecting documents by UNID. The select
method covers that, producing a QueryConverterResult
object to ferry the important information back to DefaultDominoDocumentCollectionManager
.
The core work of QueryConverter
is in getCondition
, where it performs an AST-to-DQL conversion. JNoSQL has a couple mechanisms for querying entities: explicit Java-based queries, implicit queries based on repository definition, or a SQL-like query language. Regardless of what the higher level does to query, though, it comes to the Communication driver as this tree of objects (technically, the driver can handle the last specially, but by default it arrives parsed).
Fortunately, this sort of work is a common sort of idiom. You start with the top node of the tree, handle it based on its type and, as needed, recurse down into the next node. So if, for example, the top node is an EQUALS
, all this converter needs to do is return the DQL representation of "this field equals that value", and so forth for other comparison operators. If it encounters AND
, OR
, or NOT
, then the job changes to making a composite query of that operator plus the results of converting whatever the operator is applied to - which is where the recursion back into the same method comes in.
Future Work
The main immediate work to do here is enhancing the data conversion: handling more outgoing Domino item types and incoming Java object types. A good deal of this can be done as-is, but doing some other parts reliably will be best done by changing out the specific Notes API in use. I used lotus.domino
because it's present already, but it's a placeholder for sure.
There are also a bunch of efficiency tweaks I can make: more lazy loading in conversion, optimizing data fetching for specific queries, and logging DQL explain results for developers.
Beyond that, I'll have to consider if it's worth adding extensions to the mapping side. As I mentioned, the Darwino driver has some extensions for its JSQL language and similar concepts, and it's possible that it'd be worth adding similar things for Domino, in particular direct FT searching. That said, DQL does a pretty good job being the all-consuming target, and so translating JNoSQL queries to DQL may suffice to extract what performance Domino can provide.
So we'll see. A lot of this will be based on what I need when I actually put this into real use, since right now it's partly hypothetical. In any event, I'm looking forward to finding places where I can use this instead of explicitly coding to Notes API objects for sure.