WOQL - rewrite in order

[matko]

We've been discussing reworking our query language, WOQL. While we offer the document API and GraphQL as more simple means to work with data, an actual query language is needed to do more complex queries. Unfortunately, WOQL isn't very easy to work with at times. A lot of its features grew over time, rather than out of a prefigured design.

At some point we would like to embark on an improvement project for WOQL. This discussion serves as a place to gather ideas for what this improvement should entail.

first-class treatment of documents and dictionaries

WOQL was originally written for pure triple manipulation, but TerminusDB has evolved into a document graph database. We need WOQL to work better with this document model.
Core elements are here in WOQL right now, but need refinement and better ergonomics for common use patterns.

Better typing

modalities

All woql words should have clearly defined modalities, so we can perform reorders.

static checking

Types should be statically checked where possible.

Good runtime errors for type errors

When a type cannot be statically checked, there should be a good runtime check that produces a proper error that users can parse.

Automatic generation of clients

WOQL's definition, including all its typing and modalities, should be declaratively described in a way that allows clients to be auto-generated.

query compilation without concrete context

Right now we require a concrete context to run a query, which means we need to recompile the query for each request. This slows things down unnecessarily for queries that are repeated often but with slightly different arguments. Among other places this happens internally in TerminusDB, so everything would just be faster if we could precompile those queries.

score-driven reordering

We have a basic reordering strategy now by directly defining a partial order on certain woql words. This approach should be made more generic using a scoring, preferably driven by actual data.

[hpvd]

when going for big work under the hood, perhaps it's worth to think about this rework with the background of the discussion and its comments about Support of industry standard QL/GPM - openCypher - GQL

[annainfo]

Now at https://github.com/orgs/terminusdb/discussions/2088 "Support ISO GQL:2024 : avoid vendor lock-in, facilitate benchmarking/comparability"

[hoijnet]

My PR on the WOQL-TS library is making progress and it would be great to get it merged as already that initial step is actually used in production already at dfrnt.com. I have added the ability for it to support WOQL as a DSL from a string into WOQL AST, with initial support for triples and quads and a few tests using the acorn parser.

const woqlStr = `
and(triple(doc, "rdf:type", type),
  not(quad(type, "sys:subdocument", unbound, "schema")),
)
`
const woql = parseWoqlString(woqlStr);
const postBody = prepareWoqlHttpPostBody(woql);
const result = await terminusClient.sendCustomRequest("POST", `http://localhost:6363/api/woql/admin/sandbox/local/branch/main`, postBody);

I think the following should be part of it:

1. Ability to parse WOQL AST and convert to the DSL string form (canonicalized)

I have made the following assumptions that should be validated together for the canonical WOQL DSL form:

• The functional style of variable creation is the default (makes it easy to port over to javascript)
• The WOQL. prefix is dropped (but should probably be an optional flag in the generator that comes in the future
• Math and string manipulations should be reasonably possible to express using the javascript equivalents for a fluid expressions language
• DSL identifiers that are undefined are treated as WOQL variables
• camelCasing is the new default but those used to the snake_casing and having a bit of code already may find it challenging to switch. It does look a lot neater so I'm undecided but wanted to bring it up!

An example of what WOQL could look like as DSL, with calculations happening in TerminusDB and not in the client. Below would be a submitted to the new client as a string. It is likely we should add a regular keyword that can take such a string and convert to WOQL as it would enable WOQL composition of multiple parts.

and(
  triple(doc,"amount", amount),
  triple(rule,"limit", limit)
  eq(amount,expression(limit*1.2 + 30)
)

I have also has a thought around the snake_casing that seems to move in the direction of camelCasing in the new client. It does follow how most of typescript is authored, but it's important to have cross-compatibility in the DSL so it's probably an important core language decision to make so it's easier to port.