Embedding database into applications

[amerlyq]

TerminusDB as far as I understood is a server-oriented solution.
Is there any leeway in current architecture to be evolved/forked into something like sqlite / LMDB (i.e. single-file ~20-200MB)?
Or will it require complete redesign from scratch and ditching Prolog on the way?

There is major deficiency in the field of embedded graph db solutions (if any).
Basically, I would liked to use TerminusDB like git -- and keep/sync the same knowledgebase between different clients/platforms -- Android, Firefox/Extensions, Desktop/Python, etc.
But this niche feels empty.

[matko]

There are ways to make this work in the future, but nothing is currently on the roadmap.

Architecturally, the database is split up in a 'core' part which defines all operations, and a 'server' part which slaps an HTTP frontend on top of that. This means other setups are possible. One example is the CLI tool we're developing, which supports a lot of the same operations from the command line. Likewise, an embedded version which exposes these operations through function calls is also possible.

One thing to take into account though is that we make use of SWI-Prolog. Therefore, embedding TerminusDB effectively means embedding SWI-Prolog as well. This is doable, but it won't be a small embeddable like sqlite is. I'd expect something in the order of tens of megabytes of RAM overhead. But yes, this could be a single DLL that you just load to have TerminusDB available through some C API, which would also allow bindings to be written for other languages.

Another thing to take into account is that our storage format is not the best for the embedded use case. Unlike sqlite, which stores a database in a single file, we split things out into many tiny files. Depending on your use case this may or may not be a problem.

[amerlyq]

Thanks for the reply!
It basically confirmed my unconsoling conclusions on the embedded usage.

Nevertheless, RAM/disk overhead <100MB don't seem like too much for 2020+ devices, and usecase is worth trying.
So I have some expectations for the aforementioned CLI tool to study how exactly embedding may look like. But looks like it will be CLI client to some TerminusDB server, and not standalone application? No way, then?

The only real showstopper is current storage format.
Is it mappable to keyvalue store, so all these small files could be hidden inside single file?
Serving data from/on-top mmaped LMDB (read-preference) or LevelDB (write-preference) would be nice.

[rrooij]

@amerlyq Just to clarify, the CLI client is not dependent on a running server and works standalone. However, if you want to push/pull it needs a remote server to push/pull from.

Just querying without running a server is possible from the CLI as well. You can test it out yourself with the Linux binary. ./terminusdb store init --key root will initialize a new store, run ./terminusdb query _system 't(X,Y,Z)' to query the system database.

[m3talsmith]

This may not be a big deal if:

1. The server could be installed through an install package alongside the application using it
2. The server configuration can:
   1. Be placed in a common location per platform from the application installer
   2. Or a running configuration can be discovered from the application installer

In other words, it would be nice to have an isolated instance of TerminusDB, but a flexible install pattern could be enough.

[GavinMendelGleason]

We now have a snap which may help solve the application installer problem for linux users.