eXist-db Indexer for Algolia

eXist Indexer for Algolia is a configurable index plug-in for the eXist-db native XML database. It uses eXist's own indexing mechanisms to create, upload and incrementally sync local indexes with Algolia's cloud services.

eXist Indexer for Algolia in action: autocomplete search on http://raskovnik.org

Installation

The Algolia indexer is dependent on https://github.com/BCDH/cql-module follow the install instructions in the README.

It's probably a good idea to start with a clean database, which means a completely clean $EXIST_HOME/webapp/WEB-INF/data folder.

• The Index plugin requires at least eXist version 3.0.

• The plugin can be built from source code using SBT or downloaded from Maven Central.

• Make sure eXist is not running

• Place the jar file named like exist-algolia-index-assembly-2.13_1.0.0.jar into eXist's lib/user, lib for eXist-db-6.x.x.

• Modify eXist's conf.xml file by adding the following line to the indexer/modules section:

<module id="algolia-index"
    class="org.humanistika.exist.index.algolia.AlgoliaIndex"
    application-id="YOUR-ALGOLIA-APPLICATION-ID"
    admin-api-key="YOUR-ALGOLIA-ADMIN-API-KEY"/>

just for exist-6.x.x

• add the dependency in etc/startup.xml

<dependency>
    <groupId>Your group id</groupId>
    <artifactId>AlgoliaIndex</artifactId>
    <version>1.1.0</version>
    <relativePath>exist-algolia-index-assembly-1.1.0-SNAPSHOT.jar</relativePath> <!-- this should reflect the exact filename you placed in the lib folder in lib folder -->
</dependency>

• Startup eXist.

• For the Collection(s) that you want to index with Algolia, you need to add an Algolia index configuration to eXist's collection.xconf file. See instructions.

• You can then reindex the collection with the data in eXist the way you normally do, and it should be added to Algolia.

You can use the Algolia Dashboard to examine the index. Alternatively, you can also set up eXist to to log changes to our Algolia Index. See instructions.

Configuration

For a single collection in eXist, you can put data into one or more indexes in Algolia, just create an "index" element inside the "algolia" element for each index and give it the name of the Algolia index, if the index doesn't exist in Algolia it will be automatically created for you.

For incremental indexing to work, you need to have two sets of unique ids, one for each document in the collection (documentId) and one for each rootObject (nodeId).

<collection xmlns="http://exist-db.org/collection-config/1.0">
    <index>

        <algolia>
            <namespaceMappings>
                <namespaceMapping>
                    <prefix>xml</prefix>
                    <namespace>http://www.w3.org/XML/1998/namespace</namespace>
                </namespaceMapping>
            </namespaceMappings>
            <index name="my-algolia-index-1" documentId="/path/to/unique-id/@хml:id" visibleBy="/path/to/unique-id">
                <rootObject path="/path/to/element" nodeId="@xml:id">
                    <attribute name="f1" path="/further/patha"/>
                    <attribute name="f2" path="/further/pathb" type="integer"/>
                    <object name="other" path="/further/pathc">
                        <map path="/x" type="boolean"/>
                   </object>
                </rootObject>
            </index>
        </algolia>

    </index>
</collection>

An Optional VisibleBy attribute can be used to restrict data access when searching the Algolia index

A rootObject is equivalent to an object inside an Algolia Index. We create one "rootObject" either for each document, or document fragment (if you specify a path attribute on the rootObject).

An attribute (represents a JSON object attribute, not to be confused with an XML attribute) is a simple key/value pair that is extracted from the XML and placed into the Algolia object ("rootObject" as we call it). All of the text nodes or attribute values indicated by the "path" on the "attribute" element will be serialized to a string (and then converted if you set an explicit "type" attribute).

The path for an "attribute" may point to either an XML element or XML attribute node. Paths must be simple, you can use namespace prefixes in the path, but you must also set the namespaceMappings element in the collection.xconf.

The XML Schema file https://github.com/BCDH/exist-algolia-index/blob/master/src/main/resources/xsd/exist-algolia-index-config.xsd defines and documents the index configuration.

An object represents a JSON object, and this is where things become fun, we basically serialize the XML node pointed to by the "path" attribute on the "object" element to a JSON equivalent. This allows you to create highly complex and structured objects in the Algolia index from your XML.

The name attribute that is available on the "attribute" and "object" elements allows you to set the name of the field in the JSON object of the Algolia index, this means that name names of your data fields can be different in Algolia to eXist if you wish.

limiting Objects access to certain users

You can limit data access by setting the visibleBy attribute in collection.xconf then matching the path in your XML data preferably in the header You can use this example from out test suit

xml: https://github.com/BCDH/exist-algolia-index/tree/master/src/test/resources/integration/user-specified-visibleBy/VSK.TEST.xml

collection.xconf https://github.com/BCDH/exist-algolia-index/tree/master/src/test/resources/integration/user-specified-visibleBy/collection.xconf

Enable logging in eXist (optional)

You can see what we are sending to Algolia by adding the following to your $EXIST_HOME/log4j2.xml file:

Add this as a child of the <Appenders> element:

<RollingRandomAccessFile name="algolia.index"
        filePattern="${logs}/algolia-index.${rollover.file.pattern}.log.gz"
        fileName="${logs}/algolia-index.log">
    <Policies>
        <SizeBasedTriggeringPolicy size="${rollover.max.size}"/>
    </Policies>
    <DefaultRolloverStrategy max="${rollover.max}"/>
    <PatternLayout pattern="${exist.file.pattern}"/>
</RollingRandomAccessFile>

And add this as a child of the <Loggers> element:

<Logger name="org.humanistika.exist.index.algolia" additivity="false" level="trace">
    <AppenderRef ref="algolia.index"/>
</Logger>

The log output will then appear in $EXIST_HOME/webapp/WEB-INF/logs/algolia-index.log the next time eXist is started.

Current limitations

• you can't use paths with predicates in index configuration. Support for predicates will be added at a later stage.
• when you backup eXist, you should now also make a backup copy of $EXIST_HOME/webapp/WEB-INF/data/algolia-index as that holds the local representation of what is on the remote Algolia Server. Support for adding locally stored Algolia indexes to the backup/restore procedure may be added in the future.

Building from Source

$ git clone https://github.com/BCDH/exist-algolia-index.git
$ cd exist-algolia-index
$ sbt assembly

The assembled binary can then be found is in the folder target/scala-2.13.

Acknowledgements

Hats off to Adam Retter for sharing his superb programming skills with us in this project.

This tool has been developed in the context of an ongoing BCDH project: Raskovnik - A Serbian Dictionary Platform (together with the Institute of Serbian Language). The project has received funding from the Ministry of Culture and Information of the Republic of Serbia.