Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions docs/decisions/0003-search-api-core-query-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ Two tiers: `search-*` is backend you compose; `search-api-*` is the surface you
| Tier | Package | Responsibility |
| ----------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| backend | `@lde/search` | field model · `SearchQuery` · filter semantics · engine port |
| backend | `@lde/search-typesense` | engine adapter: collection schema · query/filter compiler · `search()` |
| backend | `@lde/search-typesense` | engine adapter: collection definition · query/filter compiler · `search()` |
| API surface | `@lde/search-api-graphql` | field model + `SearchQuery` → GraphQL schema (runtime configuration; see [ADR 4](./0004-search-api-graphql-surface.md)) |
| API surface | `@lde/search-api-rest` | OpenAPI + route handlers (later, thin over the core) |

Expand All @@ -50,7 +50,7 @@ behind the adapter and is swappable with no consumer impact. Nothing engine-spec
### Field model

The engine-neutral description of a queryable field. **One `SearchField` declaration drives
four consumers** – projection (RDF→flat document), the engine collection schema, the query
four consumers** – projection (RDF→flat document), the engine collection definition, the query
semantics, and the GraphQL surface – so they cannot drift. SHACL is one possible source
(see the mapping below), not a dependency: a hand-written declaration is just as valid.

Expand Down Expand Up @@ -102,15 +102,15 @@ interface SearchField {
}

// One root type (one SHACL NodeShape); a whole deployment’s declaration is the
// SearchSchema, a map of SearchTypes keyed by type IRI. searchSchema() validates every
// SearchSchema, a map of SearchTypes keyed by class IRI. searchSchema() validates every
// declaration (the declaration-time counterpart of the port’s assertValidQuery) and the
// type is BRANDED (nominal): searchSchema() is the only constructor, so consumers never
// re-validate hand-built maps.
interface SearchType {
readonly name: string; // logical API name ('Dataset') – names the type in every surface,
// declared (like each field's name), never derived from the IRI, so vocabulary
// churn cannot silently rename the public contract
readonly type: string; // sh:targetClass
readonly class: string; // sh:targetClass
readonly fields: readonly SearchField[];
}
```
Expand All @@ -122,7 +122,7 @@ eventual generator emits it unchanged. A field with **no `path`** is a derived f
computed by its `derive` function rather than projected from the IR – yet it still carries full
query/schema/output behavior. The physical field names a declaration fans out to (`${name}_search_${locale}`,
`${name}_sort_${locale}`, `${name}_search`) follow one convention owned by
`@lde/search`, so projection, collection schema and query compiler agree. The `status_rank`
`@lde/search`, so projection, collection definition and query compiler agree. The `status_rank`
tie-break sort is a **deployment-specific delta**, never in `@lde/search`. Grouped facets need
no field-model mechanism at all: a deployment `derive` function materializes group tokens (e.g.
`group:rdf`) into the field’s own values – see Consequences. `relevance` is _not_ a delta:
Expand Down
2 changes: 1 addition & 1 deletion docs/decisions/0004-search-api-graphql-surface.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ filter inputs, reference types) are created once and reused across root types.

```ts
function buildGraphQLSchema(
schema: SearchSchema, // every root type, keyed by type IRI
schema: SearchSchema, // every root type, keyed by class IRI
options?: {
types?: Record<
string, // SearchType.name (the logical API name); entries are optional fine-tuning
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ collection to search by label.
The Dataset Register declares its labels data as a `SearchType` (e.g. per
entity kind), adds its collection(s) to `collections`, and sets
`labelSource` on each reference field. Label collections are rebuilt via
`buildCollectionSchema` so the physical label fields exist.
`buildCollectionDefinition` so the physical label fields exist.
- Reference facet buckets are labelled from the facet field’s own source,
exactly as hit references are.
- The facet-by-name typeahead (issue #534, with #533’s value-query IR) gets
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ several ([#590](https://github.com/ldelements/lde/issues/590)).
One validated `SearchSchema` declares several root types. The Dataset Register
indexes four: `datasets` plus the Organization / Class / TerminologySource label
collections its references resolve against (ADR 8). Each type derives its own
Typesense collection schema, so the types cannot share a collection – they are
Typesense collection definition, so the types cannot share a collection – they are
four independent blue/green rebuilds, each with its own versioned collection,
alias and single-flight lock.

Expand Down
18 changes: 9 additions & 9 deletions packages/search-api-graphql/test/build-schema.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ import { buildGraphQLSchema, type SearchContext } from '../src/build-schema.js';

const schema: SearchType = {
name: 'Dataset',
type: 'http://www.w3.org/ns/dcat#Dataset',
class: 'http://www.w3.org/ns/dcat#Dataset',
fields: [
{
name: 'title',
Expand Down Expand Up @@ -590,7 +590,7 @@ describe('buildGraphQLSchema', () => {
describe('multiple root types in one schema', () => {
const PERSON: SearchType = {
name: 'Person',
type: 'https://schema.org/Person',
class: 'https://schema.org/Person',
fields: [
{
name: 'name',
Expand All @@ -611,7 +611,7 @@ describe('buildGraphQLSchema', () => {
};
const CREATIVE_WORK: SearchType = {
name: 'CreativeWork',
type: 'https://schema.org/CreativeWork',
class: 'https://schema.org/CreativeWork',
fields: [
{
name: 'title',
Expand Down Expand Up @@ -659,7 +659,7 @@ describe('buildGraphQLSchema', () => {
const engine: SearchEngine = {
schema: searchSchema(PERSON, CREATIVE_WORK),
async search(searchType: SearchType): Promise<SearchResult> {
searchedTypes.push(searchType.type);
searchedTypes.push(searchType.class);
return { total: 0, hits: [], facets: {} };
},
searchFacets: noFacets,
Expand All @@ -670,7 +670,7 @@ describe('buildGraphQLSchema', () => {
contextValue: { engine, acceptLanguage: ['nl'] },
});
expect(result.errors).toBeUndefined();
expect(searchedTypes).toEqual([PERSON.type, CREATIVE_WORK.type]);
expect(searchedTypes).toEqual([PERSON.class, CREATIVE_WORK.class]);
});

it('builds without any options: names come from the search types', () => {
Expand All @@ -686,7 +686,7 @@ describe('buildGraphQLSchema', () => {
it('throws when a reference type name collides with a root type name', () => {
const withCollidingRef: SearchType = {
name: 'CreativeWork',
type: 'https://schema.org/CreativeWork',
class: 'https://schema.org/CreativeWork',
fields: [
{
name: 'author',
Expand All @@ -705,7 +705,7 @@ describe('buildGraphQLSchema', () => {
it('rejects a duplicate root type name at declaration time', () => {
const alsoPerson: SearchType = {
name: 'Person',
type: 'https://example.org/OtherPerson',
class: 'https://example.org/OtherPerson',
fields: [{ name: 'name', kind: 'keyword', output: true }],
};
// searchSchema() rejects the duplicate; SearchSchema is branded, so a
Expand All @@ -714,7 +714,7 @@ describe('buildGraphQLSchema', () => {
/Duplicate search type name “Person”/,
);
// @ts-expect-error — a hand-built map is not a (branded) SearchSchema
void (() => buildGraphQLSchema(new Map([[PERSON.type, PERSON]])));
void (() => buildGraphQLSchema(new Map([[PERSON.class, PERSON]])));
});

it('throws on options for an unknown type and on a root-field clash', () => {
Expand All @@ -741,7 +741,7 @@ describe('und-locale text output', () => {
it('serves untagged text as a LanguageString list with a null language', async () => {
const doc: SearchType = {
name: 'Doc',
type: 'urn:example:Doc',
class: 'urn:example:Doc',
fields: [
{ name: 'summary', kind: 'text', locales: ['und'], output: true },
],
Expand Down
2 changes: 1 addition & 1 deletion packages/search-api-graphql/test/facet-batch.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import { createFacetLoader, groupFacetQueries } from '../src/facet-batch.js';

const dataset = {
name: 'Dataset',
type: 'http://www.w3.org/ns/dcat#Dataset',
class: 'http://www.w3.org/ns/dcat#Dataset',
fields: [
{ name: 'keyword', kind: 'keyword', facetable: true, filterable: true },
{ name: 'status', kind: 'keyword', facetable: true, filterable: true },
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import { printGraphQLSchema } from '../src/build-schema.js';
*/
const THING: SearchType = {
name: 'Thing',
type: 'https://example.org/Thing',
class: 'https://example.org/Thing',
fields: [
{
name: 'title',
Expand Down
4 changes: 2 additions & 2 deletions packages/search-pipeline/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ import { searchIndexWriter } from '@lde/search-pipeline';
const schema = searchSchema(
{
name: 'Dataset',
type: 'http://www.w3.org/ns/dcat#Dataset',
class: 'http://www.w3.org/ns/dcat#Dataset',
fields: [
{
name: 'title',
Expand All @@ -46,7 +46,7 @@ const schema = searchSchema(
},
{
name: 'Organization',
type: 'http://xmlns.com/foaf/0.1/Organization',
class: 'http://xmlns.com/foaf/0.1/Organization',
fields: [
{
name: 'label',
Expand Down
6 changes: 3 additions & 3 deletions packages/search-pipeline/src/search-index-writer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ export function searchIndexWriter(
// by the type IRI, which is also how a projected document names its type.
const writers = new Map<string, Writer<SearchDocument>>(
[...schema.values()].map((searchType) => [
searchType.type,
searchType.class,
writerFor(searchType),
]),
);
Expand Down Expand Up @@ -121,9 +121,9 @@ export function searchIndexWriter(
pass.quads,
schema,
)) {
const documents = byType.get(searchType.type);
const documents = byType.get(searchType.class);
if (documents === undefined) {
byType.set(searchType.type, [document]);
byType.set(searchType.class, [document]);
} else {
documents.push(document);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,12 +27,12 @@ const NAME = 'https://example.org/name';
const schema = searchSchema(
{
name: 'Dataset',
type: DATASET,
class: DATASET,
fields: [{ name: 'title', kind: 'keyword', path: TITLE, array: true }],
},
{
name: 'Organization',
type: ORGANIZATION,
class: ORGANIZATION,
fields: [{ name: 'name', kind: 'keyword', path: NAME, array: true }],
},
);
Expand Down Expand Up @@ -78,7 +78,7 @@ function blueGreenFor(
): (searchType: SearchType) => Writer<SearchDocument> {
return (searchType) =>
new BlueGreenRebuild(client, searchType, {
name: COLLECTION[searchType.type],
name: COLLECTION[searchType.class],
});
}

Expand Down Expand Up @@ -167,7 +167,7 @@ describe('searchIndexWriter over multiple Typesense collections', () => {
searchType,
) => {
const real = blueGreenFor(client)(searchType);
if (searchType.type !== ORGANIZATION) {
if (searchType.class !== ORGANIZATION) {
return real;
}
return {
Expand Down
14 changes: 7 additions & 7 deletions packages/search-pipeline/test/search-index-writer.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,12 @@ const NAME = 'https://example.org/name';
const schema = searchSchema(
{
name: 'person',
type: PERSON,
class: PERSON,
fields: [{ name: 'name', kind: 'keyword', path: NAME }],
},
{
name: 'work',
type: WORK,
class: WORK,
fields: [{ name: 'name', kind: 'keyword', path: NAME }],
},
);
Expand Down Expand Up @@ -94,7 +94,7 @@ function makeFleet(
abort: vi.fn<(error: unknown) => Promise<void>>().mockResolvedValue(),
openRunCalls: 0,
};
collections.set(searchType.type, collection);
collections.set(searchType.class, collection);

const runWriter: RunWriter<SearchDocument> = {
write: async (written, documents) => {
Expand Down Expand Up @@ -276,7 +276,7 @@ describe('searchIndexWriter', () => {
const failure = new Error('person rollback failed');
const fleet = makeFleet({
flush: (searchType) =>
searchType.type === PERSON
searchType.class === PERSON
? Promise.reject(failure)
: Promise.resolve(),
});
Expand All @@ -298,7 +298,7 @@ describe('searchIndexWriter', () => {
const failure = new Error('person reset failed');
const fleet = makeFleet({
reset: (searchType) =>
searchType.type === PERSON
searchType.class === PERSON
? Promise.reject(failure)
: Promise.resolve(),
});
Expand Down Expand Up @@ -356,7 +356,7 @@ describe('searchIndexWriter', () => {
const failure = new Error('work collection swap failed');
const fleet = makeFleet({
commit: (searchType) =>
searchType.type === WORK ? Promise.reject(failure) : Promise.resolve(),
searchType.class === WORK ? Promise.reject(failure) : Promise.resolve(),
});
const run = await openRun(fleet);

Expand All @@ -372,7 +372,7 @@ describe('searchIndexWriter', () => {
const failure = new Error('work collection swap failed');
const fleet = makeFleet({
commit: (searchType) =>
searchType.type === WORK ? Promise.reject(failure) : Promise.resolve(),
searchType.class === WORK ? Promise.reject(failure) : Promise.resolve(),
});
const run = await openRun(fleet);

Expand Down
12 changes: 6 additions & 6 deletions packages/search-typesense/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,14 @@
domain-agnostic [`@lde/search`](../search) core. **Engine-specific (Typesense) but
domain-agnostic** – you supply a `SearchType`; this package never names your
domain. It is the Typesense implementation of the `SearchEngine` port: it derives
a collection schema from the field model, compiles the neutral `SearchQuery` into
a collection definition from the field model, compiles the neutral `SearchQuery` into
Typesense search params, runs it, reconstructs the engine-neutral `SearchResult`,
and manages the search index lifecycle as transactional `@lde/pipeline`
writers (Blue/green Rebuild and In-place Rebuild).

## Collection schema and engine

`buildCollectionSchema(searchType, { name, defaultSortingField, … })` derives a
`buildCollectionDefinition(searchType, { name, defaultSortingField, … })` derives a
Typesense collection from the unified `SearchField` model — the Typesense field
type comes from each field’s `kind`, and the physical fanout (per-locale
search/sort keys) matches what the projection writes, via
Expand All @@ -20,7 +20,7 @@ search/sort keys) matches what the projection writes, via
**Memory lever.** Typesense keeps the index in RAM (with a raw copy of each
document on disk), so RAM tracks the _indexed_ surface – roughly 2–3× the size
of the fields you search, facet or sort on – not the full document.
`buildCollectionSchema` keeps that surface minimal: the `output` display labels
`buildCollectionDefinition` keeps that surface minimal: the `output` display labels
fan out to `index: false` fields, stored on disk and fetched only for a hit, so
they cost no RAM; only the folded `*_search_${locale}`, facet/reference and
`*_sort_${locale}` companions are indexed. Keeping retrieval-only fields
Expand Down Expand Up @@ -67,9 +67,9 @@ Indexing runs through two transactional writers, one per update mode – the
Both implement `@lde/pipeline`’s `Writer` – each run is
`openRun(context)` → `write` per dataset → `commit()` or `abort(error)` – so
an `@lde/pipeline` `Pipeline` drives them without branching on the mode. Both
derive the collection schema from your `SearchType` (via
`buildCollectionSchema`), and their options accept everything
`buildCollectionSchema` does (`defaultLocale`, `defaultSortingField`,
derive the collection definition from your `SearchType` (via
`buildCollectionDefinition`), and their options accept everything
`buildCollectionDefinition` does (`defaultLocale`, `defaultSortingField`,
`synonymSets`) plus the tuning knobs (`batchSize`, `lockTtlMs`).

### Blue/green Rebuild
Expand Down
18 changes: 10 additions & 8 deletions packages/search-typesense/src/blue-green-rebuild.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import type {
Writer,
} from '@lde/pipeline';
import type { Dataset } from '@lde/dataset';
import { buildCollectionSchema } from './collection-schema.js';
import { buildCollectionDefinition } from './collection-definition.js';
import { BatchImporter } from './import.js';
import { httpStatus, openLockedRun, releaseLock } from './lock.js';
import { SOURCE_FIELD, sourceDocumentsFilter } from './sweep.js';
Expand All @@ -19,7 +19,7 @@ import {
type RebuildOptions,
} from './rebuild-support.js';

/** {@link BlueGreenRebuild} options: the collection-schema options (`name` is
/** {@link BlueGreenRebuild} options: the collection-definition options (`name` is
* the logical index name the alias is kept on) plus the rebuild tuning knobs. */
export type BlueGreenRebuildOptions = RebuildOptions;

Expand Down Expand Up @@ -63,22 +63,24 @@ export class BlueGreenRebuild<
}

async openRun(context: RunContext): Promise<RunWriter<TDocument>> {
const { name, batchSize, lockTtlMs, schemaOptions } = resolveRebuildOptions(
this.options,
);
const { name, batchSize, lockTtlMs, definitionOptions } =
resolveRebuildOptions(this.options);

return openLockedRun(this.client, name, lockTtlMs, async () => {
// Create the fresh (blue) collection up front, so a failure surfaces
// before any dataset is processed. startedAt orders the versioned names;
// concurrent same-name runs are excluded by the lock.
const collection = `${name}_${Date.parse(context.startedAt)}`;
const previous = await this.aliasTarget(name);
const schema = buildCollectionSchema(this.searchType, schemaOptions);
const definition = buildCollectionDefinition(
this.searchType,
definitionOptions,
);
await this.client.collections().create({
...schema,
...definition,
name: collection,
fields: [
...(schema.fields ?? []),
...(definition.fields ?? []),
{ name: SOURCE_FIELD, type: 'string' },
],
});
Expand Down
Loading