Skip to content

Serialize RDF onto response streams through one source - #453

Merged
haed merged 2 commits into
mainfrom
haed-rdf-source-streaming
Nov 16, 2026
Merged

haed merged 2 commits into
mainfrom
haed-rdf-source-streaming

Conversation

@haed

@haed haed commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

What this changes

Six places serialized RDF onto an output stream with the same hand-written Rio loop: find, the SPARQL CONSTRUCT/DESCRIBE path (twice), and the N-Quads reads of resources, resources by IRI and the context registry. They now go through RdfSource (sempods-server, org.sempods.rdf.stream): a producer that pushes statements into an RDFHandler, with two sinks, write(format, out) and toStreamingOutput(format). No issue; a refactor that came out of a review of FindEndpoint.

Why

Streaming will matter more, on both sides. The loops had already drifted: the N-Quads reads closed the container's response stream, the others did not. Push rather than Iterable, so a producer owns what it reads from: SparqlQueryService.graphSource() opens the pod connection only while the body is written. That is the shape a streaming producer for find can take later without the endpoint changing again.

Sources carry statements only. Rio.write(model, …) and GraphQuery.evaluate(handler) both hand the writer namespaces too, so neither is used, and every response body stays byte for byte what it was.

Not covered: the canonical JSON-LD shapes (toCanonicalJsonLdEntry, toJsonLdNamedGraphs) build maps grouped by subject and context and need the whole model; the resource reads also hash the full model for their ETag first. RDF4J's JSON-LD writer buffers until endRDF(), so only N-Quads streams incrementally.

Public API. RdfSource lives in sempods-server, which already declares rio-api on api; in sempods-model it would have widened that module's published contract. SparqlQueryService gains graphSource() and loses executeGraph(…, out), executeGraphAsString() and executeTupleAsJson(), which had no caller left. RdfWriterUtil shares one loop internally and loses streamNQuads, writeJsonLd, toJsonLdGraph and the map-taking readJsonLd, likewise unused. Those were the only JSON code in sempods-model, so it no longer depends on sempods-commons-json or Jackson; a consumer that reached Jackson through it has to declare it.

Verification and documentation

  • ./gradlew test checkNoLoggingBinding checkNoTestLibrariesInPom checkNoMissingPomDependencies checkDocLinks checkImageMetadata checkPublishedSignatures buildHealth — green, 3449 tests, 0 failures (local Mongo up)
  • buildHealth — no advice for :sempods-server or :sempods-model; only the existing duplicate-class warnings elsewhere
  • New tests: RdfSourceTest (round-trip with contexts, stream left open, streaming entity writes the same document) and SparqlQueryServiceTest (empty-store CONSTRUCT, graph source reaches the pod only when written)
  • doc-review: no documentation describes these code paths, and no HTTP representation changed, so none needed updating

Before requesting review

  • Every commit is signed off — git commit -s, or dco fails
  • ./gradlew test passes
  • ./gradlew buildHealth passes — required whenever a dependency or a
    public signature moved
  • ./gradlew checkDocLinks passes — every relative link in a markdown file
    still resolves
  • doc-review ran on this PR — its updates, or the reason none were
    needed, are listed above
  • A model that did substantial work here is named in a Co-Authored-By:
    trailer, and the change is one I can defend in review

🤖 Generated with Claude Code

Six places serialized statements onto an output stream with the same
hand-written Rio loop: find, the SPARQL CONSTRUCT/DESCRIBE path (twice),
and the three N-Quads reads of resources and the context registry. They
had drifted apart: the N-Quads reads closed the container's response
stream, the others left it open.

RdfSource is a producer that pushes statements into an RDFHandler, with
two sinks: write() onto a stream, and toStreamingOutput() for a response
body. Push rather than an Iterable, so the producer owns what it reads
from. SparqlQueryService.graphSource() opens the pod connection only
while the body is written, which is the shape a streaming producer for
find can take later without the endpoint changing again.

Sources carry statements only. Neither Rio.write(model, ...) nor
GraphQuery.evaluate(handler) is used, because both hand the writer
namespaces as well, and the bytes a client receives stay as they were.

RdfSource lives in sempods-server, which already declares rio-api on
api; in sempods-model it would have widened that module's published
contract. RdfWriterUtil keeps its public functions and shares one loop
internally. executeGraph(out) and executeGraphAsString had no caller
left and are removed.

No documentation describes these code paths; the change keeps every
HTTP representation as it was.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: dast <der.danilo.stein@gmail.com>
@chatgpt-codex-connector

ghost commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T11:57:20.878185Z 562e013 New commits
🔒 Security Review ✅ Completed 2026-11-16T11:11:24.932355Z e98fb2a PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

With the streaming source in place, several helpers had no caller left:
RdfWriterUtil.streamNQuads, writeJsonLd, toJsonLdGraph and the
map-taking readJsonLd, SparqlQueryService.executeTupleAsJson, and the
SempodsUriBuilder that FindEndpoint was injected with but never used.

They were the only code in sempods-model that needed JSON: the module
now depends on neither sempods-commons-json nor Jackson, and its tests
declare the two for the expected-JSON fixtures they parse. A consumer
that reached Jackson through sempods-model has to declare it.

The groupBy calls in toJsonLdNamedGraphs work on the Model directly;
the intermediate sequence bought nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: dast <der.danilo.stein@gmail.com>
@haed
haed merged commit 4e6414f into main Nov 16, 2026
@haed
haed deleted the haed-rdf-source-streaming branch November 16, 2026 12:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant