Cold dataset open takes 6-7s on network-attached volumes (Railway) — consider dataset/manifest caching

[bobsch-io]

Hi Andrew! 👋 Bobsch here.

Quick disclaimer up front: I honestly have no clue whether this is a real issue on your side — my AI assistant (Fable 5) pinned this down while we were chasing latency in our app and suggested reporting it upstream. Sharing it in case it's helpful; feel free to close if it's expected behavior.

Setup: omnigraph-server v0.7.0, cluster mode (--cluster), deployed on Railway with a network-attached volume at /var/lib/omnigraph/cluster. Cedar policy enabled, bearer-token auth.

Measurements from real production use:

Operation	Network volume (Railway)	Local disk
First read after a branch create/open (cold)	6–7 s	milliseconds
Warm reads	80–200 ms	~10 ms
Branch merge	~1.4 s	fast
Network RTT (keep-alive)	~23 ms	—

The wire is innocent (23 ms RTT) — the cost profile points at many small Lance/manifest file operations against the network volume on a cold dataset open. Our workaround is a client-side keep-warm sweep so humans never hit the cold path, which works fine, but it treats the symptom.

Suggestion: cache opened datasets/manifests in memory server-side (or batch the small reads on open), so cold opens on network-attached storage don't multiply the per-file latency.

Happy to provide detailed timings or logs if useful. Thanks for Omnigraph — the branch/merge model is carrying our whole human-in-the-loop workflow. 🙏

[aaltshuler]

Thanks for the detailed measurements, Bobsch — the RTT-vs-cost-profile breakdown made this easy to place.

Good news: what you're describing (and the fix you suggest) shipped in the releases after v0.7.0. The 6–7s cold profile on network-attached storage came from an O(commit-history) metadata re-derivation on the read path that never warmed up — many small manifest/dataset reads multiplied by your volume's per-file latency.

• v0.7.1 (perf(engine): remove the per-query metadata re-derivation tax on warm reads #268) — server-side caching of exactly the kind you propose: warm coordinator reuse behind a cheap version probe, datasets opened by location+version, held Dataset handles, and one shared Lance Session per graph. A warm same-branch read now does one probe and zero dataset opens on repeat. This is your keep-warm sweep, built in — you should be able to delete the workaround.
• v0.7.2 (write-path cost gate + opener bypass #288/feat(engine): WriteTxn - validate schema + open each data table once per write #298) — write-path latency on object stores cut ~70% (each touched table opened once per write), and optimize now compacts the internal manifest table so metadata scans stay flat as history grows ((feat): compact the internal manifest/commit-graph tables in optimize #291).
• v0.8.0 (perf(engine): halve per-write __manifest scans #307, perf(engine): scope CSR topology index to traversed edges, reuse it cross-branch #312) — per-write manifest scans halved; the graph topology index is scoped to the edges a query traverses and reused across branches, which helps branch-heavy workflows like yours specifically.

The first cold touch of a graph after boot still pays real round trips (bounded by working set now, no longer multiplied by history), so a freshly booted server won't be milliseconds on first read — but it should be nowhere near 6–7s, and everything after that stays warm server-side.

One caveat: your ~1.4s branch merge number touches a separate, still-open cost term (a cross-branch manifest read that grows with commit depth). That one is tracked and pinned by a cost test in the repo; upgrading won't fully fix it yet.

Recommendation: upgrade to v0.8.0 and re-measure on the same Railway setup. If cold/warm reads are still off after that, please reopen with fresh numbers — timings from a real network-attached deployment like yours are genuinely useful. Thanks again!