feat(cloud): first-party OreCloud client — pip install lodedb[cloud], no separate package#90
Merged
Merged
Conversation
…dd-on pip install "lodedb[cloud]" installs the proprietary orecloud client; lodedb cloud <args> forwards untouched to its CLI, printing the install hint (exit 1) when the extra is absent. The import lives only inside the trampoline: a plain `import lodedb` stays network-free, now guarded by the import-boundary test (orecloud joins the forbidden-on-import list). The extra is deliberately NOT part of [all] — a commercial client should be an explicit choice, never a side effect. Sequencing: this cannot merge before orecloud 0.1.0 exists on PyPI — the extra is unresolvable until then (uv lock/CI fail closed on it).
idas0
marked this pull request as ready for review
July 17, 2026 23:44
LodeDB("orecloud://org/environment/store") now returns the cloud
companion's store handle (same add/search/get/remove verbs over HTTPS)
instead of requiring a separate orecloud entry point, making LodeDB(...)
the single front door for local and managed stores. The companion import
stays inside the cloud branch (install hint when absent), local-only
construction options are rejected with a targeted error, and the accepted
option set is derived from orecloud.connect's own signature so the two
cannot drift.
The settled API shape: LodeDB.cloud("user-42") is the human-facing entry
point (bare store id — org/environment resolve from the credential; the
org/environment/store triple and orecloud:// URLs work too), joining the
open_readonly/open_vector_store alternate-constructor family. The
orecloud:// dispatch in LodeDB(...) stays for config-driven cases where one
string field must express either a local path or a cloud store; both entry
points funnel through lodedb.cloud.open_cloud_target (lazy import, install
hint, connect-signature-derived option allowlist). lodedb/cloud.py is also
a PEP 562 proxy so `from lodedb.cloud import Client` gives docs a single
import root. A plain `import lodedb` still never loads the companion.
… deps-only extra Management call: no separate cloud package. The client is now first-party, following exactly the [mcp] pattern (code in the wheel, the extra installs only third-party deps): - crates/lodedb-cloud-core (was OreCloud's orecloud-core): the native transfer plane — ArtifactStore over local dirs and object storage, generation inventory/diff, managed plan/materialize, redaction policy. Its lodedb-core git-pin becomes a path dep, and building against 1.3.2 surfaced a real gap the pin had hidden: the new tvvf rescore sidecar was unknown to the inventory (fail-closed refusal), so tvvf joins STORE_KINDS and ships like tvim — rescore stores now transfer completely. - The PyO3 binding grafts into the bundled extension as the lodedb._turbovec.cloud submodule — push/pull/sync ride the same wheel as the engine; no second distribution, no version dance. - src/lodedb/cloud/ (was python/orecloud/): Client, CloudStore/connect, the transfer glue (transfer.py), the full CLI, login config, and the agent scaffolds — the lazy proxy module becomes the real package, with a PEP 562 __init__ so a plain `import lodedb` never touches httpx/pynacl (import-boundary probe now forbids those instead of orecloud). The `lodedb cloud` trampoline forwards to lodedb.cloud.cli. - cloud = ["httpx>=0.27", "pynacl>=1.5"]; orecloud leaves the lockfile. - The client test suites move in (transfer round trips through the real engine, sync lineage, CLI, connect targets, CloudStore verb wiring, the LodeDB-surface parity audit, env-file and scaffold rules) alongside the reworked seam suites — 75 cloud tests plus the crate's Rust suite.
idas0
changed the title
lodedb cloud — OreCloud as an add-onInventory correctness: - inventory tvvf by the sidecar's own epoch: the engine writes vf<vf_epoch>.tvvf (tvvf_store::base_path), not g<base_epoch>.tvvf, so the derived name failed every transfer verb on real rescore stores and the regression test had fabricated the wrong shape; journal reconstruction now uses the same naming, a real-engine rescore round trip pins it end to end, and tvvf maps to the "vector" wire kind (unknown engine kinds now fail closed instead of shipping as "state") - validate every manifest digest as 64 lowercase hex at the inventory: digests become staging file names and object keys, so a path-shaped "digest" in a remote body could otherwise escape the pull staging directory (Path(staging) / sha swallows absolute paths) Restore safety: - local restores (pull, sync pull, managed materialize) run under the engine's single-writer lock (lodedb-core newly exports engine::acquire_dir_writer_lock), so a live writer and a restore can no longer interleave and lose either side's commit - restores refuse when the destination WAL holds acknowledged records (new PendingWal error); sync --force-pull explicitly truncates them after the swap; the WAL probe path is confined to the store root (the index key can arrive from CLI or remote input) - the candidate generation is verify-opened from a hardlinked scratch layout BEFORE the pointer swap (stage/publish split of export_generation_pinned), so a checksum-consistent but semantically unopenable generation never replaces a working one; journal manifests are rebuilt strictly after the swap, and the remaining narrow crash windows are documented on restore_staged - managed materialization pins the classified local snapshot (expected_local_snapshot_id; "" pins classified-as-absent): a local commit landing between classification and materialization refuses with a sync conflict instead of being overwritten, matching the dumb sync's pinned CAS; the pull-branch WAL check is a lazy FFI call (local_wal_ops), so push and status planning never pay an O(WAL-bytes) scan Client and CLI: - native sync refusals (diverged lineage, pending WAL) exit 5 like their managed 409 twins, via the new lodedb._turbovec.cloud.SyncConflictError - push aborts its server-side session on failure instead of leaving it to expire; connect(warm=True) closes the pool on transport errors; unprovisioned search_many_by_vector keeps per-query cardinality; orecloud:// targets with empty segments are rejected instead of silently reinterpreted; token writes into an existing .env always end owner-only (0600) through a unique O_EXCL scratch file; the generated agent docs describe the SDK's actual write semantics; the README quickstart links the directory and lists keys before syncing Testing: - cargo test -p lodedb-cloud-core: 15 suites green, with new coverage for the real tvvf shape, malformed digests, writer-lock contention, the pending-WAL refusal and force-pull discard, stale-classification refusal, WAL path containment, and rejection of a semantically invalid generation - .venv/bin/python -m pytest tests/: cloud + import-boundary + core local suites green (the only failures in a slim venv are the missing embeddings extras, identical on main); repo-wide ruff clean; clippy clean
… it) The moved client suites failed CI collection with ModuleNotFoundError: nacl — the test job never installs the [cloud] extra. Two-sided fix, matching how the other optional extras are handled (mcp/PIL/mem0/cognee): the CI sync gains --extra cloud so the suites actually run there, and every suite whose module-level imports pull httpx/pynacl guards itself with pytest.importorskip so an extra-less checkout skips cleanly (the native transfer verbs and the pure-filesystem suites still run — 18 pass, 8 skip without the extra; 75 pass with it). AGENTS.md's dev sync command gains the extra too.
The Windows CI leg failed the scaffold suite: the written-paths list is built with relative_to(), which renders backslashes there, and those strings land in the CLI's JSON output — a cross-platform contract, so normalize with as_posix() rather than loosening the assertions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the OreCloud client as first-party code.
pip install "lodedb[cloud]"installs lodedb plushttpxandpynacl— the extra is a dependency list, not a second package.Changes
src/lodedb/cloud/— the Python client:Client(one credential, one bound org/environment),connect/CloudStore(duck-types the local verb surface), transfer glue, login config, agent scaffolds, and thelodedb cloudCLI. The package__init__resolves every name lazily (PEP 562), so a plainimport lodedbnever loads httpx.crates/lodedb-cloud-core— the native transfer plane (artifact stores over directories and S3-compatible object storage, generation inventory/diff, managed plan/materialize, redaction policy). Joins the workspace with a path dep onlodedb-coreinstead of a pinned git rev. The pin bump surfaced one real gap: the 1.3.2tvvfrescore sidecar was unknown to the transfer inventory (fail-closed refusal on push) —tvvfnow ships liketvim, with a regression test.lodedb._turbovec.cloud— the PyO3 binding compiled into the existing bundled extension as a submodule, so push/pull/sync ship in the same wheel as the engine.LodeDB.cloud(target, **options)(bare store id,org/environment/storetriple, ororecloud://URL; options mirrorconnect()viainspect.signature, local-only kwargs get a targeted error);LodeDB("orecloud://…")constructor dispatch for config-driven code;from lodedb.cloud import Client;lodedb cloud …. All funnel throughlodedb.cloud.open_cloud_targetand raise apip install "lodedb[cloud]"hint when the deps are absent.[cloud]extra — now["httpx>=0.27", "pynacl>=1.5"]; the import-boundary probe forbidshttpx/naclon plain import and importslodedb.clouditself in the check. Deliberately not in[all].Verification
cargo test -p lodedb-cloud-core(60+ tests incl. the new tvvf coverage) and clippy green.LodeDB.cloud(…)andLodeDB("orecloud://…")returnlodedb.cloud.serving.CloudStore; a local store pushed to a directory remote, pulled into a fresh dir, reopened, andget()returned the original text — all through the bundled native core.Follow-up (OreCloud repo, after merge + release)
The server switches its
orecloud._coreimports tolodedb._turbovec.cloud, the repo dropspython/+crates/, and theorecloud0.1.0 distribution on PyPI is retired. Egoist-Machines/OreCloud#3 was superseded by this PR and closed.