Skip to content

Commit 464d20e

Browse files
author
NestGate Team
committed
feat: streaming storage + cross-spring namespace isolation on isomorphic IPC
Wire storage.store_blob, storage.retrieve_blob, storage.retrieve_range (4 MiB chunked base64), storage.object.size, and storage.namespaces.list into the modern isomorphic IPC adapter. Evolve StorageState to family-scoped namespace model: {base}/datasets/{family_id}/{namespace}/{key}.json. All storage.* methods accept optional namespace parameter (default "shared" for cross-spring access). Resolves both primalSpring upstream gaps: large tensor retrieval and cross-spring persistent storage IPC. 11,816 tests pass. Made-with: Cursor
1 parent 70ed5d1 commit 464d20e

File tree

8 files changed

+688
-173
lines changed

8 files changed

+688
-173
lines changed

CHANGELOG.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
99

1010
## [Unreleased] - 4.7.0-dev
1111

12+
### Session 43h: Streaming storage & cross-spring namespace isolation (April 13, 2026)
13+
14+
- **Streaming large-object support on isomorphic IPC**: `storage.store_blob`, `storage.retrieve_blob`,
15+
`storage.retrieve_range` (4 MiB chunked base64), and `storage.object.size` wired into the modern
16+
isomorphic IPC adapter — resolves primalSpring upstream gap for large tensor retrieval.
17+
- **Cross-spring namespace isolation**: All `storage.*` methods now accept an optional `namespace`
18+
parameter. Default namespace is `"shared"` (cross-spring accessible). Springs can use private
19+
namespaces for isolation. Directory layout: `{base}/datasets/{family_id}/{namespace}/{key}.json`.
20+
- **`storage.namespaces.list`**: New method to enumerate available namespaces within a family.
21+
- **`StorageState` evolved**: Family-scoped (`NESTGATE_FAMILY_ID` / `FAMILY_ID` / `BIOMEOS_FAMILY_ID`
22+
env resolution), namespace directory model, blob subdirectory, segment validation.
23+
- **11 new tests**: blob store/retrieve roundtrip, chunked range reassembly, object.size,
24+
namespace isolation, shared namespace default, namespace listing, path traversal rejection,
25+
capabilities listing.
26+
- Validation: `cargo fmt`, `cargo clippy`, `cargo doc`, `cargo test` — all PASS (11,816 tests, 0 failures).
27+
1228
### Session 43g: Deep debt evolution — error types & dead code (April 13, 2026)
1329

1430
- **`Box<dyn Error>` evolved**: 5 production function signatures (`websocket.rs`, `probes.rs`×3,

CONTEXT.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ than by importing this crate graph.
2929
| **Unsafe** | `#![forbid(unsafe_code)]` on ALL crate roots (zero exceptions) |
3030
| **Lint / format** | `cargo clippy --workspace --all-targets --all-features -- -D warnings` zero warnings (pedantic + nursery); `cargo fmt --check` clean |
3131
| **Docs** | `cargo doc --workspace --no-deps` — clean in routine runs |
32-
| **Tests** | `cargo test --workspace` — 11,805 passing, 451 ignored, 0 failures (see STATUS.md) |
32+
| **Tests** | `cargo test --workspace` — 11,816 passing, 451 ignored, 0 failures (see STATUS.md) |
3333
| **Coverage** | ~81.7% line (llvm-cov) — wateringHole 80% met; 90% target pending |
3434
| **Platforms** | Linux, FreeBSD, macOS, WSL2, illumos, Android |
3535
| **Specs** | 16 specification documents under `specs/` |

README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111
- **Supply chain**: `cargo deny check bans` — PASS; `cargo tree -i ring` — no matches
1212

1313
**Metrics** (re-measure as needed; see [STATUS.md](./STATUS.md))
14-
- **Tests (last recorded)**: 11,805 passing, 451 ignored, 0 failures — run `cargo test --workspace` to refresh counts
14+
- **Tests (last recorded)**: 11,816 passing, 451 ignored, 0 failures — run `cargo test --workspace` to refresh counts
1515
- **Coverage**: ~81.7% line (`cargo llvm-cov --workspace --lib`; wateringHole minimum 80% met; org target 90% pending)
1616

1717
**Technical debt (honest)**
@@ -132,7 +132,7 @@ See [STATUS.md](./STATUS.md) for measured metrics. Verified as of 2026-04-13 (Se
132132
| Build | `cargo check --workspace --all-features --all-targets` — PASS |
133133
| Clippy | `cargo clippy --workspace --all-targets --all-features -- -D warnings` — PASS (zero warnings) |
134134
| Format | `cargo fmt --all --check` — PASS |
135-
| Tests | `cargo test --workspace` — 11,805 passing, 0 failures, 451 ignored |
135+
| Tests | `cargo test --workspace` — 11,816 passing, 0 failures, 451 ignored |
136136
| Coverage | ~81.7% line (llvm-cov) — wateringHole 80% met; 90% target pending |
137137
| Docs | `cargo doc --workspace --no-deps` — zero warnings |
138138
| Deprecated | 193 `#[deprecated]` for canonical migration; zero dead callers |

STATUS.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# NestGate - Current Status
22

3-
**Last Updated**: April 13, 2026 (Session 43gerror types & dead code evolution)
3+
**Last Updated**: April 13, 2026 (Session 43hstreaming storage & namespace isolation)
44
**Version**: 4.7.0-dev
55

66
---
@@ -12,7 +12,7 @@ Build: PASS — cargo check --workspace --all-features --all-target
1212
Clippy: PASS — cargo clippy --workspace --all-targets --all-features -- -D warnings (zero errors), as of 2026-04-13
1313
Format: CLEAN (cargo fmt --check passes), as of 2026-04-13
1414
Docs: PASS — cargo doc --workspace --no-deps (zero warnings), as of 2026-04-13
15-
Tests: 11,805 passing, 0 failures, 451 ignored (cargo test --workspace; flaky tests stabilized)
15+
Tests: 11,816 passing, 0 failures, 451 ignored (cargo test --workspace; flaky tests stabilized)
1616
Coverage: ~81.7% line (cargo llvm-cov --workspace --lib) — wateringHole 80% min met; 90% target pending
1717
Files > 750 lines: 0 (production; 9 largest refactored Sessions 43–43f — max 749 LOC; engine/gcs/azure/pool_setup all under 500)
1818
Unwrap/Expect: ZERO in production library code

capability_registry.toml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,11 +17,12 @@ protocol = "jsonrpc-2.0"
1717

1818
[capabilities.storage]
1919
domain = "storage"
20-
description = "Filesystem-backed durable key-value and blob storage"
20+
description = "Filesystem-backed durable key-value and blob storage with namespace isolation"
2121
methods = [
2222
"storage.store", "storage.retrieve", "storage.exists",
2323
"storage.delete", "storage.list", "storage.stats",
2424
"storage.store_blob", "storage.retrieve_blob", "storage.retrieve_range",
25+
"storage.object.size", "storage.namespaces.list",
2526
"storage.fetch_external",
2627
]
2728

code/crates/nestgate-rpc/src/rpc/isomorphic_ipc/unix_adapter/mod.rs

Lines changed: 99 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -60,14 +60,28 @@ struct JsonRpcError {
6060
/// Match `nestgate_core::nat_traversal::BEACON_DATASET` when core is linked.
6161
const BEACON_DATASET: &str = "_known_beacons";
6262

63-
/// Filesystem-backed storage state.
63+
/// Default namespace for cross-spring shared storage.
64+
const DEFAULT_NAMESPACE: &str = "shared";
65+
66+
/// Filesystem-backed storage state with family-scoped namespace isolation.
67+
///
68+
/// Directory layout:
69+
/// ```text
70+
/// {base}/datasets/{family_id}/{namespace}/{key}.json (JSON values)
71+
/// {base}/datasets/{family_id}/{namespace}/_blobs/{key} (binary blobs)
72+
/// ```
6473
///
65-
/// Keys are stored as individual JSON files under `{base}/datasets/default/{key}.json`.
66-
/// This replaces the former in-memory `HashMap` so IPC storage survives restarts.
74+
/// - `family_id` is resolved from env (`NESTGATE_FAMILY_ID` / `FAMILY_ID` / `BIOMEOS_FAMILY_ID`),
75+
/// defaulting to `"default"`.
76+
/// - `namespace` isolates each caller (spring). The `"shared"` namespace is the cross-spring
77+
/// meeting point, readable/writable by all springs in the family. Omitting `namespace` from
78+
/// request params defaults to `"shared"` for backward compatibility.
6779
#[derive(Clone)]
6880
pub(super) struct StorageState {
69-
/// Root directory for the default dataset.
70-
dataset_dir: PathBuf,
81+
/// Root directory for this family: `{base}/datasets/{family_id}`.
82+
family_dir: PathBuf,
83+
/// The resolved family identifier.
84+
family_id: String,
7185
#[expect(
7286
dead_code,
7387
reason = "Template storage wired when template RPC handlers are enabled"
@@ -80,43 +94,81 @@ pub(super) struct StorageState {
8094
audits: crate::rpc::audit_storage::AuditStorage,
8195
}
8296

97+
/// Resolve the family identifier from environment, with cascading fallback.
98+
fn resolve_family_id() -> String {
99+
std::env::var("NESTGATE_FAMILY_ID")
100+
.or_else(|_| std::env::var("FAMILY_ID"))
101+
.or_else(|_| std::env::var("BIOMEOS_FAMILY_ID"))
102+
.unwrap_or_else(|_| "default".to_string())
103+
}
104+
83105
impl StorageState {
84106
fn new() -> Result<Self> {
85-
let dataset_dir = get_storage_base_path().join("datasets").join("default");
86-
std::fs::create_dir_all(&dataset_dir)?;
107+
let family_id = resolve_family_id();
108+
let family_dir = get_storage_base_path().join("datasets").join(&family_id);
109+
let shared_dir = family_dir.join(DEFAULT_NAMESPACE);
110+
std::fs::create_dir_all(&shared_dir)?;
87111
Ok(Self {
88-
dataset_dir,
112+
family_dir,
113+
family_id,
89114
templates: crate::rpc::template_storage::TemplateStorage::new(),
90115
audits: crate::rpc::audit_storage::AuditStorage::new(),
91116
})
92117
}
93118

94-
/// Sanitize a key to a safe filename (reject path traversal).
95-
fn key_path(&self, key: &str) -> std::result::Result<PathBuf, (i32, Cow<'static, str>)> {
96-
if key.is_empty()
97-
|| key.contains('/')
98-
|| key.contains('\\')
99-
|| key.contains("..")
100-
|| key.starts_with('.')
119+
/// Validate a name segment (key or namespace) — reject path traversal.
120+
fn validate_segment(
121+
name: &str,
122+
field: &'static str,
123+
) -> std::result::Result<(), (i32, Cow<'static, str>)> {
124+
if name.is_empty()
125+
|| name.contains('/')
126+
|| name.contains('\\')
127+
|| name.contains("..")
128+
|| name.starts_with('.')
101129
{
102-
return Err((-32602, Cow::Borrowed("Invalid key: must be a simple name")));
130+
return Err((
131+
-32602,
132+
Cow::Owned(format!("Invalid {field}: must be a simple name")),
133+
));
103134
}
104-
Ok(self.dataset_dir.join(format!("{key}.json")))
135+
Ok(())
136+
}
137+
138+
/// Resolve a namespace directory, creating it on first access.
139+
fn namespace_dir(&self, namespace: &str) -> PathBuf {
140+
self.family_dir.join(namespace)
141+
}
142+
143+
/// Sanitize a key to a safe filename within a namespace.
144+
fn key_path(
145+
&self,
146+
namespace: &str,
147+
key: &str,
148+
) -> std::result::Result<PathBuf, (i32, Cow<'static, str>)> {
149+
Self::validate_segment(namespace, "namespace")?;
150+
Self::validate_segment(key, "key")?;
151+
Ok(self.namespace_dir(namespace).join(format!("{key}.json")))
152+
}
153+
154+
/// Blob storage path within a namespace.
155+
fn blob_path(
156+
&self,
157+
namespace: &str,
158+
key: &str,
159+
) -> std::result::Result<PathBuf, (i32, Cow<'static, str>)> {
160+
Self::validate_segment(namespace, "namespace")?;
161+
Self::validate_segment(key, "key")?;
162+
Ok(self.namespace_dir(namespace).join("_blobs").join(key))
105163
}
106164

107165
/// NAT traversal info is stored under the `_nat` sub-directory.
108166
fn nat_dir(&self) -> PathBuf {
109-
self.dataset_dir
110-
.parent()
111-
.unwrap_or(&self.dataset_dir)
112-
.join("_nat")
167+
self.family_dir.join("_nat")
113168
}
114169

115170
fn beacon_dir(&self) -> PathBuf {
116-
self.dataset_dir
117-
.parent()
118-
.unwrap_or(&self.dataset_dir)
119-
.join(BEACON_DATASET)
171+
self.family_dir.join(BEACON_DATASET)
120172
}
121173
}
122174

@@ -152,6 +204,21 @@ impl UnixSocketRpcHandler {
152204
"storage.list" => unix_adapter_handlers::handle_storage_list(state, &request).await,
153205
"storage.delete" => unix_adapter_handlers::handle_storage_delete(state, &request).await,
154206
"storage.exists" => unix_adapter_handlers::handle_storage_exists(state, &request),
207+
"storage.store_blob" => {
208+
unix_adapter_handlers::handle_storage_store_blob(state, &request).await
209+
}
210+
"storage.retrieve_blob" => {
211+
unix_adapter_handlers::handle_storage_retrieve_blob(state, &request).await
212+
}
213+
"storage.retrieve_range" => {
214+
unix_adapter_handlers::handle_storage_retrieve_range(state, &request).await
215+
}
216+
"storage.object.size" => {
217+
unix_adapter_handlers::handle_storage_object_size(state, &request).await
218+
}
219+
"storage.namespaces.list" => {
220+
unix_adapter_handlers::handle_storage_namespaces_list(state).await
221+
}
155222
"session.save" => unix_adapter_handlers::handle_session_save(state, &request).await,
156223
"session.load" => unix_adapter_handlers::handle_session_load(state, &request).await,
157224
"session.list" => unix_adapter_handlers::handle_session_list(state, &request).await,
@@ -173,17 +240,13 @@ impl UnixSocketRpcHandler {
173240
"capabilities.list" | "discover_capabilities" => {
174241
Ok(unix_adapter_handlers::capabilities_response())
175242
}
176-
"identity.get" => {
177-
let family_id =
178-
std::env::var("NESTGATE_FAMILY_ID").unwrap_or_else(|_| "default".to_string());
179-
Ok(json!({
180-
"primal": nestgate_config::constants::system::DEFAULT_SERVICE_NAME,
181-
"version": env!("CARGO_PKG_VERSION"),
182-
"domain": "storage",
183-
"license": "AGPL-3.0-or-later",
184-
"family_id": family_id
185-
}))
186-
}
243+
"identity.get" => Ok(json!({
244+
"primal": nestgate_config::constants::system::DEFAULT_SERVICE_NAME,
245+
"version": env!("CARGO_PKG_VERSION"),
246+
"domain": "storage",
247+
"license": "AGPL-3.0-or-later",
248+
"family_id": state.family_id
249+
})),
187250

188251
// nat.* — NAT traversal info uses its own sub-directory
189252
"nat.store_traversal_info" => {

0 commit comments

Comments
 (0)