Skip to content

native.md

Latest commit

 

History

History
333 lines (292 loc) · 22.2 KB

native.md

File metadata and controls

333 lines (292 loc) · 22.2 KB
Raw
title Native architecture
description The platform-specific native bindings, fd-relative beneath model, platform mechanisms, loader security, and JavaScript fallback contract.

Native architecture

@openclaw/fs-safe uses native bindings that supply mechanisms Node does not expose directly. The Rust layer is deliberately not a second policy engine. TypeScript owns trusted-root selection, path validation, archive filtering, budgets, modes, identity fencing, cleanup decisions, and error normalization. Rust receives already-decided relative operations and performs the smallest platform syscall sequence that can preserve the boundary.

Every operation that has an equivalent safe Node implementation keeps that guarded JavaScript path. Native loading is lazy; installs do not compile Rust, run postinstall code, or fetch binaries at runtime. Seven exact-version optional packages are filtered by OS, CPU, and Linux libc, so an installation receives only its matching prebuilt binding. Native-only formats fail explicitly instead of substituting a weaker implementation.

The beneath model

A trusted directory descriptor is the capability. Native operations accept that descriptor plus a validated relative path and never reconstruct authority from a process working directory. Newly created files use exclusive creation, and TypeScript compares descriptor, pathname, and expected identities before accepting results.

Conceptually, a caller grants authority to an already-open root—not to a path string that can be reinterpreted later:

validated Root handle
  └─ relative components (untrusted)
       └─ open/link/mkdir beneath the handle
            └─ compare descriptor + pathname + expected identity

The TypeScript layer validates and decides. The native layer never decides whether a path, archive entry, mode, owner, or cleanup policy is acceptable.

  • Linux uses openat2(RESOLVE_BENEATH | RESOLVE_NO_MAGICLINKS), fd-relative mkdirat/linkat/renameat/renameat2, FICLONE, and copy_file_range. Without openat2, beneath opens use the guarded fallback.
  • macOS 15.4 and newer first use openat(O_RESOLVE_BENEATH); older kernels walk components with openat(O_NOFOLLOW) and restart in-root symlinks from the pinned root. Both routes apply an F_GETPATH post-open containment detector, but directory rename races mean the result remains best-effort, not race-atomic. macOS uses renameatx_np(RENAME_EXCL) and permits fclonefileat in an owned, non-shared parent. The clone is normalized inside a private staging directory: flags, ACLs, extended attributes, and broad mode bits are cleared before no-replace publication. Clone admission obtains mode, owner, exact identity, flags, and ACL state together from the retained descriptor; immutable receipts are compared across the private staging operation and against fresh no-follow pathname identity reads. Any extended entry on the target parent or private staging directory is rejected before cloning bytes. The payload's cleared ACL and normalized descriptor facts are verified before publication and again, with a fresh published-name identity fence, before its descriptor is returned. Unsupported admission before payload creation or an unsupported clone syscall may still select the documented ordinary-copy path. After the clone creates bytes, normalization and security verification failures report terminal EIO, retaining the original error detail; successful cleanup does not make them eligible for ordinary-copy retry. Cleanup failures remain secondary diagnostics, and already-terminal publication errors such as EEXIST retain their status.
  • Windows uses handle-relative NtCreateFile with OBJ_DONT_REPARSE and FILE_OPEN_REPARSE_POINT, then explicitly rejects reparse points. Rename and hardlink operations stay rooted in already-open handles. Owner/DACL reads use GetSecurityInfo; private directories receive their protected DACL in an exclusive, handle-relative NtCreateFile call. Their created handles remain open through ACL and pathname-association checks and own any failure cleanup. N-API descriptors cross into and out of this layer only through the host executable's paired libuv descriptor bridge; missing or partial exports fail with ENOTSUP instead of trying a raw HANDLE or add-on CRT descriptor namespace.

The internal macOS inspectDarwinAcl(fd) capability reports absent, empty, or present for the opened object's extended ACL. It synchronously borrows the caller's descriptor, preserving its file position and POSIX record locks, and never reopens a pathname. Keep the descriptor open until inspection returns. Darwin's acl_get_entry returns zero for an entry; end-of-list is accepted only for the first entry of a valid, privately owned empty ACL. Unsupported, malformed, and failed inspection is not reported as absence. These facts do not classify individual ACE permissions or prove volume ownership enforcement; each caller applies its own security policy.

Linux without openat2

Linux kernels before 5.6 and containers whose seccomp policy denies openat2 can keep native mode auto or require. The addon caches one harmless openat2(".") capability probe per process. ENOSYS, or EPERM on that probe, selects the native openat fallback. An EPERM/EACCES from an application operation is still a permission error and never triggers a retry. Install syscall filters before the first native operation; a later ENOSYS fails with ENOTSUP, rather than changing the cached mechanism during a call.

The fallback opens each directory relative to a retained descriptor with O_PATH | O_DIRECTORY | O_NOFOLLOW, compares exact device/inode/type identities, and rechecks the retained parent chain before and after the final no-follow open. It rejects all symlink components, including procfs magic links, even when the low-level caller requests symlink following. Public Root policy and canonical-path admission, hardlink rejection, pinned-file checks, and mutation identity fences remain in place. openBeneath() reports best-effort: these identity samples detect replacements but cannot make a multi-component walk atomic against a hostile process renaming directories between samples. This is the same documented containment class as the macOS and JavaScript paths; applications requiring atomic beneath resolution must check the result or use OS isolation. Rejection after a mutating open does not promise rollback.

Nested no-clobber Root.move() still admits both parents and uses renameat2(RENAME_NOREPLACE). Existing destinations are never overwritten. That separate syscall/filesystem capability remains required; if unavailable, the operation fails with helper-unavailable. Turning native mode off still disables no-clobber moves because Node has no equivalent atomic rename API.

Bounded owned-tree cleanup deliberately has no openat fallback: RESOLVE_NO_XDEV rejects bind mounts even when device numbers match, which ordinary identity checks cannot reproduce. cleanupSafety: "require-bounded" fails before workspace creation with helper-unavailable; low-level cleanup opens report ENOTSUP. Compatible cleanup retains its documented behavior. Low-level O_TMPFILE anonymous opens also fail before creation with ENOTSUP in the fallback, because named-entry identity checks cannot verify an unnamed file. Public staged-write APIs use exclusive named files and remain available.

For tests, set FS_SAFE_TEST_NO_OPENAT2=1 before starting Node to force the fallback (including bounded-cleanup refusal). It is read only at the first capability probe. It has no effect on macOS or Windows. See Linux fallback testing.

Archives

Native ZIP and TAR entry reads retain a private, unpooled input Buffer in the async reader. ZIP reuses its parsed directory; TAR retains admitted member offsets. TypeScript validates and selects the requested member before reading. The internal binding borrows that allocation: it must never be mutated or detached while the reader or a read task exists. The public API only accepts a pathname and owns this buffer exclusively. An N-API reference keeps the bytes alive; a mutex serializes access to the retained ZIP cursor. Plain TAR copies only the selected range after complete admission, while gzip, zstd, and bzip2 replay bounded decompression and validate the full physical stream. Concurrent TAR reads share immutable input and own separate decoder state. Each output owns a new vector, which N-API transfers to Node without a second payload copy on runtimes supporting external buffers. No entry-read path requires temporary-file staging. Extraction retains its private staged input.

Rust streams ZIP and TAR payloads, including gzip, zstd, and bzip2. It first returns a bounded manifest. TypeScript applies the shared path, filter, strip, mode, and byte policies and returns an index-bound extraction plan. Rust then creates only those planned entries beneath a private staging descriptor.

The fs-safe-archive-core Rust workspace crate owns TAR framing, paths, types, mode decoding, GNU metadata, and byte-counted local PAX records. The native binding and bundled WASM module compile the same source. No tar::Archive or Node TAR parser reinterprets admitted identities or sizes. Executors replay admitted payload ranges after complete bounded admission; native writes retain the platform's descriptor-relative primitives, while fallback writes retain the guarded Node staging/publication boundary. ZIP behavior is unchanged. maxMetaEntryBytes bounds bodies before allocation; unsupported global/old metadata and sparse forms fail closed. See bounded local PAX support.

The bundled module also compiles the same zstd and bzip2 codec implementations used by native. In off or missing-native auto, those decoders feed the shared TAR parser through fixed 64 KiB windows in one import-free WASM session with a 256 MiB linear-memory ceiling. Gzip retains Node's built-in decoder. Complete container and TAR admission precedes policy evaluation and guarded publication; concatenated compressed members and zstd skippable frames are consumed through physical EOF. No runtime command, interpreter, download, or consumer compilation is needed for these archive routes. require stays strict, and available native operation failures do not retry through WASM. Public inspectTarArchive() still accepts only plain TAR/gzip; ZIP fallback still requires optional JSZip.

Every raw pass receives only TypeScript's resolved maxEntries, maxMetaEntryBytes, and maxDecodedBytes. Shared resolution caps metadata and decoded byte fields at JavaScript's safe-integer maximum and entry counts at 2^32 - 1 before backend selection. Large finite limits remain accepted; native conversion mirrors those caps and rejects malformed non-finite or negative direct-call values before casting. Logical member headers count before filtering/stripping; metadata records do not. maxEntryBytes and maxExtractedBytes remain exclusively in TypeScript's accepted-plan builder, after strip/filter policy, and are absent from the raw meter's interface. Bounded reads use the default count/metadata/decoded bounds; public maxBytes bounds only the requested output. TypeScript derives the internal decoded cap by safely adding maxExtractedBytes and maxArchiveBytes, clamped to the safe integer maximum. Every native pass receives that same cap and charges headers, metadata, bodies, padding, EOF blocks, and trailing zeros. It rejects overflow with archive-decoded-size-exceeds-limit; no ratio policy is implied. Extraction and entry reads drain the metered reader through physical EOF after admitted-range replay. Trailing framing or decoded-limit failures propagate before directory modes are finalized, staging is published, or selected bytes return. Native gzip uses the existing flate2 member decoder with an explicit bounded member/padding transition; JavaScript retains Node gunzip and validates its unconsumed compressed suffix. Only all-zero physical padding after a complete validated trailer is accepted, still within the original archive-byte budget. Native reads stop at framing boundaries so a rejected header does not request its body from the decoder; codec buffering can still read ahead internally. Inspection finishes the complete bounded framing pass before returning its manifest. Directory and link bodies, missing two-block EOF, and nonzero trailers reject on both backends, as detailed in raw TAR framing. Raw and padded sizes above JavaScript's safe-integer maximum reject as invalid framing before applying member budgets, including when local PAX overrides the size.

Publication and hashing

Exclusive publication tries a hardlink, then a copy-on-write clone, Linux copy_file_range, and finally the existing asynchronous JavaScript byte loop. All routes preserve wx semantics and the same source/target identity and SHA-256 fencing. Native hashing and Linux whole-file copying run on N-API async workers rather than the JavaScript event loop.

Linux range copying confirms every zero-byte result with a positioned source read at the current transfer offset, including after earlier calls copied data. If readable bytes remain, automatic Root copying resumes its byte loop from that offset; exclusive publication removes its partial target before retrying the guarded byte-copy fallback. EOF checks preserve descriptor cursors and do not bypass the byte limit.

Mode semantics

Mode Native loading Fallback
auto Try once, cache the result Use guarded JavaScript when safe; reject native-only operations
require Try once, cache the result Throw FsSafeError("helper-unavailable")
off Never attempt a binding load Use guarded JavaScript when safe; reject native-only operations

sha256FileSync() is a synchronous Node implementation in all three modes and does not load the binding. Use asynchronous sha256File() for native hashing and cancellation that can respond while JavaScript callbacks run.

Features without a safe fallback, including no-clobber Root.move() and retained-directory staging, fail with helper-unavailable when native support is absent or off. Staging is currently Linux/macOS only and rejects Windows with unsupported-platform.

Windows raw owner/DACL inspection, private-directory creation, and secure-file descriptor inspection can use a package-shipped, readable .ps1 driver and adjacent .cs source in auto or off mode when their binding or capability is unavailable. System Windows PowerShell runs the fixed driver with -File; paths remain data, with no runtime-generated helper script or encoded launcher. The Windows security fallback prerequisites apply, and unsupported or disallowed command execution fails closed. This route preserves raw ACL facts, private DACLs at creation, and descriptor-bound secure reads, and emits a path-free FS_SAFE_NATIVE_FALLBACK warning once per capability per process. Each call adds PowerShell startup and compilation overhead. require rejects missing capabilities without a command, and an available native operation's failure never triggers this fallback. See Permissions and Secure file reads for error and platform contracts.

The staged-file owner also serves POSIX native pinned writes, including streaming. Unpublished files remain at 0600; requested modes are applied through the owned file descriptor only after rename and published-entry identity validation. Post-rename chmod or sync failures retain the publication receipt and final name. Its direct-child exclusive openat hands off the descriptor before any fallible post-open checks; non-following statat compares against that descriptor with exact native identities, and cleanup uses unlinkat in the retained parent. The separate checks and unlink are not atomic conditional deletion. Windows pinned writes and other fallback-capable APIs retain their existing mechanisms. Native writers share root and parent admission, but keep their platform identity checks and leaf ownership. POSIX coordinator disposal uses SuppressedError to retain both an operation failure and a disposal failure, including their receipts; stage preparation and cleanup keep their documented error mappings. Root replacement verification borrows the published descriptor after final mode application, while a private coordinator retains the staged owner until the asynchronous check finishes. The owner never escapes that coordinator; public staging methods and receipts expose no descriptor or verification callback. Verification failures preserve the published name, and disposal still retains both verification and cleanup errors when both fail. The private verification channel carries exact bigint identity from the original owned descriptor (or the content-accepted FUSE descriptor). Root compares it against exact fd and pathname metadata; legacy helper return facts and public read metadata behavior are unchanged. Missing Windows pathname identity still requires a guarded path reopen and comparison with the original retained file; that fallback does not apply to POSIX no-read modes.

JavaScript fallback guarantees and delta

Policy-bound parent creation can refresh exact directory facts through the retained POSIX descriptor. The optional native observer compares descriptor and no-follow pathname metadata and verifies the descriptor's canonical path before and after the observation. Ordinary paths can share that observation within one synchronous admission phase; callbacks, mutations, and later phases require fresh evidence. Unsupported helpers retain the guarded pathname checks. Policy and denied-path decisions remain in TypeScript.

Public policy does not change with the selected mechanism: traversal and link rejection, archive filters/limits/modes, exclusive target creation, source and target identity fencing, publication cleanup receipts, and secret/lock policy remain TypeScript-owned. What changes is the syscall strength or availability:

The table compares underlying mechanisms. On Node, public Root.open(), Root.read(), and Root.openWritable() use guarded Node file opens and report containment: "best-effort" in every native mode. require checks availability when an operation requests native support; it does not upgrade those results. See Root containment guarantees.

Capability Native path Guarded JavaScript path
Native beneath opens and Root mutations Descriptor-relative beneath operations. Pinned writes create parents and publish both replacement and no-replace targets relative to open directory descriptors. No-clobber Root.move() admits both parents and uses the native no-replace rename. Native openBeneath() reports kernel-atomic with Linux openat2 and best-effort with the guarded Linux fallback, macOS, and Windows. macOS uses O_RESOLVE_BENEATH when available plus an F_GETPATH detector, while Windows rejects reparse traversal in the object-manager call. Reports best-effort: component-wise alias checks, no-follow opens where Node exposes them, private temp/rename, and post-operation identity verification. No-clobber Root.move() is unsupported because a check followed by a replacing rename is unsafe. A same-privilege peer can replace a writable parent after a guard assertion but before Node resolves another pathname mutation; the mutation may land outside the intended root before the post-check detects it.
ZIP/TAR/gzip Rust streaming decode and fd-relative output creation. Optional JSZip or bundled WASM TAR into guarded private staging, then the same guarded merge policy.
Zstd/bzip2 TAR Rust streaming decode and fd-relative output creation. Bundled WASM codecs feed the shared Rust TAR parser, then guarded private staging and the same merge policy; no optional codec dependency.
Publication copy Clone, Linux copy_file_range, async native SHA-256. Exclusive wx byte loop and Node SHA-256 with the same content/identity fences.
rename-noreplace Atomic platform no-replace rename. Unsupported; no emulation by check-then-rename.
Windows DACL read Direct GetSecurityInfo; the public facts API exposes ordered basic allow/deny ACE SIDs, masks, and decoded flags without trust policy. Secure-file reads query the borrowed open descriptor and compare its 32-bit volume serial and 64-bit file-index projection with Node's bigint receipt. The packaged PowerShell/C# bridge preserves raw facts and inspects the borrowed descriptor for secure reads, with the same Node identity comparison; its command failures reject. Structured .NET pathname reporting retains its separate compatibility query.
Windows private directory Creation-time protected DACL. The packaged PowerShell/C# bridge applies the protected DACL at creation and retains exact handles through identity validation and failure cleanup. Command failures reject.

Use off in CI to keep the fallback contract exercised. Use require when a deployment depends on the stronger mechanism or a native-only feature; do not infer native loading from timing.

Loader security

Importing fs-safe never executes a child process. Linux libc selection uses the Node process report, then the ELF PT_INTERP field of process.execPath, then conventional musl library filenames. An installed compatibility loader does not override the running executable's interpreter. If all probes are inconclusive, the loader conservatively attempts the glibc package and lets normal module loading fail into auto fallback. The loader requires only the package selected from the detected target; it never probes unrelated packages, downloads code, or runs a postinstall step. A missing or incompatible binary silently selects the JavaScript fallback in auto, throws typed helper-unavailable in require, and is never inspected in off. Tests reject child_process, exec, or spawn usage in the loader.

Related pages