@openclaw/fs-safe/archive extracts ZIP and TAR archives behind one API, with traversal checks, blocked-link-type rejection, and entry-count and byte budgets. When the native binding is available for the current platform, Rust streams ZIP, TAR, gzip, zstd, and bzip2 while TypeScript remains the sole policy owner; every accepted output is created fd-relative in a private staging root. Extraction then merges through the same safe-open boundary used by direct writes — a symlinked entry can't trick the merge into following an out-of-tree path.
TAR admission uses one Rust core compiled into both the native binding and a
bundled, import-free WebAssembly module. In off, or auto when the native
binding is unavailable, extraction and bounded entry reads use that module for
plain TAR, gzip, zstd, and bzip2. Zstd and bzip2 use bundled WASM builds of the
same codec implementations used by native; gzip uses Node's built-in decoder.
These TAR routes work with all optional dependencies omitted and need no
runtime interpreter, download, install script, or consumer compiler toolchain.
ZIP fallback still requires optional jszip.
The shared TAR parser reuses the already-validated owned path for ordinary members. Original header names and USTAR prefixes still undergo validation even when PAX or GNU metadata supplies an override; effective override paths retain their separate checks. Empty USTAR prefixes retain field decoding and padding checks; path validation applies to nonempty prefixes. Joining an admitted prefix and name with a separator preserves their checked components, so the parser does not repeat the same component validation on the joined path.
auto prefers an available native binding; a native operation failure is
terminal and never retries through WASM. require rejects a missing binding
with FsSafeError("helper-unavailable"), including for zstd/bzip2 suffix
resolution. The separate inspectTarArchive() API still accepts only plain TAR
and gzip.
import { extractArchive, resolveArchiveKind } from "@openclaw/fs-safe/archive";await extractArchive({
archivePath: "/srv/uploads/plugin.zip",
destDir: "/srv/workspace/plugins/plugin",
kind: "zip", // optional; resolveArchiveKind() can infer
timeoutMs: 15_000, // hard budget; active destination mutation is joined
stripComponents: 0, // tar-style strip-leading-dirs
entryModes: "clamp", // default; use "preserve" for archive rwx bits
entryUmask: 0, // default; remove these bits from final modes
entryFilter: ({ path, kind, size }) => "extract",
onFiltered: "reject-archive", // default; opt into "skip-entry" explicitly
limits: {
maxArchiveBytes: 256 * 1024 * 1024,
maxEntries: 50_000,
maxExtractedBytes: 512 * 1024 * 1024,
maxEntryBytes: 256 * 1024 * 1024,
maxMetaEntryBytes: 1024 * 1024,
maxEntryPathComponents: 256,
},
});type ExtractArchiveOptions = {
archivePath: string; // absolute path to the archive
destDir: string; // absolute destination directory; must already exist
timeoutMs: number; // positive elapsed-time budget; <= 0/non-finite disables it
durable?: boolean; // false; opt into syncing published files and directories before completion
kind?: ArchiveKind; // "zip" | "tar" | "tar-zstd" | "tar-bzip2"
stripComponents?: number; // strip N leading dirs from entry paths
tarGzip?: boolean; // when archive is .tar.gz/.tgz
limits?: ArchiveExtractLimits;
logger?: ArchiveLogger; // { info?, warn? }
entryModes?: "clamp" | "preserve";
entryUmask?: number; // integer 0..0o777; defaults to 0
entryFilter?: (entry: { path: string; kind: ArchiveEntryKind; size: number }) =>
"extract" | "skip";
onFiltered?: "reject-archive" | "skip-entry";
};durable defaults to false. Private staging never fsyncs, and publication copies
defer opted-in durability until the complete merge succeeds. With durable: true, the final pass syncs each
published file once (at most eight concurrently), then each published directory
once, deepest first, and finally the destination directory. All work stays inside
the extraction deadline; active syncs are joined before rejection. File sync
failures use the same error surface as Root.copyIn(); directory I/O failures
also reject, with the existing platform limitations on directory flushing.
Deadline checks use a monotonic clock, including before queued mutations start
and before reporting success. Synchronous caller code can delay the timer, but
cannot permit the next operation after the budget expires. This does not
interrupt a callback halfway through execution or replace its own thrown error;
active destination mutations are still joined before timeout rejection.
Files whose final mode prevents reading, including 0o000 and write-only files,
sync once through the copy's retained descriptor during publication. Permissions
are never widened to reopen them. Directory modes are finalized after the file
pass, with a descriptor pinned before chmod for syncing restrictive directories.
Existing inaccessible directories are never widened; an existing search-only
directory must become readable in its final mode if no readable sync descriptor
can be acquired before chmod.
The default suits extractions into temporary or reconstructible locations.
It skips all file and directory syncs while preserving atomic file publication,
mode enforcement, identity checks, and containment checks. Successful durable: true
extraction syncs file contents and directory entries before returning; failures
can leave a partially published tree as described below.
For a crash-safe install workflow, extract with the default into a scratch
directory, then apply the caller's durability policy: sync the staged files and
directories before publishing with replaceDirectoryAtomic,
and sync the affected parent directories afterward. The directory swap alone
does not sync the staged tree. Alternatively, pass durable: true when extracted
files must be on stable storage before the extraction call returns, subject to
the platform's flushing guarantees. A plain fsync on macOS does not flush the
drive cache.
entryModes defaults to "clamp": directories become 0o755; files become
0o644, or 0o755 when the archived owner-execute bit is set. "preserve"
keeps archived read/write/execute bits. Both policies strip setuid, setgid, and
sticky bits, and neither applies archived ownership. An explicit zero mode,
including a mode containing only stripped special bits, stays zero under
"preserve". Absent metadata defaults to 0o644 for files and 0o755 for
directories; ZIP UNIX creator records with zero attributes are explicit zero,
while non-UNIX ZIP records use the absent-metadata defaults.
entryUmask removes permission bits after the selected mode policy: final modes
are the policy result & ~entryUmask. It applies to files, explicit directories,
and implicit parent directories, including existing destination directories.
The destination root and private staging modes are unchanged. The default 0
preserves existing behavior; invalid masks reject before extraction begins.
fs-safe neither reads nor changes the process umask. Pass
entryUmask: process.umask() explicitly when that is the caller's policy.
Windows retains the POSIX-mode limitations described below.
TAR mode fields containing only NUL/ASCII-space padding use absent defaults. Both backends recognize GNU binary modes, including signed values, within JavaScript's safe-integer range before masking permission bits. Malformed or unsupported mode fields consistently fall back to zero, matching the former native behavior. This replaces JavaScript's decoder-dependent octal prefix parsing, defaulting, or rejection for malformed fields. Ordinary octal, absent, explicit zero, and supported GNU binary fields retain their behavior.
Final modes remain separate from private working staging permissions: files
stay 0o600 and directories 0o700 until publication. Files receive their final
mode through the guarded copy's owned writer descriptor. Directories are pinned
before descending and finalized after their children, including empty and
restrictive directories. Explicit accepted directory modes win regardless of
archive order; implicit parents receive 0o755 & ~entryUmask. Existing destination directories
also receive the requested final mode. They are never temporarily widened to
allow child writes; insufficient write/search access still rejects.
Directory mode changes use a retained no-follow read descriptor when possible.
On macOS x64/arm64, read-denied directories can use a retained search descriptor.
On Linux x64/arm64, the Node-only search route retains an O_PATH descriptor and
changes modes through its exact /proc/self/fd/N reference after verifying the
procfs namespace and followed identity. The descriptor stays open through the
operation and verification; original root, ancestor and named-directory checks
still apply. This route trusts host mount-namespace integrity and does not claim
atomic ancestry checks or protection against privileged mount replacement.
Readable directories do not depend on procfs. A Linux search-only directory
needing a mode change requires accessible, genuine procfs; unavailable or
untrusted authority rejects explicitly instead of silently accepting a wrong
mode. Other unsupported search-only routes also fail closed. Windows retains
its existing bounded lack of POSIX mode enforcement. Best-effort mode handling
applies only to the mode change itself: an authority or deadline check that
fails immediately before dispatch still propagates, including a custom
one-shot structural check. A check failure observed immediately after dispatch
is retained while post-dispatch authority verification and final mode inspection
settle, then propagated with its exact JavaScript value, including falsy values.
Extraction and TAR inspection first copy the admitted source into a private staging file. This copy reuses at most 512 KiB of scratch space, reduced for small inputs and capped by the archive byte limit plus one overflow-probe byte. Each read stays within the remaining budget plus that probe; deadline checks surround reads, and short writes finish before the buffer is reused.
Native extraction is deliberately split into two phases. Rust first reports an
entry manifest without creating paths. TypeScript validates paths, applies
stripComponents, filters, limits, and mode policy, then passes an explicit
accepted-entry plan back to Rust. Rust owns raw-stream admission, decompression,
and fd-relative mkdirBeneath/exclusive-open writes. This keeps filter policy identical
between native and JavaScript paths rather than reimplementing it in Rust.
ZIP extraction and bounded reads admit every physical central-directory record and its referenced local header before either decoder can normalize or collapse names. Raw names and valid Unicode Path names must pass traversal checks before stripping, filtering, or selecting a requested member; duplicate or colliding names reject with entry-path, even in unrelated or skipped members. Materially conflicting local/central or Unicode interpretations, malformed critical metadata, and ambiguous framing reject with ArchiveFormatError. Harmless internal separator and dot-component equivalence is allowed only after validation; every raw and Unicode interpretation must also agree on whether its name ends in / or \. Ordinary legacy filename decoding remains backend-selected. Native ZIP extraction groups nearby metadata reads into at most two 4 KiB read-ahead buffers per admission pass; larger records retain separately bounded reads. Buffered record views remain stable across eviction, and cached work periodically yields for deadline checks.
ZIP preflight and bounded reads apply extraction's case-insensitive NFC collision policy to the complete archive, including entries that extraction would strip or skip. Known UTF-8 and Unicode Path names are checked during physical admission; legacy names retain backend-selected decoding and undergo the same collision check before callbacks or selected bytes are returned.
ZIP end-record admission searches the bounded comment window for signatures
while retaining complete comment-length and ambiguity checks. Dense signature
sequences fall back to the bounded byte scan.
The separate readZipCentralDirectoryEntryCount(buffer) hint uses bounded
reverse searches for comments and retains its latest-valid-record selection;
it does not replace strict archive admission.
ZIP admission establishes each entry's kind before callbacks: a high-word UNIX symlink type takes precedence regardless of creator, followed by the DOS directory bit, the exact UNIX directory type, or a terminal slash/backslash. Native manifests must agree with this kind, physical index, size, known path, and UNIX-creator mode before extraction or any member read. Bounded ZIP reads retain this metadata from their single admission pass without another input copy or scan.
Portable ZIP preflight, extraction, and reads bind every decoder insertion to
its admitted physical record before JSZip can coerce its type or discard its
payload. Names, physical order, original directory/permission metadata,
compression method, compressed and decoded sizes, and CRC must agree. The
private loader then applies the admitted kind, preserving UNIX-only directory
attributes, backslash-only directories, and high-word symlinks from any creator.
Symlinks remain subject to the existing filter and blocked-link policy.
UNIX socket and block-device type bits do not turn regular-file payloads into
empty directories. Directory and symlink callbacks receive their physical
declared sizes; directory bodies are not published as files. Unsupported
link-like types remain visible as other and are safely omitted when accepted.
UNIX creator metadata and permission defaults remain as described above.
The loader adapter belongs to one private JSZip instance and is removed after
loading, including failure. Public preflight still returns ordinary JSZip entry
objects, with directory keys ending in / and recognizable symlink type bits.
Compressed data is retained even for declared-zero entries, so an empty-size
claim cannot bypass payload-size or CRC verification during extraction or reads.
Extraction and bounded reads retain the admitted CRC and size independently of
mutable decoder objects, so later decoder changes cannot redefine the expected
payload integrity. Public preflight archives retain ordinary JSZip mutation
and entry-reading behavior.
Portable bounded reads select the admitted canonical name and reject a selected
decoder entry that no longer belongs to that archive and name before reading its
payload. Extraction retains the admitted names and physical order independently
of later changes to the decoder's public files object.
Within one ZIP entry, identical local and central name bytes reuse the same decoded validation. Unicode Path admission is shared only when both the raw names and the complete Unicode fields match; different fields still verify their own CRC and interpretation. Shared backing memory is checked independently. Decoded name validation is not reused across entries or archives. UTF-8-flagged ASCII names in nonshared backing memory reuse their raw-path validation, and an identical decoded spelling reuses its canonical key. Shared name bytes still undergo independent decoding and validation; Unicode Path fields retain their own CRC and interpretation checks.
stripComponents removes leading nonempty, non-. path components after
normalizing separators. For example, ./pkg/hello.txt with
stripComponents: 1 extracts to hello.txt on both backends. Entries with no
remaining components are skipped before the filter callback, but still count
toward maxEntries and undergo traversal validation. JavaScript TAR extraction
copies the admitted payload range to this accepted output path, so depth checks,
collision checks, writes, and mode application agree.
An entryFilter sees the validated canonical effective archive path before
stripping, entry kind, and declared size. On every JavaScript and native
ZIP/TAR backend (including gzip, zstd, and bzip2), backslashes become /,
empty and . components are removed, and trailing separators are removed from
directory paths. For example, ./pkg//state\cache/value is presented as
pkg/state/cache/value, even with stripComponents: 1. Case and Unicode
spelling are preserved. Local PAX path, GNU long-name, and supported ZIP
Unicode Path names use the same canonicalization.
A leading ~ is a literal archive entry name and is never expanded to the
user's home directory during extraction, publication, or durability checks.
Callbacks follow physical archive order, including ZIP names that look like
integer object keys. The public ZIP loader's files object retains ordinary
JavaScript object enumeration and mutation behavior.
Raw paths undergo traversal, absolute/drive-path, and NUL validation before
canonicalization; normalization cannot turn an unsafe path into an accepted
one. Stripping and output collision checks use this same canonical identity.
On Windows, archive admission also rejects reserved device segments such as
NUL, CON.txt, and nul .txt with ArchiveSecurityError("entry-path"),
before extraction or bounded member reads. Ignored trailing spaces in the stem
before an extension do not bypass this check. Ordinary members with these names
remain valid on POSIX; this is a host-specific device guard, not a portable-name
restriction.
Filters that compare exact strings should use canonical pre-strip paths,
including directory names without a trailing /.
Returning "skip" rejects the whole archive unless onFiltered is
explicitly "skip-entry". Runtime values other than "reject-archive" and
"skip-entry" reject before extraction starts instead of falling through to
skip behavior. Path traversal and archive-wide entry-count checks still apply
to skipped entries.
maxEntryBytes and maxExtractedBytes charge only entries accepted after
stripping and filtering. Skipping a large member does not consume these payload
budgets. The separate complete-stream decoded limit still applies to all TAR
content, including skipped or fully stripped members.
For example, a fleet restore can omit regenerated cache entries while rejecting any other policy mismatch by default:
await extractArchive({
archivePath: snapshotPath,
destDir: restoreRoot,
timeoutMs: 30_000,
entryFilter: ({ path: entryPath, kind }) =>
kind === "directory" && entryPath === "state/cache"
? "skip"
: entryPath.startsWith("state/cache/")
? "skip"
: "extract",
onFiltered: "skip-entry",
limits: { maxEntries: 50_000, maxEntryPathComponents: 64 },
});If skipping was not explicitly part of the restore contract, omit
onFiltered; the first "skip" then rejects the complete archive with
ArchiveSecurityError("entry-filtered").
Both TAR routes finish bounded admission before TypeScript policy evaluation, so a rejected plan never starts extraction. The JavaScript path owns its input, decoder, and WASM parser streams, joining their teardown on validation, write, or timeout failure.
TAR character devices, block devices, and FIFOs are presented to the filter as
kind: "other". Accepted entries of these types reject with
ArchiveSecurityError("entry-link"); an explicit "skip-entry" filter can omit
them. GNU typeflag D (GNUDumpDir) is a directory on both backends, including
its filter kind, canonical path, and directory creation policy. Its declared
body size follows the existing TAR strip/filter payload budgets; dump contents
are not restored as files.
Unsupported logical TAR records, including volume headers (V), Solaris ACL
records (A), inodes (I), continuations (M), and unrecognized typeflags,
still undergo entry counting, raw/effective path validation, stripping, depth
and output collision checks in physical order. Each remaining record reaches
entryFilter once with its canonical pre-strip path, kind: "other", and
declared effective size. A filter skip rejects with "entry-filtered" unless
onFiltered: "skip-entry" is explicit. Accepted unsupported records are safely
omitted and do not consume output payload budgets. The shared core admits these records explicitly. GNU long names describe one such
record and are then cleared; local PAX on unsupported types and GNU sparse
S records retain their existing fail-closed format policy.
If kind is omitted, the helper calls resolveArchiveKind(archivePath) and throws if the extension is not recognized. Pass kind explicitly when the archive name doesn't carry the type (e.g. content-addressed names). Archive inputs must remain regular files from preview through descriptor admission; POSIX opens are no-follow and nonblocking, so a FIFO swap cannot stall before deadline checks resume. On Windows, archive source and destination filesystem paths reject NTFS alternate-stream and directory-index namespace spellings such as file:stream and dir::$INDEX_ALLOCATION; ordinary colon-bearing POSIX names remain valid. A positive finite timeoutMs is a wall-clock budget; zero, negative, NaN, and infinity disable the deadline. Non-mutating work rejects promptly when the budget expires. If a live destination mutation is already in flight, rejection waits only for that mutation and any rollback to finish; no later destination mutation can begin.
Extraction captures the destination's lossless filesystem identity before any
entry filter runs and retains that capability through final publication. If a
filter or concurrent actor renames or replaces the destination, extraction
rejects with destination-symlink-traversal before publishing into the
replacement. This check uses bigint device and inode identities so large native
identifiers cannot compare equal after JavaScript number rounding.
The destination merge is nontransactional: each file is published atomically,
but completed files and directories can remain when a later copy, post-copy
check, mode application, or deadline fails. This also applies to
mergeExtractedTreeIntoDestination(). Guarded Root.copyIn() owns unpublished
stage cleanup and preserves completed publications. The archive merge never
acquires rollback authority over the current destination name. Replacing a
source path after publication rejects the merge with path-mismatch while
preserving the admitted bytes already published, or any later destination edit.
A failure before publication preserves a pre-existing file, and rejection does
not grant authority to delete a substituted file or alias. Failed extraction does not
restore overwritten contents. Active destination mutations and their guarded
cleanup still finish before rejection; no later destination mutation begins.
Portable ZIP output is not eligible for publication until its stream closes or
the defensive FileHandle close succeeds. If that fallback close rejects,
extractArchive() propagates the error and publishes no entry from the staged
tree. Cleanup retains its best-effort FileHandle close; it does not transfer
the descriptor to a raw or native closer.
New directories whose finalization was never reached can retain their
private working mode after failure. Failure cleanup closes retained descriptors;
it does not run a cleanup chmod sweep or roll back the archive. The public merge
helper still derives modes from its external source tree and must be able to
read that source; it never chmods an unreadable external source to admit it.
It retains the source root and each active child directory's exact identity
through traversal and copy verification. Each source file is opened once,
admitted against that root and its earlier exact identity observation, and
copied from the admitted descriptor; public file modes use that descriptor's
ordinary permission bits. Replacing a source ancestor or leaf rejects the
merge before replacement bytes can be published. These checks do not provide
a snapshot against writes to the same source inode.
That helper retains per-copy durability and immediate postorder directory-mode
finalization; the deferred pass described above belongs to extractArchive().
type ArchiveExtractLimits = {
maxArchiveBytes?: number; // refuse if archivePath stat'd size exceeds this
maxEntries?: number; // refuse before extracting if entry count > this
maxExtractedBytes?: number; // cap total payload bytes accepted after strip/filter
maxEntryBytes?: number; // cap one accepted entry after strip/filter
maxMetaEntryBytes?: number; // refuse one PAX/GNU metadata body above this
maxEntryPathComponents?: number; // bound output path depth after stripComponents
};Defaults exist for each (DEFAULT_MAX_ARCHIVE_BYTES_ZIP, DEFAULT_MAX_ENTRIES, DEFAULT_MAX_EXTRACTED_BYTES, DEFAULT_MAX_ENTRY_BYTES, DEFAULT_MAX_META_ENTRY_BYTES, DEFAULT_MAX_ENTRY_PATH_COMPONENTS). An explicit zero remains zero rather than selecting the default. maxEntries counts every archive entry, including entries removed by stripComponents or an explicit filter. The path-component default is 256. It is evaluated after stripComponents and before TypeScript accepts an entry for either JavaScript or native extraction, so rejected entries cannot cause implicit parent-directory creation. The same resolved 1 MiB metadata default applies to the native and WASM core.
A limit violation throws ArchiveLimitError. Its constant and string code are:
| Constant | Code |
|---|---|
ARCHIVE_SIZE_EXCEEDS_LIMIT |
archive-size-exceeds-limit |
ENTRY_COUNT_EXCEEDS_LIMIT |
archive-entry-count-exceeds-limit |
EXTRACTED_SIZE_EXCEEDS_LIMIT |
archive-extracted-size-exceeds-limit |
DECODED_SIZE_EXCEEDS_LIMIT |
archive-decoded-size-exceeds-limit |
ENTRY_EXTRACTED_SIZE_EXCEEDS_LIMIT |
archive-entry-extracted-size-exceeds-limit |
META_ENTRY_SIZE_EXCEEDS_LIMIT |
archive-meta-entry-size-exceeds-limit |
ENTRY_PATH_COMPONENTS_EXCEEDS_LIMIT |
archive-entry-path-components-exceeds-limit |
MANIFEST_SIZE_EXCEEDS_LIMIT |
archive-manifest-size-exceeds-limit |
MANIFEST_SIZE_EXCEEDS_LIMIT is an active internal TAR admission limit, shared
by JavaScript and native extraction and bounded reads. Each logical member,
including ignored, filtered, and fully stripped members, charges
64 + 2 * UTF-8 byte length of its effective pre-strip path before emission or
retention. PAX/GNU metadata headers do not themselves charge a member cost.
The allowance is independent of maxArchiveBytes: derive a per-member path
allowance of max(256, min(maxMetaEntryBytes, max(1, maxEntryPathComponents) * 256)),
apply the same 64-byte overhead and doubled path cost, multiply by maxEntries,
and cap the total at 64 MiB using saturating arithmetic. Zero and very large
public limits remain deterministic. There is no public maxManifestBytes
option; this charged manifest budget supplements the decoded and metadata
limits rather than bounding the complete process heap.
Catch and branch on the code to surface a meaningful response to the caller.
Entry policy failures throw ArchiveSecurityError. Its entry-related codes
are "entry-path", "entry-link", and "entry-filtered"; destination-race
codes remain "destination-not-directory", "destination-symlink", and
"destination-symlink-traversal".
- Path traversal: entries with
.., absolute paths, NUL bytes, or Windows drive-relative segments such asC:secretandnested/C:secretare rejected (ArchiveSecurityError). On Windows, path segments containing:are also rejected as alternate data stream names before either backend writes to the filesystem. - Symlink/hardlink entries: rejected by default, including ZIP entries whose Unix mode says symlink while their name ends in a slash or their DOS directory bit is set. An explicit
entryFilterwithonFiltered: "skip-entry"can omit these entries. Some archives ship symlink/hardlink entries that point outside the destination once resolved;extractArchivedoes not follow them. - Ambiguous output names: duplicate names and distinct names that collide after
stripComponents, case normalization, or Unicode normalization are rejected instead of relying on backend- or volume-specific overwrite order. - TOCTOU during merge: extraction first writes to a private temp dir, then merges into
destDirusing the same boundary checks asroot().write(). Destination symlink swaps are checked with the selected platform mechanism; non-Linux routes retain the best-effort race window documented in the security model. - Zip bombs:
maxExtractedBytesandmaxEntryBytesapply to post-decompression bytes, so highly-compressed payloads hit the cap before they exhaust disk. - Corrupt ZIP payloads: streamed output must match both the central-directory CRC and declared uncompressed size before it can leave private staging.
- Gzip container integrity: every concatenated gzip member must have a complete valid header, body, CRC32, and ISIZE trailer. A completed member may be followed by all-zero compressed-container padding (including system-tar stdout padding), bounded by the original archive-byte limit. The padding must remain zero through physical EOF; nonzero bytes or another member after padding reject. Truncation and corruption reject before publication or selected bytes return on both backends. Compressed padding is separate from decoded TAR EOF and does not bypass its checks.
- Slow-loris archives:
timeoutMsis a hard wall-clock budget for non-mutating work. Extraction is aborted on overrun; if a destination mutation is already in flight, that mutation and rollback are joined before rejection so archive-controlled publication cannot continue afterward. - Metadata bombs: a streaming pass-through reader rejects oversized PAX, GNU long-name, and GNU long-link bodies before buffering their bodies. It understands octal and base-256 fixed sizes and validates bounded local PAX bodies before using their size overrides for member framing. Original archive bytes remain unchanged.
Native gzip, zstd, and bzip2 readers check cancellation before refilling compressed input and before each decoded read, including buffered output. These checks apply to file extraction and in-memory member reads; they cannot interrupt an already-running filesystem read or a decoder step using already-buffered input. Portable zstd/bzip2 decoding checks cancellation between bounded codec steps and periodically yields to the event loop, including while consuming output-free members. An individual WASM call cannot be interrupted. Teardown joins the input, parser, and any Node decoder streams before disposing their shared WASM state.
Extraction and bounded reads admit the complete decoded TAR stream through the shared Rust core. This applies to plain TAR, gzip, zstd, and bzip2 on native and fallback paths. The native and WASM builds enforce the same framing rules:
- Every nonzero header must have a valid unsigned octal checksum, delimited
within its field. Checksum validation precedes metadata allocation and member
policy. Fixed name, prefix, and linkname fields require strict UTF-8 and NUL
padding. Raw hardlink (
1) and symlink (2) headers require a nonempty linkname; every other type, including PAX/GNU metadata, requires an empty linkname. This check precedes metadata handling and member/filter policy. - Directory (
5), hardlink (1), and symlink (2) raw headers must declare zero body bytes, whether or not local PAX metadata is present. Valid zero-size links remain subject to the existing link/filter policy. - EOF requires two consecutive, complete 512-byte zero blocks at a header boundary. A header after just one zero block, a missing/partial EOF marker, and any nonzero bytes after EOF reject. Additional zero padding after EOF may have any byte length within the decoded ceiling; zero blocks inside a declared member body are payload.
- Headers and padded bodies must be complete. Size fields accept unsigned
octal with ASCII-space/NUL padding or supported positive base-256 encoding;
malformed numbers and non-padding bytes after a NUL reject. Raw sizes and
padded sizes must fit
Number.MAX_SAFE_INTEGER, even with PAX overrides, before member budgets are considered.
Framing failures use ArchiveFormatError("archive-header-invalid"), except
invalid UTF-8 or nonzero bytes after the first NUL in fixed name, linkname,
and USTAR prefix fields, which use ArchiveSecurityError("entry-path").
Missing linknames on links and nonempty linknames on non-links still use the
format error. PAX x and GNU long-name/long-link L/K payloads retain their
existing support and metadata limits; the zero-body rule is not applied to all
non-regular types.
PAX effective sizes determine regular-member framing. Admission preserves input
bytes and emits an ordered manifest with exact effective names, types, modes,
sizes, and decoded payload offsets. TypeScript owns filtering, stripping,
collisions, permissions, and accepted-output limits. Executors replay admitted
ranges from the immutable staged input; no second TAR parser interprets PAX,
GNU names, or payload lengths. Native writes remain descriptor-relative;
JavaScript writes use the shared guarded private staging and pinned-write helpers.
Original member names and USTAR prefixes are validated even when overridden. Non-padding bytes after a fixed path field's NUL terminator reject. The core enforces the 255-byte component ceiling under NFC and NFD, including Hangul expansion. Every replay drains and validates physical EOF before publication or returning selected bytes. Unrequested, filtered, and stripped members cannot bypass validation. Decompression remains streaming; no complete decoded archive is retained in memory or written to a decoded spool.
The WASM transport has fixed 64 KiB input/output windows, one pending member event, and a 256 MiB maximum linear memory per isolated session. The parser and portable zstd/bzip2 decoder share that session and memory ceiling. JavaScript gzip decoding also emits chunks of at most 64 KiB for both staged files and buffered inputs. Metadata is bounded before allocation; codec allocation failure rejects. Stream backpressure bounds queued chunks, and completion or error releases the session's parser and decoder state after stream teardown. The manifest retains the existing charged budget below; linear memory is an additional execution resource bound, not a new public limit option.
Portable zstd/bzip2 decoding consumes every concatenated member through physical
EOF and verifies container integrity, including available checksums. Zstd
skippable frames are consumed without becoming TAR data. Truncated members and
trailing non-container bytes reject with ArchiveFormatError before filters,
publication, or selected bytes are returned. Decoded TAR EOF and byte-budget
checks still apply across member boundaries; a second TAR after EOF is not
silently ignored. The gzip-only compressed-padding policy above does not extend
to zstd/bzip2 containers.
The raw meter enforces maxEntries before consuming each logical member's body,
including members later skipped by filtering or stripping. PAX/GNU metadata
headers do not count as members; their payloads use maxMetaEntryBytes.
The meter does not receive maxEntryBytes or maxExtractedBytes: those payload
budgets apply only after strip/filter acceptance, using declared effective
sizes and excluding block padding. JavaScript's entry checker and the native
accepted-plan builder retain this shared policy. Every TAR admission/parser
pass has a separate absolute decoded ceiling:
maxExtractedBytes + maxArchiveBytes, safely clamped to
Number.MAX_SAFE_INTEGER (768 MiB with defaults). It counts every admitted
decoded byte: headers, bodies, metadata, all block padding, both EOF blocks,
and zero padding after EOF. It bounds complete decoding before parser policy,
including all filtered/stripped content; cumulative metadata and zero tails
cannot bypass this bound. Exceeding this ceiling throws
ArchiveLimitError("archive-decoded-size-exceeds-limit").
The same TypeScript helper derives the ceiling for JavaScript and every native
TAR pass. Before selecting a backend, it caps internal metadata/decoded limits at
Number.MAX_SAFE_INTEGER and logical entry counts at 2^32 - 1. Larger finite
options such as Number.MAX_VALUE remain valid; high-level payload budgets keep
their large values. The decoded ceiling uses clamped maxExtractedBytes and
archive overhead with safe addition. Ordinary limits, including
zero and the existing defaulting/rounding rules, retain their behavior.
There is no new public option. This is an absolute decoded admission
cap, not a decompression-ratio policy; bounded stream/codec read-ahead remains.
There is no independent TAR parser decompression-ratio threshold.
Extraction and single-entry reads accept one nonempty local POSIX x header
(USTAR or GNU header format) immediately before one regular/contiguous file,
directory, symlink, or hardlink. path, linkpath, and size override that
member only. Effective paths still pass traversal validation before stripping,
then the output paths pass depth and collision checks. The filter receives the
canonical effective pre-strip path, followed by link policy checks. PAX never
permits link creation. Effective sizes drive framing, filters, and the existing output-byte
budgets; maxEntries still counts members, not their metadata headers.
Records must have exact byte lengths, ASCII keys, a final newline, and no
duplicate keys or unconsumed bytes. path and linkpath must be nonempty strict
UTF-8 without NUL. Unicode, a leading BOM, numeric-looking names, and embedded
newlines preserve their exact spelling; newlines inside a byte-counted value
are data. Windows filesystem filename restrictions still apply during creation.
Ownership names retain the existing nonempty printable-ASCII contract. Raw name,
USTAR prefix, and link fields still require strict UTF-8 and NUL padding even
when metadata overrides them. Raw link targets must be present only on links.
size, uid, and gid
must be canonical unsigned decimal safe integers (zero is valid; signs, leading
zeros, fractions, and exponents are not). Padded member sizes must also fit the
safe integer range. Raw and effective directory/link sizes must both be zero;
non-directory paths ending with a separator and linkpath on non-links are
rejected rather than allowing parser-specific type or framing changes.
The descriptive allowlist is mtime, atime, ctime (signed decimal seconds
with optional fractional digits, within JavaScript's Date range), uid, gid,
uname, and gname. These attributes are accepted but not restored to the
destination. LIBARCHIVE.xattr.* and SCHILY.xattr.* with nonempty ASCII
alphanumeric/dot/underscore/hyphen suffixes are also accepted as inert metadata,
never restored as extended attributes. Their values are byte-counted and may
contain NUL, non-UTF8 bytes, or newlines, including macOS provenance metadata.
Global g, old X, old GNU N, empty/dangling/repeated local headers, mixed
PAX/GNU extension chains, unknown keys, charset declarations, ACL extensions,
and all sparse extensions (including GNU.sparse.*, SCHILY.filetype,
SCHILY.realsize, and SCHILY.size) fail closed with
ArchiveFormatError("archive-header-invalid"). GNU sparse extension blocks are still
metered in 512-byte units before rejection, preserving metadata-limit errors
for excessive chains. The per-body maxMetaEntryBytes limit bounds PAX storage
and duplicate-key state; one local header per member prevents local metadata
chains without introducing a new limit or changing defaults.
The shared core buffers GNU long-name L and long-link K bodies within
maxMetaEntryBytes. A body must contain a nonempty
UTF-8 name, with either no NUL or exactly one terminal NUL. Embedded NULs,
additional terminal NULs, bytes after a NUL, and invalid UTF-8 reject with
ArchiveFormatError("archive-header-invalid"). The core preserves original
archive bytes, including the optional terminator and block padding.
One logical member may have at most one L and one K, in either order.
Repeated metadata of either kind, mixed PAX/GNU chains in either direction,
and GNU metadata without a following member reject with the same format error.
Pending metadata is cleared only when its described member is admitted;
metadata records do not count toward maxEntries.
An L name undergoes raw-path validation before parser normalization, stripping,
or filtering; unsafe paths reject with ArchiveSecurityError("entry-path").
The validated name remains pending until its described header arrives. An
effective name ending in / or \ requires raw directory type 5 or D;
other types reject with ArchiveFormatError before filtering, preventing the
parsers from disagreeing about a member's type.
K validates encoding and NUL structure without authorizing link creation.
Normal link/filter policy still governs the described member. Canonical
pre-strip filter paths, decoded-stream ceilings, and physical EOF checks apply
to plain/gzip TAR and zstd/bzip2 alike.
Inspect accepted TAR members without creating an extracted tree. This operation
uses the same complete Rust/WASM admission and TypeScript extraction planner as
extractArchive, with zero stripping. It detects plain TAR or gzip from the
input bytes; ZIP, zstd, and bzip2 are not part of this inspection API.
import { inspectTarArchive } from "@openclaw/fs-safe/archive";
const entries = await inspectTarArchive({
archivePath: "/srv/uploads/tree.tar.gz",
timeoutMs: 30_000,
limits: {
maxArchiveBytes: 16 * 1024 * 1024,
maxEntries: 5_000,
maxEntryBytes: 16 * 1024 * 1024,
maxExtractedBytes: 64 * 1024 * 1024,
},
entryFilter: ({ kind }) => kind === "file" || kind === "directory" ? "extract" : "skip",
onFiltered: "reject-archive",
});InspectTarArchiveOptions accepts archivePath, timeoutMs, limits,
entryFilter, and onFiltered, with the same defaults and error classes as
extraction. The result is a frozen array of frozen InspectedTarEntry records:
{ path: string; kind: "file" | "directory"; size: number }, in archive order.
size is the effective declared payload length. path is extraction's
canonical pre-strip identity: case, Unicode spelling, BOM, and embedded LF are
preserved, while separators and dot components follow the existing archive path
contract. No human-readable tar listing or escape decoding is involved.
Full framing, gzip integrity, EOF, metadata, decoded-byte, and manifest-budget validation finishes before the caller's filter runs. The shared planner then applies traversal, collision, depth, blocked-type, and accepted-payload limits. A failure returns no partial result. Filter callbacks are decisions, not admission receipts: later policy, collision, or budget checks can still reject the archive. Only the resolved Promise/result is authorization-worthy; do not perform irreversible actions from a filter callback.
Root-only records count toward entry limits but produce no result; PAX/GNU metadata headers are not members. Only accepted file/directory members appear, not implicit parent directories. Unsupported records follow extraction's omission policy unless the filter rejects them, as in the example. AppleDouble records encoded as ordinary files are ordinary members, not hidden metadata.
Inspection pins and privately stages its input, then cleans up that copy. It
neither creates destination paths nor tests destination permissions or platform
filename restrictions. Its result is evidence about those inspected bytes, not
an extraction capability or a promise that a later file at archivePath is
unchanged. Callers making authorization decisions must retain the same private
immutable archive or verify byte identity before extracting with matching
filter/limit settings. Extraction always performs its own admission and guarded
publication. Native off, auto, and require retain their existing selection
and availability semantics; inspection does not fall back after native failure.
import { resolveArchiveKind, type ArchiveKind } from "@openclaw/fs-safe/archive";
const kind = resolveArchiveKind("upload.zip"); // "zip"
const tar = resolveArchiveKind("upload.tar.gz"); // "tar"
const zstd = resolveArchiveKind("upload.tar.zst"); // "tar-zstd" in auto/off; require checks native
const unknown = resolveArchiveKind("upload.bin"); // nullRecognizes:
*.zip→"zip"*.tar,*.tar.gz,*.tgz→"tar"*.tar.zst,*.tar.zstd,*.tzst→"tar-zstd"*.tar.bz2,*.tbz2,*.tbz→"tar-bzip2"
Returns null for unknown extensions; check the result before calling
extractArchive if the filename is caller-controlled. Recognized zstd and bzip2
TAR extensions resolve in auto and off even without a native binding, using
the bundled codecs for subsequent extraction or reads. Explicit require
still checks native availability during suffix resolution and throws
FsSafeError("helper-unavailable") when the binding cannot load.
For a deployment that requires native archive processing, configure native mode before the first archive call so a missing binding fails at the boundary:
import { configureFsSafeNative } from "@openclaw/fs-safe/config";
import { extractArchive } from "@openclaw/fs-safe/archive";
configureFsSafeNative({ mode: "require" });
await extractArchive({
archivePath: "/srv/restore/snapshot.tar.zst",
destDir: "/srv/restore/staging",
kind: "tar-zstd",
timeoutMs: 60_000,
});readArchiveEntry(archivePath, entryPath, { maxBytes, kind? }) reads one
regular-file entry into a bounded Buffer without extracting a tree. It reads
the input through an identity-checked descriptor, rejects a requested link or
directory, and rejects duplicate entry names anywhere in the archive. Unrequested
links and directories do not prevent reading a regular file; no links are followed
or created. It verifies ZIP CRC and declared size,
and throws ArchiveLimitError if the requested entry's output exceeds
maxBytes. ZIP output within that cap must match the declared uncompressed
size exactly; either a shorter or longer payload throws
ArchiveFormatError("archive-header-invalid") before bytes are returned on
both JavaScript and native backends.
For TAR, maxBytes applies only to that requested entry: a larger
unrequested member remains valid within the default archive admission limits.
TAR traversal uses default entry-count, compressed-input, and metadata limits,
plus the 768 MiB decoded ceiling derived from default extracted/archive byte
limits. It does not apply payload budgets to unrequested members. ZIP
inputs retain the archive subpath's 256 MiB compressed-input ceiling.
With a native binding it uses the same Rust decoders as extraction, including
zstd and bzip2 TAR. Without native, the guarded fallback uses bundled WASM for
TAR admission and zstd/bzip2 decoding, Node gunzip for gzip, and optional JSZip
for ZIP. Native require still rejects an unavailable binding.
Archive member reads retain their private in-memory input without a disk
snapshot. JavaScript ZIP member reads reuse their completed physical admission
when loading the decoder, which still checks its decoded names and entry count.
The native ZIP reader retains the private allocation and parsed directory across worker-thread
inspection and reading without an extra archive-byte copy.
Decompression still allocates its bounded output; Node receives that native
allocation without another copy where external buffers are supported.
Native TAR retains the fully admitted member offsets alongside the same input
allocation. Plain TAR copies only the selected payload range after full archive
validation. Gzip, zstd, and bzip2 replay bounded decompression and still validate
all framing, trailers, and physical padding before returning. The fallback also
retains admitted member offsets. After full admission, plain TAR copies the
selected range directly from its private snapshot; gzip, zstd, and bzip2 replay
bounded decompression through the same parser. WASM transport and selected
output use owned copies, so reusable codec windows cannot escape to callers.
Returned buffers own their bytes, so changing a result cannot modify an archive
reader or retain an unrelated part of the input through its backing ArrayBuffer.
Requested paths and effective member names use extraction's canonical pre-strip
identity: backslashes become /, and repeated separators and . components
are removed after raw-path validation. For example, ./pkg//value and
pkg\value both address pkg/value, including supported GNU/PAX and ZIP
Unicode Path names. Case and Unicode spelling are preserved. Requests ending
in / or \ still reject as non-files. Canonical duplicate members reject
before an unrelated requested entry can be returned.
const rawManifest = await readArchiveEntry(uploadPath, "package/manifest.json", {
maxBytes: 64 * 1024,
});
const manifest = JSON.parse(rawManifest.toString("utf8")) as PluginManifest;
validatePluginManifest(manifest);The archive subpath also exports the helpers extractArchive is built on. Most callers will not need them, but they are stable and documented:
| Function | Purpose |
|---|---|
withStagedArchiveDestination(opts) |
Creates a private staging dir outside the destination, calls your run(stagingDir), then cleans it up. |
mergeExtractedTreeIntoDestination(opts) |
The merge step alone — staged tree → destination through boundary checks. |
prepareArchiveDestinationDir(destDir) |
Canonicalizes and asserts the destination directory. |
prepareArchiveOutputPath({ destinationDir, destinationRealDir, relPath, outPath, originalPath, isDirectory, deadline? }) |
Validates and prepares parents for an already-resolved entry output path. |
loadZipArchiveWithPreflight(buffer, limits?) |
Loads a JSZip from a Buffer or Uint8Array with size/entry-count preflight before unzipping. |
readZipCentralDirectoryEntryCount(buffer) |
Returns the entry count from an already-loaded ZIP Buffer or Uint8Array without decoding payloads. |
createTarEntryPreflightChecker(opts) |
Returns a per-entry checker for use as a tar.x onReadEntry hook. |
These let you build custom extractors that share the same safety machinery — for example, a streaming uploader that wants to refuse archives with too many entries before reading any payloads.
archive-entry exports a handful of low-level helpers for entry-path normalization:
import {
isWindowsDrivePath,
normalizeArchiveEntryPath,
resolveArchiveOutputPath,
stripArchivePath,
validateArchiveEntryPath,
} from "@openclaw/fs-safe/archive";validateArchiveEntryPath(raw, opts)— throwsArchiveSecurityErrorfor.., absolute, NUL-containing, drive-relative, or otherwise unsafe entry paths, including alternate data stream names on Windows.normalizeArchiveEntryPath(raw)— converts backslashes in the entry path to forward slashes.stripArchivePath(entryPath, n)— normalize separators, drop empty and.components, then strip the leading N components, returningnullif none remain.resolveArchiveOutputPath({ rootDir, relPath, originalPath, escapeLabel? })— combines the validated relative path with the root and rejects escapes using the original archive path for diagnostics.isWindowsDrivePath(value)— detects drive-relative segments such asC:secretornested/C:secretthat should be rejected.
Validate attacker-controlled paths before calling normalization or stripping
helpers. After validation, stripArchivePath(entryPath, 0) returns the canonical
pre-strip identity used by extraction filters (or null for an empty path).
import { extractArchive, ArchiveLimitError, ARCHIVE_LIMIT_ERROR_CODE } from "@openclaw/fs-safe/archive";
try {
await extractArchive({
archivePath: upload.path,
destDir: targetDir,
kind: "zip",
timeoutMs: 30_000,
limits: {
maxArchiveBytes: 100 * 1024 * 1024,
maxEntries: 10_000,
maxExtractedBytes: 200 * 1024 * 1024,
maxEntryBytes: 50 * 1024 * 1024,
},
});
} catch (err) {
if (err instanceof ArchiveLimitError) {
return reply(413, { code: err.code, message: err.message });
}
throw err;
}const kind: ArchiveKind = mime === "application/zip" ? "zip" : "tar";
await extractArchive({ archivePath, destDir, kind, timeoutMs: 10_000 });import { withTempWorkspace } from "@openclaw/fs-safe/temp";
import { replaceDirectoryAtomic } from "@openclaw/fs-safe/atomic";
await withTempWorkspace({ rootDir: "/srv/site/tmp", prefix: "extract-" }, async (ws) => {
await extractArchive({
archivePath: upload.path,
destDir: ws.dir,
timeoutMs: 30_000,
});
await replaceDirectoryAtomic({
stagedDir: ws.dir,
targetDir: "/srv/site/plugin",
});
});- Atomic writes —
replaceDirectoryAtomicfor staged directory replacement. - Temp workspaces — extract into a private workspace and commit as one step.
- Errors —
FsSafeErrorcodes the underlying writes can raise. - Migrating to 0.5 — clamp-default and native-format upgrade checklist.
extractArchivesource.