diff --git a/docs/StrictProvenance.md b/docs/StrictProvenance.md index 5f3081a33..8f23604f1 100644 --- a/docs/StrictProvenance.md +++ b/docs/StrictProvenance.md @@ -1,6 +1,30 @@ # StrictProvenance Architectures -To aid code auditing and support of novel architectures, such as CHERI, which explicitly track pointer *provenance* and *bounds*, `snmalloc` makes heavy use of a `CapPtr` wrapper type around `T*` values. +## What is Strict Provenance? + +Some architectures, such as CHERI (including Arm's Morello), explicitly consider pointer *provenance* and *bounds* in addition to their target *addresses*. +Adding these considerations to the architecture enables software to constrain uses of *particular pointers* in ways that are not available with traditional protection mechanisms. +For example, while code may *have* a pointer that spans its entire C stack, it may construct a pointer that authorizes access only to a particular stack allocation (e.g., a buffer) and use this latter pointer while copying data. +Even if an attacker is able to control the length of the copy, the bounds imposed upon pointers involved can ensure that an overflow is impossible. +(On the other hand, if the attacker can influence both the *bounds* and the copy length, an overflow may still be possible; in practice, however, the two concerns are often sufficiently separated.) +For `malloc()` in particular, it is enormously beneficial to be able to impose bounds on returned pointers: it becomes impossible for allocator clients to use a pointer from `malloc()` to access adjacent allocations! +(*Temporal* concerns still apply, in that live allocations can overlap prior, now-dead allocations. +Stochastic defenses are employed within `snmalloc` and deterministic defenses are ongoing research at MSR.) + +Borrowing terminology from CHERI, we speak of the **authority** (to a subset of the address space) held by a pointer and will justify actions in terms of this authority.[^mmu-perms] +While many kinds of authority can be envisioned, herein we will mean either + +* *spatial* authority to read/write/execute within a single *interval* within the address space, or +* *vmem* authority to request modification of the virtual page mappings for a given range of addresses. + +We may **bound** the authority of a pointer, deriving a new pointer with a subset of its progenitor's authority; this is assumed to be an ambient action requiring no additional authority. +Dually, given two pointers, one with a subset of the other's authority, we may **amplify** the less-authorized, constructing a pointer with the same address but with increased authority (up to the held superset authority).[^amplifier-state] + +## snmalloc Support For Strict Provenance + +### Static Annotations With CapPtr + +To aid code auditing, `snmalloc` makes heavy use of a `CapPtr` wrapper type around `T*` values. You can think of the annotation `B` on a `CapPtr` as capturing something about the role of the pointer, e.g.: * A pointer to a whole chunk or slab, derived from an internal `void*`. @@ -20,7 +44,7 @@ The remainder of this document... * provides a summary of the constraints imposed on clients, * describes the `StrictProvenance` `capptr_*` functions provided by `ds/ptrwrap.h`, the Architecture Abstraction Layer (AAL), and the Platform Abstraction Layer (PAL). -## Limitations +### Limitations The `CapPtr` and `capptr_*` primitives and derived functions are intended to guide developers in useful directions; they are not security mechanisms in and of themselves. For non-CHERI architectures, the whole edifice crumbles in the face of an overzealous `reinterpret_cast<>` or `unsafe_*ptr` call. @@ -37,8 +61,10 @@ This trend of aliasing continues into higher-level abstractions, such as the fre ### How do I safely get an ordinary pointer to reveal to the client? -Neglecting platform-specific details of getting authority to address space and associating memory in the first place, almost all memory manipulated by `snmalloc` comes from the `AddressSpaceManager`. -Its `reserve(size)` method returns a `capptr::Chunk`; this pointer conveys full authority to the region of `size` at which it points. +Almost all memory manipulated by `snmalloc` frontends comes via the backend's `alloc_chunk` method. +The returned a `capptr::Chunk`; this pointer is spatially bounded to the returned region (which is at least as big as requested). +This pointer is, however, not restricted in its ability to manipulate the address space within its bounds; this permits the frontend to call `mmap` and `madvise` on pages therein. + To derive a pointer that is suitable for client use, we must * further spatially refine the pointer: adjust its offset with `pointer_offset` and use `capptr_bound` and @@ -60,14 +86,14 @@ Nevertheless, if adding a new kind of deallocation, we suggest following the exi * Begin by calling `p_wild = capptr_from_client(p_raw)` to annotate it as `AllocWild` and avoid using the raw form thereafter. -* Check the `Wild` pointer for domestication with `p_tame = capptr_domesticate(state_ptr, p_wild)`; `p_tame` will be a `capptr::Alloc` and will alias `p_wild` or will be `nullptr`. +* Check the `Wild` pointer for domestication with `p_tame = capptr_domesticate(state_ptr, p_wild)`; `p_tame` will be a `capptr::Alloc` and will alias `p_wild` or will be `nullptr`. At this point, we have no more use for `p_wild`. * We may now probe the Pagemap; either `p_tame` is a pointer we have given out or `nullptr`, or this access may trap (especially on platforms where domestication is just a rubber stamp). - This will give us access to the associated `MetaEntry` and, if necessary, a `Chunk`-bounded pointer to the entire backing region. + This will give us access to the associated `MetaEntry` and, in general, a (path to a) `Chunk`-bounded pointer to the entire backing region. * If desired, we can now validate other attributes of the provided capability, including its length, base, and permissions. -In fact, we can even go further and *reconstruct* the capability we would have given out for the indicated allocation, allowing for exact comparison. + In fact, we can even go further and *reconstruct* the capability we would have given out for the indicated allocation, allowing for exact comparison. Eventually we would like to reliably detect references to free objects as part of these flows, especially as frees can change the type of metadata found at the head of a chunk. When that is possible, we will add guidance that only reads of non-pointer scalar types are to be performed until after such tests have confirmed the object's liveness. @@ -79,36 +105,18 @@ Because `CapPtr` are not the kinds of pointers C++ expects to manipulate, Instead, `CapPtr` exposes `as_void()`, `template as_static()`, and `template as_reinterpret()` to perform `static_cast`, `static_cast`, and `reinterpret_cast` (respectively). Please use the first viable option from this list, reserving `reinterpret_cast` for more exciting circumstances. -## StrictProvenance in More Detail - -Tracking pointer *provenance* and *bounds* enables software to constrain uses of *particular pointers* in ways that are not available with traditional protection mechanisms. -For example, while code may *have* a pointer that spans its entire C stack, it may construct a pointer that authorizes access only to a particular stack allocation (e.g., a buffer) and use this latter pointer while copying data. -Even if an attacker is able to control the length of the copy, the bounds imposed upon pointers involved can ensure that an overflow is impossible. -(On the other hand, if the attacker can influence both the *bounds* and the copy length, an overflow may still be possible; in practice, however, the two concerns are often sufficiently separated.) -For `malloc()` in particular, it is enormously beneficial to be able to impose bounds on returned pointers: it becomes impossible for allocator clients to use a pointer from `malloc()` to access adjacent allocations! -(*Temporal* concerns still apply, in that live allocations can overlap prior, now-dead allocations. -Stochastic defenses are employed within `snmalloc` and deterministic defenses are ongoing research at MSR.) - -Borrowing terminology from CHERI, we speak of the **authority** (to a subset of the address space) held by a pointer and will justify actions in terms of this authority.[^mmu-perms] -While many kinds of authority can be envisioned, herein we will mean either - -* *spatial* authority to read/write/execute within a single *interval* within the address space, or -* *vmmap* authority to request modification of the virtual page mappings for a given range of addresses. - -We may **bound** the authority of a pointer, deriving a new pointer with a subset of its progenitor's authority; this is assumed to be an ambient action requiring no additional authority. -Dually, given two pointers, one with a subset of the other's authority, we may **amplify** the less-authorized, constructing a pointer with the same address but with increased authority (up to the held superset authority).[^amplifier-state] - ## Constraints Imposed Upon Allocations `snmalloc` ensures that returned pointers are bounded to no more than the slab entry used to back each allocation. +That is, **no two live allocations will have overlapping bounds**. It may be useful, mostly for debugging, to more precisely bound returned pointers to the actual allocation size,[^bounds-precision] but this is not required for security. -The pointers returned from `alloc()` will also be stripped of their *vmmap* authority, if supported by the platform, ensuring that clients cannot manipulate the page mapping underlying `snmalloc`'s address space. +The pointers returned from `alloc()` will also be stripped of their *vmem* authority, if supported by the platform, ensuring that clients cannot manipulate the page mapping underlying `snmalloc`'s address space. `realloc()`-ation has several policies that may be sensible. -We choose a fairly simple one for the moment: resizing in ways that do not change the backing allocation's `snmalloc` size class are left in place, while any change to the size class triggers an allocate-copy-deallocate sequence. -Even if `realloc()` leaves the object in place, the returned pointer should have its authority bounded as if this were a new allocation (and so may have less authority than `realloc()`'s pointer argument if sub-slab-entry bounds are being applied). -(Notably, this policy is compatible with the existence of size-parameterized deallocation functions: the result of `realloc()` is always associated with the size class corresponding to the requested size. -By contrast, shrinking in place in ways that changed the size class would require tracking the largest size ever associated with the allocation.) +As a holdover from snmalloc v1, we have a fairly simple policy: resizing in ways that do not change the backing allocation's `snmalloc` size class are left in place, while any change to the size class triggers an allocate-copy-deallocate sequence. +Even if `realloc()` leaves the object in place, the returned pointer should have its authority bounded as if this were a new allocation (and so the result may be a subset of the input to `realloc()`). + +Because snmalloc v2 no longer benefits from being provided the size of an allocated object (in, for example, dealloc), we may wish to adopt policies that allow objects to shrink in place beyond the lower bound of their sizeclass. ## Impact of Constraints On Deallocation @@ -117,7 +125,16 @@ These operations relied on being able to take pointers out of bounds, and so pos The current edition of `snmalloc` instead follows pointers (starting from TLS or global roots), using address arithmetic only to derive indicies into these metadata pointers. When the allocator client returns memory (or otherwise refers to an allocation), we will be careful to use the *lower bound* address, not the indicated address per se, for looking up the allocation. -The indicated address may be out of bounds, while `StrictProvenance` architectures should ensure that bounds are monotonically non-increasing, and so the lower bound will always be within the original allocation. +The indicated address may be out of bounds, while `StrictProvenance` architectures should ensure that bounds are monotonically non-increasing, and so either + +* the lower bound will always be within the original allocation. +* the pointer provided by the user will have zero length. + +If we must detach an address from the pointer, as in deallocation, we will generally reject zero-length pointers, as if they were nullptr. + +At the moment, **we permit a pointer to any part of an object to deallocate that object**. +`snmalloc`'s design ensures that we will not create a new, free "object" at an interior pointer but will, instead, always be able to find the beginning and end of the object in question. +In the future we are likely to be more strict, requiring authority to *the entire object* or at least *its lowest-address pointer-sized word* to free it. ## Object Lookup @@ -125,24 +142,25 @@ The indicated address may be out of bounds, while `StrictProvenance` architectur To ensure that this function is not used as an amplification oracle, it must construct a return pointer with the same validity as its input even as it internally accesses metadata. We make `external_pointer` use `pointer_offset` on the user-provided pointer, ensuring that the result has no more authority than the client already held. -XXX It may be worth requiring that the input pointer authorize the entire object? -What are the desired security properties here? - # Adapting the Implementation ## Design Overview For the majority of operations, no `StrictProvenance`-specific reasoning, beyond applying bounds, need be entertained. -However, as regions of memory move into and out of an `AddressSpaceManagerCore` and `ChunkAllocator`, care must be taken to recover (and preserve) the internal, *vmmap*-authorizing pointers from the user's much more tightly bounded pointers. +However, as regions of memory move out of (and back into) the allocator's client and fast free lists, care must be taken to recover (and preserve) the internal, *vmem*-authorizing pointers from the user's much more tightly bounded pointers. We store these internal pointers inside metadata, at different locations for each state: -* For free chunks in `AddressSpaceManagerCore`s, the `next` pointers themselves will be internal pointers. - That is, the head of each list in the `AddressSpaceManagerCore` and the (coerced) next pointers in each `MetaEntry` will be suitable for internal use. +* For free chunks in the "large `Range`" `Pipe`, we expect the `Range` objects themselves to work with these pointers. + In practice, these ranges place these pointers in global state or the `Pagemap` `MetaEntry`s directly. -* Once outside the `AddressSpaceManager`, chunks have a `Metaslab` associated with them, and we can store internal pointers therein (in the `MetaCommon` `chunk` field). +* Once outside the "large `Range`" `Pipe`, chunks holding heap objects will have a `SlabMetadata` structure associated with them, and we can store these high-authority pointers therein. + (Specifically, the `StrictProvenanceBackendSlabMetadata` class adds an `arena` member to the `SlabMetadata` in use.) -Within each slab, there is one or more free list of objects. +* Metadata chunks themselves, however, do not have `SlabMetadata` structures and are managed using a "small `Range`" `Pipe`. + These require special handling, considered below. + +Within each (data) slab, there is (at least) one free list of objects. We take the position that free list entries should be suitable for return, i.e., with authority bounded to their backing slab entry. (However, the *contents* of free memory may be dangerous to expose to the user and require clearing prior to handing out.) @@ -150,7 +168,7 @@ We take the position that free list entries should be suitable for return, i.e., We introduce a multi-dimensional space of bounds. The facets are `enum class`-es in `snmalloc::capptr::dimension`. -* `Spatial` captures the intended spatial extent / role of the pointer: either `Alloc`-ation or `Chunk`. +* `Spatial` captures the intended spatial extent / role of the pointer: `Alloc`-ation, `Chunk`, or an entire `Arena`. * `AddressSpaceControl` captures whether the pointer conveys control of its address space. @@ -161,11 +179,12 @@ This is enforced (loosely) using the `ConceptBound` C++20 concept. The namespace `snmalloc::capptr::bounds` contains particular points in the space of `capptr::bound<>` types: -* bounded to a region of more than `MAX_SIZECLASS_SIZE` bytes with address space control, `Chunk`; -* bounded to a region of more than `MAX_SIZECLASS_SIZE` bytes without address space control, `ChunkUser`; +* bounded to a large region of the address space with address space control, `Arena`; +* bounded to at least `MIN_CHUNK_SIZE` bytes with address space control, `Chunk`; +* bounded to at least `MIN_CHUNK_SIZE` bytes without address space control, `ChunkUser`; * bounded to a smaller region but with address space control, `AllocFull`; * bounded to a smaller region and without address space control, `Alloc`; -* bounded to a smaller region, without address space control, and unverified, `AllocWild`. +* unverified but presumed to be to an `Alloc`-ation, `AllocWild`. ## Primitive Architectural Operations @@ -173,17 +192,17 @@ Several new functions are introduced to AALs to capture primitives of the Archit * `CapPtr capptr_bound(CapPtr a, size_t sz)` spatially bounds the pointer `a` to have authority ranging only from its current target to its current target plus `sz` bytes (which must be within `a`'s authority). No imprecision in authority is permitted. - The bounds annotations must obey `capptr_is_spatial_refinement`. + The bounds annotations must obey `capptr_is_spatial_refinement`: the spatial dimension may change, but the others must be constant. Ultimately, all address space manipulated by `snmalloc` comes from its Platform's primitive allocator. An **arena** is a region returned by that provider. -The `AddressSpaceManager` divides arenas into large allocations and manages their life cycles. +The "large `Range`" `Pipe` serves to carve up `Arena`s; `Arena` pointers become `Chunk`s in the `backend`'s `alloc_chunk`. `snmalloc`'s (new, as of `snmalloc2`) heap layouts ensure that metadata associated with any object are reachable through globals, meaning no explicit amplification is required. ## Primitive Platform Operations * `CapPtr capptr_to_user_address_control(CapPtr f)` sheds authority over the address space from the `CapPtr`, on platforms where that is possible. -On CheriBSD, specifically, this strips the `VMMAP` software permission, ensuring that clients cannot have the kernel manipulate heap pages. +On CheriBSD, specifically, this strips the `VMAP` software permission, ensuring that clients cannot have the kernel manipulate heap pages. The annotation `Bout` is *computed* as a function of `Bin`. In future architectures, this is increasingly likely to be a no-op. @@ -195,17 +214,37 @@ The annotation `Bout` is *computed* as a function of `Bin`. ## Constructed Operators -* `capptr_chunk_is_alloc` converts a `capptr::ChunkUser` to a `capptr::Alloc` with no additional bounding; it is intended to ease auditing. +* `capptr_from_client` wraps a `void *` as a `capptr::AllocWild` to mark it as unverified. + +* `capptr_chunk_is_alloc` converts a `capptr::ChunkUser` to a `capptr::Alloc` without computational effect; it is intended to ease auditing. * `capptr_reveal` converts a `capptr::Alloc` to a `void*`, annotating where we mean to return a pointer to the user. * `capptr_reveal_wild` converts a `capptr::AllocWild` to a `void*`, annotating where we mean to return a *wild* pointer to the user (in `external_pointer`, e.g., where the result is just an offset of the user's pointer). +## Metadata Bounds Handling + +We presently envision three policies for handling metadata: + +1. Metadata kept within `snmalloc` is always `Arena`-bounded; metadata handed to the user (the `Allocator`s themselves) are exactly bounded as `Alloc`. + Recycling internal metadata needs no amplification, and, as `Allocator`s are never deallocated, there is never a need to amplify an `Allocator*` back to a `Chunk`- or `Arena`-bounded pointer. + +2. All metadata is exactly bounded as `Alloc`. + Here, the "small `Range`" `Pipe` will require the ability to amplify back to `Chunk`- or `Arena`-bounded pointers when internal metadata is recycled. + We believe this is straightforwardly possible using a "provenance capturing `Range`" above the outermost `Range` in the small `Range` `Pipe`. + +3. No metadata is handed to the user; instead, opaque handles to `Allocator`s are given out. + (CHERI sealed capabilities are an excellent candidate for such handles, but are beyond the scope of this document.) + Here, it is no longer essential to bound any metadata pointers at all, though we may find it useful as a defence in depth. + + +**At the time of this writing, policy 1 is in effect**; pointers to `Allocator`s are bounded only at the periphery of `snmalloc`. + # Endnotes [^mmu-perms]: Pointer authority generally *intersects* with MMU-based authorization. For example, software using a pointer with both write and execute authority will still find that it cannot write to pages considered read-only by the MMU nor will it be able to execute non-executable pages. -Generally speaking, `snmalloc` requires only read-write access to memory it manages and merely passes through other permissions, with the exception of *vmmap*, which it removes from any pointer it returns. +Generally speaking, `snmalloc` requires only read-write access to memory it manages and merely passes through other permissions, with the exception of *vmem*, which it removes from any pointer it returns. [^amplifier-state]: As we are largely following the fat pointer model and its evolution into CHERI capabilities, we achieve amplification through a *stateful*, *software* mechanism, rather than an architectural mechanism. Specifically, the amplification mechanism will retain a superset of any authority it may be asked to reconstruct. diff --git a/src/snmalloc/aal/aal.h b/src/snmalloc/aal/aal.h index 413c284b0..98606ebd9 100644 --- a/src/snmalloc/aal/aal.h +++ b/src/snmalloc/aal/aal.h @@ -212,7 +212,8 @@ namespace snmalloc "capptr_bound must preserve non-spatial CapPtr dimensions"); UNUSED(size); - return CapPtr(a.template as_static().unsafe_ptr()); + return CapPtr::unsafe_from( + a.template as_static().unsafe_ptr()); } }; } // namespace snmalloc @@ -245,6 +246,17 @@ namespace snmalloc template constexpr static bool aal_supports = (AAL::aal_features & F) == F; + + /* + * The backend's leading-order response to StrictProvenance is entirely + * within its data structures and not actually anything to do with the + * architecture. Rather than test aal_supports or + * defined(__CHERI_PURE_CAPABILITY__) or such therein, using this + * backend_strict_provenance flag makes it easy to test a lot of machinery + * on non-StrictProvenance architectures. + */ + static constexpr bool backend_strict_provenance = + aal_supports; } // namespace snmalloc #ifdef __POINTER_WIDTH__ diff --git a/src/snmalloc/aal/aal_cheri.h b/src/snmalloc/aal/aal_cheri.h index 4774dde67..587fdf126 100644 --- a/src/snmalloc/aal/aal_cheri.h +++ b/src/snmalloc/aal/aal_cheri.h @@ -86,7 +86,7 @@ namespace snmalloc } void* pb = __builtin_cheri_bounds_set_exact(a.unsafe_ptr(), size); - return CapPtr(static_cast(pb)); + return CapPtr::unsafe_from(static_cast(pb)); } }; } // namespace snmalloc diff --git a/src/snmalloc/aal/address.h b/src/snmalloc/aal/address.h index 1f528f980..f37391802 100644 --- a/src/snmalloc/aal/address.h +++ b/src/snmalloc/aal/address.h @@ -34,7 +34,8 @@ namespace snmalloc inline CapPtr pointer_offset(CapPtr base, size_t diff) { - return CapPtr(pointer_offset(base.unsafe_ptr(), diff)); + return CapPtr::unsafe_from( + pointer_offset(base.unsafe_ptr(), diff)); } /** @@ -51,7 +52,8 @@ namespace snmalloc inline CapPtr pointer_offset_signed(CapPtr base, ptrdiff_t diff) { - return CapPtr(pointer_offset_signed(base.unsafe_ptr(), diff)); + return CapPtr::unsafe_from( + pointer_offset_signed(base.unsafe_ptr(), diff)); } /** @@ -137,7 +139,8 @@ namespace snmalloc SNMALLOC_CONCEPT(capptr::ConceptBound) bounds> inline CapPtr pointer_align_down(CapPtr p) { - return CapPtr(pointer_align_down(p.unsafe_ptr())); + return CapPtr::unsafe_from( + pointer_align_down(p.unsafe_ptr())); } template @@ -174,7 +177,8 @@ namespace snmalloc SNMALLOC_CONCEPT(capptr::ConceptBound) bounds> inline CapPtr pointer_align_up(CapPtr p) { - return CapPtr(pointer_align_up(p.unsafe_ptr())); + return CapPtr::unsafe_from( + pointer_align_up(p.unsafe_ptr())); } template @@ -204,7 +208,8 @@ namespace snmalloc inline CapPtr pointer_align_down(CapPtr p, size_t alignment) { - return CapPtr(pointer_align_down(p.unsafe_ptr(), alignment)); + return CapPtr::unsafe_from( + pointer_align_down(p.unsafe_ptr(), alignment)); } /** @@ -228,7 +233,8 @@ namespace snmalloc inline CapPtr pointer_align_up(CapPtr p, size_t alignment) { - return CapPtr(pointer_align_up(p.unsafe_ptr(), alignment)); + return CapPtr::unsafe_from( + pointer_align_up(p.unsafe_ptr(), alignment)); } /** diff --git a/src/snmalloc/backend/backend.h b/src/snmalloc/backend/backend.h index 1463fa946..2870a5fe1 100644 --- a/src/snmalloc/backend/backend.h +++ b/src/snmalloc/backend/backend.h @@ -22,6 +22,10 @@ namespace snmalloc using Pal = PAL; using SlabMetadata = typename PagemapEntry::SlabMetadata; +#ifdef __cpp_concepts + static_assert(IsSlabMeta_Arena); +#endif + public: /** * Provide a block of meta-data with size and align. @@ -35,10 +39,10 @@ namespace snmalloc * does not avail itself of this degree of freedom. */ template - static capptr::Chunk + static capptr::Arena alloc_meta_data(LocalState* local_state, size_t size) { - capptr::Chunk p; + capptr::Arena p; if (local_state != nullptr) { p = local_state->get_meta_range().alloc_range_with_leftover(size); @@ -84,7 +88,7 @@ namespace snmalloc return {nullptr, nullptr}; } - auto p = local_state.object_range.alloc_range(size); + capptr::Arena p = local_state.get_object_range()->alloc_range(size); #ifdef SNMALLOC_TRACING message<1024>("Alloc chunk: {} ({})", p.unsafe_ptr(), size); @@ -97,14 +101,14 @@ namespace snmalloc #ifdef SNMALLOC_TRACING message<1024>("Out of memory"); #endif - return {p, nullptr}; + return {nullptr, nullptr}; } + meta->arena_set(p); typename Pagemap::Entry t(meta, ras); Pagemap::set_metaentry(address_cast(p), size, t); - p = Aal::capptr_bound(p, size); - return {p, meta}; + return {Aal::capptr_bound(p, size), meta}; } /** @@ -140,14 +144,17 @@ namespace snmalloc Pagemap::get_metaentry(address_cast(alloc)).get_slab_metadata()); Pagemap::set_metaentry(address_cast(alloc), size, t); + /* + * On CHERI, the passed alloc has had its bounds narrowed to just the + * Chunk, and so we retrieve the Arena-bounded cap for use in the + * remainder of the backend. + */ + capptr::Arena arena = slab_metadata.arena_get(alloc); + local_state.get_meta_range().dealloc_range( - capptr::Chunk(&slab_metadata), sizeof(SlabMetadata)); + capptr::Arena::unsafe_from(&slab_metadata), sizeof(SlabMetadata)); - // On non-CHERI platforms, we don't need to re-derive to get a pointer to - // the chunk. On CHERI platforms this will need to be stored in the - // SlabMetadata or similar. - capptr::Chunk chunk{alloc.unsafe_ptr()}; - local_state.object_range.dealloc_range(chunk, size); + local_state.get_object_range()->dealloc_range(arena, size); } template diff --git a/src/snmalloc/backend/fixedglobalconfig.h b/src/snmalloc/backend/fixedglobalconfig.h index a48a55f71..1ad18441f 100644 --- a/src/snmalloc/backend/fixedglobalconfig.h +++ b/src/snmalloc/backend/fixedglobalconfig.h @@ -78,9 +78,9 @@ namespace snmalloc // Push memory into the global range. range_to_pow_2_blocks( - capptr::Chunk(heap_base), + capptr::Arena::unsafe_from(heap_base), heap_length, - [&](capptr::Chunk p, size_t sz, bool) { + [&](capptr::Arena p, size_t sz, bool) { typename LocalState::GlobalR g; g.dealloc_range(p, sz); }); @@ -108,8 +108,8 @@ namespace snmalloc return CapPtr< T, - typename B::template with_wildness>( - p.unsafe_ptr()); + typename B::template with_wildness>:: + unsafe_from(p.unsafe_ptr()); } }; } diff --git a/src/snmalloc/backend/meta_protected_range.h b/src/snmalloc/backend/meta_protected_range.h index 46cdbc56c..5c5795cc0 100644 --- a/src/snmalloc/backend/meta_protected_range.h +++ b/src/snmalloc/backend/meta_protected_range.h @@ -103,13 +103,18 @@ namespace snmalloc Pagemap>, SmallBuddyRange>; - public: - using Stats = StatsCombiner; - ObjectRange object_range; MetaRange meta_range; + public: + using Stats = StatsCombiner; + + ObjectRange* get_object_range() + { + return &object_range; + } + MetaRange& get_meta_range() { return meta_range; diff --git a/src/snmalloc/backend/standard_range.h b/src/snmalloc/backend/standard_range.h index d07c9a504..225a9a3e5 100644 --- a/src/snmalloc/backend/standard_range.h +++ b/src/snmalloc/backend/standard_range.h @@ -22,7 +22,7 @@ namespace snmalloc template< typename PAL, typename Pagemap, - typename Base = EmptyRange, + typename Base = EmptyRange<>, size_t MinSizeBits = MinBaseSizeBits()> struct StandardLocalState : BaseLocalStateConstants { @@ -44,23 +44,31 @@ namespace snmalloc static constexpr size_t page_size_bits = bits::next_pow2_bits_const(PAL::page_size); + public: // Source for object allocations and metadata // Use buddy allocators to cache locally. - using ObjectRange = Pipe< + using LargeObjectRange = Pipe< Stats, LargeBuddyRange< LocalCacheSizeBits, LocalCacheSizeBits, Pagemap, - page_size_bits>, - SmallBuddyRange>; + page_size_bits>>; + + private: + using ObjectRange = Pipe; + + ObjectRange object_range; public: // Expose a global range for the initial allocation of meta-data. using GlobalMetaRange = Pipe; // Where we get user allocations from. - ObjectRange object_range; + LargeObjectRange* get_object_range() + { + return object_range.template ancestor(); + } // Where we get meta-data allocations from. ObjectRange& get_meta_range() diff --git a/src/snmalloc/backend_helpers/cheri_slabmetadata_mixin.h b/src/snmalloc/backend_helpers/cheri_slabmetadata_mixin.h new file mode 100644 index 000000000..2e0d75d00 --- /dev/null +++ b/src/snmalloc/backend_helpers/cheri_slabmetadata_mixin.h @@ -0,0 +1,89 @@ +#pragma once +#include "../pal/pal.h" + +namespace snmalloc +{ + /** + * In CHERI, we must retain, internal to the allocator, the authority to + * entire backing arenas, as there is no architectural mechanism to splice + * together two capabilities. Additionally, these capabilities will retain + * the VMAP software permission, conveying our authority to manipulate the + * address space mappings for said arenas. + * + * We stash these pointers inside the SlabMetadata structures for parts of + * the address space for which SlabMetadata exists. (In other parts of the + * system, we will stash them directly in the pagemap.) This requires that + * we inherit from the FrontendSlabMetadata. + */ + template + class StrictProvenanceSlabMetadataMixin : public SlabMetadata + { + template< + SNMALLOC_CONCEPT(ConceptPAL) A1, + typename A2, + typename A3, + typename A4> + friend class BackendAllocator; + + capptr::Arena arena; + + /* Set the arena pointer */ + void arena_set(capptr::Arena a) + { + arena = a; + } + + /* + * Retrieve the stashed pointer for a chunk; the caller must ensure that + * this is the correct arena for the indicated chunk. The latter is unused + * except in debug builds, as there is no architectural amplification. + */ + capptr::Arena arena_get(capptr::Alloc c) + { + SNMALLOC_ASSERT(address_cast(arena) == address_cast(c)); + UNUSED(c); + return arena; + } + }; + + /** + * A dummy implementation of StrictProvenanceBackendSlabMetadata that has no + * computational content, for use on non-StrictProvenance architectures. + */ + template + struct LaxProvenanceSlabMetadataMixin : public SlabMetadata + { + /* On non-StrictProvenance architectures, there's nothing to do */ + void arena_set(capptr::Arena) {} + + /* Just a type sleight of hand, "amplifying" the non-existant bounds */ + capptr::Arena arena_get(capptr::Alloc c) + { + return capptr::Arena::unsafe_from(c.unsafe_ptr()); + } + }; + +#ifdef __cpp_concepts + /** + * Rather than having the backend test backend_strict_provenance in several + * places and doing sleights of hand with the type system, we encapsulate + * the amplification + */ + template + concept IsSlabMeta_Arena = requires(T* t, capptr::Arena p) + { + { + t->arena_set(p) + } + ->ConceptSame; + } + &&requires(T* t, capptr::Alloc p) + { + { + t->arena_get(p) + } + ->ConceptSame>; + }; +#endif + +} // namespace snmalloc diff --git a/src/snmalloc/backend_helpers/commitrange.h b/src/snmalloc/backend_helpers/commitrange.h index de67a1367..2bbd7a583 100644 --- a/src/snmalloc/backend_helpers/commitrange.h +++ b/src/snmalloc/backend_helpers/commitrange.h @@ -18,9 +18,14 @@ namespace snmalloc static constexpr bool ConcurrencySafe = ParentRange::ConcurrencySafe; + using ChunkBounds = typename ParentRange::ChunkBounds; + static_assert( + ChunkBounds::address_space_control == + capptr::dimension::AddressSpaceControl::Full); + constexpr Type() = default; - capptr::Chunk alloc_range(size_t size) + CapPtr alloc_range(size_t size) { SNMALLOC_ASSERT_MSG( (size % PAL::page_size) == 0, @@ -33,7 +38,7 @@ namespace snmalloc return range; } - void dealloc_range(capptr::Chunk base, size_t size) + void dealloc_range(CapPtr base, size_t size) { SNMALLOC_ASSERT_MSG( (size % PAL::page_size) == 0, diff --git a/src/snmalloc/backend_helpers/defaultpagemapentry.h b/src/snmalloc/backend_helpers/defaultpagemapentry.h index 7001f4511..dc95ce653 100644 --- a/src/snmalloc/backend_helpers/defaultpagemapentry.h +++ b/src/snmalloc/backend_helpers/defaultpagemapentry.h @@ -1,6 +1,7 @@ #pragma once #include "../mem/mem.h" +#include "cheri_slabmetadata_mixin.h" namespace snmalloc { @@ -9,7 +10,7 @@ namespace snmalloc * The following class could be replaced by: * * ``` - * using DefaultPagemapEntry = FrontendMetaEntry; + * using DefaultPagemapEntry = FrontendMetaEntry; * ``` * * The full form here provides an example of how to extend the pagemap @@ -17,7 +18,8 @@ namespace snmalloc * constructs meta entries, it only ever reads them or modifies them in * place. */ - class DefaultPagemapEntry : public FrontendMetaEntry + template + class DefaultPagemapEntryT : public FrontendMetaEntry { /** * The private initialising constructor is usable only by this back end. @@ -43,22 +45,28 @@ namespace snmalloc * metadata in meta entries when they are first constructed. */ SNMALLOC_FAST_PATH - DefaultPagemapEntry(FrontendSlabMetadata* meta, uintptr_t ras) - : FrontendMetaEntry(meta, ras) + DefaultPagemapEntryT(SlabMetadata* meta, uintptr_t ras) + : FrontendMetaEntry(meta, ras) {} /** * Copy assignment is used only by the pagemap. */ - DefaultPagemapEntry& operator=(const DefaultPagemapEntry& other) + DefaultPagemapEntryT& operator=(const DefaultPagemapEntryT& other) { - FrontendMetaEntry::operator=(other); + FrontendMetaEntry::operator=(other); return *this; } /** * Default constructor. This must be callable from the pagemap. */ - SNMALLOC_FAST_PATH DefaultPagemapEntry() = default; + SNMALLOC_FAST_PATH DefaultPagemapEntryT() = default; }; -} // namespace snmalloc \ No newline at end of file + + using DefaultPagemapEntry = DefaultPagemapEntryT, + LaxProvenanceSlabMetadataMixin>>; + +} // namespace snmalloc diff --git a/src/snmalloc/backend_helpers/empty_range.h b/src/snmalloc/backend_helpers/empty_range.h index 6507a01e3..0407478a9 100644 --- a/src/snmalloc/backend_helpers/empty_range.h +++ b/src/snmalloc/backend_helpers/empty_range.h @@ -3,6 +3,7 @@ namespace snmalloc { + template class EmptyRange { public: @@ -10,9 +11,11 @@ namespace snmalloc static constexpr bool ConcurrencySafe = true; + using ChunkBounds = B; + constexpr EmptyRange() = default; - capptr::Chunk alloc_range(size_t) + CapPtr alloc_range(size_t) { return nullptr; } diff --git a/src/snmalloc/backend_helpers/globalrange.h b/src/snmalloc/backend_helpers/globalrange.h index 3e01c3d09..760ad21d7 100644 --- a/src/snmalloc/backend_helpers/globalrange.h +++ b/src/snmalloc/backend_helpers/globalrange.h @@ -11,7 +11,7 @@ namespace snmalloc */ struct GlobalRange { - template + template> class Type : public StaticParent { using StaticParent::parent; @@ -27,15 +27,17 @@ namespace snmalloc static constexpr bool ConcurrencySafe = true; + using ChunkBounds = typename ParentRange::ChunkBounds; + constexpr Type() = default; - capptr::Chunk alloc_range(size_t size) + CapPtr alloc_range(size_t size) { FlagLock lock(spin_lock); return parent.alloc_range(size); } - void dealloc_range(capptr::Chunk base, size_t size) + void dealloc_range(CapPtr base, size_t size) { FlagLock lock(spin_lock); parent.dealloc_range(base, size); diff --git a/src/snmalloc/backend_helpers/largebuddyrange.h b/src/snmalloc/backend_helpers/largebuddyrange.h index 7d315973f..d1446d725 100644 --- a/src/snmalloc/backend_helpers/largebuddyrange.h +++ b/src/snmalloc/backend_helpers/largebuddyrange.h @@ -208,7 +208,7 @@ namespace snmalloc bits::one_at_bit(MIN_REFILL_SIZE_BITS); public: - template + template> class Type : public ContainsParent { using ContainsParent::parent; @@ -231,14 +231,14 @@ namespace snmalloc */ template std::enable_if_t - parent_dealloc_range(capptr::Chunk base, size_t size) + parent_dealloc_range(capptr::Arena base, size_t size) { static_assert( MAX_SIZE_BITS != (bits::BITS - 1), "Don't set SFINAE parameter"); parent.dealloc_range(base, size); } - void dealloc_overflow(capptr::Chunk overflow) + void dealloc_overflow(capptr::Arena overflow) { if constexpr (MAX_SIZE_BITS != (bits::BITS - 1)) { @@ -258,18 +258,19 @@ namespace snmalloc * Add a range of memory to the address space. * Divides blocks into power of two sizes with natural alignment */ - void add_range(capptr::Chunk base, size_t length) + void add_range(capptr::Arena base, size_t length) { range_to_pow_2_blocks( - base, length, [this](capptr::Chunk base, size_t align, bool) { - auto overflow = capptr::Chunk(reinterpret_cast( - buddy_large.add_block(base.unsafe_uintptr(), align))); + base, length, [this](capptr::Arena base, size_t align, bool) { + auto overflow = + capptr::Arena::unsafe_from(reinterpret_cast( + buddy_large.add_block(base.unsafe_uintptr(), align))); dealloc_overflow(overflow); }); } - capptr::Chunk refill(size_t size) + capptr::Arena refill(size_t size) { if (ParentRange::Aligned) { @@ -341,9 +342,14 @@ namespace snmalloc static constexpr bool ConcurrencySafe = false; + /* The large buddy allocator always deals in Arena-bounded pointers. */ + using ChunkBounds = capptr::bounds::Arena; + static_assert( + std::is_same_v); + constexpr Type() = default; - capptr::Chunk alloc_range(size_t size) + capptr::Arena alloc_range(size_t size) { SNMALLOC_ASSERT(size >= MIN_CHUNK_SIZE); SNMALLOC_ASSERT(bits::is_pow2(size)); @@ -356,7 +362,7 @@ namespace snmalloc return nullptr; } - auto result = capptr::Chunk( + auto result = capptr::Arena::unsafe_from( reinterpret_cast(buddy_large.remove_block(size))); if (result != nullptr) @@ -365,7 +371,7 @@ namespace snmalloc return refill(size); } - void dealloc_range(capptr::Chunk base, size_t size) + void dealloc_range(capptr::Arena base, size_t size) { SNMALLOC_ASSERT(size >= MIN_CHUNK_SIZE); SNMALLOC_ASSERT(bits::is_pow2(size)); @@ -379,8 +385,9 @@ namespace snmalloc } } - auto overflow = capptr::Chunk(reinterpret_cast( - buddy_large.add_block(base.unsafe_uintptr(), size))); + auto overflow = + capptr::Arena::unsafe_from(reinterpret_cast( + buddy_large.add_block(base.unsafe_uintptr(), size))); dealloc_overflow(overflow); } }; diff --git a/src/snmalloc/backend_helpers/logrange.h b/src/snmalloc/backend_helpers/logrange.h index 9bc1d4731..0a3f907de 100644 --- a/src/snmalloc/backend_helpers/logrange.h +++ b/src/snmalloc/backend_helpers/logrange.h @@ -14,7 +14,7 @@ namespace snmalloc template struct LogRange { - template + template> class Type : public ContainsParent { using ContainsParent::parent; @@ -24,9 +24,11 @@ namespace snmalloc static constexpr bool ConcurrencySafe = ParentRange::ConcurrencySafe; + using ChunkBounds = typename ParentRange::ChunkBounds; + constexpr Type() = default; - capptr::Chunk alloc_range(size_t size) + CapPtr alloc_range(size_t size) { #ifdef SNMALLOC_TRACING message<1024>("Call alloc_range({}) on {}", size, RangeName); @@ -39,7 +41,7 @@ namespace snmalloc return range; } - void dealloc_range(capptr::Chunk base, size_t size) + void dealloc_range(CapPtr base, size_t size) { #ifdef SNMALLOC_TRACING message<1024>( diff --git a/src/snmalloc/backend_helpers/pagemapregisterrange.h b/src/snmalloc/backend_helpers/pagemapregisterrange.h index 7201f8ff9..9a0019e33 100644 --- a/src/snmalloc/backend_helpers/pagemapregisterrange.h +++ b/src/snmalloc/backend_helpers/pagemapregisterrange.h @@ -11,7 +11,7 @@ namespace snmalloc bool CanConsolidate = true> struct PagemapRegisterRange { - template + template> class Type : public ContainsParent { using ContainsParent::parent; @@ -23,7 +23,9 @@ namespace snmalloc static constexpr bool ConcurrencySafe = ParentRange::ConcurrencySafe; - capptr::Chunk alloc_range(size_t size) + using ChunkBounds = typename ParentRange::ChunkBounds; + + CapPtr alloc_range(size_t size) { auto base = parent.alloc_range(size); diff --git a/src/snmalloc/backend_helpers/palrange.h b/src/snmalloc/backend_helpers/palrange.h index 0962e00bf..b1480a2d6 100644 --- a/src/snmalloc/backend_helpers/palrange.h +++ b/src/snmalloc/backend_helpers/palrange.h @@ -14,9 +14,11 @@ namespace snmalloc // need to be changed. static constexpr bool ConcurrencySafe = true; + using ChunkBounds = capptr::bounds::Arena; + constexpr PalRange() = default; - capptr::Chunk alloc_range(size_t size) + capptr::Arena alloc_range(size_t size) { if (bits::next_pow2_bits(size) >= bits::BITS - 1) { @@ -26,8 +28,8 @@ namespace snmalloc if constexpr (pal_supports) { SNMALLOC_ASSERT(size >= PAL::minimum_alloc_size); - auto result = - capptr::Chunk(PAL::template reserve_aligned(size)); + auto result = capptr::Arena::unsafe_from( + PAL::template reserve_aligned(size)); #ifdef SNMALLOC_TRACING message<1024>("Pal range alloc: {} ({})", result.unsafe_ptr(), size); @@ -36,7 +38,7 @@ namespace snmalloc } else { - auto result = capptr::Chunk(PAL::reserve(size)); + auto result = capptr::Arena::unsafe_from(PAL::reserve(size)); #ifdef SNMALLOC_TRACING message<1024>("Pal range alloc: {} ({})", result.unsafe_ptr(), size); diff --git a/src/snmalloc/backend_helpers/range_helpers.h b/src/snmalloc/backend_helpers/range_helpers.h index 90ee8474a..454cba0ba 100644 --- a/src/snmalloc/backend_helpers/range_helpers.h +++ b/src/snmalloc/backend_helpers/range_helpers.h @@ -4,8 +4,11 @@ namespace snmalloc { - template - void range_to_pow_2_blocks(capptr::Chunk base, size_t length, F f) + template< + size_t MIN_BITS, + SNMALLOC_CONCEPT(capptr::ConceptBound) B, + typename F> + void range_to_pow_2_blocks(CapPtr base, size_t length, F f) { auto end = pointer_offset(base, length); base = pointer_align_up(base, bits::one_at_bit(MIN_BITS)); diff --git a/src/snmalloc/backend_helpers/smallbuddyrange.h b/src/snmalloc/backend_helpers/smallbuddyrange.h index fc42d6f7d..f19163b44 100644 --- a/src/snmalloc/backend_helpers/smallbuddyrange.h +++ b/src/snmalloc/backend_helpers/smallbuddyrange.h @@ -10,20 +10,22 @@ namespace snmalloc * struct for representing the redblack nodes * directly inside the meta data. */ + template struct FreeChunk { - capptr::Chunk left; - capptr::Chunk right; + CapPtr left; + CapPtr right; }; /** * Class for using the allocations own space to store in the RBTree. */ + template class BuddyInplaceRep { public: - using Handle = capptr::Chunk*; - using Contents = capptr::Chunk; + using Handle = CapPtr, bounds>*; + using Contents = CapPtr, bounds>; static constexpr Contents null = nullptr; static constexpr Contents root = nullptr; @@ -33,17 +35,17 @@ namespace snmalloc { SNMALLOC_ASSERT((address_cast(r) & MASK) == 0); if (r == nullptr) - *ptr = capptr::Chunk( - reinterpret_cast((*ptr).unsafe_uintptr() & MASK)); + *ptr = CapPtr, bounds>::unsafe_from( + reinterpret_cast*>((*ptr).unsafe_uintptr() & MASK)); else // Preserve lower bit. *ptr = pointer_offset(r, (address_cast(*ptr) & MASK)) - .template as_static(); + .template as_static>(); } static Contents get(Handle ptr) { - return pointer_align_down<2, FreeChunk>((*ptr).as_void()); + return pointer_align_down<2, FreeChunk>((*ptr).as_void()); } static Handle ref(bool direction, Contents r) @@ -66,14 +68,16 @@ namespace snmalloc if (new_is_red != is_red(k)) { auto r = ref(false, k); - auto old_addr = pointer_align_down<2, FreeChunk>(r->as_void()); + auto old_addr = pointer_align_down<2, FreeChunk>(r->as_void()); if (new_is_red) { if (old_addr == nullptr) - *r = capptr::Chunk(reinterpret_cast(MASK)); + *r = CapPtr, bounds>::unsafe_from( + reinterpret_cast*>(MASK)); else - *r = pointer_offset(old_addr, MASK).template as_static(); + *r = pointer_offset(old_addr, MASK) + .template as_static>(); } else { @@ -85,21 +89,22 @@ namespace snmalloc static Contents offset(Contents k, size_t size) { - return pointer_offset(k, size).template as_static(); + return pointer_offset(k, size).template as_static>(); } static Contents buddy(Contents k, size_t size) { // This is just doing xor size, but with what API // exists on capptr. - auto base = pointer_align_down(k.as_void(), size * 2); + auto base = pointer_align_down>(k.as_void(), size * 2); auto offset = (address_cast(k) & size) ^ size; - return pointer_offset(base, offset).template as_static(); + return pointer_offset(base, offset) + .template as_static>(); } static Contents align_down(Contents k, size_t size) { - return pointer_align_down(k.as_void(), size); + return pointer_align_down>(k.as_void(), size); } static bool compare(Contents k1, Contents k2) @@ -145,33 +150,41 @@ namespace snmalloc struct SmallBuddyRange { - template + template> class Type : public ContainsParent { + public: + using ChunkBounds = typename ParentRange::ChunkBounds; + + private: using ContainsParent::parent; static constexpr size_t MIN_BITS = - bits::next_pow2_bits_const(sizeof(FreeChunk)); + bits::next_pow2_bits_const(sizeof(FreeChunk)); - Buddy buddy_small; + Buddy, MIN_BITS, MIN_CHUNK_BITS> buddy_small; /** * Add a range of memory to the address space. * Divides blocks into power of two sizes with natural alignment */ - void add_range(capptr::Chunk base, size_t length) + void add_range(CapPtr base, size_t length) { range_to_pow_2_blocks( - base, length, [this](capptr::Chunk base, size_t align, bool) { - capptr::Chunk overflow = - buddy_small.add_block(base.as_reinterpret(), align) + base, + length, + [this](CapPtr base, size_t align, bool) { + CapPtr overflow = + buddy_small + .add_block( + base.template as_reinterpret>(), align) .template as_reinterpret(); if (overflow != nullptr) parent.dealloc_range(overflow, bits::one_at_bit(MIN_CHUNK_BITS)); }); } - capptr::Chunk refill(size_t size) + CapPtr refill(size_t size) { auto refill = parent.alloc_range(MIN_CHUNK_SIZE); @@ -189,12 +202,9 @@ namespace snmalloc constexpr Type() = default; - capptr::Chunk alloc_range(size_t size) + CapPtr alloc_range(size_t size) { - if (size >= MIN_CHUNK_SIZE) - { - return parent.alloc_range(size); - } + SNMALLOC_ASSERT(size < MIN_CHUNK_SIZE); auto result = buddy_small.remove_block(size); if (result != nullptr) @@ -206,9 +216,9 @@ namespace snmalloc return refill(size); } - capptr::Chunk alloc_range_with_leftover(size_t size) + CapPtr alloc_range_with_leftover(size_t size) { - SNMALLOC_ASSERT(size <= MIN_CHUNK_SIZE); + SNMALLOC_ASSERT(size < MIN_CHUNK_SIZE); auto rsize = bits::next_pow2(size); @@ -224,13 +234,9 @@ namespace snmalloc return result.template as_reinterpret(); } - void dealloc_range(capptr::Chunk base, size_t size) + void dealloc_range(CapPtr base, size_t size) { - if (size >= MIN_CHUNK_SIZE) - { - parent.dealloc_range(base, size); - return; - } + SNMALLOC_ASSERT(size < MIN_CHUNK_SIZE); add_range(base, size); } diff --git a/src/snmalloc/backend_helpers/statsrange.h b/src/snmalloc/backend_helpers/statsrange.h index f38f6e999..8548be9cb 100644 --- a/src/snmalloc/backend_helpers/statsrange.h +++ b/src/snmalloc/backend_helpers/statsrange.h @@ -12,7 +12,7 @@ namespace snmalloc */ struct StatsRange { - template + template> class Type : public ContainsParent { using ContainsParent::parent; @@ -25,9 +25,11 @@ namespace snmalloc static constexpr bool ConcurrencySafe = ParentRange::ConcurrencySafe; + using ChunkBounds = typename ParentRange::ChunkBounds; + constexpr Type() = default; - capptr::Chunk alloc_range(size_t size) + CapPtr alloc_range(size_t size) { auto result = parent.alloc_range(size); if (result != nullptr) @@ -43,7 +45,7 @@ namespace snmalloc return result; } - void dealloc_range(capptr::Chunk base, size_t size) + void dealloc_range(CapPtr base, size_t size) { current_usage -= size; parent.dealloc_range(base, size); diff --git a/src/snmalloc/backend_helpers/subrange.h b/src/snmalloc/backend_helpers/subrange.h index 03c782539..8d886a2b8 100644 --- a/src/snmalloc/backend_helpers/subrange.h +++ b/src/snmalloc/backend_helpers/subrange.h @@ -12,7 +12,7 @@ namespace snmalloc template struct SubRange { - template + template> class Type : public ContainsParent { using ContainsParent::parent; @@ -24,7 +24,9 @@ namespace snmalloc static constexpr bool ConcurrencySafe = ParentRange::ConcurrencySafe; - capptr::Chunk alloc_range(size_t sub_size) + using ChunkBounds = typename ParentRange::ChunkBounds; + + CapPtr alloc_range(size_t sub_size) { SNMALLOC_ASSERT(bits::is_pow2(sub_size)); diff --git a/src/snmalloc/ds_core/ptrwrap.h b/src/snmalloc/ds_core/ptrwrap.h index e7aa85c9a..f60280025 100644 --- a/src/snmalloc/ds_core/ptrwrap.h +++ b/src/snmalloc/ds_core/ptrwrap.h @@ -67,6 +67,11 @@ namespace snmalloc * Bounded to one or more particular chunk granules */ Chunk, + /** + * Unbounded return from the kernel. These correspond, on CHERI + * platforms, to kernel-side address space reservations. + */ + Arena }; /** @@ -154,6 +159,11 @@ namespace snmalloc (S == dimension::Spatial::Alloc && AS == dimension::AddressSpaceControl::User), "Wild pointers must be annotated as tightly bounded"); + static_assert( + (S != dimension::Spatial::Arena) || + (W == dimension::Wildness::Tame && + AS == dimension::AddressSpaceControl::Full), + "Arena pointers must be restricted spatially before other dimensions"); }; // clang-format off @@ -180,8 +190,16 @@ namespace snmalloc namespace bounds { /** - * Internal access to a Chunk of memory. These flow between the ASM and - * the slab allocators, for example. + * Internal access to an entire Arena. These exist only in the backend. + */ + using Arena = bound< + dimension::Spatial::Arena, + dimension::AddressSpaceControl::Full, + dimension::Wildness::Tame>; + + /** + * Internal access to a Chunk of memory. These flow across the boundary + * between back- and front-ends, for example. */ using Chunk = bound< dimension::Spatial::Chunk, @@ -242,15 +260,7 @@ namespace snmalloc return false; } - switch (BI::spatial) - { - using namespace capptr::dimension; - case Spatial::Chunk: - return true; - - case Spatial::Alloc: - return BO::spatial == Spatial::Alloc; - } + return BO::spatial <= BI::spatial; } } // namespace capptr @@ -273,6 +283,7 @@ namespace snmalloc constexpr SNMALLOC_FAST_PATH CapPtr() : CapPtr(nullptr) {} + private: /** * all other constructions must be explicit * @@ -292,13 +303,25 @@ namespace snmalloc # pragma warning(pop) #endif + public: + /** + * The CapPtr constructor is not sufficiently intimidating, given that it + * can be used to break annotation correctness. Expose it with a better + * name. + */ + static constexpr SNMALLOC_FAST_PATH CapPtr unsafe_from(T* p) + { + return CapPtr(p); + } + /** * Allow static_cast<>-s that preserve bounds but vary the target type. */ template [[nodiscard]] SNMALLOC_FAST_PATH CapPtr as_static() const { - return CapPtr(static_cast(this->unsafe_capptr)); + return CapPtr::unsafe_from( + static_cast(this->unsafe_capptr)); } [[nodiscard]] SNMALLOC_FAST_PATH CapPtr as_void() const @@ -312,7 +335,8 @@ namespace snmalloc template [[nodiscard]] SNMALLOC_FAST_PATH CapPtr as_reinterpret() const { - return CapPtr(reinterpret_cast(this->unsafe_capptr)); + return CapPtr::unsafe_from( + reinterpret_cast(this->unsafe_capptr)); } SNMALLOC_FAST_PATH bool operator==(const CapPtr& rhs) const @@ -355,6 +379,9 @@ namespace snmalloc * Aliases for CapPtr<> types with particular bounds. */ + template + using Arena = CapPtr; + template using Chunk = CapPtr; @@ -383,7 +410,7 @@ namespace snmalloc inline SNMALLOC_FAST_PATH capptr::Alloc capptr_chunk_is_alloc(capptr::ChunkUser p) { - return capptr::Alloc(p.unsafe_ptr()); + return capptr::Alloc::unsafe_from(p.unsafe_ptr()); } /** @@ -413,7 +440,7 @@ namespace snmalloc static inline SNMALLOC_FAST_PATH capptr::AllocWild capptr_from_client(void* p) { - return capptr::AllocWild(p); + return capptr::AllocWild::unsafe_from(p); } /** @@ -427,8 +454,8 @@ namespace snmalloc { return CapPtr< T, - typename B::template with_wildness>( - p.unsafe_ptr()); + typename B::template with_wildness>:: + unsafe_from(p.unsafe_ptr()); } /** @@ -477,7 +504,7 @@ namespace snmalloc SNMALLOC_FAST_PATH CapPtr load(std::memory_order order = std::memory_order_seq_cst) noexcept { - return CapPtr(this->unsafe_capptr.load(order)); + return CapPtr::unsafe_from(this->unsafe_capptr.load(order)); } SNMALLOC_FAST_PATH void store( @@ -491,7 +518,7 @@ namespace snmalloc CapPtr desired, std::memory_order order = std::memory_order_seq_cst) noexcept { - return CapPtr( + return CapPtr::unsafe_from( this->unsafe_capptr.exchange(desired.unsafe_ptr(), order)); } diff --git a/src/snmalloc/mem/backend_concept.h b/src/snmalloc/mem/backend_concept.h index 033212507..c0f2d48f9 100644 --- a/src/snmalloc/mem/backend_concept.h +++ b/src/snmalloc/mem/backend_concept.h @@ -122,7 +122,7 @@ namespace snmalloc { Backend::template alloc_meta_data(local_state, size) } - ->ConceptSame>; + ->ConceptSame>; } &&requires( LocalState& local_state, diff --git a/src/snmalloc/mem/backend_wrappers.h b/src/snmalloc/mem/backend_wrappers.h index 605b50554..2d8b7eec2 100644 --- a/src/snmalloc/mem/backend_wrappers.h +++ b/src/snmalloc/mem/backend_wrappers.h @@ -102,8 +102,8 @@ namespace snmalloc UNUSED(ls); return CapPtr< T, - typename B::template with_wildness>( - p.unsafe_ptr()); + typename B::template with_wildness>:: + unsafe_from(p.unsafe_ptr()); } } } // namespace snmalloc diff --git a/src/snmalloc/mem/corealloc.h b/src/snmalloc/mem/corealloc.h index 98cf0b79e..aa6adc5e9 100644 --- a/src/snmalloc/mem/corealloc.h +++ b/src/snmalloc/mem/corealloc.h @@ -506,10 +506,10 @@ namespace snmalloc * The front of the queue has already been validated; just change the * annotating type. */ - auto domesticate_first = [](freelist::QueuePtr p) - SNMALLOC_FAST_PATH_LAMBDA { - return freelist::HeadPtr(p.unsafe_ptr()); - }; + auto domesticate_first = + [](freelist::QueuePtr p) SNMALLOC_FAST_PATH_LAMBDA { + return freelist::HeadPtr::unsafe_from(p.unsafe_ptr()); + }; message_queue().dequeue(key_global, domesticate_first, domesticate, cb); } else diff --git a/src/snmalloc/mem/freelist.h b/src/snmalloc/mem/freelist.h index 3c70307f4..7d677508f 100644 --- a/src/snmalloc/mem/freelist.h +++ b/src/snmalloc/mem/freelist.h @@ -270,7 +270,8 @@ namespace snmalloc inline static BQueuePtr encode_next( address_t curr, BHeadPtr next, const FreeListKey& key) { - return BQueuePtr(code_next(curr, next.unsafe_ptr(), key)); + return BQueuePtr::unsafe_from( + code_next(curr, next.unsafe_ptr(), key)); } /** @@ -294,7 +295,8 @@ namespace snmalloc inline static BHeadPtr decode_next( address_t curr, BHeadPtr next, const FreeListKey& key) { - return BHeadPtr(code_next(curr, next.unsafe_ptr(), key)); + return BHeadPtr::unsafe_from( + code_next(curr, next.unsafe_ptr(), key)); } template< @@ -543,7 +545,7 @@ namespace snmalloc Object::BHeadPtr cast_head(uint32_t ix) { - return Object::BHeadPtr( + return Object::BHeadPtr::unsafe_from( static_cast*>(head[ix])); } @@ -715,8 +717,8 @@ namespace snmalloc // this is doing a CONTAINING_RECORD like cast to get back // to the actual object. This isn't true if the builder is // empty, but you are not allowed to call this in the empty case. - auto last = - Object::BHeadPtr(Object::from_next_ptr(cast_end(0))); + auto last = Object::BHeadPtr::unsafe_from( + Object::from_next_ptr(cast_end(0))); init(); return {first, last}; } diff --git a/src/snmalloc/mem/pool.h b/src/snmalloc/mem/pool.h index 8bb716a1d..0513c141d 100644 --- a/src/snmalloc/mem/pool.h +++ b/src/snmalloc/mem/pool.h @@ -29,7 +29,7 @@ namespace snmalloc private: MPMCStack stack; FlagWord lock{}; - T* list{nullptr}; + capptr::Alloc list{nullptr}; public: constexpr PoolState() = default; @@ -121,12 +121,12 @@ namespace snmalloc static T* acquire(Args&&... args) { PoolState& pool = get_state(); - T* p = pool.stack.pop(); + auto p = capptr::Alloc::unsafe_from(pool.stack.pop()); if (p != nullptr) { p->set_in_use(); - return p; + return p.unsafe_ptr(); } auto raw = @@ -137,14 +137,18 @@ namespace snmalloc Config::Pal::error("Failed to initialise thread local allocator."); } - p = new (raw.unsafe_ptr()) T(std::forward(args)...); + p = capptr_to_user_address_control( + Aal::capptr_bound( + capptr::Arena::unsafe_from(new (raw.unsafe_ptr()) + T(std::forward(args)...)), + sizeof(T))); FlagLock f(pool.lock); p->list_next = pool.list; pool.list = p; p->set_in_use(); - return p; + return p.unsafe_ptr(); } /** @@ -185,9 +189,9 @@ namespace snmalloc static T* iterate(T* p = nullptr) { if (p == nullptr) - return get_state().list; + return get_state().list.unsafe_ptr(); - return p->list_next; + return p->list_next.unsafe_ptr(); } }; } // namespace snmalloc diff --git a/src/snmalloc/mem/pooled.h b/src/snmalloc/mem/pooled.h index 06f964895..51fc3515d 100644 --- a/src/snmalloc/mem/pooled.h +++ b/src/snmalloc/mem/pooled.h @@ -23,7 +23,7 @@ namespace snmalloc /// Used by the pool for chaining together entries when not in use. std::atomic next{nullptr}; /// Used by the pool to keep the list of all entries ever created. - T* list_next; + capptr::Alloc list_next; std::atomic_flag in_use = ATOMIC_FLAG_INIT; public: diff --git a/src/snmalloc/mem/remotecache.h b/src/snmalloc/mem/remotecache.h index 50e4c9bb4..a415a1daa 100644 --- a/src/snmalloc/mem/remotecache.h +++ b/src/snmalloc/mem/remotecache.h @@ -115,7 +115,7 @@ namespace snmalloc if constexpr (Config::Options.QueueHeadsAreTame) { auto domesticate_nop = [](freelist::QueuePtr p) { - return freelist::HeadPtr(p.unsafe_ptr()); + return freelist::HeadPtr::unsafe_from(p.unsafe_ptr()); }; remote->enqueue(first, last, key, domesticate_nop); } diff --git a/src/snmalloc/pal/pal.h b/src/snmalloc/pal/pal.h index 31b846d67..a42508526 100644 --- a/src/snmalloc/pal/pal.h +++ b/src/snmalloc/pal/pal.h @@ -90,7 +90,8 @@ namespace snmalloc CapPtr>> capptr_to_user_address_control(CapPtr p) { - return CapPtr>(p.unsafe_ptr()); + return CapPtr>::unsafe_from( + p.unsafe_ptr()); } template< diff --git a/src/snmalloc/pal/pal_freebsd.h b/src/snmalloc/pal/pal_freebsd.h index d24f4b7cf..4c5c0287d 100644 --- a/src/snmalloc/pal/pal_freebsd.h +++ b/src/snmalloc/pal/pal_freebsd.h @@ -135,7 +135,7 @@ namespace snmalloc return nullptr; } } - return CapPtr>( + return CapPtr>::unsafe_from( __builtin_cheri_perms_and( p.unsafe_ptr(), ~static_cast(CHERI_PERM_SW_VMEM))); } diff --git a/src/test/func/cheri/cheri.cc b/src/test/func/cheri/cheri.cc new file mode 100644 index 000000000..d40f0b8da --- /dev/null +++ b/src/test/func/cheri/cheri.cc @@ -0,0 +1,153 @@ +#include + +#if defined(SNMALLOC_PASS_THROUGH) || !defined(__CHERI_PURE_CAPABILITY__) +// This test does not make sense in pass-through or w/o CHERI +int main() +{ + return 0; +} +#else + +// # define SNMALLOC_TRACING + +# include +# include +# include + +# if defined(__FreeBSD__) +# include +# endif + +using namespace snmalloc; + +bool cap_len_is(void* cap, size_t expected) +{ + return __builtin_cheri_length_get(cap) == expected; +} + +bool cap_vmem_perm_is(void* cap, bool expected) +{ +# if defined(CHERI_PERM_SW_VMEM) + return !!(__builtin_cheri_perms_get(cap) & CHERI_PERM_SW_VMEM) == expected; +# else +# warning "Don't know how to check VMEM permission bit" +# endif +} + +int main() +{ + +# if defined(__FreeBSD__) + { + size_t pagesize[8]; + int err = getpagesizes(pagesize, sizeof(pagesize) / sizeof(pagesize[0])); + SNMALLOC_CHECK(err > 0); + SNMALLOC_CHECK(pagesize[0] == OS_PAGE_SIZE); + } +# endif + + auto alloc = get_scoped_allocator(); + + message("Grab small object"); + { + static const size_t sz = 128; + void* o1 = alloc->alloc(sz); + SNMALLOC_CHECK(cap_len_is(o1, sz)); + SNMALLOC_CHECK(cap_vmem_perm_is(o1, false)); + alloc->dealloc(o1); + } + + /* + * This large object is sized to end up in our alloc's local buddy allocators + * when it's released. + */ + message("Grab large object"); + ptraddr_t alarge; + { + static const size_t sz = 1024 * 1024; + void* olarge = alloc->alloc(sz); + alarge = address_cast(olarge); + SNMALLOC_CHECK(cap_len_is(olarge, sz)); + SNMALLOC_CHECK(cap_vmem_perm_is(olarge, false)); + + static_cast(olarge)[128] = 'x'; + static_cast(olarge)[128 + OS_PAGE_SIZE] = 'y'; + +# if defined(__FreeBSD__) + static constexpr int irm = + MINCORE_INCORE | MINCORE_REFERENCED | MINCORE_MODIFIED; + char ic[2]; + int err = mincore(olarge, 2 * OS_PAGE_SIZE, ic); + SNMALLOC_CHECK(err == 0); + SNMALLOC_CHECK((ic[0] & irm) == irm); + SNMALLOC_CHECK((ic[1] & irm) == irm); + message("Large object in core; good"); +# endif + + alloc->dealloc(olarge); + } + + message("Grab large object again, verify reuse"); + { + static const size_t sz = 1024 * 1024; + errno = 0; + void* olarge = alloc->alloc(sz); + int err = errno; + + SNMALLOC_CHECK(alarge == address_cast(olarge)); + SNMALLOC_CHECK(err == 0); + +# if defined(__FreeBSD__) + /* + * Verify that the zeroing took place by mmap, which should mean that the + * first two pages are not in core. This implies that snmalloc successfully + * re-derived a Chunk- or Arena-bounded pointer and used that, and its VMAP + * permission, to tear pages out of the address space. + */ + static constexpr int irm = + MINCORE_INCORE | MINCORE_REFERENCED | MINCORE_MODIFIED; + char ic[2]; + err = mincore(olarge, 2 * OS_PAGE_SIZE, ic); + SNMALLOC_CHECK(err == 0); + SNMALLOC_CHECK((ic[0] & irm) == 0); + SNMALLOC_CHECK((ic[1] & irm) == 0); + message("Large object not in core; good"); +# endif + + SNMALLOC_CHECK(static_cast(olarge)[128] == '\0'); + SNMALLOC_CHECK(static_cast(olarge)[128 + OS_PAGE_SIZE] == '\0'); + SNMALLOC_CHECK(cap_len_is(olarge, sz)); + SNMALLOC_CHECK(cap_vmem_perm_is(olarge, false)); + + alloc->dealloc(olarge); + } + + /* + * Grab another CoreAlloc pointer from the pool and examine it. + * + * CoreAlloc-s come from the metadata pools of snmalloc, and so do not flow + * through the usual allocation machinery. + */ + message("Grab CoreAlloc from pool for inspection"); + { + static_assert( + std::is_same_v>); + + LocalCache lc{&StandardConfig::unused_remote}; + auto* ca = AllocPool::acquire(&lc); + + SNMALLOC_CHECK(cap_len_is(ca, sizeof(*ca))); + SNMALLOC_CHECK(cap_vmem_perm_is(ca, false)); + + /* + * Putting ca back into the pool would require unhooking our local cache, + * and that requires accessing privates. Since it's pretty harmless to do + * so here at the end of our test, just leak it. + */ + } + + message("CHERI checks OK"); + return 0; +} + +#endif diff --git a/src/test/func/domestication/domestication.cc b/src/test/func/domestication/domestication.cc index 123a44133..c0ae60375 100644 --- a/src/test/func/domestication/domestication.cc +++ b/src/test/func/domestication/domestication.cc @@ -108,8 +108,8 @@ namespace snmalloc return CapPtr< T, - typename B::template with_wildness>( - p.unsafe_ptr()); + typename B::template with_wildness>:: + unsafe_from(p.unsafe_ptr()); } }; diff --git a/src/test/func/malloc/malloc.cc b/src/test/func/malloc/malloc.cc index ec28a7d02..1d4c31da9 100644 --- a/src/test/func/malloc/malloc.cc +++ b/src/test/func/malloc/malloc.cc @@ -55,6 +55,14 @@ void check_result(size_t size, size_t align, void* p, int err, bool null) INFO("Cheri size is {}, but required to be {}.", cheri_size, alloc_size); failed = true; } +# if defined(CHERI_PERM_SW_VMEM) + const auto cheri_perms = __builtin_cheri_perms_get(p); + if (cheri_perms & CHERI_PERM_SW_VMEM) + { + INFO("Cheri permissions include VMEM authority"); + failed = true; + } +# endif if (p != nullptr) { /*