WIP: Region module (Non-experimental stable memory API)
#516
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thinking ahead, I think we want the memory instances to be stable, so maybe a class isn't so suitable after all and we should go with a version where each operation takes a token (designating the memory), the token type is a new primitive type that is stable, and we have a separate function for allocating new tokens. The concrete representation of a token might be a pair of a rust memory ID and growable vector of Rust segments used to map offsets to physical pages. If you think that could work. I think that amounts to just exposing your multStableMemoryXXX operations directly. |
luc-blaeser
reviewed
Feb 2, 2023
src/StableMemory.mo
Outdated
| /// ignore StableMemory.grow(10); | ||
| /// let afterSize = StableMemory.size(); | ||
| /// afterSize - beforeSize // => 10 | ||
| /// ``` |
There was a problem hiding this comment.
/// let memory = StableMemory();
/// let beforeSize = memory.size();
/// ignore memory.grow(10);
/// let afterSize = memory.size();
/// afterSize - beforeSize // => 10
There was a problem hiding this comment.
Just a few thoughts (not sure whether relevant or useful):
- Maybe we should abstract the page sizes and memory address space size away, to ease a potential future upgrade to 64-bit.
- The attribute "stable" confused me at the beginning, because it might suggest that other state of Motoko is not stable. Maybe, it could be named "raw memory" or "raw stable memory". (Ideally, if we would aim for true orthogonal persistence with scalable upgrades, where there would be no need for explicit distinction between stable and flexible/non-stable state.)
- As proposed, stringent bounds checking seems like a very important aspect to me.
- As @crusso suggests, some abstraction of a synthetic pointer to stable memory would be useful, such that one can reference across different stable memory instances. I guess a class-based approach could work if it encapsulates the inner raw location and returns the pointer as class instances for convenience.
// not necessarily a stable type
class RawPointer {
memory: RawMemory;
offset: Nat64;
}
// designed as a stable type
class RawMemory {
// internal representation: e.g. addresses of the pages assigned by the allocator.
public func size(): Nat;
public func pointTo(offset: Nat64): RawPointer;
}
func allocate(size: Nat): RawMemory;
func loadNat32(location: RawPointer): Nat32;
func storeNat32(location: RawPointer, value: Nat32);
func loadPointer(location: RawPointer): RawPointer;
func storePointer(location: RawPointer, value: RawPointer);
Maybe, a source of inspiration could be the Java native access API.
There was a problem hiding this comment.
One caveat I see with that is that it would make the pointer arithmetic very awkward, unless we extend all our arithmetic operators to work with RawPointer values.
There was a problem hiding this comment.
I agree, good point. Probably, it would be simpler with immediate offsets (like JNA).
25 tasks
Makes sense. Per the discussion above, and the similar one internally on Slack, I'm revising the proposal to eschew a WDYT? |
|
See also dfinity/motoko#3768 |
matthewhammer
changed the title
StableMemory (Non-Experimental API)Region module (Non-Experimental API)
matthewhammer
changed the title
Region module (Non-Experimental API)Region module (Non-experimental stable memory API)
Draft
2 tasks
|
Closing in favour of #580, mostly so based on current next-moc, not-master. |
crusso
added a commit
that referenced
this pull request
Sep 5, 2023
…ry (#580) Note this is intended for branch next-moc. CI expected to fail for base branch next-moc. Builds fine against moc regions branch Replaces original PR #516 (that I forgot existed but also targeted master, not next-moc as required) - [ ] Update moc PR dfinity/motoko#3768, sources.nix and udpdate base doc there. - [ ] revise overheads # Region Byte-level access to isolated, (virtual) stable memory _regions_. This is a moderately lightweight abstraction over IC _stable memory_ and supports persisting regions of binary data across Motoko upgrades. Use of this module is fully compatible with Motoko's use of _stable variables_, whose persistence mechanism also uses (real) IC stable memory internally, but does not interfere with this API. It is also fully compatible with existing uses of the `ExperimentalStableMemory` library, which has a similar interface, but, only supported a single memory region, without isolation between different applications. Memory is allocated, using `grow(region, pages)`, sequentially and on demand, in units of 64KiB logical pages, starting with 0 allocated pages. New pages are zero initialized. Growth is capped by a soft limit on physical page count controlled by compile-time flag `--max-stable-pages <n>` (the default is 65536, or 4GiB). Each `load` operation loads from region relative byte address `offset` in little-endian format using the natural bit-width of the type in question. The operation traps if attempting to read beyond the current region size. Each `store` operation stores to region relative byte address `offset` in little-endian format using the natural bit-width of the type in question. The operation traps if attempting to write beyond the current region size. Text values can be handled by using `Text.decodeUtf8` and `Text.encodeUtf8`, in conjunction with `loadBlob` and `storeBlob`. The current region allocation and region contents are preserved across upgrades. NB: The IC's actual stable memory size (`ic0.stable_size`) may exceed the total page size reported by summing all regions sizes. This (and the cap on growth) are to accommodate Motoko's stable variables and bookkeeping for regions. Applications that plan to use Motoko stable variables sparingly or not at all can increase `--max-stable-pages` as desired, approaching the IC maximum (initially 8GiB, then 32Gib, currently 64Gib). All applications should reserve at least one page for stable variable data, even when no stable variables are used. Usage: ```motoko no-repl import Region "mo:base/Region"; ``` ## Type `Region` ``` motoko no-repl type Region = Prim.Types.Region ``` A stateful handle to an isolated region of IC stable memory. `Region` is a stable type and regions can be stored in stable variables. ## Value `new` ``` motoko no-repl let new : () -> Region ``` Allocate a new, isolated Region of size 0. Example: ```motoko no-repl let region = Region.new(); assert Region.size(region) == 0; ``` ## Value `id` ``` motoko no-repl let id : Region -> Nat ``` Return a Nat identifying the given region. Maybe be used for equality, comparison and hashing. NB: Regions returned by `new()` are numbered from 16 (regions 0..15 are currently reserved for internal use). Allocate a new, isolated Region of size 0. Example: ```motoko no-repl let region = Region.new(); assert Region.id(region) == 16; ``` ## Value `size` ``` motoko no-repl let size : (region : Region) -> (pages : Nat64) ``` Current size of `region`, in pages. Each page is 64KiB (65536 bytes). Initially `0`. Preserved across upgrades, together with contents of allocated stable memory. Example: ```motoko no-repl let region = Region.new(); let beforeSize = Region.size(region); ignore Region.grow(region, 10); let afterSize = Region.size(region); afterSize - beforeSize // => 10 ``` ## Value `grow` ``` motoko no-repl let grow : (region : Region, newPages : Nat64) -> (oldPages : Nat64) ``` Grow current `size` of `region` by the given number of pages. Each page is 64KiB (65536 bytes). Returns the previous `size` when able to grow. Returns `0xFFFF_FFFF_FFFF_FFFF` if remaining pages insufficient. Every new page is zero-initialized, containing byte 0x00 at every offset. Function `grow` is capped by a soft limit on `size` controlled by compile-time flag `--max-stable-pages <n>` (the default is 65536, or 4GiB). Example: ```motoko no-repl import Error "mo:base/Error"; let region = Region.new(); let beforeSize = Region.grow(region, 10); if (beforeSize == 0xFFFF_FFFF_FFFF_FFFF) { throw Error.reject("Out of memory"); }; let afterSize = Region.size(region); afterSize - beforeSize // => 10 ``` ## Value `loadNat8` ``` motoko no-repl let loadNat8 : (region : Region, offset : Nat64) -> Nat8 ``` Within `region`, load a `Nat8` value from `offset`. Traps on an out-of-bounds access. Example: ```motoko no-repl let region = Region.new(); let offset = 0; let value = 123; Region.storeNat8(region, offset, value); Region.loadNat8(region, offset) // => 123 ``` ## Value `storeNat8` ``` motoko no-repl let storeNat8 : (region : Region, offset : Nat64, value : Nat8) -> () ``` Within `region`, store a `Nat8` value at `offset`. Traps on an out-of-bounds access. Example: ```motoko no-repl let region = Region.new(); let offset = 0; let value = 123; Region.storeNat8(region, offset, value); Region.loadNat8(region, offset) // => 123 ``` ## Value `loadNat16` ``` motoko no-repl let loadNat16 : (region : Region, offset : Nat64) -> Nat16 ``` Within `region`, load a `Nat16` value from `offset`. Traps on an out-of-bounds access. Example: ```motoko no-repl let region = Region.new(); let offset = 0; let value = 123; Region.storeNat16(region, offset, value); Region.loadNat16(region, offset) // => 123 ``` ## Value `storeNat16` ``` motoko no-repl let storeNat16 : (region : Region, offset : Nat64, value : Nat16) -> () ``` Within `region`, store a `Nat16` value at `offset`. Traps on an out-of-bounds access. Example: ```motoko no-repl let region = Region.new(); let offset = 0; let value = 123; Region.storeNat16(region, offset, value); Region.loadNat16(region, offset) // => 123 ``` ## Value `loadNat32` ``` motoko no-repl let loadNat32 : (region : Region, offset : Nat64) -> Nat32 ``` Within `region`, load a `Nat32` value from `offset`. Traps on an out-of-bounds access. Example: ```motoko no-repl let region = Region.new(); let offset = 0; let value = 123; Region.storeNat32(region, offset, value); Region.loadNat32(region, offset) // => 123 ``` ## Value `storeNat32` ``` motoko no-repl let storeNat32 : (region : Region, offset : Nat64, value : Nat32) -> () ``` Within `region`, store a `Nat32` value at `offset`. Traps on an out-of-bounds access. Example: ```motoko no-repl let region = Region.new(); let offset = 0; let value = 123; Region.storeNat32(region, offset, value); Region.loadNat32(region, offset) // => 123 ``` ## Value `loadNat64` ``` motoko no-repl let loadNat64 : (region : Region, offset : Nat64) -> Nat64 ``` Within `region`, load a `Nat64` value from `offset`. Traps on an out-of-bounds access. Example: ```motoko no-repl let region = Region.new(); let offset = 0; let value = 123; Region.storeNat64(region, offset, value); Region.loadNat64(region, offset) // => 123 ``` ## Value `storeNat64` ``` motoko no-repl let storeNat64 : (region : Region, offset : Nat64, value : Nat64) -> () ``` Within `region`, store a `Nat64` value at `offset`. Traps on an out-of-bounds access. Example: ```motoko no-repl let region = Region.new(); let offset = 0; let value = 123; Region.storeNat64(region, offset, value); Region.loadNat64(region, offset) // => 123 ``` ## Value `loadInt8` ``` motoko no-repl let loadInt8 : (region : Region, offset : Nat64) -> Int8 ``` Within `region`, load a `Int8` value from `offset`. Traps on an out-of-bounds access. Example: ```motoko no-repl let region = Region.new(); let offset = 0; let value = 123; Region.storeInt8(region, offset, value); Region.loadInt8(region, offset) // => 123 ``` ## Value `storeInt8` ``` motoko no-repl let storeInt8 : (region : Region, offset : Nat64, value : Int8) -> () ``` Within `region`, store a `Int8` value at `offset`. Traps on an out-of-bounds access. Example: ```motoko no-repl let region = Region.new(); let offset = 0; let value = 123; Region.storeInt8(region, offset, value); Region.loadInt8(region, offset) // => 123 ``` ## Value `loadInt16` ``` motoko no-repl let loadInt16 : (region : Region, offset : Nat64) -> Int16 ``` Within `region`, load a `Int16` value from `offset`. Traps on an out-of-bounds access. Example: ```motoko no-repl let region = Region.new(); let offset = 0; let value = 123; Region.storeInt16(region, offset, value); Region.loadInt16(region, offset) // => 123 ``` ## Value `storeInt16` ``` motoko no-repl let storeInt16 : (region : Region, offset : Nat64, value : Int16) -> () ``` Within `region`, store a `Int16` value at `offset`. Traps on an out-of-bounds access. Example: ```motoko no-repl let region = Region.new(); let offset = 0; let value = 123; Region.storeInt16(region, offset, value); Region.loadInt16(region, offset) // => 123 ``` ## Value `loadInt32` ``` motoko no-repl let loadInt32 : (region : Region, offset : Nat64) -> Int32 ``` Within `region`, load a `Int32` value from `offset`. Traps on an out-of-bounds access. Example: ```motoko no-repl let region = Region.new(); let offset = 0; let value = 123; Region.storeInt32(region, offset, value); Region.loadInt32(region, offset) // => 123 ``` ## Value `storeInt32` ``` motoko no-repl let storeInt32 : (region : Region, offset : Nat64, value : Int32) -> () ``` Within `region`, store a `Int32` value at `offset`. Traps on an out-of-bounds access. Example: ```motoko no-repl let region = Region.new(); let offset = 0; let value = 123; Region.storeInt32(region, offset, value); Region.loadInt32(region, offset) // => 123 ``` ## Value `loadInt64` ``` motoko no-repl let loadInt64 : (region : Region, offset : Nat64) -> Int64 ``` Within `region`, load a `Int64` value from `offset`. Traps on an out-of-bounds access. Example: ```motoko no-repl let region = Region.new(); let offset = 0; let value = 123; Region.storeInt64(region, offset, value); Region.loadInt64(region, offset) // => 123 ``` ## Value `storeInt64` ``` motoko no-repl let storeInt64 : (region : Region, offset : Nat64, value : Int64) -> () ``` Within `region`, store a `Int64` value at `offset`. Traps on an out-of-bounds access. Example: ```motoko no-repl let region = Region.new(); let offset = 0; let value = 123; Region.storeInt64(region, offset, value); Region.loadInt64(region, offset) // => 123 ``` ## Value `loadFloat` ``` motoko no-repl let loadFloat : (region : Region, offset : Nat64) -> Float ``` Within `region`, loads a `Float` value from the given `offset`. Traps on an out-of-bounds access. Example: ```motoko no-repl let region = Region.new(); let offset = 0; let value = 1.25; Region.storeFloat(region, offset, value); Region.loadFloat(region, offset) // => 1.25 ``` ## Value `storeFloat` ``` motoko no-repl let storeFloat : (region : Region, offset : Nat64, value : Float) -> () ``` Within `region`, store float `value` at the given `offset`. Traps on an out-of-bounds access. Example: ```motoko no-repl let region = Region.new(); let offset = 0; let value = 1.25; Region.storeFloat(region, offset, value); Region.loadFloat(region, offset) // => 1.25 ``` ## Value `loadBlob` ``` motoko no-repl let loadBlob : (region : Region, offset : Nat64, size : Nat) -> Blob ``` Within `region,` load `size` bytes starting from `offset` as a `Blob`. Traps on an out-of-bounds access. Example: ```motoko no-repl import Blob "mo:base/Blob"; let region = Region.new(); let offset = 0; let value = Blob.fromArray([1, 2, 3]); let size = value.size(); Region.storeBlob(region, offset, value); Blob.toArray(Region.loadBlob(region, offset, size)) // => [1, 2, 3] ``` ## Value `storeBlob` ``` motoko no-repl let storeBlob : (region : Region, offset : Nat64, value : Blob) -> () ``` Within `region, write `blob.size()` bytes of `blob` beginning at `offset`. Traps on an out-of-bounds access. Example: ```motoko no-repl import Blob "mo:base/Blob"; let region = Region.new(); let offset = 0; let value = Blob.fromArray([1, 2, 3]); let size = value.size(); Region.storeBlob(region, offset, value); Blob.toArray(Region.loadBlob(region, offset, size)) // => [1, 2, 3] ```
mergify bot
pushed a commit
to dfinity/motoko
that referenced
this pull request
Sep 6, 2023
…3768) Runtime-system implementation of the API described in dfinity/motoko-base#516 Checklist: - [x] pass existing tests for existing (experimental) stable memory, using region0 as that stable memory. - [x] distinct regions are isolated (see `test/run-drun/stable-regions-are-isolated.mo`). - [x] serialize/deserialize `Region` type. - [x] migration path for existing (experimental) stable memory, into region0 of new region manager. - [x] trap when Region.new fails? - [x] add version to metadata - [x] basic perf test - [x] maybe remove (most of) Region0.rs - [x] align errors - [ ] improve test region0/stable-mem-big-blog.mo to check contents of large read/writes (done in halves) - [ ] restrict access to new somehow (can be done later, but nicer to have on release). - [x] rts_stable_mem_size? - [x] max-stable-pages (limits physical pages, i.e. include meta-data) - [ ] registers not stack allocation? - [x] clean up asserts on tag ranges - [x] lazy metadata allocation? - [x] add Region.mo to next_moc - [x] document Region.mo - [ ] perf maybe: turn ic_mem_fns into ordinary externs where poss - affects perf IIRC from experience of @luc-blaeser (NOTE, I couldn't see this myself after a simple experiment) - [x] elim Region0.rs operations wrappers and just do the conditional compilation in compile.ml - bench shows inlining should be a win. - [ ] simplify chunked read writes to iterate over vec_pages entries directly should be cheaper. - [x] specify Region type in manual TODO separately: - [ ] add stable-regions.md to portal doc (sidebars.js) - [ ] gracefully fail compilation of stablemem/region ops by generating trapping functions - [ ] protect access to Region.new See also: - [Forum post, for community discussion.](https://forum.dfinity.org/t/motoko-stable-regions/19182) - **StableBTree** edits needed to upgrade to this system: - sardariuss/MotokoStableBTree#4 - sardariuss/MotokoStableBTreeTest#1
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR will likely be somewhat long-lived.
WIP because it doesn't make sense to merge until the compiler primitives work as expected.
See also the companion PR for the compiler dfinity/motoko#3768