feat: Region.mo library offering isolated regions of IC stable memory

[crusso]

Note this is intended for branch next-moc.
CI expected to fail for base branch next-moc.
Builds fine against moc regions branch

Replaces original PR #516 (that I forgot existed but also targeted master, not next-moc as required)

• Update moc PR Multiple stable memory regions (and a new primitive type Region). motoko#3768, sources.nix and udpdate base doc there.
• revise overheads

Region

Byte-level access to isolated, (virtual) stable memory regions.

This is a moderately lightweight abstraction over IC stable memory and supports persisting
regions of binary data across Motoko upgrades.
Use of this module is fully compatible with Motoko's use of
stable variables, whose persistence mechanism also uses (real) IC stable memory internally, but does not interfere with this API.
It is also fully compatible with existing uses of the ExperimentalStableMemory library, which has a similar interface, but,
only supported a single memory region, without isolation between different applications.

Memory is allocated, using grow(region, pages), sequentially and on demand, in units of 64KiB logical pages, starting with 0 allocated pages.
New pages are zero initialized.
Growth is capped by a soft limit on physical page count controlled by compile-time flag
--max-stable-pages <n> (the default is 65536, or 4GiB).

Each load operation loads from region relative byte address offset in little-endian
format using the natural bit-width of the type in question.
The operation traps if attempting to read beyond the current region size.

Each store operation stores to region relative byte address offset in little-endian format using the natural bit-width of the type in question.
The operation traps if attempting to write beyond the current region size.

Text values can be handled by using Text.decodeUtf8 and Text.encodeUtf8, in conjunction with loadBlob and storeBlob.

The current region allocation and region contents are preserved across upgrades.

NB: The IC's actual stable memory size (ic0.stable_size) may exceed the
total page size reported by summing all regions sizes.
This (and the cap on growth) are to accommodate Motoko's stable variables and bookkeeping for regions.
Applications that plan to use Motoko stable variables sparingly or not at all can
increase --max-stable-pages as desired, approaching the IC maximum (initially 8GiB, then 32Gib, currently 64Gib).
All applications should reserve at least one page for stable variable data, even when no stable variables are used.

Usage:

import Region "mo:base/Region";

Type Region

type Region = Prim.Types.Region

A stateful handle to an isolated region of IC stable memory.
Region is a stable type and regions can be stored in stable variables.

Value new

let new : () -> Region

Allocate a new, isolated Region of size 0.

Example:

let region = Region.new();
assert Region.size(region) == 0;

Value id

let id : Region -> Nat

Return a Nat identifying the given region.
Maybe be used for equality, comparison and hashing.
NB: Regions returned by new() are numbered from 16
(regions 0..15 are currently reserved for internal use).
Allocate a new, isolated Region of size 0.

Example:

let region = Region.new();
assert Region.id(region) == 16;

Value size

let size : (region : Region) -> (pages : Nat64)

Current size of region, in pages.
Each page is 64KiB (65536 bytes).
Initially 0.
Preserved across upgrades, together with contents of allocated
stable memory.

Example:

let region = Region.new();
let beforeSize = Region.size(region);
ignore Region.grow(region, 10);
let afterSize = Region.size(region);
afterSize - beforeSize // => 10

Value grow

let grow : (region : Region, newPages : Nat64) -> (oldPages : Nat64)

Grow current size of region by the given number of pages.
Each page is 64KiB (65536 bytes).
Returns the previous size when able to grow.
Returns 0xFFFF_FFFF_FFFF_FFFF if remaining pages insufficient.
Every new page is zero-initialized, containing byte 0x00 at every offset.
Function grow is capped by a soft limit on size controlled by compile-time flag
--max-stable-pages <n> (the default is 65536, or 4GiB).

Example:

import Error "mo:base/Error";

let region = Region.new();
let beforeSize = Region.grow(region, 10);
if (beforeSize == 0xFFFF_FFFF_FFFF_FFFF) {
  throw Error.reject("Out of memory");
};
let afterSize = Region.size(region);
afterSize - beforeSize // => 10

Value loadNat8

let loadNat8 : (region : Region, offset : Nat64) -> Nat8

Within region, load a Nat8 value from offset.
Traps on an out-of-bounds access.

Example:

let region = Region.new();
let offset = 0;
let value = 123;
Region.storeNat8(region, offset, value);
Region.loadNat8(region, offset) // => 123

Value storeNat8

let storeNat8 : (region : Region, offset : Nat64, value : Nat8) -> ()

Within region, store a Nat8 value at offset.
Traps on an out-of-bounds access.

Example:

let region = Region.new();
let offset = 0;
let value = 123;
Region.storeNat8(region, offset, value);
Region.loadNat8(region, offset) // => 123

Value loadNat16

let loadNat16 : (region : Region, offset : Nat64) -> Nat16

Within region, load a Nat16 value from offset.
Traps on an out-of-bounds access.

Example:

let region = Region.new();
let offset = 0;
let value = 123;
Region.storeNat16(region, offset, value);
Region.loadNat16(region, offset) // => 123

Value storeNat16

let storeNat16 : (region : Region, offset : Nat64, value : Nat16) -> ()

Within region, store a Nat16 value at offset.
Traps on an out-of-bounds access.

Example:

let region = Region.new();
let offset = 0;
let value = 123;
Region.storeNat16(region, offset, value);
Region.loadNat16(region, offset) // => 123

Value loadNat32

let loadNat32 : (region : Region, offset : Nat64) -> Nat32

Within region, load a Nat32 value from offset.
Traps on an out-of-bounds access.

Example:

let region = Region.new();
let offset = 0;
let value = 123;
Region.storeNat32(region, offset, value);
Region.loadNat32(region, offset) // => 123

Value storeNat32

let storeNat32 : (region : Region, offset : Nat64, value : Nat32) -> ()

Within region, store a Nat32 value at offset.
Traps on an out-of-bounds access.

Example:

let region = Region.new();
let offset = 0;
let value = 123;
Region.storeNat32(region, offset, value);
Region.loadNat32(region, offset) // => 123

Value loadNat64

let loadNat64 : (region : Region, offset : Nat64) -> Nat64

Within region, load a Nat64 value from offset.
Traps on an out-of-bounds access.

Example:

let region = Region.new();
let offset = 0;
let value = 123;
Region.storeNat64(region, offset, value);
Region.loadNat64(region, offset) // => 123

Value storeNat64

let storeNat64 : (region : Region, offset : Nat64, value : Nat64) -> ()

Within region, store a Nat64 value at offset.
Traps on an out-of-bounds access.

Example:

let region = Region.new();
let offset = 0;
let value = 123;
Region.storeNat64(region, offset, value);
Region.loadNat64(region, offset) // => 123

Value loadInt8

let loadInt8 : (region : Region, offset : Nat64) -> Int8

Within region, load a Int8 value from offset.
Traps on an out-of-bounds access.

Example:

let region = Region.new();
let offset = 0;
let value = 123;
Region.storeInt8(region, offset, value);
Region.loadInt8(region, offset) // => 123

Value storeInt8

let storeInt8 : (region : Region, offset : Nat64, value : Int8) -> ()

Within region, store a Int8 value at offset.
Traps on an out-of-bounds access.

Example:

let region = Region.new();
let offset = 0;
let value = 123;
Region.storeInt8(region, offset, value);
Region.loadInt8(region, offset) // => 123

Value loadInt16

let loadInt16 : (region : Region, offset : Nat64) -> Int16

Within region, load a Int16 value from offset.
Traps on an out-of-bounds access.

Example:

let region = Region.new();
let offset = 0;
let value = 123;
Region.storeInt16(region, offset, value);
Region.loadInt16(region, offset) // => 123

Value storeInt16

let storeInt16 : (region : Region, offset : Nat64, value : Int16) -> ()

Within region, store a Int16 value at offset.
Traps on an out-of-bounds access.

Example:

let region = Region.new();
let offset = 0;
let value = 123;
Region.storeInt16(region, offset, value);
Region.loadInt16(region, offset) // => 123

Value loadInt32

let loadInt32 : (region : Region, offset : Nat64) -> Int32

Within region, load a Int32 value from offset.
Traps on an out-of-bounds access.

Example:

let region = Region.new();
let offset = 0;
let value = 123;
Region.storeInt32(region, offset, value);
Region.loadInt32(region, offset) // => 123

Value storeInt32

let storeInt32 : (region : Region, offset : Nat64, value : Int32) -> ()

Within region, store a Int32 value at offset.
Traps on an out-of-bounds access.

Example:

let region = Region.new();
let offset = 0;
let value = 123;
Region.storeInt32(region, offset, value);
Region.loadInt32(region, offset) // => 123

Value loadInt64

let loadInt64 : (region : Region, offset : Nat64) -> Int64

Within region, load a Int64 value from offset.
Traps on an out-of-bounds access.

Example:

let region = Region.new();
let offset = 0;
let value = 123;
Region.storeInt64(region, offset, value);
Region.loadInt64(region, offset) // => 123

Value storeInt64

let storeInt64 : (region : Region, offset : Nat64, value : Int64) -> ()

Within region, store a Int64 value at offset.
Traps on an out-of-bounds access.

Example:

let region = Region.new();
let offset = 0;
let value = 123;
Region.storeInt64(region, offset, value);
Region.loadInt64(region, offset) // => 123

Value loadFloat

let loadFloat : (region : Region, offset : Nat64) -> Float

Within region, loads a Float value from the given offset.
Traps on an out-of-bounds access.

Example:

let region = Region.new();
let offset = 0;
let value = 1.25;
Region.storeFloat(region, offset, value);
Region.loadFloat(region, offset) // => 1.25

Value storeFloat

let storeFloat : (region : Region, offset : Nat64, value : Float) -> ()

Within region, store float value at the given offset.
Traps on an out-of-bounds access.

Example:

let region = Region.new();
let offset = 0;
let value = 1.25;
Region.storeFloat(region, offset, value);
Region.loadFloat(region, offset) // => 1.25

Value loadBlob

let loadBlob : (region : Region, offset : Nat64, size : Nat) -> Blob

Within region, load size bytes starting from offset as a Blob.
Traps on an out-of-bounds access.

Example:

import Blob "mo:base/Blob";

let region = Region.new();
let offset = 0;
let value = Blob.fromArray([1, 2, 3]);
let size = value.size();
Region.storeBlob(region, offset, value);
Blob.toArray(Region.loadBlob(region, offset, size)) // => [1, 2, 3]

Value storeBlob

let storeBlob : (region : Region, offset : Nat64, value : Blob) -> ()

Within region, write blob.size()bytes ofblobbeginning atoffset`.
Traps on an out-of-bounds access.

Example:

import Blob "mo:base/Blob";

let region = Region.new();
let offset = 0;
let value = Blob.fromArray([1, 2, 3]);
let size = value.size();
Region.storeBlob(region, offset, value);
Blob.toArray(Region.loadBlob(region, offset, size)) // => [1, 2, 3]

[crusso]

Minor fixes

[ggreif]

Rubber-stamping!