feat: indexable `Blob`s #424

ggreif · 2025-11-18T10:52:59Z

See dfinity/motoko#5018.

TODO:

also do this for base!
at method
get method
doctests

https://github.com/dfinity/motoko/pull/5018/files TODO: - [ ] also do this for `base`! - [ ] `index` method - [ ] test

github-actions · 2025-11-18T10:54:04Z

✨ Documentation preview for bda9424:

https://dfinity.github.io/motoko-core/pull/424 (source code)

github-actions · 2025-11-18T10:54:35Z

Benchmark Results

bench/ArrayBuilding.bench.mo $({\color{green}-0.19\%})$

Large known-size array building

Compares performance of different data structures for building arrays of known size.

Instructions: ${\color{green}-0.13\%}$
Heap: ${\color{gray}0\%}$
Stable Memory: ${\color{gray}0\%}$
Garbage Collection: ${\color{green}-0.05\%}$

Instructions

	1000	100000	1000000
List	542_157 $({\color{green}-1.11\%})$	47_994_641 $({\color{green}-0.68\%})$	475_069_697 $({\color{green}-0.65\%})$
Buffer	342_007 $({\color{red}+0.00\%})$	33_903_437 $({\color{red}+0.00\%})$	339_003_652 $({\color{red}+0.00\%})$
pure/List	302_137 $({\color{red}+0.00\%})$	30_003_592 $({\color{red}+0.00\%})$	300_055_974 $({\color{red}+0.00\%})$
VarArray ?T	180_528 $({\color{red}+0.00\%})$	17_802_958 $({\color{red}+0.00\%})$	178_003_173 $({\color{red}+0.00\%})$
VarArray T	160_815 $({\color{red}+0.00\%})$	15_803_245 $({\color{red}+0.00\%})$	158_003_460 $({\color{red}+0.00\%})$
Array (baseline)	42_697 $({\color{red}+0.00\%})$	4_003_127 $({\color{red}+0.00\%})$	40_003_342 $({\color{red}+0.00\%})$

Heap

	1000	100000	1000000
List	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
Buffer	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
pure/List	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
VarArray ?T	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
VarArray T	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
Array (baseline)	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$

Garbage Collection

	1000	100000	1000000
List	9.96 KiB $({\color{green}-0.97\%})$	797.46 KiB $({\color{green}-0.01\%})$	7.67 MiB $({\color{green}-0.00\%})$
Buffer	8.71 KiB $({\color{gray}0\%})$	782.15 KiB $({\color{gray}0\%})$	7.63 MiB $({\color{gray}0\%})$
pure/List	19.95 KiB $({\color{gray}0\%})$	1.91 MiB $({\color{gray}0\%})$	19.07 MiB $({\color{gray}0\%})$
VarArray ?T	8.24 KiB $({\color{gray}0\%})$	781.68 KiB $({\color{gray}0\%})$	7.63 MiB $({\color{gray}0\%})$
VarArray T	8.23 KiB $({\color{gray}0\%})$	781.67 KiB $({\color{gray}0\%})$	7.63 MiB $({\color{gray}0\%})$
Array (baseline)	4.3 KiB $({\color{gray}0\%})$	391.02 KiB $({\color{gray}0\%})$	3.82 MiB $({\color{gray}0\%})$

bench/FromIters.bench.mo $({\color{gray}0\%})$

Benchmarking the fromIter functions

Columns describe the number of elements in the input iter.

Instructions: ${\color{gray}0\%}$
Heap: ${\color{gray}0\%}$
Stable Memory: ${\color{gray}0\%}$
Garbage Collection: ${\color{gray}0\%}$

Instructions

	100	10_000	100_000
Array.fromIter	48_764 $({\color{gray}0\%})$	4_712_025 $({\color{gray}0\%})$	47_103_135 $({\color{gray}0\%})$
List.fromIter	31_698 $({\color{gray}0\%})$	3_061_541 $({\color{gray}0\%})$	30_603_553 $({\color{gray}0\%})$
List.fromIter . Iter.reverse	50_297 $({\color{gray}0\%})$	4_832_563 $({\color{gray}0\%})$	48_305_477 $({\color{gray}0\%})$

Heap

	100	10_000	100_000
Array.fromIter	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
List.fromIter	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
List.fromIter . Iter.reverse	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$

Garbage Collection

	100	10_000	100_000
Array.fromIter	2.76 KiB $({\color{gray}0\%})$	234.79 KiB $({\color{gray}0\%})$	2.29 MiB $({\color{gray}0\%})$
List.fromIter	3.51 KiB $({\color{gray}0\%})$	312.88 KiB $({\color{gray}0\%})$	3.05 MiB $({\color{gray}0\%})$
List.fromIter . Iter.reverse	5.11 KiB $({\color{gray}0\%})$	469.17 KiB $({\color{gray}0\%})$	4.58 MiB $({\color{gray}0\%})$

bench/ListBufferNewArray.bench.mo $({\color{green}-6.13\%})$

List vs. Buffer for creating known-size arrays

Performance comparison between List and Buffer for creating a new array.

Instructions: ${\color{green}-1.94\%}$
Heap: ${\color{gray}0\%}$
Stable Memory: ${\color{gray}0\%}$
Garbage Collection: ${\color{green}-4.19\%}$

Instructions

	0 (baseline)	1	5	10	100 (for loop)
List	1_367 $({\color{green}-11.64\%})$	2_687 $({\color{green}-7.85\%})$	8_667 $({\color{green}-4.19\%})$	13_462 $({\color{green}-3.48\%})$	73_164 $({\color{green}-1.88\%})$
pure/List	1_247 $({\color{gray}0\%})$	1_355 $({\color{gray}0\%})$	2_439 $({\color{gray}0\%})$	3_801 $({\color{gray}0\%})$	31_868 $({\color{gray}0\%})$
Buffer	2_119 $({\color{gray}0\%})$	2_271 $({\color{gray}0\%})$	3_518 $({\color{gray}0\%})$	5_085 $({\color{gray}0\%})$	36_640 $({\color{gray}0\%})$

Heap

	0 (baseline)	1	5	10	100 (for loop)
List	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
pure/List	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
Buffer	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$

Garbage Collection

	0 (baseline)	1	5	10	100 (for loop)
List	476 B $({\color{green}-17.36\%})$	516 B $({\color{green}-16.23\%})$	676 B $({\color{green}-12.89\%})$	784 B $({\color{green}-11.31\%})$	1.84 KiB $({\color{green}-5.05\%})$
pure/List	360 B $({\color{gray}0\%})$	380 B $({\color{gray}0\%})$	460 B $({\color{gray}0\%})$	560 B $({\color{gray}0\%})$	2.3 KiB $({\color{gray}0\%})$
Buffer	856 B $({\color{gray}0\%})$	864 B $({\color{gray}0\%})$	896 B $({\color{gray}0\%})$	936 B $({\color{gray}0\%})$	1.62 KiB $({\color{gray}0\%})$

bench/PriorityQueues.bench.mo $({\color{green}-2.68\%})$

Different priority queue implementations

_Compare the performance of the following priority queue implementations:

PriorityQueue: Binary heap implementation over List.
PriorityQueueSet: Wrapper over Set<(T, Nat)>._

Instructions: ${\color{green}-2.70\%}$
Heap: ${\color{gray}0\%}$
Stable Memory: ${\color{gray}0\%}$
Garbage Collection: ${\color{red}+0.02\%}$

Instructions

	A) PriorityQueue	B) PriorityQueueSet
1.) 100000 operations (push:pop = 1:1)	568_913_057 $({\color{green}-4.79\%})$	522_729_861 $({\color{gray}0\%})$
2.) 100000 operations (push:pop = 2:1)	707_495_424 $({\color{green}-4.77\%})$	809_693_415 $({\color{gray}0\%})$
3.) 100000 operations (push:pop = 10:1)	336_409_578 $({\color{green}-6.01\%})$	873_181_028 $({\color{gray}0\%})$
4.) 100000 operations (only push)	176_982_954 $({\color{green}-8.02\%})$	886_824_792 $({\color{gray}0\%})$
5.) 50000 pushes, then 50000 pops	745_226_615 $({\color{green}-4.04\%})$	961_776_534 $({\color{gray}0\%})$
6.) 50000 pushes, then 25000 "pop;push"es	504_254_228 $({\color{green}-4.76\%})$	922_137_111 $({\color{gray}0\%})$

Heap

	A) PriorityQueue	B) PriorityQueueSet
1.) 100000 operations (push:pop = 1:1)	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
2.) 100000 operations (push:pop = 2:1)	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
3.) 100000 operations (push:pop = 10:1)	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
4.) 100000 operations (only push)	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
5.) 50000 pushes, then 50000 pops	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
6.) 50000 pushes, then 25000 "pop;push"es	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$

Garbage Collection

	A) PriorityQueue	B) PriorityQueueSet
1.) 100000 operations (push:pop = 1:1)	15.07 MiB $({\color{red}+0.24\%})$	17.43 MiB $({\color{gray}0\%})$
2.) 100000 operations (push:pop = 2:1)	19.73 MiB $({\color{gray}0\%})$	19.32 MiB $({\color{gray}0\%})$
3.) 100000 operations (push:pop = 10:1)	8.67 MiB $({\color{gray}0\%})$	12.64 MiB $({\color{gray}0\%})$
4.) 100000 operations (only push)	3.87 MiB $({\color{gray}0\%})$	9.96 MiB $({\color{gray}0\%})$
5.) 50000 pushes, then 50000 pops	22.03 MiB $({\color{red}+0.02\%})$	26.2 MiB $({\color{gray}0\%})$
6.) 50000 pushes, then 25000 "pop;push"es	14.22 MiB $({\color{gray}0\%})$	18.44 MiB $({\color{gray}0\%})$

bench/PureListStackSafety.bench.mo $({\color{gray}0\%})$

List Stack safety

Check stack-safety of the following pure/List-related functions.

Instructions: ${\color{gray}0\%}$
Heap: ${\color{gray}0\%}$
Stable Memory: ${\color{gray}0\%}$
Garbage Collection: ${\color{gray}0\%}$

Instructions


pure/List.split	24_602_524 $({\color{gray}0\%})$
pure/List.all	7_901_014 $({\color{gray}0\%})$
pure/List.any	8_001_390 $({\color{gray}0\%})$
pure/List.map	23_103_767 $({\color{gray}0\%})$
pure/List.filter	21_104_188 $({\color{gray}0\%})$
pure/List.filterMap	27_404_742 $({\color{gray}0\%})$
pure/List.partition	21_304_994 $({\color{gray}0\%})$
pure/List.join	33_105_326 $({\color{gray}0\%})$
pure/List.flatten	24_805_667 $({\color{gray}0\%})$
pure/List.take	24_605_664 $({\color{gray}0\%})$
pure/List.drop	9_904_119 $({\color{gray}0\%})$
pure/List.foldRight	19_105_768 $({\color{gray}0\%})$
pure/List.merge	31_808_584 $({\color{gray}0\%})$
pure/List.chunks	51_510_344 $({\color{gray}0\%})$
pure/Queue	142_662_505 $({\color{gray}0\%})$

Heap


pure/List.split	272 B $({\color{gray}0\%})$
pure/List.all	272 B $({\color{gray}0\%})$
pure/List.any	272 B $({\color{gray}0\%})$
pure/List.map	272 B $({\color{gray}0\%})$
pure/List.filter	272 B $({\color{gray}0\%})$
pure/List.filterMap	272 B $({\color{gray}0\%})$
pure/List.partition	272 B $({\color{gray}0\%})$
pure/List.join	272 B $({\color{gray}0\%})$
pure/List.flatten	272 B $({\color{gray}0\%})$
pure/List.take	272 B $({\color{gray}0\%})$
pure/List.drop	272 B $({\color{gray}0\%})$
pure/List.foldRight	272 B $({\color{gray}0\%})$
pure/List.merge	272 B $({\color{gray}0\%})$
pure/List.chunks	272 B $({\color{gray}0\%})$
pure/Queue	272 B $({\color{gray}0\%})$

Garbage Collection


pure/List.split	3.05 MiB $({\color{gray}0\%})$
pure/List.all	328 B $({\color{gray}0\%})$
pure/List.any	328 B $({\color{gray}0\%})$
pure/List.map	3.05 MiB $({\color{gray}0\%})$
pure/List.filter	3.05 MiB $({\color{gray}0\%})$
pure/List.filterMap	3.05 MiB $({\color{gray}0\%})$
pure/List.partition	3.05 MiB $({\color{gray}0\%})$
pure/List.join	3.05 MiB $({\color{gray}0\%})$
pure/List.flatten	3.05 MiB $({\color{gray}0\%})$
pure/List.take	3.05 MiB $({\color{gray}0\%})$
pure/List.drop	328 B $({\color{gray}0\%})$
pure/List.foldRight	1.53 MiB $({\color{gray}0\%})$
pure/List.merge	4.58 MiB $({\color{gray}0\%})$
pure/List.chunks	7.63 MiB $({\color{gray}0\%})$
pure/Queue	18.31 MiB $({\color{gray}0\%})$

bench/Queues.bench.mo $({\color{gray}0\%})$

Different queue implementations

Compare the performance of the following queue implementations:

pure/Queue: The default immutable double-ended queue implementation.
- Pros: Good amortized performance, meaning that the average cost of operations is low O(1).
- Cons: In worst case, an operation can take O(size) time rebuilding the queue as demonstrated in the Pop front 2 elements scenario.
pure/RealTimeQueue
- Pros: Every operation is guaranteed to take at most O(1) time and space.
- Cons: Poor amortized performance: Instruction cost is on average 3x for pop and 8x for push compared to pure/Queue.
mutable Queue
- Pros: Also O(1) guarantees with a lower constant factor than pure/RealTimeQueue. Amortized performance is comparable to pure/Queue.
- Cons: It is mutable and cannot be used in shared types (not shareable).

Instructions: ${\color{gray}0\%}$
Heap: ${\color{gray}0\%}$
Stable Memory: ${\color{gray}0\%}$
Garbage Collection: ${\color{gray}0\%}$

Instructions

	pure/Queue	pure/RealTimeQueue	mutable Queue
Initialize with 2 elements	3_092 $({\color{gray}0\%})$	2_304 $({\color{gray}0\%})$	3_040 $({\color{gray}0\%})$
Push 500 elements	90_713 $({\color{gray}0\%})$	744_219 $({\color{gray}0\%})$	219_284 $({\color{gray}0\%})$
Pop front 2 elements	86_966 $({\color{gray}0\%})$	4_446 $({\color{gray}0\%})$	3_847 $({\color{gray}0\%})$
Pop 150 front&back	92_095 $({\color{gray}0\%})$	304_908 $({\color{gray}0\%})$	124_581 $({\color{gray}0\%})$

Heap

	pure/Queue	pure/RealTimeQueue	mutable Queue
Initialize with 2 elements	324 B $({\color{gray}0\%})$	300 B $({\color{gray}0\%})$	352 B $({\color{gray}0\%})$
Push 500 elements	8.08 KiB $({\color{gray}0\%})$	8.17 KiB $({\color{gray}0\%})$	19.8 KiB $({\color{gray}0\%})$
Pop front 2 elements	240 B $({\color{gray}0\%})$	240 B $({\color{gray}0\%})$	192 B $({\color{gray}0\%})$
Pop 150 front&back	-4.42 KiB $({\color{gray}0\%})$	-492 B $({\color{gray}0\%})$	-11.45 KiB $({\color{gray}0\%})$

Garbage Collection

	pure/Queue	pure/RealTimeQueue	mutable Queue
Initialize with 2 elements	508 B $({\color{gray}0\%})$	444 B $({\color{gray}0\%})$	456 B $({\color{gray}0\%})$
Push 500 elements	10.1 KiB $({\color{gray}0\%})$	137.84 KiB $({\color{gray}0\%})$	344 B $({\color{gray}0\%})$
Pop front 2 elements	12.19 KiB $({\color{gray}0\%})$	528 B $({\color{gray}0\%})$	424 B $({\color{gray}0\%})$
Pop 150 front&back	15.61 KiB $({\color{gray}0\%})$	49.66 KiB $({\color{gray}0\%})$	12.1 KiB $({\color{gray}0\%})$

Note: Renamed benchmarks cannot be compared. Refer to the current baseline for manual comparison.

rvanasa · 2025-11-18T15:26:27Z

I think we should use the same pattern as established in List, i.e. a get function which returns an optional and an at function which traps outside of the index range.

rvanasa · 2025-11-19T15:26:12Z

src/Blob.mo

+  /// ```
+  ///
+  /// Runtime: `O(1)`
+  public func at(blob : Blob, index : Nat) : Nat8 = blob.get index;


Realizing this might be confusing with contextual dot notation...

Oh, you mean the missing parens?

Thinking of the discrepancy where blob.get(i) returns a Nat8 whereas Blob.get(blob, i) would return an optional ?Nat8.

Not sure if there's much we can do about this aside from a breaking change to the compiler or using a different convention for the Blob module compared to List (which might confuse LLMs and humans).

Oh, I get it now. But the same situation probably for Text if that provides indexing at all. That wouldn't be O(1), though.

feat: indexable Blobs

6072003

https://github.com/dfinity/motoko/pull/5018/files TODO: - [ ] also do this for `base`! - [ ] `index` method - [ ] test

func get

7ff6663

ggreif force-pushed the gabor/blob-index branch from c713824 to 7ff6663 Compare November 18, 2025 21:56

ggreif added 3 commits November 18, 2025 22:58

API

6e3d1de

fix

19df92a

func at

fb3adce

ggreif force-pushed the gabor/blob-index branch from 55ce12f to fb3adce Compare November 18, 2025 22:09

update Changelog.md

bda9424

ggreif self-assigned this Nov 18, 2025

ggreif added the enhancement New feature or request label Nov 18, 2025

ggreif marked this pull request as ready for review November 18, 2025 22:19

ggreif requested a review from a team as a code owner November 18, 2025 22:19

ggreif requested review from christoph-dfinity, crusso and rvanasa November 19, 2025 14:25

rvanasa reviewed Nov 19, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: indexable `Blob`s #424

feat: indexable `Blob`s #424

ggreif commented Nov 18, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Nov 18, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Nov 18, 2025

Large known-size array building

Benchmarking the fromIter functions

List vs. Buffer for creating known-size arrays

Different priority queue implementations

List Stack safety

Different queue implementations

Uh oh!

rvanasa commented Nov 18, 2025 •

edited

Loading

Uh oh!

rvanasa Nov 19, 2025

Uh oh!

ggreif Nov 19, 2025

Uh oh!

rvanasa Nov 19, 2025 •

edited

Loading

Uh oh!

ggreif Nov 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat: indexable Blobs #424

Are you sure you want to change the base?

feat: indexable Blobs #424

Conversation

ggreif commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 18, 2025

Benchmark Results

Large known-size array building

Benchmarking the fromIter functions

List vs. Buffer for creating known-size arrays

Different priority queue implementations

List Stack safety

Different queue implementations

Uh oh!

rvanasa commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rvanasa Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

ggreif Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

rvanasa Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ggreif Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat: indexable `Blob`s #424

feat: indexable `Blob`s #424

ggreif commented Nov 18, 2025 •

edited

Loading

github-actions bot commented Nov 18, 2025 •

edited

Loading

rvanasa commented Nov 18, 2025 •

edited

Loading

rvanasa Nov 19, 2025 •

edited

Loading