Skip to content

Releases: onflow/atree

v0.8.1

27 Nov 18:08
3b39aec
Compare
Choose a tag to compare

What's Changed

This release adds the PersistentSlabStorage.HasUnsavedChanges() function.

HasUnsaveChanges() will be used by Cadence and Flow to optimize storage of domain registers to reduce payload count and mtrie count by 28.8% each (using zero downtime migration). Most of that effort will be at onflow/cadence and onflow/flow-go repos.

  • Add function to check if address has unsaved changes in storage by @fxamacker in #450
Changes to CI and version bumps...

Full Changelog: v0.8.0...v0.8.1

v0.8.0

04 Sep 15:43
2a525f5
Compare
Choose a tag to compare

Atree v0.8.0 introduces payload inlining and metadata deduplication to improve memory use, storage size, and speed of various servers that use payloads (not limited to EN). This improves network stability and performance as we handle future data growth.

Inlining and deduplication reduces payload count and mtrie nodes. This reduces size and growth rate of execution state.

  • Inlining stores small arrays and maps (previously stored in their own payload) into an existing payload.
  • Deduplication removes redundant metadata (e.g. Cadence struct field names and type information) to reduce payload size.

This release passed multiple types of tests, including 1034 hours of smoke tests that concluded on August 19, 2024.

Results exceeded goals mentioned in Scaling Execution Node (forum announcement).

Note

Sept. 4, 2024: 🏆 Atree inlined 500 million payloads (-61%) and eliminated over 1 billion mtrie nodes on mainnet!

Impact of these payload and mtrie node reductions include:

  • Reduced RAM use by hundreds of GB on each mainnet execution node.
  • Reduced SSD storage use (e.g. checkpoint file sizes).
  • Reduced network bandwidth for chunk data packs (mentioned by Peter).
  • Speedup transactions by 5-8% (mentioned by Jan).
  • Speedup network upgrades (e.g. Cadence 1.0 migrations, future migrations).
  • Future efficiency of other servers, components, databases (files/cache/index), etc. that benefit from fewer payloads.

Sept. 30, 2024: 🏆 Beyond execution nodes, bluesign shared a 5x speedup in tinyAN bootstrap time!
"btw amazing work this atree inlining, my tinyAN bootstrap time improved like 5x"

Future: We can replace nonatree domain payloads with atree payloads to reduce mainnet payload count by another 20-30%.

While devnet (shown below) had better percentage improvements, mainnet eliminated more mtrie nodes (over 1 billion)!

Memory Reduction Before/After Atree Inlining & Deduplication (Devnet Aug 14, 2024)

image

Mtrie Nodes and Atree Payloads (Aug 14, 2024 Devnet)Atree Payload Sizes (bytes)

Total Counts

Before After Change
mtrie nodes 1,182,195,945 343,949,052 -838,246,893
atree payloads 455,399,623 102,283,931 -353,115,692

Total Sizes

Before After Change
mtrie nodes 113.49 GB 33.02 GB -80.47 GB
atree payloads 87.74 GB 69.28 GB -18.46 GB
combined 201.23 GB 102.30 GB -98.93 GB
Before After
count 455,399,623 102,283,931
mean 192.7 677.4
std 996.4 2,065.7
min 13 22
25% 88 199
50% 139 665
75% 206 1,043
99% 876 1,267
99.5% 1,078 1,453
max 5,853,892 5,853,892

What's Changed (since v0.6.0)

  • Change OrderedMap.Get() to return Value instead of Storable by @fxamacker in #318
  • Change Array.Get() to return Value instead of Storable by @fxamacker in #316
  • Optimize max map value size to reduce number of registers by @fxamacker in #314
  • Refactor to remove lint warnings by @fxamacker in #319
  • Add Array.ID() and OrderedMap.ID() by @fxamacker in #321
  • Rename ID to ValueID by @fxamacker in #325
  • Refactor creating new StorableSlab by @fxamacker in #324
  • Unexport SlabID fields to prevent misuse by @fxamacker in #323
  • Rename StorageID related types, vars, and funcs by @fxamacker in #322
  • Add support for iterating only loaded values by @fxamacker in #311
  • Refactor hasPointer for register inlining by @fxamacker in #327
  • Reduce size of encoded array slab and bump version by @fxamacker in #330
  • Add ignore filters to dependabot.yml by @fxamacker in #302
  • Reduce encoded size of map and bump version by @fxamacker in #331
  • Update README.md to describe Atree by @fxamacker in #332
  • Update responsible disclosure link by @jribbink in #333
  • Fix slab size when resetting mutable storable in OrderedMap by @fxamacker in #337
  • Omit empty next slab ID in encoded map data slab by @fxamacker in #340
  • Omit empty next slab ID in encoded array data slab by @fxamacker in #339
  • Fix slab size when resetting mutable storable in Array by @fxamacker in #336
  • Refactor encoding version and flag to add more flags by @fxamacker in #338
  • Add readonly iterators and support value mutations only from non-readonly iterators by @fxamacker in #345
  • Atree Register Inlining and Data Deduplication by @fxamacker in #342
  • Update smoke test for atree inlining by @fxamacker in #348
  • Update for Cadence integration for atree inlining and deduplication by @fxamacker in #352
  • Make smoke tests check recently added data deduplication feature by @fxamacker in #350
  • Remove ContainerStorable.EncodeAsElement by @fxamacker in #354
  • Add feature to support mutation for array and map iterators by @fxamacker in #359
  • Add support for changing type info of atree arrays (atree inlining branch) by @fxamacker in #376
  • Add support for changing type info of atree maps (atree inlining branch) by @fxamacker in #377
  • Reduce RAM and persistent storage by deduplicating inlined dict type info by @fxamacker in #369
  • Use encoded type info to deduplicate extra data by @fxamacker in #381
  • Fix error type for external errors during serialization by @fxamacker in #382
  • Add PersistentSlabStorage.GetAllChildReferences() for atree inlining by @fxamacker in #392
  • Add feature to enable atree inlining migration to fix references to non-existent registers by @fxamacker in #388
  • Fix migration filter for old unreferenced slabs (atree inlining feature branch) by @fxamacker in #396
  • Add NonderterministicFastCommit to speed up migrations when ordering isn't required by @fxamacker in #403
  • Add BatchPreload to decode slabs in parallel and cache by @fxamacker in #404
  • Check mutation of elements from readonly map iterator by @fxamacker in #410
  • Check mutation of elements from readonly array iterator by @fxamacker in #411
  • Update comment for NondeterministicFastCommit by @fxamacker in #412
  • Update copyright notice to Flow Foundation by @fxamacker in #415
  • Update copyright notice to Flow Foundation (in main branch) by @fxamacker in #417
  • Update feature/array-map-inlining (atree inlining feature branch) by @fxamacker in #428
  • Merge feature/array-map-inlining (atree inlining feature branch) to main by @fxamacker in #429
  • Add functions to check availablility of CBOR tag numbers by @fxamacker in #434
  • Add SlabID.Address() by @fxamacker in #435
  • Make smoke tests check if CBOR tag nums are reserved by @fxamacker in #436
Changes to CI and version bumps...

Read more

v0.8.0-rc.6

19 Aug 14:31
141ec5c
Compare
Choose a tag to compare
v0.8.0-rc.6 Pre-release
Pre-release

Release 0.8.x (since v0.8.0-rc.1) introduces atree inlining and deduplication, so the data format is a breaking change from 0.7.x.

What's Changed Since 0.8.0-rc.5

To reduce risks, new functions were added to help projects using Atree avoid producing undecodable data in storage.

  • Add IsCBORTagNumberRangeAvailable() and ReservedCBORTagNumberRange() to check availablility of CBOR tag numbers by @fxamacker in #434

  • Add SlabID.Address() by @fxamacker in #435

Changes to CI and version bumps...

Full Changelog: v0.8.0-rc.5...v0.8.0-rc.6

v0.8.0-rc.5

29 Jul 16:07
1fec437
Compare
Choose a tag to compare
v0.8.0-rc.5 Pre-release
Pre-release

Release 0.8.x (since v0.8.0-rc.1) introduces atree inlining and deduplication, so the data format is a breaking change from 0.7.x.

What's Changed Since v0.6.0 (main branch)

v0.8.0-rc.5 merges atree inlining feature branch to main branch after extensive testing.

  • Change OrderedMap.Get() to return Value instead of Storable by @fxamacker in #318
  • Change Array.Get() to return Value instead of Storable by @fxamacker in #316
  • Optimize max map value size to reduce number of registers by @fxamacker in #314
  • Refactor to remove lint warnings by @fxamacker in #319
  • Add Array.ID() and OrderedMap.ID() by @fxamacker in #321
  • Rename ID to ValueID by @fxamacker in #325
  • Refactor creating new StorableSlab by @fxamacker in #324
  • Unexport SlabID fields to prevent misuse by @fxamacker in #323
  • Rename StorageID related types, vars, and funcs by @fxamacker in #322
  • Add support for iterating only loaded values by @fxamacker in #311
  • Refactor hasPointer for register inlining by @fxamacker in #327
  • Reduce size of encoded array slab and bump version by @fxamacker in #330
  • Add ignore filters to dependabot.yml by @fxamacker in #302
  • Reduce encoded size of map and bump version by @fxamacker in #331
  • Update README.md to describe Atree by @fxamacker in #332
  • Update responsible disclosure link by @jribbink in #333
  • Fix slab size when resetting mutable storable in OrderedMap by @fxamacker in #337
  • Omit empty next slab ID in encoded map data slab by @fxamacker in #340
  • Omit empty next slab ID in encoded array data slab by @fxamacker in #339
  • Fix slab size when resetting mutable storable in Array by @fxamacker in #336
  • Refactor encoding version and flag to add more flags by @fxamacker in #338
  • Add readonly iterators and support value mutations only from non-readonly iterators by @fxamacker in #345
  • Atree Register Inlining and Data Deduplication by @fxamacker in #342
  • Update smoke test for atree inlining by @fxamacker in #348
  • Update for Cadence integration for atree inlining and deduplication by @fxamacker in #352
  • Make smoke tests check recently added data deduplication feature by @fxamacker in #350
  • Remove ContainerStorable.EncodeAsElement by @fxamacker in #354
  • Add feature to support mutation for array and map iterators by @fxamacker in #359
  • Add support for changing type info of atree arrays (atree inlining branch) by @fxamacker in #376
  • Add support for changing type info of atree maps (atree inlining branch) by @fxamacker in #377
  • Reduce RAM and persistent storage by deduplicating inlined dict type info by @fxamacker in #369
  • Use encoded type info to deduplicate extra data by @fxamacker in #381
  • Fix error type for external errors during serialization by @fxamacker in #382
  • Add PersistentSlabStorage.GetAllChildReferences() for atree inlining by @fxamacker in #392
  • Add feature to enable atree inlining migration to fix references to non-existent registers by @fxamacker in #388
  • Fix migration filter for old unreferenced slabs (atree inlining feature branch) by @fxamacker in #396
  • Add NonderterministicFastCommit to speed up migrations when ordering isn't required by @fxamacker in #403
  • Add BatchPreload to decode slabs in parallel and cache by @fxamacker in #404
  • Check mutation of elements from readonly map iterator by @fxamacker in #410
  • Check mutation of elements from readonly array iterator by @fxamacker in #411
  • Update comment for NondeterministicFastCommit by @fxamacker in #412
  • Update copyright notice to Flow Foundation by @fxamacker in #415
  • Update copyright notice to Flow Foundation (in main branch) by @fxamacker in #417
  • Update feature/array-map-inlining (atree inlining feature branch) by @fxamacker in #428
  • Merge feature/array-map-inlining (atree inlining feature branch) to main by @fxamacker in #429
Changes to CI and version bumps...

New Contributors

Full Changelog: v0.6.0...v0.8.0-rc.5

v0.8.0-rc.4

25 Jul 19:24
caf04d2
Compare
Choose a tag to compare
v0.8.0-rc.4 Pre-release
Pre-release

Release 0.8.x (since v0.8.0-rc.1) introduces atree inlining and deduplication, so the data format is a breaking change from 0.7.x.

What's Changed Since v0.8.0-rc.3

Changes in v0.8.0-rc.4 are limited to updated copyright notice, comments, and dependency version bumps.

More details...

Full Changelog: v0.8.0-rc.3...v0.8.0-rc.4

v0.7.0-rc.3

25 Jul 19:15
8d3ce7d
Compare
Choose a tag to compare
v0.7.0-rc.3 Pre-release
Pre-release

The v0.7.x does not include atree inlining. Atree inlining is introduced in v0.8.0-rc.1 with breaking changes to data format.

What's Changed Since v0.7.0-rc.2

  • Update copyright notice to Flow Foundation (in feature/stable-cadence) by @fxamacker in #416

Full Changelog: v0.7.0-rc.2...v0.7.0-rc.3

v0.8.0-rc.3

21 May 22:04
688791c
Compare
Choose a tag to compare
v0.8.0-rc.3 Pre-release
Pre-release

Release 0.8.x (since v0.8.0-rc.1) introduces atree inlining and deduplication, so the data format is a breaking change from 0.7.x.

What's Changed Since v0.8.0-rc.2

  • Check mutation of elements from readonly map iterator by @fxamacker in #410
  • Check mutation of elements from readonly array iterator by @fxamacker in #411

Full Changelog: v0.8.0-rc.2...v0.8.0-rc.3

v0.8.0-rc.2

14 May 14:26
73e00ec
Compare
Choose a tag to compare
v0.8.0-rc.2 Pre-release
Pre-release

Release 0.8.x (since v0.8.0-rc.1) introduces atree inlining and deduplication, so the data format is a breaking change from 0.7.x.

What's Changed Since v0.8.0-rc.1

  • Add NonderterministicFastCommit to speed up migrations when ordering isn't required by @fxamacker in #403
  • Add BatchPreload to decode slabs in parallel and cache by @fxamacker in #404

These new features are intended to be used by state migration programs in onflow/flow-go to speedup migrations.

Full Changelog: v0.8.0-rc.1...v0.8.0-rc.2

v0.7.0-rc.2

14 May 18:23
8c70b0e
Compare
Choose a tag to compare
v0.7.0-rc.2 Pre-release
Pre-release

The v0.7.x does not include atree inlining. Atree inlining is introduced in v0.8.0-rc.1 with breaking changes to data format.

What's Changed Since v0.7.0-rc.1

  • Add NonderterministicFastCommit to speed up migrations (for branch without atree inlining) by @fxamacker in #406
  • Add BatchPreload to decode slabs in parallel and cache (for branch without atree inlining) by @fxamacker in #407

These new features are intended to be used by state migration programs in onflow/flow-go to speedup migrations.

Full Changelog: v0.7.0-rc.1...v0.7.0-rc.2

v0.8.0-rc.1

01 May 21:00
e4400b2
Compare
Choose a tag to compare
v0.8.0-rc.1 Pre-release
Pre-release

Release 0.8.0-rc.1 introduces atree inlining and deduplication, so the data format is a breaking change.

Atree inlining and deduplication is primarily in these PRs:

  • Atree Register Inlining and Data Deduplication in #342
  • Add readonly iterators and support value mutations only from non-readonly iterators in #345
  • Reduce RAM and persistent storage by deduplicating inlined dict type info in #369

NOTE: Atree Register Inlining Project includes other work: migration (in onflow/flow-go) and integration (in onflow/cadence).

What's Changed

  • Change OrderedMap.Get() to return Value instead of Storable by @fxamacker in #318
  • Change Array.Get() to return Value instead of Storable by @fxamacker in #316
  • Optimize max map value size to reduce number of registers by @fxamacker in #314
  • Refactor to remove lint warnings by @fxamacker in #319
  • Add Array.ID() and OrderedMap.ID() by @fxamacker in #321
  • Rename ID to ValueID by @fxamacker in #325
  • Refactor creating new StorableSlab by @fxamacker in #324
  • Unexport SlabID fields to prevent misuse by @fxamacker in #323
  • Rename StorageID related types, vars, and funcs by @fxamacker in #322
  • Add support for iterating only loaded values by @fxamacker in #311
  • Refactor hasPointer for register inlining by @fxamacker in #327
  • Reduce size of encoded array slab and bump version by @fxamacker in #330
  • Reduce encoded size of map and bump version by @fxamacker in #331
  • Update README.md to describe Atree by @fxamacker in #332
  • Update responsible disclosure link by @jribbink in #333
  • Fix slab size when resetting mutable storable in OrderedMap by @fxamacker in #337
  • Omit empty next slab ID in encoded map data slab by @fxamacker in #340
  • Omit empty next slab ID in encoded array data slab by @fxamacker in #339
  • Fix slab size when resetting mutable storable in Array by @fxamacker in #336
  • Refactor encoding version and flag to add more flags by @fxamacker in #338
  • Add readonly iterators and support value mutations only from non-readonly iterators by @fxamacker in #345
  • Atree Register Inlining and Data Deduplication by @fxamacker in #342
  • Update smoke test for atree inlining by @fxamacker in #348
  • Update for Cadence integration for atree inlining and deduplication by @fxamacker in #352
  • Make smoke tests check recently added data deduplication feature by @fxamacker in #350
  • Remove ContainerStorable.EncodeAsElement by @fxamacker in #354
  • Add feature to support mutation for array and map iterators by @fxamacker in #359
  • Add support for changing type info of atree arrays (atree inlining branch) by @fxamacker in #376
  • Add support for changing type info of atree maps (atree inlining branch) by @fxamacker in #377
  • Reduce RAM and persistent storage by deduplicating inlined dict type info by @fxamacker in #369
  • Use encoded type info to deduplicate extra data by @fxamacker in #381
  • Fix error type for external errors during serialization by @fxamacker in #382
  • Add PersistentSlabStorage.GetAllChildReferences() for atree inlining by @fxamacker in #392
  • Add feature to enable atree inlining migration to fix references to non-existent registers by @fxamacker in #388
  • Fix migration filter for old unreferenced slabs (atree inlining feature branch) by @fxamacker in #396
CI / Dependabot

New Contributors

Full Changelog: v0.6.0...v0.8.0-rc.1