Skip to content

v0.8.0
Compare
Choose a tag to compare
@fxamacker fxamacker released this 04 Sep 15:43
· 48 commits to main since this release
v0.8.0
2a525f5
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.

Atree v0.8.0 introduces payload inlining and metadata deduplication to improve memory use, storage size, and speed of various servers that use payloads (not limited to EN). This improves network stability and performance as we handle future data growth.

Inlining and deduplication reduces payload count and mtrie nodes. This reduces size and growth rate of execution state.

  • Inlining stores small arrays and maps (previously stored in their own payload) into an existing payload.
  • Deduplication removes redundant metadata (e.g. Cadence struct field names and type information) to reduce payload size.

This release passed multiple types of tests, including 1034 hours of smoke tests that concluded on August 19, 2024.

Results exceeded goals mentioned in Scaling Execution Node (forum announcement).

Note

Sept. 4, 2024: 🏆 Atree inlined 500 million payloads (-61%) and eliminated over 1 billion mtrie nodes on mainnet!

Impact of these payload and mtrie node reductions include:

  • Reduced RAM use by hundreds of GB on each mainnet execution node.
  • Reduced SSD storage use (e.g. checkpoint file sizes).
  • Reduced network bandwidth for chunk data packs (mentioned by Peter).
  • Speedup transactions by 5-8% (mentioned by Jan).
  • Speedup network upgrades (e.g. Cadence 1.0 migrations, future migrations).
  • Future efficiency of other servers, components, databases (files/cache/index), etc. that benefit from fewer payloads.

Sept. 30, 2024: 🏆 Beyond execution nodes, bluesign shared a 5x speedup in tinyAN bootstrap time!
"btw amazing work this atree inlining, my tinyAN bootstrap time improved like 5x"

Future: We can replace nonatree domain payloads with atree payloads to reduce mainnet payload count by another 20-30%.

While devnet (shown below) had better percentage improvements, mainnet eliminated more mtrie nodes (over 1 billion)!

Memory Reduction Before/After Atree Inlining & Deduplication (Devnet Aug 14, 2024)

image

Mtrie Nodes and Atree Payloads (Aug 14, 2024 Devnet)Atree Payload Sizes (bytes)

Total Counts

Before After Change
mtrie nodes 1,182,195,945 343,949,052 -838,246,893
atree payloads 455,399,623 102,283,931 -353,115,692

Total Sizes

Before After Change
mtrie nodes 113.49 GB 33.02 GB -80.47 GB
atree payloads 87.74 GB 69.28 GB -18.46 GB
combined 201.23 GB 102.30 GB -98.93 GB
Before After
count 455,399,623 102,283,931
mean 192.7 677.4
std 996.4 2,065.7
min 13 22
25% 88 199
50% 139 665
75% 206 1,043
99% 876 1,267
99.5% 1,078 1,453
max 5,853,892 5,853,892

What's Changed (since v0.6.0)

  • Change OrderedMap.Get() to return Value instead of Storable by @fxamacker in #318
  • Change Array.Get() to return Value instead of Storable by @fxamacker in #316
  • Optimize max map value size to reduce number of registers by @fxamacker in #314
  • Refactor to remove lint warnings by @fxamacker in #319
  • Add Array.ID() and OrderedMap.ID() by @fxamacker in #321
  • Rename ID to ValueID by @fxamacker in #325
  • Refactor creating new StorableSlab by @fxamacker in #324
  • Unexport SlabID fields to prevent misuse by @fxamacker in #323
  • Rename StorageID related types, vars, and funcs by @fxamacker in #322
  • Add support for iterating only loaded values by @fxamacker in #311
  • Refactor hasPointer for register inlining by @fxamacker in #327
  • Reduce size of encoded array slab and bump version by @fxamacker in #330
  • Add ignore filters to dependabot.yml by @fxamacker in #302
  • Reduce encoded size of map and bump version by @fxamacker in #331
  • Update README.md to describe Atree by @fxamacker in #332
  • Update responsible disclosure link by @jribbink in #333
  • Fix slab size when resetting mutable storable in OrderedMap by @fxamacker in #337
  • Omit empty next slab ID in encoded map data slab by @fxamacker in #340
  • Omit empty next slab ID in encoded array data slab by @fxamacker in #339
  • Fix slab size when resetting mutable storable in Array by @fxamacker in #336
  • Refactor encoding version and flag to add more flags by @fxamacker in #338
  • Add readonly iterators and support value mutations only from non-readonly iterators by @fxamacker in #345
  • Atree Register Inlining and Data Deduplication by @fxamacker in #342
  • Update smoke test for atree inlining by @fxamacker in #348
  • Update for Cadence integration for atree inlining and deduplication by @fxamacker in #352
  • Make smoke tests check recently added data deduplication feature by @fxamacker in #350
  • Remove ContainerStorable.EncodeAsElement by @fxamacker in #354
  • Add feature to support mutation for array and map iterators by @fxamacker in #359
  • Add support for changing type info of atree arrays (atree inlining branch) by @fxamacker in #376
  • Add support for changing type info of atree maps (atree inlining branch) by @fxamacker in #377
  • Reduce RAM and persistent storage by deduplicating inlined dict type info by @fxamacker in #369
  • Use encoded type info to deduplicate extra data by @fxamacker in #381
  • Fix error type for external errors during serialization by @fxamacker in #382
  • Add PersistentSlabStorage.GetAllChildReferences() for atree inlining by @fxamacker in #392
  • Add feature to enable atree inlining migration to fix references to non-existent registers by @fxamacker in #388
  • Fix migration filter for old unreferenced slabs (atree inlining feature branch) by @fxamacker in #396
  • Add NonderterministicFastCommit to speed up migrations when ordering isn't required by @fxamacker in #403
  • Add BatchPreload to decode slabs in parallel and cache by @fxamacker in #404
  • Check mutation of elements from readonly map iterator by @fxamacker in #410
  • Check mutation of elements from readonly array iterator by @fxamacker in #411
  • Update comment for NondeterministicFastCommit by @fxamacker in #412
  • Update copyright notice to Flow Foundation by @fxamacker in #415
  • Update copyright notice to Flow Foundation (in main branch) by @fxamacker in #417
  • Update feature/array-map-inlining (atree inlining feature branch) by @fxamacker in #428
  • Merge feature/array-map-inlining (atree inlining feature branch) to main by @fxamacker in #429
  • Add functions to check availablility of CBOR tag numbers by @fxamacker in #434
  • Add SlabID.Address() by @fxamacker in #435
  • Make smoke tests check if CBOR tag nums are reserved by @fxamacker in #436
Changes to CI and version bumps...

Special thanks (alphabetically):

  • Bastian for feature requests, approving proposed designs, numerous PR reviews, and feedback! 🙌
  • Dete for suggestion to deduplicate data when encoding (in 2021)! 🙏
  • Ramtin for suggestion to inline atree registers, approving the proposed designs, and feedback! 💪

New Contributors

Full Changelog: v0.6.0...v0.8.0

Contributors

jribbink, dependabot, and fxamacker
Assets 2
Loading