10th April 2025 - Third Height Co-orindated Upgrade on Mainnet26 at Block Height 109384800

What's Changed

Cadence

• Update to Cadence v1.3.4 and atree v0.10.0 by @turbolent in #7260

Data Availability - Websocket streaming

• Implement integration test for new websockets by @UlyanaAndrukhiv in #6907
• check for correct err in ReadJSON method by @illia-malachyn in #7065
• OK response should not contain error message by @illia-malachyn in #7037
• Properly handle subscription errors in data providers by @illia-malachyn in #7046
• Pass irrecoverable context to ws handler by @illia-malachyn in #7154
• Changed max subscription error code to 429 by @Guitarheroua in #7158
• Refactor validator into own module by @peterargue in #7102
• Refactor Access API RPC handler into rpc engine by @peterargue in #7103
• Refactor common Access API structs into models by @peterargue in #7104
• Refactor access subscriptions by @peterargue in #7105
• Fix and refactor validation logic in data providers by @illia-malachyn in #7140
• update websockets cli flag by @peterargue in #7252

EVM

• Introduce the dryCall method under EVM contract & CadenceOwnedAccount resource by @m-Peter in #6838

Execution

• Migrate last executed block from badger to pebble by @zhangchiqing in #7117

Storage

• Update pebble dir path by @zhangchiqing in #7141
• Release Pebble batch resource to reduce memory use by @fxamacker in #7153

Util

• Update util to read results from pebble by @zhangchiqing in #7092
• Read storage stats by @zhangchiqing in #7131

Misc

• Remove anti-pattern of ignoring irrecoverable error channel by @peterargue in #7155
• v0.40 disable block profiler by @zhangchiqing in #7223
• Jan/hcu prep v0.40 compatibility by @j1010001 in #7249

Full Changelog: v0.39.5...v0.40.0

v0.38.4

What's Changed

• [v0.38] Backport: Use SafetyData to avoid double-proposing instead of myLastProposedView by @jordanschalm in #7079

Full Changelog: v0.38.3...v0.38.4

Protocol Upgrade - EFM Recovery at Block View 20504725, Block Height 108630638

What's Changed

⬆️ Coordinated Upgrades for all Node Roles

This is a mechanism built into the Protocol State which enables view-coordinated upgrades to the Protocol State data model. This mechanism can be leveraged to implement coordinated behavioural changes ("protocol upgrades") for all node roles, including for backward-incompatible upgrades. Previously such upgrades were supported only for Execution Nodes and the Execution stack via the Height Coordinated Upgrade mechanism and VersionBeacon smart contract.

This release completes the upgrade feature and implements the first such coordinated upgrade, which will be used to enable EFM Recovery (see below). Further details are in the working group meeting notes.

Related PRs

• KVStore Model Upgrade & Chunk.ServiceEventCount Model Upgrade Logic by @jordanschalm in #6796
• Chunk Data Model supports per-chunk service event mapping by @jordanschalm in #6744
• DKG Result Submission Backward Compatibility by @jordanschalm in #6827
• [BFT] Database migration for flow.DKGEndState by @durkmurder in #6861
• CBOR Encoding is compatible with stricter network decode mode by @jordanschalm in #6879

🌟 Epoch Fallback Mode Recovery

Epoch Fallback Mode (EFM) is a operational mode triggered when an epoch transition fails for any reason. While in EFM, the current epoch is extended in perpetuity and epoch transitions are disabled. This has the effect of disabling changes to the node operator committee. Also, automatic reward payout is disabled and reverts to a manual governance transaction.

EFM Recovery provides a mechanism to recover from epoch failures using a governance transaction. Previously, recovery from EFM required a network upgrade (spork). With EFM Recovery, a governance transaction is used to resume epoch transitions. Further details are in the design document.

EFM Recovery is available in this release, but must be enabled via a coordinated upgrade (see above).

Related PRs

• Feature: EFM Recovery by @jordanschalm in #6885
• [EFM] Added a service event to update EpochExtensionViewCount by @durkmurder in #6272
• [EFM] Telemetry consumers for KV store state machines by @durkmurder in #6291
• [EFM] Add EjectIdentity service event by @durkmurder in #6296
• [EFM] minor polishing by @AlexHentschel in #6309
• Updated documentation of FinalizationSafetyThreshold by @AlexHentschel in #6384
• [EFM] Renamed EpochCommitSafetyThreshold -> FinalizationSafetyThreshold by @durkmurder in #6318
• [EFM] Documentation of mapping from NodeID -> DKG index by @AlexHentschel in #6428
• [EFM] Extended documentation where generalized protocol logic meets bootstrapping implementation by @AlexHentschel in #6452
• [EFM] Update protocol.DKG to use IndexMap by @durkmurder in #6338
• [EFM] Update DKG engine to submit a valid DKG index map by @durkmurder in #6490
• [EFM] Updated type conversion of EpochCommit by @durkmurder in #6494
• [EFM] Enable previously disabled tests by @durkmurder in #6496
• [EFM] Updated type conversion of EpochRecover by @durkmurder in #6506
• EFM Recovery: Fixes from Benchnet Testing by @jordanschalm in #6620
• EFM Integration Test Part 2 by @kc1116 in #6424
• [EFM] Refactored encoding and decoding of DKG data during bootstrapping by @durkmurder in #6745
• updating the term DKG key as it is outdated by @AlexHentschel in #6735
• Random beacon comments by @tarakby in #6590
• [EFM] Recoverable Random Beacon State Machine by @durkmurder in #6771
• Re-enable test TestProtocolVersionUpgrade by @jordanschalm in #6752
• [EFM] EpochCommit is backward compatible by @durkmurder in #6795
• [BFT] Node ejected before epoch recovery by @durkmurder in #6632
• [BFT] Epoch Recovery integration test by @durkmurder in #6823
• [EFM] Recoverable Random Beacon State Machine follow up updates by @durkmurder in #6815
• [EFM] Fix bugs from Benchnet testing by @jordanschalm in #6898
• Use SafetyData to avoid double-proposing instead of myLastProposedView by @jordanschalm in #6921
• [EFM] Extra flags for the recovery command line tool by @durkmurder in #6935
• [EFM] Extra consistency checks for DKG Index Map by @durkmurder in #6936
• Add metric for current kvstore version by @jordanschalm in #6953
• [EFM] Unit tests for excluding/including recovery epoch participants by @durkmurder in #6968

🪨 Storage Layer

• [Storage Refactor] Init pebble DB in scaffold. by @zhangchiqing in #6949

⛰️ Access API

• [Access] Disable registers db pruning by default by @peterargue in #7017
• Validation of error message on transaction replay is broken by @m-Peter in #7026
• Generate random string with maxLen symbols instead of UUID by @illia-malachyn in #7007
• [Access] Update model Block REST API Response by @UlyanaAndrukhiv in #6884
• Add feature flag for new websockets stream API by @illia-malachyn in #7024
• Return error if --state-stream-addr no configured. by @illia-malachyn in #7010
• Updates core contracts and transactions to use correct import placeholders by @joshuahannan in #7027
• [Access] Tuning API logging by @peterargue in #7048
• Update EVM ViewProvider to return block View at its end state by @m-Peter in #7050

❓ Miscellaneous

• Rename MonotonousCounter to MonotonicCounter by @peterargue in #6998
• Refactor Consensus Sealing Engine+Core: engine.Unit -> ComponentManager by @tim-barry in #6934
• Use ExecutionParameters contract in bootstrapping by @joshuahannan in #6986

Breaking Changes

• ServiceEventCount field added to Chunk. After the Protocol State Upgrade to version 2, this field will be activated. When active, this field's presence changes the canonical encoding and hashing for Chunk and all entities which use it (for example: ExecutionResult and Block). Clients which are not using a compatible version of flow-go will compute incorrect IDs for these entities.

Full Changelog: v0.38.3...v0.39.0

18th Feb 2025 Second Height Co-orindated Upgrade on Mainnet26 at block height 103983000

What's Changed

• [v0.38] Update to Cadence v1.3.3 by @turbolent in #7060
• add access compatibility with cadence fixes by @j1010001 in #7063

Full Changelog: v0.38.2...v0.38.3

18th Feb 2025 Height Co-ordinated Upgrade on Mainnet26 at block height 103957477

The Flow network has been upgraded!

This release brings a major enhancement to on-chain state storage with Account Storage Format v2, improving both the scalability and resiliency of the network.

✅ What is Included in This Upgrade?

• Optimized Account Storage Format v2 for improved efficiency and scalability
• Zero-Downtime State Migration - an innovative new method to seamlessly roll out the new account storage format on the live network without requiring any downtime.

🔹 What’s New in Account Storage Format v2?

The Account Storage Format v2 enhances how the on-chain state is stored and processed, resulting in significantly reduced memory usage on the Execution nodes.

The Execution State is a Merkle tree, which holds the user data (account token balances, NFTs, smart contracts deployed etc.) as payloads in its leaves. This update optimizes the storage format for the payloads resulting in fewer, more compactly serialized payloads. Fewer payloads in turn reduce the size of the MTrie, making Flow more efficient in handling state.

With this update,

• The number of payloads will be reduced by 25-30%, reducing the on-disk storage footprint of the execution data. This will improve RAM, Disk and CPU efficiency, not only for execution nodes but also for all databases, indexers, and caches that store and serve execution data such as the access node.
• The number of vertices in the MTrie will also be reduced by 25-30% reducing the in-memory storage footprint of the data for execution nodes, which hold the execution state in memory. With this size reduction, execution nodes will see more than an 11% reduction in memory usage (RAM) improving their stability and capacity to accommodate memory spikes during peak usage.

This improvement will also help slow down future memory growth rate as new accounts are created, further ensuring that Flow scales efficiently.
Read more about this change here 👉 GitHub Issue #3584

🔹How Will Account Storage v2 Be Rolled Out? Zero-Downtime Migration!

To deploy the new Account Storage Format without any network downtime, an innovative approach called Zero-Downtime Migration has been introduced.

Zero-Downtime Migration uses system transactions to migrate small batches of accounts, running in the background without affecting network operations. The ambient process takes several days to complete.

With this approach, the new Account Storage Format will be deployed gradually and seamlessly without requiring any action from developers or users, ensuring zero disruption to the network.

🔜 Upcoming improvements

This upgrade also lays the foundation for several upcoming improvements that will be rolled out as part of the next set of updates. These include:

• Websocket Streaming API endpoints aimed at improving the user experience of dApps.
• The ability to deploy protocol upgrades including downwards-incompatible changes for all node types to the live Mainnet - without requiring any network downtime!
• Easy recovery from epoch transition failures, again requiring no downtime.

This upgrade marks another significant step in Flow’s journey toward scaling to petabytes of on-chain data. 🚀

---

❗Breaking change

There is a breaking change to streaming Access API in this upgrade.

Several AccessAPI and ExecutionDataAPI streaming endpoints include a message_index field in the response which indicates the order of messages sent by the server so the client can detect if they missed processing any. Previously, some endpoints used 1 as the first index in a stream while others used 0. After the HCU, all endpoints will be consistent regarding message_index and start from index 0.
APIs that will affected (Protobuf definition) -

1. SubscribeEvents - field message_index
2. SendAndSubscribeTransactionStatuses - field message_index
3. SubscribeAccountStatuses - field message_index

These streaming APIs are fairly recent, therefore most apps will not be impacted.

---

⚠️ Deprecation

• The protocol_version field of the NodeVersionInfo endpoint of the Access API is deprecated and will be removed in a future release.

---

Commit list

Account Storage Format v2 Migration

• Cadence Account Storage Map Migration by @turbolent in #6761
• Enable storage format v2 for all but mainnet by @janezpodhostnik in #6864
• Enable storage format v2 on mainnet by @janezpodhostnik in #6988

NOTE: Account Storage Format v2 is implemented in PRs in repo onflow/cadence.

Zero downtime (Recovering from failed epoch switchover)

• Add test cases documenting CBOR's behaviour with omitted and extra fields by @jordanschalm in #6880

Data Availability

• Reduce logging from version control by @peterargue in #6342
• Reduce logging on some access endpoints by @peterargue in #6340
• Update event parsing validator to support ResourceDestroyed events by @peterargue in #6356
• Updated go-ds-pebble version by @UlyanaAndrukhiv in #6411
• Add compatible range to Node Version Info endpoint by @UlyanaAndrukhiv in #6294
• Added metrics for transaction validator by @Guitarheroua in #6239
• Stop Control feature for AN by @Guitarheroua in #6202
• Implemented options to fail or warn when payer is not able to pay by @Guitarheroua in #6297
• Use Indexed height when checking payer balance by @AndriiDiachuk in #6292
• Enforce prune threshold by @Guitarheroua in #6345
• Improve error messages when querying old blocks by @UlyanaAndrukhiv in #6554
• Store Transaction Result error messages in db by @UlyanaAndrukhiv in #6468
• Refactor executionNodesForBlockID by encapsulating its parameters into a struct by @UlyanaAndrukhiv in #6499
• Improve bounds checking in rest converters by @AndriiDiachuk in #6567
• Make REST message size limit configurable by @AndriiDiachuk in #6596
• Change SendAndSubscribe endpoint's MessageIndex to start at 0 by @AndriiDiachuk in #6598
• Add util command to backfill tx error messages db by @UlyanaAndrukhiv in #6525
• Refactor REST Routes Package by @UlyanaAndrukhiv in #6616
• Add support for ignoring version beacon events for compatible versions by @peterargue in #6535
• Add new websocket handler and skeleton for its deps by @illia-malachyn in #6630
• Add implementation BlockProvider by @UlyanaAndrukhiv in #6636
• Implement keepalive routine with ping-ponging to ws connection in ws controller by @UlyanaAndrukhiv in #6757
• Add unit test for websocket controller by @illia-malachyn in #6762
• Fix mocking in connection unittests by @peterargue in #6822
• Add implementation for events data providers and account statuses data providers by @AndriiDiachuk in #6766
• ws controller error handling by @illia-malachyn in #6798
• Implement inactivity tracker by @UlyanaAndrukhiv in #6817
• Add implementation for transaction statuses data providers by @AndriiDiachuk in #6818
• Response limit tracker by @Guitarheroua in #6814
• Make WebSocket responses from data providers consistent with Access REST API responses by @UlyanaAndrukhiv in #6802
• Fix access connection tests for latest version of testing library by @peterargue in #6917
• Unify subscription id with client message id by @illia-malachyn in #6847
• Heartbeat interval not reset correctly in streaming endpoints. by @AndriiDiachuk in #6848
• Make response for data providers consistent. by @AndriiDiachuk in #6846
• Add deprecate message to legacy ws handler and controller by @illia-malachyn in #6918
• Add arguments getter for data providers by @UlyanaAndrukhiv in htt...

Jan 27th 2025 Height Co-orindated Upgrade on Mainnet26 at block height 101584244

What's Changed

FVM

• Get execution version from snapshot instead of state - v0.37 port by @janezpodhostnik in #6883
• Move execution parameter to separate account - V0.37 by @janezpodhostnik in #6891

Cadence

• Port internal fixes by @turbolent in #6937

Collection

• Removes unused first height field from LN builder by @jordanschalm in #6829
• Reduce log level in provider by @peterargue in #6931

CI/CD

• Update Makefile to support private util builds by @sjonpaulbrown in #6797
• Jan/port pr6924 to v0.37 by @j1010001 in #6925

Util

• Backport v0.37 cmd add verify execution result by @zhangchiqing in #6791
• Backport 0.37 fix verify execution result by @zhangchiqing in #6805
• Backport v0.37 add nWorker to verify execution result by @zhangchiqing in #6831
• Log verification progress by @zhangchiqing in #6834
• Backport v0.37 add query last executed sealed block by @zhangchiqing in #6920
• Backport: Add util command to debug transaction by @turbolent in #6923

Full Changelog: v0.37.22...v0.37.36

Nov 21st 2024 Height Co-orindated Upgrade on Mainnet26 at block height 94361765

What's Changed

Data Availability

• Add a Time To Seal metric to access node to track time it takes to seal a transaction by @AndriiDiachuk in #6605
• Backport transaction insufficient balance by @Guitarheroua in #6549
• Fixed issues where time-to-seal was NaN by @AndriiDiachuk in #6625
• Marking v0.37.22 as backward compatible by @vishalchangrani in #6741

CI/CD

• Remove input by @Kay-Zee in #6618
• Add support for building private dependencies in CI by @sjonpaulbrown in #6728

FVM

• Copy slice when setting block hash list - Port by @janezpodhostnik in #6740

Cadence

• Update to Cadence v1.0.3 by @turbolent in #6753

Full Changelog: v0.37.20...v0.37.22

October 29th 2024 Height Co-orindated Upgrade on Mainnet26 at block height 91874965

What's Changed

• Marking v0.37.18 as backward compatible in terms of script execution by @vishalchangrani in #6595
• Expose version boundary to cadence interface - port by @janezpodhostnik in #6597
• Update to Cadence v1.0.2 by @turbolent in #6607

Full Changelog: v0.37.17...v0.37.20

October 8th 2024 Height Co-orindated Upgrade on Mainnet26 at block height 89610514

What's Changed

• Backport v0.37 [Bootstrap] Fix duplication detection in root block finalization by @zhangchiqing in #6485
• cherrypick commits from PR: #6513 by @vishalchangrani in #6530
• [EVM] port of #6451 - Adding commitment over state update by @ramtinms in #6476
• [EVM] port of 6456 - add state update checksum to tx executed events by @ramtinms in #6477
• [Access] Add support for ignoring version beacon events for compatible versions by @peterargue in #6536
• Update to cadence v1.0.1 by @SupunS in #6548

Full Changelog: v0.37.16...v0.37.17

Mainnet 26

What's Changed

Cadence

• Port internal 6971 by @turbolent in #6502
• Reject entitled Account.capabilities.get/borrow/publish by @turbolent in #6500

Network

• Add Libp2p Gossipsub Peer Gater - v0.37 by @peterargue in #6493
• switching off peer-scoring by default by @vishalchangrani in #6473
• Change sample size log from warn to debug by @kc1116 in #6460

EVM

• Port PR 6430 (Support EVM registers in check-storage) to v0.37 by @fxamacker in #6454
• port #6467 - patch evm tracer by @ramtinms in #6475
• backport of 6480 - handle UInt,Int types for ABI encoding/decoding by @ramtinms in #6481
• port 6486 - Add boundary checks for ABI encoding values of type UInt & Int by @ramtinms in #6492

FVM

• Fix metering invalidation - port by @janezpodhostnik in #6461

Execution

• Lower retry interval in EN requester engine by @jordanschalm in #6436

Tools

• Fix cluster assignment >2/3 internal check by @jordanschalm in #6432
• Recover validation error pretty printing panic by @turbolent in #6434
• update to transit script pull command to download file if md5 of remote file is not known by @vishalchangrani in #6453
• adding an option to boostrapcmd to pull candidate access nodes in addition to proposed nodes by @vishalchangrani in #6489
• Remove internal partners from root block creation by @zhangchiqing in #6471

Misc

• minor revision of log, to clarify the root causes of likely benign error log by @AlexHentschel in #6420
• Transaction Validator recovers from script parse panic by @jordanschalm in #6443

Full Changelog: v0.37.10...v0.37.16