Add feature to enable atree inlining migration to fix references to non-existent registers #388
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In testnet, broken references seem to have resulted from a bug that was fixed 2 years ago by onflow/cadence#1565. A broken reference is a `StorageID` referencing a non-existent register. So far, only 10 registers in testnet (none on mainnet) were found to contain broken references. This commit adds a feature to enable migration programs in onflow/flow-go to fix broken references in maps. `FixLoadedBrokenReferences()` traverses loaded slabs and replaces broken map (if any) with empty map having the same StorageID and also removes all slabs in the old map. Limitations: - only fix broken references in map (this is intentional) - only traverse loaded slabs in deltas and cache IMPORTANT: This should not be used to silently fix unknown problems. It should only be used by migration programs to fix known problems which were determined to be appropriate to fix in this manner.
|
LGTM! As this is basically the same as #387, I have the same feedback. Once that other PR is in, it would be good to update this PR in a similar way |
Co-authored-by: Bastian Müller <[email protected]>
Currently, calls to this function can be limited by migration programs by specifying the 9 testnet accounts affected by 10 registers with broken references on testnet. This commit allows callers to implement additional restrictions, so calling FixLoadedBrokenReferences for the affected 9 testnet accounts can be even more limited at the callers discretion. In practice, this change is not expected to produce different migration results because full migration tests before this change correctly fixed the 10 known registers in the 9 testnet accounts. This commit added predicate func(old Value) bool to PersistentSlabStorage.FixLoadedBrokenReferences() to control whether to fix a atree.Value containing broken references. Also modified PersistentSlabStorage.FixLoadedBrokenReferences() to return fixed storage IDs and skipped storage IDs. Both returned values are of type map[StorageID][]StorageID, with key as root slab ID and value as all slab IDs containing broken references connected to the root. Also added more tests and improved existing tests.
This commit adds GetAllChildReferences() which essentially wraps an existing internal function.
6 tasks
…ences-for-atree-inlining Add PersistentSlabStorage.GetAllChildReferences() for atree inlining
This was referenced Apr 17, 2024
fxamacker
added a commit
to onflow/flow-go
that referenced
this pull request
Apr 22, 2024
This commit adds --fix-testnet-slabs-with-broken-references flag to the migration program. It uses a feature from onflow/atree#388 to fix 10 testnet registers affecting 9 testnet accounts. The 9 testnet accounts are hardcoded in this commit to prevent accidentally applying this fix to any other accounts. In testnet, broken references seem to have resulted from a bug that was fixed 2 years ago by onflow/cadence#1565. A broken reference is a `StorageID` referencing a non-existent register. So far, only 10 registers in testnet (none on mainnet) were found to contain broken references.
turbolent
pushed a commit
to onflow/flow-go
that referenced
this pull request
Apr 23, 2024
This commit adds --fix-testnet-slabs-with-broken-references flag to the migration program. It uses a feature from onflow/atree#388 to fix 10 testnet registers affecting 9 testnet accounts. The 9 testnet accounts are hardcoded in this commit to prevent accidentally applying this fix to any other accounts. In testnet, broken references seem to have resulted from a bug that was fixed 2 years ago by onflow/cadence#1565. A broken reference is a `StorageID` referencing a non-existent register. So far, only 10 registers in testnet (none on mainnet) were found to contain broken references.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Updates #292 #386
Problem
Broken references were found in testnet data and they seem to have resulted from a bug that was fixed 2 years ago by onflow/cadence#1565.
A broken reference is a
StorageIDreferencing a non-existent register. So far, only 10 registers in testnet (none on mainnet) were found to contain broken references.Solution
Port from non-inlining version of the solution:
This PR adds a feature to enable migration programs in onflow/flow-go to fix broken references in maps.
FixLoadedBrokenReferences()traverses loaded slabs and replaces broken map (if any) with empty map having the sameStorageIDand also removes all slabs in the old map.Limitations:
IMPORTANT: This should not be used to silently fix unknown problems. It should only be used by migration programs to fix known problems which were determined to be appropriate to fix in this manner.
TODO:
mainbranchFiles changedin the Github PR explorer