Implement beacon db pruner #14687
Implement beacon db pruner #14687
Conversation
dB2510
force-pushed
the
beacondbpruning
branch
from
0138619 to
df84c28
Compare
|
No flag to customize the retention period? |
Will include it in this PR. |
dB2510
force-pushed
the
beacondbpruning
branch
from
7eb6d40 to
20aafa7
Compare
dB2510
force-pushed
the
beacondbpruning
branch
from
4f2164b to
b373d0d
Compare
| @@ -590,26 +636,26 @@ func (s *Store) SaveRegistrationsByValidatorIDs(ctx context.Context, ids []primi | |||
| } | |||
|
|
|||
| // blockRootsByFilter retrieves the block roots given the filter criteria. | |||
| func blockRootsByFilter(ctx context.Context, tx *bolt.Tx, f *filters.QueryFilter) ([][]byte, error) { | |||
| func blockRootsByFilter(ctx context.Context, tx *bolt.Tx, f *filters.QueryFilter) ([][]byte, map[primitives.Slot][][]byte, error) { | |||
There was a problem hiding this comment.
Seems a bit wasteful to return the roots in a slice and map. Can you make this work with just the map?
| defer span.End() | ||
|
|
||
| // Perform all deletions in a single transaction for atomicity | ||
| return s.db.Update(func(tx *bolt.Tx) error { |
There was a problem hiding this comment.
I think you are not deleting everything you should. Looking at bucket names, there are also:
checkpointBucket(if this bucket stores what the name suggests, then if we remove an old checkpoint block, this bucket will still contain the checkpoint)stateValidatorsBucket(I think this one is very important because validators are a big chunk of the state)blockParentRootIndicesBucketstateSlotIndicesBucketfinalizedBlockRootsIndexBucketblockRootValidatorHashesBucketoriginCheckpointBlockRootKey(I don't know the implications of removing this, it's best to ask @kasey about it. The bucket is used in a few places, but after the first pruning the origin checkpoint will "move". Shouldn't we update the value in the bucket accordingly?)
| // Delete block from cache | ||
| s.blockCache.Del(string(root)) | ||
| // Delete state summary from cache | ||
| s.stateSummaryCache.delete([32]byte(root)) |
There was a problem hiding this comment.
This might not be very important to fix, but if the transaction fails these deletions will not roll back. Maybe it would be better to store a slice of roots and then clear caches after the transaction completes? The downside of this is allocating the slice.
| for { | ||
| select { | ||
| case <-p.ctx.Done(): | ||
| return |
There was a problem hiding this comment.
Can we debug log something?
| case <-p.ctx.Done(): | ||
| return | ||
| case <-p.done: | ||
| return |
There was a problem hiding this comment.
Can we debug log something?
|
Additional comment:
|
Co-authored-by: Radosław Kapka <[email protected]>
Co-authored-by: Preston Van Loon <[email protected]>
Co-authored-by: Radosław Kapka <[email protected]>
What type of PR is this?
Feature
What does this PR do? Why is it needed?
This PR implements beacon db pruning based on weak subjectivity period. This is needed to avoid storing historical blocks beyond a point for non-archival nodes.
Which issues(s) does this PR fix?
#8787
Other notes for review
This PR implements a pruner service based on weak subjectivity period to periodically delete blocks and states beyond weak subjectivity period to maintain a constant-size database.
Acknowledgements