[Badger Batch] Chunk Data Packs and ConsumeProgress #6391
Conversation
zhangchiqing
force-pushed
the
leo/badger-chunk-queue
branch
from
9d87c56
to
5c6c7fe
Compare
| if err != nil { | ||
| return 0, fmt.Errorf("failed to retrieve processed index: %w", err) | ||
| } | ||
| return processed, nil | ||
| } | ||
|
|
||
| // InitProcessedIndex insert the default processed index to the storage layer, can only be done once. | ||
| // initialize for the second time will return storage.ErrAlreadyExists |
There was a problem hiding this comment.
With batch updates, it's not easy to prevent dirty reads in this call when checking the case that might return ErrAlreadyExists. We would need to introduce a lock.
Instead of using lock, I decided to move this check to a Factory struct, since it's easy for the owner of the Factory to ensure initializing in a single thread, and then the returned consumer progress could be concurrent-safe. This eliminates the uninitialized state for the ConsumerProgress, since the caller wouldn't have the object unless it's been initialized by the factory.
zhangchiqing
changed the base branch from
leo/badger-batch-approvals
to
master
zhangchiqing
force-pushed
the
leo/badger-chunk-queue
branch
from
5c6c7fe
to
161566c
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6391 +/- ##
==========================================
- Coverage 41.50% 41.44% -0.06%
==========================================
Files 2013 2021 +8
Lines 143577 143867 +290
==========================================
+ Hits 59590 59631 +41
- Misses 77813 78061 +248
- Partials 6174 6175 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
zhangchiqing
requested review from
jordanschalm and
AlexHentschel
and removed request for
ramtinms
zhangchiqing
changed the base branch from
master
to
leo/badger-batch-approvals
zhangchiqing
force-pushed
the
leo/badger-chunk-queue
branch
from
5f3f7f3
to
13aa4c3
Compare
|
Close for now. Found a better way to refactor. |
|
Will re-implement once #6466 is merged. |
This PR refactored the storage module of chunk data packs and consumer progress to use badger batch updates.
There are many file changes, primarily due to a refactor that implements the factory pattern, which addresses a possible API misuse. I've provided further explanation in the comments below.