Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Access] Mark execution data fulfilled after all consumers finish processing #6217

Conversation

UlyanaAndrukhiv
Copy link
Contributor

@UlyanaAndrukhiv UlyanaAndrukhiv commented Jul 15, 2024

Closes: #6138

Context

In this pull request:

  • Added supporting of handling 2 cases ( newly indexed blocks on the indexer and newly synced execution data on downloader) for marking a height as fulfilled for the Execution data pruner.
    In the case where indexing is disabled, but execution sync and pruning are enabled - pruner registers Download as ExecutionDataProducer and marks a height as fulfilled as soon as the data for a block is stored.
    In the case where indexing and execution sync enabled - pruner checks the lower processed value.

  • Added integration test, updated existing functional tests for pruner

@codecov-commenter
Copy link

codecov-commenter commented Jul 15, 2024

Codecov Report

Attention: Patch coverage is 69.53642% with 46 lines in your changes missing coverage. Please review.

Project coverage is 41.55%. Comparing base (09712a4) to head (d157091).

Files Patch % Lines
cmd/access/node_builder/access_node_builder.go 0.00% 12 Missing ⚠️
cmd/observer/node_builder/observer_builder.go 0.00% 12 Missing ⚠️
...xecutiondatasync/execution_data/mock/downloader.go 41.66% 4 Missing and 3 partials ⚠️
...tasync/execution_data/processed_height_recorder.go 37.50% 5 Missing ⚠️
module/state_synchronization/indexer/indexer.go 88.37% 3 Missing and 2 partials ⚠️
module/executiondatasync/pruner/pruner.go 92.50% 2 Missing and 1 partial ⚠️
engine/execution/ingestion/engine.go 94.73% 1 Missing ⚠️
...ule/executiondatasync/execution_data/downloader.go 80.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6217      +/-   ##
==========================================
+ Coverage   41.54%   41.55%   +0.01%     
==========================================
  Files        1992     1993       +1     
  Lines      141553   141618      +65     
==========================================
+ Hits        58804    58853      +49     
- Misses      76625    76641      +16     
  Partials     6124     6124              
Flag Coverage Δ
unittests 41.55% <69.53%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@peterargue peterargue left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for fixing the race condition. added a few comments, but I think you're on the right track

module/executiondatasync/pruner/pruner.go Outdated Show resolved Hide resolved
module/executiondatasync/pruner/pruner.go Outdated Show resolved Hide resolved
module/executiondatasync/pruner/pruner.go Outdated Show resolved Hide resolved
@UlyanaAndrukhiv UlyanaAndrukhiv marked this pull request as ready for review July 22, 2024 19:17
Copy link
Contributor

@peterargue peterargue left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @UlyanaAndrukhiv! looks good.

@peterargue peterargue added this pull request to the merge queue Jul 29, 2024
Merged via the queue into onflow:master with commit f296f51 Jul 29, 2024
55 checks passed
@peterargue peterargue deleted the UlyanaAndrukhiv/6138-execution-data-sync-improvements branch July 29, 2024 23:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Access] Mark execution data fulfilled after all consumers finish processing
4 participants