Skip to content

Conversation

kalloc
Copy link
Contributor

@kalloc kalloc commented May 24, 2025

Observed a case where the pipeline continues running even after all datasources have finished their work.
For example, when using RpcBlockCrawler with a defined end_slot, the pipeline does not terminate automatically upon reaching the end.

--

From Deluge with love!

@kalloc kalloc marked this pull request as draft May 24, 2025 14:05
@kalloc kalloc changed the title fix(core): pipeline doesn’t stop automatically after all registered datasources complete (WIP) fix(core): pipeline doesn’t stop automatically after all registered datasources complete May 24, 2025
@kalloc kalloc marked this pull request as ready for review May 24, 2025 18:08
@kalloc
Copy link
Contributor Author

kalloc commented May 24, 2025

@sonicfromnewyoke hi, I found that the RPC block crawler datasource does not exit after complete crawling. I wrote a reproductive example (also maybe possible as just example) and resolved the issue.
Moreover, I removed referencing to a sender channel (&sender) in datasources because moving or .clone is enough, channel already based on Arc. If you do not agree with it, I can return.

Copy link
Collaborator

@sonicfromnewyoke sonicfromnewyoke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wow, this is huge 🤩
run lt locally, everything works good

@sonicfromnewyoke sonicfromnewyoke merged commit 1a83d89 into sevenlabs-hq:main May 24, 2025
5 checks passed
@sonicfromnewyoke
Copy link
Collaborator

@kalloc thank you for the so precise debug and fix 💪

@DerZwergGimli
Copy link

DerZwergGimli commented Jun 6, 2025

can we make sure this is in a new version maybe 0.8.2 since using crates.io

[cargo.toml]

carbon-core = 0.8.1
carbon-rpc-block-crawler-datasource = 0.8.1

still gives the described issue (the Pipeline won't stop - only referencing the git repo resolves this)

[cargo.toml]

carbon-core = { git = "https://github.com/sevenlabs-hq/carbon", version = "0.8.1" }
carbon-rpc-block-crawler-datasource = { git = "https://github.com/sevenlabs-hq/carbon", version = "0.8.1" }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants