Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for "one-shot" sources #19

Open
nhairs opened this issue Sep 2, 2024 · 0 comments
Open

Add support for "one-shot" sources #19

nhairs opened this issue Sep 2, 2024 · 0 comments
Labels
new feature New feature open for work Issue can be picked up by anyone

Comments

@nhairs
Copy link
Owner

nhairs commented Sep 2, 2024

Some sources (e.g. source.file.DirectoriesAndFiles) will run out of jobs and throw StopIteration. Whilst this is correctly caught and the SourceWorker will shutdown we should monitor all the source workers and if they all shutdown then we should shutdown the whole application.

This might need a new state as the sources should not be shutdown/cleaned up until after the sink workers have shutdown as the sources will need to ack the jobs.

@nhairs nhairs added new feature New feature open for work Issue can be picked up by anyone labels Sep 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new feature New feature open for work Issue can be picked up by anyone
Projects
None yet
Development

No branches or pull requests

1 participant