Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move changefeed listeners to web service #1057

Open
heyheyjp opened this issue Jul 26, 2017 · 2 comments
Open

Move changefeed listeners to web service #1057

heyheyjp opened this issue Jul 26, 2017 · 2 comments

Comments

@heyheyjp
Copy link
Collaborator

Currently, the workers set up listening to db changefeeds and, effectively, queue items for their own work queues. This is problematic because if changes to the database occur that should be handled by a worker while the worker is not running, that change event is lost and the job is never processed. Instead of connecting the changefeed listeners in the worker process, we should do it in the standing web service. This also makes it easier to choose to alternative mechanisms for background task execution (instead of having always-running worker dynos).

@jeffreywescott
Copy link
Contributor

For what it's worth, these used to be in the web service, but it caused performance problems during "heavy" load (i.e., voting), since JavaScript is single-threaded.

@heyheyjp
Copy link
Collaborator Author

heyheyjp commented Jul 26, 2017

Ah. Good to know. We might find this interesting in thinking about how to attack the issue:
https://www.rethinkdb.com/docs/sharding-and-replication/#running-a-proxy-node

@heyheyjp heyheyjp reopened this Jul 26, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants