Split server into "overseer" and "app" processes

[7sempra]

Allows us to shut down the server in order to do DB maintenance, import new data, etc.

First use case: updating the SDE import. We really want to shut down the server while we're mucking about with the SDE definitions.

• nf start now starts the overseer process
• Overseer spawns the app process
• App process runs the express server and tasks queue
• Overseer and app process can communicate via node's built-in communication system
• Overseer runs its own, minimal, express server (on a different, high port like 8080). It exposes some simple UI to start/stop the app process.
• Could eventually migrate task management out of app process and into overseer (each task would be its own, spawned process).

[ayust]

What functionality does the overseer process provide beyond dokku's normal app management?

[7sempra]

So we need some way to interact with the DB while the server is not running. For example, when updating the SDE, we don't want the app server running the background and reading partially-true data. We could acquire locks on those tables, but the SDE import may take a while due to the need to rebuild full text indices and we then have the burden of remembering to acquire read locks everywhere that touches an SDE table.

We could manually shut down the app server container, then start a new container that just performs the maintenance, then shut that container down, then start the app server again. That's pretty laborious, though (and can't be automated easily or invoked from a web UI). Is there another option?

[ayust]

Could just add a 'maintenance mode' flag to the app server that results in all requests returning a 'Maintenance in progress' response that doesn't require any db queries to serve, no?