add --ephemeral mode for persistent worker

[0xB10C]

When passing --ephemeral to cirrus worker run, the worker will accept one task and then exit the process once the task completed.

This can be used inside e.g. ephemeral VMs which should be shutdown after each task. A user has to take care of cleaning up after the worker has finished.

resolves #809

[0xB10C]

This still needs a bit of work on the backend side, I think. Ideally, the backend would understand that the worker is ephemeral: the worker should tell the backend in some proto message, possibly WorkerInfo. The backend should clean up ephemeral workers faster than normal persistent workers.

When I run ephemeral workers with unique names, my worker pool grows quite fast with mostly inactive, old workers still present. When I reuse the same name, scheduling tasks after the VM comes back online sometimes doesn't work (the backend probably thinks it already assigned a task).

Would love your input on this @fkorotkov.

[fkorotkov]

Seem we'll need an option for the worker to unregister itself similar to cirrus worker pause command. Let us look into this next week after Thanksgiving.

[0xB10C]

Seem we'll need an option for the worker to unregister itself similar to cirrus worker pause command. Let us look into this next week after Thanksgiving.

Sounds good! I remember trying to use SetDisabled, but that turned out to only pause the worker and unpausing didn't fix the scheduling problems I had.

[0xB10C]

I've observed some of my --ephemeral runners not getting a task and log level=error msg="failed to poll upstream https://grpc.cirrus-ci.com:443: rpc error: code = AlreadyExists desc = entity already exists: app: \"s~cirrus-ci-production\"\npath <\n Element {\n type: \"PersistentWorkerTaskAssignment\"\n id: 0x1226f1681c0000\n }\n>\n"

[0xB10C]

rebase and resolved conflicts

[0xB10C]

I don't think the cirrus team is interested in merging this.

I would have hoped for clearer communication on this from start, as this would have saved me a bunch of time working on this..