Continuous testing reruns can give different results from 'fresh' run

[holly-cummins]

When debugging #58, I've noticed that if I add or remove a problem annotation, or disable and enable tests, the new tests will run, but the results aren't accurate – that is, they're different from what I'd get if I stopped and restarted.

I assume the issue is that some global state is being changed on startup, and that's what causes #58, and that also causes this. The two fixes may be the same, but tracking both issues separately to make sure it's resolved.