Flaky matcher

[paul-butcher]

What does this change?

Fixes #2807

Part of this is simply to clarify intent, but the fundamental fixes (I believe) are:

• replace the spinlock waits with eventually.
  • I'm not sure if this actually makes a difference, but I think it's more correct
  • I did see some instances where the spinlock version failed but the Patience version succeeded.
• Ensure that A starts first
  • One instance of this problem seemed to be that A never had to wait for C to start (because it had already started), which also meant that C never waited (because C finished before A acquired the lock)
• Lock A before waiting for C
  • As above. Sometimes C didn't have to wait for the A lock to be released, because it got there before the lock was applied.

I have also made some changes to clarify the intent of the waits and make things easier to debug.

How to test

Look at buildkite. Retry the build associated with this PR several times. I have already run it a few times on the associated branch.

Without this fix, I would expect to see at least some failure if run five times.

How can we measure success?

We no longer have to retry the matcher task to get our PRs to build successfully.

Have we considered potential risks?

This is a change to an unreliable test. The only risk is that I haven't actually fixed the problem, but even so, this approach is an improvement on what was there before