fix: ring buffer deadlock

[IronCore864]

Fix an issue where the ring buffer might deadlock because rwlock and iterator mutex are acquired in a different order.

Fixes: #508

Notes:

1. As discussed with Harry and Ben, rb.Write isn't adjusted because the rwlock and iterator mutex are acquired at different times.
2. Besides an internal close function, an internal positions function is created for the same reason.
3. The test case here is not added because it was supposed to be an easy way to reproduce the deadlock. Adding an additional benchmark test is not useful, and the deadlock issue can be caught by the TestDeadlock test case anyway.
4. All three functions signalIterators, releaseIterators and removeIterator are adjusted (although technical only releaseIterators needed to be ) because they share a common naming pattern so it's nicer to share a common implementation pattern as well: all internal functions don't operate locks.

Benchmark test, manual test to reproduce the 3-way deadlock issue, and race test are all done and passed.

[benhoyt]

Looking good, thanks. Just a couple of nit comments to see if we can tidy up a couple of things.

[IronCore864]

I have refactored according to the code review from Ben and Dima, and then the final tests with -race and manual deadlock testing passed successfully. @hpidcock please take a look.

[benhoyt]

Looks good to me, and passes my deadlock tests locally.

[benhoyt]

I'm going to merge this optimistically so we can move ahead with c3cb594, but @hpidcock -- we'd still appreciate your review-after-the-fact when you get a chance.

[hpidcock]

I'm going to merge this optimistically so we can move ahead with c3cb594, but @hpidcock -- we'd still appreciate your review-after-the-fact when you get a chance.

LGTM. Looks like all the previous locking behaviour is preserved with the consistent ordering to avoid deadlocks.