refactor: use trylock

[awill1988]

This attempts to deal with write contention by using TryLock of the RWMutex and recurse if the lock cannot be obtained which will check for the key again.

Related issue(s)

#3866

Checklist

• I have read the contributing guidelines.
• I have referenced an issue containing the design document if my change
  introduces a new feature.
• I am following the
  contributing code guidelines.
• I have read the security policy.
• I confirm that this pull request does not address a security
  vulnerability. If this pull request addresses a security vulnerability, I
  confirm that I got the approval (please contact
  [email protected]) from the maintainers to push
  the changes.
• I have added tests that prove my fix is effective or that my feature
  works.
• I have added or changed the documentation.

Further Comments

The idea is that when the lock is not obtained for writing we can leverage the implicit latency of the database statement to get the key after recursing.

[awill1988]

I'm beginning to think the failures on conformance are false negatives b/c I had a conformance pass with static sleep value of 3ms.. and I've got conformance passing locally without sleep, so maybe #3866 works on its own 🤷🏻

[awill1988]

🤷🏻