Improve resiliency for cross-service requests

[donaldgray]

Various bits of the solution make HTTP requests to other service. We don't have particularly robust handling of failures, we should consider adding some resiliency - the most obvious way is by utilising https://github.com/App-vNext/Polly but there may be alternatives.

Can this be natively wired into YARP? If not is it on roadmap? (Possibly related: #457).

We need to think about how different services handle errors for services, it's unlikely a blanked approach will fit all use cases.