Improve the cache hint mechanism / job scheduling

[tmcgilchrist]

Background

Improve the cache hint mechanism / job scheduling for tezos-ci. For now the scheduler is simple, it assumes 600 seconds for an uncached job, 10s if the cache hint matches. An exact match on the cache hint doesn't happen in practice, notably because it's a string and not a string Current.t so it can only be "generic" things. However if the cache hint is too broad, then the scheduler will send all jobs to one machine because it's always better to get the "10s" jobs instead of 600s, even if in practice these "cached" jobs are slower.

Summary

What I'd propose is a progressive match (like github's cache hint), for example
<tezos opam commit>:<MR#>:<commit>:<test name>
and scheduler would send the job to the available worker with the largest common prefix.

Outcome

Improved cache hint support is available on tezos-ci cluster and being used for scheduling jobs.