Define an experiment on a subset of crates from the github bot #460
Description
Full crater runs now take about a week, which led @eddyb to combine a few unrelated PRs into a single crater run (see rust-lang/rust#64398). Unfortunately, one of those PRs caused an issue in about 600 crates, so we can't definitively say that the other PRs would have passed.
Ideally, we would be able to specify a list of crates when triggering an experiment from the github bot (e.g. as a link to a gist containing the crate identifiers) and run one experiment for each PR on the failing crates. This is already possible from the CLI via config.toml, but setting up crater locally to build arbitrary rustc commits is not trivial.
@pietroalbini I'm going to look into this. Let me know if you have thoughts.