syzbot: cover more configurations

[tarasmadan]

Is your feature request related to a problem? Please describe.
Inspired by the kitchen discussion we had yesterday and #6292 (comment) .

It would be nice to be able to cover more combinations of

1. architecture (x86, arm, risc-v etc.)
2. sanitizers (kasan, kmsan, kcsan etc.)
3. trees (more public, more private)
4. kernel_configs
5. attack_point (root, user, capabilities etc.)

Multiplying all these options we're getting too many target configurations.
The alternative may be to have a few instances focused on the coverage extension and the workers who are actually executing this corpus on the target systems.

I'm 98% sure this approach was previously discussed. @dvyukov, am I duplicating some existing issue/doc?

Experiments required

1. How many bugs are found by the programs that don't add new coverage?
2. TODO

[dvyukov]

am I duplicating some existing issue/doc?

I don't remember any existing issues off the top of my head. It was discussed that we could generate hundreds of permutations for gVisor flags, which is a similar problem. Simliarly, we could generate permutations for qemu flags, currently we also have 2 fixed sets. There are few combinations for options for GCE VMs as well.

[ramosian-glider]

Following an offline discussion, we could probably introduce a new type of dashboard clients that would take an existing corpus and replay it on several different kernel builds / command line flag configurations without fuzzing on a daily basis.

[tarasmadan]

Snapshot based machine (~25 execs/second) will be able to replay 90k (current corpus looks comparable) programs in 1 hour.
It means 24 configurations/day. It is the complexity of our upstream infra.
Every tail report (#6332) will be about the specific program from the corpus.

[dvyukov]

Note: it's only about finding bugs. Once we found a bug, we need to do cause bisection, then periodic re-testing and fix bisection, and patch testing.