Block scaled dot HLO op

[sergey-kozub]

Block scaled dot is an operation that accepts quantized inputs, then dequantizes and performs the dot op. (NVidia) Blackwell has the MMA unit that does this at the hardware level.

There are a few options how this could be implemented in XLA:

1. A graph of XLA ops - this requires pattern matching, which will fail if the cost model fusion decides to put the dequantize and the dot op in separate fusion computations.
2. A custom call, this is currently implemented in the BlockScalingRewriter pass.
3. A specific HLO op (block-scaled-dot), taking quantized input and scale tensors as inputs - PR#22141 proposes to introduce this.

The question to discuss is, whether it's worth introducing a new HLO op (block-scaled-dot), or whether we could just continue using options (1) and/or (2) in the meanwhile.