Gliner quantization

[carloronsi]

Hi everyone! I've tried the quantization code present in this report:

https://github.com/urchade/GLiNER/blob/main/examples/convert_to_onnx.ipynb

Followed step-by-step, but the quantized model (last cell) is not returning any entity (if the threshold is reduced, texts are labeled incorrectly).

Am I missing something? Is there an additional step required to make the quantized model work?

Thank you all!

[urchade]

@Ingvarstep

[miguelwon]

I'm also having a similar problem. When quantizing to 8bit I'm measuring a quite strong drop in performance. More than I usually see for this type of models.

[merlihson]

Have you, guys, tried Quantization Aware Training? Can it technically work with GliNER?