*2-bit quantization produces \name\ instead of "name" in JSON output, making tool calling unreliable. 4-bit is the production configuration.
I wonder if using json-repair here might get past this ?
BoundryML looks like another solution https://docs.boundaryml.com/home
I'm sure there will be other issues past the tool calling issue, but it would be interesting to see what they are.
I wonder if using json-repair here might get past this ?
BoundryML looks like another solution https://docs.boundaryml.com/home
I'm sure there will be other issues past the tool calling issue, but it would be interesting to see what they are.