Debugging performance

[pdufour]

Hi! Thanks for building this awesome library, I'm trying to figure out how many tokens / s are generated by this so I can compare performance to other libraries like https://github.com/abetlen/llama-cpp-python. This gives you a debug output like the following:
Output generated in 266.13 seconds (1.50 tokens/s, 398 tokens, context 627) Any way to get a similar output when running the repl command? Thanks!

[philpax]

You can't easily get this with the repl mode because it's optimised for back and forth, but try the --stats option for infer and let me know if that tells you what you need!

[danforbes]

These two examples both demonstrate piping some inference stats to the terminal: inference.rs & vicuna-chat.rs

[pdufour]

Perfect, thanks!