Support browser-based models

[kentcdodds]

Any chance we could get support for something like this? https://www.npmjs.com/package/@mlc-ai/web-llm (check this demo: https://llm-tool-calling.nico.dev/)

That way people don't have to worry about configuring a whole LLM just to test things out.

[fredericbarthelet]

Hi @kentcdodds and thanks for proposing the use of in-browser LLM support to avoid having to bring your own key and rely on an external LLM provider.

We plan on supporting more models than Sonnet and Haiku as described in #2. Web LLM could be the default option as it requires no further configuration. Would you be willing to work on a pull request to add this option once we rework the sidebar Model section?

I have only 2 concerns ATM that'll need to be tested if this is added:

• one would be related to package size - I guess adding web-llm as a dependency will dramatically increase the whole project size. I'd like the package to still be started with npx @alpic-ai/grizzly in less than 10s under normal conditions
• the other would be related to model responses in the various scenario - model configured is used in 3 places: playground, safety check, tool confusion benchmark generation. I expect the latest to behave less than optimally with light models. To be checked as well.

[kentcdodds]

I haven't used https://www.npmjs.com/package/@mlc-ai/web-llm myself, but my impression is that it downloads models on-demand. At least, the "unpacked" size reported by npm is 13.1MB which sounds big until you compare it to Next.js which is 127MB. Perhaps it could be an optional dependency?