Feature request: add support for additional models

[dereklance]

It would be nice if we could support models like llama-3.2-1b-instruct. Or at least provide some docs on how to add it ourselves. It looks like we can compile from source using the scripts/build-runtime.sh and add additional supported models, but I can't seem to figure out which mlc-llm tag to check out to make the script work. when i use v0.19.0 the script seems to miss the TVM FFI headers. thanks for this library though, it works great with the models it supports out of the box!

[grabbou]

Will add smaller model, I don’t think there’s much runtime overhead. The compilation process with MLC isn’t the easiest, my goal would be to provide simple npx command that works out of the box.

Let’s keep it open and I’ll follow up next week.

[dereklance]

Will add smaller model, I don’t think there’s much runtime overhead. The compilation process with MLC isn’t the easiest, my goal would be to provide simple npx command that works out of the box.

Let’s keep it open and I’ll follow up next week.

thanks for the quick response, and npx command would be great. in the meantime i could probably get things working, but just would need to know whether you guys are using mlc-llm v0.19.0 tag or some other version?

edit: never mind, i got it working

[grabbou]

Since you got it running, I would be happy to take some notes / inspiration from you. I tried to make some notes in the bash script for future self, hope that worked for you too!