New local model: Gemma 3

No description provided.

While testing Gemma 3 with MLC-LLM, we encountered several issues:

Lack of Multimodal Support: Gemma 3 was originally designed as a multimodal model. However, we found that the current MLC-LLM implementation lacks image processing.
bfloat16 Problem: Vulkan’s support for bfloat16 is still experimental as of version 1.4.311 source. We confirmed that MLC-LLM currently does not implement support for Vulkan bfloat16, meaning additional implementation work is required for proper bfloat16 support.
Result Disparity between mlc-llm & reference(HF transformers): We compared outputs between transformer using bfloat16 and mlc-llm models using q0f32, we observed degraded performance in mlc-llm's run. This discrepancy strongly suggests that there may be a bug in MLC-LLM’s Python model implementation. (further investigation is warranted)
QAT4: Gemma 3 provides a QAT-specific variant. We found that using non-QAT parameters leads to even more severe degradation in model performance.

We plan to address each of these issues. However, this may take the time and effort, these tasks will be excluded from the 0.0.3 milestone.

Navigation Menu