Manual context-length input capped at 2048 for architectures outside the auto-fit validated family list (e.g. deepseek_v4)

[True2456]

Summary

For architectures not in context_fit.py's validated family list (gemma4, qwen3_5/qwen3_6), fit_batched_vlm_context correctly and intentionally declines to auto-fit a context length — it returns None, and get_runtime_load_info then returns {}. That's a reasonable safety choice (avoiding a guessed value that could OOM). But the app UI then falls back to a small fixed default (2048) for the manual context-length control too, even though:

1. The model's own config.json declares a much larger native limit (max_position_embeddings), which is already being read successfully elsewhere in the same fit function.
2. The backend has no actual problem running at a much higher context — lms load <model> --context-length 8192 works correctly and generates normally, proving the limitation is UI-only, not a real backend constraint.

So users with an architecture outside the validated family list are silently capped at 2048 tokens in the GUI, with no way to raise it short of discovering the CLI --context-length override exists.

Repro

1. Load any MLX text model whose architecture isn't gemma4 or qwen3_5/qwen3_6 (e.g. deepseek_v4) via the GUI's model-load dialog.
2. Try to set context length above 2048 in the slider/input — not possible.
3. Compare: lms load <model> --context-length 8192 loads successfully and the model generates normally at that context length (confirmed via lms ps showing CONTEXT 8192, and a real generation through the local /v1/chat/completions API).

Root cause (traced in mlx_engine)

• model_kit/batched_vision/context_fit.py: fit_batched_vlm_context only sets validated_family = True for gemma4 and qwen3_5/qwen3_6. For every other family it still computes a best-effort fit, but returns None ("leaving context unchanged") whenever that best-effort result is <= MIN_FITTED_CONTEXT_TOKENS (4096) — which discards the real, already-available max_position_embeddings value read a few lines earlier in the same function.
• generate.py's get_runtime_load_info then returns {} when effective_context_length is None.
• Somewhere in the app frontend (not in this repo, so unverified from here), an empty/missing context_length in that response results in the manual slider defaulting to and capping at 2048, rather than falling back to the model's own declared max_position_embeddings as the ceiling (distinct from the auto-fit default).

Suggested fix

Two independent asks, either would resolve this:

1. When fit_batched_vlm_context returns None for an unvalidated family, still surface the model's own max_position_embeddings (already read into max_context_length inside the function) as an upper bound for the manual control, separately from the safety-motivated auto-fit default. i.e. "we won't auto-pick a large value for an architecture we haven't validated memory behavior for" and "the user can't manually enter a value the config itself declares as valid" don't need to be the same restriction.
2. Alternatively, on the frontend: when context_length is absent from get_runtime_load_info's response, don't hard-cap the manual input at a small constant — allow arbitrary manual entry (as the CLI already does via --context-length), perhaps with a warning that the value hasn't been auto-validated for this architecture.

Environment

• LM Studio 0.4.20 (build 1), macOS
• mlx-llm backend (nax-advsimd), engine build app-mlx-generate-mac26-arm64@33
• Model: local MLX conversion of deepseek-ai/DeepSeek-V4-Flash (deepseek_v4 architecture), text-only, max_position_embeddings: 1048576

[True2456]

Closing this — my root-cause analysis was wrong, and the real cause is both simpler and not in this repo.

I attributed the 2048 cap to context_fit.py declining to auto-fit for architectures outside the validated family list. That was a guess based on reading the code path, and it does not hold up. The actual cause is that LM Studio's model classifier produces an empty metadata.configJson ({}) for this model, so contextLength falls back to a small default (4096 in model-index-cache.json) and the UI caps accordingly. The architecture is irrelevant.

The real trigger, isolated by bisection: tokenizer_config.json using the HuggingFace AddedToken object form for bos_token/eos_token/pad_token:

"bos_token": {"__type": "AddedToken", "content": "<｜begin▁of▁sentence｜>", "lstrip": false, "normalized": true, "rstrip": false, "single_word": false}

Converting those three to plain strings immediately populates configJson with arch, modelType, quantizationBits, quantizationMode, chatTemplate, and the correct contextLength: 1048576. Nothing else changes.

Minimal repro (no large model needed — I used ~600-byte directories containing a tiny model.safetensors saved with metadata={"format": "mlx"}, a config.json, and a tokenizer_config.json): two otherwise-identical model directories, one with the object form and one with plain strings. The object-form one indexes with configJson: {} and contextLength: 4096; the string one indexes correctly. The AddedToken object form is standard HF output, so this affects any checkpoint that carries it.

I independently ruled out, each with its own controlled test: config.json size (a 140KB config indexes fine), the deepseek_v4 architecture itself (indexes fine), the presence of vision_config, and a 102GB single-file model.safetensors.

Since the failure is in the classifier that writes model-index-cache.json rather than anything in mlx_engine, this is the wrong repo for it. Apologies for the noise, and for the incorrect analysis in the original report.

One caveat worth recording for anyone who finds this: converting to plain strings fixes the metadata but then causes LM Studio to stop delegating prompt templating to mlx_vlm and fail on every request with TypeError: Cannot read properties of undefined (reading 'template') at temp_createMlxPredictionArgs. So it is not a usable workaround as-is.

#357 is unaffected by any of this and still stands on its own.