feat(server): Add OpenAI /v1/models/{model_id} endpoint - #108
Conversation
|
Tried on 2 x RTX 6000 Ada (sm_89) serving Qwen3.8-Flash-Next (RadixArk NVFP4) at TP=2, offload backend, fp8 KV pool of 8 x 262,144 tokens, merged onto my deploy branch (main af71ba4 + #385/#386/#389/#392/#354 and ten other open PRs), tests run on the box, then put in production.
|
|
Hey @gdevenyi, I'm not super familiar with all the different endpoints with OpenAI provide - but I asked AI to provide a summary of the current implementation. For my specific use case - e.g. agentic coding, this was all that was missing to get it working.
With this PR locally on Windows, I can provide proof that the server now replies to requests.
I can also provide proof that Visual Studio Preview (Insiders [12113.136]) allows you to specify the locally running FreeToken instance and have it apply changes to files (similar to claude). You simply have to click "Manage models", "Add model provider", pick "Open AI" and set the BaseURL to your running instance of FreeToken and then enter the model id exactly as it is displayed at: http://127.0.0.1:1919/v1/models with the correct "id".
|



In Microsoft Visual Studio, you can use local AI endpoints. When you point it at the OpenAI compatible API, it makes a request to:
GET http://127.0.0.1:1919/v1/models
and returns:
Afterwards, Visual Studio then makes a follow up request to:
GET http://127.0.0.1:1919/v1/models/Qwen3.6-35B-A3B-NVFP4 and currently gets a 404, even though the model really is loaded. This then prevents FreeToken server from being used as Visual Studio assumes the OpenAPI server and/or model name is invalid.
This is a small OpenAI-compatibility gap. The OpenAI API has a “retrieve model” operation corresponding to GET /v1/models/{model}, and some clients evidently use it to validate a manually specified model name.
This change leaves the existing /v1/models endpoint unchanged and only adds the missing OpenAI-compatible retrieve-model operation. The existing route currently constructs its ModelCard using the same context-length and reasoning-effort metadata shown above.