Getting FIM right with StarCoder2

[CordMemescape]

Hey hivemind.

I use VSCode and Twinny with Ollama, and for the most part, things work okay. However, I have issues with FIM where sometimes just the right code completion is provided, or just the right comment is made, but most of the time I get a bunch of extra stuff appended to the complete. Generally this is in the form of <file_sep><fim_prefix><fim_suffix>t. appended to the sensible part of the completion.

I use deepseek-coder:6.7b-instruct for chat and starcoder2-3b-q4_0 for fim. My provider setup is as follows:

Label: DeepSeek 6.7B Chat
    Type: Chat
    Provider: ollama
    Protocol: http
    Model name: deepseek-coder:6.7b-instruct
    Host: 0.0.0.0
    Port: 11434
    API path: /v1/chat/completions
    API key:

Label: StarCoder2 3B FIM
    Type: FIM
    Fim template: starcoder
    Provider: ollama
    Protocol: http
    Model name: starcoder2:3b-q4_0
    Host: 0.0.0.0
    Port: 11434
    API path: /api/generate
    API key:

The Twinny config is default for everything except:
    Twinny: Debounce Wait: 900
    Twinny: File Context Enabled: True
    Twinny: Completion Cache Enabled: True
    Twinny: Keep Alive: 15m

The templates are all default.

Any thoughts as to what I need to tweak/fix in order to ensure that the completion are just completions and don't contain all the extraneous stuff?

TIA.

[rjmacarthy]

Hello. Afaik Starcoder2 messed up a little the FIM in training so this is a model flaw which doesn't stop at the right place. I've also noticed it and unfortunately the added prefix after the completion is often too random to strip out.

[CordMemescape]

Aw hell, that's rather a pain - other than the added prefix, Starcoder2 works pretty well. I like it because I can run it locally without hammering resources and it's pretty responsive.

Any thoughts or suggestions for a lightweight replacement?

Also, thanks for the reply, much appreciated!

[rjmacarthy]

I like codellama-code or deepseek base models for fim and codestrall for chat.

[CordMemescape]

I'll give them a try and see how performant they are. Any suggestion on the recommended model sizes?

[rjmacarthy]

You could try 7b for codellama-code, maybe 3b Starcoder1 and 6b deepseek-coder base models only for fim and codestrall 12b.

[CordMemescape]

Ta, I'll give them a go!