My Local Coding Model Recommendations (After Testing 50+ Models)

[charly1r]

UPDATE: I will be updating this section as often as I can so you don't have to read the whole thread to find the model results, I will be using humaneval and LCB for the evaluations, I am not a pro if a result does not look good for you please bench it and share your results. My hardware 9070XT + 3060TI(8GB) = 24GB VRAM, 32 RAM, 9800X3D CPU. All models can be downloaded using LM Studio, just paste the model name and download the Quant that fits your hardware, I added the links to huggingface as well.

Ok guys I will stop testing for now unless something or someone finds a good model to test, for now 27B has the top intelligence and if you want speed MOE.

🏆 Visual Leaderboard

🏆 Rank	Model	Quant	TPS	LCB+HEval	Notes
🥇	DavidAU 711 Fable Fusion Heretic NEO MAX MTP	Q4_K_S	41.91	84.47	🏆 Current leader. Tested with 80k context. Best overall benchmark. (HF)
🥈	DavidAU 711 Fable Fusion Heretic NEO MAX MTP	Q4_K_M	42.93	84.35	Previous leader. Fastest of the top 3. (HF)
🥉	DavidAU 711 Fable Fusion Heretic NEO MAX	Q4_K_M	24.65	83.81	Original non-MTP release. Slow but smart. (HF)
4	Architect Polaris2 F451 Heretic	Q4_K_M	26.88	81.16	Heretic version of DavidAU 711. (HF)
5	Tess-4-27B	Q4_K_M	26.94	80.94	Excellent all-around coding model. (HF)
6	Kwaipilot_KAT-Coder-V2.5-Dev-APEX-MTP-I-Compact	APEX	117.59	74.93	Fastest model tested so far. (HF)
7	Genesis Hermes V3 APEX Compact	APEX	100.30	70.57	Excellent speed with solid coding performance. (HF)
8	REAM-192 Heretic APEX IBalanced	Q5_K_M	81.38	69.73	Balanced Q5 model. (HF)
9	Defiant 9B Heretic MAX MTP	Q8_0	90.79	69.56	Strongest 9B model tested. (HF)
10	Genesis Hermes V5 APEX Compact	APEX	100.45	65.08	High-throughput compact model. (HF)
11	Genesis Hermes V4 MTP	MTP	103.14	64.78	High-speed MTP variant. (HF)
12	Qwen3.6-27B-A3B Coder	—	105.06	64.49	Official coder baseline. (HF)
13	Genesis Hermes V6	—	105.08	59.70	Latest Hermes release. (HF)

🏅 Top 3

🥇	🥈	🥉
DavidAU 711 MTP Q4_K_S	DavidAU 711 MTP Q4_K_M	DavidAU 711 Q4_K_M
84.47	84.35	83.81
🚀 41.91 TPS	🚀 42.93 TPS	🚀 24.65 TPS

📊 Performance Snapshot

LCB+HEval

84.47 ████████████████████████████████████████  🥇 DavidAU 711 Q4_K_S MTP
84.35 ██████████████████████████████████████▉  🥈 DavidAU 711 Q4_K_M MTP
83.81 █████████████████████████████████████▋   🥉 DavidAU 711 Q4_K_M
81.16 ██████████████████████████████████       Architect Polaris2
80.94 █████████████████████████████████        Tess-4-27B
74.93 ██████████████████████████████           Kwaipilot KAT-Coder V2.5 Dev
70.57 ███████████████████████████              Genesis Hermes V3
69.73 ██████████████████████████▉             REAM-192
69.56 ██████████████████████████▊             Defiant 9B
65.08 ████████████████████████                 Genesis Hermes V5
64.78 ███████████████████████▉                Genesis Hermes V4
64.49 ███████████████████████▊                Qwen3.6-27B-A3B Coder
59.70 █████████████████████                   Genesis Hermes V6

[charly1r]

For anyone who comes across this post, I wanted to share part of my AI journey.

I primarily run local models on consumer hardware (RX 9070 XT 16GB + 3060ti 8GB) and spend a lot of time testing coding agents and models.

One thing I learned quickly is that for local agents, context size becomes the biggest bottleneck, not raw benchmark scores.

I eventually settled on Little-Coder because it is:

• Batteries included
• Lightweight
• Easy to run locally
• Has strong anti-loop protection (my favorite feature, which I jokingly call "Loopinus Maximus")

Many coding agents get stuck in loops and require constant babysitting. Little-Coder has been much more resilient in that regard.

If you have any model reccomendation please post, all of us are always looking for the holy grail.

Important: These are not scientific benchmarks. I turned the process into a blind tournament and used ChatGPT to generate prompts, evaluate submissions, and keep scoring consistent.

---

Models Tested on 90K Context

Over 50 coding models were evaluated.

Finalists:

🥇 Qwen3.6-35B-A3B-REAM-192-heretic-APEX-IQuality-Q5_K_M

🥈 Qwen3.6-28B-A3B-REAP20-Q4_K_M

🥉 Qwen3.6-35B-A3B-Uncensored-Claude-Genesis-V3-APEX-Compact

4️⃣ deepreinforce-ai_Ornith-1.0-35B-IQ4_XS

❌ Qwen-3.5-28B-A3B-REAP.i1-Q4_K_S

---

Benchmark Series

The finalists competed in five blind engineering challenges:

Benchmark 1 – Deluxe Snake

Built a premium arcade-quality Snake game with progression, effects, achievements, menus, save systems, and multiple game modes.

Tested: Creativity, product thinking, game architecture, and gameplay quality.

Winner: 🥇 REAM-192

---

Benchmark 2 – Personal Task Manager

Built a fully functional CLI task manager with persistence and recovery.

Tested: Software engineering fundamentals, architecture, validation, and reliability.

Winner: 🥇 REAM-192

---

Benchmark 3 – Mini Version Control System

Built a simplified VCS with snapshots, history, and state management.

Tested: State modeling, abstraction, architecture, and complexity handling.

Winner: 🥇 REAM-192

---

Benchmark 4 – Log Analyzer

Built a log parsing and analysis system.

Tested: Parsing, validation, reporting, and verification culture.

Winner: 🥇 REAM-192 (close finish)

---

Benchmark 5 – Loopinus Maximus

The trap benchmark.

Models had to:

• Build an application
• List assumptions
• Challenge those assumptions
• Find failures
• Diagnose failures
• Repair failures
• Explain remaining uncertainty

The goal was not to prove correctness.

The goal was to determine how well the model could discover when it was wrong.

Tested: Root-cause analysis, falsification, debugging discipline, and intellectual honesty.

Winner: 🥇 REAM-192 (photo finish)

---

Final Assessment

🥇 REAM-192

The strongest overall engineering model.

Strengths:

• Best engineering judgment
• Strong architecture
• Excellent debugging discipline
• Consistent across all benchmarks

---

🥈 REAP20

The most balanced challenger.

Strengths:

• Strong verification culture
• Excellent risk analysis
• Consistently near the top

---

🥉 Genesis

The biggest surprise.

Strengths:

• Excellent testing discipline
• Strong falsification mindset
• Consistently overperformed expectations

---

4️⃣ Ornith

Most improved model.

Strengths:

• Creative solutions
• Strong reasoning
• Improved significantly throughout the tournament

---

My Recommendations

If I could keep only three coding models installed today:

1. 🥇 REAM-192
2. 🥈 REAP20
3. 🥉 Genesis

After five engineering benchmarks, REAM-192 remained the most complete coding model I tested.

@Qm-jmz

These are the huggingface URL's.

https://huggingface.co/keithnull/Qwen3.6-35B-A3B-REAM-192-heretic-APEX-GGUF

https://huggingface.co/DjeDjeB/Qwen3.6-28B-REAP20-A3B-GGUF

https://huggingface.co/LuffyTheFox/Qwen3.6-35B-A3B-Uncensored-Claude-Genesis-GGUF

Let us know how do they treat you.

[Qm-jmz]

Can you share your configuration, especially the context settings? Thanks much!

[charly1r]

Sure, don't take my word for it, but I've been getting really good results with the configuration below.

I haven't completely eliminated the SWA bug (forcing full prompt re-processing due to lack of cache data) that is currently affecting the newer Qwen models, but over the last two days I've only seen a single occurrence during heavy testing. Hopefully the llama.cpp team will get to the bottom of it soon.

Related discussion:

ggml-org/llama.cpp#22746 (comment)

My Hardware

• AMD RX 9070 XT 16GB
• NVIDIA RTX 3060 Ti 8GB
• 32GB System RAM
• 24GB Total VRAM

Obviously adjust the configuration to fit your own hardware.

Notes

• This model has vision support, so I load the mmproj, but it isn't required if you don't need vision.
• I use --tensor-split 60,40.
• It is not the absolute fastest split I tested, but I prefer keeping some VRAM available on the primary GPU for Windows, browsers, terminals, VSCode, etc.
• If you remove -sm layer, llama.cpp does a surprisingly good job distributing layers automatically, but after testing several configurations I still preferred 60/40 for daily use.

I spent a few days benchmarking the same workload across different tensor splits and recorded the results below.

Tensor Split Comparison

Split	RX 9070 XT VRAM	RX Shared	3060 Ti VRAM	3060 Shared	Prompt TPS	Initial TPS	Sustained TPS	Notes
60/40	13.0 GB	1.7 GB	7.2 GB	0.0 GB	~2430	~85.8	77-80	Best overall workstation balance
65/35	13.7 GB	2.1 GB	6.9 GB	0.1 GB	~2547	~86.4	75-77	Fastest startup
Auto	13.2 GB	2.9 GB	6.5 GB	0.1 GB	~2440	~86.3	76-78	Auto selected roughly 67/33
70/30	14.0 GB	2.8 GB	6.5 GB	0.1 GB	~2469	~80.5	78-80	Best sustained inference speed
75/25	13.4 GB	3.8 GB	5.4 GB	0.1 GB	~1887	~75.5	67-69	Worst performer in my testing

My Takeaway

These numbers are specific to my setup:

• Qwen3.6-35B-A3B-REAM-192 (Q5_K_M)
• llama.cpp Vulkan
• 90k context
• Long coding sessions with Little-Coder

A few things stood out:

• 75/25 consistently performed the worst.
• 65/35 produced the highest startup TPS, but performance tapered off as context grew.
• Auto allocation landed very close to 67/33 and behaved similarly to 65/35.
• 70/30 produced the best sustained throughput during long sessions.
• 60/40 even though a little slower than 70/30 left more breathing room on the primary GPU.

In the end I settled on 60/40 for now I will keep an eye how it performs for the next week or so.

Technically 70/30 is a little faster, but this machine is also my daily Windows workstation. While the model is running I still have browsers, terminals, file explorers, logs, screenshots, and other tools open.

For me, giving up a small amount of performance in exchange for more VRAM headroom and a more responsive desktop was worth it.

Your mileage may vary, but hopefully the numbers help someone else avoid spending a weekend testing every tensor split like I did 😄.

llama-server.exe `
  -m "C:/Games/Projects/models/keithnull/Qwen3.6-35B-A3B-REAM-192-heretic-APEX-GGUF/Qwen3.6-35B-A3B-REAM-192-heretic-APEX-IQuality-Q5_K_M.gguf" `
  --mmproj "C:\Games\Projects\models\keithnull\Qwen3.6-35B-A3B-REAM-192-heretic-APEX-GGUF\mmproj-REAP-26B-F16.gguf" `
  -ngl 99 `
  -np 1 `
  --ctx-size 90000 `
  --no-mmap `
  --kv-unified `
  --parallel 1 `
  --flash-attn on `
  --no-warmup `
  --temp 0.6 `
  --top-p 0.95 `
  --top-k 20 `
  --min-p 0.05 `
  --repeat-penalty 1.05 `
  --presence-penalty 1.5 `
  -t 7 `
  -ctk q4_0 `
  -ctv q4_0 `
  -b 2048 `
  -ub 512 `
  --cache-ram 8192 `
  --ctx-checkpoints 8 `
  --cache-prompt `
  --no-context-shift `
  --checkpoint-min-step 1024 `
  -sm layer `
  -ts 60,40 `
  --host 127.0.0.1 `
  --port 8080 2>&1 | Tee-Object -FilePath ("C:\1AIAPPS\llama.cpp\logs\llama-server-" + (Get-Date -Format "yyyy-MM-dd_HH-mm-ss") + ".log")

[Qm-jmz]

It was a great help. My hardware is a 4090 with 16GB of VRAM and 32GB of RAM. Although I haven't actually tried it yet, maybe I should try the 64K context and keep the rest of the configuration as default?

[charly1r]

Well the 4090 has 24GB VRAM check and make sure what you have.
If have 24GB VRAM the REAM model is pretty good otherwise you will need a smaller quant like Q3.
64K is fine and fits better in 16GB VRAM. I just do long coding prompts for testing.

[Qm-jmz]

4090 laptop. And you’re right. I’ll probably need to use Q3 quant. At that point, the rankings might change again. But your ranking system could still help me save a lot of time and effort!

[charly1r]

Then remove this:
-sm layer -ts 60,40

That is for dual GPUs, just look if the models I posted on the first message have Q3 and are:
Below 16gb VRAM
that leave space for the cache and a little room for the OS.

[itayinbarr]

Carlos — this is an incredible writeup, thank you. (And thanks @Qm-jmz for the config deep-dive; that tensor-split table is genuinely useful reference for anyone on a dual-GPU box.)

On your methodology question — "is ChatGPT-as-judge an OK empirical approach, or should I use real benchmarking software?": it's a great smoke test, and honestly a fun one, but I'd treat it as directional rather than decisive. Two things worth knowing about LLM-as-judge:

• Judge bias — verbosity/length bias, position bias, and self-preference (a judge tends to favor outputs stylistically like its own) mean a single judge on freeform prompts can rank confidently but not always reproducibly.
• Variance — n=1 per task is noisy with local quants; the same model can win or lose a round on a reroll.

So for ranking models I'd lean on the harness that ships in benchmarks/ (Aider Polyglot + the terminal-bench profile) — deterministic pass/fail on real edits, reproducible, and directly comparable to the paper's numbers. Best of both worlds: keep your creative prompts (deluxe snake is a great stress test that pass/fail harnesses don't capture — product sense, "is it fun") as a qualitative layer on top of a pass/fail harness doing the quantitative ranking. Your tournament is a legitimately useful signal; I just wouldn't retire a model on it alone.

On the models themselves — REAM-192, REAP20, Genesis: I'll pull them and run them through the little-coder harness on my own hardware this week before I add anything to models.json (I like to verify scaffold-fit myself first — cross-model rankings shift a lot with quant + context on real coding tasks). I'll report numbers back here. The 27B-MTP-pi-reasoning one you flagged as weak — noted, I'll skip it.

Really appreciate the legwork. This is exactly the kind of community testing that makes the model picks better.

[charly1r]

@itayinbarr I take back my earlier comment about Qwen 3.6 27B. I judged it too quickly.

On my split-GPU setup, I get around 30 TPS with Qwen 3.6 27B compared to roughly 80 TPS with Qwen 3.5 35B A3B. However, there's an important tradeoff: the 27B dense model consistently produces better code.

For the last 3 days, I've been running the same prompts through both models, and the 27B outputs have generally been superior. The downside is lower TPS, which means more waiting time. The upside is cleaner code that requires fewer fixes and less follow-up work.

I also noticed that GPT tends to have preferences toward certain models, so I ran a blind tournament. For each round, I only provided the generated code and labeled the submissions something like "A-Snake" and "B-Snake." In the next round, I shuffled the labels so A and B represented different models. After everything was ranked, I revealed which model corresponded to each entry. GPT's response was basically: "Now I see why you did a blind tournament lol."

That said, I agree that something like an Aider+Bench profile would be a more rigorous way to compare them.

So, REAM 192 is still a great model. It's fast and gets you about 90% of the way there. But if you're chasing that extra 10% in code quality, the 27B is the better choice.

@Qm-jmz

The 27B currently comes in two main variants: Standard and MTP.

The MTP version requires roughly 2 GB of additional VRAM and, at least in my testing, provides only about a 15% speed increase. If you're running a 16 GB card, I'd recommend the non-MTP version since it fits much more comfortably in VRAM.

For 24 GB cards, I'm currently testing:
https://huggingface.co/kdqemre/Qwen3.6-27B-Omnimerge-v4-MTP-Q4_K_L-GGUF
Even on 24 GB, it's a tight fit.

For 16 GB VRAM, you'll probably need to drop down to a Q3_K_S quant. There are many variants available, but I'd start by comparing a few 27B builds against A3B using your own prompts. Sometimes the finetune matters more than the base model, and in some cases Unsloth variants may produce better results.

Some variants worth testing:

• https://huggingface.co/unsloth/Qwen3.6-27B-GGUF
• https://huggingface.co/mradermacher/Qwen3.6-27B-Heretic2-Uncensored-Finetune-Thinking-GGUF
• https://huggingface.co/mradermacher/Qwen3.6-27B-Heretic2-Uncensored-Finetune-Thinking-i1-GGUF
• https://huggingface.co/HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive
• https://huggingface.co/DavidAU/Qwen3.6-27B-Heretic-Uncensored-FINETUNE-NEO-CODE-Di-IMatrix-MAX-GGUF

One final note: make sure you're downloading the non-MTP versions when that's what you intend. Some model titles can be misleading, so double-check the actual model details before downloading.

[itayinbarr]

Love this update, Carlos — and the blind, shuffled-label tournament is a real step up; that's exactly the fix for the self-preference/position bias I was flagging, and GPT's "now I see why you did a blind tournament lol" is chef's kiss. That's a genuinely credible qualitative signal now.

The dense-27B vs A3B observation is the interesting one: slower (~30 vs ~80 TPS on your split) but cleaner code that needs fewer follow-up fixes. That tradeoff — raw throughput vs. fewer correction round-trips — is precisely the kind of thing a pass/fail harness surfaces well (total wall-clock-to-green, not just tok/s), so it's a great candidate to put through Aider Polyglot / terminal-bench. I've added the 27B non-MTP (unsloth + the Heretic2 finetunes you listed) to my test batch alongside REAM-192 / REAP20 / Genesis, and I'll report per-model pass rates and time-to-solution so the speed/quality tradeoff is visible rather than hidden.

Good call on the MTP note too (≈2 GB extra VRAM for ~15% speed) — I'll flag non-MTP as the safer default for 16 GB cards when I write up results. Thanks for keeping at this; the model picks are getting real data behind them because of it.

[Qm-jmz]

Throughput and resource usage tests of Qwen3.6-35B-A3B-REAM-192-heretic-APEX-ICompact-Q3_K_L on my computer

Test Date: 2026-07-04

Environment

• GPU: NVIDIA GeForce RTX 4090 Laptop GPU, 16 GB VRAM
• CPU: Intel Core i9-14900HX, 24 cores / 32 threads
• Memory: 32 GB
• llama.cpp: b9844 (6c5de1cc8)
• Model: Qwen3.6-35B-A3B-REAM-192-heretic-APEX-ICompact-Q3_K_L.gguf (12.453 GiB)

This test was designed and executed entirely by Codex.The test scripts are as follows. Each group is tested using actual /completion requests. For short tests, approximately 8K tokens are used as input. For long-context tests, 60K and 84K tokens are used respectively.

"""Benchmark Qwen3.6 Q3_K_L llama-server configurations on this workstation.

The benchmark measures real HTTP inference rather than treating a listening
socket as success.  It records load time, GPU memory, cold prompt throughput,
generation throughput, and prompt-cache reuse.
"""

from __future__ import annotations

import csv
import argparse
import json
import pathlib
import subprocess
import time
import urllib.error
import urllib.request


REPO = pathlib.Path(r"xxx")
SERVER = REPO / "llama" / "llama-server.exe"
MODEL = (
    REPO
    / "models"
    / "Qwen3.6-35B-A3B-REAM-192-heretic-APEX-GGUF"
    / "Qwen3.6-35B-A3B-REAM-192-heretic-APEX-ICompact-Q3_K_L.gguf"
)
LOG_DIR = REPO / "test-logs"
PORT = 8888
BASE_URL = f"http://127.0.0.1:{PORT}"
DEFAULT_PROMPT_TOKENS = 8192


STANDARD_CONFIGS = [
    {"name": "gpu_q4_32k", "ctx": 32768, "cache": "q4_0", "cpu_moe": False},
    {"name": "gpu_q4_64k", "ctx": 65536, "cache": "q4_0", "cpu_moe": False},
    {"name": "gpu_q4_90k", "ctx": 90000, "cache": "q4_0", "cpu_moe": False},
    {"name": "cpu_moe_q4_64k", "ctx": 65536, "cache": "q4_0", "cpu_moe": True},
    {"name": "cpu_moe_q4_90k", "ctx": 90000, "cache": "q4_0", "cpu_moe": True},
    {"name": "gpu_q8_32k", "ctx": 32768, "cache": "q8_0", "cpu_moe": False},
]

STRESS_CONFIGS = [
    {
        "name": "gpu_q4_64k_stress",
        "ctx": 65536,
        "cache": "q4_0",
        "cpu_moe": False,
        "prompt_tokens": 60000,
        "predict_tokens": 64,
    },
    {
        "name": "gpu_q4_90k_stress",
        "ctx": 90000,
        "cache": "q4_0",
        "cpu_moe": False,
        "prompt_tokens": 84000,
        "predict_tokens": 64,
    },
    {
        "name": "gpu_q8_64k_stress",
        "ctx": 65536,
        "cache": "q8_0",
        "cpu_moe": False,
        "prompt_tokens": 60000,
        "predict_tokens": 64,
    },
    {
        "name": "gpu_q8_90k_stress",
        "ctx": 90000,
        "cache": "q8_0",
        "cpu_moe": False,
        "prompt_tokens": 84000,
        "predict_tokens": 64,
    },
]


def post_json(path: str, payload: dict, timeout: int = 600) -> dict:
    data = json.dumps(payload).encode("utf-8")
    request = urllib.request.Request(
        BASE_URL + path,
        data=data,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(request, timeout=timeout) as response:
        return json.loads(response.read().decode("utf-8"))


def wait_until_ready(proc: subprocess.Popen, timeout: int = 180) -> tuple[bool, float, str]:
    started = time.perf_counter()
    last_error = ""
    while time.perf_counter() - started < timeout:
        if proc.poll() is not None:
            return False, time.perf_counter() - started, f"server exited with {proc.returncode}"
        try:
            with urllib.request.urlopen(BASE_URL + "/health", timeout=2) as response:
                if response.status == 200:
                    return True, time.perf_counter() - started, ""
        except (urllib.error.URLError, TimeoutError, ConnectionError) as exc:
            last_error = str(exc)
        time.sleep(0.5)
    return False, time.perf_counter() - started, f"health timeout: {last_error}"


def gpu_memory() -> tuple[float, float]:
    completed = subprocess.run(
        [
            "nvidia-smi",
            "--query-gpu=memory.used,memory.free",
            "--format=csv,noheader,nounits",
        ],
        capture_output=True,
        text=True,
        check=False,
    )
    if completed.returncode != 0:
        return 0.0, 0.0
    used, free = completed.stdout.strip().split(",")[:2]
    return float(used.strip()), float(free.strip())


def build_args(config: dict) -> list[str]:
    args = [
        str(SERVER),
        "-m", str(MODEL),
        "--host", "127.0.0.1",
        "--port", str(PORT),
        "-ngl", "99",
        "-np", "1",
        "-c", str(config["ctx"]),
        "--no-mmap",
        "--kv-unified",
        "--flash-attn", "on",
        "--no-warmup",
        "-ctk", config["cache"],
        "-ctv", config["cache"],
        "-b", "2048",
        "-ub", "512",
        "--cache-ram", "8192",
        "--ctx-checkpoints", "8",
        "--cache-prompt",
        "--no-context-shift",
        "--checkpoint-min-step", "1024",
        "--metrics",
        "--jinja",
    ]
    if config["cpu_moe"]:
        args.extend(["--n-cpu-moe", "999"])
    return args


def stop_process(proc: subprocess.Popen) -> None:
    if proc.poll() is not None:
        return
    proc.terminate()
    try:
        proc.wait(timeout=15)
    except subprocess.TimeoutExpired:
        proc.kill()
        proc.wait(timeout=10)


def token_count(text: str) -> int:
    response = post_json("/tokenize", {"content": text, "add_special": False}, timeout=120)
    return len(response.get("tokens", []))


def make_prompt(target_tokens: int) -> tuple[str, int]:
    line = (
        "\n// Existing module: validate inputs, preserve invariants, handle failures, "
        "and keep tests deterministic."
    )
    repeats = 900
    for _ in range(4):
        text = line * repeats
        count = token_count(text)
        if not count:
            raise RuntimeError("/tokenize returned no tokens")
        repeats = max(1, int(repeats * target_tokens / count))
    prompt = line * repeats
    return prompt, token_count(prompt)


def completion(prompt: str, n_predict: int) -> tuple[dict, float]:
    payload = {
        "prompt": prompt,
        "n_predict": n_predict,
        "temperature": 0.0,
        "seed": 42,
        "cache_prompt": True,
    }
    started = time.perf_counter()
    response = post_json("/completion", payload)
    return response, time.perf_counter() - started


def timing_fields(response: dict, wall_seconds: float, prefix: str) -> dict:
    timings = response.get("timings", {})
    return {
        f"{prefix}_wall_seconds": round(wall_seconds, 3),
        f"{prefix}_prompt_tokens": timings.get("prompt_n", 0),
        f"{prefix}_prompt_tps": round(float(timings.get("prompt_per_second", 0)), 2),
        f"{prefix}_generated_tokens": timings.get("predicted_n", 0),
        f"{prefix}_generation_tps": round(float(timings.get("predicted_per_second", 0)), 2),
    }


def run_config(config: dict) -> dict:
    name = config["name"]
    log_path = LOG_DIR / f"benchmark-{name}.log"
    baseline_used, _ = gpu_memory()
    result = {
        **config,
        "loaded": False,
        "load_seconds": 0.0,
        "gpu_baseline_mib": baseline_used,
        "gpu_loaded_mib": 0.0,
        "gpu_delta_mib": 0.0,
        "gpu_peak_mib": 0.0,
        "input_tokens": 0,
        "error": "",
        "log": str(log_path),
    }
    print(f"\n[TEST] {name}: ctx={config['ctx']} cache={config['cache']} cpu_moe={config['cpu_moe']}", flush=True)
    with log_path.open("w", encoding="utf-8") as log_file:
        proc = subprocess.Popen(
            build_args(config),
            cwd=REPO,
            stdout=log_file,
            stderr=subprocess.STDOUT,
        )
        try:
            ready, load_seconds, error = wait_until_ready(proc)
            result["load_seconds"] = round(load_seconds, 2)
            if not ready:
                result["error"] = error
                print(f"  FAILED: {error}", flush=True)
                return result

            result["loaded"] = True
            loaded_used, _ = gpu_memory()
            result["gpu_loaded_mib"] = loaded_used
            result["gpu_delta_mib"] = max(0.0, loaded_used - baseline_used)
            result["gpu_peak_mib"] = loaded_used
            print(
                f"  loaded={load_seconds:.1f}s gpu={loaded_used:.0f} MiB "
                f"(delta {result['gpu_delta_mib']:.0f})",
                flush=True,
            )

            completion("Return exactly: warmup", 8)
            prompt, count = make_prompt(config.get("prompt_tokens", DEFAULT_PROMPT_TOKENS))
            result["input_tokens"] = count
            predict_tokens = config.get("predict_tokens", 128)

            cold, cold_wall = completion(
                prompt + "\nTask: identify three concrete correctness risks in this module.",
                predict_tokens,
            )
            result.update(timing_fields(cold, cold_wall, "cold"))
            after_cold, _ = gpu_memory()
            result["gpu_peak_mib"] = max(result["gpu_peak_mib"], after_cold)

            cached, cached_wall = completion(
                prompt
                + "\nTask: identify three concrete correctness risks in this module."
                + "\nAdditional constraint: rank them by severity.",
                predict_tokens,
            )
            result.update(timing_fields(cached, cached_wall, "cached"))
            after_cached, _ = gpu_memory()
            result["gpu_peak_mib"] = max(result["gpu_peak_mib"], after_cached)
            print(
                f"  cold pp={result['cold_prompt_tps']:.1f} t/s, "
                f"tg={result['cold_generation_tps']:.1f} t/s; "
                f"cached wall={result['cached_wall_seconds']:.2f}s; "
                f"peak_gpu={result['gpu_peak_mib']:.0f} MiB",
                flush=True,
            )
        except Exception as exc:  # keep the matrix running after OOM/API failures
            result["error"] = f"{type(exc).__name__}: {exc}"
            print(f"  ERROR: {result['error']}", flush=True)
        finally:
            stop_process(proc)
            time.sleep(3)
    return result


def write_results(results: list[dict], suite: str, output: str | None = None) -> None:
    stem = output or ("benchmark-summary" if suite == "standard" else "benchmark-stress-summary")
    json_path = LOG_DIR / f"{stem}.json"
    csv_path = LOG_DIR / f"{stem}.csv"
    json_path.write_text(json.dumps(results, indent=2), encoding="utf-8")
    fieldnames = sorted({key for result in results for key in result})
    with csv_path.open("w", newline="", encoding="utf-8-sig") as csv_file:
        writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
        writer.writeheader()
        writer.writerows(results)
    print(f"\nResults: {json_path}\n         {csv_path}", flush=True)


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--suite", choices=("standard", "stress"), default="standard")
    parser.add_argument("--name", action="append", help="run only the named configuration")
    parser.add_argument("--output", help="result filename stem under test-logs")
    args = parser.parse_args()
    LOG_DIR.mkdir(exist_ok=True)
    if not SERVER.exists() or not MODEL.exists():
        raise SystemExit("llama-server or Q3_K_L model is missing")
    configs = STANDARD_CONFIGS if args.suite == "standard" else STRESS_CONFIGS
    if args.name:
        requested = set(args.name)
        configs = [config for config in configs if config["name"] in requested]
        missing = requested - {config["name"] for config in configs}
        if missing:
            raise SystemExit(f"unknown configuration(s): {', '.join(sorted(missing))}")
    results = []
    for config in configs:
        result = run_config(config)
        results.append(result)
        write_results(results, args.suite, args.output)


if __name__ == "__main__":
    main()

Results

Configuration	Input	Peak Video Memory	Prompt Tokens/s	Generated Tokens/s	Conclusion
Full GPU, Q4 KV, 32K	8K	13,134 MiB	1,489	67.8	Fast, but with limited context
Full GPU, Q4 KV, 64K	8K	13,410 MiB	1,515	67.8	Recommended by default
Full GPU, Q4 KV, 90K	8K	13,618 MiB	1,498	67.4	Available for long sessions
CPU-MoE, Q4 KV, 64K	8K	2,432 MiB	642	25.1	Used solely for video memory fallback
CPU-MoE, Q4 KV, 90K	8K	2,620 MiB	642	26.2	Used solely for video memory fallback
Full GPU, Q8 KV, 32K	8K	13,294 MiB	1,525	67.1	No throughput gain
Full GPU, Q4 KV, 64K	60K	13,410 MiB	1,170	37.9	Stress test passed
Full GPU, Q4 KV, 90K	84K	13,618 MiB	1,021	30.7	Stress test passed
Full GPU, Q8 KV, 64K	60K	13,730 MiB	1,179	37.3	Occupies approximately 320 MiB extra
Full GPU, Q8 KV, 90K	84K	14,058 MiB	1,015	29.2	Occupies approximately 440 MiB extra

--n-cpu-moe 999 keeps the main MoE processing on the CPU. This results in very low GPU memory usage, but the local generation speed decreases by about 60%. The Q8 KV mode doesn’t improve throughput, and it also consumes an additional 320–440 MiB of memory due to its complex processing requirements. This performance test does not evaluate any differences in output quality between Q8 and Q4.

[Qm-jmz]

In fact, this is for reference only, and there are some points I have doubts about that I have not yet verified.

[Qm-jmz]

@itayinbarr I take back my earlier comment about Qwen 3.6 27B. I judged it too quickly.

On my split-GPU setup, I get around 30 TPS with Qwen 3.6 27B compared to roughly 80 TPS with Qwen 3.5 35B A3B. However, there's an important tradeoff: the 27B dense model consistently produces better code.

For the last 3 days, I've been running the same prompts through both models, and the 27B outputs have generally been superior. The downside is lower TPS, which means more waiting time. The upside is cleaner code that requires fewer fixes and less follow-up work.

I also noticed that GPT tends to have preferences toward certain models, so I ran a blind tournament. For each round, I only provided the generated code and labeled the submissions something like "A-Snake" and "B-Snake." In the next round, I shuffled the labels so A and B represented different models. After everything was ranked, I revealed which model corresponded to each entry. GPT's response was basically: "Now I see why you did a blind tournament lol."

That said, I agree that something like an Aider+Bench profile would be a more rigorous way to compare them.

So, REAM 192 is still a great model. It's fast and gets you about 90% of the way there. But if you're chasing that extra 10% in code quality, the 27B is the better choice.

@Qm-jmz

The 27B currently comes in two main variants: Standard and MTP.

The MTP version requires roughly 2 GB of additional VRAM and, at least in my testing, provides only about a 15% speed increase. If you're running a 16 GB card, I'd recommend the non-MTP version since it fits much more comfortably in VRAM.

For 24 GB cards, I'm currently testing: https://huggingface.co/kdqemre/Qwen3.6-27B-Omnimerge-v4-MTP-Q4_K_L-GGUF Even on 24 GB, it's a tight fit.

For 16 GB VRAM, you'll probably need to drop down to a Q3_K_S quant. There are many variants available, but I'd start by comparing a few 27B builds against A3B using your own prompts. Sometimes the finetune matters more than the base model, and in some cases Unsloth variants may produce better results.

Some variants worth testing:

• https://huggingface.co/unsloth/Qwen3.6-27B-GGUF • https://huggingface.co/mradermacher/Qwen3.6-27B-Heretic2-Uncensored-Finetune-Thinking-GGUF • https://huggingface.co/mradermacher/Qwen3.6-27B-Heretic2-Uncensored-Finetune-Thinking-i1-GGUF • https://huggingface.co/HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive • https://huggingface.co/DavidAU/Qwen3.6-27B-Heretic-Uncensored-FINETUNE-NEO-CODE-Di-IMatrix-MAX-GGUF

One final note: make sure you're downloading the non-MTP versions when that's what you intend. Some model titles can be misleading, so double-check the actual model details before downloading.

Are there some tips on choosing community models, and how to quickly select a reliable and easy-to-use model. Also, based on your recommendation, I am downloading https://huggingface.co/ManniX-ITA/Qwen3.6-27B-Omnimerge-v4-GGUF/blob/main/Qwen3.6-27B-Omnimerge-v4-Q3_K_S.gguf, similar tests to those conducted before may be performed later.

[Qm-jmz]

Omnimerge 27B Q3_K_S Native device configuration benchmark

Test date: 2026-07-07

Environment

• GPU：NVIDIA GeForce RTX 4090 Laptop GPU，16 GB VRAM
• CPU: Intel Core i9-14900HX, 24 cores / 32 threads
• Memory: 32 GB
• llama.cpp：b9844 (6c5de1cc8)
• Model: 'Qwen3.6-27B-Omnimerge-v4-Q3_K_S.gguf' (11.246 GiB)
• Architecture metadata: 'qwen35' dense, 64 layers, KV heads=4, native context=262144

Each group passed the real '/completion' request test.

Result

Configure	Enter	Memory peak	Prompt tok/s	Generate tok/s	Conclusion
Full GPU, Q4 KV, 32K	8K	13,348 MiB	770	17.4	Available, with the largest available video memory
Full GPU, Q4 KV, 48K	8K	13,707 MiB	764	19.6	You can use
Full GPU, Q4 KV, 64K	8K	14,086 MiB	761	19.5	Default recommendation
Full GPU, Q4 KV, 90K	8K	14,637 MiB	757	19.4	Long context can be used
Full GPU, Q8 KV, 32K	8K	13,855 MiB	758	19.3	No significant gains in throughput
Full GPU, Q8 KV, 64K	8K	15,111 MiB	701	19.2	Video memory is too tight, not recommended to default
Full GPU, Q4 KV, 48K	44K	13,895 MiB	662	13.9	Passed the stress test
Full GPU, Q4 KV, 64K	60K	14,218 MiB	642	15.5	Passed the stress test
Full GPU, Q4 KV, 90K	84K	14,745 MiB	566	11.5*	Passed the stress test
Full GPU, Q8 KV, 32K	28K	13,777 MiB	697	17.1	Q8 still shows no significant gains
Full GPU, Q4 KV, 128K	8K	15,471 MiB	720	18.2	Extreme mode, very limited VRAM margin
Full GPU, Q4 KV, 128K	120K	15,470 MiB	490	10.6*	Passed the ultimate pressure test

'*': 90K/128K long prompt The model quickly stops on its own in the first round of requests, cold start generation rate is unreliable; The table uses the 64-token decode rate of cache continuation as a reference.

parameters:

-ngl 99
--ctx-size 65536
--flash-attn on
-ctk q4_0
-ctv q4_0
--no-mmap
--kv-unified
-np 1
-b 2048
-ub 512
--cache-ram 8192
--ctx-checkpoints 8
--cache-prompt
--no-context-shift
--checkpoint-min-step 1024
--jinja

[charly1r]

@Qm-jmz Great testing and thanks for sharing your results.

Just to clarify, my post was intended as an informal community discussion based on my personal experience using different models. The goal is simply to help others who are trying to find a model that works well for their hardware and use cases.

Everyone is welcome to share their own experiences and experiments—we're all here to help each other find that elusive "holy grail" model! 🚀

Regarding your 90k and 120k context tests and the model stopping unexpectedly, I've seen very similar behavior before.

In some cases, a model appears to load successfully and allocates most of the available VRAM. However, when the first prompt is submitted—even something as simple as "hello"—the session immediately exits back to the terminal prompt.

From what I've observed, this usually indicates that loading the model and allocating the KV cache is not the entire memory requirement. The model often needs additional VRAM (and sometimes a small amount of Shared GPU Memory) once it begins processing the first prompt. The increase can be minimal, but if the system is already operating at the edge of available memory, that extra requirement can be enough to cause the session to fail.

That's one of the reasons I recommended the Q3 quantizations for your setup. They generally provide a bit more memory headroom, which can make the difference between a stable session and one that crashes as soon as prompt ingestion begins.

Thanks again for running these tests and sharing the data—it helps everyone better understand the practical limits of these models in real-world use.