Stop Asking What Model to Run. You Only Have 2 Options

Source: Agent Native, Stop Asking What Model to Run. You Only Have 2 Options (Medium, 2026-07-02): https://agentnativedev.medium.com/stop-asking-what-model-to-run-you-only-have-2-options-67304ed23c46

This is a summary in my own words, not the article itself; read the original for the full text.

The core argument

“Which model should we run?” is usually the wrong first question. When a local model looks incompetent, the stack around it is more often at fault than the weights: a wrong chat template, a context window too large for the hardware, a harsh quantization, no tool-call parser, lossy compaction, or an unrestricted shell. Those are harness failures blamed on the model.

The better question is what the smallest agent stack is that a team can standardize on, measure and operate. The proposed answer is deliberately dull:

  1. One default local coding model, plus one fallback or second opinion.
  2. Both behind the same API, with every agent going through the same harness.
  3. Evaluation on repository-level tasks, changing one variable at a time.

The “two options” title riffs on a deliberately absurd Reddit claim that only two local models exist. The literal claim is wrong; the instinct is right, because a short default list serves a team better than dozens of models across dozens of hardware setups. As the author puts it, “model roulette is not engineering, it is entertainment.”

What decides whether a model feels smart

  • Five factors outweigh the model: hardware, quantization, context size, memory bandwidth, and the harness.
    • The identical model can be stable on one setup and broken on another.
  • A coding agent is a loop, not a chat: read the repo, plan, pick files, edit, run checks, read the errors, patch, summarize, and stop safely.
  • Harness quality covers chat templates, tool-call parsing, handling of reasoning tokens, file selection, compaction, shell permissions, test feedback, patch format, retries and stop conditions.

Decouple the agent from the runner

  • llama.cpp, vLLM, SGLang, LM Studio, Ollama and hosted providers all offer an OpenAI-compatible /v1/chat/completions endpoint.
  • The agent should speak only that API, so the runner can change without touching the product. Harnesses such as OpenCode, Cline and Qwen-Agent can target any such endpoint.

Pick by job, not by leaderboard

  • Qwen 3.6 27B (dense): the default when correctness matters. Every token uses all parameters, which makes multi-file code work more predictable.
    • 24 GB at a good quant, or 40–80 GB at FP8/BF16; 262,144-token native context.
  • Qwen 3.6 35B-A3B (mixture of experts, about 3B active): when hardware is the limit, on 12–24 GB cards with careful quantization and a sized context.
    • All weights still need memory; only compute scales with the active count.
    • When weights overflow VRAM into host memory, bandwidth becomes the bottleneck, which explains wildly different speeds on the same GPU.
  • Google Gemma 4 (31B dense, 26B-A4B MoE): language-heavy work such as conversation, translation, tone, explanation, multimodal documents, and release notes.
  • Small models (about 0.6B–14B) for routing, classification, extraction, summaries, edge use and cheap batch jobs; cloud only for what local cannot do.

Note: The model names are a mid-2026 snapshot and will date; the selection method will not.

Quantization changes behavior, not just size

  • Lower bit widths erode knowledge, numerical stability, tool-call formatting, syntax, instruction following, long-context recall, stopping, and multilingual quality.
  • Rough guide: Q4 is the floor for serious work, Q5 better, Q8 comfortable but heavy, Q3 sometimes acceptable, Q2 never the default for a coding agent.
  • Test a quant on real tasks, not on whether it loads. A small model at good precision often wins short or strict-JSON work against a crushed large one; a low-bit large model can still win on reasoning.

The worked case in this course

The capstone that follows tests this argument on an NVIDIA inference microservice (NVIDIA-NIM), and the outcome makes it concrete: the model was never the variable that mattered. Diagnosing Tool Calling on an NVIDIA NIM shows that tool calling needs the model’s output convention and the server’s parser to agree, and that a mismatch returns a normal-looking response with no tool call. NVIDIA NIM Operational Traps collects failures where the container reports healthy yet does not do its job. NIM Tool Calling Defect Report traces one case to its root: the container silently discarded the tool-calling settings, so no choice of model could have fixed it.