# Which Local AI Model Should You Run?
Canonical source: [https://isaiuseful.com/local-models](https://isaiuseful.com/local-models)
Agent-first model guide · checked 15 August 2026
A model package loading is only the first gate. A useful local agent also needs memory and tokens for **instructions, tool schemas, repository files, tool results, reasoning and the answer** . Choose the job first, then balance capability, context and memory.
- [Choose an agent role](#roles)
- [Route local + frontier work](#routing)
- [Use a phone as the controller](#mobile-control)
- [Budget the context](#context)
- [Match your memory](#memory)
**4**
agent roles
WORKER · CODER · RESEARCHER · PLANNER
**3**
context numbers
ADVERTISED · CONFIGURED · USABLE
**20–25%**
minimum memory headroom
OS · HARNESS · KV CACHE
Role selector
## Which local AI model fits each kind of work?
Size is a rough capability band, not an IQ score. Architecture and post-training can move a model up or down; your quant, runtime and tool grammar can move it again.
Practical band
**20–35B · Q4/Q6**
16–32K minimum · 32–64K preferred
### The everyday local-agent band.
Suitable for bounded multi-file edits, debugging with a reproduction, test-driven implementation and a small set of reliable tools. Keep a human checkpoint before migrations, security changes or broad refactors.
### Escalate the plan
For an ambiguous feature, let a 70–120B or strong hosted model produce the plan and review criteria. Save that plan as a file, compact or start a clean task, then let the cheaper 20–35B worker implement it.
Model band
Good default assignments
Expect trouble with
**2–4B**
Routing, tagging, format conversion, autocomplete and tightly constrained extraction.
Reliable planning, fuzzy tool choice, long chains and precise multilingual nuance.
**7–12B**
Grounded Q&A, drafting, single-file code, one or two simple tools and supervised workers.
Autonomous repo-wide work, conflicting requirements and recovery after several failed steps.
**20–35B**
Everyday coding agents, multi-document synthesis, structured research and several focused tools.
High-stakes judgment, novel architecture and long autonomous runs without tests or checkpoints.
**70–120B**
Planning, review, ambiguous debugging, cross-file refactors and coordinating specialist workers.
Guaranteed frontier parity, perfect long-context recall or unattended production changes.
**400B+**
Hosted or multi-GPU frontier open-weight work when quality justifies infrastructure. [Plan the provider tier →](https://isaiuseful.com/cloud-models.html.md)
Single 128 GB machines; active MoE parameters do not remove the need to store and load the full checkpoint.
High fidelity
### Q6 / Q8
Use when exact phrasing, code reliability or multilingual nuance matters and a smaller high-precision model already meets the task.
Start here
### Q4_K_M / official QAT
The normal quality/size starting point. Four-bit compression usually preserves broad ability, but agentic application accuracy can degrade more than perplexity suggests.
Trade carefully
### Q3
Useful when it unlocks a meaningfully larger model. Re-run tool use, structured output and language tests; subtle failures appear before chat becomes obviously bad.
Last-resort fit
### Q2 / IQ2
Use only after testing the exact build on your tasks. Very-low-bit quants can make a larger model load, but tool calls, structured output, code edits and multilingual nuance may degrade unevenly.
**Heuristic, not a benchmark grade.** The role bands synthesize model-card capability curves and practical agent constraints. [ACBench](https://isaiuseful.com/benchmarks.html.md#acbench) found that compression effects differ by model, task and quantization method; validate the exact build through the harness and tasks you will use.
- [Understand the ACBench signal →](https://isaiuseful.com/benchmarks.html.md#acbench)
- [Read the ACBench paper →](https://arxiv.org/abs/2505.19433)
- [See llama.cpp quant sizes →](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md)
Hybrid agent architecture
## Route each step—not the whole application—to the right model.
A local model does not need to replace the frontier model. Let the strongest model resolve ambiguity, move routine execution to a cheaper specialist, and escalate when the trace shows difficulty.
> Visual: Hybrid model-routing architecture
**Visual entries (display order):**
- 01 · plan **Frontier planner** Settles architecture, constraints, review criteria and unfamiliar failure modes.
- 02 · route **Model-neutral gateway** Selects a target for each turn from task, policy, cost, latency and recent tool signals.
- 03 · execute + escalate **Specialist models → tools → frontier** Local or hosted workers handle bounded calls; repeated errors, loops or uncertainty route back up.
Current local worker
### Nemotron 3.5 Lightning
NVIDIA’s 30B-total / 3B-active hybrid MoE targets high-volume tool calls, validation and delegated execution. The official NVFP4 checkpoint is about 20.1 GiB before runtime and cache, making 32 GB the sensible personal-system floor.
Keep the seam stable
### Route to roles, not vendor IDs.
Name targets such as `planner` , `private-worker` and `frontier-review` . Map those aliases to a local Gemma/Muse/Qwen-class endpoint, a European hosted service or a frontier API without rewriting application logic.
Measure useful work
### Tokens are an input—not the score.
Benchmark task success, total workflow cost, completion time, retries and frontier-escalation frequency. Include human review and failed runs so a cheap but unreliable route cannot look efficient.
Current maturity boundary
### Switchyard is an experiment, not a production default.
NeMo Switchyard provides provider translation plus classifier, stage, escalation and custom routing. Its repository labels the software pre-alpha and not for production use; isolate it behind a gateway contract, pin a revision and keep a tested fallback.
**Promising vendor and partner evidence—not a universal saving.** NVIDIA reports up to 4× the output speed of similar-sized models and a 30% faster 10,000-task agent run at comparable accuracy. On a separate 145-task internal suite, LangChain reported that an escalation route between Lightning and Claude Opus 4.8 cut total cost 74% while sending 7% of calls to the frontier model, but accuracy fell from 86% to 80%. Reproduce the comparison on your own task mix, prices, providers and failure costs.
- [NVIDIA Lightning launch and benchmark context →](https://developer.nvidia.com/blog/nvidia-nemotron-3-5-lightning-delivers-fast-accurate-specialized-task-execution-for-long-running-agents/)
- [NVIDIA Switchyard architecture →](https://developer.nvidia.com/blog/route-ai-agent-workloads-across-models-with-nvidia-nemo-switchyard/)
- [LangChain’s 145-task benchmark →](https://www.langchain.com/blog/switchyard-agent-routing-benchmark)
- [Switchyard source, licence and maturity warning →](https://github.com/NVIDIA-NeMo/Switchyard)
- [Plan a hosted or data-centre Lightning route →](https://isaiuseful.com/cloud-models.html.md#nemotron-35-lightning)
Pocket control
## Let the phone drive without pretending it hosts the model.
Remote access and local inference are separate decisions. The phone can carry the conversation and approvals while an authenticated gateway, model and tools stay on an always-on machine you control.
> Visual: Phone-to-home-model control path
**Visual entries (display order):**
- 01 · phone **Browser · app · messaging** Collects the request, shows progress and asks for approval. It may be only a control surface.
- 02 · gateway **Identity · sessions · tool policy** Authenticates the client, owns the agent loop and decides which backend and capabilities are available.
- 03 · model host **Ollama · LM Studio · vLLM** Loads weights and serves inference on the home machine, workstation or private server.
Easiest browser route
### Open WebUI + Ollama
Run both on the host, make only the authenticated WebUI reachable over a private route and open it in the phone browser. Keep Ollama itself on localhost or an internal network.
- [Open WebUI documentation →](https://docs.openwebui.com/)
Native companion route
### OpenClaw + home gateway
Keep the gateway and session state on the always-on host, then pair an iPhone or Android companion. Start with remote chat; add device-node capabilities only when the workflow needs them.
- [OpenClaw remote access →](https://docs.openclaw.ai/gateway/remote)
Coding-agent cockpit
### Orca mobile + desktop
Pair Orca's iPhone or Android companion with its desktop runtime to see worktrees, agent status and session output, then reply or dictate from the phone. The desktop remains the source of truth.
- [Orca mobile companion →](https://www.onorca.dev/docs/mobile)
Use an app you already have
### Hermes + messaging
Reach the agent through a supported messaging channel while its model runs elsewhere. Treat that channel provider as part of the data path, even when inference stays at home.
- [Hermes messaging gateway →](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/)
**Private-by-default rule:** expose the authenticated interface or gateway through a Tailnet or SSH tunnel. Do not forward a raw model, retrieval or agent-control port through the router. Test from cellular data, confirm the execution-location label and verify that a denied tool still fails. A model that actually runs on the phone is a different architecture, with device limits on model fit, sustained speed, memory, heat and battery.
- [OpenClaw iPhone setup →](https://docs.openclaw.ai/platforms/ios)
- [OpenClaw Android setup →](https://docs.openclaw.ai/platforms/android)
- [Orca private remote server →](https://www.onorca.dev/docs/remote-servers)
- [LM Studio API authentication →](https://lmstudio.ai/docs/developer/core/authentication)
- [See an always-on host topology →](https://isaiuseful.com/remote-spark.html.md#topology)
Agent-safe memory
## How much memory is left after the model loads?
These defaults assume one interactive agent, a Q4-class model, the OS and runtime, plus useful KV-cache headroom. “Chat fit” can be larger; “agent fit” must survive tool use and growing context.
16 GB agent fit: 7–12B Q4 with 8–16K context. A 14 GB model may load, but leaves almost no runway for a harness.
Shortlist for this machine
### Start with these 16 GB candidates.
These are working configurations, not weight-only fits. Context is a conservative configured starting point for one interactive agent; increase it only after watching memory and prompt-processing time.
Multilingual worker
### Gemma 4 12B
**QAT Q4 · start at 8–16K**
Best first candidate for multilingual drafting, retrieval-grounded work and bounded code changes.
- [Open family record →](#gemma)
Tool-aware worker
### Qwen3.5 9B
**Q4 · start at 8–16K**
A compact coding and tool-use baseline with broad runtime packaging.
- [Open family record →](#qwen)
Moonshot research model
### Moonlight 16B‑A3B
**Community 4-bit · 8K maximum**
The small Moonshot option. It is useful for experiments, but it is not a miniature Kimi K3.
- [Open family record →](#kimi)
Older edge baseline
### GPT‑OSS 20B
**14 GB MXFP4 · short context**
Retained for compatibility testing. It can load in 16 GB, but leaves almost no margin and is not a current first pick.
- [Open family record →](#gpt-oss)
**Looking for the viral Kimi demo?** Kimi K3 is a 2.8T server-scale model, not a 16 GB download. The Moonshot card above is an older small research checkpoint. [Plan the provider tier →](https://isaiuseful.com/cloud-models.html.md)
16 GB · worker
### 7–12B Q4 · 8–16K
**Agent-safe start:** Gemma 4 12B QAT Q4 (about 6.7–7.4 GB) or Qwen3.5 9B Q4 (6.6 GB). Moonlight 16B‑A3B is a small Moonshot research option in community 4-bit builds, but its model limit is only 8K.
**Chat fit is not agent fit:** GPT‑OSS 20B is designed to run within 16 GB and its Ollama package is 14 GB, but that leaves almost no margin for macOS, long context or tool results.
24 GB · strong worker
### 12–20B Q4 · 16–32K
**Current start:** Gemma 4 12B at Q6/Q8 or Muse Glimmer’s official 17 GB quant. Gemma 4 26B‑A4B Q4 at 14.4 GB is also viable when the runtime supports it well; treat 24 GB Muse as a tighter agent configuration.
**Tight:** Qwen3.8‑27B FP8 (16.38 GB; supported serving backends), Qwen3.6 27B Q4 (17 GB; broader desktop packaging), GLM‑4.7‑Flash (19 GB) and 32B Q4 packages load on paper but sacrifice the context and tool runway that makes a harness useful.
32 GB · everyday agent
### 20–31B low-bit · 16–32K
**Ranked start:** Gemma 4 26B‑A4B Q4 first, Muse Glimmer 30B second and Qwen3.8‑27B FP8 third. Qwen remains the coding-heavy choice on a supported backend and leaves about 15 GB before runtime allocation.
**Fallback route:** GLM‑4.7‑Flash Q4 and older broadly packaged models remain compatibility options where current Gemma, Muse or Qwen builds are unavailable.
64 GB · choose your bias
### Context route or capability route
**Ranked start:** Gemma 4 31B for the strongest current family fit, Muse Glimmer for agent tools and review, then Qwen3.8‑27B FP8 for coding-heavy work and parallel sequences. Use the older DeepSeek R1 Distill 70B only when a reasoning acceptance set proves the trade.
**Do not spend memory for its own sake:** Qwen says the FP8 package’s performance is nearly identical to the original model. The 55.56 GB full checkpoint is a fidelity test route, not the default 64 GB configuration.
128 GB · planner or team
### One 120B or several specialists
**Ranked start:** run Gemma 4 at high precision, Muse Glimmer for one or more local agents, or Qwen3.8‑27B FP8 with parallel headroom. Keep GPT‑OSS 120B as an older compatibility baseline only when the exact acceptance set rewards its 65 GB package.
Qwen3.5 122B‑A10B Q4 (81 GB) remains a capability-first stretch, not the automatic Qwen default. Choose it only when your tasks beat Qwen3.8‑27B by enough to justify the lost context and concurrency runway.
01
### Weights share; context multiplies
Parallel agents can share one loaded model, but each active sequence adds KV-cache pressure and its own growing history.
02
### Set context deliberately
A model may advertise 128K–1M while the runtime loads 8K. Raising it costs memory and prompt-processing time.
03
### Prefer Q4_K_M or official QAT
Start here, then compare Q6/Q8 or a larger Q3 only on a repeatable tool, code and language test set.
04
### Measure completed work
Task success, retries and human corrections matter more than tokens per second or a leaderboard point.
Context budget
## 128K advertised is not 128K for your files.
The configured window must hold the harness, system and project instructions, MCP schemas, conversation, retrieved files, tool results, hidden reasoning where applicable and the next answer.
01
### Advertised
**What the model card allows**
The training limit: often 128K, 256K or more. It says nothing about your runtime setting, speed or memory fit.
02
### Configured
**What the runtime actually loads**
A runtime can be set to 8K even when the model supports far more. The inference server and agent client may also impose separate limits.
03
### Usable
**What remains for the task**
Configured window minus instructions, tool definitions, history, output reserve and safety buffer. This is the number to plan around.
Coding fit
**Fits with 6.2K tokens of runway**
The initial slice fits. The remaining range is roughly 1 additional 16 KiB tool-result batch, or 445–1,039 more nonblank code lines before compaction.
> Visual: 32K configured: 10K harness, 8.3K supplied material, 8K output reserve, 6.2K free.
Harness
**10K**
*Working set*
**8.3K**
Output reserve
**8K**
Runway
**6.2K**
**4**
files
**480**
relevant lines
**12 KiB**
tool output
**350**
plan words
**A coding task is not “the repo”**
Load a deliberate working set
Count the files and relevant lines the agent must see now. Search and retrieval can fetch more later; dumping the full tree spends context before work begins.
**Tool rounds eat the runway**
Diffs, tests and errors accumulate
A long command result may cost more than the prompt that caused it. Prefer focused tests, clipped logs and fresh reads over carrying every failed attempt.
**Plan → save → compact → build**
Use capability where it propagates
Let a stronger model resolve ambiguity and write the plan plus acceptance criteria. Start the worker with that artifact and only the files needed for the next task.
**This is a token-occupancy estimator, not a tokenizer or RAM calculator.** Coding mode uses transparent planning midpoints: 10 tokens per relevant nonblank line, 4 bytes per token for logs/diffs, and 0.75 prose words per token. Real repositories, languages, tokenizers, harness prompts and hidden model work vary widely. Use the range as a loading plan, then inspect the harness’s live context view.
- [LM Studio context fields →](https://lmstudio.ai/docs/developer/rest/list)
- [Copilot context management →](https://docs.github.com/en/copilot/concepts/agents/copilot-cli/context-management)
Family deep dive
## Inspect the families behind your memory shortlist.
Families are ordered by current recommendation, not download size: **Gemma 4 first, Muse Glimmer second and Qwen3.8 third.** An **older** label marks the relevant local line as pre‑2026 and keeps it for compatibility or research—not as a first pick. Fit chips still show where a package can be practical, not where its maximum context will fit.
room for useful context
tight or runtime-specific
workstation, multi-GPU or cloud
01
Gemma 4
Multilingual European default with unusually strong Estonian proof.
**16 GB**
12B QAT Q4 · about 6.7–7.4 GB weights
**24 GB**
26B‑A4B Q4 · official estimate 14.4 GB
**32 GB**
31B Q4 · official estimate 17.5 GB
**64 GB**
31B SFP8 · official estimate 34.9 GB
**128 GB**
31B BF16 · official estimate 69.9 GB
01 · Current local default
### A European model choice with a demanding Estonian proof point.
Google reports pre-training across 140+ languages and out-of-the-box support for 35+. Independent TartuNLP evaluation of **Gemma 3** found strong Estonian instruction following, grammar and word-meaning results at 12B and 27B. That makes Estonian an excellent stress test for the family—not a promise for every language or automatic evidence for Gemma 4.
**All Gemma 4 sizes**
E2B
E4B
12B
26B · 4B active
31B
Base and instruction-tuned checkpoints; Google also publishes quantization-aware-trained variants.
**Scale inside one family**
Google model-card scores · higher is better
- [MMLU‑Pro](https://isaiuseful.com/benchmarks.html.md#mmlu-pro)
**60.0**
E2B
- [MMLU‑Pro](https://isaiuseful.com/benchmarks.html.md#mmlu-pro)
**77.2**
12B
- [MMLU‑Pro](https://isaiuseful.com/benchmarks.html.md#mmlu-pro)
**85.2**
31B
- [LiveCodeBench v6](https://isaiuseful.com/benchmarks.html.md#livecodebench)
**44.0**
E2B
- [LiveCodeBench v6](https://isaiuseful.com/benchmarks.html.md#livecodebench)
**72.0**
12B
- [LiveCodeBench v6](https://isaiuseful.com/benchmarks.html.md#livecodebench)
**80.0**
31B
- [GitHub →](https://github.com/google-deepmind/gemma)
- [Hugging Face collection →](https://huggingface.co/collections/google/gemma-4)
- [Ollama tags →](https://ollama.com/library/gemma4)
- [LM Studio family →](https://lmstudio.ai/models/gemma-4)
- [Open 12B in LM Studio →](https://lmstudio.ai/deeplink?name=gemma-4-12b&owner=google)
- [Gemma 4 paper →](https://arxiv.org/abs/2607.02770)
- [Official model card and scores →](https://ai.google.dev/gemma/docs/core/model_card_4)
- [TartuNLP Estonian evaluation →](https://huggingface.co/tartuNLP/llama-estllm-prototype-0825)
- [Gemma Scope, ShieldGemma and companion models →](https://ai.google.dev/gemma/docs)
02
Muse Glimmer
Meta’s current 30B local-agent launch for tools, coding and image input.
**24 GB**
Official K‑Quant‑17GB · 16.76 GB weights · start below maximum context
**32 GB**
K‑Quant‑17GB plus optional 1.40 GB vision encoder and 1.63 GB DFlash drafter
**64 GB**
Official K‑Quant‑Dynamic · 19.65 GB weights · target hardware 64 GB
**128 GB**
DGX Spark owner runs · 5–10.5 tok/s base · about 23–38 tok/s with DFlash
02 · Current agent launch
### A real 30B release for one device—with unusually clear package boundaries.
**Muse Glimmer 30B** is a dense model distilled from Muse Spark and released under Apache 2.0 for agentic work, tool use, coding, multilingual tasks and text-plus-image input. Meta publishes a 131,072+ model context and training across more than 100 languages. The official GGUF repository contains a 16.76 GB K‑Quant‑17GB build and a 19.65 GB dynamic build; image input needs the separate 1.40 GB perception file, while the optional 1.63 GB DFlash drafter accelerates generation without changing accepted output. Two early DGX Spark owner runs measured 5–10.5 tok/s without DFlash and about 23–38 tok/s with the official drafter set to 15 speculative tokens; useful evidence, but not a standardized cross-engine benchmark. Those file sizes make 24–32 GB systems credible targets, but they do not make the full context cheap or guarantee that every launch-day client understands the new architecture.
**Official released artifacts**
29.6B total · dense
17 GB local quant
19.65 GB dynamic quant
1.40 GB vision encoder
1.63 GB DFlash drafter
131,072+ model context
Meta targets the two quants at 24/32 GB and 64 GB respectively. Its announcement says Ollama and LM Studio integrations are coming in the days after launch, so pin a current llama.cpp or supported Transformers build and test the complete scaffold.
**Muse Glimmer vendor scorecard**
High reasoning; harnesses differ; compare the full table
- [MCP‑Atlas public](https://isaiuseful.com/benchmarks.html.md#mcp-atlas)
**75.5**
Tool and server use
- [SWE‑bench Pro](https://isaiuseful.com/benchmarks.html.md#swe-bench)
**51.2**
Agentic coding
- [Terminal‑Bench 2.1](https://isaiuseful.com/benchmarks.html.md#terminal-bench)
**51.7**
With Terminus‑2
- [OSWorld Verified](https://isaiuseful.com/benchmarks.html.md#osworld)
**65.9**
Qwen3.6‑27B: 75.6
- [Meta launch, local-speed evidence and packaging timeline →](https://research.meta.ai/blog/introducing-muse-glimmer-open-agentic-model)
- [Full-precision model card and benchmark table →](https://huggingface.co/meta-models/Muse-Glimmer-30B)
- [Official GGUF files and exact sizes →](https://huggingface.co/meta-models/Muse-Glimmer-30B-GGUF/tree/main)
- [Meta evaluation methodology →](https://research.meta.ai/static/muse-glimmer-methodology)
- [Meta developer documentation →](https://developer.meta.com/ai/models/muse-glimmer/)
- [Usage policy and agent safety guidance →](https://huggingface.co/meta-models/Muse-Glimmer-30B/blob/main/USAGE_POLICY.md)
- [DGX Spark llama.cpp, DFlash and extended-context owner test →](https://www.reddit.com/r/LocalLLaMA/comments/1vl9adk/i_ran_muse_glimmer_1m_context_all_tests_passed/)
- [DGX Spark vLLM DFlash owner test →](https://www.reddit.com/r/LocalLLM/comments/1vm4j0i/muse_glimmer_30b_on_dgx_spark_using_dflash_is/)
- [Spark capacity and performance boundary →](https://isaiuseful.com/remote-spark.html.md#spark-reviews)
03
Qwen3.8 / Qwen
Current all-rounder for multilingual, coding and agentic work.
**16 GB**
Qwen3.5 9B Q4 · 6.6 GB
**24 GB**
Qwen3.8 27B FP8 · 16.38 GB · short-context fit
**32 GB**
Qwen3.8 27B FP8 · useful context runway
**64 GB**
Qwen3.8 27B FP8 · long context or parallel agents
**128 GB**
Qwen3.8 27B FP8 · high-concurrency agent team
03 · Current all-rounder
### Qwen3.8‑27B is the practical Qwen pick for a 32 GB machine.
It handles text, images and video, can switch thinking on or off, and ships in a 16.38 GB FP8 build. Start with 32 GB for one useful local agent; 64 GB gives you more context or parallel work. Qwen reports that the FP8 build performs close to the full 55.56 GB checkpoint, but test it on your own tasks.
**Qwen3.8‑27B release**
27B · dense
16.38 GB · FP8
55.56 GB · full
262K · native context
1M · extensible
Text + image + video
Controllable thinking
Official serving recipes: Transformers, vLLM, SGLang and TokenSpeed.
**Released Qwen3.8 scorecard**
Qwen model card · vendor-reported · higher is better
- [LiveCodeBench v6](https://isaiuseful.com/benchmarks.html.md#livecodebench)
**90.3**
Competitive coding
- [OSWorld‑Verified](https://isaiuseful.com/benchmarks.html.md#osworld)
**84.3**
Computer use
- [Terminal‑Bench 2.1](https://isaiuseful.com/benchmarks.html.md#terminal-bench)
**73.0**
Terminus scaffold
CoWorkBench
**70.7**
Long-horizon office work
- [SWE‑bench Pro](https://isaiuseful.com/benchmarks.html.md#swe-bench)
**61.7**
Agentic coding
IFBench
**79.5**
Instruction following
- [Qwen3.8‑27B model card, scores and full checkpoint →](https://huggingface.co/Qwen/Qwen3.8-27B)
- [Official Qwen3.8‑27B FP8 package →](https://huggingface.co/Qwen/Qwen3.8-27B-FP8)
- [SGLang Qwen3.8 cookbook →](https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.8-27B)
- [vLLM Qwen3.8 recipe →](https://recipes.vllm.ai/Qwen/Qwen3.8-27B)
- [TokenSpeed Qwen3.8 recipe →](https://lightseek.org/tokenspeed/recipes/models#qwen3-8)
- [Qwen Agent →](https://github.com/QwenLM/Qwen-Agent)
- [Qwen Code →](https://github.com/QwenLM/qwen-code)
04
Nemotron
Lightning for routed local execution; larger models for planning and orchestration.
**16 GB**
Nano 4B Q4 · 2.8 GB
**32 GB**
Lightning 30B‑A3B NVFP4 · 20.1 GiB before runtime and cache
**64 GB**
Nano 30B‑A3B Q8 · 34 GB; Lightning BF16 is too tight for agent use
**128 GB**
Super 120B‑A12B low-bit · backend-specific; verify support
04 · Current NVIDIA specialist
### Lightning is the local execution worker; Super and Ultra move up the planning ladder.
**Nemotron 3.5 Lightning** is a 30B-total / 3B-active hybrid MoE released for high-volume agent execution. Its official NVFP4 repository is about 20.1 GiB, and NVIDIA documents DGX Spark, Jetson and GeForce RTX 5090 routes plus data-centre serving. Treat 32 GB as the practical personal-system floor: runtime state, its optional 1.26 GiB DSpark drafter and useful context still need memory. The model’s OpenMDW 1.1 terms are not the earlier NVIDIA Open Model License.
**Current family**
Nano · 4B
Nano · 30B / 3B active
Lightning 3.5 · 30B / 3B active
Nano Omni · 30B / 3B active
Super · 120B / 12B active
Ultra · 550B / 55B active
Lightning supports up to a one-million-token model window; available memory and the serving recipe determine the usable context.
**Lightning → Super**
NVIDIA model-card evaluations; harness settings matter
- [MMLU‑Pro](https://isaiuseful.com/benchmarks.html.md#mmlu-pro)
**81.9**
Lightning BF16
- [MMLU‑Pro](https://isaiuseful.com/benchmarks.html.md#mmlu-pro)
**83.6**
Super 120B FP8
- [GPQA](https://isaiuseful.com/benchmarks.html.md#gpqa)
**75.4**
Lightning BF16
- [GPQA](https://isaiuseful.com/benchmarks.html.md#gpqa)
**79.4**
Super 120B FP8
- [SWE‑bench Verified](https://isaiuseful.com/benchmarks.html.md#swe-bench)
**51.6**
Lightning BF16
- [Terminal‑Bench 2.1](https://isaiuseful.com/benchmarks.html.md#terminal-bench)
**24.6**
Lightning BF16
- [Lightning launch and local routes →](https://developer.nvidia.com/blog/nvidia-nemotron-3-5-lightning-delivers-fast-accurate-specialized-task-execution-for-long-running-agents/)
- [Lightning NVFP4 card and recipe →](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4)
- [Lightning BF16 card and scores →](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16)
- [Use Lightning in a hybrid route →](#routing)
- [Nemotron recipes and data →](https://github.com/NVIDIA-NeMo/Nemotron)
- [Nemotron 3 Nano card →](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16)
- [Super card and hardware floor →](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8)
- [Ultra deployment tier →](https://isaiuseful.com/cloud-models.html.md#nemotron-3-ultra)
- [NeMo Guardrails →](https://github.com/NVIDIA-NeMo/NeMo-Guardrails)
05
GLM
GLM‑4.7‑Flash for personal machines; GLM‑5.2 for self-hosting; GLM‑5.3 for current hosted long-horizon work.
**16 GB**
Older GLM‑4 9B Q4 · about 5–6 GB
**24 GB**
GLM‑4.7‑Flash 30B‑A3B Q4 · 19 GB · tight
**32 GB**
GLM‑4.7‑Flash Q4 · 19 GB · 16–32K starting context
**64 GB**
GLM‑4.7‑Flash Q8 or BF16 · package-dependent
**748 GB**
GLM‑5.2 · 753B / 40B active · official 465 GB NVIDIA NVFP4 checkpoint
05 · Current mixed deployment
### Choose GLM‑5.3 for hosted work and GLM‑5.2 for self-hosting.
GLM‑5.3 is Z.ai’s newer model for long-running coding and agent tasks. It is available through Coding Plan, while the general API and downloadable weights are still pending. If you need to run GLM yourself today, use GLM‑5.2; its official 465 GB NVFP4 build is aimed at DGX Station-class hardware. On a normal local machine, GLM‑4.7‑Flash is the practical choice.
**Relevant family members**
GLM‑4 · 9B
GLM‑4.7‑Flash · 30B / 3B active
GLM‑5.1 · 744B / 40B active
GLM‑5.2 · 753B / 40B active · weights live
GLM‑5.3 · same base · hosted release
GLM‑5.3 weights, licence and serving recipes remain pending; do not reuse GLM‑5.2 hardware results as GLM‑5.3 measurements.
**GLM‑5.3 launch scorecard**
Z.ai-reported · protocols differ · higher is better
- [Terminal‑Bench 2.1](https://isaiuseful.com/benchmarks.html.md#terminal-bench)
**88.2**
Claude Code 2.1.207; six-hour timeout
- [Terminal‑Bench 3.0](https://isaiuseful.com/benchmarks.html.md#terminal-bench)
**28.3**
avg@3; max effort; 400K context
- [Agents’ Last Exam CLI](https://isaiuseful.com/benchmarks.html.md#agents-last-exam)
**28.5**
Official protocol; max effort; 1M context
- [HLE + tools](https://isaiuseful.com/benchmarks.html.md#hle)
**62.5**
300K context; context management; LLM judge
**GLM‑5.3 migration recipe**
Do not change only the model ID. Z.ai says thinking must remain enabled: replace `thinking.type: "disabled"` with `"enabled"` , then choose `reasoning_effort` `low` , `high` or `max` . Start existing non-thinking workloads at `low` ; use `max` for coding acceptance tests. A disabled-thinking request will fail.
- [GLM‑5.3 announcement, benchmarks and methods →](https://z.ai/blog/glm-5.3)
- [GLM‑5.3 API status and migration guide →](https://docs.z.ai/guides/llm/glm-5.3)
- [GLM Coding Plan and coding-agent setup →](https://docs.z.ai/devpack/overview)
- [GLM‑5 family repository →](https://github.com/zai-org/GLM-5)
- [GLM‑5.2 model card, weights and licence →](https://huggingface.co/zai-org/GLM-5.2)
- [Official NVIDIA GLM‑5.2 NVFP4 checkpoint →](https://huggingface.co/nvidia/GLM-5.2-NVFP4)
- [GLM‑5.2 DGX Station measurements →](https://isaiuseful.com/dgx-station.html.md#sweet-spot)
- [GLM‑4.7‑Flash model card →](https://huggingface.co/zai-org/GLM-4.7-Flash)
06
DeepSeek
Older distilled reasoning baselines locally; current frontier releases belong on servers.
**16 GB**
R1‑0528 Qwen3 8B Q4 · 5.2 GB
**24 GB**
R1 Distill 32B Q4 · 20 GB · tight
**32 GB**
R1 Distill 32B Q4 · 20 GB
**64 GB**
R1 Distill Llama 70B Q4 · 43 GB
**Server**
V3.2 and full R1 · 671B total · multi-GPU/cloud
06 · Older reasoning baseline
### Keep the distills as reasoning baselines, not first picks.
The local R1 distills predate the current generation. Use R1‑0528‑Qwen3‑8B for compatibility tests, 32B for the useful middle and 70B on a 64 GB-class machine only when your acceptance set rewards them. DeepSeek‑V3.2 and full R1 are 671B MoE deployments and belong on multi-GPU servers or hosted endpoints.
**R1 releases**
1.5B
7B
8B
14B
32B
70B
671B full
Distilled variants are based on Qwen or Llama checkpoints; V3.2 is a separate 671B MoE family.
**Original R1 distill scaling**
DeepSeek-reported pass@1
- [AIME 2024](https://isaiuseful.com/benchmarks.html.md#aime)
**28.9**
1.5B
- [AIME 2024](https://isaiuseful.com/benchmarks.html.md#aime)
**69.7**
14B
- [AIME 2024](https://isaiuseful.com/benchmarks.html.md#aime)
**72.6**
32B
- [LiveCodeBench](https://isaiuseful.com/benchmarks.html.md#livecodebench)
**16.9**
1.5B
- [LiveCodeBench](https://isaiuseful.com/benchmarks.html.md#livecodebench)
**53.1**
14B
- [LiveCodeBench](https://isaiuseful.com/benchmarks.html.md#livecodebench)
**57.2**
32B
- [GitHub →](https://github.com/deepseek-ai/DeepSeek-R1)
- [Hugging Face 8B →](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B)
- [Ollama size ladder →](https://ollama.com/library/deepseek-r1/tags)
- [LM Studio model →](https://lmstudio.ai/models/deepseek/deepseek-r1-0528-qwen3-8b)
- [Open 8B in LM Studio →](https://lmstudio.ai/deeplink?name=deepseek-r1-0528-qwen3-8b&owner=deepseek)
- [DeepSeek‑R1 paper →](https://arxiv.org/abs/2501.12948)
- [V3.2 model card and scores →](https://huggingface.co/deepseek-ai/DeepSeek-V3.2)
- [DeepEP communication library →](https://github.com/deepseek-ai/DeepEP)
07
Mistral
Older compact local fallback; Small 4 is a separate server-class current option.
**16 GB**
Ministral 3 14B Q4 · roughly 9–10 GB
**24 GB**
Ministral 3 14B Q8 or Devstral 24B Q4
**32 GB**
Devstral Small 24B Q6/Q8 · package-dependent
**64 GB**
High-precision 24B; Small 4 119B needs aggressive low-bit quant
**128 GB**
Small 4 119B low-bit · runtime-specific; server-first
07 · Older compact fallback
### Ministral 3 is a compatibility fallback, not the local default.
The 3B, 8B and 14B Ministral models are compact, multimodal and permissively licensed. Mistral Small 4 combines instruct, reasoning and coding modes at 119B total/6.5B active; treat local low-bit builds as experimental until your runtime lists the exact architecture.
**Current main families**
Ministral 3 · 3B
8B
14B
Small 4 · 119B / 6.5B active
Large 3 · 675B / 41B active
Base, instruct and reasoning variants exist; Devstral is the coding-specialist companion line.
**Current GPQA evaluation metadata**
Separate Hugging Face model-card runs; not a controlled head-to-head
- [GPQA Diamond](https://isaiuseful.com/benchmarks.html.md#gpqa)
**71.2**
Small 4 119B
- [GPQA Diamond](https://isaiuseful.com/benchmarks.html.md#gpqa)
**67.2**
Large 3 675B
- [GitHub inference →](https://github.com/mistralai/mistral-inference)
- [Hugging Face Ministral 14B →](https://huggingface.co/mistralai/Ministral-3-14B-Instruct-2512)
- [Ollama tags →](https://ollama.com/library/ministral-3/tags)
- [LM Studio model →](https://lmstudio.ai/models/mistralai/ministral-3-14b)
- [Open 14B in LM Studio →](https://lmstudio.ai/deeplink?name=ministral-3-14b&owner=mistralai)
- [Ministral 3 paper →](https://arxiv.org/abs/2601.08584)
- [Small 4 card and score →](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603)
- [Large 3 card, score and deployment →](https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512)
- [Mistral fine-tuning toolkit →](https://github.com/mistralai/mistral-finetune)
08
Kimi / Moonshot
Older small research checkpoints locally; Kimi K3 is a current server-scale release.
**16 GB**
Moonlight 16B‑A3B · community 4-bit · 8K maximum context
**32 GB**
Moonlight 16B‑A3B at higher precision; still an 8K research model
**64 GB**
Kimi Linear 48B‑A3B community 4-bit · runtime-specific · test 32–128K first
**128 GB**
Kimi Linear 48B‑A3B BF16-class fit · leave room for context and runtime
**K3**
2.8T · 1M model limit · weights live · multi-GPU
08 · Older local research line
### Small Moonshot models are older research choices; Kimi K3 is server-scale.
**Kimi K3** now has released weights, but its 2.8T parameters, native vision and one-million-token model limit still make it a multi-GPU or hosted deployment—not a personal-machine recommendation. Moonlight 16B‑A3B and experimental Kimi Linear 48B‑A3B are the smaller downloadable Moonshot options; both predate the current generation and should be treated as research checkpoints, not miniature K3 substitutes.
**Do not collapse these into one size ladder**
Moonlight · 16B / 3B active · 8K
Kimi Linear · 48B / 3B active · 1M model limit
K2.6 · 1T / 32B active
K3 · 2.8T · 1M model limit
Kimi Linear uses custom code and a specialist long-context architecture. Its one-million-token support does not make one million tokens practical on a 64 or 128 GB personal machine.
**Small checkpoints are research choices**
Vendor-reported; not a K3 capability proxy
- [Moonlight MMLU‑Pro](https://isaiuseful.com/benchmarks.html.md#mmlu-pro)
**42.4**
16B‑A3B; 8K context
- [Moonlight HumanEval](https://isaiuseful.com/benchmarks.html.md#humaneval)
**48.1**
16B‑A3B
- [Kimi Linear MMLU‑Pro](https://isaiuseful.com/benchmarks.html.md#mmlu-pro)
**51.0**
48B‑A3B; evaluated at 4K
- [Kimi Linear RULER](https://isaiuseful.com/benchmarks.html.md#ruler)
**84.3**
48B‑A3B; evaluated at 128K
- [Official Kimi K3 launch and weight-release date →](https://www.kimi.com/blog/kimi-k3)
- [Play the embedded Kimi K3 introduction →](https://isaiuseful.com/cloud-models.html.md#kimi-k3)
- [Moonshot AI model overview →](https://www.moonshot.ai/)
- [Kimi K2.6 weights →](https://huggingface.co/moonshotai/Kimi-K2.6)
- [Moonlight 16B‑A3B model card →](https://huggingface.co/moonshotai/Moonlight-16B-A3B-Instruct)
- [Kimi Linear 48B‑A3B model card →](https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct)
- [Independent Estonian benchmark including Kimi K2 →](https://huggingface.co/tartuNLP/llama-estllm-prototype-0825)
09
Llama 3 + 4
Older ecosystem baseline with broad quant and compatibility support.
**16 GB**
Llama 3.1 8B Q8 or 3.2 3B high precision
**24 GB**
Llama 3.2 Vision 11B Q8 · package-dependent
**64 GB**
Llama 3.3 70B Q4 · 43 GB
**128 GB**
Llama 4 Scout 109B total · community low-bit quant
09 · Older ecosystem baseline
### Use Llama only when compatibility matters more than current capability.
Llama 3.1 8B remains a safe runtime test and Llama 3.3 70B is the clean 64 GB step. Llama 4 Scout is 109B total/17B active and may fit at 128 GB in community quants, but Meta’s official full-precision path is multi-GPU. Check the custom license and exact runtime.
**Shipped family sizes**
3 · 8B
70B
3.1 · 8B
70B
405B
3.2 · 1B
3B
11B vision
90B vision
3.3 · 70B
4 Scout · 109B / 17B active
4 Maverick · 400B / 17B active
**Llama 4 instruct scores**
Meta model card
- [MMLU‑Pro](https://isaiuseful.com/benchmarks.html.md#mmlu-pro)
**74.3**
Scout
- [MMLU‑Pro](https://isaiuseful.com/benchmarks.html.md#mmlu-pro)
**80.5**
Maverick
- [GPQA](https://isaiuseful.com/benchmarks.html.md#gpqa)
**57.2**
Scout
- [GPQA](https://isaiuseful.com/benchmarks.html.md#gpqa)
**69.8**
Maverick
- [LiveCodeBench](https://isaiuseful.com/benchmarks.html.md#livecodebench)
**32.8**
Scout
- [LiveCodeBench](https://isaiuseful.com/benchmarks.html.md#livecodebench)
**43.4**
Maverick
- [GitHub and model cards →](https://github.com/meta-llama/llama-models)
- [Hugging Face Llama 3.3 →](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct)
- [Ollama 70B tags →](https://ollama.com/library/llama3.3/tags)
- [LM Studio 8B →](https://lmstudio.ai/models/meta-llama/llama-3.1-8b)
- [Open 8B in LM Studio →](https://lmstudio.ai/deeplink?name=llama-3.1-8b&owner=meta-llama)
- [Llama 4 Scout card and scores →](https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct)
- [Llama Stack →](https://github.com/meta-llama/llama-stack)
- [Llama Guard 4, Prompt Guard 2 and LlamaFirewall →](https://ai.meta.com/blog/ai-defenders-program-llama-protection-tools/)
10
GPT‑OSS
Older MoE compatibility baseline; useful footprints, no longer a first recommendation.
**16 GB**
20B MXFP4 · 14 GB package · short-context edge fit
**24 GB**
20B MXFP4 · 14 GB · comfortable
**32 GB**
20B MXFP4 · space for longer context
**64 GB**
120B package is 65 GB; stay on 20B here
**128 GB**
120B MXFP4 · 65 GB package
10 · Older compatibility baseline
### Keep 20B and 120B for compatibility tests, not as defaults.
Both prior-year models use native MXFP4 MoE weights and the Harmony response format. Prefer the current top-three families unless GPT‑OSS wins the exact runtime, tool or footprint test. OpenAI says 20B can run within 16 GB and 120B within 80 GB; a 16 GB personal machine still has almost no margin after loading the 14 GB 20B package.
**All GPT‑OSS sizes**
20B · 3.6B active
120B · 5.1B active
**20B → 120B**
Current Hugging Face evaluation metadata
- [MMLU‑Pro](https://isaiuseful.com/benchmarks.html.md#mmlu-pro)
**73.6**
20B
- [MMLU‑Pro](https://isaiuseful.com/benchmarks.html.md#mmlu-pro)
**80.8**
120B
- [GPQA Diamond](https://isaiuseful.com/benchmarks.html.md#gpqa)
**58.6**
20B
- [GPQA Diamond](https://isaiuseful.com/benchmarks.html.md#gpqa)
**80.8**
120B
- [GitHub and reference runtimes →](https://github.com/openai/gpt-oss)
- [Hugging Face 20B and scores →](https://huggingface.co/openai/gpt-oss-20b)
- [Hugging Face 120B and scores →](https://huggingface.co/openai/gpt-oss-120b)
- [Ollama tags →](https://ollama.com/library/gpt-oss/tags)
- [LM Studio model →](https://lmstudio.ai/models/openai/gpt-oss-20b)
- [Open 20B in LM Studio →](https://lmstudio.ai/deeplink?name=gpt-oss-20b&owner=openai)
- [Official release and evaluation →](https://openai.com/index/introducing-gpt-oss/)
- [Model-card paper →](https://arxiv.org/abs/2508.10925)
- [GPT‑OSS Safeguard companion models →](https://openai.com/index/introducing-gpt-oss-safeguard/)
Also high-profile
## Four more families—and one separate specialist.
The four language-model families fill useful gaps in reasoning, enterprise licensing, open research and Estonian work. H3 sits outside the agent ranking because it generates audio and video.
16 GB · 14B Q4 9.1 GB
### Microsoft Phi‑4
A compact MIT-licensed reasoning model. Microsoft reports [MMLU](https://isaiuseful.com/benchmarks.html.md#mmlu-pro) 84.8, [GPQA](https://isaiuseful.com/benchmarks.html.md#gpqa) 56.1 and [HumanEval](https://isaiuseful.com/benchmarks.html.md#humaneval) 82.6 for the 14B base instruction model; reasoning, mini and multimodal Phi‑4 variants also ship.
**Sizes**
14B
mini
multimodal
reasoning
- [Phi Cookbook →](https://github.com/microsoft/PhiCookBook)
- [Hugging Face + paper + scores →](https://huggingface.co/microsoft/phi-4)
- [Ollama tags →](https://ollama.com/library/phi4/tags)
- [LM Studio model →](https://lmstudio.ai/models/microsoft/phi-4)
8–32 GB · Apache 2.0
### IBM Granite 4
Enterprise-oriented models with governance disclosures, hybrid Mamba/Transformer variants and compact footprints. Shipped language sizes include Micro, H‑Micro, H‑Tiny, H‑Small and dense 8B; evaluate the exact card because scores vary by variant.
**Sizes**
Micro
H‑Micro
H‑Tiny
H‑Small
8B
- [GitHub, sizes and disclosures →](https://github.com/ibm-granite/granite-4.0-language-models)
- [Hugging Face collection →](https://huggingface.co/collections/ibm-granite/granite-40-models)
- [Ollama tags →](https://ollama.com/library/granite4/tags)
- [Granite Guardian →](https://github.com/ibm-granite/granite-guardian)
16 / 32 GB · fully open research
### AI2 OLMo 3
A rare option with training code, data, checkpoints and detailed recipes. The family ships 7B and 32B Base, Instruct and Think variants with 65,536-token context; use 7B Q4/Q8 at 16 GB and 32B Q4 at 24–32 GB.
**Sizes**
7B
32B
Base
Instruct
Think
- [GitHub training code →](https://github.com/allenai/OLMo)
- [Hugging Face card and scores →](https://huggingface.co/allenai/Olmo-3-1025-7B)
- [OLMo 3 paper →](https://allenai.org/papers/olmo3)
- [Ollama tags →](https://ollama.com/library/olmo3/tags)
Estonian specialist · prototype
### TartuNLP EstLLM 8B
A locally testable Estonian-focused research checkpoint and evaluation suite. The authors clearly label it an early prototype with 4K context and no multi-turn chat; use it as a benchmark and fine-tuning reference, not an automatic production default.
**Size**
8B
BF16 checkpoint
community quants
- [Understand the Estonian benchmark suite →](https://isaiuseful.com/benchmarks.html.md#ifeval)
- [Model card and Estonian tables →](https://huggingface.co/tartuNLP/llama-estllm-prototype-0825)
- [EstLLM paper →](https://arxiv.org/abs/2603.02041)
- [Open the EKI benchmark →](https://mõõdupuu.eki.ee/benchmark/bib_bench)
128 GB Spark specialist · video + stereo audio
### MiniMax H3 is local—but it is not an assistant LLM or a simple BF16 load.
Its local H3‑Base takes text, images, video and audio and generates 4–15-second, 24 fps video with native 32 kHz stereo audio. One released BF16 FL2VA task package is about 134.2 GiB on disk—already beyond Spark’s 128 GB unified capacity before activations. NVIDIA’s single-Spark path fits by using a pruned FP8 DiT, FP8 Qwen3‑VL conditioner and the released VAEs; its vendor benchmark cuts one 5-second, 480p, 50-step job from 710.6 to 181.3 seconds. That is strong fit and optimization evidence, not real-time generation.
The fully local release stops at 768p H3‑Base. MiniMax’s recommended 2K result still calls hosted Context‑IR and Regenerate‑2K modules, and the initial open release omits its native sparse-attention implementation. The community license also excludes the EU, UK, United States and South Korea; users there need a separate MiniMax license rather than treating the public weights as permission to run them.
**Released route**
33B dense DiT
4–15 seconds
768p local base
24 fps
32 kHz stereo
181.3 s Spark benchmark
- [Official model card and local workflows →](https://huggingface.co/MiniMaxAI/MiniMax-H3)
- [NVIDIA GB10 runtime, precision and config →](https://github.com/NVlabs/Sana/tree/sol-engine/models/minimax_h3/GB10)
- [NVIDIA on-device benchmark →](https://nvlabs.github.io/Sana/Sol-Engine/H3-OnDevice/)
- [Read the territorial and commercial license terms →](https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/LICENSE)
- [Request a license for an excluded territory →](https://platform.minimax.io/h3-license)
- [Read the Spark-specific decision note →](https://isaiuseful.com/remote-spark.html.md#minimax-h3-spark)
Capability evidence
## Benchmarks shortlist. Work samples decide.
Compare sizes inside the same family and table first. Then test the failure that costs you money: wrong tool calls, incomplete edits, weak language, invented citations or human cleanup.
Good comparison
### Same family, same card
Gemma 4 E2B → 12B → 31B on the same [MMLU‑Pro](https://isaiuseful.com/benchmarks.html.md#mmlu-pro) table shows a meaningful capability curve.
Qualified comparison
### Same named benchmark
Check version, prompt, pass@1 versus consensus, tool access and whether the result is vendor-reported.
Required comparison
### Your 30–100 tasks
Score multilingual quality, tool selection, argument accuracy, code edits, recovery and the amount of human correction required.
European multilingual proof
### Estonian is the demo, not the border.
Estonian is a useful low-resource, morphologically rich stress test. TartuNLP’s independent table reports [IFEval‑et](https://isaiuseful.com/benchmarks.html.md#ifeval) of 0.756 for Gemma 3 12B and 0.766 for 27B; Llama 3.3 70B scores 0.771 and Kimi K2 0.789. Gemma 3 27B also posts 0.817 on [Grammar‑et](https://isaiuseful.com/benchmarks.html.md#ifeval) and 0.953 on [Word‑Meanings‑et](https://isaiuseful.com/benchmarks.html.md#ifeval) . That is impressive evidence for the candidate list, while every production language pair still needs its own prompts.
**0.756**
Gemma 3 12B ·
- [IFEval‑et](https://isaiuseful.com/benchmarks.html.md#ifeval)
**0.766**
Gemma 3 27B ·
- [IFEval‑et](https://isaiuseful.com/benchmarks.html.md#ifeval)
**0.817**
Gemma 3 27B ·
- [Grammar‑et](https://isaiuseful.com/benchmarks.html.md#ifeval)
**0.953**
Gemma 3 27B ·
- [Word‑Meanings‑et](https://isaiuseful.com/benchmarks.html.md#ifeval)
Inference layer
## First decide how the model will run.
The inference runtime loads weights, allocates context and exposes an API. It is not the agent harness. Confirm acceleration, chat template, structured output and tool-call format before connecting an agent.
Inspect + prototype
### LM Studio
Best first stop for discovery, MLX/GGUF downloads, chat-template inspection and an OpenAI-compatible local API. Check both configured and model-maximum context; they are separate values.
- [LM Studio →](https://lmstudio.ai/)
Simple local service
### Ollama
Fast route to a reproducible pull command and local API for coding harnesses. Inspect the exact tag and context setting: “latest” can hide quantization and file-size differences.
- [Ollama →](https://ollama.com/)
Apple-native
### MLX + llama.cpp
Use MLX when an Apple-silicon-native package exists; use llama.cpp for explicit GGUF control, broader quant choices and portable serving.
- [MLX LM →](https://github.com/ml-explore/mlx-lm)
- [llama.cpp →](https://github.com/ggml-org/llama.cpp)
Workstation/server
### vLLM or SGLang
Prefer these on supported NVIDIA systems when concurrency, continuous batching, tensor parallelism and an always-on endpoint matter.
- [vLLM →](https://docs.vllm.ai/)
- [SGLang →](https://docs.sglang.ai/)
Agent + specification layers
## Then choose who plans, acts and preserves intent.
A harness owns the loop around the model: tools, memory, permissions and retries. A specification workflow stores an agreed plan outside the conversation. Both can call a local inference API; neither makes a weak model reason like a frontier model.
> Visual: Separation between inference, harness and specification workflow
**Visual entries (display order):**
- 01 · inference **LM Studio · Ollama · llama.cpp · vLLM** Loads the model and serves tokens.
- 02 · agent harness **Hermes Agent · OpenClaw** Runs tools, sessions, memory and approval gates.
- 03 · durable intent **OpenSpec · Specflow** Keeps plans, tasks and acceptance criteria outside chat history.
Agent harness
### Hermes Agent
Use Hermes when you want a general agent loop with tools, skills, memory and multiple model-provider endpoints. Point it at the inference server; then evaluate the model through Hermes rather than assuming chat quality transfers.
- [Hermes Agent repository →](https://github.com/nousresearch/hermes-agent)
Agent harness
### OpenClaw
Use OpenClaw for an always-on personal agent, channels and remote clients. Keep tool authority narrower than model capability, authenticate the gateway and treat local inference as one backend—not the harness itself.
- [OpenClaw →](https://openclaw.ai/)
Specification workflow
### OpenSpec
Stores a proposal, tasks and spec deltas in the repository before implementation. This is a clean hand-off: use a stronger model to settle the plan, then give a smaller worker the approved artifacts and relevant files.
- [OpenSpec repository →](https://github.com/Fission-AI/OpenSpec)
Planning methodology
### Specflow
SpecStory’s open methodology moves from intent to roadmap, workplans, execution and refinement. It helps preserve decisions between agent sessions; it does not serve a model or enforce an inference format.
- [Specflow repository →](https://github.com/specstoryai/specflow)
Scale route
## Move from personal memory to a deskside frontier tier.
RTX Spark is offered for slim laptops and small desktops; DGX Spark is a compact dedicated system; DGX Station adds a 748 GB coherent tier. What matters is available memory, memory bandwidth, backend support and whether the model runs on the client or a remote host.
Personal computer
**16–64 GB Mac, laptop or desktop**
Interactive use, private documents and one-user local APIs. LM Studio, Ollama, MLX or llama.cpp.
Spark-class computer · native NVFP4
**128 GB · ≈150–165B planning fit**
The speculative band reserves 20–25% for runtime and cache; NVIDIA's up-to-200B figure is a tighter capacity ceiling. Validate the exact GB10 kernel.
Linked workstations · native NVFP4
**256 GB · ≈295–330B planning fit**
NVIDIA’s ConnectX path advertises up to 405B and demonstrates Qwen‑235B NVFP4. The lower band keeps practical headroom; software and model architecture still decide whether it works.
Station, cluster or cloud · native support varies
**≈860–960B Station planning fit**
DGX Station can plausibly hold a mixed-NVFP4 model in that band across coherent memory. Blackwell datacenter nodes and racks scale further; H100/H200 do not provide native NVFP4 W4A4.
- [Compare licensable models and NVFP4 servers →](https://isaiuseful.com/cloud-models.html.md#nvfp4)
Separate tier · DGX Station
### 748 GB makes the official NVFP4 GLM‑5.2 a single-node fit—not a guaranteed fast one.
Station combines 252 GB of 7.1 TB/s HBM3e with 496 GB of 396 GB/s LPDDR5X. NVIDIA's 465 GB GLM‑5.2 NVFP4 checkpoint fits across the coherent pool, but spills beyond HBM and still needs runtime and cache headroom. The linked Station measurements belong to GLM‑5.2; GLM‑5.3 weights and serving recipes were not public when checked. Station is most compelling when 70–300B models live in HBM, or when local access to a larger low-bit model is more valuable than cloud-scale throughput.
**Sweet-spot rule**
Use Spark for 20–35B dense models and sparse 30–120B MoEs. Consider Station for daily 70–120B high-precision work, 200–400B low-bit work, or controlled 400B–1T experiments. Rent first when that top tier is occasional.
- [Open the Station buyer's guide →](https://isaiuseful.com/dgx-station.html.md)
- [Compare Spark and Station →](https://isaiuseful.com/dgx-station.html.md#comparison)
- [Check the independent evidence gap →](https://isaiuseful.com/dgx-station.html.md#reviews)
**Parameter ceilings are not memory guarantees.** NVFP4 planning bands reserve 20–25% of advertised memory and assume roughly 5.0–5.2 effective bits per stored parameter. Architecture, mixed-precision layers, KV cache, multimodal towers, memory tier and backend support can move the result. A model package can load while leaving too little memory for useful context, tool traffic or concurrency.
- [NVIDIA NVFP4 format and Blackwell support →](https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/)
- [Official RTX Spark laptops and desktops →](https://www.nvidia.com/en-us/products/rtx-spark/)
- [Official DGX Spark specifications →](https://www.nvidia.com/en-us/products/workstations/dgx-spark/)
- [Official DGX Station specifications →](https://www.nvidia.com/en-eu/products/workstations/dgx-station/)
- [Price NVFP4-capable servers →](https://isaiuseful.com/cloud-models.html.md#nvfp4)
- [Operate a remote Spark safely →](https://isaiuseful.com/remote-spark.html.md)
Method
## What “compatible” means here.
The core catalogue was checked against official model cards, repositories and current Ollama/LM Studio packages in July 2026; dated launch records and specialist routes were rechecked through 15 August 2026.
### Agent-safe
The default package leaves roughly 20–25% of system memory plus a practical starting context target. It is a planning rule, not a throughput guarantee.
### Tight
The weights can load, but context, parallel requests or other applications may trigger swap or out-of-memory errors. Tight fits are labelled in amber and should start with short context.
### Server
The official model card expects multi-GPU or vendor-specific infrastructure, or no verified LM Studio/Ollama local package was found. Cloud-only Ollama listings are not counted as local support.
### Quantization
Q4 sizes are package-level approximations unless the vendor publishes a memory table. QAT, GGUF Q4_K_M, MLX 4-bit, NVFP4 and MXFP4 are not interchangeable. Native NVFP4 W4A4 requires Blackwell-generation Tensor Cores plus a supported kernel; Hopper fallback is not the same compute path.
**Nemotron 3.5 Lightning** weights, licence, model-card scores, repository sizes, DGX Spark recipe and routing evidence were checked on 12 August 2026; NVIDIA’s speed figures and the separate LangChain routing study remain workload-specific evidence. Muse Glimmer’s official model card and GGUF files were rechecked on 15 August 2026; Meta’s RTX 5090 speed data and two early DGX Spark owner reports remain workload- and configuration-specific, with no standardized third-party Muse benchmark yet. Other fast-moving names and the NVFP4 hardware boundary were rechecked on 26 July 2026. **GLM‑5.2** is the available open 753B server-class release; NVIDIA's official 465 GB NVFP4 checkpoint and the creator-run Station measurements are the current local evidence. **GLM‑5.3** availability, API migration requirements and vendor benchmark methods were checked on 15 August 2026; its weights, licence and local serving recipes were not yet public. **Kimi K3** now has downloadable weights as well as product and API access, but its 2.8T scale still keeps it in the hosted or multi-GPU tier; the smaller Moonshot checkpoints are older research lines, not K3 substitutes. Benchmark scores are vendor-reported unless an independent source is named.
The short answer
Gemma 4 first for the strongest current local family. Muse Glimmer second for portable agents. Qwen3.8 third for coding and breadth. Nemotron and GLM cover specialist routes. Keep GPT‑OSS, Llama and the older local lines as baselines—not defaults. Keep room for the work.
- [Choose an agent role](#roles)
- [Design the hybrid route](#routing)
- [Budget the context](#context)
- [Scale to a model cloud](https://isaiuseful.com/cloud-models.html.md)
- [Choose a workflow](https://isaiuseful.com/guides.html.md#chooser)