A model package loading is only the first gate. A useful local agent also needs memory and tokens for instructions, tool schemas, repository files, tool results, reasoning and the answer. Choose the job first, then balance capability, context and memory.
Size is a rough capability band, not an IQ score. Architecture and post-training can move a model up or down; your quant, runtime and tool grammar can move it again.
Good for extraction, classification, rewriting, translation checks, test generation, search-query drafting and one well-described tool call at a time.
Do not make it the unsupervised owner of ambiguous multi-file changes. Retrieval can supply missing facts; it cannot supply judgment the model does not have.
Suitable for bounded multi-file edits, debugging with a reproduction, test-driven implementation and a small set of reliable tools. Keep a human checkpoint before migrations, security changes or broad refactors.
For an ambiguous feature, let a 70–120B or strong hosted model produce the plan and review criteria. Save that plan as a file, compact or start a clean task, then let the cheaper 20–35B worker implement it.
A focused MCP server can give a modest model live facts and deterministic actions. Use allow-listed tools with distinct names, small schemas and bounded results; require citations or structured evidence.
Every schema and result consumes context. Too many similar tools increase selection errors, while a large search dump can crowd out the question. Route only the relevant tool set to each agent.
Use the strongest model for architecture, unfamiliar code, competing constraints, failure analysis and final review. A stronger planner can reduce expensive wandering by several weaker implementers.
Large does not mean infallible or “equivalent to Sonnet.” Keep acceptance tests, explicit constraints and human review. At 128 GB, context headroom may be worth more than squeezing in a larger low-bit quant.
Routing, tagging, format conversion, autocomplete and tightly constrained extraction.
Reliable planning, fuzzy tool choice, long chains and precise multilingual nuance.
Grounded Q&A, drafting, single-file code, one or two simple tools and supervised workers.
Autonomous repo-wide work, conflicting requirements and recovery after several failed steps.
Everyday coding agents, multi-document synthesis, structured research and several focused tools.
High-stakes judgment, novel architecture and long autonomous runs without tests or checkpoints.
Planning, review, ambiguous debugging, cross-file refactors and coordinating specialist workers.
Guaranteed frontier parity, perfect long-context recall or unattended production changes.
Hosted or multi-GPU frontier open-weight work when quality justifies infrastructure. Plan the provider tier →
Single 128 GB machines; active MoE parameters do not remove the need to store and load the full checkpoint.
Use when exact phrasing, code reliability or multilingual nuance matters and a smaller high-precision model already meets the task.
The normal quality/size starting point. Four-bit compression usually preserves broad ability, but agentic application accuracy can degrade more than perplexity suggests.
Useful when it unlocks a meaningfully larger model. Re-run tool use, structured output and language tests; subtle failures appear before chat becomes obviously bad.
Use only after testing the exact build on your tasks. Very-low-bit quants can make a larger model load, but tool calls, structured output, code edits and multilingual nuance may degrade unevenly.
A local model does not need to replace the frontier model. Let the strongest model resolve ambiguity, move routine execution to a cheaper specialist, and escalate when the trace shows difficulty.
Settles architecture, constraints, review criteria and unfamiliar failure modes.
Selects a target for each turn from task, policy, cost, latency and recent tool signals.
Local or hosted workers handle bounded calls; repeated errors, loops or uncertainty route back up.
NVIDIA’s 30B-total / 3B-active hybrid MoE targets high-volume tool calls, validation and delegated execution. The official NVFP4 checkpoint is about 20.1 GiB before runtime and cache, making 32 GB the sensible personal-system floor.
Name targets such as planner, private-worker and frontier-review. Map those aliases to a local Gemma/Muse/Qwen-class endpoint, a European hosted service or a frontier API without rewriting application logic.
Benchmark task success, total workflow cost, completion time, retries and frontier-escalation frequency. Include human review and failed runs so a cheap but unreliable route cannot look efficient.
NeMo Switchyard provides provider translation plus classifier, stage, escalation and custom routing. Its repository labels the software pre-alpha and not for production use; isolate it behind a gateway contract, pin a revision and keep a tested fallback.
Remote access and local inference are separate decisions. The phone can carry the conversation and approvals while an authenticated gateway, model and tools stay on an always-on machine you control.
Collects the request, shows progress and asks for approval. It may be only a control surface.
Authenticates the client, owns the agent loop and decides which backend and capabilities are available.
Loads weights and serves inference on the home machine, workstation or private server.
Run both on the host, make only the authenticated WebUI reachable over a private route and open it in the phone browser. Keep Ollama itself on localhost or an internal network.
Open WebUI documentation →Keep the gateway and session state on the always-on host, then pair an iPhone or Android companion. Start with remote chat; add device-node capabilities only when the workflow needs them.
OpenClaw remote access →Pair Orca's iPhone or Android companion with its desktop runtime to see worktrees, agent status and session output, then reply or dictate from the phone. The desktop remains the source of truth.
Orca mobile companion →Reach the agent through a supported messaging channel while its model runs elsewhere. Treat that channel provider as part of the data path, even when inference stays at home.
Hermes messaging gateway →These defaults assume one interactive agent, a Q4-class model, the OS and runtime, plus useful KV-cache headroom. “Chat fit” can be larger; “agent fit” must survive tool use and growing context.
16 GB agent fit: 7–12B Q4 with 8–16K context. A 14 GB model may load, but leaves almost no runway for a harness.
These are working configurations, not weight-only fits. Context is a conservative configured starting point for one interactive agent; increase it only after watching memory and prompt-processing time.
Best first candidate for multilingual drafting, retrieval-grounded work and bounded code changes.
Open family record →A compact coding and tool-use baseline with broad runtime packaging.
Open family record →The small Moonshot option. It is useful for experiments, but it is not a miniature Kimi K3.
Open family record →Retained for compatibility testing. It can load in 16 GB, but leaves almost no margin and is not a current first pick.
Open family record →Looking for the viral Kimi demo? Kimi K3 is a 2.8T server-scale model, not a 16 GB download. The Moonshot card above is an older small research checkpoint. Plan the provider tier →
Agent-safe start: Gemma 4 12B QAT Q4 (about 6.7–7.4 GB) or Qwen3.5 9B Q4 (6.6 GB). Moonlight 16B‑A3B is a small Moonshot research option in community 4-bit builds, but its model limit is only 8K.
Chat fit is not agent fit: GPT‑OSS 20B is designed to run within 16 GB and its Ollama package is 14 GB, but that leaves almost no margin for macOS, long context or tool results.
Current start: Gemma 4 12B at Q6/Q8 or Muse Glimmer’s official 17 GB quant. Gemma 4 26B‑A4B Q4 at 14.4 GB is also viable when the runtime supports it well; treat 24 GB Muse as a tighter agent configuration.
Tight: Qwen3.8‑27B FP8 (16.38 GB; supported serving backends), Qwen3.6 27B Q4 (17 GB; broader desktop packaging), GLM‑4.7‑Flash (19 GB) and 32B Q4 packages load on paper but sacrifice the context and tool runway that makes a harness useful.
Ranked start: Gemma 4 26B‑A4B Q4 first, Muse Glimmer 30B second and Qwen3.8‑27B FP8 third. Qwen remains the coding-heavy choice on a supported backend and leaves about 15 GB before runtime allocation.
Fallback route: GLM‑4.7‑Flash Q4 and older broadly packaged models remain compatibility options where current Gemma, Muse or Qwen builds are unavailable.
Ranked start: Gemma 4 31B for the strongest current family fit, Muse Glimmer for agent tools and review, then Qwen3.8‑27B FP8 for coding-heavy work and parallel sequences. Use the older DeepSeek R1 Distill 70B only when a reasoning acceptance set proves the trade.
Do not spend memory for its own sake: Qwen says the FP8 package’s performance is nearly identical to the original model. The 55.56 GB full checkpoint is a fidelity test route, not the default 64 GB configuration.
Ranked start: run Gemma 4 at high precision, Muse Glimmer for one or more local agents, or Qwen3.8‑27B FP8 with parallel headroom. Keep GPT‑OSS 120B as an older compatibility baseline only when the exact acceptance set rewards its 65 GB package.
Qwen3.5 122B‑A10B Q4 (81 GB) remains a capability-first stretch, not the automatic Qwen default. Choose it only when your tasks beat Qwen3.8‑27B by enough to justify the lost context and concurrency runway.
Parallel agents can share one loaded model, but each active sequence adds KV-cache pressure and its own growing history.
A model may advertise 128K–1M while the runtime loads 8K. Raising it costs memory and prompt-processing time.
Start here, then compare Q6/Q8 or a larger Q3 only on a repeatable tool, code and language test set.
Task success, retries and human corrections matter more than tokens per second or a leaderboard point.
The configured window must hold the harness, system and project instructions, MCP schemas, conversation, retrieved files, tool results, hidden reasoning where applicable and the next answer.
The training limit: often 128K, 256K or more. It says nothing about your runtime setting, speed or memory fit.
A runtime can be set to 8K even when the model supports far more. The inference server and agent client may also impose separate limits.
Configured window minus instructions, tool definitions, history, output reserve and safety buffer. This is the number to plan around.
The initial slice fits. The remaining range is roughly 1 additional 16 KiB tool-result batch, or 445–1,039 more nonblank code lines before compaction.
Count the files and relevant lines the agent must see now. Search and retrieval can fetch more later; dumping the full tree spends context before work begins.
A long command result may cost more than the prompt that caused it. Prefer focused tests, clipped logs and fresh reads over carrying every failed attempt.
Let a stronger model resolve ambiguity and write the plan plus acceptance criteria. Start the worker with that artifact and only the files needed for the next task.
Families are ordered by current recommendation, not download size: Gemma 4 first, Muse Glimmer second and Qwen3.8 third. An older label marks the relevant local line as pre‑2026 and keeps it for compatibility or research—not as a first pick. Fit chips still show where a package can be practical, not where its maximum context will fit.
Google reports pre-training across 140+ languages and out-of-the-box support for 35+. Independent TartuNLP evaluation of Gemma 3 found strong Estonian instruction following, grammar and word-meaning results at 12B and 27B. That makes Estonian an excellent stress test for the family—not a promise for every language or automatic evidence for Gemma 4.
Muse Glimmer 30B is a dense model distilled from Muse Spark and released under Apache 2.0 for agentic work, tool use, coding, multilingual tasks and text-plus-image input. Meta publishes a 131,072+ model context and training across more than 100 languages. The official GGUF repository contains a 16.76 GB K‑Quant‑17GB build and a 19.65 GB dynamic build; image input needs the separate 1.40 GB perception file, while the optional 1.63 GB DFlash drafter accelerates generation without changing accepted output. Two early DGX Spark owner runs measured 5–10.5 tok/s without DFlash and about 23–38 tok/s with the official drafter set to 15 speculative tokens; useful evidence, but not a standardized cross-engine benchmark. Those file sizes make 24–32 GB systems credible targets, but they do not make the full context cheap or guarantee that every launch-day client understands the new architecture.
It handles text, images and video, can switch thinking on or off, and ships in a 16.38 GB FP8 build. Start with 32 GB for one useful local agent; 64 GB gives you more context or parallel work. Qwen reports that the FP8 build performs close to the full 55.56 GB checkpoint, but test it on your own tasks.
Nemotron 3.5 Lightning is a 30B-total / 3B-active hybrid MoE released for high-volume agent execution. Its official NVFP4 repository is about 20.1 GiB, and NVIDIA documents DGX Spark, Jetson and GeForce RTX 5090 routes plus data-centre serving. Treat 32 GB as the practical personal-system floor: runtime state, its optional 1.26 GiB DSpark drafter and useful context still need memory. The model’s OpenMDW 1.1 terms are not the earlier NVIDIA Open Model License.
GLM‑5.3 is Z.ai’s newer model for long-running coding and agent tasks. It is available through Coding Plan, while the general API and downloadable weights are still pending. If you need to run GLM yourself today, use GLM‑5.2; its official 465 GB NVFP4 build is aimed at DGX Station-class hardware. On a normal local machine, GLM‑4.7‑Flash is the practical choice.
Do not change only the model ID. Z.ai says thinking must remain enabled: replace thinking.type: "disabled" with "enabled", then choose reasoning_effort low, high or max. Start existing non-thinking workloads at low; use max for coding acceptance tests. A disabled-thinking request will fail.
The local R1 distills predate the current generation. Use R1‑0528‑Qwen3‑8B for compatibility tests, 32B for the useful middle and 70B on a 64 GB-class machine only when your acceptance set rewards them. DeepSeek‑V3.2 and full R1 are 671B MoE deployments and belong on multi-GPU servers or hosted endpoints.
The 3B, 8B and 14B Ministral models are compact, multimodal and permissively licensed. Mistral Small 4 combines instruct, reasoning and coding modes at 119B total/6.5B active; treat local low-bit builds as experimental until your runtime lists the exact architecture.
Kimi K3 now has released weights, but its 2.8T parameters, native vision and one-million-token model limit still make it a multi-GPU or hosted deployment—not a personal-machine recommendation. Moonlight 16B‑A3B and experimental Kimi Linear 48B‑A3B are the smaller downloadable Moonshot options; both predate the current generation and should be treated as research checkpoints, not miniature K3 substitutes.
Llama 3.1 8B remains a safe runtime test and Llama 3.3 70B is the clean 64 GB step. Llama 4 Scout is 109B total/17B active and may fit at 128 GB in community quants, but Meta’s official full-precision path is multi-GPU. Check the custom license and exact runtime.
Both prior-year models use native MXFP4 MoE weights and the Harmony response format. Prefer the current top-three families unless GPT‑OSS wins the exact runtime, tool or footprint test. OpenAI says 20B can run within 16 GB and 120B within 80 GB; a 16 GB personal machine still has almost no margin after loading the 14 GB 20B package.
The four language-model families fill useful gaps in reasoning, enterprise licensing, open research and Estonian work. H3 sits outside the agent ranking because it generates audio and video.
A compact MIT-licensed reasoning model. Microsoft reports MMLU 84.8, GPQA 56.1 and HumanEval 82.6 for the 14B base instruction model; reasoning, mini and multimodal Phi‑4 variants also ship.
Enterprise-oriented models with governance disclosures, hybrid Mamba/Transformer variants and compact footprints. Shipped language sizes include Micro, H‑Micro, H‑Tiny, H‑Small and dense 8B; evaluate the exact card because scores vary by variant.
A rare option with training code, data, checkpoints and detailed recipes. The family ships 7B and 32B Base, Instruct and Think variants with 65,536-token context; use 7B Q4/Q8 at 16 GB and 32B Q4 at 24–32 GB.
A locally testable Estonian-focused research checkpoint and evaluation suite. The authors clearly label it an early prototype with 4K context and no multi-turn chat; use it as a benchmark and fine-tuning reference, not an automatic production default.
Compare sizes inside the same family and table first. Then test the failure that costs you money: wrong tool calls, incomplete edits, weak language, invented citations or human cleanup.
Gemma 4 E2B → 12B → 31B on the same MMLU‑Pro table shows a meaningful capability curve.
Check version, prompt, pass@1 versus consensus, tool access and whether the result is vendor-reported.
Score multilingual quality, tool selection, argument accuracy, code edits, recovery and the amount of human correction required.
Estonian is a useful low-resource, morphologically rich stress test. TartuNLP’s independent table reports IFEval‑et of 0.756 for Gemma 3 12B and 0.766 for 27B; Llama 3.3 70B scores 0.771 and Kimi K2 0.789. Gemma 3 27B also posts 0.817 on Grammar‑et and 0.953 on Word‑Meanings‑et. That is impressive evidence for the candidate list, while every production language pair still needs its own prompts.
The inference runtime loads weights, allocates context and exposes an API. It is not the agent harness. Confirm acceleration, chat template, structured output and tool-call format before connecting an agent.
Best first stop for discovery, MLX/GGUF downloads, chat-template inspection and an OpenAI-compatible local API. Check both configured and model-maximum context; they are separate values.
LM Studio →Fast route to a reproducible pull command and local API for coding harnesses. Inspect the exact tag and context setting: “latest” can hide quantization and file-size differences.
Ollama →Use MLX when an Apple-silicon-native package exists; use llama.cpp for explicit GGUF control, broader quant choices and portable serving.
Prefer these on supported NVIDIA systems when concurrency, continuous batching, tensor parallelism and an always-on endpoint matter.
A harness owns the loop around the model: tools, memory, permissions and retries. A specification workflow stores an agreed plan outside the conversation. Both can call a local inference API; neither makes a weak model reason like a frontier model.
Loads the model and serves tokens.
Runs tools, sessions, memory and approval gates.
Keeps plans, tasks and acceptance criteria outside chat history.
Use Hermes when you want a general agent loop with tools, skills, memory and multiple model-provider endpoints. Point it at the inference server; then evaluate the model through Hermes rather than assuming chat quality transfers.
Hermes Agent repository →Use OpenClaw for an always-on personal agent, channels and remote clients. Keep tool authority narrower than model capability, authenticate the gateway and treat local inference as one backend—not the harness itself.
OpenClaw →Stores a proposal, tasks and spec deltas in the repository before implementation. This is a clean hand-off: use a stronger model to settle the plan, then give a smaller worker the approved artifacts and relevant files.
OpenSpec repository →SpecStory’s open methodology moves from intent to roadmap, workplans, execution and refinement. It helps preserve decisions between agent sessions; it does not serve a model or enforce an inference format.
Specflow repository →RTX Spark is offered for slim laptops and small desktops; DGX Spark is a compact dedicated system; DGX Station adds a 748 GB coherent tier. What matters is available memory, memory bandwidth, backend support and whether the model runs on the client or a remote host.
Interactive use, private documents and one-user local APIs. LM Studio, Ollama, MLX or llama.cpp.
The speculative band reserves 20–25% for runtime and cache; NVIDIA's up-to-200B figure is a tighter capacity ceiling. Validate the exact GB10 kernel.
NVIDIA’s ConnectX path advertises up to 405B and demonstrates Qwen‑235B NVFP4. The lower band keeps practical headroom; software and model architecture still decide whether it works.
DGX Station can plausibly hold a mixed-NVFP4 model in that band across coherent memory. Blackwell datacenter nodes and racks scale further; H100/H200 do not provide native NVFP4 W4A4.
Compare licensable models and NVFP4 servers →Station combines 252 GB of 7.1 TB/s HBM3e with 496 GB of 396 GB/s LPDDR5X. NVIDIA's 465 GB GLM‑5.2 NVFP4 checkpoint fits across the coherent pool, but spills beyond HBM and still needs runtime and cache headroom. The linked Station measurements belong to GLM‑5.2; GLM‑5.3 weights and serving recipes were not public when checked. Station is most compelling when 70–300B models live in HBM, or when local access to a larger low-bit model is more valuable than cloud-scale throughput.
Use Spark for 20–35B dense models and sparse 30–120B MoEs. Consider Station for daily 70–120B high-precision work, 200–400B low-bit work, or controlled 400B–1T experiments. Rent first when that top tier is occasional.
Parameter ceilings are not memory guarantees. NVFP4 planning bands reserve 20–25% of advertised memory and assume roughly 5.0–5.2 effective bits per stored parameter. Architecture, mixed-precision layers, KV cache, multimodal towers, memory tier and backend support can move the result. A model package can load while leaving too little memory for useful context, tool traffic or concurrency.
The core catalogue was checked against official model cards, repositories and current Ollama/LM Studio packages in July 2026; dated launch records and specialist routes were rechecked through 15 August 2026.
The default package leaves roughly 20–25% of system memory plus a practical starting context target. It is a planning rule, not a throughput guarantee.
The weights can load, but context, parallel requests or other applications may trigger swap or out-of-memory errors. Tight fits are labelled in amber and should start with short context.
The official model card expects multi-GPU or vendor-specific infrastructure, or no verified LM Studio/Ollama local package was found. Cloud-only Ollama listings are not counted as local support.
Q4 sizes are package-level approximations unless the vendor publishes a memory table. QAT, GGUF Q4_K_M, MLX 4-bit, NVFP4 and MXFP4 are not interchangeable. Native NVFP4 W4A4 requires Blackwell-generation Tensor Cores plus a supported kernel; Hopper fallback is not the same compute path.
Gemma 4 first for the strongest current local family. Muse Glimmer second for portable agents. Qwen3.8 third for coding and breadth. Nemotron and GLM cover specialist routes. Keep GPT‑OSS, Llama and the older local lines as baselines—not defaults. Keep room for the work.