Choose the job first. Then budget capability, context and memory for instructions, tools, source files, results, reasoning and the answer.
Size is a rough capability band, not an IQ score. Architecture and post-training can move a model up or down; your quant, runtime and tool grammar can move it again.
Good for extraction, classification, rewriting, translation checks, test generation, search-query drafting and one well-described tool call at a time.
Do not make it the unsupervised owner of ambiguous multi-file changes. Retrieval can supply missing facts; it cannot supply judgment the model does not have.
Suitable for bounded multi-file edits, debugging with a reproduction, test-driven implementation and a small set of reliable tools. Keep a human checkpoint before migrations, security changes or broad refactors.
For an ambiguous feature, let a 70-120B or strong hosted model produce the plan and review criteria. Save that plan as a file, compact or start a clean task, then let the cheaper 20-35B worker implement it.
A focused MCP server can give a modest model live facts and deterministic actions. Use allow-listed tools with distinct names, small schemas and bounded results; require citations or structured evidence.
Every schema and result consumes context. Too many similar tools increase selection errors, while a large search dump can crowd out the question. Route only the relevant tool set to each agent.
Use the strongest model for architecture, unfamiliar code, competing constraints, failure analysis and final review. A stronger planner can reduce expensive wandering by several weaker implementers.
Large does not mean infallible or “equivalent to Sonnet.” Keep acceptance tests, explicit constraints and human review. At 128 GB, context headroom may be worth more than squeezing in a larger low-bit quant.
Routing, tagging, format conversion, autocomplete and tightly constrained extraction.
Reliable planning, fuzzy tool choice, long chains and precise multilingual nuance.
Grounded Q&A, drafting, single-file code, one or two simple tools and supervised workers.
Autonomous repo-wide work, conflicting requirements and recovery after several failed steps.
Everyday coding agents, multi-document synthesis, structured research and several focused tools.
High-stakes judgment, novel architecture and long autonomous runs without tests or checkpoints.
Planning, review, ambiguous debugging, cross-file refactors and coordinating specialist workers.
Guaranteed frontier parity, perfect long-context recall or unattended production changes.
Hosted or multi-GPU frontier open-weight work when quality justifies infrastructure. Plan the provider tier →
Single 128 GB machines; active MoE parameters do not remove the need to store and load the full checkpoint.
Use when exact phrasing, code reliability or multilingual nuance matters and a smaller high-precision model already meets the task.
The normal quality/size starting point. Four-bit compression usually preserves broad ability, but agentic application accuracy can degrade more than perplexity suggests.
Useful when it unlocks a meaningfully larger model. Re-run tool use, structured output and language tests; subtle failures appear before chat becomes obviously bad.
Use only after testing the exact build on your tasks. Very-low-bit quants can make a larger model load, but tool calls, structured output, code edits and multilingual nuance may degrade unevenly.
A local model does not need to replace the frontier model. Let the strongest model resolve ambiguity, move routine execution to a cheaper specialist, and escalate when the trace shows difficulty.
Settles architecture, constraints, review criteria and unfamiliar failure modes.
Selects a target for each turn from task, policy, cost, latency and recent tool signals.
Local or hosted workers handle bounded calls; repeated errors, loops or uncertainty route back up.
NVIDIA’s 30B-total / 3B-active hybrid MoE targets high-volume tool calls, validation and delegated execution. The official NVFP4 checkpoint is about 20.1 GiB before runtime and cache, making 32 GB the sensible personal-system floor.
Name targets such as planner, private-worker and frontier-review. Map those aliases to a local Gemma/Muse/Qwen-class endpoint, a European hosted service or a frontier API without rewriting application logic.
Benchmark task success, total workflow cost, completion time, retries and frontier-escalation frequency. Include human review and failed runs so a cheap but unreliable route cannot look efficient.
NeMo Switchyard provides provider translation plus classifier, stage, escalation and custom routing. Its repository labels the software pre-alpha and not for production use; isolate it behind a gateway contract, pin a revision and keep a tested fallback.
Remote access and local inference are separate decisions. The phone can carry the conversation and approvals while an authenticated gateway, model and tools stay on an always-on machine you control.
Collects the request, shows progress and asks for approval. It may be only a control surface.
Authenticates the client, owns the agent loop and decides which backend and capabilities are available.
Loads weights and serves inference on the home machine, workstation or private server.
Run both on the host, make only the authenticated WebUI reachable over a private route and open it in the phone browser. Keep Ollama itself on localhost or an internal network.
Open WebUI documentation →Keep the gateway and session state on the always-on host, then pair an iPhone or Android companion. Start with remote chat; add device-node capabilities only when the workflow needs them.
OpenClaw remote access →Pair Orca's iPhone or Android companion with its desktop runtime to see worktrees, agent status and session output, then reply or dictate from the phone. The desktop remains the source of truth.
Orca mobile companion →Reach the agent through a supported messaging channel while its model runs elsewhere. Treat that channel provider as part of the data path, even when inference stays at home.
Hermes messaging gateway →These defaults assume one interactive agent, a Q4-class model, the OS and runtime, plus useful KV-cache headroom. “Chat fit” can be larger; “agent fit” must survive tool use and growing context.
16 GB agent fit: 7-12B Q4 with 8-16K context. A 14 GB model may load, but leaves almost no runway for a harness.
These are working configurations, not weight-only fits. Context is a conservative configured starting point for one interactive agent; increase it only after watching memory and prompt-processing time.
Best first candidate for multilingual drafting, retrieval-grounded work and bounded code changes.
Open family record →A compact coding and tool-use baseline with broad runtime packaging.
Open family record →The small Moonshot option. It is useful for experiments, but it is not a miniature Kimi K3.
Open family record →Retained for compatibility testing. It can load in 16 GB, but leaves almost no margin and is not a current first pick.
Open family record →Looking for the viral Kimi demo? Kimi K3 is a 2.8T server-scale model, not a 16 GB download. The Moonshot card above is an older small research checkpoint. Plan the provider tier →
Agent-safe start: Gemma 4 12B QAT Q4 (about 6.7-7.4 GB) or Qwen3.5 9B Q4 (6.6 GB). Moonlight 16B-A3B is a small Moonshot research option in community 4-bit builds, but its model limit is only 8K.
Chat fit is not agent fit: GPT-OSS 20B is designed to run within 16 GB and its Ollama package is 14 GB, but that leaves almost no margin for macOS, long context or tool results.
Current start: Gemma 4 12B at Q6/Q8 or Muse Glimmer’s official 17 GB quant. Gemma 4 26B-A4B Q4 at 14.4 GB is also viable when the runtime supports it well; treat 24 GB Muse as a tighter configuration and require a short tool-loop acceptance test.
Tight: Qwen3.8-27B FP8 (16.38 GB; supported serving backends), Qwen3.6 27B Q4 (17 GB; broader desktop packaging), GLM-4.7-Flash (19 GB) and 32B Q4 packages load on paper but sacrifice the context and tool runway that makes a harness useful.
Ranked start: Gemma 4 26B-A4B Q4 first, Muse Glimmer 30B second only after its configured harness passes a bounded read-then-edit test, and Qwen3.8-27B FP8 third. Qwen remains the coding-heavy choice on a supported backend and leaves about 15 GB before runtime allocation.
Fallback route: GLM-4.7-Flash Q4 and older broadly packaged models remain compatibility options where current Gemma, Muse or Qwen builds are unavailable.
Ranked start: Gemma 4 31B for the strongest current family fit, Muse Glimmer after a tool-loop acceptance test, then Qwen3.8-27B FP8 for coding-heavy work and parallel sequences. Use the older DeepSeek R1 Distill 70B only when a reasoning acceptance set proves the trade.
Do not spend memory for its own sake: Qwen says the FP8 package’s performance is nearly identical to the original model. The 55.56 GB full checkpoint is a fidelity test route, not the default 64 GB configuration.
Ranked start: run Gemma 4 at high precision, Muse Glimmer for agents only after a tool-loop acceptance test, or Qwen3.8-27B FP8 with parallel headroom. Keep GPT-OSS 120B as an older compatibility baseline only when the exact acceptance set rewards its 65 GB package.
Qwen3.5 122B-A10B Q4 (81 GB) remains a capability-first stretch, not the automatic Qwen default. Choose it only when your tasks beat Qwen3.8-27B by enough to justify the lost context and concurrency runway.
Parallel agents can share one loaded model, but each active sequence adds KV-cache pressure and its own growing history.
A model may advertise 128K-1M while the runtime loads 8K. Raising it costs memory and prompt-processing time.
Start here, then compare Q6/Q8 or a larger Q3 only on a repeatable tool, code and language test set.
Task success, retries and human corrections matter more than tokens per second or a leaderboard point.
The configured window must hold the harness, system and project instructions, MCP schemas, conversation, retrieved files, tool results, hidden reasoning where applicable and the next answer.
The training limit: often 128K, 256K or more. It says nothing about your runtime setting, speed or memory fit.
A runtime can be set to 8K even when the model supports far more. The inference server and agent client may also impose separate limits.
Configured window minus instructions, tool definitions, history, output reserve and safety buffer. This is the number to plan around.
The initial slice fits. The remaining range is roughly 1 additional 16 KiB tool-result batch, or 445-1,039 more nonblank code lines before compaction.
Count the files and relevant lines the agent must see now. Search and retrieval can fetch more later; dumping the full tree spends context before work begins.
A long command result may cost more than the prompt that caused it. Prefer focused tests, clipped logs and fresh reads over carrying every failed attempt.
Let a stronger model resolve ambiguity and write the plan plus acceptance criteria. Start the worker with that artifact and only the files needed for the next task.
Families are ordered by current recommendation, not download size: Gemma 4 first, Muse Glimmer second with a tool-loop warning and Qwen3.8 third. An older label marks the relevant local line as pre-2026 and keeps it for compatibility or research, not as a first pick. Fit chips still show where a package can be practical, not where its maximum context will fit.
Google reports pre-training across 140+ languages and out-of-the-box support for 35+. Independent TartuNLP evaluation of Gemma 3 found strong Estonian instruction following, grammar and word-meaning results at 12B and 27B. That makes Estonian an excellent stress test for the family, not a promise for every language or automatic evidence for Gemma 4.
Muse Glimmer 30B is a dense model distilled from Muse Spark and released under Apache 2.0 for agentic work, tool use, coding, multilingual tasks and text-plus-image input. Meta publishes a 131,072+ model context and training across more than 100 languages. The official GGUF repository contains a 16.76 GB K-Quant-17GB build and a 19.65 GB dynamic build; image input needs the separate 1.40 GB perception file, while the optional 1.63 GB DFlash drafter accelerates generation without changing accepted output. Two early DGX Spark owner runs measured 5-10.5 tok/s without DFlash and about 23-38 tok/s with the official drafter set to 15 speculative tokens; useful evidence, but not a standardized cross-engine benchmark. Those file sizes make 24-32 GB systems credible targets, but they do not make the full context cheap or guarantee that every launch-day client understands the new architecture.
It handles text, images and video, can switch thinking on or off, and ships in a 16.38 GB FP8 build. Start with 32 GB for one useful local agent; 64 GB gives you more context or parallel work. Qwen reports that the FP8 build performs close to the full 55.56 GB checkpoint, but test it on your own tasks.
Nemotron 3.5 Lightning is a 30B-total / 3B-active hybrid MoE released for high-volume agent execution. Its official NVFP4 repository is about 20.1 GiB, and NVIDIA documents DGX Spark, Jetson and GeForce RTX 5090 routes plus data-centre serving. Treat 32 GB as the practical personal-system floor: runtime state, its optional 1.26 GiB DSpark drafter and useful context still need memory. The model’s OpenMDW 1.1 terms are not the earlier NVIDIA Open Model License.
GLM-5.3 is Z.ai’s newer model for long-running coding and agent tasks. It is available through Coding Plan, while the general API and downloadable weights are still pending. If you need to run GLM yourself today, use GLM-5.2; its official 465 GB NVFP4 build is aimed at DGX Station-class hardware. On a normal local machine, GLM-4.7-Flash is the practical choice.
Do not change only the model ID. Z.ai says thinking must remain enabled: replace thinking.type: "disabled" with "enabled", then choose reasoning_effort low, high or max. Start existing non-thinking workloads at low; use max for coding acceptance tests. A disabled-thinking request will fail.
The local R1 distills predate the current generation. Use R1-0528-Qwen3-8B for compatibility tests, 32B for the useful middle and 70B on a 64 GB-class machine only when your acceptance set rewards them. DeepSeek-V3.2 and full R1 are 671B MoE deployments and belong on multi-GPU servers or hosted endpoints.
The 3B, 8B and 14B Ministral models are compact, multimodal and permissively licensed. Mistral Small 4 combines instruct, reasoning and coding modes at 119B total/6.5B active; treat local low-bit builds as experimental until your runtime lists the exact architecture.
Kimi K3 now has released weights, but its 2.8T parameters, native vision and one-million-token model limit still make it a multi-GPU or hosted deployment, not a personal-machine recommendation. Moonlight 16B-A3B and experimental Kimi Linear 48B-A3B are the smaller downloadable Moonshot options; both predate the current generation and should be treated as research checkpoints, not miniature K3 substitutes.
Llama 3.1 8B remains a safe runtime test and Llama 3.3 70B is the clean 64 GB step. Llama 4 Scout is 109B total/17B active and may fit at 128 GB in community quants, but Meta’s official full-precision path is multi-GPU. Check the custom license and exact runtime.
Both prior-year models use native MXFP4 MoE weights and the Harmony response format. Prefer the current top-three families unless GPT-OSS wins the exact runtime, tool or footprint test. OpenAI says 20B can run within 16 GB and 120B within 80 GB; a 16 GB personal machine still has almost no margin after loading the 14 GB 20B package.
This is a filtered list, not a copy of a hosted catalogue. Every media entry has released weights, a credible local or workstation route, and creator-published quality or benchmark evidence. Hosted-only products, LoRAs, duplicate endpoints and utility wrappers stay out.
A compact MIT-licensed reasoning model. Microsoft reports MMLU 84.8, GPQA 56.1 and HumanEval 82.6 for the 14B base instruction model; reasoning, mini and multimodal Phi-4 variants also ship.
Enterprise-oriented models with governance disclosures, hybrid Mamba/Transformer variants and compact footprints. Shipped language sizes include Micro, H-Micro, H-Tiny, H-Small and dense 8B; evaluate the exact card because scores vary by variant.
A rare option with training code, data, checkpoints and detailed recipes. The family ships 7B and 32B Base, Instruct and Think variants with 65,536-token context; use 7B Q4/Q8 at 16 GB and 32B Q4 at 24-32 GB.
A locally testable Estonian-focused research checkpoint and evaluation suite. The authors clearly label it an early prototype with 4K context and no multi-turn chat; use it as a benchmark and fine-tuning reference, not an automatic production default.
The 4B and 9B checkpoints combine text-to-image generation and multi-reference editing in one four-step model. Black Forest Labs says the Apache-2.0 4B model fits about 13 GB VRAM on an RTX 3090 or 4070-class GPU; the stronger 9B model fits about 29 GB and uses the FLUX Non-Commercial License. Both are actual WaveSpeed endpoints, but their released weights make local use credible without the API.
Tongyi-MAI's 6B image model targets photorealism and bilingual text rendering in a much smaller package than the server leaders. The creator says it runs on consumer GPUs with under 16 GB VRAM and publishes the code and weights. That makes the Turbo checkpoint a genuine local alternative, not merely a WaveSpeed product label.
Tencent's 8.3B text-to-video and image-to-video release is the strongest current lightweight video candidate in this catalogue. Tencent reports state-of-the-art open-model quality and motion coherence, publishes training and inference code, and documents a 14 GB minimum with model offloading. The step-distilled 480p image-to-video checkpoint runs in 8 or 12 steps and was measured by Tencent at under 75 seconds on an RTX 4090.
The 5B Wan checkpoint handles both text-to-video and image-to-video at 720p and 24 fps on one 24 GB RTX 4090 with offloading. Alibaba describes it as one of the fastest open 720p models and publishes the weights and inference code. The larger A14B route belongs on the server page; this 5B checkpoint is the consumer-workstation cut.
MiniMax Music 3 generates complete songs of up to five minutes from lyrics and a detailed music description, including vocals and 32 kHz stereo audio. The official Diffusers path fits the full-precision pipeline in under 24 GB VRAM; automatic CPU offload uses about 22 GB, and layer-by-layer language-model streaming can reduce the GPU requirement to 8 GB at the cost of speed. Inference currently requires CUDA and does not stream the generated audio.
Qwen releases 0.6B and 1.7B speech models for custom voices, voice design and cloning. On Qwen's Seed-TTS evaluation, the 12 Hz 1.7B Base checkpoint reports 0.77 Chinese and 1.24 English word error rates, ahead of the commercial and open baselines listed in that table. The official project does not state a minimum VRAM, so treat the small parameter count as a starting signal and measure the complete audio stack on the target GPU.
ACE-Step is the clearest local music-generation addition in the hosted catalogue. The standard model can run with under 4 GB VRAM, while its 4B XL decoder needs at least 12 GB with offload and recommends 20 GB. The creators report under ten seconds per full song on an RTX 3090 and publish benchmark, profiling and GPU-tier tools, including Mac, AMD, Intel and CUDA paths.
Tencent releases the full 3.3B shape and 2B PBR texture pipeline, including training code. Its creator evaluation leads the listed open and closed baselines on shape following and textured-asset metrics. Hardware scales by task: 10 GB for shape, 21 GB for texture, or 29 GB for the complete pipeline. Note that its community licence excludes the EU, UK and South Korea.
Meta's current Segment Anything release handles text-prompted image segmentation, video segmentation and multi-object tracking. The official checkpoint is about 3.44 GB. On Meta's public tables, SAM 3.1 improves six of seven video-object-segmentation benchmarks over SAM 3 and adds an approximately sevenfold 128-object speedup on one H100 test. The official project does not publish a consumer-GPU minimum, so measure your prompt count, image size and video workload before promising a device tier.
Compare sizes inside the same family and table first. Then test the failure that costs you money: wrong tool calls, incomplete edits, weak language, invented citations or human cleanup.
Gemma 4 E2B → 12B → 31B on the same MMLU-Pro table shows a meaningful capability curve.
Check version, prompt, pass@1 versus consensus, tool access and whether the result is vendor-reported.
Score multilingual quality, tool selection, argument accuracy, code edits, recovery and the amount of human correction required.
Estonian is a useful low-resource, morphologically rich stress test. TartuNLP’s independent table reports IFEval-et of 0.756 for Gemma 3 12B and 0.766 for 27B; Llama 3.3 70B scores 0.771 and Kimi K2 0.789. Gemma 3 27B also posts 0.817 on Grammar-et and 0.953 on Word-Meanings-et. That is impressive evidence for the candidate list, while every production language pair still needs its own prompts.
A local result comes from the checkpoint and every layer that prepares, computes, samples and parses it. Reproduce that contract before deciding the model itself is weak.
Record the exact checkpoint revision, quantization, tokenizer or encoding code, chat template, reasoning mode, tool parser and sampling settings. A family name is not a reproducible model.
Pin the runtime and container commit, driver, GPU type, attention backend, kernels, tensor parallelism, speculative decoder, KV-cache format, batch limits and context ceiling.
Replay real long-context work with tool calls and exact literals. Failures can cluster around prompt content rather than rise smoothly at one universal context length.
Use labelled answers, executable tool and code checks, repeat runs and raw request records. Score quality, latency and throughput separately; one attractive tok/s number proves none of the others.
The inference runtime loads weights, allocates context and exposes an API. It is not the agent harness. Confirm acceleration, chat template, structured output and tool-call format before connecting an agent.
Best first stop for discovery, MLX/GGUF downloads, chat-template inspection and an OpenAI-compatible local API. Check both configured and model-maximum context; they are separate values.
LM Studio →Fast route to a reproducible pull command and local API for coding harnesses. Inspect the exact tag and context setting: “latest” can hide quantization and file-size differences.
Ollama →Use MLX when an Apple-silicon-native package exists; use llama.cpp for explicit GGUF control, broader quant choices and portable serving.
Prefer these on supported NVIDIA systems when concurrency, continuous batching, tensor parallelism and an always-on endpoint matter.
A harness owns the loop around the model: tools, memory, permissions and retries. A specification workflow stores an agreed plan outside the conversation. Both can call a local inference API; neither makes a weak model reason like a frontier model.
Loads the model and serves tokens.
Runs tools, sessions, memory and approval gates.
Keeps plans, tasks and acceptance criteria outside chat history.
Use Hermes when you want a general agent loop with tools, skills, memory and multiple model-provider endpoints. Point it at the inference server; then evaluate the model through Hermes rather than assuming chat quality transfers.
Hermes Agent repository →Use OpenClaw for an always-on personal agent, channels and remote clients. Keep tool authority narrower than model capability, authenticate the gateway and treat local inference as one backend, not the harness itself.
OpenClaw →Stores a proposal, tasks and spec deltas in the repository before implementation. This is a clean hand-off: use a stronger model to settle the plan, then give a smaller worker the approved artifacts and relevant files.
OpenSpec repository →SpecStory’s open methodology moves from intent to roadmap, workplans, execution and refinement. It helps preserve decisions between agent sessions; it does not serve a model or enforce an inference format.
Specflow repository →RTX Spark is offered for slim laptops and small desktops; DGX Spark is a compact dedicated system; DGX Station adds a 748 GB coherent tier. What matters is available memory, memory bandwidth, backend support and whether the model runs on the client or a remote host.
Interactive use, private documents and one-user local APIs. LM Studio, Ollama, MLX or llama.cpp.
The speculative band reserves 20-25% for runtime and cache; NVIDIA's up-to-200B figure is a tighter capacity ceiling. Validate the exact GB10 kernel.
NVIDIA’s ConnectX path advertises up to 405B and demonstrates Qwen-235B NVFP4. The lower band keeps practical headroom; software and model architecture still decide whether it works.
DGX Station can plausibly hold a mixed-NVFP4 model in that band across coherent memory. Blackwell datacenter nodes and racks scale further; H100/H200 do not provide native NVFP4 W4A4.
Compare licensable models and NVFP4 servers →Station combines 252 GB of 7.1 TB/s HBM3e with 496 GB of 396 GB/s LPDDR5X. NVIDIA's 465 GB GLM-5.2 NVFP4 checkpoint fits across the coherent pool, but spills beyond HBM and still needs runtime and cache headroom. The linked Station measurements belong to GLM-5.2; GLM-5.3 weights and serving recipes were not public when checked. Station is most compelling when 70-300B models live in HBM, or when local access to a larger low-bit model is more valuable than cloud-scale throughput.
Use Spark for 20-35B dense models and sparse 30-120B MoEs. Consider Station for daily 70-120B high-precision work, 200-400B low-bit work, or controlled 400B-1T experiments. Rent first when that top tier is occasional.
Parameter ceilings are not memory guarantees. NVFP4 planning bands reserve 20-25% of advertised memory and assume roughly 5.0-5.2 effective bits per stored parameter. Architecture, mixed-precision layers, KV cache, multimodal towers, memory tier and backend support can move the result. A model package can load while leaving too little memory for useful context, tool traffic or concurrency.
The core catalogue was checked against official model cards, repositories and current Ollama/LM Studio packages in July 2026; dated launch records and specialist routes were rechecked through 15 August 2026.
The default package leaves roughly 20-25% of system memory plus a practical starting context target. It is a planning rule, not a throughput guarantee.
The weights can load, but context, parallel requests or other applications may trigger swap or out-of-memory errors. Tight fits are labelled in amber and should start with short context.
The official model card expects multi-GPU or vendor-specific infrastructure, or no verified LM Studio/Ollama local package was found. Cloud-only Ollama listings are not counted as local support.
Q4 sizes are package-level approximations unless the vendor publishes a memory table. QAT, GGUF Q4_K_M, MLX 4-bit, NVFP4 and MXFP4 are not interchangeable. Native NVFP4 W4A4 requires Blackwell-generation Tensor Cores plus a supported kernel; Hopper fallback is not the same compute path.
Gemma 4 first for the strongest current local family. Muse Glimmer second only after tool-loop acceptance. Qwen3.8 third for coding and breadth. Nemotron and GLM cover specialist routes. Keep GPT-OSS, Llama and the older local lines as baselines, not defaults. Keep room for the work.