RAG + prompting
Retrieve documents at run time. Best for changing facts, private records and answers that need citations.
Data: governed documentsDoes not place the knowledge reliably inside the weights.Guides · Model adaptation · checked 27 July 2026
Should You Use RAG, Fine-Tuning or Pretraining?
Start with the measured failure, then choose the lightest intervention that can fix it. Most organisations need retrieval or a focused tune—not a foundation model trained from scratch.
4distinct intervention routesRETRIEVAL TO PRETRAINING
1measured gap before trainingBENCHMARK THE BASELINE
0evaluation gates skippedTEST EVERY CHECKPOINT
Interactive decision tree
When should you use RAG instead of fine-tuning?
Select every gap that applies. The recommendation can combine retrieval and training because production assistants usually need more than one layer.
Start with RAG
Give the model governed access to current knowledge.
Index the allowed documents, retrieve evidence into context and require citations. Measure that baseline before changing weights.
RAGFine-tuneContinueScratch
Retrieval does not fix deep language fluency or reliably teach a new output protocol.
The four rungs
“Our own data” can mean four different systems.
The more of the foundation you change, the more data rights, compute, evaluation and rollback discipline you inherit.
SFT / LoRA / preference tuning
Train on demonstrations or preference pairs to change format, tone, policies and task behavior.
Data: labelled examplesCan overfit style or damage other capabilities without a regression suite.Continued pretraining
Resume the next-token objective on a licensed language or domain corpus, then post-train again.
Data: large raw corpusUseful for vocabulary and cultural grounding; materially harder than SFT.Training from scratch
Design the tokenizer and recipe, initialize weights and pretrain across a massive balanced corpus.
Data: foundation-scale mixOnly justified when sovereignty and underrepresentation outweigh the programme risk.Method boundaries draw on Meta’s Llama 3 report, Google’s Gemma tuning guidance, the InstructGPT paper and public adaptation reports for ALLaM, NorwAI and EuroLLM-22B.
Model architecture
Another scaling axis: reuse depth.
A looped or recurrent-depth Transformer stores fewer unique layers and applies some of them repeatedly. The proposition is fewer unique weights, with more sequential computation available for a problem.
Each layer normally owns a separate set of parameters.
The hidden state is refined by repeatedly applying some of the same parameters.
Unique parameters + training data + recurrent computation
The recurrence is latent computation inside one model computation. It is not the model printing a longer chain of thought.
Effective depth without storing every layer.
Weight tying can reduce weight memory versus an equally deep untied model. Iterative refinement may suit reasoning and algorithmic tasks; implementations with variable passes can expose adaptive test-time compute. That is worth testing on memory-constrained local or edge systems.
Memory saved is not compute saved.
Extra passes usually add latency and accelerator work. Fewer unique weights may hold less factual capacity, gains can plateau, and training stability, halting, KV-cache design and serving support remain active engineering problems. A model that fits can still run slowly.
Architecture is only one system layer.
Recurrence does not replace retrieval, tools, memory, agents or post-training. A national-language model still needs strong language, culture, instruction and evaluation data; the architecture choice is separate from the language-data strategy.
Three different loops
Ask what repeats—and where.
Looped Transformer
The model reapplies shared neural-network blocks during one forward computation.
hidden state → shared block
→ refined state → shared block → outputReasoning-token loop
A conventional autoregressive model emits extra reasoning or scratchpad tokens before the answer.
token → token → token → answerAgent loop
An external harness calls a model, tools and memory repeatedly. Its persistence says nothing conclusive about recurrent blocks inside the model.
plan → act → observe → update
→ verify → repeatResearch checkpoints you can run
Study the architecture before betting a product on it.
Ouro
ByteDance/Ouro-1.4B and ByteDance/Ouro-1.4B-Thinking are open Looped Language Models pretrained for iterative latent computation; official 2.6B base and Thinking variants are also public. Treat them as research models and compare against a mature conventional model at similar runtime cost.
Huginn
tomg-group-umd/huginn-0125 is an approximately 3.5B-parameter recurrent-depth proof of concept. Its official implementation exposes recurrence depth, making it useful for architecture experiments—not a polished default local assistant.
Loopie
The July 2026 paper reports layer-level recurrence in MoE models with 6B total / approximately 0.6B active parameters and 20B total / approximately 2B active. No official weights or serving code were linked or discoverable at this check, so this is a significant research result—not yet a deployment recommendation.
Read the Loopie paper →Does Fable or Mythos use recurrent depth?
It is not publicly known. Anthropic describes Claude Fable 5 and Claude Mythos 5 as the same underlying model with different safeguards and access arrangements, and reports unusually strong long-horizon autonomy. Anthropic has not publicly identified a Looped Transformer, recurrent-depth block, OpenMythos-style recurrence or another hidden-state looping design. The observed persistence can also come from long-horizon training, adaptive reasoning effort, an agent harness, context compaction, persistent files and notes, sub-agents, repeated verification and training to recover after failures.
OpenMythos describes itself as an independent, community-built theoretical reconstruction based on public research and speculation. It is not leaked Anthropic code and is not evidence of Anthropic’s architecture.
Behaviour can suggest an architectural hypothesis, but persistence observed through an agent product is not enough to reverse-engineer the neural architecture underneath it.
Training decision
Fine-tuning usually cannot create native recurrent depth.
| Intervention | What it can do | Architecture boundary |
|---|---|---|
| RAG | Supply current or private evidence at inference time. | Does not change model depth. |
| LoRA / SFT | Specialise the behaviour of a looped checkpoint. | Normally does not convert a conventional Transformer into native recurrent depth. |
| Continued pretraining | Adapt an existing looped checkpoint to a language or domain. | Preserves the checkpoint’s basic architecture. |
| Training from scratch | Design and pretrain a genuinely new recurrent architecture. | The cleanest route—and the highest programme burden. Retrofitting recurrence into pretrained models exists, but remains experimental. |
Decision rule
Start with an existing looped checkpoint when studying the architecture. Do not redesign a national or enterprise model around recurrence until it beats a conventional baseline on the same data, compute, latency and task suite.
Bounded experiment
Compare completed work, not parameter labels.
- Pair the models.Use Ouro 1.4B Thinking and a mature conventional 1–4B model at comparable precision and on the same hardware.
- Test the work.Measure arithmetic and algorithms, multi-step instructions, retrieval-grounded QA, Estonian quality and tool-call formatting.
- Measure operations.Record completed-task accuracy, latency, peak memory and total compute or energy where measurable.
- Vary depth carefully.Change recurrent passes only where the official implementation supports it; record where quality improves, plateaus or falls.
Sub-1B specialization
Small models become useful when the job becomes specific.
A sub-1B model is rarely convincing as a miniature general-purpose chatbot. It can be a credible, cheap language-processing component inside ordinary software.
Useful mental model
Large models solve unfamiliar problems. Micro-models perform familiar jobs extremely cheaply. The smaller the model, the narrower and better-tested its contract should be.
Turn messy language into known fields.
Classify intent or documents, rewrite a query, and extract entities, requirements or metadata filters.
IntentEntitiesFilters
Connect language to deterministic software.
Select a search path, API or internal function; rerank a small candidate set; and emit a validated JSON plan.
ToolsRerankingJSON
Finish a bounded, grounded artifact.
Write a short cited summary or template, flag spam or sensitive content, and support local or offline actions where the device permits.
TemplatesModerationLocal
Architectural correction
Fine-tune the behavior; retrieve the facts. Teach customer language, intent labels, filter schemas, tool traces, output contracts and tone. Keep prices, policies, compatibility rules and catalogue content in pages, databases, APIs or search indexes, then retrieve verified passages at run time.
01AskReceive the customer’s natural-language question.
02InterpretClassify intent, extract filters and write a search plan.
03RetrieveUse lexical search for exact matches and vectors for semantic candidates.
04RerankSelect the few passages that best support the requested decision.
05ComposeGive the model only allowlisted evidence and stable source IDs.
06ValidateCheck schema, claims, calculations and permissions before rendering.
Browser delivery boundary
In-browser inference removes the server inference queue and can keep text on the device, but it shifts model download, memory, battery and compatibility costs to the visitor. Make a substantial download explicit and keep a WASM, server or non-AI fallback.
| Parameter band | 4-bit weights-only floor | Practical public-website boundary |
|---|---|---|
| Up to 150M | Up to ≈75 MB | Easy to justify for classifiers, embeddings, entity extraction and specialized transformations when the measured feature earns the download. |
| 270M–360M | ≈135–180 MB | Reasonable for an explicit AI-powered feature after opt-in, progress feedback and testing on representative phones and laptops. |
| Around 500M | ≈250 MB | Viable for a valuable local feature, but the visitor should knowingly start the download and have a graceful fallback. |
| Around 1B | ≈500 MB | Technically possible, but usually too heavy for an invisible enhancement on a normal public site. |
The size column is arithmetic, not a package quote: four-bit weights require roughly 0.5 bytes per parameter. Tokenizers, metadata and runtime files increase the download; activations and the KV cache increase working memory. Real speed, memory pressure and output quality depend on the exact model, quantization, context, browser and device.
Workflow loop
A useful model programme is an evaluation loop.
The deploy gate is not the end. Production failures become new tests; they do not flow directly into training data.
Inference engineering
Training is only half the model programme.
Products, evaluations, synthetic-data generation and reinforcement-learning rollouts all run inference. They do not share the same latency, throughput, cost or reproducibility target.
Protect user latency under load.
Measure time to first token, inter-token latency, p95/p99 tails, throughput per accelerator, errors and cost per accepted result with realistic prompt lengths, output lengths and concurrency.
TTFT + ITLTail latencyAvailability
Maximize useful, reproducible output.
Offline generation can trade single-request latency for batching and aggregate throughput. Pin the model, dataset, prompt template, sampler and engine, then retain outputs so a score or corpus can be reproduced.
Batch throughputCostReproducibility
Treat rollout as a distributed data path.
Rollout workers must serve the intended policy and reward models, refresh weights safely and return versioned trajectories. Slow or unstable inference can idle the rest of the training loop.
RolloutsWeight updatesFault recovery
| Layer | Learn first | Working proof |
|---|---|---|
| 1 · Model foundation | Modern Python, PyTorch tensor execution and Transformer anatomy: attention, feed-forward or expert blocks, tensor shapes, dtypes and memory use. | Load a pinned model and reproduce reference outputs and numerical tolerances. |
| 2 · Serving engines | SGLang and vLLM request paths; prefill versus decode; KV-cache allocation; continuous batching; prefix caching; structured output and speculative decoding. | Serve the same supported model through both engines, preserve the API contract and compare TTFT, inter-token latency and throughput. |
| 3 · Performance | Profiling, roofline reasoning, GPU memory and bandwidth, kernel launches and synchronization. Add CUDA or Triton for kernel work and C++ for native extensions, bindings and framework internals. | Locate one measured bottleneck, change one variable and keep correctness within an explicit tolerance. |
| 4 · Distributed scale | Processes, queues, backpressure and failure recovery; collectives and tensor, pipeline, data or expert parallelism; NCCL plus the role of NVLink and InfiniBand. | Explain when the run is compute-, memory- or communication-bound, then demonstrate multi-GPU scaling and restart behavior. |
| 5 · Production evidence | Load generation, request tracing, numerical-stability tests, regression suites, admission control, observability, release pinning and rollback. | Publish a workload-specific service target and a benchmark report with model, engine, hardware, quantization, traffic shape and failure tests. |
Do you need C++?
Not to begin. Python, PyTorch, Transformer inference math and disciplined benchmarking are the entry layer. C++ becomes important when you change native framework code, bindings, memory movement or kernel launch paths; CUDA or Triton matters when you change the GPU kernels themselves. An application developer consuming an inference API may never need that depth, while an inference-framework engineer eventually will.
Language + culture
A national model is more than fluent output.
It should work across local language, institutions, culture, safety norms and actual public or enterprise tasks—and preserve evidence of where its data came from.
Language and world model
Licensed web, books, news, archives, Wikipedia, parliamentary and legal text, science, maths and code.
Cross-language coverage
Translation memories, bilingual text and careful translation of high-value material for lower-resource coverage.
Useful behavior
Local QA, summarisation, public-service workflows, document work, coding and structured-output demonstrations.
Boundaries
Ranked answers, refusal edge cases, abuse prompts, jailbreaks and culturally grounded safety judgments.
Actions
Function-call schemas, tool results, recovery paths and multi-step workflow traces with permission boundaries.
Anti-self-deception
Held-out local tests for idioms, geography, institutions, law, culture, safety, cost and production tasks.
Open-data example · checked 14 August 2026. OpenWALDO is developing a public corpus and training toolchain that carries source, license, count and hash records from selected data into model artifacts. It is worth inspecting if “open weights” are not enough for your definition of open AI; its license identifiers are assertions, not legal proof, and the resulting model still needs independent quality and safety evaluation.
European starting point
EuroLLM-22B covers all 24 official EU languages plus 11 additional languages, including Estonian. That makes it a relevant base or benchmark candidate—not automatic proof that it passes your local tasks.
Interactive compute + memory planner
See the order of magnitude before the purchase order.
The compute estimate uses the common dense-transformer heuristic of roughly 6 × parameters × training tokens. It is planning math—not a quote or a promise.
Approximate memory per model replica—
Planning estimateBase weights —Train state —Activations/runtime —
This excludes data engineering, storage, networking, checkpoints, failed runs, evaluation, staff and serving.
The memory visual assumes a 4-bit base plus 15% load overhead for QLoRA, a BF16 base for LoRA, and about 16 bytes per parameter before activation reserve for full Adam-style training. Sharding changes per-device fit; long sequences and large batches can make activation memory much higher.
Public scale references
The final run is not the programme.
Published disclosures show orders of magnitude, not transferable price quotes. Different architectures, data mixes and clusters make direct cost comparisons approximate.
Focused instruct tuningNorwAI · 7.5B
38.4 H100-hours
Continued pretrainingNorwAI disclosed runs
4.4K–31.1K H100-hours
Final foundation runBLOOM · 176B
1.083M A100-hours
Wider research programmeBigScience / BLOOM
3.46M GPU-hours
Sources: NorwAI technical report; BLOOM carbon and training disclosure; and the wider programme accounting in Estimating the Carbon Footprint of BLOOM. Accelerator generations and accounting boundaries differ.
Evaluation contract
A tuned model can improve and still be worse.
Ship only when the target gain survives general capability, safety, cost and production-like checks.
Did the intended task improve?
Held-out local prompts, exact output contract and representative languages.
What did the model forget?
General reasoning, multilingual performance, calibration and base-model strengths.
Did refusals or tool behavior move?
Injection, sensitive data, dangerous requests, permissions and false tool calls.
Can you afford the new behavior?
Tokens, latency, memory, energy, concurrency, retries and human review time.
Can you reproduce and roll back?
Data manifest, code, base hash, adapter, hyperparameters, eval artifacts and owner.
What failed
Six ways a model programme goes wrong.
These are not arguments against training. They are reasons to preserve evidence, human data, rollback paths and a cheaper baseline.
Fine-tuning was used as a database.
Facts still became stale, citations disappeared and each update required another training run.
Recovery: retrieval first; tune behavior only.The model learned from its descendants.
Nature experiments found that indiscriminate recursive training on model-generated data loses distribution tails and degrades later models.
Read the primary study →Only the successful run was costed.
BLOOM’s final run used about 1.08M A100-hours; the wider project accounted for 3.46M GPU-hours.
Inspect the accounting boundary →The public score rose; the local task did not.
Prompt templates, contamination and harness choices can move scores without improving the production distribution.
Review benchmark traps →The corpus could not be documented or reused.
Unclear copyright, personal-data basis or source provenance can stop release after compute has already been spent.
Recovery: make the data manifest a release artifact.The model trained successfully and failed economically.
A larger model or longer reasoning trace raised latency, memory and review cost beyond the workflow’s value.
Recovery: benchmark total cost per accepted result.Open models + distillation
Jensen Huang’s case—and the boundary around it.
Huang argues that learning from other systems is fundamental. The legal and operational question is not whether distillation exists, but what data, contract, privacy and intellectual-property permissions govern a specific use.
Axios · 22 July 2026
“Learning from AI … is fundamental to intelligence.”
Huang said AI systems will increasingly learn from other AI-generated knowledge and argued that policy should target contract, privacy or other misconduct rather than prohibit the technique broadly.
“The world needs both frontier closed models and frontier open models.”
NVIDIA joined 26 other named organizations in a letter arguing that open weights expand access, competition, control, safety research and sovereignty. The letter also acknowledges that released weights are hard to trace or reverse and calls for targeted legal and commercial treatment of unlawful extraction.
This is an advocacy position signed by NVIDIA and others—not neutral evidence that every open release is safe or lawful.
Distillation can compress capability, create training examples, evaluate or validate another model.
Terms, access controls, privacy, copyright, trade-secret and competition rules still apply to the actual collection and use.
Edge cases + obligations
The hard costs sit outside the training script.
This is operational guidance, not legal advice. Scope obligations with qualified counsel and the competent authority for the actual provider, model, system and market.
Document the model and training content.
EU guidance says GPAI providers must maintain technical documentation, support downstream providers, implement a Union-copyright policy and publish a sufficiently detailed training-content summary.
Read Commission guidance →Make AI interaction and generated content legible.
From 2 August 2026, covered providers and deployers face transparency duties including disclosure of AI interaction, machine-readable marking and notices for specified deepfakes or public-interest content.
Open the July 2026 guidelines →A model is not anonymous by assertion.
The EDPB says anonymity and legitimate-interest analysis are case-specific, including whether people can be identified or personal data extracted by querying the model.
Read Opinion 28/2024 →Track rights before tokenization.
The EU AI Act keeps training-content summary and copyright-policy duties relevant even for many open-weight routes. Store source, licence, collection date, restrictions and transformations.
Read the AI Act →Measure experiments and serving, not just the final run.
NIST calls for documented energy, water and emissions impacts. The IEA reports that AI-focused data-centre electricity use grew 50% in 2025 while per-task efficiency improved rapidly.
Read IEA 2026 analysis →The model is not the product.
Budget corpus pipelines, evaluation, retrieval, serving, tool permissions, observability, security review, incident response, user support and model retirement.
Map the lifecycle with NIST →The Commission’s training-content template FAQ says failure to publish a required summary can trigger enforcement from 2 August 2026, with potential fines up to 3% of prior-year worldwide turnover or €15 million, whichever is higher. Check the official scope, transition and penalty guidance.
Who builds what
Four teams, four definitions of progress.
A model programme fails when every team thinks the deliverable is “the model.”
Fund a measured workflow.
- Baseline RAG before approving training.
- Choose one sovereign or language-sensitive use case.
- Own rights, risk appetite and the deploy gate.
Own behavior and reproducibility.
- Datasets → Transformers → PEFT / TRL → evals.
- Version prompts, data, adapters and output contracts.
- Hand the inference track a representative workload, not a demo prompt.
Operate an AI factory, not a GPU rack.
- Storage, network, scheduler and checkpoint recovery.
- Isolate training from production inference.
- Measure utilization, energy and cost per accepted output.
Buy speed without surrendering evidence.
- Require exportable weights or adapters where promised.
- Keep your data manifest and evaluation set.
- Separate service claims from reproducible artifacts.
European commercial route
Mistral markets customisation, self-hosting and work from fine-tuning through pretraining. Use it to buy delivery speed while negotiating data, evaluation, portability and operating boundaries.
Review Mistral custom model training →Ecosystem route
EuroLLM, Hugging Face, public corpora, universities, national libraries, media archives and design partners build more internal capability—but require stronger programme ownership.
AI Now Summit 2026 · video library
See how custom AI moves from model to institution.
These 32 Mistral-hosted talks are first-party conference perspectives, organized by the decision they can inform. They are useful implementation context—not independent evidence that every deployment claim generalizes. Pair the physical-AI talk with the task-first robotics route and its field acceptance gate.
Showing all 32 talks.
Strategy · 42:44Opening keynote
Strategy · 16:14A CIO’s vision on the AI industrial revolution
Industry · 14:20How EDF and Mistral are reinventing France’s electricity
Public systems · 17:00How HTX is scaling AI for public safety
Agents · 19:41How agentic AI is reinventing large organizations
Industry · 9:50Transforming a global group with sovereign technology
Public systems · 20:19Perspectives on AI for defense
Industry · 25:44Scaling AI to support the energy transition
Public systems · 49:00How nations can develop resilient, citizen-centric AI systems
Models · 20:29The machines behind the machines
Industry · 22:26Scaling enterprise value with sovereign AI
Industry · 17:44A culture-driven approach to innovation
Agents · 21:31Scaling secure and transparent workflows
Industry · 14:28How Airbus is powering Europe’s AI industrial revolution
Industry · 24:56How La Banque Postale is building a human-centric future for banking
Public systems · 9:42Talk by Benjamin Haddad, Minister Delegate for Europe
Public systems · 16:52Shaping France’s AI future: CDC’s roadmap to digital autonomy
Industry · 25:12Mistral models powering Alexa+
Strategy · 5:26Closing keynote: The end of AI as we know it
Models · 28:08Co-developing scaffolds and models hand-in-hand
Models · 28:09AI for earth observation: Building the future with EVE
Strategy · 29:52AI infrastructure is the new critical infrastructure
Public systems · 28:27Luxembourg’s sovereign AI playbook for Europe
Agents · 27:56Building interconnected AI for complex operations
Models · 28:08Channelling the power of LLMs to unlock ancient archives
Models · 14:33Mistral robotics and physical AI
Agents · 21:20Rethinking the architecture of agentic systems
Agents · 20:43Best practices for building autonomous AI workflows
Public systems · 27:27Advancing innovation at the European Patent Office
Strategy · 32:05The AI sovereignty paradox: Scalable ecosystems for trusted adoption
Models · 29:24Domain AI models fine-tuned with proprietary knowledge
Models · 36:12Building custom code models for Ericsson proprietary silicon
The practical answer
Adapt a strong open base. Preserve the data trail. Let evaluation earn the next rung.