How to convert your agent from LLMs to SLMs without breaking it
NVIDIA Research's 6-step LLM-to-SLM conversion algorithm, with case studies and practical numbers. Migrate your agent from expensive frontier models to small language models.
TL;DR: NVIDIA Research published a position paper arguing small language models aren’t just cheaper but structurally better for most agent tasks. More importantly, they included a 6-step algorithm for migrating existing agents from LLMs to SLMs, plus case studies with replacement estimates. The conversion process is practical and achievable for most agent stacks today.
Key takeaways:
- NVIDIA Research proposes a 6-step LLM-to-SLM conversion algorithm: instrument, curate, cluster, select, fine-tune, iterate
- Case studies show 40-70% of agent LLM queries can be replaced by specialized SLMs
- Code agency (a controller script orchestrating) makes SLM substitution easier than LM agency (the model orchestrating itself)
- 10-30x inference cost reduction. A 7B SLM costs roughly Rs 18,000/month ($216) vs Rs 3.6 lakh ($4,320) for GPT-4o at 100K tool calls
- The heterogeneous agent pattern: SLM by default, LLM for the hard stuff
My agent called GPT-4o 47 times in one coding session last week. I checked the logs afterwards. 43 of those calls were doing the same two things: extracting intent from a user request and formatting the response into a tool call. The model spent 43 rounds of Rs 1.70 per call deciding whether the user wanted to “search their notes” or “create a new note.”
A 3B model can do that. It can do it faster, locally, and for free after the first Rs 18,000 of electricity.
I missed this because I never looked at what my agent was asking the model to do. I assumed every call needed GPT-4o-level reasoning. The logs showed otherwise.
What I found is that NVIDIA Research had already done this work. They published it as a position paper in June 2025 with a title that sounds like hype but reads like a migration playbook.
What does NVIDIA argue?
The paper makes three claims. They call them V1, V2, and V3.
V1: SLMs are sufficiently powerful. Phi-2 at 2.7B parameters matches 30B models on commonsense reasoning and code generation cited in the paper’s Section 3.1. SmolLM2 at 1.7B matches 14B contemporaries. DeepSeek-R1-Distill-Qwen-7B outperforms Claude 3.5 Sonnet and GPT-4o on specific benchmarks. The evidence is dense across model families from Microsoft, NVIDIA, HuggingFace, and DeepSeek.
V2: SLMs are inherently more suitable. Agent tasks are repetitive, scoped, and non-conversational. A model that handles one conversation type — say, tool call formatting — will do it better than a generalist that handles everything.
V3: SLMs are 10-30x more economical. This is the easy one. A 7B model costs a fraction of a 70B model to serve. No parallelization needed. Runs on consumer hardware.
The paper includes a correspondence section. One reader pointed out that reasoning LLMs are sometimes pitched as an alternative to agentic workflows. NVIDIA’s Belcak agreed: reasoning has been put on a pedestal where some argue it could replace orchestration. It can’t.
What is code agency vs LM agency?
This distinction is the key insight that makes the conversion algorithm work. The paper introduces two modes of agentic architecture.
Language model agency. The LLM is both the brain and the skeleton. It decides what to do, calls tools, interprets results, and manages the flow. Every loop iteration goes through the model. This is how most agents work today.
Code agency. A dedicated controller script handles orchestration. The language model fills specific, narrow slots. Intent classification here. Response formatting there. Tool call generation in between. The controller code manages the loop.
The kitchen analogy for this is a restaurant where the head chef does everything versus a restaurant where the head chef focuses on the pass and the line cooks execute the station work.
LANGUAGE MODEL AGENCY | CODE AGENCY
──────────────────────────────────────|─────────────────────────
model: "user wants to create a note" | classifier returns: "create_note"
model: "extracting title: 'meeting'" | router calls: extract_title(content)
model: "formatting tool call..." | formatter: build_tool_call("create_note")
model: "calling tool..." | harness: execute(tool_call)
model: "interpreting: 'note created'" | harness: format_response("done")
model: "generating response..." | template: "Note created. Anything else?"The agent column is a genuine transcript of what an LLM does on an agent loop. Every line is a model invocation. The code agency column uses the model for exactly one thing: classification. Everything else is a deterministic function.
This is the structural pattern that makes SLM substitution possible. If your agent uses LM agency, you’re paying GPT-4o to loop through a for-else-if chain that a 30-line Python script could handle. The NVIDIA algorithm converts you from LM agency to code agency, then substitutes SLMs where they fit.
How does NVIDIA’s 6-step conversion algorithm work?
The algorithm is the practical core of the paper. Six steps, designed to be painless.
S1: Instrument your agent
Deploy logging on every non-human-interface agent call. Capture input prompts, output responses, tool call contents, and latency. Encrypted pipelines with role-based access.
You can’t convert what you haven’t measured. If you don’t know what your agent is asking the model to do, you don’t know which calls can be replaced.
# Pseudocode for what instrumentation looks like
agent_logger = AgentLogger(
capture_prompts=True,
capture_responses=True,
capture_tool_calls=True,
storage="encrypted_s3",
)S2: Curate and filter
Collect 10,000 to 100,000 examples. This is the rule of thumb for SLM fine-tuning. Remove PII, PHI, and sensitive data. Paraphrase named entities and numerical details without compromising the general information content.
The paper cites a range of tools for automated dataset preparation. The key number: 10K examples is enough to start seeing reliable improvements from fine-tuning.
S3: Cluster into tasks
Use unsupervised clustering on the collected prompts. Identify recurring patterns. Common clusters from real agents include:
| Task cluster | Example | Frequency |
|---|---|---|
| Intent recognition | Is this a search or a create request? | 30-40% |
| Entity extraction | Pull date, title, tags from user input | 20-30% |
| Response formatting | Turn tool output into natural language | 15-25% |
| Tool selection | Which tool matches this request? | 10-20% |
| Code generation | Generate boilerplate for specific patterns | 10-15% |
The granularity depends on your agent. A coding agent will have different clusters than a customer support agent.
S4: Select candidate SLMs
For each cluster, pick one or more candidate SLMs. Criteria: instruction following, reasoning, context window, licensing, deployment footprint.
The paper gives specific model recommendations across multiple size tiers:
| Size | Model | Why it works for agents |
|---|---|---|
| 1-3B | SmolLM2, Hymba-1.5B | Intent classification, simple routing |
| 3-8B | Phi-3.5, DeepSeek-R1-Distill | Tool calling, instruction following, code gen |
| 7-14B | Phi-4, Nemotron-H 9B | Complex extraction, multi-step formatting |
| 8-14B | xLAM-2-8B, Gemma 4 12B | Agent-specific fine-tuned, tool use |
S5: Fine-tune specialized SLMs
Prepare a task-specific dataset for each cluster. Use LoRA or QLoRA to keep costs down. Optionally use knowledge distillation — train the SLM to mimic the LLM’s outputs on the specific task.
This is where 10K examples become enough. A focused dataset for a single task cluster — say, formatting tool calls — lets a 3B model match a frontier model on that specific skill. The model doesn’t need to know the entire internet. It needs to know how to turn “create a note titled groceries with tags shopping” into create_note(title="groceries", tags=["shopping"]).
S6: Iterate and refine
Retrain the SLMs and the router periodically with new data. Agent usage patterns change. New clusters emerge. Old ones become irrelevant. The continuous improvement loop returns to S2.
This is the step most teams skip. They convert once and declare victory. Agent tasks drift. The SLM drifts with them if you don’t retrain.
What do the case studies show?
NVIDIA ran three case studies on real open-source agents. The numbers are honest. No 100% replacement claimed.
| Agent | What it does | Queries replaceable by SLMs |
|---|---|---|
| MetaGPT | Multi-agent software company simulation | ~60% |
| Open Operator | Workflow automation (API calls, orchestration) | ~40% |
| Cradle | GUI automation via screenshot input | ~70% |
MetaGPT at 60% makes sense. Its multi-agent setup involves structured role-based interactions. Drafting system design documents, generating boilerplate code, producing test cases — these are repetitive and follow templates. Architectural reasoning and adaptive debugging stay on LLMs.
Open Operator at 40% is the lowest. Workflow automation involves multi-step reasoning, maintaining conversation flow, and orchestrating across services. The paper notes that “simple command parsing and routing” work on SLMs but “complex tasks requiring multi-step reasoning” do not. That 40% floor is the baseline for tasks that are genuinely deterministic.
Cradle at 70% is the highest. GUI automation involves repetitive click sequences and pre-learned interaction patterns. Dynamic GUI adaptation and unstructured error resolution still need LLMs.
The pattern is clear: the more repetitive and structured the task, the higher the replacement rate.
Where should you NOT use SLMs?
The paper acknowledges alternative views. Three worth discussing.
Complex multi-step reasoning with branching logic. When your agent has to evaluate five possible paths, simulate outcomes, and pick one, a 7B model will hallucinate more than a 70B model. The representational capacity difference is real. The paper’s position: use a large model for this. SLM everything else.
Broad world knowledge for obscure topics. If your agent needs to reason about a niche domain with sparse training data, a small model is more likely to fabricate. The paper is honest about this. SLMs match LLMs on well-scoped tasks but lag on open-domain reasoning.
Long-horizon agentic tasks. Maintaining coherence across hundreds of tool calls over hours is hard. Small models forget context faster. The paper suggests this is an engineering problem — structured context management, not model capability — but acknowledges that larger models handle it better today.
The tradeoff section in the paper quotes their correspondence: reasoning LLMs are sometimes presented as an alternative to agentic workflows. They aren’t. A reasoning LLM is still an LLM on the same expensive infrastructure. The arithmetic doesn’t change.
What does this mean for your agent stack?
The practical outcome is a tiered architecture. Three layers, three model sizes.
Layer 1: SLM (1-8B). Runs locally or on a cheap endpoint. Handles intent recognition, entity extraction, tool call formatting, response templating. This is 40-70% of your agent’s model calls.
Layer 2: Mid model (8-14B). Handles classification, summarization, structured analysis. Deployed on a single GPU or API endpoint. This is 20-35% of your calls.
Layer 3: LLM (70B+ or frontier API). Handles complex planning, multi-step reasoning, and novel tasks. Called sparingly. This is 5-15% of your calls.
The agent harness routes between layers. It doesn’t guess. The routing logic is deterministic. Based on task type, not model votes. Intent recognition and formatting always hit Layer 1. Architectural reasoning hits Layer 3. The harness is the differentiator, not the model.
What is the cost difference?
The paper conservatively estimates 10-30x cost reduction. Here is what that looks like for a typical agent at 100K tool calls per month.
| Model tier | Cost per 1M input tokens | Cost per 100K tool calls | Monthly cost |
|---|---|---|---|
| GPT-4o | Rs 36 ($0.435) | ~Rs 36,000 ($432) | Rs 3.6 lakh ($4,320) |
| Claude 3.5 Sonnet | Rs 24 ($0.30) | ~Rs 24,000 ($288) | Rs 2.4 lakh ($2,880) |
| Fine-tuned 7B SLM (API) | Rs 3.6 ($0.044) | ~Rs 3,600 ($43) | Rs 36,000 ($432) |
| Fine-tuned 7B SLM (local) | Rs 0 (fixed hardware) | ~Rs 0 (marginal) | Rs 18,000 ($216) electricity |
Numbers assume 1K input tokens per tool call on average. Local SLM pricing assumes a Mac Mini or similar running 24/7. SLM API pricing assumes provider rates for fine-tuned models.
The gap widens as call volume grows. At 500K calls per month, the local SLM still costs Rs 18,000. The GPT-4o setup costs Rs 18 lakh ($21,600).
FAQ
How long does the conversion take? The paper’s algorithm is designed for incremental migration. Start with S1 (instrumentation) which takes a day. By week two you have enough data to cluster. Production routing can begin in 2-4 weeks depending on your agent complexity.
Do I need to be an ML engineer to do this? Not for the first three steps. S1-S3 are engineering work. Logging, clustering, analysis. S4-S5 need ML skills for fine-tuning. S6 is engineering again. You can stop after S3 and use off-the-shelf SLMs without fine-tuning.
What if my agent uses LM agency? NVIDIA’s conversion algorithm implicitly converts LM agency to code agency as part of S1-S3. The instrumentation reveals which calls the model makes that could be deterministic code instead. Replace those first. The SLM substitution follows naturally.
Does this work for voice agents or multimodal agents? The paper focuses on text-based LM agents but notes their position extends to vision-language models. Voice agents add latency constraints that favor SLMs (faster inference on device). The clustering approach works the same way.
Related Posts
- The year small models ate the benchmarks. The trend that made SLM conversion viable: 4B models now beat 70B models from last year on specific tasks.
- Agent harness architecture: the 15 essential jobs. The harness that routes between model tiers is the real differentiator. This post covers what it needs to do.
- Loop engineering: the 8 canonical production agent loops. Understanding the loop patterns helps identify which parts of your agent loop can be hardened into code agency.
Converting an agent from LLM-centric to SLM-first architecture requires instrumentation, task clustering, and a routing harness. The conversion algorithm (S1-S6) from NVIDIA Research provides a structured path. Expect 40-70% query replacement, 10-30x cost reduction, and a tiered agent architecture where model size is a routing parameter, not an identity.
This article was published on Agentic Up (https://agenticup.dev): practical guides for developers and founders building with AI agents. Reach me at [email protected]
