Bottleneck in model routing is information, not reasoning
A new paper proves execution-grounded feedback beats smarter prompting for model routing. The C-A-F loop yields +15.3% over vanilla and generalizes to unseen tasks.
TL;DR: A new paper from NUS and Alibaba proves that model routing fails because of information deficit, not reasoning limits. The C-A-F loop (Context-Action-Feedback) turns routing into a contextual bandit problem where every decision improves the next one. ACRouter, the paper’s implementation, achieves +15.3% over vanilla routing and is the only router that maintains performance on out-of-distribution tasks.
I spent months assuming the best way to pick a model for a coding task was to know which model was strongest overall. Opus 4.6 leads the benchmarks. Pick Opus.
Then I saw the performance heatmap from the Agent-as-a-Router paper.
Claude Opus 4.6 wins on average at 42.9%. But GLM-5 beats it by 86% on algorithm design. Qwen3-Max beats it by 111% on test generation. Kimi-K2.5 beats it by 30% on data science. Five different models are the best choice across nine coding dimensions.
No single model dominates. And manually picking the right one for every task doesn’t scale. The question is: how do you build a system that learns which model to use for which task, and gets better with every decision?
This paper has a concrete answer. It’s useful enough to change how I think about model selection in agent pipelines.
Key takeaways:
- The bottleneck in model routing is information deficit, not reasoning failure — adding performance stats to a vanilla LLM router yields +15.3% relative gain
- The C-A-F loop (Context-Action-Feedback) formalizes routing as a contextual bandit with cumulative regret as the metric
- ACRouter achieves 49.98% AvgPerf on in-distribution tasks and 62.50% on OOD tasks, beating static routers by a wide margin on both
- Static classifiers collapse on OOD tasks (8-21% AvgPerf) — they overfit to the training distribution
- The C-A-F pattern generalizes to tool selection, API routing, prompt strategy, and thinking-effort allocation
Why does model routing fail today?
Most routing systems treat model selection as a static classification problem. You train a classifier on a fixed dataset, deploy it, and it stays frozen. The LLM-as-a-Router approach uses a capable model like Claude Sonnet 4.6 to pick the best model for each incoming task.
The paper runs an ablation that isolates why these routers fall short of the oracle upper bound. The vanilla router scores 41.41 AvgPerf%. Adding task dimension information barely moves the needle — 41.18. But adding per-dimension performance statistics from a held-out probing set jumps it to 47.74. A +15.3% relative gain.
The same statistics also power a simple heuristic router (DimensionBest) which scores 47.50. The LLM router with the same information exceeds it. The gap to the oracle (57.00) isn’t about reasoning capability. It’s about missing information that only execution can produce — actual verification of the selected model’s output, in a sandbox, for the specific task at hand.
This is the paper’s central insight: the bottleneck is information deficit, not reasoning failure. A router that can’t learn from its own decisions is structurally incapable of closing the gap.
What is the C-A-F loop?
The paper formalizes routing as a sequential decision process they call the Context-Action-Feedback loop:
- Context — the task description plus all accumulated experience from prior decisions
- Action — selecting which model to invoke from the pool
- Feedback — the verified performance score and cost from executing that model’s output
- The feedback feeds back into the context for the next task
This is equivalent to a contextual multi-armed bandit. Each task is an arm pull. The reward combines performance and cost. The metric is cumulative regret — the running gap to a hypothetical oracle that always picks the right model.
The loop is simple to state and surprisingly powerful. Every routing decision produces new information. The information is accumulated in memory. The memory informs the next decision. Static routers can’t do this because their information state is frozen at deployment.
Kitchen analogy: The C-A-F loop is the expediter at a busy pass. The expediter sees the ticket (context), decides which station gets it (action), watches the plate come back (feedback), and remembers that this station handles this type of dish well for the next ticket. Over a service, the expediter gets faster. A static routing policy is a cook who ignores the tickets and sends every order to the same station.
How do you build ACRouter’s three modules?
ACRouter instantiates the C-A-F loop with three components.
Orchestrator. Makes the routing decision by combining three signals: the DimensionBest prior (per-dimension performance from a probing set), the top-10 historical neighbors retrieved from Memory via cosine kNN on task embeddings (voyage-code-3 or BGE-large), and task metadata. The core policy is a Qwen3.5-0.8B model fine-tuned on the probing set, combined with heuristic rules via weighted voting. It’s deliberately cheap — 0.8B parameters is tiny by frontier model standards.
Verifier. Evaluates model output and aggregates signals into a unified performance score. For executable tasks, it runs the output in a sandbox (AST parsing for code, Docker-based execution for agentic programming). For non-executable tasks, it uses LLM-as-Judge. The key design choice is that verification is automatic and happens every loop — no human in the loop, no delayed feedback.
| Scoring method | Used for |
|---|---|
| Sandbox execution (pass@1) | Code generation, bug fixing, refactoring |
| AST parsing | Syntax validation, code understanding |
| LLM-as-Judge | Test generation, code design |
| Docker-based sandbox | Agentic programming (OOD tasks) |
Memory. An online vector store keyed by task embeddings. Each entry logs the chosen model, performance score, monetary cost, and verification traces. Retrieval uses cosine kNN to find the top-10 most similar past tasks. The store is FIFO-bounded at 20K entries, committed in place after each attempt. Unlike dimension-hashed routing, this embedding-based store enables fine-grained, context-aware decisions — the router sees not just “this task is in the algorithm category” but “this task is similar to the three where GLM-5 outperformed Opus last week.”
The three modules work together in a continuous loop. The Orchestrator routes a task. The Verifier scores the result. The Memory stores the outcome. The Memory feeds the Orchestrator for the next task. No human intervention required.
What happens on out-of-distribution tasks?
This is where the paper earns its keep.
Standard practice is to train a lightweight classifier — LogReg, RouteLLM-BERT, TF-IDF+MLP — on a labeled set and deploy it. On in-distribution tasks (tasks from the same distribution as the training set), these classifiers stay within 1.3% of the best heuristic. Acceptable performance for a cheap solution.
On out-of-distribution tasks — agentic programming requiring multi-step planning, file navigation, and iterative debugging — they collapse. LogReg drops to 19.64%. RouteLLM-BERT hits 21.43%. TF-IDF+MLP scores 13.39%. All below random (31.25%).
| Router | ID AvgPerf | OOD AvgPerf |
|---|---|---|
| ACRouter | 49.98 | 62.50 |
| LinUCB (online bandit) | 46.84 | 49.82 |
| Qwen3.5-0.8B-Finetuned | 46.41 | 55.36 |
| LogReg | 47.26 | 19.64 |
| RouteLLM-BERT | 47.22 | 21.43 |
| Always-Opus 4.6 | 43.83 | 57.14 |
| Random | 38.75 | 31.25 |
The static classifiers overfit to the training distribution. When the task type shifts, they have no mechanism to adapt. Contextual bandits (LinUCB, LinTS) survive better because they update online, but they lack the context-aware reasoning that ACRouter’s Memory provides. ACRouter is the only router that maintains strength on OOD tasks.
The practical implication: if your routing system only sees the same types of tasks in production that it saw in training, a cheap static classifier is fine. If the task distribution shifts — and it will, because real-world usage always introduces new patterns — you need adaptive routing.
Who is this not for?
If you always use one model for everything, none of this matters. Many teams should keep doing that. A routing layer adds complexity, latency, and another component to debug.
If your task distribution is narrow and stable — same type of coding task, same domain, same output format — a static classifier trained on a representative sample will perform adequately at lower cost.
The C-A-F loop costs more per decision than a static lookup. The Verifier needs a sandbox. The Memory needs storage and retrieval. The Orchestrator needs inference budget. The paper’s Perf/$ numbers show ACRouter at 3.79 on ID tasks versus 6.27 for LogReg. You pay for the adaptability.
Tradeoff: The C-A-F loop is most valuable when your task distribution shifts over time. If you build agents that handle heterogeneous requests — different coding languages, different complexity levels, different output requirements — the information accumulation pays for itself. If every task looks like the last one, a static policy is cheaper and nearly as good.
What should you build today?
Start with the ablation that costs nothing. The paper shows that simply adding per-dimension performance statistics to your routing prompt yields a +15.3% gain. Before building a full C-A-F loop, collect your execution data and surface it at decision time.
If you want the full loop, the architecture is straightforward: an Orchestrator (cheap model + nearest-neighbor retrieval), a Verifier (sandbox execution + automated scoring), and a Memory (vector store with FIFO eviction). The paper’s implementation uses a 0.8B fine-tuned model, a Docker sandbox, and a simple BGE-large embedding store. None of it requires frontier infrastructure.
The broader insight is more valuable than the implementation. The C-A-F pattern — decide, execute, verify, remember — applies to any agent that must choose among heterogeneous options. Tool selection. API endpoint routing. Prompt strategy. Thinking-effort allocation. Every time your agent makes a choice, it should observe the outcome and feed it back into the next choice.
The C-A-F loop is a general pattern for any agent that must choose among options and improve from experience. The key architectural insight: separate the decision (Orchestrator) from the verification (Verifier) from the memory (Memory) so each can improve independently.
FAQ
What is the main finding of the Agent-as-a-Router paper? The paper proves that the bottleneck in LLM model routing is information deficit — the router lacks execution-grounded feedback about how well each model actually performed on similar tasks — rather than a reasoning failure. Simply adding per-dimension performance statistics to a vanilla LLM router improves it by 15.3% relative, beating a heuristic router with the same prior information.
What is the C-A-F loop? The Context-Action-Feedback loop formalizes routing as a sequential decision process. At each task, the router observes the context (task description plus accumulated experience), selects a model to invoke (action), receives verification feedback (score and cost), and stores that feedback in memory for the next decision.
Does ACRouter cost more than always picking the best single model? No. ACRouter achieves higher average performance than always picking Opus 4.6 (49.98% vs 43.83% on in-distribution tasks) while costing less per unit of performance (Perf/$ 3.79 vs 1.29). The Verifier and Memory add some overhead, but smarter routing saves more than it spends.
How do static routers perform on out-of-distribution tasks? Poorly. Lightweight classifiers like LogReg and RouteLLM-BERT collapse on OOD tasks, dropping to 8-21% AvgPerf — below random (31.25%). They overfit to the training distribution. ACRouter maintains 62.50% on OOD, the highest among all routers tested.
Can the C-A-F loop be used for things other than model routing? Yes. The paper explicitly generalizes the pattern to tool selection, API endpoint selection, prompt strategy selection, and thinking-effort allocation. Any agent that must choose among heterogeneous options and improve from feedback can use the same Context-Action-Feedback structure.
Related Posts
- Model-agnostic agents: when the same architecture works across 7 providers. Building agents that switch models freely — a complementary approach to routing.
- Loop engineering: production agent loop patterns. The 8 canonical patterns for agent orchestration. The C-A-F loop is one more pattern to add.
- NVIDIA rebuilt its stack for agents. Here’s what that means.. Hardware trends that make per-task model routing more practical — cheaper inference means more decisions to route.
This article was published on Agentic Up (https://agenticup.dev): practical guides for developers and founders building with AI agents. Reach me at [email protected]
