The phase transition hiding in your agent's skill library
Research from UBC shows skill selection accuracy stays high up to ~90 skills, then drops sharply. Not gradually. Semantic confusability drives the collapse, not library size alone.
TL;DR: UBC researchers formalized converting multi-agent systems into single agents with skill libraries, saving 54% tokens and 50% latency while matching accuracy. But there’s a catch: skill selection accuracy stays stable up to a critical threshold (~80-90 skills), then collapses sharply . Not gradually. Semantic confusability among similar skills drives the collapse, not library size alone. Hierarchical routing restores accuracy from ~45% to ~85% at 120+ skills.
Key takeaways:
- MAS → SAS compilation saves 54% tokens and 50% latency while matching accuracy on GSM8K, HumanEval, and HotpotQA
- Selection accuracy follows a phase transition: stable up to ~80-90 skills, then drops to ~20% at 200 skills
- Semantic confusability, not library size, drives the collapse. Unique skills stay at 100% accuracy
- Hierarchical routing (coarse-to-fine) restores ~72-85% accuracy at 120+ skills
- Keep flat skill libraries under ~50 skills, audit overlap, adopt hierarchy beyond that
Here’s a puzzle I keep running into.
Every team I talk to is building multi-agent systems. They have a writer agent, a reviewer agent, a coder agent, a debugger agent, a planner agent. The architecture diagrams look like network topologies. Token costs look like AWS bills.
The natural alternative, one agent with a library of skills, gets dismissed as a toy. It shouldn’t be.
What the paper found
Xiaoxiao Li at UBC (CIFAR AI Chair) formalized something I’ve felt but couldn’t prove. A multi-agent system can be compiled into a single agent with skills by encoding each agent’s behavior as a skill descriptor and execution policy. The math works out: pipeline, router-workers, and iterative refinement architectures are all “compilable.”
The results on three benchmarks:
| Metric | MAS | SAS (compiled) | Change |
|---|---|---|---|
| GSM8K accuracy | 94.0% | 92.0% | -2.0% |
| HumanEval accuracy | 100.0% | 100.0% | 0.0% |
| HotpotQA accuracy | 84.0% | 88.0% | +4.0% |
| Avg. tokens | ~2,389 | ~1,060 | -53.7% |
| Avg. latency | ~9,821ms | ~5,022ms | -49.5% |
| API calls | 3-4 | 1 | -75% |
The SAS uses one API call. One context window. No inter-agent handoff overhead.
That alone is useful. But the paper’s real contribution is the question it asks next: what happens when the skill library grows?
The phase transition
Li tested selection accuracy across libraries from 5 to 200 skills, controlling for semantic similarity. The pattern is not a gentle slope. It’s a cliff.
At |S| ≤ 20, accuracy stays above 90%. Between 30-75 skills, it holds steady. At 80-90 skills, it starts dropping. At 200 skills, accuracy is about 20%.
The fitted scaling law:
ACC ≈ α / (1 + (|S|/κ)^γ)
Where κ ≈ 85-92 (the capacity threshold where accuracy halves) and γ > 1.5 (super-linear decay (the phase transition)).
The cognitive science parallel is deliberate. Hick’s Law says human reaction time scales logarithmically with choice count. It breaks down beyond ~8 options. Li shows LLM skill selection has an analogous capacity limit, with a phase transition instead of a log curve.
What actually breaks selection
Here’s the part that matters for builders. Library size alone isn’t the problem. Confusability is.
In a controlled experiment, Li created skill libraries with three conditions:
- No competitors: Each skill is semantically unique
- Low confusability: 1 semantically similar “competitor” per skill
- High confusability: 2 competitors per skill
At |S| = 20, the no-competitor condition scored 100% accuracy. Adding one competitor dropped it 7-30%. Adding two dropped it 17-63%.
This means: if you have 10 well-separated skills, selection is near-perfect. If those 10 skills include “summarize text,” “summarize code,” “summarize meeting notes,” and “summarize email,” you’re in the danger zone.
Skill descriptors matter. “Process data” is not a skill name. “Compute rolling 7-day average across time-series metrics” — that’s a skill.
The fix: hierarchy
Flat selection degrades because the model must scan all options at once. Hierarchical routing breaks the decision into two stages:
- Pick a cluster (10-40 categories, below the capacity threshold)
- Pick a skill within that cluster (2-3 options, trivial by comparison)
At |S| = 120, flat accuracy is ~45-63%. Hierarchical accuracy is ~72-85%. For GPT-4o-mini, the improvement is +37-40% absolute.
The paper tested two hierarchies: naive domain grouping and confusability-aware grouping. Both worked similarly when domain boundaries happened to align with confusability clusters. The key design rule: first-stage categories must be semantically distinct. Put confusable skills together in the second stage where the small pool makes selection easy.
When NOT to use this
Three conditions make compilation fail:
- Parallel sampling: Independent agents running in parallel, best-of-n selection
- Private information: Agents with hidden state
- Adversarial objectives: Debate, opponent modeling
These require true parallelism or information asymmetry that a single context window can’t replicate. For everything else (pipelines, router-workers, iterative refinement), compilation is faithful.
The practical playbook
- Keep flat libraries under ~50 skills. Above that, accuracy degrades non-linearly.
- Audit semantic overlap before adding skills. If two skills sound similar, merge or differentiate them.
- Adopt hierarchy at scale. Group by domain, keep first-stage categories distinct, keep second-stage pools tiny.
- Write specific descriptors. “Compute sum of values” beats “process data.”
- Match model to library size. Stronger models have slightly higher capacity thresholds, but hierarchy helps all models.
What’s still open
This is a UBC technical report, not a peer-reviewed paper. The experiments use synthetic skill libraries, not real-world agent skills from production systems. The model coverage is limited to GPT-4o-mini and GPT-4o. End-to-end task accuracy (not just selection accuracy) needs more study.
But the phase transition finding is robust: R² > 0.97 on the scaling law fit, consistent across two models, and grounded in established cognitive theory.
The implication for builders: if your agent has more than 50 skills and you’re using flat selection, you’ve already hit the phase transition. You just haven’t measured it yet.
FAQ
What is a skill in this context? A skill is a schema-bounded operation with a semantic descriptor, an input-output signature, and an execution policy. It’s like a tool, but instead of being automatically triggered, the model must choose it based on semantic match.
Does this mean multi-agent systems are obsolete? No. Many production agent systems need parallel execution, private state, or heterogeneous models. All of those fail the compilability conditions. But for sequential pipeline architectures, a compiled SAS matches performance at half the cost.
How do I know if my skills are too similar? If you can’t describe each skill in one sentence that distinguishes it from all others, they’re too similar.
Is this the same as Anthropic’s agent skills? The paper builds on Anthropic’s concept of agent skills and formalizes the scaling behavior that Anthropic’s documentation doesn’t cover.
Related Posts
- When better models aren’t better agents. The scaling of agent capability doesn’t follow model capability curves either.
- Code as agent harness: a survey of 7 production systems. Different approaches to agent architecture, including single-agent and multi-agent patterns.
- Why agents break in production. The failure modes that matter when you ship an agent system.
This article was published on Agentic Up (https://agenticup.dev): practical guides for developers and founders building with AI agents. Reach me at [email protected]
