What is the difference between outcome evaluation and process evaluation for agents?

Outcome evaluation checks whether the final task passed. Process evaluation checks whether the agent selected the right skill, followed its steps, composed it with others correctly, and reflected on the output. An agent can pass by trial and error while doing all four dimensions wrong.

What are the four dimensions of agentic skill-use in SkillCoach?

Skill selection (did the agent choose the correct skill?), skill following (did it execute the key steps in order?), skill composition (did it chain skills in the correct workflow?), and skill-grounded reflection (did it validate the output against the skill's specifications?).

How do self-evolving rubrics work?

The rubrics start from a seed template and update based on real agent rollouts. Each iteration uses a calibration set of trajectories to judge, propose patches, and validate on a held-out set. The evolved rubrics become more faithful evaluation instruments over time.

What improvement did SkillCoach's rubrics achieve?

Evolved rubrics improved gold-keypoint coverage by +12pp, reduced hallucination rate from 2% to 0%, improved filtering consistency by +14pp, and provided stronger supervision signals than outcome-only filtering for SFT training.

How does SkillCoach connect to the skill decomposition problem?

The existing skill router post covers the decompose-retrieve bottleneck — getting the right skills to the agent. SkillCoach covers the post-retrieval problem: once the agent has the skills, did it use them correctly? The two problems are sequential and together define the full routing pipeline.

Your agent passed the test. It still used skills wrong.

Final task success hides process failures. SkillCoach introduces self-evolving rubrics across four dimensions of agentic skill-use: selection, following, composition, and reflection.

TL;DR: The SkillCoach paper (Zhu et al., arXiv 2607.01874, July 2026) introduces self-evolving rubrics for evaluating agentic skill-use across four dimensions: selection, following, composition, and reflection. The core insight is that final task success is too coarse. An agent can pass the verifier by trial and error while picking wrong skills, skipping steps, and composing in the wrong order. Process evaluation exposes failures that outcome accuracy hides. Evolved rubrics also produce better SFT training data than outcome-only filtering.

Key takeaways:

Outcome accuracy hides process failures. An agent can pass tests while selecting wrong skills, skipping steps, and ignoring output checks

SkillCoach defines four measurable dimensions: skill selection, skill following, skill composition, and skill-grounded reflection

Self-evolving rubrics improve from seed versions: +12pp keypoint coverage, 0% hallucination rate, +14pp filtering consistency

Process-filtered trajectories produce stronger SFT models than outcome-only filtered data

SkillCoach and the SkillWeaver decomposition problem are sequential. Together they define the full routing pipeline

I shipped a skill router last month. It retrieved the right skills. It composed them in the right order. The final output passed every test.

Then I watched the trajectory.

The agent selected three skills to complete one task. The first was a distractor. It did nothing useful but did not break anything either. The second skill ran, produced a partial output, and the agent sent that output to the third skill without validating it. The third skill’s spec required a pre-flight check. The agent skipped it. The final output passed because the first skill’s useless work happened to leave the pipeline in a passable state.

The agent looked competent in the logs. Final accuracy said 100%. The trajectory told a different story.

This is the problem the SkillCoach paper solves. Final verifier success is too coarse for both evaluation and training. You need process-level evaluation. And the process criteria should come from the skills themselves.

The four dimensions

SkillCoach defines agentic skill-use as a meta-ability with four measurable dimensions. Each one can fail independently.

Skill selection. Did the agent choose the correct skill from the library? In realistic repositories with overlapping skills, agents routinely pick distractors. These are skills that do something related but not what the task requires. The output may still pass if the distractor produces a coincidentally correct result.

Skill following. Did the agent execute the key steps in order? Skills encode domain procedures. Skipping a validation step, reordering a pipeline, or missing a required sub-task are following failures. They are invisible in final accuracy if the output still happens to work.

Skill composition. Did the agent chain multiple skills in the correct workflow? Complex tasks need skills in sequence. The wrong order, missing handoffs, or parallel execution when serial is required. These are composition failures.

Skill-grounded reflection. Did the agent validate the output against the skill’s own specifications? Many skills include post-execution checks. Agents that skip these produce outputs that pass the task verifier but violate the skill’s quality criteria.

The benchmark validates these dimensions. Evolved rubrics improved gold-keypoint coverage by 12 percentage points. Hallucination rate dropped from 2% to 0%. Filtering consistency improved by 14 points. Every dimension improved from seed to evolved version.

Self-evolving rubrics

The mechanism is straightforward. SkillCoach starts with a seed rubric template for each skill-dependent task. It runs real agent rollouts against the task, uses a calibration set to judge trajectories against the current rubric, proposes patches where the rubric misses failures or flags successes incorrectly, and validates the patched rubric on a held-out set.

The rubrics evolve without human annotation. They become more faithful evaluation instruments over successive rounds because they learn from actual agent behaviour in the specific task environment.

This matters because fixed rubrics are brittle. A rubric written before the agent runs cannot anticipate every failure mode the agent discovers through execution. Self-evolving rubrics update as the failure surface reveals itself.

Process quality is not outcome quality

This is the paper’s central distinction.

An external verifier checks whether the task passed. The rubric checks how the agent used its skills to get there. The verifier and the rubric are kept separate. An agent that passes the verifier through trial and error gets a high outcome score and a low process score. An agent that uses skills correctly but produces a borderline output gets a low outcome score and a high process score.

The two signals together tell you what to fix. Low process with high outcome means the agent is groping its way to success. It will fail on the next task. Low outcome with high process means the skills are right but the task difficulty does not match the skills. Raise the skill ceiling, not the evaluation.

This is the tradeoff in the centre. Outcome evaluation tells you whether the agent passed. Process evaluation tells you whether the agent learned anything reusable.

Better training data

The evolved rubrics do more than evaluate. They also filter trajectories for supervised fine-tuning.

Outcome-only filtering treats all verifier-passing trajectories as positive demonstrations. This is how agents learn brittle behaviours. Selecting wrong skills that did not break anything. Skipping steps that happened to be unnecessary for this particular input. Ignoring validation checks that would have caught real errors.

SkillCoach’s rubric-filtered trajectories exclude these. The result is stronger SFT models trained on fewer but higher-quality demonstrations. Trajectories that demonstrate correct skill-use behaviour.

Connecting to the routing pipeline

The existing post on your skill router is decomposing wrong covered the front half of the problem. The SkillWeaver paper from Alibaba showed that decomposition (not retrieval) is the bottleneck in compositional skill routing. Standard decomposition achieves only 34.2% category recall. The fix is retrieve-first-decompose-second.

SkillCoach covers the back half. Once the agent has the right skills, did it use them correctly? Selection, following, composition, reflection.

The two problems are sequential. Without correct decomposition, the agent gets the wrong skills. Without process evaluation, you cannot tell whether the agent used the right skills correctly. The pipeline is incomplete without both.

The phase transition in skill libraries adds a third dimension. Selection accuracy collapses at around 80 to 90 skills. Below that threshold, routing works. Above it, hierarchical grouping is required.

Together, these three papers define the practical limits of skill-based agents today. Decomposition, selection, and process evaluation. Each one is a ceiling. Each one has a measurable fix.

FAQ

Should I replace my test suite with process rubrics? No. The verifier and the rubric serve different purposes. The verifier checks task success. The rubric checks skill-use quality. Keep both. Use the rubric to diagnose failures the verifier misses.

How many trajectories do I need to evolve useful rubrics? The paper uses 10 calibration trajectories per iteration and validates on a held-out set. The exact number depends on task complexity. The seed rubric matters more than the pool size.

Can process rubrics be automated? Yes. That is the paper’s contribution. The rubrics self-evolve without human annotation. The human role is to define the seed template and the external verifier.

What is the tradeoff? Process evaluation adds overhead. You need to log trajectories, run the rubric, and maintain the seed templates. The cost is worth it when failures are invisible in final accuracy. That applies to most production settings.

Does this work for any skill library? The paper tests on SkillsBench and SkillLearnBench tasks. The four dimensions are general, but the seed rubric template needs task-specific framing. Generic rubrics produce generic signals.

Your skill router is decomposing wrong. Decomposition, not retrieval, is the bottleneck in skill routing.
The phase transition hiding in your agent’s skill library. Selection accuracy collapses at around 80 to 90 skills.
Your context window is a workspace, not a filing cabinet. Four context engineering strategies.

This article was published on Agentic Up (https://agenticup.dev): practical guides for developers and founders building with AI agents. Reach me at [email protected].