Agent Result Evaluator

/hub:eval — Evaluate Agent Results

by alirezarezvani · MIT license · ★ 26,349 on GitHub · updated Aug 30

Evaluate and rank agent results by metric or LLM judge for an AgentHub session. Use when the user runs /hub:eval or asks to score, compare, or pick a winner among completed AgentHub agents.

---
name: "eval"
description: "Evaluate and rank agent results by metric or LLM judge for an AgentHub session. Use when the user runs /hub:eval or asks to score, compare, or pick a winner among completed AgentHub agents."
command: /hub:eval
---

# /hub:eval — Evaluate Agent Results

Rank all agent results for a session. Supports metric-based evaluation (run a command), LLM judge (compare diffs), or hybrid.

## Usage

```
/hub:eval                           # Eval latest session using configured criteria
/hub:eval 20260317-143022           # Eval specific session
/hub:eval --judge                   # Force LLM judge mode (ignore metric config)
```

Overview

Rank all agent results for a session. Supports metric-based evaluation (run a command), LLM judge (compare diffs), or hybrid.

Manual install (npx, ZIP)

Claude Codenpx skills add alirezarezvani/claude-skills --skill eval -g -a claude-code

Claude Code (manual)

mkdir -p ~/.claude/skills && curl -sL https://codeload.github.com/alirezarezvani/claude-skills/tar.gz/19392f7a0826 \
  | tar -xz -C ~/.claude/skills --strip-components=4 claude-skills-19392f7a0826/engineering/agenthub/skills/eval

Windows (PowerShell)

New-Item -ItemType Directory -Force "$HOME\.claude\skills" | Out-Null; Invoke-WebRequest "https://m.agentalley.io/zip/a5b1fde8cc6babeec6ac/eval.zip" -OutFile "$env:TEMP\eval.zip"; Expand-Archive "$env:TEMP\eval.zip" -DestinationPath "$HOME\.claude\skills" -Force

Claude.ai / desktopDownload the ZIP and upload it under Settings → Capabilities → Skills.

Codex, Cursor, other agentsPoint them at SKILL.md. The other files are plain text.

Connects with Agent Alley MCP · agentalley.io/mcp

Files included

SKILL.md2.4 KB

Read the files here

alirezarezvani/19392f71 file

SKILL.md2.4 KB

Shared references2SKILL.md reads these from elsewhere in the repo. Not in the ZIP.

Get them on GitHub ↗

SKILL.md

Show the full text81 lines

eval/SKILL.md81 lines · 2.4 KB

PreviewCode

Outline

RawView on GitHub

/hub:eval — Evaluate Agent Results

Rank all agent results for a session. Supports metric-based evaluation (run a command), LLM judge (compare diffs), or hybrid.

Usage

/hub:eval                           # Eval latest session using configured criteria
/hub:eval 20260317-143022           # Eval specific session
/hub:eval --judge                   # Force LLM judge mode (ignore metric config)

What It Does

Metric Mode (eval command configured)

Run the evaluation command in each agent's worktree:

python {skill_path}/scripts/result_ranker.py \
  --session {session-id} \
  --eval-cmd "{eval_cmd}" \
  --metric {metric} --direction {direction}

Output:

RANK  AGENT       METRIC      DELTA      FILES
1     agent-2     142ms       -38ms      2
2     agent-1     165ms       -15ms      3
3     agent-3     190ms       +10ms      1

Winner: agent-2 (142ms)

LLM Judge Mode (no eval command, or --judge flag)

For each agent:

Get the diff: git diff {base_branch}...{agent_branch}
Read the agent's result post from .agenthub/board/results/agent-{i}-result.md
Compare all diffs and rank by:
- Correctness — Does it solve the task?
- Simplicity — Fewer lines changed is better (when equal correctness)
- Quality — Clean execution, good structure, no regressions

Present rankings with justification.

Example LLM judge output for a content task:

RANK  AGENT    VERDICT                               WORD COUNT
1     agent-1  Strong narrative, clear CTA            1480
2     agent-3  Good data points, weak intro           1520
3     agent-2  Generic tone, no differentiation       1350

Winner: agent-1 (strongest narrative arc and call-to-action)

Hybrid Mode

Run metric evaluation first
If top agents are within 10% of each other, use LLM judge to break ties
Present both metric and qualitative rankings

After Eval

Update session state:

python {skill_path}/scripts/session_manager.py --update {session-id} --state evaluating

Tell the user:
- Ranked results with winner highlighted
- Next step: /hub:merge to merge the winner
- Or /hub:merge {session-id} --agent {winner} to be explicit

1	---
2	name: "eval"
3	description: "Evaluate and rank agent results by metric or LLM judge for an AgentHub session. Use when the user runs /hub:eval or asks to score, compare, or pick a winner among completed AgentHub agents."
4	command: /hub:eval
5	---
6
7	# /hub:eval — Evaluate Agent Results
8
9	Rank all agent results for a session. Supports metric-based evaluation (run a command), LLM judge (compare diffs), or hybrid.
10
11	## Usage
12
13	```
14	/hub:eval # Eval latest session using configured criteria
15	/hub:eval 20260317-143022 # Eval specific session
16	/hub:eval --judge # Force LLM judge mode (ignore metric config)
17	```
18
19	## What It Does
20
21	### Metric Mode (eval command configured)
22
23	Run the evaluation command in each agent's worktree:
24
25	```bash
26	python {skill_path}/scripts/result_ranker.py \
27	--session {session-id} \
28	--eval-cmd "{eval_cmd}" \
29	--metric {metric} --direction {direction}
30	```
31
32	Output:
33	```
34	RANK AGENT METRIC DELTA FILES
35	1 agent-2 142ms -38ms 2
36	2 agent-1 165ms -15ms 3
37	3 agent-3 190ms +10ms 1
38
39	Winner: agent-2 (142ms)
40	```
41
42	### LLM Judge Mode (no eval command, or --judge flag)
43
44	For each agent:
45	1. Get the diff: `git diff {base_branch}...{agent_branch}`
46	2. Read the agent's result post from `.agenthub/board/results/agent-{i}-result.md`
47	3. Compare all diffs and rank by:
48	- Correctness — Does it solve the task?
49	- Simplicity — Fewer lines changed is better (when equal correctness)
50	- Quality — Clean execution, good structure, no regressions
51
52	Present rankings with justification.
53
54	Example LLM judge output for a content task:
55	```
56	RANK AGENT VERDICT WORD COUNT
57	1 agent-1 Strong narrative, clear CTA 1480
58	2 agent-3 Good data points, weak intro 1520
59	3 agent-2 Generic tone, no differentiation 1350
60
61	Winner: agent-1 (strongest narrative arc and call-to-action)
62	```
63
64	### Hybrid Mode
65
66	1. Run metric evaluation first
67	2. If top agents are within 10% of each other, use LLM judge to break ties
68	3. Present both metric and qualitative rankings
69
70	## After Eval
71
72	1. Update session state:
73	```bash
74	python {skill_path}/scripts/session_manager.py --update {session-id} --state evaluating
75	```
76
77	2. Tell the user:
78	- Ranked results with winner highlighted
79	- Next step: `/hub:merge` to merge the winner
80	- Or `/hub:merge {session-id} --agent {winner}` to be explicit
81