Skills · Data & AI

Eval Harness First

Unverified●31/40

Build the evaluation harness that gates every fine-tuning run — golden sets, per-failure-mode graders, judge calibration, and base-model baselines. Use when starting a fine-tuning effort, when converting traces into an eval set, or when calibrating a judge against human labels.

Originally by wshobson · MIT

Claude Code◐PartialHas SKILL.md but declares no allowed-tools — Claude Code will ask for permission each time

Cursor◐PartialPlain prose you can paste in — but no Cursor rules file

Codex◐PartialPlain prose you can paste in — but no AGENTS.md

Gemini CLI◐PartialPlain prose you can paste in

Copilot◐PartialPlain prose you can paste in — but no Copilot instructions file

npx agentalley add eval-harness-first

This command does not work yet — the CLI is still being built. Until then, use Raw in the reader below to take the file.

Who is stuck, and on what

The whole source

No sign-in, no blur, nothing truncated

eval-harness-first/SKILL.md210 lines7.8 KBRawView on GitHub

Frontmatter — 2 properties

name	eval-harness-first
description	Build the evaluation harness that gates every fine-tuning run — golden sets, per-failure-mode graders, judge calibration, and base-model baselines. Use when starting a fine-tuning effort, when converting traces into an eval set, or when calibrating a judge against human labels.

1	---
2	name: eval-harness-first
3	description: Build the evaluation harness that gates every fine-tuning run — golden sets, per-failure-mode graders, judge calibration, and base-model baselines. Use when starting a fine-tuning effort, when converting traces into an eval set, or when calibrating a judge against human labels.
4	---A5 — No allowed-tools declared — no way to tell what this skill may touch
5
6	# Eval Harness First
7
8	The Phase 0 gate for the whole plugin:
9	`finetuning-method-selection` and every downstream
10	skill assume this harness exists before a training
11	config gets written. The harness is not a run-end
12	side artifact — it is the data-curation engine. The
13	same labeled traces that build the goldens feed
14	training data, minus an explicit holdout.
15
16	Input: production/agent traces if they exist, or
17	a task spec if they don't, plus labelers willing to
18	grade ≥100 examples.
19	Output format: the `eval/` directory below —
20	goldens, graders, drift suite, and the base-model
21	baseline that later phases gate on.
22
23	## The Gate
24
25	No eval harness, no fine-tune. Skip to a training
26	config and there is nothing to measure against,
27	nothing to catch regressions, and no labeled data
28	to train on. The flywheel:
29
30	1. Collect traces — production/agent spans, or
31	synthetic tasks if none exist yet.
32	2. Error analysis — open coding on ≥100 traces,
33	axial coding into 4–8 failure buckets.
34	3. One grader per bucket — deterministic first;
35	calibrated LLM-judge only for genuinely
36	subjective criteria.
37	4. Prioritize by frequency × severity × value.
38	5. **The labeled traces feed dataset curation, minus
39	an explicit holdout.** Every `eval/goldens.jsonl`
40	ID stays excluded from training data by ID.
41	6. Train.
42	7. Re-run the same harness on the checkpoint —
43	not a different, looser one.
44	8. Drift detection feeds back to step 2 — new
45	production failure modes re-open error analysis.
46
47	Steps 2–4 build the harness; steps 5–8 are why it
48	must exist first — it is both the training data
49	source and the checkpoint's exit gate.
50
51	## Building Goldens
52
53	- From traces, when they exist: run error
54	analysis — open coding on ≥100 real traces (read
55	them, tag failures in your own words, no fixed
56	taxonomy yet), then axial coding to collapse those
57	tags into 4–8 named failure buckets. Fewer than 4
58	means the coding pass was too shallow; more than 8
59	means buckets need merging. Exception:
60	single-failure-surface tasks (e.g. strict-schema
61	extraction) may land at 1–2 buckets with per-field
62	sub-metrics inside one grader — don't invent
63	artificial splits with no evidence behind them.
64	- Synthetic, when traces don't exist yet:
65	dimension-based generation — enumerate the axes
66	that matter (task type, difficulty, edge case,
67	persona) and sample the cross-product; free-
68	generated prompts cluster around whatever's
69	easiest to write.
70	- Goldens are versioned like code — commit
71	`eval/goldens.jsonl`, diff it in review, tag it per
72	release. It doubles as the CI regression suite.
73
74	## Graders
75
76	One grader per failure bucket from error analysis —
77	not one for the whole eval set. A single blended
78	score hides which bucket regressed.
79
80	- Deterministic first. Regex, schema validation,
81	or execution checks are cheaper, reproducible, and
82	need no calibration.
83	- **LLM-judge only for genuinely subjective
84	criteria** — tone, faithfulness, "which response
85	is better" — where no deterministic check can
86	express it.
87	- Binary pass/fail over Likert. A 1–5 or 1–10
88	scale is noisier to calibrate and harder to apply
89	consistently; collapse to pass/fail.
90	- **Drift-suite MMLU-style scoring: prefer logprob
91	over generate-and-extract** — a tight token budget
92	makes generate-and-extract parse-brittle for models
93	that preamble, conflating format compliance with
94	the knowledge being measured. Templates for all
95	four grader shapes and this scoring note:
96	`references/grader-templates.md`.
97
98	## Judge Calibration Is a Prerequisite
99
100	Any bucket routed to an LLM-judge needs calibration
101	before its verdicts count for anything beyond
102	exploration — a hard prerequisite, not a
103	nice-to-have. **N/A when no bucket routes to a
104	judge** — an all-deterministic harness has nothing
105	to calibrate; state that rather than leaving this
106	section unaddressed.
107
108	- Label ≥100 items, split train/dev/**sealed
109	test** (report once, no re-touching after).
110	- Report TPR and TNR, not one blended accuracy
111	number — a judge can hit 90% by always saying
112	"pass" on a skewed set.
113	- Pin the judge to a fixed model snapshot and
114	recalibrate on judge-model change, quarterly
115	regardless.
116	- **The judge must come from a different model family
117	than the model under test.**
118	- A judge that misses the agreed TPR/TNR bar ships
119	advisory-only — flags for human review, never
120	gates a promotion. Full protocol, bias correction,
121	and recalibration checklist:
122	`references/judge-calibration.md`.
123
124	## The Baseline
125
126	Before Phase 1 (method selection) starts, run the
127	full harness — goldens plus the capability-drift
128	suite — against the unmodified base model. This is
129	the number every later checkpoint gets compared
130	against.
131
132	`eval/baseline-<model>.json` is the gate token. No
133	baseline file, no comparison basis for
134	`checkpoint-promotion` — a checkpoint that "looks
135	better" against nothing measured isn't a finding.
136
137	## Directory Contract
138
139	```
140	eval/
141	├── goldens.jsonl # labeled traces + synthetic goldens, versioned
142	├── graders/ # one module per failure bucket
143	│ ├── schema_compliance.py
144	│ ├── exact_match.py
145	│ └── rubric_judge.py
146	├── drift-suite.yaml # frozen benchmarks + 200-500 domain-adjacent items
147	└── baseline-<model>.json # gate token: harness + drift suite vs the base model
148	runs/
149	└── <run-id>/
150	└── results.json # per-run harness output, one per checkpoint
151	```
152
153	`eval/` persists across runs and lives outside
154	`runs/` — the fixed measuring stick, not a run
155	artifact. `runs/` is disposable; `eval/` is not.
156	Never let a run script write into `eval/`. **Canonical
157	location:** every per-trace `results.json` — the
158	Phase 0 baseline included — lives at
159	`runs/<run-id>/results.json`, never under
160	`eval/runs/...`; an instruction requesting the
161	latter is wrong, not this contract.
162
163	### Phase 0 Exit Checklist
164
165	Before `finetuning-method-selection`, confirm:
166
167	1. ≥100 traces open-coded; 4–8 failure buckets (N/A
168	floor for synthetic goldens on a single-failure-
169	surface task — see the Building Goldens exception;
170	bucket count then comes from post-baseline error
171	analysis instead).
172	2. `eval/goldens.jsonl` committed and versioned.
173	3. One grader per bucket, deterministic first.
174	4. Judges calibrated — TPR/TNR, snapshot pinned,
175	different family (**N/A when no bucket routes to
176	an LLM-judge**; state that explicitly).
177	5. `eval/drift-suite.yaml` frozen.
178	6. `eval/baseline-<model>.json` written.
179
180	Missing any of the six (or its stated N/A)? Not
181	Phase 0 complete — `/finetune` checks the baseline
182	file before a run.
183
184	## Related Skills
185
186	General-purpose evaluation guidance (dashboards, A/B
187	testing, non-fine-tuning harnesses) lives in the
188	`llm-application-dev` plugin's `llm-evaluation`
189	skill — this skill covers only the fine-tuning
190	coupling: goldens that double as training data, and
191	the baseline that gates a checkpoint.
192
193	- `finetuning-method-selection` — routes here first.
194	- `dataset-curation` — formats these traces into
195	training rows.
196	- `trace-to-training-data` — turns graded traces into
197	training examples.
198	- `checkpoint-promotion` — consumes
199	`baseline-<model>.json`, re-runs this harness on
200	each candidate checkpoint.
201
202	## References
203
204	- `references/grader-templates.md` — runnable grader
205	examples per shape, plus a `drift-suite.yaml`
206	example and MMLU logprob-scoring note.
207	- `references/judge-calibration.md` — the
208	calibration protocol, including the all-
209	deterministic N/A path.
210

Reviews

Installed this one?Write the first review and take the Trailblazer badge.

Reviews only open after a real install, so this is empty — and we leave it empty rather than invent one.

Alternatives

Also in Data & AI

Task Coordination StrategiesDecompose complex tasks, design dependency graphs, and coordinate multi-agent work with proper task descriptions and workload balancing. Use this skill when breaking down work for agent teams, managing task dependencies, or monitoring team progress.Data & AI · MIT◐◐◐◐◐●35/40 Ebay Seller ToolsData & AI · MIT·····●34/40 Tough Decision Advisor: Every Angle ConsideredHand in a decision you're stuck on. Get back a clear breakdown of every angle — the trade-offs, the risks, the blind spot, and a recommended path.Data & AI · Business · MIT license●····●32/40 DHDNA Profiler — Cognitive Pattern ExtractionPaste any email, proposal, or note someone wrote, and get back a plain-language read on how they think, what drives their decisions, and how they communicate.Data & AI · Research & analysis · MIT license●····●32/40

Verification score

Safety23/25

A1Destructive commands6/6
A2Data exfiltration6/6
A3Hardcoded secrets4/4
A4Prompt-injection guard5/5
A5Permission footprint · line 42/4

Transparency8.2/15

B1No obfuscation5/5
B2Declares dependencies0/4
B3Documented3.2/4
B4Pinned version0/2

Effectivenessnot scored yet

Needs real runs and real usage before it can be measured.

Costnot scored yet

Needs real runs and real usage before it can be measured.

Real-world usenot scored yet

Needs real runs and real usage before it can be measured.

31.2/40 on the part a machine can score. The badge only reaches Verified once Effectiveness and Cost have actually run.

wshobson

Original author

LicenceMIT

Repowshobson/agents

Repo stars39,748

Licence found inrepo

View on GitHub This is mine

README badge

The SVG endpoint is still being built. This is the final markdown shape.

[![AgentAlley](https://agentalley.ai/badge/eval-harness-first.svg)](https://agentalley.ai/s/eval-harness-first)

Submit your own