AgentAlley

Skills · Data & AI

Trace To Training Data

Unverified●31/40

Convert evaluation traces and production logs into SFT examples and preference pairs. Use when graded traces or failure examples exist and need to become training data, when applying rejection sampling to model outputs, or when building DPO pairs from passing and failing runs.

Originally by wshobson · MIT

Claude Code◐PartialHas SKILL.md but declares no allowed-tools — Claude Code will ask for permission each time
Cursor◐PartialPlain prose you can paste in — but no Cursor rules file
Codex◐PartialPlain prose you can paste in — but no AGENTS.md
Gemini CLI◐PartialPlain prose you can paste in
Copilot◐PartialPlain prose you can paste in — but no Copilot instructions file
npx agentalley add trace-to-training-data

This command does not work yet — the CLI is still being built. Until then, use Raw in the reader below to take the file.

Who is stuck, and on what

Convert evaluation traces and production logs into SFT examples and preference pairs. Use when graded traces or failure examples exist and need to become training data, when applying rejection sampling to model outputs, or when building DPO pairs from passing and failing runs.

The whole source

No sign-in, no blur, nothing truncated
trace-to-training-data/SKILL.md216 lines6.1 KBRawView on GitHub
Frontmatter — 2 properties
nametrace-to-training-data
descriptionConvert evaluation traces and production logs into SFT examples and preference pairs. Use when graded traces or failure examples exist and need to become training data, when applying rejection sampling to model outputs, or when building DPO pairs from passing and failing runs.
1---
2name: trace-to-training-data
3description: Convert evaluation traces and production logs into SFT examples and preference pairs. Use when graded traces or failure examples exist and need to become training data, when applying rejection sampling to model outputs, or when building DPO pairs from passing and failing runs.
4---A5 — No allowed-tools declared — no way to tell what this skill may touch
5 
6# Trace To Training Data
7 
8This skill assumes `eval-harness-first`
9already graded the traces being
10converted here — goldens, graders,
11and `runs/<run-id>/results.json`
12all exist before conversion
13starts. This is the flywheel edge
14that skill names in its own flow:
15"the same labeled traces become
16the training set." Conversion
17happens here; grading already
18happened upstream.
19 
20**Input:** graded traces —
21`eval/goldens.jsonl` plus
22`runs/<run-id>/results.json`, each
23row carrying a `task_id`, a
24`verdict` from the grader, and a
25`reward` when the task supports a
26scalar score (judge score,
27execution partial-credit, or an
28RLVR verifier):
29 
30```json
31{"task_id": "t-042", "trace_id": "t-042-a3",
32 "messages": [{"role": "user", "content": "..."}],
33 "verdict": "pass", "reward": 0.91,
34 "grader": "exact_match"}
35```
36 
37**Output format:** rows shaped
38exactly like `dataset-curation`'s
39Format Selection table — SFT
40`messages` rows or DPO
41`prompt`/`chosen`/`rejected`
42pairs — so this skill's output is
43that skill's input with no
44reshaping step in between.
45 
46## The Principle
47 
48The eval harness already did the
49labeling work: every trace in
50`results.json` carries a verdict,
51and often a reward, before this
52skill ever touches it. Converting
53a graded trace into a training
54row is mechanical — pick a shape
55from `dataset-curation`'s table,
56map fields, write JSONL.
57**Curation is the work that
58remains** — which traces clear a
59quality bar, which pairs are
60informative, and which rows must
61never enter the training set at
62all.
63 
64Treat any conversion step that
65requires re-judging a trace as a
66sign the harness is missing a
67grader, not a gap this skill
68should paper over. A trace with
69no verdict or reward isn't
70convertible yet — route it back
71to `eval-harness-first` first,
72don't hand-label it here to
73unblock conversion.
74 
75## SFT From Traces
76 
77- **Keep the top-reward fraction
78 of successful trajectories**,
79 not every passing one. Rank
80 passing traces by reward and
81 take a fraction (the
82 Agent-lightning pattern) rather
83 than every trace that merely
84 cleared the pass bar — a trace
85 that barely passed is a weaker
86 SFT signal than one that scored
87 well above threshold.
88- **Expert-corrected failures
89 become gold SFT examples
90 directly** (the Langfuse
91 pattern) — when a human edits a
92 failing trace's output into a
93 correct one, that correction
94 needs no reward threshold; a
95 human already validated it.
96 Route corrections straight into
97 the SFT set.
98- **Step-level masking beats
99 whole-trajectory discard for
100 multi-step traces.** When only
101 some steps in a multi-step
102 trajectory are bad, mask the
103 loss on the bad steps and keep
104 the good ones, rather than
105 discarding the whole trajectory.
106 SRFT reports 32.2% vs. 30.9% on
107 SWE-bench for step-level critic
108 masking over trajectory discard
109 — a real, if modest, gap from
110 the finer-grained cut.
111 
112## Preference Pairs From Traces
113 
114- **Build pairs from
115 passing-vs-failing trajectories
116 on the SAME task**, never from
117 unrelated best- and
118 worst-scoring traces pulled
119 across different tasks —
120 cross-task pairs teach the
121 model to prefer one task over
122 another, not one response over
123 another.
124- **Select the rejected member at
125 μ−2σ of the reward distribution
126 for that task, never the
127 absolute minimum.**
128 `preference-optimization`'s
129 Pair Construction section owns
130 the full selection formula;
131 this skill supplies the graded
132 trajectories it consumes.
133- **Judge-scored delta selection
134 cuts pair volume without
135 cutting signal.** Score each
136 candidate pair by
137 chosen-minus-rejected judge
138 delta and keep only the
139 highest-delta subset — the top
140 5k of a 16.5k candidate pool
141 matched the full pool's
142 downstream result. Build the
143 full candidate set first, then
144 filter by delta; don't cap
145 generation at 5k up front.
146 
147## Hygiene
148 
149- **Scan for secrets and PII before any row ships,
150 and redact what's found.** Traces sourced from
151 production logs can carry credentials, API keys,
152 tokens, or customer data — run a secret/PII scan
153 over every SFT and DPO row and redact matches;
154 conversion fails closed (the row is dropped, not
155 shipped with the raw content) if sensitive fields
156 remain after redaction. Never commit secrets.
157- **Eval goldens must never leak
158 into training data.** Hold
159 every `eval/goldens.jsonl` ID
160 out of every converted SFT and
161 DPO set — a trace that also
162 appears as a golden trains on
163 the exact item the checkpoint
164 gets graded against later,
165 silently inflating every
166 subsequent eval run.
167- **Dedup against the training
168 set**, not just within the
169 newly converted rows —
170 exact-match or
171 embedding-similarity, matching
172 `dataset-curation`'s dedup
173 method field, run against
174 whatever training data already
175 exists before this batch merges
176 in.
177- **Provenance goes into the
178 dataset card.** Every converted
179 row must trace back to its
180 source `run_id` and `trace_id`
181 — `dataset-curation`'s
182 Provenance field checks for
183 exactly this link back to
184 `trace-to-training-data`
185 output; a row with no traceable
186 source isn't ready to merge.
187 
188## Related Skills
189 
190- `eval-harness-first` — produces
191 the graded traces this skill
192 converts; a trace with no
193 verdict or reward isn't
194 convertible yet, route it back
195 there before conversion.
196- `dataset-curation` — owns the
197 target formats and the dataset
198 card this skill's provenance
199 data feeds; converted rows must
200 match its Format Selection
201 table field names exactly, not
202 an approximation of them.
203- `preference-optimization` —
204 consumes the DPO pairs this
205 skill builds and owns the full
206 μ−2σ rejection-selection
207 formula referenced above.
208 
209Worked JSONL-to-JSONL conversions
210— graded trace to SFT row, trace
211pair to DPO pair, correction to
212SFT row, the rejection-sampling
213loop, and the goldens-holdout
214check — live in
215`references/conversion-recipes.md`.
216 

Reviews

Installed this one?Write the first review and take the Trailblazer badge.

Reviews only open after a real install, so this is empty — and we leave it empty rather than invent one.

Alternatives

Also in Data & AI
Task Coordination StrategiesDecompose complex tasks, design dependency graphs, and coordinate multi-agent work with proper task descriptions and workload balancing. Use this skill when breaking down work for agent teams, managing task dependencies, or monitoring team progress.Data & AI · MIT◐◐◐◐◐●35/40Ebay Seller ToolsData & AI · MIT·····●34/40Tough Decision Advisor: Every Angle ConsideredHand in a decision you're stuck on. Get back a clear breakdown of every angle — the trade-offs, the risks, the blind spot, and a recommended path.Data & AI · Business · MIT license●····●32/40DHDNA Profiler — Cognitive Pattern ExtractionPaste any email, proposal, or note someone wrote, and get back a plain-language read on how they think, what drives their decisions, and how they communicate.Data & AI · Research & analysis · MIT license●····●32/40