AgentAlley

Skills · Data & AI

Preference Optimization

Unverified●31/40

Align a fine-tuned model with preference data using DPO, ORPO, KTO, or SimPO. Use when preference pairs or thumbs-up/down feedback exist, when choosing between preference-optimization methods, or when a DPO run needs hyperparameters or debugging.

Originally by wshobson · MIT

Claude Code◐PartialHas SKILL.md but declares no allowed-tools — Claude Code will ask for permission each time
Cursor◐PartialPlain prose you can paste in — but no Cursor rules file
Codex◐PartialPlain prose you can paste in — but no AGENTS.md
Gemini CLI◐PartialPlain prose you can paste in
Copilot◐PartialPlain prose you can paste in — but no Copilot instructions file
npx agentalley add preference-optimization

This command does not work yet — the CLI is still being built. Until then, use Raw in the reader below to take the file.

Who is stuck, and on what

Align a fine-tuned model with preference data using DPO, ORPO, KTO, or SimPO. Use when preference pairs or thumbs-up/down feedback exist, when choosing between preference-optimization methods, or when a DPO run needs hyperparameters or debugging.

The whole source

No sign-in, no blur, nothing truncated
preference-optimization/SKILL.md203 lines7.7 KBRawView on GitHub
Frontmatter — 2 properties
namepreference-optimization
descriptionAlign a fine-tuned model with preference data using DPO, ORPO, KTO, or SimPO. Use when preference pairs or thumbs-up/down feedback exist, when choosing between preference-optimization methods, or when a DPO run needs hyperparameters or debugging.
1---
2name: preference-optimization
3description: Align a fine-tuned model with preference data using DPO, ORPO, KTO, or SimPO. Use when preference pairs or thumbs-up/down feedback exist, when choosing between preference-optimization methods, or when a DPO run needs hyperparameters or debugging.
4---A5 — No allowed-tools declared — no way to tell what this skill may touch
5 
6# Preference Optimization
7 
8This skill assumes `finetuning-method-selection`
9already routed here because the data shape is
10preference pairs or unpaired thumbs-up/down
11feedback, not demonstrations (that's
12`lora-qlora-recipes`) or a verifiable reward
13signal (that's `grpo-rlvr-training`). What
14follows is method selection among the DPO family,
15the evidence for how much that selection actually
16matters, the production training pattern, and how
17to build the pairs in the first place.
18 
19**Input:** a routing decision (preference
20optimization) plus preference pairs or unpaired
21feedback, usually from an SFT checkpoint.
22**Output format:** a validated method choice plus
23a config — the kwarg values in
24`references/method-configs.md`, not free-form
25advice — that `llm-finetuning-training-engineer`
26consumes directly.
27 
28## Method Selection
29 
30| Data shape | Method | Key parameters |
31|---|---|---|
32| Preference pairs, default case | **DPO** | β=0.1, LR 5e-7–1e-6, 1–2 epochs |
33| Memory-bound or no SFT checkpoint | **ORPO** | reference-free, fused SFT+preference in one loss |
34| Unpaired thumbs-up/down | **KTO** | binary label per example, no pairing needed |
35| Length bias observed, sweep budget available | **SimPO** | reference-free; see sweep grid below |
36 
37- **DPO is the safe default.** Use β=0.1 and a
38 learning rate of 5e-7 to 1e-6 for 1–2 epochs.
39 This LR is *lower* than the SFT LR that produced
40 the checkpoint being aligned — porting an SFT-
41 scale LR into a DPO run is the most common
42 misconfiguration here, not an edge case.
43- **ORPO** routes in when memory is the
44 constraint, or when there's no separate SFT
45 checkpoint to start from — it's reference-free
46 and fuses the SFT and preference objectives into
47 one loss, skipping the separate SFT pass and the
48 reference-model memory cost DPO carries.
49- **KTO** routes in when feedback is unpaired
50 binary signal (thumbs-up/down) rather than
51 matched preference pairs — don't force unpaired
52 feedback into synthetic pairs to use DPO instead.
53- **SimPO** fixes DPO's length bias but only pays
54 off with disciplined sweeping — its published
55 gains are a ceiling reported under a tuned sweep,
56 not a baseline any single config will reproduce.
57 Route here only when there's sweep budget; use
58 DPO instead if there isn't.
59- **Classic RLHF (reward model + PPO) is retired**
60 outside frontier labs. Don't reach for it in a
61 production pipeline — every method above is
62 cheaper and better-supported for the same data
63 shapes.
64 
65### Worked Examples
66 
67- *"We have an SFT checkpoint and clean paired
68 preference data, no length-bias complaints yet."*
69 → default case → **DPO** at β=0.1.
70- *"Reviewers click thumbs-up/down per response;
71 nothing is paired."* → unpaired signal →
72 **KTO**, not DPO — don't synthesize pairs to
73 force DPO onto unpaired data.
74- *"GPU budget doesn't cover a separate SFT pass
75 plus a DPO reference model."* → memory-bound,
76 no separate checkpoint → **ORPO**.
77- *"DPO output favors longer answers regardless of
78 quality, and there's time to run a sweep."* →
79 length bias plus sweep budget → **SimPO**. Skip
80 it if the sweep budget isn't actually there.
81 
82## The Low-Leverage Truth
83 
84A 2026 240-H100-run study (arXiv 2603.19335) is
85the load-bearing evidence behind the table above:
86**loss-function choice is worth roughly 1
87percentage point of leverage, model scale is
88worth roughly 50.** Zero of 20 DPO variants tested
89beat vanilla DPO. Rankings also **invert with
90scale** — a variant that wins in a small pilot can
91lose at deployment size.
92 
93Two practical consequences:
94 
95- Don't spend a routing decision agonizing over
96 DPO-variant bake-offs. The table above is
97 sufficient; deeper variant selection is
98 low-leverage compared to data quality and scale.
99- **Validate at deployment scale before trusting a
100 ranking.** A method comparison run on a small
101 pilot model doesn't transfer to the production
102 size class — re-check the winner once scale
103 changes.
104 
105This is also why the Method Selection table above
106is deliberately short: it encodes the ~1pp lever,
107not a ranking of DPO variants that the same study
108shows doesn't hold up across scale. Treat any
109variant-selection advice that isn't in that table
110— including advice that claims a specific variant
111"wins" — as unproven until it's been validated at
112the target deployment size.
113 
114## Production Pattern: Iterative On-Policy DPO
115 
116A single offline DPO pass on a static preference
117dataset is a starting point, not the production
118pattern. The policy drifts away from the
119distribution the pairs were sampled from as
120training proceeds, and a static dataset goes stale
121against that drift. Production pipelines run DPO
122iteratively and on-policy instead:
123 
1241. Sample completions from the current policy
125 checkpoint.
1262. Score or rank the completions (reward model,
127 judge, or task grader).
1283. Run a DPO pass using the current checkpoint as
129 the reference model.
1304. The resulting checkpoint becomes both the new
131 policy *and* the new reference for the next
132 round.
133 
134Repeat. Each round's reference model is the prior
135round's output, not a fixed initial checkpoint —
136that's what keeps the preference signal on-policy
137instead of scoring against an increasingly stale
138distribution.
139 
140A single-pass DPO run is still a reasonable first
141iteration — it just isn't the whole pipeline. Plan
142for at least one more round once the first
143checkpoint exists, rather than treating pass one
144as the finished artifact.
145 
146## Pair Construction
147 
148Build DPO/ORPO pairs from **same-task
149passing-vs-failing trajectories** — two attempts
150at the same underlying task, not unrelated
151best-and-worst examples pulled from different
152tasks. Within that trajectory set, select the
153rejected member at **μ−2σ of the reward
154distribution, never the minimum**. Naive
155best-vs-worst pair construction (max reward vs.
156absolute minimum) degrades as scale increases; the
157μ−2σ selection is more robust to the same scale
158sensitivity the low-leverage study surfaced above.
159 
160```
161sorted_by_reward = sort(trajectories, key=reward)
162chosen = sorted_by_reward[-1] # highest reward
163mu, sigma = mean(rewards), stdev(rewards)
164rejected = closest(sorted_by_reward, mu - 2 * sigma)
165# NOT sorted_by_reward[0] — the absolute minimum
166# is the naive best-vs-worst construction that
167# degrades as scale increases.
168```
169 
170For the mechanics of turning graded traces into
171these pairs — including rejection sampling and
172judge-scored delta selection — see
173`trace-to-training-data`.
174 
175## References
176 
177Complete TRL config blocks per method —
178`DPOConfig`, `ORPOConfig`, `KTOConfig`, and the
179SimPO sweep grid — plus Unsloth wrappers and a
180catastrophic-forgetting note live in
181`references/method-configs.md`. Those configs use
182the same current-TRL API conventions established
183in `lora-qlora-recipes`'s
184`references/unsloth-trl-mapping.md`
185(`processing_class`, not `tokenizer=`).
186 
187`references/method-configs.md` also carries the
188catastrophic-forgetting note: a too-high learning
189rate is the usual cause when a preference-tuned
190checkpoint loses general capability, and the fix
191is almost always to drop the LR toward the low end
192of the range in the Method Selection table above
193before reaching for any other remediation.
194 
195Related skills: `finetuning-method-selection`
196routes here once preference pairs or unpaired
197feedback exist; `lora-qlora-recipes` produces the
198SFT checkpoint DPO/KTO/SimPO align (ORPO's
199fused path can skip it); `trace-to-training-data`
200converts passing/failing trajectories into the
201pairs this skill's Pair Construction section
202consumes.
203 

Reviews

Installed this one?Write the first review and take the Trailblazer badge.

Reviews only open after a real install, so this is empty — and we leave it empty rather than invent one.

Alternatives

Also in Data & AI
Task Coordination StrategiesDecompose complex tasks, design dependency graphs, and coordinate multi-agent work with proper task descriptions and workload balancing. Use this skill when breaking down work for agent teams, managing task dependencies, or monitoring team progress.Data & AI · MIT◐◐◐◐◐●35/40Ebay Seller ToolsData & AI · MIT·····●34/40Tough Decision Advisor: Every Angle ConsideredHand in a decision you're stuck on. Get back a clear breakdown of every angle — the trade-offs, the risks, the blind spot, and a recommended path.Data & AI · Business · MIT license●····●32/40DHDNA Profiler — Cognitive Pattern ExtractionPaste any email, proposal, or note someone wrote, and get back a plain-language read on how they think, what drives their decisions, and how they communicate.Data & AI · Research & analysis · MIT license●····●32/40