Skills · Data & AI

GRPO & RLVR Training

Unverified●32/40

Train reasoning and verifiable-task behavior with GRPO and reinforcement learning from verifiable rewards (RLVR). Use when task success is algorithmically checkable (math, code, tool calls, structured output), when designing GRPO reward functions, or when a GRPO run diverges or reward-hacks.

Originally by wshobson · MIT

Claude Code◐PartialHas SKILL.md but declares no allowed-tools — Claude Code will ask for permission each time

Cursor◐PartialPlain prose you can paste in — but no Cursor rules file

Codex◐PartialPlain prose you can paste in — but no AGENTS.md

Gemini CLI◐PartialPlain prose you can paste in

Copilot◐PartialPlain prose you can paste in — but no Copilot instructions file

npx agentalley add grpo-rlvr-training

This command does not work yet — the CLI is still being built. Until then, use Raw in the reader below to take the file.

Who is stuck, and on what

The whole source

No sign-in, no blur, nothing truncated

grpo-rlvr-training/SKILL.md206 lines7.6 KBRawView on GitHub

Frontmatter — 2 properties

name	grpo-rlvr-training
description	Train reasoning and verifiable-task behavior with GRPO and reinforcement learning from verifiable rewards (RLVR). Use when task success is algorithmically checkable (math, code, tool calls, structured output), when designing GRPO reward functions, or when a GRPO run diverges or reward-hacks.

1	---
2	name: grpo-rlvr-training
3	description: Train reasoning and verifiable-task behavior with GRPO and reinforcement learning from verifiable rewards (RLVR). Use when task success is algorithmically checkable (math, code, tool calls, structured output), when designing GRPO reward functions, or when a GRPO run diverges or reward-hacks.
4	---A5 — No allowed-tools declared — no way to tell what this skill may touch
5
6	# GRPO & RLVR Training
7
8	This skill assumes `finetuning-method-selection`
9	already routed here because the target behavior
10	has a verifiable pass/fail signal — not
11	demonstrations (`lora-qlora-recipes`) or
12	preference pairs (`preference-optimization`).
13	What follows is when RL is the right tool, the
14	reference recipe, the mandatory reward-inspection
15	gate, and how to pick a GRPO variant when the
16	base recipe misbehaves.
17
18	Input: a routing decision (RLVR via GRPO)
19	plus a verifier (code executor, test suite,
20	schema checker, or grader) for the target task.
21	Output format: a validated GRPO config — the
22	kwarg values in `references/grpo-memory.md` and
23	the reward functions in
24	`references/reward-functions.md`, not free-form
25	advice — that `llm-finetuning-training-engineer`
26	consumes directly.
27
28	## When RL Applies
29
30	GRPO+RLVR only pays off when task success is
31	algorithmically checkable — a unit test
32	passes, a parser accepts the output, a tool call
33	matches an expected schema, a math answer matches
34	a ground truth. If grading the output requires
35	human judgment or a subjective rubric, that's an
36	eval-harness and judge-calibration problem first
37	— see `eval-harness-first` — not a reason to skip
38	straight to RL.
39
40	Before opening a GRPO run, confirm the model can
41	sometimes succeed on the target task already.
42	RL sharpens an existing capability by reweighting
43	toward the samples that already work; it does not
44	install a capability from zero.
45
46	- **The model never succeeds, even at low
47	temperature across many samples:** the gap is
48	format or task understanding, not policy
49	refinement. Route back to SFT first
50	(`lora-qlora-recipes`) and only return to this
51	skill once the base success rate is nonzero.
52	- **The model succeeds sometimes,
53	inconsistently:** this is the GRPO sweet spot —
54	proceed to The Recipe below.
55
56	The standing rule for the whole plugin: **DPO for
57	taste, GRPO for reasoning.** If the signal is a
58	preference between two acceptable outputs, that's
59	`preference-optimization`, not this skill.
60
61	## The Recipe
62
63	The reference recipe is TRL's `GRPOTrainer` with
64	vLLM-backed generation:
65
66	```python
67	from trl import GRPOConfig, GRPOTrainer
68
69	grpo_args = GRPOConfig(
70	output_dir="./outputs-grpo",
71	use_vllm=True,
72	vllm_mode="colocate", # single GPU; "server" for multi-GPU
73	num_generations=8, # floor — fewer starves the group-relative baseline
74	learning_rate=5e-7, # settled range for GRPO
75	beta=0.01, # KL coefficient vs the reference policy
76	per_device_train_batch_size=8,
77	gradient_accumulation_steps=4,
78	bf16=True,
79	logging_steps=10,
80	seed=3407,
81	)
82
83	trainer = GRPOTrainer(
84	model=SFT_CHECKPOINT,
85	args=grpo_args,
86	reward_funcs=[format_reward, correctness_reward], # references/reward-functions.md
87	train_dataset=prompts, # prompt-only — GRPO generates its own completions
88	processing_class=tokenizer,
89	)
90
91	trainer.train()
92	```
93
94	- `vllm_mode="colocate"` runs generation and
95	training on the same GPU — the default for a
96	single-GPU box.
97	- `vllm_mode="server"` points at a separate
98	vLLM server process and is the multi-GPU path —
99	generation and training don't compete for the
100	same device.
101	- `num_generations` ≥ 8 is a floor, not a
102	suggestion: GRPO's advantage estimate is
103	relative to the group mean, and fewer than 8
104	samples per prompt produces a noisy baseline.
105	- Reward is composite — a format reward (did
106	the output parse / match the required
107	structure) plus a correctness reward (did the
108	answer verify). A well-formed-but-wrong answer
109	and a malformed one should not score
110	identically; correctness alone loses that
111	signal.
112	- `learning_rate=5e-7` and `beta=0.01` are
113	the settled starting point; deviate only after
114	the base run is stable and reward-inspected
115	(below).
116
117	Memory sizing for this recipe by target size
118	class: `references/grpo-memory.md`.
119
120	## The Inspection Rule
121
122	**Run the reward function against 50–100 sampled
123	outputs and manually read the results before
124	starting the actual training run.** This is a
125	gate, not a one-time sanity check.
126
127	If the reward function's judgment disagrees with
128	a human reading of that sample, fix the reward
129	function first. Training against an uninspected
130	reward, or tuning hyperparameters to compensate
131	for one silently scoring the wrong thing, is how
132	a run reward-hacks: the model optimizes cleanly
133	toward the wrong target, and that doesn't surface
134	as a training-loop bug.
135
136	This inspection is a Phase 1 gate input for
137	`/finetune` — the same 50–100-sample read that
138	catches a broken reward function here is what that
139	command checks for before it lets a GRPO brief
140	proceed.
141
142	Complete reward function implementations to
143	inspect against — exact-match, schema-validation,
144	unit-test-execution, a length-penalty wrapper, and
145	a rubric-as-reward judge pattern:
146	`references/reward-functions.md`.
147
148	## Variant Selection
149
150	The base recipe above is the default. Reach for a
151	variant only when a specific failure mode shows
152	up, not preemptively:
153
154	\| Failure mode \| Variant \| Why \|
155	\|---\|---\|---\|
156	\| Entropy collapse / degenerate long chain-of-thought \| DAPO \| Decouples clip bounds and relaxes the KL penalty that over-regularizes exploration on long reasoning traces \|
157	\| Reward or output length trends up regardless of quality \| Dr.GRPO \| Removes GRPO's length-normalization bias so reward tracks correctness, not completion length \|
158	\| Training a mixture-of-experts model \| GSPO \| Moves the importance-sampling ratio to the sequence level instead of per-token — per-token ratios are unstable on MoE routing, so GSPO is required here, not optional \|
159
160	Start with plain GRPO. Watch for the specific
161	symptom — collapsing entropy on long CoT, a
162	length-reward correlation, or MoE instability —
163	and only then swap in the matching variant above.
164	Don't pre-select a variant before the base recipe
165	has actually shown the failure mode.
166
167	## VLM RL Is Reference-Only
168
169	Vision-language RL is **not executed by this
170	plugin in v1** — it's documented here for
171	context, not as a runnable path. Tooling is
172	fragmented across ms-swift and EasyR1-derived
173	forks with no one-line TRL command yet, and naive
174	text-only GRPO applied to a VLM tends to
175	reward-hack by optimizing the text-reasoning trace
176	while ignoring the image — the model learns to
177	sound right without looking at the input. A VLM
178	RL run is a research spike outside this skill's
179	supported recipe, not a variant of The Recipe
180	above.
181
182	## References
183
184	- `references/reward-functions.md` — complete
185	Python reward functions (exact-match
186	correctness, schema validation, unit-test
187	execution, a length-penalty wrapper, and a
188	rubric-as-reward judge pattern) to inspect under
189	The Inspection Rule before any training run.
190	- `references/grpo-memory.md` — memory sizing by
191	target size class, vLLM sleep-mode and
192	optimizer-state tactics, Unsloth's long-context
193	RL chunking, and the DGX Spark bandwidth caveat
194	for decode-heavy rollouts.
195
196	Related skills: `finetuning-method-selection`
197	routes here once a verifiable pass/fail signal
198	exists; `preference-optimization` is the sibling
199	skill for preference pairs rather than verifiable
200	rewards; `eval-harness-first` covers judge
201	calibration for any reward that isn't purely
202	code-checkable. On DGX Spark, defer to the
203	`dgx-spark-ops` plugin's skills, when installed,
204	for the memory/thermal remediation ladder this
205	skill's memory table doesn't cover.
206

Reviews

Installed this one?Write the first review and take the Trailblazer badge.

Reviews only open after a real install, so this is empty — and we leave it empty rather than invent one.

Alternatives

Also in Data & AI

Task Coordination StrategiesDecompose complex tasks, design dependency graphs, and coordinate multi-agent work with proper task descriptions and workload balancing. Use this skill when breaking down work for agent teams, managing task dependencies, or monitoring team progress.Data & AI · MIT◐◐◐◐◐●35/40 Ebay Seller ToolsData & AI · MIT·····●34/40 Tough Decision Advisor: Every Angle ConsideredHand in a decision you're stuck on. Get back a clear breakdown of every angle — the trade-offs, the risks, the blind spot, and a recommended path.Data & AI · Business · MIT license●····●32/40 DHDNA Profiler — Cognitive Pattern ExtractionPaste any email, proposal, or note someone wrote, and get back a plain-language read on how they think, what drives their decisions, and how they communicate.Data & AI · Research & analysis · MIT license●····●32/40

Verification score

Safety23/25

A1Destructive commands6/6
A2Data exfiltration6/6
A3Hardcoded secrets4/4
A4Prompt-injection guard5/5
A5Permission footprint · line 42/4

Transparency9/15

B1No obfuscation5/5
B2Declares dependencies0/4
B3Documented4/4
B4Pinned version0/2

Effectivenessnot scored yet

Needs real runs and real usage before it can be measured.

Costnot scored yet

Needs real runs and real usage before it can be measured.

Real-world usenot scored yet

Needs real runs and real usage before it can be measured.

32.0/40 on the part a machine can score. The badge only reaches Verified once Effectiveness and Cost have actually run.

wshobson

Original author

LicenceMIT

Repowshobson/agents

Repo stars39,748

Licence found inrepo

View on GitHub This is mine

README badge

The SVG endpoint is still being built. This is the final markdown shape.

[![AgentAlley](https://agentalley.ai/badge/grpo-rlvr-training.svg)](https://agentalley.ai/s/grpo-rlvr-training)

Submit your own