Training Check
Unverified●37/40Claude Code●WorksValid SKILL.md that declares allowed-tools
Cursor·UnknownWe have not crawled the repo tree, so we will not guess
Codex·UnknownWe have not crawled the repo tree, so we will not guess
Gemini CLI·UnknownThe spec defines no detection rule for Gemini
Copilot·UnknownWe have not crawled the repo tree, so we will not guess
npx agentalley add training-checkWho is stuck, and on what
Periodically check WandB metrics during training to catch problems early (NaN, loss divergence, idle GPUs). Avoids wasting GPU hours on broken runs. Use when training is running and you want automated health checks.
The whole source
Frontmatter — 4 properties
| name | training-check |
|---|---|
| description | Periodically check WandB metrics during training to catch problems early (NaN, loss divergence, idle GPUs). Avoids wasting GPU hours on broken runs. Use when training is running and you want automated health checks. |
| argument-hint | [wandb-run-path] |
| allowed-tools | Bash(*), Read, Grep, Glob, Write, Edit, mcp__codex__codex, mcp__codex__codex-reply |
| 1 | --- |
| 2 | name: training-check |
| 3 | description: Periodically check WandB metrics during training to catch problems early (NaN, loss divergence, idle GPUs). Avoids wasting GPU hours on broken runs. Use when training is running and you want automated health checks. |
| 4 | argument-hint: "[wandb-run-path]" |
| 5 | allowed-tools: Bash(*), Read, Grep, Glob, Write, Edit, mcp__codex__codex, mcp__codex__codex-reply |
| 6 | --- |
| 7 | |
| 8 | # Training Check |
| 9 | |
| 10 | Periodically read WandB metrics during training to catch problems early. Do not wait until training finishes to discover it was a waste of GPU time. |
| 11 | |
| 12 | > ⏱ This skill is **correctly** cron-wired (see below): it polls |
| 13 | > machine-checkable training health (NaN / divergence / idle GPU) — the additive |
| 14 | > external-wait shape in |
| 15 | > [`shared-references/external-cadence.md`](../shared-references/external-cadence.md). |
| 16 | > The occasional Codex call for an ambiguous metric is a **one-shot** check per |
| 17 | > tick, not a multi-round verdict loop, so it stays additive — it never grows |
| 18 | > into a wrapped verdict skill. |
| 19 | |
| 20 | ## Context: $ARGUMENTS |
| 21 | |
| 22 | ## Constants |
| 23 | |
| 24 | - WANDB_ENTITY and WANDB_PROJECT: read from CLAUDE.md or passed as argument (format: `entity/project/run_id`) |
| 25 | - CHECK_INTERVAL: starts at 10 minutes, then gradually increases if consistently healthy: 10 min → 20 min → 30 min → 60 min (cap) |
| 26 | - REVIEWER_MODEL = `gpt-6-astra` — used via Codex MCP for ambiguous cases only |
| 27 | |
| 28 | ## When to Use |
| 29 | |
| 30 | - After training is confirmed running (session alive, loss decreasing for first few steps) |
| 31 | - Set up via CronCreate to fire periodically during training |
| 32 | - **This skill checks training QUALITY, not process HEALTH.** Process health (session alive, GPU utilization) is [watchdog.py](../../tools/watchdog.py)'s job. |
| 33 | |
| 34 | ## Workflow |
| 35 | |
| 36 | ### Step 1: Read WandB Metrics |
| 37 | |
| 38 | ```python |
| 39 | import wandb |
| 40 | api = wandb.Api() |
| 41 | run = api.run("<entity>/<project>/<run_id>") |
| 42 | history = run.history() |
| 43 | ``` |
| 44 | |
| 45 | If WandB is unreachable (API error, network issue), fall back to reading the log file directly via SSH: |
| 46 | ```bash |
| 47 | ssh server "tail -100 /path/to/training.log" |
| 48 | ``` |
| 49 | |
| 50 | Check these signals: |
| 51 | - **Loss trend**: Is training loss decreasing over the last N steps? |
| 52 | - **Eval metrics**: Are evaluation metrics improving (or at least not degrading)? |
| 53 | - **NaN / Inf**: Any NaN or Inf values in loss or gradients? |
| 54 | - **Spikes**: Sudden large jumps in loss (>10x normal variance)? |
| 55 | - **Learning rate**: Is the schedule behaving as expected? |
| 56 | - **Gradient norm**: Exploding or vanishing? |
| 57 | |
| 58 | ### Step 2: Judgment |
| 59 | |
| 60 | | Signal | Judgment | Action | |
| 61 | |--------|----------|--------| |
| 62 | | NaN/Inf in loss | **Clearly bad** | Stop training, investigate | |
| 63 | | Loss diverging (increasing for >N steps) | **Clearly bad** | Stop training, investigate | |
| 64 | | Eval metrics significantly worse than baseline | **Clearly bad** | Stop training, investigate | |
| 65 | | Loss decreasing, metrics improving | **Clearly fine** | Continue, increase check interval | |
| 66 | | Loss flat but not diverging | **Unsure** | → Step 3 (Codex judgment) | |
| 67 | | Metrics noisy, can't tell trend | **Unsure** | → Step 3 (Codex judgment) | |
| 68 | | Slightly worse than baseline but still early | **Unsure** | → Step 3 (Codex judgment) | |
| 69 | |
| 70 | ### Step 3: Codex Judgment (only when unsure) |
| 71 | |
| 72 | Only escalate to Codex when the signal is ambiguous. For clearly good or clearly bad signals, act directly. |
| 73 | |
| 74 | ``` |
| 75 | mcp__codex__codex: |
| 76 | model: gpt-6-astra |
| 77 | config: {"model_reasoning_effort": "xhigh"} |
| 78 | prompt: | |
| 79 | TRAINING HEALTH CHECK — need your judgment on ambiguous metrics. |
| 80 | |
| 81 | Run: <entity>/<project>/<run_id> |
| 82 | Current epoch/step: X / Y total |
| 83 | Training loss (last 10 checkpoints): [values] |
| 84 | Eval metrics (last 3 evals): [values] |
| 85 | Baseline reference: [numbers from paper/reproduction] |
| 86 | |
| 87 | What I'm unsure about: [specific concern] |
| 88 | |
| 89 | Please respond with exactly one of: |
| 90 | - STOP: clearly problematic, should kill training |
| 91 | - CONTINUE: looks fine, check again next interval |
| 92 | - WAIT: not enough data to judge, check again sooner |
| 93 | ``` |
| 94 | |
| 95 | ### Step 4: Act |
| 96 | |
| 97 | | Decision | Action | |
| 98 | |----------|--------| |
| 99 | | **Stop** | Kill the training session. Save the WandB run URL, key metrics, and reason for stopping. Log to project notes for debugging. | |
| 100 | | **Continue** | Do nothing. Will be invoked again at next interval (increase interval if consistently healthy). | |
| 101 | | **Wait** | Do nothing but keep the current short interval (don't increase). | |
| 102 | |
| 103 | ## Integration with Watchdog |
| 104 | |
| 105 | Training-check and [watchdog.py](../../tools/watchdog.py) operate at different levels: |
| 106 | |
| 107 | | Layer | Tool | What it checks | Frequency | |
| 108 | |-------|------|----------------|-----------| |
| 109 | | Process health | watchdog.py | Session alive? GPU active? | Every 60s (continuous) | |
| 110 | | Training quality | training-check | Loss trend? Metrics improving? | Every 10-60 min (periodic) | |
| 111 | |
| 112 | Use both together: |
| 113 | - Watchdog catches crashes and idle GPUs immediately |
| 114 | - Training-check catches subtle quality issues (loss plateau, metric degradation) |
| 115 | |
| 116 | ## Rules |
| 117 | |
| 118 | - Do not stop training on first sign of noise — some loss spikes are normal. Look at **trends over multiple checkpoints**. |
| 119 | - When stopping training, always save the WandB run URL and key metrics as evidence. |
| 120 | - If both WandB and log files are unreachable, report the connectivity issue and try again next interval. Do not assume training is broken. |
| 121 | - Gradually increase check interval when healthy (10 → 20 → 30 → 60 min). Reset to 10 min after any anomaly. |
| 122 | - This skill is meant to be automated via CronCreate — do not ask the user whether to set it up. Just set it. |
| 123 | |
| 124 | ## CronCreate Setup Example |
| 125 | |
| 126 | ``` |
| 127 | After training is confirmed stable: |
| 128 | CronCreate (recurring, every 10 minutes initially): |
| 129 | "Run /training-check for wandb run <entity>/<project>/<run_id>" |
| 130 | ``` |
| 131 | |
| 132 | As the check interval increases, delete the old CronCreate job and create a new one with the longer interval. |
| 133 |
Reviews
Installed this one?Write the first review and take the Trailblazer badge.
Alternatives
Research Review via External Reviewer Backend (ultra reasoning)Get a deep critical review of research from an external reviewer backend (Codex or manual). Use when user says "review my research", "help me review", "get external review", or wants critical feedback on research ideas, papers, or experimental results.●····●38/40Changelog AutomationAutomate changelog generation from commits, PRs, and releases following Keep a Changelog format. Use when setting up release workflows, generating release notes, or standardizing commit conventions.◐····●36/40Hermes Tweet> Install and operate Hermes Tweet, a Hermes Agent plugin for X/Twitter research, timeline reading, tweet analysis, and approval-gated private or state-changing operations. Use this skill when installing Hermes Tweet, researching X/Twitter accounts, monitoring launch signals, investigating mentions, auditing giveaways, or preparing gated X operations. Use proactively when a Hermes Agent workflow needs current X/Twitter context. Requires XQUIK_API_KEY for read and action tools.◐····●36/40Social Publishing> Schedule and publish social media posts across 13 platforms (X, LinkedIn, Instagram, Facebook Pages, TikTok, Discord, Telegram, YouTube, Reddit, WordPress, Pinterest) via the SocialClaw API. Use when the user wants to publish, schedule, or manage social media content programmatically. Requires SOCIALCLAW_API_KEY.◐····●34/40