Vision-Language SFT
Unverified●32/40Claude Code◐PartialHas SKILL.md but declares no allowed-tools — Claude Code will ask for permission each time
Cursor◐PartialPlain prose you can paste in — but no Cursor rules file
Codex◐PartialPlain prose you can paste in — but no AGENTS.md
Gemini CLI◐PartialPlain prose you can paste in
Copilot◐PartialPlain prose you can paste in — but no Copilot instructions file
npx agentalley add vision-sftWho is stuck, and on what
Fine-tune vision-language models (VLMs) with supervised learning on image+text data. Use when adapting a VLM to a visual domain or task, configuring frozen-vision-tower LoRA, or debugging a VLM fine-tune that trains without learning.
The whole source
Frontmatter — 2 properties
| name | vision-sft |
|---|---|
| description | Fine-tune vision-language models (VLMs) with supervised learning on image+text data. Use when adapting a VLM to a visual domain or task, configuring frozen-vision-tower LoRA, or debugging a VLM fine-tune that trains without learning. |
| 1 | --- |
| 2 | name: vision-sft |
| 3 | description: Fine-tune vision-language models (VLMs) with supervised learning on image+text data. Use when adapting a VLM to a visual domain or task, configuring frozen-vision-tower LoRA, or debugging a VLM fine-tune that trains without learning. |
| 4 | ---A5 — No allowed-tools declared — no way to tell what this skill may touch |
| 5 | |
| 6 | # Vision-Language SFT |
| 7 | |
| 8 | This skill assumes `finetuning-method-selection` |
| 9 | already routed here: the data shape is |
| 10 | image+text demonstrations, not preference pairs |
| 11 | or a verifiable reward signal, and the base is a |
| 12 | vision-language model rather than a text-only |
| 13 | one. `lora-qlora-recipes` covers the text-only |
| 14 | LoRA/QLoRA recipe this skill specializes for the |
| 15 | vision tower and projector; read that skill first |
| 16 | if the LoRA fundamentals (rank, alpha, target |
| 17 | modules) aren't already familiar. |
| 18 | |
| 19 | **Input:** an image+text dataset and a VLM base |
| 20 | model already picked from the model catalog. |
| 21 | **Output format:** a validated adapter config — |
| 22 | which components are frozen, LoRA target modules, |
| 23 | and a `min_pixels`/`max_pixels` budget — that |
| 24 | `llm-finetuning-training-engineer` consumes |
| 25 | directly when it generates a runnable script. |
| 26 | |
| 27 | ## Quick Reference |
| 28 | |
| 29 | | Situation | Default | |
| 30 | |---|---| |
| 31 | | Adapting behavior on familiar images | Frozen tower+projector, LoRA r=8–16, α=16–32 | |
| 32 | | Visual domain shift | Unfreeze last-6 ViT layers, vision LR 5–10x lower | |
| 33 | | Doesn't fit in bf16 at target rank | QLoRA — frozen vision tower only | |
| 34 | | `fast_inference=True` | `finetune_vision_layers=False` | |
| 35 | | Loss normal, eval not improving | Check the Two Silent Killers below first | |
| 36 | |
| 37 | ## The Consensus Recipe |
| 38 | |
| 39 | Freeze the vision tower and the projector. Put |
| 40 | LoRA on the LLM only, all-linear (the same |
| 41 | attention + MLP target list as text-only SFT — |
| 42 | see `lora-qlora-recipes`), at **r=8–16, |
| 43 | α=16–32**. This is the settled default for |
| 44 | adapting a VLM's behavior without disturbing how |
| 45 | it sees. |
| 46 | |
| 47 | - **The vision tower and projector stay frozen by |
| 48 | default.** They already encode a general visual |
| 49 | representation; retraining them is rarely |
| 50 | necessary and adds risk without adding |
| 51 | capability for most tasks. |
| 52 | - **LoRA rank runs lower than the text-only |
| 53 | general default** (r=8–16 here vs r=16–32 for |
| 54 | text-only SFT) because the LLM-only adapter is |
| 55 | adapting behavior, not injecting new visual |
| 56 | knowledge. |
| 57 | - **QLoRA is permitted only with a frozen vision |
| 58 | tower.** Quantizing the base while also |
| 59 | unfreezing and training vision layers is |
| 60 | unsupported and unstable — treat this as a hard |
| 61 | pairing rule, not a tunable. If the vision tower |
| 62 | needs to unfreeze, drop QLoRA and use bf16 LoRA |
| 63 | instead. |
| 64 | |
| 65 | ```python |
| 66 | # freeze tower + projector; LoRA on LLM only |
| 67 | for name, param in model.named_parameters(): |
| 68 | if "vision_tower" in name or "projector" in name: |
| 69 | param.requires_grad = False |
| 70 | |
| 71 | target_modules = [ |
| 72 | "q_proj", "k_proj", "v_proj", "o_proj", |
| 73 | "gate_proj", "up_proj", "down_proj", |
| 74 | ] # LLM-only, all-linear — r=8-16, alpha=16-32 |
| 75 | ``` |
| 76 | |
| 77 | ## When to Unfreeze |
| 78 | |
| 79 | Unfreezing vision layers is a deliberate |
| 80 | escalation, not a default decision — reach |
| 81 | for it only when the domain shift is |
| 82 | visual, not textual. |
| 83 | |
| 84 | - **Unfreeze only for visual domain |
| 85 | shift.** If the task is teaching new |
| 86 | behavior on images the tower already |
| 87 | understands (charts, everyday photos), |
| 88 | the frozen-tower recipe above is |
| 89 | sufficient. Unfreeze when the visual |
| 90 | domain itself is unfamiliar to the |
| 91 | tower — satellite imagery, medical |
| 92 | scans, dense technical diagrams — and |
| 93 | the frozen-tower recipe plateaus. |
| 94 | - **Last-6 ViT layers is the sweet |
| 95 | spot.** Unfreezing the final six |
| 96 | vision-transformer layers (not the |
| 97 | whole tower) measured **+1.7pt DocVQA |
| 98 | at ~1.75x training cost** over the |
| 99 | frozen baseline. Treat six layers as |
| 100 | the ceiling worth paying for; going |
| 101 | further spends compute without a |
| 102 | matched result. |
| 103 | - **Vision LR must run 5–10x lower than |
| 104 | the LLM LR when unfrozen.** The vision |
| 105 | tower's pretrained representation is |
| 106 | more fragile than the LLM's adapter; |
| 107 | the same LR for both risks overwriting |
| 108 | the visual representation faster than |
| 109 | the LLM adapter can compensate. |
| 110 | - **High LoRA rank on the patch- |
| 111 | embedding layer risks NaN.** If patch |
| 112 | embedding is in the unfrozen set, keep |
| 113 | its rank low and watch early-step loss |
| 114 | closely — one of the most fragile |
| 115 | places to apply LoRA in a VLM. |
| 116 | |
| 117 | ## The Two Silent Killers |
| 118 | |
| 119 | Both produce a run that trains without error and |
| 120 | without learning: the loss curve looks normal, |
| 121 | the model doesn't improve, and neither throws an |
| 122 | exception — both need an explicit pre-training |
| 123 | check, not just a clean training log. |
| 124 | |
| 125 | - **Image-tag/count mismatch.** Every image |
| 126 | placeholder token in the templated text must |
| 127 | map 1:1 to a media item actually passed to the |
| 128 | collator. A mismatch (one placeholder, zero or |
| 129 | two images attached; or an image with no |
| 130 | placeholder) doesn't error in most collators — |
| 131 | it silently misaligns image and text, and the |
| 132 | model "trains but learns nothing." Validate the |
| 133 | 1:1 placeholder-to-media mapping before training |
| 134 | starts, on every example, not just a sample. |
| 135 | Full validation-checklist detail: |
| 136 | `references/collators-and-pitfalls.md`. |
| 137 | - **`min_pixels`/`max_pixels` resolution budget.** |
| 138 | This pair is the single most consequential |
| 139 | hyperparameter for quality and memory in VLM |
| 140 | SFT — more than rank, alpha, or LR. Too low |
| 141 | silently downsamples images below what the task |
| 142 | needs (small document text becomes unreadable |
| 143 | even though training "succeeds"); too high blows |
| 144 | the activation memory budget or forces too small |
| 145 | a batch to train stably. Set it deliberately per |
| 146 | dataset, don't leave it at a framework default. |
| 147 | |
| 148 | ## Unsloth Specifics |
| 149 | |
| 150 | - **`UnslothVisionDataCollator`** is the collator |
| 151 | Unsloth expects for VLM SFT — it handles the |
| 152 | image-tag alignment and per-architecture |
| 153 | processor contract described in |
| 154 | `references/collators-and-pitfalls.md`. Don't |
| 155 | substitute a text-only collator for VLM data. |
| 156 | - **`finetune_vision_layers=False` is required |
| 157 | when `fast_inference=True`.** vLLM cannot serve |
| 158 | LoRA adapters on vision layers, so a fast- |
| 159 | inference setup that also unfreezes vision |
| 160 | layers fails at serve time even if training |
| 161 | succeeds. If the recipe calls for unfreezing the |
| 162 | last-6 ViT layers (see When to Unfreeze above), |
| 163 | fast inference is off the table for that run — |
| 164 | choose one or the other, not both. |
| 165 | |
| 166 | ## Model Choice |
| 167 | |
| 168 | Base VLM choice is out of scope for this skill — |
| 169 | it lives in one place, the model catalog at |
| 170 | `finetuning-method-selection`'s |
| 171 | `references/model-catalog.md`. This skill and its |
| 172 | references describe recipes by architecture |
| 173 | family only, never by recommending one model over |
| 174 | another. |
| 175 | |
| 176 | VLM reinforcement learning (VLM-GRPO) is |
| 177 | reference-only in this plugin — the fragmented |
| 178 | tooling and reward-hacking failure modes specific |
| 179 | to VLM-RL are covered in `grpo-rlvr-training`, |
| 180 | not here. This skill's scope stops at supervised |
| 181 | fine-tuning. |
| 182 | |
| 183 | ## Failure Modes |
| 184 | |
| 185 | The recurring mistake across every section above |
| 186 | is treating a clean loss curve as proof the run |
| 187 | is healthy. A normal-looking curve is consistent |
| 188 | with **both** a working run **and** either silent |
| 189 | killer, since the model trains on *something* |
| 190 | either way — just not the aligned image-text |
| 191 | signal when a killer is present. A flat eval score |
| 192 | next to a normal loss curve means re-run the |
| 193 | checklist in `references/collators-and-pitfalls.md` |
| 194 | before touching any hyperparameter. |
| 195 | |
| 196 | ## References |
| 197 | |
| 198 | - `references/collators-and-pitfalls.md` — per- |
| 199 | architecture collator table, dataset-format |
| 200 | examples with image placeholders, a pre- |
| 201 | training validation checklist, and the two- |
| 202 | stage projector-alignment recipe as an advanced |
| 203 | pattern. |
| 204 | |
| 205 | Related skills: `finetuning-method-selection` |
| 206 | routes here; `lora-qlora-recipes` covers the |
| 207 | text-only LoRA fundamentals this skill |
| 208 | specializes; `grpo-rlvr-training` covers VLM-RL |
| 209 | (reference-only); `dataset-curation` covers |
| 210 | image+text dataset preparation this skill doesn't. |
| 211 |
Reviews
Installed this one?Write the first review and take the Trailblazer badge.
Alternatives
Task Coordination StrategiesDecompose complex tasks, design dependency graphs, and coordinate multi-agent work with proper task descriptions and workload balancing. Use this skill when breaking down work for agent teams, managing task dependencies, or monitoring team progress.◐◐◐◐◐●35/40Ebay Seller Tools·····●34/40Tough Decision Advisor: Every Angle ConsideredHand in a decision you're stuck on. Get back a clear breakdown of every angle — the trade-offs, the risks, the blind spot, and a recommended path.●····●32/40DHDNA Profiler — Cognitive Pattern ExtractionPaste any email, proposal, or note someone wrote, and get back a plain-language read on how they think, what drives their decisions, and how they communicate.●····●32/40