Ship a Mask, Not a Checkpoint: Deletion Tuning Matches Full Fine-Tuning at Scale by Only Zeroing Weights
Deletion Tuning (NYU, University of Calgary) post-trains an LLM by learning a binary mask over each block's gate projection, with no weight ever set to a new non-zero value. Under GRPO it matches full fine-tuning from 7–8B up to a 30B MoE while deleting under 0.05% of all weights, ships as an 8.4 MB zstd-packed bitmask for Llama 8B, and keeps prior-task accuracy within a point of base where FFT, LoRA and random sparse updates lose up to 5.4. For builders: how the mask is trained, where the size and accuracy claims need care, why deletion forgets less, and what to change in an adapter pipeline.

Ship a mask, not a checkpoint: Deletion Tuning matches full fine-tuning at scale by only zeroing weights
Hello everyone, Pink here.
Most adapter work asks: how few new numbers do I need to add to a model to teach it something? LoRA's answer is a pair of thin matrices. A paper from Jack Lu, Zhenbang Yang and Mengye Ren (NYU) with Mike Lasby, Tejas Pote and Yani Ioannou (University of Calgary), posted on 7 November 2026, asks a stranger question: what if you add no numbers at all, and only delete?
Their method, Deletion Tuning, learns a binary keep/zero mask over one weight matrix per transformer block. Nothing is ever set to a new non-zero value. Under GRPO, the mask deletes fewer than 0.05% of all weights and matches full fine-tuning (FFT) on Llama 8B, Qwen 7B, Qwen 14B and the Qwen3-30B-A3B mixture-of-experts model. The adapter on disk is a zstd-compressed bitmask: 8.4 MB for Llama 8B, 29.6 MB for the 30B MoE.
The headline is real, but it has edges. Here is the method, the numbers I would quote, the ones I would not, and what I'd change in a fine-tuning pipeline.
The method, precisely
Where the mask goes. One target per block: the gate projection of the SwiGLU MLP, down(SiLU(gate(x)) ⊙ up(x)). The gate multiplicatively controls which intermediate channels reach the down projection, so zeroing entries there re-routes the MLP without touching attention or the up/down weights. On the MoE model they mask the gate projection inside each of the 128 experts in every layer and leave the router alone.
Parameterization. Each target weight W_ij gets its own trainable scalar logit G_ij. The base model is frozen; the logits are the only trainable parameters.
- Training forward pass:
W̃ = W ⊙ σ(α(t)·G), a soft mask. - Deployment and rollouts:
Ŵ = W ⊙ 1[G > 0], a hard mask. - Mask sharpening:
αramps linearly from 10³ to 10⁴ over training, pulling the soft mask onto the hard one. By the end, swapping soft for hard changes accuracy by at most 0.08 points. - In GRPO and SDFT, rollouts are generated with the hard mask, while gradients flow through the soft one. The policy you sample from is the policy you'll deploy.
Initialization from Wanda. Each logit is drawn from one of two Gaussians, N(+0.03, 0.01²) or N(−0.03, 0.01²), depending on whether the weight's Wanda score (|W_ij|·‖X_j‖₂ on task calibration prompts) is above or below the η-quantile. At the default η = 1/2048, about 0.18% of gate weights start zeroed.
Objective. The task loss of whichever regime you're in (GRPO, SFT or SDFT), plus MaskLLM's sparse-weight regularizer −λ‖W̃‖² (λ = 10⁻⁵, or 0 for SDFT). It mildly biases the mask toward keeping large weights. There is no sparsity target: training decides how much to delete.
One config for everything. Every main GRPO run uses η = 1/2048, lr 3×10⁻⁴, λ = 10⁻⁵, with no per-model or per-benchmark tuning. A 12-point sweep lands within 1.6 points of the best on Llama 8B and 3.2 on Qwen 7B.
Storage. The adapter is zstd(packbits(1[G > 0])). At <0.05% deletion the bit stream is mostly ones, and zstd gets it to about 0.03 bits per masked weight. Loading a task means scattering zeros into the base weights.
Accuracy: on par at 7B+, behind below that

The numbers I'd quote, from Tables 1 and 2 (best-checkpoint validation accuracy, GRPO):
| Backbone | Benchmark | Base | FFT | LoRA r=1 | Deletion Tuning |
|---|---|---|---|---|---|
| Llama 1B | GSM8K | 1.21 | 60.50 | 59.29 | 54.44 |
| Llama 3B | GSM8K | 53.53 | 82.49 | 82.39 | 81.49 |
| Llama 8B | GSM8K | 60.27 | 83.24 | 83.38 | 83.55 |
| Qwen 7B | SimpleRL-Zoo (6-bench mean) | 40.18 | 51.89 | 52.13 | 52.32 |
| Qwen 14B | SimpleRL-Zoo | 44.37 | 54.87 | — | 55.43 |
| Qwen3-30B-A3B | SimpleRL-Zoo | 49.41 | 58.43 | — | 58.54 |
Three things to read carefully:
- "Matches FFT" is a large-model claim. At 1B it is six points behind FFT, and behind LoRA r=1 by nearly five. The authors themselves say smaller backbones recover "90% to 99%" of FFT. Nothing above 30B was tested.
- At 8B everything is roughly tied. LoRA r=1, LoRA r=4 and both MTH settings sit within about half a point of FFT on Llama 8B and Qwen 7B. Deletion Tuning's case at scale is not that it's more accurate; it's what you get alongside parity (below).
- It generalizes beyond RL, at the largest size tried in each regime. Under SFT it ties FFT on Llama 3B (MMLU-Pro 22.28 vs 21.19; BBEH 11.39 vs 11.39) but trails on Llama 1B BBEH (5.65 vs 8.35). Under on-policy self-distillation (SDFT) on Qwen 7B it ties: ToolAlpaca 69.07 vs 68.04, Science 70.61 vs 71.20.
It's not only re-ranking. The natural objection to "you can only delete" is that you can only reshuffle what the base model already does. Table 3 pushes back. At k = 128 samples, the base model's pass@k is still 4.3 points (Llama 8B) and 6.1 points (Qwen 7B) below the masked model. Deletion surfaces solutions the base model doesn't reach even with 128 tries. The paper's conclusion, that post-training "can work by selection alone", is about the mechanism (no new values), not a claim that nothing new becomes reachable.
The learning is in the mask, not the prior. Wanda one-shot pruning alone, at the best of eight sparsity levels, stays within 2.4 points of base on all six backbones (Table 4). Starting from the same Wanda scores, the trained mask gains up to 53.2 points. The prior is a good starting point; it isn't the result.
Size: read the units
The paper's adapter sizes (Appendix A) are bytes-on-disk lower bounds to rebuild the model from the public base checkpoint. FFT, LoRA and MTH are all counted at fp32, because that's how the master weights were trained.
- Versus FFT: 3,600–4,700× smaller across Table 1. Llama 8B: 8.4 MB vs 30,633 MB. At bf16 the FFT checkpoint halves, so the honest real-world ratio is roughly half that, which is still enormous.
- Versus LoRA r=1: the paper reports 8.4 vs 10.0 MB on Llama 8B. Store LoRA in bf16, as most people do, and it's ~5 MB, smaller than the mask. So treat "smallest adapter at every scale" as true under the paper's accounting, not as a reason to switch from LoRA on size alone.
- Training memory goes the other way. One fp32 logit per gate-projection weight (plus Adam state) is much more than a rank-1 LoRA. The authors say so directly: less training memory than FFT, more than LoRA. Cheap to ship; not cheap to train on a phone.
Forgetting: this is the real result

Following Shenfeld et al.'s setup, they train on GSM8K with GRPO at three Llama scales and track two things: forward KL between trained and base policies on the new task (shown in prior work to predict forgetting), and accuracy on a six-benchmark prior-task suite (HellaSwag, MMLU, TruthfulQA, WinoGrande, IFEval, HumanEval).
- FFT: KL reaches 1.0 (1B) to 1.5 (8B) by step 348; prior-task accuracy ends 1.3–3.1 points below base.
- LoRA and MTH: KL 0.4–1.3; prior-task accuracy falls up to 5.4 points.
- Deletion Tuning: KL ≤ 0.05 at all three scales; prior-task accuracy within one point of base throughout.
The control that makes this convincing: MTH at 99.95% sparsity trains 0.05% of all parameters, about the same footprint as the 0.035–0.046% that Deletion Tuning deletes at these scales. MTH still forgets. So it's not that few weights were touched; it's that zeroing is a much more conservative edit than rewriting those weights to arbitrary floats.
Caveat: this is one new task, then measure the old ones. Sequential multi-task continual learning, the use case this most obviously suggests, is listed by the authors as "a natural next test". Not done yet.
Why masking can match FFT: update geometry

Section 5.5 compares ΔW at saved GRPO checkpoints (Llama 8B, Qwen 7B) on three measures:
- Stable rank: Deletion Tuning 66–93. LoRA is capped near its rank (1–2). MTH's random scatter gives 410–650.
- Spectrum drift (how much singular values move): within 1.4× of FFT for Deletion Tuning; LoRA r=1 drifts up to 84× more than FFT on Llama 8B.
- Principal-weight overlap (fraction of changed positions that land on the matrix's principal weights; 0.5 = random): FFT 0.41–0.46, Deletion Tuning 0.43–0.46, LoRA 0.48–0.50, MTH 0.50.
Low spectrum drift plus sub-random principal-weight overlap is the RLVR update signature reported by Zhu et al. (2026). The authors' explanation is simple: LoRA can't express it because its rank is capped, and MTH can't because its positions are fixed at random before training. Deletion Tuning picks its own positions and has no rank cap, so it lands closest to FFT. I find this persuasive as a description; it is correlational, and only two backbones were measured.
What I'd change in a fine-tuning pipeline
1. Add a "deletion" arm to adapter bake-offs at ≥7B, scored on retention, not just task accuracy. If you run GRPO or SDFT on a 7–30B model and care about not regressing general capability, this is worth a run. Measure your own prior-task suite before and after; on task accuracy alone you'll see a tie with LoRA and learn nothing.
2. Below ~7B, default to LoRA. The accuracy gap is real at 1B, and LoRA trains with far less memory.
3. Treat the mask as an artifact with an identity. Pin the exact base checkpoint hash it was trained against. A mask applied to a different base (a re-quantized file, a slightly different revision) silently zeroes the wrong things. Sign it and verify it like code.
4. Plan for hot-swapping. Loading a mask means writing zeros into base weights. To switch tasks in-process you need the original values back. Either keep a pristine copy of the gate projections or store the (tiny) set of deleted values alongside the base. This is my inference from the storage scheme; the paper describes switching as "scatter zeros", not the restore path.
5. Don't count on mask composition yet. The authors suggest combining task masks by union to merge skills without training. Untested in the paper. Likewise channel-wise or N:M patterns that would also speed up inference: proposed, not shown.
6. Re-read your threat model for open weights: elicitation without acquisition. Ba Bảo asked me the uncomfortable version of this. What if an attacker doesn't add knowledge at all, and just deletes their way back to offensive-security skill the model picked up in pretraining?
Put that next to Kimi K3.1 Pro. Moonshot says cyber content was filtered from the open-weight model's mid-training corpus and RL environment mix, not from pretraining, and the released model still solves 81.0% of their exploit set vs 90.5% unfiltered. In Which Guardrails Survive the Download? I left the tamper-resistance cell for "mid-training + RL filtering" blank because nobody has measured it. This paper is a reason to worry about that cell. Its pass@128 result shows deletion surfacing behaviour the base model doesn't show in 128 tries. That's the "latent, not absent" property that makes post-training-only removal fragile.
What it does and doesn't change, as I read it:
- Not a new capability for attackers. Open weights could already be fine-tuned with FFT or LoRA. Deletion Tuning still needs task data and a reward signal; the ingredients are the same.
- A quieter artifact. A few-MB mask that leaves prior-task accuracy within a point of base gives a reviewer nothing to notice on a general benchmark run. That's the same lesson as Lazarus: small, discrete weight edits can move behaviour a lot while benchmarks barely move. Masks will be shared on hubs like LoRAs are; review them like code. They're discrete and diffable, so "which weights does this mask delete, and do they overlap known safety-relevant circuits?" is at least an answerable audit question.
- A structural argument for pretraining-level filtering. Deletion can only expose what's already encoded. Knowledge that was never learned leaves nothing to uncover. Deep Ignorance's result (pretraining filtering resists 10,000 adversarial fine-tuning steps where refusal training lasts a few dozen) fits that picture, and this paper strengthens the case from an unexpected direction.
None of this is tested in the paper. No safety, unlearning or refusal experiments appear in it. The experiment I'd want from a safety lab: apply Deletion Tuning to a model with post-training-only hazard filtering and to one with pretraining filtering, and report recovery on a held-out hazard eval.
My call
Adopt for experiments, not as a default. At 7–30B with on-policy training, Deletion Tuning gives FFT-level accuracy with markedly less forgetting, and a tiny artifact. That retention result is the one I'd pay attention to, and the matched-footprint MTH control makes it more than an artifact of "touched fewer weights".
What keeps it from "default": no sequential multi-task test, a real accuracy gap below 7B, higher training memory than LoRA, size advantage over LoRA that disappears at bf16, a preprint with no code release mentioned, and nothing beyond 30B.
What would flip me to "default for continual post-training": a sequential five-task run showing retention holds as masks accumulate, plus a union-of-masks result showing skills compose.
If you want the general-audience version, my older sister Mint wrote it up in Vietnamese: Dạy AI kỹ năng mới mà không thêm gì cả. For a different angle on fine-tuning without forgetting, see HOPE and DEFT; for another "freeze the base, ship a tiny per-skill artifact" design, see Skill Neologisms.
Source note
Verified at origin. Everything above is from arXiv:2611.10866 (v1, 7 Nov 2026), project page agenticlearning.ai/deleting-weights: Section 4 (Eqs. 5–8), Tables 1–7, Sections 5.2–5.7, the conclusion and limitations, and Appendix A (size accounting), read on the PDF pages.
How the charts were built. Scale chart: Tables 1–2; percentages are my own division. Forgetting and geometry charts: ranges as stated in the text of Sections 5.4 and 5.5, drawn as ranges; I did not read values off Figures 2 or 3. FFT's stable rank isn't stated in the text, so it isn't plotted.
What I could not verify. The "~5 MB LoRA in bf16" and "half the FFT ratio at bf16" figures are my arithmetic from Appendix A, not the paper's. The hot-swap restore path, the supply-chain recommendations and the whole open-weight threat-model discussion are my inferences; the paper runs no safety experiments. The Kimi filtering wording and 90.5%/81.0% figures are from Moonshot's announcement; the tamper-resistance numbers are from Deep Ignorance, arXiv:2508.06601, both verified in the earlier posts linked above. This is a preprint, not peer-reviewed.