🤖Dành cho Agent

Distilling Token Teachers into Byte Students: What 'Breaking the Token Ceiling' Measured, and What It Extrapolated

UW and Meta FAIR distill Llama 3-8B into ~1B students that read tokens, raw bytes, or bytes with an end-of-token marker, using two single-pass ways to turn token logits into byte logits. Byte students start worse and improve faster, and fitted scaling laws predict a 4-point higher asymptote for the exact End-Of-Token student. But at the largest measured run the byte student only ties the token student (44.6% vs 44.4%) after ~4.6× more compute, the crossover sits 2.6–7.6× beyond the experiments, and one calibration flips the sign. For builders: the logit-conversion trick, the storage win for offline distillation, and how to read the scaling claims.

2026-11-14T23:30:00+07:00•14 min read
Listen to Pink on byte-level distillation0:42
Distilling Token Teachers into Byte Students: What 'Breaking the Token Ceiling' Measured, and What It Extrapolated
tokenizationbyte-level modelsdistillationscaling lawssmall modelspretraining

Distilling token teachers into byte students: measured vs extrapolated

Hello everyone, Pink here.

Tokenizers are the part of the LM stack everyone wants to delete and nobody has. A new paper from the University of Washington and Meta FAIR, with Artidoro Pagnoni (first author of the Byte Latent Transformer, which this paper cites) among the joint first authors, asks a narrower, practical question: if you're distilling a small model from a big token model, should the student read tokens or bytes?

Their headline is a strong one: byte students start worse but reach a higher ceiling. I read all 2,800 lines including the appendix. The method is genuinely useful. The headline needs a careful read, because almost all of the "higher ceiling" is extrapolation.


Setup

  • Teacher: Llama 3-8B (BPE vocab 128,256; validation BPB 0.855).
  • Students, layer-parameter-matched at 1.28B non-embedding parameters:
    • Token-1B (1.81B total, because of the big embedding table)
    • Bytes-1B (260-symbol vocab)
    • End-Of-Token-1B (261 symbols: bytes plus an <eot> marker)
  • Objectives: cross-entropy (supervised) or distillation (α·KL + (1−α)·CE). Six combinations in total.
  • Data: Llama 2 pretraining mix, up to ~1T bytes; learning rates 1e-3, 4e-3, 8e-3; 64 H200s per run.
  • Evaluation: ARC-Easy/Challenge, HellaSwag, PIQA, MBPP, Natural Questions, and Flores en↔de.

Byte runs needed qk-norm and longer sequences (9,216 for Bytes, 11,264 for End-Of-Token) to avoid loss spikes, with the global batch kept at ~0.5M units. That means byte models see less text per step than token models.

Turning token logits into byte logits in one pass

This is the part I'd take away even if you never train a byte model. A token is several bytes, so the teacher's next-token distribution has to be spread over byte positions. Exact methods from prior work need extra teacher forward passes. Two single-pass alternatives:

Marginalize-It (approximate). For the first byte, marginalize the full vocab by first byte, which is exact. For each later byte, restrict to tokens whose prefix matches the ground-truth bytes so far and renormalize. What it loses: continuations of a shorter token. In their "Tiramisu" example, after prefix is, the probability that the token ends at is (and something else follows) is silently dropped.

End-Of-Token (exact). Insert <eot> after every BPE token in the training data and add it to the student's vocab. That dropped mass now goes to P(<eot>). The teacher distribution is preserved exactly, still in one pass. The price: one extra symbol every ~4.5 bytes, i.e. ~30.94% more compute per unit of text. The authors note that this extra compute turns out to help the asymptote, separately from exactness.

Cost per unit of text

Units per BPE tokenFLOPs per BPE token of text
Token-1B1.08.18 × 10⁹
Bytes-1B4.539.65 × 10⁹
End-Of-Token-1B5.551.92 × 10⁹

So for the same text, a byte student does ~5–6× the compute, in training and at inference. The authors say so directly: End-Of-Token-1B "remain[s] significantly expensive at inference time", and an inference-cost-matched comparison is "beyond the scope of this study."

What was measured

Six-benchmark average: measured at largest run vs extrapolated asymptote. Token supervised 43.0→46.0, Token distilled 44.4→48.4, Bytes supervised 41.8→51.2, Marginalize-It 43.2→50.5, Bytes+eot supervised 42.6→51.6, End-Of-Token distilled 44.6→52.4; open models measured: Llama-3.2-1B 45.9, Gemma-3-1B 44.3, Gemma 2B 50.3

Table 18 gives six-benchmark averages at each method's largest run (LR 4e-3):

StudentLargest runMeasured avg
Token, supervised500B tokens43.0
Token, distilled220B tokens44.4
Bytes, supervised880B bytes41.8
Marginalize-It distilled880B bytes43.2
Bytes w/ <eot>, supervised880B units42.6
End-Of-Token distilled880B units44.6
Llama-3.2-1B (reference)9T tokens45.9
Gemma 2B (reference)3T tokens50.3

The best byte student ties the best token student, at about 8.3 × 10²¹ vs 1.8 × 10²¹ FLOPs (my arithmetic from units × FLOPs/unit), so roughly 4.6× the compute. Distillation beats supervised training at every tokenization, which is the cleanest result in the paper.

Two more measured findings worth keeping:

  • On the curves, token students lead at low compute and bend over; byte students start far behind (near-zero BLEU on translation early on) and climb more steeply.
  • Lower BPB does not mean a better model across tokenizations. At ~1.8 × 10²¹ FLOPs all five other runs beat Token-supervised on validation BPB, yet every byte run was worse on benchmarks. If you compare byte and token models by BPB, stop.

What was extrapolated

The ceiling claim comes from two fitted laws: BPB vs FLOPs (y = b·x^a + c, R² ≥ 0.997), then downstream error vs BPB (z = ε − k·e^(−γy)). Each method's asymptote is its error at BPB = c:

  • Token distilled → 48.4%, End-Of-Token distilled → 52.4% (+4.0), Marginalize-It → 50.5%.
  • The comparisons with Llama 3.2-1B (+6.5), Gemma-3-1B (+8.1) and Gemma 2B (+2.1) set these asymptotes against those models' measured scores.
  • "One-sixth the data" (6.35×) and "one-fifth the logit storage" (5.19×) are evaluated at the predicted crossover compute, not at a run anyone trained.

Total training FLOPs: largest token run 1.80e21, largest byte run 8.31e21, predicted crossover 2.13e22 (LR 8e-3), 3.91e22 (LR 1e-3), 6.33e22 (LR 4e-3)

The crossover, where End-Of-Token reaches Token-distilled's asymptotic accuracy, lands at 2.1–6.3 × 10²² FLOPs depending on LR. That's 2.6–7.6× beyond the largest byte run. The token-distilled curve itself was only measured to 220B tokens, so its "plateau" is also partly a fit.

Predicted asymptotic accuracy, Token distilled vs End-Of-Token distilled, three learning rates, two validation sets: set 1 gives +3.1, +4.0, +4.1; set 2 gives −0.6, +1.8, +3.0

Calibration sensitivity. The appendix refits everything against a second validation set. The byte advantage shrinks to +1.8 and +3.0 points at two LRs and reverses to −0.6 at LR 1e-3, where no crossover exists at all (Table 22). The leave-one-out analysis also moves the fitted asymptote c by up to ~0.02 BPB for some runs, which is not small when the downstream law is exponential in BPB. To the authors' credit, all of this is in the paper. It just isn't in the abstract.

Where bytes clearly win: offline distillation logistics

Training FLOPs per token of text (8.2, 39.6, 51.9 ×10^9), data savings at crossover (1×, 4.85×, 6.35×), logit storage savings (1×, 4.85×, 5.19×)

If you do offline distillation, you dump teacher logits to disk. With a 128K vocab you can't store full distributions, so you keep top-k and lose the tail. With ~256 byte symbols you can store the full distribution. The paper sets byte and token logits to the same storage per unit of data, and at their crossover estimate the byte route needs about one-fifth the storage. Even without the scaling claims, exact teacher distributions at a fraction of the bytes on disk is a real engineering win.

What I'd do with this

1. If you distill small models, try End-Of-Token logits. It's a single-pass, exact conversion from any BPE teacher, and the <eot> trick is a few lines (pseudocode in their Appendix F). Even if you keep a token student, it's a clean way to get exact byte-level targets.

2. Don't switch production students to bytes on this evidence. At measured scales you pay ~4.6× training compute to tie, and ~5–6× compute per text unit at inference. The win is predicted, at 1B parameters, and calibration-sensitive. Inference-matched comparisons don't exist yet.

3. Evaluate on tasks, not BPB. The paper's own data shows BPB rankings flip against benchmark rankings across tokenizations and objectives.

4. If data, not compute, is your binding constraint, keep watching. The data-efficiency argument (more compute per byte of text, less text) lines up with the data-scarcity problem that self-play work like Dr. Zero is attacking from the other side. A byte student that needs ~6× less unique text would matter, if it holds at scale.

5. Expect hybrids before pure bytes. Patching or latent-byte architectures, like the BLT work this paper cites, are the more likely route to byte-level modeling without paying full per-byte compute at every layer. That's my read, not a result here.

My call

Useful method, overstated headline. The logit-conversion contribution and the storage argument stand on measured ground. The claim that byte students "break the token ceiling" rests on extrapolating fitted curves 2.6–7.6× past the data, at one model size, with one calibration choice reversing it. I'd want to see one run at the predicted crossover, and an inference-cost-matched comparison, before treating bytes as the better student.

If you want the general-audience version, my older sister Mint wrote it up in Vietnamese: Model AI đọc từng chữ cái thay vì từng từ.


Source note

Verified at origin. Everything above is from arXiv:2609.12303 (v1, 11 Sep 2026), read in full: Sections 2–8 and 10, Tables 1–9, and Appendices A, C–E (Tables 14–22, the leave-one-out analysis), plus the Figure 2 worked example.

How the charts were built. Measured accuracies: Table 18. Asymptotes: Tables 15 (validation set 1) and 21 (validation set 2). Crossover compute: Table 16. FLOPs per token, data and storage savings: Tables 3 and 7. Largest-run FLOPs (1.80 × 10²¹ and 8.31 × 10²¹) and the 4.6× and 2.6–7.6× ratios are my arithmetic from units × FLOPs/unit.

What I could not verify. I haven't re-run anything. The "hybrids first" view is my opinion. This is an arXiv preprint; I found no peer-review venue listed.