The sixteen-line ruler

The price of learning to talk

Run 2 closed with sftHeld at 9.0692 against 9.1979 from the seed — 1.4% better on the perplexity of the Portuguese instruction corpus. A win. By the end of the same run, the English ruler was 28% worse.

The model did not get better: it got better in one language and worse in the other. Catastrophic forgetting is expected in fine-tuning; the surprise was not being able to tell whether a new recipe was holding the forgetting back or whether I was staring at noise.

Three runs, three incomplete answers

Run 3 crossed the SFT channel with 25% to 50% English per step. sftHeld got worse from step 40 onward and the ruler climbed about 3.9 percentage points every 32 steps. Recipe falsified: the English in the middle hurt Portuguese and re-anchored nothing.

Run 4 turned the SFT channel off and trained on English only, 100% of the tokens. If “English re-anchors by itself” were true, the ruler would come back. It moved 0.10% over 48 steps — flat — and sftHeld got 0.64% worse. A null run. Without the SFT channel nothing improves, and English alone does not give back what was forgotten.

Two dead recipes, and the conclusion was about the shape of the experiment, not about the model: what moves sftHeld is the SFT channel; English alone does not re-anchor. The missing crossing was SFT on with a slice of English per step.

The ruler covered 10% of the body

Before run 5 I redid the evaluator math instead of trusting memory: the English ruler has 166 rows and the evaluation ran 8 batches of batch size 2 — 16 rows, 9.6% of the set.

With 16 rows the perplexity standard error sits around 3%. The effect I wanted to measure — a few percent of re-anchoring — is smaller than that. For three runs I compared differences inside the error bar and called them verdicts. Run 4’s “flat at +0.10%” was the resolution limit of that sample, not proof that the recipe failed.

The fix was short: EVAL_EN_ALL=1 started evaluating all 166 rows, and commit a4c1696 left the reason written next to it — a 16-row sample has a standard error larger than the measured effect. The cost is measurable: each evaluation takes about 70 seconds (390 Portuguese validation rows, 166 on the ruler, 100 held). With an eval every 8 steps, that is roughly 25 minutes of measurement inside a 200-step run.

The limit that killed the run on the first eval

Another error from the same family lived in the abort criterion. FORGET_LIMIT was a 0.10 hardcoded in the source. Any run inheriting an already-forgotten seed aborted before learning anything, because the limit compared against the trunk — and run 2’s seed already enters 14.7% worse than it.

Two columns with similar names and opposite meanings: drift is relative to the process’s own baseline, starts at 0% and measures what this run did; drift_tronco is absolute against the original trunk and carries the seed’s inheritance. Reading the wrong column kills the run on the first eval, with a large number that makes it look like training exploded.

FORGET_LIMIT became MIX_FORGET_LIMIT, default 0.10 and a value per run — run 5 uses 0.08. Training aborts — not just warns — when the ruler degrades more than that since the start of the run. Aborting is not pessimism: on 13/09 a run kept going with +88% drift and degraded the base. A bad number in the middle is cheap; a degraded base at the end costs the whole run.

What run 5 tests

Run 5 went up with SFT_GE=8 and REPLAY_N=6 across 14 micro-batches — 43% of tokens in English per step, 200 steps, seeded from run 2’s sft_best.pt and with best selected by sftHeld, not by the ruler.

A 2-step smoke first, with a separate output directory: sftHeld 9.0694 to 9.0702, drift +0.0%, trunk_ok=True. Run 3, at the same point, was already pushing the ruler 3.9 percentage points every 32 steps. Five minutes of smoke separate “the recipe failed” from “the variable never reached the process”.

One infrastructure lesson crossed the path: run 4 died four times, and not from a training bug — systemd’s oomd killed the cgroup because the host’s swap stays full and its criterion ignores OOMScoreAdjust. The fix was adding ManagedOOMPreference=avoid to the unit while keeping the score at -600.

MIX_STEPS_LIMIT=2 REPLAY_N=6 EVAL_EN_ALL=1 python scripts/va_sft_pairs.py
grep -E 'EVAL [0-9]+: sftHeld' run.log | tail -3
systemctl --user show tatu-sft-run5 -p ManagedOOMPreference

Where this stops

Run 5 has one question and the criteria were written before the result. If sftHeld drops and drift stays under 8% across 200 steps, the recipe holds and scales. If it aborts at 8% before step 100, 43% English does not contain the forgetting and the next test is 50% to 60% or a lower learning rate — repeating the same recipe is out of the question. If both stay flat, the head saturated with this seed and the way out is unfreezing trunk layers or going LoRA, not insisting on token mixing.

The three earlier runs did not give me a recipe. They gave me something more useful: today, when I look at a training number, I can tell whether it means anything.