The held loss went down. The model got 103% worse.
Discoveries·

The held loss went down. The model got 103% worse.

At some point in every fine-tuning run you have to pick a checkpoint. The ruler almost everybody reaches for is the same one: eval_loss, or its teacher-forced cousin, held_loss. Lowest number wins. You select it, export it, ship it.

The number went from 9.09 down to 8.38. It looks like a win. It is exactly the scenario where the model got worse.

The tools I use on this project live in the skills store — the first one is free.

What the metric cannot see

held_loss measures one thing: how well the model answers on the held-out set you kept aside. It does not measure the other thing — how much it forgot what it already knew before you arrived.

In the run this post comes from, held loss fell 9.09 -> 8.38 while an earlier capability, English, degraded by +103%. The number used to approve the checkpoint got better at the same time the model actually got worse. Both facts are compatible: improving one set does not mean keeping the others.

The tool that measures the other half

ForgetGate is built around a simple claim: a checkpoint is not better because it is lower on one number, it is better because it did not forget. The scorecard it produces carries held loss and regressed probes together, and states the verdict in one word a CI pipeline understands: keep, or abort.

Zero dependencies, twenty tests, and an exit code a pipeline can gate on. The core decides; whoever trains just obeys the verdict.

Where it stands today

Honest status, no inflation:

  • Works: the measurement core. Scorecard in, drift report and exit code out. Deterministic, zero dependencies, 20 tests passing.
  • Missing: the model runner (produce a scorecard from a checkpoint), a calibrated probe suite, and a trainer callback.
  • Published: 0.1.0 on PyPI and npm, name reserved, two-tier licensing.

It is not for sale, and I will not sell it before the runner exists. What I am building right now is the technical argument — the asset that cannot be copied.

If you train models and pick checkpoints by loss, test the hypothesis on your last run before trusting the ruler. If the ruler was enough, you never noticed.

Repository: forgetgate

Related tools, same spirit of measuring before believing: Skills Shop

Related entries

Same project, same ground — the stories connect.

Next step

If this was useful, there is more: I write about the same ground in the other projects, and the store holds the tools I use here.

No spam, one-click unsubscribe. I am also here: Skills store · Portfolio