The preview server that failed in zero seconds
LifeLog·

The preview server that failed in zero seconds

The symptom came back in a new suit

This site’s E2E suite runs against a local preview server. It serves the freshly built output and navigates through it for close to forty-five minutes. That is not an infrastructure detail: it is what separates “the tests passed” from “the tests passed because the server was alive”. On November 12th the kernel killed the preview process mid-run — the runner marks child processes as the OOM killer’s preferred target, and the preview, the most expensive process in the suite, pays that bill first. The record has timestamps: the kill happened at 11:53:55; the server was gone for good at 12:17:51; and from there until 12:33:33, 242 assertions failed with net::ERR_CONNECTION_REFUSED. The product was intact. What was dead was the server.

The answer was a watchdog: a small supervisor that restarts the preview whenever the process exits, records exit code and uptime, and backs off when the exit is too fast to be normal. I wrote it convinced the class was solved — a server killed by OOM stops being a code-level red.

The tools I use on this project live in the skills store — the first one is free.

The watchdog became the problem

At 00:52 on November 17th CI went red again with the same message: the preview never opened the port. The difference was the watchdog log, which now existed: preview exited rc=1 after 0s. Zero seconds. A real HTTP server takes a few seconds to load the build output; exiting in zero seconds is not a server failure, it is a launch failure — the process never existed at all.

The cause was in the wrapper. To lower the odds of the OOM killer picking the preview, the watchdog prefixed the command with a utility that adjusts the process memory preference. That utility requires privilege. Without privilege it fails — and here is the detail that cost a night: when it fails, it does not execute the command placed after the separator. No stderr, no traceback: an immediate exit code 1 and a server that was never started. The watchdog, correct in its logic, did exactly what it should — it restarted. It restarted nothing, with growing waits between attempts.

The second defect, behind the first

I fixed the wrapper and ran again. The symptom did not change: rc=1 after 0s. That is where I almost lost the night by treating one diagnosis as if it were the only cause.

In CI the project installs dependencies with the repo’s own package manager, and the Astro binary lives in node_modules/.bin. The generic launcher resolved the command through PATH — which on my machine had the binary and on the runner did not. Identical result: exit 1, zero seconds, no message. Two independent faults with a single signature. While the first defect existed, the second was invisible; after fixing the first, the log said the same thing, which makes anyone conclude the fix did not work.

The fix: probe before using, and say which path won

The rule this episode leaves behind is plain: never prefix a command with a wrapper that can fail silently without first proving that it works. The probe is a real, cheap, disposable execution:

CHILDREN_PRIO=()
ADJ_NOTE="off(sem privilegio)"
if command -v choom >/dev/null 2>&1 && choom -n -300 -- true >/dev/null 2>&1; then
  CHILDREN_PRIO=(choom -n -300 --)
  ADJ_NOTE="on(-300)"
fi

If the probe passes, the wrapper goes in; if it does not, the command runs clean and the log records off(no privilege) — the operator reads the decision instead of guessing it. The alternative was keeping the optimization as an invisible requirement, and that is precisely how the preview stopped starting.

The second defect required explicit binary resolution: the project-local binary first, the package manager executor second, and generic resolution only as a last resort — with the chosen path printed in the log.

The third part was observability, and it may be the most useful of all. The failing step printed only the watchdog log, which said rc=1 and nothing else. Astro’s own stderr went to a separate file that was never read, so any real cause stayed invisible behind the summary. Now, when the port does not open, the diagnostic prints both: the watchdog record and the child’s output.

The same commit (3005a4d, six files, +36/−17) also carried a lint correction: two generic security rules fired on test files — reading a path built in a fixture, indexing an object by a dynamic key — and pushed the warning total above the project ceiling. Zero real risk, red CI. They are now disabled only in the test scope, with the reason written into the configuration.

What this changes

An exit code together with uptime is a signature, not a cause. rc=1 after zero seconds means “I never started”; rc=1 after three seconds means “I started and failed”. Those are opposite diagnoses, and the watchdog log now carries both fields exactly to separate them.

The second lesson is about wrappers: CPU limits, priority adjustments, timeouts, nice — anything placed in front of a command can swallow that command when it fails, and the symptom then shows up far from the culprit. Testing the wrapper with one real execution costs milliseconds and saves a night.

Next step

Sweep the rest of the CI chain for the same pattern: every wrapper that injects a prefix into a process needs a probe before use, and every failing step must print the output of whatever failed, not just the summary from whatever was watching.

Related entries

Same project, same ground — the stories connect.

Next step

If this was useful, there is more: I write about the same ground in the other projects, and the store holds the tools I use here.

No spam, one-click unsubscribe. I am also here: Skills store · Portfolio