The tests that measured the machine — a green that depended on which computer ran it
Discoveries·

The tests that measured the machine — a green that depended on which computer ran it

A test that passed alone and failed in company

The symptom was specific and unflattering: the same test, with not one line changed, gave three different answers depending on how it was executed. Alone, green. Filtered by name, green. On the full suite, red — always the same one, always in the same position.

That pattern has an old name in software testing: order dependency. But the label explains nothing on its own, and the lazy investigation stops at “run it in parallel and it goes away”. The real cost showed up somewhere unexpected: the automated critic that signs off the delivery score got stuck at 60 for three consecutive rounds, always blocked by a single red test. The score would not climb because the result was not reproducible — not because the product was broken.

The tools I use on this project live in the skills store — the first one is free.

Commit afca95a closed that loop with a plain proof: full suite, 4,132 tests, zero failures. Before it, the same command returned exactly one error, and only on the complete run.

The cause lived in the import, not in the test

The failing test checked something reasonable: that a write never touches the user’s real data directory, only a temporary directory owned by the test itself. It did so by comparing the resolved path against that case’s temporary directory.

The problem was that another test file, when imported, pointed the environment variable defining the data directory at some temporary directory — created at import time, at module scope — and never restored the previous value. On the full suite that file was imported earlier, and every subsequent test inherited a temporary directory belonging to someone else. The isolation test saw a path that was not its own and failed, correctly.

Running alone, the contaminating file never entered collection. So the test passed. In other words: the red was true, and the green was a lie.

_antes = os.environ.get("APP_HOME")
os.environ["APP_HOME"] = tempfile.mkdtemp(prefix="worker-")

import worker  # the import only needs the directory; the suite does not

if _antes is None:
    os.environ.pop("APP_HOME", None)
else:
    os.environ["APP_HOME"] = _antes

The cure was not more isolation — it was undoing. Save the previous value, change it only for the necessary window, restore it immediately afterwards. Environment isolation is the job of a fixture with automatic teardown; the import needs nothing more than that.

Seven ways of measuring the machine

Commit daf693f, on the same day, closed eight failures. None of them was a product defect — all of them were the test asking the wrong environment.

The portability guard scanned comments as if they were code. It hunted for absolute machine paths in executable code, but the sweep also caught comments and docstrings. Three out of three “leaks” found were historical migration records, not paths in use. The filter now scans only executable code.

A plugin carried an absolute path from an operating system that no longer exists. Two path insertions from an old environment, kept after the migration: dead code and, worse, a record of private structure inside a public file. They now derive from the user’s directory with an existence check.

The network call detector skipped the line whenever it saw the function name anywhere. A parameter named after it was enough for the detector to ignore a real call right after an import. Comparison now requires an assignment starting the line — a real rebinding. With that, eight call sites entered the surface and received an explicit justification with a suppression marker instead of staying invisible.

A test indexed the provider list by fixed position. The list had five entries when the test was written; on 24 October two left and four remained. The index became an out-of-range access and the test died by position, not by behaviour. Resolution moved to name:

por_nome = {p["name"]: p for p in provedores}
primeiro = por_nome["local"]

An autonomy test measured the machine instead of the chain. It zeroed the key dictionary and expected the call to fall through to the next provider. But there is a local model server that answers with no key at all — so the chain “worked” for a reason the test did not control. With a stub on the call and triage switched off, the test measures the policy, not the availability of the machine.

Three security guards returned success without having anything to run. The static analyser they call was not installed in the virtual environment, and the absence turned into a skip state — not a failure. As a result, a real weak-hash finding stayed masked. Installing the tool lit all three at once, and using a hash as a structural signature now declares explicitly that it is not for security.

Why skipping is not passing

The point that ties the whole story together: there is a difference between “I verified and it is right” and “I could not verify”. In a health report that is the difference between green and grey. In a test suite it is the difference between coverage and theatre.

A guard that cannot run must fail loudly. A test that depends on external state must declare the dependency and control it. A test that indexes by position must index by identity. And a test that measures the environment must be rewritten to measure the contract — because a green that only appears on your computer is not information, it is coincidence.

What this changes

The suite stopped producing order-dependent results: 4,132 tests passing, zero failures, on the complete run. The 108 tests in the corrected files went green without exception. And the three security guards that used to skip now actually run.

More relevant than the number: red means something again. When everything passes for reasons other than “the code is right”, green stops being a signal.

Next step

Repeat the sweep for other places where the absence of a tool becomes a skip instead of a failure: any guard that depends on an external binary must state, in its own output, that it ran — and fail when it did not.

Related entries

Same project, same ground — the stories connect.

Next step

If this was useful, there is more: I write about the same ground in the other projects, and the store holds the tools I use here.

No spam, one-click unsubscribe. I am also here: Skills store · Portfolio