Serguey Asael Shinder: A guard tested on a bench is not tested, and three of mine lied on the first real run
I spent a day building a small pipeline with guards in it — checks that refuse to let work through when something is wrong. I did the responsible thing and tested each guard in both directions: a case that must fire, a case that must stay silent. Every one passed. The installation check was green. I then ran the pipeline on real work for the first time, and three of the guards were wrong.
The interesting part is not that they were wrong. It is how. Not one of them fired on a genuine defect and missed it. All three fired on correct work.
The first blocked the step that was supposed to yield control while a long job ran — the exact moment when yielding is the right behaviour, because the job signals back when it finishes. The guard was watching for "is something still running" when the failure it was built for is "the step finished and nobody started the next one". Those look identical for about a second and are opposite in meaning.
The second decided that a file was unfinished work because its name contained a word. The file was a normal output of the pipeline, produced every run. The check had been written against a filename because the filename was available; the thing it actually needed to read was a verdict inside the document.
The third reported the time of every log line as the moment it was read, because the lines carried a timestamp without a date and the parser did not recognise that shape. The ordering was right, so nothing looked broken. The absolute times were fiction.
What the bench cannot see
Each of those guards was individually correct. The self-tests were real, ran in both directions, and I would write them the same way again. What they could not contain was the thing that actually broke: the transition between one step and the next. A bench exercises a component with inputs you chose. A real run exercises it with the state the previous step happened to leave behind, at the moment it happened to leave it.
That distinction has a practical consequence I have started applying elsewhere. When a check passes its unit test, I now write down what it has not yet been exposed to — specifically, which other step's output it will see first in production. That sentence is usually enough to reveal that I have tested the guard against the input I imagined rather than the input it will get.
And the harder discipline, which I keep relearning: a guard that fires on correct work is worse than no guard, because the lesson it teaches is to bypass it. Each of these three had to be fixed within the hour, not because the false alarm was expensive, but because the habit of overriding it would have been.