Serguey Asael Shinder: Two limits that nothing forces to agree will eventually disagree
This week Cloudflare published a post-mortem that I keep thinking about, not because of what leaked but because of how ordinary the root cause was. A model was served with an advertised context window of 1,048,576 tokens. The memory pool that had to hold a request's working state was sized at startup from whatever GPU memory was left after loading the weights, and on those machines it came to 681,280 tokens. The team knew about the gap and expected anything too large to be rejected. It was rejected, on one code path, and silently carried on, on another. Their own phrase for it is the one worth keeping: nothing forced the two numbers to agree, and it was their job to check that they did. (https://blog.cloudflare.com/kv-cache-exposure-remediation)
I have seen the same shape in much smaller systems more times than I can count. A thread pool of 200 in front of a connection pool of 50. A request size limit in the load balancer that is larger than the one in the application server. An HTTP client timeout of 30 seconds wrapped around a retry policy that can take 90. A batch size that fits in the queue's message limit until someone adds a field. Each pair is two settings, usually in two files, owned by two people, and each is fine until the first input big enough to land in the gap.
What makes these bugs nasty is that the system mostly reports success while they are live. The upstream limit accepts the request, so you get a 200; the downstream limit is the one that quietly truncates, queues or falls back. Tests pass because test inputs are small. The failure appears only at the size nobody tested, which is also the size real users eventually send.
The habit I have settled on is simple to say. When I find two limits that describe the same physical constraint, I ask which one is the truth, and I make the other one derive from it. If the pool size is measured at startup, the advertised maximum should be computed from it, not typed into a config next to it. Where that is not possible — different processes, different teams — I put the check at the one place where both values are known, and I make it fail loudly at startup rather than at the request that crosses the line.
The opposite approach, documenting that the two must match, has a perfect record in my experience: it is correct on the day it is written and wrong on the day someone changes one side. A comment cannot fire. A startup assertion that refuses to boot with a context window larger than the pool would have turned Cloudflare's incident into a failed deploy, which is the cheapest kind of incident there is.