Serguey Asael Shinder: The speed was in the idle time
Two unrelated results from the past few days found their gains in the same place.
Headstart, a patch series for the Rust compiler and Cargo, lets a crate start compiling as soon as its dependency's interface has been checked, instead of waiting for every function body. On 16 cores, clean builds of real projects ran up to 42% faster. The author is clear about where the time came from: cores the build would otherwise leave idle. On 4 cores the gain shrinks, because there were fewer idle cores to use.
A Kubernetes benchmark of node swap on local NVMe packed up to three times as many Python sandboxes onto a node. The memory it freed was memory the sandboxes held while waiting for the next prompt. When the authors squeezed further and pushed memory that was actually in use into swap, a kernel build got more than 40% slower.
Neither result made the work faster. The compiler did not check a function body any quicker; the sandboxes did not run any leaner. Both found capacity that was allocated, paid for and doing nothing, and put it to use.
That is worth remembering because it suggests a different first question when a system feels too slow or too expensive. The usual instinct is to make the busy part faster: a better algorithm, a bigger machine. Often the cheaper win is to ask what is reserved but not working:
- Waiting, not computing. In a dependency chain, how much of the elapsed time is each step waiting for a predecessor to finish something it does not need?
- Held, not used. How much of the memory, connections or threads a process holds is in use at any moment, and how much is insurance?
- Serial by habit. Which steps run one after another only because they always have?
And the limit. The Kubernetes numbers show what happens past the idle part: once swap had to hold the working set, the gain turned into a 40% loss. Idle capacity is real, but it is finite, and the measurement that finds it is the same one that tells you when you have used it up. Both projects published that measurement. That is the part to copy.