SSerguey Asael Shinder
Java coding notes: the JVM, and writing software that lasts

Serguey Asael Shinder: An example in a specification is code someone will copy, so test it like code

· by Serguey Asael Shinder / Serguey Shinder

On 9 November the draft Bitcoin proposal BIP 392 changed two lines. The specification text was already right: it says the scan and spend keys come from the child paths 1h/0 and 0h/0. Two of its examples did not follow that rule, and both derived the keys from the same path. Nobody had to misread the prose to get it wrong; they only had to copy an example.

That is the normal way examples fail. The rule is reviewed as prose, carefully, by people who know the domain. The example is read as an illustration, so reviewers check that it looks plausible, not that it obeys the rule above it. Then the example becomes the most used part of the document, because a developer in a hurry scrolls past the rule to the snippet.

Java libraries have the same weak spot in three places. Javadoc {@code} blocks are never compiled. README snippets drift as APIs change, and nobody notices until an issue arrives saying the example does not build. And configuration examples, a YAML block or a properties file, are often the first thing a user pastes into production.

Three habits reduce this, and none is expensive.

Serguey Asael Shinder: An example in a specification is code someone will copy, so test it like code
An example in a specification is code someone will copy, so test it like code — Serguey Asael Shinder

First, make examples executable. Keep them as real test methods and include them in documentation, so a change that breaks the example breaks the build. Since release 18 javadoc has a @snippet tag, introduced by JEP 413, whose stated goal includes making it easier to validate the code fragments in API documentation; the JEP is explicit that javadoc itself does not compile or run them and leaves that to external tools, which is where your tests come in.

Second, check examples against the rule, not just for compilation. A snippet that compiles can still contradict the text, as in BIP 392, where both versions were syntactically valid descriptors. If the rule is checkable, write the check: an assertion in the example test that the derived path matches the documented one.

Third, review examples last and separately. A reviewer who reads the rule and then the example tends to see what the rule says. Reading the example alone and asking "what would someone who copies this get?" finds the gap faster.

The cost of a wrong example rises with time. BIP 392 is a draft, so the fix landed before wallets shipped code built on it. A wrong snippet in a released library's README has already been copied, and the fix only reaches the people who look again.