Serguey Asael Shinder: A linter that stays silent on new syntax looks exactly like a clean codebase
Checkstyle 14.2.0, released on 27 September, fixed two kinds of bugs that look alike in a changelog and behave very differently in a team.
One was a false positive: UnusedLocalVariable flagged an unnamed variable _ inside an anonymous class. Somebody's build went red, they wrote a reproduction with the exact command line, and it was fixed. The other kind was false negatives: LambdaParameterName and NeedBraces silently passed lambdas inside a switch rule. Nobody's build went red. Nobody was annoyed. The check simply did not look at code written with newer syntax, and the report said what it always says when there is nothing to find.
I think this asymmetry is worth designing for, because it is not specific to Checkstyle. Every tool that inspects code, from linters to security scanners to coverage reports, is written against the language as it was when the rule was written. When the language grows, the rule does not fail. It stops seeing part of the code, and "no findings" is the output for both "checked and fine" and "never checked".
A false positive announces itself. It costs somebody time, so they complain, and the complaint is the bug report. That is also why false positives are dangerous in a different way: if they are not fixed fast, people learn to suppress them, and the suppression outlives the bug.
A false negative has no announcer. The only way to find one is to feed the tool something it must flag and see whether it does. So when my team moves to a new language level, I do three things before I trust the reports again.
Write a known-bad sample in the new syntax. A lambda without braces inside a switch rule, a badly named record component, whatever the rules claim to catch. If the tool is silent on a file that must fail, the rule does not cover that syntax yet, whatever the documentation says.
Keep that sample in the repository. A negative control that lives next to the configuration runs on every upgrade of the tool, which is when coverage changes.
Read the "false negative" section of each release note. Tools list these fixes quietly, and each one is a record of code the tool had not been reading. The fix does not tell you what slipped through before it; only running the new version over the old code does.
A clean report is evidence only if the tool could have produced a dirty one. That is easy to forget exactly when it matters, right after an upgrade that made everything look fine.