Serguey Asael Shinder: Every command that adds configuration needs one that takes it away
Payara's October release adds two small commands, delete-jvm-option and delete-system-properties. On a release-notes page they look like housekeeping. I think they point at something most tools get wrong for years: the command that adds a setting arrives long before the command that removes it.
When that happens, automation becomes one-directional. A script can create a JVM option, register a property, open a port, add a user. Undoing it means logging in, opening an admin console or editing a file by hand. The setup script and the real state of the server start to drift the first time somebody makes that manual change, and they never quite meet again.
The cost shows up in three places:
- Rollback. A deployment that added configuration cannot be rolled back by running the previous version of the script, because the previous script never had to remove anything. The new setting survives the rollback.
- Testing. An environment that can only accumulate configuration cannot be returned to a known state. Each test run starts from whatever the last one left behind.
- Trust. When some changes are scripted and others manual, nobody can say with confidence what the script will produce on a clean machine. People stop rebuilding environments because they are afraid to.
The fix is a design rule rather than a tool: for every operation that creates state, ship the operation that deletes it at the same time. Same command style, same permissions, same scripting support. If the product has create-x, it has delete-x in the same release, and the delete is idempotent — removing something that is already gone is a success, not an error, so a script can run it without first asking.
The same rule applies inside our own code. A feature flag that can be switched on by an API call but only switched off by a database edit; a migration with an up and no down; a cache that can be filled programmatically but only cleared by restarting the service. Each of these is a one-way door, and each one eventually forces a manual step at the worst possible moment.
The question I now ask when reviewing anything that writes configuration is simple: how do I get back to the state before this ran, using the same tool? If the answer involves a person and a console, the work is not finished yet.