Two DNS records, two servers gone

At 11:34 UTC on Wednesday, a small team ran an apply that was supposed to add two DNS records. A minute later their monitoring went red. Both production servers were gone.

The team runs an events site on Hetzner Cloud and describes its infrastructure with OpenTofu, the open source fork of Terraform. You write down the servers, records and volumes you want, the tool compares that with what exists, and it shows you a plan before it touches anything. That plan is the safety net. Most of us read its last line, something like "2 to add, 0 to change, 0 to destroy", and say yes. They wrote it all up in a refreshingly honest postmortem.

The plan they checked really did say 2 to add. It was a targeted plan: with -target, the tool only looks at the resources you name, here the two records. The runbook then split the plan and the apply into two commands, and the flag got lost on the way to the second one. So the apply planned the whole stack again, and the whole stack had drifted.

The drift was in user_data, the cloud-init script a server runs on its first boot. The scripts had changed since the last apply, and a changed user_data cannot be patched onto a running server. The only way to apply it is to destroy the server and create a new one. The second plan said "5 to add, 3 to destroy", and that was the one that ran.

Then the bad luck stacked up. A replacement destroys first and creates second, and when the new servers were requested, Hetzner had no room for that server type in any of its central European locations. The team rebuilt on a bigger type, reattached the database volume and let Flux pull the cluster back from git. The site answered again at 12:14, forty minutes after the apply. No database rows were lost. The logs, the alert history and every Web Push subscription were, and the monthly bill nearly tripled until the cheaper servers are back.

The nice thing about this postmortem is how ordinary the cause is. Nobody typed destroy. A flag fell out between two commands, and a summary line was the only guard. The fix is just as ordinary: save the plan you checked with tofu plan -out=prod.plan, then run tofu apply prod.plan, so the thing you approved is the thing that runs. They also told the tool to ignore changes to user_data, so a server is only rebuilt when someone asks for it.

Worth checking in your own pipeline this week: whether the plan shown for review and the apply that runs are the same run. If the apply plans again, the review is a guess. Terraform and OpenTofu both apply a saved plan file as is, and prevent_destroy on the few resources you can never lose is one line.

Sources: The postmortem (GitHub issue), OpenTofu: plan and -out, Hetzner Cloud provider: hcloud_server, OpenTofu: lifecycle and prevent_destroy