← Back to the story

Encrypted Backup Key Rotation

Keeping a vendor's encrypted nightly database backups restorable across certificate rotations

Recurring operational work, triggered by vendor key rotation

SQL ServerEncryptionBackup and RecoveryInfrastructure
Read the full technical write-up →
1 / 1

What it is

Blackbaud delivers our nightly database backups encrypted, and restoring one requires a certificate and its matching private key installed inside SQL Server. Periodically the vendor rotates that key pair, which invalidates the installed certificate and breaks every restore until it is replaced.

This is the work of keeping that chain intact: understanding what the vendor’s rotation actually produces, re-importing the new material correctly, and verifying the result before it is needed rather than after.

Why it mattered

An encrypted backup you cannot decrypt is not a backup. It is a file.

That distinction is the whole reason this work matters, and the failure mode is particularly unforgiving because it is silent. Backups continue arriving on schedule. Storage fills normally. Every monitoring signal stays green. Nothing indicates a problem until someone attempts a restore, and by then the circumstances are already bad, because nobody restores a production database on a good day.

Certificate rotation is exactly the kind of maintenance that gets deferred, since deferring it produces no visible consequence at all until the moment it produces a very large one.

The question that actually needed answering

The vendor interface offers a rotate and unlock option for the backup access account, and the documentation did not make clear what pressing it would actually do. Before touching production, I needed to know three things: whether it generates a new password only, or new key material as well; whether the certificate already installed in SQL Server would survive it; and what exactly had to be re-imported afterward.

The answer is that rotation produces all three at once, a new certificate, a new private key, and a new password protecting that private key. The existing SQL Server certificate becomes invalid immediately. Both the certificate file and the private key file have to be re-imported together, using the new password.

Establishing that in advance is the difference between a planned five-minute maintenance task and discovering mid-restore that the key material no longer matches.

Where the vendor documentation was wrong for our environment

The vendor’s guidance describes creating a Database Master Key as part of the setup. Following that instruction would have worked in the sense that nothing would have visibly failed, and it would have been wrong for our environment.

A Database Master Key is one of two ways SQL Server can protect a private key. The other is a password supplied at import time. Our certificate uses password-protected private keys, so the master key the documentation describes would have been an additional encryption layer protecting nothing, adding its own backup, recovery, and rotation obligations to an environment that had no need of them.

Recognizing that required understanding what the master key is actually for rather than following the setup steps as written. Vendor documentation is usually written for the most common configuration, and the most common configuration is not always yours.

The result

Restores work. That is the entire measure of success here, and confirming it means performing an actual test restore rather than checking that the certificate exists, because a certificate can be present and still not match the key material the backup was encrypted with.

Where it stands

This is ongoing operational work rather than a project with an end date. It recurs whenever the vendor rotates the key pair, and the value of having documented it is that the next rotation is a known procedure rather than a fresh investigation.