In three of the ransomware incidents our team responded to last year, the backup job had been reporting success for months. In all three, the data could not be recovered from it. Nobody had lied and nothing was misconfigured in an obvious way. The jobs really had run, and they really had finished green. They just could not produce a working system when it mattered.
This is the most common serious gap we find during assessments, and it is entirely avoidable. A backup is a claim. A restore is evidence. Until you have performed the restore, you have a filing system with good intentions.
The six failure modes
Every failed recovery we have worked on falls into one of these categories.
1. The backup was encrypted along with production
The backup server was domain-joined, the backup repository was a plain SMB share, and the service account had write access to it. The attacker found all three in under an hour. If your backup storage can be reached and modified using credentials that exist in your production directory, it is not a backup. It is a second copy waiting in the same blast radius.
2. Something important was never in scope
New virtual machines get built and never added to a job. A database moves to a new host. An application starts writing to a second volume. The backup software has no idea, and the job stays green because everything it was told to protect really was protected. We routinely find that between five and fifteen per cent of production workloads are outside the backup scope at any given time.
3. The backup is crash-consistent, not application-consistent
The files are all there, but the database will not mount because it was captured mid-write with no quiesce. This is the failure that looks fine until the moment you need it, because file-level verification passes perfectly.
4. Retention is shorter than dwell time
Attackers commonly sit inside an environment for weeks before they trigger encryption. If you keep fourteen days of backups and the intrusion started five weeks ago, every restore point you hold contains the attacker. Retention has to be measured against how long a compromise might have gone unnoticed, not against how much storage you happened to buy.
5. Nobody can actually perform the restore
The one engineer who understood the backup platform has left. The documentation is a screenshot from 2021. The console password is in a password manager that is itself only accessible from a domain-joined machine, and the domain is down.
6. The restore works but takes far too long
Recovery from an off-site copy over a slow link is technically possible and commercially useless. If the numbers say eleven days and the business can survive two, you do not have a plan, you have an arithmetic problem you have not done yet.
The quarterly drill
This takes half a day and it is the single highest-value routine we run with clients. Do it with the same people who would be on shift during a real incident, not just the person who owns the backup platform.
- Pick the target in advance, not the sample. Choose one tier-one workload and one thing nobody expects, such as a file share or a certificate authority. Do not let the team pre-warm it.
- Restore into an isolated network. Never into production. A sandboxed VLAN with no route out is the whole point, and it also lets you scan the restored system for malware before you would ever trust it.
- Time everything. Record when the decision was made, when the restore started, when the system booted, and when the application actually served a request. The last number is your real recovery time, and it is usually much larger than the first estimate.
- Prove the application, not the file. Log in. Run a report. Query yesterday's transactions. A VM that boots is not a service that works.
- Restore from the off-site or immutable copy at least once a year. This is the copy that matters in a ransomware event and the one that gets tested least.
- Do one run without the specialist. Hand the runbook to a competent engineer who does not own the platform. If they cannot follow it, the runbook is wrong, and you have just learned that cheaply.
- Write down what broke. Every drill surfaces something. Log it, assign it, fix it before the next one.
The number that matters. Not "did the backup run" but "how long from decision to service restored, verified by a person who was not the one who built it". Track that number over time. If it is not falling, the drills are not producing change.
What good looks like
| Control | Weak | Strong |
|---|---|---|
| Repository | SMB share on the domain | Hardened repository or object storage with a retention lock |
| Credentials | Domain admin service account | Isolated local account, multi-factor on the console |
| Copies | One, on site | Three copies, two media, one off site, one immutable |
| Retention | 14 days | Long enough to predate a plausible dwell time, typically 60 days or more |
| Verification | Job status is green | Automated sandbox boot and application check every week |
| Documentation | Tribal knowledge | Runbook proven by someone who did not write it |
Start this week
You do not need a project to begin. Pick one system, restore it into an isolated network, and time how long it takes to serve a real request. Whatever that exercise surfaces is worth more than another month of green job reports.
If you would like us to run the first drill with your team, it is part of the free assessment. We bring the sandbox, the checklist and an engineer who has done the real thing under pressure.