## Kubernetes Restore Drills That Actually Prove Recovery
Most Kubernetes backup conversations start with the wrong question: Are we backing up the cluster?
A better question is: Can we restore the business service, with its data, secrets, ingress, dependencies, and expected behavior, inside the recovery time the business needs?
That distinction matters. A scheduled backup job can complete successfully for months while the organization remains unable to recover from a real outage. Common gaps include persistent volumes that were never snapshotted, secrets that were restored into the wrong namespace, ingress records that still point to production, applications that boot but cannot authenticate to external services, and restore procedures that only one engineer understands.
Kubernetes makes this harder because an application is rarely a single object. A working service may include Deployments, StatefulSets, ConfigMaps, Secrets, PersistentVolumeClaims, ServiceAccounts, RBAC rules, Ingress objects, cert-manager certificates, DNS records, object storage buckets, container registry access, and external databases. Restoring the YAML is not the same as restoring the service.
This article walks through a practical approach to Kubernetes restore drills for small and midsize businesses, nonprofits, and IT teams that operate production clusters but do not have a dedicated disaster recovery department. The examples use Velero because it is widely adopted and supports both Kubernetes API object backup and persistent volume snapshots, but the principles apply to other backup platforms as well.
## Backups Are Evidence Collection, Restores Are Proof
A backup is evidence that data was copied somewhere. A restore drill is proof that the copied data can be used to resume service.
For Kubernetes, a meaningful restore drill should answer six questions:
1. Can we rebuild the namespace and required Kubernetes objects?
2. Can we recover persistent data to a usable point in time?
3. Can the application start with correct configuration and secrets?
4. Can users or upstream systems reach the restored service?
5. Can the service pass functional tests, not just pod readiness checks?
6. Can the team complete the process within the documented recovery time objective?
The last question is often the most uncomfortable. A restore procedure that works only after six hours of manual debugging may be technically successful but operationally unacceptable if the business requires a one-hour RTO.
## Define the Recovery Contract Before Writing Commands
Before designing the drill, define what recovery means for each application. This should be a short written contract between IT and the business owner of the service.
For each workload, document:
– Recovery Time Objective, or RTO: how long the service can be unavailable
– Recovery Point Objective, or RPO: how much data loss is acceptable
– Critical user journey: the smallest test that proves the service is useful
– Data stores: PVCs, managed databases, object storage, queues, caches
– External dependencies: identity provider, payment gateway, SMTP, APIs, DNS
– Restore priority: which service comes first during a broad outage
A realistic contract might look like this:
“`text
Application: donor-portal
Namespace: donor-prod
RTO: 2 hours
RPO: 15 minutes
Critical test: user can log in, view donor record, create test pledge, and generate receipt PDF
Data: PostgreSQL managed database, uploads PVC, Redis cache disposable
External dependencies: Entra ID SSO, SMTP relay, S3-compatible object storage
Restore priority: Tier 1
“`
Notice that the critical test is not pod is running. It describes a business function.
## Use an Isolated Recovery Environment
Never run an untested restore directly into production unless the incident demands it. A routine drill should restore into an isolated recovery cluster or at least a separate namespace with strict guardrails.
The safest pattern is a separate non-production cluster with:
– No production DNS records pointed at it
– Separate load balancer or private ingress
– Restricted outbound access where possible
– Separate cloud identity or reduced permissions
– A clear naming convention such as dr-test or recovery-lab
This prevents accidental email blasts, duplicate payment processing, or data writes to production dependencies.
If you cannot maintain a standing recovery cluster, use infrastructure as code to create one on demand. For example, a Terraform workspace or GitOps environment can provision a small temporary cluster, install the ingress controller, install Velero, run the restore, collect evidence, and destroy the environment after the test.
## Install Velero With Volume Snapshot Support
Velero can back up Kubernetes API objects and, depending on the plugin and storage class, persistent volumes. The exact installation varies by cloud provider and storage platform. A simplified installation pattern looks like this:
“`bash
velero install \
–provider aws \
–plugins velero/velero-plugin-for-aws:v1.10.0 \
–bucket company-k8s-backups \
–backup-location-config region=us-east-1 \
–snapshot-location-config region=us-east-1 \
–secret-file ./credentials-velero
“`
For clusters using CSI snapshots, confirm that the required snapshot components and VolumeSnapshotClass objects exist:
“`bash
kubectl get volumesnapshotclass
kubectl get crd | grep volumesnapshot
“`
A typical VolumeSnapshotClass may look like this:
“`yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
name: csi-default-snapclass
labels:
velero.io/csi-volumesnapshot-class: true
driver: ebs.csi.aws.com
deletionPolicy: Retain
“`
The label tells Velero which snapshot class to use. The deletion policy should match your retention requirements. Retain can be safer for recovery but requires cleanup discipline.
## Label Applications for Restore Scope
One mistake teams make is backing up entire clusters without defining application boundaries. During a restore, you should be able to select the exact namespace and related resources needed for a service.
Use consistent labels across resources:
“`yaml
metadata:
labels:
app.kubernetes.io/name: donor-portal
app.kubernetes.io/part-of: fundraising-platform
backup.computerbutler.com/tier: tier-1
“`
Then create backups by namespace, label selector, or both:
“`bash
velero backup create donor-portal-nightly \
–include-namespaces donor-prod \
–selector app.kubernetes.io/part-of=fundraising-platform \
–snapshot-volumes \
–ttl 720h
“`
After the backup completes, inspect it rather than assuming success:
“`bash
velero backup describe donor-portal-nightly –details
velero backup logs donor-portal-nightly
“`
Look specifically for skipped persistent volume claims, snapshot errors, excluded resources, and plugin warnings.
## Capture Cluster Dependencies That Backups Often Miss
A namespace backup may not include everything an application needs. Kubernetes clusters commonly store important dependencies outside the application namespace.
Examples include:
– ClusterIssuer objects used by cert-manager
– IngressClass configuration
– StorageClass definitions
– CustomResourceDefinitions and their controllers
– ExternalSecrets or SealedSecrets controllers
– NetworkPolicies in shared namespaces
– Service mesh configuration
– Shared image pull secrets
– DNS automation such as external-dns
For a restore drill, create a dependency checklist. Some items should be restored from backup, while others should be installed by the platform baseline before application restore.
A practical split is:
– Platform baseline: ingress controller, CNI, CSI driver, cert-manager, external secrets operator, monitoring agents
– Application restore: namespace, Deployments, StatefulSets, Services, ConfigMaps, Secrets, PVCs, Ingress, app-specific CRDs
This avoids restoring old cluster-level infrastructure into a newer cluster where it may not belong.
## Build a Repeatable Restore Runbook
A restore drill should not be a hero exercise. If only your senior Kubernetes engineer can perform it, the organization has a people dependency problem.
Create a runbook with exact commands, expected outputs, decision points, and rollback steps. Store it in version control alongside infrastructure documentation.
A simple restore sequence might look like this:
“`bash
# 1. Confirm recovery cluster context
kubectl config current-context
# 2. Confirm Velero can see backups
velero backup get
# 3. Create a restore from the selected backup
velero restore create donor-portal-dr-2026-08 \
–from-backup donor-portal-nightly \
–namespace-mappings donor-prod:donor-dr
# 4. Watch restore progress
velero restore get
velero restore describe donor-portal-dr-2026-08 –details
# 5. Inspect restored objects
kubectl get all -n donor-dr
kubectl get pvc -n donor-dr
kubectl get ingress -n donor-dr
“`
Namespace mapping is useful for drills because it lets you restore production objects into a test namespace. However, it can expose assumptions in application configuration. If environment variables, hostnames, certificates, or hardcoded namespace references assume donor-prod, document and fix them.
## Validate Persistent Volumes, Not Just Claims
A restored PVC can bind successfully while the application data inside it is missing, stale, or incompatible. Always validate the actual data.
For a simple application volume, you can launch a temporary inspection pod:
“`yaml
apiVersion: v1
kind: Pod
metadata:
name: pvc-inspector
namespace: donor-dr
spec:
restartPolicy: Never
containers:
– name: shell
image: alpine:3.20
command:
– sh
– -c
– sleep 3600
volumeMounts:
– name: data
mountPath: /data
volumes:
– name: data
persistentVolumeClaim:
claimName: uploads
“`
Apply it and inspect the mounted data:
“`bash
kubectl apply -f pvc-inspector.yaml
kubectl exec -n donor-dr pvc-inspector — ls -lah /data
kubectl exec -n donor-dr pvc-inspector — find /data -maxdepth 2 -type f | head
“`
For databases running inside Kubernetes, use application-level consistency checks. Filesystem snapshots are not always sufficient for transactional databases unless the backup process coordinates quiescing, crash consistency, or database-native dumps.
For PostgreSQL, a validation command might be:
“`bash
kubectl exec -n donor-dr deploy/donor-postgres — pg_isready
kubectl exec -n donor-dr deploy/donor-postgres — psql -U appuser -d donor -c ‘select count(*) from pledges;’
“`
If your production database is a managed service outside Kubernetes, your restore drill must include that platform too. A Kubernetes restore that points to the live production database is not a disaster recovery test. It is a risky application redeployment.
## Prevent Restored Workloads From Harming Production
Restored applications can be dangerous if they reconnect to production systems. Common examples include sending real customer emails, processing queued payments twice, writing to production object storage, or registering duplicate webhooks.
Use one or more of these controls during drills:
– Restore into a namespace with default-deny egress NetworkPolicy
– Override secrets with test credentials
– Disable CronJobs until explicitly approved
– Block SMTP and payment provider endpoints at the firewall
– Use separate DNS names such as donor-dr.internal.example.org
– Scale workers to zero until validation begins
You can disable CronJobs after restore with:
“`bash
kubectl get cronjob -n donor-dr
kubectl patch cronjob receipt-sender -n donor-dr -p ‘{spec:{suspend:true}}’
“`
A basic default-deny egress policy looks like this:
“`yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-egress
namespace: donor-dr
spec:
podSelector: {}
policyTypes:
– Egress
“`
Then add only the egress destinations required for the drill, such as the test identity provider or internal package repository.
## Test Ingress, TLS, and DNS Deliberately
Many restore drills stop when pods become ready. Users, however, experience the application through DNS, TLS, ingress, authentication, and browser behavior.
Create a separate recovery hostname and validate it end to end:
“`bash
kubectl get ingress -n donor-dr
curl -Ik https://donor-dr.example.org
curl -s https://donor-dr.example.org/healthz
“`
Check the certificate issuer and expiration:
“`bash
echo | openssl s_client -connect donor-dr.example.org:443 -servername donor-dr.example.org 2>/dev/null | openssl x509 -noout -issuer -subject -dates
“`
If the application uses SSO, verify that redirect URIs, callback URLs, and session cookies work in the recovery hostname. Authentication is one of the most common restore surprises because identity providers often restrict allowed redirect URLs.
## Add Functional Tests to the Drill
A good restore drill ends with a business-level test. This does not have to be a full QA suite, but it should prove the restored service performs its core function.
For a web application, that may include:
– Load login page
– Authenticate with test user
– Read an existing record
– Create a test transaction
– Upload and retrieve a file
– Generate a report
– Confirm background worker processes a job
You can automate part of this with a lightweight smoke test container:
“`yaml
apiVersion: batch/v1
kind: Job
metadata:
name: donor-portal-smoke-test
namespace: donor-dr
spec:
template:
spec:
restartPolicy: Never
containers:
– name: smoke
image: curlimages/curl:8.9.1
command:
– sh
– -c
– |
curl -fsS https://donor-dr.example.org/healthz
curl -fsS https://donor-dr.example.org/api/version
“`
For deeper validation, use Playwright, Cypress, k6, Postman CLI, or application-specific scripts. The important point is to make success measurable.
## Measure RTO and RPO During the Drill
If you do not time the drill, you have not tested the RTO.
Record timestamps for:
– Incident declaration or drill start
– Recovery cluster available
– Backup selected
– Restore started
– Restore completed
– Application pods ready
– Data validation completed
– User-facing smoke test passed
– Business owner signoff
For RPO, identify the newest recovered transaction or data record. If the backup schedule promises a 15-minute RPO but the newest restored record is four hours old, the backup system may be running on the wrong schedule or failing silently.
A simple drill record can be stored as Markdown:
“`text
Drill: donor-portal quarterly restore
Date: 2026-08-20
Backup used: donor-portal-nightly-20260820-010000
Restore start: 09:05
Pods ready: 09:31
Smoke test passed: 09:44
Business validation passed: 10:12
Measured RTO: 67 minutes
Measured RPO: 11 minutes
Issues: SSO redirect URI missing for recovery hostname
Owner: IT operations
“`
The issues section is the most valuable part. Restore drills should produce improvements.
## Automate What Is Safe, Keep Human Approval Where Needed
The goal is not to remove humans from disaster recovery. The goal is to remove guesswork.
Good automation candidates include:
– Creating the recovery namespace
– Applying network guardrails
– Starting the Velero restore
– Waiting for pods and PVCs
– Running smoke tests
– Collecting logs and events
– Producing a drill report
Human approval should remain for actions that can affect production data, external communications, DNS cutover, or customer-facing declarations.
A simple shell wrapper can reduce operator error:
“`bash
#!/usr/bin/env bash
set -euo pipefail
APP=donor-portal
SOURCE_NS=donor-prod
TARGET_NS=donor-dr
BACKUP=$1
RESTORE=${APP}-restore-$(date +%Y%m%d-%H%M)
kubectl create namespace ${TARGET_NS} –dry-run=client -o yaml | kubectl apply -f –
kubectl apply -f networkpolicy-default-deny-egress.yaml
velero restore create ${RESTORE} \
–from-backup ${BACKUP} \
–namespace-mappings ${SOURCE_NS}:${TARGET_NS}
velero restore describe ${RESTORE} –details
kubectl rollout status deploy -n ${TARGET_NS} –timeout=15m
kubectl get pods,pvc,ingress -n ${TARGET_NS}
“`
This is not a complete DR platform, but it provides repeatability.
## Common Failure Modes to Watch For
Kubernetes restore drills tend to uncover the same classes of problems.
### Missing or unusable persistent volume snapshots
The backup completed, but the PVC was skipped or restored empty. Verify snapshot support for each storage class and test data inside the volume.
### Secrets restored but no longer valid
Certificates, API keys, or database passwords may have rotated since the backup. Decide whether secrets should be restored from backup, recreated from a secrets manager, or injected by an external secrets operator.
### Cluster-scoped resources not included
Applications that depend on CRDs, ClusterRoles, admission policies, or ingress classes may fail in a clean recovery cluster. Define the platform baseline separately.
### Hardcoded production hostnames
Applications may contain environment variables, callback URLs, or config files that assume production DNS. Recovery environments need planned overrides.
### Restored workers process real queues
Background workers may connect to production message brokers and process live jobs. Disable workers until dependencies are isolated.
### Backup retention does not match compliance needs
Operational backups for quick recovery are not the same as long-term retention for legal, compliance, or audit requirements. Keep those policies distinct.
## How Often Should You Run Restore Drills?
For tier-one services, quarterly is a reasonable minimum. Highly regulated or revenue-critical environments may need monthly drills or automated restore validation after major platform changes.
Also run a drill after:
– Migrating storage classes or CSI drivers
– Changing backup tools or cloud accounts
– Upgrading Kubernetes across major versions
– Moving ingress controllers
– Refactoring namespaces or Helm releases
– Changing identity providers or secrets management
– Onboarding a new critical application
A restore drill is especially important before you need it. During an incident, the team is already dealing with pressure, incomplete information, and business impact.
## Practical Summary and Key Takeaways
Kubernetes backup success is not the same as business recovery. A real restore drill proves that the application, data, configuration, secrets, ingress, DNS, and external dependencies can be recovered in a controlled and measurable way.
Start by defining RTO, RPO, and the critical user journey for each application. Restore into an isolated environment, validate persistent data directly, prevent restored workloads from touching production systems, and run functional smoke tests that reflect how the business actually uses the service. Measure the drill, document the gaps, and improve the runbook after every exercise.
Key takeaways:
– Treat restores as proof, not routine backup administration.
– Validate application behavior, not only Kubernetes object creation.
– Test PVC contents and database consistency, not just PVC binding.
– Use isolated recovery environments with egress controls and test credentials.
– Document exact commands, owners, timestamps, and outcomes.
– Automate repeatable steps while keeping risky production-impacting actions under human approval.
– Run drills regularly, especially after platform, storage, ingress, or identity changes.
For organizations that rely on Kubernetes but do not have time to build a full internal disaster recovery practice, a managed IT partner can help design the runbooks, automate the safe steps, and facilitate restore exercises. The result is more than a backup report. It is confidence that when something breaks, the business knows how to recover.
