## Why Kubernetes restore drills matter more than backup jobs
Many organizations can point to a dashboard that says their Kubernetes backups completed successfully last night. Far fewer can prove that those backups can rebuild a production namespace, reconnect persistent volumes, restore secrets, reapply ingress, and bring an application back online within the recovery time objective the business expects.
That gap is where outages become expensive. Kubernetes adds flexibility, but it also adds moving parts: API objects, custom resources, persistent volumes, storage classes, external load balancers, DNS records, certificates, container images, secrets, and identity integrations. A green backup status does not necessarily mean all of those pieces can be recovered together.
This article walks through a practical approach to Kubernetes backup drills for SMBs, nonprofits, and enterprise IT teams running production workloads on Kubernetes. The focus is not simply installing a backup tool. The goal is to build a repeatable restore exercise that proves recovery works, produces evidence for leadership and auditors, and exposes engineering gaps before a ransomware event, cloud outage, accidental deletion, or failed upgrade does.
The examples use Velero because it is widely adopted and works across many Kubernetes environments, but the method applies equally to other backup platforms. The important part is the discipline: define the recovery objective, restore into an isolated target, validate the application, document the result, and improve the runbook.
## Start with the recovery question the business actually cares about
Before writing a single command, clarify what the restore drill must prove. A useful test is not, Can we restore Kubernetes objects? It is, Can we recover the business service to an agreed point in time and prove users can work again?
For each application, document these items:
– Recovery time objective, or RTO: how long the service may be unavailable.
– Recovery point objective, or RPO: how much data loss is acceptable.
– Critical namespaces and dependencies.
– Persistent volumes and storage classes.
– External systems such as databases, object storage, identity providers, DNS, payment gateways, and SMTP relays.
– Required secrets, certificates, and image pull credentials.
– Validation tests that prove the application works.
For example, a donor management system for a nonprofit might tolerate a four-hour RTO but only a fifteen-minute RPO during a fundraising event. A reporting dashboard may tolerate a longer RPO because data can be regenerated. A customer portal may require both fast recovery and strong evidence that no data was restored into the wrong tenant or environment.
This context determines the backup design. If the application uses a managed cloud database outside the cluster, Kubernetes backup alone is not enough. If the application stores uploads on persistent volumes, object storage, or both, the restore drill must validate all of them.
## Understand what must be backed up in Kubernetes
A Kubernetes application is usually composed of more than Deployments and Services. A complete backup plan should account for at least four layers.
### Cluster API resources
These include Namespaces, Deployments, StatefulSets, DaemonSets, Services, ConfigMaps, Secrets, Ingress objects, ServiceAccounts, Roles, RoleBindings, NetworkPolicies, PersistentVolumeClaims, and Custom Resources.
Custom Resource Definitions deserve special attention. Many modern platforms rely on CRDs for certificates, service meshes, database operators, external secrets, monitoring, and ingress controllers. Restoring a custom resource without the corresponding operator or CRD version can fail or produce a partially functioning application.
### Persistent data
PersistentVolumeClaims may be backed by cloud block disks, SAN volumes, distributed storage, or CSI snapshots. Some workloads, such as PostgreSQL, MySQL, Elasticsearch, and message queues, may need application-aware backups rather than crash-consistent volume snapshots. A snapshot can be useful, but it is not automatically a clean database backup.
### Cluster-adjacent services
Load balancers, DNS records, certificates, container registries, identity providers, cloud IAM roles, storage buckets, and managed databases often live outside Kubernetes. A restore drill that ignores them may look successful in the cluster while users still cannot reach the service.
### Infrastructure and configuration source of truth
If you use Terraform, Pulumi, Helm, Kustomize, or GitOps tooling such as Argo CD or Flux, your restore process should use those sources of truth. Backups should not become the only place where critical configuration exists. A healthy recovery model combines declarative infrastructure with point-in-time data recovery.
## Build a restore architecture, not just a backup schedule
A reliable Kubernetes backup strategy usually includes the following elements:
– Object backups stored outside the cluster.
– Volume snapshots or file-level backups for persistent data.
– Immutable or locked backup storage where supported.
– Separate credentials for backup operations.
– Restore testing into a non-production cluster or isolated namespace.
– Git-based application manifests and infrastructure definitions.
– Monitoring and alerting for failed backups and stale snapshots.
– Written runbooks with expected RTO and RPO.
A common mistake is storing backups in the same cloud account, region, or administrative boundary as the cluster. If an attacker compromises cluster-admin credentials and the backup storage uses the same broad credentials, the backups may be deleted before anyone starts recovery. Use separate identities, least privilege, object lock where available, and lifecycle policies that retain enough restore points without exposing unnecessary cost.
## Example: install Velero with object storage and CSI support
The exact installation command depends on your cloud provider and storage location. The pattern is consistent: create a backup storage location, grant Velero limited permissions, install the server components, and enable CSI features if you rely on volume snapshots.
A simplified installation command looks like this:
“`bash
velero install \
–provider aws \
–plugins velero/velero-plugin-for-aws:v1.10.0 \
–bucket company-k8s-backups-prod \
–backup-location-config region=us-east-1 \
–snapshot-location-config region=us-east-1 \
–features=EnableCSI \
–secret-file ./velero-credentials
“`
For Azure, Google Cloud, and S3-compatible platforms, the provider, plugin, and storage configuration change, but the recovery principles do not. In production, avoid using long-lived administrator credentials. Prefer cloud-native workload identity where possible, or tightly scoped service principals with permission only to read and write the required backup objects and snapshots.
After installation, verify the backup storage location:
“`bash
velero backup-location get
velero snapshot-location get
kubectl get pods -n velero
“`
You want the backup location to show as available. If it does not, fix that before scheduling backups. A backup system that cannot reach its repository will quietly become shelfware.
## Label applications so backups are intentional
Do not rely only on broad cluster-wide backups. They have value, especially before upgrades, but application-level recovery is easier when workloads are clearly labeled.
Example namespace and labels:
“`yaml
apiVersion: v1
kind: Namespace
metadata:
name: donor-portal
labels:
backup-tier: gold
business-service: donor-portal
data-classification: confidential
“`
Example backup command for a specific namespace:
“`bash
velero backup create donor-portal-manual-001 \
–include-namespaces donor-portal \
–snapshot-volumes \
–ttl 720h
“`
Check backup progress:
“`bash
velero backup describe donor-portal-manual-001 –details
velero backup logs donor-portal-manual-001
“`
The `–details` output is important. Confirm that the expected persistent volumes were included. If the output says no volumes were snapshotted when you expected three database volumes, stop and investigate.
## Account for quiescing and application-aware backups
Volume snapshots are often crash-consistent, not application-consistent. For stateless web services, that may be fine. For databases, it may not be.
There are several ways to improve consistency:
– Use the database engine’s native backup tools and store dumps in durable object storage.
– Use an operator that supports backup and restore workflows.
– Temporarily quiesce writes before taking a snapshot.
– Use pre-backup and post-backup hooks where appropriate.
Velero supports hooks that run commands inside containers before or after backup. For example, a workload might expose a maintenance command that flushes buffers or pauses writes briefly:
“`yaml
apiVersion: v1
kind: Pod
metadata:
name: app-worker
namespace: donor-portal
annotations:
pre.hook.backup.velero.io/container: worker
pre.hook.backup.velero.io/command: ‘[/bin/sh, -c, /app/bin/pause-writes.sh]’
post.hook.backup.velero.io/container: worker
post.hook.backup.velero.io/command: ‘[/bin/sh, -c, /app/bin/resume-writes.sh]’
“`
Do not add hooks blindly. Test them carefully. A hook that hangs can cause backup problems; a hook that pauses writes too long can create an outage. For many databases, native tools such as `pg_dump`, WAL archiving, MySQL binary logs, or vendor-specific operators are safer than trying to freeze a live volume.
## Design the restore drill environment
A restore drill should not overwrite production. Use one of these patterns:
1. Restore into a separate Kubernetes cluster in the same cloud account.
2. Restore into a separate cluster in a different account or subscription.
3. Restore into an isolated namespace with modified ingress and secrets.
4. Restore selected workloads into a lab cluster for validation.
The best choice depends on risk and cost. For ransomware readiness, restoring into a separate administrative boundary is stronger. For routine monthly drills, a staging cluster may be sufficient.
Make sure the drill environment has compatible Kubernetes versions, storage classes, ingress controllers, CRDs, and operators. A restore can fail if production uses a storage class named `fast-ssd` but the test cluster only has `standard`. You can either create matching storage classes or use restore resource modifiers to map resources to the target environment.
Before the drill, capture baseline information:
“`bash
kubectl get ns donor-portal –show-labels
kubectl get pvc -n donor-portal
kubectl get ingress -n donor-portal
kubectl get deploy,statefulset,svc -n donor-portal
velero backup get
“`
This gives you something to compare against after the restore.
## Run a controlled restore into a test namespace
If restoring into the same cluster for a limited drill, use a namespace mapping. This avoids colliding with production resources.
“`bash
velero restore create donor-portal-restore-test-001 \
–from-backup donor-portal-manual-001 \
–namespace-mappings donor-portal:donor-portal-drill
“`
Watch the restore:
“`bash
velero restore describe donor-portal-restore-test-001 –details
velero restore logs donor-portal-restore-test-001
kubectl get all -n donor-portal-drill
kubectl get pvc -n donor-portal-drill
“`
In many environments, you will need to prevent the restored application from sending real emails, processing payments, or connecting to production third-party services. That means the drill runbook should include safe configuration overrides.
One approach is to patch selected ConfigMaps and Secrets after restore:
“`bash
kubectl patch configmap app-settings -n donor-portal-drill \
–type merge \
-p ‘{“data”:{“EMAIL_MODE”:”sandbox”,”PAYMENT_MODE”:”test”}}’
“`
If you use GitOps, a cleaner approach is to maintain a recovery overlay. The backup restores data and stateful objects, while GitOps applies environment-specific configuration.
## Validate the restored service like a user would
A restore drill is not complete when pods are running. Kubernetes can report healthy pods while the application is unusable. Build validation steps into the runbook.
Useful checks include:
– Pods are ready and not restarting.
– PVCs are bound to expected storage classes.
– Migrations or startup jobs completed successfully.
– Application logs show successful database connections.
– Internal service discovery works.
– Ingress responds on the drill hostname.
– Authentication works with test identity configuration.
– Key user workflows succeed.
– Data exists and matches the expected restore point.
Example command checks:
“`bash
kubectl get pods -n donor-portal-drill
kubectl get events -n donor-portal-drill –sort-by=.lastTimestamp
kubectl logs -n donor-portal-drill deploy/donor-web –tail=100
kubectl exec -n donor-portal-drill deploy/donor-web — /app/bin/healthcheck
“`
For HTTP validation, use a drill hostname rather than production DNS:
“`bash
curl -I https://donor-portal-drill.example.org/health
curl https://donor-portal-drill.example.org/version
“`
For data validation, define safe queries or application-level checks. For example, verify that a known test record exists, that the record count is within an expected range, and that the latest transaction timestamp aligns with the RPO.
## Measure RTO and RPO with evidence
Every drill should produce a short evidence package. This is useful for internal improvement, cyber insurance questionnaires, compliance audits, and executive reporting.
At minimum, record:
– Application name and business owner.
– Backup name and timestamp.
– Restore target environment.
– Start and end time of the drill.
– Actual RTO achieved.
– Actual RPO observed.
– Validation steps and results.
– Screenshots or command output showing success.
– Issues found and remediation tasks.
– Names of participants.
You can capture useful details with commands like:
“`bash
velero backup describe donor-portal-manual-001 –details > evidence-backup.txt
velero restore describe donor-portal-restore-test-001 –details > evidence-restore.txt
kubectl get pods,pvc,ingress -n donor-portal-drill -o wide > evidence-k8s.txt
“`
Store the evidence somewhere separate from the cluster, such as a ticketing system, documentation platform, or compliance evidence repository. The point is not bureaucracy. The point is being able to prove, six months later, that the organization tested recovery and fixed what it found.
## Common failure modes uncovered by restore drills
Restore drills often reveal problems that normal monitoring misses.
### Missing CRDs or operators
A backup may contain custom resources, but the target cluster may not have the CRDs installed. Install platform components such as cert-manager, ingress controllers, external-secrets, monitoring operators, and database operators before restoring dependent applications.
### Storage class mismatch
PVCs may remain pending because the target cluster lacks the source storage class. Standardize storage class names where possible, or document mappings for each recovery environment.
### Secrets restored but no longer valid
Secrets may reference expired certificates, rotated passwords, disabled service accounts, or cloud identities unavailable in the drill environment. Backup systems can restore a Secret object, but they cannot guarantee the credential still works.
### External dependencies not included
The cluster may restore successfully while the application fails because DNS, firewall rules, WAF policies, cloud IAM, or managed databases were not included in the recovery plan.
### Backups not immutable
If backup administrators, cluster administrators, and cloud administrators all share broad rights, an attacker can delete primary workloads and backups. Immutability, retention locks, separate accounts, and restricted credentials matter.
### Git drift
If production has manual hotfixes not represented in Git, the restored environment may behave differently. Restore drills are a practical way to detect configuration drift and reinforce GitOps discipline.
## How often should Kubernetes restore drills run?
The right cadence depends on business criticality. A practical baseline is:
– Quarterly restore drills for critical production applications.
– Monthly automated restore checks for high-risk or high-value systems.
– Before and after major Kubernetes upgrades.
– After changing storage classes, backup tools, ingress controllers, or identity systems.
– After onboarding a new production workload.
Not every test needs to be a full disaster recovery exercise. Mix lightweight automated restores with deeper tabletop and technical drills. For example, run a weekly job that restores a small representative namespace into a disposable cluster, and run a quarterly application-owner drill for the most important workloads.
## Automating a lightweight restore verification
For mature teams, a restore drill can become part of operations automation. The following shell outline is intentionally simple, but it shows the workflow:
“`bash
#!/usr/bin/env bash
set -euo pipefail
APP_NS=donor-portal
DRILL_NS=donor-portal-drill
BACKUP_NAME=donor-portal-auto-$(date +%Y%m%d%H%M)
RESTORE_NAME=${BACKUP_NAME}-restore
velero backup create ${BACKUP_NAME} \
–include-namespaces ${APP_NS} \
–snapshot-volumes \
–wait
kubectl delete ns ${DRILL_NS} –ignore-not-found=true
velero restore create ${RESTORE_NAME} \
–from-backup ${BACKUP_NAME} \
–namespace-mappings ${APP_NS}:${DRILL_NS} \
–wait
kubectl wait –for=condition=available \
deploy/donor-web -n ${DRILL_NS} –timeout=300s
kubectl exec -n ${DRILL_NS} deploy/donor-web — /app/bin/healthcheck
velero restore describe ${RESTORE_NAME} –details
“`
In production, add guardrails: prevent accidental deletion of real namespaces, use sandbox integrations, collect logs, notify the operations team, and clean up resources after the test. For regulated environments, route results to your ticketing or governance system.
## Security considerations for backup platforms
A Kubernetes backup system is highly sensitive. It may access Secrets, persistent data, and cloud snapshots. Treat it as critical infrastructure.
Recommended controls include:
– Use least-privilege RBAC for backup service accounts.
– Restrict who can create restores into production namespaces.
– Encrypt backup repositories and snapshots.
– Enable object lock or immutability where supported.
– Separate backup administration from daily cluster administration.
– Monitor delete operations against backup buckets and snapshots.
– Rotate credentials and prefer workload identity over static keys.
– Include backup systems in incident response plans.
Also consider privacy requirements. If backups contain regulated data, restoring into a test environment may create a compliance issue unless that environment has equivalent controls. Masking, access restrictions, and written procedures matter.
## Practical summary and key takeaways
Kubernetes backup success is not measured by completed jobs. It is measured by restored applications, validated data, documented evidence, and a recovery process the team can execute under pressure.
Key takeaways:
– Define RTO and RPO per application before designing the backup job.
– Back up API resources, persistent data, custom resources, and external dependencies.
– Use Velero or a comparable tool, but do not confuse tool installation with recovery readiness.
– Test restores into an isolated environment, not over production.
– Validate the restored service with real application checks, not just pod status.
– Capture evidence for leadership, auditors, cyber insurance, and continuous improvement.
– Protect backup storage with least privilege, encryption, immutability, and administrative separation.
– Repeat drills regularly and after major infrastructure changes.
For organizations relying on Kubernetes for customer portals, internal applications, analytics platforms, or nonprofit operations, restore drills are one of the most valuable engineering habits you can build. They turn backup from an assumption into a tested capability.
