Backup, restore, and disaster recovery

A recoverable Koldan service requires coordinated protection of PostgreSQL, Temporal persistence, S3-compatible object storage, identity, and deployment inputs. Replication and PersistentVolumes improve availability, but neither replaces an independent backup. Prove the recovery design through isolated restores and scheduled site-promotion exercises.

Define the recovery contract

Before production use, assign a recovery owner and define recovery point objectives (RPOs) and recovery time objectives (RTOs) for the complete service and for each persistent dependency. Record which data loss and unavailable features the organization will accept if the dependencies cannot be restored to the same point.

Maintain a recovery record containing:

• backup frequency, retention, encryption, immutability, and off-cluster or off-account copies;
• the PostgreSQL base-backup, snapshot, and WAL retention needed to meet the database RPO;
• the method used to coordinate PostgreSQL, Temporal persistence, and object-storage recovery points;
• replication topology, measured lag, last durable checkpoint, and manual promotion authority;
• identity-provider, certificate, secret, encryption-key, and license recovery owners;
• exact chart, image, PostgreSQL, pgvector, Temporal, and other dependency versions needed by each recovery set;
• key-management and break-glass access that remains available during a cluster, account, or regional outage;
• fencing, DNS, load-balancer, validation, promotion, and failback procedures;
• the results and duration of the latest restore and disaster-recovery exercise.

Do not describe an RPO only in terms of backup-job frequency. WAL archive gaps, asynchronous replication lag, incomplete object replication, or unavailable encryption keys can produce an older usable recovery point.

Protect every required state store

Helm release metadata can contain rendered Secrets and populated values. Restrict access to it and to every backup that contains it. Never commit database dumps, exported values, credentials, private keys, license documents, or identity-provider exports to the source repository.

Protect the encryption key independently from PostgreSQL while binding its version to each database recovery set. Restoring the database without the matching koldan.security.encryption-key can leave persisted OIDC credentials and webhook secrets unreadable. Protect the API-key HMAC secret with the same recovery record when existing API keys must survive restoration. A different koldan.api.api-keys.hmac-secret does not corrupt the database, but every stored API-key hash becomes unusable and all clients require replacement keys.

The bundled PostgreSQL dependency exposes postgresql.backup.enabled and postgresql.backup.cronjob.*. If used, monitor the CronJob and move its output to protected storage outside the cluster failure domain. A scheduled dump retained only on a PVC does not provide continuous point-in-time recovery, role and infrastructure recovery, or protection from cluster-wide loss.

Design PostgreSQL protection

Use at least one database-native or provider-managed recovery method as the primary PostgreSQL backup. Combine methods when the RPO, recovery scope, or audit requirements demand it.

Distinguish snapshot consistency

An application-consistent database snapshot is taken through a PostgreSQL-aware or provider-supported process that establishes a recoverable database point. It can protect database transaction consistency while writes continue, but it does not by itself align PostgreSQL with S3 or Temporal.

A crash-consistent volume snapshot represents storage as it might appear after abrupt power loss. PostgreSQL can normally perform crash recovery only when all required volumes were captured atomically and the storage method is supported for the deployed topology. A snapshot of one live data volume while WAL or tablespaces are elsewhere is not a valid backup. Treat crash-consistent snapshots as a recovery method only after repeated restore tests; otherwise use them as a supplement to database-native backups.

Protect WAL for PITR

PITR requires a successful base backup and every WAL segment from that backup through the target time. Store WAL outside the database and cluster failure domain, encrypt it, prevent premature expiration, and alert on archive delay, missing segments, and an RPO-sized gap. Record the database timeline and the intended timestamp or transaction recovery target for each incident.

Replication slots and a healthy standby are not a WAL archive. A standby can receive corruption or deletion, and retained WAL on the primary can disappear with the primary. Verify recovery by creating a new PostgreSQL service from the protected base backup and WAL stream.

Preserve roles, grants, and pgvector

Record database ownership, login and non-login roles, grants, default privileges, required server parameters, and the authentication and TLS configuration needed after complete cluster loss. Logical backup designs must explicitly protect global roles because a database-only dump does not recreate every cluster-level object. Treat exported role definitions and password verifiers as sensitive data.

The Koldan database requires the PostgreSQL vector extension supplied by pgvector. Record the PostgreSQL major version and:

SELECT extname, extversion
FROM pg_extension
WHERE extname = 'vector';

Before restoring a physical backup, install compatible pgvector software on every target database server. For a logical restore, enable the provider-supported extension or install its package before restoring objects that depend on vector. Recreate the extension with an administrative identity if the restore method does not do so; do not grant the Koldan application login superuser privileges. See PostgreSQL.

Design S3-compatible protection

Enable object versioning when the provider and retention policy support it. Versioning makes it possible to recover a previous object after overwrite or deletion, but it is not an independent backup if an administrator, lifecycle rule, compromised identity, or account loss can permanently remove all versions.

For required recovery points:

• replicate objects to a separate site, region, or account and monitor replication failures and lag;
• retain a checkpoint or inventory that identifies which object versions were durable at the recovery site;
• protect destination credentials and encryption keys independently from the primary site;
• decide explicitly whether delete markers, historical versions, metadata, tags, and retention settings replicate;
• test restores of multipart objects, large objects, old versions, and objects encrypted with every active key;
• use object lock or equivalent immutable retention where required and supported, after validating its operational and compliance consequences;
• ensure lifecycle rules cannot expire the only recoverable version before the database and backup retention windows close.

Cross-region or site-to-site replication is normally asynchronous. Its configured status is not evidence that every database-referenced object has arrived. Monitor actual replication backlog and age, and include the last verified durable checkpoint in promotion decisions.

Coordinate PostgreSQL and object-storage recovery

PostgreSQL and S3 do not share a transaction boundary. Do not restore their independently latest copies and assume consistency.

Prefer one of these recovery-point methods:

1. Quiesce new Koldan writes, allow active work to settle, record the time, create a database-consistent recovery point, and protect all object versions required at that point.
2. When writes cannot stop, retain PostgreSQL PITR and versioned object history long enough to select a tested common point. Record database timestamps, backup identifiers, object inventories or versions, and replication checkpoints.
3. For a DR replica, promote only to a point for which both the database and required objects are confirmed durable at the target site.

Use this decision table during restore:

Apply the same principle to Temporal persistence: select Temporal default and visibility stores that belong to the intended service recovery boundary. Do not claim workflow completion from a database or object check alone.

Create a coordinated recovery set

1. Confirm the most recent isolated restore passed and the backup destinations have protected capacity.
2. Record the release revision, chart version, image digests, dependency versions, PostgreSQL timeline, pgvector version, and backup start time.
3. Record object-replication lag, last durable object checkpoint, Temporal health, and database WAL archive health.
4. If the technologies cannot establish an acceptable common point while writes continue, stop admitting new HTTP and gRPC work and allow active work to settle within the maintenance window.
5. Fence or pause Koldan writers through approved workload and traffic controls. Keep PostgreSQL, Temporal, and object storage available until their supported backup procedures complete.
6. Take the database-consistent PostgreSQL backup or provider snapshot. For PITR, confirm the base backup is complete and WAL through the declared boundary is archived.
7. Protect both Temporal persistence stores and their namespace and retention configuration using the supported database procedure.
8. Confirm every required S3 object version is present in the protected copy or beyond the recorded replication checkpoint. Capture a provider inventory or equivalent evidence where available.
9. Export or protect identity-provider and other enabled dependency state through their supported tools.
10. Protect the immutable chart package, values, secret-source references, certificates, CA bundles, license, and key-management references in the approved encrypted store.
11. Resume workloads and traffic, then verify authentication, readiness, new database/object writes, and background processing.
12. Verify backup logs, sizes, integrity checks, encryption metadata, retention, off-site copies, WAL continuity, and replication status. Record the achieved common recovery point and any RPO exception.

Restore after complete PostgreSQL loss

Perform database recovery in an isolated network or namespace. Fence Koldan workloads from the target database so no startup migration, scheduled work, or application write occurs before validation.

1. Select the declared PostgreSQL recovery point and identify the compatible engine, major version, server parameters, encoding/collation requirements, pgvector package, backup, WAL timeline, and encryption keys.
2. Provision a new PostgreSQL service without exposing it to Koldan. Apply the production availability, storage, TLS, auditing, and backup controls before cutover.
3. Restore cluster roles and required global objects through the protected administrative process. Recreate ownership and membership without displaying password verifiers or credentials.
4. Install pgvector on every target server or enable it through the managed provider. Confirm a compatible extension version is available.
5. Restore the physical backup and replay WAL to the approved recovery target, or restore the logical database with ownership, grants, and extensions. Do not mix files from unrelated backups or PostgreSQL major versions.
6. Verify PostgreSQL completed recovery without missing WAL, invalid pages, unexpected read-only state, or extension-loading errors. Confirm the expected database owner, schemas, grants, and vector extension.
7. Run database-native integrity and consistency checks appropriate to the backup method. Compare row counts and representative records with backup evidence where available.
8. Restore S3 and Temporal to the selected common boundary before allowing Koldan to connect.
9. Take a protected post-restore checkpoint. Keep the failed source and original backups unchanged for investigation and another recovery attempt.
10. Allow only the isolated validation deployment to connect. Start the recorded Koldan version and watch for unexpected database changes. If it proposes an unplanned migration, stop it and recheck the chart, image, and recovery point.

Restore and validate the complete service

Restore into an isolated cluster, namespace, or network segment first. Block public traffic, outbound webhooks, scheduled automation, and other external side effects until validation is complete.

1. Declare the common target recovery point and preserve the failed environment and backup evidence where possible.
2. Recreate namespace policies, service accounts, registry access, storage classes, secret-store integration, staging DNS, network controls, and certificate trust.
3. Restore identity-provider state and validate the issuer, clients, signing keys, role mappings, federation dependencies, and time synchronization.
4. Restore PostgreSQL using the complete-loss procedure above, including roles, grants, required parameters, extensions, and pgvector.
5. Restore the Temporal default and visibility stores and the recorded namespace/retention configuration. Confirm Temporal frontend and persistence health before worker workloads start.
6. Restore S3 objects, versions, metadata, bucket policies, and encryption-key access to the coordinated boundary. Read representative versions with the recovery identity.
7. Restore or rebuild search and coordination services through their tested procedures.
8. Restore update-data storage if used, then restore configuration, secret references, certificates, CA bundles, and license material without placing sensitive values in terminal history or unprotected files.
9. Render the recorded Koldan chart and image versions. Keep Koldan replicas stopped or database access fenced until the render and restored dependencies have been reviewed.
10. Start Koldan in a controlled change window. Watch startup, readiness, database activity, Temporal pollers and backlog, object access, authentication, and provider errors.
11. Complete every validation check below. Have a second operator approve the recovery point and cutover evidence.
12. Fence the failed site against application and database writes, then direct the load balancer or DNS to the restored service. DNS change alone is not a write fence.
13. Monitor errors, latency, queue backlog, database locks/connections, object misses, replication, and identity-provider events closely after traffic resumes.

Do not let an unreviewed newer chart start against restored data. Startup can apply persistent changes that make returning to the recorded version unsafe.

Validation checklist

• The intended PostgreSQL timeline and recovery point are recorded; WAL recovery completed without a gap.
• Expected database roles, ownership, grants, schemas, and required extensions exist; vector returns an extension version.
• Sampled database records retrieve the intended S3 object versions, including records created near the recovery boundary.
• S3 encryption keys, policies, lifecycle rules, versioning, and replication status are correct for the recovery site.
• Temporal frontend, namespace, default store, visibility store, expected task-queue pollers, and new background work are healthy.
• All required Deployments become Ready without restart loops or unexpected migrations.
• Staging DNS, TLS chains, HTTP/gRPC routing, and private dependency connections use the intended recovery endpoints.
• A test identity authenticates and receives the expected authorization.
• Representative API reads and writes succeed using non-production validation data.
• Search-dependent and provider-dependent checks succeed without sending unintended production notifications or webhooks.
• Certificates, credentials, encryption keys, and license validity are confirmed without exposing their contents.
• Monitoring, alerting, WAL archiving, replication, and the next scheduled backup operate from the recovered environment.
• The achieved RPO and RTO, validation evidence, exceptions, and cutover approval are recorded.

Prepare a disaster-recovery site

Keep the standby site fenced from production writes until an authorized promotion. Replication reduces recovery time; it also copies accidental deletion, corruption, or malicious changes. Retain immutable, point-in-time backups independently of every replica.

Promote the DR site

1. Declare the incident, promotion authority, target RPO, observed PostgreSQL and S3 replication lag, and last common checkpoint.
2. Stop or fence the primary site's public traffic and all application writers. Revoke or block its database and object-storage write path where practical. Confirm loss of inbound traffic alone cannot leave hidden writers active.
3. Stop replication only through the provider's supported promotion procedure. Record final PostgreSQL replay position/timeline and object-replication checkpoint.
4. Decide whether to promote the latest replicated point or restore an earlier coordinated backup. If S3 is behind PostgreSQL, advance object recovery or choose an earlier database point.
5. Promote PostgreSQL and S3 access at the DR site. Restore Temporal, identity, secrets, certificates, and license as required.
6. Keep Koldan isolated while completing the validation checklist. Confirm the old site cannot regain write access automatically.
7. Shift load-balancer or DNS traffic only after validation and a second-operator approval.
8. Record actual data loss, RPO/RTO achievement, residual replication gaps, and the new authoritative site.

Do not advertise automatic failover unless the complete fencing, dependency promotion, recovery-point selection, and application validation sequence is implemented and repeatedly tested. Koldan documentation does not imply active-active writes or exactly-once cross-site recovery.

Fail back safely

Treat failback as another migration, not as reversal of a DNS record.

1. Keep the repaired former primary fenced and take a new protected backup of the current authoritative site.
2. Rebuild or re-seed PostgreSQL, S3, Temporal persistence, and identity state from the authoritative site using their supported procedures. Do not blindly reverse a stream that can reintroduce stale state.
3. Re-establish replication and wait until measured lag is within the approved cutback boundary.
4. Validate the rebuilt site in isolation, including pgvector, object versions, Temporal queues, identity, certificates, secrets, license, and outbound integrations.
5. Schedule a controlled write pause, record the final common checkpoint, fence the DR site, promote the rebuilt site, and repeat the cutover checklist.
6. Re-establish DR protection in the intended direction and confirm independent backups continue.

Failure decisions

Run isolated restores on a schedule and after material changes to PostgreSQL, pgvector, Temporal, S3, identity, encryption, chart versions, or backup policy. Exercise loss of the entire database service and promotion of the DR site, not only restoration of one table or PVC. Track every gap as a production risk and retest after remediation.