Production readiness and high availability

Use this guide to turn a working Koldan installation into a production service that can tolerate routine maintenance and the loss of a node or availability zone. It defines a recommended operating baseline; it does not replace load testing or the availability designs of PostgreSQL, Temporal, object storage, identity, search, and coordination services.

Define the availability contract

Before choosing replica counts, record:

• the service-level objective and the HTTP, gRPC, and background-processing paths it covers;
• the recovery point objective (RPO) and recovery time objective (RTO), with an owner for each persistent dependency;
• whether the design must tolerate a pod, node, availability-zone, cluster, or region failure;
• the reduced-capacity envelope that must remain after the largest required failure;
• maintenance windows, maximum drain time, and the authority to stop incoming work;
• the backup, restore, failover, and failback procedures that prove the objectives.

Replica counts protect only stateless compute. They do not make a single database, single persistent volume, single load balancer, or single failure domain highly available.

Use a production baseline

The following counts are practical starting recommendations, not universal sizing guarantees. Increase them when load tests, failure-domain count, or downstream capacity require it.

The common chart values remain installation scaffolding. The medium-cluster capacity profile is the supported multi-node starting point, but its fixed replica and resource values must still be qualified against the site's failure domains, dependencies, and capacity contract.

Provide failure-domain capacity

Use at least three schedulable worker nodes distributed across the availability zones in scope. For zone-failure tolerance, keep enough unallocated CPU, memory, ephemeral storage, volume-attachment capacity, IP addresses, and provider quota in the surviving zones to place the required replicas.

Calculate the post-failure envelope, not only normal utilization:

surviving requested capacity
  >= required Ready replicas after failure
     × tested per-pod requests
     + rollout or rescheduling headroom

Ensure the cluster autoscaler can add nodes in every intended zone, but do not depend on scale-up time to meet an immediate availability objective. Reserve scarce GPU or other accelerator capacity in more than one failure domain when an enabled workload requires it.

Set CPU, memory, and ephemeral-storage requests for every enabled workload. Requests allow the scheduler to make a credible failure-domain placement decision and are required for CPU- or memory-utilization HPAs. Set limits from measured peak behavior; an undersized memory or ephemeral-storage limit can convert load into restarts or eviction.

Spread Koldan replicas

The current chart exposes nodeSelector, tolerations, and arbitrary affinity for web, grpc, each engine instance, and each worker pool. It does not expose topologySpreadConstraints for these Koldan Deployments.

Use required pod anti-affinity for host separation and preferred anti-affinity for zone distribution. The following is a supported HTTP example because it uses the chart's web.affinity field and stable workload label:

web:
  replicaCount: 2
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              dixilang.com/submodule: web
              app.kubernetes.io/instance: <release-name>
          topologyKey: kubernetes.io/hostname
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchLabels:
                dixilang.com/submodule: web
                app.kubernetes.io/instance: <release-name>
            topologyKey: topology.kubernetes.io/zone

Apply the same pattern to grpc.affinity with dixilang.com/submodule: grpc. For a worker pool, set workerPools.<pool>.affinity and select dixilang.com/worker-pool: <pool>. For an engine instance, select its dixilang.com/engine-type and dixilang.com/engine-id labels. Include app.kubernetes.io/instance: <release-name> in every selector so one release does not affect another release in the same namespace.

The chart's shared workflowsWorker.affinity default already prefers worker pods on different hosts. A non-empty workerPools.<pool>.affinity replaces that shared default for the pool, so include every required constraint in the pool-specific value.

Preferred zone anti-affinity improves distribution but is not a strict skew guarantee. If policy requires a bounded zone skew, inject a reviewed topology-spread constraint through an approved admission policy or manage a downstream chart customization. That mechanism is outside the current Koldan chart and must be revalidated on every upgrade.

Protect voluntary disruptions

The current Koldan templates do not create PodDisruptionBudgets (PDBs) for the HTTP API, gRPC API, engines, or worker pools, and there is no Koldan Helm value that enables them. Manage PDBs for these Deployments through a separately versioned platform policy or manifest set.

For two replicas, a common starting policy is maxUnavailable: 1. Select one workload narrowly; for example:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: koldan-web
  namespace: <namespace>
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      dixilang.com/submodule: web
      app.kubernetes.io/instance: <release-name>

Create separate PDBs for gRPC, each worker pool, and each required engine instance. Confirm selectors against the rendered release before applying them. A PDB limits voluntary evictions; it does not prevent hardware failure, zone loss, pod crashes, or force deletion. Never set a budget that makes node maintenance impossible.

Several bundled dependencies do expose their own PDB controls. For the pinned charts these include zookeeper.pdb, keycloak.pdb, minio.podDisruptionBudget, and the per-component Temporal podDisruptionBudget maps. Configure and render each dependency independently.

Make persistent storage recoverable

Select storage classes that support the required zones, reclaim behavior, encryption, snapshots, IOPS, throughput, and online expansion. Set alerts before 70% usage and validate the expansion procedure before relying on it. A replicated storage class does not replace an application-consistent backup.

Use measured data plus recovery and maintenance headroom:

requested capacity
  >= restored baseline
     + growth through the expansion-response window
     + WAL, compaction, temporary, and upgrade overhead
     + at least 30% free-space headroom

Snapshot and restore tests must prove that volumes can be recreated in a surviving zone. Verify allowVolumeExpansion, volume binding mode, snapshot class, attachment limits, and recovery permissions for the selected storage class.

Operate PostgreSQL as an HA service

For production, prefer an external managed PostgreSQL s ervice or an operator-managed cluster with automated failover, continuous backup, and tested recovery. CloudNativePG is one example of an independent Kubernetes PostgreSQL operator; its installation, support, upgrades, backup storage, and failover policies remain the platform team's responsibility and are not installed or managed by the Koldan chart.

Use at least three database instances across failure domains when the database platform and RPO design support that topology. Configure Koldan to the writable/failover endpoint, not to a fixed pod or read-only service. Choose synchronous-replication policy deliberately: it can reduce data loss but can also stop writes when the required quorum or zone is unavailable.

The Koldan database requires pgvector. Install a PostgreSQL-major-compatible pgvector package in every image or server that can become primary, create the vector extension in the Koldan database, and verify its version after failover and restore. See PostgreSQL.

Calculate the connection ceiling with HPA maximums and failure behavior:

Koldan pool ceiling = sum(maximum replicas × maximum Hikari pool size)
database budget     = Koldan ceiling
                    + Temporal and identity pools
                    + migrations, monitoring, backup, and administration
                    + failover headroom

A connection pooler such as PgBouncer can protect the database from connection churn, but it does not add transaction, lock, CPU, or I/O capacity. Validate the chosen pooling mode with Koldan and migration behavior before production use. Monitor connection use, replica lag, WAL and disk growth, failover time, backup age, and restore tests.

Operate Temporal for production

Prefer a separately operated or managed Temporal service when the organization already has a supported production topology. If the pinned Temporal chart dependency is used, its server.replicaCount default is one and is not HA. A production starting point is three replicas for each enabled server role, explicit PDBs, resource requests, host/zone spreading, and an HA SQL persistence service for both the default and visibility stores.

The pinned dependency supports component overrides under:

• temporal.server.frontend
• temporal.server.history
• temporal.server.matching
• temporal.server.worker

Each component accepts replicaCount, resources, affinity, topologySpreadConstraints, and podDisruptionBudget. Set component replicaCount explicitly when defining its PDB, render all four Deployments and PDBs, and verify quorum-independent Temporal membership before production traffic. Do not change temporal.server.config.persistence.numHistoryShards after the initial Temporal deployment.

Keep the Koldan worker pools at two or more replicas when background-processing availability is required. Their replica and concurrency changes affect all queues assigned to a pool; use the mappings and capacity procedure in Temporal. Alert on poller loss, backlog, schedule-to-start latency, persistence errors, and frontend availability.

Operate ZooKeeper, search, identity, and S3

ZooKeeper

The packaged ZooKeeper dependency defaults to one member. For production, use at least three voting members across hosts and preferably zones:

zookeeper:
  install: true
  replicaCount: 3
  podAntiAffinityPreset: hard
  pdb:
    create: true
    maxUnavailable: 1
  persistence:
    size: 8Gi
    storageClass: <expandable-zone-aware-class>

The pinned ZooKeeper subchart also accepts topologySpreadConstraints. Render its labels before adding a selector. Confirm that all three members are Ready, one leader is elected, followers are synchronized, and Koldan remains connected while one member is stopped. Do not voluntarily disrupt a majority.

Search

The Koldan values reduce the bundled Elasticsearch dependency to one replica and an 8 GiB claim. For production, use an appropriately supported managed or separately operated HA search cluster, or validate a three-member packaged topology with hard host anti-affinity, zone placement, shard replicas, a one-member disruption limit, snapshots, and recovery tests. Search replicas and shard layout must be sized together; adding pods without replica shards does not protect indexed data.

Identity

Run at least two identity-provider replicas behind an HA ingress and use an HA database. For bundled Keycloak, the pinned subchart exposes keycloak.replicaCount, podAntiAffinityPreset, topologySpreadConstraints, and pdb. Protect signing keys, realm/client configuration, federation settings, and database state. Test login, token refresh, logout, and signing-key recovery after a pod, node, and database failover.

S3-compatible storage

The packaged MinIO override is mode: standalone with one replica and is not HA. Prefer a multi-zone durable service. If operating the pinned MinIO dependency, use its distributed mode and quorum rules, persistent volumes across failure domains, anti-affinity or topology spread, a disruption budget, encryption, versioning or replication as policy requires, and tested node and zone recovery. The pinned chart documents four replicas as the minimum distributed layout; capacity and failure tolerance still depend on the number and placement of drives.

Do not use bucket lifecycle expiration as a backup. Keep PostgreSQL records, object versions, and Temporal persistence recoverable to a mutually consistent point.

Make ingress and DNS redundant

• Run at least two ingress or Route-controller replicas on separate nodes and zones.
• Use load balancers that health-check only Ready endpoints and support the selected HTTP, gRPC, TLS, timeout, and connection-draining behavior.
• Keep HTTP and gRPC certificates, private CA trust, and DNS names valid in every failure path.
• Use a DNS TTL that permits planned failover without creating excessive query load, and rehearse endpoint changes before an incident.
• Retain sufficient load-balancer, node-port, IP-address, and security-policy quota in each surviving zone.
• Confirm long-lived gRPC sessions reconnect within the stated objective; replica count cannot preserve an individual connection to a failed pod.

Control disruption and rollout

Koldan exposes the Deployment strategy for HTTP, gRPC, discovery, and worker pools. The supported capacity profiles use maxSurge: 0 and maxUnavailable: 1 to avoid scheduling an additional large pod during rollout. Engines still use the Kubernetes default rolling strategy, and the chart does not expose general minReadySeconds, lifecycle hooks, or Koldan application PDBs.

Before a node drain or rollout:

1. Confirm every affected workload has more than one Ready replica and sufficient capacity after one replica is removed.
2. Confirm external PDBs and dependency PDBs select the rendered pods and permit the intended maintenance.
3. Check Temporal queue backlog, database and provider headroom, storage attachment health, and ingress endpoints.
4. Keep the worker-pool termination grace at its packaged 45-second baseline unless shutdown measurements require more time. Long transcription activities heartbeat and retry rather than holding a terminating pod for their full runtime. The gRPC and engine templates default to 330 seconds because their connection-drain behavior differs; verify real client and workload behavior rather than assuming any duration guarantees completion.
5. Drain or roll one failure domain at a time. Wait for replacement readiness and dependency recovery before continuing.
6. Watch error rate, latency, reconnects, Temporal retries/backlog, database connections, and S3 failures until the full fleet is stable.

Do not combine a chart upgrade, database failover, certificate rotation, and node maintenance in one change window unless the combined recovery path has been rehearsed.

Monitor and recover

At minimum, alert on:

• desired versus Ready replicas, unschedulable pods, restart rate, eviction, and zone imbalance;
• HTTP/gRPC availability, latency, error rate, and load-balancer healthy-target count;
• Temporal frontend errors, task-queue pollers, backlog, schedule-to-start latency, and persistence health;
• PostgreSQL availability, connection saturation, replication lag, pgvector availability, WAL/disk growth, and backup age;
• S3 request errors, latency, capacity, replication, and missing objects;
• ZooKeeper quorum/member health, search cluster health, and identity login/token failures;
• PVC usage/inodes, snapshot failures, certificate expiry, and secret/license rotation age.

Back up PostgreSQL, both Temporal persistence stores, S3 objects, identity data, ZooKeeper/search state or their tested rebuild inputs, update-data PVC contents, chart artifacts, configuration, Secrets, certificates, and license material. Run isolated restore rehearsals and compare achieved RPO/RTO with the declared targets. See Backup and restore.

Prove failure tolerance

Run these tests in a production-like environment before go-live and after material topology changes:

Abort a test if it approaches data loss, quorum loss, or unplanned service impact. Preserve events, metrics, logs, and timing so the result can be audited.