Upgrade and rollback

Upgrade Koldan with the chart, images, release notes, and documentation from the same release. Treat a release as a coordinated change to application workloads and enabled dependencies, not as an image-tag update.

Choose the change strategy

Release-specific compatibility and required actions take precedence over this general runbook. Do not infer rollback compatibility from a successful manifest render.

Prerequisites

• The supported source version and upgrade path are confirmed from the target release notes.
• The target chart and every image are pinned to immutable versions or digests.
• The exact production values and dependency topology have passed an upgrade rehearsal in a non-production environment.
• A coordinated backup has been restored successfully with the current release.
• Monitoring, synthetic checks, alert routing, maintenance communications, and rollback owners are ready.
• The cluster has enough CPU, memory, storage, IP, and database capacity for old and new pods to overlap during rollout.

Capture the current state

Record the release and cluster state before changing it:

helm history koldan --namespace koldan
helm get values koldan --namespace koldan --all > current-values.yaml
helm get manifest koldan --namespace koldan > current-manifest.yaml

kubectl --namespace koldan get deployment,statefulset,pod,hpa,pvc
kubectl --namespace koldan get events --sort-by=.lastTimestamp

The exported Helm values and manifest can contain credentials, license data, certificate material, and other sensitive settings. Store them only in the approved encrypted change record, restrict access, and remove the local copies after the recovery window.

Also record:

• chart version, application version, and image digests;
• desired/Ready replicas and HPA conditions;
• current error rate, latency, backlog, PostgreSQL connections, and storage headroom;
• the latest verified backup identifiers and recovery point;
• active certificate, identity-provider, S3, Temporal, and database endpoints.

Prepare the target release

1. Read the release notes for required source versions, maintenance needs, database changes, dependency versions, removed values, and rollback limitations.
2. Diff the old and target values.yaml. Reapply operator-owned overrides to a clean values file based on the new release. Do not use --reuse-values; it can retain removed values and omit new release defaults.
3. Keep chart-managed lists, including workerPools.<pool>.workers, out of the operator values file. Carry forward supported overrides such as capacityOverrides instead.
4. Recalculate worker-provider concurrency and PostgreSQL connection ceilings for any replica, pool, or default change.
5. Confirm StorageClass behavior, PVC expansion, disruption controls, and node placement for all enabled dependencies.
6. Take and verify a coordinated backup immediately before the production change.

Lint and render the exact chart package:

helm lint ./koldan --values operator-values.yaml

helm template koldan ./koldan \
  --namespace koldan \
  --values operator-values.yaml \
  > rendered.yaml

Treat rendered.yaml as sensitive because the chart renders some credentials and the license into Kubernetes Secrets. Inspect it in a protected workspace, then remove it according to the change procedure.

Review the diff for:

• removed or renamed workloads and Services;
• image references and pull credentials;
• Secret and ConfigMap changes and their pod-template checksums;
• Route, Ingress, load-balancer, DNS, and TLS behavior;
• replica counts, HPAs, resource requests/limits, placement, and termination grace;
• PVC names, access modes, StorageClasses, and requested sizes;
• bundled dependency version or persistence changes;
• application properties and database connection pools.

Perform the upgrade

Start the maintenance window or change observation before running Helm. If the release requires quiescence, block new HTTP and gRPC traffic and allow active work to reach the release-specific safe point.

helm upgrade koldan ./koldan \
  --namespace koldan \
  --values operator-values.yaml \
  --atomic \
  --timeout 30m

Use a timeout based on measured image pulls, scheduling, volume attachment, dependency startup, and application initialization. --atomic waits and reverses Kubernetes resources if the upgrade fails; it does not undo database migrations, object changes, identity-provider changes, or external infrastructure changes.

Observe the rollout in another terminal:

kubectl --namespace koldan get pod --watch
kubectl --namespace koldan get events --sort-by=.lastTimestamp

Do not declare success when Helm exits alone. Confirm that old pods terminate, new pods become Ready, and no workload remains on an unintended image.

Verify the release

Complete these checks before ending the maintenance window:

1. helm status and helm history show the intended deployed revision.
2. Every expected Deployment has its required available replicas; HPAs have valid targets and metrics.
3. Pods have the intended image digest, configuration checksum, resource settings, and Secret mounts.
4. Liveness and readiness are stable with no restart, scheduling, mount, or image-pull loop.
5. PostgreSQL shows acceptable connections, locks, migration status, query latency, disk, and WAL behavior.
6. Temporal frontend and persistence are healthy; new work completes and existing backlog does not regress.
7. Representative object reads and writes succeed through the configured S3 identity.
8. OIDC login/token validation and expected authorization succeed.
9. Public HTTP and gRPC checks succeed through DNS, TLS, and the production routing path.
10. Error rate, latency, provider throttling, worker backlog, and storage growth remain within the tested operating envelope.
11. Metrics, logs, dashboards, and alerts include the new pods and release labels.

Continue heightened observation for a normal traffic cycle. Record the final revision, verification evidence, and any changed capacity limits.

Handle a failed upgrade

Pause further changes and preserve pod events, current and previous-container logs, Helm status/history, and dependency signals. Then choose the recovery path:

For a compatible Helm rollback:

helm history koldan --namespace koldan

helm rollback koldan <known-good-revision> \
  --namespace koldan \
  --wait \
  --timeout 30m

Repeat the full release verification after rollback. Confirm that restored Kubernetes configuration still matches any credentials, certificates, DNS, database, Temporal, and S3 state that changed during the failed attempt.

Close the change

Remove protected local render/export files, retain the approved audit record, and confirm the next scheduled backup covers the upgraded state. Update the capacity baseline and runbook when the release changes operational signals or procedures. Do not delete the pre-upgrade backup until both the rollback window and the organization's backup-retention requirement have passed.