Troubleshooting

Start with the affected operator-visible function, establish the scope and start time, and then follow the request path through Kubernetes and its dependencies. Preserve evidence before restarting or rolling back; repeated changes can destroy the only useful previous-container logs and make recovery harder.

First response

1. Record the incident time, affected HTTP or gRPC operation, scope, external symptom, and a correlation identifier when available. Do not capture credentials or request content.
2. Check current alerts and recent changes to Helm, GitOps, DNS, certificates, secrets, identity, network policy, nodes, storage, and external providers.
3. Determine whether the failure affects all traffic, one endpoint, new work, in-flight work, one replica, one node/zone, or one dependency.
4. Protect recovery: stop the active rollout if it is still progressing, preserve backups, and reduce or block new traffic when continued writes could worsen inconsistent or saturated state.
5. Inspect Kubernetes state before restarting anything.

helm status koldan --namespace koldan
helm history koldan --namespace koldan

kubectl --namespace koldan get deployment,statefulset,pod,hpa,pvc
kubectl --namespace koldan get events --sort-by=.lastTimestamp
kubectl --namespace koldan describe pod <pod-name>
kubectl --namespace koldan logs <pod-name> --since=30m
kubectl --namespace koldan logs <pod-name> --previous

Helm output and logs can contain sensitive configuration or tenant metadata. Keep evidence in an approved incident store and redact it before sharing.

Symptom map

Pod does not start or become Ready

For Pending pods, read the scheduler event; do not reduce requests blindly. Confirm namespace quotas, node allocatable resources, node selectors, taints/tolerations, affinity, PVC binding/topology, and image-pull access. If a PVC is pending, verify its StorageClass, access mode, capacity, provisioner events, and zone constraints.

For startup or restart failures, check status.containerStatuses[].lastState, exit code, and reason. OOMKilled requires comparing peak memory with both the container limit and node pressure. A probe failure after startup requires checking the management endpoint and application log, not merely increasing probe delays.

Compare the installed release with the intended protected values file. Verify that referenced ConfigMaps, Secrets, certificate files, and license material exist and are readable without printing their contents. Check that the expected dependency DNS names resolve from the namespace.

If one new replica fails while old replicas remain healthy, keep the healthy replicas serving and correct the rollout. If all replicas fail after a change, reinstate traffic controls and use the upgrade recovery decision.

Encryption-key or API-key HMAC failure

Treat koldan.security.encryption-key and koldan.api.api-keys.hmac-secret as release-wide values. Compare secret-manager versions, pod-template configuration revisions, and pod creation times without printing the values.

The current chart does not wire an arbitrary Kubernetes Secret into these properties. Confirm that the final pod specification contains the approved environment injection, or account for the fact that populated config values reside in the ConfigMap and Helm release metadata. See Configuration and secrets.

HTTP or gRPC is unavailable

Trace from outside inward:

1. DNS resolves to the intended load balancer or Route/Ingress.
2. The certificate is valid for the requested hostname and the full chain is served.
3. Firewall/security-group and load-balancer health checks permit the configured port and protocol.
4. Route/Ingress termination matches the backend protocol; gRPC retains HTTP/2 through the selected path.
5. The Kubernetes Service has Ready endpoints and selects the intended pods.
6. The public port matches web.service.port or grpc.service.port and any advertised address/port configuration.
7. A direct in-cluster request succeeds before changing the public network layer.

Distinguish connection timeout, connection refusal, TLS handshake failure, HTTP gateway response, authentication response, and application error. They originate at different layers. Do not disable TLS verification or expose a management port as a workaround.

Certificate or private-CA failure

Check the requested hostname against the certificate SAN, validity period, issuer, complete chain, private-key match, key usage, and system time. Identify which hop terminates TLS: client load balancer/Route/Ingress, Koldan gRPC, or an external dependency.

For outbound failures, confirm customCA.enabled, the expected PEM entries, Secret creation, mount presence, and that every affected pod rolled after the trust change. Test from the same namespace and network path as the failing pod. Keep hostname verification enabled.

During CA rotation, retain old and new trust anchors through the overlap period. If recovery is required, restore the prior matched certificate/key and trust set through Helm; do not mix a previous key with a new certificate.

PostgreSQL failure

Classify the message before acting:

Do not solve connection exhaustion by repeatedly restarting all clients; reconnect storms can worsen it. Stop unsafe scaling, protect administration capacity, and either reduce admitted load/pool ceilings or increase tested database capacity. Do not manually alter schema history.

Temporal failure or growing backlog

Verify the configured frontend address, TCP 7233, TLS/authentication, namespace, and clock synchronization. Then check Temporal frontend, matching, history, and persistence health using the Temporal platform's supported tools.

Confirm all expected worker-pool Deployments are Ready and that the deployed chart version owns their current configuration. Compare arrival and completion rates, schedule-to-start delay, retry/error patterns, and downstream provider latency. A Ready worker can still be unable to complete work when PostgreSQL, S3, or a provider is saturated.

Classify queue symptoms before changing capacity:

Use task-queue names for Temporal observations and worker names for Helm capacityOverrides. The version-matched mapping is listed in Temporal. Do not rename queues, create replacement pollers, or edit Temporal state during an incident.

Do not increase replicas or concurrency until the constrained provider and PostgreSQL connection budget have headroom. Do not manually delete, replay, or edit Temporal persistence during incident response. For incomplete persistence recovery, stop worker workloads and follow the tested coordinated restore.

S3-compatible storage failure

Check the configured endpoint, DNS, TLS/CA, bucket, region, addressing mode, credential or workload identity, policy, quota, throttling, encryption-key access, lifecycle actions, and replication state. Test with the same identity and network path Koldan uses; success from an administrator laptop does not establish pod access.

For missing content, correlate the database record with object versions and provider audit logs. Check expiration/archive policies and the recovery point used for both systems. Do not create an empty replacement object or delete the database record to clear the symptom.

Authentication failure

Confirm the configured issuer is exactly the value returned and used in tokens. Check discovery and JWK reachability from the Koldan namespace, TLS trust, signing-key rotation, client ID/secret, token audience, redirect URI, and system time. Then verify role claims and mappings.

If LDAP federation is used, additionally check bind access, search base and filter, username attribute, group mapping, LDAP TLS trust, and provider-side connection limits. Preserve an administrator recovery path that does not depend on the failed federation.

Distinguish one user's authorization failure from all users' authentication failure. Do not broaden role mappings or enable uncontrolled sign-up as an incident workaround.

Scheduling, eviction, or saturation

Review CPU throttling, memory working set/OOM, node and pod ephemeral storage, PVC capacity, node pressure, HPA events, quotas, and placement across failure domains. Then correlate PostgreSQL connections/locks, Temporal backlog, object-store throttling, and downstream provider capacity.

If load exceeds the tested envelope, reduce admitted traffic first. Scale only after applying the formulas and checks in Scaling and capacity. More concurrency can turn a recoverable queue into cascading dependency failure.

Recover and close

After the immediate fix:

1. Repeat public HTTP/gRPC, authentication, database read/write, object read/write, and new-work completion checks.
2. Confirm desired and Ready replicas, restarts, error/latency, backlog, dependencies, and storage return to stable behavior.
3. Verify monitoring and backup jobs affected by the incident.
4. Remove temporary access or traffic controls and confirm normal routing.
5. Record the trigger, impact, evidence, corrective action, recovery time, and follow-up owner.
6. Add or improve the alert, dashboard, capacity guard, or runbook step that would shorten recurrence.

If recovery requires a chart reversal or persistent restore, use Upgrade and rollback or Backup and restore. Never treat a pod restart as resolution until the failed end-to-end operation has been verified.