Health and monitoring

Put health checks, metrics, centralized logs, and actionable alerts in place before admitting production traffic. Monitor both Koldan and every service it depends on; a Ready pod does not prove that an end-to-end request can complete.

Prerequisites

A cluster monitoring system that can discover and scrape pod-local endpoints.
Central log storage with access controls and retention appropriate for operational metadata.
Alert routing with an identified responder and an escalation path.
Dashboards for Kubernetes, PostgreSQL, Temporal, object storage, the identity provider, and any enabled inference provider.
A synthetic check that can authenticate and exercise representative HTTP and gRPC operations without using production data.

Use the right signal

Signal	What it establishes	Operator action
Liveness probe	The process can continue running.	Investigate repeated restarts; do not use liveness as a service-availability measure.
Readiness probe	Kubernetes may send traffic to the pod.	Page on insufficient Ready replicas and correlate with dependency health.
Public HTTP/gRPC check	DNS, TLS, routing, authentication, and an API path work from a client network.	Use as the primary external availability signal.
Queue and backlog signals	Accepted work is being consumed at the required rate.	Compare arrival rate, completion rate, worker capacity, and Temporal health.
Dependency telemetry	PostgreSQL, Temporal, S3, identity, and enabled providers have capacity and are reachable.	Resolve dependency saturation before adding Koldan concurrency.
Backup and certificate status	Recovery material and trust remain usable.	Alert before expiry and on failed or unverified backups.

Health and metrics endpoints

Application deployments created by the chart use these management paths:

/actuator/health/liveness
/actuator/health/readiness
/actuator/prometheus

The HTTP and gRPC management ports are 5193 and 5195. Worker-pool ports come from workerPools.<pool>.managementPort; the packaged defaults are 5196, 5197, and 5198. Packaged engine deployments use 5200. Treat the current chart as the source of truth for the installed release.

Management ports are cluster-internal. Do not add them to a public Service, Ingress, Route, or load balancer. Limit scraper access with network policy where the platform supports it.

To check a specific pod without exposing its management port:

kubectl --namespace koldan port-forward pod/<pod-name> \
  <management-port>:<management-port>

curl --fail --silent --show-error \
  http://127.0.0.1:<management-port>/actuator/health/readiness

An HTTP success from this test confirms only that pod's readiness endpoint. Continue with a public synthetic request and dependency checks.

Configure metrics collection

The chart can install optional Prometheus and Grafana dependencies with prometheus.install and grafana.install. Their packaged values are a starting point: review persistence, retention, authentication, TLS, resources, placement, and backup before production use. An existing platform monitoring service can scrape the same management endpoints.

After configuring monitoring target selection, confirm every expected target is present and UP. The scrape endpoint is pod-local, so verify that the collector reaches the named management container ports. Do not assume that installing Prometheus alone makes every pod a scrape target.

At minimum, retain enough metric history to distinguish a transient spike from sustained saturation and to compare behavior before and after a deployment. Preserve these dimensions where available: cluster, namespace, release, workload, pod, service, response status, and dependency. Avoid tenant or request content in metric labels.

Collect useful logs

Collect stdout and stderr from every Koldan pod and enabled chart dependency. Attach Kubernetes metadata at ingestion time, including namespace, workload, pod, container, image, node, and restart count. Keep clocks synchronized so an API error can be correlated with dependency logs.

Restrict log access and redact at the collector when required. Logs can contain tenant identifiers, object names, endpoint addresses, and correlation identifiers. Never add access tokens, passwords, license contents, private keys, or complete request payloads to dashboards or alert notifications.

During an incident, preserve both current and previous-container logs:

kubectl --namespace koldan logs <pod-name> --since=30m
kubectl --namespace koldan logs <pod-name> --previous

Alerting baseline

Choose thresholds from observed production behavior and required service objectives. Each alert should identify the affected service, include a dashboard and runbook link, and state the first safe diagnostic action.

Condition	Corroborating signals	First response
External check fails	Ingress/Route and load-balancer health, Ready endpoints, HTTP/gRPC failures	Determine whether the fault is routing, Koldan, identity, or a dependency.
Available replicas fall below requirement	Deployment availability, pod conditions, restarts, scheduling events	Protect remaining capacity and inspect the newest failing pod.
Restart loop or OOM kill	Restart count, exit reason, memory working set, limits	Preserve previous logs; correct resource or configuration cause before restarting repeatedly.
Error rate or latency changes materially	Request rate, response class, pod CPU/memory, dependency latency	Compare to the most recent deployment and isolate the saturated tier.
Temporal backlog grows	Arrival/completion rates, worker readiness, Temporal persistence latency	Confirm downstream headroom before scaling workers.
PostgreSQL approaches a limit	Active/waiting connections, query latency, locks, CPU, I/O, disk/WAL	Stop unsafe scaling and reduce load or increase tested database capacity.
Object operations fail or slow	Provider errors, throttling, latency, quota, bucket growth	Check endpoint, policy, capacity, and lifecycle/replication state.
Storage approaches capacity	PVC usage, database and object growth, node ephemeral storage	Extend capacity through the storage provider's supported procedure; do not delete live data manually.
Certificate, credential, or license nears expiry	Issuer/provider expiry inventory and application warnings	Rotate through the documented procedure before the maintenance window closes.
Backup is missing, stale, or failed	Job status, off-cluster copy, restore-test age	Treat the deployment as outside its recovery objective until a backup and verification succeed.

Prevent alert storms by routing one dependency incident to the team that owns that dependency while retaining Koldan symptoms as context. Do not silence availability alerts solely because pods remain Ready.

Grafana authentication

If Grafana is enabled, use a dedicated OIDC client with its exact HTTPS callback URL. Map only approved monitoring roles, disable anonymous access and uncontrolled sign-up, keep TLS verification enabled, and store the client secret through the approved secret workflow. Test an administrator recovery login before depending on OIDC exclusively.

The chart includes a Koldan dashboard when grafana.install is enabled. Confirm its queries match the labels emitted by your scraper and use it as a starting view, not as complete production coverage.

Production-readiness verification

Before go-live and after monitoring changes:

Confirm all expected scrape targets are present and current.
Confirm dashboards populate for every enabled replica and dependency.
Exercise synthetic HTTP and gRPC checks through the production network path.
Trigger representative warning and critical alerts in a non-production environment.
Confirm notifications reach the responder with the correct runbook and labels.
Stop one non-critical test replica and verify that readiness, availability, logs, and alerts agree.
Confirm monitoring remains available when a Koldan workload is unhealthy.

If telemetry disappears during a rollout, pause the rollout and restore observability before continuing. A change that cannot be observed cannot be safely verified or scaled.