Scaling and capacity

Koldan ships two supported capacity profiles. Select the deployment case first, then qualify it against the site's acceptance rate, processing-start latency, completion target, sustained media-hours per hour, and supported burst. A profile is a resource baseline, not a universal throughput guarantee.

Choose the deployment case

Install a profile first and place site-specific images, endpoints, credentials, ingress, storage classes, and provider settings in a later values file so they take precedence:

# One 24-vCPU / 64-GiB / 2-TiB node
helm upgrade --install koldan ./charts/koldan \
  --namespace koldan --create-namespace \
  -f ./charts/koldan/profiles/single-node.yaml \
  -f ./site-values.yaml

# Existing organizational Kubernetes/OpenShift cluster
helm upgrade --install koldan ./charts/koldan \
  --namespace koldan --create-namespace \
  -f ./charts/koldan/profiles/medium-cluster.yaml \
  -f ./site-values.yaml

The single-node profile requests 18.45 CPU and 44.75 GiB memory in steady state. This leaves 5.55 CPU and 19.25 GiB for the operating system, Kubernetes, transient pressure, and maintenance. It uses one replica and is not highly available; node maintenance or failure interrupts service. Its persistent-volume starting points are 1 TiB for object storage, 64 GiB each for PostgreSQL and search, 16 GiB for ZooKeeper, and 10 GiB for update data. Backups must leave the VM and its underlying failure domain.

The medium-cluster profile requests approximately 71.75 CPU and 111.5 GiB before ingress, monitoring, model servers, and platform services. Use at least three 32-vCPU/64-GiB worker nodes as the reference topology. This is only a starting point: add nodes or change pod sizes when allocatable capacity, failure-domain placement, storage topology, model serving, or the site's measured workload requires it.

Scale Koldan from site service objectives, measured demand, saturation, and pending work. Increasing replicas is safe only when nodes, PostgreSQL, object storage, identity, and enabled processing providers all retain sufficient headroom.

Prerequisites

• Resource requests and limits based on workload measurements.
• Metrics for request rate, error rate, latency, pending background work, pod CPU/memory/ephemeral storage, and dependency saturation.
• A current inventory of replica counts, per-pod concurrency, database pools, and provider capacity.
• A tested load profile that represents file sizes, durations, languages, and enabled features used in the environment.
• A site capacity contract that defines expected request acceptance, processing start, completion time, sustained throughput, burst behavior, and priority reservations.
• A rollback value set and an operator who can reduce incoming load if a change causes saturation.

Do not size from registered or concurrent user count alone. Two environments with the same user count can have very different request shapes and processing cost.

Define the site capacity contract

Capacity is a product commitment before it is an infrastructure setting. Decide what workload Koldan must support before choosing replicas or concurrency.

Record the following for every site:

Do not convert registered-user count directly into capacity. Use observed or contractually bounded work units. For speech processing, audio duration and channel count are usually more predictive than job count; a multichannel file can create several concurrent inference operations. For summaries, input and output tokens matter more than request count.

Choose one primary operating priority and document the secondary objective:

• Balanced: protect stable completion throughput and dependencies while keeping ordinary pending time low. This is the packaged default posture.
• Acceptance-first: keep request handling responsive and accept supported work during bursts, allowing processing to wait when execution capacity is full.
• Immediate-start: provision enough warm capacity that supported bursts begin processing within the agreed start-time target.
• Completion-throughput: maximize completed work per hour, even when some accepted jobs wait before starting.
• Interactive reservation: reserve capacity for streaming or other latency-sensitive traffic before allocating the remainder to offline and batch work.

These objectives are not interchangeable. Increasing concurrency may reduce time waiting to start but worsen completion time if it overloads CPU, storage, the database, or an inference server. Accepting work quickly does not require starting all of it immediately. Conversely, immediate start requires the complete processing path—not only Koldan worker capacity—to be provisioned for the burst.

Koldan worker settings can shape concurrency by workload type, but they do not by themselves reserve physical resources or provide strict per-tenant ordering and fairness. Do not promise premium-tier or tenant-specific priority unless the site's admission, routing, resource-reservation, and provider-capacity design enforces it end to end.

Select the capacity control

The chart creates HPAs for the HTTP API, gRPC API, and compute worker deployment. Inference, orchestration, and packaged engine instances use explicit replicas. When an HPA is enabled, the corresponding static replicaCount is not rendered into that Deployment.

Configure API and compute autoscaling

Set CPU and memory requests before enabling utilization-based scaling; Kubernetes calculates utilization relative to requests. Configure:

web:
  resources:
    requests:
      cpu: <measured-request>
      memory: <measured-request>
  autoscaling:
    enabled: true
    minReplicas: <minimum-ready-capacity>
    maxReplicas: <dependency-safe-maximum>
    targetCPUUtilizationPercentage: <tested-target>

grpc:
  resources:
    requests:
      cpu: <measured-request>
      memory: <measured-request>
  autoscaling:
    enabled: true
    minReplicas: <minimum-ready-capacity>
    maxReplicas: <dependency-safe-maximum>
    targetCPUUtilizationPercentage: <tested-target>

workerPools:
  compute:
    resources:
      requests:
        cpu: <measured-request>
        memory: <measured-request>
    autoscaling:
      enabled: true
      minReplicas: <minimum-ready-capacity>
      maxReplicas: <node-and-storage-safe-maximum>
      targetCPUUtilizationPercentage: <tested-target>

The chart also renders targetMemoryUtilizationPercentage when set. Memory-based HPA signals do not work well for workloads whose memory does not fall after traffic subsides; validate scale-down behavior before enabling them.

Confirm the cluster metrics API is healthy, render the chart, and verify each HPA targets the intended Deployment. Set API maxReplicas from downstream and database limits. Set compute maxReplicas from allocatable CPU, node and pod ephemeral storage, database connections, and the aggregate concurrency created by every compute replica. Retain enough minReplicas for maintenance and expected failure domains.

Size worker pools

The compute pool can use workerPools.compute.autoscaling; when enabled, budget against maxReplicas rather than replicaCount. Change inference and orchestration replicas explicitly with workerPools.<pool>.replicaCount. Use workerPools.<pool>.capacityOverrides to change supported per-worker concurrency; do not copy or replace the chart-managed workers lists in operator values because those lists can change with a release.

Each worker-pool replica serves every background workload assigned to that pool. Scaling a pool therefore changes aggregate capacity for all of those workloads, not only the one that currently has pending work. Use the installed chart's worker mapping and defaults when calculating the effect of a change.

Understand the profile worker defaults

The profile worker capacities target stable mixed-workload performance on their specified pod resources. Starting more work is not necessarily faster: contention can reduce completed work per hour and increase tail latency.

Capacity is configured per background workload, not as one shared pool-wide ceiling. Different workloads in the same pod can run concurrently. With one packaged replica, the theoretical maxima are:

These totals are upper bounds, not reserved resources or a promise that every workload can sustain its maximum simultaneously. The in-process coordination cache and thread ceiling are shared across the orchestration pod. Likewise, CPU, memory, ephemeral storage, the database pool, and network connections are shared by every workload in a pool.

Treat these values as balanced starting points, not benchmark results for every installation. Measure speech throughput in media-hours per hour using the site's representative duration, codec, channel count, language, and enabled features. Raise a workload limit only when time waiting to start exceeds the site target, the configured concurrency remains fully used, and the worker and dependency serving that work have measured headroom. Cache pressure is different: increase the coordination cache or thread ceiling only when eviction or reconstruction cost is material and the orchestration pod retains memory and thread headroom.

Capacity overrides are maps keyed by the chart-defined worker name. Keep workerPools.<pool>.workers chart-managed; replacing that list can remove background processing introduced by a release. Helm deep-merges override values, so inspect the rendered worker ConfigMap before applying a change. The supported worker names and capacity fields are listed in the Helm values reference.

For every constrained provider, maintain this invariant:

usable provider capacity = tested efficient capacity
                         - reserved real-time capacity
                         - failover and safety headroom

pool replicas × per-replica workload concurrency
  ≤ usable provider capacity

The shipped profiles use maxSurge: 0 and maxUnavailable: 1, so they do not add a large replacement pod during rollout. If a site changes that strategy, calculate the peak as replicas + maxSurge, not only the steady-state replica count.

For example, if a provider sustains 24 concurrent operations, 6 are reserved for other traffic, and 3 are held as headroom, Koldan capacity must not exceed 15. With 3 pool replicas, cap the relevant workload at 5 operations per replica.

One inference-pool replica participates in several provider-backed workloads. Recalculate the invariant separately for every enabled provider whenever that pool's replica count changes. More replicas with unchanged concurrency can overload a provider even while Kubernetes CPU remains low.

Do not scale a pool only from total pending work. Compare backlog and time waiting to start by workload type, then identify whether the limiting resource is Koldan compute, PostgreSQL, object storage, or a downstream provider. One blocked provider does not justify increasing the whole pool.

Use workerPools.<pool>.terminationGracePeriodSeconds where a worker needs longer than the cluster default to stop safely. Validate scale-down with real-duration test jobs and watch whether work completes or is safely retried before reducing production replicas.

Budget PostgreSQL connections

Every enabled application pod can hold a connection pool. Calculate the maximum, not the idle count:

Koldan connection ceiling =
  Σ(maximum workload replicas × per-pod maximum pool size)

required PostgreSQL capacity =
  Koldan connection ceiling
  + identity and background-processing infrastructure pools when they share PostgreSQL
  + migration, backup, monitoring, and administration sessions
  + failover and safety reserve

Inventory these chart controls where enabled:

• web.hikari.maximumPoolSize
• grpc.hikari.maximumPoolSize
• engineK2.hikari.maximumPoolSize and engineSlibe.hikari.maximumPoolSize
• workerPools.<pool>.hikari.maximum-pool-size

For HPAs, use maxReplicas in the calculation. For engine instances, sum every instance replica. Confirm PostgreSQL max_connections, memory, CPU, I/O, locks, and failover capacity can sustain the result. A connection pooler does not remove transaction, query, or lock limits.

Plan compute and storage capacity

Observe peak ephemeral-storage use under representative concurrent jobs, including temporary data and retry overlap. Set requests high enough for scheduling and limits high enough for the tested peak. An ephemeral-storage limit breach can evict or restart a pod even when CPU and memory are healthy.

Forecast each durable store independently:

Never use manual database-row, bucket-object, or PVC-file deletion as a capacity response. Apply supported retention and provider maintenance procedures.

Perform a scaling change

1. Record the trigger, baseline traffic, backlog, latency, errors, resource use, and dependency headroom.
2. Calculate provider concurrency and PostgreSQL connection ceilings at the proposed maximum.
3. Change one limiting dimension in the operator values file. Coordinate replica and concurrency changes in the same rollout when the capacity invariant requires it.
4. Render the exact release and inspect Deployments, HPAs, resources, and worker ConfigMaps. For a capacity override, verify the exact worker key and ensure every chart-managed worker remains in its pool.
5. Apply an atomic Helm upgrade during an observed change window.
6. Wait for the automatic affected-pool rollout, then confirm desired and Ready replicas, pod placement, scrape targets, database connections, and active workers for every background workload in the pool.
7. Run representative traffic and watch error, latency, backlog, throttling, and storage signals through at least one normal demand cycle.
8. Record the new tested envelope and update alerts and capacity forecasts.

If errors, provider throttling, database waits, eviction, or backlog growth worsens, stop increasing traffic. Restore the previous values with Helm, verify every replica and dependency, and reduce admitted load until the original capacity envelope is stable. A Helm rollback does not undo provider-side quota or database changes; reverse those separately through their controlled procedures.