Configuration and secrets
Use a version-controlled, non-secret values file as the declared state of a Koldan installation. Supply credentials, private keys, the license, and other sensitive values from an approved secret-delivery process at deployment time.
Prerequisites
Before changing configuration, record the Helm release name, namespace, chart version, current release revision, configuration owner, and recovery contact for each external dependency. Keep the chart package for the installed Koldan version available: its values.yaml is the version-matched configuration contract.
Configuration contract
The chart builds the application configuration in this order:
- Packaged application defaults.
- Chart-generated overrides.
- Operator configuration supplied through the top-level
configvalue.
Values in config therefore take precedence over packaged application defaults. Use normal Helm values for deployment concerns such as replicas, Services, Ingresses, Routes, resources, persistence, and bundled dependencies. Use config only for supported application properties:
config: |
koldan:
site:
name: Example
spring:
datasource:
url: jdbc:postgresql://postgresql.example.net:5432/koldan?sslmode=verify-full
username: <database-user>
password: <injected-database-password>
The example shows structure, not a safe secret-storage pattern. The chart renders config into a ConfigMap. Any secret placed there is readable to subjects that can read that ConfigMap. Several other chart inputs, including OAuth client secrets and license, are rendered into pod configuration or Kubernetes Secrets but remain present in Helm release data. Restrict access to the namespace and Helm release objects accordingly.
Do not edit chart-generated ConfigMaps, Secrets, Deployments, or StatefulSets directly. Helm replaces those changes on the next release operation.
Set production root secrets before first startup
Create the following site-specific values before any production workload starts. Do not use the packaged development defaults.
| Property | Production requirement | Consumers | Consequence of loss or unplanned change |
|---|---|---|---|
koldan.security.encryption-key |
Base64 encoding of exactly 32 random bytes. Keep one value consistent across every Koldan workload and replica that connects to the same database. | Koldan workloads that encrypt or decrypt persisted credentials and webhook secrets | A wrong key causes decryption failures. Losing the key can make encrypted database values unrecoverable. Replacing it without re-encrypting existing data breaks access to that data. |
koldan.api.api-keys.hmac-secret |
Site-unique high-entropy secret of at least 32 random bytes. Keep one value consistent across all HTTP and gRPC API replicas. | HTTP and gRPC API-key authentication | Changing it invalidates every existing API key immediately. The current release has no dual-HMAC-key overlap. |
Generate the values in a protected administrative environment and write them directly to access-controlled files or a secret manager; do not print them into CI logs:
umask 077
openssl rand -base64 32 > koldan-encryption-key.txt
openssl rand -base64 32 > koldan-api-hmac-secret.txt
The first command produces the required Base64 representation of a 256-bit encryption key. Do not reuse either value between independent environments. A disaster-recovery site that will attach to a copy of the same database must recover the same encryption key. It must also recover the same HMAC secret if existing API keys are expected to remain valid after failover.
Reference secret-backed environment variables from application configuration:
config: |
koldan:
security:
encryption-key: ${KOLDAN_ENCRYPTION_KEY}
api:
api-keys:
hmac-secret: ${KOLDAN_API_API_KEYS_HMAC_SECRET}
The current chart does not provide a general existingSecret, extraEnv, or envFrom interface for these two values. The configuration above works only when an approved deployment overlay, admission controller, or secret-injection mechanism adds both environment variables to every consuming Deployment. Render the final pod specifications and verify the secretKeyRef or equivalent injection; merely creating a Kubernetes Secret does not connect it to Koldan.
If the platform has no reviewed injection mechanism, the current chart requires the populated values to be supplied under config. That path writes them into the chart-managed ConfigMap and Helm release record. Treat both objects, rendered manifests, GitOps state, namespace backups, and support bundles as secret-bearing, and obtain an explicit security approval before production use.
Complete the required-secret inventory
The two root secrets above are always required for production. The following values are required when their corresponding integration is enabled:
| Capability | Required sensitive input |
|---|---|
| PostgreSQL | Database password or workload identity accepted by the external service |
| Default OIDC tenant and service authentication | Default-tenant client secret, engine client secret where used, and each chart workload OAuth client secret |
| S3-compatible storage | Access/secret key pair unless the selected integration uses a verified workload-identity path |
| LLM provider | Provider API key, cloud credential, or supported workload identity |
| Bundled dependencies | PostgreSQL, Keycloak, MinIO, Grafana, and Temporal credentials selected by the enabled subcharts |
| TLS and private trust | Serving private keys, client keys where mTLS is used, and protected CA-management inputs |
| Koldan license | The issued license document |
No production installation may retain <changeme>, an empty required credential, or a packaged development secret. Record an owner, source-of-truth location, rotation or recovery method, and consuming workloads for every entry.
Handle sensitive values
Treat at least the following as secrets: the license, database and S3 credentials, OIDC client secrets, LLM provider tokens, cloud credential files, private keys, encryption keys, registry credentials, and bundled-dependency administrator passwords.
- Keep populated secret files out of source control, build artifacts, shell history, tickets, and CI logs.
- Prefer short-lived workload identity where both Koldan and the provider support it.
- If a value must pass through Helm, render the complete values file only in a protected workspace, apply it through a masked CI job or equivalent controlled process, and delete the transient copy according to organizational policy.
- Avoid
--setfor secrets because command lines are commonly retained in shell history and process or CI output. - Grant each workload and dependency a distinct identity with only the required permissions.
- Limit
get,list, andwatchaccess to Secrets and Helm release objects. Kubernetes base64 encoding is not encryption. - Enable encryption at rest for the cluster secret store and back up the key-management configuration needed for recovery.
See Application properties for supported property names and defaults, and Helm values for the chart interface.
Plan and apply a change
- Export the current non-secret configuration and note the active Helm revision. Back up dependent state when the change can affect stored data.
- Change one operational concern at a time and have another operator review security-sensitive changes.
-
Render the exact chart version locally and inspect the affected resources:
-
Check the rendered output for unexpected plaintext secrets before retaining or sharing it. A rendered manifest is sensitive whenever populated secrets are supplied.
-
Apply the change with release waiting and automatic rollback:
-
Confirm all affected workloads become Ready and exercise the dependency or endpoint changed.
Rotate a credential
Where the provider supports overlapping credentials, issue a second credential first. Update Koldan, wait for every affected pod to use the new credential, test the complete operation, and only then revoke the old credential. If overlap is impossible, schedule a maintenance window and define the rollback credential before starting.
A Helm change restarts workloads whose pod templates include the changed chart input. Verify the pod creation timestamps rather than assuming every workload restarted. Restart any affected workload that still has the old mounted or environment value, then verify readiness before revocation.
Rotate the API-key HMAC secret
There is no overlap between old and new HMAC secrets. Before changing koldan.api.api-keys.hmac-secret, inventory API-key consumers, arrange an alternate authenticated path for issuing replacements, and schedule a coordinated cutover. Roll HTTP and gRPC API replicas to the new value together. Every previously issued API key then fails validation and must be replaced. Retain the prior HMAC secret only in the protected rollback store for the approved rollback window; rolling back makes keys issued under the new secret invalid instead.
Protect the encryption key
Do not rotate koldan.security.encryption-key as an ordinary Helm change. The current configuration interface accepts only one key and provides no configuration-only procedure to decrypt with an old key while re-encrypting with a new one. Back up the key independently of the database, restrict restore access, and test recovering it into an isolated environment. If exposure requires replacement after encrypted data exists, keep the deployment fenced and use a release-specific, tested data re-encryption procedure before changing the configured key.
After either planned change, confirm all consuming replicas use one configuration revision. Verify API-key authentication through HTTP and gRPC where enabled, and verify operations that read previously stored encrypted credentials without printing their plaintext.
Recover from a failed change
Use pod events and logs to distinguish rendering, scheduling, startup, authentication, and dependency failures. Correct the forward configuration when possible. To restore the prior Helm revision:
helm history koldan --namespace koldan
helm rollback koldan <revision> --namespace koldan --wait --timeout 15m
A Helm rollback does not undo an external credential revocation, database migration, object deletion, or provider-side policy change. Restore those systems separately when applicable.
| Symptom | Check |
|---|---|
| Value has no effect | Property spelling and indentation; active Helm revision; rendered application-userdefined.yaml; pod creation time |
Pod starts with an unresolved ${...} value |
The referenced environment variable or mounted source is not available to that pod |
| Authentication fails after rotation | All replicas rolled; new credential permissions; old credential was not revoked prematurely |
| All existing API keys return authentication failures | koldan.api.api-keys.hmac-secret changed, differs between replicas, or fell back to the development default |
| Startup reports that the encryption key is not 256 bits | koldan.security.encryption-key is not valid Base64 for exactly 32 bytes, or an environment placeholder was not resolved |
| Existing stored credentials or webhook secrets cannot be decrypted | The configured encryption key differs from the key used to encrypt the database values; restore the correct key before allowing writes |
| Rollback remains unhealthy | External state changed outside Helm, or the prior credential/certificate is no longer valid |