LLM providers

Configure an external large language model (LLM) provider only when an enabled and licensed Koldan capability requires one. Provider accounts, network access, data-governance controls, model availability, quota, and credentials are operator-managed.

Prerequisites

Before enabling a provider, approve the data residency and retention terms, endpoint location, model or deployment ID, authentication method, rate and token quotas, expected concurrency, cost controls, timeout policy, and incident owner. Permit egress only to the required endpoint and ensure DNS and TLS work from the Koldan namespace.

Provider model catalogs and names change independently of Koldan. Use a model ID currently enabled in your provider account and supported by that provider's API; do not copy a sample model name into production without validating it.

Configuration contract

Provider declarations are a named map under koldan.llm.providers. Each entry has a stable operator-selected ID and one supported type:

config: |
  koldan:
    llm:
      providers:
        primary:
          type: <openai|azure-openai|ollama|bedrock|gemini|anthropic|vertex-ai>
          <provider-type>:
            model: <provider-model-or-deployment-id>

Koldan capability settings reference the provider ID and may override its model. The supported routing keys in this release are:

• koldan.llm.session-history-summary
• koldan.llm.session-history-autotitle
• koldan.llm.ai-simple-text-summary
• koldan.llm.speech-service-transcription-summary

Each accepts provider and optional model. Configure only the capabilities enabled for the deployment. For example:

config: |
  koldan:
    llm:
      providers:
        primary:
          type: ollama
          ollama:
            base-url: https://llm.example.net
            model: <model-id>
            timeout: 5m
      session-history-summary:
        provider: primary

Provider IDs referenced by a capability must exist. Every provider declaration requires its own model or deployment ID. A capability-level model changes the model for that capability; it does not make the provider-level value optional.

Protect credentials

API keys, static cloud credentials, credential JSON files, and sensitive custom headers are secrets. Values under the chart's config block are rendered into a ConfigMap; a populated API key there is not protected as a Kubernetes Secret. Follow Configuration and secrets, restrict Helm and namespace access, and prefer provider workload identity where supported.

A string such as ${PROVIDER_API_KEY} is only a reference: it does not inject an environment variable into a pod. Use it only when your approved deployment customization makes that variable available to every Koldan workload that consumes the provider. Verify the rendered pod specification before relying on it.

Disable provider request-and-response logging for production data unless the resulting content has been explicitly approved for the log destination and retention policy.

OpenAI and compatible endpoints

Use type: openai for the OpenAI Chat API or a compatible endpoint validated with the installed release.

primary:
  type: openai
  openai:
    api-key: <injected-api-key>
    model: <model-id>
    timeout: 5m

When using a gateway or compatible API, validate its base-URL path, authentication headers, streaming behavior, error format, and supported request parameters. API compatibility is not implied by an OpenAI-like product label.

Azure OpenAI

Use type: azure-openai with an Azure resource endpoint and deployment name.

primary:
  type: azure-openai
  azure-openai:
    endpoint: https://<resource-name>.openai.azure.com/
    api-key: <injected-api-key>
    deployment-name: <deployment-id>
    timeout: 5m

For workload identity, set use-azure-credential: true, omit api-key, assign the Kubernetes workload identity access to the Azure resource, and make the Azure default credential chain available to the consuming pods. Do not assume a developer workstation or Azure CLI login is visible inside a pod.

Ollama

Use type: ollama for an operator-managed Ollama endpoint.

Do not use the localhost default unless Ollama runs in the same pod. For an in-cluster or external service, use its routable HTTPS URL, restrict access with NetworkPolicy and authentication controls available at the endpoint or gateway, and size model memory and concurrency independently from Koldan.

primary:
  type: ollama
  ollama:
    base-url: https://llm.example.net
    model: <model-id>
    timeout: 5m

Amazon Bedrock

Use type: bedrock for a model enabled in the selected AWS Region.

Prefer pod workload identity, such as an IAM role associated with the chart service account, and omit both static credential properties. Grant only the required Bedrock invocation actions for approved resources. If static credentials are unavoidable, provide both values; a partial pair falls back to the default credential chain and is a configuration error from an operational perspective.

primary:
  type: bedrock
  bedrock:
    region: <aws-region>
    model: <model-id>
    timeout: 5m

Google AI Gemini

Use type: gemini with a Google AI API key.

primary:
  type: gemini
  gemini:
    api-key: <injected-api-key>
    model: <model-id>
    timeout: 5m

Enable returned thinking content only after reviewing its data-handling and logging implications.

Anthropic

Use type: anthropic with an Anthropic API key.

primary:
  type: anthropic
  anthropic:
    api-key: <injected-api-key>
    model: <model-id>
    timeout: 5m

Google Cloud Vertex AI

Use type: vertex-ai for Gemini models hosted through Vertex AI.

Vertex AI uses Google Cloud Application Default Credentials (ADC). Prefer Workload Identity on GKE. Where a service-account JSON file is required, create a Kubernetes Secret whose key is google-credentials.json, then use the chart's existing-Secret interface:

googleCredentials:
  enabled: true
  existingSecret: koldan-google-credentials

config: |
  koldan:
    llm:
      providers:
        primary:
          type: vertex-ai
          vertex-ai:
            project: <project-id>
            location: <location>
            model: <model-id>

The chart mounts the credential in the Koldan API and workflow-worker workloads and sets GOOGLE_APPLICATION_CREDENTIALS. Do not populate googleCredentials.json in an ordinary values file: that alternate chart path creates a Secret but also retains the JSON in Helm release data. A host gcloud login is not an authentication method for cluster pods.

Verify and operate

After rendering and applying the configuration:

1. Confirm only the intended Koldan workloads received the credential or workload identity.
2. Verify endpoint DNS, TCP, TLS, and proxy behavior from the Koldan namespace.
3. Confirm the identity can invoke the selected model but cannot administer unrelated provider resources.
4. Exercise one small representative request through each enabled Koldan capability.
5. Verify timeout and provider-unavailability behavior without changing security controls.
6. Monitor request count, latency, error codes, throttling, token usage, quota, and spend without logging prompts or responses by default.

Rotate an API key by issuing a second key, updating and rolling all consuming workloads, testing every configured capability, and revoking the old key. For workload identity, apply and test the new role binding before removing the old permission.