Models & Reasoning

Changed in v0.107.0

Chat models are configured exclusively through the Model Offering Catalog (model_offerings.jsonc). The legacy per-provider variables (AZURE_OPENAI_MODELS, ANTHROPIC_MODELS, OPENAI_MODELS, FOUNDRY_MODELS, MODEL_MAX_CONTEXT_TOKENS, ANTHROPIC_HOSTING, and the per-provider chat endpoint/key variables) have been removed. The *_MODELS snippets shown below in the per-provider sections describe what each offering needs -- put those values on an offering in the catalog instead. The shared Azure credential variables and OPENAI_API_KEY are retained.

Multi-model switching

Author several offerings in model_offerings.jsonc and switch between them mid-conversation:

A model selector appears above the chat input (hidden when only one model is configured)
Per-session model selection persists across reloads
Regenerate with a different model -- click the chevron on Regenerate
Each assistant message shows which model generated it
All models share the same Tools, Skills, and MCP integrations

Per-model context-window limits are set by each offering's context_window field (defaulting to 128000 when unset). The context-window progress bar above the input updates automatically when you switch models.

Model Offering Catalog

Every chat model is managed from a single JSONC file. Point MODEL_OFFERINGS_FILE at it (default model_offerings.jsonc, resolved against the working directory); it is the single source of truth for model routing. A non-demo deployment with no catalog boots normally with a startup warning (so the Model Settings screen stays reachable to fix it); chat is unavailable and returns a message pointing at chatwalaau models add / the Model Settings screen the moment you try it (DEMO_MODE=true runs without one).

Each offering describes one model:

{
  "auth_profiles": { "gateway": { "api_key_env": "GATEWAY_API_KEY" } },
  "offerings": [
    // The default chat model (at least one chat offering is required).
    {
      "id": "gpt-5.2",
      "provider": "azure-openai",
      "model_ref": "gpt-5.2",              // the real deployment name
      "endpoint": "${AZURE_OPENAI_ENDPOINT}",
      "default": true,
      "context_window": 400000
    },
    // Claude via an Anthropic-compatible gateway (per-offering hosting).
    {
      "id": "claude-opus-4-8",
      "provider": "anthropic",
      "hosting": "foundry",
      "model_ref": "claude-opus-4-8",
      "base_url": "https://my-gateway.example.com/anthropic",
      "auth_profile": "gateway"
    },
    // A non-OpenAI Foundry family: turn off the OpenAI reasoning controls.
    {
      "id": "deepseek-v4",
      "provider": "foundry",
      "model_ref": "my-deepseek-deployment",
      "endpoint": "${FOUNDRY_PROJECT_ENDPOINT}",
      "family": "bare"
    },
    // Optional: at most one embeddings offering and one image offering.
    {
      "id": "embeddings",
      "provider": "azure-openai",
      "model_ref": "text-embedding-3-small",
      "endpoint": "${AZURE_OPENAI_ENDPOINT}",
      "operations": ["embeddings"]
    }
  ]
}

Why it helps:

Gateways. One endpoint that fronts several model families becomes several offerings sharing an endpoint / base_url -- no new provider to configure.
Mixed hostings. Direct Claude and Foundry-hosted Claude can coexist in one instance (per-offering hosting), which the single global ANTHROPIC_HOSTING could not express.
One required chat model, everything else optional. At least one chat offering is required; embeddings and image offerings are optional. With several chat models, the default: true one (or the first) is used when no model is specified.

Rules of thumb:

Secrets never live in the file. Reference an environment variable by name (api_key_env) or a named auth_profiles entry, or rely on the shared Entra ID credential lanes. ${VAR} in endpoint / base_url is filled from the environment.
family overrides the per-model options a gateway model advertises: openai-reasoning (effort + verbosity), anthropic-adaptive, or bare (no generation controls -- use this for non-reasoning gateway models).
A malformed catalog fails startup fast with a message naming the offending offering, so a typo never silently drops your models.
DEMO_MODE always uses the legacy lane. The legacy *_MODELS variables are now deprecated in favor of the catalog (still supported when no catalog is present).
Catalog invariants (enforced by the CLI, the GUI, and the server): at least one chat offering and exactly one default among them; unique ids; at most one embeddings and one image offering; hosting applies to the anthropic provider only.

When a deployment cannot serve a feature

Some features are supplied by the model's deployment, and whether they work depends on how that deployment was created -- not on the model.

Claude on Microsoft Foundry: pick the right hosting option

Foundry offers two hosting options, chosen when you create the deployment. Choosing "Default settings" in the portal picks Hosted on Azure, which is the restricted one:

	Hosted on Azure	Hosted on Anthropic
Where inference runs	Azure infrastructure	Anthropic infrastructure
Hosted web search	❌ Not supported	✅ Supported
Native structured outputs	❌ Not supported	✅ Supported

Requests that use an unsupported feature are rejected with 400 by design. The symptom looks like this, and every turn fails:

web search not supported in your workspace

The fix is to recreate the deployment, not to change any setting in ChatWalaʻau:

Foundry portal → model catalog → your Claude model → Deploy → Custom settings
Expand Model version settings and pick the version labeled Hosted on Anthropic
Note the deployment name and point the offering's model_ref at it

Inside one Foundry resource the endpoint URL and authentication are unchanged -- only the deployment name changes.

The tool version is not the problem

Anthropic publishes three web search tool versions. All three are server-side tools, so none of them works on a Hosted-on-Azure deployment, and the two newer ones also require code execution — which is unsupported there too.

If you must stay on a restricted deployment

Tell the offering what its deployment cannot serve, and ChatWalaʻau stops sending it. In Model Settings, or directly in the file:

{
  "id": "claude-opus-5-foundry",
  "provider": "anthropic",
  "model_ref": "claude-opus-5",
  "hosting": "foundry",
  "capabilities": {
    "web_search": false,
    "native_structured_output": false
  }
}

The two settings behave differently, and the difference matters:

Setting	What "Not available" does
Hosted web search	Removes the capability. There is no substitute. The web-search guidance is also removed from the system prompt, so the agent will not claim it can search — it will not invent citations.
Native structured output	Changes the mechanism, not the outcome. Structured Output keeps working through the forced-tool-use fallback and still returns JSON conforming to your schema.

Nothing disappears silently: a withheld tool appears in the prompt dump as excluded with the reason provider capability, and the server logs one line naming the offering.

Leave both on Default unless a request actually fails because of them. Default means "the provider decides", which is how every offering behaved before these settings existed.

Another way to get Claude and web search on Foundry

Register the deployment under the native foundry provider instead — its platform-supplied web search is unaffected by the Anthropic restriction. The trade-off is that generation options there are detected by deployment name, so a Claude deployment shows none. Recreating it as Hosted on Anthropic is usually better.

The order models appear in

The order of the chat offerings in the file is the order the model selector shows. Put your most-used model first and it appears first.

The default: true flag decides only which model is preselected for a new chat. It does not move that model to the top of the list -- a default sitting third in the file appears third, and is still the one preselected.

Changed in v0.106.0

Before v0.106.0 the default model was always hoisted to the top of the selector, regardless of where it sat in the file. If your default is not already first, the dropdown order will look different after upgrading -- the default now appears where it actually is. Nothing about which model answers has changed. To get the old look, drag it to the top in Model Settings.

Authoring offerings (CLI and GUI)

You do not have to hand-write the JSONC. The catalog is operator-configurable two ways, and both read and write the same MODEL_OFFERINGS_FILE, so you can mix and match. list still reflects a running server; the authoring commands and the screen below manage the file itself.

CLI (offline, no server required). The chatwalaau models commands read and write the local catalog file, creating it if it does not exist:

# Interactive wizard: author and append one offering
$ chatwalaau models add
Provider [azure-openai / anthropic / openai / foundry]: anthropic
Model ref: claude-opus-4-8
Operation [chat / embeddings / image]: chat
Offering id: claude-opus-4-8
Default chat model? [y/N]: y
Base URL: https://api.anthropic.com
Hosting [direct / foundry]: direct
Family (optional):
Context window (optional): 200000
API key environment variable name: ANTHROPIC_API_KEY
Wrote model_offerings.jsonc (1 offering).

# Edit or remove an existing offering by id
$ chatwalaau models edit claude-opus-4-8
$ chatwalaau models remove claude-opus-4-8

# List the models a running server is serving (read-only, unchanged)
$ chatwalaau models list

The wizard prompts only for the name of the API-key environment variable (api_key_env), never the secret value. chatwalaau init can also run this "set up your first model" step right after it writes .env (skip it with chatwalaau init --no-model, or in any non-interactive run), so you can configure a model before or after init.

GUI (Model Settings screen). Open the Model Settings screen from the gear/sliders icon in the chat sidebar footer (next to the info icon). The left pane is a searchable settings list (with a refresh); the right pane is the Model Offering Catalog editor. Compose offerings grouped by operation -- Chat (at least one; exactly one default), Embeddings (0-1), and Image (0-1). Each referenced environment variable shows a detected / not set indicator, so you can confirm the key is present without ever seeing its value. Save applies immediately via in-process hot reload -- no restart -- and a progress indicator shows while the change is applied.

Each offering shows as a compact row (id, provider, model, default) that expands on click when you want to edit it. Drag the grip handle to reorder -- that order is exactly what the chat model selector shows.

note

Saving from the GUI writes clean JSON, so hand-written comments in model_offerings.jsonc are not preserved. (Named auth_profiles entries are preserved.) CLI edits take effect the next time the server starts; GUI saves hot-reload the running server in place. No new environment variable is introduced -- both paths use MODEL_OFFERINGS_FILE (default model_offerings.jsonc).

Task model assignments

Besides the chat models you pick in the selector, ChatWalaʻau runs a few background helper models -- to generate a chat's title, extract user memory, curate agent memory, summarize a Teams meeting, and turn a natural-language question into an ontology SPARQL query. Each of these is assigned to one of your existing chat offerings, so it inherits that offering's provider, endpoint, and credentials automatically.

GUI. In the Model Settings screen, the Task model assignments section lists each task with a dropdown of your chat offerings plus Follow session / default.
CLI. chatwalaau models role list shows the tasks and current assignments; chatwalaau models role set <role> <offering-id> assigns one and chatwalaau models role clear <role> unsets it.
File. They live in an optional top-level roles block in model_offerings.jsonc, e.g. "roles": { "session_title": "gpt-4o-mini" }.

Leaving a task unassigned uses the chat's own model (then the catalog default), which is the default behavior. The role keys are session_title, user_memory_extraction, agent_memory_curation, meeting_summary, and ontology_nl.

note

Since v0.109.0 these assignments replaced the dedicated SESSION_TITLE_MODEL, USER_MEMORY_EXTRACTION_MODEL, AGENT_MEMORY_CURATION_MODEL, TEAMS_MEETING_SUMMARY_MODEL, and ONTOLOGY_NL_MODEL environment variables, which were removed. A leftover variable is ignored (with a startup advisory naming the successor role).

Anthropic (Claude) provider

Enable Claude models alongside Azure OpenAI -- they appear in the same selector and can be picked per turn. Anthropic is disabled by default; leaving ANTHROPIC_MODELS unset is a no-op.

Two hostings, selected by one variable:

# direct (Anthropic public API) | foundry (Anthropic on Azure AI Foundry)
ANTHROPIC_HOSTING=direct
ANTHROPIC_MODELS=claude-sonnet-4-5-20250929,claude-haiku-4-5

Direct hosting:

ANTHROPIC_API_KEY=sk-ant-...
# ANTHROPIC_BASE_URL=https://your-gateway.example.com   # optional proxy

Foundry hosting (Anthropic on Azure AI Foundry) -- supply exactly one endpoint and one auth method:

ANTHROPIC_HOSTING=foundry
ANTHROPIC_FOUNDRY_RESOURCE=my-aifoundry          # resource name (subdomain) only
# OR: ANTHROPIC_FOUNDRY_BASE_URL=https://my-aifoundry.services.ai.azure.com/anthropic/

# Auth A -- API key (sent as the api-key header):
ANTHROPIC_FOUNDRY_API_KEY=<foundry-key>
# Auth B -- Entra ID: leave the API key EMPTY and reuse AZURE_CREDENTIAL_MODE
#           (cli | managed-identity | default) + AZURE_TENANT_ID.

warning

ANTHROPIC_FOUNDRY_RESOURCE is the Azure AI Services resource name (the subdomain), not the Foundry project URL. For Entra ID auth, do not place a token in ANTHROPIC_FOUNDRY_API_KEY -- it is sent verbatim as the api-key header and returns HTTP 401.

Anthropic requires max_tokens on every request as a hard output cap:

ANTHROPIC_MAX_TOKENS=8192

Hosted web search works out of the box for Claude (web_search_20250305). Every other agent feature works on either provider as long as the model supports tool calling. Speech-to-text, text-to-speech, image generation, and RAG embedding run on their own dedicated Azure models, independent of the chat provider.

OpenAI (direct) provider

Enable models from OpenAI's own public API alongside Azure OpenAI and Anthropic -- they appear in the same selector and can be picked per turn. OpenAI is disabled by default; leaving OPENAI_MODELS unset is a no-op.

Authentication is by API key only:

OPENAI_MODELS=gpt-5.1
OPENAI_API_KEY=sk-...
# OPENAI_BASE_URL=https://your-gateway.example.com   # optional, OpenAI-compatible gateways

The current release supports reasoning models (for example gpt-5.x / o-series). They behave exactly like Azure OpenAI reasoning models -- same reasoning-effort and verbosity controls, hosted web search (country-scoped via WEB_SEARCH_COUNTRY), structured output, and background responses. Non-reasoning models (gpt-4o / gpt-4.1) are planned for a later release.

Microsoft Foundry provider

Enable reasoning models deployed in a Microsoft Foundry project alongside the other providers -- they appear in the same selector and can be picked per turn. Foundry is disabled by default; leaving FOUNDRY_MODELS unset is a no-op.

FOUNDRY_MODELS=gpt-5.1
FOUNDRY_PROJECT_ENDPOINT=https://<resource>.services.ai.azure.com/api/projects/<project>

Authentication is by Entra ID only, reusing the same Azure credential setup as Azure OpenAI: AZURE_CREDENTIAL_MODE (cli via az login, managed-identity on Azure-hosted compute, or default) plus the AZURE_TENANT_ID pin. There is no API-key option -- Foundry project endpoints authenticate with Entra ID; grant the signed-in identity a Foundry data-plane role (for example Azure AI User) on the project resource. AZURE_OPENAI_API_KEY does not apply to this provider.

Any Foundry chat-model deployment can be listed, and the generation controls match the model family: OpenAI reasoning deployments (gpt-5.x / o-series names) expose the same reasoning-effort and verbosity controls as Azure OpenAI reasoning models, while other families (for example DeepSeek) show no generation controls -- requests to them stay free of parameters those models reject. Web search (with citations) and structured output work across families. Background responses are not yet enabled for Foundry models in this release.

Native Foundry vs Claude-on-Foundry

This provider serves models natively from a Foundry project endpoint (FOUNDRY_*). It is separate from the Anthropic provider's Foundry hosting option (ANTHROPIC_HOSTING=foundry + ANTHROPIC_FOUNDRY_*), which serves Claude models through the Foundry Anthropic route.

Model ids must be unique across all providers (AZURE_OPENAI_MODELS, ANTHROPIC_MODELS, OPENAI_MODELS, FOUNDRY_MODELS); a collision is rejected at startup. The model selector lists Azure first, then Anthropic, then OpenAI, then Foundry, and the default model is the first configured Azure model when any is set.

Per-message generation options

Next to the model picker is a generation-options panel: one compact control for each option the selected model accepts. The available options, their allowed values, and defaults are served by the backend (GET /api/model, the model_options catalog) -- the UI renders only what the model advertises, so a control that a model cannot use never appears. There is no environment variable; each option's per-session choice persists across reloads, and the resolved values are shown next to the model name and token usage and persist with the session.

Reasoning effort

Sets how hard the model reasons, per message.

Provider	Levels	Default	Mechanism
Azure OpenAI (gpt-5.5 / gpt-5.4)	`low`, `medium`, `high`, `xhigh`	`medium`	`reasoning.effort`
OpenAI (direct, gpt-5.x / o-series)	`low`, `medium`, `high`, `xhigh`	`medium`	`reasoning.effort`
Microsoft Foundry (gpt-5.x / o-series deployments)	`low`, `medium`, `high`, `xhigh`	`medium`	`reasoning.effort`
Anthropic (Opus 4.8 / 4.7)	`low`, `medium`, `high`, `xhigh`, `max`	`xhigh`	adaptive thinking + `output_config.effort`

Verbosity (Azure OpenAI gpt-5.x)

Sets how terse or expansive the visible answer is, independent of how hard the model reasons. Offered only on Azure OpenAI gpt-5.x (low / medium / high, default medium); it maps to the OpenAI Responses API text.verbosity control. The default selection is output-neutral -- the request is unchanged until you pick a non-default value.

Why no temperature / top_p?

Both currently supported model families are reasoning / adaptive-thinking models. Anthropic Opus 4.7/4.8 reject temperature, top_p, and top_k with HTTP 400, and Azure OpenAI gpt-5.x reasoning models do not accept them either. The options catalog therefore does not offer sampling parameters for these models. The mechanism is general, though: when a future non-reasoning model is added, it can advertise temperature / top_p and the matching slider appears automatically.

Prompt caching (input-token cost)

Every model call re-sends a large, stable prefix -- the system prompt plus the full tool schemas. On a long turn (especially a coding tool loop, where one message fans out into many sequential model calls) that prefix is billed again and again. Prompt caching marks the stable prefix as cacheable so it is billed once and re-read cheaply on the following calls. It is output-transparent: the model's replies are identical -- only billing and latency change.

It is on by default and provider-agnostic:

Anthropic (Claude): the backend injects cache_control breakpoints on the system block (which also covers the tool definitions) and the last few conversation messages, so in an agentic/coding tool loop the conversation tail -- tool results, file contents, reasoning -- re-reads at a large discount instead of being re-billed at full price on every model call.
Azure OpenAI: prompt caching is automatic for prefixes of ~1024 tokens or more; nothing to configure.

For very long tool loops, the sliding-window history compaction shifts the conversation prefix and lowers the conversation cache hit-rate. If caching is your priority, raise COMPACTION_KEEP_LAST_GROUPS (keeps a wider stable window) and/or set ANTHROPIC_PROMPT_CACHE_TTL=1h (survives multi-minute pauses).

# Master toggle (default true). Set false to disable caching entirely.
PROMPT_CACHE_ENABLED=true

# Anthropic cache lifetime: 5m (default) or 1h (extended cache). Unknown -> 5m.
ANTHROPIC_PROMPT_CACHE_TTL=5m

When the active provider reports cache usage, the per-message token readout includes cache_read_input_tokens / cache_write_input_tokens, so you can see the savings directly.

tip

If credit usage still feels high on Claude, also lower the reasoning effort per message (the selector above). The Anthropic default is xhigh, which spends a lot of output tokens thinking on every step; medium or low is plenty for many coding edits.

Structured output (JSON)

Constrain the assistant's answer to JSON -- optionally to a specific JSON Schema -- so the reply is machine-readable instead of prose. It works across all base providers (Azure OpenAI, Anthropic, and OpenAI) and is a per-message opt-in: when it is off, requests are unchanged.

In the chat UI. A { } Structured Output toggle sits next to the model options above the input. Turn it on and the next answer comes back as JSON, rendered as a copyable code block. Two modes:

Generic JSON -- just toggle on; the model returns a well-formed JSON object.
Explicit schema -- open the small editor (the +schema / edit button) and paste a JSON Schema. The answer is constrained to your schema. An empty or invalid schema falls back to generic JSON.

The toggle is hidden for any model that does not support structured output, and a model that only supports a best-effort fallback is annotated as such.

What "no schema" means depends on the model

Turning the toggle on without writing a schema falls back to a default schema, and that default is a property of the provider:

Provider	Default when no schema is supplied
Azure OpenAI / OpenAI	Any JSON object (`{"type": "object", "additionalProperties": true}`)
Anthropic (Claude)	`{"answer": "..."}` -- an object with a single `answer` string

The difference is not a ChatWalaʻau choice: Anthropic's structured outputs require additionalProperties: false on every object, do not allow it to be omitted, and offer no way to request free-form JSON, so "any JSON object" cannot be expressed there. Rather than refuse the feature, those models fall back to a minimal valid shape -- and the schema editor states the exact shape you will get, so it is never a surprise.

Supply your own schema whenever the shape matters. An explicit schema always overrides the default, on every provider.

A schema that satisfies the strict rules works on every provider:

{
  "type": "object",
  "properties": { "answer": { "type": "string" } },
  "required": ["answer"],
  "additionalProperties": false
}

Every array needs "items", every object needs "additionalProperties": false, and every declared property must appear in "required". The editor lists any violation with the path of the offending sub-schema before you send.

note

Web search and structured output can't be used together (the providers reject that combination), so while structured output is on, web search is skipped for that turn. Other tools (coding, weather, and so on) keep working.

tip

Strict, native structured output (the default for the shipped models) guarantees the shape on the provider side, so a separate validation step is unnecessary. ChatWalaʻau still does a light, non-blocking check and shows a small JSON badge on the message -- amber if the output could not be parsed (for example a very long answer that was cut off). It never blocks or rewrites the answer.

Over the API. The OpenAI-compatible Responses API accepts the standard text.format field, so your existing OpenAI SDK code works unchanged -- see OpenAI-compatible API.

Reliability: transient provider errors

Model providers occasionally return a transient server error (HTTP 5xx) -- "The server had an error processing your request." In a coding session one question fans out into many sequential model calls, so the chance of hitting one at least once per turn is higher.

ChatWalaʻau handles this for you:

Automatic retry before any output. If the error happens before the answer has started streaming, the backend resends the request automatically (a small, bounded number of attempts with a short backoff). This is safe because nothing has been shown or saved yet, so the model simply restarts the turn. While it retries, the chat shows a brief "Temporary server error -- retrying..." notice so the run reads as in progress, not stalled.
No silent duplication. If the error happens after the answer has begun streaming, it is not retried (a retry could duplicate output). You get a clear "temporary server error -- please resend" message; just send again, or start a new chat if the conversation is very long.

Errors that a retry would not fix are reported distinctly and are never retried:

Condition	What you see	What to do
Out of credits / quota	A billing message naming the provider	Add credits / raise the quota, or switch to another configured model
Rate limited (429)	A rate-limit message	Wait a moment and retry, lower the reasoning effort, or raise the deployment quota
Transient 5xx after output	"temporary server error -- please resend"	Resend; start a new chat if the conversation is very long

Multi-model switching​

Model Offering Catalog​

When a deployment cannot serve a feature​

Claude on Microsoft Foundry: pick the right hosting option​

If you must stay on a restricted deployment​

The order models appear in​

Authoring offerings (CLI and GUI)​

Task model assignments​

Anthropic (Claude) provider​

OpenAI (direct) provider​

Microsoft Foundry provider​

Per-message generation options​

Reasoning effort​

Verbosity (Azure OpenAI gpt-5.x)​

Prompt caching (input-token cost)​

Structured output (JSON)​

What "no schema" means depends on the model​

Reliability: transient provider errors​

Multi-model switching

Model Offering Catalog

When a deployment cannot serve a feature

Claude on Microsoft Foundry: pick the right hosting option

If you must stay on a restricted deployment

The order models appear in

Authoring offerings (CLI and GUI)

Task model assignments

Anthropic (Claude) provider

OpenAI (direct) provider

Microsoft Foundry provider

Per-message generation options

Reasoning effort

Verbosity (Azure OpenAI gpt-5.x)

Prompt caching (input-token cost)

Structured output (JSON)

What "no schema" means depends on the model

Reliability: transient provider errors