Skip to main content

LLM providers

Qry talks to several model providers at once. Each conversation, notebook cell, dashboard assist and Forge translation names the model it uses, and users pick from the models an administrator has enabled for them. This page covers the admin side: connecting providers, enabling models and controlling cost.

Supported providers​

ProviderModelsHow Qry reaches them
AnthropicClaudeAnthropic API key, or Google Vertex AI, or Azure AI Foundry
OpenAIGPTOpenAI API key, or Azure AI Foundry
GoogleGeminiGemini API key, or Google Vertex AI
Vertex AI Model GardenOpen models published in Model GardenYour Google Cloud project's credentials
OpenAI-compatible endpointAny model served with the OpenAI protocolA base URL and key — for models you host

Reaching a model through your own cloud account (Vertex AI or Azure AI Foundry) keeps prompts under the agreements you already have with that cloud.

Connecting a provider​

All provider settings live in Admin › System configuration › AI Models.

Anthropic, OpenAI and Gemini APIs​

Paste the provider's API key. For an OpenAI-compatible server, also set OpenAI API Base URL.

Google Vertex AI​

Turn on Use Vertex AI and set the project and region. Gemini and Claude are then called through Vertex AI with the installation's Google Cloud identity; no API key is needed. The pod's identity needs the aiplatform.user role on that project.

Web search with Claude on Vertex AI​

The chat's web search runs, with Claude, as Anthropic's web_search tool. On Vertex AI that tool is governed by the organization policy constraint vertexai.allowedPartnerModelFeatures. Google denies the tool by default on Claude 4.6 and later, and on the Claude 4.5 models from 2 December 2026. Until the policy allows it, a web search with Claude fails with HTTP 400 FAILED_PRECONDITION, and the error names the value that is missing. Questions that do not search the web are not affected.

Allowing publishers/anthropic:web_search permits web search on every Claude model, including models released later, without enabling any other partner-model feature. Someone with roles/orgpolicy.policyAdmin sets it on the Vertex project:

cat > vertex-partner-features.yaml <<'EOF'
name: projects/<project>/policies/vertexai.allowedPartnerModelFeatures
spec:
rules:
- values:
allowedValues:
- publishers/anthropic:web_search
EOF
gcloud org-policies set-policy vertex-partner-features.yaml

This replaces the project's own policy for the constraint. If the project already allows other values, list them in allowedValues as well. Check the result with:

gcloud org-policies describe vertexai.allowedPartnerModelFeatures --project <project> --effective

The change applied within a minute when we tested it. If the Organization Policy API is not enabled on the project, add --billing-project with a project where it is enabled. Gemini's web search uses Google Search grounding, which this constraint does not govern.

Azure AI Foundry​

Turn on Enable Azure AI Foundry and give the Foundry Resource Name (the name, not a URL). One resource serves both GPT (OpenAI protocol) and Claude (Anthropic protocol).

  • Leave Foundry API Key empty to authenticate with the pod's Entra ID identity; it then needs the Cognitive Services User role on the resource.
  • Embedding Deployment and Whisper Deployment make the same resource the backend for text embeddings and dictation. Changing the embedding model means re-indexing what was indexed with the previous one.
  • The endpoint overrides are only for sovereign clouds or private endpoints.

Vertex AI Model Garden​

Turn on Enable Model Garden, and optionally set its project and region. It uses the existing Google Cloud credentials.

Defaults​

Default LLM Service and the default model of each provider are used when a call names no model — for example, a background summary.

Enabling models and deciding who uses them​

Admin › Model permissions holds the model catalogue:

  • Sync brings in the models a provider currently offers; Add model registers one by hand (a Model Garden model, or one behind an OpenAI-compatible endpoint).
  • Each model has its context window, output limit, pricing and capabilities (tools, images, reasoning). Pricing is what usage costs are computed from, so keep it filled in.
  • Disable a model to take it out of every picker.
  • Default permissions and per-group permissions decide which enabled models each group may use.

Some features take their own model — the notebook assistant, skill authoring, retrieval reranking — and have a setting for it in System configuration. A setting that names a model which is later disabled or removed falls back to another model; the operator CLI's qry models audit lists those settings.

Reliability​

Calls retry with exponential backoff on rate-limit and overload responses (429, 503, Anthropic's 529), using each provider SDK's own retry logic. There is no automatic switch to another provider: if a provider is down, the models it serves fail until it is back, and users can pick another model.

Long conversations​

Long conversations are compacted so they stay within the model's context window:

  • Layer 0 (truncation) — large tool results are kept in history as metadata plus five sample rows (default 8 KB); the full result stays available to the interface.
  • Layer 1 (server-side) — Anthropic's tool-use clearing drops old tool calls and results (default 120K tokens).
  • Layer 2 (client-side) — the conversation is summarised in one extra call (default 150K tokens), only when Layer 1 is not enough.

Most installations never need to change the defaults.

Cost controls​

  • Budgets — Admin › Budget management sets spend limits per user, per group or for the whole organisation, with alerts as they are approached.
  • Model permissions — keep the most expensive models for the groups that need them.
  • Scheduled task quotas — see Task quota templates.
  • Cost attribution — Admin › Cost attribution shows who and what spent it.

Common issues​

429 errors despite low traffic. The provider's rate limit is per account. If several installations share one account, they share the limit; raise it with the provider or split the accounts.

Compaction runs often. Raise the Layer 1 and Layer 2 thresholds, or lower the Layer 0 budget so history grows more slowly.

Answers change after switching model mid-conversation. The new model did not produce the earlier turns. Start a new conversation when consistency matters.

Vertex AI works locally but not from the cluster. The pod's identity is missing aiplatform.user on the Vertex project.

Web search with Claude fails with "Organization Policy constraint constraints/vertexai.allowedPartnerModelFeatures violated". The Vertex project does not allow Claude's web search tool. See Web search with Claude on Vertex AI.

Embedding changes do not take effect. The embedding service caches its configuration; restart the Celery workers after changing it — see Embedding configuration.

See also​

Running Qry without a model provider​

Qry also works with no model provider at all — for example an installation that only shares data products through Nexus. With none configured:

  • Qry opens on Nexus (or the Catalog if Nexus is turned off) instead of the chat.
  • What needs a model is hidden: Messages, Workspaces, Skills, Memory, Forge and ML Hub, and the AI buttons in Nexus, notebooks, dashboards and Lakeflow. Search (⌘K) finds by name and does not offer to ask Qry.
  • Data sources, the Catalog, notebooks, dashboards, Lakeflow jobs, Nexus with its assistants connector, users, groups and SSO work as usual.

To keep this mode even with a provider configured — a key set for something else, or AI not part of the contract — turn on Hide AI features in Admin → System Configuration → Application.

Configuring a provider later (with that setting off) brings everything back; nothing needs reinstalling.