LLM Gateway

All LLM traffic of agent runs goes through the gateway (ADR-0011: LLM Gateway for Credentials and Usage Metering): it authenticates the run, injects the provider credential configured in Core, and forwards the request. Agents never hold provider keys.

Anthropic API

The gateway serves the Anthropic API under /v1/ (e.g. POST /v1/messages), including streaming (server-sent events are forwarded as they arrive).

Authentication

A Ballet run token with audience gateway and capability llm.invoke, in Authorization: Bearer <token> or x-api-key: <token>. The token’s cust/proj claims select the credential: the project’s override, else the organization default (REST API, LLM credentials).

Configuring an agent

Point the agent at the gateway and give it the run token as its API key, e.g. for Claude Code:

ANTHROPIC_BASE_URL=http://gateway:8082
ANTHROPIC_API_KEY=<run token>
Errors

In the Anthropic error format ({"type": "error", "error": {"type", "message"}}):

Status

error.type

Cause

401

authentication_error

Missing or invalid run token

403

permission_error

Token lacks llm.invoke, or no credential is configured

502

api_error

Credential lookup or provider unreachable

Errors returned by the provider itself are passed through unchanged.

Embeddings

POST /v1/embeddings is OpenAI-compatible ({"model", "input": string | [string]} → {"object": "list", "model", "data": [{"index", "embedding"}], "usage"}) and serves hybrid search (ADR-0021: Hybrid Vector Search over All Content).

Models

hash-256 (default, [embeddings] default_model) is computed by the gateway itself: deterministic feature hashing of words and word pairs into 256 dimensions. It needs no provider and captures lexical overlap only — suitable for development, tests and offline use. Any other model is forwarded to the organization’s openai credential (OpenAI or any OpenAI-compatible API via base_url).

Callers

Run tokens with llm.invoke (scope from the token), or service tokens with llm.embed (Knowledge, Core indexing), which must name the organization in X-Ballet-Organization (and optionally X-Ballet-Project) for credentials and metering. kit/embed provides the Gateway client and the Hash embedder.

Usage metering

The gateway reads token usage from every response — the usage object of JSON responses, or message_start / message_delta events of streams, without buffering them — and attributes it to the run token’s organization, project and ticket:

  • Prometheus (gateway /metrics): ballet_llm_tokens_total (labels organization, project, model, token_type = input|output|cache_read|cache_write) and ballet_llm_requests_total (organization, project, model, status).

  • Core: records are delivered in batches every 2 seconds to POST /internal/v1/usage (usage.write) and kept while Core is unreachable (up to 100 000). Per-ticket and per-model totals are available at GET /api/v1/projects/{project}/usage (REST API).

Trying it locally

core/cmd/devtoken mints run tokens from Core’s key file, and gateway/cmd/fake-anthropic serves a fake Messages API with usage (both development only). Point a project credential at the fake (base_url http://127.0.0.1:9900, key sk-fake) to test without a real provider:

go build -o bin/devtoken ./core/cmd/devtoken
TOKEN=$(bin/devtoken -keys data/token-keys.json -organization acme -project WEB \
  -ticket WEB-1 -aud gateway -caps llm.invoke)
curl localhost:8082/v1/messages -H "x-api-key: $TOKEN" \
  -H "anthropic-version: 2023-06-01" -H "content-type: application/json" \
  -d '{"model":"claude-haiku-4-5","max_tokens":16,"messages":[{"role":"user","content":"hi"}]}'