AI
Workers AI
Run Workers AI text generation and embedding models on OpenOrange through OpenRouter or any OpenAI-compatible provider.
Runs on Any OpenAI-compatible provider (OpenRouter by default, or ORANGE_AI_BASE_URL)
- Works
run()with text generation, streams, tool calls, JSON output, and embeddings, plusmodels()- Differs
- Models run on the provider you choose, OpenRouter by default, and each
@cf/model maps to one of its models - Missing
- Image, speech, translation, and reranker models,
env.AI.gateway(),autorag(),aiSearch(), andtoMarkdown() - Migration
- Not needed Models run on the provider you configure.
- Configure
ai- Checks
- 1 pass
On OpenOrange, env.AI sends each model call to an OpenAI-compatible provider:
OpenRouter by default, another hosted API, or a model
on your machine. Your Workers AI code and Wrangler configuration run unchanged.
Quickstart
You need an OpenRouter API key, or the base URL of another OpenAI-compatible API.
-
Bind the AI binding
Add an
aientry towrangler.jsonc, as for Wrangler.{ "name": "workers-ai", "main": "src/index.ts", "compatibility_date": "2026-09-03", "ai": { "binding": "AI" } } -
Call a model
Call
env.AI.run()with a Workers AI model name, as on Cloudflare.interface Env { AI: Ai; } export default { async fetch(request, env): Promise<Response> { const { response } = (await env.AI.run("@cf/meta/llama-3.1-8b-instruct", { messages: [{ role: "user", content: await request.text() }], })) as { response?: string }; return new Response(response); }, } satisfies ExportedHandler<Env>; -
Set your provider key
Put the key in
.dev.vars. For OpenAI, Groq, a local Ollama, or another provider, see Providers.OPENROUTER_API_KEY=sk-or-...On a cluster, pass the same file with
orange deploy --secrets-file .dev.vars. -
Run it
orange devshows the binding on an OpenAI-compatible API.orange dev Binding Resource Runs on env.AI Workers AI OpenAI-compatible APIAsk a question. The Worker returns the model’s answer.
curl localhost:8787 -d 'What is a Durable Object?'
For streaming, tool calls, embeddings, and an OpenAI-shape model, see
examples/workers-ai.
How it works
env.AIis a port of workerd’s AI client. Eachrun()becomes a request to your provider’s/chat/completionsor/embeddingsAPI, with no Rivet Actor in between.- Each
@cf/model name maps to one provider model. Common models map to OpenRouter out of the box, andORANGE_AI_MODELSadds the rest. See Models. - Results and streams keep the shape that Cloudflare returns for each model,
and
orange devlogs each call with its model, status, and latency.
AI context Compatibility, limits and details
Written for LLMs and coding agents: every edge case, measurement, and known deviation. You do not need it to use the feature.
Compatibility
| Feature | Status | Notes |
|---|---|---|
run() text generation with prompt or messages | Supported | For each mapped model. |
stream: true | Supported | Server-Sent Events in the model’s shape. The framing is not checked against recordings from Cloudflare. |
tools and tool calls | Supported | |
response_format | Supported | response is the model’s JSON text. |
| Embeddings | Supported | { shape, data }, in input order. |
returnRawResponse: true | Supported | Returns the Response, also a failed one. |
models() | Partial | Lists only mapped models. Applies search and task, and ignores the other filters. |
The gateway option of run() | Partial | Goes to the provider directly, without gateway caching or logs. aiGatewayLogId is null, and orange dev warns once. |
Responses API input (input and instructions), as for gpt-oss | Not supported | Throws. Pass messages instead. |
| Image, speech, translation, and reranker models | Not supported | No mapping, so run() throws. |
env.AI.gateway(), autorag(), aiSearch(), and toMarkdown() | Not supported | Throw NotSupportedError. |
websocket: true, and ReadableStream or FormData inputs | Not supported | Throw NotSupportedError. |
new Ai() from cloudflare:ai | Not supported | Throws NotSupportedError. Use the ai binding. |
Three differences reach most projects:
- The text comes from another host, so answers differ from Cloudflare’s for the same model name. The result shape follows the Workers AI schema.
- A model without a mapping never falls back to another model.
run()throws until you map it inORANGE_AI_MODELS. - Cloudflare’s default
max_tokensis not applied. The provider’s default applies.
Limits
Orange sets no limits of its own on model calls.
| What | Limit | What happens over it |
|---|---|---|
| Requests and tokens | Your provider’s rate limits and quotas | run() throws InferenceUpstreamError with <HTTP status>: <provider message>, such as 429: .... With returnRawResponse: true, it returns the failed Response instead. |
Overlapping calls on one env.AI | None set by Orange | Each call keeps its own options. |
Providers
| Provider | Set in .dev.vars | Notes |
|---|---|---|
| OpenRouter (default) | OPENROUTER_API_KEY=<API key> | The built-in model mappings use OpenRouter model IDs. |
| OpenAI | ORANGE_AI_BASE_URL=https://api.openai.com/v1ORANGE_AI_API_KEY=<API key> | Map each model with ORANGE_AI_MODELS. |
| Groq | ORANGE_AI_BASE_URL=https://api.groq.com/openai/v1ORANGE_AI_API_KEY=<API key> | Map each model with ORANGE_AI_MODELS. |
| Together AI | ORANGE_AI_BASE_URL=https://api.together.ai/v1ORANGE_AI_API_KEY=<API key> | Map each model with ORANGE_AI_MODELS. |
| Fireworks AI | ORANGE_AI_BASE_URL=https://api.fireworks.ai/inference/v1ORANGE_AI_API_KEY=<API key> | Map each model with ORANGE_AI_MODELS. |
| Any OpenAI-compatible API | ORANGE_AI_BASE_URL=<base URL>ORANGE_AI_API_KEY=<API key> | The API must serve /chat/completions, and /embeddings for embedding models. |
| Ollama on your machine | ORANGE_AI_BASE_URL=http://127.0.0.1:11434/v1 | No key. Map each model to one that the server has. |
| LM Studio on your machine | ORANGE_AI_BASE_URL=http://127.0.0.1:1234/v1 | No key. Map each model to one that the server has. |
Another provider uses its own model IDs, so add a mapping in ORANGE_AI_MODELS
for each model that the Worker calls. For a local Ollama:
ORANGE_AI_BASE_URL=http://127.0.0.1:11434/v1
ORANGE_AI_MODELS='{"@cf/meta/llama-3.1-8b-instruct":{"model":"llama3.1:8b","task":"text"}}'Settings
Put the settings in .dev.vars or the process environment. A value in
.dev.vars wins.
| Variable | Default | Meaning |
|---|---|---|
ORANGE_AI_API_KEY | OPENROUTER_API_KEY | Bearer token for the provider. ORANGE_AI_API_KEY wins when both are set. |
ORANGE_AI_BASE_URL | https://openrouter.ai/api/v1 | Base URL of the OpenAI-compatible API. A trailing / is dropped. |
ORANGE_AI_MODELS | none | JSON that adds or replaces model mappings. |
- Without
ORANGE_AI_BASE_URL, a key is required. If no key is set,orange devprints one warning at startup, andrun()throws. See Errors. - With
ORANGE_AI_BASE_URLset, the key is optional. Without a key, Orange sends noAuthorizationheader, and prints no warning. orange doctorchecks eachaibinding. It reportsenv.AI · OpenRouterwith a key,env.AI · <base URL>withORANGE_AI_BASE_URL, and an action with neither.
Models
Each Workers AI model name maps to one provider model. A text model also has the result shape that Cloudflare returns for it:
legacy:{ response, tool_calls, usage }. Streams senddata: {"response":"..."}events.openai: an OpenAI chat completion,{ id, object, created, model, choices, usage }. Streams send OpenAIchat.completion.chunkevents.
| Workers AI model | Provider model | Task | Shape |
|---|---|---|---|
@cf/meta/llama-3.3-70b-instruct-fp8-fast | meta-llama/llama-3.3-70b-instruct | Text | legacy |
@cf/meta/llama-3.1-8b-instruct | meta-llama/llama-3.1-8b-instruct | Text | legacy |
@cf/meta/llama-3.1-8b-instruct-fast | meta-llama/llama-3.1-8b-instruct | Text | legacy |
@cf/zai-org/glm-4.7-flash | z-ai/glm-4.7-flash | Text | openai |
@cf/moonshotai/kimi-k2.7-code | moonshotai/kimi-k2.7-code | Text | openai |
@cf/openai/gpt-oss-120b | openai/gpt-oss-120b | Text | openai |
@cf/openai/gpt-oss-20b | openai/gpt-oss-20b | Text | openai |
@cf/baai/bge-base-en-v1.5 | baai/bge-base-en-v1.5 | Embeddings | |
@cf/baai/bge-large-en-v1.5 | baai/bge-large-en-v1.5 | Embeddings | |
@cf/baai/bge-m3 | baai/bge-m3 | Embeddings |
The shapes come from workerd’s model types. GLM 4.7 Flash and Kimi K2.7 Code
return ChatCompletionsOutput. gpt-oss returns it for messages input.
OpenRouter does not serve @cf/baai/bge-small-en-v1.5, so it has no mapping.
ORANGE_AI_MODELS merges over this table. Each entry needs a provider model
and a task, text or embeddings. shape is optional and defaults to
legacy:
ORANGE_AI_MODELS='{"@cf/meta/llama-3-8b-instruct":{"model":"meta-llama/llama-3-8b-instruct","task":"text"},"@cf/zai-org/glm-5.2":{"model":"z-ai/glm-5.2","task":"text","shape":"openai"}}'Map an embedding model only to the same model on the provider. Another model returns vectors that do not match the vectors you stored before.
A bad ORANGE_AI_MODELS fails every call. run() throws
InferenceUpstreamError with 3000: <message>, and models() throws
AiInternalError with the message:
- A value that is not JSON:
ORANGE_AI_MODELS is not valid JSON: <cause>. - A value that is not a JSON object:
ORANGE_AI_MODELS must be a JSON object that maps Workers AI model names to {"model", "task"}. - An entry with another shape:
ORANGE_AI_MODELS entry "<name>" must be {"model": "<provider model>", "task": "text" | "embeddings", "shape"?: "legacy" | "openai"}.
Requests and results
env.AI sends workerd’s AI requests to an in-process backend, which
translates them for the provider:
| Call | Provider request | Result |
|---|---|---|
run() with a legacy text model | POST {base}/chat/completions | { response, tool_calls, usage } |
run() with an openai text model | POST {base}/chat/completions | The provider’s chat completion |
run() with stream: true | Same, with stream: true | text/event-stream |
run() with an embedding model | POST {base}/embeddings, with text as input | { shape, data } |
models() | none | The mapped models, each described as Served by OpenOrange as <provider model>. |
- A
promptbecomes one user message. - For a
legacymodel,messages,max_tokens,temperature,top_p,top_k,seed,frequency_penalty,presence_penalty,repetition_penalty,tools, andresponse_formatare forwarded. Legacy tools,{ name, description, parameters }, are wrapped as OpenAI{ type: "function", function }tools. Ajson_schemaformat is wrapped underjson_schema.schemawith the nameresponse. - For an
openaimodel, every input field exceptpromptis forwarded unchanged. - An embedding
textis a string or an array of strings. - In the
legacyshape, each tool call has both the{ name, arguments }shape and the{ id, type, function }shape, as Workers AI declares. - A
legacystream sendsdata: {"response":"..."}events. One last event holds the tool calls and usage, thendata: [DONE]. - An
openaistream forwards each provider chunk, thendata: [DONE]. - Comment lines in the provider’s stream, such as OpenRouter’s
: OPENROUTER PROCESSING, are dropped. - Libraries that read Workers AI streams, such as
workers-ai-provider, parse both shapes. orange devlogs each call to stderr as[ai] <Workers AI model> -> <provider model> <status> <latency>ms.
Errors
A provider or backend error throws InferenceUpstreamError with
<code>: <message>, unless the call passed returnRawResponse: true.
| Situation | What the Worker sees |
|---|---|
No key, and no ORANGE_AI_BASE_URL | 10000: Workers AI needs an API key for https://openrouter.ai/api/v1. Set ORANGE_AI_API_KEY or OPENROUTER_API_KEY in .dev.vars or the environment. At startup, orange dev prints the same text with Calls to env.AI.run() throw until it is set. |
| A model without a mapping | 5007: Model "@cf/baai/bge-small-en-v1.5" has no OpenOrange mapping. Add it to ORANGE_AI_MODELS. No provider call is made. |
| The provider answers with an error | <HTTP status>: <provider message>, such as 429: Rate limit exceeded: free-models-per-day. |
| The provider cannot be reached | 3000: Could not reach the AI provider at <base URL>: <cause> |
| The provider fails during a stream | The stream fails with Error: The AI provider failed during the stream: <message>, not an InferenceUpstreamError. |
Responses API input (input) without messages or prompt | 5006: OpenOrange does not support Responses API input (`input`) yet. Pass `messages` instead. |
Text generation without prompt or messages | 5006: Text generation needs a `prompt` string or a `messages` array. |
Embeddings without text | 5006: Text embeddings need a `text` string or array of strings. |
env.AI.gateway() | NotSupportedError: env.AI.gateway() is not supported by OpenOrange. Pass the `gateway` option to env.AI.run() instead. OpenOrange ignores gateway caching and logs. |
autorag(), aiSearch(), toMarkdown() | NotSupportedError, with Query your vector store with fetch()., Query your search index with fetch()., or Convert documents with a library in the Worker. |
websocket: true | NotSupportedError: Workers AI WebSocket mode is not supported by OpenOrange. Call run() without `websocket` to use the provider over HTTP. |
new Ai() from cloudflare:ai | NotSupportedError: cloudflare:ai Ai is not implemented by OpenOrange because it needs Cloudflare's AI service. Call the model provider's HTTP API with fetch(). |
Deviations
These add to the differences under Compatibility.
- An
openaimodel returns the provider’s chat completion as it is. Itsidandmodelfields are the provider’s values. lastRequestIdisnull, andgetLogs()returns an empty array.- Each
run()keeps its own options. workerd stores them on the binding while a request is in flight, so overlapping calls could see each other’s.
Conformance
tests/bindings/ai-binding.test.tsruns a Worker underorange devagainst a mock provider, which is a real HTTP server. It covers both result shapes, tool calls, streaming in both shapes, provider429and500errors, a provider error during a stream, embeddings, thegatewayoption, unmapped models, a missing OpenRouter key, a local provider without a key, and overlapping calls with different options.tests/examples/runnable-examples.test.tsrunsexamples/workers-aiagainst a mock provider: chat, streaming, a tool call, embeddings, anopenai-shape model,returnRawResponse,models(), a provider error, and a provider that is down.tests/cli/doctor.test.tschecks the binding’s row inorange doctor.- The generated
Aitype is not registered in the type-conformance checks. - No test calls a real provider.
Recorded results
Recorded results from the conformance run of 2026-11-10 (Orange ea40348), 1,670 of 1,851 tests passing across every area. Each check is works when every result passed and partial when some did. The full feature matrix is on the compatibility page.
Workers AI + AI Gateway
| Check | Status | Results |
|---|---|---|
| Workers AI | Works |
|