Skip to content
AI / Workers AI

AI

Workers AI

Run Workers AI text generation and embedding models on OpenOrange through OpenRouter or any OpenAI-compatible provider.

Pluggable

Runs on Any OpenAI-compatible provider (OpenRouter by default, or ORANGE_AI_BASE_URL)

Works
run() with text generation, streams, tool calls, JSON output, and embeddings, plus models()
Differs
Models run on the provider you choose, OpenRouter by default, and each @cf/ model maps to one of its models
Missing
Image, speech, translation, and reranker models, env.AI.gateway(), autorag(), aiSearch(), and toMarkdown()
Migration
Not needed Models run on the provider you configure.
Configure
ai
Checks
1 pass

On OpenOrange, env.AI sends each model call to an OpenAI-compatible provider: OpenRouter by default, another hosted API, or a model on your machine. Your Workers AI code and Wrangler configuration run unchanged.

Quickstart

You need an OpenRouter API key, or the base URL of another OpenAI-compatible API.

  1. Bind the AI binding

    Add an ai entry to wrangler.jsonc, as for Wrangler.

    {
      "name": "workers-ai",
      "main": "src/index.ts",
      "compatibility_date": "2026-09-03",
      "ai": { "binding": "AI" }
    }
  2. Call a model

    Call env.AI.run() with a Workers AI model name, as on Cloudflare.

    interface Env {
      AI: Ai;
    }
    
    export default {
      async fetch(request, env): Promise<Response> {
        const { response } = (await env.AI.run("@cf/meta/llama-3.1-8b-instruct", {
          messages: [{ role: "user", content: await request.text() }],
        })) as { response?: string };
        return new Response(response);
      },
    } satisfies ExportedHandler<Env>;
  3. Set your provider key

    Put the key in .dev.vars. For OpenAI, Groq, a local Ollama, or another provider, see Providers.

    OPENROUTER_API_KEY=sk-or-...

    On a cluster, pass the same file with orange deploy --secrets-file .dev.vars.

  4. Run it

    orange dev shows the binding on an OpenAI-compatible API.

    orange dev
      Binding   Resource     Runs on
      env.AI    Workers AI   OpenAI-compatible API

    Ask a question. The Worker returns the model’s answer.

    curl localhost:8787 -d 'What is a Durable Object?'

For streaming, tool calls, embeddings, and an OpenAI-shape model, see examples/workers-ai.

How it works

  • env.AI is a port of workerd’s AI client. Each run() becomes a request to your provider’s /chat/completions or /embeddings API, with no Rivet Actor in between.
  • Each @cf/ model name maps to one provider model. Common models map to OpenRouter out of the box, and ORANGE_AI_MODELS adds the rest. See Models.
  • Results and streams keep the shape that Cloudflare returns for each model, and orange dev logs each call with its model, status, and latency.
AI context Compatibility, limits and details

Written for LLMs and coding agents: every edge case, measurement, and known deviation. You do not need it to use the feature.

Compatibility

FeatureStatusNotes
run() text generation with prompt or messagesSupportedFor each mapped model.
stream: trueSupportedServer-Sent Events in the model’s shape. The framing is not checked against recordings from Cloudflare.
tools and tool callsSupported
response_formatSupportedresponse is the model’s JSON text.
EmbeddingsSupported{ shape, data }, in input order.
returnRawResponse: trueSupportedReturns the Response, also a failed one.
models()PartialLists only mapped models. Applies search and task, and ignores the other filters.
The gateway option of run()PartialGoes to the provider directly, without gateway caching or logs. aiGatewayLogId is null, and orange dev warns once.
Responses API input (input and instructions), as for gpt-ossNot supportedThrows. Pass messages instead.
Image, speech, translation, and reranker modelsNot supportedNo mapping, so run() throws.
env.AI.gateway(), autorag(), aiSearch(), and toMarkdown()Not supportedThrow NotSupportedError.
websocket: true, and ReadableStream or FormData inputsNot supportedThrow NotSupportedError.
new Ai() from cloudflare:aiNot supportedThrows NotSupportedError. Use the ai binding.

Three differences reach most projects:

  • The text comes from another host, so answers differ from Cloudflare’s for the same model name. The result shape follows the Workers AI schema.
  • A model without a mapping never falls back to another model. run() throws until you map it in ORANGE_AI_MODELS.
  • Cloudflare’s default max_tokens is not applied. The provider’s default applies.

Limits

Orange sets no limits of its own on model calls.

WhatLimitWhat happens over it
Requests and tokensYour provider’s rate limits and quotasrun() throws InferenceUpstreamError with <HTTP status>: <provider message>, such as 429: .... With returnRawResponse: true, it returns the failed Response instead.
Overlapping calls on one env.AINone set by OrangeEach call keeps its own options.

Providers

ProviderSet in .dev.varsNotes
OpenRouter (default)OPENROUTER_API_KEY=<API key>The built-in model mappings use OpenRouter model IDs.
OpenAIORANGE_AI_BASE_URL=https://api.openai.com/v1
ORANGE_AI_API_KEY=<API key>
Map each model with ORANGE_AI_MODELS.
GroqORANGE_AI_BASE_URL=https://api.groq.com/openai/v1
ORANGE_AI_API_KEY=<API key>
Map each model with ORANGE_AI_MODELS.
Together AIORANGE_AI_BASE_URL=https://api.together.ai/v1
ORANGE_AI_API_KEY=<API key>
Map each model with ORANGE_AI_MODELS.
Fireworks AIORANGE_AI_BASE_URL=https://api.fireworks.ai/inference/v1
ORANGE_AI_API_KEY=<API key>
Map each model with ORANGE_AI_MODELS.
Any OpenAI-compatible APIORANGE_AI_BASE_URL=<base URL>
ORANGE_AI_API_KEY=<API key>
The API must serve /chat/completions, and /embeddings for embedding models.
Ollama on your machineORANGE_AI_BASE_URL=http://127.0.0.1:11434/v1No key. Map each model to one that the server has.
LM Studio on your machineORANGE_AI_BASE_URL=http://127.0.0.1:1234/v1No key. Map each model to one that the server has.

Another provider uses its own model IDs, so add a mapping in ORANGE_AI_MODELS for each model that the Worker calls. For a local Ollama:

ORANGE_AI_BASE_URL=http://127.0.0.1:11434/v1
ORANGE_AI_MODELS='{"@cf/meta/llama-3.1-8b-instruct":{"model":"llama3.1:8b","task":"text"}}'

Settings

Put the settings in .dev.vars or the process environment. A value in .dev.vars wins.

VariableDefaultMeaning
ORANGE_AI_API_KEYOPENROUTER_API_KEYBearer token for the provider. ORANGE_AI_API_KEY wins when both are set.
ORANGE_AI_BASE_URLhttps://openrouter.ai/api/v1Base URL of the OpenAI-compatible API. A trailing / is dropped.
ORANGE_AI_MODELSnoneJSON that adds or replaces model mappings.
  • Without ORANGE_AI_BASE_URL, a key is required. If no key is set, orange dev prints one warning at startup, and run() throws. See Errors.
  • With ORANGE_AI_BASE_URL set, the key is optional. Without a key, Orange sends no Authorization header, and prints no warning.
  • orange doctor checks each ai binding. It reports env.AI · OpenRouter with a key, env.AI · <base URL> with ORANGE_AI_BASE_URL, and an action with neither.

Models

Each Workers AI model name maps to one provider model. A text model also has the result shape that Cloudflare returns for it:

  • legacy: { response, tool_calls, usage }. Streams send data: {"response":"..."} events.
  • openai: an OpenAI chat completion, { id, object, created, model, choices, usage }. Streams send OpenAI chat.completion.chunk events.
Workers AI modelProvider modelTaskShape
@cf/meta/llama-3.3-70b-instruct-fp8-fastmeta-llama/llama-3.3-70b-instructTextlegacy
@cf/meta/llama-3.1-8b-instructmeta-llama/llama-3.1-8b-instructTextlegacy
@cf/meta/llama-3.1-8b-instruct-fastmeta-llama/llama-3.1-8b-instructTextlegacy
@cf/zai-org/glm-4.7-flashz-ai/glm-4.7-flashTextopenai
@cf/moonshotai/kimi-k2.7-codemoonshotai/kimi-k2.7-codeTextopenai
@cf/openai/gpt-oss-120bopenai/gpt-oss-120bTextopenai
@cf/openai/gpt-oss-20bopenai/gpt-oss-20bTextopenai
@cf/baai/bge-base-en-v1.5baai/bge-base-en-v1.5Embeddings
@cf/baai/bge-large-en-v1.5baai/bge-large-en-v1.5Embeddings
@cf/baai/bge-m3baai/bge-m3Embeddings

The shapes come from workerd’s model types. GLM 4.7 Flash and Kimi K2.7 Code return ChatCompletionsOutput. gpt-oss returns it for messages input. OpenRouter does not serve @cf/baai/bge-small-en-v1.5, so it has no mapping.

ORANGE_AI_MODELS merges over this table. Each entry needs a provider model and a task, text or embeddings. shape is optional and defaults to legacy:

ORANGE_AI_MODELS='{"@cf/meta/llama-3-8b-instruct":{"model":"meta-llama/llama-3-8b-instruct","task":"text"},"@cf/zai-org/glm-5.2":{"model":"z-ai/glm-5.2","task":"text","shape":"openai"}}'

Map an embedding model only to the same model on the provider. Another model returns vectors that do not match the vectors you stored before.

A bad ORANGE_AI_MODELS fails every call. run() throws InferenceUpstreamError with 3000: <message>, and models() throws AiInternalError with the message:

  • A value that is not JSON: ORANGE_AI_MODELS is not valid JSON: <cause>.
  • A value that is not a JSON object: ORANGE_AI_MODELS must be a JSON object that maps Workers AI model names to {"model", "task"}.
  • An entry with another shape: ORANGE_AI_MODELS entry "<name>" must be {"model": "<provider model>", "task": "text" | "embeddings", "shape"?: "legacy" | "openai"}.

Requests and results

env.AI sends workerd’s AI requests to an in-process backend, which translates them for the provider:

CallProvider requestResult
run() with a legacy text modelPOST {base}/chat/completions{ response, tool_calls, usage }
run() with an openai text modelPOST {base}/chat/completionsThe provider’s chat completion
run() with stream: trueSame, with stream: truetext/event-stream
run() with an embedding modelPOST {base}/embeddings, with text as input{ shape, data }
models()noneThe mapped models, each described as Served by OpenOrange as <provider model>.
  • A prompt becomes one user message.
  • For a legacy model, messages, max_tokens, temperature, top_p, top_k, seed, frequency_penalty, presence_penalty, repetition_penalty, tools, and response_format are forwarded. Legacy tools, { name, description, parameters }, are wrapped as OpenAI { type: "function", function } tools. A json_schema format is wrapped under json_schema.schema with the name response.
  • For an openai model, every input field except prompt is forwarded unchanged.
  • An embedding text is a string or an array of strings.
  • In the legacy shape, each tool call has both the { name, arguments } shape and the { id, type, function } shape, as Workers AI declares.
  • A legacy stream sends data: {"response":"..."} events. One last event holds the tool calls and usage, then data: [DONE].
  • An openai stream forwards each provider chunk, then data: [DONE].
  • Comment lines in the provider’s stream, such as OpenRouter’s : OPENROUTER PROCESSING, are dropped.
  • Libraries that read Workers AI streams, such as workers-ai-provider, parse both shapes.
  • orange dev logs each call to stderr as [ai] <Workers AI model> -> <provider model> <status> <latency>ms.

Errors

A provider or backend error throws InferenceUpstreamError with <code>: <message>, unless the call passed returnRawResponse: true.

SituationWhat the Worker sees
No key, and no ORANGE_AI_BASE_URL10000: Workers AI needs an API key for https://openrouter.ai/api/v1. Set ORANGE_AI_API_KEY or OPENROUTER_API_KEY in .dev.vars or the environment. At startup, orange dev prints the same text with Calls to env.AI.run() throw until it is set.
A model without a mapping5007: Model "@cf/baai/bge-small-en-v1.5" has no OpenOrange mapping. Add it to ORANGE_AI_MODELS. No provider call is made.
The provider answers with an error<HTTP status>: <provider message>, such as 429: Rate limit exceeded: free-models-per-day.
The provider cannot be reached3000: Could not reach the AI provider at <base URL>: <cause>
The provider fails during a streamThe stream fails with Error: The AI provider failed during the stream: <message>, not an InferenceUpstreamError.
Responses API input (input) without messages or prompt5006: OpenOrange does not support Responses API input (`input`) yet. Pass `messages` instead.
Text generation without prompt or messages5006: Text generation needs a `prompt` string or a `messages` array.
Embeddings without text5006: Text embeddings need a `text` string or array of strings.
env.AI.gateway()NotSupportedError: env.AI.gateway() is not supported by OpenOrange. Pass the `gateway` option to env.AI.run() instead. OpenOrange ignores gateway caching and logs.
autorag(), aiSearch(), toMarkdown()NotSupportedError, with Query your vector store with fetch()., Query your search index with fetch()., or Convert documents with a library in the Worker.
websocket: trueNotSupportedError: Workers AI WebSocket mode is not supported by OpenOrange. Call run() without `websocket` to use the provider over HTTP.
new Ai() from cloudflare:aiNotSupportedError: cloudflare:ai Ai is not implemented by OpenOrange because it needs Cloudflare's AI service. Call the model provider's HTTP API with fetch().

Deviations

These add to the differences under Compatibility.

  • An openai model returns the provider’s chat completion as it is. Its id and model fields are the provider’s values.
  • lastRequestId is null, and getLogs() returns an empty array.
  • Each run() keeps its own options. workerd stores them on the binding while a request is in flight, so overlapping calls could see each other’s.

Conformance

  • tests/bindings/ai-binding.test.ts runs a Worker under orange dev against a mock provider, which is a real HTTP server. It covers both result shapes, tool calls, streaming in both shapes, provider 429 and 500 errors, a provider error during a stream, embeddings, the gateway option, unmapped models, a missing OpenRouter key, a local provider without a key, and overlapping calls with different options.
  • tests/examples/runnable-examples.test.ts runs examples/workers-ai against a mock provider: chat, streaming, a tool call, embeddings, an openai-shape model, returnRawResponse, models(), a provider error, and a provider that is down.
  • tests/cli/doctor.test.ts checks the binding’s row in orange doctor.
  • The generated Ai type is not registered in the type-conformance checks.
  • No test calls a real provider.
Recorded results

Recorded results from the conformance run of 2026-11-10 (Orange ea40348), 1,670 of 1,851 tests passing across every area. Each check is works when every result passed and partial when some did. The full feature matrix is on the compatibility page.

Workers AI + AI Gateway
Check Status Results
Workers AI Works
  • Pass: example: workers-ai