Skip to content
Reference / OpenTelemetry and Prometheus

Reference

OpenTelemetry and Prometheus

Export traces with OpenTelemetry, scrape Prometheus metrics, and read the logs of a deployed Worker.

Each Durable Object runs as a Rivet Actor, so Rivet’s traces, metrics, and logs cover your objects. You turn them on with environment variables, and your code does not change.

Inside orange serve, your Worker calls your Durable Objects, which run as Rivet Actors. RivetKit sends their traces over OTLP to a collector such as Jaeger, Tempo, or Honeycomb, and Prometheus scrapes the runner's metrics endpoint.

Inside orange serve, your Worker calls your Durable Objects, which run as Rivet Actors. RivetKit sends their traces over OTLP to a collector such as Jaeger, Tempo, or Honeycomb, and Prometheus scrapes the runner's metrics endpoint.

Quickstart

You need a Rivet Cloud cluster and an OTLP collector, such as Jaeger, Tempo, or Honeycomb.

  1. Send traces to your collector

    orange deploy \
      --var OTEL_SERVICE_NAME:my-app \
      --var OTEL_EXPORTER_OTLP_TRACES_ENDPOINT:https://collector.example.com/v1/traces
  2. Find an object's spans

    Each request and RPC call to a Durable Object is one span. Its rivet.actor.key is the object’s ID.

  3. Open metrics and logs

    On Rivet Cloud these need no setup. The Rivet dashboard shows your cluster’s metrics, and its Logs page shows your Worker’s output.

Try it locally

orange dev reads the same variables from your shell.

export OTEL_SERVICE_NAME=my-app
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://collector.example.com/v1/traces
orange dev

Self-host

Pass the same variables to the container. Set RIVETKIT_METRICS_PORT to serve Prometheus metrics, and scrape them from a sidecar in the same pod.

docker run \
  -e OTEL_SERVICE_NAME=my-app \
  -e OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://collector.example.com/v1/traces \
  -e RIVETKIT_METRICS_PORT=9464 \
  -e RIVET_ENDPOINT=... my-worker

Autoscale dynamic Workers

Scale runners on orange_worker_loader_slot_utilization and the slot queue wait, not on CPU. See Dynamic Worker metrics for an example.

Learn more

AI context Spans, metrics, and log output in detail

Written for LLMs and coding agents: every edge case, measurement, and known deviation. You do not need it to use the feature.

Traces

  • With an OTLP endpoint set, RivetKit exports a span for each Durable Object request and RPC call, with the SQLite work inside it. The Worker’s own handlers get no spans.
  • Each span carries rivet.actor.name, which is the class in kebab case and a short hash, such as greeter-5c8cd02c, and rivet.actor.key, which is the object’s ID.
  • --var also puts the variable in the Worker’s env. See Variables and secrets.
  • If the collector needs an API key in OTEL_EXPORTER_OTLP_HEADERS, pass it with --secrets-file, not --var.

Metrics

RivetKit serves Prometheus metrics at http://127.0.0.1:$RIVETKIT_METRICS_PORT/metrics.

  • Rivet Cloud sets RIVETKIT_METRICS_PORT on each runner, collects the metrics, and shows them in the Rivet dashboard. orange deploy refuses to set it with --var.
  • Self-hosted: the endpoint listens on 127.0.0.1 only, so docker run -p does not expose it. Scrape it from the same network namespace, such as a sidecar in the same Kubernetes pod.
  • orange dev serves metrics when RIVETKIT_METRICS_PORT is set, from its first Worker.
  • With several Workers, only the first Worker’s runtime serves metrics, so they cover that Worker’s objects only.
MetricWhat it shows
rivet_rivetkit_actor_active_countAwake actors, by actor_name
rivet_rivetkit_actor_http_requests_activeRequests and RPC calls in progress
rivet_rivetkit_actor_inbox_depthWork waiting to run

Dynamic Worker metrics

A Worker with a worker_loaders binding adds these to the same endpoint.

MetricTypeWhat it shows
orange_worker_loader_workersgaugeDynamic Workers that hold a slot
orange_worker_loader_workers_maxgaugeORANGE_WORKER_LOADER_MAX_WORKERS
orange_worker_loader_slot_utilizationgaugeSlots in use divided by slots, from 0 to 1
orange_worker_loader_queue_depthgaugeDynamic Workers that wait for a slot
orange_worker_loader_queue_wait_secondshistogramHow long each dynamic Worker waited for a slot
orange_worker_loader_start_secondshistogramFrom a slot to a loaded dynamic Worker
orange_worker_loader_vm_boot_secondshistogramHow long one agentOS VM takes to boot
orange_worker_loader_pool_claims_totalcounterVM claims by mode (network, deny) and result: hit (prewarmed), warming (waited for a prewarm), miss (booted on demand)
orange_worker_loader_pool_readygaugePrewarmed VMs by mode
orange_worker_loader_vmsgaugeagentOS VMs by state: live, pooled, booting, stopping
orange_worker_loader_vms_maxgaugeORANGE_WORKER_LOADER_MAX_VMS
orange_worker_loader_vm_queue_depthgaugeDynamic Workers with a slot that wait for room in the VM budget
orange_worker_loader_rejections_totalcounterWorkerLoaderLimitError refusals by code: queue_full, queue_timeout
orange_runtime_resident_memory_bytesgaugeRSS by process: runner, and on Linux children, which includes the agentOS sidecar
orange_runtime_memory_limit_bytesgaugeThe runner’s cgroup memory limit, when it has one

Useful queries:

# Pool miss rate
sum(rate(orange_worker_loader_pool_claims_total{result="miss"}[5m]))
  / sum(rate(orange_worker_loader_pool_claims_total[5m]))
# Slot queue wait, p95
histogram_quantile(0.95, sum by (le) (rate(orange_worker_loader_queue_wait_seconds_bucket[5m])))
# Memory headroom
sum(orange_runtime_resident_memory_bytes) / orange_runtime_memory_limit_bytes

The agentOS sidecar’s own request queue is not visible to Orange, so it has no metric.

Autoscaling

With a Kubernetes HorizontalPodAutoscaler and a Prometheus adapter:

  • Scale on orange_worker_loader_slot_utilization, with a target of about 0.7, and on the p95 of orange_worker_loader_queue_wait_seconds. A busy slot can mean a dynamic Worker that waits on I/O, so CPU does not show dynamic Worker demand. Do not scale on CPU.
  • Set behavior.scaleDown.stabilizationWindowSeconds longer than ORANGE_WORKER_LOADER_IDLE_TIMEOUT_MS (30 s by default), such as 300 s. An idle dynamic Worker holds its slot until the timeout, so a shorter window scales down while slots still look busy, and then scales up again.
  • Alert on orange_worker_loader_rejections_total. Each refusal is a 503 or a WorkerLoaderLimitError your Worker caught.
  • Size the pod’s memory limit for ORANGE_WORKER_LOADER_MAX_VMS VMs, and watch orange_runtime_resident_memory_bytes against orange_runtime_memory_limit_bytes.
  • Orange has no readiness endpoint, so a full runner still receives requests. Its 503s carry Retry-After.

Logs

  • orange dev prints each request and your Worker’s console output.
  • orange serve prints the console output and RivetKit’s warnings and errors.
  • Self-hosted, read the container’s output.
  • Set RIVET_LOG_LEVEL=info to add RivetKit’s info logs.