Production-grade LLM routing with budget ceilings, tiered escalation, and multi-provider failover.

30-Second Demo

pip install tokenwise-llm

# Route a query to the best model within budget
tokenwise route "Debug this segfault" --strategy best_quality --budget 0.05

# Decompose a task, execute it, track spend
tokenwise plan "Write a Python function to validate email addresses, \
  then write unit tests for it" --budget 0.05 --execute

Example output:

Plan: 4 steps | Budget: $0.05 | Estimated: $0.0002

Status: Success | Total cost: $0.0007 | Budget remaining: $0.0493

If a step fails, TokenWise automatically escalates to a stronger model and retries within budget.

Why

Most LLM frameworks optimize for capability. TokenWise optimizes for cost governance. You declare a budget per request — TokenWise enforces it, selects the best model within that constraint, and escalates only when needed. No hidden traffic allocation, no implicit ceilings. All decisions are explicit and per-request.

Quick Start

Set your API key

export OPENROUTER_API_KEY="sk-or-..."

TokenWise uses OpenRouter as the default gateway. Set OPENAI_API_KEY, ANTHROPIC_API_KEY, or GOOGLE_API_KEY to bypass OpenRouter for those providers.

Route a query

tokenwise route "Write a haiku about Python"

Route with budget ceiling

tokenwise route "Debug this segfault" --strategy best_quality --budget 0.05

Plan and execute

tokenwise plan "Build a REST API" --budget 0.50 --execute

Inspect spend

tokenwise ledger --summary

Routing strategies

Budget is a universal parameter on all strategies. Pass budget_strict=False to fall back to best-effort.

Python API

from tokenwise import Router, Planner, Executor

# Route a single query (with structured trace)
router = Router()
model, trace = router.route_with_trace("Explain quantum computing", strategy="balanced", budget=0.10)
print(f"{model.id} (${model.input_price}/M input)")
print(f"Request: {trace.request_id}, state: {trace.termination_state.value}")

# Plan and execute a complex task
planner = Planner()
plan = planner.plan(task="Build a REST API for a todo app", budget=0.50)

executor = Executor()
result = executor.execute(plan)
print(f"Cost: ${result.total_cost:.4f}, success: {result.success}")

OpenAI-compatible proxy

tokenwise serve --port 8000

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
response = client.chat.completions.create(
    model="auto",  # TokenWise picks the best model
    messages=[{"role": "user", "content": "Hello!"}],
)

🦞 OpenClaw integration

Use TokenWise as a drop-in proxy for OpenClaw — see the integration guide.

Core Features

• Budget-aware routing — cost ceilings enforced via max_tokens caps with conservative estimation (details).
• Tiered escalation — budget, mid, flagship; escalates upward on failure, never downward. Monotonic mode available (TOKENWISE_ESCALATION_POLICY=monotonic).
• Capability-aware fallback — routes and fallbacks filtered by code, reasoning, math, or general.
• Task decomposition — LLM-powered planning with per-step model assignment and async DAG scheduling.
• Routing traces — structured RoutingTrace on every request with request_id, escalation records, termination state, and budget tracking.
• Risk gate — opt-in rule-based check that blocks destructive or ambiguous queries before routing (TOKENWISE_RISK_GATE_ENABLED=true).
• Cost ledger — structured per-call accounting including failures and retries, persisted to JSONL.
• Multi-provider failover — OpenRouter, OpenAI, Anthropic, and Google with connection pooling.

Benchmark: Cost–Quality Frontier

X-axis: average cost per task (USD). Y-axis: success rate (%). The star marks the TokenWise escalation strategy, which uses Router.route() with escalating strategies (cheapest → balanced → best_quality) and escalates to higher tiers when a task fails validation (max 1 escalation per tier). Baselines use a single fixed model for all tasks.

On this 20-task benchmark set, TokenWise Escalation is the only strategy reaching 100% success, while reducing average cost per task by ~5x versus Flagship Only.

Results from a single fixed-seed run. Model pricing and outputs may vary over time.

In multi-step workflows (10–50 steps), the per-step savings compound: a 5x reduction per step means 5x for the entire workflow. With more steps there are more opportunities for escalation to save on easy sub-tasks while still escalating when needed.

Success metric: 15 of 20 tasks have dedicated validators (reasoning correctness, code structure, substantiveness checks). The remaining 5 simple tasks use a length-check fallback (>20 chars). Validators are defined in benchmarks/strategy_pareto.py.

Budget: $0.03/task soft target — used for Router model filtering but not enforced as a hard ceiling.

How to reproduce

# Requires an OpenRouter API key (or direct provider keys)
export OPENROUTER_API_KEY="sk-or-..."

uv sync --group benchmark
uv run python benchmarks/strategy_pareto.py

Generated artifacts:

Use --dry-run to preview the plan without making API calls.

Limitations

• Small task set (20 tasks across 4 categories); not a comprehensive LLM benchmark.
• Validators are heuristic — they check for known correct patterns, not full semantic correctness.
• Model pricing and availability change over time; results are provider-dependent.
• Single-run results; no confidence intervals. The run date is printed in the script output.
• Escalation in the benchmark uses a validation-retry loop; in production, the Executor escalates on execution failure.

Comparison

Most routing tools optimize per-request model choice. TokenWise treats routing as a workflow-level control system.

High-level comparison (as of February 2026). Corrections welcome via issues.

How It Works

Architecture

┌───────────────────────────────────────────────────────┐
│                       TokenWise                       │
│                                                       │
│  ┌────────────┐  ┌────────────┐  ┌────────────┐       │
│  │   Router   │  │  Planner   │  │  Executor  │       │
│  │            │  │            │  │            │       │
│  │  1. Detect │  │  Breaks    │  │  Runs the  │       │
│  │  scenario  │  │  task into │  │  plan,     │       │
│  │  2. Route  │  │  steps +   │  │  tracks    │       │
│  │  within    │  │  assigns   │  │  spend,    │       │
│  │  budget    │  │  models    │  │  retries   │       │
│  └─────┬──────┘  └─────┬──────┘  └─────┬──────┘       │
│        │               │               │              │
│        └───────────────┼───────────────┘              │
│                        ▼                              │
│          ┌──────────────────────────┐                 │
│          │    ProviderResolver      │  ← LLM calls    │
│          │                          │                 │
│          │  OpenAI    · Anthropic   │                 │
│          │  Google    · OpenRouter  │                 │
│          └──────────────────────────┘                 │
│                                                       │
│            ┌──────────────┐                           │
│            │   Registry   │  ← metadata + pricing     │
│            └──────────────┘                           │
└───────────────────────────────────────────────────────┘

Router pipeline

The router uses a two-stage pipeline: detect (capabilities + complexity) then route (filter by budget, apply strategy: cheapest / balanced / best_quality).

Planner and Executor

Planner decomposes a task into subtasks using a cheap LLM, assigns the optimal model to each step within budget, and auto-downgrades expensive steps if over budget.

Executor runs the plan via async DAG scheduling, tracks actual cost via CostLedger, and escalates to stronger models on failure (flagship before mid, filtered by capability).

If executor.execute(plan) is called inside an existing event loop (Jupyter, FastAPI), it falls back to sequential execution. Use await executor.aexecute(plan) directly for concurrent DAG scheduling in async code.

Observability

Every execution produces a structured trace:

result = executor.execute(plan)

# Routing trace with termination state and escalation records
if result.routing_trace:
    rt = result.routing_trace
    print(f"Request: {rt.request_id}, state: {rt.termination_state.value}")
    print(f"Model: {rt.initial_model} -> {rt.final_model}")
    for esc in rt.escalations:
        print(f"  Escalated: {esc.from_model} -> {esc.to_model} ({esc.reason_code.value})")

# Per-step results and cost ledger
for sr in result.step_results:
    print(f"Step {sr.step_id}: model={sr.model_id}, "
          f"cost=${sr.actual_cost:.4f}, escalated={sr.escalated}")

for entry in result.ledger.entries:
    print(f"  {entry.reason}: {entry.model_id} "
          f"({entry.input_tokens}in/{entry.output_tokens}out) "
          f"${entry.cost:.6f} {'ok' if entry.success else 'FAIL'}")

print(f"Total: ${result.total_cost:.4f}, "
      f"wasted: ${result.ledger.wasted_cost:.4f}, "
      f"remaining: ${result.budget_remaining:.4f}")

Example output when step 1 fails and escalates:

Request: a3f1b2c4d5e6, state: completed
Model: openai/gpt-4.1-mini -> openai/gpt-4.1
  Escalated: openai/gpt-4.1-mini -> openai/gpt-4.1 (model_error)
Step 1: model=openai/gpt-4.1, cost=$0.0052, escalated=True
  step 1 attempt 1: openai/gpt-4.1-mini (82in/0out) $0.000000 FAIL
  step 1 escalation attempt 1: openai/gpt-4.1 (82in/204out) $0.001800 ok
Total: $0.0052, wasted: $0.0000, remaining: $0.9948

Budget Semantics

TokenWise enforces budget ceilings by capping max_tokens before each LLM call. Input token counts are estimated using a chars / 4 heuristic with a 1.2x safety margin — not a tokenizer. The budget ceiling is real and enforced, but small overruns are possible when the heuristic underestimates input tokens. A future release will support pluggable tokenizer-based estimation for stricter guarantees.

Known Limitations (v0.5)

All previous limitations have been resolved:

• Planner cost not budgeted — tracked and deducted (v0.4)
• Linear execution — parallel DAG scheduling (v0.4)
• No persistent spend tracking — JSONL ledger (v0.4)
• No routing audit trail — structured RoutingTrace (v0.5)
• No escalation policy control — monotonic mode (v0.5)

Configuration

TokenWise reads configuration from environment variables and an optional config file (~/.config/tokenwise/config.yaml).

# ~/.config/tokenwise/config.yaml
default_strategy: balanced
default_budget: 0.50
planner_model: openai/gpt-4.1-mini
escalation_policy: flexible     # or monotonic
trace_level: basic              # or verbose
risk_gate:
  enabled: false
  irreversible_block: true
  ambiguity_threshold: 0.9

Development

git clone https://github.com/itsarbit/tokenwise.git
cd tokenwise
uv sync
uv run pytest
uv run ruff check src/ tests/
uv run mypy src/

src/tokenwise/
├── models.py        # Pydantic data models
├── config.py        # Settings from env vars and config file
├── registry.py      # ModelRegistry — fetches/caches models
├── router.py        # Two-stage pipeline: scenario → strategy
├── planner.py       # Decomposes tasks, assigns models
├── executor.py      # Runs plans, tracks spend, escalates
├── risk_gate.py     # Rule-based risk gate (destructive/ambiguous queries)
├── ledger_store.py  # Persistent JSONL spend history
├── cli.py           # Typer CLI
├── proxy.py         # FastAPI OpenAI-compatible proxy
├── providers/       # LLM provider adapters
│   ├── openrouter.py
│   ├── openai.py
│   ├── anthropic.py
│   ├── google.py
│   └── resolver.py  # Maps model IDs → provider instances
└── data/
    └── model_capabilities.json

Philosophy

LLM systems should be treated like distributed systems. That means clear failure semantics, explicit cost ceilings, predictable escalation, and observability. TokenWise is designed with that philosophy.

Background reading: LLM Routers Are Not Enough — the blog post that motivated TokenWise's design.

License

MIT