API Cost Accounting¶

ScoreBench shows an API-equivalent model cost for each priced run. This is the cost implied by token usage and the standard synchronous public API price table embedded in the server. It is useful for comparing strategies under a common budget, but it is not an invoice and does not attempt to value a Claude Max, ChatGPT, enterprise, or other subscription plan.

The token and cost views deliberately count cache reads differently. Working tokens exclude cached-input reads so repeatedly loading the same context does not look like new work. API-equivalent cost includes those reads at the model's cached-input price. Cache writes remain part of working tokens and use their separate price where the provider publishes one.

The dashboard and Export Studio support Cost on the x-axis. Lower-cost candidate snapshots appear earlier on that axis, so the score chart answers "what quality did this run reach for this API budget?" Export Studio's Best candidate view can compare one best-scoring point per run by either working tokens or cost.

Price Table¶

The current table is version 2026-08-13. Prices are USD per million tokens. A dash means the provider does not publish a separately billed cache-write category for that model; cache misses use the regular input price.

Sources: OpenAI model documentation, Anthropic pricing, xAI pricing, Kimi K3 pricing, and Z.AI pricing.

The standard table deliberately excludes long-context surcharges, regional uplifts, batch discounts, priority or fast-mode charges, tools, infrastructure, taxes, and subscription economics. For example, provider-specific long-context pricing is not inferred from aggregate run counters.

Derivation Order¶

ScoreBench chooses the strongest available measurement for every candidate:

1. A runner-reported cost_usd, when present.
2. A cumulative input/output/cache token breakdown priced with the table above.
3. A final run-level breakdown allocated to candidates in proportion to each candidate's cumulative working-token total.
4. A legacy aggregate estimate using 80% input and 20% output tokens:

cost = working_tokens * (0.8 * input_rate + 0.2 * output_rate) / 1,000,000

Methods 3 and 4 are estimates. A partial token breakdown is also an estimate when a separately priced cache category is missing. The UI prefixes every estimated value with ~, shows the derivation in candidate details, and leaves unknown or composite model names unpriced instead of guessing.

Aggregate estimates cannot recover historical cache reads and can materially understate cache-heavy agent runs. They are intended to preserve a useful historical comparison until new runs provide full token categories; they are not backfilled into the database.

Runner Contract¶

For the best cost accuracy, report cumulative run-relative categories on every submission and in the final harness run usage event:

harness submit candidate.py \
  --input-tokens 120000 \
  --output-tokens 6000 \
  --cache-creation-tokens 4000 \
  --cache-read-tokens 900000 \
  --total-tokens 130000 \
  --tokens-total-source launcher_usage

For canonical disjoint counters, --input-tokens means fresh input and --cache-read-tokens means cached input. OpenAI/Codex instead reports input_tokens as an inclusive total. Pass that raw total as --input-tokens, its cached subset as --cached-input-tokens, and any cache-write subset as --cache-creation-tokens; the CLI subtracts both subsets before sending the canonical breakdown. Do not pass both cache-read flags, pre-normalize an inclusive count, or fabricate a split.

Grok's native inputTokens and totalTokens both include cached reads. Grok runs must send disjoint fresh input, output, and cache-read counters; the server rejects an aggregate-only Grok snapshot because it cannot recover working tokens from that value. The ScoreBench skill's token_usage.py --grok-jsonl path parses the active Grok session safely. For other providers, when only the aggregate is available, continue sending the exact aggregate and let the dashboard mark its cost estimate.

Maintenance¶

Prices live in challenge_harness/model_pricing.py; the report payload includes the pricing version and source URLs. Updating prices requires a code review, test changes, and a version-date bump. Report regeneration uses the currently deployed table, so the displayed figure is a current-price comparison rather than historical provider billing at the candidate's submission date.