Harness Middleware Protocol¶

The harness is the source of truth for A/B experiments. Agents do not talk to Tensara, HighLoad, Kaggle, or other exercise connectors directly. They talk to harnessd; harnessd owns connector credentials, candidate numbering, immutable artifacts, submissions, score payloads, usage metadata, and reports.

For an end-to-end operator and agent workflow guide, start with docs/harness-guide.md. This file is the lower-level HTTP and middleware contract.

Roles¶

• expctl: controller CLI for run setup, credential setup, usage snapshots, and reports.
• harnessd: local HTTP middleware. It authenticates agents with per-arm bearer tokens or scoped exercise tokens and submits to the configured connector.
• harness: lightweight agent CLI. It packages candidate files and calls harnessd.
• Venue connector: one connector per platform/deployment, such as local_tensara, public_tensara, highload, cpumode, gpumode, vliw, or paradigm_puzzles.

Agent Context¶

The preferred agent path starts in the Harness web UI. The user creates an exercise API key bound to one user, credential profile, connector, and exercise, then passes only:

export HARNESS_URL=https://scorebench.dev/
export HARNESS_RUN_TOKEN=hrun_...

When an agent runs inside an older harness workspace, the CLI can still discover its local middleware URL and scoped arm token from .harness/context.json:

harness context

The token is not printed by harness context. HARNESS_URL and HARNESS_ARM_TOKEN remain supported as legacy launcher overrides, but new agent sessions should use HARNESS_RUN_TOKEN.

Venue API keys, cookies, browser sessions, and rate-limit state stay on the middleware side. In isolated Docker mode, credential files are mounted only into the harnessd service, not the agent services.

Harness UI users are local username/password records. The Account page can add users, and its Exercise API Keys section binds each key to the signed-in user who creates it. The agent receives only the exercise API key, not the user's UI password.

Agent Commands¶

harness context
harness exercise
harness exercise --exercise sum_of_prime_numbers
harness run start --id run001 \
  --skills scorebench \
  --model gpt-5-codex \
  --coding-harness Codex \
  --effort high \
  --autonomy autonomous
harness run current
harness run progress
harness submit path/to/solution --notes "short result note" --total-tokens 126000 --tokens-total-source agent_claim
harness submit path/to/solution --exercise sum_of_prime_numbers --notes "short result note" --total-tokens 126000 --tokens-total-source agent_claim
harness invalidate --reason "exploit: caches venue inputs instead of solving generally"
harness invalidate <candidate_id> --reason "bug: undefined behavior produced an invalid score"
harness best
harness history

--exercise also accepts the alias --problem. It is request scoped: it tells the connector which problem to read or submit for that one call, without mutating the run configuration or any other arm. If omitted, the middleware uses the run's configured default exercise.

Run access is created in the web UI. The run name can either be pre-bound in the web UI or left blank for the agent to choose. A blank run name creates an exercise-scoped token whose server-side scope includes:

{
  "user_name": "admin",
  "connector": "highload",
  "exercise": "sum_of_prime_numbers",
  "credential_profile": "skill-research",
  "needs_run_name": true
}

The agent must start or continue a run before the first submission:

harness run start --id run001 \
  --strategy "progress logging skill with perf access" \
  --hypothesis "perf-guided changes should reduce score faster" \
  --skills scorebench,problem-agnostic-optimization \
  --model gpt-5-codex \
  --coding-harness Codex \
  --effort high \
  --autonomy autonomous

--model and --effort are required: harness run start fails without them (supplied on the command line, or inherited from a pre-bound exercise API key created with --model/--effort) so every run is attributable in reports. --coding-harness records the actual agent interface separately from the model: normally Claude uses Claude Code, GPT/OpenAI uses Codex, DeepSeek uses Deep Code, Grok uses Grok Build, Kimi (including model ID k3) uses Kimi Code, and GLM uses ZCode. Known families are inferred for legacy clients; crossed/custom setups must set it explicitly. A Claude model inside Codex records Codex. --strategy, --hypothesis, --label, --skills, and --autonomy are optional; --skills and --autonomy are strongly encouraged and produce non-fatal warnings when missing.

Use --prompt or --prompt-file to persist the original run instructions for the dashboard. Pre-bound exercise API keys may carry the same original_prompt metadata, and harness admin launch records its goal and additional prompt automatically. Prompt text is stored once on the logical run, not duplicated on every candidate.

On GPU-backed connectors, harness run start --gpu <gpu> (for example --gpu H100 on tensara) declares the GPU the whole run targets. Submissions in that run inherit it as their connector gpu option, and a conflicting harness submit --gpu is rejected so a run never mixes GPUs. Without a run-level GPU the connector default applies (T4 for tensara connectors), and the GPU actually used is still recorded per candidate for report filtering.

If a previous run already exists for the same user/profile/exercise, the harness rejects the start request and names the previous run. Continue it with that run id, or retry with --confirm-new-run only when the user explicitly wants a new independent run.

If the web UI creates the token with a run name, the token is already scoped to that run and needs_run_name is false. The agent should use harness run current to inspect it rather than switching it to a different run.

harness submit requires a cumulative token snapshot in the same call. total_tokens is not optional, and it must be run-relative unless a supervised runner records a run-start baseline. Before optimization work starts or resumes, the agent must send a run ping so reports can derive elapsed time from server timestamps and remove dead time between sessions:

harness run ping --event start --note "starting work"
harness run ping --event resume --note "resuming work"

The ping records a middleware timestamp only. It is not an agent claim about thinking time.

At the end of a run, or before handing control back to the user, record the final run-level usage measurement. Reports prefer this final run usage over per-submission inference:

harness run usage \
  --total-tokens 10643192 \
  --input-tokens 301811787 \
  --cached-input-tokens 292001024 \
  --output-tokens 832429 \
  --reasoning-output-tokens 329446 \
  --usage-source codex_usage \
  --usage-confidence exact \
  --tokens-total-source final_goal_usage

If only total_tokens is available, send that for providers whose aggregate is a working-token count. Do not invent missing input/output breakdown fields. Grok is an explicit exception: its aggregate includes cached reads, so a Grok run must use the ScoreBench skill's token_usage.py --grok-jsonl parser and send fresh input, output, and cache-read counters.

harness submit sol.cu \
  --exercise sum_of_prime_numbers \
  --label baseline \
  --notes "first viable kernel" \
  --idempotency-key baseline-001 \
  --input-tokens 120000 \
  --output-tokens 6000 \
  --total-tokens 126000 \
  --tokens-delta 126000 \
  --usage-source agent_reported \
  --usage-confidence estimated \
  --tokens-total-source agent_claim \
  --meta session_id=abc123 \
  --meta hypothesis=baseline

total_tokens is mandatory. Token deltas and cost are accepted only as measurements, not as trusted facts. Agent-reported timing fields are rejected; use harness run ping instead.

Working token total¶

Reports and dashboards use a working token total — input_tokens + output_tokens + cache_creation_tokens — as the headline token count. Cache reads are re-reads of already-counted context and are deliberately excluded; counting them inflates the total by roughly conversation-length times turn-count. Send the categories distinctly:

harness submit sol.cu \
  --input-tokens 57578 \
  --output-tokens 437510 \
  --cache-creation-tokens 1191658 \
  --cache-read-tokens 998452654 \
  --total-tokens 1000139400 \
  --tokens-total-source agent_claim

When the input/output breakdown is present the harness computes the working total itself (recording tokens_total_source: harness_working_tokens), so cache reads in --total-tokens never inflate what the dashboard shows; the cache-read count is retained for the per-candidate breakdown and API-equivalent cost. If you can supply only --total-tokens, the harness uses it as-is — it cannot un-inflate a total it did not break down or recover cached reads for pricing. For that reason, aggregate-only usage is rejected when the active run model is Grok.

The canonical fields are provider-agnostic: with --cache-read-tokens, --input-tokens must be the uncached, non-cache-write input. Anthropic already reports disjoint categories. OpenAI/Codex instead reports an inclusive input_tokens count. Pass that raw total with --cached-input-tokens and any cache-write subset with --cache-creation-tokens; the CLI converts them to the canonical disjoint fields. Do not pass both cache-read flags. In every case the working total excludes cache reads, while API-equivalent cost includes them at the model's cached-input rate. Grok also reports inclusive input, but its native field names and per-turn semantics differ; use the skill's --grok-jsonl parser rather than translating totalTokens manually.

Submission resource limits¶

Every run is protected by server-side candidate, artifact, token, and rate budgets. GET /context returns the effective submission_limits so workers and supervisors can inspect the contract before submitting. Defaults are:

submission_limits:
  max_candidates_per_run: 1000
  max_bundle_bytes_per_run: 2147483648
  max_total_tokens_per_run: 100000000
  rate_window_seconds: 600
  max_submissions_per_window: 20
  same_content_window_seconds: 600
  max_same_content_submissions_per_window: 1
  token_warning_floor: 5000000
  token_warning_per_elapsed_hour: 10000000

An experiment can override these positive values at the top level of experiment.yaml. Candidate, artifact, and token exhaustion returns HTTP 409; the rolling run and same-content rate limits return HTTP 429 with Retry-After and a structured rate_limit object. Both windows are derived from persisted candidate timestamps, so daemon restarts do not reset them. Checks run under the same service lock as candidate allocation. An exact retry with a previously accepted idempotency key is resolved before the budgets, so a client can safely recover a lost response without consuming another candidate or venue submission. The same-content window applies only when a new idempotency key would create a new candidate for an unchanged bundle. When a connector records an explicit cooldown with nextSubmissionAt, that venue timestamp releases the bundle instead of adding a second middleware wait.

The token ceiling applies to the normalized working total. When input/output categories are supplied, cache reads remain recorded but do not count against the token budget. Aggregate-only totals cannot be corrected this way. Totals that are below the hard limit but unusually high for the run's elapsed time are accepted with token_usage: suspect and a trust warning instead of silently distorting comparisons.

HTTP Endpoints¶

The agent / coding-harness pairing bootstrap is the only unauthenticated worker endpoint:

• POST /api/pair with JSON { "code": "PZL-..." } redeems a short-lived, single-use code and returns one pre-bound scoped run token. The server stores only the code hash, rate-limits attempts, and returns the token only in the successful response. The scorebench pair CLI command saves it in the local user config with mode 0600, scoped to the selected workspace.

Creating, listing, monitoring, and revoking pairings requires the owning browser session. Once paired, the worker uses the standard bearer-token protocol below; pairing does not weaken or broaden its scope.

All arm endpoints require:

Authorization: Bearer <HARNESS_RUN_TOKEN>

Legacy arm tokens use the same bearer header.

Endpoints:

• GET /exercise: exercise statement and connector metadata visible to this arm. For HighLoad this includes languages, defaultLanguage, and defaultFilenames.
• GET /exercise?exercise=<problem_id>: request-scoped exercise statement override for connectors that support multiple problems.
• GET /context: token-bound user, credential profile, connector, exercise, run context, and effective submission limits.
• POST /run/start: legacy arm-token endpoint to start or reactivate the current agent run.
• POST /run/ping: record a server timestamp for run start/resume/heartbeat; required before submit.
• GET /run/current: read the active agent run for this arm.
• GET /run/progress: canonical latest-submission active time, elapsed time, token total, sources, measurement timestamps, and current submission allowance for the exact run-token scope. This read does not create an activity heartbeat.
• POST /run/trace: upload one sanitized gzip NDJSON trace artifact for the exact run-token scope.
• GET /run/traces: list trace artifacts for the exact run-token scope.
• GET /run/trace?trace_id=<id>: download one scoped gzip trace artifact. Trace uploads and reads do not create activity heartbeats.
• GET /best: best scored candidate visible to this arm.
• GET /history: candidates for this arm only.
• GET /solutions: connector-visible solution list for the scoped exercise.
• GET /leaderboard: connector-visible leaderboard for the scoped exercise.
• GET /public-solution?solution_id=<id>: connector-visible metadata for a public or website-visible solution id. This is distinct from GET /solution.
• GET /challenge-page?section=<section>: connector-visible challenge section, such as HighLoad leaderboard or generators.
• GET /solve-form: redacted connector solve-form defaults, such as HighLoad compiler, compiler flags, availableLanguages, and defaultFilenames.
• GET /solution?solution_id=<id>: solution details only for connector ids submitted by the current run.
• POST /submit: immutable candidate submission.
• POST /invalidate: mark a visible candidate invalid without deleting its bundle, score payload, logs, or connector evidence.
• POST /reinstate: restore a visible invalidated candidate by appending a linked audit event.

GET /run/progress¶

GET /run/progress requires a run-scoped bearer token. Legacy arm tokens are rejected because they do not identify one authoritative run. Existing run tokens with own-history:read remain compatible; newly created tokens also carry run-progress:read.

The response is designed for supervisors and agents that need accounting progress without parsing dashboard HTML or treating /best as the latest measurement:

{
  "scope": {
    "kind": "run_token",
    "run_id": "run001"
  },
  "progress": {
    "schema_version": 1,
    "run_id": "run001",
    "active_seconds": 14764.6,
    "elapsed_seconds": 15120.7,
    "tokens_total": 4780000,
    "active_seconds_source": "server_timestamps_with_run_pings",
    "elapsed_seconds_source": "server_timestamps",
    "tokens_total_source": "claude_code_jsonl",
    "measured_at": "2026-07-29T10:20:30Z",
    "tokens_measured_at": "2026-07-29T10:21:00Z",
    "latest_candidate_measured_at": "2026-07-29T10:15:00Z",
    "idle_gap_seconds": 900,
    "idle_gap_removed_seconds": 0,
    "latest_candidate_id": "candidate-id",
    "latest_candidate_status": "accepted",
    "candidate_count": 33
  },
  "submission": {
    "can_submit": true,
    "blocked_by": null,
    "retry_after_seconds": 0,
    "next_submission_at": null,
    "general": {
      "window_seconds": 600.0,
      "limit": 20,
      "used": 2,
      "remaining": 18,
      "can_submit": true
    },
    "same_content": {
      "window_seconds": 600.0,
      "limit": 1,
      "checked": false
    }
  }
}

active_seconds and elapsed_seconds are measured through the latest trusted candidate or run ping, identified by measured_at; they do not advance merely because a watcher polls this endpoint. latest_candidate_measured_at remains separate so callers can see when timing extends beyond the last submission. Active time uses the same 15-minute unsupported-gap cap and run-ping heuristic as reports. A later run usage event may provide a newer token measurement, so tokens_measured_at is also separate.

This endpoint is intentionally different from /best. /best describes the best-scoring candidate and therefore exposes time-to-best, which can be older than the run's latest trusted progress. A run with start/activity pings can accumulate measured time before its first candidate. With neither candidates nor trusted pings, active and elapsed time are zero, measured_at is null, and their source is no_submitted_candidates.

Run trace artifacts¶

Run traces are post-run observability artifacts. Discovery, normalization, redaction, size limiting, and compression belong to the installed ScoreBench skill; the server only validates and stores the completed gzip. Trace operations require a bound, started run token. Legacy arm tokens are rejected.

Upload the gzip body directly:

POST /run/trace HTTP/1.1
Authorization: Bearer hrun_...
Content-Type: application/gzip
Content-Length: ...
X-Scorebench-Trace-Id: trace_<first-32-hex-of-sha256>
X-Scorebench-Trace-Sha256: <full-sha256>

The first uncompressed NDJSON record is a bounded manifest:

{
  "format": "scorebench-run-trace",
  "schema_version": 1,
  "provider": "codex",
  "session_id": "session-id",
  "event_count": 128,
  "normalization": {
    "private_reasoning_included": false
  }
}

The trace ID is content-derived, so retrying the same upload is idempotent. Compressed content is limited to 50 MiB and uncompressed content to 64 MiB. Each run can retain at most three traces and 128 MiB of compressed trace data in aggregate. Replaying an already stored trace is still allowed after those limits are reached. The server validates the complete gzip stream, its hash, and the manifest. It rejects artifacts claiming to contain private reasoning. Trace files live outside SQLite, candidate bundles, and generated report payloads.

Master-token endpoint:

• POST /usage_snapshot: external launcher usage snapshot. Requires X-Harness-Admin-Token.

POST /submit Schema¶

Required fields:

{
  "run_id": "run001",
  "bundle_b64": "<base64 tar.gz>",
  "bundle_sha256": "<sha256 hex>",
  "manifest": {
    "root": ".",
    "files": []
  }
}

run_id is mandatory for raw HTTP clients. The CLI fills it from GET /run/current. With HARNESS_RUN_TOKEN, the value must match the server-side run bound to the token.

Optional fields:

{
  "exercise": "sum_of_prime_numbers",
  "notes": "short human note",
  "label": "baseline",
  "connector_options": {
    "compiler": "gcc_cpp",
    "compiler_options": "-O3 -march=native",
    "language": "cpp",
    "system": "raptor_cove_p"
  },
  "idempotency_key": "baseline-001",
  "metadata": {
    "session_id": "abc123",
    "hypothesis": "baseline"
  },
  "usage": {
    "input_tokens": 120000,
    "output_tokens": 6000,
    "total_tokens": 126000,
    "tokens_delta": 126000,
    "cost_usd": 1.25,
    "source": "agent_reported",
    "confidence": "estimated",
    "tokens_total_source": "agent_claim"
  }
}

exercise may also be sent as problem, metadata.exercise, or metadata.problem by minimal clients. The middleware copies it into candidate metadata as both exercise and problem_id, includes it in the immutable request hash, and passes it to the connector for that submission only.

connector_options is optional and narrowly scoped to the selected connector for this submission. Supported keys are compiler, compiler_options, language, solution_file, system, gpu, leaderboard, submission_mode, profile_brev, and benchmark_index; top-level aliases with the same names are accepted for simple clients. Unknown connector option keys are rejected. For CPU.mode, valid compiler ids are currently rustc, clang_cpp, gcc_cpp, and clang_asm. For HighLoad, compiler, compiler_options, and language map directly to the website solve form fields compiler, compilerArgs, and lang. The language is the venue language id for the current candidate, such as CPP, RUST, GO, CSHARP, or ZIG; callers should inspect harness solve-form instead of assuming C++. Changing only compiler flags is a legitimate new candidate and should use a new idempotency key.

HighLoad's built-in language ids are CPP, GO, RUST, CSHARP, and ZIG. Connector config may add additional safe ids through languages, langs, or language_ids. harness solve-form --language RUST requests the upstream /solve/RUST form for the scoped exercise and returns the selected language, available language ids, default filenames, compiler id, and compiler arguments.

The middleware also injects normalized run metadata into every accepted candidate:

{
  "agent_run_id": "run001",
  "agent_run_label": "montgomery p31 baseline",
  "agent_run_strategy": "deterministic Miller-Rabin with Montgomery multiplication",
  "agent_run_hypothesis": "small-prime filtering plus Montgomery arithmetic lowers score without correctness loss",
  "agent_run_candidate_seq": 1,
  "agent_run_candidate_id": "run001-c001",
  "agent_run_candidate_timestamp": "2026-06-22T12:00:00Z"
}

candidate_seq and the immutable candidate storage id remain arm-global so database rows and artifacts never change identity. agent_run_candidate_seq restarts at 1 for each run, and reports use it for human-facing candidate labels such as c001-baseline. Older candidates without run metadata fall back to the arm-global sequence.

If a request tries to provide conflicting run metadata, the middleware rejects it instead of silently rewriting it.

Top-level usage keys are accepted as aliases for usage.* for simple clients. usage.total_tokens or top-level total_tokens is required; submissions without it are rejected before candidate allocation and connector submission.

Trust Model¶

The harness records two categories of data:

• Authoritative middleware data: timestamp, arm, candidate sequence, bundle hash, artifact path, connector stdout/stderr, raw score payload, and status.
• Reported data: token counts, cost, and arbitrary metadata.
• Derived middleware data: elapsed time from server timestamps, run pings, candidate submission timestamps, and API-equivalent cost estimates from the versioned public list-price table.

Reported data is never silently promoted to authoritative. The submit response and reports include trust fields:

• trusted: exact or parsed data from a launcher/middleware/API meter.
• agent_reported: supplied by the solving agent.
• suspect: accepted for auditability, but outside the elapsed-time sanity envelope and visibly warned in reports.
• missing: not supplied. New submissions are rejected when total token usage is missing, so this value only appears in older runs or non-token fields.

Sanity checks either reject unsafe requests before candidate allocation or add warnings without discarding the candidate:

• missing or mismatched run id is rejected before candidate allocation.
• missing idempotency key.
• agent-reported timing fields are rejected.
• total_tokens lower than the previous recorded token total.
• tokens_delta inconsistent with the previous token total.
• total_tokens unusually high for elapsed run time (accepted as suspect).
• same content hash submitted under a new candidate.
• candidate label reused with different content.

If an idempotency key is reused with the same content hash and the same exercise scope, the middleware returns the original candidate instead of resubmitting. If the key is reused with different content or a different exercise, the request is rejected.

Negative numeric usage values are rejected.

POST /invalidate Schema¶

Agents can invalidate only candidates visible to their current token scope. This is for submissions that were accepted by the connector but later found to be exploity, invalid, or based on a false assumption. Invalidation is not deletion: the middleware keeps the immutable bundle, score rows, raw connector payload, usage metadata, and trace evidence, and appends an audit row.

The candidate status becomes invalidated. Invalidated candidates remain in history and report exports for auditability, but are excluded from best, best-so-far curves, promotion decisions, and dashboard winner calculations.

Request:

{
  "candidate_id": "profile_local_tensara_josu_cand_0533",
  "reason": "exploit: memoizes exact matrix inputs instead of general multiplication",
  "metadata": {
    "class": "exploit"
  }
}

candidate_id is optional. If omitted, Harness invalidates the latest candidate visible to the token's run scope. reason is required and should state why the candidate should no longer count.

CLI:

harness invalidate \
  profile_local_tensara_josu_cand_0533 \
  --reason "exploit: memoizes exact matrix inputs instead of general multiplication" \
  --meta class=exploit

Response:

{
  "candidate": {"id": "profile_local_tensara_josu_cand_0533", "status": "invalidated"},
  "invalidation": {
    "invalidated_at": "2026-07-03T12:00:00Z",
    "invalidated_by": "admin",
    "reason": "exploit: memoizes exact matrix inputs instead of general multiplication"
  },
  "best": {}
}

Invalidated candidates remain in history, TSV exports, logs, and dashboards with status INVALIDATED, but they are excluded from best, best-so-far curves, promotion decisions, and winner calculations.

GET /history includes an audit object on every candidate row. For an invalidated candidate it contains the reason, actor, timestamp, metadata, and any later reinstatement, so a worker can distinguish a venue failure from an explicit correctness decision.

Concrete operational example:

harness invalidate profile_local_tensara_josu_cand_0533 \
  --reason "device-side exact input comparison and cached output reuse is not a valid general square matrix multiplication submission"

For the regenerated local_tensara / square-matmul dashboard, that candidate is shown as INVALIDATED, and the run falls back to profile_local_tensara_josu_cand_0529 at 8078.938681941922 us.

POST /reinstate Schema¶

Reinstatement reverses the candidate's current exclusion without removing or editing the invalidation record. The candidate must be visible to the current token and currently invalidated.

{
  "candidate_id": "profile_local_tensara_josu_cand_0533",
  "reason": "contract correction: the documented input domain permits this specialization",
  "metadata": {
    "class": "contract_correction"
  }
}

Harness restores the status captured in the invalidation metadata. For a legacy invalidation that predates status capture, it conservatively infers scored, failed, or submitted from preserved score evidence. An operator can pass restore_status explicitly when that inference is insufficient.

harness reinstate profile_local_tensara_josu_cand_0533 \
  --reason "contract correction: the documented input domain permits this specialization" \
  --meta class=contract_correction

The response contains the restored candidate, the append-only reinstatement event, and the newly computed scoped best.

Deterministic Outputs¶

expctl report <run_dir> writes:

• reports/report.json: complete structured data.
• reports/report.csv: candidate table for spreadsheets.
• reports/progress.tsv: canonical all-arm progress log:

timestamp   experiment_id   run_id  agent_id    variant exercise    problem_id  candidate   score   score_unit  decision    status  tokens_total    tokens_delta    elapsed_seconds server_elapsed_seconds  exercise_submission_id  code_sha256 label   warnings

• reports/progress-details.tsv: all-arm debug log with token, elapsed-time source, and trust fields.
• reports/<arm>-perf.tsv: per-arm deterministic log with the reference columns:

timestamp   candidate   score   decision    tokens_total    tokens_delta    wall_seconds    label

• reports/report.html: static run-scoped summary page.
• reports/strategy-compare.html: primary exercise-level comparison dashboard. It aggregates sibling runs with the same connector and exercise, plus optional external baselines from runs/external_players/<connector>/<exercise>.json. It includes exercise picking, strategy toggles, best-score trajectories, failed/rejected markers, wall idle-gap compression, token expenditure, API-equivalent cost, and a summary table. Cost derivation and uncertainty are documented in API Cost Accounting.
• Web route /ui/reports/: redirects to /ui/reports/strategy-compare.html, the chart dashboard.
• Web route /ui/runs: searchable personal Runs index with isolated dashboard links and three independent controls. Hide locally removes a run from its owner's normal comparisons without deleting evidence; Publish publicly adds it to anonymous public dashboards and is off by default; Delete permanently removes the run, submissions, bundles, and ledger rows after confirmation.
• Web route /ui/keys: create and manage scoped exercise API keys.
• Web route /ui/docs/: generated MkDocs documentation with search.
• reports/strategy-compare-<connector>-<exercise>.html: one comparison page per discovered exercise under the same runs root.
• reports/comparison.svg and reports/<arm>_progress.svg: lightweight charts.

The TSV files are generated from SQLite and immutable artifacts, never from agent prose logs.

For new rows, the canonical progress.tsv run_id column is the web/token-bound run id. Legacy rows without run metadata fall back to the arm id. Exercise-level strategy comparison dashboards split harness data by agent_run_id when present, so multiple independent runs from the same arm can be compared objectively. Run and candidate payloads also expose coding_harness; dashboards can filter it and Export Studio can group or split comparisons by it independently of model.

GitHub PR Venues¶

PR-based exercises are modeled as a transport, not as special logic in every connector. A github_pr submission still uses the same POST /submit schema and the same report fields. The normalized remote id is stored in exercise_submission_id as:

github:owner/repo#123@<head_sha>

Default mode: dry_run is deterministic and side-effect free. It computes the server-generated branch, PR title/body marker, PR URL shape, head SHA surrogate, and state-machine events without touching GitHub. This is useful for tests and for checking PR policy without creating public noise.

mode: gh_cli is opt-in and requires server-side config:

connector: github_pr
connector_config:
  mode: gh_cli
  base_repo: owner/exercise-repo
  base_ref: main
  repo_path: /srv/harness/checkouts/exercise-repo
  target_path: submissions/agent-a
  head_owner: github-user-or-fork-owner
  draft: true

The daemon can load a per-arm connectors.github_pr.env_file containing GH_TOKEN=.... Agents never receive that token. The generated branch name is server-controlled:

arena/<connector>/<problem>/<experiment>/<arm>/<candidate>-<content_sha12>

For public exercise repos, use a conservative policy such as pr_only_final or pr_only_on_promote. If both A/B arms open public PRs during the run, record the experiment as public-contaminated because agents may observe each other's work.

For ECDSA.fail specifically, the official scoring flow should remain the ECDSA.fail CLI/API-key path unless the organizers require PRs for official scoring. The GitHub PR transport is best treated as an optional publish step for that connector.

CPU.mode Venue¶

The cpumode connector uses https://cpu.mattstuchlik.com/api:

• GET /api/challenges/{exercise} for exercise metadata.
• POST /api/challenges/{exercise}/submissions for source submissions.
• GET /api/jobs/{job_id} for benchmark results.

The daemon stores CPU.mode auth in connector env files. The web UI accepts a regular cpumode_... API token or an authenticated cpu_mode_session=... cookie. When a parent credential is provided, it can use it once to mint a scoped cpumode_... API token and store that token. Existing and scripted credentials can still use any supported shape:

CPUMODE_API_TOKEN=...
CPU_MODE_TOKEN=...
CPU_MODE_TOKEN_FILE=/secure/cpumode-token.txt
CPUMODE_SESSION_COOKIE=cpu_mode_session=...
CPUMODE_AUTH=Bearer cpumode_...

CPU.mode agent tokens are stored through CPUMODE_API_TOKEN; the web UI keeps the setup path simple by collecting the authenticated cookie and doing the mint server-side.

The normalized score is the selected job's result_time_ns with score type cpu_time_ns. Set connector_config.system to choose the target system, for example raptor_cove_p; all raw job payloads are preserved in the candidate artifact.

GPU Mode Venue¶

The gpumode connector submits through the official Popcorn CLI. The daemon stores the Popcorn identity, creates an isolated connector home per credential profile/run, writes that home's .popcorn.yaml, and runs:

popcorn submit --no-tui --leaderboard <exercise> --gpu <gpu> --mode <mode> --output <artifact> <submission.py>

The stored credential is the cli_id value from ~/.popcorn.yaml after running popcorn register discord or popcorn register github:

POPCORN_CLI_ID=...

Submissions must contain a single Python file, usually submission.py. If a bundle contains multiple Python files, pass --solution-file or connector_options.solution_file so the connector knows which file to send. Popcorn directives such as #!POPCORN leaderboard ... and #!POPCORN gpu ... can still live in the source, but the harness scope remains the source of truth for the exercise/run/credential profile.

Useful connector options:

{
  "gpu": "B200",
  "submission_mode": "test",
  "leaderboard": "qr_v2",
  "profile_brev": "true",
  "benchmark_index": "0",
  "solution_file": "submission.py"
}

submit and refresh are Popcorn proxy calls. The normalized score is the selected Popcorn run score in seconds with score type gpumode_score_seconds, but the response also preserves the visible Popcorn CLI payload and normalized public case results:

connector_response.raw.popcorn.command
connector_response.raw.popcorn.stdout
connector_response.raw.popcorn.stderr
connector_response.raw.popcorn.output
connector_response.raw.popcorn.text
connector_response.raw.popcorn.parsed
connector_response.case_results
connector_response.case_summary

The connector performs an authenticated read of /user/submissions/<submission_id> after submit, refresh, and solution fetches because Popcorn's submissions show text omits runs[].result. Public benchmark/test cases are normalized with typed parameters and nanosecond timing fields. Secret runs retain their aggregate run/score metadata, but their detailed result bodies are stripped before storage and response serialization. If structured details are temporarily unavailable, submit output in Popcorn's rendered benchmark format is parsed as a bounded fallback.

refresh uses popcorn submissions show <id> --no-code. To fetch the same view directly through the harness, run:

harness solution <submission_id> --no-code

The full source-including Popcorn view remains available through:

harness solution <submission_id>

Agents must use these Harness calls instead of calling popcorn directly. The run token can only fetch submissions recorded under its scoped run. Running harness refresh <candidate_id> after this connector update can enrich an older candidate with case details if Popcorn still retains the submission.

Paradigm Puzzles Venue¶

paradigm_puzzles stores one server-side PARADIGM_PUZZLES_API_KEY=pp_... credential and exposes the active API-documented exercises through run-token scope. Workers submit ordinary bundles through POST /submit; they never receive the bearer key.

The optional /ui/paradigm/agent flow creates a pre-bound run and generates a prompt for an agent or coding harness running on the user's computer. It accepts an explicit effort, arbitrary duration or no limit, and optional parameters. Named profiles and HTTP(S) reference URLs are private to their owning user; ScoreBench does not fetch or execute those URLs or parameters. It does not execute an agent on ScoreBench. The prompt explicitly installs the required ScoreBench skill and CLI, then uses the one-time /api/pair bootstrap above. The owner-only monitor reads canonical run progress; it can be closed and revisited while the process continues independently.

The connector selects the upstream codec from the scoped exercise: JSON source, structured packing JSON, or multipart file upload. It preserves SSE progress and result events, normalizes each challenge's metric, and records an authoritative lower or higher direction with the score. Report grouping uses the persisted direction, so maximizing Paradigm challenges are not interpreted using the host experiment's lower-is-better default.

AMM, Prop AMM, QEC, and Packing are validated before submission. SSE exercises check authenticated cooldown state before submitting. Lean Semantics returns a remote submission id in submitted state and supports POST /refresh; the other documented endpoints normally return a terminal score in the initial response. Paradigm leaderboard, submission-list, and submission-detail reads are proxied through the existing connector read endpoints.

See docs/connectors.md#paradigm-puzzles for file names, payloads, metrics, and directions.

Credential Storage¶

The web UI stores connector credentials as named profiles under the shared runs root:

runs/credentials/<connector>/<credential-name>.env
runs/credentials/<connector>/<credential-name>.json

These profiles are selectable when creating exercise API keys. A run token binds one user, one credential profile, one connector, one exercise, and optionally one run; agents cannot use that token to switch to other profiles or connectors. Exercise API keys can be archived, revoked, or deleted from the Account page. Only active run tokens authenticate.

The same credential profile may back several independent run tokens in parallel. Those tokens intentionally do not share a read surface. For a scoped run token, best, history, refresh, and solution are filtered to rows with the same user, credential profile, connector, exercise, and run id. In particular, solution only fetches a connector source when the requested remote solution id was recorded by a submission from that same run. This lets the owner compare runs in the web dashboard while preventing agents from inspecting sibling runs through the harness API, even when the connector credential itself could see them.

New web UI credential env files are encrypted at rest:

# harness-secret-env:v1
HARNESS_SECRET_ENV=<fernet payload>

The local decrypt key is stored at runs/credentials/.harness_secret_key with 0600 permissions. This is local at-rest protection; the daemon can decrypt the secrets while running. Production deployments can set SCOREBENCH_SECRET_ENV_KEY_FILE (legacy alias: HARNESS_SECRET_ENV_KEY_FILE) to an absolute path outside the run-data tree so database or credential-directory snapshots do not contain both ciphertext and key. Copy the existing key to that path before setting the variable; the daemon fails closed if an existing encrypted credential cannot be decrypted. Connector loaders still support legacy plaintext env files for scripted and older runs.

The older YAML-oriented CLI flow remains available:

expctl credentials setup experiment.yaml

The interactive flow supports any number of agents and any configured connector. It writes YAML-bound credential env files under:

CONFIG_DIR/.harness_credentials/<experiment_id>/users/<user>/<connector>.env

All credential env files are written with 0600 permissions. The web UI never renders secret values back to the browser, and the YAML flow stores only env-file paths in experiment.yaml.