Harness Connector Reference¶

This reference describes the connector layer used by Harness. It is for operators who configure experiments, maintain connector credentials, or need to understand what a solving agent can submit through harness submit.

The agent contract is the same for every connector:

export HARNESS_URL=https://scorebench.dev/
export HARNESS_RUN_TOKEN=hrun_...
harness context
harness exercise
harness run start --id run001 \
  --skills scorebench \
  --model gpt-5-codex \
  --coding-harness Codex \
  --effort high \
  --autonomy autonomous
harness run ping --event start --note "starting work"
harness submit path/to/solution --total-tokens 123456 --tokens-total-source agent_claim
harness refresh
harness best
harness history

Agents receive only the scoped Harness run token. Harness keeps connector API keys, cookies, CLI homes, browser sessions, rate-limit state, raw responses, and normalized scores on the daemon side.

If a connector accepts a candidate that later proves invalid, exploity, or based on a false assumption, the agent should invalidate it through Harness instead of calling the platform directly or rewriting history:

harness invalidate <candidate_id> --reason "exploit: ..."

Omit <candidate_id> only when invalidating the latest candidate visible to the current run token. Invalidation keeps the bundle, raw connector payloads, scores, logs, token data, and history intact, but changes the candidate status to invalidated and excludes it from best, best-so-far curves, promotion decisions, and dashboard winner calculations.

If a later contract review proves the invalidation itself was wrong, restore the candidate without erasing that history:

harness reinstate <candidate_id> --reason "contract correction: ..."

History and reports expose the reason, actor, and timestamp for both events.

Connector Model¶

Connector implementations live in challenge_harness/connectors/. The registry currently supports:

Every connector returns a ConnectorResult with:

• status: scored, submitted, rejected, or failed.
• metric_value: normalized numeric score when available.
• score_type: metric namespace used by reports.
• direction: optional connector-authoritative lower or higher direction.
• raw: connector-specific evidence, remote ids, commands, and redacted request metadata.
• stdout, stderr, and error: useful operator diagnostics.

Harness stores both the normalized fields and the raw connector evidence. Do not discard raw connector payloads when adding or changing a connector; reports and debugging rely on them.

Connector-authoritative direction is stored with each score. This matters on a shared deployment where one connector minimizes runtime while another maximizes edge or Elo. Exercise reports derive their direction from those persisted score rows instead of assuming the serving experiment's default direction.

Configuration Layers¶

Connector settings can come from several places. Prefer experiment-level connector_config for shared defaults and arm-level connectors.<name> for per-arm credentials or per-agent overrides. Exact merge order is connector-specific, because older experiment files used a few legacy arm fields.

Common sources are:

• Experiment-level connector and exercise.
• Experiment-level connector_config.
• Arm-level connectors.<connector_name>.
• Arm-level connector_config.
• Legacy arm fields such as connector_env_file or connector_gpu.
• Request-scoped harness submit connector options.

Common request-scoped options accepted by harness submit are:

--exercise / --problem
--language / --lang
--compiler
--compiler-options / --compiler-args
--system
--solution-file
--gpu
--leaderboard
--submission-mode / --mode
--profile-brev
--benchmark-index

Unknown connector option keys are rejected by the service. This keeps agent submissions deterministic and prevents typo-driven silent behavior changes.

Credential profiles are configured through the web UI or:

./expctl credentials setup experiment.yaml
./expctl credentials add experiment.yaml --arm agent_a --connector highload --secret-env AGENT_A_SECRET

Generated credential env files default to:

CONFIG_DIR/.harness_credentials/<experiment_id>/users/<user>/<connector>.env

New web UI credentials are stored under the shared run root:

runs/credentials/<connector>/<credential-name>.env
runs/credentials/<connector>/<credential-name>.json

Secret values stay on the daemon side. Agent workspaces should receive only HARNESS_URL and HARNESS_RUN_TOKEN.

Fake¶

Use fake for local smoke tests, report tests, and harness development. It does not call an external service and does not need credentials.

Inputs¶

The connector scans all text files in the submitted bundle.

• A file containing score=<number> or score:<number> returns scored.
• Any file containing FAIL returns failed.
• Any file containing REJECT returns rejected.
• Missing score marker returns failed.

The default direction is lower-is-better unless the experiment objective says otherwise.

Config¶

experiment_id: demo_ab
connector: fake
exercise: fake_score
objective:
  direction: lower
  target: 10
arms:
  - name: skill
    treatment: optimization_skill_enabled
    user: demo_skill
  - name: control
    treatment: no_skill
    user: demo_control
connector_config:
  exercise_text: "Submit a file containing score=<number>. Lower is better."

Useful keys:

• exercise_text: statement returned by harness exercise.
• direction: included in the exercise response for local test fixtures.

Result Shape¶

score_type is local. The raw payload includes the parsed score, arm name, and Harness candidate id.

Local Tensara¶

Use local_tensara for the private Tensara deployment. It defaults to:

https://tensara.62.171.174.233.sslip.io

Aliases are tensara, tensara_local, and local-tensara.

Credentials¶

Credential profiles store a Tensara API key:

TENSARA_API_KEY=tsra_...

The connector also accepts:

TENSARA_TOKEN=...

The key must come from the target Tensara deployment. A Harness hrun_... token is not a Tensara key, and a public Tensara key is not interchangeable with a private deployment key.

Exercise Reads¶

harness exercise resolves the statement in this order:

1. connector_config.exercise_text, if set.
2. Local problem markdown from problem_root, when direct API mode is enabled.
3. A local direct-API fallback statement with the problem URL and problem roots.
4. tensara problem --json <problem> when direct API is disabled.

Relevant config:

connector_config:
  problem_root: /home/josu/dev/tensara-selfhost/problems
  fetch_problems: true
  authenticate_problem_reads: true

Submission Modes¶

local_tensara defaults to mode: submit.

When direct_api is enabled, and the mode is submit, the connector posts code directly to the Tensara deployment's SSE direct-submit endpoint. This is the default for local_tensara. It sends:

• problemSlug
• code
• language
• gpuType

It parses terminal SSE events such as ACCEPTED, BENCHMARKED, WRONG_ANSWER, COMPILE_ERROR, RUNTIME_ERROR, and RATE_LIMIT_EXCEEDED. ACCEPTED and BENCHMARKED become scored; other terminal non-success states become rejected or failed.

When direct API mode is disabled, the connector shells out to the Tensara CLI:

tensara submit --problem <problem> --solution <file> -g <gpu> --language <language>
tensara checker --json --problem <problem> --solution <file> -g <gpu>
tensara benchmark --json --problem <problem> --solution <file> -g <gpu>
tensara sample --json --problem <problem> --solution <file> -g <gpu>

checker, benchmark, and sample are explicit diagnostic modes. Normal competition runs should use submit.

Config¶

connector: local_tensara
exercise: leaky-relu
objective:
  metric: runtime_us
  direction: lower
arms:
  - name: agent_a
    treatment: skill_a
    user: tensara_a
    connector_env_file: /secure/tensara_a.env
connector_config:
  cli: tensara
  mode: submit
  gpu: T4
  language: cuda
  solution_file: sol.cu
  base_url: https://tensara.62.171.174.233.sslip.io
  direct_api: true
  direct_submit_path: /api/submissions/direct-submit
  timeout_seconds: 1800

Useful keys:

• base_url or api_base_url: target Tensara API host.
• problem_url_base: override problem page links.
• problem_root: local problem markdown roots.
• direct_api: enable or disable direct SSE submit.
• direct_submit_path: direct submit endpoint path or full URL.
• cli: Tensara CLI binary.
• mode: submit, checker, benchmark, or sample.
• gpu: Tensara GPU target, default T4. The GPU used is recorded on every candidate and shown in the strategy dashboards; runs on different GPUs are never mixed in one dashboard view unless the viewer opts in (see GPU tagging and filtering).
• language: cuda, python, mojo, cute, or cutile.
• solution_file: file inside the submitted bundle.
• authenticate: true, false, or auto.
• authenticate_problem_reads: whether tensara problem should authenticate.
• fetch_problems: merge the live CLI problem list into the UI exercise list.
• connector_homes_dir: parent directory for isolated per-arm CLI homes.

If solution_file is omitted, the connector chooses a likely source file in this order: .cu, .py, .mojo, .cute, .cutile, then the first file.

Each arm gets an isolated CLI home by default:

runs/<experiment>/connector_homes/local_tensara/<arm>

Result Shape¶

Direct API results use score_type: runtime_us. Raw evidence includes the endpoint, HTTP status, problem, GPU, language, remote submission id, terminal status, and a bounded SSE event window.

CLI results use the CLI mode as score_type. Raw evidence includes command, return code, arm name, GPU, connector home, parsed JSON or stdout text, and parsed runtime when present.

GPU Tagging and Filtering¶

Scores from different GPUs are not comparable, so the harness tracks the GPU per run and per candidate:

• Pick the GPU per run: harness run start --gpu H100 ... (also available on harness admin create-run-token --gpu and harness admin launch --gpu, and as gpu in the run-token API payload). A run declared with a GPU pins every submission in it: a conflicting harness submit --gpu is rejected, so a run can never mix GPUs.
• Without a run-level GPU, submissions use harness submit --gpu or the connector default (connector_config.gpu, T4 for tensara connectors).
• The GPU the connector actually evaluated on is recorded in each candidate's metadata and surfaced per point and per strategy in reports, TSV exports, and the dashboards.

The strategy-compare dashboards get a GPU filter next to Model/Effort. When runs with more than one GPU exist for an exercise, the filter defaults to the current GPU (the serving run's configured GPU) so different GPUs are not mixed silently. Selecting All GPUs (mixed) — or extra GPUs — is an explicit viewer choice and is kept in the gpus= URL parameter. Runs recorded before GPU metadata existed inherit their run's configured GPU. External player baselines may declare a gpu field in their JSON to participate in the same filtering.

Public Tensara¶

Use public_tensara for public Tensara at:

https://tensara.org

Aliases are public-tensara, tensara_public, and tensara-public.

Credentials¶

Create a public Tensara API key at:

https://tensara.org/cli

Store it as a public_tensara credential profile:

TENSARA_API_KEY=tsra_...

Behavior¶

public_tensara subclasses the local Tensara connector with public defaults:

• base_url: https://tensara.org
• problem_url_base: https://tensara.org/problems
• fetch_problems: true
• direct_api: false
• authenticate_problem_reads: false

Problem reads use:

tensara problem --json <problem>

Submissions use the Tensara CLI by default. The same mode, gpu, language, solution_file, authenticate, and connector_homes_dir settings from local_tensara apply.

Config¶

experiment_id: public_tensara_leaky_relu_example
connector: public_tensara
exercise: leaky-relu
objective:
  metric: runtime_us
  direction: lower
arms:
  - name: agent_a
    treatment: public_tensara_example
    user: tensara_public_a
    connector_env_file: /secure/public_tensara_a.env
connector_config:
  cli: tensara
  mode: submit
  gpu: T4
  language: cuda
  solution_file: sol.cu
  authenticate: true

Result Shape¶

The connector returns the same CLI result shape as local_tensara: command, return code, parsed JSON or text output, metric when parseable, and status scored for successful CLI return codes or rejected for non-zero CLI return codes.

HighLoad¶

Use highload for HighLoad.fun compute challenges. It defaults to:

https://highload.fun

Credentials¶

HighLoad uses an authenticated cookie saved as a named credential profile. Supported env shapes are:

HIGHLOAD_COOKIE="a=session-a; b=session-b"

or:

HIGHLOAD_COOKIE_NAME=a
HIGHLOAD_COOKIE_VALUE=session-a

If HIGHLOAD_COOKIE is a bare value instead of a name=value cookie header, the connector treats it as the value for HIGHLOAD_COOKIE_NAME, defaulting to a.

Exercise Reads¶

By default, harness exercise returns configured text or a simple Harness submission instruction. Set fetch_exercise_page: true to fetch and strip text from:

/challenges/compute/<exercise>/<section>

fetch_exercises or fetch_exercise_list can merge the public compute list from /challenges/compute/list into the UI exercise picker.

Agents can also inspect HighLoad website data through Harness without receiving the cookie:

harness solve-form --language <LANG>
harness leaderboard
harness solutions --page 1 --lang <LANG>
harness inspect-solution <solution_id>
harness challenge-page generators

Use the language for the current candidate, for example CPP, RUST, GO, CSHARP, or ZIG. Harness passes safe language ids through to HighLoad and does not restrict submissions to C++.

HighLoad language metadata is exposed through harness exercise and harness solve-form:

• languages: supported venue language ids known to Harness.
• defaultLanguage: language selected by connector config or arm override.
• defaultFilenames: filename expected by the venue for known languages.
• availableLanguages: solve-form response language ids for the current scope.

Built-in default filenames are:

If connector config adds a safe language id that does not have a known filename, Harness defaults the upstream payload filename to main.txt unless the bundle or --solution-file provides a better match.

Run tokens remain exercise-scoped. A token for sum_of_prime_numbers cannot use these commands to read order_book. harness solution <solution_id> is still reserved for solution ids submitted by the current run; use harness inspect-solution for public or connector-visible HighLoad solution metadata from other players.

harness solve-form redacts the CSRF token and default source, but returns visible defaults such as compiler and compilerArgs. Treat the compiler and compiler flags as part of the candidate hypothesis. If a candidate only changes flags, use a new label/idempotency key and explain that in --notes.

Submission¶

HighLoad submits by loading the solve form, extracting the CSRF token, and posting form fields back to:

/challenges/compute/<exercise>/solve/<language>

The connector sends:

• challengeId
• lang
• compiler
• compilerArgs
• solution, as JSON mapping filenames to source text

Submission-time overrides:

harness submit main.rs \
  --language RUST \
  --compiler <compiler-id> \
  --compiler-options "<compiler-args>" \
  --total-tokens ...

The raw connector response records the actual compiler and compilerArgs sent upstream so reports can distinguish source-code changes from flag-only changes.

If the submitted bundle has one text file and its filename does not match the HighLoad default for the selected language, Harness renames it in the upstream payload to the expected default such as main.cpp, main.rs, or main.go.

Polling, Refresh, And Solution Fetch¶

Set poll_status: true to poll the upstream status API immediately after submit:

/api/solutions/status/v1

The connector converts the hex solution id to the integer id expected by the status API. If the status response reports success without an obvious score, it fetches solution detail pages and parses score-like text.

harness refresh [candidate_id] works for HighLoad candidates with a recorded solution id. It returns submitted while HighLoad is still testing, scored when a score is parsed, and failed for terminal error states.

harness solution <solution_id> fetches the HighLoad solution page through Harness and returns stripped text plus parsed source payload when available.

Config¶

experiment_id: highload_three_agents_example
connector: highload
exercise: order_book
objective:
  metric: score
  direction: higher
arms:
  - name: agent_a
    treatment: skill_a
    user: highload_a
    connector_env_file: /secure/highload_a.env
connector_config:
  language: RUST
  compiler: <compiler-id>
  compilerArgs: <compiler-args>
  rate_limit_seconds: 30
  poll_status: true

Useful keys:

• base_url: HighLoad host.
• language or lang: upstream language, for example CPP, RUST, GO, CSHARP, or ZIG; additional safe ids are passed through when configured.
• languages, langs, or language_ids: optional additional safe language ids to expose beside the built-in CPP, GO, RUST, CSHARP, and ZIG.
• compiler: upstream compiler id.
• compilerArgs or compiler_args: compiler flags.
• rate_limit_seconds: per-arm minimum interval between upstream requests.
• timeout_seconds: upstream request timeout.
• state_file: custom rate-limit state file.
• fetch_exercises: fetch public compute exercise list.
• fetch_exercise_page: fetch a challenge page for harness exercise.
• exercise_section: path suffix for exercise-page fetches.

Each arm gets a separate rate-limit state by default:

runs/<experiment>/connector_homes/highload/<arm>/.highload_rate_limit/last_request_at

Result Shape¶

score_type is submit. Raw evidence includes upstream HTTP status, redirect location, solution id, feedback alerts, submitted filenames, status API response, and any fetched solution page text used to parse a score.

CPU.mode¶

Use cpumode for CPU.mode challenges at:

https://cpu.mattstuchlik.com

Aliases are cpu_mode and cpu-mode.

Credentials¶

The UI accepts either a CPU.mode API token or an authenticated session cookie. For a parent credential, Harness can mint a scoped cpumode_... agent token for the named credential profile.

Supported env shapes include:

CPUMODE_API_TOKEN=cpumode_...
CPU_MODE_API_TOKEN=cpumode_...
CPUMODE_TOKEN=cpumode_...
CPU_MODE_TOKEN=cpumode_...
CPUMODE_TOKEN_FILE=/secure/cpumode-token.txt
CPU_MODE_TOKEN_FILE=/secure/cpumode-token.txt
CPUMODE_AUTH="Bearer cpumode_..."
CPUMODE_SESSION_COOKIE="cpu_mode_session=..."
CPU_MODE_SESSION_COOKIE="cpu_mode_session=..."

If both a bearer token and a cookie are present, the bearer token wins.

Exercise Reads¶

By default, harness exercise fetches:

/api/challenges/<exercise>

The returned statement includes title, description, languages, compilers, compiler options, limits, and raw payload. Set fetch_exercise: false to avoid the API read and return configured text instead.

The exercise picker starts with built-in challenge ids and can merge live /api/challenges results.

Submission¶

CPU.mode submits one source file to:

/api/challenges/<exercise>/submissions

The connector sends:

• source
• language
• compiler_options
• compiler, only when it is one of rustc, clang_cpp, gcc_cpp, or clang_asm

Language is inferred from solution_file or the source suffix when not configured:

• .rs -> rust
• .cpp, .cc, .cxx, .c++ -> cpp
• .s, .asm -> asm

If multiple files match the language, set --solution-file or connector_config.solution_file.

Polling¶

By default, CPU.mode polls returned job ids until they finish:

/api/jobs/<job_id>

The connector selects the scored job matching system when possible, defaulting to raptor_cove_p. It normalizes result_time_ns as cpu_time_ns.

Config¶

experiment_id: cpumode_counting_bytes_ab
connector: cpumode
exercise: counting_bytes
objective:
  metric: cpu_time_ns
  direction: lower
arms:
  - name: agent_a
    treatment: skill_a
    user: cpumode_a
    connector_env_file: /secure/cpumode_a.env
connector_config:
  language: rust
  compiler: rustc
  compiler_options: --edition=2024 -O -C target-cpu=native -C target-feature=+crt-static
  system: raptor_cove_p
  poll_jobs: true
  poll_interval_seconds: 5
  poll_timeout_seconds: 1800

Useful keys:

• base_url: CPU.mode API host.
• language or lang: rust, cpp, or asm.
• compiler: rustc, clang_cpp, gcc_cpp, or clang_asm.
• compiler_options or compilerOptions: compiler flags.
• solution_file or source_file: source path inside the bundle.
• system: target job system to prefer.
• poll_jobs: whether to poll jobs after submit.
• poll_interval_seconds: polling interval.
• poll_timeout_seconds: total polling timeout.
• dry_run: record a redacted request without submitting.

Result Shape¶

score_type is cpu_time_ns. Raw evidence includes the solution id, remote submission id, submit response, polled jobs, selected job, and redacted request metadata. Source is redacted from stored request evidence.

GPU Mode¶

Use gpumode for GPU Mode / Popcorn leaderboards. It defaults to:

https://site--bot--dxfjds728w5v.code.run

Aliases are gpu_mode, gpu-mode, popcorn, popcorn_cli, and popcorn-cli.

Credentials¶

GPU Mode uses the Popcorn CLI identity. The preferred UI flow starts browser auth for GitHub or Discord and stores the resulting CLI id. Scripted setups can store:

POPCORN_CLI_ID=...

or a full Popcorn config:

POPCORN_CONFIG_YAML="cli_id: ..."

The connector writes the credential into an isolated per-arm .popcorn.yaml with 0600 permissions before running Popcorn.

Exercise Reads¶

harness exercise returns leaderboard metadata from /leaderboards when available, sanitized to remove secret/private fields and file contents. If no description is available, it returns a generic Popcorn submission statement.

fetch_exercises: true can merge live leaderboard names into the UI exercise picker.

Submission¶

GPU Mode submissions must contain one Python file, usually submission.py. If the bundle has multiple Python files, pass --solution-file.

The connector runs:

popcorn submit --no-tui \
  --leaderboard <leaderboard> \
  --output <connector_home>/results/<candidate>.txt \
  --gpu <gpu> \
  --mode <mode> \
  <submission.py>

Defaults:

• gpu: B200
• mode: test
• score_scope: secret
• timeout_seconds: 3900

For B200 Brev profiling, pass --profile-brev and optionally --benchmark-index. That switches the Popcorn command to --profile-brev, sets mode to profile, and uses B200_Brev.

Refresh And Solution Fetch¶

harness refresh [candidate_id] runs:

popcorn submissions show <submission_id> --no-code

The connector also reads the authenticated Popcorn submission-details JSON to recover runs[].result, because submissions show omits benchmark/test case objects. Only public run results are retained; detailed results from runs marked secret are removed before the payload is stored or returned.

harness solution <submission_id> proxies:

popcorn submissions show <submission_id>

Add --no-code to omit source where Popcorn supports it:

harness solution 123 --no-code

Agents should not call popcorn directly while working under a Harness run. Harness is the Popcorn proxy so the ledger retains the normalized score and the original Popcorn evidence together.

Config¶

experiment_id: gpumode_qr_example
connector: gpumode
exercise: qr_v2
objective:
  metric: gpumode_score_seconds
  direction: lower
arms:
  - name: agent_a
    treatment: skill_a
    user: gpumode_a
    connector_env_file: /secure/gpumode_a.env
connector_config:
  binary: popcorn
  gpu: B200
  mode: leaderboard
  score_scope: secret
  timeout_seconds: 3900

Useful keys:

• api_url or base_url: Popcorn API URL.
• binary or popcorn_binary: Popcorn CLI path.
• gpu: Popcorn GPU, default B200.
• mode or submission_mode: Popcorn mode.
• leaderboard: override exercise/leaderboard.
• solution_file or source_file: Python file inside the bundle.
• score_scope: secret, public, or all scored runs.
• profile_brev: use Popcorn --profile-brev.
• benchmark_index: benchmark index for Brev profile mode.
• fetch_exercises: fetch live leaderboard list.
• fetch_exercise_page: fetch metadata for harness exercise.
• fetch_submission_details: fetch structured public per-case results after submit/refresh/solution calls; defaults to true.
• submission_details_timeout_seconds: timeout for the details read; defaults to 5 seconds and never exceeds the connector timeout.
• dry_run: record command metadata without running Popcorn.

Each arm gets an isolated Popcorn home by default:

runs/<experiment>/connector_homes/gpumode/<arm>

Result Shape¶

score_type is gpumode_score_seconds. Raw evidence includes request metadata, redacted command, connector home, return code, output file text, submission id, remote submission id, parsed result, and a first-class Popcorn proxy payload:

connector_response.raw.popcorn.command
connector_response.raw.popcorn.stdout
connector_response.raw.popcorn.stderr
connector_response.raw.popcorn.output
connector_response.raw.popcorn.text
connector_response.raw.popcorn.parsed
connector_response.raw.popcorn.truncated
connector_response.case_results
connector_response.case_summary
connector_response.raw.case_results
connector_response.raw.case_summary

case_results contains public benchmark/test cases in venue order. Every case has kind, index, mode, runner, status, and spec; semicolon-delimited spec fields are also exposed as typed parameters. Benchmark timings are exact Popcorn nanoseconds under mean_ns, error_ns, best_ns, and worst_ns when available. case_summary.source identifies structured submission details or the bounded rendered-text fallback. The raw Popcorn text fields and case count are bounded to avoid unbounded response growth. Detailed secret-run case data is never exposed.

For an older stored candidate, run harness refresh <candidate_id> to append a new score record with per-case details when Popcorn still retains the remote submission.

Paradigm Puzzles¶

Use paradigm_puzzles for the active submission endpoints documented at https://www.paradigm.xyz/puzzles/api-docs. Aliases are paradigm, paradigm-puzzles, and paradigm_puzzle.

Credentials¶

Create a personal key from the Paradigm Puzzles API page and store it as a named ScoreBench credential profile:

PARADIGM_PUZZLES_API_KEY=pp_...

The connector also recognizes PUZZLES_API_KEY and PARADIGM_API_KEY in operator-managed env files for compatibility, but new profiles always use the canonical key. A solving agent receives only its hrun_... ScoreBench token; the Paradigm bearer key never enters the candidate bundle, CLI output, raw score payload, or trace log. Paradigm derives the credited X account from the key.

Because a key pasted into chat or logs is compromised, regenerate it from the Paradigm API page or its authenticated key-management endpoint before using the profile.

Agent / Coding-Harness Handoff¶

The signed-in /ui/paradigm/agent page creates a pre-bound run and a prompt for an agent or coding harness that remains on the user's own computer. It is not remote agent hosting. Users select effort, arbitrary duration or no limit, and optional harness parameters. Codex and Claude Code expose known model and effort presets. Claude Code includes low, medium, high, extra high, and max, with an explicit arbitrary-value path for other models, effort levels, and coding harnesses. Users can save private harness profiles with an HTTP(S) project or documentation URL; ScoreBench records and displays the URL but never fetches or executes it.

The dashboard connector picker includes the complete active Paradigm challenge catalog. A challenge with report data opens its comparison dashboard; a challenge without report data opens /ui/paradigm/agent with that challenge selected.

Skills are selected independently. ScoreBench is mandatory, Problem-Agnostic Optimization is an optional built-in example, and each user can privately save additional named skills with HTTP(S) source or documentation URLs. The prompt makes each selected skill and its local installation explicit, updates the ScoreBench CLI from the current deployment, and pairs a dedicated workspace using a short-lived code.

Pairing codes expire after 30 minutes, are single-use, and are persisted only as hashes. A successful scorebench pair exchanges the code for the normal scoped hrun_... token; the CLI stores it locally with mode 0600 and does not print it. The browser monitor authenticates as the owning user and reads run-scoped status without receiving that token. Closing the page is safe, but the agent or harness process must continue running for work to continue.

The Paradigm credential never leaves ScoreBench during this flow. Agents still submit only through ScoreBench and remain subject to validation, cooldown, idempotency, accounting, and connector evidence rules below.

Exercises And Payloads¶

The connector exposes the twelve currently documented submit-capable API challenges:

--label becomes the upstream submission name where that challenge accepts a name. Use --solution-file when a multi-file bundle contains more than one file with the expected extension. Packing accepts either a raw semicircle array or an object containing name and semicircles.

Chess and Addition have lexicographic leaderboards but ScoreBench stores one numeric metric. Chess encodes cleared level first and uses inverse parameter count only as a within-level tie-break. Addition puts qualified candidates above unqualified candidates, uses inverse parameter count among qualified candidates, and retains accuracy as progress before qualification. The upstream level, pass/qualification flag, accuracy or score percentage, and parameter count stay in connector_response.raw.response.

The API navigation also lists historical or closed puzzles. They are not placed in the submit dropdown unless the current API documentation exposes an active submission contract. For example, Attention Kernel has public historical reads but its challenge page says submissions are closed.

Validation, Cooldowns, And Status¶

Before consuming a cooldown, the connector calls Paradigm's public validation endpoint for AMM, Prop AMM, QEC, and Packing. A validation failure becomes a rejected ScoreBench candidate and no cooldown or submit request is made.

For the ten SSE challenges, the connector then reads the authenticated cooldown endpoint. If canSubmit is false, the candidate fails locally with nextSubmissionAt recorded in raw evidence; it is not sent upstream. Set prevalidate: false or check_cooldown: false only for a replacement deployment whose API intentionally lacks those endpoints.

SSE stage, progress, result, and error events are preserved in raw evidence. vliw returns a synchronous scored 202. lean-semantics returns a pending 202; ScoreBench records its submission id and harness refresh polls GET /lean-semantics/submissions/{id} until adjudication is terminal.

Public reads remain available through the scoped ScoreBench token:

scorebench leaderboard
scorebench solutions
scorebench inspect-solution <submission_id>
scorebench solution <own-submission-id> --no-code

Do not call Paradigm directly from a worker run. Direct calls bypass immutable bundles, idempotency, usage accounting, cooldown evidence, and refresh state.

Config¶

connector: paradigm_puzzles
exercise: qec
objective:
  metric: paradigm_errors_per_million
  direction: lower
arms:
  - name: agent_a
    treatment: manual
    user: user_a
    connector_env_file: /secure/paradigm-agent-a.env
connector_config:
  solution_file: decoder.py
  timeout_seconds: 1800
  prevalidate: true
  check_cooldown: true

The connector's challenge-specific direction overrides the experiment default for persisted candidate scores and per-exercise reports. A single-exercise deployment should still set objective.direction correctly for clarity and for reports generated before the first candidate has scored.

VLIW¶

Use vliw for the VLIW kernel challenge. It defaults to:

http://127.0.0.1:8790

This loopback endpoint is an SSH tunnel to the private ScoreBench judge. Aliases are vliw-challenge and vliw_challenge. The judge uses the pinned upstream problem from Anthropic's original performance take-home: candidates optimize a kernel for a simulated VLIW machine and are scored by simulated cycles (lower is better).

The connector exposes exactly two exercises: without-indices (the default) checks final values, while with-indices checks final values and final tree indices. The exercise id is part of each queued job, so runs, tokens, and dashboards stay separated. A substituted deployment can override the list with a connector_config.exercises array.

The pinned Input.generate contract initializes every lane's tree index to zero. A candidate may specialize to this documented zero-start input domain. Arbitrary nonzero initial indices are not part of either exercise unless the run's original prompt explicitly adds that stricter requirement. This is different from caching outputs for exact generated inputs, which remains an invalid benchmark shortcut.

Credentials¶

VLIW is credentialless at the connector layer. Create the run with the ScoreBench main profile in the Runs page, or omit --credential in the CLI. The agent receives only its scoped hrun_... ScoreBench run token. No Mastodon session cookie or other VLIW venue credential is stored.

Submission Model¶

A candidate is one Python file (default perf_takehome.py) defining the kernel builder class. For each submission the connector:

1. Copies the candidate bundle into a per-candidate work directory and writes the pinned problem module (problem.py, fetched from base_url + problem_path and cached per arm; problem_file uses a local copy instead).
2. Runs KernelBuilder().build_kernel(10, 2047, 256, 16) in a subprocess and serializes kb.instrs (candidate print() output cannot corrupt this; the instruction list is written to a file).
3. Submits gzip-compressed JSON {exercise, instrs, source_sha256} to /api/submit with a content-derived idempotency key. Candidate Python is not sent to or executed by the judge.
4. Waits briefly for an idle judge. If another job is running, ScoreBench stores the judge job id as a pending submission and refreshes it asynchronously.

The judge persists jobs in SQLite and one resource-limited worker subprocess scores them sequentially against nine deterministic tests. passed responses score with the worst observed cycle count; failed correctness checks are rejected.

Config¶

Everything venue-specific is overridable so a replacement deployment only needs config changes:

connector: vliw
exercise: without-indices
connector_config:
  base_url: http://127.0.0.1:8790
  auth_mode: none
  submit_path: /api/submit
  history_path: /api/my-submissions
  scoreboard_path: /api/scoreboard
  problem_path: /static/problem.py
  public_problem_url: https://raw.githubusercontent.com/anthropics/original_performance_takehome/5452f74bd977807ac2e74f3d29432b9df6f25197/problem.py
  # problem_file: /local/copy/problem.py
  module_name: perf_takehome
  kernel_class: KernelBuilder
  build_method: build_kernel
  instrs_attr: instrs
  build_kernel_args: [10, 2047, 256, 16]
  python_bin: python3
  timeout_seconds: 120
  extract_timeout_seconds: 180
  submit_wait_seconds: 10
  poll_interval_seconds: 0.5
  circuit_failure_threshold: 3
  circuit_open_seconds: 120

public_problem_url is returned by scorebench exercise for agent setup. It is separate from the private base_url + problem_path used by ScoreBench while extracting and submitting instructions.

Result Shape¶

score_type is cycles. Raw evidence includes the endpoint, HTTP status, judge job id, source hash, solution file, module name, build arguments, instruction count, queue state, and the judge response (passed, cycles, per-test cycles, or the error message).

The connector permits only one upstream scoring request at a time. After repeated HTTP 5xx or transport failures it temporarily opens a circuit and emits retryable, retry_after_seconds, failure_kind, and circuit_breaker in the raw result. A circuit-open or busy result explicitly means no upstream request was made; agents should retain the candidate and wait for the reported retry interval instead of immediately resubmitting.

GitHub PR¶

Use github_pr for PR-backed workflows where the "submission" is a pull request or a deterministic PR identity. It is a transport connector, not a benchmark runner.

Alias: github-pr. The connector is currently hidden from the web UI connector lists (credentials page and run-key creation); configure it through the experiment config or the CLI credentials wizard.

Credentials¶

Real PR mode uses a GitHub token:

GH_TOKEN=github_pat_...

Dry-run mode does not need credentials and is the default.

Exercise Reads¶

harness exercise returns the PR transport settings visible to the run:

• mode
• base repository
• base ref
• PR policy

Submission Modes¶

mode: dry_run is deterministic and side-effect free. It hashes the candidate bundle, derives a stable branch name, title, body, fake PR number, fake head SHA, and remote_submission_id, then records status checking.

mode: gh_cli is opt-in and creates or updates real pull requests using a server-side git checkout and the GitHub CLI. It requires:

• repo_path: local git checkout on the daemon side.
• base_repo: owner/repo.
• base_ref: target branch.
• valid GitHub CLI auth or GH_TOKEN from the credential env file.

The transport checks out a branch, copies candidate files into target_path, commits changes when needed, force-pushes the branch with lease, and opens or updates a draft PR by default.

Config¶

experiment_id: github_pr_demo
connector: github_pr
exercise: point_add
objective:
  metric: score
  direction: lower
arms:
  - name: agent_a
    treatment: pr_publish
    user: gh_user_a
connector_config:
  mode: dry_run
  base_repo: ecdsafail/ecdsafail-exercise
  base_ref: main
  source_connector: ecdsa_fail
  policy: pr_only_final
  draft: true

Useful keys:

• mode: dry_run or gh_cli.
• base_repo: owner/repo.
• base_owner and base_repo: alternate split form.
• base_ref: target branch, default main.
• source_connector: logical upstream challenge connector for branch naming.
• branch_prefix: branch namespace, default arena.
• policy or pr_policy: policy label stored in raw payload.
• repo_path: daemon-side checkout for gh_cli.
• target_path: destination path inside the checkout, default ..
• clean_target: remove target_path before copying candidate files.
• remote: git remote, default origin.
• head_owner and head_repo: fork/head configuration.
• draft: create draft PRs by default.
• maintainer_can_modify: allow maintainer edits by default.
• gh_cli: GitHub CLI binary, default gh.
• timeout_seconds: subprocess timeout.

Result Shape¶

score_type is github_pr. Status is submitted; the raw payload's transport state is usually checking. Raw evidence includes PR URL, PR number, head ref, head SHA, base repo/ref, content hash, branch/PR events, and remote_submission_id in this form:

github:<owner>/<repo>#<pr_number>@<head_sha>

Use github_pr when the challenge explicitly wants PR publication or PR-based review. If a platform has its own scoring CLI/API, keep that platform connector as the official scoring path and use github_pr only as a publish transport.

Adding A Connector¶

When adding a connector:

1. Implement ChallengeConnector.exercise() and ChallengeConnector.evaluate().
2. Return a complete ConnectorResult with raw evidence and redacted request data.
3. Add a credential schema when the connector uses secrets.
4. Register the connector in challenge_harness/connectors/__init__.py.
5. Keep credentials on the daemon side; never require agents to hold platform secrets.
6. Add focused connector tests and a service-level submission test when behavior affects refresh, solution fetch, reports, idempotency, or token scope.
7. Update this document and any example YAML needed by operators.

Connector code should not decide experiment ownership, token accounting, idempotency policy, or dashboard semantics. Those belong in challenge_harness.service and challenge_harness.report.