Harness Architecture¶

Harness is a local-first middleware for objective challenge optimization runs. Agents do not talk directly to HighLoad, Tensara, CPU.mode, GitHub PR challenges, or future platforms. They talk to Harness. Harness owns credentials, submissions, result normalization, immutable evidence, logging, and reports.

The core rule is:

agents submit candidates; Harness owns the venue connection and the ledger

Repository Boundary¶

This repository contains the server side:

• HTTP daemon and web UI
• agent-facing CLI
• connector layer for challenge platforms
• SQLite ledger and immutable candidate bundles
• credential storage and scoped run tokens
• deterministic reports and dashboards

The agent instructions live separately in the Harness Agent skill repository:

https://github.com/josusanmartin/scorebench-skill

The skill should teach an agent how to use the Harness API and CLI. It should not contain connector secrets, server internals, or challenge platform API keys.

High-Level Shape¶

                              external challenge platforms
                              +----------------------------+
                              | HighLoad / Tensara / CPU   |
                              | GitHub PR / Kaggle / ...   |
                              +-------------+--------------+
                                            ^
                                            |
                                    connector layer
                                            |
+-------------+       +---------------------+---------------------+
| Harness user | ----> | harness daemon / web UI                   |
| browser     |       | challenge_harness.daemon                  |
+-------------+       +---------------------+---------------------+
                                            |
                                            v
                                  HarnessService
                            challenge_harness.service
                                            |
             +------------------------------+------------------------------+
             |                              |                              |
             v                              v                              v
       SQLite ledger                 credential store              report builder
 challenge_harness.db          secret env files / metadata       challenge_harness.report
             ^
             |
+------------+------------+
| solving agent workspace |
| harness CLI + run token |
+-------------------------+

GitHub can render the same system as a Mermaid diagram:

flowchart TB
    Admin["Harness user browser"]
    Agent["Solving agent workspace"]
    CLI["harness CLI"]
    Daemon["Harness daemon and web UI<br/>challenge_harness.daemon"]
    Service["HarnessService<br/>challenge_harness.service"]
    DB[("SQLite ledger<br/>harness.sqlite")]
    Credentials["Credential store<br/>encrypted secret files and metadata"]
    Bundles["Immutable candidate bundles"]
    Reports["Reports and dashboards<br/>challenge_harness.report"]
    Connectors["Connector layer<br/>challenge_harness/connectors"]
    Platforms["External challenge platforms<br/>HighLoad / Tensara / CPU.mode / GitHub PR / ..."]

    Admin --> Daemon
    Agent --> CLI
    CLI --> Daemon
    Daemon --> Service
    Service --> DB
    Service --> Credentials
    Service --> Bundles
    Service --> Reports
    Service --> Connectors
    Connectors --> Platforms
    Reports --> Daemon

There are two important security boundaries:

1. Harness users manage their own credentials and create scoped exercise or run tokens.
2. Agents receive only a scoped Harness token, never the underlying connector credential.

The public deployment can expose more than one product origin without running multiple writers against the SQLite ledger. scorebench.dev serves the general benchmark product, while paradigm.scorebench.dev serves the Paradigm Puzzles agent workflow. The daemon derives the product surface from an exact hostname allowlist, changes navigation and landing behavior, filters runs and credential controls, and scopes generated report payloads to the matching connector. Both origins still share the service layer, queue, credentials, and authoritative ledger.

Main Components¶

Daemon And Web UI¶

challenge_harness.daemon runs the local or public HTTP service. It serves:

• user login
• credential management
• exercise/run token creation
• dashboards and reports
• agent API endpoints used by the harness CLI
• static assets such as CSS, reports, and favicon

The daemon is intentionally thin. It parses HTTP requests, renders pages, and delegates most business logic to HarnessService.

HarnessService¶

challenge_harness.service.HarnessService is the orchestration layer. It owns the rules for:

• admin authentication and session behavior
• credential profile lookup
• exercise/run token validation
• run start, run current, run progress, run ping, and usage accounting
• run-scoped post-run trace artifact validation and retrieval
• candidate submission validation
• idempotency and duplicate-content checks
• connector execution
• pending result refresh
• report generation triggers

If the behavior affects correctness, security, or accounting, it belongs in the service layer rather than in HTML templates or connector-specific code.

SQLite Ledger¶

challenge_harness.db.HarnessStore is the durable ledger. The database lives in the run directory as harness.sqlite.

The ledger records the facts that reports should trust:

• experiment and run metadata
• credential profile bindings
• scoped exercise/run tokens
• candidate sequence numbers
• content hashes
• immutable bundle locations
• submission timestamps
• connector status and raw response metadata
• normalized scores
• token snapshots and usage deltas
• run heartbeats and elapsed-time evidence

SQLite is used because the system is local-first, transactional, easy to back up, and simple to inspect when debugging.

Sanitized agent traces are intentionally not stored in SQLite. The skill produces bounded gzip NDJSON only after a run; the service validates and stores it under artifacts/run_traces/ using a hash-derived run scope and trace ID. Trace upload/download is observational, never triggers report generation, and does not affect run activity or candidate state.

Agent CLI¶

challenge_harness.cli provides the harness command used by solving agents. It reads:

• HARNESS_URL
• HARNESS_RUN_TOKEN

Then it calls the daemon for:

• harness context
• harness exercise
• harness run start
• harness run current
• harness run progress
• harness run ping
• harness run usage
• harness submit
• harness invalidate
• harness reinstate
• harness refresh
• harness best
• harness history
• harness solution

The CLI creates deterministic candidate bundles from files or directories and sends them to the server. It does not need platform credentials.

harness run progress is the stable supervisor-facing accounting contract. It uses the same canonical timing and token normalization as reports, is restricted to one run-token scope, and does not add a heartbeat when polled. Dashboard markup and /best are presentation and best-candidate views, not progress measurement APIs.

Connector Layer¶

Connectors live under challenge_harness/connectors/. A connector translates a Harness candidate into a platform-specific action and normalizes the result.

Connectors can also expose scoped discovery data through the daemon, such as HighLoad leaderboards, solution lists, challenge-page sections, and solve-form defaults. HighLoad discovery includes language metadata and default filenames so agents can choose CPP, GO, RUST, CSHARP, ZIG, or configured safe language ids without receiving the underlying cookie.

Current connector shapes include:

• highload: cookie-authenticated HighLoad submissions and status refresh
• local_tensara: Tensara CLI/API-key backed submissions to the private/local Tensara deployment
• public_tensara: Tensara CLI/API-key backed submissions to public Tensara
• cpumode: CPU.mode credential and submission integration
• gpumode: GPU Mode / Popcorn proxy submissions
• vliw: credentialless, queued ScoreBench VLIW judging scored in cycles
• github_pr: PR-based challenge transport
• fake: deterministic test connector

Connectors are deliberately small. They should know how to talk to a platform, but they should not decide experiment policy, token accounting, run ownership, or dashboard semantics.

Submission Flow¶

The normal flow is:

1. A Harness user logs in to the Harness UI.
2. The user saves one or more named credential profiles for a connector.
3. The user creates an exercise API key scoped to:
   user + connector + credential profile + exercise.
4. The user gives the generated environment block to the agent.
5. Agent runs `harness context` and `harness exercise`.
6. Agent starts or resumes a named run with `harness run start`.
7. Agent submits candidates with `harness submit`.
8. Harness validates token scope, run name, exercise, idempotency, token usage,
   and candidate metadata.
9. Harness writes the candidate bundle and ledger rows before connector work.
10. Connector submits or evaluates against the external platform.
11. Harness stores the normalized result and raw evidence.
12. If a candidate is later found invalid, the scoped agent can run
    `harness invalidate` to mark it `invalidated` without deleting evidence.
13. If a contract review reverses that decision, `harness reinstate` appends a
    linked audit event and restores the captured prior status.
14. Reports are generated from the ledger and immutable artifacts.

The same flow as a sequence diagram:

sequenceDiagram
    participant User as Harness user UI
    participant Agent as Solving agent
    participant CLI as harness CLI
    participant Daemon as Harness daemon
    participant Service as HarnessService
    participant Store as SQLite ledger
    participant Connector as Connector
    participant Platform as External challenge
    participant Report as Report builder

    User->>Daemon: Save connector credential profile
    Daemon->>Service: create credential profile
    Service->>Store: store profile metadata
    Service-->>Daemon: credential saved

    User->>Daemon: Create scoped exercise API key
    Daemon->>Service: create run token scope
    Service->>Store: store token record and scope
    Service-->>Daemon: show token once
    User-->>Agent: HARNESS_URL + HARNESS_RUN_TOKEN

    Agent->>CLI: harness context / exercise
    CLI->>Daemon: authenticated request
    Daemon->>Service: validate token
    Service->>Store: load token scope
    Service-->>CLI: scoped connector + exercise context

    Agent->>CLI: harness run start --id run001 --skills ... --model ... --coding-harness ... --effort ... --autonomy ...
    CLI->>Daemon: start or resume run
    Daemon->>Service: validate run name and policy
    Service->>Store: record run metadata
    Service-->>CLI: active run

    Agent->>CLI: harness submit solution file
    CLI->>Daemon: candidate bundle + metadata + token snapshot
    Daemon->>Service: validate and allocate candidate
    Service->>Store: write candidate row and evidence
    Service->>Connector: submit/evaluate with server-held credential
    Connector->>Platform: platform-specific request
    Platform-->>Connector: raw result or pending status
    Connector-->>Service: normalized result + raw evidence
    Service->>Store: store score/status/evidence
    Service->>Report: regenerate or mark reports stale
    Service-->>CLI: authoritative submission response

This lets the agent focus on solving the problem. The agent should not need to know whether the backend is a form POST, a CLI command, a GitHub pull request, or a polling API.

Credential Model¶

Credentials are named profiles per connector. Examples:

connector: highload
profile: skill-research
secret: HIGHLOAD_COOKIE

connector: local_tensara
profile: no-skill
secret: TENSARA_API_KEY

connector: cpumode
profile: quant-agent
secret: CPUMODE_API_TOKEN

The web UI manages these profiles. Agents never receive these secrets.

The run token given to an agent is scoped to a specific user, connector, credential profile, and exercise. This keeps runs isolated:

agent token can submit to one exercise with one credential profile
agent token cannot list credentials
agent token cannot inspect sibling runs
agent token cannot switch to another connector

Run bearer tokens are shown only in the creation or rotation response. SQLite stores a SHA-256 token digest and prefix, not recoverable plaintext. Reissuing an active token creates a replacement with the same scope and immediately revokes the old token.

User passwords use salted PBKDF2-SHA256 hashes. A successful login against a legacy unsalted SHA-256 record upgrades that record in place; login and registration attempts are throttled per client.

Resource Boundaries¶

Run-scoped submission budgets are enforced under the candidate-allocation lock:

• candidate count per run
• cumulative compressed candidate-bundle bytes per run
• normalized working-token total per run
• distinct candidates per rolling time window
• repeated submissions of the same bundle per rolling time window

The defaults are conservative and can be overridden by the experiment's submission_limits mapping. Submission windows are calculated from persisted candidate timestamps and survive daemon restarts. Idempotent response recovery happens before budget consumption, so replaying the original request key remains safe even while a window is full. Post-run traces have independent per-upload, count, and aggregate compressed-byte limits. HTTP request bodies are length-bounded and read with a socket timeout; the request ceiling includes base64 expansion of the 64 MiB candidate-bundle limit, and the threaded HTTP server has a fixed worker ceiling plus a smaller semaphore for concurrently buffered bodies above 1 MiB. Sensitive dynamic responses and trace downloads are marked Cache-Control: no-store.

Cross-run code-similarity warnings combine an exact normalized-token digest, 64-bit simhash for structural edits, and a compact bottom-k shingle-containment sketch for additive padding. Indexed simhash and containment anchors nominate a small candidate set before scoring. Solutions with 16-79 normalized tokens get exact-match coverage only; approximate checks start at 80 tokens to limit false positives on short boilerplate. For exercises with a trusted public starter, ScoreBench removes that starter's normalized shingles before comparison and requires at least 32 residual shingles, including for exact matches. This keeps tiny identical edits around the starter from becoming evidence of copied solution code. Fuzzy checks additionally require at least 80 residual shingles on both sides; the total token count cannot make a small customization look substantial. Residual containment also uses a 0.90 minimum instead of the raw-code 0.72 threshold and requires residual sizes to be within 2x. Exact and SimHash matches keep their normal rules above the residual floor. The registry is keyed by connector and exercise, so only an explicit, versioned template can be discounted; learned corpus-common code is never silently ignored. Existing raw fingerprints remain compatible through residual containment, so enabling a template does not require a database backfill for new submissions. Historical annotations can be rebuilt from immutable, hash-verified bundles with scripts/recalculate_code_similarity.py; the command is dry-run by default, creates an online SQLite backup before applying, preserves unrelated warnings, and is expected to produce a zero-change second dry-run. Production maintenance uses --registered-template-scopes-only so unrelated binary challenge bundles are not traversed. Similarity remains an audit warning, not an automatic rejection.

The offline replay summary includes non-source metadata for every resulting flag: candidate and run IDs, score, match reason, distance, containment, and residual sizes. This makes threshold changes auditable before a guarded apply without exposing submitted code.

Connector Responsibilities¶

A connector should provide a narrow contract:

• declare its credential schema for the UI
• list or validate supported exercises when possible
• return an exercise statement or instructions
• submit or evaluate a candidate bundle
• refresh a pending result when the platform is asynchronous
• normalize score, status, direction, and remote IDs
• preserve useful raw response metadata for debugging

A connector should not:

• trust agent-provided scores
• read another credential profile
• decide A/B grouping
• mutate unrelated runs
• hide platform errors from the service layer

Conceptually, every connector sits behind the same boundary:

flowchart LR
    Bundle["Candidate bundle<br/>source files + metadata"]
    Scope["Validated token scope<br/>user + connector + credential + exercise + run"]
    Service["HarnessService"]
    Schema["Connector credential schema"]
    Connector["Connector implementation"]
    Credential["Server-held credential profile"]
    Platform["Challenge platform"]
    Result["Normalized AdapterResult<br/>status + score + direction + remote id + raw evidence"]
    Store[("SQLite ledger")]

    Bundle --> Service
    Scope --> Service
    Service --> Schema
    Service --> Credential
    Service --> Connector
    Credential --> Connector
    Connector --> Platform
    Platform --> Connector
    Connector --> Result
    Result --> Service
    Service --> Store

Database And Evidence Model¶

Harness stores both normalized data and evidence.

Normalized data is what the dashboard uses:

candidate_id
run_name
credential_profile
connector
exercise
status
score
score_unit
direction
tokens_total
tokens_delta
active_seconds
remote_submission_id

Evidence is what lets us debug and audit:

server_received_at
server_completed_at
request hash
bundle hash
source hash
raw connector response
connector logs
trace id
remote URL or submission id
refresh attempts
warnings

The dashboard should be reproducible from the ledger and immutable bundles. It should not depend on agent-written progress files as authoritative data.

The current database has implementation-specific details, but the core ledger relationship is:

erDiagram
    USERS ||--o{ CREDENTIAL_PROFILES : owns
    USERS ||--o{ RUN_TOKENS : receives
    CREDENTIAL_PROFILES ||--o{ RUN_TOKENS : scopes
    CONNECTORS ||--o{ CREDENTIAL_PROFILES : defines
    CONNECTORS ||--o{ EXERCISES : provides
    EXERCISES ||--o{ RUN_TOKENS : scopes
    RUN_TOKENS ||--o{ RUNS : starts
    RUNS ||--o{ CANDIDATES : contains
    CANDIDATES ||--|| BUNDLES : stores
    CANDIDATES ||--o{ SUBMISSION_RESULTS : records
    CANDIDATES ||--o{ USAGE_SNAPSHOTS : accounts
    CANDIDATES ||--o{ TRACE_EVENTS : explains
    SUBMISSION_RESULTS ||--o{ RAW_EVIDENCE : preserves

    USERS {
        string username
        string role
    }
    CONNECTORS {
        string name
        string credential_schema
    }
    CREDENTIAL_PROFILES {
        string name
        string connector
        string owner_user
        string encrypted_secret_ref
    }
    EXERCISES {
        string connector
        string exercise_id
        string title
    }
    RUN_TOKENS {
        string token_hash
        string user
        string connector
        string credential_profile
        string exercise_id
    }
    RUNS {
        string run_name
        string credential_profile
        string exercise_id
    }
    CANDIDATES {
        string candidate_id
        string content_sha256
        string status
    }
    SUBMISSION_RESULTS {
        string remote_submission_id
        float score
        string status
        string direction
    }

Timing, Token, And Cost Accounting¶

Harness can trust server-observed facts:

• when the request arrived
• when the connector returned
• candidate sequence number
• bundle hash
• raw platform response
• normalized platform score

Harness cannot inherently know true model token usage unless the runner or provider reports it. For that reason token data is recorded with provenance.

Agents are required to submit token snapshots when using the Harness workflow. Harness stores totals and computes deltas server-side. If token data is missing, the service should reject or warn according to the current policy rather than silently producing empty token charts.

Provider aggregates are not universally comparable. In particular, Grok's aggregate includes cached reads, so the service requires a disjoint input/output/cache-read breakdown for Grok models and recomputes the working total server-side.

Run pings and usage snapshots give the dashboard a better view of active work than wall-clock timestamps alone. This matters when an agent session expires, idles, or is interrupted for unrelated reasons.

Active time is explicitly an estimate. Reports retain its source, raw wall value, and cumulative idle time removed for each candidate. Candidates where at least half of clock time was excluded receive a distinct chart marker and tooltip explanation. If submitted timestamps are unavailable, the fallback caps each inter-candidate gap independently instead of collapsing all later candidates to one timestamp.

Cost is a report-time derived field, not an authoritative provider invoice. The report builder prices model token categories with the versioned hardcoded table in challenge_harness.model_pricing; legacy aggregate-only runs receive a visibly marked estimate. Unknown and composite models remain unpriced. See API Cost Accounting.

Pending And Stale Results¶

Some platforms return a result immediately. Others take minutes and require polling.

The connector layer can return a pending status. Harness records that state and uses refresh paths to complete the result later:

submitted -> pending -> accepted
submitted -> pending -> rejected
submitted -> pending -> error

This is important for platforms such as HighLoad, where the challenge page may show completed submissions later than the original submit request. A stale refresh worker or explicit harness refresh should reconcile those results so the ledger does not permanently undercount completed attempts.

Reports And Dashboard¶

challenge_harness.report builds deterministic report artifacts from the SQLite ledger and candidate bundles.

Important outputs include:

• strategy comparison dashboard
• per-exercise report JSON
• run summaries
• TSV/CSV exports for analysis
• raw evidence links where available

The primary dashboard is the strategy comparison view. It compares runs by:

• connector
• exercise
• credential profile
• run name
• score trajectory
• active time
• wall time
• token spend
• API-equivalent model cost
• failures and rejected candidates
• promotions and best-so-far changes

Invalidated candidates are preserved for auditability but excluded from best, best-so-far curves, promotion decisions, and dashboard winner calculations.

The dashboard should answer:

Which strategy improved fastest under the same budget?
Which strategy reached the best final score?
Which strategy spent fewer tokens or less active time?
Which strategy reached a score under the lowest API-equivalent cost?
Which failures were platform errors versus rejected candidates?

The reporting pipeline is one-way: reports are derived from the ledger, not the other way around.

Report generation is isolated from the live submission path:

• writable service connections use SQLite WAL mode;
• report builders open every ledger with a true read-only, query-only connection and never run migrations;
• every output is written to a temporary file and atomically replaced;
• .report-manifest.json records a source fingerprint for each artifact, so a targeted refresh cannot make unrelated stale pages appear fresh;
• concurrent refresh requests are coalesced, and requests that arrive during a render are handled by a bounded catch-up loop;
• if the source changes while a report is rendering, the generated artifact remains stale and is rendered again;
• a normal stale page is served immediately while its replacement renders in the background.

Full generation still writes the compatibility JSON, CSV, TSV, SVG, compare, and export artifacts. A stale exercise page uses incremental generation: exercise topology and run-count summaries come from the manifest, only the requested exercise data is rebuilt, and only the requested compare/data or export artifacts are atomically replaced.

Interactive compare pages contain the application shell rather than the full candidate history. The shell fetches compact strategy-compare-<connector>-<exercise>.json; a deterministic .json.gz sidecar is retained as an artifact, while the daemon caches viewer-scoped gzip responses so privacy filtering does not require repeated parsing or compression. Dashboard query filters can be applied before JSON transfer. Export pages stay self-contained so downloaded presentations work offline.

Generated comparison payloads retain all runs. The HTTP layer scopes each JSON response at request time: authenticated owners receive their locally visible runs, while anonymous and Public runs views receive only runs explicitly published by their owners. Local hide state therefore cannot remove a run for another viewer or accidentally make a private run public.

The admin web keeps chart viewing and report management on separate routes:

• /ui/reports/ redirects to /ui/reports/strategy-compare.html, the interactive chart dashboard.
• /ui/runs is the personal Runs index: searchable run history, isolated dashboard links, local hide/show, explicit public publication, and permanent deletion. Publication is an allowlist: a run without a publication row is private.
• /ui/keys creates and manages scoped exercise API keys under Account.
• /ui/docs/ serves the generated MkDocs site and its search index from the run directory.

flowchart LR
    DB[("SQLite ledger")]
    Bundles["Immutable bundles"]
    Raw["Raw connector evidence"]
    Builder["Report builder<br/>challenge_harness.report"]
    JSON["report.json<br/>exercise-report.json<br/>compact dashboard JSON + gzip"]
    TSV["TSV / CSV exports"]
    HTML["strategy-compare.html<br/>application shell"]
    Manifest[".report-manifest.json<br/>freshness + summaries + timings"]
    UI["Web UI report pages"]

    DB --> Builder
    Bundles --> Builder
    Raw --> Builder
    Builder --> JSON
    Builder --> TSV
    Builder --> HTML
    Builder --> Manifest
    JSON --> UI
    HTML --> UI

Logging And Tracing¶

Harness should log enough to debug connector issues without exposing secrets.

Useful log layers are:

• request log: HTTP method, path, status, duration, trace id
• service log: run token scope, candidate id, validation decisions
• connector log: platform request attempts, remote IDs, status transitions
• trace log: structured per-submission timeline
• error log: exceptions, connector failures, refresh failures

Secrets must be redacted from logs. The right debugging handle is the trace id, candidate id, remote submission id, and credential profile name, not the underlying cookie or API key.

GET /health includes a reports object with the active worker state, queued scope, and last generation metrics. reports.generate.completed trace events include total time, per-phase time, selected scopes, candidate and artifact counts, payload sizes, and whether the source changed during generation.

Adding A New Connector¶

To add a connector:

1. Create challenge_harness/connectors/<connector_name>.py.
2. Define the credential schema used by the web UI.
3. Register the connector in challenge_harness/connectors/__init__.py.
4. Implement exercise lookup or validation.
5. Implement submit/evaluate.
6. Implement refresh if results are asynchronous.
7. Normalize status, score, direction, and remote submission ID.
8. Preserve raw response metadata for debugging.
9. Add tests with fake HTTP responses or a fake CLI.
10. Update the Harness Agent skill only if the agent-facing workflow changes.

The target is a connector that is boring and auditable. Platform-specific quirks should stay inside the connector, while run policy and experiment accounting stay in the Harness service and ledger.

Operational Modes¶

Harness can run locally or behind a public HTTPS reverse proxy.

Local mode is useful for private experiments:

browser -> http://127.0.0.1:<port>/ui
agent   -> http://127.0.0.1:<port>

Public mode is useful when multiple users need access:

browser -> HTTPS host -> reverse proxy -> harness daemon
agent   -> HTTPS host -> reverse proxy -> harness daemon

In public mode, per-user credential isolation matters. A signed-in user should only see and manage their own credential profiles unless explicit admin tooling is added for cross-user operations.

Design Principles¶

• Keep agents lightweight: one run token, simple CLI commands, no platform keys.
• Keep the ledger authoritative: reports come from SQLite and immutable bundles.
• Keep connectors narrow: submit, refresh, normalize, preserve evidence.
• Keep credentials server-side: agents never receive cookies or API keys.
• Keep reports deterministic: fixed ordering, stable schemas, reproducible HTML.
• Keep failures visible: connector errors should be recorded and returned.
• Keep experiments isolated: one credential profile and one run scope per token.