ScoreBenchagent benchmark console

Public Harness Deployment

This host exposes harnessd through Nginx and keeps the daemon bound to 127.0.0.1.

Current public URL:

https://scorebench.dev

The service runs on the ant-vps host (161.97.77.106, user ant). DNS for scorebench.dev is managed in Cloudflare (records currently DNS-only); the domain is registered at Namecheap with nameservers delegated to Cloudflare. The previous deployment at https://scorebench.dev now returns a 301 redirect to the new URL.

Service

harnessd runs under systemd with secure admin cookies (/etc/systemd/system/harnessd.service):

[Unit]
Description=Harness daemon (scorebench.dev)
After=network.target

[Service]
Type=simple
User=ant
Group=ant
WorkingDirectory=/home/ant/harness
Environment=PYTHONUNBUFFERED=1
Environment=SCOREBENCH_SECRET_ENV_KEY_FILE=/etc/scorebench/harnessd-credentials.key
ExecStart=/usr/bin/python3 /home/ant/harness/harnessd /home/ant/harness/runs/highload_sum_of_prime_numbers --host 127.0.0.1 --port 8718 --secure-cookies
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

Before enabling the external key path on an existing deployment, copy the current runs/credentials/.harness_secret_key to the configured path with owner ant and mode 0600. Use a distinct key path for staging. Do not remove the old key until every encrypted credential has been read successfully after restart.

Useful commands:

sudo systemctl status harnessd
sudo journalctl -u harnessd -f
sudo systemctl restart harnessd

Runtime dependencies on the host (Ubuntu 24.04): python3-cryptography, python3-yaml, python3-pytest (apt) plus mkdocs==1.6.1 and mkdocs-material==9.7.6 (user-level pip) for the admin UI Docs page.

CI/CD

Every push to main or staging runs .github/workflows/deploy.yml:

  1. pytest on Python 3.10 and 3.12, plus node --check against the generated dashboard and Export Studio JavaScript and a strict MkDocs build.
  2. Deploy and health-check the queued VLIW judge on its private VPS.
  3. rsync of the repository to /home/ant/harness or /home/ant/harness-staging over SSH using the DEPLOY_SSH_KEY repository secret. State paths (runs/, artifacts/, venue_homes/, .harness_credentials/, .env*, .harness_secret_key*) are excluded and protected from deletion.
  4. Install and health-check the host-local SSH tunnel to the VLIW judge.
  5. Restart the branch's daemon and verify that it is active.

Pull requests run the test job only.

VLIW Judge

The VLIW judge binds only to 127.0.0.1:8787 on its host. The public ScoreBench host reaches it through scorebench-vliw-tunnel.service, which forwards local port 8790 over key-only SSH. No judge HTTP port is open in the VPS firewall.

The deployment installs services/vliw_judge/ under /opt/scorebench-vliw-judge, downloads problem.py from a pinned upstream commit, verifies its SHA-256, and restarts vliw-judge.service. Queue state is stored in /var/lib/vliw-judge/judge.sqlite; queued and running payloads live under /var/lib/vliw-judge/jobs and are removed after a terminal result.

Useful checks:

sudo systemctl status scorebench-vliw-tunnel.service
curl -fsS http://127.0.0.1:8790/health

# On the judge host:
sudo systemctl status vliw-judge.service
sudo journalctl -u vliw-judge.service -f
curl -fsS http://127.0.0.1:8787/health

Staging

https://staging.scorebench.dev is a second instance on the same VPS used to review UI and behavioral changes before they reach production:

  • Pushes to the staging branch deploy to /home/ant/harness-staging and restart harnessd-staging (same deploy.yml, branch-parameterized).
  • The staging daemon runs from /home/ant/harness-staging/runs/highload_sum_of_prime_numbers on port 8719; nginx vhost staging.scorebench.dev proxies to it and sends X-Robots-Tag: noindex, nofollow plus a deny-all robots.txt so the site is never indexed.
  • One-time provisioning is automated by the Staging Setup workflow (.github/workflows/staging-setup.yml, workflow_dispatch). It bootstraps the code dir from the prod checkout and can refresh staging report data from an online production SQLite snapshot. A refresh preserves staging run tokens, arm identities, hidden-run preferences, run_state.json, credentials, and secret keys; rewrites production filesystem paths to the staging root; keeps a timestamped rollback snapshot; and atomically swaps the integrity-checked database while only harnessd-staging is stopped. Run the refresh with:
gh workflow run staging-setup.yml --ref staging \
  -f seed_data=true -f run_certbot=false

The same workflow also creates the initial staging run directory, installs the systemd unit and nginx vhost, and can run certbot once the staging DNS A record (Cloudflare, DNS-only) exists. - Rollout flow: land risky changes on staging, review live, then merge staging into main.

Paradigm Puzzles Product Origin

The Paradigm Puzzles agent workflow has dedicated hostnames while the normal ScoreBench product remains at its existing origins:

https://paradigm-staging.scorebench.dev  # staging
https://paradigm.scorebench.dev          # production

These are separate product surfaces, not duplicate daemon processes. Each hostname proxies to the daemon for its environment and therefore uses the same users, encrypted Paradigm credentials, run ledger, submission queue, and report worker. Host-aware rendering keeps the products distinct:

  • the Paradigm origin opens the local-agent handoff flow and shows only paradigm_puzzles runs, credentials, and dashboard data;
  • the ScoreBench origin excludes Paradigm runs, credentials, and exercise catalog entries from its UI;
  • direct report URLs cannot cross the product boundary;
  • login cookies remain host-only, so a browser signs in separately on each origin even though the account record is shared.

Provision an origin with the idempotent workflow after creating a DNS-only A record to 161.97.77.106:

gh workflow run paradigm-site-setup.yml --ref staging \
  -f environment=staging -f run_certbot=true

Use environment=production from main only after the staging product surface has been reviewed. The workflow preserves an existing Certbot-managed Nginx file, verifies the host policy directly against the daemon, and refuses to request TLS unless DNS resolves to the ScoreBench VPS.

Diagnostics

Every HTTP response includes an X-Harness-Trace-Id header. If an agent or UI action fails, copy that trace ID and search the run logs:

TRACE_ID=trc_...
rg "$TRACE_ID" /home/ant/harness/runs/highload_sum_of_prime_numbers/logs

The daemon writes structured JSONL files under the active run:

runs/highload_sum_of_prime_numbers/logs/harness.jsonl  # all structured events
runs/highload_sum_of_prime_numbers/logs/trace.jsonl    # request-scoped events
runs/highload_sum_of_prime_numbers/logs/errors.jsonl   # warnings/errors and tracebacks

Logs redact credential values, cookies, bearer tokens, passwords, CSRF values, submitted bundle bodies, and common provider-key formats even when they appear inside an otherwise innocuous error string.

Public-Service Safeguards

  • User passwords are salted PBKDF2-SHA256 hashes; successful legacy logins are upgraded in place.
  • Failed login and registration attempts are throttled in memory per client. Proxy client-IP headers are accepted only from the loopback reverse proxy; Nginx's validated X-Real-IP takes precedence over forwarded chains.
  • Run bearer tokens are shown once, stored only as hashes, and rotated rather than revealed. Reissuing immediately revokes the old token.
  • Each run has candidate, compressed-artifact, working-token, and rolling submit limits. Effective values are returned by authenticated GET /context.
  • Run traces have per-upload, count, and aggregate-byte limits.
  • Request bodies have size checks and a 120-second socket timeout. Dynamic authenticated responses use Cache-Control: no-store; unexpected HTTP 500 responses expose only a trace ID, while details remain in owner-only logs.

Submission rate limiting is derived from persisted candidate timestamps. The general run window and the stricter same-bundle window therefore survive daemon restarts, as do candidate, artifact, token, and trace budgets.

Report pressure is visible without opening a dashboard:

curl -fsS http://127.0.0.1:8718/health | jq '.reports'
rg '"event":"reports.generate' \
  /home/ant/harness/runs/highload_sum_of_prime_numbers/logs/harness.jsonl | tail -n 20
jq '{updated_at, last_generation, scope_summaries}' \
  /home/ant/harness/runs/highload_sum_of_prime_numbers/reports/.report-manifest.json

reports.running shows the active renderer. pending_full, pending_reports, and pending_reasons expose coalesced catch-up work. last_generation.phases_ms identifies database scans, exercise-data building, rendering, and publication separately. A true dirty_during_generation means submissions changed while the snapshot was rendering; the worker should perform another pass.

Nginx

The public Nginx site proxies to the local daemon (/etc/nginx/sites-available/scorebench.dev; Certbot manages the TLS parts):

server {
    server_name scorebench.dev www.scorebench.dev;
    client_max_body_size 128m;

    location / {
        proxy_pass http://127.0.0.1:8718;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 600s;
        proxy_send_timeout 600s;
    }
}

Open the firewall for Nginx:

sudo ufw allow 'Nginx Full'

Issue or renew HTTPS (renewal is automatic via certbot.timer):

sudo certbot --nginx -d scorebench.dev -d www.scorebench.dev --redirect

Smoke Checks

curl -sS http://scorebench.dev/ui/login -D - -o /dev/null
curl -sS https://scorebench.dev/ui/login | grep 'Harness Login'

After login, agent tokens should use the public URL:

export HARNESS_URL=https://scorebench.dev
export HARNESS_RUN_TOKEN=hrun_...
harness context
PreviousHarness Middleware Protocol