add reproducible cross-model benchmark matrix (#107)

This commit is contained in:
Joseph Magly authored and GitHub committed 2026-07-20 14:52:32 -04:00
1 parent 58193d688d
commit 937c6e9837
10 files changed
+842 -4

No files matched your search

+60
View File
@@ -0,0 +1,60 @@
# Model and Harness Comparisons
T3MP3ST model comparisons are built from committed benchmark receipts. The
matrix command aggregates existing artifacts; it never starts containers,
calls a model, or contacts a target.
## Reproduce the committed comparison
```bash
npm run bench:model-matrix
npm run verify:model-matrix
```
The committed Cybench matrix compares Claude Opus 4.7 and 4.8 on the same 15
task IDs, `live-tools` harness label, artifact schema, and pass@1 policy. Its
JSON contains every source artifact path; its Markdown output provides the
human-readable table.
Outcome categories stay separate:
- `success`: the committed strict oracle verdict detected the expected flag;
- `failure`: the benchmark ran but did not satisfy the oracle;
- `abstention`: the artifact explicitly records a refusal, abstention, or
no-action outcome;
- `infrastructure_error`: the run failed because of a timeout, API, spawn, or
equivalent harness error;
- `skipped`: the expected artifact is unavailable.
These are system results: model, provider, harness, runtime, tools, target
class, and attempt policy all matter. Historical dates and model versions also
differ. Do not interpret the table as an isolated model-quality ranking.
## Run arbitrary model variants
The refusal-frontier harness accepts any comma-separated OpenRouter model IDs
without source edits:
```bash
# Offline: prints the exact shared corpus and call count; spends no API quota.
node scripts/refusal-frontier.mjs \
--models anthropic/model-a,openai/model-b \
--classes crypto,memory \
--n 3 \
--dry-run
# API-backed and cost-bearing: requires OPENROUTER_API_KEY.
node scripts/refusal-frontier.mjs \
--models anthropic/model-a,openai/model-b \
--classes crypto,memory \
--n 3 \
--at comparison-2026-07
```
Every selected model receives the same classes, five-rung corpus, sample count,
judge, and concurrency settings. JSON output retains raw responses and judge
rationales; Markdown output renders model-by-model and cross-model deltas.
Use the fields in [CONTRIBUTION_RECEIPTS](CONTRIBUTION_RECEIPTS.md) when adding
new results. Never commit credentials or private target data, and never present
live external runs without a target-specific authorization receipt.