add reproducible cross-model benchmark matrix
This commit is contained in:
10 files changed
+842
-4
No files matched your search
@@ -0,0 +1,60 @@
|
||||
# Model and Harness Comparisons
|
||||
|
||||
T3MP3ST model comparisons are built from committed benchmark receipts. The
|
||||
matrix command aggregates existing artifacts; it never starts containers,
|
||||
calls a model, or contacts a target.
|
||||
|
||||
## Reproduce the committed comparison
|
||||
|
||||
```bash
|
||||
npm run bench:model-matrix
|
||||
npm run verify:model-matrix
|
||||
```
|
||||
|
||||
The committed Cybench matrix compares Claude Opus 4.7 and 4.8 on the same 15
|
||||
task IDs, `live-tools` harness label, artifact schema, and pass@1 policy. Its
|
||||
JSON contains every source artifact path; its Markdown output provides the
|
||||
human-readable table.
|
||||
|
||||
Outcome categories stay separate:
|
||||
|
||||
- `success`: the committed strict oracle verdict detected the expected flag;
|
||||
- `failure`: the benchmark ran but did not satisfy the oracle;
|
||||
- `abstention`: the artifact explicitly records a refusal, abstention, or
|
||||
no-action outcome;
|
||||
- `infrastructure_error`: the run failed because of a timeout, API, spawn, or
|
||||
equivalent harness error;
|
||||
- `skipped`: the expected artifact is unavailable.
|
||||
|
||||
These are system results: model, provider, harness, runtime, tools, target
|
||||
class, and attempt policy all matter. Historical dates and model versions also
|
||||
differ. Do not interpret the table as an isolated model-quality ranking.
|
||||
|
||||
## Run arbitrary model variants
|
||||
|
||||
The refusal-frontier harness accepts any comma-separated OpenRouter model IDs
|
||||
without source edits:
|
||||
|
||||
```bash
|
||||
# Offline: prints the exact shared corpus and call count; spends no API quota.
|
||||
node scripts/refusal-frontier.mjs \
|
||||
--models anthropic/model-a,openai/model-b \
|
||||
--classes crypto,memory \
|
||||
--n 3 \
|
||||
--dry-run
|
||||
|
||||
# API-backed and cost-bearing: requires OPENROUTER_API_KEY.
|
||||
node scripts/refusal-frontier.mjs \
|
||||
--models anthropic/model-a,openai/model-b \
|
||||
--classes crypto,memory \
|
||||
--n 3 \
|
||||
--at comparison-2026-07
|
||||
```
|
||||
|
||||
Every selected model receives the same classes, five-rung corpus, sample count,
|
||||
judge, and concurrency settings. JSON output retains raw responses and judge
|
||||
rationales; Markdown output renders model-by-model and cross-model deltas.
|
||||
|
||||
Use the fields in [CONTRIBUTION_RECEIPTS](CONTRIBUTION_RECEIPTS.md) when adding
|
||||
new results. Never commit credentials or private target data, and never present
|
||||
live external runs without a target-specific authorization receipt.
|
||||
Reference in new issue
Block a user