61 lines
2.2 KiB
Markdown
61 lines
2.2 KiB
Markdown
# Model and Harness Comparisons
|
|
|
|
T3MP3ST model comparisons are built from committed benchmark receipts. The
|
|
matrix command aggregates existing artifacts; it never starts containers,
|
|
calls a model, or contacts a target.
|
|
|
|
## Reproduce the committed comparison
|
|
|
|
```bash
|
|
npm run bench:model-matrix
|
|
npm run verify:model-matrix
|
|
```
|
|
|
|
The committed Cybench matrix compares Claude Opus 4.7 and 4.8 on the same 15
|
|
task IDs, `live-tools` harness label, artifact schema, and pass@1 policy. Its
|
|
JSON contains every source artifact path; its Markdown output provides the
|
|
human-readable table.
|
|
|
|
Outcome categories stay separate:
|
|
|
|
- `success`: the committed strict oracle verdict detected the expected flag;
|
|
- `failure`: the benchmark ran but did not satisfy the oracle;
|
|
- `abstention`: the artifact explicitly records a refusal, abstention, or
|
|
no-action outcome;
|
|
- `infrastructure_error`: the run failed because of a timeout, API, spawn, or
|
|
equivalent harness error;
|
|
- `skipped`: the expected artifact is unavailable.
|
|
|
|
These are system results: model, provider, harness, runtime, tools, target
|
|
class, and attempt policy all matter. Historical dates and model versions also
|
|
differ. Do not interpret the table as an isolated model-quality ranking.
|
|
|
|
## Run arbitrary model variants
|
|
|
|
The refusal-frontier harness accepts any comma-separated OpenRouter model IDs
|
|
without source edits:
|
|
|
|
```bash
|
|
# Offline: prints the exact shared corpus and call count; spends no API quota.
|
|
node scripts/refusal-frontier.mjs \
|
|
--models anthropic/model-a,openai/model-b \
|
|
--classes crypto,memory \
|
|
--n 3 \
|
|
--dry-run
|
|
|
|
# API-backed and cost-bearing: requires OPENROUTER_API_KEY.
|
|
node scripts/refusal-frontier.mjs \
|
|
--models anthropic/model-a,openai/model-b \
|
|
--classes crypto,memory \
|
|
--n 3 \
|
|
--at comparison-2026-07
|
|
```
|
|
|
|
Every selected model receives the same classes, five-rung corpus, sample count,
|
|
judge, and concurrency settings. JSON output retains raw responses and judge
|
|
rationales; Markdown output renders model-by-model and cross-model deltas.
|
|
|
|
Use the fields in [CONTRIBUTION_RECEIPTS](CONTRIBUTION_RECEIPTS.md) when adding
|
|
new results. Never commit credentials or private target data, and never present
|
|
live external runs without a target-specific authorization receipt.
|