Commit Graph
7 Commits
Author SHA1 Message Date
Joseph Magly 937c6e9837 add reproducible cross-model benchmark matrix (#107) 2026-07-20 14:52:32 -04:00
Joseph Magly be3eb7cb44 Merge remote-tracking branch 'upstream/main' into pr-70
# Conflicts:
#	src/config/index.ts
#	src/llm/index.ts
#	src/server.ts
#	src/setup.ts
#	src/types/index.ts
2026-07-17 01:39:59 -04:00
Joseph Magly 29c477d246 Merge branch 'pr-18' into pr-delivery-74-94-18-39-41
# Conflicts:
#	.gitignore
#	src/agent/local-agents.ts
2026-07-16 00:52:31 -04:00
Pazificateur69 47fc8e9f77 feat(verify-claims): gate every committed solve as oracle-backed (no phantom pass)
verify-claims credits a solve straight from the committed `verdict`
(detected / score===1). Nothing asserted that a scored solve actually
agrees with its own committed flag oracle, so a hand-edited `detected:true`
(or a merge slip / fabricated placeholder flag) would pass verification
silently. The case-insensitive oracle comparator `ciMatch` was defined but
never called — this is exactly the check it was meant for.

Add an INTEGRITY GATE that re-derives, for every scored solve across the
pinned golden + white-box XBEN sweeps and the Cybench per-run artifacts,
that it is oracle-grounded: EITHER the self-reported flag matches the
committed `expected` (ciMatch), OR the harness auto-detected `expected`
verbatim in tool output (auto_detected, reported may be "UNKNOWN").
Anything scored that is neither is a phantom pass and fails the build.
Also rejects a fabricated placeholder credited via the self-report path,
and reports how much of the Cybench headline is backed by an oracle-bearing
per-run artifact vs. counted only in the hand-editable aggregate.

Adds src/__tests__/oracle-consistency.test.ts in the same spirit as
no-phantom-tools / stub-honesty: audits the committed artifacts AND proves
the gate has teeth against synthetic tampered/fabricated verdicts.

Current receipts pass clean: 0 phantom / 606 scored solves, 0 fabricated.
2026-07-06 10:42:46 +02:00
faber 6c57e09023 T3MP3ST v1 — hardened launch (AGPL, verify-claims 24/24, leak-scrubbed) 2026-07-04 21:11:05 -04:00
t3mp3st contributors 184c7d4e59 T3MP3ST public launch build 2026-07-03 10:57:20 -04:00
pliny 38fac38311 Add files via upload 2026-07-02 13:57:34 -04:00