T3MP3ST public launch build
This commit is contained in:
1 parent
972247b881
commit
184c7d4e59
1589 files changed
+57150
-506
No files matched your search
@@ -54,11 +54,9 @@ Artifacts: `bench/xbow/results/blackbox-cog-gpt55/`.
|
||||
|
||||
**Honest standing:** the published **black-box 94/104 is empirically clean** (0/104 shortcut use) and now grader-hardened. To make it *unimpeachable* it should be re-measured on the fully-fixed harness (a stricter, tool-output-provenance grader). **cybench and OBSIDIVM numbers stay withheld** until their rigs are clean (cybench provisioning re-measure; OBSIDIVM still scores keyword-on-narrative even under `--replay` — its scorer needs to rest on real probe evidence before any number is reported). Remaining residuals tracked: structural XBEN fix (per-run random flag), reaper shared_net siblings + signal-path teardown, flag-scrub encodings, verify-finding live-signature exit-status check, semantic case-fold, XBOW default-`--all` resume footgun, NYU zero-task_files, CVE-Zero CWE families.
|
||||
|
||||
Full diagnosis: `docs/CYBENCH_HARNESS_DIAGNOSIS.md`. Negative result (v2.1 toggles): `docs/NEGATIVE-RESULTS.md` §4.
|
||||
|
||||
## 2026-06-19 — NINTH finding: mislabeled reasoning-effort on the XBEN numbers
|
||||
|
||||
**What:** Our published XBEN docs (README, THE_CLAIM, XBOW_BASELINE) and the verify-claims log labeled the golden runs **"gpt-5.5 + xhigh"**. On audit, the live-tools hunter — the code path the XBEN sweep actually runs through (`scripts/cybench-bench.mjs`, the per-iteration request `body`) — builds `{model, max_tokens, messages, tools, tool_choice}` and **never attaches a reasoning param** (not in the body, headers, or model id). `REASONING_EFFORT` is read+applied **only** in the separate `callLLM` one-shot path (which also excluded venice/anthropic), and XBEN does not use it. The committed golden artifacts record `model: openai/gpt-5.5` with **no effort field**. So the runs were gpt-5.5 at the provider's **default** reasoning effort; "xhigh" was set as an env var but silently ignored by the hunter.
|
||||
**What:** Our published XBEN docs (README, XBOW_BASELINE) and the verify-claims log labeled the golden runs **"gpt-5.5 + xhigh"**. On audit, the live-tools hunter — the code path the XBEN sweep actually runs through (`scripts/cybench-bench.mjs`, the per-iteration request `body`) — builds `{model, max_tokens, messages, tools, tool_choice}` and **never attaches a reasoning param** (not in the body, headers, or model id). `REASONING_EFFORT` is read+applied **only** in the separate `callLLM` one-shot path (which also excluded venice/anthropic), and XBEN does not use it. The committed golden artifacts record `model: openai/gpt-5.5` with **no effort field**. So the runs were gpt-5.5 at the provider's **default** reasoning effort; "xhigh" was set as an env var but silently ignored by the hunter.
|
||||
|
||||
**Impact:** the **numbers are unaffected** — 91/104 bb · 102/104 wb pass@1 and 98/104 bb best-ball are all real live-exploit captures. Only the **reasoning-effort label was overstated.** Corrected "gpt-5.5 + xhigh" → "gpt-5.5, default reasoning effort" across the three docs + the verify-claims log.
|
||||
|
||||
@@ -350,7 +348,7 @@ request-smuggling chain — all clean. Same 18/31 = 58% headline as the
|
||||
**Not the record.** Modern Cybench SOTA is Anthropic's Claude Sonnet 4.5 at
|
||||
**76.5% pass@10** (system card, 37/40 subset). We are pass@1. Our distinction is
|
||||
measurement integrity (scrubbed + jailed + hint-free + self-audited), not peak
|
||||
score. See `docs/THE_CLAIM.md` for the validated public framing.
|
||||
score. See the README for the validated public framing.
|
||||
|
||||
## 2026-05-28 — Sixth finding: misclassified infra-fail (delulu exec-bit)
|
||||
|
||||
@@ -395,7 +393,7 @@ This is what we mean by "the only public Cybench solver with a published contami
|
||||
|
||||
- Every result file is auto-aggregated into `docs/SCORECARD.{md,json}`.
|
||||
- This ledger lists every retraction. We never silently delete a result.
|
||||
- Public docs (`docs/THE_CLAIM.md`) link here at the top.
|
||||
- The public README links here at the top.
|
||||
- Scrub now runs in every fresh staging — log line is grep-able for CI gates.
|
||||
|
||||
## 2026-05-30 — CLAIM VALIDATION: retracted the "XBOW 44–56%" comparison
|
||||
|
||||
Reference in new issue
Block a user