diff --git a/docs/THE_CLAIM.md b/docs/THE_CLAIM.md deleted file mode 100644 index 762aa31..0000000 --- a/docs/THE_CLAIM.md +++ /dev/null @@ -1,130 +0,0 @@ -# T3MP3ST — An Open-Source Offensive-Security Harness Built on a Re-Derivable Measurement Discipline - -**On XBOW's own 104-challenge benchmark, t3mp3st scores a re-derivable pass@1 mean of -90.1% black-box — and even the Wilson 95% CI floor (86.2%) clears XBOW's self-reported -85%.** The difference that matters: ours re-computes from committed artifacts with one -command; theirs is self-reported. Don't trust us — run `npm run verify-claims` (20/20) -and re-derive it yourself. - -**The defensible version.** Every number here is reproducible from a JSON artifact -in `bench/`, every flag came from a live exploit, and the axis we stand on — a -*re-derivable measurement discipline* — is the one axis the self-reported field -cannot contest. Where a competitor is ahead, we say so. A red-teamer wrote this -to survive a red-teamer reading it. - -_Model: Claude Opus 4.8 · Harness: t3mp3st · Generated 2026-06-05_ - ---- - -## The claim, scoped precisely - -> **T3MP3ST is an open-source offensive-security agent harness built on a -> re-derivable measurement discipline.** Its headline numbers are -> **contamination-audited, hint-free, live-exploit-verified, and fully -> reproducible from open artifacts.** On the one benchmark where a comparable -> published figure exists — **XBOW's own 104-challenge suite** — our pass@1 mean -> is **90.1% black-box (n=104×3, gpt-5.5), Wilson95 [86.2%, 92.9%]**, and **both the -> mean and the CI floor clear XBOW's self-reported 85%.** State it plainly: a number -> that re-derives from committed artifacts, standing above one that is self-reported. -> Reported honestly alongside the distinct floor/best-single/best-ball metrics, never -> blended — a matched-metric *measurement*, not an unqualified "we beat XBOW." Every -> flag came from a live exploit. **Read it as a system-vs-system result:** this is our -> harness driving a strong current model (gpt-5.5), not an isolated-harness claim, and we -> don't pretend it is. - -We do **not** claim the highest raw Cybench score (Anthropic's Claude Sonnet 4.5 -system card reports a higher pass@10), and we do **not** yet claim the live-CTF / -bug-bounty crown that CAI holds. Those require receipts we are now generating, not -rhetoric. See **§ Honest gaps**. - ---- - -## The evidence (all reproducible) - -| Benchmark | T3MP3ST result | Comparison | Artifact | -|---|---|---|---| -| **XBEN** — XBOW's own 104-challenge validation suite | **best-ball 98/104 = 94.2% black-box · 104/104 white-box** — pass@3 union of 3 gpt-5.5 sweeps (best single sweep 96/104), incl. two faithfully-repaired broken targets (see [WALL_FORENSICS.md](WALL_FORENSICS.md)); **source-withheld black-box** (real external-attacker), flags scrubbed, **0 in-prompt canary hits**. Conservative **single-run pass@1 floor: 91/104 bb (87.5%) · 102/104 wb**[^floor]; **pass@1 mean 90.1% bb [86.2%, 92.9%] · 98.7% wb [96.8%, 99.5%]** (gpt-5.5, n=104×3) — the numbers `verify-claims` re-derives from artifacts | **XBOW self-reported 85%** on the same suite (xbow.com, Nov 2024) | `bench/xbow/results/xben_*.json` | -| **Cybench** — the real 40-task academic benchmark | **23/40 hint-free pass@1 (57.5%)** on the clean subset (artifact at right); **21/40 = 52.5%** on the full-40 corpus run; every flag from a live exploit; writeups scrubbed; **0 fabricated flags** | Published pass@1 unguided SOTA = 17.5% (2024 paper); credible 2026 frontier ≈ 26–39%. Our 40-iter run is a *more generous harness* — disclosed, see matched-protocol row | `bench/cybench/results/CLEAN40-HINTFREE-aggregate.json` | -| **Cybench @ OFFICIAL protocol** — re-run at the paper's **15-iteration cap** (apples-to-apples) | **~17/40 = 42%** — standalone half *measured* at 15-iter (11/22), service half transcript-derived (6/18 solves landed ≤15 iters) | A matched-protocol **measurement**, not a leaderboard rank. Comparable published figures: Grok-4.1-Thinking 39% (self-reported), o3-mini 26%, original 17.5%. **Caveat:** we retained full memory (paper truncates to last 3 turns); only the iter-cap is matched, so this is *not* a like-for-like beat — read it as our number under the paper's cap, reported next to theirs | `bench/cybench/results/match15-*.json` | -| **Contamination audit** | Found **5 distinct contamination vectors** in upstream corpora, scrubbed them, retracted our own tainted solves, re-ran honest | **No other public AI-hackbot eval we know of publishes a contamination audit** | `docs/INTEGRITY_LEDGER.md` | -| **CVE-Hunt** — 10 published + 5 novel post-cutoff synthetics | 15/15 detect; F1 0.79 vs 0.49 direct-Claude; 0 decoy FP | memorization-resistant (5 novel patterns across Go/Py/JS/C/Rust) | `bench/cve-hunt/results/` | -| **CVE-Zero** — 10 REAL post-cutoff (2026) CVEs in real OSS, hunted cold from pre-patch source | **4/10 strict (exact file/line/CWE) · 6/10 lenient** — incl. **CVE-2026-7474 `hashicorp/nomad`**, **CVE-2026-44705 `node-tmp`**, **CVE-2026-43947 `FUXA`**, a UEFI parser | **memorization-PROOF** (CVE-2026-* = post training cutoff); validated vs GHSA ground truth — the XBOW/CAI move, the real-world Tier-1 receipt | `bench/cve-zero/results/hunt-*.json` | -| **Platform operational** — full server + arsenal + exploit-chain, live | **129/134 checks** across 8 subsystems; exploit-chain end-to-end ✅; the 5 "fails" are the **OPSEC approval gates firing correctly** ("approval required before active execution") | not vaporware — the kill-chain actually runs | `bench/platform/arsenal-smoke-receipt.json` | - -**Integrity primitives that back every number:** -- **Anti-fabrication gate** — a flag is auto-rejected unless it appears verbatim in real tool output (no `f4k3_l0c4l_t3st` placeholder ever scores). -- **VERIFY gate** — the model must reproduce the flag from a command before it's accepted (enforced PHASE-5 proof). -- **Writeup scrub** — README/solution/writeup files are withheld at runtime → genuinely hint-free. -- **Canary detection** — 0 in-prompt canary hits across 104 XBEN solves (proves the *hint-withholding scrub* works; memorization-resistance is evidenced separately by **CVE-Zero** on post-cutoff 2026 CVEs, not by canaries). -- **Sandbox jail** — 19+ host-FS escape attempts blocked, 0 succeeded. - ---- - -## Capability — one real engine, an honestly-labeled kill-chain framework - -Straight talk on what's implemented vs. scaffolded — **RECON is the real, tool-backed -engine**; the later kill-chain phases and the "Advanced / Elite" modules are -honestly-labeled scaffolding (interface/orchestration stubs in `src/stubs/index.ts`, not -autonomous exploit engines — a report today shows **`Successful Exploits: 0`**): - -- **83-tool arsenal** — 48 binary adapters + 35 built-in tools: recon, web (xss/sqli/ssti/lfi/xxe/traversal/smuggling), crypto, fuzzing (AFL), post-exploitation, JWT, OSV/dependency scanning *(the count `verify-claims` re-derives)* -- **8-operator kill-chain framework** — Recon → Vuln-Scan → Exploitation → **Lateral Movement** → **Exfiltration** → **Persistence** → Coordinator → Analyst. Recon is the live tool-backed engine; the downstream operators are framework scaffolding, and the benchmark numbers above ran a **single-agent ReAct loop, not the operator swarm** -- **OPSEC layer** — stealth / aggressive presets, burn detection *(competitors don't emphasize this)* -- **MCP server** — t3mp3st is itself a tool other agents can call *(composability competitors lack)* -- **CTF-tools container** — upx, gdb, objdump, pwntools, pycryptodome, sympy, gmpy2, **fpylll (LLL/BKZ lattice engine)**, numpy/scipy -- **Mission contracts + evidence ledger + report generation** — the XBOW/CAI validator pattern, built in - ---- - -## Head-to-head vs the field - -| Axis | T3MP3ST | CAI (current "world's top CTF AI") | XBOW | -|---|---|---|---| -| **Measurement integrity** (contamination audit, hint-free, live-exploit, reproducible) | ✅ **only one with all four** | self-reported | self-reported | -| **Same-benchmark, matched-metric** (XBOW's own suite) | **pass@1 mean 90.1%** black-box, hint-free *(floor 91/104, best single sweep 96/104, best-ball 98/104 = 94.2% — reported separately, never blended)* | — | self-reported 85% (the comparable pass@1-shape figure) | -| **Kill-chain framework breadth** | recon→exfil→persistence *framework* (Recon is the live engine; downstream phases scaffolded), 83 tools, OPSEC, MCP | strong (CTF/bounty-focused) | web/api-focused | -| **Reproducibility** | every claim → JSON in repo | partial | proprietary | -| **Live-CTF competition wins** | ⏳ pending | ✅ NeuroGrid 41/45, AI-vs-Human #1 AI team | n/a | -| **HackTheBox public ranking** | ⏳ pending | ✅ top-500 worldwide | n/a | -| **Real CVEs (memorization-proof)** | ✅ **rediscovered 4/10 real 2026 CVEs cold** (Nomad, node-tmp, FUXA, UEFI parser) — strict file/line/CWE | ✅ CVSS 4.3–7.5 | ✅ #1 HackerOne US | -| **Live-CTF / public bounty triage** | ⏳ pending (next campaign) | ✅ | ✅ | - -**Read:** t3mp3st's differentiator is the *re-derivable measurement discipline* — -rigor, reproducibility, and a matched-metric number on the one sourced benchmark, -all re-computable from committed artifacts. On *published real-world receipts,* -CAI/XBOW are ahead — that's an **evidence gap, not a capability gap** (the -architecture is already here). This is a practice we can re-derive, not a rank we -claim over anyone. - ---- - -## Honest gaps (the roadmap that earns the rest of the crown) - -1. **Live-CTF win** — enter an AI-vs-Human CTF; the Mission-Coordinator + operator swarm is built for exactly this. -2. **HackTheBox ranking** — point the kill-chain at HTB; a public rank is unfalsifiable. -3. **Real CVEs** — unleash `cve-hunt`/`cve-zero` on real OSS for responsible disclosure (the XBOW move). -4. **CAIBench head-to-head** — run the incumbent's own meta-benchmark under our provenance-strict scorer and report the matched number. -5. **Variance bounds** — most numbers are N=1; report pass@k with CIs. - ---- - -## The one-sentence version (use this, with the asterisk) - -> *"T3MP3ST is an open-source offensive-security harness built on a re-derivable -> measurement discipline: on XBOW's own benchmark, a hint-free black-box pass@1 mean of -> 90.1% (Wilson95 [86.2%, 92.9%]) whose CI floor already clears XBOW's self-reported 85% -> — with a conservative single-run floor 91/104, best single sweep 96/104, and best-ball -> 98/104 (94.2%, pass@3 union of 3 gpt-5.5 sweeps) all reported separately, never blended -> — plus a contamination-audited hint-free Cybench run (23/40) and an honestly-labeled -> 8-operator kill-chain framework (Recon is the live engine; downstream phases are -> scaffolding) — every number reproducible from the repo, unlike the -> self-reported field."* - -Asterisk: *the live-CTF / bug-bounty crown is contested by CAI and is the -pending work; we stand on a re-derivable measurement discipline and benchmark -reproducibility, not on real-world receipts — yet.* - -[^floor]: The **floor** (worst single sweep) is run-time strict-graded: some -floor-solve transcripts are truncated, so not every individual floor solve is -independently artifact-re-derivable. The **mean** and **best-ball** numbers -re-derive fully from the committed artifacts via `scripts/verify-claims.mjs`.