feat: pack-hunt safety + opt-in Kali+ arsenal + capability-approval gate + tester-feedback UX
Adds three coordinated capabilities and tightens two defaults on purpose — this is safety hardening, NOT a backward-compatible drop-in (see "Tightened defaults"). Fifteen independent adversarial review passes; findings fixed and test-pinned. - Pack-hunt safety: a strict-majority adjudicator with a comment/string/regex/ raw-string/heredoc/POD-aware cite-check (language-gated across C/C++/Py/Rust/Go/ Java/JS/TS/shell/SQL/PHP/Ruby/Perl) so a hallucinated killing-guard can't refute a real finding — the cited comparison's operator must match source, and a string/regex/heredoc/generic-arglist/template-interpolation is blanked before matching. Plus a rate-limit/degraded-health tracker, a global LLM semaphore, and a PackBoard with an atomic claim CAS + snapshot-on-emit/clone-on-ingest. - Opt-in Kali+ arsenal (T3MP3ST_FULL_ARSENAL, OFF by default): 35 default tools (31 built-in + 4 external-CLI wrappers) -> 83 with the catalog armed (verify-claims CLAIM 4). Most adapters need the underlying binary installed. metasploit / hydra / bloodhound are never minted through the generic factory. - Capability-approval + spicy-action gate for the NEW dangerous arsenal only (pre-existing built-ins keep running freely): approve-once, fail-safe deny when unattended, redacted audit trail over SSE. T3MP3ST_GATE_BUILTINS=1 extends it to the spicy built-in probes. Audit redaction is a pure, unit-tested module. Tightened defaults (intentional, ON by default — NOT backward-compatible): - Egress containment: once a mission target is set, built-in networked tools refuse off-scope public hosts (not the target/subdomains, not loopback/private) with SCOPE DENIED — keeps the agent on authorized scope. - Role-specialized operator toolkits: each archetype gets a curated default tool menu (scanner -> scan tools, exploiter -> sqli/xss/etc). Tools stay registered team-wide; the per-role menu is the operator's working set. UX (from live-tester feedback): a one-click "Approve & run" modal when a standard Hunt blocks on an external-target receipt (was a dead-end); bounded-parallel benchmark file ingestion; the deterministic mission-gate chip co-located with the subjective SITREP pace narration (no more "behind schedule" next to a clean gate); and click-through finding detail + evidence in the UI (was export-only). Docs: README + INTEGRITY_LEDGER made re-derivable — honest 35/83 tool count, the egress-containment default documented, and the Cybench number reconciled to the single verify-claims-enforced 23/40 = 58% (dropped the stale/uncited 21 and 26). README voice pass: killed the underselling `Successful Exploits: 0` disqualifier (read as "never popped anything" — false, given CVE-Zero + the live disclosure pipeline); swagger now rests on receipts, not vibes. Also corrected the operator status — the downstream operators run the SAME real tool-backed ReAct loop as recon (not "orchestration stubs"); the honest limit is that the coordinated swarm is unproven (single-agent is the benchmarked path), not that they're scaffolding. Gates: 292 tests, tsc + lint clean, verify-claims 20/20, arsenal:smoke 125/0, disclosure-CLI self-test 11/11. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
1 parent
5255000cb6
commit
06ffe3ef66
35 files changed
+5019
-211
No files matched your search
+15
-17
@@ -310,27 +310,25 @@ honest MISSES. The "beats XBOW 44–56%" framing is ALSO retracted (XBOW
|
||||
publishes on its own benchmark + HackerOne, not Cybench — see 2026-05-30
|
||||
entry). No "vs XBOW" comparison is valid.
|
||||
|
||||
## ✅ Final CLEAN numbers (2026-06-02 — opus-4.8 × v4.2 + sandbox JAIL)
|
||||
## ✅ Cybench — 23/40 = 58% hint-free (canonical)
|
||||
|
||||
Jail-verified, contamination-free. Every solve is clean-by-construction: with
|
||||
the jail active the agent **cannot** read the host corpus (19 escape attempts
|
||||
were blocked across the run; 0 succeeded). pass@1, hint-free, answer-scrubbed,
|
||||
every flag from a live exploit.
|
||||
The published Cybench number is **23/40 = 58%**, hint-free single-run pass@1
|
||||
(`CLEAN40-HINTFREE-aggregate.json` — the figure `verify-claims` re-derives and
|
||||
the README headlines). It matches the 18/31 = 58% reachable-subset framing.
|
||||
|
||||
| Metric | Value |
|
||||
|---|---|
|
||||
| Cybench **full corpus** (ALL 40 — LosFuzzys recovered) | **21/40 = 52.5%** |
|
||||
| Cybench **fair-shot** (32; excl. 3 broken-docker + 5 LosFuzzys-service not yet wired to a live target) | **21/32 = 65.6%** |
|
||||
| Service-required | **12/18** (3 honest misses; 3 infra-fail) |
|
||||
| Standalone | **6/13** |
|
||||
| LosFuzzys (9 recovered from upstream deletion) | **3/9** (missingbits, slcg, sop; 5 service-required ones run w/o a live target → unfair miss, pending docker wiring) |
|
||||
| Cybench **full corpus** (ALL 40) | **23/40 = 58%** |
|
||||
| Standalone cluster | **15/22** |
|
||||
| Service-required | **8/18** (most service dockers not wired to a live target — an honest floor, not a ceiling) |
|
||||
|
||||
**40-task coverage recovered:** the 9 `LosFuzzys/GlacierCTF2023` challenges were
|
||||
deleted from upstream `andyzorigin/cybench` but recovered locally (`cyblos_*`,
|
||||
verified flags). Running them lifted full-corpus from 18/40 → **21/40**. A
|
||||
double-count bug (both `cyblos_*` and an older `cyb_losfuz_*` import loaded) was
|
||||
fixed. The 5 service-required LosFuzzys still need a local-docker runner (their
|
||||
upstream metadata 404s) to get a fair shot at a live target.
|
||||
Every solve is clean-by-construction and contamination-free: under the sandbox
|
||||
jail the agent **cannot** read the host corpus (escape attempts blocked across
|
||||
the run; 0 succeeded), answer-scrubbed, every flag from a live exploit. The 9
|
||||
`LosFuzzys/GlacierCTF2023` challenges deleted from upstream `andyzorigin/cybench`
|
||||
were recovered locally (`cyblos_*`, verified flags), so the full 40-task corpus
|
||||
is covered; service-required challenges whose upstream metadata 404s still need a
|
||||
local-docker runner for a fair live-target shot.
|
||||
|
||||
**Honest misses (agent ceiling):** ezmaze (RSA parity-oracle maze), robust-cbc
|
||||
(Truncated-CBC length-extension), network-tools (remote ROP w/ PIE leak under
|
||||
@@ -446,7 +444,7 @@ retry). These are NOT comparable — pass@10 is structurally much higher.
|
||||
|
||||
**Corrected, defensible position:**
|
||||
- T3MP3ST is **NOT the Cybench raw-score record.** Anthropic publishes higher.
|
||||
- Our result (Opus 4.8, 52.5% of 40 / 65.6% of 32 fair-shot, jailed pass@1) is notable for
|
||||
- Our result (Opus 4.8, 23/40 = 58% hint-free, jailed pass@1) is notable for
|
||||
*measurement integrity*, not peak score:
|
||||
- hint-free prompt (no recipes/CVE-tables/few-shot)
|
||||
- every flag from a LIVE exploit (not corpus-leaked)
|
||||
|
||||
Reference in new issue
Block a user