mm/t3mp3st
0
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
52ff30de914d998d3c8c66b93c9938eb9d6508ad
t3mp3st/README.md
T
N3thunt3r69andJoseph Magly 52ff30de91 feat(llm): resume the Claude Code session across a kill-chain phase (#149) 
* feat(llm): resume the Claude Code session across a kill-chain phase

Every ReAct iteration flattens the growing transcript into a brand-new
prompt and spawns a fresh `claude -p` process. Within one task that is
a quadratic resend; across a mission it also means each spawn re-pays
Claude Code's own CLAUDE.md/skills/MCP bootstrap tax from scratch,
since T3MP3ST never told the CLI these calls belong to one session.

LocalAgentAdapter now tracks the Claude Code session id from the JSON
envelope and passes it back in with --resume on every later call, so
the CLI carries the accumulated transcript itself and the repeated
part is billed at cache-read pricing instead of resent. The session
is scoped to one kill-chain phase: TempestCommand drops it for every
operator right after advancePhase(), since local-agent operators are
spawned once for the whole mission and would otherwise carry one
session across all 7 phases. A stale or expired session id fails fast
(confirmed against the real CLI: nonzero exit, no JSON on stdout) and
localAgentChat retries once without --resume rather than failing the
task over it.

Verified against the real CLI: a resumed call recalled prior context
correctly and dropped from $0.151 to $0.008 (cache_read_input_tokens
absorbing the prior turn instead of cache_creation), and an invalid
--resume id triggered the fallback and still succeeded. Ran live
against an authorized target: the same session id held across all 4
recon tasks in one phase, then a newly spawned operator's first call
in the next phase carried no --resume, confirming both the resume and
the reset.

Full suite: 764/764, no regressions.

* fix(llm): send only the delta on a resumed Claude Code call

jmagly's PR #149 review caught it: LocalAgentAdapter.chat() built its
prompt from the full messages array unconditionally, so a resumed
call sent the whole growing transcript again on top of a session that
already had it. The resend the feature exists to remove was still
happening, and the resumed session's own history now grew on top of
it too. Existing tests only asserted that sessionId reached the CLI,
never what was actually in the prompt, so this passed green.

LocalAgentAdapter now tracks the exact messages array object and how
many of its entries were already sent. A later call against the SAME
array (AgentLoop growing one task's transcript via push) sends only
the new tail. A DIFFERENT array (a new task) still sends everything,
since the resumed session has never seen that task's content even
though the CLI session itself carries over. The stale-session
fallback in localAgentChat now takes a separate fallbackPrompt: a
fresh session has no history, so its retry gets the full transcript,
never the delta sized for the failed resumed attempt.

Verified against the real CLI with a synthetic multi-round exchange
(~750 tokens of padding added between each call): promptTokens went
19127 -> 19625 -> 20135 -> 20645, a flat ~510 tokens per round
regardless of the accumulating transcript, instead of resending or
compounding. Added the regression coverage the review asked for: a
same-array resumed call excludes the prior turn's content while
retaining the tool contract, a new-array call still gets the full
prompt, and the fallback retry receives fallbackPrompt rather than
the delta, checked both at the LocalAgentAdapter call-shape level and
against a real spawned CLI script.

Full suite: 769/769, no regressions.

* fix(test): import LLMMessage in local-agent-tool-calling.test.ts

Hosted CI caught it, not my local run: the previous commit used
LLMMessage[] type annotations in two new tests without importing the
type, so tsc failed with TS2304 while vitest itself stayed green
(esbuild strips type annotations without checking them). I ran
typecheck once before writing those tests and never reran it after,
so this went out uncaught.

Full suite: 769/769. npx tsc --noEmit: clean.

* fix(llm): isolate per-operator Claude sessions and narrow the fallback

A self-review after the last round turned up a more serious version of
what jmagly caught: TempestCommand built exactly one LLMBackbone and
handed the SAME instance to every operator's AgentLoop (and to each
OperatorAgent's own decompose-on-failure path). LocalAgentAdapter's
new session/delta state lived on that shared instance, so one
operator's Claude Code session could get resumed by a completely
different operator, with the second operator's unrelated task content
sent into the first operator's conversation. Every test for the
resume feature so far constructed independent LLMBackbone instances
per test, which is not how spawnOperator() actually wires things, so
nothing caught it.

spawnOperator() now builds a fresh LLMBackbone per operator from the
same config, used for both that operator's AgentLoop and its own
decompose fallback, so state stays consistent within one operator and
isolated from every other one. OperatorCell.spawnOperator() takes an
optional llm override to carry that through; existing callers that
don't pass one keep the cell's shared default.

Two narrower issues from the same review:

localAgentChat's stale-session fallback retried on ANY failure of a
resumed call, not just a confirmed stale session. A real network
error or auth failure would silently double latency on a retry that
could not fix it, and compounded with LLMBackbone's own retry ladder
into up to 6 CLI spawns for one failing turn. The fallback now only
fires for the exact empirically-confirmed stale-session error text
("No conversation found with session ID").

The character-estimate usage fallback (used only when the JSON
envelope fails to parse) estimated from the delta prompt instead of
the full transcript. A resumed call's real promptTokens reflects the
whole cached conversation, confirmed empirically in the last round
(19127 -> 19625 -> 20135 -> 20645), so estimating from the small delta
text instead undercounted the budget check by an order of magnitude
in that one edge case.

One finding from the review was deliberately left alone: resending
the tool contract on every delta call is real overhead, but jmagly's
review explicitly required the contract stay on every call ("retaining
... the tool contract") and the existing test already pins that. Cutting
it would contradict that requirement for a small, bounded savings.

New coverage: an isolation test that spawns two operators through a
real TempestCommand and confirms neither's LocalAgentAdapter nor its
session leaks into the other, a test proving a non-stale failure
propagates without a fallback retry, and a test proving the usage
estimate reflects the full transcript on a mid-session parse failure.

Full suite: 772/772, no regressions.

* fix(llm): reset the Claude session at every task boundary

jmagly's third review: switching from delta to full-prompt sending on
a new messages array wasn't enough, since LocalAgentAdapter.chat()
still passed the OLD claudeSessionId either way. A second task on the
same operator inherited the first task's persisted Claude session,
carrying its system prompt, target data, and tool output (including
anything attacker-controlled) across a boundary T3MP3ST already has a
single sanctioned channel for: PackBoard.

chat() now clears claudeSessionId whenever the incoming messages
array is not the one it last saw, before building the request. A
different array means AgentLoop.run() started a new task; the call
after that reset carries no --resume, so it opens a genuinely fresh
session instead of resuming the old one under a bigger prompt. Within
one task the array is the same object across iterations (AgentLoop
grows it via push), so resumption and the delta-only send from the
last round are untouched.

Rewrote the tests that asserted continuity across separate array
literals, since that shape doesn't happen in AgentLoop and was masking
this exact gap: they now grow one array via push to represent a real
task, matching the pattern already used for the delta tests. Added
the test jmagly asked for directly: one operator, two sequential
tasks, task 2 has no session id, both at the LocalAgentAdapter level
and through a real TempestCommand.spawnOperator().

Full suite: 773/773, no regressions.

---------

Co-authored-by: Joseph Magly <1159087+jmagly@users.noreply.github.com>
2026-08-23 17:12:27 -04:00

342 lines
26 KiB
Markdown
Executable File
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 🌩️ T3MP3ST 🌩️
<!-- ⊰ sharp eye on the raw source. there's a flag for the curious: T3MP3ST{r3c31pt5_n0t_v1b3z} — the one that counts, you earn: run `npm run verify-claims`. LOVE PLINY ⊱ -->
```
▄▄▄█████▓▓█████ ███▄ ▄███▓ ██▓███ ▓█████ ██████ ▄▄▄█████▓
▓ ██▒ ▓▒▓█ ▀ ▓██▒▀█▀ ██▒▓██░ ██▒▓█ ▀ ▒██ ▒ ▓ ██▒ ▓▒
▒ ▓██░ ▒░▒███ ▓██ ▓██░▓██░ ██▓▒▒███ ░ ▓██▄ ▒ ▓██░ ▒░
░ ▓██▓ ░ ▒▓█ ▄ ▒██ ▒██ ▒██▄█▓▒ ▒▒▓█ ▄ ▒ ██▒░ ▓██▓ ░
▒██▒ ░ ░▒████▒▒██▒ ░██▒▒██▒ ░ ░░▒████▒▒██████▒▒ ▒██▒ ░
▒ ░░ ░░ ▒░ ░░ ▒░ ░ ░▒▓▒░ ░ ░░░ ▒░ ░▒ ▒▓▒ ▒ ░ ▒ ░░
░ ░ ░ ░░ ░ ░░▒ ░ ░ ░ ░░ ░▒ ░ ░ ░
░ ░ ░ ░ ░░ ░ ░ ░ ░ ░
░ ░ ░ ░ ░ ░
```
<div align="center">
**A multi-agent offensive-security framework, built to turn the AI coding agent you already run into a zero-day hunter.**
![scores: re-derivable](https://img.shields.io/badge/scores-re--derivable-brightgreen) &nbsp; ![verify-claims 27/27](https://img.shields.io/badge/verify--claims-27%2F27-brightgreen) &nbsp; ![PRs welcome](https://img.shields.io/badge/PRs-welcome-purple) &nbsp; ![License: AGPL-3.0](https://img.shields.io/badge/License-AGPL--3.0-blue)
</div>
**Your AI coding agent is already a hacker — T3MP3ST hands it an arsenal.**
Point it at an authorized target and the kill chain runs itself: **recon → exploit → report**, from a browser War Room or the CLI, driven by the agent you're *already* signed into — Claude Code, Codex, Hermes, OpenCode, Oh My Pi — or a model you run **fully offline** (Ollama, LM Studio, vLLM). No new API keys, no cloud tenant, no second bill. Your agent is the brain; T3MP3ST is the war machine bolted around it. **Self-hosted storm. Keyless warfare.** ⚡
And it won't ask you to take its word for it. On **XBOW's own 104-challenge suite it scores 90.1% pass@1** — above XBOW's self-reported 85% — alongside hint-free CTF solves and a **cold hunt on real, post-cutoff CVEs the model had never seen**. Every number in this README recomputes from committed data with one command (`npm run verify-claims`). Loud about the mission, honest about the build — the [status table](#what-ships-today) says exactly what's live, what's scaffolding, and what's still roadmap; full receipts in [Benchmarks](#benchmarks).
Three things set it apart:
1. **Reproducible.** Every number in this README recomputes from committed data — `npm run verify-claims` re-derives all of them, 27/27 green. A claim that can't be reproduced doesn't ship. No trust-me numbers, ever.
2. **Keyless.** The AI coding agent already on your machine is the backbone. No API keys, no second bill, no gatekeeper.
3. **Honest about scope.** The [status table](#what-ships-today) marks exactly what's stable, experimental, or roadmap — because red-teaming shouldn't be a priesthood, and it damn sure shouldn't run on vibes.
**Jump to** → [Quick start](#quick-start) · [Usage guide](docs/GETTING_STARTED.md) · [Developer guide](docs/DEVELOPER_GUIDE.md) · [Updating](#updating-from-upstream) · [What it hunts](#what-it-hunts) · [What ships today](#what-ships-today) · [Benchmarks](#benchmarks) · [Architecture](#architecture) · [Docs](#documentation)
## ⚠️ Authorized use only
T3MP3ST is an **offensive** security tool, built for **authorized** testing, research, and education. Point it **only** at systems you own or have **explicit, written permission** to test. Unauthorized access to computers, networks, or data is illegal in most jurisdictions — **you alone are responsible** for how you use this software and for staying inside the law and your rules of engagement. Bring the storm to *your* targets, not someone else's.
T3MP3ST is provided **as-is under the AGPL-3.0 license, with no warranty and no liability** for any damage, loss, or misuse. The authors do not endorse, support, or condone unauthorized activity. Get permission. Stay in scope. Don't be a menace. 🫡
## Why it exists
Offensive security sits behind years of practice and expensive tooling. The bet behind T3MP3ST is that a coordinated agent swarm puts real bug-hunting in reach of people who never got the invite, across web apps, CTFs, smart contracts, source code, and embedded/robotics OSS. That is an ambitious bet, and the sections below are careful to separate what already works from what is still a bet.
## What it hunts
| Domain | What it does | Status |
|---|---|---|
| 🕸️ **Web apps** | Black-box, external-attacker recon → exploit (XBEN suite) | ✅ Stable |
| 🚩 **CTF** | Hint-free, sandbox-jailed solves (Cybench) | ✅ Stable |
| 🤖 **Robotics / OT / embedded** | Coordinated-disclosure pipeline for OSS vuln hunting (OSV + live-PoC + refuter) | ✅ Pipeline stable |
| 📂 **Source code** | White-box repo analysis with blind master-builder decomposition | ✅ Multi-language ingest (web-tree-sitter) |
| 💰 **Smart contracts** | Damn Vulnerable DeFi | ⚠️ reproduction, not novel discovery |
| ☁️ **Cloud (IaC)** | Misconfig-detection benchmark (`cloud:bench`) + opt-in cloud arsenal (aws/az/gcloud + scoutsuite/cloudfox/pmapper; pacu gated) | 🚧 IaC-misconfig scaffolding — live-cloud exploitation not yet benchmarked |
| 📱 **Mobile** | Built-in static analyzer (manifest misconfig + secret/cleartext detection, `mobile:bench`) + opt-in arsenal (mobsfscan/objection/drozer; frida gated) | 🚧 static-detection scaffolding — dynamic exploitation not benchmarked |
| 🔩 **Binary / RE** | Decompiled-output sink detector (unsafe-copy / format-string / cmd-injection / int-overflow, `binary:bench`) + opt-in arsenal (ghidra/radare2/objdump/checksec/strings; gdb gated) | 🚧 static sink-detection scaffolding — solving/pwn not benchmarked |
## Quick start
Fastest path to a running War Room (keyless, ~2 min to set up; mission time depends on the target):
```bash
npm install
npm run server # War Room → http://127.0.0.1:3333/ui/
```
In the War Room, open **Settings** and connect a local agent (Claude Code / Codex / Hermes / OpenCode / Oh My Pi). Then describe a target to **Op Admiral** in plain English and launch. The agent you connected is the brain. No key required.
Prefer to bring a key? Set one and skip the connect step:
```bash
export OPENROUTER_API_KEY=... # or VENICE_API_KEY / ANTHROPIC_API_KEY / OPENAI_API_KEY
export XAI_API_KEY=... # Grok Build (grok-build-0.1) — xAI's coding model, native tool-calling
export NOVITA_API_KEY=... # Novita AI's hosted OpenAI-compatible API
```
Novita AI is an opt-in third-party hosted provider. Requests send the selected
model id, prompts/context, generated output, and request metadata to Novita;
your API key authenticates those requests. Review Novita's current
[privacy policy](https://novita.ai/legal/privacy-policy) and
[terms](https://novita.ai/legal/terms-of-service) before using sensitive target
data. T3MP3ST does not claim independent assurance for sensitive workloads.
Slow local agents can be given more room with `T3MP3ST_LOCAL_AGENT_TIMEOUT_MS`
for each CLI call, `T3MP3ST_TASK_TIMEOUT_MS` for mission tasks, and
`T3MP3ST_GENERAL_TIMEOUT_MS` for planning requests. Values are milliseconds.
Claude Code session reuse is disabled by default because a resumed coding-agent
session may retain provider-side context and ambient tool authority outside the
Arsenal receipt boundary. Operators who explicitly trust their local Claude
Code configuration as a separate execution authority may set
`T3MP3ST_TRUST_CLAUDE_SESSION=1`; reuse remains isolated to one task and the
choice must not be treated as an Arsenal-enforced sandbox.
Or run it **fully offline** on your own model — no key, no cloud. Defaults to Ollama; point it at any OpenAI-compatible server (LM Studio, vLLM, llama.cpp):
```bash
ollama serve && ollama pull llama3 # or an OpenAI-compatible server
export TEMPEST_LOCAL_BASE_URL=http://localhost:11434/api # LM Studio: http://localhost:1234/v1
export TEMPEST_LOCAL_MODEL=llama3
npm run build # only needed from a git clone
npx tempest # → "Change default provider" → local
```
Tool-calling works on any local model (it's driven over text), so the Arsenal runs even on models without native function-calling.
Check the numbers for yourself:
```bash
npm run verify-claims # re-derives every headline from committed JSON in bench/
```
Step-by-step operator usage lives in [Getting Started](docs/GETTING_STARTED.md). Library/SDK usage, the HTTP API, and MCP setup live in [docs/](docs/).
### Docker
Run T3MP3ST API server in a container (localhost only, not exposed externally):
```bash
cp .env.example .env # configure API keys
docker compose up -d # API → http://localhost:3333
docker compose logs -f # view logs
```
**Security Note:** Container binds to `127.0.0.1:3333` - accessible only from localhost, not exposed to network.
Test the API:
```bash
curl http://localhost:3333/api/health
curl http://localhost:3333/api/bounty/platforms
```
Execute commands inside the container:
```bash
docker compose exec app npm run verify-claims
docker compose exec app npm run cve:bench
```
Full deployment guide: [docs/DOCKER.md](docs/DOCKER.md).
## Updating from upstream
If you installed from a release tarball or copied the tree instead of tracking `git pull`, use the built-in updater to sync with [github.com/elder-plinius/T3MP3ST](https://github.com/elder-plinius/T3MP3ST) without losing local secrets or bench output. It shows a numbered plan, asks **y/N** before changing anything, then runs `npm install`.
```bash
npm run update # interactive sync from upstream main
npm run update:dry # preview only — no git or npm changes
npm run update:hard # hard reset to upstream/main (still restores protected paths)
```
Works on Windows (PowerShell), macOS, Linux, and WSL. Requires **git** and **npm** on your PATH.
### Safety modes
The updater is destructive **only when explicitly requested**:
| Command | What it does | Local changes |
|---|---|---|
| `npm run update` | Default interactive sync | `git merge upstream/main`; protected paths backed up and restored |
| `npm run update:dry` | Read-only preview | No git init, fetch, merge, reset, or `npm install`; safe on tarball installs |
| `npm run update:hard` | Opt-in hard reset | `git reset --hard upstream/main`; protected paths still restored |
All non-dry-run modes ask **y/N** before changing anything. Pass `--force` only in trusted automation. On the first-time path (no commits yet), the updater replaces the working tree with the upstream snapshot, but protected paths are backed up first and restored afterward.
### Protected paths (inside the repo)
Before replacing files, the updater backs up anything on disk that matches [`scripts/update-protected.txt`](scripts/update-protected.txt), then restores it after sync. **Only paths that exist locally are affected** — if you never created them, nothing happens.
| Path | Why it's protected |
|---|---|
| `.env`, `.env.*` | API keys and local env overrides. `!.env.example` is an exception — the template is **not** protected so upstream can update it. |
| `.keys.local` | One-off key paste file loaded by bench scripts (e.g. `VENICE_API_KEY`) without touching `.env`. |
| `.keys.bounty.json` | HackerOne / Bugcrowd / similar platform credentials. |
| `bench/cybench/corpus-stage/` | Large cloned Cybench corpus (not redistributed; expensive to re-download). |
| `bench/cybench/service-stage/`, `bench/cybench/challenges/` | Per-run Cybench Docker staging and challenge trees (regenerable, but slow to rebuild). |
| `bench/xbow/stage/`, `bench/xbow/challenges/` | XBOW/XBEN challenge staging (large third-party trees). |
| `bench/wild-hunt/` | Cold-hunt findings, PoCs, disclosure drafts, and campaign results — pre-coordination vuln material. |
| `bench/decomposition-results/` | White-box decomposition run JSON (may contain unreported analysis). |
| `bench/refusal-frontier/` | Refusal-boundary probe artifacts (raw model responses). |
| `bench/nyu/` | Staged NYU CTF content from `nyu-prep.mjs`. |
| `docs/disclosures/` | Generated vendor disclosure packages (`disclosure-gen` output). |
| `reports/` | Engagement and hunt reports (persistent output; same tree as Docker volume mounts in deployment setups). |
| `evidence/` | PoCs, screenshots, and other finding evidence kept across updates. |
Add your own patterns in `scripts/update-protected.local.txt` (optional local overlay; same glob syntax as the main manifest).
### Never touched (outside the repo)
These live outside the project tree — an update never reads or writes them:
- `%APPDATA%\t3mp3st-nodejs\Config\config.json` (or the macOS/Linux `conf` store path) — saved by `npm run setup`
- War Room browser **localStorage** on the War Room origin
- Local agent auth (`~/.codex`, `%LOCALAPPDATA%\hermes`, etc.)
## What ships today
The framework is an 8-operator kill chain, and this table won't blow smoke about it. **Recon is a live, tool-backed engine** — and the teeth are already real: 90.1% pass@1 on XBEN, 8/10 held-out post-cutoff CVEs pinned to exact file/line/CWE, and a coordinated-disclosure pipeline that's live enough to have drafts held for vendor coordination right now. What's *not* proven is the swarm. Each downstream operator — Exploiter, Infiltrator, Exfiltrator, Ghost — runs the **same real, tool-backed ReAct loop as recon** (real exploit tools, not stubs), but the headline numbers came from a single agent, not the coordinated 8-operator cell, and end-to-end swarm exploitation is unbenchmarked and still unreliable. The engine is real; the swarm is the part still earning its stripes. Loud where we've earned it, blunt about the rest.
| Component | Status | Notes |
|---|---|---|
| Re-derivable measurement (`verify-claims`) | ✅ Stable | every headline recomputes from committed artifacts |
| Recon engine | ✅ Stable | drives nmap / DNS / HTTP / fingerprinting; every finding traces to real tool output |
| Mission engine + War Room + Op Admiral | ✅ Stable | keyless through a connected local agent |
| Arsenal, MCP server, HTTP API | ✅ Stable | 36 built-in tools by default; 109 with the opt-in `T3MP3ST_FULL_ARSENAL` (+73 adapters, with dangerous/catalog-only drivers — metasploit, hydra, pacu, frida — behind narrow approved paths rather than generic execution) — both counts re-derive via `verify-claims`. `security_recon` over MCP |
| Egress-scope containment | ✅ Stable (on by default) | once a mission target is set, built-in networked tools refuse off-scope public hosts — not the target/subdomains, not loopback/private (`SCOPE DENIED`) — a tightened default, not a bare tool runner |
| Coordinated-disclosure pipeline | ✅ Stable | OSV novelty + live PoC + refuter panel + CVSS; drafts only, a human sends |
| White-box source analysis | ⚠️ Experimental | Multi-language ingest via web-tree-sitter (Python/JS/TS/Go/Java/C/C++); Python retains its regex parser, while other languages fail open to no extracted blocks; multi-model decomposition costs more tokens, not fewer |
| DeFi (Damn Vulnerable DeFi) | ⚠️ Experimental | reproduces known exploit classes; not novel discovery |
| Exploiter / Infiltrator / Exfiltrator / Ghost | ⚠️ Experimental | run the real tool-backed ReAct loop (same engine as recon); unproven as a coordinated swarm — single-agent is the benchmarked path, live swarm exploitation still unreliable |
| Advanced modules (cloud, persistence, swarm, cognition) | 🚧 Planned | interface-only in `src/stubs/` |
| Self-improvement loop | 🧪 Research | records lessons + proposals today; feeding them back into planning is roadmap |
Full feature-by-feature breakdown: [FEATURES.md](FEATURES.md).
## Coverage by domain
Where the storm reaches today — and where it's headed. Same discipline as everything else: a domain is ✅ only when there's a receipt behind it.
| Domain | What it covers | Status |
|---|---|---|
| 🕸️ **Web** | apps, APIs, auth flows, OWASP Top 10 | ✅ **Core** — XBEN 90.1% pass@1 |
| 📂 **Code** | white-box source audits, SAST-style vuln hunting | ✅ **Proven (hunt result)** — held-out CVE-Zero: single-agent 8/10 exact file/line/CWE, 10/10 found (7 languages); the repo-ingest *engine* itself is still ⚠️ experimental |
| 🚩 **CTF** | wargames, practice ranges, challenges | ✅ **Proven** — Cybench 23/40 hint-free |
| 🔌 **Network / Infra** | recon, service/stack fingerprinting; lateral + privesc | ✅ recon (live nmap/DNS/HTTP engine) · ⚠️ lateral/privesc experimental |
| 🤖 **Embedded / IoT / OT** | firmware, robotics, ICS/SCADA OSS | ✅ **CVE pipeline live** — coordinated-disclosure drafts held for vendors |
| 📦 **Supply chain** | dependency audits, install-without-confirmation | ⚠️ **Real** — dedicated class; hit a CWE-829 on the held-out set |
| 💰 **Blockchain** | smart contracts, DeFi, Solidity | ⚠️ **Reproduction only** — Damn Vulnerable DeFi, not novel discovery |
| ☁️ **Cloud** | AWS/GCP/Azure misconfig, IAM, serverless | 🚧 **In development** |
| 📱 **Mobile** | Android/iOS app security | 🚧 **In development** |
| 🏢 **Identity / AD** | Kerberos, pass-the-hash, AD attacks | 🚧 **In development** |
| 🔐 **Binary / RE** | overflows, ROP, exploit dev | 🚧 **In development** — needs specialized tooling |
The class/squad architecture means new domains *compose* rather than fork — each is a loadout (specialist classes + arsenal + target adapter + a benchmark). 🚧 domains ship dark until they have a number.
## Benchmarks
Headline results. Each recomputes from the committed JSON with `npm run verify-claims`; full methodology and caveats are in the linked docs.
| Suite | Result | Context |
|---|---|---|
| **XBEN** — XBOW's 104-challenge suite, black-box | **pass@1 mean 90.1%** (Wilson-95 86.2–92.9), floor 91/104 · gpt-5.5 | XBOW self-reports 85% on the same suite; ours re-derives the graded verdict from committed artifacts (raw transcripts stripped for privacy) |
| **XBEN** — white-box (reported separately) | pass@1 98.7%, best-ball 104/104 · gpt-5.5 | never blended with the black-box number |
| **Cybench** — 40-task academic bench, Opus 4.8, no hints | **23/40 (58%) hint-free, single-run pass@1** (`verify-claims`-enforced) | not the raw-score record (Anthropic: 76.5% pass@10); every flag graded against the committed oracle |
| **Cybench model matrix** — identical 15-task committed subset, pass@1 | **Opus 4.7 vs 4.8**, with per-task source receipts and separate failure/abstention/infrastructure outcomes | [rebuild and inspect the model/harness matrix](docs/MODEL_MATRIX.md); historical system comparison, not an isolated model ranking |
| **CVE-Zero** — 10 real post-cutoff (2026) CVEs, **held-out**, 7 languages | **single-agent 8/10 exact file/line/CWE** (verified all-exact, stable) · **10/10 found** (full pack) | **memorization- & fitting-proof**: post-cutoff, and the hardened prompts were never tuned on these; `verify-claims` recomputes it. n=10, directional; the swarm's edge here is recall, not a coordination-beats-solo proof |
**How to read these:**
- Every solved flag is graded against a committed ground-truth oracle — not a self-report — and `verify-claims` recomputes the pass/fail. Raw per-step transcripts are stripped for operator privacy, so you re-check the **graded verdict**, not the raw tool output. Zero fabricated, enforced by an anti-fitting guard that runs on every push.
- Black-box (source withheld) and white-box (source staged) are reported separately and never blended.
- These ran a **single-agent ReAct loop, not the 8-operator swarm.** The swarm is framework architecture; it is not what scored these numbers.
- Results are system-vs-system: this harness driving a strong current model, not an isolated-harness claim.
The number isn't the flex — the **receipt** is. A keyless, open-source harness that hands you the re-run instead of asking you to trust it: clone it, run `npm run verify-claims`, and every verdict above recomputes from its committed oracle in front of you.
Deeper reading: [WALL_FORENSICS](docs/WALL_FORENSICS.md) (per-challenge misses), [CYBENCH](docs/CYBENCH.md), [INTEGRITY_LEDGER](docs/INTEGRITY_LEDGER.md) (contamination audit and every retraction), [OBSIDIVM](docs/OBSIDIVM.md) (our own live web range).
## Documentation
| Doc | Contents |
|---|---|
| [Docs index](docs/README.md) | operator, developer, benchmark, and release documentation map |
| [Getting Started](docs/GETTING_STARTED.md) | install, first launch, first safe mission, CLI basics, updates, and troubleshooting |
| [Developer Guide](docs/DEVELOPER_GUIDE.md) | source map, scripts, SDK usage, extension points, and release checks |
| [API Reference](docs/API_REFERENCE.md) | local HTTP API route groups and integration notes |
| [MCP Guide](docs/MCP_GUIDE.md) | MCP server setup and `security_recon` usage |
| [FEATURES.md](FEATURES.md) | feature-by-feature status (`[x]` shipped / `[~]` partial / `[ ]` planned) |
| [SCOPE_AND_AUTHORIZATION](docs/SCOPE_AND_AUTHORIZATION.md) | authority model, scope receipts, evidence and retest rules |
| [VERIFIED_PROVENANCE](docs/VERIFIED_PROVENANCE.md) | how findings become tool-proven instead of model-asserted |
| [CONTRIBUTION_RECEIPTS](docs/CONTRIBUTION_RECEIPTS.md) | PR receipt template for scope, run mode, model/harness labels, redaction, and verification |
| [MODEL_MATRIX](docs/MODEL_MATRIX.md) | Reproducible cross-model benchmark matrix and arbitrary variant-test model selection |
| [TEAM_PREVIEW](docs/TEAM_PREVIEW.md) | first-run path and review script |
| [INSTALL_MATRIX](docs/INSTALL_MATRIX.md) | macOS / Linux readiness table |
| [ARSENAL_ACTIVATION_PLAN](docs/ARSENAL_ACTIVATION_PLAN.md) | optional external-tool setup |
| [PULL_REQUEST_DELIVERY](docs/PULL_REQUEST_DELIVERY.md) | contributor and maintainer checklist for scoped, reviewable PRs |
| [CYBENCH](docs/CYBENCH.md) · [WALL_FORENSICS](docs/WALL_FORENSICS.md) · [INTEGRITY_LEDGER](docs/INTEGRITY_LEDGER.md) · [COGNITIVE_ARCHITECTURE](docs/COGNITIVE_ARCHITECTURE.md) | benchmark methodology |
| [RELEASE_CHECKLIST](docs/RELEASE_CHECKLIST.md) | the gates a release must pass |
## Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ T3MP3ST COMMAND │
├─────────────────────────────────────────────────────────────────┤
│ MISSION CONTROL ◄── TARGET MODEL ──► ARSENAL (TOOLS) │
│ ▲ │
│ AGENT CELL: RECON · SCANNER · EXPLOITER · INFILTRATOR · │
│ EXFILTRATOR · GHOST · COORDINATOR · ANALYST │
│ ▲ │
│ EVIDENCE VAULT · CREDENTIAL STORE · FINDINGS LEDGER │
│ ▲ │
│ OPSEC LAYER · COMMS CHANNEL · LLM BACKBONE │
└─────────────────────────────────────────────────────────────────┘
```
Operators map to MITRE ATT&CK and Cyber Kill Chain phases (recon is live; later phases are scaffolded):
| Operator | Phase | MITRE | Function |
|---|---|---|---|
| **Recon** | Reconnaissance | TA0043 | OSINT, network discovery, asset enumeration |
| **Scanner** | Discovery | TA0007 | vulnerability scanning, service fingerprinting |
| **Exploiter** | Initial Access | TA0001 | exploitation, payload delivery |
| **Infiltrator** | Lateral Movement | TA0008 | post-exploitation, privilege escalation |
| **Exfiltrator** | Collection / Exfil | TA0009/10 | data extraction, credential harvesting |
| **Ghost** | Persistence | TA0003 | persistence, stealth, cleanup |
| **Coordinator** | Command & Control | TA0011 | mission control, orchestration |
| **Analyst** | Analysis | — | pattern analysis, reporting |
**Providers:** OpenRouter, Venice, Anthropic, OpenAI, or a keyless local agent (Claude Code / Codex / Hermes / OpenCode / Oh My Pi). Set `OPENROUTER_API_KEY` / `VENICE_API_KEY` / `ANTHROPIC_API_KEY`, or connect an agent in Settings.
**Integrations:** `node dist/mcp-server.js` exposes `security_recon` to MCP-aware agents. `npm run server` starts the HTTP API (`POST /api/mission/start`, `GET /api/mission/status`, and more). Full reference in [docs/](docs/).
## Contributing — join the swarm
Red-teaming shouldn't be a priesthood. Bring an adapter, a prompt pack, a runbook, a new arsenal tool, or a bug report.
**One rule, non-negotiable:** everything here is for **authorized testing only**. Owned, scoped, or consenting targets. Build for defenders, or don't build it here.
1. Fork it, branch it.
2. Open a PR with tests. If you touch a headline number, `npm run verify-claims` has to stay green.
Release process and gates: [RELEASE_CHECKLIST](docs/RELEASE_CHECKLIST.md).
## License
AGPL-3.0. See [LICENSE](LICENSE).
---
<div align="center">
*Fortes fortuna iuvat* — fortune favors the bold.
⊰•-•✧ LOVE PLINY ✧•-•⊱ 🌩️
</div>
Reference in new issue View Git Blame Copy Permalink