docs: add usage and developer documentation
Add user and developer documentation plus Pagenary docsite publishing support. Includes Getting Started, Developer Guide, API and MCP references, Pagenary tenant metadata, npm scripts for sync/build/check/serve, and validation coverage for the generated docsite. Closes #102
This commit is contained in:
Joseph Magly authored and GitHub committed
1 parent
7b1d12a07c
commit
0c35a833b9
39 files changed
+4920
-2
No files matched your search
@@ -0,0 +1,180 @@
|
||||
---
|
||||
title: "Obsidivm"
|
||||
summary: "Local web range and evaluation notes."
|
||||
description: "Local web range and evaluation notes."
|
||||
audience: ["researcher", "operator"]
|
||||
category: "benchmarks"
|
||||
status: "current"
|
||||
source: "T3MP3ST repository"
|
||||
sourcePath: "docs/OBSIDIVM.md"
|
||||
updated: "2026-07-20"
|
||||
---
|
||||
# T3MP3ST ⇄ OBSIDIVM
|
||||
|
||||
T3MP3ST is the weapon. OBSIDIVM is the range. This integration makes them
|
||||
peanut butter and chocolate: t3mp3st runs ops, OBSIDIVM provides targets +
|
||||
canonical scoring, and the two flow in both directions.
|
||||
|
||||
```
|
||||
t3mp3st (:3333) OBSIDIVM (:4200)
|
||||
┌─────────────────────────┐ ┌──────────────────────────┐
|
||||
│ General + Operators │ ──────▶ │ /api/spec │
|
||||
│ Pliny endpoints │ ──────▶ │ /api/score/text │
|
||||
│ Evidence + Finding │ ──────▶ │ /api/runs │
|
||||
│ ledger │ │ /api/runs/:id/sessions │
|
||||
│ │ │ /api/deploy /destroy │
|
||||
└──────────┬──────────────┘ └─────────────┬────────────┘
|
||||
▲ │
|
||||
│ ▼
|
||||
┌──────────┴──────────────┐ ┌──────────────────────────┐
|
||||
│ :8889 agent shim │ ◀────── │ warroom.html (UI) │
|
||||
│ (t3mp3st-as-PLINY-CODE) │ ◀────── │ evolve.py (engine) │
|
||||
│ /api/launch /sessions │ │ │
|
||||
│ /api/session/log │ │ │
|
||||
│ /api/events │ │ │
|
||||
└─────────────────────────┘ └──────────────────────────┘
|
||||
```
|
||||
|
||||
## What you get
|
||||
|
||||
| Capability | How |
|
||||
|---|---|
|
||||
| **Use t3mp3st to attack OBSIDIVM** | `obsidivm-bench.mjs` reads targets from `/api/spec`, dispatches t3mp3st missions, posts transcripts to `/api/score/text`, records run in OBSIDIVM ledger. |
|
||||
| **Use OBSIDIVM warroom to drive t3mp3st** | `obsidivm-agent.mjs` exposes the `:8889` PLINY CODE protocol; OBSIDIVM warroom + `evolve.py` see t3mp3st as their agent without modification. |
|
||||
| **Multi-generation evolution** | OBSIDIVM's `/api/evolve/start` re-launches sessions and analyzes weaknesses. Pointing it at the `:8889` shim turns t3mp3st into the evolved agent. |
|
||||
| **Canonical scoring** | All scoring goes through `obsidium_spec.py`'s `score_text()` — same evidence rules used by warroom + tests. No drift. |
|
||||
|
||||
## Files added
|
||||
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| `scripts/obsidivm-bridge.mjs` | Thin HTTP client around OBSIDIVM's API. CLI + ES module. |
|
||||
| `scripts/obsidivm-bench.mjs` | End-to-end driver: dispatch hunter → score → record run. |
|
||||
| `scripts/obsidivm-agent.mjs` | `:8889` PLINY CODE protocol shim; t3mp3st-as-agent. |
|
||||
|
||||
## npm scripts
|
||||
|
||||
```bash
|
||||
npm run obsidivm # bridge CLI help
|
||||
npm run obsidivm:spec # dump OBSIDIVM target/expected list
|
||||
npm run obsidivm:bench # full-suite bench (defaults to stub hunter)
|
||||
npm run obsidivm:bench:stub # explicit stub mode (no LLM)
|
||||
npm run obsidivm:bench:live # direct LLM hunter
|
||||
npm run obsidivm:bench:t3mp3st # drive through t3mp3st platform
|
||||
npm run obsidivm:agent # serve :8889 for warroom + evolve.py
|
||||
```
|
||||
|
||||
## Setup once
|
||||
|
||||
```bash
|
||||
# 1. Start OBSIDIVM
|
||||
cd ~/Desktop/workspace/OBSIDIVM
|
||||
python3 range.py # listens on :4200
|
||||
|
||||
# 2. (Optional) Start t3mp3st server if you want the :8889 shim
|
||||
cd ~/Desktop/t3mp3st
|
||||
npm run server # listens on :3333
|
||||
|
||||
# 3. Drop your LLM key (one-time, persists in macOS Keychain)
|
||||
npm run keys set OPENROUTER_API_KEY # or ANTHROPIC_API_KEY / OPENAI_API_KEY
|
||||
```
|
||||
|
||||
## Mode A — t3mp3st attacks OBSIDIVM
|
||||
|
||||
```bash
|
||||
# Full suite (14 targets), stub hunter (no LLM, sanity-check pipeline):
|
||||
npm run obsidivm:bench:stub
|
||||
|
||||
# Single target, live LLM hunter:
|
||||
npm run obsidivm:bench:live -- --target dvwa
|
||||
|
||||
# Drive through t3mp3st's General planner / Pliny endpoints:
|
||||
npm run obsidivm:bench:t3mp3st -- --target juice --report /tmp/juice.json
|
||||
```
|
||||
|
||||
Each run prints a per-target table with letter grade (`A` through `F`) and
|
||||
suite aggregate, and posts a run record to OBSIDIVM's ledger
|
||||
(`~/.obsidium/runs/`) with all per-target sessions attached.
|
||||
|
||||
### Stub-mode calibration
|
||||
|
||||
```
|
||||
[B ] dvwa 12/23 weighted=71.07% [C+] juice 6/12 65.57%
|
||||
[B ] webgoat 7/11 weighted=73.85% [B ] shepherd 4/7 74.29%
|
||||
[B+] wordpress 6/9 weighted=81.08% [C+] hackazon 5/10 66.67%
|
||||
[C ] dvga 3/6 weighted=58.82% [C ] vampi 3/6 58.82%
|
||||
[C+] wrongsec 3/6 weighted=66.67% [B ] bwapp 6/9 79.25%
|
||||
[C+] bodgeit 3/6 weighted=64.29% [C+] pygoat 4/7 66.67%
|
||||
[B ] log4shell 3/5 weighted=70.59% [C+] mutillidae 4/8 63.41%
|
||||
|
||||
SUITE: 69/125 expected findings, weighted avg 68.65%, grade C+
|
||||
```
|
||||
|
||||
The stub hunter intentionally exercises only the first half of each target's
|
||||
expected-findings list — confirming the scoring + ledger path works.
|
||||
|
||||
## Mode B — OBSIDIVM drives t3mp3st (warroom + evolve.py)
|
||||
|
||||
```bash
|
||||
# In one terminal:
|
||||
npm run server # t3mp3st :3333
|
||||
|
||||
# In another:
|
||||
npm run obsidivm:agent # :8889 PLINY CODE shim
|
||||
|
||||
# Then open OBSIDIVM warroom — the agent dot turns green and clicking
|
||||
# "quick attack" on any target POSTs to :8889/api/launch, which routes
|
||||
# to t3mp3st's /api/general/auto with auto-approval.
|
||||
```
|
||||
|
||||
### Protocol the shim implements
|
||||
|
||||
| Endpoint | Behavior |
|
||||
|---|---|
|
||||
| `POST /api/launch {prompt}` | Extracts target URL + command from prompt. POSTs t3mp3st `/api/general/auto`. Auto-clears approval gates. Returns `{id, session_id, status: "launching"}`. |
|
||||
| `GET /api/sessions` | In-memory session table augmented with status from t3mp3st ledger. |
|
||||
| `GET /api/session/log?id=<sid>` | Queries t3mp3st findings/evidence/sitreps for the mapped mission. Returns transcript ready for `/api/score/text`. |
|
||||
| `GET /api/events` | SSE bridge. Emits `session_started`, `session_dispatched`, `session_failed`, `stopped_all`, heartbeats. |
|
||||
| `POST /api/stop-all` | Marks all sessions stopped (t3mp3st mission cancellation is local-state only — t3mp3st does not expose a kill endpoint). |
|
||||
| `GET /health` | Shim health + connected provider. |
|
||||
|
||||
### evolve.py integration
|
||||
|
||||
OBSIDIVM's `evolve.py` polls `:8889/api/sessions` and re-launches via
|
||||
`/api/launch`. As long as the shim is running, evolution loops re-fire
|
||||
t3mp3st missions automatically — no `evolve.py` modification needed.
|
||||
|
||||
## Programmatic use
|
||||
|
||||
```javascript
|
||||
import { obsidivm } from './scripts/obsidivm-bridge.mjs';
|
||||
|
||||
const o = obsidivm();
|
||||
const spec = await o.getSpec(); // 14 targets, 125 expected findings
|
||||
const verdict = await o.scoreText('dvwa', myAgentTranscript);
|
||||
// → { found, total, weighted_percent, grade, results: [...] }
|
||||
const run = await o.createRun({ agent: 'my-hunter' });
|
||||
await o.attachSession(run.run_id, {
|
||||
target_id: 'dvwa',
|
||||
score: verdict,
|
||||
transcript: myAgentTranscript,
|
||||
});
|
||||
```
|
||||
|
||||
## Roadmap
|
||||
|
||||
- [ ] **Live exploit replay** — actually execute PoCs against deployed Docker
|
||||
targets (current bench uses agent transcripts; behavioral validation
|
||||
would close the loop)
|
||||
- [ ] **t3mp3st `code_audit` operator** — give the General real source-audit
|
||||
tools so orchestrated mode produces structured findings instead of
|
||||
agent prose
|
||||
- [ ] **Cross-bench fusion** — combine CVE-Replay bench (source) + OBSIDIVM
|
||||
bench (live targets) into a single fitness function for evolution
|
||||
- [ ] **Per-CWE blind-spot heat-map** — generated from OBSIDIVM run-ledger
|
||||
history; drives corpus/operator/prompt mutation
|
||||
- [ ] **CloudGoat tier** — wire `obsidivm-bench.mjs` to also drive the 16
|
||||
AWS scenarios (requires AWS creds; currently web-tier only)
|
||||
- [ ] **Self-improvement loop** — after each run, run t3mp3st's
|
||||
`/api/learning/run-review`, accept high-confidence proposals, write
|
||||
them as resource-pack updates → next bench iteration uses tighter prompts
|
||||
Reference in new issue
Block a user