Joseph Magly
0c35a833b9
Add user and developer documentation plus Pagenary docsite publishing support. Includes Getting Started, Developer Guide, API and MCP references, Pagenary tenant metadata, npm scripts for sync/build/check/serve, and validation coverage for the generated docsite. Closes #102
181 lines
8.4 KiB
Markdown
181 lines
8.4 KiB
Markdown
---
|
|
title: "Obsidivm"
|
|
summary: "Local web range and evaluation notes."
|
|
description: "Local web range and evaluation notes."
|
|
audience: ["researcher", "operator"]
|
|
category: "benchmarks"
|
|
status: "current"
|
|
source: "T3MP3ST repository"
|
|
sourcePath: "docs/OBSIDIVM.md"
|
|
updated: "2026-07-20"
|
|
---
|
|
# T3MP3ST ⇄ OBSIDIVM
|
|
|
|
T3MP3ST is the weapon. OBSIDIVM is the range. This integration makes them
|
|
peanut butter and chocolate: t3mp3st runs ops, OBSIDIVM provides targets +
|
|
canonical scoring, and the two flow in both directions.
|
|
|
|
```
|
|
t3mp3st (:3333) OBSIDIVM (:4200)
|
|
┌─────────────────────────┐ ┌──────────────────────────┐
|
|
│ General + Operators │ ──────▶ │ /api/spec │
|
|
│ Pliny endpoints │ ──────▶ │ /api/score/text │
|
|
│ Evidence + Finding │ ──────▶ │ /api/runs │
|
|
│ ledger │ │ /api/runs/:id/sessions │
|
|
│ │ │ /api/deploy /destroy │
|
|
└──────────┬──────────────┘ └─────────────┬────────────┘
|
|
▲ │
|
|
│ ▼
|
|
┌──────────┴──────────────┐ ┌──────────────────────────┐
|
|
│ :8889 agent shim │ ◀────── │ warroom.html (UI) │
|
|
│ (t3mp3st-as-PLINY-CODE) │ ◀────── │ evolve.py (engine) │
|
|
│ /api/launch /sessions │ │ │
|
|
│ /api/session/log │ │ │
|
|
│ /api/events │ │ │
|
|
└─────────────────────────┘ └──────────────────────────┘
|
|
```
|
|
|
|
## What you get
|
|
|
|
| Capability | How |
|
|
|---|---|
|
|
| **Use t3mp3st to attack OBSIDIVM** | `obsidivm-bench.mjs` reads targets from `/api/spec`, dispatches t3mp3st missions, posts transcripts to `/api/score/text`, records run in OBSIDIVM ledger. |
|
|
| **Use OBSIDIVM warroom to drive t3mp3st** | `obsidivm-agent.mjs` exposes the `:8889` PLINY CODE protocol; OBSIDIVM warroom + `evolve.py` see t3mp3st as their agent without modification. |
|
|
| **Multi-generation evolution** | OBSIDIVM's `/api/evolve/start` re-launches sessions and analyzes weaknesses. Pointing it at the `:8889` shim turns t3mp3st into the evolved agent. |
|
|
| **Canonical scoring** | All scoring goes through `obsidium_spec.py`'s `score_text()` — same evidence rules used by warroom + tests. No drift. |
|
|
|
|
## Files added
|
|
|
|
| File | Role |
|
|
|---|---|
|
|
| `scripts/obsidivm-bridge.mjs` | Thin HTTP client around OBSIDIVM's API. CLI + ES module. |
|
|
| `scripts/obsidivm-bench.mjs` | End-to-end driver: dispatch hunter → score → record run. |
|
|
| `scripts/obsidivm-agent.mjs` | `:8889` PLINY CODE protocol shim; t3mp3st-as-agent. |
|
|
|
|
## npm scripts
|
|
|
|
```bash
|
|
npm run obsidivm # bridge CLI help
|
|
npm run obsidivm:spec # dump OBSIDIVM target/expected list
|
|
npm run obsidivm:bench # full-suite bench (defaults to stub hunter)
|
|
npm run obsidivm:bench:stub # explicit stub mode (no LLM)
|
|
npm run obsidivm:bench:live # direct LLM hunter
|
|
npm run obsidivm:bench:t3mp3st # drive through t3mp3st platform
|
|
npm run obsidivm:agent # serve :8889 for warroom + evolve.py
|
|
```
|
|
|
|
## Setup once
|
|
|
|
```bash
|
|
# 1. Start OBSIDIVM
|
|
cd ~/Desktop/workspace/OBSIDIVM
|
|
python3 range.py # listens on :4200
|
|
|
|
# 2. (Optional) Start t3mp3st server if you want the :8889 shim
|
|
cd ~/Desktop/t3mp3st
|
|
npm run server # listens on :3333
|
|
|
|
# 3. Drop your LLM key (one-time, persists in macOS Keychain)
|
|
npm run keys set OPENROUTER_API_KEY # or ANTHROPIC_API_KEY / OPENAI_API_KEY
|
|
```
|
|
|
|
## Mode A — t3mp3st attacks OBSIDIVM
|
|
|
|
```bash
|
|
# Full suite (14 targets), stub hunter (no LLM, sanity-check pipeline):
|
|
npm run obsidivm:bench:stub
|
|
|
|
# Single target, live LLM hunter:
|
|
npm run obsidivm:bench:live -- --target dvwa
|
|
|
|
# Drive through t3mp3st's General planner / Pliny endpoints:
|
|
npm run obsidivm:bench:t3mp3st -- --target juice --report /tmp/juice.json
|
|
```
|
|
|
|
Each run prints a per-target table with letter grade (`A` through `F`) and
|
|
suite aggregate, and posts a run record to OBSIDIVM's ledger
|
|
(`~/.obsidium/runs/`) with all per-target sessions attached.
|
|
|
|
### Stub-mode calibration
|
|
|
|
```
|
|
[B ] dvwa 12/23 weighted=71.07% [C+] juice 6/12 65.57%
|
|
[B ] webgoat 7/11 weighted=73.85% [B ] shepherd 4/7 74.29%
|
|
[B+] wordpress 6/9 weighted=81.08% [C+] hackazon 5/10 66.67%
|
|
[C ] dvga 3/6 weighted=58.82% [C ] vampi 3/6 58.82%
|
|
[C+] wrongsec 3/6 weighted=66.67% [B ] bwapp 6/9 79.25%
|
|
[C+] bodgeit 3/6 weighted=64.29% [C+] pygoat 4/7 66.67%
|
|
[B ] log4shell 3/5 weighted=70.59% [C+] mutillidae 4/8 63.41%
|
|
|
|
SUITE: 69/125 expected findings, weighted avg 68.65%, grade C+
|
|
```
|
|
|
|
The stub hunter intentionally exercises only the first half of each target's
|
|
expected-findings list — confirming the scoring + ledger path works.
|
|
|
|
## Mode B — OBSIDIVM drives t3mp3st (warroom + evolve.py)
|
|
|
|
```bash
|
|
# In one terminal:
|
|
npm run server # t3mp3st :3333
|
|
|
|
# In another:
|
|
npm run obsidivm:agent # :8889 PLINY CODE shim
|
|
|
|
# Then open OBSIDIVM warroom — the agent dot turns green and clicking
|
|
# "quick attack" on any target POSTs to :8889/api/launch, which routes
|
|
# to t3mp3st's /api/general/auto with auto-approval.
|
|
```
|
|
|
|
### Protocol the shim implements
|
|
|
|
| Endpoint | Behavior |
|
|
|---|---|
|
|
| `POST /api/launch {prompt}` | Extracts target URL + command from prompt. POSTs t3mp3st `/api/general/auto`. Auto-clears approval gates. Returns `{id, session_id, status: "launching"}`. |
|
|
| `GET /api/sessions` | In-memory session table augmented with status from t3mp3st ledger. |
|
|
| `GET /api/session/log?id=<sid>` | Queries t3mp3st findings/evidence/sitreps for the mapped mission. Returns transcript ready for `/api/score/text`. |
|
|
| `GET /api/events` | SSE bridge. Emits `session_started`, `session_dispatched`, `session_failed`, `stopped_all`, heartbeats. |
|
|
| `POST /api/stop-all` | Marks all sessions stopped (t3mp3st mission cancellation is local-state only — t3mp3st does not expose a kill endpoint). |
|
|
| `GET /health` | Shim health + connected provider. |
|
|
|
|
### evolve.py integration
|
|
|
|
OBSIDIVM's `evolve.py` polls `:8889/api/sessions` and re-launches via
|
|
`/api/launch`. As long as the shim is running, evolution loops re-fire
|
|
t3mp3st missions automatically — no `evolve.py` modification needed.
|
|
|
|
## Programmatic use
|
|
|
|
```javascript
|
|
import { obsidivm } from './scripts/obsidivm-bridge.mjs';
|
|
|
|
const o = obsidivm();
|
|
const spec = await o.getSpec(); // 14 targets, 125 expected findings
|
|
const verdict = await o.scoreText('dvwa', myAgentTranscript);
|
|
// → { found, total, weighted_percent, grade, results: [...] }
|
|
const run = await o.createRun({ agent: 'my-hunter' });
|
|
await o.attachSession(run.run_id, {
|
|
target_id: 'dvwa',
|
|
score: verdict,
|
|
transcript: myAgentTranscript,
|
|
});
|
|
```
|
|
|
|
## Roadmap
|
|
|
|
- [ ] **Live exploit replay** — actually execute PoCs against deployed Docker
|
|
targets (current bench uses agent transcripts; behavioral validation
|
|
would close the loop)
|
|
- [ ] **t3mp3st `code_audit` operator** — give the General real source-audit
|
|
tools so orchestrated mode produces structured findings instead of
|
|
agent prose
|
|
- [ ] **Cross-bench fusion** — combine CVE-Replay bench (source) + OBSIDIVM
|
|
bench (live targets) into a single fitness function for evolution
|
|
- [ ] **Per-CWE blind-spot heat-map** — generated from OBSIDIVM run-ledger
|
|
history; drives corpus/operator/prompt mutation
|
|
- [ ] **CloudGoat tier** — wire `obsidivm-bench.mjs` to also drive the 16
|
|
AWS scenarios (requires AWS creds; currently web-tier only)
|
|
- [ ] **Self-improvement loop** — after each run, run t3mp3st's
|
|
`/api/learning/run-review`, accept high-confidence proposals, write
|
|
them as resource-pack updates → next bench iteration uses tighter prompts
|