* test(obsidivm): RED — bridge should give an actionable error when the service is down The obsidivm:* benchmarks fetch from the OBSIDIVM range service; when it is not running the bridge throws a bare 'fetch failed' (surfaced as 'FATAL: fetch failed'), naming neither the service, the URL, nor how to start it. Pins that a connection failure becomes an actionable, self-service error. Fails on main: the message is 'fetch failed'. Refs #156 * fix(obsidivm): actionable error when the range service is unreachable The obsidivm:* benchmarks fetch from the OBSIDIVM range service; when it is not running the bridge's shared call() let the raw 'fetch failed' propagate, which surfaced as an opaque 'FATAL: fetch failed' (#156) naming neither the service, the URL, nor how to start it — even in --hunter stub, whose grading step still needs it. Wrap the fetch in call() (the single choke point for every OBSIDIVM request) so a connection failure becomes: 'OBSIDIVM service unreachable at <url> ... Start it per docs/OBSIDIVM.md (python3 range.py, listens on :4200), or set OBSIDIVM_URL', flagged err.unreachable to distinguish it from an HTTP-status error; a timeout is reported distinctly. Clarify in the bench --help/usage that every hunter mode (stub included) needs the OBSIDIVM service, live/t3mp3st also a key. Adds scripts/obsidivm-bridge.d.mts so the new test types cleanly. Closes #156 * fix(obsidivm): distinguish invalid-URL from unreachable; test timeout path Adversarial review of the diff found two gaps: - A malformed OBSIDIVM_URL (ERR_INVALID_URL) was folded into the 'service unreachable — is it running?' path, misdirecting the user to start a service when the real fix is the URL. Now branched to a distinct 'Invalid OBSIDIVM base URL … set OBSIDIVM_URL' error, no err.unreachable flag. - The timeout (AbortError) branch had no coverage. Added a test using a never-responding TCP server + short timeoutMs asserting 'timed out after Nms' and NOT unreachable; and a test for the invalid-URL branch. Also completed the .d.mts stub with the cloudgoat* methods it was missing. Refs #156
This commit is contained in:
1 parent
c352f34843
commit
41f0c14039
4 files changed
+163
-8
No files matched your search
@@ -7,7 +7,9 @@
|
||||
* 2. Dispatch a t3mp3st hunter against the target's URL.
|
||||
* - --hunter=live : direct LLM (OpenRouter / Anthropic / OpenAI auto-detect)
|
||||
* - --hunter=t3mp3st: drive t3mp3st's /api/general/auto (full platform)
|
||||
* - --hunter=stub : synthesized transcript (no LLM, for plumbing tests)
|
||||
* - --hunter=stub : synthesized transcript (no LLM, for plumbing tests).
|
||||
* NB: only the HUNTER is offline — the OBSIDIVM service
|
||||
* is still required for the spec (step 1) and scoring (3).
|
||||
* 3. Submit the agent transcript to OBSIDIVM's /api/score/text.
|
||||
* 4. Persist the run via OBSIDIVM's /api/runs ledger + per-target sessions.
|
||||
* 5. Print a per-target table + suite aggregate (weighted % + grade).
|
||||
@@ -105,9 +107,12 @@ function help() {
|
||||
Options:
|
||||
--target <id> Target id (repeatable). Default: all 14 OBSIDIVM targets.
|
||||
--hunter <stub|live|t3mp3st>
|
||||
stub: synthesized transcript (no LLM)
|
||||
stub: synthesized transcript (no LLM; the OBSIDIVM
|
||||
service is still required for spec + scoring)
|
||||
live: direct LLM call
|
||||
t3mp3st: drive t3mp3st's /api/general/auto
|
||||
All modes need the OBSIDIVM service (see --obsidivm / docs/OBSIDIVM.md); live
|
||||
and t3mp3st additionally need an LLM key.
|
||||
--obsidivm <url> OBSIDIVM base URL (default http://127.0.0.1:4200)
|
||||
--t3mp3st <url> t3mp3st base URL (default http://127.0.0.1:3333)
|
||||
--model <name> LLM model id (default claude-opus-4-7)
|
||||
|
||||
Reference in new issue
Block a user