Lyubomir Bozhinov
6d4e01756a
* test(binary): RED — no rule for memcpy/memmove with a non-constant length Adds two corpus fixtures + direct assertions for a memcpy/memmove sink: - memcpy-wire-length (bad): a wire-decoded length into a fixed buffer, expect B-MEMCPY. Fails on main — the ruleset has no memcpy rule. - memcpy-bounded-control (clean): two safe memcpys (sizeof-bounded + a constant length). This forces a DISCRIMINATING rule: a naive /memcpy/ match false-positives here and fails the zero-false-positive gate. Refs #150 * feat(binary): detect memcpy/memmove with a non-constant length Adds B-MEMCPY to the decompiled-vuln ruleset — the classic unbounded-copy overflow. The framework's own reasoning layer treats this as its textbook target (src/orchestration/prompts.ts:60,80 — 'a wire-controlled length flowing into an unchecked memcpy'; orchestration/index.ts:11), but the static pre-filter had no rule for it: strcpy was covered, its memcpy(dst, src, len) sibling was not. The rule inspects the length (3rd) argument and fires only when it is non-constant — a variable, struct field, deref, or computed/decoded value — staying silent on a sizeof(...) or numeric-literal length. Validated against a 22-case dangerous/safe matrix (perfect discrimination). Corpus 6+2 -> 7+3; the new sizeof/constant-bounded control keeps the rule honest — a bare /memcpy/ match would false-positive on it. Closes #150 * fix(binary): stop B-MEMCPY false-positive on nested-call args; disclose limits Adversarial review found a real false positive: a comma inside a nested call in arg 1/2 (memcpy(dst, get_src(a,b), sizeof(dst))) mis-split the naive [^,]+,[^,]+, arg list, so a sizeof-BOUNDED call fired. Restricting args 1-2 to [^,()]+ (paren-free) removes it — verified the bounded nested-call form is now silent while every common non-constant length still fires. The trade is a memcpy whose 1st/2nd arg is itself a call is not matched (an acceptable miss for a directional pre-filter, now disclosed). Rule comment rewritten to honestly enumerate the directional misses (leading- digit computed length, sizeof-plus-term, nested-call args, inline comments, __memcpy_chk/memcpy_s/wmemcpy variants). Test expanded into an in-tree discrimination matrix (fires / silent / pinned disclosed-limit cases) so the scope boundary is auditable and any future tightening surfaces deliberately. Refs #150 * fix(binary): match pointer casts in B-MEMCPY args; correct disclosed limits A second adversarial pass on the shipped rule caught that the previous paren-free tightening ([^,()]+) over-corrected: it also stopped matching pointer CASTS on args 1-2 — memcpy((void *)dst, (void *)src, len) — which are the most common memcpy shape in real decompiled output. The rule was silently missing the main case. Replace the arg matcher with one level of BALANCED parens, (?:[^,()]|\([^()]*\))+, so a cast (void *)dst and a comma-bearing call get(a,b) are each consumed as ONE arg. This fires on casts again, still fires on a call-bearing arg with a non-constant length (now a true positive, previously a disclosed miss), and keeps the original nested-call-with-bounded-length false positive silent. Verified no catastrophic backtracking on pathological input. Also corrects the disclosed-limits comment: the previous 'arg is itself a call' miss is now only DOUBLY-nested parens, and the inline-comment note was backwards (a comment obscuring a sizeof/literal false-fires, it does not cause a miss) — restated accurately, with decompiled output not emitting inline comments noted. Tests pin the cast fires and the variant exclusions. Refs #150
Benchmark artifacts
Every headline number in the top-level README re-derives from the committed result
JSONs in this directory. Run npm run verify-claims from the repo root — it recomputes
each claim from these files under a provenance-strict scorer and fails if any number
drifts from its artifact.
Each result file records the structured verdict for a challenge run: the score / detection, canary status, the target id, and (for CVE hunts) the finding metadata. Those are the fields the scorer reads.
A note on transcripts
The raw per-run agent transcripts are intentionally omitted from this public release.
During a run, an operator's recon commands (find, ls, env) get captured verbatim
into the transcript, which means they can photograph the author's local machine. Rather
than hand-scrub an unbounded capture surface, the release ships the structured verdicts
only. This is lossless for reproducibility: verify-claims never reads the transcript
field — it re-derives everything from the verdicts — so every number here remains
independently checkable. To regenerate full transcripts, re-run the harness against the
same challenge suites from scratch.