From 151d3555c678654421ed61a67da19f9aa4714ed9 Mon Sep 17 00:00:00 2001 From: Lyubomir Bozhinov <12250247+lyubomir-bozhinov@users.noreply.github.com> Date: Mon, 20 Jul 2026 18:41:21 +0300 Subject: [PATCH] feat(ingest): multi-language white-box ingest (web-tree-sitter) (#75) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(ingest): pin web-tree-sitter@0.25.10 + tree-sitter-wasms@0.1.13; add multilang-ingest design docs * feat(ingest): multi-language grammar registry (web-tree-sitter, 8 langs, fail-open); 100% cov * feat(ingest): language-aware param-name splitter (py/go/ts/java/c); 100% cov * feat(ingest): tree-sitter CodeBlock extractor with wall-clock parse budget + fail-open * feat(ingest): cross-lang sinks + multilang config + wire tree-sitter into ingestRepository (sync) * feat(ingest): repoint white-box production callers to multilang config + prod regression test * server-init-wiring: load multi-language grammars once at bootstrap initGrammars() runs after LLM init, before app.listen. Internally fail-open (never rejects): on failure the grammar registry stays empty and ingest falls back to the Python regex parser. * adversarial-tests: hostile-input hardening for untrusted target source Malformed per-language source, wrong-language content, binary/non-UTF8, empty, unicode identifiers, 20k-func large file (wall-clock budget), deeply nested input, and a symlink-loop crawl — all must fail-open, never throw or spin. Drives parseFileMultiLang + ingestRepository/crawl. * readme-status + docs: flip white-box status to multi-language ingest README source-code rows now state web-tree-sitter multi-language ingest (fail-open to Python regex). whitebox.ts scope caveat + no-source error message updated off the stale Python-only wording. * coverage-setup: enforce 100% coverage gate on the three new ingest files vitest.config.ts adds a coverage-only block (v8, perFile thresholds=100) scoped to ts-grammars.ts, param-split.ts, ts-parse.ts. Test discovery and env stay on defaults. Edited files gate on new-branch coverage via the regression suites; the server.ts initGrammars bootstrap line is the documented smoke-covered carve-out. * review: fix WASM heap leak, language-scoped fail-open, sink recall/precision Adversarial security + strict code review (two independent subagents) returned one blocking bug and several correctness findings; remediated: BLOCKING #1 — native WASM memory leak: web-tree-sitter Tree/Parser objects were never .delete()'d. Both reviewers reproduced a process-global wedge: after ~7 large parses the shared WASM heap exhausts, every subsequent parse aborts (caught by fail-open), and non-Python ingest silently returns [] forever. Fix: free the tree in a finally; delete the parser in resetParser. Regression test parses ~8MB through the real shared parser 12x. regex, which mis-parses brace/class syntax into corrupt blocks (wrong line spans, lost methods). Now non-.py fail-open returns [] (honest absence); the Python regex only ever runs on .py. Isolated ts-parse-fallback test. http(s).request, axios(config), C exec-family); negative lookbehind stops bare system(/popen(/fetch( from matching Python receivers (os.system etc. still matched, corpus rankings unchanged). Recall+precision tests added. operator sees non-Python degradation. (one-line call-graph edges, no decorators, C++ member methods). * review round 2: fix sink-regex regressions, real timeout test, deterministic leak guard Third review round (correctness + security re-attack + test-quality) against the round-1 fixes surfaced issues in the fixes themselves; remediated: - Sink-regex regressions I introduced: (1) my fetch lookbehind dropped window.fetch/this.fetch (top JS SSRF idiom) — reverted to \bfetch\(; (2) bare .Get/.Post/.Do flagged db.Get/cache.Get/queue.Do as SSRF — receiver-qualified to [Cc]lient. Recall+precision tests cover both; regex is ReDoS-free (timed). - Placebo leak test: the 12x function-dense loop did NOT wedge the parser on machines with heap headroom (proven: passed with tree.delete removed). Replaced with a deterministic vi.spyOn(Tree.prototype,'delete') assertion, verified RED without the fix. - Real (non-mocked) parse-timeout test: parseFileMultiLang gains an injectable budgetMs; a 0ms budget drives the real progressCallback cancellation to []. - runWhiteboxAnalysis swap now has coverage (spy on orchestrator.run) so the 2nd production caller can't silently revert to the Python-only config. - Un-vacuoused weak tests: unicode now asserts the extracted block; nested/ wrong-language wrapped in invariants() + timing bounds. - CI now runs npm run test:coverage (the 100% gate was defined but unenforced). - Security: documented maxFileBytes as the control bounding a single parse's peak WASM memory below the 2GB heap-abort ceiling. - Stale 'fail-open to Python regex' docs fixed (ts-parse, whitebox, server, adversarial); circular-import note added; temp-dir cleanup; honest coverage wording in spec + config. * docs(ingest): align fallback-contract text with the shipped language-scoped fail-open Review feedback (#75, @jmagly): the design docs + one ts-grammars comment still described the pre-implementation 'fall back to the Python regex' model for non-Python files. The shipped contract is language-scoped: .py uses parseFile; any non-.py file with a missing/unsupported/failed grammar or a parse timeout returns [] (never the Python regex, which would mis-parse brace/class syntax into corrupt blocks). Updated every stale reference in multilang-ingest-spec.md, multilang-ingest-plan.md, and ts-grammars.ts so the docs match the security boundary. Docs/comment only — no code change. * fix(test): make the ReDoS wall-clock guard robust under coverage instrumentation Review feedback (#75, @jmagly): the CI step this PR adds (npm run test:coverage = vitest run --coverage) runs the full suite under V8 coverage, and that overhead pushes the pathological-brace DoS timing guard in local-agent-tool-calling.test.ts:69 over its <300ms bound (320 < 300 fails on CI). Raise the bound to 1000ms — still 2x below the ~2100ms quadratic-ReDoS regression it guards against, so it fails hard if the regression returns, but no longer false-fails under coverage/slow CI. Chose the reviewer's option (a) (robust bound) over scoping test:coverage, so the gate stays a simple full-suite run and no test-file enumeration can rot. Verified: npm run test:coverage is green and the 3 new files stay at 100%. * fix(ingest): cover pointer-returning C functions --------- Co-authored-by: Joseph Magly <1159087+jmagly@users.noreply.github.com> --- .github/workflows/ci.yml | 3 + .gitignore | 3 + README.md | 4 +- docs/design/multilang-ingest-plan.md | 243 ++++++++++++++++++ docs/design/multilang-ingest-spec.md | 124 +++++++++ package-lock.json | 233 ++++++++++++++++- package.json | 6 +- .../local-agent-tool-calling.test.ts | 5 +- src/__tests__/multilang-config.test.ts | 24 ++ src/__tests__/param-split.test.ts | 49 ++++ src/__tests__/sink-patterns.test.ts | 69 +++++ src/__tests__/ts-grammars.test.ts | 69 +++++ src/__tests__/ts-parse-adversarial.test.ts | 136 ++++++++++ src/__tests__/ts-parse-fallback.test.ts | 31 +++ src/__tests__/ts-parse-multilang-rank.test.ts | 58 +++++ src/__tests__/ts-parse.test.ts | 95 +++++++ src/__tests__/whitebox-multilang.test.ts | 59 +++++ src/recon/code-ingest.ts | 53 +++- src/recon/param-split.ts | 77 ++++++ src/recon/ts-grammars.ts | 149 +++++++++++ src/recon/ts-parse.ts | 133 ++++++++++ src/recon/whitebox.ts | 44 +++- src/server.ts | 7 + vitest.config.ts | 32 +++ 24 files changed, 1681 insertions(+), 25 deletions(-) create mode 100644 docs/design/multilang-ingest-plan.md create mode 100644 docs/design/multilang-ingest-spec.md create mode 100644 src/__tests__/multilang-config.test.ts create mode 100644 src/__tests__/param-split.test.ts create mode 100644 src/__tests__/sink-patterns.test.ts create mode 100644 src/__tests__/ts-grammars.test.ts create mode 100644 src/__tests__/ts-parse-adversarial.test.ts create mode 100644 src/__tests__/ts-parse-fallback.test.ts create mode 100644 src/__tests__/ts-parse-multilang-rank.test.ts create mode 100644 src/__tests__/ts-parse.test.ts create mode 100644 src/__tests__/whitebox-multilang.test.ts create mode 100644 src/recon/param-split.ts create mode 100644 src/recon/ts-grammars.ts create mode 100644 src/recon/ts-parse.ts create mode 100644 vitest.config.ts diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 666ee56..27ec232 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -19,6 +19,9 @@ jobs: - run: npm run lint - run: npm run typecheck - run: npm test + # Enforce the 100% coverage gate on the new multi-language ingest files + # (vitest exits non-zero if a per-file threshold is missed). + - run: npm run test:coverage - run: npm run doctor # Honesty gates: prove the headline numbers re-derive and that no # challenge-fitting has crept in. Public contributors / CI can verify. diff --git a/.gitignore b/.gitignore index 674a84c..c20c52d 100755 --- a/.gitignore +++ b/.gitignore @@ -65,3 +65,6 @@ scripts/.export-denylist.local !.aiwg/addons/** server.pid + +# vitest v8 coverage output +coverage/ diff --git a/README.md b/README.md index b9e1f54..8125b10 100755 --- a/README.md +++ b/README.md @@ -53,7 +53,7 @@ Offensive security sits behind years of practice and expensive tooling. The bet | 🕸️ **Web apps** | Black-box, external-attacker recon → exploit (XBEN suite) | ✅ Stable | | 🚩 **CTF** | Hint-free, sandbox-jailed solves (Cybench) | ✅ Stable | | 🤖 **Robotics / OT / embedded** | Coordinated-disclosure pipeline for OSS vuln hunting (OSV + live-PoC + refuter) | ✅ Pipeline stable | -| 📂 **Source code** | White-box repo analysis with blind master-builder decomposition | ⚠️ Python-only ingest | +| 📂 **Source code** | White-box repo analysis with blind master-builder decomposition | ✅ Multi-language ingest (web-tree-sitter) | | 💰 **Smart contracts** | Damn Vulnerable DeFi | ⚠️ reproduction, not novel discovery | | ☁️ **Cloud (IaC)** | Misconfig-detection benchmark (`cloud:bench`) + opt-in cloud arsenal (aws/az/gcloud + scoutsuite/cloudfox/pmapper; pacu gated) | 🚧 IaC-misconfig scaffolding — live-cloud exploitation not yet benchmarked | | 📱 **Mobile** | Built-in static analyzer (manifest misconfig + secret/cleartext detection, `mobile:bench`) + opt-in arsenal (mobsfscan/objection/drozer; frida gated) | 🚧 static-detection scaffolding — dynamic exploitation not benchmarked | @@ -194,7 +194,7 @@ The framework is an 8-operator kill chain, and this table won't blow smoke about | Arsenal, MCP server, HTTP API | ✅ Stable | 35 built-in tools by default; 108 with the opt-in `T3MP3ST_FULL_ARSENAL` (+73 adapters, with dangerous/catalog-only drivers — metasploit, hydra, pacu, frida — behind narrow approved paths rather than generic execution) — both counts re-derive via `verify-claims`. `security_recon` over MCP | | Egress-scope containment | ✅ Stable (on by default) | once a mission target is set, built-in networked tools refuse off-scope public hosts — not the target/subdomains, not loopback/private (`SCOPE DENIED`) — a tightened default, not a bare tool runner | | Coordinated-disclosure pipeline | ✅ Stable | OSV novelty + live PoC + refuter panel + CVSS; drafts only, a human sends | -| White-box source analysis | ⚠️ Experimental | Python-only regex ingest; multi-model decomposition costs more tokens, not fewer | +| White-box source analysis | ⚠️ Experimental | Multi-language ingest via web-tree-sitter (Python/JS/TS/Go/Java/C/C++); Python retains its regex parser, while other languages fail open to no extracted blocks; multi-model decomposition costs more tokens, not fewer | | DeFi (Damn Vulnerable DeFi) | ⚠️ Experimental | reproduces known exploit classes; not novel discovery | | Exploiter / Infiltrator / Exfiltrator / Ghost | ⚠️ Experimental | run the real tool-backed ReAct loop (same engine as recon); unproven as a coordinated swarm — single-agent is the benchmarked path, live swarm exploitation still unreliable | | Advanced modules (cloud, persistence, swarm, cognition) | 🚧 Planned | interface-only in `src/stubs/` | diff --git a/docs/design/multilang-ingest-plan.md b/docs/design/multilang-ingest-plan.md new file mode 100644 index 0000000..c92ccab --- /dev/null +++ b/docs/design/multilang-ingest-plan.md @@ -0,0 +1,243 @@ +# Plan: T3MP3ST-MLINGEST — Multi-language white-box ingest (web-tree-sitter) + +## Executive Summary +- **Goal:** Replace the Python-only `parseFile` stage with a portable `web-tree-sitter` extractor emitting the existing `CodeBlock` shape; reuse the security-ranking pipeline verbatim; language-scoped fail-open (`.py` keeps the Python regex; other languages return `[]` on a grammar miss / parse error — never the Python regex). +- **Complexity:** Medium-low (no async ripple — see Arch). **Risk:** Low-Medium. **Diff:** 3 new source files (`param-split`, `ts-grammars`, `ts-parse`) + edits to 3 (`code-ingest`, `whitebox`, `server`) + coverage config + test suites. +- **Spec:** `2026-07-10-multilang-ingest-design.md`. **Reviews folded:** architect-review (async→sync pivot + 5 defects), test-automator (RED discrimination + granularity). + +## 🚨 Critical Implementation Standards +- Fail-open mandatory (language-scoped): `.py` → `parseFile`; a non-`.py` grammar init/parse failure → `[]` (never the Python regex). Ingest never crashes a mission. +- Security ranking reused, not rewritten (`classify`/`prioritize`/`reachability`/`context-pack` untouched). +- vitest green, `tsc --noEmit` clean, `eslint` clean. No TODO/HACK/dead code; every failure path handled. +- **100% coverage of our code (hard gate).** Statements/branches/functions/lines = 100% on every NEW file (`param-split.ts`, `ts-grammars.ts`, `ts-parse.ts`) and 100% of every new branch/function added to edited files (`code-ingest.ts`, `whitebox.ts`). Sole carve-out: the **single** `await initGrammars(...)` wiring line at `server.ts` bootstrap — `initGrammars` itself is 100% unit-tested in `grammar-registry`; the wiring line is exercised by `npm run smoke` at the ship gate. (`cli.ts` is NOT wired — the CLI never ingests; ingest is reached only via `server.ts:6060/6126`.) Coverage is necessary-not-sufficient — the review gate confirms assertions are meaningful, not line-touching. +- **Adversarial phase mandatory (this parses untrusted target source).** A crafted hostile-input suite + a `security-test-engineer` subagent actively trying to break the built code; all defects fixed before review. +- **Strict subagent review is the merge gate.** `/code-review-pr-strict` methodology (parallel specialists, security-critical detection FIRST, false-positive elimination) on the full diff; every confirmed finding resolved; re-review until clean. + +## Architecture — SYNC ingest + bootstrap init (revised per architect §2) +web-tree-sitter: only `Parser.init()`/`Language.load(wasm)` are async (one-time); `parser.parse()` is **sync**. So we hoist init to bootstrap and keep the ingest path synchronous — **`ingestRepository` stays sync**, eliminating the entire async ripple through `whitebox.ts`/`server.ts`/tests. + +``` +bootstrap (server start / CLI main): await initGrammars(exts) // load all grammars once into a module singleton; try/catch → empty registry on failure (fail-open) +───────────────────────────────────── +crawl (multi-lang includeExts) + → ingestRepository (SYNC, unchanged signature) — per file: + parseFileMultiLang(path, content, ext) // SYNC + ├─ ext == .py → parseFile(...) // Python regex, unchanged + ├─ non-.py, getGrammar(ext) loaded? → parser.parse() → query → nodeToCodeBlock[] → CodeBlock[] + └─ non-.py, no grammar / not-yet-loaded / timeout / error → [] // FAIL-OPEN (never the Python regex) + → buildCallGraph → findEntryPoints → reachability → classify → prioritize (UNCHANGED) + → context-pack → orchestrator (UNCHANGED) +``` +Race: a mission firing before init resolves simply fail-opens for that call (`.py` → Python regex, other languages → `[]`) — degrade, never crash. Grammars eager-loaded at bootstrap (~few MB, one-time) — the cost of a sync hot path. + +**Killed by the sync pivot:** the async signature change on `ingestRepository`; the `ingest-limits.test.ts` await churn (D1); the unguarded-init mission crash (D5, now a bootstrap try/catch). +**NOT killed — still required (validator BLOCKER):** `whitebox.ts` must still be edited to repoint BOTH production callers (`ingestRepoToSourceContext` L125, `runWhiteboxAnalysis` L205) from `createPythonIngestConfig` → `createMultiLangIngestConfig`. This config-swap is independent of sync/async; without it the multilang extractor only ever sees `.py` in production and the PR is a no-op. Node `whitebox-wiring` below. + +## Current State (grounded in /tmp/T3MP3ST) +- `CodeBlock = {id, path, name, kind, lineStart, lineEnd, params, decorators, body}` (`id=\`${path}::${name}@${lineStart}\``; `decorators=[]` for non-Python). +- `parseFile` (code-ingest.ts:409, stays as the `.py` path only); `parseParams` (decl L328, Python-only). +- `ingestRepository` (L749, **stays sync**); sink regexes L169–201; `crawl` L238. +- **Two** production `ingestRepository` callers, both in `whitebox.ts`: `ingestRepoToSourceContext` (L125) and `runWhiteboxAnalysis` (L205) — both pass `createPythonIngestConfig`. (`server.ts:6060` calls `ingestRepoToSourceContext`, not `ingestRepository` directly.) Both must be repointed — see `whitebox-wiring`. + +## Node ownership split (execution-offload datapoint) +- **Owned (local:false — specialized / cross-cutting / verification):** `deps-assets`, `coverage-setup`, `grammar-registry` (+`initGrammars`), `server-init-wiring`, `ts-parse`, `ingest-wiring`, `whitebox-wiring`, `adversarial-tests`. (8) +- **Offloaded (local:true — mechanical single-region, committed RED test):** `param-split`, `sink-patterns`, `multilang-config`, `readme-status`. (4) +- 12 manifest nodes; the Adversarial-exploration and Strict-review **phases** below are subagent-driven gates, not manifest nodes. +- **Retained-`local:false`, deliberate:** `nodeToCodeBlock` (pure map inside `ts-parse`) is offload-able in principle, but a discriminating RED test for it needs a real tree-sitter Node (no deterministic offline parse without the loaded grammar), so its natural RED test *is* the `ts-parse` integration test — extracting it would create a node whose test can't be cleanly pre-committed. Conscious retention per validator finding 5. + +**Execution prerequisite:** every `local:true` node's `accept` runs a pre-committed RED test file that **no node creates** — the orchestrator commits all RED baselines to the repo *before* running the manifest (Promise-Theory: the local model never authors its own test). Running the manifest cold without this step makes those accepts fail with "no test files found." + +## RED baselines (committed before impl; local model sees only the test) + +**`param-split`** — `src/__tests__/param-split.test.ts` (＋discriminating same-raw case per tester) +```ts +import { describe, it, expect } from 'vitest'; +import { splitParamList } from '../recon/param-split.js'; +describe('splitParamList', () => { + it('python: drops self, strips annotations/defaults/*args', () => + expect(splitParamList('self, url: str, timeout=5, *args', 'py')).toEqual(['url','timeout'])); + it('go name-first', () => expect(splitParamList('ctx context.Context, id string', 'go')).toEqual(['ctx','id'])); + it('java type-first', () => expect(splitParamList('String url, int max', 'java')).toEqual(['url','max'])); + it('ts name: type', () => expect(splitParamList('url: string, opts?: Opts', 'ts')).toEqual(['url','opts'])); + it('empty', () => expect(splitParamList('', 'ts')).toEqual([])); + it('SAME raw, different lang → forces real dispatch', () => { // ← tester fix + expect(splitParamList('a b', 'go')).toEqual(['a']); // name-first + expect(splitParamList('a b', 'java')).toEqual(['b']); // type-first + }); +}); +``` + +**`sink-patterns`** — `src/__tests__/sink-patterns.test.ts` (scoped to 2 contiguous regexes; ＋false-positive guards) +```ts +import { describe, it, expect } from 'vitest'; +import { DANGEROUS_SINK_RE, OUTBOUND_REQUEST_RE } from '../recon/code-ingest.js'; +describe('cross-language sinks', () => { + it('matches new sinks', () => { + for (const s of ['Runtime.getRuntime().exec(cmd)','exec.Command("sh")','child_process.exec(x)','system(buf)','popen(cmd)']) + expect(DANGEROUS_SINK_RE.test(s)).toBe(true); + }); + it('regression: python sinks still match', () => expect(DANGEROUS_SINK_RE.test('os.system(x)')).toBe(true)); + it('no FP on benign / System.out / filesystem ids', () => { // ← tester fix + expect(DANGEROUS_SINK_RE.test('const total = sum(a, b)')).toBe(false); + expect(DANGEROUS_SINK_RE.test('System.out.println("debug")')).toBe(false); + expect(DANGEROUS_SINK_RE.test('fileSystem.readFile(x)')).toBe(false); + }); + it('outbound covers go/js http', () => { + for (const s of ['http.Get(u)','fetch(u)','axios.get(u)']) expect(OUTBOUND_REQUEST_RE.test(s)).toBe(true); + }); +}); +``` + +**`multilang-config`** — `src/__tests__/multilang-config.test.ts` (all exts ＋negative per tester) +```ts +import { describe, it, expect } from 'vitest'; +import { mkdtempSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; import { tmpdir } from 'node:os'; +import { crawl, createMultiLangIngestConfig } from '../recon/code-ingest.js'; +describe('createMultiLangIngestConfig', () => { + const EXTS = ['py','js','ts','tsx','go','java','c','cpp']; + it('crawls every supported ext, excludes non-source', () => { + const root = mkdtempSync(join(tmpdir(), 'mling-')); + for (const e of EXTS) writeFileSync(join(root, `f.${e}`), '\n'); + writeFileSync(join(root, 'd.md'), '# ignore'); + const files = crawl(createMultiLangIngestConfig(root)); + for (const e of EXTS) expect(files.some(f => f.endsWith(`f.${e}`))).toBe(true); // no degenerate hardcode + expect(files.some(f => f.endsWith('d.md'))).toBe(false); // no wildcard/drop-filter + }); +}); +``` + +**`readme-status`** — accept grep (corrected; all three rows, "regex" word — per tester finding 2). + +## Owned-node acceptance tests (real assertions committed up front — tester finding 5) +- **`grammar-registry`** — `ts-grammars.test.ts`: registry has entries for all 8 exts with non-empty query + `@name`/`@def` captures; **`initGrammars` fail-open**: forcing a bad wasm path leaves the registry empty and `getParser` returns undefined, **no throw** (covers D5). +- **`ts-parse`** — `ts-parse.test.ts`: parse a TS and a Go function → assert `name/params/lineStart/body`; unsupported non-`.py` `.zzz` → returns `[]`, no throw; `.py` → byte-identical to `parseFile`. +- **`server-init-wiring`** — `tsc --noEmit` clean; the single bootstrap line covered by `npm run smoke` at the ship gate (`initGrammars` itself is 100% unit-tested in `grammar-registry`). +- **`ingest-wiring`** (headline regression) — `ts-parse-multilang-rank.test.ts`: temp repo `.py`+`.go`+`.ts`+`.java`, each a sink-bearing function; `ingestRepository(createMultiLangIngestConfig(root))` (sync) → non-`.py` blocks present, classified `attack_surface`, priority > a neutral block; **PLUS a cross-language reachability case** (Go/TS entry-point → reachable callee gets the reach bonus) to guard `buildCallGraph` name-matching on non-Python ids (tester finding 5). Existing `code-ingest.test.ts` unchanged (sync preserved). +- **`whitebox-wiring`** (production-path regression) — `whitebox-multilang.test.ts`: temp Go+TS repo through `ingestRepoToSourceContext(root)` → asserts non-`.py` source in the returned `sourceContext` (the real server/CLI entry, not the config directly); `tsc --noEmit` + `eslint` green; `whitebox-containment` tests still green. + +## Execution Manifest +```json +{ + "execution-manifest": [ + { + "id": "deps-assets", + "files": ["package.json"], + "change": "Add web-tree-sitter dependency and bundle prebuilt grammar .wasm assets for py/js/ts/tsx/go/java/c/cpp (pinned versions), ensuring they ship to dist. package.json + build/copy only — no source parser code here. Environmental — author yourself.", + "accept": "npm ci >/dev/null 2>&1 && node -e \"require('web-tree-sitter')\" && test -d node_modules/web-tree-sitter", + "local": false + }, + { + "id": "coverage-setup", + "files": ["package.json", "vitest.config.ts"], + "change": "Add @vitest/coverage-v8 devDependency and create vitest.config.ts containing ONLY a coverage block (provider v8; include the new files src/recon/param-split.ts, ts-grammars.ts, ts-parse.ts; per-file thresholds statements/branches/functions/lines = 100). Do NOT add test.include/root/environment keys — the existing zero-config `vitest run src` behavior must be byte-preserved. Owned (config).", + "accept": "npx vitest run src >/dev/null 2>&1 && npx vitest run --coverage src/__tests__/redact.test.ts 2>&1 | grep -qiE 'coverage|% ' && test -f vitest.config.ts", + "local": false + }, + { + "id": "grammar-registry", + "files": ["src/recon/ts-grammars.ts", "src/__tests__/ts-grammars.test.ts"], + "change": "Create ts-grammars.ts: extension->{languageId, wasmFile, query (tree-sitter S-expr with @name/@def), captures}; a wasm-path resolver; initGrammars(exts) (async, one-time, cached, try/catch -> empty registry on failure); getParser(ext) sync accessor. Specialized queries + fail-open init — author yourself. Author ts-grammars.test.ts as the RED baseline.", + "accept": "npx vitest run src/__tests__/ts-grammars.test.ts", + "local": false + }, + { + "id": "param-split", + "files": ["src/recon/param-split.ts"], + "change": "Create param-split.ts exporting splitParamList(raw, lang): bare param identifiers per test_param-split. Python drops self/cls, strips *args and ':'-annotations and '='-defaults; Go/TS name-first (name before ':' or space); Java type-first (name is last token of each comma part). Must branch on lang (the 'a b' case proves it). Empty -> [].", + "accept": "npx vitest run src/__tests__/param-split.test.ts", + "forbid": ["new_deps"], + "kind": "create", + "local": true + }, + { + "id": "ts-parse", + "files": ["src/recon/ts-parse.ts", "src/__tests__/ts-parse.test.ts"], + "change": "Create ts-parse.ts: nodeToCodeBlock(node, source, path) -> full CodeBlock (id, kind, lineStart/lineEnd from node position, params via splitParamList on the parameters-node text, decorators [], body via node text); parseFileMultiLang(path, content, ext): CodeBlock[] — SYNC — `.py` → parseFile; non-.py via getGrammar(ext), returning [] on no-grammar/unsupported/any error (never the Python regex). MUST bound parse time (wall-clock via progressCallback) so a pathological input cannot spin — on timeout, treat as parse failure and return []. Integration crux — author yourself.", + "accept": "npx vitest run src/__tests__/ts-parse.test.ts", + "local": false + }, + { + "id": "sink-patterns", + "files": ["src/recon/code-ingest.ts"], + "change": "Additively extend ONLY the two contiguous consts DANGEROUS_SINK_RE and OUTBOUND_REQUEST_RE (L173-179) with cross-language sinks (Java Runtime.exec/ProcessBuilder, Go exec.Command/http.Get, C system(/popen(, JS child_process/fetch/axios) per test_sink-patterns, WITHOUT dropping existing Python patterns and WITHOUT matching benign ids like System.out/fileSystem. Export both consts for the test. (SINK_EVIDENCE_RES enrichment deferred to a follow-up — keeps this a single region.)", + "accept": "npx vitest run src/__tests__/sink-patterns.test.ts", + "forbid": ["new_deps"], + "local": true + }, + { + "id": "multilang-config", + "files": ["src/recon/code-ingest.ts"], + "change": "Add createMultiLangIngestConfig(repoRoot): IngestConfig mirroring createPythonIngestConfig with includeExts covering exactly py/js/ts/tsx/go/java/c/cpp (leading dots), per test_multilang-config (all included, .md excluded).", + "accept": "npx vitest run src/__tests__/multilang-config.test.ts", + "forbid": ["new_deps"], + "local": true + }, + { + "id": "server-init-wiring", + "files": ["src/server.ts"], + "change": "Invoke `await initGrammars(SUPPORTED_EXTS)` ONCE at server bootstrap (before app.listen) so grammars are loaded before any ingest request; on init failure log and continue (registry empty -> .py via parseFile, other languages -> []). server.ts is the ONLY ingest entrypoint (6060/6126); do NOT wire cli.ts (the CLI never ingests). Cross-cutting entrypoint — author yourself.", + "accept": "npx tsc --noEmit", + "local": false + }, + { + "id": "ingest-wiring", + "files": ["src/recon/code-ingest.ts", "src/__tests__/ts-parse-multilang-rank.test.ts"], + "change": "In ingestRepository (KEEP SYNC), dispatch each file through parseFileMultiLang(path, content, ext) (language-scoped fail-open: .py -> parseFile, other languages -> []), preserving byte/file ceilings. Author the headline multilang-rank regression test (non-.py blocks extracted + attack_surface ranked + cross-language reachability). Do NOT change ingestRepository's signature or any caller. Author yourself.", + "accept": "npx vitest run src/__tests__/ts-parse-multilang-rank.test.ts src/__tests__/code-ingest.test.ts", + "local": false + }, + { + "id": "whitebox-wiring", + "files": ["src/recon/whitebox.ts", "src/__tests__/whitebox-multilang.test.ts"], + "change": "BLOCKING production wiring: in whitebox.ts repoint BOTH ingestRepository calls (ingestRepoToSourceContext L125 [sync], runWhiteboxAnalysis L205 [already async — leave async]) from createPythonIngestConfig to createMultiLangIngestConfig — a pure argument swap, no signature change. Author the regression test whitebox-multilang.test.ts driving a temp Go+TS repo through ingestRepoToSourceContext and asserting non-.py source appears in the returned sourceContext (proves the feature reaches production, not just tests). Verify existing whitebox-containment tests stay green. Author yourself.", + "accept": "npx vitest run src/__tests__/whitebox-multilang.test.ts src/__tests__/whitebox-containment.test.ts && npx tsc --noEmit && npm run lint", + "local": false + }, + { + "id": "adversarial-tests", + "files": ["src/__tests__/ts-parse-adversarial.test.ts"], + "change": "Author a hostile-input suite exercising the FULL wired path (parseFileMultiLang + ingestRepository): malformed/syntactically-broken source per language; a .go file containing Python (and vice-versa); binary/non-UTF8 bytes; a multi-MB generated file and a deeply-nested/very-long-line file (byte/file ceilings hold); a PATHOLOGICAL input that would make tree-sitter spin (asserts parse returns within the timeout and fail-opens — proves the ts-parse timeout is real, not aspirational); a symlink and a symlink-loop under the repo root (crawl must not escape the contained root or hang); empty file; unicode identifiers; a file that makes tree-sitter error. Assert: never throws (non-.py fail-opens to [], .py to parseFile), bounded wall-time, CodeBlock invariants hold (lineStart<=lineEnd, body non-empty when name present, params is string[]), ingest resolves. Owned (crafted adversarial inputs).", + "accept": "npx vitest run src/__tests__/ts-parse-adversarial.test.ts", + "local": false + }, + { + "id": "readme-status", + "files": ["README.md"], + "change": "Flip the two rows that literally say 'Python-only' to multi-language (web-tree-sitter), Python-regex fail-open noted: the 'What it hunts' row (⚠️ Python-only ingest, ~L56) and the 'What ships today' row (Python-only regex ingest, ~L115). Leave the 'Coverage by domain' Code row (~L130, 'experimental') alone — it is a maturity axis, not the Python-only axis, and stays honest until the PR proves out. (Multi-region doc edit — the single-region rule is waived for docs.)", + "accept": "grep -qi 'web-tree-sitter' README.md && ! grep -qi 'Python-only regex ingest' README.md && ! grep -qi 'Python-only ingest' README.md", + "forbid": ["new_deps"], + "local": true + } + ] +} +``` + +## Risk Assessment (post-review) +- ~~[High] async ripple~~ → **eliminated** by the sync + bootstrap-init architecture. +- **[Med] grammar wasm sourcing/version** — owned `deps-assets`; pin versions; the fail-open init means a bad bundle preserves Python ingest while non-Python files yield `[]`, not a crash. +- **[Med] cross-language call-graph** — `buildCallGraph` name-matching may mis-handle non-Python ids; guarded by the reachability assertion in the headline test. +- **[Low] sink-regex FP** — guarded by `System.out`/`fileSystem` negatives. +- **[Low] bootstrap race** — pre-init-completion ingest preserves Python parsing while non-Python files yield `[]`. + +## Rollout — gated phases (each can bounce back) +1. **Build.** Branch `feat/multilang-ingest`; land the 12 nodes in manifest order (fleet: `start-fleet.sh`). I implement the 8 owned; cascade fills the 4 local. +2. **Coverage gate (hard).** `npx vitest run --coverage` → **100%** on the new files (thresholds fail the run otherwise); manually confirm every new branch/function in `code-ingest.ts`/`whitebox.ts` is hit (diff-coverage), carve-out only the `server.ts`/`cli.ts` bootstrap lines (smoke-covered). Gap → add tests, re-run. +3. **Adversarial phase.** `ts-parse-adversarial.test.ts` green, then dispatch a **`security-test-engineer`** subagent to actively break the built code against a hostile-repo corpus (malformed/mixed-language/binary/pathological inputs, resource-exhaustion, parser hangs). Every defect fixed; re-run coverage. +4. **Strict review gate.** Apply the **`/code-review-pr-strict`** methodology via parallel specialist subagents on the FULL diff — **security-critical detection runs first**, then correctness/perf/maintainability, with false-positive elimination and the mandatory verification protocol. Confirm coverage is meaningful assertions (no line-touching). Resolve every confirmed finding; **re-review until clean**. +5. **Ship.** `npm test` + `tsc --noEmit` + `eslint` + `npm run smoke` (covers the server bootstrap init line) + coverage all green → open PR; body carries spec §4 decision record, the node split, coverage report, adversarial summary, and review sign-off. +- Follow-ups (own issues): SINK_EVIDENCE_RES cross-language enrichment; cxpak `serve` phase-2 ranking upgrade. + +## Success Criteria +- [ ] Non-`.py` functions extracted + security-ranked (headline regression green, incl. cross-lang reachability). +- [ ] **Feature reaches production** — `whitebox-multilang.test.ts` drives multilang through `ingestRepoToSourceContext` (not just `createMultiLangIngestConfig` directly). +- [ ] Fail-open verified at BOTH boundaries: init failure (empty registry) and per-file unsupported ext. +- [ ] Python behavior byte-unchanged (existing suite green, no test edits). +- [ ] Both README "Python-only" rows flipped. `tsc`/`eslint`/`vitest` all green (eslint gated in `whitebox-wiring`). +- [ ] **100% coverage** on new files + all new branches in edited files (carve-out: bootstrap lines) — coverage gate green. +- [ ] **Adversarial phase** passed: crafted-input suite green AND `security-test-engineer` subagent found no unfixed break. +- [ ] **Strict review** (`/code-review-pr-strict`, security-first) clean — all confirmed findings resolved, assertions verified meaningful. + +--- +**Status:** Draft — pending human sign-off +**Date:** 2026-07-10 diff --git a/docs/design/multilang-ingest-spec.md b/docs/design/multilang-ingest-spec.md new file mode 100644 index 0000000..f38baf4 --- /dev/null +++ b/docs/design/multilang-ingest-spec.md @@ -0,0 +1,124 @@ +# Multi-language white-box ingest for T3MP3ST + +**Status:** Draft design (for an upstream PR to `elder-plinius/T3MP3ST`) +**Date:** 2026-07-10 +**Scope:** Replace the Python-only ingest parser with a portable multi-language parser, reusing T3MP3ST's existing security-ranking pipeline unchanged. + +--- + +## 1. Problem + +T3MP3ST markets white-box source hunting as a proven strength ("Code: ✅ Proven — CVE-Zero 8/10 across 7 languages"), but the ingest **engine** only reads Python. `src/recon/code-ingest.ts:6` says so itself: + +> "This is a PYTHON-ONLY, regex-based 'AST-lite' PROTOTYPE … The real version is tree-sitter (or a language server) + multi-language." + +The README status table flags it: **Code → ⚠️ Python-only ingest**. Point the white-box flow at a Go/Java/C/TS repo and `crawl` keeps only `.py`; every non-Python file is invisible to the hunt. + +## 2. Goal / Non-goals + +**Goal.** White-box ingest parses many languages, feeding T3MP3ST's *existing* security-ranking so multi-language repos get the same attack-surface-ordered analysis Python repos get today. + +**Non-goals.** +- Not changing the security model (`classify`/`prioritize`/`reachability`/`context-pack`) — it is reused verbatim. +- Not adding an external binary, native-compiled dependency, or new API key. +- Not building cross-language taint/call-graph analysis in this PR (see §9, phase 2). + +## 3. Key insight — the seam is one stage + +The ingest is a pipeline (`whitebox.ts`): `code-ingest (SELECT) → context-pack (BUDGET) → orchestrator (REASON)`. Within SELECT, only **one** stage is language-bound: + +- `parseFile()` (`code-ingest.ts:409`) — Python `def`/`class` regex → `CodeBlock[]`. **Language-bound.** +- Everything downstream operates on `CodeBlock {name, params, body, line, …}` via text regexes over the block body/name/params: + - `buildCallGraph`, `findEntryPoints`, `reachability` — walk `CodeBlock` names. Language-agnostic. + - `classify` (per-block `Exposure`) and `prioritize` (score) — run `DANGEROUS_SINK_RE`, `RISKY_PARAM_RE`, `OUTBOUND_REQUEST_RE`, `SECURITY_CONTROL_NAME_RE` over the block **body/params**. Mechanism is language-agnostic. + +So the change is surgical: **replace `parseFile` with a multi-language block extractor that emits the same `CodeBlock` shape.** No downstream code changes. + +## 4. Decision record — how to parse (embedded ADR; T3MP3ST has no ADR dir) + +**Decision: `web-tree-sitter` (WASM), as the default on every platform.** + +Options considered: + +| Option | Nature | Rejected because | +|---|---|---| +| Extend regex per language | fragile per-language regex | The exact trap the file header warns against; unmaintainable across 40 grammars. | +| `tree-sitter` (node-tree-sitter) | **native** N-API C++ addon; fastest | Compiles native code at install (node-gyp/toolchain) or needs per-OS/ABI prebuilds. T3MP3ST has **zero** native deps today and ships via `npx tempest` / `npm install` onto arbitrary machines, **fully offline**, cross-platform (open Windows bug #18). Native compile is the #1 install-failure mode in exactly that scenario. | +| Per-language JS parsers (`@babel/parser`…) | pure JS, one language each | Cannot cover many languages. | +| Shell out to cxpak (external Rust tool) | multi-language, but external | Adds a binary install (brew/cargo/docker); and cxpak exposes **no** `{name, params, body, span}` symbol interface (CLI = markdown prose; `serve` = analyses, not a symbol table; `lsp` = MCP-over-stdio, no `documentSymbol`) — verified. Cannot cheaply supply `CodeBlock`s. | +| **`web-tree-sitter` (WASM)** | official WASM build of tree-sitter | **Chosen.** | + +**Rationale.** web-tree-sitter is the WASM sibling of the native binding — same grammars, same parse quality — with **zero compile step**, running identically on any OS/Node/offline. It preserves T3MP3ST's pure-JS, portable, keyless character. The native binding's ~2–5× parse-speed edge is worthless here: ingest parses a repo **once per mission**, dwarfed by LLM latency. Grammar `.wasm` files are sourced from a prebuilt collection (e.g. `tree-sitter-wasms`; exact package/coverage confirmed at implementation time), not built locally. + +**Consequence / accepted cost.** Async grammar init; a few MB of bundled wasm for the top languages (lazy-loaded by extension); slightly more setup code than a `require()` grammar. + +## 5. Architecture + +``` +crawl (broadened includeExts) + → parseFileMultiLang(path, content) ← NEW: web-tree-sitter → CodeBlock[] + ├─ ext == .py → parseFile() (existing Python regex, unchanged) + ├─ non-.py, grammar loaded → tree-sitter query → CodeBlock[] + └─ non-.py, no grammar / timeout / err → [] (never the Python regex) ← fail-open + → buildCallGraph → findEntryPoints → reachability → classify → prioritize (UNCHANGED) + → context-pack (BUDGET) → orchestrator (REASON) (UNCHANGED) +``` + +**SYNC architecture (revised — grammar init hoisted to bootstrap so the ingest path stays synchronous).** web-tree-sitter's `Parser.init()`/`Language.load()` are async (one-time); `parser.parse()` is sync. So init runs once at bootstrap and the per-file parse is sync — `ingestRepository` keeps its sync signature and there is no async ripple. + +**New units:** +- `src/recon/ts-grammars.ts` — `initGrammars(exts): Promise` (one-time, cached, `try/catch` → empty registry on failure = fail-open); `getParser(ext): Parser | undefined` (sync accessor); the ext→{query, captures, wasm} registry. +- `src/recon/ts-parse.ts` — `parseFileMultiLang(path, content, ext): CodeBlock[]` (**sync**): `.py` → `parseFile()`; non-`.py` with a loaded grammar → `parser.parse()` (with a **bounded parse time** — a hostile input must not spin) → query → `nodeToCodeBlock`; non-`.py` with no grammar / timeout / error → `[]` (never the Python regex — routing brace/`class` syntax through it would emit corrupt blocks). `nodeToCodeBlock` fills the full `CodeBlock` shape (`name` from `@name`, `body`/`lineStart`/`lineEnd` from node position, `params` via `splitParamList(text, lang)`, `decorators []`, `kind`). +- Per-language query table (top ~8: `py, js, ts, tsx, go, java, c, cpp`). cxpak's grammar queries are a reference for the capture patterns. + +**Wiring.** (1) Bootstrap: `await initGrammars(...)` once at **`server.ts` startup** (the only ingest entrypoint — `server.ts:6060/6126`; the CLI never ingests, so `cli.ts` is not wired). (2) `ingestRepository()` (`code-ingest.ts:749`, **stays sync**) dispatches each file through `parseFileMultiLang(...)` (language-scoped fail-open: `.py` → `parseFile`, other languages → `[]`). (3) **Production caller swap (required):** both `whitebox.ts` callers of `ingestRepository` — `ingestRepoToSourceContext` (L125) and `runWhiteboxAnalysis` (L205) — move from `createPythonIngestConfig` to `createMultiLangIngestConfig`; without this the extractor only sees `.py` in production. + +## 6. Security-ranking reuse & cross-language sinks + +`classify`/`prioritize` are reused unchanged. Their regexes (`DANGEROUS_SINK_RE`, etc.) are Python-tuned; cross-language sinks that already overlap (`eval(`, `exec(`, `system(`) still fire, so ranking **degrades gracefully**, never breaks. This PR **additively** broadens the sink/param patterns with a few cross-language entries (`Runtime.exec`, `ProcessBuilder`, `exec.Command`, `os/exec`, `system(`, `popen(`, `child_process`), each with a test. Deeper per-language tuning is out of scope (incremental follow-up). + +## 7. Fail-open + +- `.py` always → `parseFile` (Python files unaffected). A non-`.py` file with a missing/unsupported/failed grammar → `[]` (honest absence) — **never** the Python regex, which would mis-parse brace/`class` syntax into corrupt blocks (wrong line spans, lost methods). This language-scoping is part of the security boundary. +- web-tree-sitter init throws in a hostile env → the registry stays empty; `.py` still ingests via the Python regex, every other language yields `[]` (and a one-time `console.warn`). **Ingest never crashes a mission.** + +## 8. Test plan (vitest — matches their suite; deterministic, no model calls) + +1. **Multi-language extraction (headline regression).** Temp repo (`mkdtempSync`, mirroring `code-ingest.test.ts`) with `.py` + `.go` + `.ts` files, each holding a function with a known sink. Assert non-`.py` functions are extracted as `CodeBlock`s **and** land as `attack_surface`/high-priority after `classify`/`prioritize`. This test *is* the proof the limitation is fixed. +2. **CodeBlock fidelity.** For a known TS/Go function, assert `name`, `params`, `line`, and `body` (contains the sink line) are correct. +3. **Fail-open — unsupported ext.** A `.zig` (or grammar-absent, non-`.py`) file returns `[]`, no throw (a `.py` file still routes to `parseFile`). +4. **Fail-open — grammar init failure.** Force `initGrammars` to fail (bad wasm path) → registry stays empty, `getGrammar` returns undefined, `.py` routes to `parseFile` and every other language returns `[]`; `ingestRepository` resolves normally, mission-safe. +5. **Cross-language sink patterns.** Unit-test each added regex against a positive and negative snippet. +6. *(optional, `--ignored` like `spine-live.test.ts`)* real wasm load against a fixture repo — live smoke. + +**Acceptance:** `npm test` green, `npm run typecheck` clean; test 1 proves multi-language, tests 3–4 prove fail-open. + +## 9. PR scope + +**In:** `ts-grammars.ts` (registry + `initGrammars`/`getParser`) + `ts-parse.ts` (`parseFileMultiLang`/`nodeToCodeBlock`) + `param-split.ts`; `web-tree-sitter` dep + grammar wasm assets (top ~8 langs); bootstrap `initGrammars` call (server/CLI); `ingestRepository`/`crawl` wiring (**sync**, broadened exts) + `createMultiLangIngestConfig`; **`whitebox.ts` caller swap (both call sites)**; additive cross-language sink patterns; tests; README status flip (both "Python-only" rows). + +**Out (name in PR as deliberate):** cxpak integration (phase 2); deep per-language sink tuning; the `serve`-backed cross-language call-graph/taint upgrade; node-tree-sitter fast path. + +**Size target:** 3 new source files (`param-split`, `ts-grammars`, `ts-parse`) + edits to 4 (`code-ingest`, `whitebox`, `server`, `cli`) + coverage config + test suites (unit, headline regression, production-path, adversarial). Larger than a trivial PR because of the mandated quality gates (§12), but still one cohesive feature — deliberately not a mega-diff (cf. #61/#17, closed for sprawl). + +## 10. Phase 2 (noted, not this PR) — cxpak as an optional ranking upgrade + +When installed, cxpak's `serve` endpoints supply genuinely better multi-language facts than T3MP3ST's regex heuristics: `/call_graph` (cross-language edges), `/api_surface` (entry points with file+line), `/data_flow` (taint paths). A later, fail-open, opt-in PR can let cxpak *upgrade* the ranking inputs (entry points, reachability, sinks) while the web-tree-sitter parser from phase 1 remains the universal default. Not required for the headline win. + +## 11. Risks + +- **Grammar wasm sourcing/versioning** — confirm the collection package and language coverage at implementation time; pin versions. +- **production caller swap** — the multilang config must be repointed at BOTH `whitebox.ts` `ingestRepository` callers (L125, L205); a regression test drives `ingestRepoToSourceContext` end-to-end so the feature can't silently no-op in production. *(The async ripple is designed out by the sync + bootstrap-init architecture.)* +- **bootstrap race** — a mission firing before `initGrammars` resolves still parses Python through `parseFile`; non-Python files yield `[]` for that call. Entrypoints `await` init before serving. +- **Param extraction variance** across grammars — mitigated by falling back to slicing the signature line when the parameters node is absent. +- **Bundle size** — mitigated by lazy-loading grammars per encountered extension rather than eager-loading all. + +## 12. Quality gates (mandatory — all three block merge) + +1. **100% coverage of our code.** vitest coverage (`@vitest/coverage-v8`, `vitest.config.ts` with a coverage-only block) with per-file thresholds = 100% (statements/branches/functions/lines), **CI-enforced** via `npm run test:coverage`, on every NEW file (`param-split.ts`, `ts-grammars.ts`, `ts-parse.ts`). The new branches/functions added to edited files (`code-ingest.ts` sink patterns + `createMultiLangIngestConfig` + dispatch, both `whitebox.ts` caller swaps) are **covered by targeted regression tests** (`sink-patterns`, `ts-parse-multilang-rank`, `whitebox-multilang` — including a spy test on the `runWhiteboxAnalysis` swap so it can't silently revert), but not per-file-threshold-gated: a whole-file 100% threshold is unachievable on large pre-existing files. Sole coverage carve-out: the single `await initGrammars(...)` wiring line at `server.ts` bootstrap (`initGrammars` itself is 100% unit-tested; the line is `npm run smoke`-covered). Coverage is necessary-not-sufficient — gate 3 verifies the covering assertions are meaningful (no line-touching / cheater tests). + +2. **Adversarial local testing.** Because ingest parses **untrusted target source**, a crafted hostile-input suite (`ts-parse-adversarial.test.ts`) drives the full wired path against malformed/mixed-language/binary/oversized inputs, a **pathological input that would make tree-sitter spin** (asserting the `ts-parse` parse-timeout fail-opens — not aspirational), and **symlink/loop entries** (crawl must not escape the contained root or hang); asserts fail-open + CodeBlock invariants + bounded wall-time/bytes. Then a `security-test-engineer` subagent actively attempts to break the built code (parser hangs, resource exhaustion, mission crash) against a hostile-repo corpus. All defects fixed before review. + +3. **Strict subagent code review.** The `/code-review-pr-strict` methodology on the full diff: parallel specialist subagents with **security-critical detection first**, false-positive elimination, and the mandatory verification protocol. Every confirmed finding resolved; re-review until clean. This is the merge gate. + +Gate order at rollout: build → coverage → adversarial → strict review → ship; any gate can bounce back to build. diff --git a/package-lock.json b/package-lock.json index 065aee3..fc6c6ff 100755 --- a/package-lock.json +++ b/package-lock.json @@ -23,7 +23,9 @@ "gradient-string": "^2.0.2", "inquirer": "^9.2.15", "ora": "^8.0.1", - "tsx": "^4.7.0" + "tree-sitter-wasms": "0.1.13", + "tsx": "^4.7.0", + "web-tree-sitter": "0.25.10" }, "bin": { "t3mp3st": "dist/cli.js", @@ -37,6 +39,7 @@ "@types/gradient-string": "^1.1.5", "@types/inquirer": "^9.0.7", "@types/node": "^20.11.0", + "@vitest/coverage-v8": "4.1.9", "eslint": "^9.39.2", "typescript": "^5.3.3", "typescript-eslint": "^8.51.0", @@ -46,6 +49,66 @@ "node": ">=18.0.0" } }, + "node_modules/@babel/helper-string-parser": { + "version": "7.29.7", + "resolved": "https://registry.npmjs.org/@babel/helper-string-parser/-/helper-string-parser-7.29.7.tgz", + "integrity": "sha512-Pb5ijPrZ89GDH8223L4UP8i6QApWxs04RbPQJTeWDV0/keR2E36MeKnyr6LYmUUvqRRI+Iv87SuF1W6ErINzYw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=6.9.0" + } + }, + "node_modules/@babel/helper-validator-identifier": { + "version": "7.29.7", + "resolved": "https://registry.npmjs.org/@babel/helper-validator-identifier/-/helper-validator-identifier-7.29.7.tgz", + "integrity": "sha512-qehxGkRj55h/ff8EMaJ+cYhyaKlHIxqYDn682wQD7RNp9UujOQsHog2uS0r2vzr4pW+sXf90NeeayjcNaX3fFg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=6.9.0" + } + }, + "node_modules/@babel/parser": { + "version": "7.29.7", + "resolved": "https://registry.npmjs.org/@babel/parser/-/parser-7.29.7.tgz", + "integrity": "sha512-hnORnjP/1P/zFEndoeX+n+t1RwWRJiJpM/jO7FW32Kn9r5+sJB2JWOdYo4L6k78j15eCwY3Gm/7364B1EMwtNg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@babel/types": "^7.29.7" + }, + "bin": { + "parser": "bin/babel-parser.js" + }, + "engines": { + "node": ">=6.0.0" + } + }, + "node_modules/@babel/types": { + "version": "7.29.7", + "resolved": "https://registry.npmjs.org/@babel/types/-/types-7.29.7.tgz", + "integrity": "sha512-4zBIxpPzowiZpusoFkyGVwakdRJUyuH5PxQ/PrqghfdFWWasvnCdPfQXHrenDai+gyLARulZjZowCOj6fjT4pA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@babel/helper-string-parser": "^7.29.7", + "@babel/helper-validator-identifier": "^7.29.7" + }, + "engines": { + "node": ">=6.9.0" + } + }, + "node_modules/@bcoe/v8-coverage": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/@bcoe/v8-coverage/-/v8-coverage-1.0.2.tgz", + "integrity": "sha512-6zABk/ECA/QYSCQ1NGiVwwbQerUCZ+TQbp64Q3AgmfNvurHH0j8TtXa1qbShXA6qqkpAj4V5W8pP6mLe1mcMqA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=18" + } + }, "node_modules/@colors/colors": { "version": "1.5.0", "resolved": "https://registry.npmjs.org/@colors/colors/-/colors-1.5.0.tgz", @@ -768,6 +831,16 @@ "node": ">=18" } }, + "node_modules/@jridgewell/resolve-uri": { + "version": "3.1.2", + "resolved": "https://registry.npmjs.org/@jridgewell/resolve-uri/-/resolve-uri-3.1.2.tgz", + "integrity": "sha512-bRISgCIjP20/tbWSPWMEi54QVPRZExkuD9lJL+UIxUKtwVJA8wW1Trb1jMs1RFXo1CBTNZ/5hpC9QvmKWdopKw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=6.0.0" + } + }, "node_modules/@jridgewell/sourcemap-codec": { "version": "1.5.5", "resolved": "https://registry.npmjs.org/@jridgewell/sourcemap-codec/-/sourcemap-codec-1.5.5.tgz", @@ -775,6 +848,17 @@ "dev": true, "license": "MIT" }, + "node_modules/@jridgewell/trace-mapping": { + "version": "0.3.31", + "resolved": "https://registry.npmjs.org/@jridgewell/trace-mapping/-/trace-mapping-0.3.31.tgz", + "integrity": "sha512-zzNR+SdQSDJzc8joaeP8QQoCQr8NuYx2dIIytl1QeBEZHJ9uW6hebsrYgbz8hJwUQao3TWCMtmfV8Nu1twOLAw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@jridgewell/resolve-uri": "^3.1.0", + "@jridgewell/sourcemap-codec": "^1.4.14" + } + }, "node_modules/@modelcontextprotocol/sdk": { "version": "1.29.0", "resolved": "https://registry.npmjs.org/@modelcontextprotocol/sdk/-/sdk-1.29.0.tgz", @@ -1864,6 +1948,37 @@ "url": "https://opencollective.com/typescript-eslint" } }, + "node_modules/@vitest/coverage-v8": { + "version": "4.1.9", + "resolved": "https://registry.npmjs.org/@vitest/coverage-v8/-/coverage-v8-4.1.9.tgz", + "integrity": "sha512-G9/lgqibheLVBDRuya45EbsEXTYcWoSG+TLg7i2axuzx0Eq62eXn+aWXyaVdV5vKvFSWd6ywcX8hA7la9Pvu8g==", + "dev": true, + "license": "MIT", + "dependencies": { + "@bcoe/v8-coverage": "^1.0.2", + "@vitest/utils": "4.1.9", + "ast-v8-to-istanbul": "^1.0.0", + "istanbul-lib-coverage": "^3.2.2", + "istanbul-lib-report": "^3.0.1", + "istanbul-reports": "^3.2.0", + "magicast": "^0.5.2", + "obug": "^2.1.1", + "std-env": "^4.0.0-rc.1", + "tinyrainbow": "^3.1.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + }, + "peerDependencies": { + "@vitest/browser": "4.1.9", + "vitest": "4.1.9" + }, + "peerDependenciesMeta": { + "@vitest/browser": { + "optional": true + } + } + }, "node_modules/@vitest/expect": { "version": "4.1.9", "resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-4.1.9.tgz", @@ -2134,6 +2249,18 @@ "node": ">=12" } }, + "node_modules/ast-v8-to-istanbul": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/ast-v8-to-istanbul/-/ast-v8-to-istanbul-1.0.4.tgz", + "integrity": "sha512-0bC0/4bTSrnwdhU3IsZDwEdojvuPrSg59OYZfKsLRtJZ0u8VBx9DebfqqG8bRdCC0I7vjgxmPi41P0lpkhJHtA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@jridgewell/trace-mapping": "^0.3.31", + "estree-walker": "^3.0.3", + "js-tokens": "^10.0.0" + } + }, "node_modules/atomically": { "version": "2.1.0", "resolved": "https://registry.npmjs.org/atomically/-/atomically-2.1.0.tgz", @@ -3600,6 +3727,13 @@ "node": ">=16.9.0" } }, + "node_modules/html-escaper": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/html-escaper/-/html-escaper-2.0.2.tgz", + "integrity": "sha512-H2iMtd0I4Mt5eYiapRdIDjp+XzelXQ0tFE4JS7YFwFevXXMmOp9myNrUvCg0D6ws8iqkRPBfKHgbwig1SmlLfg==", + "dev": true, + "license": "MIT" + }, "node_modules/http-errors": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-2.0.1.tgz", @@ -3988,6 +4122,45 @@ "integrity": "sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw==", "license": "ISC" }, + "node_modules/istanbul-lib-coverage": { + "version": "3.2.2", + "resolved": "https://registry.npmjs.org/istanbul-lib-coverage/-/istanbul-lib-coverage-3.2.2.tgz", + "integrity": "sha512-O8dpsF+r0WV/8MNRKfnmrtCWhuKjxrq2w+jpzBL5UZKTi2LeVWnWOmWRxFlesJONmc+wLAGvKQZEOanko0LFTg==", + "dev": true, + "license": "BSD-3-Clause", + "engines": { + "node": ">=8" + } + }, + "node_modules/istanbul-lib-report": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/istanbul-lib-report/-/istanbul-lib-report-3.0.1.tgz", + "integrity": "sha512-GCfE1mtsHGOELCU8e/Z7YWzpmybrx/+dSTfLrvY8qRmaY6zXTKWn6WQIjaAFw069icm6GVMNkgu0NzI4iPZUNw==", + "dev": true, + "license": "BSD-3-Clause", + "dependencies": { + "istanbul-lib-coverage": "^3.0.0", + "make-dir": "^4.0.0", + "supports-color": "^7.1.0" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/istanbul-reports": { + "version": "3.2.0", + "resolved": "https://registry.npmjs.org/istanbul-reports/-/istanbul-reports-3.2.0.tgz", + "integrity": "sha512-HGYWWS/ehqTV3xN10i23tkPkpH46MLCIMFNCaaKNavAXTF1RkqxawEPtnjnGZ6XKSInBKkiOA5BKS+aZiY3AvA==", + "dev": true, + "license": "BSD-3-Clause", + "dependencies": { + "html-escaper": "^2.0.0", + "istanbul-lib-report": "^3.0.0" + }, + "engines": { + "node": ">=8" + } + }, "node_modules/jose": { "version": "6.1.3", "resolved": "https://registry.npmjs.org/jose/-/jose-6.1.3.tgz", @@ -3997,6 +4170,13 @@ "url": "https://github.com/sponsors/panva" } }, + "node_modules/js-tokens": { + "version": "10.0.0", + "resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-10.0.0.tgz", + "integrity": "sha512-lM/UBzQmfJRo9ABXbPWemivdCW8V2G8FHaHdypQaIy523snUjog0W71ayWXTjiR+ixeMyVHN2XcpnTd/liPg/Q==", + "dev": true, + "license": "MIT" + }, "node_modules/js-yaml": { "version": "4.3.0", "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-4.3.0.tgz", @@ -4392,6 +4572,34 @@ "@jridgewell/sourcemap-codec": "^1.5.5" } }, + "node_modules/magicast": { + "version": "0.5.3", + "resolved": "https://registry.npmjs.org/magicast/-/magicast-0.5.3.tgz", + "integrity": "sha512-pVKE4UdSQ7DvHzivsCIFx2BJn1mHG6KsyrFcaxFx6tONdneEuThrDx0Cj3AMg58KyN4pzYT+LHOotxDQDjNvkw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@babel/parser": "^7.29.3", + "@babel/types": "^7.29.0", + "source-map-js": "^1.2.1" + } + }, + "node_modules/make-dir": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/make-dir/-/make-dir-4.0.0.tgz", + "integrity": "sha512-hXdUTZYIVOt1Ex//jAQi+wTZZpUpwBj/0QsOzqegb3rGMMeJiSEu5xLHnYfBrRV4RH2+OCSOO95Is/7x1WJ4bw==", + "dev": true, + "license": "MIT", + "dependencies": { + "semver": "^7.5.3" + }, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, "node_modules/math-intrinsics": { "version": "1.1.0", "resolved": "https://registry.npmjs.org/math-intrinsics/-/math-intrinsics-1.1.0.tgz", @@ -5517,6 +5725,15 @@ "node": ">=0.6" } }, + "node_modules/tree-sitter-wasms": { + "version": "0.1.13", + "resolved": "https://registry.npmjs.org/tree-sitter-wasms/-/tree-sitter-wasms-0.1.13.tgz", + "integrity": "sha512-wT+cR6DwaIz80/vho3AvSF0N4txuNx/5bcRKoXouOfClpxh/qqrF4URNLQXbbt8MaAxeksZcZd1j8gcGjc+QxQ==", + "license": "Unlicense", + "dependencies": { + "tree-sitter-wasms": "^0.1.11" + } + }, "node_modules/ts-api-utils": { "version": "2.3.0", "resolved": "https://registry.npmjs.org/ts-api-utils/-/ts-api-utils-2.3.0.tgz", @@ -5870,6 +6087,20 @@ "defaults": "^1.0.3" } }, + "node_modules/web-tree-sitter": { + "version": "0.25.10", + "resolved": "https://registry.npmjs.org/web-tree-sitter/-/web-tree-sitter-0.25.10.tgz", + "integrity": "sha512-Y09sF44/13XvgVKgO2cNDw5rGk6s26MgoZPXLESvMXeefBf7i6/73eFurre0IsTW6E14Y0ArIzhUMmjoc7xyzA==", + "license": "MIT", + "peerDependencies": { + "@types/emscripten": "^1.40.0" + }, + "peerDependenciesMeta": { + "@types/emscripten": { + "optional": true + } + } + }, "node_modules/when-exit": { "version": "2.1.5", "resolved": "https://registry.npmjs.org/when-exit/-/when-exit-2.1.5.tgz", diff --git a/package.json b/package.json index 740a141..71410fe 100755 --- a/package.json +++ b/package.json @@ -55,6 +55,7 @@ "mcp": "tsx src/mcp-server.ts", "mcp:prod": "node dist/mcp-server.js", "test": "vitest run src && node scripts/test-ops-preflight.mjs", + "test:coverage": "vitest run --coverage", "test:watch": "vitest src", "lint": "eslint src/**/*.ts", "typecheck": "tsc --noEmit", @@ -120,7 +121,9 @@ "gradient-string": "^2.0.2", "inquirer": "^9.2.15", "ora": "^8.0.1", - "tsx": "^4.7.0" + "tree-sitter-wasms": "0.1.13", + "tsx": "^4.7.0", + "web-tree-sitter": "0.25.10" }, "devDependencies": { "@eslint/js": "^9.39.2", @@ -130,6 +133,7 @@ "@types/gradient-string": "^1.1.5", "@types/inquirer": "^9.0.7", "@types/node": "^20.11.0", + "@vitest/coverage-v8": "4.1.9", "eslint": "^9.39.2", "typescript": "^5.3.3", "typescript-eslint": "^8.51.0", diff --git a/src/__tests__/local-agent-tool-calling.test.ts b/src/__tests__/local-agent-tool-calling.test.ts index e22952b..01e0d93 100644 --- a/src/__tests__/local-agent-tool-calling.test.ts +++ b/src/__tests__/local-agent-tool-calling.test.ts @@ -66,7 +66,10 @@ describe('parseTextToolCalls — happy path + drift tolerance', () => { it('is DoS-safe on pathological brace input (was: quadratic ReDoS)', () => { const t0 = Date.now(); expect(parseTextToolCalls('{'.repeat(80000))).toBeUndefined(); - expect(Date.now() - t0).toBeLessThan(300); // bounded — the old greedy regex took ~2100ms + // Bounded — the old greedy regex took ~2100ms. Generous ceiling (2x below that + // regression target) so V8 coverage instrumentation / slow CI can't false-fail + // this wall-clock guard; it still fails hard if the quadratic ReDoS returns. + expect(Date.now() - t0).toBeLessThan(1000); }); }); diff --git a/src/__tests__/multilang-config.test.ts b/src/__tests__/multilang-config.test.ts new file mode 100644 index 0000000..f5cd51c --- /dev/null +++ b/src/__tests__/multilang-config.test.ts @@ -0,0 +1,24 @@ +import { describe, it, expect } from 'vitest'; +import { mkdtempSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; +import { crawl, createMultiLangIngestConfig } from '../recon/code-ingest.js'; + +describe('createMultiLangIngestConfig', () => { + const EXTS = ['py', 'js', 'ts', 'tsx', 'go', 'java', 'c', 'cpp']; + + it('crawls every supported extension and excludes non-source', () => { + const root = mkdtempSync(join(tmpdir(), 'mling-')); + for (const e of EXTS) writeFileSync(join(root, `f.${e}`), 'x\n'); + writeFileSync(join(root, 'd.md'), '# ignore'); + writeFileSync(join(root, 'e.txt'), 'ignore'); + + const files = crawl(createMultiLangIngestConfig(root)); + + for (const e of EXTS) { + expect(files.some((f) => f.endsWith(`f.${e}`)), `includes .${e}`).toBe(true); + } + expect(files.some((f) => f.endsWith('d.md'))).toBe(false); + expect(files.some((f) => f.endsWith('e.txt'))).toBe(false); + }); +}); diff --git a/src/__tests__/param-split.test.ts b/src/__tests__/param-split.test.ts new file mode 100644 index 0000000..95502b9 --- /dev/null +++ b/src/__tests__/param-split.test.ts @@ -0,0 +1,49 @@ +import { describe, it, expect } from 'vitest'; +import { splitParamList } from '../recon/param-split.js'; + +describe('splitParamList', () => { + it('python: drops self, strips annotations/defaults, drops *args', () => + expect(splitParamList('self, url: str, timeout=5, *args', 'py')).toEqual(['url', 'timeout'])); + it('go name-first, drops types', () => + expect(splitParamList('ctx context.Context, id string', 'go')).toEqual(['ctx', 'id'])); + it('java type-first, keeps trailing name', () => + expect(splitParamList('String url, int max', 'java')).toEqual(['url', 'max'])); + it('ts name: type, strips optional marker', () => + expect(splitParamList('url: string, opts?: Opts', 'ts')).toEqual(['url', 'opts'])); + it('empty', () => expect(splitParamList('', 'ts')).toEqual([])); + it('SAME raw, different lang → forces real dispatch', () => { + expect(splitParamList('a b', 'go')).toEqual(['a']); // name-first + expect(splitParamList('a b', 'java')).toEqual(['b']); // type-first + }); + it('strips surrounding parens (tree-sitter params node text)', () => + expect(splitParamList('(url string)', 'go')).toEqual(['url'])); + it('c pointers: type-first, strips *', () => + expect(splitParamList('char *url, int n', 'c')).toEqual(['url', 'n'])); + it('generics do not split params', () => + expect(splitParamList('m Map, n int', 'go')).toEqual(['m', 'n'])); + it('python drops bare * and / markers', () => + expect(splitParamList('a, *, b, /', 'py')).toEqual(['a', 'b'])); + it('python drops cls', () => + expect(splitParamList('cls, x', 'py')).toEqual(['x'])); + it('ts rest param + array type', () => + expect(splitParamList('...args: number[]', 'ts')).toEqual(['args'])); + it('cpp references and pointers stripped', () => + expect(splitParamList('const std::string &ref, int* p', 'cpp')).toEqual(['ref', 'p'])); + it('c array-suffix name stripped', () => + expect(splitParamList('char buf[]', 'c')).toEqual(['buf'])); + it('bracket + brace nesting does not split', () => { + expect(splitParamList('a int[2], b T{}', 'go')).toEqual(['a', 'b']); + }); + it('unbalanced close bracket is tolerated', () => + expect(splitParamList('a)x, b', 'go')).toEqual(['a', 'b'])); + it('non-identifier params are dropped', () => + expect(splitParamList('*, ---', 'go')).toEqual([])); + it('trailing comma tolerated', () => + expect(splitParamList('a,', 'go')).toEqual(['a'])); + it('empty segment (double comma) dropped', () => + expect(splitParamList('a,,b', 'go')).toEqual(['a', 'b'])); + it('ts name-less param dropped', () => + expect(splitParamList(': int, x: str', 'ts')).toEqual(['x'])); + it('java name-less param dropped', () => + expect(splitParamList('=v, int x', 'java')).toEqual(['x'])); +}); diff --git a/src/__tests__/sink-patterns.test.ts b/src/__tests__/sink-patterns.test.ts new file mode 100644 index 0000000..1f170e0 --- /dev/null +++ b/src/__tests__/sink-patterns.test.ts @@ -0,0 +1,69 @@ +import { describe, it, expect } from 'vitest'; +import { DANGEROUS_SINK_RE, OUTBOUND_REQUEST_RE } from '../recon/code-ingest.js'; + +describe('cross-language sinks', () => { + it('matches new cross-language sinks', () => { + for (const s of [ + 'Runtime.getRuntime().exec(cmd)', + 'exec.Command("sh")', + 'child_process.exec(x)', + 'system(buf)', + 'popen(cmd)', + 'new ProcessBuilder(cmd)', + ]) { + expect(DANGEROUS_SINK_RE.test(s), s).toBe(true); + } + }); + + it('regression: existing python sinks still match', () => { + for (const s of ['os.system(x)', 'eval(y)', 'subprocess.run(z)', 'yaml.load(f)']) { + expect(DANGEROUS_SINK_RE.test(s), s).toBe(true); + } + }); + + it('no false positive on benign / System.out / filesystem identifiers', () => { + for (const s of ['const total = sum(a, b)', 'System.out.println("debug")', 'fileSystem.readFile(x)']) { + expect(DANGEROUS_SINK_RE.test(s), s).toBe(false); + } + }); + + it('outbound covers go net/http and js fetch/axios', () => { + for (const s of ['http.Get(u)', 'http.Post(u, b)', 'fetch(u)', 'axios.get(u)']) { + expect(OUTBOUND_REQUEST_RE.test(s), s).toBe(true); + } + }); + + it('recall: idiomatic outbound/exec forms missed by the first-pass patterns', () => { + // Go http.NewRequest+client.Do, Go client.Get, Node http(s).request, axios(config), + // C exec-family, and receiver-qualified fetch — all real attack-surface idioms. + for (const s of [ + 'req, _ := http.NewRequest("GET", u, nil)', + 'resp, _ := client.Do(req)', + 'exec.CommandContext(ctx, cmd)', + 'execve(path, argv, envp)', + 'execvp(cmd, args)', + 'https.request(options, cb)', + 'axios({ url: u })', + 'return window.fetch(url)', // qualified fetch must still match (JS SSRF idiom) + 'this.fetch(userUrl)', + ]) { + expect(DANGEROUS_SINK_RE.test(s), s).toBe(true); + } + for (const s of ['client.Get(u)', 'client.Do(req)', 'http.request(u, cb)', 'axios({url:u})', 'window.fetch(u)']) { + expect(OUTBOUND_REQUEST_RE.test(s), s).toBe(true); + } + }); + + it('precision: bare-call sinks and generic .Get/.Do methods on unrelated receivers are not flagged', () => { + // `.system(`/`.popen(` on an unrelated object must not be flagged as the + // C/OS bare call. (os.system / os.popen keep matching via their own patterns.) + expect(DANGEROUS_SINK_RE.test('solarSystem.system(planets)'), 'receiver .system(').toBe(false); + expect(DANGEROUS_SINK_RE.test('os.system(cmd)'), 'os.system still').toBe(true); + // Generic keyed-store methods must NOT be mistaken for outbound HTTP — the + // .Get/.Post/.Do alternatives are receiver-qualified to `client`. + for (const s of ['return db.Get(id)', 'cache.Get(userID)', 'queue.Do(task)', 'session.Post(data)', 'config.get(key)']) { + expect(OUTBOUND_REQUEST_RE.test(s), s).toBe(false); + expect(DANGEROUS_SINK_RE.test(s), s).toBe(false); + } + }); +}); diff --git a/src/__tests__/ts-grammars.test.ts b/src/__tests__/ts-grammars.test.ts new file mode 100644 index 0000000..67cde34 --- /dev/null +++ b/src/__tests__/ts-grammars.test.ts @@ -0,0 +1,69 @@ +import { describe, it, expect, beforeEach } from 'vitest'; +import { Parser } from 'web-tree-sitter'; +import { + initGrammars, + getGrammar, + supportedExts, + __resetGrammarsForTest, +} from '../recon/ts-grammars.js'; + +describe('ts-grammars registry', () => { + beforeEach(() => __resetGrammarsForTest()); + + it('advertises the multi-language extension set', () => { + expect(supportedExts()).toEqual( + expect.arrayContaining(['.py', '.js', '.ts', '.tsx', '.go', '.java', '.c', '.cpp']), + ); + }); + + it('loads every grammar with a working def-query (multi-language, real parse)', async () => { + await initGrammars(); + const samples: Record = { + '.py': 'def fetch(url):\n return get(url)\n', + '.js': 'function fetch(url){return get(url)}\n', + '.ts': 'export function fetch(url: string){return get(url)}\n', + '.tsx': 'export function F(p: {a:number}){ return null; }\n', + '.go': 'package m\nfunc Fetch(url string) error { return nil }\n', + '.java': 'class A { void run(String u){} }\n', + '.c': 'int fetch(char *url){ return 0; }\n', + '.cpp': 'int fetch(char *url){ return 0; }\n', + }; + for (const ext of supportedExts()) { + const g = getGrammar(ext); + expect(g, `grammar for ${ext}`).toBeDefined(); + const p = new Parser(); + p.setLanguage(g!.language); + const tree = p.parse(samples[ext]); + const names = g!.query + .matches(tree!.rootNode) + .flatMap((m) => m.captures.filter((c) => c.name === 'name').map((c) => c.node.text)); + expect(names.length, `captures for ${ext}`).toBeGreaterThan(0); + } + }); + + it('is idempotent — a second init keeps the loaded registry', async () => { + await initGrammars(['.go']); + await initGrammars(['.py']); // second call is a no-op (already initialized) + expect(getGrammar('.go')).toBeDefined(); + expect(getGrammar('.py')).toBeUndefined(); + }); + + it('fail-open: per-grammar load failure skips that ext, no throw', async () => { + await expect( + initGrammars(['.go'], () => { throw new Error('bad wasm path'); }), + ).resolves.toBeUndefined(); + expect(getGrammar('.go')).toBeUndefined(); + }); + + it('fail-open: global init failure leaves the registry empty, no throw', async () => { + await expect( + initGrammars(supportedExts(), (n) => n, () => Promise.reject(new Error('runtime init failed'))), + ).resolves.toBeUndefined(); + for (const ext of supportedExts()) expect(getGrammar(ext)).toBeUndefined(); + }); + + it('ignores unknown extensions', async () => { + await initGrammars(['.zzz']); + expect(getGrammar('.zzz')).toBeUndefined(); + }); +}); diff --git a/src/__tests__/ts-parse-adversarial.test.ts b/src/__tests__/ts-parse-adversarial.test.ts new file mode 100644 index 0000000..693405b --- /dev/null +++ b/src/__tests__/ts-parse-adversarial.test.ts @@ -0,0 +1,136 @@ +/** + * Adversarial hardening — ingest parses UNTRUSTED target source. A crafted file + * must never throw, hang, or crash a mission; worst case is honest absence ([]) + * — a non-.py file is NEVER routed to the Python regex. Drives the full wired + * path (parseFileMultiLang + ingestRepository/crawl). + */ +import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest'; +import { mkdtempSync, writeFileSync, mkdirSync, symlinkSync, rmSync } from 'fs'; +import { join } from 'path'; +import { tmpdir } from 'os'; +import { Tree } from 'web-tree-sitter'; +import { initGrammars, __resetGrammarsForTest } from '../recon/ts-grammars.js'; +import { parseFileMultiLang } from '../recon/ts-parse.js'; +import { ingestRepository, createMultiLangIngestConfig, type CodeBlock } from '../recon/code-ingest.js'; + +const tmpRoots: string[] = []; + +beforeAll(async () => { + __resetGrammarsForTest(); + await initGrammars(); +}); + +afterAll(() => { + for (const r of tmpRoots) rmSync(r, { recursive: true, force: true }); +}); + +const invariants = (blocks: CodeBlock[]) => { + expect(Array.isArray(blocks)).toBe(true); + for (const b of blocks) { + expect(b.lineStart).toBeLessThanOrEqual(b.lineEnd); + expect(Array.isArray(b.params)).toBe(true); + if (b.name) expect(b.body.length).toBeGreaterThan(0); + } +}; + +describe('adversarial parseFileMultiLang', () => { + it('malformed source per language does not throw', () => { + for (const [ext, src] of [ + ['.go', 'func ('], + ['.ts', 'class {{{ function ('], + ['.java', 'class A { void ('], + ['.c', 'int f( { { {'], + ] as const) { + expect(() => invariants(parseFileMultiLang(`x${ext}`, src, ext))).not.toThrow(); + } + }); + + it('wrong-language content (Python in a .go file) yields structurally valid output', () => { + expect(() => + invariants(parseFileMultiLang('x.go', 'def fetch(url):\n return get(url)\n', '.go')), + ).not.toThrow(); + }); + + it('binary / non-UTF8 bytes do not throw', () => { + const bin = Buffer.from([0x00, 0xff, 0xfe, 0x10, 0x80, 0x00, 0x41]).toString('binary'); + expect(() => invariants(parseFileMultiLang('x.go', bin, '.go'))).not.toThrow(); + }); + + it('empty file yields no blocks, no throw', () => { + expect(parseFileMultiLang('e.go', '', '.go')).toEqual([]); + }); + + it('unicode identifiers are extracted correctly', () => { + const blocks = parseFileMultiLang('u.go', 'package m\nfunc Ünïçødé(x int) int { return x }\n', '.go'); + const fn = blocks.find((b) => b.name === 'Ünïçødé'); + expect(fn, 'unicode function name captured').toBeDefined(); + expect(fn!.params).toEqual(['x']); + }); + + it('large file parses within the wall-clock budget and still extracts', () => { + let src = 'package m\n'; + for (let i = 0; i < 20000; i++) src += `func F${i}(url string) error { return http.Get(url) }\n`; + const t0 = Date.now(); + const blocks = parseFileMultiLang('big.go', src, '.go'); + expect(Date.now() - t0).toBeLessThan(10000); // bounded — never spins + expect(blocks.length).toBeGreaterThan(0); + }); + + it('a real parse tripping the wall-clock budget fails open to [] (not mocked)', () => { + // Drive the REAL tree-sitter progressCallback path with a 0ms budget: the + // callback returns true on its first invocation, tree-sitter cancels, parse + // returns null, and the fail-open returns []. Proves the actual cancellation + // wiring — not a mock that hard-returns null. + let src = 'package m\n'; + for (let i = 0; i < 5000; i++) src += `func F${i}(a int) int { return a }\n`; + const t0 = Date.now(); + const blocks = parseFileMultiLang('slow.go', src, '.go', undefined, 0); + expect(blocks).toEqual([]); // cancellation path taken + expect(Date.now() - t0).toBeLessThan(5000); // cancelled early, not run to completion + }); + + it('deeply nested input does not blow the stack, hang, or corrupt output', () => { + const deep = 'package m\nfunc F() { ' + '{'.repeat(2000) + '}'.repeat(2000) + ' }\n'; + const t0 = Date.now(); + expect(() => invariants(parseFileMultiLang('d.go', deep, '.go'))).not.toThrow(); + expect(Date.now() - t0).toBeLessThan(10000); + }); + + it('frees the parsed tree exactly once per parse that produced one (leak-fix mechanism)', () => { + // Deterministic, machine-independent guard for the WASM Tree/Parser leak: + // assert tree.delete() is actually invoked. (A resource-exhaustion repro is + // hardware/heap-ceiling dependent and green-only on machines with headroom.) + const del = vi.spyOn(Tree.prototype, 'delete'); + try { + del.mockClear(); + parseFileMultiLang('ok.go', 'package m\nfunc F(){ return }\n', '.go'); // success path + expect(del, 'success path frees its tree').toHaveBeenCalledTimes(1); + + del.mockClear(); + parseFileMultiLang('bad.ts', 'class C { m( { { {', '.ts'); // error-recovery still yields a tree + expect(del, 'error-recovery path frees its tree').toHaveBeenCalledTimes(1); + } finally { + del.mockRestore(); + } + }); +}); + +describe('adversarial crawl / ingest', () => { + it('symlink loop under the repo root terminates (crawl does not follow symlink dirs)', () => { + const root = mkdtempSync(join(tmpdir(), 'advcrawl-')); + tmpRoots.push(root); + const sub = join(root, 'sub'); + mkdirSync(sub); + writeFileSync(join(sub, 'a.go'), 'package m\nfunc A() {}\n'); + try { + symlinkSync(root, join(sub, 'loop'), 'dir'); // self-referential loop + } catch { + console.warn('[skip] symlinks unavailable in this environment — loop guard unverified'); + return; + } + const t0 = Date.now(); + const result = ingestRepository(createMultiLangIngestConfig(root)); + expect(Date.now() - t0).toBeLessThan(10000); // did not spin on the loop + expect(result.analysisUnits.some((u) => u.block.name === 'A')).toBe(true); + }); +}); diff --git a/src/__tests__/ts-parse-fallback.test.ts b/src/__tests__/ts-parse-fallback.test.ts new file mode 100644 index 0000000..412a57b --- /dev/null +++ b/src/__tests__/ts-parse-fallback.test.ts @@ -0,0 +1,31 @@ +/** + * Language-scoped fail-open: the Python regex parser must ONLY ever run on `.py`. + * This file deliberately never calls initGrammars(), so the registry is empty and + * every getGrammar() returns undefined — the "grammars failed to load / not yet + * loaded" production scenario. A brace/`class` language routed to the Python regex + * in that state would emit a corrupt block (wrong line span, methods lost); the + * contract is honest absence ([]) instead. + */ +import { describe, it, expect, beforeAll } from 'vitest'; +import { __resetGrammarsForTest } from '../recon/ts-grammars.js'; +import { parseFileMultiLang } from '../recon/ts-parse.js'; +import { parseFile } from '../recon/code-ingest.js'; + +beforeAll(() => __resetGrammarsForTest()); // ensure the registry is empty; do NOT init + +describe('parseFileMultiLang — language-scoped fallback (no grammars loaded)', () => { + it('a Java `class` file returns [] — never a mangled Python-regex block', () => { + // If this routed to parseFile, the Python CLASS_RE would match `class A` and + // emit exactly one (wrong) block; assert it does not. + expect(parseFileMultiLang('A.java', 'class A {\n void run(String c){}\n}\n', '.java')).toEqual([]); + }); + + it('a JS `class` file returns [] with no grammar (would false-match the Python regex)', () => { + expect(parseFileMultiLang('a.js', 'class Svc {\n run(cmd){ exec(cmd); }\n}\n', '.js')).toEqual([]); + }); + + it('.py still routes to the Python regex even with no grammars loaded (byte-identical)', () => { + const src = 'def fetch(url):\n return get(url)\n'; + expect(parseFileMultiLang('a.py', src, '.py')).toEqual(parseFile('a.py', src)); + }); +}); diff --git a/src/__tests__/ts-parse-multilang-rank.test.ts b/src/__tests__/ts-parse-multilang-rank.test.ts new file mode 100644 index 0000000..5120f81 --- /dev/null +++ b/src/__tests__/ts-parse-multilang-rank.test.ts @@ -0,0 +1,58 @@ +import { describe, it, expect, beforeAll } from 'vitest'; +import { mkdtempSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; +import { initGrammars, __resetGrammarsForTest } from '../recon/ts-grammars.js'; +import { ingestRepository, createMultiLangIngestConfig } from '../recon/code-ingest.js'; + +beforeAll(async () => { + __resetGrammarsForTest(); + await initGrammars(); +}); + +describe('multi-language ingest → security ranking (headline regression)', () => { + function repo(): string { + const root = mkdtempSync(join(tmpdir(), 'mlrank-')); + // Go: Handler calls Fetch; Fetch is an SSRF surface (identifier param + outbound). + writeFileSync( + join(root, 'svc.go'), + 'package m\n' + + 'func Handler(url string) error {\n\treturn Fetch(url)\n}\n' + + 'func Fetch(url string) error {\n\treturn http.Get(url)\n}\n' + + 'func Add(a int, b int) int {\n\treturn a + b\n}\n', + ); + // TS: command-exec sink. + writeFileSync(join(root, 'run.ts'), 'export function runCmd(cmd: string) { return exec(cmd); }\n'); + // Java: Runtime.exec sink. + writeFileSync(join(root, 'A.java'), 'class A { void run(String cmd){ Runtime.getRuntime().exec(cmd); } }\n'); + // Python still works alongside (regex path). + writeFileSync(join(root, 'p.py'), 'def load(path):\n return open(path)\n'); + return root; + } + + it('extracts non-.py functions and security-ranks them as attack_surface', () => { + const { analysisUnits } = ingestRepository(createMultiLangIngestConfig(repo())); + const by = (name: string) => analysisUnits.find((u) => u.block.name === name); + + // multi-language extraction reached the pipeline (Go/TS/Java + Python) + for (const name of ['Fetch', 'runCmd', 'run', 'load']) { + expect(by(name), `block ${name} extracted`).toBeDefined(); + } + + // sink-bearing functions are attack_surface, out-ranking a neutral block + for (const name of ['Fetch', 'runCmd', 'run']) { + expect(by(name)!.exposure, `${name} exposure`).toBe('attack_surface'); + } + expect(by('Fetch')!.priority).toBeGreaterThan(by('Add')!.priority); + expect(by('Add')!.exposure).toBe('neutral'); + }); + + it('call graph links non-Python identifiers (cross-language reachability guard)', () => { + const { analysisUnits } = ingestRepository(createMultiLangIngestConfig(repo())); + const handler = analysisUnits.find((u) => u.block.name === 'Handler')!; + const fetch = analysisUnits.find((u) => u.block.name === 'Fetch')!; + // buildCallGraph matched the Go call "Fetch(url)" inside Handler by name. + expect(handler.callees).toContain(fetch.block.id); + expect(fetch.callers).toContain(handler.block.id); + }); +}); diff --git a/src/__tests__/ts-parse.test.ts b/src/__tests__/ts-parse.test.ts new file mode 100644 index 0000000..b6a28c7 --- /dev/null +++ b/src/__tests__/ts-parse.test.ts @@ -0,0 +1,95 @@ +import { describe, it, expect, beforeAll } from 'vitest'; +import type { Parser } from 'web-tree-sitter'; +import { initGrammars, __resetGrammarsForTest } from '../recon/ts-grammars.js'; +import { parseFileMultiLang, nodeToCodeBlock } from '../recon/ts-parse.js'; +import { parseFile } from '../recon/code-ingest.js'; + +beforeAll(async () => { + __resetGrammarsForTest(); // clear any singleton state leaked from another test file + await initGrammars(); +}); + +describe('parseFileMultiLang', () => { + it('extracts a Go function as a CodeBlock', () => { + const src = 'package m\nfunc Fetch(url string) error {\n return http.Get(url)\n}\n'; + const fn = parseFileMultiLang('a.go', src, '.go').find((b) => b.name === 'Fetch'); + expect(fn).toBeDefined(); + expect(fn!.params).toEqual(['url']); + expect(fn!.kind).toBe('function'); + expect(fn!.lineStart).toBe(2); + expect(fn!.body).toContain('http.Get(url)'); + expect(fn!.id).toBe('a.go::Fetch@2'); + expect(fn!.decorators).toEqual([]); + }); + + it('extracts a TS class + method', () => { + const src = 'class Svc {\n run(cmd: string) { exec(cmd); }\n}\n'; + const blocks = parseFileMultiLang('s.ts', src, '.ts'); + expect(blocks.find((b) => b.name === 'Svc')?.kind).toBe('class'); + const m = blocks.find((b) => b.name === 'run'); + expect(m?.kind).toBe('method'); + expect(m?.params).toEqual(['cmd']); + }); + + it('extracts a Java constructor as a method', () => { + const blocks = parseFileMultiLang('A.java', 'class A {\n A(int x){}\n}\n', '.java'); + expect(blocks.find((b) => b.name === 'A' && b.kind === 'method')).toBeDefined(); + expect(blocks.find((b) => b.name === 'A' && b.kind === 'class')).toBeDefined(); + }); + + it('extracts C functions with pointer return types', () => { + const blocks = parseFileMultiLang( + 'fetch.c', + 'const char *fetch_url(const char *url) { return url; }\n', + '.c', + ); + expect(blocks).toEqual([ + expect.objectContaining({ + name: 'fetch_url', + kind: 'function', + params: ['url'], + }), + ]); + }); + + it('.py routes to the regex parser (byte-identical)', () => { + const src = 'def fetch(url):\n return get(url)\n'; + expect(parseFileMultiLang('a.py', src, '.py')).toEqual(parseFile('a.py', src)); + }); + + it('unsupported ext returns [] (no throw, no Python-regex on non-.py)', () => { + expect(parseFileMultiLang('a.zzz', 'func F(){}', '.zzz')).toEqual([]); + }); + + it('fail-open on parse timeout (parse returns null) → []', () => { + const nullParser = () => + ({ setLanguage() {}, parse: () => null, delete() {} }) as unknown as Parser; + expect(parseFileMultiLang('a.go', 'package m\nfunc F(){}\n', '.go', nullParser)).toEqual([]); + }); + + it('fail-open on parser error (parse throws) → []', () => { + const throwParser = () => + ({ + setLanguage() {}, + parse() { + throw new Error('boom'); + }, + delete() {}, + }) as unknown as Parser; + expect(parseFileMultiLang('a.go', 'package m\nfunc F(){}\n', '.go', throwParser)).toEqual([]); + }); +}); + +describe('nodeToCodeBlock', () => { + it('maps a class node (no params) to kind=class with empty params', () => { + const fake = { + type: 'class_declaration', + startPosition: { row: 4 }, + endPosition: { row: 9 }, + text: 'class C {}', + } as never; + const name = { text: 'C' } as never; + const cb = nodeToCodeBlock(fake, name, undefined, 'x.ts', 'ts'); + expect(cb).toMatchObject({ name: 'C', kind: 'class', lineStart: 5, lineEnd: 10, params: [], id: 'x.ts::C@5' }); + }); +}); diff --git a/src/__tests__/whitebox-multilang.test.ts b/src/__tests__/whitebox-multilang.test.ts new file mode 100644 index 0000000..b87d7a4 --- /dev/null +++ b/src/__tests__/whitebox-multilang.test.ts @@ -0,0 +1,59 @@ +import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest'; +import { mkdtempSync, mkdirSync, writeFileSync, realpathSync } from 'fs'; +import { join } from 'path'; +import { tmpdir } from 'os'; +import { initGrammars, __resetGrammarsForTest } from '../recon/ts-grammars.js'; +import { ingestRepoToSourceContext, runWhiteboxAnalysis } from '../recon/whitebox.js'; +import { DecompositionOrchestrator } from '../orchestration/index.js'; + +let base: string; +let repo: string; +let prevRoot: string | undefined; + +beforeAll(async () => { + __resetGrammarsForTest(); + await initGrammars(); + base = realpathSync(mkdtempSync(join(tmpdir(), 't3mp3st-wbml-'))); + prevRoot = process.env.T3MP3ST_REPO_ROOT; + process.env.T3MP3ST_REPO_ROOT = base; + repo = join(base, 'repo'); + mkdirSync(repo); + writeFileSync(join(repo, 'svc.go'), 'package m\nfunc Fetch(url string) error {\n\treturn http.Get(url)\n}\n'); + writeFileSync(join(repo, 'run.ts'), 'export function runCmd(cmd: string) {\n\treturn exec(cmd);\n}\n'); + writeFileSync(join(repo, 'p.py'), 'def load(path):\n return open(path)\n'); +}); + +afterAll(() => { + if (prevRoot === undefined) delete process.env.T3MP3ST_REPO_ROOT; + else process.env.T3MP3ST_REPO_ROOT = prevRoot; +}); + +describe('ingestRepoToSourceContext (production white-box entry) — multi-language', () => { + it('non-.py source reaches the packed sourceContext (feature is not a no-op in prod)', () => { + const { sourceContext } = ingestRepoToSourceContext(repo); + // Go and TS blocks — proof the multilang config swap actually took effect on the + // real production path, not just via a direct createMultiLangIngestConfig call. + expect(sourceContext).toContain('Fetch'); + expect(sourceContext).toContain('runCmd'); + // Python still ingested alongside. + expect(sourceContext).toContain('load'); + }); + + it('runWhiteboxAnalysis (2nd production caller) also feeds multilang source to the orchestrator', async () => { + // Guards the second createMultiLangIngestConfig swap site — a one-token + // revert to createPythonIngestConfig would drop non-.py content silently. + // Stub orch.run so no real LLM calls fire; capture the sourceContext it gets. + const runSpy = vi + .spyOn(DecompositionOrchestrator.prototype, 'run') + .mockResolvedValue({} as never); + try { + await runWhiteboxAnalysis({ repoPath: repo, objective: 'x' }); + expect(runSpy).toHaveBeenCalledTimes(1); + const sourceContext = runSpy.mock.calls[0][1] as string; + expect(sourceContext).toContain('Fetch'); // Go reached the orchestrator + expect(sourceContext).toContain('runCmd'); // TS reached the orchestrator + } finally { + runSpy.mockRestore(); + } + }); +}); diff --git a/src/recon/code-ingest.ts b/src/recon/code-ingest.ts index 418645d..0623240 100644 --- a/src/recon/code-ingest.ts +++ b/src/recon/code-ingest.ts @@ -37,7 +37,8 @@ */ import { readdirSync, readFileSync, statSync } from 'fs'; -import { join, sep } from 'path'; +import { join, sep, extname } from 'path'; +import { parseFileMultiLang } from './ts-parse.js'; import { estimateTokens, type SourceFile, @@ -169,14 +170,23 @@ const ENTRY_NAME_RES: RegExp[] = [ const SECURITY_CONTROL_NAME_RE = /valid|verif|auth|authoriz|authentic|permission|sanitiz|escap|csrf|check_|require_|is_allowed/i; -// Dangerous sinks that make a block part of the attack surface. -const DANGEROUS_SINK_RE = - /requests\.(get|post|put)|urllib|urlopen|httpx|socket\.|subprocess|os\.system|\beval\(|\bexec\(|pickle\.loads|yaml\.load|cursor\.execute|\.raw\(|open\(/; +// Dangerous sinks that make a block part of the attack surface. Python patterns +// plus cross-language sinks (Java/Go/C/JS) for the multi-language ingest. +// Bare-call patterns (system/popen/exec-family/fetch) use a `(?= maxTotalBytes) { truncated = true; console.warn(`[code-ingest] ingest reached maxTotalBytes ceiling (${maxTotalBytes}) after ${totalBytes} bytes; remaining files skipped — raise IngestConfig.maxTotalBytes to analyze more.`); @@ -841,6 +854,30 @@ export function createPythonIngestConfig(repoRoot: string): IngestConfig { }; } +/** + * Multi-language ingest config: same ceilings as the Python config, but crawls + * the languages the tree-sitter extractor supports. Non-source files are + * excluded by `includeExts`. Used by the white-box analysis entrypoints. + * + * SECURITY: `maxFileBytes` (1 MB) is a security control here, not just a perf + * knob — it caps the peak WASM linear-memory a single tree-sitter parse can + * demand. A dense/obfuscated file large enough to approach web-tree-sitter's + * 2 GB heap ceiling makes the native allocator abort() (caught + fail-open, but + * the WASM high-water-mark cannot be reclaimed for the process's life). The 1 MB + * cap keeps that unreachable (worst measured blow-up ~370x → ~5-6x margin). Do + * NOT raise it without re-testing that margin. + */ +export function createMultiLangIngestConfig(repoRoot: string): IngestConfig { + return { + repoRoot, + includeExts: ['.py', '.js', '.ts', '.tsx', '.go', '.java', '.c', '.cpp'], + excludeGlobs: [...DEFAULT_EXCLUDES], + maxFileBytes: 1_000_000, + maxFiles: 50_000, + maxTotalBytes: 1_000_000_000, + }; +} + // ============================================================================= // COMPOSITION WITH context-pack — rendering & security-ordered packing // ============================================================================= diff --git a/src/recon/param-split.ts b/src/recon/param-split.ts new file mode 100644 index 0000000..4b56614 --- /dev/null +++ b/src/recon/param-split.ts @@ -0,0 +1,77 @@ +/** + * Language-aware parameter-name extractor for the multi-language ingest. + * + * tree-sitter hands us a parameter-list as raw source text; different languages + * order name and type differently (Python/Go/TS are name-first, Java/C/C++ are + * type-first). This returns the bare identifier names so the security-ranking + * layer's `RISKY_PARAM_RE` can match on them exactly as it did for Python. + */ + +/** Split on top-level commas only — commas inside (), [], {}, <> are nested. */ +function splitTopLevel(s: string): string[] { + const parts: string[] = []; + let depth = 0; + let cur = ''; + for (const ch of s) { + if (ch === '(' || ch === '[' || ch === '{' || ch === '<') depth++; + else if (ch === ')' || ch === ']' || ch === '}' || ch === '>') depth = Math.max(0, depth - 1); + if (ch === ',' && depth === 0) { + parts.push(cur); + cur = ''; + } else { + cur += ch; + } + } + if (cur.trim()) parts.push(cur); + return parts; +} + +/** First identifier-shaped token in a string, or null. */ +function firstId(s: string): string | null { + const m = s.match(/[A-Za-z_$][A-Za-z0-9_$]*/); + return m ? m[0] : null; +} + +function paramName(partRaw: string, lang: string): string | null { + const part = partRaw.trim(); + if (!part) return null; + + if (lang === 'py') { + if (part.startsWith('*')) return null; // *args / **kwargs / bare * + const p = part.split('=')[0].split(':')[0].trim(); + if (!p || p === '/' || p === 'self' || p === 'cls') return null; + return firstId(p); + } + + if (lang === 'ts' || lang === 'js') { + // name-first: name is the token before ':' or '=', minus a trailing '?'. + let p = part.replace(/^\.\.\./, '').split(':')[0].split('=')[0].trim(); + p = p.replace(/\?$/, '').trim(); + const toks = p.split(/\s+/).filter(Boolean); // drop TS modifiers (public/readonly) + return firstId(toks[toks.length - 1] ?? ''); + } + + if (lang === 'go') { + // name-first: first token of the group ("a, b int" already split → "a", "b int"). + return firstId(part); + } + + // java / c / c++ / unknown: type-first → name is the last token; strip * & []. + const p = part.split('=')[0].trim(); + const toks = p.split(/\s+/).filter(Boolean); + const last = (toks[toks.length - 1] ?? '').replace(/^[*&]+/, '').replace(/\[\]$/, ''); + return firstId(last); +} + +/** Bare parameter identifiers from a raw parameter list, per language dialect. */ +export function splitParamList(raw: string, lang: string): string[] { + let s = raw.trim(); + if (s.startsWith('(') && s.endsWith(')')) s = s.slice(1, -1).trim(); + if (!s) return []; + const out: string[] = []; + for (const part of splitTopLevel(s)) { + const name = paramName(part, lang); + if (name) out.push(name); + } + return out; +} diff --git a/src/recon/ts-grammars.ts b/src/recon/ts-grammars.ts new file mode 100644 index 0000000..0d565e9 --- /dev/null +++ b/src/recon/ts-grammars.ts @@ -0,0 +1,149 @@ +/** + * Multi-language grammar registry for white-box ingest. + * + * Wraps web-tree-sitter (WASM) so `parseFileMultiLang` can extract CodeBlocks + * from ~8 languages, replacing the Python-only regex parser. Grammars are + * loaded ONCE at bootstrap (`initGrammars`) — `Parser.init()`/`Language.load()` + * are async, but `parser.parse()` is sync, so the ingest path stays sync. + * + * Fail-open by design: if init throws (bad/missing wasm, version skew), the + * registry is left empty. Python still routes to `parseFile`; non-Python files + * yield no blocks — ingest never crashes a mission. + */ +import { Parser, Language, Query } from 'web-tree-sitter'; +import { createRequire } from 'module'; + +const require = createRequire(import.meta.url); + +/** A loaded grammar: the tree-sitter Language plus the compiled def-query. */ +export interface GrammarEntry { + language: Language; + query: Query; + /** Language id, for param-splitting dialect (py/js/ts/go/java/c/cpp). */ + lang: string; +} + +/** + * Per-extension spec: which prebuilt wasm (from tree-sitter-wasms) and the + * tree-sitter query capturing function/method/class definitions. Every pattern + * binds `@name` (the identifier) and `@def` (the whole definition node); + * `@params` is bound where the grammar exposes a parameter list. + */ +const SPECS: Record = { + '.py': { + wasm: 'tree-sitter-python', lang: 'py', + query: ` + (function_definition name: (identifier) @name parameters: (parameters) @params) @def + (class_definition name: (identifier) @name) @def + `, + }, + '.js': { + wasm: 'tree-sitter-javascript', lang: 'js', + query: ` + (function_declaration name: (identifier) @name parameters: (formal_parameters) @params) @def + (method_definition name: (property_identifier) @name parameters: (formal_parameters) @params) @def + (class_declaration name: (identifier) @name) @def + `, + }, + '.ts': { + wasm: 'tree-sitter-typescript', lang: 'ts', + query: ` + (function_declaration name: (identifier) @name parameters: (formal_parameters) @params) @def + (method_definition name: (property_identifier) @name parameters: (formal_parameters) @params) @def + (class_declaration name: (type_identifier) @name) @def + `, + }, + '.tsx': { + wasm: 'tree-sitter-tsx', lang: 'ts', + query: ` + (function_declaration name: (identifier) @name parameters: (formal_parameters) @params) @def + (method_definition name: (property_identifier) @name parameters: (formal_parameters) @params) @def + (class_declaration name: (type_identifier) @name) @def + `, + }, + '.go': { + wasm: 'tree-sitter-go', lang: 'go', + query: ` + (function_declaration name: (identifier) @name parameters: (parameter_list) @params) @def + (method_declaration name: (field_identifier) @name parameters: (parameter_list) @params) @def + `, + }, + '.java': { + wasm: 'tree-sitter-java', lang: 'java', + query: ` + (method_declaration name: (identifier) @name parameters: (formal_parameters) @params) @def + (constructor_declaration name: (identifier) @name parameters: (formal_parameters) @params) @def + (class_declaration name: (identifier) @name) @def + `, + }, + '.c': { + wasm: 'tree-sitter-c', lang: 'c', + query: ` + (function_definition declarator: (function_declarator declarator: (identifier) @name parameters: (parameter_list) @params)) @def + (function_definition declarator: (pointer_declarator declarator: (function_declarator declarator: (identifier) @name parameters: (parameter_list) @params))) @def + `, + }, + '.cpp': { + wasm: 'tree-sitter-cpp', lang: 'cpp', + query: ` + (function_definition declarator: (function_declarator declarator: (identifier) @name parameters: (parameter_list) @params)) @def + `, + }, +}; + +const registry = new Map(); +let initialized = false; + +/** Extensions (with leading dot) the registry can parse. */ +export function supportedExts(): string[] { + return Object.keys(SPECS); +} + +/** + * Load grammars once. Idempotent. Fail-open: any error leaves the (possibly + * partial) registry in place and marks init done — callers extract nothing for + * exts that never loaded (non-.py fail-open is empty, not the Python regex). + * `wasmResolve` lets a test force a bad path to exercise the failure branch. + */ +export async function initGrammars( + exts: string[] = supportedExts(), + wasmResolve: (name: string) => string = (name) => require.resolve(`tree-sitter-wasms/out/${name}.wasm`), + init: () => Promise = () => Parser.init(), +): Promise { + if (initialized) return; + try { + await init(); + for (const ext of exts) { + const spec = SPECS[ext]; + if (!spec) continue; + try { + const language = await Language.load(wasmResolve(spec.wasm)); + const query = new Query(language, spec.query); + registry.set(ext, { language, query, lang: spec.lang }); + } catch { + // per-ext fail-open: skip this grammar, keep the rest + } + } + } catch { + // global fail-open: registry stays empty → non-.py ingest extracts nothing. + // Log once so an operator scanning a non-Python repo sees the degradation + // (the `initialized` guard means this never retries for the process). + console.warn( + '[ts-grammars] grammar init failed; multi-language ingest disabled for this process ' + + '(non-Python files will yield no blocks). Python ingest is unaffected.', + ); + } finally { + initialized = true; + } +} + +/** A loaded grammar for `ext`, or undefined (→ caller returns [] for non-.py; .py uses parseFile). */ +export function getGrammar(ext: string): GrammarEntry | undefined { + return registry.get(ext); +} + +/** Test-only: reset init state + registry so a fresh `initGrammars` runs. */ +export function __resetGrammarsForTest(): void { + registry.clear(); + initialized = false; +} diff --git a/src/recon/ts-parse.ts b/src/recon/ts-parse.ts new file mode 100644 index 0000000..3b093e8 --- /dev/null +++ b/src/recon/ts-parse.ts @@ -0,0 +1,133 @@ +/** + * Multi-language CodeBlock extractor — the tree-sitter replacement for the + * Python-only `parseFile`. Emits the SAME `CodeBlock` shape so the downstream + * security-ranking pipeline (classify/prioritize/reachability) is reused + * verbatim. + * + * Fail-open on every path: `.py` uses the Python regex parser; any other + * language with a missing grammar, parse timeout, or parser error yields `[]` + * (honest absence) — never the Python regex, which would mis-parse brace/`class` + * syntax into corrupt blocks. A hostile input cannot crash or hang a mission — + * the parse is wall-clock time-bounded and every error is caught. + */ +import { Parser, type Node } from 'web-tree-sitter'; +import { getGrammar } from './ts-grammars.js'; +import { splitParamList } from './param-split.js'; +// Deliberate circular import with code-ingest.ts (it imports parseFileMultiLang). +// Safe because both cross-references are used only inside function bodies, never +// at module-evaluation time — do NOT hoist parseFile/CodeBlock to top-level use. +import { parseFile, type CodeBlock } from './code-ingest.js'; + +/** Hard per-file parse bound (wall-clock ms). A pathological input hits this and fail-opens. */ +const PARSE_BUDGET_MS = 5000; + +let sharedParser: Parser | null = null; +function getParser(): Parser { + if (!sharedParser) sharedParser = new Parser(); + return sharedParser; +} +/** + * Drop the parser after a timeout/error so the next parse starts clean. MUST + * free the native (WASM-backed) parser first: web-tree-sitter has no GC-based + * finalizer, so a dropped-but-not-deleted parser leaks its linear-memory + * allocation. Left unfreed, repeated failures exhaust the shared WASM heap and + * permanently wedge every subsequent parse (they abort, get caught, and yield + * [] forever). Frees the module singleton, which is correct because production + * always uses the default `getParser` (`makeParser` is a test-only seam). + */ +function resetParser(): void { + sharedParser?.delete(); + sharedParser = null; +} + +function kindOf(nodeType: string): CodeBlock['kind'] { + if (nodeType.includes('class')) return 'class'; + if (nodeType.includes('method') || nodeType.includes('constructor')) return 'method'; + return 'function'; +} + +/** Map a captured definition node (+ its name/params captures) to a CodeBlock. */ +export function nodeToCodeBlock( + def: Node, + nameNode: Node, + paramsNode: Node | undefined, + path: string, + lang: string, +): CodeBlock { + const name = nameNode.text; + const lineStart = def.startPosition.row + 1; // tree-sitter rows are 0-indexed + const lineEnd = def.endPosition.row + 1; + return { + id: `${path}::${name}@${lineStart}`, + path, + name, + kind: kindOf(def.type), + lineStart, + lineEnd, + params: paramsNode ? splitParamList(paramsNode.text, lang) : [], + decorators: [], + body: def.text, + }; +} + +/** + * Extract CodeBlocks from one file. `.py` keeps the regex parser (benchmark + * stability); other languages use tree-sitter. Fail-open is language-scoped: the + * Python regex parser only ever runs on `.py`. For any other language a missing + * grammar, parse timeout, or parser error returns `[]` (nothing extracted) — + * never the Python regex, which would mis-parse brace/`class` syntax and emit + * corrupt blocks (wrong line spans, lost methods) rather than honest absence. + * + * Time-bounding note: only the `parse()` call is wall-clock bounded (via the + * `budgetMs` arg, injectable so a test can drive the real cancellation path). + * When called through `ingestRepository` the content is capped at `maxFileBytes` + * (1 MB) upstream, which also bounds the near-linear match/conversion loop and + * the WASM peak-memory demand (see createMultiLangIngestConfig). + * + * ponytail: unify Python onto tree-sitter only if CVE-Zero is re-baselined. + */ +export function parseFileMultiLang( + path: string, + content: string, + ext: string, + makeParser: () => Parser = getParser, + budgetMs: number = PARSE_BUDGET_MS, +): CodeBlock[] { + if (ext === '.py') return parseFile(path, content); + const g = getGrammar(ext); + if (!g) return []; // unsupported / grammar not loaded — honest absence, not Python-regex garbage + let tree: ReturnType | undefined; + try { + const p = makeParser(); + p.setLanguage(g.language); + // Bound parse wall-time: progressCallback returning true cancels the parse + // (→ parse returns null). setTimeoutMicros is deprecated/broken in 0.25. + const start = Date.now(); + tree = p.parse(content, undefined, { + progressCallback: () => Date.now() - start > budgetMs, + }); + if (!tree) { + resetParser(); // parse cancelled (budget exceeded) + return []; + } + const blocks: CodeBlock[] = []; + for (const match of g.query.matches(tree.rootNode)) { + const caps: Record = {}; + for (const c of match.captures) caps[c.name] = c.node; + blocks.push(nodeToCodeBlock(caps.def, caps.name, caps.params, path, g.lang)); + } + return blocks; + } catch { + // Fail-open. A native WASM OOM abort (huge/dense input near the 2GB heap + // ceiling) surfaces here too; it is caught, not fatal, but leaves the shared + // heap pinned high — the maxFileBytes cap keeps that unreachable in prod. + resetParser(); + return []; // any parser error → fail-open (empty, not Python-regex garbage) + } finally { + // Free the native WASM-backed tree. nodeToCodeBlock already copied every + // field out as a plain JS string, so nothing dangles after delete(). + // Without this the linear-memory heap grows unbounded and eventually + // poisons the shared parser for the rest of the process. + tree?.delete(); + } +} diff --git a/src/recon/whitebox.ts b/src/recon/whitebox.ts index 547dd7f..587cdc6 100644 --- a/src/recon/whitebox.ts +++ b/src/recon/whitebox.ts @@ -8,16 +8,36 @@ * itself makes (worker + orchestrator models). It is otherwise pure of network * side effects. * - * SCOPE CAVEAT (inherited from code-ingest): the ingest layer is a PYTHON-ONLY, - * regex "AST-lite" prototype — see recon/code-ingest.ts. It selects which Python - * blocks to analyze first by security exposure; anything non-`.py` or imported - * from outside the scan scope is invisible. Do NOT describe this as "any repo". + * SCOPE CAVEAT (inherited from code-ingest): the ingest layer extracts blocks via + * web-tree-sitter for Python/JS/TS/Go/Java/C/C++ (Python uses the regex "AST-lite" + * path; other languages fail-open to [] — never the Python regex — on a grammar + * miss or parse error) — see recon/code-ingest.ts + recon/ts-parse.ts. It selects which + * blocks to analyze first by security exposure; a language without a bundled grammar, + * or code imported from outside the scan scope, is invisible. Do NOT describe this as + * "any repo" or "any language". + * + * KNOWN v1 EXTRACTION LIMITS (fail-safe — these shapes are silently NOT extracted or + * under-linked, never mis-extracted; all strictly better than the prior Python-only + * ingest, and tracked as follow-ups): + * - JS/TS: only `function`/`method`/`class` declarations are captured. Arrow-function + * and function-expression definitions (`const f = () => …`) are not — idiomatic in + * modern TS, so their sinks may go unranked. + * - C++: only free functions are captured. In-class and out-of-line (`Class::method`) + * member methods are not — a materially narrower C++ story than "full support". + * - Entry-point elevation for non-Python code relies on name heuristics only; no + * language emits `decorators`, so annotation-based routes (Spring `@GetMapping`, + * TS `@Get()`, Express `app.get(...)`) are not elevated to `exposed_externally`. + * - Call-graph/reachability: buildCallGraph strips a block's signature line before + * scanning for callees (kept unchanged for Python-benchmark stability), so a + * SINGLE-LINE non-Python definition (`func F(u) { return G(u) }`) misses the call + * on that line — the block is still extracted and sink-classified, only the edge + * is lost. */ import { config } from '../config/index.js'; import { ingestRepository, - createPythonIngestConfig, + createMultiLangIngestConfig, packAnalysisUnits, } from './code-ingest.js'; import { DecompositionOrchestrator } from '../orchestration/index.js'; @@ -218,7 +238,7 @@ export function ingestRepoToSourceContext( ): RepoSourceContext { const source = resolveRepoSourceForAnalysis(repoPath); try { - const result = ingestRepository(createPythonIngestConfig(source.repoPath)); + const result = ingestRepository(createMultiLangIngestConfig(source.repoPath)); const packed = packAnalysisUnits(result.analysisUnits, tokenBudget); return { sourceContext: packed.text, @@ -302,21 +322,21 @@ export async function runWhiteboxAnalysis( // 1) ingest source, security-rank the blocks try { - const result = ingestRepository(createPythonIngestConfig(source.repoPath)); + const result = ingestRepository(createMultiLangIngestConfig(source.repoPath)); // 2) pack into a generous source view (orchestrator re-packs per-worker itself) const packed = packAnalysisUnits(result.analysisUnits, DEFAULT_ANALYSIS_SOURCE_BUDGET); const sourceContext = packed.text; // Fail loud ONLY when there is genuinely NO source to analyze: 0 ingestable source files - // (non-Python or empty repo). A valid Python repo that has files but no def/class (a - // module-level-only script) yields 0 blocks -> empty packed text; that used to run the - // orchestrator, so we do NOT regress it into a 500 — gate the throw on the file count, not - // on empty packed text, and let a files>0 repo proceed as before. + // (no supported-language files, or empty repo). A valid repo that has files but no + // def/class/func (a module-level-only script) yields 0 blocks -> empty packed text; that + // used to run the orchestrator, so we do NOT regress it into a 500 — gate the throw on the + // file count, not on empty packed text, and let a files>0 repo proceed as before. if (result.stats.files === 0) { throw new Error( 'white-box analysis has no analyzable source: the repo has 0 ingestable source files ' + - '(non-Python, or empty). Point repoPath at a Python codebase.', + '(no supported-language source, or empty). Point repoPath at a code repository.', ); } diff --git a/src/server.ts b/src/server.ts index 838b273..8721c72 100755 --- a/src/server.ts +++ b/src/server.ts @@ -32,6 +32,7 @@ import { createTargetFromUrl, createTargetFromIP } from './target/index.js'; import type { OperatorArchetype, LLMProvider } from './types/index.js'; import { listOperatorPrompts, setOperatorOverride, resetOperatorOverride, type OperatorOverride } from './operators/index.js'; import { ingestRepoToSourceContext, runWhiteboxAnalysis, resolveRepoSourceForAnalysis, RepoCloneError, RepoPathError } from './recon/whitebox.js'; +import { initGrammars } from './recon/ts-grammars.js'; import { redactCredential } from './evidence/index.js'; const execFileAsync = promisify(execFile); @@ -7789,6 +7790,12 @@ async function startServer() { await loadPersistedState(); llm = await initLLM(); + // Load multi-language grammars once, before any white-box ingest request. + // initGrammars is internally fail-open (never rejects): on failure the + // registry stays empty, Python still ingests via the regex parser, and other + // languages yield no blocks (it logs a warning). + await initGrammars(); + // Graceful-shutdown flush: on SIGTERM/SIGINT (Ctrl-C, docker stop, systemctl restart) write // any pending debounced snapshot before exiting, so the persistState debounce can't lose the // last <1s of state. Registered here (the server-start path) so it never affects test imports. diff --git a/vitest.config.ts b/vitest.config.ts new file mode 100644 index 0000000..c94219c --- /dev/null +++ b/vitest.config.ts @@ -0,0 +1,32 @@ +import { defineConfig } from 'vitest/config'; + +// Coverage-only configuration. Test discovery, environment, and everything else +// stay on Vitest defaults — this file exists solely to enforce the multi-language +// ingest quality gate: 100% coverage (statements/branches/functions/lines) on the +// three fully-new source files this feature adds. +// +// Edited files (code-ingest.ts, whitebox.ts, server.ts) are NOT in `include`: a +// whole-file 100% threshold is unachievable on large pre-existing files, and the +// gate for them is "new branches covered", verified by the multilang/regression +// suites rather than a file-level threshold. The single server.ts bootstrap line +// (`await initGrammars()`) is the documented carve-out — smoke-covered, and +// initGrammars itself is unit-tested to 100% in ts-grammars.test.ts. +export default defineConfig({ + test: { + coverage: { + provider: 'v8', + include: [ + 'src/recon/ts-grammars.ts', + 'src/recon/param-split.ts', + 'src/recon/ts-parse.ts', + ], + thresholds: { + perFile: true, + statements: 100, + branches: 100, + functions: 100, + lines: 100, + }, + }, + }, +});