Lyubomir BozhinovandJoseph Magly
151d3555c6
* chore(ingest): pin web-tree-sitter@0.25.10 + tree-sitter-wasms@0.1.13; add multilang-ingest design docs * feat(ingest): multi-language grammar registry (web-tree-sitter, 8 langs, fail-open); 100% cov * feat(ingest): language-aware param-name splitter (py/go/ts/java/c); 100% cov * feat(ingest): tree-sitter CodeBlock extractor with wall-clock parse budget + fail-open * feat(ingest): cross-lang sinks + multilang config + wire tree-sitter into ingestRepository (sync) * feat(ingest): repoint white-box production callers to multilang config + prod regression test * server-init-wiring: load multi-language grammars once at bootstrap initGrammars() runs after LLM init, before app.listen. Internally fail-open (never rejects): on failure the grammar registry stays empty and ingest falls back to the Python regex parser. * adversarial-tests: hostile-input hardening for untrusted target source Malformed per-language source, wrong-language content, binary/non-UTF8, empty, unicode identifiers, 20k-func large file (wall-clock budget), deeply nested input, and a symlink-loop crawl — all must fail-open, never throw or spin. Drives parseFileMultiLang + ingestRepository/crawl. * readme-status + docs: flip white-box status to multi-language ingest README source-code rows now state web-tree-sitter multi-language ingest (fail-open to Python regex). whitebox.ts scope caveat + no-source error message updated off the stale Python-only wording. * coverage-setup: enforce 100% coverage gate on the three new ingest files vitest.config.ts adds a coverage-only block (v8, perFile thresholds=100) scoped to ts-grammars.ts, param-split.ts, ts-parse.ts. Test discovery and env stay on defaults. Edited files gate on new-branch coverage via the regression suites; the server.ts initGrammars bootstrap line is the documented smoke-covered carve-out. * review: fix WASM heap leak, language-scoped fail-open, sink recall/precision Adversarial security + strict code review (two independent subagents) returned one blocking bug and several correctness findings; remediated: BLOCKING #1 — native WASM memory leak: web-tree-sitter Tree/Parser objects were never .delete()'d. Both reviewers reproduced a process-global wedge: after ~7 large parses the shared WASM heap exhausts, every subsequent parse aborts (caught by fail-open), and non-Python ingest silently returns [] forever. Fix: free the tree in a finally; delete the parser in resetParser. Regression test parses ~8MB through the real shared parser 12x. regex, which mis-parses brace/class syntax into corrupt blocks (wrong line spans, lost methods). Now non-.py fail-open returns [] (honest absence); the Python regex only ever runs on .py. Isolated ts-parse-fallback test. http(s).request, axios(config), C exec-family); negative lookbehind stops bare system(/popen(/fetch( from matching Python receivers (os.system etc. still matched, corpus rankings unchanged). Recall+precision tests added. operator sees non-Python degradation. (one-line call-graph edges, no decorators, C++ member methods). * review round 2: fix sink-regex regressions, real timeout test, deterministic leak guard Third review round (correctness + security re-attack + test-quality) against the round-1 fixes surfaced issues in the fixes themselves; remediated: - Sink-regex regressions I introduced: (1) my fetch lookbehind dropped window.fetch/this.fetch (top JS SSRF idiom) — reverted to \bfetch\(; (2) bare .Get/.Post/.Do flagged db.Get/cache.Get/queue.Do as SSRF — receiver-qualified to [Cc]lient. Recall+precision tests cover both; regex is ReDoS-free (timed). - Placebo leak test: the 12x function-dense loop did NOT wedge the parser on machines with heap headroom (proven: passed with tree.delete removed). Replaced with a deterministic vi.spyOn(Tree.prototype,'delete') assertion, verified RED without the fix. - Real (non-mocked) parse-timeout test: parseFileMultiLang gains an injectable budgetMs; a 0ms budget drives the real progressCallback cancellation to []. - runWhiteboxAnalysis swap now has coverage (spy on orchestrator.run) so the 2nd production caller can't silently revert to the Python-only config. - Un-vacuoused weak tests: unicode now asserts the extracted block; nested/ wrong-language wrapped in invariants() + timing bounds. - CI now runs npm run test:coverage (the 100% gate was defined but unenforced). - Security: documented maxFileBytes as the control bounding a single parse's peak WASM memory below the 2GB heap-abort ceiling. - Stale 'fail-open to Python regex' docs fixed (ts-parse, whitebox, server, adversarial); circular-import note added; temp-dir cleanup; honest coverage wording in spec + config. * docs(ingest): align fallback-contract text with the shipped language-scoped fail-open Review feedback (#75, @jmagly): the design docs + one ts-grammars comment still described the pre-implementation 'fall back to the Python regex' model for non-Python files. The shipped contract is language-scoped: .py uses parseFile; any non-.py file with a missing/unsupported/failed grammar or a parse timeout returns [] (never the Python regex, which would mis-parse brace/class syntax into corrupt blocks). Updated every stale reference in multilang-ingest-spec.md, multilang-ingest-plan.md, and ts-grammars.ts so the docs match the security boundary. Docs/comment only — no code change. * fix(test): make the ReDoS wall-clock guard robust under coverage instrumentation Review feedback (#75, @jmagly): the CI step this PR adds (npm run test:coverage = vitest run --coverage) runs the full suite under V8 coverage, and that overhead pushes the pathological-brace DoS timing guard in local-agent-tool-calling.test.ts:69 over its <300ms bound (320 < 300 fails on CI). Raise the bound to 1000ms — still 2x below the ~2100ms quadratic-ReDoS regression it guards against, so it fails hard if the regression returns, but no longer false-fails under coverage/slow CI. Chose the reviewer's option (a) (robust bound) over scoping test:coverage, so the gate stays a simple full-suite run and no test-file enumeration can rot. Verified: npm run test:coverage is green and the 3 new files stay at 100%. * fix(ingest): cover pointer-returning C functions --------- Co-authored-by: Joseph Magly <1159087+jmagly@users.noreply.github.com>
125 lines
14 KiB
Markdown
125 lines
14 KiB
Markdown
# Multi-language white-box ingest for T3MP3ST
|
||
|
||
**Status:** Draft design (for an upstream PR to `elder-plinius/T3MP3ST`)
|
||
**Date:** 2026-07-10
|
||
**Scope:** Replace the Python-only ingest parser with a portable multi-language parser, reusing T3MP3ST's existing security-ranking pipeline unchanged.
|
||
|
||
---
|
||
|
||
## 1. Problem
|
||
|
||
T3MP3ST markets white-box source hunting as a proven strength ("Code: ✅ Proven — CVE-Zero 8/10 across 7 languages"), but the ingest **engine** only reads Python. `src/recon/code-ingest.ts:6` says so itself:
|
||
|
||
> "This is a PYTHON-ONLY, regex-based 'AST-lite' PROTOTYPE … The real version is tree-sitter (or a language server) + multi-language."
|
||
|
||
The README status table flags it: **Code → ⚠️ Python-only ingest**. Point the white-box flow at a Go/Java/C/TS repo and `crawl` keeps only `.py`; every non-Python file is invisible to the hunt.
|
||
|
||
## 2. Goal / Non-goals
|
||
|
||
**Goal.** White-box ingest parses many languages, feeding T3MP3ST's *existing* security-ranking so multi-language repos get the same attack-surface-ordered analysis Python repos get today.
|
||
|
||
**Non-goals.**
|
||
- Not changing the security model (`classify`/`prioritize`/`reachability`/`context-pack`) — it is reused verbatim.
|
||
- Not adding an external binary, native-compiled dependency, or new API key.
|
||
- Not building cross-language taint/call-graph analysis in this PR (see §9, phase 2).
|
||
|
||
## 3. Key insight — the seam is one stage
|
||
|
||
The ingest is a pipeline (`whitebox.ts`): `code-ingest (SELECT) → context-pack (BUDGET) → orchestrator (REASON)`. Within SELECT, only **one** stage is language-bound:
|
||
|
||
- `parseFile()` (`code-ingest.ts:409`) — Python `def`/`class` regex → `CodeBlock[]`. **Language-bound.**
|
||
- Everything downstream operates on `CodeBlock {name, params, body, line, …}` via text regexes over the block body/name/params:
|
||
- `buildCallGraph`, `findEntryPoints`, `reachability` — walk `CodeBlock` names. Language-agnostic.
|
||
- `classify` (per-block `Exposure`) and `prioritize` (score) — run `DANGEROUS_SINK_RE`, `RISKY_PARAM_RE`, `OUTBOUND_REQUEST_RE`, `SECURITY_CONTROL_NAME_RE` over the block **body/params**. Mechanism is language-agnostic.
|
||
|
||
So the change is surgical: **replace `parseFile` with a multi-language block extractor that emits the same `CodeBlock` shape.** No downstream code changes.
|
||
|
||
## 4. Decision record — how to parse (embedded ADR; T3MP3ST has no ADR dir)
|
||
|
||
**Decision: `web-tree-sitter` (WASM), as the default on every platform.**
|
||
|
||
Options considered:
|
||
|
||
| Option | Nature | Rejected because |
|
||
|---|---|---|
|
||
| Extend regex per language | fragile per-language regex | The exact trap the file header warns against; unmaintainable across 40 grammars. |
|
||
| `tree-sitter` (node-tree-sitter) | **native** N-API C++ addon; fastest | Compiles native code at install (node-gyp/toolchain) or needs per-OS/ABI prebuilds. T3MP3ST has **zero** native deps today and ships via `npx tempest` / `npm install` onto arbitrary machines, **fully offline**, cross-platform (open Windows bug #18). Native compile is the #1 install-failure mode in exactly that scenario. |
|
||
| Per-language JS parsers (`@babel/parser`…) | pure JS, one language each | Cannot cover many languages. |
|
||
| Shell out to cxpak (external Rust tool) | multi-language, but external | Adds a binary install (brew/cargo/docker); and cxpak exposes **no** `{name, params, body, span}` symbol interface (CLI = markdown prose; `serve` = analyses, not a symbol table; `lsp` = MCP-over-stdio, no `documentSymbol`) — verified. Cannot cheaply supply `CodeBlock`s. |
|
||
| **`web-tree-sitter` (WASM)** | official WASM build of tree-sitter | **Chosen.** |
|
||
|
||
**Rationale.** web-tree-sitter is the WASM sibling of the native binding — same grammars, same parse quality — with **zero compile step**, running identically on any OS/Node/offline. It preserves T3MP3ST's pure-JS, portable, keyless character. The native binding's ~2–5× parse-speed edge is worthless here: ingest parses a repo **once per mission**, dwarfed by LLM latency. Grammar `.wasm` files are sourced from a prebuilt collection (e.g. `tree-sitter-wasms`; exact package/coverage confirmed at implementation time), not built locally.
|
||
|
||
**Consequence / accepted cost.** Async grammar init; a few MB of bundled wasm for the top languages (lazy-loaded by extension); slightly more setup code than a `require()` grammar.
|
||
|
||
## 5. Architecture
|
||
|
||
```
|
||
crawl (broadened includeExts)
|
||
→ parseFileMultiLang(path, content) ← NEW: web-tree-sitter → CodeBlock[]
|
||
├─ ext == .py → parseFile() (existing Python regex, unchanged)
|
||
├─ non-.py, grammar loaded → tree-sitter query → CodeBlock[]
|
||
└─ non-.py, no grammar / timeout / err → [] (never the Python regex) ← fail-open
|
||
→ buildCallGraph → findEntryPoints → reachability → classify → prioritize (UNCHANGED)
|
||
→ context-pack (BUDGET) → orchestrator (REASON) (UNCHANGED)
|
||
```
|
||
|
||
**SYNC architecture (revised — grammar init hoisted to bootstrap so the ingest path stays synchronous).** web-tree-sitter's `Parser.init()`/`Language.load()` are async (one-time); `parser.parse()` is sync. So init runs once at bootstrap and the per-file parse is sync — `ingestRepository` keeps its sync signature and there is no async ripple.
|
||
|
||
**New units:**
|
||
- `src/recon/ts-grammars.ts` — `initGrammars(exts): Promise<void>` (one-time, cached, `try/catch` → empty registry on failure = fail-open); `getParser(ext): Parser | undefined` (sync accessor); the ext→{query, captures, wasm} registry.
|
||
- `src/recon/ts-parse.ts` — `parseFileMultiLang(path, content, ext): CodeBlock[]` (**sync**): `.py` → `parseFile()`; non-`.py` with a loaded grammar → `parser.parse()` (with a **bounded parse time** — a hostile input must not spin) → query → `nodeToCodeBlock`; non-`.py` with no grammar / timeout / error → `[]` (never the Python regex — routing brace/`class` syntax through it would emit corrupt blocks). `nodeToCodeBlock` fills the full `CodeBlock` shape (`name` from `@name`, `body`/`lineStart`/`lineEnd` from node position, `params` via `splitParamList(text, lang)`, `decorators []`, `kind`).
|
||
- Per-language query table (top ~8: `py, js, ts, tsx, go, java, c, cpp`). cxpak's grammar queries are a reference for the capture patterns.
|
||
|
||
**Wiring.** (1) Bootstrap: `await initGrammars(...)` once at **`server.ts` startup** (the only ingest entrypoint — `server.ts:6060/6126`; the CLI never ingests, so `cli.ts` is not wired). (2) `ingestRepository()` (`code-ingest.ts:749`, **stays sync**) dispatches each file through `parseFileMultiLang(...)` (language-scoped fail-open: `.py` → `parseFile`, other languages → `[]`). (3) **Production caller swap (required):** both `whitebox.ts` callers of `ingestRepository` — `ingestRepoToSourceContext` (L125) and `runWhiteboxAnalysis` (L205) — move from `createPythonIngestConfig` to `createMultiLangIngestConfig`; without this the extractor only sees `.py` in production.
|
||
|
||
## 6. Security-ranking reuse & cross-language sinks
|
||
|
||
`classify`/`prioritize` are reused unchanged. Their regexes (`DANGEROUS_SINK_RE`, etc.) are Python-tuned; cross-language sinks that already overlap (`eval(`, `exec(`, `system(`) still fire, so ranking **degrades gracefully**, never breaks. This PR **additively** broadens the sink/param patterns with a few cross-language entries (`Runtime.exec`, `ProcessBuilder`, `exec.Command`, `os/exec`, `system(`, `popen(`, `child_process`), each with a test. Deeper per-language tuning is out of scope (incremental follow-up).
|
||
|
||
## 7. Fail-open
|
||
|
||
- `.py` always → `parseFile` (Python files unaffected). A non-`.py` file with a missing/unsupported/failed grammar → `[]` (honest absence) — **never** the Python regex, which would mis-parse brace/`class` syntax into corrupt blocks (wrong line spans, lost methods). This language-scoping is part of the security boundary.
|
||
- web-tree-sitter init throws in a hostile env → the registry stays empty; `.py` still ingests via the Python regex, every other language yields `[]` (and a one-time `console.warn`). **Ingest never crashes a mission.**
|
||
|
||
## 8. Test plan (vitest — matches their suite; deterministic, no model calls)
|
||
|
||
1. **Multi-language extraction (headline regression).** Temp repo (`mkdtempSync`, mirroring `code-ingest.test.ts`) with `.py` + `.go` + `.ts` files, each holding a function with a known sink. Assert non-`.py` functions are extracted as `CodeBlock`s **and** land as `attack_surface`/high-priority after `classify`/`prioritize`. This test *is* the proof the limitation is fixed.
|
||
2. **CodeBlock fidelity.** For a known TS/Go function, assert `name`, `params`, `line`, and `body` (contains the sink line) are correct.
|
||
3. **Fail-open — unsupported ext.** A `.zig` (or grammar-absent, non-`.py`) file returns `[]`, no throw (a `.py` file still routes to `parseFile`).
|
||
4. **Fail-open — grammar init failure.** Force `initGrammars` to fail (bad wasm path) → registry stays empty, `getGrammar` returns undefined, `.py` routes to `parseFile` and every other language returns `[]`; `ingestRepository` resolves normally, mission-safe.
|
||
5. **Cross-language sink patterns.** Unit-test each added regex against a positive and negative snippet.
|
||
6. *(optional, `--ignored` like `spine-live.test.ts`)* real wasm load against a fixture repo — live smoke.
|
||
|
||
**Acceptance:** `npm test` green, `npm run typecheck` clean; test 1 proves multi-language, tests 3–4 prove fail-open.
|
||
|
||
## 9. PR scope
|
||
|
||
**In:** `ts-grammars.ts` (registry + `initGrammars`/`getParser`) + `ts-parse.ts` (`parseFileMultiLang`/`nodeToCodeBlock`) + `param-split.ts`; `web-tree-sitter` dep + grammar wasm assets (top ~8 langs); bootstrap `initGrammars` call (server/CLI); `ingestRepository`/`crawl` wiring (**sync**, broadened exts) + `createMultiLangIngestConfig`; **`whitebox.ts` caller swap (both call sites)**; additive cross-language sink patterns; tests; README status flip (both "Python-only" rows).
|
||
|
||
**Out (name in PR as deliberate):** cxpak integration (phase 2); deep per-language sink tuning; the `serve`-backed cross-language call-graph/taint upgrade; node-tree-sitter fast path.
|
||
|
||
**Size target:** 3 new source files (`param-split`, `ts-grammars`, `ts-parse`) + edits to 4 (`code-ingest`, `whitebox`, `server`, `cli`) + coverage config + test suites (unit, headline regression, production-path, adversarial). Larger than a trivial PR because of the mandated quality gates (§12), but still one cohesive feature — deliberately not a mega-diff (cf. #61/#17, closed for sprawl).
|
||
|
||
## 10. Phase 2 (noted, not this PR) — cxpak as an optional ranking upgrade
|
||
|
||
When installed, cxpak's `serve` endpoints supply genuinely better multi-language facts than T3MP3ST's regex heuristics: `/call_graph` (cross-language edges), `/api_surface` (entry points with file+line), `/data_flow` (taint paths). A later, fail-open, opt-in PR can let cxpak *upgrade* the ranking inputs (entry points, reachability, sinks) while the web-tree-sitter parser from phase 1 remains the universal default. Not required for the headline win.
|
||
|
||
## 11. Risks
|
||
|
||
- **Grammar wasm sourcing/versioning** — confirm the collection package and language coverage at implementation time; pin versions.
|
||
- **production caller swap** — the multilang config must be repointed at BOTH `whitebox.ts` `ingestRepository` callers (L125, L205); a regression test drives `ingestRepoToSourceContext` end-to-end so the feature can't silently no-op in production. *(The async ripple is designed out by the sync + bootstrap-init architecture.)*
|
||
- **bootstrap race** — a mission firing before `initGrammars` resolves still parses Python through `parseFile`; non-Python files yield `[]` for that call. Entrypoints `await` init before serving.
|
||
- **Param extraction variance** across grammars — mitigated by falling back to slicing the signature line when the parameters node is absent.
|
||
- **Bundle size** — mitigated by lazy-loading grammars per encountered extension rather than eager-loading all.
|
||
|
||
## 12. Quality gates (mandatory — all three block merge)
|
||
|
||
1. **100% coverage of our code.** vitest coverage (`@vitest/coverage-v8`, `vitest.config.ts` with a coverage-only block) with per-file thresholds = 100% (statements/branches/functions/lines), **CI-enforced** via `npm run test:coverage`, on every NEW file (`param-split.ts`, `ts-grammars.ts`, `ts-parse.ts`). The new branches/functions added to edited files (`code-ingest.ts` sink patterns + `createMultiLangIngestConfig` + dispatch, both `whitebox.ts` caller swaps) are **covered by targeted regression tests** (`sink-patterns`, `ts-parse-multilang-rank`, `whitebox-multilang` — including a spy test on the `runWhiteboxAnalysis` swap so it can't silently revert), but not per-file-threshold-gated: a whole-file 100% threshold is unachievable on large pre-existing files. Sole coverage carve-out: the single `await initGrammars(...)` wiring line at `server.ts` bootstrap (`initGrammars` itself is 100% unit-tested; the line is `npm run smoke`-covered). Coverage is necessary-not-sufficient — gate 3 verifies the covering assertions are meaningful (no line-touching / cheater tests).
|
||
|
||
2. **Adversarial local testing.** Because ingest parses **untrusted target source**, a crafted hostile-input suite (`ts-parse-adversarial.test.ts`) drives the full wired path against malformed/mixed-language/binary/oversized inputs, a **pathological input that would make tree-sitter spin** (asserting the `ts-parse` parse-timeout fail-opens — not aspirational), and **symlink/loop entries** (crawl must not escape the contained root or hang); asserts fail-open + CodeBlock invariants + bounded wall-time/bytes. Then a `security-test-engineer` subagent actively attempts to break the built code (parser hangs, resource exhaustion, mission crash) against a hostile-repo corpus. All defects fixed before review.
|
||
|
||
3. **Strict subagent code review.** The `/code-review-pr-strict` methodology on the full diff: parallel specialist subagents with **security-critical detection first**, false-positive elimination, and the mandatory verification protocol. Every confirmed finding resolved; re-review until clean. This is the merge gate.
|
||
|
||
Gate order at rollout: build → coverage → adversarial → strict review → ship; any gate can bounce back to build.
|