t3mp3st/docs/design/multilang-ingest-spec.md

Multi-language white-box ingest for T3MP3ST

Status: Draft design (for an upstream PR to elder-plinius/T3MP3ST) Date: 2026-07-10 Scope: Replace the Python-only ingest parser with a portable multi-language parser, reusing T3MP3ST's existing security-ranking pipeline unchanged.

---

1. Problem

T3MP3ST markets white-box source hunting as a proven strength ("Code: ✅ Proven — CVE-Zero 8/10 across 7 languages"), but the ingest engine only reads Python. src/recon/code-ingest.ts:6 says so itself:

"This is a PYTHON-ONLY, regex-based 'AST-lite' PROTOTYPE … The real version is tree-sitter (or a language server) + multi-language."

The README status table flags it: Code → ⚠️ Python-only ingest. Point the white-box flow at a Go/Java/C/TS repo and crawl keeps only .py; every non-Python file is invisible to the hunt.

2. Goal / Non-goals

Goal. White-box ingest parses many languages, feeding T3MP3ST's existing security-ranking so multi-language repos get the same attack-surface-ordered analysis Python repos get today.

Non-goals.

• Not changing the security model (classify/prioritize/reachability/context-pack) — it is reused verbatim.
• Not adding an external binary, native-compiled dependency, or new API key.
• Not building cross-language taint/call-graph analysis in this PR (see §9, phase 2).

3. Key insight — the seam is one stage

The ingest is a pipeline (whitebox.ts): code-ingest (SELECT) → context-pack (BUDGET) → orchestrator (REASON). Within SELECT, only one stage is language-bound:

• parseFile() (code-ingest.ts:409) — Python def/class regex → CodeBlock[]. Language-bound.
• Everything downstream operates on CodeBlock {name, params, body, line, …} via text regexes over the block body/name/params:
  • buildCallGraph, findEntryPoints, reachability — walk CodeBlock names. Language-agnostic.
  • classify (per-block Exposure) and prioritize (score) — run DANGEROUS_SINK_RE, RISKY_PARAM_RE, OUTBOUND_REQUEST_RE, SECURITY_CONTROL_NAME_RE over the block body/params. Mechanism is language-agnostic.

So the change is surgical: replace parseFile with a multi-language block extractor that emits the same CodeBlock shape. No downstream code changes.

4. Decision record — how to parse (embedded ADR; T3MP3ST has no ADR dir)

Decision: web-tree-sitter (WASM), as the default on every platform.

Options considered:

Option	Nature	Rejected because
Extend regex per language	fragile per-language regex	The exact trap the file header warns against; unmaintainable across 40 grammars.
tree-sitter (node-tree-sitter)	native N-API C++ addon; fastest	Compiles native code at install (node-gyp/toolchain) or needs per-OS/ABI prebuilds. T3MP3ST has zero native deps today and ships via npx tempest / npm install onto arbitrary machines, fully offline, cross-platform (open Windows bug #18). Native compile is the #1 install-failure mode in exactly that scenario.
Per-language JS parsers (@babel/parser…)	pure JS, one language each	Cannot cover many languages.
Shell out to cxpak (external Rust tool)	multi-language, but external	Adds a binary install (brew/cargo/docker); and cxpak exposes no {name, params, body, span} symbol interface (CLI = markdown prose; serve = analyses, not a symbol table; lsp = MCP-over-stdio, no documentSymbol) — verified. Cannot cheaply supply CodeBlocks.
web-tree-sitter (WASM)	official WASM build of tree-sitter	Chosen.

Rationale. web-tree-sitter is the WASM sibling of the native binding — same grammars, same parse quality — with zero compile step, running identically on any OS/Node/offline. It preserves T3MP3ST's pure-JS, portable, keyless character. The native binding's ~2–5× parse-speed edge is worthless here: ingest parses a repo once per mission, dwarfed by LLM latency. Grammar .wasm files are sourced from a prebuilt collection (e.g. tree-sitter-wasms; exact package/coverage confirmed at implementation time), not built locally.

Consequence / accepted cost. Async grammar init; a few MB of bundled wasm for the top languages (lazy-loaded by extension); slightly more setup code than a require() grammar.

5. Architecture

crawl (broadened includeExts)
  → parseFileMultiLang(path, content)          ← NEW: web-tree-sitter → CodeBlock[]
      ├─ ext == .py                          → parseFile()  (existing Python regex, unchanged)
      ├─ non-.py, grammar loaded             → tree-sitter query → CodeBlock[]
      └─ non-.py, no grammar / timeout / err → []  (never the Python regex)   ← fail-open
  → buildCallGraph → findEntryPoints → reachability → classify → prioritize   (UNCHANGED)
  → context-pack (BUDGET) → orchestrator (REASON)                             (UNCHANGED)

SYNC architecture (revised — grammar init hoisted to bootstrap so the ingest path stays synchronous). web-tree-sitter's Parser.init()/Language.load() are async (one-time); parser.parse() is sync. So init runs once at bootstrap and the per-file parse is sync — ingestRepository keeps its sync signature and there is no async ripple.

New units:

• src/recon/ts-grammars.ts — initGrammars(exts): Promise<void> (one-time, cached, try/catch → empty registry on failure = fail-open); getParser(ext): Parser | undefined (sync accessor); the ext→{query, captures, wasm} registry.
• src/recon/ts-parse.ts — parseFileMultiLang(path, content, ext): CodeBlock[] (sync): .py → parseFile(); non-.py with a loaded grammar → parser.parse() (with a bounded parse time — a hostile input must not spin) → query → nodeToCodeBlock; non-.py with no grammar / timeout / error → [] (never the Python regex — routing brace/class syntax through it would emit corrupt blocks). nodeToCodeBlock fills the full CodeBlock shape (name from @name, body/lineStart/lineEnd from node position, params via splitParamList(text, lang), decorators [], kind).
• Per-language query table (top ~8: py, js, ts, tsx, go, java, c, cpp). cxpak's grammar queries are a reference for the capture patterns.

Wiring. (1) Bootstrap: await initGrammars(...) once at server.ts startup (the only ingest entrypoint — server.ts:6060/6126; the CLI never ingests, so cli.ts is not wired). (2) ingestRepository() (code-ingest.ts:749, stays sync) dispatches each file through parseFileMultiLang(...) (language-scoped fail-open: .py → parseFile, other languages → []). (3) Production caller swap (required): both whitebox.ts callers of ingestRepository — ingestRepoToSourceContext (L125) and runWhiteboxAnalysis (L205) — move from createPythonIngestConfig to createMultiLangIngestConfig; without this the extractor only sees .py in production.

6. Security-ranking reuse & cross-language sinks

classify/prioritize are reused unchanged. Their regexes (DANGEROUS_SINK_RE, etc.) are Python-tuned; cross-language sinks that already overlap (eval(, exec(, system() still fire, so ranking degrades gracefully, never breaks. This PR additively broadens the sink/param patterns with a few cross-language entries (Runtime.exec, ProcessBuilder, exec.Command, os/exec, system(, popen(, child_process), each with a test. Deeper per-language tuning is out of scope (incremental follow-up).

7. Fail-open

• .py always → parseFile (Python files unaffected). A non-.py file with a missing/unsupported/failed grammar → [] (honest absence) — never the Python regex, which would mis-parse brace/class syntax into corrupt blocks (wrong line spans, lost methods). This language-scoping is part of the security boundary.
• web-tree-sitter init throws in a hostile env → the registry stays empty; .py still ingests via the Python regex, every other language yields [] (and a one-time console.warn). Ingest never crashes a mission.

8. Test plan (vitest — matches their suite; deterministic, no model calls)

1. Multi-language extraction (headline regression). Temp repo (mkdtempSync, mirroring code-ingest.test.ts) with .py + .go + .ts files, each holding a function with a known sink. Assert non-.py functions are extracted as CodeBlocks and land as attack_surface/high-priority after classify/prioritize. This test is the proof the limitation is fixed.
2. CodeBlock fidelity. For a known TS/Go function, assert name, params, line, and body (contains the sink line) are correct.
3. Fail-open — unsupported ext. A .zig (or grammar-absent, non-.py) file returns [], no throw (a .py file still routes to parseFile).
4. Fail-open — grammar init failure. Force initGrammars to fail (bad wasm path) → registry stays empty, getGrammar returns undefined, .py routes to parseFile and every other language returns []; ingestRepository resolves normally, mission-safe.
5. Cross-language sink patterns. Unit-test each added regex against a positive and negative snippet.
6. (optional, --ignored like spine-live.test.ts) real wasm load against a fixture repo — live smoke.

Acceptance: npm test green, npm run typecheck clean; test 1 proves multi-language, tests 3–4 prove fail-open.

9. PR scope

In: ts-grammars.ts (registry + initGrammars/getParser) + ts-parse.ts (parseFileMultiLang/nodeToCodeBlock) + param-split.ts; web-tree-sitter dep + grammar wasm assets (top ~8 langs); bootstrap initGrammars call (server/CLI); ingestRepository/crawl wiring (sync, broadened exts) + createMultiLangIngestConfig; whitebox.ts caller swap (both call sites); additive cross-language sink patterns; tests; README status flip (both "Python-only" rows).

Out (name in PR as deliberate): cxpak integration (phase 2); deep per-language sink tuning; the serve-backed cross-language call-graph/taint upgrade; node-tree-sitter fast path.

Size target: 3 new source files (param-split, ts-grammars, ts-parse) + edits to 4 (code-ingest, whitebox, server, cli) + coverage config + test suites (unit, headline regression, production-path, adversarial). Larger than a trivial PR because of the mandated quality gates (§12), but still one cohesive feature — deliberately not a mega-diff (cf. #61/#17, closed for sprawl).

10. Phase 2 (noted, not this PR) — cxpak as an optional ranking upgrade

When installed, cxpak's serve endpoints supply genuinely better multi-language facts than T3MP3ST's regex heuristics: /call_graph (cross-language edges), /api_surface (entry points with file+line), /data_flow (taint paths). A later, fail-open, opt-in PR can let cxpak upgrade the ranking inputs (entry points, reachability, sinks) while the web-tree-sitter parser from phase 1 remains the universal default. Not required for the headline win.

11. Risks

• Grammar wasm sourcing/versioning — confirm the collection package and language coverage at implementation time; pin versions.
• production caller swap — the multilang config must be repointed at BOTH whitebox.ts ingestRepository callers (L125, L205); a regression test drives ingestRepoToSourceContext end-to-end so the feature can't silently no-op in production. (The async ripple is designed out by the sync + bootstrap-init architecture.)
• bootstrap race — a mission firing before initGrammars resolves still parses Python through parseFile; non-Python files yield [] for that call. Entrypoints await init before serving.
• Param extraction variance across grammars — mitigated by falling back to slicing the signature line when the parameters node is absent.
• Bundle size — mitigated by lazy-loading grammars per encountered extension rather than eager-loading all.

12. Quality gates (mandatory — all three block merge)

1. 100% coverage of our code. vitest coverage (@vitest/coverage-v8, vitest.config.ts with a coverage-only block) with per-file thresholds = 100% (statements/branches/functions/lines), CI-enforced via npm run test:coverage, on every NEW file (param-split.ts, ts-grammars.ts, ts-parse.ts). The new branches/functions added to edited files (code-ingest.ts sink patterns + createMultiLangIngestConfig + dispatch, both whitebox.ts caller swaps) are covered by targeted regression tests (sink-patterns, ts-parse-multilang-rank, whitebox-multilang — including a spy test on the runWhiteboxAnalysis swap so it can't silently revert), but not per-file-threshold-gated: a whole-file 100% threshold is unachievable on large pre-existing files. Sole coverage carve-out: the single await initGrammars(...) wiring line at server.ts bootstrap (initGrammars itself is 100% unit-tested; the line is npm run smoke-covered). Coverage is necessary-not-sufficient — gate 3 verifies the covering assertions are meaningful (no line-touching / cheater tests).

2. Adversarial local testing. Because ingest parses untrusted target source, a crafted hostile-input suite (ts-parse-adversarial.test.ts) drives the full wired path against malformed/mixed-language/binary/oversized inputs, a pathological input that would make tree-sitter spin (asserting the ts-parse parse-timeout fail-opens — not aspirational), and symlink/loop entries (crawl must not escape the contained root or hang); asserts fail-open + CodeBlock invariants + bounded wall-time/bytes. Then a security-test-engineer subagent actively attempts to break the built code (parser hangs, resource exhaustion, mission crash) against a hostile-repo corpus. All defects fixed before review.

3. Strict subagent code review. The /code-review-pr-strict methodology on the full diff: parallel specialist subagents with security-critical detection first, false-positive elimination, and the mandatory verification protocol. Every confirmed finding resolved; re-review until clean. This is the merge gate.

Gate order at rollout: build → coverage → adversarial → strict review → ship; any gate can bounce back to build.