feat(ingest): multi-language white-box ingest (web-tree-sitter) (#75)

* chore(ingest): pin web-tree-sitter@0.25.10 + tree-sitter-wasms@0.1.13; add multilang-ingest design docs

* feat(ingest): multi-language grammar registry (web-tree-sitter, 8 langs, fail-open); 100% cov

* feat(ingest): language-aware param-name splitter (py/go/ts/java/c); 100% cov

* feat(ingest): tree-sitter CodeBlock extractor with wall-clock parse budget + fail-open

* feat(ingest): cross-lang sinks + multilang config + wire tree-sitter into ingestRepository (sync)

* feat(ingest): repoint white-box production callers to multilang config + prod regression test

* server-init-wiring: load multi-language grammars once at bootstrap

initGrammars() runs after LLM init, before app.listen. Internally
fail-open (never rejects): on failure the grammar registry stays empty
and ingest falls back to the Python regex parser.

* adversarial-tests: hostile-input hardening for untrusted target source

Malformed per-language source, wrong-language content, binary/non-UTF8,
empty, unicode identifiers, 20k-func large file (wall-clock budget),
deeply nested input, and a symlink-loop crawl — all must fail-open, never
throw or spin. Drives parseFileMultiLang + ingestRepository/crawl.

* readme-status + docs: flip white-box status to multi-language ingest

README source-code rows now state web-tree-sitter multi-language ingest
(fail-open to Python regex). whitebox.ts scope caveat + no-source error
message updated off the stale Python-only wording.

* coverage-setup: enforce 100% coverage gate on the three new ingest files

vitest.config.ts adds a coverage-only block (v8, perFile thresholds=100)
scoped to ts-grammars.ts, param-split.ts, ts-parse.ts. Test discovery and
env stay on defaults. Edited files gate on new-branch coverage via the
regression suites; the server.ts initGrammars bootstrap line is the
documented smoke-covered carve-out.

* review: fix WASM heap leak, language-scoped fail-open, sink recall/precision

Adversarial security + strict code review (two independent subagents)
returned one blocking bug and several correctness findings; remediated:

BLOCKING #1 — native WASM memory leak: web-tree-sitter Tree/Parser objects
were never .delete()'d. Both reviewers reproduced a process-global wedge:
after ~7 large parses the shared WASM heap exhausts, every subsequent parse
aborts (caught by fail-open), and non-Python ingest silently returns []
forever. Fix: free the tree in a finally; delete the parser in resetParser.
Regression test parses ~8MB through the real shared parser 12x.

regex, which mis-parses brace/class syntax into corrupt blocks (wrong line
spans, lost methods). Now non-.py fail-open returns [] (honest absence); the
Python regex only ever runs on .py. Isolated ts-parse-fallback test.

http(s).request, axios(config), C exec-family); negative lookbehind stops
bare system(/popen(/fetch( from matching Python receivers (os.system etc.
still matched, corpus rankings unchanged). Recall+precision tests added.

operator sees non-Python degradation.

(one-line call-graph edges, no decorators, C++ member methods).

* review round 2: fix sink-regex regressions, real timeout test, deterministic leak guard

Third review round (correctness + security re-attack + test-quality) against
the round-1 fixes surfaced issues in the fixes themselves; remediated:

- Sink-regex regressions I introduced: (1) my fetch lookbehind dropped
  window.fetch/this.fetch (top JS SSRF idiom) — reverted to \bfetch\(;
  (2) bare .Get/.Post/.Do flagged db.Get/cache.Get/queue.Do as SSRF —
  receiver-qualified to [Cc]lient. Recall+precision tests cover both;
  regex is ReDoS-free (timed).
- Placebo leak test: the 12x function-dense loop did NOT wedge the parser
  on machines with heap headroom (proven: passed with tree.delete removed).
  Replaced with a deterministic vi.spyOn(Tree.prototype,'delete') assertion,
  verified RED without the fix.
- Real (non-mocked) parse-timeout test: parseFileMultiLang gains an injectable
  budgetMs; a 0ms budget drives the real progressCallback cancellation to [].
- runWhiteboxAnalysis swap now has coverage (spy on orchestrator.run) so the
  2nd production caller can't silently revert to the Python-only config.
- Un-vacuoused weak tests: unicode now asserts the extracted block; nested/
  wrong-language wrapped in invariants() + timing bounds.
- CI now runs npm run test:coverage (the 100% gate was defined but unenforced).
- Security: documented maxFileBytes as the control bounding a single parse's
  peak WASM memory below the 2GB heap-abort ceiling.
- Stale 'fail-open to Python regex' docs fixed (ts-parse, whitebox, server,
  adversarial); circular-import note added; temp-dir cleanup; honest coverage
  wording in spec + config.

* docs(ingest): align fallback-contract text with the shipped language-scoped fail-open

Review feedback (#75, @jmagly): the design docs + one ts-grammars comment
still described the pre-implementation 'fall back to the Python regex'
model for non-Python files. The shipped contract is language-scoped: .py
uses parseFile; any non-.py file with a missing/unsupported/failed grammar
or a parse timeout returns [] (never the Python regex, which would mis-parse
brace/class syntax into corrupt blocks). Updated every stale reference in
multilang-ingest-spec.md, multilang-ingest-plan.md, and ts-grammars.ts so
the docs match the security boundary. Docs/comment only — no code change.

* fix(test): make the ReDoS wall-clock guard robust under coverage instrumentation

Review feedback (#75, @jmagly): the CI step this PR adds (npm run
test:coverage = vitest run --coverage) runs the full suite under V8
coverage, and that overhead pushes the pathological-brace DoS timing guard
in local-agent-tool-calling.test.ts:69 over its <300ms bound (320 < 300
fails on CI). Raise the bound to 1000ms — still 2x below the ~2100ms
quadratic-ReDoS regression it guards against, so it fails hard if the
regression returns, but no longer false-fails under coverage/slow CI.

Chose the reviewer's option (a) (robust bound) over scoping test:coverage,
so the gate stays a simple full-suite run and no test-file enumeration can
rot. Verified: npm run test:coverage is green and the 3 new files stay at
100%.

* fix(ingest): cover pointer-returning C functions

---------

Co-authored-by: Joseph Magly <1159087+jmagly@users.noreply.github.com>

src/recon/code-ingest.ts

@@ -37,7 +37,8 @@
  */
 
 import { readdirSync, readFileSync, statSync } from 'fs';
-import { join, sep } from 'path';
+import { join, sep, extname } from 'path';
+import { parseFileMultiLang } from './ts-parse.js';
 import {
   estimateTokens,
   type SourceFile,
@@ -169,14 +170,23 @@ const ENTRY_NAME_RES: RegExp[] = [
 const SECURITY_CONTROL_NAME_RE =
   /valid|verif|auth|authoriz|authentic|permission|sanitiz|escap|csrf|check_|require_|is_allowed/i;
 
-// Dangerous sinks that make a block part of the attack surface.
-const DANGEROUS_SINK_RE =
-  /requests\.(get|post|put)|urllib|urlopen|httpx|socket\.|subprocess|os\.system|\beval\(|\bexec\(|pickle\.loads|yaml\.load|cursor\.execute|\.raw\(|open\(/;
+// Dangerous sinks that make a block part of the attack surface. Python patterns
+// plus cross-language sinks (Java/Go/C/JS) for the multi-language ingest.
+// Bare-call patterns (system/popen/exec-family/fetch) use a `(?<![\w.])` guard so
+// they match a C/JS free call `system(x)` but NOT a Python/JS method call on a
+// receiver `foo.system(x)` — that keeps the existing Python corpus's rankings
+// stable (its `os.system`/`os.popen` are matched by the qualified patterns).
+export const DANGEROUS_SINK_RE =
+  /requests\.(get|post|put)|urllib|urlopen|httpx|socket\.|subprocess|os\.system|\beval\(|\bexec\(|pickle\.loads|yaml\.load|cursor\.execute|\.raw\(|open\(|exec\.Command(?:Context)?|Runtime\.getRuntime|ProcessBuilder|(?<![\w.])system\(|(?<![\w.])popen\(|(?<![\w.])exec(?:l|v)[pe]?\(|http\.(Get|Post|NewRequest)|https?\.request\(|[Cc]lient\.(Do|Get|Post)\(|\bfetch\(|axios[.(]/;
 
 // Outbound-request sinks specifically (subset of the above) — used for the
-// SSRF/IDOR "identifier param + outbound request" signal.
-const OUTBOUND_REQUEST_RE =
-  /requests\.(get|post|put)|urllib|urlopen|httpx|socket\./;
+// SSRF/IDOR "identifier param + outbound request" signal. Cross-language: Go
+// net/http (http.Get/Post/NewRequest and *client*.Do/Get/Post), Node http(s).request,
+// JS fetch/axios. `fetch` uses \b (matches window.fetch/this.fetch too); the Go
+// client methods are receiver-qualified to `client` so a mundane db.Get/cache.Get
+// is NOT mistaken for an outbound request.
+export const OUTBOUND_REQUEST_RE =
+  /requests\.(get|post|put)|urllib|urlopen|httpx|socket\.|http\.(Get|Post|get|post|NewRequest)|https?\.request\(|[Cc]lient\.(Do|Get|Post)\(|\bfetch\(|axios[.(]/;
 
 // URL/identifier-shaped param names.
 const RISKY_PARAM_RE = /url|uri|endpoint|host|addr|id$|_id|path|file|name/i;
@@ -764,7 +774,10 @@ export function ingestRepository(config: IngestConfig): IngestResult {
     }
     processedFiles += 1;
     totalBytes += content.length;
-    allBlocks.push(...parseFile(path, content));
+    // Multi-language dispatch: .py uses the legacy regex parser; supported
+    // non-Python languages use tree-sitter and yield [] when their grammar is
+    // unavailable. Stays sync — parseFileMultiLang does not await.
+    allBlocks.push(...parseFileMultiLang(path, content, extname(path).toLowerCase()));
     if (maxTotalBytes !== undefined && totalBytes >= maxTotalBytes) {
       truncated = true;
       console.warn(`[code-ingest] ingest reached maxTotalBytes ceiling (${maxTotalBytes}) after ${totalBytes} bytes; remaining files skipped — raise IngestConfig.maxTotalBytes to analyze more.`);
@@ -841,6 +854,30 @@ export function createPythonIngestConfig(repoRoot: string): IngestConfig {
   };
 }
 
+/**
+ * Multi-language ingest config: same ceilings as the Python config, but crawls
+ * the languages the tree-sitter extractor supports. Non-source files are
+ * excluded by `includeExts`. Used by the white-box analysis entrypoints.
+ *
+ * SECURITY: `maxFileBytes` (1 MB) is a security control here, not just a perf
+ * knob — it caps the peak WASM linear-memory a single tree-sitter parse can
+ * demand. A dense/obfuscated file large enough to approach web-tree-sitter's
+ * 2 GB heap ceiling makes the native allocator abort() (caught + fail-open, but
+ * the WASM high-water-mark cannot be reclaimed for the process's life). The 1 MB
+ * cap keeps that unreachable (worst measured blow-up ~370x → ~5-6x margin). Do
+ * NOT raise it without re-testing that margin.
+ */
+export function createMultiLangIngestConfig(repoRoot: string): IngestConfig {
+  return {
+    repoRoot,
+    includeExts: ['.py', '.js', '.ts', '.tsx', '.go', '.java', '.c', '.cpp'],
+    excludeGlobs: [...DEFAULT_EXCLUDES],
+    maxFileBytes: 1_000_000,
+    maxFiles: 50_000,
+    maxTotalBytes: 1_000_000_000,
+  };
+}
+
 // =============================================================================
 // COMPOSITION WITH context-pack — rendering & security-ordered packing
 // =============================================================================