feat(ingest): multi-language white-box ingest (web-tree-sitter) (#75)
* chore(ingest): pin web-tree-sitter@0.25.10 + tree-sitter-wasms@0.1.13; add multilang-ingest design docs * feat(ingest): multi-language grammar registry (web-tree-sitter, 8 langs, fail-open); 100% cov * feat(ingest): language-aware param-name splitter (py/go/ts/java/c); 100% cov * feat(ingest): tree-sitter CodeBlock extractor with wall-clock parse budget + fail-open * feat(ingest): cross-lang sinks + multilang config + wire tree-sitter into ingestRepository (sync) * feat(ingest): repoint white-box production callers to multilang config + prod regression test * server-init-wiring: load multi-language grammars once at bootstrap initGrammars() runs after LLM init, before app.listen. Internally fail-open (never rejects): on failure the grammar registry stays empty and ingest falls back to the Python regex parser. * adversarial-tests: hostile-input hardening for untrusted target source Malformed per-language source, wrong-language content, binary/non-UTF8, empty, unicode identifiers, 20k-func large file (wall-clock budget), deeply nested input, and a symlink-loop crawl — all must fail-open, never throw or spin. Drives parseFileMultiLang + ingestRepository/crawl. * readme-status + docs: flip white-box status to multi-language ingest README source-code rows now state web-tree-sitter multi-language ingest (fail-open to Python regex). whitebox.ts scope caveat + no-source error message updated off the stale Python-only wording. * coverage-setup: enforce 100% coverage gate on the three new ingest files vitest.config.ts adds a coverage-only block (v8, perFile thresholds=100) scoped to ts-grammars.ts, param-split.ts, ts-parse.ts. Test discovery and env stay on defaults. Edited files gate on new-branch coverage via the regression suites; the server.ts initGrammars bootstrap line is the documented smoke-covered carve-out. * review: fix WASM heap leak, language-scoped fail-open, sink recall/precision Adversarial security + strict code review (two independent subagents) returned one blocking bug and several correctness findings; remediated: BLOCKING #1 — native WASM memory leak: web-tree-sitter Tree/Parser objects were never .delete()'d. Both reviewers reproduced a process-global wedge: after ~7 large parses the shared WASM heap exhausts, every subsequent parse aborts (caught by fail-open), and non-Python ingest silently returns [] forever. Fix: free the tree in a finally; delete the parser in resetParser. Regression test parses ~8MB through the real shared parser 12x. regex, which mis-parses brace/class syntax into corrupt blocks (wrong line spans, lost methods). Now non-.py fail-open returns [] (honest absence); the Python regex only ever runs on .py. Isolated ts-parse-fallback test. http(s).request, axios(config), C exec-family); negative lookbehind stops bare system(/popen(/fetch( from matching Python receivers (os.system etc. still matched, corpus rankings unchanged). Recall+precision tests added. operator sees non-Python degradation. (one-line call-graph edges, no decorators, C++ member methods). * review round 2: fix sink-regex regressions, real timeout test, deterministic leak guard Third review round (correctness + security re-attack + test-quality) against the round-1 fixes surfaced issues in the fixes themselves; remediated: - Sink-regex regressions I introduced: (1) my fetch lookbehind dropped window.fetch/this.fetch (top JS SSRF idiom) — reverted to \bfetch\(; (2) bare .Get/.Post/.Do flagged db.Get/cache.Get/queue.Do as SSRF — receiver-qualified to [Cc]lient. Recall+precision tests cover both; regex is ReDoS-free (timed). - Placebo leak test: the 12x function-dense loop did NOT wedge the parser on machines with heap headroom (proven: passed with tree.delete removed). Replaced with a deterministic vi.spyOn(Tree.prototype,'delete') assertion, verified RED without the fix. - Real (non-mocked) parse-timeout test: parseFileMultiLang gains an injectable budgetMs; a 0ms budget drives the real progressCallback cancellation to []. - runWhiteboxAnalysis swap now has coverage (spy on orchestrator.run) so the 2nd production caller can't silently revert to the Python-only config. - Un-vacuoused weak tests: unicode now asserts the extracted block; nested/ wrong-language wrapped in invariants() + timing bounds. - CI now runs npm run test:coverage (the 100% gate was defined but unenforced). - Security: documented maxFileBytes as the control bounding a single parse's peak WASM memory below the 2GB heap-abort ceiling. - Stale 'fail-open to Python regex' docs fixed (ts-parse, whitebox, server, adversarial); circular-import note added; temp-dir cleanup; honest coverage wording in spec + config. * docs(ingest): align fallback-contract text with the shipped language-scoped fail-open Review feedback (#75, @jmagly): the design docs + one ts-grammars comment still described the pre-implementation 'fall back to the Python regex' model for non-Python files. The shipped contract is language-scoped: .py uses parseFile; any non-.py file with a missing/unsupported/failed grammar or a parse timeout returns [] (never the Python regex, which would mis-parse brace/class syntax into corrupt blocks). Updated every stale reference in multilang-ingest-spec.md, multilang-ingest-plan.md, and ts-grammars.ts so the docs match the security boundary. Docs/comment only — no code change. * fix(test): make the ReDoS wall-clock guard robust under coverage instrumentation Review feedback (#75, @jmagly): the CI step this PR adds (npm run test:coverage = vitest run --coverage) runs the full suite under V8 coverage, and that overhead pushes the pathological-brace DoS timing guard in local-agent-tool-calling.test.ts:69 over its <300ms bound (320 < 300 fails on CI). Raise the bound to 1000ms — still 2x below the ~2100ms quadratic-ReDoS regression it guards against, so it fails hard if the regression returns, but no longer false-fails under coverage/slow CI. Chose the reviewer's option (a) (robust bound) over scoping test:coverage, so the gate stays a simple full-suite run and no test-file enumeration can rot. Verified: npm run test:coverage is green and the 3 new files stay at 100%. * fix(ingest): cover pointer-returning C functions --------- Co-authored-by: Joseph Magly <1159087+jmagly@users.noreply.github.com>
This commit is contained in:
Lyubomir BozhinovandJoseph Magly authored and GitHub committed
1 parent
440dedd9c1
commit
151d3555c6
24 files changed
+1681
-25
No files matched your search
@@ -19,6 +19,9 @@ jobs:
|
||||
- run: npm run lint
|
||||
- run: npm run typecheck
|
||||
- run: npm test
|
||||
# Enforce the 100% coverage gate on the new multi-language ingest files
|
||||
# (vitest exits non-zero if a per-file threshold is missed).
|
||||
- run: npm run test:coverage
|
||||
- run: npm run doctor
|
||||
# Honesty gates: prove the headline numbers re-derive and that no
|
||||
# challenge-fitting has crept in. Public contributors / CI can verify.
|
||||
|
||||
@@ -65,3 +65,6 @@ scripts/.export-denylist.local
|
||||
!.aiwg/addons/**
|
||||
|
||||
server.pid
|
||||
|
||||
# vitest v8 coverage output
|
||||
coverage/
|
||||
@@ -53,7 +53,7 @@ Offensive security sits behind years of practice and expensive tooling. The bet
|
||||
| 🕸️ **Web apps** | Black-box, external-attacker recon → exploit (XBEN suite) | ✅ Stable |
|
||||
| 🚩 **CTF** | Hint-free, sandbox-jailed solves (Cybench) | ✅ Stable |
|
||||
| 🤖 **Robotics / OT / embedded** | Coordinated-disclosure pipeline for OSS vuln hunting (OSV + live-PoC + refuter) | ✅ Pipeline stable |
|
||||
| 📂 **Source code** | White-box repo analysis with blind master-builder decomposition | ⚠️ Python-only ingest |
|
||||
| 📂 **Source code** | White-box repo analysis with blind master-builder decomposition | ✅ Multi-language ingest (web-tree-sitter) |
|
||||
| 💰 **Smart contracts** | Damn Vulnerable DeFi | ⚠️ reproduction, not novel discovery |
|
||||
| ☁️ **Cloud (IaC)** | Misconfig-detection benchmark (`cloud:bench`) + opt-in cloud arsenal (aws/az/gcloud + scoutsuite/cloudfox/pmapper; pacu gated) | 🚧 IaC-misconfig scaffolding — live-cloud exploitation not yet benchmarked |
|
||||
| 📱 **Mobile** | Built-in static analyzer (manifest misconfig + secret/cleartext detection, `mobile:bench`) + opt-in arsenal (mobsfscan/objection/drozer; frida gated) | 🚧 static-detection scaffolding — dynamic exploitation not benchmarked |
|
||||
@@ -194,7 +194,7 @@ The framework is an 8-operator kill chain, and this table won't blow smoke about
|
||||
| Arsenal, MCP server, HTTP API | ✅ Stable | 35 built-in tools by default; 108 with the opt-in `T3MP3ST_FULL_ARSENAL` (+73 adapters, with dangerous/catalog-only drivers — metasploit, hydra, pacu, frida — behind narrow approved paths rather than generic execution) — both counts re-derive via `verify-claims`. `security_recon` over MCP |
|
||||
| Egress-scope containment | ✅ Stable (on by default) | once a mission target is set, built-in networked tools refuse off-scope public hosts — not the target/subdomains, not loopback/private (`SCOPE DENIED`) — a tightened default, not a bare tool runner |
|
||||
| Coordinated-disclosure pipeline | ✅ Stable | OSV novelty + live PoC + refuter panel + CVSS; drafts only, a human sends |
|
||||
| White-box source analysis | ⚠️ Experimental | Python-only regex ingest; multi-model decomposition costs more tokens, not fewer |
|
||||
| White-box source analysis | ⚠️ Experimental | Multi-language ingest via web-tree-sitter (Python/JS/TS/Go/Java/C/C++); Python retains its regex parser, while other languages fail open to no extracted blocks; multi-model decomposition costs more tokens, not fewer |
|
||||
| DeFi (Damn Vulnerable DeFi) | ⚠️ Experimental | reproduces known exploit classes; not novel discovery |
|
||||
| Exploiter / Infiltrator / Exfiltrator / Ghost | ⚠️ Experimental | run the real tool-backed ReAct loop (same engine as recon); unproven as a coordinated swarm — single-agent is the benchmarked path, live swarm exploitation still unreliable |
|
||||
| Advanced modules (cloud, persistence, swarm, cognition) | 🚧 Planned | interface-only in `src/stubs/` |
|
||||
|
||||
@@ -0,0 +1,243 @@
|
||||
# Plan: T3MP3ST-MLINGEST — Multi-language white-box ingest (web-tree-sitter)
|
||||
|
||||
## Executive Summary
|
||||
- **Goal:** Replace the Python-only `parseFile` stage with a portable `web-tree-sitter` extractor emitting the existing `CodeBlock` shape; reuse the security-ranking pipeline verbatim; language-scoped fail-open (`.py` keeps the Python regex; other languages return `[]` on a grammar miss / parse error — never the Python regex).
|
||||
- **Complexity:** Medium-low (no async ripple — see Arch). **Risk:** Low-Medium. **Diff:** 3 new source files (`param-split`, `ts-grammars`, `ts-parse`) + edits to 3 (`code-ingest`, `whitebox`, `server`) + coverage config + test suites.
|
||||
- **Spec:** `2026-07-10-multilang-ingest-design.md`. **Reviews folded:** architect-review (async→sync pivot + 5 defects), test-automator (RED discrimination + granularity).
|
||||
|
||||
## 🚨 Critical Implementation Standards
|
||||
- Fail-open mandatory (language-scoped): `.py` → `parseFile`; a non-`.py` grammar init/parse failure → `[]` (never the Python regex). Ingest never crashes a mission.
|
||||
- Security ranking reused, not rewritten (`classify`/`prioritize`/`reachability`/`context-pack` untouched).
|
||||
- vitest green, `tsc --noEmit` clean, `eslint` clean. No TODO/HACK/dead code; every failure path handled.
|
||||
- **100% coverage of our code (hard gate).** Statements/branches/functions/lines = 100% on every NEW file (`param-split.ts`, `ts-grammars.ts`, `ts-parse.ts`) and 100% of every new branch/function added to edited files (`code-ingest.ts`, `whitebox.ts`). Sole carve-out: the **single** `await initGrammars(...)` wiring line at `server.ts` bootstrap — `initGrammars` itself is 100% unit-tested in `grammar-registry`; the wiring line is exercised by `npm run smoke` at the ship gate. (`cli.ts` is NOT wired — the CLI never ingests; ingest is reached only via `server.ts:6060/6126`.) Coverage is necessary-not-sufficient — the review gate confirms assertions are meaningful, not line-touching.
|
||||
- **Adversarial phase mandatory (this parses untrusted target source).** A crafted hostile-input suite + a `security-test-engineer` subagent actively trying to break the built code; all defects fixed before review.
|
||||
- **Strict subagent review is the merge gate.** `/code-review-pr-strict` methodology (parallel specialists, security-critical detection FIRST, false-positive elimination) on the full diff; every confirmed finding resolved; re-review until clean.
|
||||
|
||||
## Architecture — SYNC ingest + bootstrap init (revised per architect §2)
|
||||
web-tree-sitter: only `Parser.init()`/`Language.load(wasm)` are async (one-time); `parser.parse()` is **sync**. So we hoist init to bootstrap and keep the ingest path synchronous — **`ingestRepository` stays sync**, eliminating the entire async ripple through `whitebox.ts`/`server.ts`/tests.
|
||||
|
||||
```
|
||||
bootstrap (server start / CLI main): await initGrammars(exts) // load all grammars once into a module singleton; try/catch → empty registry on failure (fail-open)
|
||||
─────────────────────────────────────
|
||||
crawl (multi-lang includeExts)
|
||||
→ ingestRepository (SYNC, unchanged signature) — per file:
|
||||
parseFileMultiLang(path, content, ext) // SYNC
|
||||
├─ ext == .py → parseFile(...) // Python regex, unchanged
|
||||
├─ non-.py, getGrammar(ext) loaded? → parser.parse() → query → nodeToCodeBlock[] → CodeBlock[]
|
||||
└─ non-.py, no grammar / not-yet-loaded / timeout / error → [] // FAIL-OPEN (never the Python regex)
|
||||
→ buildCallGraph → findEntryPoints → reachability → classify → prioritize (UNCHANGED)
|
||||
→ context-pack → orchestrator (UNCHANGED)
|
||||
```
|
||||
Race: a mission firing before init resolves simply fail-opens for that call (`.py` → Python regex, other languages → `[]`) — degrade, never crash. Grammars eager-loaded at bootstrap (~few MB, one-time) — the cost of a sync hot path.
|
||||
|
||||
**Killed by the sync pivot:** the async signature change on `ingestRepository`; the `ingest-limits.test.ts` await churn (D1); the unguarded-init mission crash (D5, now a bootstrap try/catch).
|
||||
**NOT killed — still required (validator BLOCKER):** `whitebox.ts` must still be edited to repoint BOTH production callers (`ingestRepoToSourceContext` L125, `runWhiteboxAnalysis` L205) from `createPythonIngestConfig` → `createMultiLangIngestConfig`. This config-swap is independent of sync/async; without it the multilang extractor only ever sees `.py` in production and the PR is a no-op. Node `whitebox-wiring` below.
|
||||
|
||||
## Current State (grounded in /tmp/T3MP3ST)
|
||||
- `CodeBlock = {id, path, name, kind, lineStart, lineEnd, params, decorators, body}` (`id=\`${path}::${name}@${lineStart}\``; `decorators=[]` for non-Python).
|
||||
- `parseFile` (code-ingest.ts:409, stays as the `.py` path only); `parseParams` (decl L328, Python-only).
|
||||
- `ingestRepository` (L749, **stays sync**); sink regexes L169–201; `crawl` L238.
|
||||
- **Two** production `ingestRepository` callers, both in `whitebox.ts`: `ingestRepoToSourceContext` (L125) and `runWhiteboxAnalysis` (L205) — both pass `createPythonIngestConfig`. (`server.ts:6060` calls `ingestRepoToSourceContext`, not `ingestRepository` directly.) Both must be repointed — see `whitebox-wiring`.
|
||||
|
||||
## Node ownership split (execution-offload datapoint)
|
||||
- **Owned (local:false — specialized / cross-cutting / verification):** `deps-assets`, `coverage-setup`, `grammar-registry` (+`initGrammars`), `server-init-wiring`, `ts-parse`, `ingest-wiring`, `whitebox-wiring`, `adversarial-tests`. (8)
|
||||
- **Offloaded (local:true — mechanical single-region, committed RED test):** `param-split`, `sink-patterns`, `multilang-config`, `readme-status`. (4)
|
||||
- 12 manifest nodes; the Adversarial-exploration and Strict-review **phases** below are subagent-driven gates, not manifest nodes.
|
||||
- **Retained-`local:false`, deliberate:** `nodeToCodeBlock` (pure map inside `ts-parse`) is offload-able in principle, but a discriminating RED test for it needs a real tree-sitter Node (no deterministic offline parse without the loaded grammar), so its natural RED test *is* the `ts-parse` integration test — extracting it would create a node whose test can't be cleanly pre-committed. Conscious retention per validator finding 5.
|
||||
|
||||
**Execution prerequisite:** every `local:true` node's `accept` runs a pre-committed RED test file that **no node creates** — the orchestrator commits all RED baselines to the repo *before* running the manifest (Promise-Theory: the local model never authors its own test). Running the manifest cold without this step makes those accepts fail with "no test files found."
|
||||
|
||||
## RED baselines (committed before impl; local model sees only the test)
|
||||
|
||||
**`param-split`** — `src/__tests__/param-split.test.ts` (＋discriminating same-raw case per tester)
|
||||
```ts
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { splitParamList } from '../recon/param-split.js';
|
||||
describe('splitParamList', () => {
|
||||
it('python: drops self, strips annotations/defaults/*args', () =>
|
||||
expect(splitParamList('self, url: str, timeout=5, *args', 'py')).toEqual(['url','timeout']));
|
||||
it('go name-first', () => expect(splitParamList('ctx context.Context, id string', 'go')).toEqual(['ctx','id']));
|
||||
it('java type-first', () => expect(splitParamList('String url, int max', 'java')).toEqual(['url','max']));
|
||||
it('ts name: type', () => expect(splitParamList('url: string, opts?: Opts', 'ts')).toEqual(['url','opts']));
|
||||
it('empty', () => expect(splitParamList('', 'ts')).toEqual([]));
|
||||
it('SAME raw, different lang → forces real dispatch', () => { // ← tester fix
|
||||
expect(splitParamList('a b', 'go')).toEqual(['a']); // name-first
|
||||
expect(splitParamList('a b', 'java')).toEqual(['b']); // type-first
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
**`sink-patterns`** — `src/__tests__/sink-patterns.test.ts` (scoped to 2 contiguous regexes; ＋false-positive guards)
|
||||
```ts
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { DANGEROUS_SINK_RE, OUTBOUND_REQUEST_RE } from '../recon/code-ingest.js';
|
||||
describe('cross-language sinks', () => {
|
||||
it('matches new sinks', () => {
|
||||
for (const s of ['Runtime.getRuntime().exec(cmd)','exec.Command("sh")','child_process.exec(x)','system(buf)','popen(cmd)'])
|
||||
expect(DANGEROUS_SINK_RE.test(s)).toBe(true);
|
||||
});
|
||||
it('regression: python sinks still match', () => expect(DANGEROUS_SINK_RE.test('os.system(x)')).toBe(true));
|
||||
it('no FP on benign / System.out / filesystem ids', () => { // ← tester fix
|
||||
expect(DANGEROUS_SINK_RE.test('const total = sum(a, b)')).toBe(false);
|
||||
expect(DANGEROUS_SINK_RE.test('System.out.println("debug")')).toBe(false);
|
||||
expect(DANGEROUS_SINK_RE.test('fileSystem.readFile(x)')).toBe(false);
|
||||
});
|
||||
it('outbound covers go/js http', () => {
|
||||
for (const s of ['http.Get(u)','fetch(u)','axios.get(u)']) expect(OUTBOUND_REQUEST_RE.test(s)).toBe(true);
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
**`multilang-config`** — `src/__tests__/multilang-config.test.ts` (all exts ＋negative per tester)
|
||||
```ts
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { mkdtempSync, writeFileSync } from 'node:fs';
|
||||
import { join } from 'node:path'; import { tmpdir } from 'node:os';
|
||||
import { crawl, createMultiLangIngestConfig } from '../recon/code-ingest.js';
|
||||
describe('createMultiLangIngestConfig', () => {
|
||||
const EXTS = ['py','js','ts','tsx','go','java','c','cpp'];
|
||||
it('crawls every supported ext, excludes non-source', () => {
|
||||
const root = mkdtempSync(join(tmpdir(), 'mling-'));
|
||||
for (const e of EXTS) writeFileSync(join(root, `f.${e}`), '\n');
|
||||
writeFileSync(join(root, 'd.md'), '# ignore');
|
||||
const files = crawl(createMultiLangIngestConfig(root));
|
||||
for (const e of EXTS) expect(files.some(f => f.endsWith(`f.${e}`))).toBe(true); // no degenerate hardcode
|
||||
expect(files.some(f => f.endsWith('d.md'))).toBe(false); // no wildcard/drop-filter
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
**`readme-status`** — accept grep (corrected; all three rows, "regex" word — per tester finding 2).
|
||||
|
||||
## Owned-node acceptance tests (real assertions committed up front — tester finding 5)
|
||||
- **`grammar-registry`** — `ts-grammars.test.ts`: registry has entries for all 8 exts with non-empty query + `@name`/`@def` captures; **`initGrammars` fail-open**: forcing a bad wasm path leaves the registry empty and `getParser` returns undefined, **no throw** (covers D5).
|
||||
- **`ts-parse`** — `ts-parse.test.ts`: parse a TS and a Go function → assert `name/params/lineStart/body`; unsupported non-`.py` `.zzz` → returns `[]`, no throw; `.py` → byte-identical to `parseFile`.
|
||||
- **`server-init-wiring`** — `tsc --noEmit` clean; the single bootstrap line covered by `npm run smoke` at the ship gate (`initGrammars` itself is 100% unit-tested in `grammar-registry`).
|
||||
- **`ingest-wiring`** (headline regression) — `ts-parse-multilang-rank.test.ts`: temp repo `.py`+`.go`+`.ts`+`.java`, each a sink-bearing function; `ingestRepository(createMultiLangIngestConfig(root))` (sync) → non-`.py` blocks present, classified `attack_surface`, priority > a neutral block; **PLUS a cross-language reachability case** (Go/TS entry-point → reachable callee gets the reach bonus) to guard `buildCallGraph` name-matching on non-Python ids (tester finding 5). Existing `code-ingest.test.ts` unchanged (sync preserved).
|
||||
- **`whitebox-wiring`** (production-path regression) — `whitebox-multilang.test.ts`: temp Go+TS repo through `ingestRepoToSourceContext(root)` → asserts non-`.py` source in the returned `sourceContext` (the real server/CLI entry, not the config directly); `tsc --noEmit` + `eslint` green; `whitebox-containment` tests still green.
|
||||
|
||||
## Execution Manifest
|
||||
```json
|
||||
{
|
||||
"execution-manifest": [
|
||||
{
|
||||
"id": "deps-assets",
|
||||
"files": ["package.json"],
|
||||
"change": "Add web-tree-sitter dependency and bundle prebuilt grammar .wasm assets for py/js/ts/tsx/go/java/c/cpp (pinned versions), ensuring they ship to dist. package.json + build/copy only — no source parser code here. Environmental — author yourself.",
|
||||
"accept": "npm ci >/dev/null 2>&1 && node -e \"require('web-tree-sitter')\" && test -d node_modules/web-tree-sitter",
|
||||
"local": false
|
||||
},
|
||||
{
|
||||
"id": "coverage-setup",
|
||||
"files": ["package.json", "vitest.config.ts"],
|
||||
"change": "Add @vitest/coverage-v8 devDependency and create vitest.config.ts containing ONLY a coverage block (provider v8; include the new files src/recon/param-split.ts, ts-grammars.ts, ts-parse.ts; per-file thresholds statements/branches/functions/lines = 100). Do NOT add test.include/root/environment keys — the existing zero-config `vitest run src` behavior must be byte-preserved. Owned (config).",
|
||||
"accept": "npx vitest run src >/dev/null 2>&1 && npx vitest run --coverage src/__tests__/redact.test.ts 2>&1 | grep -qiE 'coverage|% ' && test -f vitest.config.ts",
|
||||
"local": false
|
||||
},
|
||||
{
|
||||
"id": "grammar-registry",
|
||||
"files": ["src/recon/ts-grammars.ts", "src/__tests__/ts-grammars.test.ts"],
|
||||
"change": "Create ts-grammars.ts: extension->{languageId, wasmFile, query (tree-sitter S-expr with @name/@def), captures}; a wasm-path resolver; initGrammars(exts) (async, one-time, cached, try/catch -> empty registry on failure); getParser(ext) sync accessor. Specialized queries + fail-open init — author yourself. Author ts-grammars.test.ts as the RED baseline.",
|
||||
"accept": "npx vitest run src/__tests__/ts-grammars.test.ts",
|
||||
"local": false
|
||||
},
|
||||
{
|
||||
"id": "param-split",
|
||||
"files": ["src/recon/param-split.ts"],
|
||||
"change": "Create param-split.ts exporting splitParamList(raw, lang): bare param identifiers per test_param-split. Python drops self/cls, strips *args and ':'-annotations and '='-defaults; Go/TS name-first (name before ':' or space); Java type-first (name is last token of each comma part). Must branch on lang (the 'a b' case proves it). Empty -> [].",
|
||||
"accept": "npx vitest run src/__tests__/param-split.test.ts",
|
||||
"forbid": ["new_deps"],
|
||||
"kind": "create",
|
||||
"local": true
|
||||
},
|
||||
{
|
||||
"id": "ts-parse",
|
||||
"files": ["src/recon/ts-parse.ts", "src/__tests__/ts-parse.test.ts"],
|
||||
"change": "Create ts-parse.ts: nodeToCodeBlock(node, source, path) -> full CodeBlock (id, kind, lineStart/lineEnd from node position, params via splitParamList on the parameters-node text, decorators [], body via node text); parseFileMultiLang(path, content, ext): CodeBlock[] — SYNC — `.py` → parseFile; non-.py via getGrammar(ext), returning [] on no-grammar/unsupported/any error (never the Python regex). MUST bound parse time (wall-clock via progressCallback) so a pathological input cannot spin — on timeout, treat as parse failure and return []. Integration crux — author yourself.",
|
||||
"accept": "npx vitest run src/__tests__/ts-parse.test.ts",
|
||||
"local": false
|
||||
},
|
||||
{
|
||||
"id": "sink-patterns",
|
||||
"files": ["src/recon/code-ingest.ts"],
|
||||
"change": "Additively extend ONLY the two contiguous consts DANGEROUS_SINK_RE and OUTBOUND_REQUEST_RE (L173-179) with cross-language sinks (Java Runtime.exec/ProcessBuilder, Go exec.Command/http.Get, C system(/popen(, JS child_process/fetch/axios) per test_sink-patterns, WITHOUT dropping existing Python patterns and WITHOUT matching benign ids like System.out/fileSystem. Export both consts for the test. (SINK_EVIDENCE_RES enrichment deferred to a follow-up — keeps this a single region.)",
|
||||
"accept": "npx vitest run src/__tests__/sink-patterns.test.ts",
|
||||
"forbid": ["new_deps"],
|
||||
"local": true
|
||||
},
|
||||
{
|
||||
"id": "multilang-config",
|
||||
"files": ["src/recon/code-ingest.ts"],
|
||||
"change": "Add createMultiLangIngestConfig(repoRoot): IngestConfig mirroring createPythonIngestConfig with includeExts covering exactly py/js/ts/tsx/go/java/c/cpp (leading dots), per test_multilang-config (all included, .md excluded).",
|
||||
"accept": "npx vitest run src/__tests__/multilang-config.test.ts",
|
||||
"forbid": ["new_deps"],
|
||||
"local": true
|
||||
},
|
||||
{
|
||||
"id": "server-init-wiring",
|
||||
"files": ["src/server.ts"],
|
||||
"change": "Invoke `await initGrammars(SUPPORTED_EXTS)` ONCE at server bootstrap (before app.listen) so grammars are loaded before any ingest request; on init failure log and continue (registry empty -> .py via parseFile, other languages -> []). server.ts is the ONLY ingest entrypoint (6060/6126); do NOT wire cli.ts (the CLI never ingests). Cross-cutting entrypoint — author yourself.",
|
||||
"accept": "npx tsc --noEmit",
|
||||
"local": false
|
||||
},
|
||||
{
|
||||
"id": "ingest-wiring",
|
||||
"files": ["src/recon/code-ingest.ts", "src/__tests__/ts-parse-multilang-rank.test.ts"],
|
||||
"change": "In ingestRepository (KEEP SYNC), dispatch each file through parseFileMultiLang(path, content, ext) (language-scoped fail-open: .py -> parseFile, other languages -> []), preserving byte/file ceilings. Author the headline multilang-rank regression test (non-.py blocks extracted + attack_surface ranked + cross-language reachability). Do NOT change ingestRepository's signature or any caller. Author yourself.",
|
||||
"accept": "npx vitest run src/__tests__/ts-parse-multilang-rank.test.ts src/__tests__/code-ingest.test.ts",
|
||||
"local": false
|
||||
},
|
||||
{
|
||||
"id": "whitebox-wiring",
|
||||
"files": ["src/recon/whitebox.ts", "src/__tests__/whitebox-multilang.test.ts"],
|
||||
"change": "BLOCKING production wiring: in whitebox.ts repoint BOTH ingestRepository calls (ingestRepoToSourceContext L125 [sync], runWhiteboxAnalysis L205 [already async — leave async]) from createPythonIngestConfig to createMultiLangIngestConfig — a pure argument swap, no signature change. Author the regression test whitebox-multilang.test.ts driving a temp Go+TS repo through ingestRepoToSourceContext and asserting non-.py source appears in the returned sourceContext (proves the feature reaches production, not just tests). Verify existing whitebox-containment tests stay green. Author yourself.",
|
||||
"accept": "npx vitest run src/__tests__/whitebox-multilang.test.ts src/__tests__/whitebox-containment.test.ts && npx tsc --noEmit && npm run lint",
|
||||
"local": false
|
||||
},
|
||||
{
|
||||
"id": "adversarial-tests",
|
||||
"files": ["src/__tests__/ts-parse-adversarial.test.ts"],
|
||||
"change": "Author a hostile-input suite exercising the FULL wired path (parseFileMultiLang + ingestRepository): malformed/syntactically-broken source per language; a .go file containing Python (and vice-versa); binary/non-UTF8 bytes; a multi-MB generated file and a deeply-nested/very-long-line file (byte/file ceilings hold); a PATHOLOGICAL input that would make tree-sitter spin (asserts parse returns within the timeout and fail-opens — proves the ts-parse timeout is real, not aspirational); a symlink and a symlink-loop under the repo root (crawl must not escape the contained root or hang); empty file; unicode identifiers; a file that makes tree-sitter error. Assert: never throws (non-.py fail-opens to [], .py to parseFile), bounded wall-time, CodeBlock invariants hold (lineStart<=lineEnd, body non-empty when name present, params is string[]), ingest resolves. Owned (crafted adversarial inputs).",
|
||||
"accept": "npx vitest run src/__tests__/ts-parse-adversarial.test.ts",
|
||||
"local": false
|
||||
},
|
||||
{
|
||||
"id": "readme-status",
|
||||
"files": ["README.md"],
|
||||
"change": "Flip the two rows that literally say 'Python-only' to multi-language (web-tree-sitter), Python-regex fail-open noted: the 'What it hunts' row (⚠️ Python-only ingest, ~L56) and the 'What ships today' row (Python-only regex ingest, ~L115). Leave the 'Coverage by domain' Code row (~L130, 'experimental') alone — it is a maturity axis, not the Python-only axis, and stays honest until the PR proves out. (Multi-region doc edit — the single-region rule is waived for docs.)",
|
||||
"accept": "grep -qi 'web-tree-sitter' README.md && ! grep -qi 'Python-only regex ingest' README.md && ! grep -qi 'Python-only ingest' README.md",
|
||||
"forbid": ["new_deps"],
|
||||
"local": true
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Risk Assessment (post-review)
|
||||
- ~~[High] async ripple~~ → **eliminated** by the sync + bootstrap-init architecture.
|
||||
- **[Med] grammar wasm sourcing/version** — owned `deps-assets`; pin versions; the fail-open init means a bad bundle preserves Python ingest while non-Python files yield `[]`, not a crash.
|
||||
- **[Med] cross-language call-graph** — `buildCallGraph` name-matching may mis-handle non-Python ids; guarded by the reachability assertion in the headline test.
|
||||
- **[Low] sink-regex FP** — guarded by `System.out`/`fileSystem` negatives.
|
||||
- **[Low] bootstrap race** — pre-init-completion ingest preserves Python parsing while non-Python files yield `[]`.
|
||||
|
||||
## Rollout — gated phases (each can bounce back)
|
||||
1. **Build.** Branch `feat/multilang-ingest`; land the 12 nodes in manifest order (fleet: `start-fleet.sh`). I implement the 8 owned; cascade fills the 4 local.
|
||||
2. **Coverage gate (hard).** `npx vitest run --coverage` → **100%** on the new files (thresholds fail the run otherwise); manually confirm every new branch/function in `code-ingest.ts`/`whitebox.ts` is hit (diff-coverage), carve-out only the `server.ts`/`cli.ts` bootstrap lines (smoke-covered). Gap → add tests, re-run.
|
||||
3. **Adversarial phase.** `ts-parse-adversarial.test.ts` green, then dispatch a **`security-test-engineer`** subagent to actively break the built code against a hostile-repo corpus (malformed/mixed-language/binary/pathological inputs, resource-exhaustion, parser hangs). Every defect fixed; re-run coverage.
|
||||
4. **Strict review gate.** Apply the **`/code-review-pr-strict`** methodology via parallel specialist subagents on the FULL diff — **security-critical detection runs first**, then correctness/perf/maintainability, with false-positive elimination and the mandatory verification protocol. Confirm coverage is meaningful assertions (no line-touching). Resolve every confirmed finding; **re-review until clean**.
|
||||
5. **Ship.** `npm test` + `tsc --noEmit` + `eslint` + `npm run smoke` (covers the server bootstrap init line) + coverage all green → open PR; body carries spec §4 decision record, the node split, coverage report, adversarial summary, and review sign-off.
|
||||
- Follow-ups (own issues): SINK_EVIDENCE_RES cross-language enrichment; cxpak `serve` phase-2 ranking upgrade.
|
||||
|
||||
## Success Criteria
|
||||
- [ ] Non-`.py` functions extracted + security-ranked (headline regression green, incl. cross-lang reachability).
|
||||
- [ ] **Feature reaches production** — `whitebox-multilang.test.ts` drives multilang through `ingestRepoToSourceContext` (not just `createMultiLangIngestConfig` directly).
|
||||
- [ ] Fail-open verified at BOTH boundaries: init failure (empty registry) and per-file unsupported ext.
|
||||
- [ ] Python behavior byte-unchanged (existing suite green, no test edits).
|
||||
- [ ] Both README "Python-only" rows flipped. `tsc`/`eslint`/`vitest` all green (eslint gated in `whitebox-wiring`).
|
||||
- [ ] **100% coverage** on new files + all new branches in edited files (carve-out: bootstrap lines) — coverage gate green.
|
||||
- [ ] **Adversarial phase** passed: crafted-input suite green AND `security-test-engineer` subagent found no unfixed break.
|
||||
- [ ] **Strict review** (`/code-review-pr-strict`, security-first) clean — all confirmed findings resolved, assertions verified meaningful.
|
||||
|
||||
---
|
||||
**Status:** Draft — pending human sign-off
|
||||
**Date:** 2026-07-10
|
||||
@@ -0,0 +1,124 @@
|
||||
# Multi-language white-box ingest for T3MP3ST
|
||||
|
||||
**Status:** Draft design (for an upstream PR to `elder-plinius/T3MP3ST`)
|
||||
**Date:** 2026-07-10
|
||||
**Scope:** Replace the Python-only ingest parser with a portable multi-language parser, reusing T3MP3ST's existing security-ranking pipeline unchanged.
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem
|
||||
|
||||
T3MP3ST markets white-box source hunting as a proven strength ("Code: ✅ Proven — CVE-Zero 8/10 across 7 languages"), but the ingest **engine** only reads Python. `src/recon/code-ingest.ts:6` says so itself:
|
||||
|
||||
> "This is a PYTHON-ONLY, regex-based 'AST-lite' PROTOTYPE … The real version is tree-sitter (or a language server) + multi-language."
|
||||
|
||||
The README status table flags it: **Code → ⚠️ Python-only ingest**. Point the white-box flow at a Go/Java/C/TS repo and `crawl` keeps only `.py`; every non-Python file is invisible to the hunt.
|
||||
|
||||
## 2. Goal / Non-goals
|
||||
|
||||
**Goal.** White-box ingest parses many languages, feeding T3MP3ST's *existing* security-ranking so multi-language repos get the same attack-surface-ordered analysis Python repos get today.
|
||||
|
||||
**Non-goals.**
|
||||
- Not changing the security model (`classify`/`prioritize`/`reachability`/`context-pack`) — it is reused verbatim.
|
||||
- Not adding an external binary, native-compiled dependency, or new API key.
|
||||
- Not building cross-language taint/call-graph analysis in this PR (see §9, phase 2).
|
||||
|
||||
## 3. Key insight — the seam is one stage
|
||||
|
||||
The ingest is a pipeline (`whitebox.ts`): `code-ingest (SELECT) → context-pack (BUDGET) → orchestrator (REASON)`. Within SELECT, only **one** stage is language-bound:
|
||||
|
||||
- `parseFile()` (`code-ingest.ts:409`) — Python `def`/`class` regex → `CodeBlock[]`. **Language-bound.**
|
||||
- Everything downstream operates on `CodeBlock {name, params, body, line, …}` via text regexes over the block body/name/params:
|
||||
- `buildCallGraph`, `findEntryPoints`, `reachability` — walk `CodeBlock` names. Language-agnostic.
|
||||
- `classify` (per-block `Exposure`) and `prioritize` (score) — run `DANGEROUS_SINK_RE`, `RISKY_PARAM_RE`, `OUTBOUND_REQUEST_RE`, `SECURITY_CONTROL_NAME_RE` over the block **body/params**. Mechanism is language-agnostic.
|
||||
|
||||
So the change is surgical: **replace `parseFile` with a multi-language block extractor that emits the same `CodeBlock` shape.** No downstream code changes.
|
||||
|
||||
## 4. Decision record — how to parse (embedded ADR; T3MP3ST has no ADR dir)
|
||||
|
||||
**Decision: `web-tree-sitter` (WASM), as the default on every platform.**
|
||||
|
||||
Options considered:
|
||||
|
||||
| Option | Nature | Rejected because |
|
||||
|---|---|---|
|
||||
| Extend regex per language | fragile per-language regex | The exact trap the file header warns against; unmaintainable across 40 grammars. |
|
||||
| `tree-sitter` (node-tree-sitter) | **native** N-API C++ addon; fastest | Compiles native code at install (node-gyp/toolchain) or needs per-OS/ABI prebuilds. T3MP3ST has **zero** native deps today and ships via `npx tempest` / `npm install` onto arbitrary machines, **fully offline**, cross-platform (open Windows bug #18). Native compile is the #1 install-failure mode in exactly that scenario. |
|
||||
| Per-language JS parsers (`@babel/parser`…) | pure JS, one language each | Cannot cover many languages. |
|
||||
| Shell out to cxpak (external Rust tool) | multi-language, but external | Adds a binary install (brew/cargo/docker); and cxpak exposes **no** `{name, params, body, span}` symbol interface (CLI = markdown prose; `serve` = analyses, not a symbol table; `lsp` = MCP-over-stdio, no `documentSymbol`) — verified. Cannot cheaply supply `CodeBlock`s. |
|
||||
| **`web-tree-sitter` (WASM)** | official WASM build of tree-sitter | **Chosen.** |
|
||||
|
||||
**Rationale.** web-tree-sitter is the WASM sibling of the native binding — same grammars, same parse quality — with **zero compile step**, running identically on any OS/Node/offline. It preserves T3MP3ST's pure-JS, portable, keyless character. The native binding's ~2–5× parse-speed edge is worthless here: ingest parses a repo **once per mission**, dwarfed by LLM latency. Grammar `.wasm` files are sourced from a prebuilt collection (e.g. `tree-sitter-wasms`; exact package/coverage confirmed at implementation time), not built locally.
|
||||
|
||||
**Consequence / accepted cost.** Async grammar init; a few MB of bundled wasm for the top languages (lazy-loaded by extension); slightly more setup code than a `require()` grammar.
|
||||
|
||||
## 5. Architecture
|
||||
|
||||
```
|
||||
crawl (broadened includeExts)
|
||||
→ parseFileMultiLang(path, content) ← NEW: web-tree-sitter → CodeBlock[]
|
||||
├─ ext == .py → parseFile() (existing Python regex, unchanged)
|
||||
├─ non-.py, grammar loaded → tree-sitter query → CodeBlock[]
|
||||
└─ non-.py, no grammar / timeout / err → [] (never the Python regex) ← fail-open
|
||||
→ buildCallGraph → findEntryPoints → reachability → classify → prioritize (UNCHANGED)
|
||||
→ context-pack (BUDGET) → orchestrator (REASON) (UNCHANGED)
|
||||
```
|
||||
|
||||
**SYNC architecture (revised — grammar init hoisted to bootstrap so the ingest path stays synchronous).** web-tree-sitter's `Parser.init()`/`Language.load()` are async (one-time); `parser.parse()` is sync. So init runs once at bootstrap and the per-file parse is sync — `ingestRepository` keeps its sync signature and there is no async ripple.
|
||||
|
||||
**New units:**
|
||||
- `src/recon/ts-grammars.ts` — `initGrammars(exts): Promise<void>` (one-time, cached, `try/catch` → empty registry on failure = fail-open); `getParser(ext): Parser | undefined` (sync accessor); the ext→{query, captures, wasm} registry.
|
||||
- `src/recon/ts-parse.ts` — `parseFileMultiLang(path, content, ext): CodeBlock[]` (**sync**): `.py` → `parseFile()`; non-`.py` with a loaded grammar → `parser.parse()` (with a **bounded parse time** — a hostile input must not spin) → query → `nodeToCodeBlock`; non-`.py` with no grammar / timeout / error → `[]` (never the Python regex — routing brace/`class` syntax through it would emit corrupt blocks). `nodeToCodeBlock` fills the full `CodeBlock` shape (`name` from `@name`, `body`/`lineStart`/`lineEnd` from node position, `params` via `splitParamList(text, lang)`, `decorators []`, `kind`).
|
||||
- Per-language query table (top ~8: `py, js, ts, tsx, go, java, c, cpp`). cxpak's grammar queries are a reference for the capture patterns.
|
||||
|
||||
**Wiring.** (1) Bootstrap: `await initGrammars(...)` once at **`server.ts` startup** (the only ingest entrypoint — `server.ts:6060/6126`; the CLI never ingests, so `cli.ts` is not wired). (2) `ingestRepository()` (`code-ingest.ts:749`, **stays sync**) dispatches each file through `parseFileMultiLang(...)` (language-scoped fail-open: `.py` → `parseFile`, other languages → `[]`). (3) **Production caller swap (required):** both `whitebox.ts` callers of `ingestRepository` — `ingestRepoToSourceContext` (L125) and `runWhiteboxAnalysis` (L205) — move from `createPythonIngestConfig` to `createMultiLangIngestConfig`; without this the extractor only sees `.py` in production.
|
||||
|
||||
## 6. Security-ranking reuse & cross-language sinks
|
||||
|
||||
`classify`/`prioritize` are reused unchanged. Their regexes (`DANGEROUS_SINK_RE`, etc.) are Python-tuned; cross-language sinks that already overlap (`eval(`, `exec(`, `system(`) still fire, so ranking **degrades gracefully**, never breaks. This PR **additively** broadens the sink/param patterns with a few cross-language entries (`Runtime.exec`, `ProcessBuilder`, `exec.Command`, `os/exec`, `system(`, `popen(`, `child_process`), each with a test. Deeper per-language tuning is out of scope (incremental follow-up).
|
||||
|
||||
## 7. Fail-open
|
||||
|
||||
- `.py` always → `parseFile` (Python files unaffected). A non-`.py` file with a missing/unsupported/failed grammar → `[]` (honest absence) — **never** the Python regex, which would mis-parse brace/`class` syntax into corrupt blocks (wrong line spans, lost methods). This language-scoping is part of the security boundary.
|
||||
- web-tree-sitter init throws in a hostile env → the registry stays empty; `.py` still ingests via the Python regex, every other language yields `[]` (and a one-time `console.warn`). **Ingest never crashes a mission.**
|
||||
|
||||
## 8. Test plan (vitest — matches their suite; deterministic, no model calls)
|
||||
|
||||
1. **Multi-language extraction (headline regression).** Temp repo (`mkdtempSync`, mirroring `code-ingest.test.ts`) with `.py` + `.go` + `.ts` files, each holding a function with a known sink. Assert non-`.py` functions are extracted as `CodeBlock`s **and** land as `attack_surface`/high-priority after `classify`/`prioritize`. This test *is* the proof the limitation is fixed.
|
||||
2. **CodeBlock fidelity.** For a known TS/Go function, assert `name`, `params`, `line`, and `body` (contains the sink line) are correct.
|
||||
3. **Fail-open — unsupported ext.** A `.zig` (or grammar-absent, non-`.py`) file returns `[]`, no throw (a `.py` file still routes to `parseFile`).
|
||||
4. **Fail-open — grammar init failure.** Force `initGrammars` to fail (bad wasm path) → registry stays empty, `getGrammar` returns undefined, `.py` routes to `parseFile` and every other language returns `[]`; `ingestRepository` resolves normally, mission-safe.
|
||||
5. **Cross-language sink patterns.** Unit-test each added regex against a positive and negative snippet.
|
||||
6. *(optional, `--ignored` like `spine-live.test.ts`)* real wasm load against a fixture repo — live smoke.
|
||||
|
||||
**Acceptance:** `npm test` green, `npm run typecheck` clean; test 1 proves multi-language, tests 3–4 prove fail-open.
|
||||
|
||||
## 9. PR scope
|
||||
|
||||
**In:** `ts-grammars.ts` (registry + `initGrammars`/`getParser`) + `ts-parse.ts` (`parseFileMultiLang`/`nodeToCodeBlock`) + `param-split.ts`; `web-tree-sitter` dep + grammar wasm assets (top ~8 langs); bootstrap `initGrammars` call (server/CLI); `ingestRepository`/`crawl` wiring (**sync**, broadened exts) + `createMultiLangIngestConfig`; **`whitebox.ts` caller swap (both call sites)**; additive cross-language sink patterns; tests; README status flip (both "Python-only" rows).
|
||||
|
||||
**Out (name in PR as deliberate):** cxpak integration (phase 2); deep per-language sink tuning; the `serve`-backed cross-language call-graph/taint upgrade; node-tree-sitter fast path.
|
||||
|
||||
**Size target:** 3 new source files (`param-split`, `ts-grammars`, `ts-parse`) + edits to 4 (`code-ingest`, `whitebox`, `server`, `cli`) + coverage config + test suites (unit, headline regression, production-path, adversarial). Larger than a trivial PR because of the mandated quality gates (§12), but still one cohesive feature — deliberately not a mega-diff (cf. #61/#17, closed for sprawl).
|
||||
|
||||
## 10. Phase 2 (noted, not this PR) — cxpak as an optional ranking upgrade
|
||||
|
||||
When installed, cxpak's `serve` endpoints supply genuinely better multi-language facts than T3MP3ST's regex heuristics: `/call_graph` (cross-language edges), `/api_surface` (entry points with file+line), `/data_flow` (taint paths). A later, fail-open, opt-in PR can let cxpak *upgrade* the ranking inputs (entry points, reachability, sinks) while the web-tree-sitter parser from phase 1 remains the universal default. Not required for the headline win.
|
||||
|
||||
## 11. Risks
|
||||
|
||||
- **Grammar wasm sourcing/versioning** — confirm the collection package and language coverage at implementation time; pin versions.
|
||||
- **production caller swap** — the multilang config must be repointed at BOTH `whitebox.ts` `ingestRepository` callers (L125, L205); a regression test drives `ingestRepoToSourceContext` end-to-end so the feature can't silently no-op in production. *(The async ripple is designed out by the sync + bootstrap-init architecture.)*
|
||||
- **bootstrap race** — a mission firing before `initGrammars` resolves still parses Python through `parseFile`; non-Python files yield `[]` for that call. Entrypoints `await` init before serving.
|
||||
- **Param extraction variance** across grammars — mitigated by falling back to slicing the signature line when the parameters node is absent.
|
||||
- **Bundle size** — mitigated by lazy-loading grammars per encountered extension rather than eager-loading all.
|
||||
|
||||
## 12. Quality gates (mandatory — all three block merge)
|
||||
|
||||
1. **100% coverage of our code.** vitest coverage (`@vitest/coverage-v8`, `vitest.config.ts` with a coverage-only block) with per-file thresholds = 100% (statements/branches/functions/lines), **CI-enforced** via `npm run test:coverage`, on every NEW file (`param-split.ts`, `ts-grammars.ts`, `ts-parse.ts`). The new branches/functions added to edited files (`code-ingest.ts` sink patterns + `createMultiLangIngestConfig` + dispatch, both `whitebox.ts` caller swaps) are **covered by targeted regression tests** (`sink-patterns`, `ts-parse-multilang-rank`, `whitebox-multilang` — including a spy test on the `runWhiteboxAnalysis` swap so it can't silently revert), but not per-file-threshold-gated: a whole-file 100% threshold is unachievable on large pre-existing files. Sole coverage carve-out: the single `await initGrammars(...)` wiring line at `server.ts` bootstrap (`initGrammars` itself is 100% unit-tested; the line is `npm run smoke`-covered). Coverage is necessary-not-sufficient — gate 3 verifies the covering assertions are meaningful (no line-touching / cheater tests).
|
||||
|
||||
2. **Adversarial local testing.** Because ingest parses **untrusted target source**, a crafted hostile-input suite (`ts-parse-adversarial.test.ts`) drives the full wired path against malformed/mixed-language/binary/oversized inputs, a **pathological input that would make tree-sitter spin** (asserting the `ts-parse` parse-timeout fail-opens — not aspirational), and **symlink/loop entries** (crawl must not escape the contained root or hang); asserts fail-open + CodeBlock invariants + bounded wall-time/bytes. Then a `security-test-engineer` subagent actively attempts to break the built code (parser hangs, resource exhaustion, mission crash) against a hostile-repo corpus. All defects fixed before review.
|
||||
|
||||
3. **Strict subagent code review.** The `/code-review-pr-strict` methodology on the full diff: parallel specialist subagents with **security-critical detection first**, false-positive elimination, and the mandatory verification protocol. Every confirmed finding resolved; re-review until clean. This is the merge gate.
|
||||
|
||||
Gate order at rollout: build → coverage → adversarial → strict review → ship; any gate can bounce back to build.
|
||||
Generated
+232
-1
@@ -23,7 +23,9 @@
|
||||
"gradient-string": "^2.0.2",
|
||||
"inquirer": "^9.2.15",
|
||||
"ora": "^8.0.1",
|
||||
"tsx": "^4.7.0"
|
||||
"tree-sitter-wasms": "0.1.13",
|
||||
"tsx": "^4.7.0",
|
||||
"web-tree-sitter": "0.25.10"
|
||||
},
|
||||
"bin": {
|
||||
"t3mp3st": "dist/cli.js",
|
||||
@@ -37,6 +39,7 @@
|
||||
"@types/gradient-string": "^1.1.5",
|
||||
"@types/inquirer": "^9.0.7",
|
||||
"@types/node": "^20.11.0",
|
||||
"@vitest/coverage-v8": "4.1.9",
|
||||
"eslint": "^9.39.2",
|
||||
"typescript": "^5.3.3",
|
||||
"typescript-eslint": "^8.51.0",
|
||||
@@ -46,6 +49,66 @@
|
||||
"node": ">=18.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@babel/helper-string-parser": {
|
||||
"version": "7.29.7",
|
||||
"resolved": "https://registry.npmjs.org/@babel/helper-string-parser/-/helper-string-parser-7.29.7.tgz",
|
||||
"integrity": "sha512-Pb5ijPrZ89GDH8223L4UP8i6QApWxs04RbPQJTeWDV0/keR2E36MeKnyr6LYmUUvqRRI+Iv87SuF1W6ErINzYw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=6.9.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@babel/helper-validator-identifier": {
|
||||
"version": "7.29.7",
|
||||
"resolved": "https://registry.npmjs.org/@babel/helper-validator-identifier/-/helper-validator-identifier-7.29.7.tgz",
|
||||
"integrity": "sha512-qehxGkRj55h/ff8EMaJ+cYhyaKlHIxqYDn682wQD7RNp9UujOQsHog2uS0r2vzr4pW+sXf90NeeayjcNaX3fFg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=6.9.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@babel/parser": {
|
||||
"version": "7.29.7",
|
||||
"resolved": "https://registry.npmjs.org/@babel/parser/-/parser-7.29.7.tgz",
|
||||
"integrity": "sha512-hnORnjP/1P/zFEndoeX+n+t1RwWRJiJpM/jO7FW32Kn9r5+sJB2JWOdYo4L6k78j15eCwY3Gm/7364B1EMwtNg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@babel/types": "^7.29.7"
|
||||
},
|
||||
"bin": {
|
||||
"parser": "bin/babel-parser.js"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=6.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@babel/types": {
|
||||
"version": "7.29.7",
|
||||
"resolved": "https://registry.npmjs.org/@babel/types/-/types-7.29.7.tgz",
|
||||
"integrity": "sha512-4zBIxpPzowiZpusoFkyGVwakdRJUyuH5PxQ/PrqghfdFWWasvnCdPfQXHrenDai+gyLARulZjZowCOj6fjT4pA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@babel/helper-string-parser": "^7.29.7",
|
||||
"@babel/helper-validator-identifier": "^7.29.7"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=6.9.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@bcoe/v8-coverage": {
|
||||
"version": "1.0.2",
|
||||
"resolved": "https://registry.npmjs.org/@bcoe/v8-coverage/-/v8-coverage-1.0.2.tgz",
|
||||
"integrity": "sha512-6zABk/ECA/QYSCQ1NGiVwwbQerUCZ+TQbp64Q3AgmfNvurHH0j8TtXa1qbShXA6qqkpAj4V5W8pP6mLe1mcMqA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@colors/colors": {
|
||||
"version": "1.5.0",
|
||||
"resolved": "https://registry.npmjs.org/@colors/colors/-/colors-1.5.0.tgz",
|
||||
@@ -768,6 +831,16 @@
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@jridgewell/resolve-uri": {
|
||||
"version": "3.1.2",
|
||||
"resolved": "https://registry.npmjs.org/@jridgewell/resolve-uri/-/resolve-uri-3.1.2.tgz",
|
||||
"integrity": "sha512-bRISgCIjP20/tbWSPWMEi54QVPRZExkuD9lJL+UIxUKtwVJA8wW1Trb1jMs1RFXo1CBTNZ/5hpC9QvmKWdopKw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=6.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@jridgewell/sourcemap-codec": {
|
||||
"version": "1.5.5",
|
||||
"resolved": "https://registry.npmjs.org/@jridgewell/sourcemap-codec/-/sourcemap-codec-1.5.5.tgz",
|
||||
@@ -775,6 +848,17 @@
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@jridgewell/trace-mapping": {
|
||||
"version": "0.3.31",
|
||||
"resolved": "https://registry.npmjs.org/@jridgewell/trace-mapping/-/trace-mapping-0.3.31.tgz",
|
||||
"integrity": "sha512-zzNR+SdQSDJzc8joaeP8QQoCQr8NuYx2dIIytl1QeBEZHJ9uW6hebsrYgbz8hJwUQao3TWCMtmfV8Nu1twOLAw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@jridgewell/resolve-uri": "^3.1.0",
|
||||
"@jridgewell/sourcemap-codec": "^1.4.14"
|
||||
}
|
||||
},
|
||||
"node_modules/@modelcontextprotocol/sdk": {
|
||||
"version": "1.29.0",
|
||||
"resolved": "https://registry.npmjs.org/@modelcontextprotocol/sdk/-/sdk-1.29.0.tgz",
|
||||
@@ -1864,6 +1948,37 @@
|
||||
"url": "https://opencollective.com/typescript-eslint"
|
||||
}
|
||||
},
|
||||
"node_modules/@vitest/coverage-v8": {
|
||||
"version": "4.1.9",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/coverage-v8/-/coverage-v8-4.1.9.tgz",
|
||||
"integrity": "sha512-G9/lgqibheLVBDRuya45EbsEXTYcWoSG+TLg7i2axuzx0Eq62eXn+aWXyaVdV5vKvFSWd6ywcX8hA7la9Pvu8g==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@bcoe/v8-coverage": "^1.0.2",
|
||||
"@vitest/utils": "4.1.9",
|
||||
"ast-v8-to-istanbul": "^1.0.0",
|
||||
"istanbul-lib-coverage": "^3.2.2",
|
||||
"istanbul-lib-report": "^3.0.1",
|
||||
"istanbul-reports": "^3.2.0",
|
||||
"magicast": "^0.5.2",
|
||||
"obug": "^2.1.1",
|
||||
"std-env": "^4.0.0-rc.1",
|
||||
"tinyrainbow": "^3.1.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://opencollective.com/vitest"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@vitest/browser": "4.1.9",
|
||||
"vitest": "4.1.9"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"@vitest/browser": {
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"node_modules/@vitest/expect": {
|
||||
"version": "4.1.9",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-4.1.9.tgz",
|
||||
@@ -2134,6 +2249,18 @@
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/ast-v8-to-istanbul": {
|
||||
"version": "1.0.4",
|
||||
"resolved": "https://registry.npmjs.org/ast-v8-to-istanbul/-/ast-v8-to-istanbul-1.0.4.tgz",
|
||||
"integrity": "sha512-0bC0/4bTSrnwdhU3IsZDwEdojvuPrSg59OYZfKsLRtJZ0u8VBx9DebfqqG8bRdCC0I7vjgxmPi41P0lpkhJHtA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@jridgewell/trace-mapping": "^0.3.31",
|
||||
"estree-walker": "^3.0.3",
|
||||
"js-tokens": "^10.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/atomically": {
|
||||
"version": "2.1.0",
|
||||
"resolved": "https://registry.npmjs.org/atomically/-/atomically-2.1.0.tgz",
|
||||
@@ -3600,6 +3727,13 @@
|
||||
"node": ">=16.9.0"
|
||||
}
|
||||
},
|
||||
"node_modules/html-escaper": {
|
||||
"version": "2.0.2",
|
||||
"resolved": "https://registry.npmjs.org/html-escaper/-/html-escaper-2.0.2.tgz",
|
||||
"integrity": "sha512-H2iMtd0I4Mt5eYiapRdIDjp+XzelXQ0tFE4JS7YFwFevXXMmOp9myNrUvCg0D6ws8iqkRPBfKHgbwig1SmlLfg==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/http-errors": {
|
||||
"version": "2.0.1",
|
||||
"resolved": "https://registry.npmjs.org/http-errors/-/http-errors-2.0.1.tgz",
|
||||
@@ -3988,6 +4122,45 @@
|
||||
"integrity": "sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw==",
|
||||
"license": "ISC"
|
||||
},
|
||||
"node_modules/istanbul-lib-coverage": {
|
||||
"version": "3.2.2",
|
||||
"resolved": "https://registry.npmjs.org/istanbul-lib-coverage/-/istanbul-lib-coverage-3.2.2.tgz",
|
||||
"integrity": "sha512-O8dpsF+r0WV/8MNRKfnmrtCWhuKjxrq2w+jpzBL5UZKTi2LeVWnWOmWRxFlesJONmc+wLAGvKQZEOanko0LFTg==",
|
||||
"dev": true,
|
||||
"license": "BSD-3-Clause",
|
||||
"engines": {
|
||||
"node": ">=8"
|
||||
}
|
||||
},
|
||||
"node_modules/istanbul-lib-report": {
|
||||
"version": "3.0.1",
|
||||
"resolved": "https://registry.npmjs.org/istanbul-lib-report/-/istanbul-lib-report-3.0.1.tgz",
|
||||
"integrity": "sha512-GCfE1mtsHGOELCU8e/Z7YWzpmybrx/+dSTfLrvY8qRmaY6zXTKWn6WQIjaAFw069icm6GVMNkgu0NzI4iPZUNw==",
|
||||
"dev": true,
|
||||
"license": "BSD-3-Clause",
|
||||
"dependencies": {
|
||||
"istanbul-lib-coverage": "^3.0.0",
|
||||
"make-dir": "^4.0.0",
|
||||
"supports-color": "^7.1.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=10"
|
||||
}
|
||||
},
|
||||
"node_modules/istanbul-reports": {
|
||||
"version": "3.2.0",
|
||||
"resolved": "https://registry.npmjs.org/istanbul-reports/-/istanbul-reports-3.2.0.tgz",
|
||||
"integrity": "sha512-HGYWWS/ehqTV3xN10i23tkPkpH46MLCIMFNCaaKNavAXTF1RkqxawEPtnjnGZ6XKSInBKkiOA5BKS+aZiY3AvA==",
|
||||
"dev": true,
|
||||
"license": "BSD-3-Clause",
|
||||
"dependencies": {
|
||||
"html-escaper": "^2.0.0",
|
||||
"istanbul-lib-report": "^3.0.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=8"
|
||||
}
|
||||
},
|
||||
"node_modules/jose": {
|
||||
"version": "6.1.3",
|
||||
"resolved": "https://registry.npmjs.org/jose/-/jose-6.1.3.tgz",
|
||||
@@ -3997,6 +4170,13 @@
|
||||
"url": "https://github.com/sponsors/panva"
|
||||
}
|
||||
},
|
||||
"node_modules/js-tokens": {
|
||||
"version": "10.0.0",
|
||||
"resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-10.0.0.tgz",
|
||||
"integrity": "sha512-lM/UBzQmfJRo9ABXbPWemivdCW8V2G8FHaHdypQaIy523snUjog0W71ayWXTjiR+ixeMyVHN2XcpnTd/liPg/Q==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/js-yaml": {
|
||||
"version": "4.3.0",
|
||||
"resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-4.3.0.tgz",
|
||||
@@ -4392,6 +4572,34 @@
|
||||
"@jridgewell/sourcemap-codec": "^1.5.5"
|
||||
}
|
||||
},
|
||||
"node_modules/magicast": {
|
||||
"version": "0.5.3",
|
||||
"resolved": "https://registry.npmjs.org/magicast/-/magicast-0.5.3.tgz",
|
||||
"integrity": "sha512-pVKE4UdSQ7DvHzivsCIFx2BJn1mHG6KsyrFcaxFx6tONdneEuThrDx0Cj3AMg58KyN4pzYT+LHOotxDQDjNvkw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@babel/parser": "^7.29.3",
|
||||
"@babel/types": "^7.29.0",
|
||||
"source-map-js": "^1.2.1"
|
||||
}
|
||||
},
|
||||
"node_modules/make-dir": {
|
||||
"version": "4.0.0",
|
||||
"resolved": "https://registry.npmjs.org/make-dir/-/make-dir-4.0.0.tgz",
|
||||
"integrity": "sha512-hXdUTZYIVOt1Ex//jAQi+wTZZpUpwBj/0QsOzqegb3rGMMeJiSEu5xLHnYfBrRV4RH2+OCSOO95Is/7x1WJ4bw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"semver": "^7.5.3"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=10"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/sindresorhus"
|
||||
}
|
||||
},
|
||||
"node_modules/math-intrinsics": {
|
||||
"version": "1.1.0",
|
||||
"resolved": "https://registry.npmjs.org/math-intrinsics/-/math-intrinsics-1.1.0.tgz",
|
||||
@@ -5517,6 +5725,15 @@
|
||||
"node": ">=0.6"
|
||||
}
|
||||
},
|
||||
"node_modules/tree-sitter-wasms": {
|
||||
"version": "0.1.13",
|
||||
"resolved": "https://registry.npmjs.org/tree-sitter-wasms/-/tree-sitter-wasms-0.1.13.tgz",
|
||||
"integrity": "sha512-wT+cR6DwaIz80/vho3AvSF0N4txuNx/5bcRKoXouOfClpxh/qqrF4URNLQXbbt8MaAxeksZcZd1j8gcGjc+QxQ==",
|
||||
"license": "Unlicense",
|
||||
"dependencies": {
|
||||
"tree-sitter-wasms": "^0.1.11"
|
||||
}
|
||||
},
|
||||
"node_modules/ts-api-utils": {
|
||||
"version": "2.3.0",
|
||||
"resolved": "https://registry.npmjs.org/ts-api-utils/-/ts-api-utils-2.3.0.tgz",
|
||||
@@ -5870,6 +6087,20 @@
|
||||
"defaults": "^1.0.3"
|
||||
}
|
||||
},
|
||||
"node_modules/web-tree-sitter": {
|
||||
"version": "0.25.10",
|
||||
"resolved": "https://registry.npmjs.org/web-tree-sitter/-/web-tree-sitter-0.25.10.tgz",
|
||||
"integrity": "sha512-Y09sF44/13XvgVKgO2cNDw5rGk6s26MgoZPXLESvMXeefBf7i6/73eFurre0IsTW6E14Y0ArIzhUMmjoc7xyzA==",
|
||||
"license": "MIT",
|
||||
"peerDependencies": {
|
||||
"@types/emscripten": "^1.40.0"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"@types/emscripten": {
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"node_modules/when-exit": {
|
||||
"version": "2.1.5",
|
||||
"resolved": "https://registry.npmjs.org/when-exit/-/when-exit-2.1.5.tgz",
|
||||
|
||||
+5
-1
@@ -55,6 +55,7 @@
|
||||
"mcp": "tsx src/mcp-server.ts",
|
||||
"mcp:prod": "node dist/mcp-server.js",
|
||||
"test": "vitest run src && node scripts/test-ops-preflight.mjs",
|
||||
"test:coverage": "vitest run --coverage",
|
||||
"test:watch": "vitest src",
|
||||
"lint": "eslint src/**/*.ts",
|
||||
"typecheck": "tsc --noEmit",
|
||||
@@ -120,7 +121,9 @@
|
||||
"gradient-string": "^2.0.2",
|
||||
"inquirer": "^9.2.15",
|
||||
"ora": "^8.0.1",
|
||||
"tsx": "^4.7.0"
|
||||
"tree-sitter-wasms": "0.1.13",
|
||||
"tsx": "^4.7.0",
|
||||
"web-tree-sitter": "0.25.10"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@eslint/js": "^9.39.2",
|
||||
@@ -130,6 +133,7 @@
|
||||
"@types/gradient-string": "^1.1.5",
|
||||
"@types/inquirer": "^9.0.7",
|
||||
"@types/node": "^20.11.0",
|
||||
"@vitest/coverage-v8": "4.1.9",
|
||||
"eslint": "^9.39.2",
|
||||
"typescript": "^5.3.3",
|
||||
"typescript-eslint": "^8.51.0",
|
||||
|
||||
@@ -66,7 +66,10 @@ describe('parseTextToolCalls — happy path + drift tolerance', () => {
|
||||
it('is DoS-safe on pathological brace input (was: quadratic ReDoS)', () => {
|
||||
const t0 = Date.now();
|
||||
expect(parseTextToolCalls('{'.repeat(80000))).toBeUndefined();
|
||||
expect(Date.now() - t0).toBeLessThan(300); // bounded — the old greedy regex took ~2100ms
|
||||
// Bounded — the old greedy regex took ~2100ms. Generous ceiling (2x below that
|
||||
// regression target) so V8 coverage instrumentation / slow CI can't false-fail
|
||||
// this wall-clock guard; it still fails hard if the quadratic ReDoS returns.
|
||||
expect(Date.now() - t0).toBeLessThan(1000);
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { mkdtempSync, writeFileSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { crawl, createMultiLangIngestConfig } from '../recon/code-ingest.js';
|
||||
|
||||
describe('createMultiLangIngestConfig', () => {
|
||||
const EXTS = ['py', 'js', 'ts', 'tsx', 'go', 'java', 'c', 'cpp'];
|
||||
|
||||
it('crawls every supported extension and excludes non-source', () => {
|
||||
const root = mkdtempSync(join(tmpdir(), 'mling-'));
|
||||
for (const e of EXTS) writeFileSync(join(root, `f.${e}`), 'x\n');
|
||||
writeFileSync(join(root, 'd.md'), '# ignore');
|
||||
writeFileSync(join(root, 'e.txt'), 'ignore');
|
||||
|
||||
const files = crawl(createMultiLangIngestConfig(root));
|
||||
|
||||
for (const e of EXTS) {
|
||||
expect(files.some((f) => f.endsWith(`f.${e}`)), `includes .${e}`).toBe(true);
|
||||
}
|
||||
expect(files.some((f) => f.endsWith('d.md'))).toBe(false);
|
||||
expect(files.some((f) => f.endsWith('e.txt'))).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,49 @@
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { splitParamList } from '../recon/param-split.js';
|
||||
|
||||
describe('splitParamList', () => {
|
||||
it('python: drops self, strips annotations/defaults, drops *args', () =>
|
||||
expect(splitParamList('self, url: str, timeout=5, *args', 'py')).toEqual(['url', 'timeout']));
|
||||
it('go name-first, drops types', () =>
|
||||
expect(splitParamList('ctx context.Context, id string', 'go')).toEqual(['ctx', 'id']));
|
||||
it('java type-first, keeps trailing name', () =>
|
||||
expect(splitParamList('String url, int max', 'java')).toEqual(['url', 'max']));
|
||||
it('ts name: type, strips optional marker', () =>
|
||||
expect(splitParamList('url: string, opts?: Opts', 'ts')).toEqual(['url', 'opts']));
|
||||
it('empty', () => expect(splitParamList('', 'ts')).toEqual([]));
|
||||
it('SAME raw, different lang → forces real dispatch', () => {
|
||||
expect(splitParamList('a b', 'go')).toEqual(['a']); // name-first
|
||||
expect(splitParamList('a b', 'java')).toEqual(['b']); // type-first
|
||||
});
|
||||
it('strips surrounding parens (tree-sitter params node text)', () =>
|
||||
expect(splitParamList('(url string)', 'go')).toEqual(['url']));
|
||||
it('c pointers: type-first, strips *', () =>
|
||||
expect(splitParamList('char *url, int n', 'c')).toEqual(['url', 'n']));
|
||||
it('generics do not split params', () =>
|
||||
expect(splitParamList('m Map<int, string>, n int', 'go')).toEqual(['m', 'n']));
|
||||
it('python drops bare * and / markers', () =>
|
||||
expect(splitParamList('a, *, b, /', 'py')).toEqual(['a', 'b']));
|
||||
it('python drops cls', () =>
|
||||
expect(splitParamList('cls, x', 'py')).toEqual(['x']));
|
||||
it('ts rest param + array type', () =>
|
||||
expect(splitParamList('...args: number[]', 'ts')).toEqual(['args']));
|
||||
it('cpp references and pointers stripped', () =>
|
||||
expect(splitParamList('const std::string &ref, int* p', 'cpp')).toEqual(['ref', 'p']));
|
||||
it('c array-suffix name stripped', () =>
|
||||
expect(splitParamList('char buf[]', 'c')).toEqual(['buf']));
|
||||
it('bracket + brace nesting does not split', () => {
|
||||
expect(splitParamList('a int[2], b T{}', 'go')).toEqual(['a', 'b']);
|
||||
});
|
||||
it('unbalanced close bracket is tolerated', () =>
|
||||
expect(splitParamList('a)x, b', 'go')).toEqual(['a', 'b']));
|
||||
it('non-identifier params are dropped', () =>
|
||||
expect(splitParamList('*, ---', 'go')).toEqual([]));
|
||||
it('trailing comma tolerated', () =>
|
||||
expect(splitParamList('a,', 'go')).toEqual(['a']));
|
||||
it('empty segment (double comma) dropped', () =>
|
||||
expect(splitParamList('a,,b', 'go')).toEqual(['a', 'b']));
|
||||
it('ts name-less param dropped', () =>
|
||||
expect(splitParamList(': int, x: str', 'ts')).toEqual(['x']));
|
||||
it('java name-less param dropped', () =>
|
||||
expect(splitParamList('=v, int x', 'java')).toEqual(['x']));
|
||||
});
|
||||
@@ -0,0 +1,69 @@
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { DANGEROUS_SINK_RE, OUTBOUND_REQUEST_RE } from '../recon/code-ingest.js';
|
||||
|
||||
describe('cross-language sinks', () => {
|
||||
it('matches new cross-language sinks', () => {
|
||||
for (const s of [
|
||||
'Runtime.getRuntime().exec(cmd)',
|
||||
'exec.Command("sh")',
|
||||
'child_process.exec(x)',
|
||||
'system(buf)',
|
||||
'popen(cmd)',
|
||||
'new ProcessBuilder(cmd)',
|
||||
]) {
|
||||
expect(DANGEROUS_SINK_RE.test(s), s).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
it('regression: existing python sinks still match', () => {
|
||||
for (const s of ['os.system(x)', 'eval(y)', 'subprocess.run(z)', 'yaml.load(f)']) {
|
||||
expect(DANGEROUS_SINK_RE.test(s), s).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
it('no false positive on benign / System.out / filesystem identifiers', () => {
|
||||
for (const s of ['const total = sum(a, b)', 'System.out.println("debug")', 'fileSystem.readFile(x)']) {
|
||||
expect(DANGEROUS_SINK_RE.test(s), s).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
it('outbound covers go net/http and js fetch/axios', () => {
|
||||
for (const s of ['http.Get(u)', 'http.Post(u, b)', 'fetch(u)', 'axios.get(u)']) {
|
||||
expect(OUTBOUND_REQUEST_RE.test(s), s).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
it('recall: idiomatic outbound/exec forms missed by the first-pass patterns', () => {
|
||||
// Go http.NewRequest+client.Do, Go client.Get, Node http(s).request, axios(config),
|
||||
// C exec-family, and receiver-qualified fetch — all real attack-surface idioms.
|
||||
for (const s of [
|
||||
'req, _ := http.NewRequest("GET", u, nil)',
|
||||
'resp, _ := client.Do(req)',
|
||||
'exec.CommandContext(ctx, cmd)',
|
||||
'execve(path, argv, envp)',
|
||||
'execvp(cmd, args)',
|
||||
'https.request(options, cb)',
|
||||
'axios({ url: u })',
|
||||
'return window.fetch(url)', // qualified fetch must still match (JS SSRF idiom)
|
||||
'this.fetch(userUrl)',
|
||||
]) {
|
||||
expect(DANGEROUS_SINK_RE.test(s), s).toBe(true);
|
||||
}
|
||||
for (const s of ['client.Get(u)', 'client.Do(req)', 'http.request(u, cb)', 'axios({url:u})', 'window.fetch(u)']) {
|
||||
expect(OUTBOUND_REQUEST_RE.test(s), s).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
it('precision: bare-call sinks and generic .Get/.Do methods on unrelated receivers are not flagged', () => {
|
||||
// `.system(`/`.popen(` on an unrelated object must not be flagged as the
|
||||
// C/OS bare call. (os.system / os.popen keep matching via their own patterns.)
|
||||
expect(DANGEROUS_SINK_RE.test('solarSystem.system(planets)'), 'receiver .system(').toBe(false);
|
||||
expect(DANGEROUS_SINK_RE.test('os.system(cmd)'), 'os.system still').toBe(true);
|
||||
// Generic keyed-store methods must NOT be mistaken for outbound HTTP — the
|
||||
// .Get/.Post/.Do alternatives are receiver-qualified to `client`.
|
||||
for (const s of ['return db.Get(id)', 'cache.Get(userID)', 'queue.Do(task)', 'session.Post(data)', 'config.get(key)']) {
|
||||
expect(OUTBOUND_REQUEST_RE.test(s), s).toBe(false);
|
||||
expect(DANGEROUS_SINK_RE.test(s), s).toBe(false);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,69 @@
|
||||
import { describe, it, expect, beforeEach } from 'vitest';
|
||||
import { Parser } from 'web-tree-sitter';
|
||||
import {
|
||||
initGrammars,
|
||||
getGrammar,
|
||||
supportedExts,
|
||||
__resetGrammarsForTest,
|
||||
} from '../recon/ts-grammars.js';
|
||||
|
||||
describe('ts-grammars registry', () => {
|
||||
beforeEach(() => __resetGrammarsForTest());
|
||||
|
||||
it('advertises the multi-language extension set', () => {
|
||||
expect(supportedExts()).toEqual(
|
||||
expect.arrayContaining(['.py', '.js', '.ts', '.tsx', '.go', '.java', '.c', '.cpp']),
|
||||
);
|
||||
});
|
||||
|
||||
it('loads every grammar with a working def-query (multi-language, real parse)', async () => {
|
||||
await initGrammars();
|
||||
const samples: Record<string, string> = {
|
||||
'.py': 'def fetch(url):\n return get(url)\n',
|
||||
'.js': 'function fetch(url){return get(url)}\n',
|
||||
'.ts': 'export function fetch(url: string){return get(url)}\n',
|
||||
'.tsx': 'export function F(p: {a:number}){ return null; }\n',
|
||||
'.go': 'package m\nfunc Fetch(url string) error { return nil }\n',
|
||||
'.java': 'class A { void run(String u){} }\n',
|
||||
'.c': 'int fetch(char *url){ return 0; }\n',
|
||||
'.cpp': 'int fetch(char *url){ return 0; }\n',
|
||||
};
|
||||
for (const ext of supportedExts()) {
|
||||
const g = getGrammar(ext);
|
||||
expect(g, `grammar for ${ext}`).toBeDefined();
|
||||
const p = new Parser();
|
||||
p.setLanguage(g!.language);
|
||||
const tree = p.parse(samples[ext]);
|
||||
const names = g!.query
|
||||
.matches(tree!.rootNode)
|
||||
.flatMap((m) => m.captures.filter((c) => c.name === 'name').map((c) => c.node.text));
|
||||
expect(names.length, `captures for ${ext}`).toBeGreaterThan(0);
|
||||
}
|
||||
});
|
||||
|
||||
it('is idempotent — a second init keeps the loaded registry', async () => {
|
||||
await initGrammars(['.go']);
|
||||
await initGrammars(['.py']); // second call is a no-op (already initialized)
|
||||
expect(getGrammar('.go')).toBeDefined();
|
||||
expect(getGrammar('.py')).toBeUndefined();
|
||||
});
|
||||
|
||||
it('fail-open: per-grammar load failure skips that ext, no throw', async () => {
|
||||
await expect(
|
||||
initGrammars(['.go'], () => { throw new Error('bad wasm path'); }),
|
||||
).resolves.toBeUndefined();
|
||||
expect(getGrammar('.go')).toBeUndefined();
|
||||
});
|
||||
|
||||
it('fail-open: global init failure leaves the registry empty, no throw', async () => {
|
||||
await expect(
|
||||
initGrammars(supportedExts(), (n) => n, () => Promise.reject(new Error('runtime init failed'))),
|
||||
).resolves.toBeUndefined();
|
||||
for (const ext of supportedExts()) expect(getGrammar(ext)).toBeUndefined();
|
||||
});
|
||||
|
||||
it('ignores unknown extensions', async () => {
|
||||
await initGrammars(['.zzz']);
|
||||
expect(getGrammar('.zzz')).toBeUndefined();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,136 @@
|
||||
/**
|
||||
* Adversarial hardening — ingest parses UNTRUSTED target source. A crafted file
|
||||
* must never throw, hang, or crash a mission; worst case is honest absence ([])
|
||||
* — a non-.py file is NEVER routed to the Python regex. Drives the full wired
|
||||
* path (parseFileMultiLang + ingestRepository/crawl).
|
||||
*/
|
||||
import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest';
|
||||
import { mkdtempSync, writeFileSync, mkdirSync, symlinkSync, rmSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { Tree } from 'web-tree-sitter';
|
||||
import { initGrammars, __resetGrammarsForTest } from '../recon/ts-grammars.js';
|
||||
import { parseFileMultiLang } from '../recon/ts-parse.js';
|
||||
import { ingestRepository, createMultiLangIngestConfig, type CodeBlock } from '../recon/code-ingest.js';
|
||||
|
||||
const tmpRoots: string[] = [];
|
||||
|
||||
beforeAll(async () => {
|
||||
__resetGrammarsForTest();
|
||||
await initGrammars();
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
for (const r of tmpRoots) rmSync(r, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
const invariants = (blocks: CodeBlock[]) => {
|
||||
expect(Array.isArray(blocks)).toBe(true);
|
||||
for (const b of blocks) {
|
||||
expect(b.lineStart).toBeLessThanOrEqual(b.lineEnd);
|
||||
expect(Array.isArray(b.params)).toBe(true);
|
||||
if (b.name) expect(b.body.length).toBeGreaterThan(0);
|
||||
}
|
||||
};
|
||||
|
||||
describe('adversarial parseFileMultiLang', () => {
|
||||
it('malformed source per language does not throw', () => {
|
||||
for (const [ext, src] of [
|
||||
['.go', 'func ('],
|
||||
['.ts', 'class {{{ function ('],
|
||||
['.java', 'class A { void ('],
|
||||
['.c', 'int f( { { {'],
|
||||
] as const) {
|
||||
expect(() => invariants(parseFileMultiLang(`x${ext}`, src, ext))).not.toThrow();
|
||||
}
|
||||
});
|
||||
|
||||
it('wrong-language content (Python in a .go file) yields structurally valid output', () => {
|
||||
expect(() =>
|
||||
invariants(parseFileMultiLang('x.go', 'def fetch(url):\n return get(url)\n', '.go')),
|
||||
).not.toThrow();
|
||||
});
|
||||
|
||||
it('binary / non-UTF8 bytes do not throw', () => {
|
||||
const bin = Buffer.from([0x00, 0xff, 0xfe, 0x10, 0x80, 0x00, 0x41]).toString('binary');
|
||||
expect(() => invariants(parseFileMultiLang('x.go', bin, '.go'))).not.toThrow();
|
||||
});
|
||||
|
||||
it('empty file yields no blocks, no throw', () => {
|
||||
expect(parseFileMultiLang('e.go', '', '.go')).toEqual([]);
|
||||
});
|
||||
|
||||
it('unicode identifiers are extracted correctly', () => {
|
||||
const blocks = parseFileMultiLang('u.go', 'package m\nfunc Ünïçødé(x int) int { return x }\n', '.go');
|
||||
const fn = blocks.find((b) => b.name === 'Ünïçødé');
|
||||
expect(fn, 'unicode function name captured').toBeDefined();
|
||||
expect(fn!.params).toEqual(['x']);
|
||||
});
|
||||
|
||||
it('large file parses within the wall-clock budget and still extracts', () => {
|
||||
let src = 'package m\n';
|
||||
for (let i = 0; i < 20000; i++) src += `func F${i}(url string) error { return http.Get(url) }\n`;
|
||||
const t0 = Date.now();
|
||||
const blocks = parseFileMultiLang('big.go', src, '.go');
|
||||
expect(Date.now() - t0).toBeLessThan(10000); // bounded — never spins
|
||||
expect(blocks.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it('a real parse tripping the wall-clock budget fails open to [] (not mocked)', () => {
|
||||
// Drive the REAL tree-sitter progressCallback path with a 0ms budget: the
|
||||
// callback returns true on its first invocation, tree-sitter cancels, parse
|
||||
// returns null, and the fail-open returns []. Proves the actual cancellation
|
||||
// wiring — not a mock that hard-returns null.
|
||||
let src = 'package m\n';
|
||||
for (let i = 0; i < 5000; i++) src += `func F${i}(a int) int { return a }\n`;
|
||||
const t0 = Date.now();
|
||||
const blocks = parseFileMultiLang('slow.go', src, '.go', undefined, 0);
|
||||
expect(blocks).toEqual([]); // cancellation path taken
|
||||
expect(Date.now() - t0).toBeLessThan(5000); // cancelled early, not run to completion
|
||||
});
|
||||
|
||||
it('deeply nested input does not blow the stack, hang, or corrupt output', () => {
|
||||
const deep = 'package m\nfunc F() { ' + '{'.repeat(2000) + '}'.repeat(2000) + ' }\n';
|
||||
const t0 = Date.now();
|
||||
expect(() => invariants(parseFileMultiLang('d.go', deep, '.go'))).not.toThrow();
|
||||
expect(Date.now() - t0).toBeLessThan(10000);
|
||||
});
|
||||
|
||||
it('frees the parsed tree exactly once per parse that produced one (leak-fix mechanism)', () => {
|
||||
// Deterministic, machine-independent guard for the WASM Tree/Parser leak:
|
||||
// assert tree.delete() is actually invoked. (A resource-exhaustion repro is
|
||||
// hardware/heap-ceiling dependent and green-only on machines with headroom.)
|
||||
const del = vi.spyOn(Tree.prototype, 'delete');
|
||||
try {
|
||||
del.mockClear();
|
||||
parseFileMultiLang('ok.go', 'package m\nfunc F(){ return }\n', '.go'); // success path
|
||||
expect(del, 'success path frees its tree').toHaveBeenCalledTimes(1);
|
||||
|
||||
del.mockClear();
|
||||
parseFileMultiLang('bad.ts', 'class C { m( { { {', '.ts'); // error-recovery still yields a tree
|
||||
expect(del, 'error-recovery path frees its tree').toHaveBeenCalledTimes(1);
|
||||
} finally {
|
||||
del.mockRestore();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('adversarial crawl / ingest', () => {
|
||||
it('symlink loop under the repo root terminates (crawl does not follow symlink dirs)', () => {
|
||||
const root = mkdtempSync(join(tmpdir(), 'advcrawl-'));
|
||||
tmpRoots.push(root);
|
||||
const sub = join(root, 'sub');
|
||||
mkdirSync(sub);
|
||||
writeFileSync(join(sub, 'a.go'), 'package m\nfunc A() {}\n');
|
||||
try {
|
||||
symlinkSync(root, join(sub, 'loop'), 'dir'); // self-referential loop
|
||||
} catch {
|
||||
console.warn('[skip] symlinks unavailable in this environment — loop guard unverified');
|
||||
return;
|
||||
}
|
||||
const t0 = Date.now();
|
||||
const result = ingestRepository(createMultiLangIngestConfig(root));
|
||||
expect(Date.now() - t0).toBeLessThan(10000); // did not spin on the loop
|
||||
expect(result.analysisUnits.some((u) => u.block.name === 'A')).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,31 @@
|
||||
/**
|
||||
* Language-scoped fail-open: the Python regex parser must ONLY ever run on `.py`.
|
||||
* This file deliberately never calls initGrammars(), so the registry is empty and
|
||||
* every getGrammar() returns undefined — the "grammars failed to load / not yet
|
||||
* loaded" production scenario. A brace/`class` language routed to the Python regex
|
||||
* in that state would emit a corrupt block (wrong line span, methods lost); the
|
||||
* contract is honest absence ([]) instead.
|
||||
*/
|
||||
import { describe, it, expect, beforeAll } from 'vitest';
|
||||
import { __resetGrammarsForTest } from '../recon/ts-grammars.js';
|
||||
import { parseFileMultiLang } from '../recon/ts-parse.js';
|
||||
import { parseFile } from '../recon/code-ingest.js';
|
||||
|
||||
beforeAll(() => __resetGrammarsForTest()); // ensure the registry is empty; do NOT init
|
||||
|
||||
describe('parseFileMultiLang — language-scoped fallback (no grammars loaded)', () => {
|
||||
it('a Java `class` file returns [] — never a mangled Python-regex block', () => {
|
||||
// If this routed to parseFile, the Python CLASS_RE would match `class A` and
|
||||
// emit exactly one (wrong) block; assert it does not.
|
||||
expect(parseFileMultiLang('A.java', 'class A {\n void run(String c){}\n}\n', '.java')).toEqual([]);
|
||||
});
|
||||
|
||||
it('a JS `class` file returns [] with no grammar (would false-match the Python regex)', () => {
|
||||
expect(parseFileMultiLang('a.js', 'class Svc {\n run(cmd){ exec(cmd); }\n}\n', '.js')).toEqual([]);
|
||||
});
|
||||
|
||||
it('.py still routes to the Python regex even with no grammars loaded (byte-identical)', () => {
|
||||
const src = 'def fetch(url):\n return get(url)\n';
|
||||
expect(parseFileMultiLang('a.py', src, '.py')).toEqual(parseFile('a.py', src));
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,58 @@
|
||||
import { describe, it, expect, beforeAll } from 'vitest';
|
||||
import { mkdtempSync, writeFileSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { initGrammars, __resetGrammarsForTest } from '../recon/ts-grammars.js';
|
||||
import { ingestRepository, createMultiLangIngestConfig } from '../recon/code-ingest.js';
|
||||
|
||||
beforeAll(async () => {
|
||||
__resetGrammarsForTest();
|
||||
await initGrammars();
|
||||
});
|
||||
|
||||
describe('multi-language ingest → security ranking (headline regression)', () => {
|
||||
function repo(): string {
|
||||
const root = mkdtempSync(join(tmpdir(), 'mlrank-'));
|
||||
// Go: Handler calls Fetch; Fetch is an SSRF surface (identifier param + outbound).
|
||||
writeFileSync(
|
||||
join(root, 'svc.go'),
|
||||
'package m\n' +
|
||||
'func Handler(url string) error {\n\treturn Fetch(url)\n}\n' +
|
||||
'func Fetch(url string) error {\n\treturn http.Get(url)\n}\n' +
|
||||
'func Add(a int, b int) int {\n\treturn a + b\n}\n',
|
||||
);
|
||||
// TS: command-exec sink.
|
||||
writeFileSync(join(root, 'run.ts'), 'export function runCmd(cmd: string) { return exec(cmd); }\n');
|
||||
// Java: Runtime.exec sink.
|
||||
writeFileSync(join(root, 'A.java'), 'class A { void run(String cmd){ Runtime.getRuntime().exec(cmd); } }\n');
|
||||
// Python still works alongside (regex path).
|
||||
writeFileSync(join(root, 'p.py'), 'def load(path):\n return open(path)\n');
|
||||
return root;
|
||||
}
|
||||
|
||||
it('extracts non-.py functions and security-ranks them as attack_surface', () => {
|
||||
const { analysisUnits } = ingestRepository(createMultiLangIngestConfig(repo()));
|
||||
const by = (name: string) => analysisUnits.find((u) => u.block.name === name);
|
||||
|
||||
// multi-language extraction reached the pipeline (Go/TS/Java + Python)
|
||||
for (const name of ['Fetch', 'runCmd', 'run', 'load']) {
|
||||
expect(by(name), `block ${name} extracted`).toBeDefined();
|
||||
}
|
||||
|
||||
// sink-bearing functions are attack_surface, out-ranking a neutral block
|
||||
for (const name of ['Fetch', 'runCmd', 'run']) {
|
||||
expect(by(name)!.exposure, `${name} exposure`).toBe('attack_surface');
|
||||
}
|
||||
expect(by('Fetch')!.priority).toBeGreaterThan(by('Add')!.priority);
|
||||
expect(by('Add')!.exposure).toBe('neutral');
|
||||
});
|
||||
|
||||
it('call graph links non-Python identifiers (cross-language reachability guard)', () => {
|
||||
const { analysisUnits } = ingestRepository(createMultiLangIngestConfig(repo()));
|
||||
const handler = analysisUnits.find((u) => u.block.name === 'Handler')!;
|
||||
const fetch = analysisUnits.find((u) => u.block.name === 'Fetch')!;
|
||||
// buildCallGraph matched the Go call "Fetch(url)" inside Handler by name.
|
||||
expect(handler.callees).toContain(fetch.block.id);
|
||||
expect(fetch.callers).toContain(handler.block.id);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,95 @@
|
||||
import { describe, it, expect, beforeAll } from 'vitest';
|
||||
import type { Parser } from 'web-tree-sitter';
|
||||
import { initGrammars, __resetGrammarsForTest } from '../recon/ts-grammars.js';
|
||||
import { parseFileMultiLang, nodeToCodeBlock } from '../recon/ts-parse.js';
|
||||
import { parseFile } from '../recon/code-ingest.js';
|
||||
|
||||
beforeAll(async () => {
|
||||
__resetGrammarsForTest(); // clear any singleton state leaked from another test file
|
||||
await initGrammars();
|
||||
});
|
||||
|
||||
describe('parseFileMultiLang', () => {
|
||||
it('extracts a Go function as a CodeBlock', () => {
|
||||
const src = 'package m\nfunc Fetch(url string) error {\n return http.Get(url)\n}\n';
|
||||
const fn = parseFileMultiLang('a.go', src, '.go').find((b) => b.name === 'Fetch');
|
||||
expect(fn).toBeDefined();
|
||||
expect(fn!.params).toEqual(['url']);
|
||||
expect(fn!.kind).toBe('function');
|
||||
expect(fn!.lineStart).toBe(2);
|
||||
expect(fn!.body).toContain('http.Get(url)');
|
||||
expect(fn!.id).toBe('a.go::Fetch@2');
|
||||
expect(fn!.decorators).toEqual([]);
|
||||
});
|
||||
|
||||
it('extracts a TS class + method', () => {
|
||||
const src = 'class Svc {\n run(cmd: string) { exec(cmd); }\n}\n';
|
||||
const blocks = parseFileMultiLang('s.ts', src, '.ts');
|
||||
expect(blocks.find((b) => b.name === 'Svc')?.kind).toBe('class');
|
||||
const m = blocks.find((b) => b.name === 'run');
|
||||
expect(m?.kind).toBe('method');
|
||||
expect(m?.params).toEqual(['cmd']);
|
||||
});
|
||||
|
||||
it('extracts a Java constructor as a method', () => {
|
||||
const blocks = parseFileMultiLang('A.java', 'class A {\n A(int x){}\n}\n', '.java');
|
||||
expect(blocks.find((b) => b.name === 'A' && b.kind === 'method')).toBeDefined();
|
||||
expect(blocks.find((b) => b.name === 'A' && b.kind === 'class')).toBeDefined();
|
||||
});
|
||||
|
||||
it('extracts C functions with pointer return types', () => {
|
||||
const blocks = parseFileMultiLang(
|
||||
'fetch.c',
|
||||
'const char *fetch_url(const char *url) { return url; }\n',
|
||||
'.c',
|
||||
);
|
||||
expect(blocks).toEqual([
|
||||
expect.objectContaining({
|
||||
name: 'fetch_url',
|
||||
kind: 'function',
|
||||
params: ['url'],
|
||||
}),
|
||||
]);
|
||||
});
|
||||
|
||||
it('.py routes to the regex parser (byte-identical)', () => {
|
||||
const src = 'def fetch(url):\n return get(url)\n';
|
||||
expect(parseFileMultiLang('a.py', src, '.py')).toEqual(parseFile('a.py', src));
|
||||
});
|
||||
|
||||
it('unsupported ext returns [] (no throw, no Python-regex on non-.py)', () => {
|
||||
expect(parseFileMultiLang('a.zzz', 'func F(){}', '.zzz')).toEqual([]);
|
||||
});
|
||||
|
||||
it('fail-open on parse timeout (parse returns null) → []', () => {
|
||||
const nullParser = () =>
|
||||
({ setLanguage() {}, parse: () => null, delete() {} }) as unknown as Parser;
|
||||
expect(parseFileMultiLang('a.go', 'package m\nfunc F(){}\n', '.go', nullParser)).toEqual([]);
|
||||
});
|
||||
|
||||
it('fail-open on parser error (parse throws) → []', () => {
|
||||
const throwParser = () =>
|
||||
({
|
||||
setLanguage() {},
|
||||
parse() {
|
||||
throw new Error('boom');
|
||||
},
|
||||
delete() {},
|
||||
}) as unknown as Parser;
|
||||
expect(parseFileMultiLang('a.go', 'package m\nfunc F(){}\n', '.go', throwParser)).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('nodeToCodeBlock', () => {
|
||||
it('maps a class node (no params) to kind=class with empty params', () => {
|
||||
const fake = {
|
||||
type: 'class_declaration',
|
||||
startPosition: { row: 4 },
|
||||
endPosition: { row: 9 },
|
||||
text: 'class C {}',
|
||||
} as never;
|
||||
const name = { text: 'C' } as never;
|
||||
const cb = nodeToCodeBlock(fake, name, undefined, 'x.ts', 'ts');
|
||||
expect(cb).toMatchObject({ name: 'C', kind: 'class', lineStart: 5, lineEnd: 10, params: [], id: 'x.ts::C@5' });
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,59 @@
|
||||
import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest';
|
||||
import { mkdtempSync, mkdirSync, writeFileSync, realpathSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { initGrammars, __resetGrammarsForTest } from '../recon/ts-grammars.js';
|
||||
import { ingestRepoToSourceContext, runWhiteboxAnalysis } from '../recon/whitebox.js';
|
||||
import { DecompositionOrchestrator } from '../orchestration/index.js';
|
||||
|
||||
let base: string;
|
||||
let repo: string;
|
||||
let prevRoot: string | undefined;
|
||||
|
||||
beforeAll(async () => {
|
||||
__resetGrammarsForTest();
|
||||
await initGrammars();
|
||||
base = realpathSync(mkdtempSync(join(tmpdir(), 't3mp3st-wbml-')));
|
||||
prevRoot = process.env.T3MP3ST_REPO_ROOT;
|
||||
process.env.T3MP3ST_REPO_ROOT = base;
|
||||
repo = join(base, 'repo');
|
||||
mkdirSync(repo);
|
||||
writeFileSync(join(repo, 'svc.go'), 'package m\nfunc Fetch(url string) error {\n\treturn http.Get(url)\n}\n');
|
||||
writeFileSync(join(repo, 'run.ts'), 'export function runCmd(cmd: string) {\n\treturn exec(cmd);\n}\n');
|
||||
writeFileSync(join(repo, 'p.py'), 'def load(path):\n return open(path)\n');
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
if (prevRoot === undefined) delete process.env.T3MP3ST_REPO_ROOT;
|
||||
else process.env.T3MP3ST_REPO_ROOT = prevRoot;
|
||||
});
|
||||
|
||||
describe('ingestRepoToSourceContext (production white-box entry) — multi-language', () => {
|
||||
it('non-.py source reaches the packed sourceContext (feature is not a no-op in prod)', () => {
|
||||
const { sourceContext } = ingestRepoToSourceContext(repo);
|
||||
// Go and TS blocks — proof the multilang config swap actually took effect on the
|
||||
// real production path, not just via a direct createMultiLangIngestConfig call.
|
||||
expect(sourceContext).toContain('Fetch');
|
||||
expect(sourceContext).toContain('runCmd');
|
||||
// Python still ingested alongside.
|
||||
expect(sourceContext).toContain('load');
|
||||
});
|
||||
|
||||
it('runWhiteboxAnalysis (2nd production caller) also feeds multilang source to the orchestrator', async () => {
|
||||
// Guards the second createMultiLangIngestConfig swap site — a one-token
|
||||
// revert to createPythonIngestConfig would drop non-.py content silently.
|
||||
// Stub orch.run so no real LLM calls fire; capture the sourceContext it gets.
|
||||
const runSpy = vi
|
||||
.spyOn(DecompositionOrchestrator.prototype, 'run')
|
||||
.mockResolvedValue({} as never);
|
||||
try {
|
||||
await runWhiteboxAnalysis({ repoPath: repo, objective: 'x' });
|
||||
expect(runSpy).toHaveBeenCalledTimes(1);
|
||||
const sourceContext = runSpy.mock.calls[0][1] as string;
|
||||
expect(sourceContext).toContain('Fetch'); // Go reached the orchestrator
|
||||
expect(sourceContext).toContain('runCmd'); // TS reached the orchestrator
|
||||
} finally {
|
||||
runSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -37,7 +37,8 @@
|
||||
*/
|
||||
|
||||
import { readdirSync, readFileSync, statSync } from 'fs';
|
||||
import { join, sep } from 'path';
|
||||
import { join, sep, extname } from 'path';
|
||||
import { parseFileMultiLang } from './ts-parse.js';
|
||||
import {
|
||||
estimateTokens,
|
||||
type SourceFile,
|
||||
@@ -169,14 +170,23 @@ const ENTRY_NAME_RES: RegExp[] = [
|
||||
const SECURITY_CONTROL_NAME_RE =
|
||||
/valid|verif|auth|authoriz|authentic|permission|sanitiz|escap|csrf|check_|require_|is_allowed/i;
|
||||
|
||||
// Dangerous sinks that make a block part of the attack surface.
|
||||
const DANGEROUS_SINK_RE =
|
||||
/requests\.(get|post|put)|urllib|urlopen|httpx|socket\.|subprocess|os\.system|\beval\(|\bexec\(|pickle\.loads|yaml\.load|cursor\.execute|\.raw\(|open\(/;
|
||||
// Dangerous sinks that make a block part of the attack surface. Python patterns
|
||||
// plus cross-language sinks (Java/Go/C/JS) for the multi-language ingest.
|
||||
// Bare-call patterns (system/popen/exec-family/fetch) use a `(?<![\w.])` guard so
|
||||
// they match a C/JS free call `system(x)` but NOT a Python/JS method call on a
|
||||
// receiver `foo.system(x)` — that keeps the existing Python corpus's rankings
|
||||
// stable (its `os.system`/`os.popen` are matched by the qualified patterns).
|
||||
export const DANGEROUS_SINK_RE =
|
||||
/requests\.(get|post|put)|urllib|urlopen|httpx|socket\.|subprocess|os\.system|\beval\(|\bexec\(|pickle\.loads|yaml\.load|cursor\.execute|\.raw\(|open\(|exec\.Command(?:Context)?|Runtime\.getRuntime|ProcessBuilder|(?<![\w.])system\(|(?<![\w.])popen\(|(?<![\w.])exec(?:l|v)[pe]?\(|http\.(Get|Post|NewRequest)|https?\.request\(|[Cc]lient\.(Do|Get|Post)\(|\bfetch\(|axios[.(]/;
|
||||
|
||||
// Outbound-request sinks specifically (subset of the above) — used for the
|
||||
// SSRF/IDOR "identifier param + outbound request" signal.
|
||||
const OUTBOUND_REQUEST_RE =
|
||||
/requests\.(get|post|put)|urllib|urlopen|httpx|socket\./;
|
||||
// SSRF/IDOR "identifier param + outbound request" signal. Cross-language: Go
|
||||
// net/http (http.Get/Post/NewRequest and *client*.Do/Get/Post), Node http(s).request,
|
||||
// JS fetch/axios. `fetch` uses \b (matches window.fetch/this.fetch too); the Go
|
||||
// client methods are receiver-qualified to `client` so a mundane db.Get/cache.Get
|
||||
// is NOT mistaken for an outbound request.
|
||||
export const OUTBOUND_REQUEST_RE =
|
||||
/requests\.(get|post|put)|urllib|urlopen|httpx|socket\.|http\.(Get|Post|get|post|NewRequest)|https?\.request\(|[Cc]lient\.(Do|Get|Post)\(|\bfetch\(|axios[.(]/;
|
||||
|
||||
// URL/identifier-shaped param names.
|
||||
const RISKY_PARAM_RE = /url|uri|endpoint|host|addr|id$|_id|path|file|name/i;
|
||||
@@ -764,7 +774,10 @@ export function ingestRepository(config: IngestConfig): IngestResult {
|
||||
}
|
||||
processedFiles += 1;
|
||||
totalBytes += content.length;
|
||||
allBlocks.push(...parseFile(path, content));
|
||||
// Multi-language dispatch: .py uses the legacy regex parser; supported
|
||||
// non-Python languages use tree-sitter and yield [] when their grammar is
|
||||
// unavailable. Stays sync — parseFileMultiLang does not await.
|
||||
allBlocks.push(...parseFileMultiLang(path, content, extname(path).toLowerCase()));
|
||||
if (maxTotalBytes !== undefined && totalBytes >= maxTotalBytes) {
|
||||
truncated = true;
|
||||
console.warn(`[code-ingest] ingest reached maxTotalBytes ceiling (${maxTotalBytes}) after ${totalBytes} bytes; remaining files skipped — raise IngestConfig.maxTotalBytes to analyze more.`);
|
||||
@@ -841,6 +854,30 @@ export function createPythonIngestConfig(repoRoot: string): IngestConfig {
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Multi-language ingest config: same ceilings as the Python config, but crawls
|
||||
* the languages the tree-sitter extractor supports. Non-source files are
|
||||
* excluded by `includeExts`. Used by the white-box analysis entrypoints.
|
||||
*
|
||||
* SECURITY: `maxFileBytes` (1 MB) is a security control here, not just a perf
|
||||
* knob — it caps the peak WASM linear-memory a single tree-sitter parse can
|
||||
* demand. A dense/obfuscated file large enough to approach web-tree-sitter's
|
||||
* 2 GB heap ceiling makes the native allocator abort() (caught + fail-open, but
|
||||
* the WASM high-water-mark cannot be reclaimed for the process's life). The 1 MB
|
||||
* cap keeps that unreachable (worst measured blow-up ~370x → ~5-6x margin). Do
|
||||
* NOT raise it without re-testing that margin.
|
||||
*/
|
||||
export function createMultiLangIngestConfig(repoRoot: string): IngestConfig {
|
||||
return {
|
||||
repoRoot,
|
||||
includeExts: ['.py', '.js', '.ts', '.tsx', '.go', '.java', '.c', '.cpp'],
|
||||
excludeGlobs: [...DEFAULT_EXCLUDES],
|
||||
maxFileBytes: 1_000_000,
|
||||
maxFiles: 50_000,
|
||||
maxTotalBytes: 1_000_000_000,
|
||||
};
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// COMPOSITION WITH context-pack — rendering & security-ordered packing
|
||||
// =============================================================================
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
/**
|
||||
* Language-aware parameter-name extractor for the multi-language ingest.
|
||||
*
|
||||
* tree-sitter hands us a parameter-list as raw source text; different languages
|
||||
* order name and type differently (Python/Go/TS are name-first, Java/C/C++ are
|
||||
* type-first). This returns the bare identifier names so the security-ranking
|
||||
* layer's `RISKY_PARAM_RE` can match on them exactly as it did for Python.
|
||||
*/
|
||||
|
||||
/** Split on top-level commas only — commas inside (), [], {}, <> are nested. */
|
||||
function splitTopLevel(s: string): string[] {
|
||||
const parts: string[] = [];
|
||||
let depth = 0;
|
||||
let cur = '';
|
||||
for (const ch of s) {
|
||||
if (ch === '(' || ch === '[' || ch === '{' || ch === '<') depth++;
|
||||
else if (ch === ')' || ch === ']' || ch === '}' || ch === '>') depth = Math.max(0, depth - 1);
|
||||
if (ch === ',' && depth === 0) {
|
||||
parts.push(cur);
|
||||
cur = '';
|
||||
} else {
|
||||
cur += ch;
|
||||
}
|
||||
}
|
||||
if (cur.trim()) parts.push(cur);
|
||||
return parts;
|
||||
}
|
||||
|
||||
/** First identifier-shaped token in a string, or null. */
|
||||
function firstId(s: string): string | null {
|
||||
const m = s.match(/[A-Za-z_$][A-Za-z0-9_$]*/);
|
||||
return m ? m[0] : null;
|
||||
}
|
||||
|
||||
function paramName(partRaw: string, lang: string): string | null {
|
||||
const part = partRaw.trim();
|
||||
if (!part) return null;
|
||||
|
||||
if (lang === 'py') {
|
||||
if (part.startsWith('*')) return null; // *args / **kwargs / bare *
|
||||
const p = part.split('=')[0].split(':')[0].trim();
|
||||
if (!p || p === '/' || p === 'self' || p === 'cls') return null;
|
||||
return firstId(p);
|
||||
}
|
||||
|
||||
if (lang === 'ts' || lang === 'js') {
|
||||
// name-first: name is the token before ':' or '=', minus a trailing '?'.
|
||||
let p = part.replace(/^\.\.\./, '').split(':')[0].split('=')[0].trim();
|
||||
p = p.replace(/\?$/, '').trim();
|
||||
const toks = p.split(/\s+/).filter(Boolean); // drop TS modifiers (public/readonly)
|
||||
return firstId(toks[toks.length - 1] ?? '');
|
||||
}
|
||||
|
||||
if (lang === 'go') {
|
||||
// name-first: first token of the group ("a, b int" already split → "a", "b int").
|
||||
return firstId(part);
|
||||
}
|
||||
|
||||
// java / c / c++ / unknown: type-first → name is the last token; strip * & [].
|
||||
const p = part.split('=')[0].trim();
|
||||
const toks = p.split(/\s+/).filter(Boolean);
|
||||
const last = (toks[toks.length - 1] ?? '').replace(/^[*&]+/, '').replace(/\[\]$/, '');
|
||||
return firstId(last);
|
||||
}
|
||||
|
||||
/** Bare parameter identifiers from a raw parameter list, per language dialect. */
|
||||
export function splitParamList(raw: string, lang: string): string[] {
|
||||
let s = raw.trim();
|
||||
if (s.startsWith('(') && s.endsWith(')')) s = s.slice(1, -1).trim();
|
||||
if (!s) return [];
|
||||
const out: string[] = [];
|
||||
for (const part of splitTopLevel(s)) {
|
||||
const name = paramName(part, lang);
|
||||
if (name) out.push(name);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
@@ -0,0 +1,149 @@
|
||||
/**
|
||||
* Multi-language grammar registry for white-box ingest.
|
||||
*
|
||||
* Wraps web-tree-sitter (WASM) so `parseFileMultiLang` can extract CodeBlocks
|
||||
* from ~8 languages, replacing the Python-only regex parser. Grammars are
|
||||
* loaded ONCE at bootstrap (`initGrammars`) — `Parser.init()`/`Language.load()`
|
||||
* are async, but `parser.parse()` is sync, so the ingest path stays sync.
|
||||
*
|
||||
* Fail-open by design: if init throws (bad/missing wasm, version skew), the
|
||||
* registry is left empty. Python still routes to `parseFile`; non-Python files
|
||||
* yield no blocks — ingest never crashes a mission.
|
||||
*/
|
||||
import { Parser, Language, Query } from 'web-tree-sitter';
|
||||
import { createRequire } from 'module';
|
||||
|
||||
const require = createRequire(import.meta.url);
|
||||
|
||||
/** A loaded grammar: the tree-sitter Language plus the compiled def-query. */
|
||||
export interface GrammarEntry {
|
||||
language: Language;
|
||||
query: Query;
|
||||
/** Language id, for param-splitting dialect (py/js/ts/go/java/c/cpp). */
|
||||
lang: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-extension spec: which prebuilt wasm (from tree-sitter-wasms) and the
|
||||
* tree-sitter query capturing function/method/class definitions. Every pattern
|
||||
* binds `@name` (the identifier) and `@def` (the whole definition node);
|
||||
* `@params` is bound where the grammar exposes a parameter list.
|
||||
*/
|
||||
const SPECS: Record<string, { wasm: string; lang: string; query: string }> = {
|
||||
'.py': {
|
||||
wasm: 'tree-sitter-python', lang: 'py',
|
||||
query: `
|
||||
(function_definition name: (identifier) @name parameters: (parameters) @params) @def
|
||||
(class_definition name: (identifier) @name) @def
|
||||
`,
|
||||
},
|
||||
'.js': {
|
||||
wasm: 'tree-sitter-javascript', lang: 'js',
|
||||
query: `
|
||||
(function_declaration name: (identifier) @name parameters: (formal_parameters) @params) @def
|
||||
(method_definition name: (property_identifier) @name parameters: (formal_parameters) @params) @def
|
||||
(class_declaration name: (identifier) @name) @def
|
||||
`,
|
||||
},
|
||||
'.ts': {
|
||||
wasm: 'tree-sitter-typescript', lang: 'ts',
|
||||
query: `
|
||||
(function_declaration name: (identifier) @name parameters: (formal_parameters) @params) @def
|
||||
(method_definition name: (property_identifier) @name parameters: (formal_parameters) @params) @def
|
||||
(class_declaration name: (type_identifier) @name) @def
|
||||
`,
|
||||
},
|
||||
'.tsx': {
|
||||
wasm: 'tree-sitter-tsx', lang: 'ts',
|
||||
query: `
|
||||
(function_declaration name: (identifier) @name parameters: (formal_parameters) @params) @def
|
||||
(method_definition name: (property_identifier) @name parameters: (formal_parameters) @params) @def
|
||||
(class_declaration name: (type_identifier) @name) @def
|
||||
`,
|
||||
},
|
||||
'.go': {
|
||||
wasm: 'tree-sitter-go', lang: 'go',
|
||||
query: `
|
||||
(function_declaration name: (identifier) @name parameters: (parameter_list) @params) @def
|
||||
(method_declaration name: (field_identifier) @name parameters: (parameter_list) @params) @def
|
||||
`,
|
||||
},
|
||||
'.java': {
|
||||
wasm: 'tree-sitter-java', lang: 'java',
|
||||
query: `
|
||||
(method_declaration name: (identifier) @name parameters: (formal_parameters) @params) @def
|
||||
(constructor_declaration name: (identifier) @name parameters: (formal_parameters) @params) @def
|
||||
(class_declaration name: (identifier) @name) @def
|
||||
`,
|
||||
},
|
||||
'.c': {
|
||||
wasm: 'tree-sitter-c', lang: 'c',
|
||||
query: `
|
||||
(function_definition declarator: (function_declarator declarator: (identifier) @name parameters: (parameter_list) @params)) @def
|
||||
(function_definition declarator: (pointer_declarator declarator: (function_declarator declarator: (identifier) @name parameters: (parameter_list) @params))) @def
|
||||
`,
|
||||
},
|
||||
'.cpp': {
|
||||
wasm: 'tree-sitter-cpp', lang: 'cpp',
|
||||
query: `
|
||||
(function_definition declarator: (function_declarator declarator: (identifier) @name parameters: (parameter_list) @params)) @def
|
||||
`,
|
||||
},
|
||||
};
|
||||
|
||||
const registry = new Map<string, GrammarEntry>();
|
||||
let initialized = false;
|
||||
|
||||
/** Extensions (with leading dot) the registry can parse. */
|
||||
export function supportedExts(): string[] {
|
||||
return Object.keys(SPECS);
|
||||
}
|
||||
|
||||
/**
|
||||
* Load grammars once. Idempotent. Fail-open: any error leaves the (possibly
|
||||
* partial) registry in place and marks init done — callers extract nothing for
|
||||
* exts that never loaded (non-.py fail-open is empty, not the Python regex).
|
||||
* `wasmResolve` lets a test force a bad path to exercise the failure branch.
|
||||
*/
|
||||
export async function initGrammars(
|
||||
exts: string[] = supportedExts(),
|
||||
wasmResolve: (name: string) => string = (name) => require.resolve(`tree-sitter-wasms/out/${name}.wasm`),
|
||||
init: () => Promise<void> = () => Parser.init(),
|
||||
): Promise<void> {
|
||||
if (initialized) return;
|
||||
try {
|
||||
await init();
|
||||
for (const ext of exts) {
|
||||
const spec = SPECS[ext];
|
||||
if (!spec) continue;
|
||||
try {
|
||||
const language = await Language.load(wasmResolve(spec.wasm));
|
||||
const query = new Query(language, spec.query);
|
||||
registry.set(ext, { language, query, lang: spec.lang });
|
||||
} catch {
|
||||
// per-ext fail-open: skip this grammar, keep the rest
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// global fail-open: registry stays empty → non-.py ingest extracts nothing.
|
||||
// Log once so an operator scanning a non-Python repo sees the degradation
|
||||
// (the `initialized` guard means this never retries for the process).
|
||||
console.warn(
|
||||
'[ts-grammars] grammar init failed; multi-language ingest disabled for this process ' +
|
||||
'(non-Python files will yield no blocks). Python ingest is unaffected.',
|
||||
);
|
||||
} finally {
|
||||
initialized = true;
|
||||
}
|
||||
}
|
||||
|
||||
/** A loaded grammar for `ext`, or undefined (→ caller returns [] for non-.py; .py uses parseFile). */
|
||||
export function getGrammar(ext: string): GrammarEntry | undefined {
|
||||
return registry.get(ext);
|
||||
}
|
||||
|
||||
/** Test-only: reset init state + registry so a fresh `initGrammars` runs. */
|
||||
export function __resetGrammarsForTest(): void {
|
||||
registry.clear();
|
||||
initialized = false;
|
||||
}
|
||||
@@ -0,0 +1,133 @@
|
||||
/**
|
||||
* Multi-language CodeBlock extractor — the tree-sitter replacement for the
|
||||
* Python-only `parseFile`. Emits the SAME `CodeBlock` shape so the downstream
|
||||
* security-ranking pipeline (classify/prioritize/reachability) is reused
|
||||
* verbatim.
|
||||
*
|
||||
* Fail-open on every path: `.py` uses the Python regex parser; any other
|
||||
* language with a missing grammar, parse timeout, or parser error yields `[]`
|
||||
* (honest absence) — never the Python regex, which would mis-parse brace/`class`
|
||||
* syntax into corrupt blocks. A hostile input cannot crash or hang a mission —
|
||||
* the parse is wall-clock time-bounded and every error is caught.
|
||||
*/
|
||||
import { Parser, type Node } from 'web-tree-sitter';
|
||||
import { getGrammar } from './ts-grammars.js';
|
||||
import { splitParamList } from './param-split.js';
|
||||
// Deliberate circular import with code-ingest.ts (it imports parseFileMultiLang).
|
||||
// Safe because both cross-references are used only inside function bodies, never
|
||||
// at module-evaluation time — do NOT hoist parseFile/CodeBlock to top-level use.
|
||||
import { parseFile, type CodeBlock } from './code-ingest.js';
|
||||
|
||||
/** Hard per-file parse bound (wall-clock ms). A pathological input hits this and fail-opens. */
|
||||
const PARSE_BUDGET_MS = 5000;
|
||||
|
||||
let sharedParser: Parser | null = null;
|
||||
function getParser(): Parser {
|
||||
if (!sharedParser) sharedParser = new Parser();
|
||||
return sharedParser;
|
||||
}
|
||||
/**
|
||||
* Drop the parser after a timeout/error so the next parse starts clean. MUST
|
||||
* free the native (WASM-backed) parser first: web-tree-sitter has no GC-based
|
||||
* finalizer, so a dropped-but-not-deleted parser leaks its linear-memory
|
||||
* allocation. Left unfreed, repeated failures exhaust the shared WASM heap and
|
||||
* permanently wedge every subsequent parse (they abort, get caught, and yield
|
||||
* [] forever). Frees the module singleton, which is correct because production
|
||||
* always uses the default `getParser` (`makeParser` is a test-only seam).
|
||||
*/
|
||||
function resetParser(): void {
|
||||
sharedParser?.delete();
|
||||
sharedParser = null;
|
||||
}
|
||||
|
||||
function kindOf(nodeType: string): CodeBlock['kind'] {
|
||||
if (nodeType.includes('class')) return 'class';
|
||||
if (nodeType.includes('method') || nodeType.includes('constructor')) return 'method';
|
||||
return 'function';
|
||||
}
|
||||
|
||||
/** Map a captured definition node (+ its name/params captures) to a CodeBlock. */
|
||||
export function nodeToCodeBlock(
|
||||
def: Node,
|
||||
nameNode: Node,
|
||||
paramsNode: Node | undefined,
|
||||
path: string,
|
||||
lang: string,
|
||||
): CodeBlock {
|
||||
const name = nameNode.text;
|
||||
const lineStart = def.startPosition.row + 1; // tree-sitter rows are 0-indexed
|
||||
const lineEnd = def.endPosition.row + 1;
|
||||
return {
|
||||
id: `${path}::${name}@${lineStart}`,
|
||||
path,
|
||||
name,
|
||||
kind: kindOf(def.type),
|
||||
lineStart,
|
||||
lineEnd,
|
||||
params: paramsNode ? splitParamList(paramsNode.text, lang) : [],
|
||||
decorators: [],
|
||||
body: def.text,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract CodeBlocks from one file. `.py` keeps the regex parser (benchmark
|
||||
* stability); other languages use tree-sitter. Fail-open is language-scoped: the
|
||||
* Python regex parser only ever runs on `.py`. For any other language a missing
|
||||
* grammar, parse timeout, or parser error returns `[]` (nothing extracted) —
|
||||
* never the Python regex, which would mis-parse brace/`class` syntax and emit
|
||||
* corrupt blocks (wrong line spans, lost methods) rather than honest absence.
|
||||
*
|
||||
* Time-bounding note: only the `parse()` call is wall-clock bounded (via the
|
||||
* `budgetMs` arg, injectable so a test can drive the real cancellation path).
|
||||
* When called through `ingestRepository` the content is capped at `maxFileBytes`
|
||||
* (1 MB) upstream, which also bounds the near-linear match/conversion loop and
|
||||
* the WASM peak-memory demand (see createMultiLangIngestConfig).
|
||||
*
|
||||
* ponytail: unify Python onto tree-sitter only if CVE-Zero is re-baselined.
|
||||
*/
|
||||
export function parseFileMultiLang(
|
||||
path: string,
|
||||
content: string,
|
||||
ext: string,
|
||||
makeParser: () => Parser = getParser,
|
||||
budgetMs: number = PARSE_BUDGET_MS,
|
||||
): CodeBlock[] {
|
||||
if (ext === '.py') return parseFile(path, content);
|
||||
const g = getGrammar(ext);
|
||||
if (!g) return []; // unsupported / grammar not loaded — honest absence, not Python-regex garbage
|
||||
let tree: ReturnType<Parser['parse']> | undefined;
|
||||
try {
|
||||
const p = makeParser();
|
||||
p.setLanguage(g.language);
|
||||
// Bound parse wall-time: progressCallback returning true cancels the parse
|
||||
// (→ parse returns null). setTimeoutMicros is deprecated/broken in 0.25.
|
||||
const start = Date.now();
|
||||
tree = p.parse(content, undefined, {
|
||||
progressCallback: () => Date.now() - start > budgetMs,
|
||||
});
|
||||
if (!tree) {
|
||||
resetParser(); // parse cancelled (budget exceeded)
|
||||
return [];
|
||||
}
|
||||
const blocks: CodeBlock[] = [];
|
||||
for (const match of g.query.matches(tree.rootNode)) {
|
||||
const caps: Record<string, Node> = {};
|
||||
for (const c of match.captures) caps[c.name] = c.node;
|
||||
blocks.push(nodeToCodeBlock(caps.def, caps.name, caps.params, path, g.lang));
|
||||
}
|
||||
return blocks;
|
||||
} catch {
|
||||
// Fail-open. A native WASM OOM abort (huge/dense input near the 2GB heap
|
||||
// ceiling) surfaces here too; it is caught, not fatal, but leaves the shared
|
||||
// heap pinned high — the maxFileBytes cap keeps that unreachable in prod.
|
||||
resetParser();
|
||||
return []; // any parser error → fail-open (empty, not Python-regex garbage)
|
||||
} finally {
|
||||
// Free the native WASM-backed tree. nodeToCodeBlock already copied every
|
||||
// field out as a plain JS string, so nothing dangles after delete().
|
||||
// Without this the linear-memory heap grows unbounded and eventually
|
||||
// poisons the shared parser for the rest of the process.
|
||||
tree?.delete();
|
||||
}
|
||||
}
|
||||
+32
-12
@@ -8,16 +8,36 @@
|
||||
* itself makes (worker + orchestrator models). It is otherwise pure of network
|
||||
* side effects.
|
||||
*
|
||||
* SCOPE CAVEAT (inherited from code-ingest): the ingest layer is a PYTHON-ONLY,
|
||||
* regex "AST-lite" prototype — see recon/code-ingest.ts. It selects which Python
|
||||
* blocks to analyze first by security exposure; anything non-`.py` or imported
|
||||
* from outside the scan scope is invisible. Do NOT describe this as "any repo".
|
||||
* SCOPE CAVEAT (inherited from code-ingest): the ingest layer extracts blocks via
|
||||
* web-tree-sitter for Python/JS/TS/Go/Java/C/C++ (Python uses the regex "AST-lite"
|
||||
* path; other languages fail-open to [] — never the Python regex — on a grammar
|
||||
* miss or parse error) — see recon/code-ingest.ts + recon/ts-parse.ts. It selects which
|
||||
* blocks to analyze first by security exposure; a language without a bundled grammar,
|
||||
* or code imported from outside the scan scope, is invisible. Do NOT describe this as
|
||||
* "any repo" or "any language".
|
||||
*
|
||||
* KNOWN v1 EXTRACTION LIMITS (fail-safe — these shapes are silently NOT extracted or
|
||||
* under-linked, never mis-extracted; all strictly better than the prior Python-only
|
||||
* ingest, and tracked as follow-ups):
|
||||
* - JS/TS: only `function`/`method`/`class` declarations are captured. Arrow-function
|
||||
* and function-expression definitions (`const f = () => …`) are not — idiomatic in
|
||||
* modern TS, so their sinks may go unranked.
|
||||
* - C++: only free functions are captured. In-class and out-of-line (`Class::method`)
|
||||
* member methods are not — a materially narrower C++ story than "full support".
|
||||
* - Entry-point elevation for non-Python code relies on name heuristics only; no
|
||||
* language emits `decorators`, so annotation-based routes (Spring `@GetMapping`,
|
||||
* TS `@Get()`, Express `app.get(...)`) are not elevated to `exposed_externally`.
|
||||
* - Call-graph/reachability: buildCallGraph strips a block's signature line before
|
||||
* scanning for callees (kept unchanged for Python-benchmark stability), so a
|
||||
* SINGLE-LINE non-Python definition (`func F(u) { return G(u) }`) misses the call
|
||||
* on that line — the block is still extracted and sink-classified, only the edge
|
||||
* is lost.
|
||||
*/
|
||||
|
||||
import { config } from '../config/index.js';
|
||||
import {
|
||||
ingestRepository,
|
||||
createPythonIngestConfig,
|
||||
createMultiLangIngestConfig,
|
||||
packAnalysisUnits,
|
||||
} from './code-ingest.js';
|
||||
import { DecompositionOrchestrator } from '../orchestration/index.js';
|
||||
@@ -218,7 +238,7 @@ export function ingestRepoToSourceContext(
|
||||
): RepoSourceContext {
|
||||
const source = resolveRepoSourceForAnalysis(repoPath);
|
||||
try {
|
||||
const result = ingestRepository(createPythonIngestConfig(source.repoPath));
|
||||
const result = ingestRepository(createMultiLangIngestConfig(source.repoPath));
|
||||
const packed = packAnalysisUnits(result.analysisUnits, tokenBudget);
|
||||
return {
|
||||
sourceContext: packed.text,
|
||||
@@ -302,21 +322,21 @@ export async function runWhiteboxAnalysis(
|
||||
|
||||
// 1) ingest source, security-rank the blocks
|
||||
try {
|
||||
const result = ingestRepository(createPythonIngestConfig(source.repoPath));
|
||||
const result = ingestRepository(createMultiLangIngestConfig(source.repoPath));
|
||||
|
||||
// 2) pack into a generous source view (orchestrator re-packs per-worker itself)
|
||||
const packed = packAnalysisUnits(result.analysisUnits, DEFAULT_ANALYSIS_SOURCE_BUDGET);
|
||||
const sourceContext = packed.text;
|
||||
|
||||
// Fail loud ONLY when there is genuinely NO source to analyze: 0 ingestable source files
|
||||
// (non-Python or empty repo). A valid Python repo that has files but no def/class (a
|
||||
// module-level-only script) yields 0 blocks -> empty packed text; that used to run the
|
||||
// orchestrator, so we do NOT regress it into a 500 — gate the throw on the file count, not
|
||||
// on empty packed text, and let a files>0 repo proceed as before.
|
||||
// (no supported-language files, or empty repo). A valid repo that has files but no
|
||||
// def/class/func (a module-level-only script) yields 0 blocks -> empty packed text; that
|
||||
// used to run the orchestrator, so we do NOT regress it into a 500 — gate the throw on the
|
||||
// file count, not on empty packed text, and let a files>0 repo proceed as before.
|
||||
if (result.stats.files === 0) {
|
||||
throw new Error(
|
||||
'white-box analysis has no analyzable source: the repo has 0 ingestable source files ' +
|
||||
'(non-Python, or empty). Point repoPath at a Python codebase.',
|
||||
'(no supported-language source, or empty). Point repoPath at a code repository.',
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
@@ -32,6 +32,7 @@ import { createTargetFromUrl, createTargetFromIP } from './target/index.js';
|
||||
import type { OperatorArchetype, LLMProvider } from './types/index.js';
|
||||
import { listOperatorPrompts, setOperatorOverride, resetOperatorOverride, type OperatorOverride } from './operators/index.js';
|
||||
import { ingestRepoToSourceContext, runWhiteboxAnalysis, resolveRepoSourceForAnalysis, RepoCloneError, RepoPathError } from './recon/whitebox.js';
|
||||
import { initGrammars } from './recon/ts-grammars.js';
|
||||
import { redactCredential } from './evidence/index.js';
|
||||
|
||||
const execFileAsync = promisify(execFile);
|
||||
@@ -7789,6 +7790,12 @@ async function startServer() {
|
||||
await loadPersistedState();
|
||||
llm = await initLLM();
|
||||
|
||||
// Load multi-language grammars once, before any white-box ingest request.
|
||||
// initGrammars is internally fail-open (never rejects): on failure the
|
||||
// registry stays empty, Python still ingests via the regex parser, and other
|
||||
// languages yield no blocks (it logs a warning).
|
||||
await initGrammars();
|
||||
|
||||
// Graceful-shutdown flush: on SIGTERM/SIGINT (Ctrl-C, docker stop, systemctl restart) write
|
||||
// any pending debounced snapshot before exiting, so the persistState debounce can't lose the
|
||||
// last <1s of state. Registered here (the server-start path) so it never affects test imports.
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
import { defineConfig } from 'vitest/config';
|
||||
|
||||
// Coverage-only configuration. Test discovery, environment, and everything else
|
||||
// stay on Vitest defaults — this file exists solely to enforce the multi-language
|
||||
// ingest quality gate: 100% coverage (statements/branches/functions/lines) on the
|
||||
// three fully-new source files this feature adds.
|
||||
//
|
||||
// Edited files (code-ingest.ts, whitebox.ts, server.ts) are NOT in `include`: a
|
||||
// whole-file 100% threshold is unachievable on large pre-existing files, and the
|
||||
// gate for them is "new branches covered", verified by the multilang/regression
|
||||
// suites rather than a file-level threshold. The single server.ts bootstrap line
|
||||
// (`await initGrammars()`) is the documented carve-out — smoke-covered, and
|
||||
// initGrammars itself is unit-tested to 100% in ts-grammars.test.ts.
|
||||
export default defineConfig({
|
||||
test: {
|
||||
coverage: {
|
||||
provider: 'v8',
|
||||
include: [
|
||||
'src/recon/ts-grammars.ts',
|
||||
'src/recon/param-split.ts',
|
||||
'src/recon/ts-parse.ts',
|
||||
],
|
||||
thresholds: {
|
||||
perFile: true,
|
||||
statements: 100,
|
||||
branches: 100,
|
||||
functions: 100,
|
||||
lines: 100,
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
Reference in new issue
Block a user