feat(ingest): multi-language white-box ingest (web-tree-sitter) (#75)
* chore(ingest): pin web-tree-sitter@0.25.10 + tree-sitter-wasms@0.1.13; add multilang-ingest design docs * feat(ingest): multi-language grammar registry (web-tree-sitter, 8 langs, fail-open); 100% cov * feat(ingest): language-aware param-name splitter (py/go/ts/java/c); 100% cov * feat(ingest): tree-sitter CodeBlock extractor with wall-clock parse budget + fail-open * feat(ingest): cross-lang sinks + multilang config + wire tree-sitter into ingestRepository (sync) * feat(ingest): repoint white-box production callers to multilang config + prod regression test * server-init-wiring: load multi-language grammars once at bootstrap initGrammars() runs after LLM init, before app.listen. Internally fail-open (never rejects): on failure the grammar registry stays empty and ingest falls back to the Python regex parser. * adversarial-tests: hostile-input hardening for untrusted target source Malformed per-language source, wrong-language content, binary/non-UTF8, empty, unicode identifiers, 20k-func large file (wall-clock budget), deeply nested input, and a symlink-loop crawl — all must fail-open, never throw or spin. Drives parseFileMultiLang + ingestRepository/crawl. * readme-status + docs: flip white-box status to multi-language ingest README source-code rows now state web-tree-sitter multi-language ingest (fail-open to Python regex). whitebox.ts scope caveat + no-source error message updated off the stale Python-only wording. * coverage-setup: enforce 100% coverage gate on the three new ingest files vitest.config.ts adds a coverage-only block (v8, perFile thresholds=100) scoped to ts-grammars.ts, param-split.ts, ts-parse.ts. Test discovery and env stay on defaults. Edited files gate on new-branch coverage via the regression suites; the server.ts initGrammars bootstrap line is the documented smoke-covered carve-out. * review: fix WASM heap leak, language-scoped fail-open, sink recall/precision Adversarial security + strict code review (two independent subagents) returned one blocking bug and several correctness findings; remediated: BLOCKING #1 — native WASM memory leak: web-tree-sitter Tree/Parser objects were never .delete()'d. Both reviewers reproduced a process-global wedge: after ~7 large parses the shared WASM heap exhausts, every subsequent parse aborts (caught by fail-open), and non-Python ingest silently returns [] forever. Fix: free the tree in a finally; delete the parser in resetParser. Regression test parses ~8MB through the real shared parser 12x. regex, which mis-parses brace/class syntax into corrupt blocks (wrong line spans, lost methods). Now non-.py fail-open returns [] (honest absence); the Python regex only ever runs on .py. Isolated ts-parse-fallback test. http(s).request, axios(config), C exec-family); negative lookbehind stops bare system(/popen(/fetch( from matching Python receivers (os.system etc. still matched, corpus rankings unchanged). Recall+precision tests added. operator sees non-Python degradation. (one-line call-graph edges, no decorators, C++ member methods). * review round 2: fix sink-regex regressions, real timeout test, deterministic leak guard Third review round (correctness + security re-attack + test-quality) against the round-1 fixes surfaced issues in the fixes themselves; remediated: - Sink-regex regressions I introduced: (1) my fetch lookbehind dropped window.fetch/this.fetch (top JS SSRF idiom) — reverted to \bfetch\(; (2) bare .Get/.Post/.Do flagged db.Get/cache.Get/queue.Do as SSRF — receiver-qualified to [Cc]lient. Recall+precision tests cover both; regex is ReDoS-free (timed). - Placebo leak test: the 12x function-dense loop did NOT wedge the parser on machines with heap headroom (proven: passed with tree.delete removed). Replaced with a deterministic vi.spyOn(Tree.prototype,'delete') assertion, verified RED without the fix. - Real (non-mocked) parse-timeout test: parseFileMultiLang gains an injectable budgetMs; a 0ms budget drives the real progressCallback cancellation to []. - runWhiteboxAnalysis swap now has coverage (spy on orchestrator.run) so the 2nd production caller can't silently revert to the Python-only config. - Un-vacuoused weak tests: unicode now asserts the extracted block; nested/ wrong-language wrapped in invariants() + timing bounds. - CI now runs npm run test:coverage (the 100% gate was defined but unenforced). - Security: documented maxFileBytes as the control bounding a single parse's peak WASM memory below the 2GB heap-abort ceiling. - Stale 'fail-open to Python regex' docs fixed (ts-parse, whitebox, server, adversarial); circular-import note added; temp-dir cleanup; honest coverage wording in spec + config. * docs(ingest): align fallback-contract text with the shipped language-scoped fail-open Review feedback (#75, @jmagly): the design docs + one ts-grammars comment still described the pre-implementation 'fall back to the Python regex' model for non-Python files. The shipped contract is language-scoped: .py uses parseFile; any non-.py file with a missing/unsupported/failed grammar or a parse timeout returns [] (never the Python regex, which would mis-parse brace/class syntax into corrupt blocks). Updated every stale reference in multilang-ingest-spec.md, multilang-ingest-plan.md, and ts-grammars.ts so the docs match the security boundary. Docs/comment only — no code change. * fix(test): make the ReDoS wall-clock guard robust under coverage instrumentation Review feedback (#75, @jmagly): the CI step this PR adds (npm run test:coverage = vitest run --coverage) runs the full suite under V8 coverage, and that overhead pushes the pathological-brace DoS timing guard in local-agent-tool-calling.test.ts:69 over its <300ms bound (320 < 300 fails on CI). Raise the bound to 1000ms — still 2x below the ~2100ms quadratic-ReDoS regression it guards against, so it fails hard if the regression returns, but no longer false-fails under coverage/slow CI. Chose the reviewer's option (a) (robust bound) over scoping test:coverage, so the gate stays a simple full-suite run and no test-file enumeration can rot. Verified: npm run test:coverage is green and the 3 new files stay at 100%. * fix(ingest): cover pointer-returning C functions --------- Co-authored-by: Joseph Magly <1159087+jmagly@users.noreply.github.com>
This commit is contained in:
Lyubomir BozhinovandJoseph Magly authored and GitHub committed
1 parent
440dedd9c1
commit
151d3555c6
24 files changed
+1681
-25
No files matched your search
@@ -0,0 +1,243 @@
|
||||
# Plan: T3MP3ST-MLINGEST — Multi-language white-box ingest (web-tree-sitter)
|
||||
|
||||
## Executive Summary
|
||||
- **Goal:** Replace the Python-only `parseFile` stage with a portable `web-tree-sitter` extractor emitting the existing `CodeBlock` shape; reuse the security-ranking pipeline verbatim; language-scoped fail-open (`.py` keeps the Python regex; other languages return `[]` on a grammar miss / parse error — never the Python regex).
|
||||
- **Complexity:** Medium-low (no async ripple — see Arch). **Risk:** Low-Medium. **Diff:** 3 new source files (`param-split`, `ts-grammars`, `ts-parse`) + edits to 3 (`code-ingest`, `whitebox`, `server`) + coverage config + test suites.
|
||||
- **Spec:** `2026-07-10-multilang-ingest-design.md`. **Reviews folded:** architect-review (async→sync pivot + 5 defects), test-automator (RED discrimination + granularity).
|
||||
|
||||
## 🚨 Critical Implementation Standards
|
||||
- Fail-open mandatory (language-scoped): `.py` → `parseFile`; a non-`.py` grammar init/parse failure → `[]` (never the Python regex). Ingest never crashes a mission.
|
||||
- Security ranking reused, not rewritten (`classify`/`prioritize`/`reachability`/`context-pack` untouched).
|
||||
- vitest green, `tsc --noEmit` clean, `eslint` clean. No TODO/HACK/dead code; every failure path handled.
|
||||
- **100% coverage of our code (hard gate).** Statements/branches/functions/lines = 100% on every NEW file (`param-split.ts`, `ts-grammars.ts`, `ts-parse.ts`) and 100% of every new branch/function added to edited files (`code-ingest.ts`, `whitebox.ts`). Sole carve-out: the **single** `await initGrammars(...)` wiring line at `server.ts` bootstrap — `initGrammars` itself is 100% unit-tested in `grammar-registry`; the wiring line is exercised by `npm run smoke` at the ship gate. (`cli.ts` is NOT wired — the CLI never ingests; ingest is reached only via `server.ts:6060/6126`.) Coverage is necessary-not-sufficient — the review gate confirms assertions are meaningful, not line-touching.
|
||||
- **Adversarial phase mandatory (this parses untrusted target source).** A crafted hostile-input suite + a `security-test-engineer` subagent actively trying to break the built code; all defects fixed before review.
|
||||
- **Strict subagent review is the merge gate.** `/code-review-pr-strict` methodology (parallel specialists, security-critical detection FIRST, false-positive elimination) on the full diff; every confirmed finding resolved; re-review until clean.
|
||||
|
||||
## Architecture — SYNC ingest + bootstrap init (revised per architect §2)
|
||||
web-tree-sitter: only `Parser.init()`/`Language.load(wasm)` are async (one-time); `parser.parse()` is **sync**. So we hoist init to bootstrap and keep the ingest path synchronous — **`ingestRepository` stays sync**, eliminating the entire async ripple through `whitebox.ts`/`server.ts`/tests.
|
||||
|
||||
```
|
||||
bootstrap (server start / CLI main): await initGrammars(exts) // load all grammars once into a module singleton; try/catch → empty registry on failure (fail-open)
|
||||
─────────────────────────────────────
|
||||
crawl (multi-lang includeExts)
|
||||
→ ingestRepository (SYNC, unchanged signature) — per file:
|
||||
parseFileMultiLang(path, content, ext) // SYNC
|
||||
├─ ext == .py → parseFile(...) // Python regex, unchanged
|
||||
├─ non-.py, getGrammar(ext) loaded? → parser.parse() → query → nodeToCodeBlock[] → CodeBlock[]
|
||||
└─ non-.py, no grammar / not-yet-loaded / timeout / error → [] // FAIL-OPEN (never the Python regex)
|
||||
→ buildCallGraph → findEntryPoints → reachability → classify → prioritize (UNCHANGED)
|
||||
→ context-pack → orchestrator (UNCHANGED)
|
||||
```
|
||||
Race: a mission firing before init resolves simply fail-opens for that call (`.py` → Python regex, other languages → `[]`) — degrade, never crash. Grammars eager-loaded at bootstrap (~few MB, one-time) — the cost of a sync hot path.
|
||||
|
||||
**Killed by the sync pivot:** the async signature change on `ingestRepository`; the `ingest-limits.test.ts` await churn (D1); the unguarded-init mission crash (D5, now a bootstrap try/catch).
|
||||
**NOT killed — still required (validator BLOCKER):** `whitebox.ts` must still be edited to repoint BOTH production callers (`ingestRepoToSourceContext` L125, `runWhiteboxAnalysis` L205) from `createPythonIngestConfig` → `createMultiLangIngestConfig`. This config-swap is independent of sync/async; without it the multilang extractor only ever sees `.py` in production and the PR is a no-op. Node `whitebox-wiring` below.
|
||||
|
||||
## Current State (grounded in /tmp/T3MP3ST)
|
||||
- `CodeBlock = {id, path, name, kind, lineStart, lineEnd, params, decorators, body}` (`id=\`${path}::${name}@${lineStart}\``; `decorators=[]` for non-Python).
|
||||
- `parseFile` (code-ingest.ts:409, stays as the `.py` path only); `parseParams` (decl L328, Python-only).
|
||||
- `ingestRepository` (L749, **stays sync**); sink regexes L169–201; `crawl` L238.
|
||||
- **Two** production `ingestRepository` callers, both in `whitebox.ts`: `ingestRepoToSourceContext` (L125) and `runWhiteboxAnalysis` (L205) — both pass `createPythonIngestConfig`. (`server.ts:6060` calls `ingestRepoToSourceContext`, not `ingestRepository` directly.) Both must be repointed — see `whitebox-wiring`.
|
||||
|
||||
## Node ownership split (execution-offload datapoint)
|
||||
- **Owned (local:false — specialized / cross-cutting / verification):** `deps-assets`, `coverage-setup`, `grammar-registry` (+`initGrammars`), `server-init-wiring`, `ts-parse`, `ingest-wiring`, `whitebox-wiring`, `adversarial-tests`. (8)
|
||||
- **Offloaded (local:true — mechanical single-region, committed RED test):** `param-split`, `sink-patterns`, `multilang-config`, `readme-status`. (4)
|
||||
- 12 manifest nodes; the Adversarial-exploration and Strict-review **phases** below are subagent-driven gates, not manifest nodes.
|
||||
- **Retained-`local:false`, deliberate:** `nodeToCodeBlock` (pure map inside `ts-parse`) is offload-able in principle, but a discriminating RED test for it needs a real tree-sitter Node (no deterministic offline parse without the loaded grammar), so its natural RED test *is* the `ts-parse` integration test — extracting it would create a node whose test can't be cleanly pre-committed. Conscious retention per validator finding 5.
|
||||
|
||||
**Execution prerequisite:** every `local:true` node's `accept` runs a pre-committed RED test file that **no node creates** — the orchestrator commits all RED baselines to the repo *before* running the manifest (Promise-Theory: the local model never authors its own test). Running the manifest cold without this step makes those accepts fail with "no test files found."
|
||||
|
||||
## RED baselines (committed before impl; local model sees only the test)
|
||||
|
||||
**`param-split`** — `src/__tests__/param-split.test.ts` (＋discriminating same-raw case per tester)
|
||||
```ts
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { splitParamList } from '../recon/param-split.js';
|
||||
describe('splitParamList', () => {
|
||||
it('python: drops self, strips annotations/defaults/*args', () =>
|
||||
expect(splitParamList('self, url: str, timeout=5, *args', 'py')).toEqual(['url','timeout']));
|
||||
it('go name-first', () => expect(splitParamList('ctx context.Context, id string', 'go')).toEqual(['ctx','id']));
|
||||
it('java type-first', () => expect(splitParamList('String url, int max', 'java')).toEqual(['url','max']));
|
||||
it('ts name: type', () => expect(splitParamList('url: string, opts?: Opts', 'ts')).toEqual(['url','opts']));
|
||||
it('empty', () => expect(splitParamList('', 'ts')).toEqual([]));
|
||||
it('SAME raw, different lang → forces real dispatch', () => { // ← tester fix
|
||||
expect(splitParamList('a b', 'go')).toEqual(['a']); // name-first
|
||||
expect(splitParamList('a b', 'java')).toEqual(['b']); // type-first
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
**`sink-patterns`** — `src/__tests__/sink-patterns.test.ts` (scoped to 2 contiguous regexes; ＋false-positive guards)
|
||||
```ts
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { DANGEROUS_SINK_RE, OUTBOUND_REQUEST_RE } from '../recon/code-ingest.js';
|
||||
describe('cross-language sinks', () => {
|
||||
it('matches new sinks', () => {
|
||||
for (const s of ['Runtime.getRuntime().exec(cmd)','exec.Command("sh")','child_process.exec(x)','system(buf)','popen(cmd)'])
|
||||
expect(DANGEROUS_SINK_RE.test(s)).toBe(true);
|
||||
});
|
||||
it('regression: python sinks still match', () => expect(DANGEROUS_SINK_RE.test('os.system(x)')).toBe(true));
|
||||
it('no FP on benign / System.out / filesystem ids', () => { // ← tester fix
|
||||
expect(DANGEROUS_SINK_RE.test('const total = sum(a, b)')).toBe(false);
|
||||
expect(DANGEROUS_SINK_RE.test('System.out.println("debug")')).toBe(false);
|
||||
expect(DANGEROUS_SINK_RE.test('fileSystem.readFile(x)')).toBe(false);
|
||||
});
|
||||
it('outbound covers go/js http', () => {
|
||||
for (const s of ['http.Get(u)','fetch(u)','axios.get(u)']) expect(OUTBOUND_REQUEST_RE.test(s)).toBe(true);
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
**`multilang-config`** — `src/__tests__/multilang-config.test.ts` (all exts ＋negative per tester)
|
||||
```ts
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { mkdtempSync, writeFileSync } from 'node:fs';
|
||||
import { join } from 'node:path'; import { tmpdir } from 'node:os';
|
||||
import { crawl, createMultiLangIngestConfig } from '../recon/code-ingest.js';
|
||||
describe('createMultiLangIngestConfig', () => {
|
||||
const EXTS = ['py','js','ts','tsx','go','java','c','cpp'];
|
||||
it('crawls every supported ext, excludes non-source', () => {
|
||||
const root = mkdtempSync(join(tmpdir(), 'mling-'));
|
||||
for (const e of EXTS) writeFileSync(join(root, `f.${e}`), '\n');
|
||||
writeFileSync(join(root, 'd.md'), '# ignore');
|
||||
const files = crawl(createMultiLangIngestConfig(root));
|
||||
for (const e of EXTS) expect(files.some(f => f.endsWith(`f.${e}`))).toBe(true); // no degenerate hardcode
|
||||
expect(files.some(f => f.endsWith('d.md'))).toBe(false); // no wildcard/drop-filter
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
**`readme-status`** — accept grep (corrected; all three rows, "regex" word — per tester finding 2).
|
||||
|
||||
## Owned-node acceptance tests (real assertions committed up front — tester finding 5)
|
||||
- **`grammar-registry`** — `ts-grammars.test.ts`: registry has entries for all 8 exts with non-empty query + `@name`/`@def` captures; **`initGrammars` fail-open**: forcing a bad wasm path leaves the registry empty and `getParser` returns undefined, **no throw** (covers D5).
|
||||
- **`ts-parse`** — `ts-parse.test.ts`: parse a TS and a Go function → assert `name/params/lineStart/body`; unsupported non-`.py` `.zzz` → returns `[]`, no throw; `.py` → byte-identical to `parseFile`.
|
||||
- **`server-init-wiring`** — `tsc --noEmit` clean; the single bootstrap line covered by `npm run smoke` at the ship gate (`initGrammars` itself is 100% unit-tested in `grammar-registry`).
|
||||
- **`ingest-wiring`** (headline regression) — `ts-parse-multilang-rank.test.ts`: temp repo `.py`+`.go`+`.ts`+`.java`, each a sink-bearing function; `ingestRepository(createMultiLangIngestConfig(root))` (sync) → non-`.py` blocks present, classified `attack_surface`, priority > a neutral block; **PLUS a cross-language reachability case** (Go/TS entry-point → reachable callee gets the reach bonus) to guard `buildCallGraph` name-matching on non-Python ids (tester finding 5). Existing `code-ingest.test.ts` unchanged (sync preserved).
|
||||
- **`whitebox-wiring`** (production-path regression) — `whitebox-multilang.test.ts`: temp Go+TS repo through `ingestRepoToSourceContext(root)` → asserts non-`.py` source in the returned `sourceContext` (the real server/CLI entry, not the config directly); `tsc --noEmit` + `eslint` green; `whitebox-containment` tests still green.
|
||||
|
||||
## Execution Manifest
|
||||
```json
|
||||
{
|
||||
"execution-manifest": [
|
||||
{
|
||||
"id": "deps-assets",
|
||||
"files": ["package.json"],
|
||||
"change": "Add web-tree-sitter dependency and bundle prebuilt grammar .wasm assets for py/js/ts/tsx/go/java/c/cpp (pinned versions), ensuring they ship to dist. package.json + build/copy only — no source parser code here. Environmental — author yourself.",
|
||||
"accept": "npm ci >/dev/null 2>&1 && node -e \"require('web-tree-sitter')\" && test -d node_modules/web-tree-sitter",
|
||||
"local": false
|
||||
},
|
||||
{
|
||||
"id": "coverage-setup",
|
||||
"files": ["package.json", "vitest.config.ts"],
|
||||
"change": "Add @vitest/coverage-v8 devDependency and create vitest.config.ts containing ONLY a coverage block (provider v8; include the new files src/recon/param-split.ts, ts-grammars.ts, ts-parse.ts; per-file thresholds statements/branches/functions/lines = 100). Do NOT add test.include/root/environment keys — the existing zero-config `vitest run src` behavior must be byte-preserved. Owned (config).",
|
||||
"accept": "npx vitest run src >/dev/null 2>&1 && npx vitest run --coverage src/__tests__/redact.test.ts 2>&1 | grep -qiE 'coverage|% ' && test -f vitest.config.ts",
|
||||
"local": false
|
||||
},
|
||||
{
|
||||
"id": "grammar-registry",
|
||||
"files": ["src/recon/ts-grammars.ts", "src/__tests__/ts-grammars.test.ts"],
|
||||
"change": "Create ts-grammars.ts: extension->{languageId, wasmFile, query (tree-sitter S-expr with @name/@def), captures}; a wasm-path resolver; initGrammars(exts) (async, one-time, cached, try/catch -> empty registry on failure); getParser(ext) sync accessor. Specialized queries + fail-open init — author yourself. Author ts-grammars.test.ts as the RED baseline.",
|
||||
"accept": "npx vitest run src/__tests__/ts-grammars.test.ts",
|
||||
"local": false
|
||||
},
|
||||
{
|
||||
"id": "param-split",
|
||||
"files": ["src/recon/param-split.ts"],
|
||||
"change": "Create param-split.ts exporting splitParamList(raw, lang): bare param identifiers per test_param-split. Python drops self/cls, strips *args and ':'-annotations and '='-defaults; Go/TS name-first (name before ':' or space); Java type-first (name is last token of each comma part). Must branch on lang (the 'a b' case proves it). Empty -> [].",
|
||||
"accept": "npx vitest run src/__tests__/param-split.test.ts",
|
||||
"forbid": ["new_deps"],
|
||||
"kind": "create",
|
||||
"local": true
|
||||
},
|
||||
{
|
||||
"id": "ts-parse",
|
||||
"files": ["src/recon/ts-parse.ts", "src/__tests__/ts-parse.test.ts"],
|
||||
"change": "Create ts-parse.ts: nodeToCodeBlock(node, source, path) -> full CodeBlock (id, kind, lineStart/lineEnd from node position, params via splitParamList on the parameters-node text, decorators [], body via node text); parseFileMultiLang(path, content, ext): CodeBlock[] — SYNC — `.py` → parseFile; non-.py via getGrammar(ext), returning [] on no-grammar/unsupported/any error (never the Python regex). MUST bound parse time (wall-clock via progressCallback) so a pathological input cannot spin — on timeout, treat as parse failure and return []. Integration crux — author yourself.",
|
||||
"accept": "npx vitest run src/__tests__/ts-parse.test.ts",
|
||||
"local": false
|
||||
},
|
||||
{
|
||||
"id": "sink-patterns",
|
||||
"files": ["src/recon/code-ingest.ts"],
|
||||
"change": "Additively extend ONLY the two contiguous consts DANGEROUS_SINK_RE and OUTBOUND_REQUEST_RE (L173-179) with cross-language sinks (Java Runtime.exec/ProcessBuilder, Go exec.Command/http.Get, C system(/popen(, JS child_process/fetch/axios) per test_sink-patterns, WITHOUT dropping existing Python patterns and WITHOUT matching benign ids like System.out/fileSystem. Export both consts for the test. (SINK_EVIDENCE_RES enrichment deferred to a follow-up — keeps this a single region.)",
|
||||
"accept": "npx vitest run src/__tests__/sink-patterns.test.ts",
|
||||
"forbid": ["new_deps"],
|
||||
"local": true
|
||||
},
|
||||
{
|
||||
"id": "multilang-config",
|
||||
"files": ["src/recon/code-ingest.ts"],
|
||||
"change": "Add createMultiLangIngestConfig(repoRoot): IngestConfig mirroring createPythonIngestConfig with includeExts covering exactly py/js/ts/tsx/go/java/c/cpp (leading dots), per test_multilang-config (all included, .md excluded).",
|
||||
"accept": "npx vitest run src/__tests__/multilang-config.test.ts",
|
||||
"forbid": ["new_deps"],
|
||||
"local": true
|
||||
},
|
||||
{
|
||||
"id": "server-init-wiring",
|
||||
"files": ["src/server.ts"],
|
||||
"change": "Invoke `await initGrammars(SUPPORTED_EXTS)` ONCE at server bootstrap (before app.listen) so grammars are loaded before any ingest request; on init failure log and continue (registry empty -> .py via parseFile, other languages -> []). server.ts is the ONLY ingest entrypoint (6060/6126); do NOT wire cli.ts (the CLI never ingests). Cross-cutting entrypoint — author yourself.",
|
||||
"accept": "npx tsc --noEmit",
|
||||
"local": false
|
||||
},
|
||||
{
|
||||
"id": "ingest-wiring",
|
||||
"files": ["src/recon/code-ingest.ts", "src/__tests__/ts-parse-multilang-rank.test.ts"],
|
||||
"change": "In ingestRepository (KEEP SYNC), dispatch each file through parseFileMultiLang(path, content, ext) (language-scoped fail-open: .py -> parseFile, other languages -> []), preserving byte/file ceilings. Author the headline multilang-rank regression test (non-.py blocks extracted + attack_surface ranked + cross-language reachability). Do NOT change ingestRepository's signature or any caller. Author yourself.",
|
||||
"accept": "npx vitest run src/__tests__/ts-parse-multilang-rank.test.ts src/__tests__/code-ingest.test.ts",
|
||||
"local": false
|
||||
},
|
||||
{
|
||||
"id": "whitebox-wiring",
|
||||
"files": ["src/recon/whitebox.ts", "src/__tests__/whitebox-multilang.test.ts"],
|
||||
"change": "BLOCKING production wiring: in whitebox.ts repoint BOTH ingestRepository calls (ingestRepoToSourceContext L125 [sync], runWhiteboxAnalysis L205 [already async — leave async]) from createPythonIngestConfig to createMultiLangIngestConfig — a pure argument swap, no signature change. Author the regression test whitebox-multilang.test.ts driving a temp Go+TS repo through ingestRepoToSourceContext and asserting non-.py source appears in the returned sourceContext (proves the feature reaches production, not just tests). Verify existing whitebox-containment tests stay green. Author yourself.",
|
||||
"accept": "npx vitest run src/__tests__/whitebox-multilang.test.ts src/__tests__/whitebox-containment.test.ts && npx tsc --noEmit && npm run lint",
|
||||
"local": false
|
||||
},
|
||||
{
|
||||
"id": "adversarial-tests",
|
||||
"files": ["src/__tests__/ts-parse-adversarial.test.ts"],
|
||||
"change": "Author a hostile-input suite exercising the FULL wired path (parseFileMultiLang + ingestRepository): malformed/syntactically-broken source per language; a .go file containing Python (and vice-versa); binary/non-UTF8 bytes; a multi-MB generated file and a deeply-nested/very-long-line file (byte/file ceilings hold); a PATHOLOGICAL input that would make tree-sitter spin (asserts parse returns within the timeout and fail-opens — proves the ts-parse timeout is real, not aspirational); a symlink and a symlink-loop under the repo root (crawl must not escape the contained root or hang); empty file; unicode identifiers; a file that makes tree-sitter error. Assert: never throws (non-.py fail-opens to [], .py to parseFile), bounded wall-time, CodeBlock invariants hold (lineStart<=lineEnd, body non-empty when name present, params is string[]), ingest resolves. Owned (crafted adversarial inputs).",
|
||||
"accept": "npx vitest run src/__tests__/ts-parse-adversarial.test.ts",
|
||||
"local": false
|
||||
},
|
||||
{
|
||||
"id": "readme-status",
|
||||
"files": ["README.md"],
|
||||
"change": "Flip the two rows that literally say 'Python-only' to multi-language (web-tree-sitter), Python-regex fail-open noted: the 'What it hunts' row (⚠️ Python-only ingest, ~L56) and the 'What ships today' row (Python-only regex ingest, ~L115). Leave the 'Coverage by domain' Code row (~L130, 'experimental') alone — it is a maturity axis, not the Python-only axis, and stays honest until the PR proves out. (Multi-region doc edit — the single-region rule is waived for docs.)",
|
||||
"accept": "grep -qi 'web-tree-sitter' README.md && ! grep -qi 'Python-only regex ingest' README.md && ! grep -qi 'Python-only ingest' README.md",
|
||||
"forbid": ["new_deps"],
|
||||
"local": true
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Risk Assessment (post-review)
|
||||
- ~~[High] async ripple~~ → **eliminated** by the sync + bootstrap-init architecture.
|
||||
- **[Med] grammar wasm sourcing/version** — owned `deps-assets`; pin versions; the fail-open init means a bad bundle preserves Python ingest while non-Python files yield `[]`, not a crash.
|
||||
- **[Med] cross-language call-graph** — `buildCallGraph` name-matching may mis-handle non-Python ids; guarded by the reachability assertion in the headline test.
|
||||
- **[Low] sink-regex FP** — guarded by `System.out`/`fileSystem` negatives.
|
||||
- **[Low] bootstrap race** — pre-init-completion ingest preserves Python parsing while non-Python files yield `[]`.
|
||||
|
||||
## Rollout — gated phases (each can bounce back)
|
||||
1. **Build.** Branch `feat/multilang-ingest`; land the 12 nodes in manifest order (fleet: `start-fleet.sh`). I implement the 8 owned; cascade fills the 4 local.
|
||||
2. **Coverage gate (hard).** `npx vitest run --coverage` → **100%** on the new files (thresholds fail the run otherwise); manually confirm every new branch/function in `code-ingest.ts`/`whitebox.ts` is hit (diff-coverage), carve-out only the `server.ts`/`cli.ts` bootstrap lines (smoke-covered). Gap → add tests, re-run.
|
||||
3. **Adversarial phase.** `ts-parse-adversarial.test.ts` green, then dispatch a **`security-test-engineer`** subagent to actively break the built code against a hostile-repo corpus (malformed/mixed-language/binary/pathological inputs, resource-exhaustion, parser hangs). Every defect fixed; re-run coverage.
|
||||
4. **Strict review gate.** Apply the **`/code-review-pr-strict`** methodology via parallel specialist subagents on the FULL diff — **security-critical detection runs first**, then correctness/perf/maintainability, with false-positive elimination and the mandatory verification protocol. Confirm coverage is meaningful assertions (no line-touching). Resolve every confirmed finding; **re-review until clean**.
|
||||
5. **Ship.** `npm test` + `tsc --noEmit` + `eslint` + `npm run smoke` (covers the server bootstrap init line) + coverage all green → open PR; body carries spec §4 decision record, the node split, coverage report, adversarial summary, and review sign-off.
|
||||
- Follow-ups (own issues): SINK_EVIDENCE_RES cross-language enrichment; cxpak `serve` phase-2 ranking upgrade.
|
||||
|
||||
## Success Criteria
|
||||
- [ ] Non-`.py` functions extracted + security-ranked (headline regression green, incl. cross-lang reachability).
|
||||
- [ ] **Feature reaches production** — `whitebox-multilang.test.ts` drives multilang through `ingestRepoToSourceContext` (not just `createMultiLangIngestConfig` directly).
|
||||
- [ ] Fail-open verified at BOTH boundaries: init failure (empty registry) and per-file unsupported ext.
|
||||
- [ ] Python behavior byte-unchanged (existing suite green, no test edits).
|
||||
- [ ] Both README "Python-only" rows flipped. `tsc`/`eslint`/`vitest` all green (eslint gated in `whitebox-wiring`).
|
||||
- [ ] **100% coverage** on new files + all new branches in edited files (carve-out: bootstrap lines) — coverage gate green.
|
||||
- [ ] **Adversarial phase** passed: crafted-input suite green AND `security-test-engineer` subagent found no unfixed break.
|
||||
- [ ] **Strict review** (`/code-review-pr-strict`, security-first) clean — all confirmed findings resolved, assertions verified meaningful.
|
||||
|
||||
---
|
||||
**Status:** Draft — pending human sign-off
|
||||
**Date:** 2026-07-10
|
||||
Reference in new issue
Block a user