Joseph Magly
afc9dad1b2
* docs(policy): allow low-impact contributor courtesy cleanup * chore(aiwg): migrate to shared BT6 maintainer plugin * chore: reconcile delivery validation
4.0 KiB
4.0 KiB
UC-005 — Reproduce Claims and Benchmark Results
Metadata
- ID: UC-005
- Status: Baselined (brownfield)
- Level: User goal
- Scope: Benchmark corpora, manifests, ground truth, receipts, graders, claim verification, and CI
- Primary actor: Maintainer, contributor, or evaluator
- Goal: Re-derive public capability claims from versioned evidence and detect grading, provenance, or fitting regressions.
- Related stories: US-08, US-09, US-10
- Quality drivers: NFR-03, NFR-08, NFR-09
Preconditions
- The evaluated corpus, manifest, ground truth, and required receipts are available under their documented retention policy.
- The evaluator uses the declared harness, metric, model/configuration identity, and command.
- Sensitive or third-party material is handled according to repository policy.
Trigger
The evaluator runs claim verification, a benchmark verifier, or the corresponding CI gate.
Main Success Scenario
- The harness loads the versioned corpus/manifest and ground truth.
- It validates required receipt identity and provenance fields.
- The grader recomputes per-case outcomes from retained evidence.
- Aggregate scores and headline verdicts are recomputed using the declared metric.
- Anti-fitting checks verify separation and policy invariants.
- The verifier compares derived values with public claims.
- It emits a pass/fail result and enough context to reproduce a mismatch.
- CI blocks integration when a required claim, provenance, or anti-fitting invariant fails.
Alternate Flows
- A local evaluator runs only deterministic verification against committed artifacts.
- A live benchmark run produces new receipts, followed by deterministic grading in a separate step.
- Raw sensitive transcripts may be withheld while a declared, re-derivable graded receipt remains available; the documentation must state the reproduction boundary.
Exception Flows
- Missing or stale evidence: fail the affected claim rather than substituting an estimate.
- Model/provider outage during a live run: classify as infrastructure/run failure, not model success or failure.
- Ground-truth or manifest mismatch: stop grading and identify the conflicting artifact.
- Suspected self-fitting or corpus leakage: fail the anti-fitting gate pending review.
Postconditions
- Every passing headline claim maps to the evaluated corpus, metric, and retained evidence.
- Failed claims identify the discrepant artifact or value.
- Benchmark scope is not generalized to unsupported product maturity.
Acceptance Criteria
npm run verify-claimsderives required public claims from repository artifacts.- Seeded claim mismatches, missing provenance, or fitting violations cause a non-zero gate result.
- Receipts identify the relevant corpus/model/configuration and evaluation outcome.
- Live-run infrastructure failures remain distinct from evaluated model verdicts.
- Maturity documentation limits conclusions to the evaluated scope.
Design and Test Traceability
| Concern | Implementation | Verification |
|---|---|---|
| Claim derivation | scripts/verify-claims.mjs, benchmark manifests/receipts |
Claim verification in CI |
| Finding grading | Verification/refutation and grading scripts | Script self-tests and benchmark tests |
| Anti-fitting | No-fitting and no-self-fitting checks | Seeded anti-fitting tests |
| Gate enforcement | .github/workflows/ci.yml |
CI job status and local command parity |
References
- @.aiwg/architecture/adr-004.md — Evidence-derived public-claims decision.
- @.aiwg/architecture/adr-005.md — Current-state versus research-vision decision.
- @.aiwg/architecture/vision-alignment.md — Maturity interpretation rules.
- @scripts/verify-claims.mjs — Deterministic public-claim verifier.
- @scripts/verify-finding.mjs — Finding verification.
- @scripts/refute-finding.mjs — Candidate-refutation path.
- @scripts/test-no-fitting.mjs — Anti-fitting evidence.
- @scripts/test-no-self-fitting.mjs — Self-fitting prevention evidence.
- @.github/workflows/ci.yml — Required automated gates.