Joseph Magly
afc9dad1b2
* docs(policy): allow low-impact contributor courtesy cleanup * chore(aiwg): migrate to shared BT6 maintainer plugin * chore: reconcile delivery validation
82 lines
4.0 KiB
Markdown
82 lines
4.0 KiB
Markdown
# UC-005 — Reproduce Claims and Benchmark Results
|
|
|
|
## Metadata
|
|
|
|
- **ID:** UC-005
|
|
- **Status:** Baselined (brownfield)
|
|
- **Level:** User goal
|
|
- **Scope:** Benchmark corpora, manifests, ground truth, receipts, graders, claim verification, and CI
|
|
- **Primary actor:** Maintainer, contributor, or evaluator
|
|
- **Goal:** Re-derive public capability claims from versioned evidence and detect grading, provenance, or fitting regressions.
|
|
- **Related stories:** US-08, US-09, US-10
|
|
- **Quality drivers:** NFR-03, NFR-08, NFR-09
|
|
|
|
## Preconditions
|
|
|
|
- The evaluated corpus, manifest, ground truth, and required receipts are available under their documented retention policy.
|
|
- The evaluator uses the declared harness, metric, model/configuration identity, and command.
|
|
- Sensitive or third-party material is handled according to repository policy.
|
|
|
|
## Trigger
|
|
|
|
The evaluator runs claim verification, a benchmark verifier, or the corresponding CI gate.
|
|
|
|
## Main Success Scenario
|
|
|
|
1. The harness loads the versioned corpus/manifest and ground truth.
|
|
2. It validates required receipt identity and provenance fields.
|
|
3. The grader recomputes per-case outcomes from retained evidence.
|
|
4. Aggregate scores and headline verdicts are recomputed using the declared metric.
|
|
5. Anti-fitting checks verify separation and policy invariants.
|
|
6. The verifier compares derived values with public claims.
|
|
7. It emits a pass/fail result and enough context to reproduce a mismatch.
|
|
8. CI blocks integration when a required claim, provenance, or anti-fitting invariant fails.
|
|
|
|
## Alternate Flows
|
|
|
|
- A local evaluator runs only deterministic verification against committed artifacts.
|
|
- A live benchmark run produces new receipts, followed by deterministic grading in a separate step.
|
|
- Raw sensitive transcripts may be withheld while a declared, re-derivable graded receipt remains available; the documentation must state the reproduction boundary.
|
|
|
|
## Exception Flows
|
|
|
|
- Missing or stale evidence: fail the affected claim rather than substituting an estimate.
|
|
- Model/provider outage during a live run: classify as infrastructure/run failure, not model success or failure.
|
|
- Ground-truth or manifest mismatch: stop grading and identify the conflicting artifact.
|
|
- Suspected self-fitting or corpus leakage: fail the anti-fitting gate pending review.
|
|
|
|
## Postconditions
|
|
|
|
- Every passing headline claim maps to the evaluated corpus, metric, and retained evidence.
|
|
- Failed claims identify the discrepant artifact or value.
|
|
- Benchmark scope is not generalized to unsupported product maturity.
|
|
|
|
## Acceptance Criteria
|
|
|
|
- [ ] `npm run verify-claims` derives required public claims from repository artifacts.
|
|
- [ ] Seeded claim mismatches, missing provenance, or fitting violations cause a non-zero gate result.
|
|
- [ ] Receipts identify the relevant corpus/model/configuration and evaluation outcome.
|
|
- [ ] Live-run infrastructure failures remain distinct from evaluated model verdicts.
|
|
- [ ] Maturity documentation limits conclusions to the evaluated scope.
|
|
|
|
## Design and Test Traceability
|
|
|
|
| Concern | Implementation | Verification |
|
|
| --- | --- | --- |
|
|
| Claim derivation | `scripts/verify-claims.mjs`, benchmark manifests/receipts | Claim verification in CI |
|
|
| Finding grading | Verification/refutation and grading scripts | Script self-tests and benchmark tests |
|
|
| Anti-fitting | No-fitting and no-self-fitting checks | Seeded anti-fitting tests |
|
|
| Gate enforcement | `.github/workflows/ci.yml` | CI job status and local command parity |
|
|
|
|
## References
|
|
|
|
- @.aiwg/architecture/adr-004.md — Evidence-derived public-claims decision.
|
|
- @.aiwg/architecture/adr-005.md — Current-state versus research-vision decision.
|
|
- @.aiwg/architecture/vision-alignment.md — Maturity interpretation rules.
|
|
- @scripts/verify-claims.mjs — Deterministic public-claim verifier.
|
|
- @scripts/verify-finding.mjs — Finding verification.
|
|
- @scripts/refute-finding.mjs — Candidate-refutation path.
|
|
- @scripts/test-no-fitting.mjs — Anti-fitting evidence.
|
|
- @scripts/test-no-self-fitting.mjs — Self-fitting prevention evidence.
|
|
- @.github/workflows/ci.yml — Required automated gates.
|