# UC-005 — Reproduce Claims and Benchmark Results

## Metadata

- **ID:** UC-005
- **Status:** Baselined (brownfield)
- **Level:** User goal
- **Scope:** Benchmark corpora, manifests, ground truth, receipts, graders, claim verification, and CI
- **Primary actor:** Maintainer, contributor, or evaluator
- **Goal:** Re-derive public capability claims from versioned evidence and detect grading, provenance, or fitting regressions.
- **Related stories:** US-08, US-09, US-10
- **Quality drivers:** NFR-03, NFR-08, NFR-09

## Preconditions

- The evaluated corpus, manifest, ground truth, and required receipts are available under their documented retention policy.
- The evaluator uses the declared harness, metric, model/configuration identity, and command.
- Sensitive or third-party material is handled according to repository policy.

## Trigger

The evaluator runs claim verification, a benchmark verifier, or the corresponding CI gate.

## Main Success Scenario

1. The harness loads the versioned corpus/manifest and ground truth.
2. It validates required receipt identity and provenance fields.
3. The grader recomputes per-case outcomes from retained evidence.
4. Aggregate scores and headline verdicts are recomputed using the declared metric.
5. Anti-fitting checks verify separation and policy invariants.
6. The verifier compares derived values with public claims.
7. It emits a pass/fail result and enough context to reproduce a mismatch.
8. CI blocks integration when a required claim, provenance, or anti-fitting invariant fails.

## Alternate Flows

- A local evaluator runs only deterministic verification against committed artifacts.
- A live benchmark run produces new receipts, followed by deterministic grading in a separate step.
- Raw sensitive transcripts may be withheld while a declared, re-derivable graded receipt remains available; the documentation must state the reproduction boundary.

## Exception Flows

- Missing or stale evidence: fail the affected claim rather than substituting an estimate.
- Model/provider outage during a live run: classify as infrastructure/run failure, not model success or failure.
- Ground-truth or manifest mismatch: stop grading and identify the conflicting artifact.
- Suspected self-fitting or corpus leakage: fail the anti-fitting gate pending review.

## Postconditions

- Every passing headline claim maps to the evaluated corpus, metric, and retained evidence.
- Failed claims identify the discrepant artifact or value.
- Benchmark scope is not generalized to unsupported product maturity.

## Acceptance Criteria

- [ ] `npm run verify-claims` derives required public claims from repository artifacts.
- [ ] Seeded claim mismatches, missing provenance, or fitting violations cause a non-zero gate result.
- [ ] Receipts identify the relevant corpus/model/configuration and evaluation outcome.
- [ ] Live-run infrastructure failures remain distinct from evaluated model verdicts.
- [ ] Maturity documentation limits conclusions to the evaluated scope.

## Design and Test Traceability

| Concern | Implementation | Verification |
| --- | --- | --- |
| Claim derivation | `scripts/verify-claims.mjs`, benchmark manifests/receipts | Claim verification in CI |
| Finding grading | Verification/refutation and grading scripts | Script self-tests and benchmark tests |
| Anti-fitting | No-fitting and no-self-fitting checks | Seeded anti-fitting tests |
| Gate enforcement | `.github/workflows/ci.yml` | CI job status and local command parity |

## References

- @.aiwg/architecture/adr-004.md — Evidence-derived public-claims decision.
- @.aiwg/architecture/adr-005.md — Current-state versus research-vision decision.
- @.aiwg/architecture/vision-alignment.md — Maturity interpretation rules.
- @scripts/verify-claims.mjs — Deterministic public-claim verifier.
- @scripts/verify-finding.mjs — Finding verification.
- @scripts/refute-finding.mjs — Candidate-refutation path.
- @scripts/test-no-fitting.mjs — Anti-fitting evidence.
- @scripts/test-no-self-fitting.mjs — Self-fitting prevention evidence.
- @.github/workflows/ci.yml — Required automated gates.