chore(aiwg): adopt shared BT6 maintainer workflow (#148)

* docs(policy): allow low-impact contributor courtesy cleanup

* chore(aiwg): migrate to shared BT6 maintainer plugin

* chore: reconcile delivery validation
This commit is contained in:
Joseph Magly authored and GitHub committed 2026-08-02 17:17:28 -04:00
1 parent 8f658891f2
commit afc9dad1b2
89 files changed
+4683 -1512

No files matched your search

+76 -8
View File
@@ -1,13 +1,81 @@
# UC-005 — Reproduce Claims and Benchmark Results
**Primary actor:** Maintainer, contributor, or evaluator
**Goal:** Re-derive public capability claims from committed data and detect fitting or provenance regressions.
## Metadata
## Main Flow
- **ID:** UC-005
- **Status:** Baselined (brownfield)
- **Level:** User goal
- **Scope:** Benchmark corpora, manifests, ground truth, receipts, graders, claim verification, and CI
- **Primary actor:** Maintainer, contributor, or evaluator
- **Goal:** Re-derive public capability claims from versioned evidence and detect grading, provenance, or fitting regressions.
- **Related stories:** US-08, US-09, US-10
- **Quality drivers:** NFR-03, NFR-08, NFR-09
1. Evaluator runs claim or benchmark verification.
2. Harness reads committed corpus, receipts, and ground truth.
3. Scores and verdicts are recomputed.
4. CI fails if claims, provenance, or anti-fitting invariants do not hold.
## Preconditions
**Implementation coverage:** `bench/`, `scripts/verify-claims.mjs`, model/benchmark scripts, anti-fitting tests, `.github/workflows/ci.yml`.
- The evaluated corpus, manifest, ground truth, and required receipts are available under their documented retention policy.
- The evaluator uses the declared harness, metric, model/configuration identity, and command.
- Sensitive or third-party material is handled according to repository policy.
## Trigger
The evaluator runs claim verification, a benchmark verifier, or the corresponding CI gate.
## Main Success Scenario
1. The harness loads the versioned corpus/manifest and ground truth.
2. It validates required receipt identity and provenance fields.
3. The grader recomputes per-case outcomes from retained evidence.
4. Aggregate scores and headline verdicts are recomputed using the declared metric.
5. Anti-fitting checks verify separation and policy invariants.
6. The verifier compares derived values with public claims.
7. It emits a pass/fail result and enough context to reproduce a mismatch.
8. CI blocks integration when a required claim, provenance, or anti-fitting invariant fails.
## Alternate Flows
- A local evaluator runs only deterministic verification against committed artifacts.
- A live benchmark run produces new receipts, followed by deterministic grading in a separate step.
- Raw sensitive transcripts may be withheld while a declared, re-derivable graded receipt remains available; the documentation must state the reproduction boundary.
## Exception Flows
- Missing or stale evidence: fail the affected claim rather than substituting an estimate.
- Model/provider outage during a live run: classify as infrastructure/run failure, not model success or failure.
- Ground-truth or manifest mismatch: stop grading and identify the conflicting artifact.
- Suspected self-fitting or corpus leakage: fail the anti-fitting gate pending review.
## Postconditions
- Every passing headline claim maps to the evaluated corpus, metric, and retained evidence.
- Failed claims identify the discrepant artifact or value.
- Benchmark scope is not generalized to unsupported product maturity.
## Acceptance Criteria
- [ ] `npm run verify-claims` derives required public claims from repository artifacts.
- [ ] Seeded claim mismatches, missing provenance, or fitting violations cause a non-zero gate result.
- [ ] Receipts identify the relevant corpus/model/configuration and evaluation outcome.
- [ ] Live-run infrastructure failures remain distinct from evaluated model verdicts.
- [ ] Maturity documentation limits conclusions to the evaluated scope.
## Design and Test Traceability
| Concern | Implementation | Verification |
| --- | --- | --- |
| Claim derivation | `scripts/verify-claims.mjs`, benchmark manifests/receipts | Claim verification in CI |
| Finding grading | Verification/refutation and grading scripts | Script self-tests and benchmark tests |
| Anti-fitting | No-fitting and no-self-fitting checks | Seeded anti-fitting tests |
| Gate enforcement | `.github/workflows/ci.yml` | CI job status and local command parity |
## References
- @.aiwg/architecture/adr-004.md — Evidence-derived public-claims decision.
- @.aiwg/architecture/adr-005.md — Current-state versus research-vision decision.
- @.aiwg/architecture/vision-alignment.md — Maturity interpretation rules.
- @scripts/verify-claims.mjs — Deterministic public-claim verifier.
- @scripts/verify-finding.mjs — Finding verification.
- @scripts/refute-finding.mjs — Candidate-refutation path.
- @scripts/test-no-fitting.mjs — Anti-fitting evidence.
- @scripts/test-no-self-fitting.mjs — Self-fitting prevention evidence.
- @.github/workflows/ci.yml — Required automated gates.