chore(aiwg): adopt shared BT6 maintainer workflow (#148)
* docs(policy): allow low-impact contributor courtesy cleanup * chore(aiwg): migrate to shared BT6 maintainer plugin * chore: reconcile delivery validation
This commit is contained in:
1 parent
8f658891f2
commit
afc9dad1b2
89 files changed
+4683
-1512
No files matched your search
@@ -1,13 +1,81 @@
|
||||
# UC-005 — Reproduce Claims and Benchmark Results
|
||||
|
||||
**Primary actor:** Maintainer, contributor, or evaluator
|
||||
**Goal:** Re-derive public capability claims from committed data and detect fitting or provenance regressions.
|
||||
## Metadata
|
||||
|
||||
## Main Flow
|
||||
- **ID:** UC-005
|
||||
- **Status:** Baselined (brownfield)
|
||||
- **Level:** User goal
|
||||
- **Scope:** Benchmark corpora, manifests, ground truth, receipts, graders, claim verification, and CI
|
||||
- **Primary actor:** Maintainer, contributor, or evaluator
|
||||
- **Goal:** Re-derive public capability claims from versioned evidence and detect grading, provenance, or fitting regressions.
|
||||
- **Related stories:** US-08, US-09, US-10
|
||||
- **Quality drivers:** NFR-03, NFR-08, NFR-09
|
||||
|
||||
1. Evaluator runs claim or benchmark verification.
|
||||
2. Harness reads committed corpus, receipts, and ground truth.
|
||||
3. Scores and verdicts are recomputed.
|
||||
4. CI fails if claims, provenance, or anti-fitting invariants do not hold.
|
||||
## Preconditions
|
||||
|
||||
**Implementation coverage:** `bench/`, `scripts/verify-claims.mjs`, model/benchmark scripts, anti-fitting tests, `.github/workflows/ci.yml`.
|
||||
- The evaluated corpus, manifest, ground truth, and required receipts are available under their documented retention policy.
|
||||
- The evaluator uses the declared harness, metric, model/configuration identity, and command.
|
||||
- Sensitive or third-party material is handled according to repository policy.
|
||||
|
||||
## Trigger
|
||||
|
||||
The evaluator runs claim verification, a benchmark verifier, or the corresponding CI gate.
|
||||
|
||||
## Main Success Scenario
|
||||
|
||||
1. The harness loads the versioned corpus/manifest and ground truth.
|
||||
2. It validates required receipt identity and provenance fields.
|
||||
3. The grader recomputes per-case outcomes from retained evidence.
|
||||
4. Aggregate scores and headline verdicts are recomputed using the declared metric.
|
||||
5. Anti-fitting checks verify separation and policy invariants.
|
||||
6. The verifier compares derived values with public claims.
|
||||
7. It emits a pass/fail result and enough context to reproduce a mismatch.
|
||||
8. CI blocks integration when a required claim, provenance, or anti-fitting invariant fails.
|
||||
|
||||
## Alternate Flows
|
||||
|
||||
- A local evaluator runs only deterministic verification against committed artifacts.
|
||||
- A live benchmark run produces new receipts, followed by deterministic grading in a separate step.
|
||||
- Raw sensitive transcripts may be withheld while a declared, re-derivable graded receipt remains available; the documentation must state the reproduction boundary.
|
||||
|
||||
## Exception Flows
|
||||
|
||||
- Missing or stale evidence: fail the affected claim rather than substituting an estimate.
|
||||
- Model/provider outage during a live run: classify as infrastructure/run failure, not model success or failure.
|
||||
- Ground-truth or manifest mismatch: stop grading and identify the conflicting artifact.
|
||||
- Suspected self-fitting or corpus leakage: fail the anti-fitting gate pending review.
|
||||
|
||||
## Postconditions
|
||||
|
||||
- Every passing headline claim maps to the evaluated corpus, metric, and retained evidence.
|
||||
- Failed claims identify the discrepant artifact or value.
|
||||
- Benchmark scope is not generalized to unsupported product maturity.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `npm run verify-claims` derives required public claims from repository artifacts.
|
||||
- [ ] Seeded claim mismatches, missing provenance, or fitting violations cause a non-zero gate result.
|
||||
- [ ] Receipts identify the relevant corpus/model/configuration and evaluation outcome.
|
||||
- [ ] Live-run infrastructure failures remain distinct from evaluated model verdicts.
|
||||
- [ ] Maturity documentation limits conclusions to the evaluated scope.
|
||||
|
||||
## Design and Test Traceability
|
||||
|
||||
| Concern | Implementation | Verification |
|
||||
| --- | --- | --- |
|
||||
| Claim derivation | `scripts/verify-claims.mjs`, benchmark manifests/receipts | Claim verification in CI |
|
||||
| Finding grading | Verification/refutation and grading scripts | Script self-tests and benchmark tests |
|
||||
| Anti-fitting | No-fitting and no-self-fitting checks | Seeded anti-fitting tests |
|
||||
| Gate enforcement | `.github/workflows/ci.yml` | CI job status and local command parity |
|
||||
|
||||
## References
|
||||
|
||||
- @.aiwg/architecture/adr-004.md — Evidence-derived public-claims decision.
|
||||
- @.aiwg/architecture/adr-005.md — Current-state versus research-vision decision.
|
||||
- @.aiwg/architecture/vision-alignment.md — Maturity interpretation rules.
|
||||
- @scripts/verify-claims.mjs — Deterministic public-claim verifier.
|
||||
- @scripts/verify-finding.mjs — Finding verification.
|
||||
- @scripts/refute-finding.mjs — Candidate-refutation path.
|
||||
- @scripts/test-no-fitting.mjs — Anti-fitting evidence.
|
||||
- @scripts/test-no-self-fitting.mjs — Self-fitting prevention evidence.
|
||||
- @.github/workflows/ci.yml — Required automated gates.
|
||||
Reference in new issue
Block a user