chore(aiwg): adopt shared BT6 maintainer workflow (#148)
* docs(policy): allow low-impact contributor courtesy cleanup * chore(aiwg): migrate to shared BT6 maintainer plugin * chore: reconcile delivery validation
This commit is contained in:
1 parent
8f658891f2
commit
afc9dad1b2
89 files changed
+4683
-1512
No files matched your search
@@ -1,16 +1,123 @@
|
||||
# ADR-004 — Derive Public Claims from Versioned Evidence
|
||||
|
||||
**Status:** Accepted (retrospective)
|
||||
**Date:** 2026-07-20
|
||||
## Metadata
|
||||
|
||||
- **ADR ID:** ADR-004
|
||||
- **Status:** Accepted (retrospective baseline)
|
||||
- **Decision date:** 2026-07-20
|
||||
- **Authors:** Architecture baseline process
|
||||
- **Reviewers:** Maintainer and evaluation-review roles; named human approval is not recorded
|
||||
- **Supersedes:** None
|
||||
- **Current level:** L4 — Implemented in repository verification and CI
|
||||
|
||||
## Context
|
||||
|
||||
Offensive-AI performance claims are easy to overfit or report without reproducible evidence.
|
||||
### Problem
|
||||
|
||||
Offensive-AI findings and benchmark scores are easy to overfit, misgrade, or report without enough context to reproduce them. Model/provider outages can be confused with model failures, and successful corpus-specific results can be overstated as general product capability.
|
||||
|
||||
### Constraints
|
||||
|
||||
- Some third-party corpora or raw sensitive transcripts cannot be redistributed.
|
||||
- Model/provider behavior and live infrastructure are nondeterministic.
|
||||
- Public claims must remain auditable after documentation or implementation changes.
|
||||
- Grading, provenance, and anti-fitting must be enforceable in deterministic CI where possible.
|
||||
- Evidence retention must respect coordinated-disclosure and operator-controlled sensitive paths.
|
||||
|
||||
### Stakeholders
|
||||
|
||||
- Users and researchers need credible, scoped performance claims.
|
||||
- Maintainers need release gates that detect stale documentation.
|
||||
- Vulnerability recipients need evidence-backed findings.
|
||||
- Corpus owners and target operators need licensing/privacy boundaries respected.
|
||||
|
||||
## Source Verification and Claim Tracking
|
||||
|
||||
| Claim | Source | Verification |
|
||||
| --- | --- | --- |
|
||||
| Public claims are recomputed by repository tooling | Claim verifier and package scripts | Execute deterministic claim gate |
|
||||
| Fitting controls are tested | Anti-fitting test scripts | Seeded negative tests |
|
||||
| CI includes claim/provenance gates | CI workflow | Workflow inspection and local command parity |
|
||||
| Some raw material may be held locally | Ignore/disclosure policy and benchmark staging rules | Repository policy inspection |
|
||||
|
||||
## Decision
|
||||
|
||||
Headline claims must be recomputed from committed corpus, ground truth, and receipts. Anti-fitting, provenance, and claim-verification gates are architectural product controls.
|
||||
Headline claims and verified findings must derive from versioned corpus/manifest, ground truth, and retained receipts or evidence. Anti-fitting, provenance, finding verification, and claim verification are product architecture controls and release gates, not optional documentation tasks.
|
||||
|
||||
Every published result must state the evaluated scope: corpus, harness/metric, model/provider identity when relevant, and reproduction boundary. If raw inputs cannot be published, documentation must distinguish a re-derivable graded receipt from full raw reproduction. Missing evidence fails the claim; it is not replaced with an estimate.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
### Publish manually curated benchmark tables
|
||||
|
||||
- **Pros:** Simple presentation and low runtime cost.
|
||||
- **Cons:** Stale values, transcription errors, and no automated derivation chain.
|
||||
- **Why not chosen:** Cannot enforce public-claim integrity across releases.
|
||||
|
||||
### Retain only aggregate scores
|
||||
|
||||
- **Pros:** Small repository and simpler privacy posture.
|
||||
- **Cons:** Prevents per-case audit, regrading, and error classification.
|
||||
- **Why not chosen:** Aggregate-only evidence is insufficient for disputed findings and fitting analysis.
|
||||
|
||||
### Publish every raw transcript and corpus
|
||||
|
||||
- **Pros:** Maximum reproduction detail.
|
||||
- **Cons:** May violate licensing, privacy, security, or coordinated-disclosure constraints.
|
||||
- **Why not chosen:** Evidence detail must be maximized within lawful/safe retention boundaries, with limitations disclosed.
|
||||
|
||||
## Consequences
|
||||
|
||||
Repository size and CI cost increase, but claims are auditable. Raw sensitive transcripts may be withheld, so documentation must distinguish re-derivable graded verdicts from full raw reproduction.
|
||||
### Positive
|
||||
|
||||
- Claim drift and seeded provenance/fitting defects can block release.
|
||||
- Findings and scores are reviewable at the case level where artifacts permit.
|
||||
- Corpus-scoped language reduces misleading generalization.
|
||||
|
||||
### Negative
|
||||
|
||||
- Repository/storage and CI cost increase.
|
||||
- Receipt schemas and graders become compatibility surfaces.
|
||||
- Full independent reproduction may remain impossible for legally or sensitively withheld raw inputs.
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
| Risk | Likelihood / impact | Mitigation |
|
||||
| --- | --- | --- |
|
||||
| Receipt exists but does not support claim | Medium / High | Deterministic recomputation, verifier tests, exact claim mapping |
|
||||
| Corpus leakage/self-fitting | Medium / Critical | Split policy and no-fitting/no-self-fitting gates |
|
||||
| Infrastructure failure misclassified | Medium / High | Separate run/infrastructure outcome from model verdict |
|
||||
| Sensitive evidence published | Low / Critical | Hold policies, redaction, coordinated-disclosure paths |
|
||||
|
||||
## Implementation and Verification
|
||||
|
||||
- **Claim derivation:** `scripts/verify-claims.mjs` and package scripts.
|
||||
- **Finding lifecycle:** verification/refutation scripts and evidence modules.
|
||||
- **Anti-fitting:** dedicated no-fitting and no-self-fitting tests.
|
||||
- **Gate enforcement:** required CI jobs for claims, provenance, prompt integrity, tests, and smoke.
|
||||
- **State model:** Versioned manifests/ground truth/receipts; raw sensitive inputs may stay in explicitly local ignored staging.
|
||||
- **Regression guard:** Any change to a claim, receipt, grader, ground truth, or benchmark harness must make derivation pass from the committed state.
|
||||
|
||||
## Definition of Done
|
||||
|
||||
- [x] Evidence-derived claim policy documented.
|
||||
- [x] Deterministic claim and anti-fitting gates identified.
|
||||
- [x] Withheld-raw-input limitation made explicit.
|
||||
- [ ] Every headline claim has a machine-readable mapping to exact supporting receipts and corpus version.
|
||||
- [ ] Named evaluation-review approval is recorded.
|
||||
|
||||
## References
|
||||
|
||||
- @.aiwg/requirements/UC-005.md — Reproduction and failure flows.
|
||||
- @.aiwg/requirements/nfr-register.md — Provenance, testability, and honesty requirements.
|
||||
- @.aiwg/architecture/vision-alignment.md — Corpus-scoped maturity interpretation.
|
||||
- @scripts/verify-claims.mjs — Claim derivation implementation.
|
||||
- @scripts/verify-finding.mjs — Finding verification.
|
||||
- @scripts/refute-finding.mjs — Candidate refutation.
|
||||
- @scripts/test-no-fitting.mjs — Fitting-prevention evidence.
|
||||
- @scripts/test-no-self-fitting.mjs — Self-fitting-prevention evidence.
|
||||
- @.github/workflows/ci.yml — Automated gate orchestration.
|
||||
|
||||
## Review and Approval
|
||||
|
||||
This ADR records an implemented evaluation/release policy. Its accepted status does not imply that withheld source material is fully independently reproducible. Amendments require maintainer and evaluation review, a stated evidence-retention impact, and passing seeded negative tests.
|
||||
Reference in new issue
Block a user