t3mp3st/.aiwg/architecture/adr-004.md

# ADR-004 — Derive Public Claims from Versioned Evidence

## Metadata

- **ADR ID:** ADR-004
- **Status:** Accepted (retrospective baseline)
- **Decision date:** 2026-07-20
- **Authors:** Architecture baseline process
- **Reviewers:** Maintainer and evaluation-review roles; named human approval is not recorded
- **Supersedes:** None
- **Current level:** L4 — Implemented in repository verification and CI

## Context

### Problem

Offensive-AI findings and benchmark scores are easy to overfit, misgrade, or report without enough context to reproduce them. Model/provider outages can be confused with model failures, and successful corpus-specific results can be overstated as general product capability.

### Constraints

- Some third-party corpora or raw sensitive transcripts cannot be redistributed.
- Model/provider behavior and live infrastructure are nondeterministic.
- Public claims must remain auditable after documentation or implementation changes.
- Grading, provenance, and anti-fitting must be enforceable in deterministic CI where possible.
- Evidence retention must respect coordinated-disclosure and operator-controlled sensitive paths.

### Stakeholders

- Users and researchers need credible, scoped performance claims.
- Maintainers need release gates that detect stale documentation.
- Vulnerability recipients need evidence-backed findings.
- Corpus owners and target operators need licensing/privacy boundaries respected.

## Source Verification and Claim Tracking

| Claim | Source | Verification |
| --- | --- | --- |
| Public claims are recomputed by repository tooling | Claim verifier and package scripts | Execute deterministic claim gate |
| Fitting controls are tested | Anti-fitting test scripts | Seeded negative tests |
| CI includes claim/provenance gates | CI workflow | Workflow inspection and local command parity |
| Some raw material may be held locally | Ignore/disclosure policy and benchmark staging rules | Repository policy inspection |

## Decision

Headline claims and verified findings must derive from versioned corpus/manifest, ground truth, and retained receipts or evidence. Anti-fitting, provenance, finding verification, and claim verification are product architecture controls and release gates, not optional documentation tasks.

Every published result must state the evaluated scope: corpus, harness/metric, model/provider identity when relevant, and reproduction boundary. If raw inputs cannot be published, documentation must distinguish a re-derivable graded receipt from full raw reproduction. Missing evidence fails the claim; it is not replaced with an estimate.

## Alternatives Considered

### Publish manually curated benchmark tables

- **Pros:** Simple presentation and low runtime cost.
- **Cons:** Stale values, transcription errors, and no automated derivation chain.
- **Why not chosen:** Cannot enforce public-claim integrity across releases.

### Retain only aggregate scores

- **Pros:** Small repository and simpler privacy posture.
- **Cons:** Prevents per-case audit, regrading, and error classification.
- **Why not chosen:** Aggregate-only evidence is insufficient for disputed findings and fitting analysis.

### Publish every raw transcript and corpus

- **Pros:** Maximum reproduction detail.
- **Cons:** May violate licensing, privacy, security, or coordinated-disclosure constraints.
- **Why not chosen:** Evidence detail must be maximized within lawful/safe retention boundaries, with limitations disclosed.

## Consequences

### Positive

- Claim drift and seeded provenance/fitting defects can block release.
- Findings and scores are reviewable at the case level where artifacts permit.
- Corpus-scoped language reduces misleading generalization.

### Negative

- Repository/storage and CI cost increase.
- Receipt schemas and graders become compatibility surfaces.
- Full independent reproduction may remain impossible for legally or sensitively withheld raw inputs.

## Risks and Mitigations

| Risk | Likelihood / impact | Mitigation |
| --- | --- | --- |
| Receipt exists but does not support claim | Medium / High | Deterministic recomputation, verifier tests, exact claim mapping |
| Corpus leakage/self-fitting | Medium / Critical | Split policy and no-fitting/no-self-fitting gates |
| Infrastructure failure misclassified | Medium / High | Separate run/infrastructure outcome from model verdict |
| Sensitive evidence published | Low / Critical | Hold policies, redaction, coordinated-disclosure paths |

## Implementation and Verification

- **Claim derivation:** `scripts/verify-claims.mjs` and package scripts.
- **Finding lifecycle:** verification/refutation scripts and evidence modules.
- **Anti-fitting:** dedicated no-fitting and no-self-fitting tests.
- **Gate enforcement:** required CI jobs for claims, provenance, prompt integrity, tests, and smoke.
- **State model:** Versioned manifests/ground truth/receipts; raw sensitive inputs may stay in explicitly local ignored staging.
- **Regression guard:** Any change to a claim, receipt, grader, ground truth, or benchmark harness must make derivation pass from the committed state.

## Definition of Done

- [x] Evidence-derived claim policy documented.
- [x] Deterministic claim and anti-fitting gates identified.
- [x] Withheld-raw-input limitation made explicit.
- [ ] Every headline claim has a machine-readable mapping to exact supporting receipts and corpus version.
- [ ] Named evaluation-review approval is recorded.

## References

- @.aiwg/requirements/UC-005.md — Reproduction and failure flows.
- @.aiwg/requirements/nfr-register.md — Provenance, testability, and honesty requirements.
- @.aiwg/architecture/vision-alignment.md — Corpus-scoped maturity interpretation.
- @scripts/verify-claims.mjs — Claim derivation implementation.
- @scripts/verify-finding.mjs — Finding verification.
- @scripts/refute-finding.mjs — Candidate refutation.
- @scripts/test-no-fitting.mjs — Fitting-prevention evidence.
- @scripts/test-no-self-fitting.mjs — Self-fitting-prevention evidence.
- @.github/workflows/ci.yml — Automated gate orchestration.

## Review and Approval

This ADR records an implemented evaluation/release policy. Its accepted status does not imply that withheld source material is fully independently reproducible. Amendments require maintainer and evaluation review, a stated evidence-retention impact, and passing seeded negative tests.