diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 422bb07..8018034 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -49,6 +49,10 @@ npm test npm run doctor ``` +For changes that affect claims, run modes, agent/tool execution, target scope, +egress, redaction, reports, or benchmark artifacts, include a contribution +receipt using [`docs/CONTRIBUTION_RECEIPTS.md`](docs/CONTRIBUTION_RECEIPTS.md). + If the API is running: ```bash diff --git a/README.md b/README.md index ff06f0b..c8a1009 100755 --- a/README.md +++ b/README.md @@ -170,6 +170,7 @@ Deeper reading: [WALL_FORENSICS](docs/WALL_FORENSICS.md) (per-challenge misses), | [FEATURES.md](FEATURES.md) | feature-by-feature status (`[x]` shipped / `[~]` partial / `[ ]` planned) | | [SCOPE_AND_AUTHORIZATION](docs/SCOPE_AND_AUTHORIZATION.md) | authority model, scope receipts, evidence and retest rules | | [VERIFIED_PROVENANCE](docs/VERIFIED_PROVENANCE.md) | how findings become tool-proven instead of model-asserted | +| [CONTRIBUTION_RECEIPTS](docs/CONTRIBUTION_RECEIPTS.md) | PR receipt template for scope, run mode, model/harness labels, redaction, and verification | | [TEAM_PREVIEW](docs/TEAM_PREVIEW.md) | first-run path and review script | | [INSTALL_MATRIX](docs/INSTALL_MATRIX.md) | macOS / Linux readiness table | | [ARSENAL_ACTIVATION_PLAN](docs/ARSENAL_ACTIVATION_PLAN.md) | optional external-tool setup | diff --git a/docs/CONTRIBUTION_RECEIPTS.md b/docs/CONTRIBUTION_RECEIPTS.md new file mode 100644 index 0000000..625ce88 --- /dev/null +++ b/docs/CONTRIBUTION_RECEIPTS.md @@ -0,0 +1,99 @@ +# Contribution Receipts + +T3MP3ST already treats benchmark claims as something reviewers should be able to +re-derive. Contribution receipts apply the same discipline to pull requests: +make scope, run mode, evidence, redaction, and model/harness labels explicit +before a change is merged. + +A receipt can live in the PR description, an issue comment, or a committed +artifact when the change itself introduces a durable benchmark or claim. + +## When A Receipt Is Needed + +Include a receipt for changes that affect: + +- benchmark numbers, benchmark artifacts, scoring, or claim verification; +- local-agent, API-backed, MCP, or Arsenal execution behavior; +- target scope, egress containment, approval gates, or evidence gates; +- UI or docs labels for preview, wired, installed, gated, synthetic, live, + planning-only, tool-backed, or experimental behavior; +- redaction, source ingest, evidence storage, reports, or exported artifacts. + +A receipt is optional for narrow typo fixes, formatting-only docs edits, and +mechanical dependency-free cleanup that does not alter behavior or claims. + +## Required Fields + +| Field | What To Record | +| --- | --- | +| Change | Short summary of what changed and why. | +| Scope class | `docs_only`, `static_fixture`, `local_lab`, `ctf_range`, or `authorized_live`. | +| Target authority | Who owns or authorized the target. Use `not_applicable` for docs-only/static work. | +| Network use | `none`, `loopback`, `private_lab`, or `authorized_external`. | +| Run mode labels | For example `planning_only`, `static_test`, `mocked`, `local_agent`, `api_backed`, `tool_backed`, `approval_gated`, `swarm`, `live_authorized`. | +| Model/harness labels | Model, provider, agent runtime, harness, tool access, target class, and attempt policy. | +| Commands run | Exact commands and pass/fail result. Include skipped commands and why. | +| Artifacts | Files, logs, screenshots, reports, or benchmark outputs that support the claim. | +| Redaction | What was removed or masked, especially secrets, tokens, private keys, credentials, flags, and private target data. | +| Claims changed | State `none` when no README/docs/headline claim changed. Otherwise name the claim and how it is re-derived. | +| Abstentions/refusals | Count separately from failed attempts when model or agent behavior is measured. | +| Residual risk | Known limits, unverified paths, optional missing tools, or reviewer follow-up. | + +## Model And Harness Matrix + +Benchmark and agent-run claims should not blend model capability, harness +capability, tool availability, target choice, and run mode. Use a small matrix +when a change introduces or updates measured behavior. + +| Field | Example | +| --- | --- | +| model | `gpt-5.5`, `local-llama`, `not_applicable` | +| provider | `OpenRouter`, `Venice`, `Ollama`, `not_applicable` | +| model_version_or_date | Provider version, model date, or retrieval date if known. | +| harness | `verify-claims`, `cybench-bench`, `xbow`, `local-agent`, `manual-review`. | +| agent_runtime | `Codex`, `Claude Code`, `Hermes`, `none`, or other runtime. | +| tool_access | `none`, `read_only`, `mocked`, `local_only`, `approval_gated`, `live_authorized`. | +| target_class | `static_fixture`, `local_lab`, `ctf_range`, `authorized_external`. | +| run_mode | `single_agent`, `local_agent`, `api_backed`, `swarm`, `planning_only`. | +| attempts | Number of runs or tasks attempted. | +| successes | Number of evidence-backed successes. | +| failures | Number of attempts with evidence-backed failure. | +| abstentions | Refusals, skipped tasks, or no-action outcomes. | +| artifacts | Paths to committed artifacts or private reviewer-only receipts. | + +## PR Receipt Template + +```markdown +## Contribution Receipt + +- Change: +- Scope class: +- Target authority: +- Network use: +- Run mode labels: +- Model/harness labels: +- Commands run: + - `npm run typecheck` -> pass/fail/skipped + - `npm test` -> pass/fail/skipped + - `npm run doctor` -> pass/fail/skipped + - `npm run verify-claims` -> pass/fail/skipped +- Artifacts: +- Redaction: +- Claims changed: +- Abstentions/refusals: +- Residual risk: +``` + +## Review Rules + +- Do not accept a live or external-target result unless the receipt names the + authorization path and target scope. +- Do not accept raw secrets, tokens, credentials, private keys, recovered + passwords, or private target data in a receipt or artifact. +- Do not promote a UI/docs label from preview, planned, synthetic, or + planning-only to live/tool-backed unless a reviewer can trace the supporting + command or artifact. +- Do not blend refusals, abstentions, skipped runs, infrastructure failures, and + failed attempts into a single failure bucket when reporting model behavior. +- Do not update headline claims unless `npm run verify-claims` or the relevant + claim-specific verifier re-derives the number from committed artifacts.