Merge remote-tracking branch 'upstream/main' into pr-47
# Conflicts: # README.md
This commit is contained in:
75 files changed
+7949
-648
No files matched your search
@@ -0,0 +1,99 @@
|
||||
# Contribution Receipts
|
||||
|
||||
T3MP3ST already treats benchmark claims as something reviewers should be able to
|
||||
re-derive. Contribution receipts apply the same discipline to pull requests:
|
||||
make scope, run mode, evidence, redaction, and model/harness labels explicit
|
||||
before a change is merged.
|
||||
|
||||
A receipt can live in the PR description, an issue comment, or a committed
|
||||
artifact when the change itself introduces a durable benchmark or claim.
|
||||
|
||||
## When A Receipt Is Needed
|
||||
|
||||
Include a receipt for changes that affect:
|
||||
|
||||
- benchmark numbers, benchmark artifacts, scoring, or claim verification;
|
||||
- local-agent, API-backed, MCP, or Arsenal execution behavior;
|
||||
- target scope, egress containment, approval gates, or evidence gates;
|
||||
- UI or docs labels for preview, wired, installed, gated, synthetic, live,
|
||||
planning-only, tool-backed, or experimental behavior;
|
||||
- redaction, source ingest, evidence storage, reports, or exported artifacts.
|
||||
|
||||
A receipt is optional for narrow typo fixes, formatting-only docs edits, and
|
||||
mechanical dependency-free cleanup that does not alter behavior or claims.
|
||||
|
||||
## Required Fields
|
||||
|
||||
| Field | What To Record |
|
||||
| --- | --- |
|
||||
| Change | Short summary of what changed and why. |
|
||||
| Scope class | `docs_only`, `static_fixture`, `local_lab`, `ctf_range`, or `authorized_live`. |
|
||||
| Target authority | Who owns or authorized the target. Use `not_applicable` for docs-only/static work. |
|
||||
| Network use | `none`, `loopback`, `private_lab`, or `authorized_external`. |
|
||||
| Run mode labels | For example `planning_only`, `static_test`, `mocked`, `local_agent`, `api_backed`, `tool_backed`, `approval_gated`, `swarm`, `live_authorized`. |
|
||||
| Model/harness labels | Model, provider, agent runtime, harness, tool access, target class, and attempt policy. |
|
||||
| Commands run | Exact commands and pass/fail result. Include skipped commands and why. |
|
||||
| Artifacts | Files, logs, screenshots, reports, or benchmark outputs that support the claim. |
|
||||
| Redaction | What was removed or masked, especially secrets, tokens, private keys, credentials, flags, and private target data. |
|
||||
| Claims changed | State `none` when no README/docs/headline claim changed. Otherwise name the claim and how it is re-derived. |
|
||||
| Abstentions/refusals | Count separately from failed attempts when model or agent behavior is measured. |
|
||||
| Residual risk | Known limits, unverified paths, optional missing tools, or reviewer follow-up. |
|
||||
|
||||
## Model And Harness Matrix
|
||||
|
||||
Benchmark and agent-run claims should not blend model capability, harness
|
||||
capability, tool availability, target choice, and run mode. Use a small matrix
|
||||
when a change introduces or updates measured behavior.
|
||||
|
||||
| Field | Example |
|
||||
| --- | --- |
|
||||
| model | `gpt-5.5`, `local-llama`, `not_applicable` |
|
||||
| provider | `OpenRouter`, `Venice`, `Ollama`, `not_applicable` |
|
||||
| model_version_or_date | Provider version, model date, or retrieval date if known. |
|
||||
| harness | `verify-claims`, `cybench-bench`, `xbow`, `local-agent`, `manual-review`. |
|
||||
| agent_runtime | `Codex`, `Claude Code`, `Hermes`, `none`, or other runtime. |
|
||||
| tool_access | `none`, `read_only`, `mocked`, `local_only`, `approval_gated`, `live_authorized`. |
|
||||
| target_class | `static_fixture`, `local_lab`, `ctf_range`, `authorized_external`. |
|
||||
| run_mode | `single_agent`, `local_agent`, `api_backed`, `swarm`, `planning_only`. |
|
||||
| attempts | Number of runs or tasks attempted. |
|
||||
| successes | Number of evidence-backed successes. |
|
||||
| failures | Number of attempts with evidence-backed failure. |
|
||||
| abstentions | Refusals, skipped tasks, or no-action outcomes. |
|
||||
| artifacts | Paths to committed artifacts or private reviewer-only receipts. |
|
||||
|
||||
## PR Receipt Template
|
||||
|
||||
```markdown
|
||||
## Contribution Receipt
|
||||
|
||||
- Change:
|
||||
- Scope class:
|
||||
- Target authority:
|
||||
- Network use:
|
||||
- Run mode labels:
|
||||
- Model/harness labels:
|
||||
- Commands run:
|
||||
- `npm run typecheck` -> pass/fail/skipped
|
||||
- `npm test` -> pass/fail/skipped
|
||||
- `npm run doctor` -> pass/fail/skipped
|
||||
- `npm run verify-claims` -> pass/fail/skipped
|
||||
- Artifacts:
|
||||
- Redaction:
|
||||
- Claims changed:
|
||||
- Abstentions/refusals:
|
||||
- Residual risk:
|
||||
```
|
||||
|
||||
## Review Rules
|
||||
|
||||
- Do not accept a live or external-target result unless the receipt names the
|
||||
authorization path and target scope.
|
||||
- Do not accept raw secrets, tokens, credentials, private keys, recovered
|
||||
passwords, or private target data in a receipt or artifact.
|
||||
- Do not promote a UI/docs label from preview, planned, synthetic, or
|
||||
planning-only to live/tool-backed unless a reviewer can trace the supporting
|
||||
command or artifact.
|
||||
- Do not blend refusals, abstentions, skipped runs, infrastructure failures, and
|
||||
failed attempts into a single failure bucket when reporting model behavior.
|
||||
- Do not update headline claims unless `npm run verify-claims` or the relevant
|
||||
claim-specific verifier re-derives the number from committed artifacts.
|
||||
@@ -0,0 +1,141 @@
|
||||
# Pull Request Delivery Guide
|
||||
|
||||
Use this checklist before opening or updating a pull request. It is written for
|
||||
both human contributors and coding agents.
|
||||
|
||||
## Contributor Checklist
|
||||
|
||||
1. Start from current `main`.
|
||||
|
||||
```bash
|
||||
git fetch upstream
|
||||
git rebase upstream/main
|
||||
```
|
||||
|
||||
If the branch has drifted heavily, create a fresh branch from `main` and
|
||||
cherry-pick only the commits that belong to the PR.
|
||||
|
||||
2. Keep the diff scoped to the PR title.
|
||||
|
||||
```bash
|
||||
git diff --name-status upstream/main...HEAD
|
||||
git diff --stat upstream/main...HEAD
|
||||
```
|
||||
|
||||
The changed file list should be easy to explain from the PR title. Do not
|
||||
ship stale-base churn, generated artifacts, formatter-only rewrites, or
|
||||
unrelated deletions.
|
||||
|
||||
3. Stop before pushing if the diff removes protected project evidence.
|
||||
|
||||
These paths should not disappear unless the PR is explicitly about replacing
|
||||
them and the PR body explains the replacement:
|
||||
|
||||
- benchmark corpora under `bench/`
|
||||
- benchmark scripts under `scripts/*bench*`
|
||||
- claim/provenance docs such as `docs/VERIFIED_PROVENANCE.md`
|
||||
- redaction, provider, local-agent, and adapter safety tests
|
||||
- provider configuration, environment templates, and API-key plumbing
|
||||
|
||||
4. Run the smallest meaningful verification set.
|
||||
|
||||
For most code changes:
|
||||
|
||||
```bash
|
||||
npm run typecheck
|
||||
npm test
|
||||
npm run doctor
|
||||
```
|
||||
|
||||
For docs-only changes, note why code tests were skipped. For UI-only
|
||||
`docs/index.html` changes, extract or otherwise parse-check the page scripts.
|
||||
For benchmark or headline-claim changes, also run:
|
||||
|
||||
```bash
|
||||
npm run verify-claims
|
||||
```
|
||||
|
||||
5. Add a contribution receipt when the change affects behavior, claims, target
|
||||
scope, agent execution, evidence, reports, or safety gates.
|
||||
|
||||
Use the template below. The receipt can live in the PR body unless the
|
||||
change introduces durable benchmark artifacts.
|
||||
|
||||
6. Write the PR body for review, not just for announcement.
|
||||
|
||||
Include:
|
||||
|
||||
- what changed
|
||||
- why it changed
|
||||
- exact commands run and pass/fail/skipped result
|
||||
- any skipped verification and why
|
||||
- screenshots or artifacts for UI/reporting changes
|
||||
- residual risk or follow-up work
|
||||
|
||||
## Coding-Agent Instructions
|
||||
|
||||
When an agent prepares a PR, it must perform a final scope audit before posting.
|
||||
|
||||
Use this command and inspect every line:
|
||||
|
||||
```bash
|
||||
git diff --name-status upstream/main...HEAD
|
||||
```
|
||||
|
||||
If the output contains an unrelated deletion, stop and repair the branch before
|
||||
asking for review. Common accidental stale-base deletions include benchmark
|
||||
fixtures, provenance docs, Gemini/provider tests, local-agent tests, and
|
||||
adapter-tool safety tests.
|
||||
|
||||
Agents should prefer a fresh branch when a rebase produces noisy conflicts. The
|
||||
repair path is:
|
||||
|
||||
1. create a branch from current `main`;
|
||||
2. copy only the files needed for the requested feature or fix;
|
||||
3. rerun the focused checks;
|
||||
4. update the PR body with the resulting receipt.
|
||||
|
||||
## Contribution Receipt Template
|
||||
|
||||
```markdown
|
||||
## Contribution Receipt
|
||||
|
||||
- Change:
|
||||
- Scope class: docs_only | static_fixture | local_lab | ctf_range | authorized_live
|
||||
- Target authority:
|
||||
- Network use: none | loopback | private_lab | authorized_external
|
||||
- Run mode labels:
|
||||
- Model/harness labels:
|
||||
- Commands run:
|
||||
- `npm run typecheck` -> pass/fail/skipped
|
||||
- `npm test` -> pass/fail/skipped
|
||||
- `npm run doctor` -> pass/fail/skipped
|
||||
- `npm run verify-claims` -> pass/fail/skipped
|
||||
- Artifacts:
|
||||
- Redaction:
|
||||
- Claims changed:
|
||||
- Abstentions/refusals:
|
||||
- Residual risk:
|
||||
```
|
||||
|
||||
## Maintainer Review Checklist
|
||||
|
||||
Before approving a PR, verify:
|
||||
|
||||
- `git diff --name-status main...HEAD` matches the PR title and description;
|
||||
- no unrelated protected evidence, benchmark, provider, redaction, or safety
|
||||
tests were removed;
|
||||
- required checks or documented substitutes were run;
|
||||
- UI-only changes have at least a static script parse check;
|
||||
- benchmark, claim, scope, local-agent, Arsenal, or safety-gate changes include
|
||||
a contribution receipt;
|
||||
- risky updater, shell, Docker, network, or local-agent execution changes have
|
||||
an explicit degraded-mode and security review note.
|
||||
|
||||
If a PR fails the scope audit, ask the submitter to rebase or recreate the
|
||||
branch from current `main` before deeper review. Do not spend review cycles on
|
||||
feature details while stale-base deletions remain in the diff.
|
||||
|
||||
Only comment on PRs that need action: concrete fixes, missing verification,
|
||||
missing receipts, or a rebase/scope cleanup. Ready PRs should not receive
|
||||
process comments.
|
||||
@@ -0,0 +1,38 @@
|
||||
# Target Header Injection
|
||||
|
||||
T3MP3ST can apply authentication headers to its HTTP arsenal tools. The headers are bound to one exact origin so credentials are not sent to other authorized targets, subdomains, ports, or redirect destinations.
|
||||
|
||||
## Configuration
|
||||
|
||||
Set both values in `~/.t3mp3st/.env`:
|
||||
|
||||
```bash
|
||||
TEMPEST_TARGET_ORIGIN=https://api.target.example
|
||||
TEMPEST_TARGET_HEADERS={"Authorization":"Bearer token","X-Tenant":"acme"}
|
||||
```
|
||||
|
||||
`TEMPEST_TARGET_ORIGIN` must contain only an `http` or `https` origin: scheme, hostname, and optional port. Paths, query strings, fragments, and embedded credentials are rejected. Origin matching includes the scheme and effective port.
|
||||
|
||||
`TEMPEST_TARGET_HEADERS` must be a non-empty JSON object whose values are strings. Invalid or incomplete configuration fails closed: the request continues without configured headers.
|
||||
|
||||
Transport-level headers such as `Host`, `Content-Length`, `Connection`, `Transfer-Encoding`, and proxy authorization headers are rejected. Authentication headers including `Authorization`, `Cookie`, and application-specific API key headers are supported.
|
||||
|
||||
## Coverage And Overrides
|
||||
|
||||
Configured headers are applied to HTTP requests made by all built-in web probes and to `curl_request` when the request URL matches the configured origin. They are not applied to unrelated origins.
|
||||
|
||||
Headers supplied explicitly by a tool call override configured defaults case-insensitively. For example, `authorization` overrides a configured `Authorization` value without sending duplicates.
|
||||
|
||||
## Redirect Safety
|
||||
|
||||
Built-in requests follow redirects while retaining configured headers only for same-origin hops. Before following a cross-origin redirect, T3MP3ST removes every configured header as well as standard credential headers.
|
||||
|
||||
When configured headers are active, `curl_request` removes `-L`, `--location`, and `--location-trusted` from caller-supplied flags. This prevents custom authentication headers from being forwarded to a redirect destination.
|
||||
|
||||
## Secret Handling
|
||||
|
||||
Configured curl headers are written to a temporary mode `0600` config file rather than command-line arguments, then deleted after execution. This keeps values out of the local process list.
|
||||
|
||||
Before Arsenal records or emits a tool result, configured header values are replaced with `[REDACTED]` in output, errors, findings, and other structured result fields. This also covers targets that reflect a credential in a response.
|
||||
|
||||
Keep `~/.t3mp3st/.env` private and never commit real credentials. Rotate a credential if it appears in logs or evidence created outside the Arsenal execution path.
|
||||
@@ -0,0 +1,64 @@
|
||||
# Verified Provenance
|
||||
|
||||
T3MP3ST is evidence-first: a finding should say whether it is backed by tool
|
||||
output, context, or only model reasoning.
|
||||
|
||||
If the evidence vault says `Provenance: none` or `model-asserted`, the finding is
|
||||
not yet verified. Treat it as a hypothesis until a tool result, retest, or
|
||||
operator-supplied artifact backs the claim.
|
||||
|
||||
## What Counts As Tool-Proven
|
||||
|
||||
A finding becomes tool-proven when all of these are true:
|
||||
|
||||
- The agent requests an Arsenal tool.
|
||||
- T3MP3ST executes that tool through the harness.
|
||||
- The returned output is captured as evidence.
|
||||
- The finding records the tool name or evidence reference that supports it.
|
||||
|
||||
Final prose from a model is useful analysis, but it is not the same thing as
|
||||
tool-backed evidence.
|
||||
|
||||
## Keyless And Local-Agent Runs
|
||||
|
||||
For keyless runs, connect a local agent in the War Room Settings first: Claude
|
||||
Code, Codex, Hermes, or another supported local-agent backend.
|
||||
|
||||
The local agent must request tools through T3MP3ST. If it only writes a final
|
||||
answer in prose, the harness has no tool output to attach and findings should stay
|
||||
unverified/model-asserted.
|
||||
|
||||
For a quick sanity check:
|
||||
|
||||
1. Start the War Room with `npm run server`.
|
||||
2. Connect a local agent in Settings.
|
||||
3. Use a local or explicitly authorized target.
|
||||
4. Run a harmless reconnaissance path.
|
||||
5. Open the evidence/finding view and check that at least one finding cites a
|
||||
tool name and captured output.
|
||||
|
||||
If every finding remains `Provenance: none`, the run may still be useful as a
|
||||
planning pass, but it did not verify findings through the tool loop.
|
||||
|
||||
## Common Reasons Provenance Is Missing
|
||||
|
||||
- No API provider or local agent is connected.
|
||||
- The chosen agent produced prose instead of a tool request.
|
||||
- The tool request could not be parsed by the local-agent/tool-call bridge.
|
||||
- The tool was catalog-only, missing locally, or receipt-gated.
|
||||
- The target or action was out of scope, so the harness correctly refused it.
|
||||
- The final report promoted model analysis before a retest or evidence item was
|
||||
attached.
|
||||
|
||||
## Operator Checklist
|
||||
|
||||
Before trusting a finding:
|
||||
|
||||
- Is there an evidence item?
|
||||
- Does the evidence item include tool output, a command result, a screenshot, or
|
||||
another reproducible artifact?
|
||||
- Does the finding cite the evidence or tool name?
|
||||
- Did any receipt or scope gate apply?
|
||||
- Has the claim been retested or falsified?
|
||||
|
||||
When in doubt, keep the finding labeled as unverified and queue a retest.
|
||||
+1096
-433
File diff suppressed because it is too large.
Load diff
Reference in new issue
Block a user