Merge pull request #20 from younger-plinius/feat/local-models-and-grok-build

feat: local models + Grok Build (xai) backbones run the Arsenal
This commit is contained in:
pliny authored and GitHub committed 2026-07-05 18:05:00 -04:00
commit a4e7fe96ff
6 files changed
+214 -45

No files matched your search

+19 -3
View File
@@ -22,9 +22,11 @@
</div>
Point T3MP3ST at an authorized target and the kill chain is yours — **recon → exploit → report**, from a web War Room or the CLI, driven by the AI coding agent you're already signed into (Claude Code, Codex, Hermes). No new API keys, no cloud, no second bill. Your agent is the brain; T3MP3ST is the war machine you bolt around it. Self-hosted storm, keyless warfare. ⚡
**Your AI coding agent is already a hacker — T3MP3ST hands it an arsenal.**
The recon engine is live and tool-backed; the exploit loop is benchmark-proven — **90.1% pass@1 on XBEN**, XBOW's own 104-challenge suite, plus hint-free CTF solves and a cold hunt on real post-cutoff CVEs. Every number recomputes from committed data (`npm run verify-claims`), and the [status table](#what-ships-today) is exact about what's live, what's scaffolding, and what's still roadmap. The full 8-operator swarm is the architecture it grows into — loud about the mission, honest about where the build is. Full receipts, caveats, and the held-out CVE breakdown are in [Benchmarks](#benchmarks) below.
Point it at an authorized target and the kill chain runs itself: **recon → exploit → report**, from a browser War Room or the CLI, driven by the agent you're *already* signed into — Claude Code, Codex, Hermes — or a model you run **fully offline** (Ollama, LM Studio, vLLM). No new API keys, no cloud tenant, no second bill. Your agent is the brain; T3MP3ST is the war machine bolted around it. **Self-hosted storm. Keyless warfare.** ⚡
And it won't ask you to take its word for it. On **XBOW's own 104-challenge suite it scores 90.1% pass@1** — above XBOW's self-reported 85% — alongside hint-free CTF solves and a **cold hunt on real, post-cutoff CVEs the model had never seen**. Every number in this README recomputes from committed data with one command (`npm run verify-claims`). Loud about the mission, honest about the build — the [status table](#what-ships-today) says exactly what's live, what's scaffolding, and what's still roadmap; full receipts in [Benchmarks](#benchmarks).
Three things set it apart:
@@ -32,6 +34,8 @@ Three things set it apart:
2. **Keyless.** The AI coding agent already on your machine is the backbone. No API keys, no second bill, no gatekeeper.
3. **Honest about scope.** The [status table](#what-ships-today) marks exactly what's stable, experimental, or roadmap — because red-teaming shouldn't be a priesthood, and it damn sure shouldn't run on vibes.
**Jump to** → [Quick start](#quick-start) · [What it hunts](#what-it-hunts) · [What ships today](#what-ships-today) · [Benchmarks](#benchmarks) · [Architecture](#architecture) · [Docs](#documentation)
## ⚠️ Authorized use only
T3MP3ST is an **offensive** security tool, built for **authorized** testing, research, and education. Point it **only** at systems you own or have **explicit, written permission** to test. Unauthorized access to computers, networks, or data is illegal in most jurisdictions — **you alone are responsible** for how you use this software and for staying inside the law and your rules of engagement. Bring the storm to *your* targets, not someone else's.
@@ -66,9 +70,21 @@ In the War Room, open **Settings** and connect a local agent (Claude Code / Code
Prefer to bring a key? Set one and skip the connect step:
```bash
export OPENROUTER_API_KEY=... # or VENICE_API_KEY / ANTHROPIC_API_KEY
export OPENROUTER_API_KEY=... # or VENICE_API_KEY / ANTHROPIC_API_KEY / OPENAI_API_KEY
export XAI_API_KEY=... # Grok Build (grok-build-0.1) — xAI's coding model, native tool-calling
```
Or run it **fully offline** on your own model — no key, no cloud. Defaults to Ollama; point it at any OpenAI-compatible server (LM Studio, vLLM, llama.cpp):
```bash
ollama serve && ollama pull llama3 # or an OpenAI-compatible server
export TEMPEST_LOCAL_BASE_URL=http://localhost:11434/api # LM Studio: http://localhost:1234/v1
export TEMPEST_LOCAL_MODEL=llama3
npx tempest config # → "Change default provider" → local
```
Tool-calling works on any local model (it's driven over text), so the Arsenal runs even on models without native function-calling.
Check the numbers for yourself:
```bash
+58 -1
View File
@@ -4,7 +4,7 @@
* turn 0 and every operator abstains without ever running the Arsenal. These tests pin the fix AND
* the parser-hardening from the PR #16 audit (over-match, ReDoS, drift-abstains, string args, Codex coverage).
*/
import { describe, it, expect, vi } from 'vitest';
import { describe, it, expect, vi, afterEach } from 'vitest';
import { EventEmitter } from 'events';
// Mock the local-agent CLI bridge (LocalAgentAdapter) and the codex spawn/file read (CodexAdapter).
@@ -97,3 +97,60 @@ describe('codex backbone surfaces toolCalls (guards the CodexAdapter half of the
expect(res.toolCalls).toBeUndefined();
});
});
describe('local-model (HTTP) backbone surfaces toolCalls — the keyless-path fix for self-hosted models', () => {
const origFetch = global.fetch;
afterEach(() => { global.fetch = origFetch; });
const mockJson = (body: unknown) => {
const spy = vi.fn(async (_url: string, _init: { body: string }) => ({ ok: true, json: async () => body } as unknown as Response));
global.fetch = spy as unknown as typeof fetch;
return spy;
};
const localModel = (baseUrl?: string) => new LLMBackbone({ provider: 'local', model: 'llama3', baseUrl } as never);
it('Ollama wire (/api/chat): parses a tool request out of message.content', async () => {
mockJson({ model: 'llama3', message: { role: 'assistant', content: '```json\n{"tool_calls":[{"name":"nmap_scan","arguments":{"target":"x"}}]}\n```' } });
const res = await localModel('http://localhost:11434/api').chatWithTools([{ role: 'user', content: 'scan' }], TOOLS as never);
expect(res.toolCalls?.[0]?.name).toBe('nmap_scan');
expect(res.finishReason).toBe('tool_calls');
});
it('OpenAI-compatible wire (/v1): parses out of choices[0].message.content + hits /v1/chat/completions', async () => {
const spy = mockJson({ model: 'local', choices: [{ message: { content: '{"tool_calls":[{"name":"nmap_scan","arguments":{"target":"y"}}]}' } }] });
const res = await localModel('http://localhost:1234/v1').chatWithTools([{ role: 'user', content: 'scan' }], TOOLS as never);
expect(res.toolCalls?.[0]?.name).toBe('nmap_scan');
expect(spy.mock.calls[0][0] as string).toContain('/v1/chat/completions');
});
it('prose debrief → final answer, not a forever-abstain', async () => {
mockJson({ model: 'llama3', message: { content: 'No exploitable surface found. Final debrief.' } });
const res = await localModel().chatWithTools([{ role: 'user', content: 'scan' }], TOOLS as never);
expect(res.toolCalls).toBeUndefined();
expect(res.finishReason).toBe('stop');
});
it('describes the Arsenal to the model (so a local model knows what it can request)', async () => {
const spy = mockJson({ message: { content: 'ok' } });
await localModel().chatWithTools([{ role: 'user', content: 'scan' }], TOOLS as never);
const body = JSON.parse((spy.mock.calls[0][1] as { body: string }).body);
const sys = body.messages.find((m: { role: string }) => m.role === 'system');
expect(sys.content).toContain('nmap_scan');
});
});
describe('xai provider (Grok Build) — routes to xAI OpenAI-compatible API with the key', () => {
const origFetch = global.fetch;
afterEach(() => { global.fetch = origFetch; });
it('hits api.x.ai with the bearer key and surfaces native tool_calls', async () => {
const spy = vi.fn(async (_url: string, _init: { headers: Record<string, string> }) => ({
ok: true,
json: async () => ({
model: 'grok-build-0.1',
choices: [{ message: { content: '', tool_calls: [{ id: 'c1', function: { name: 'nmap_scan', arguments: '{"target":"x"}' } }] }, finish_reason: 'tool_calls' }],
}),
} as unknown as Response));
global.fetch = spy as unknown as typeof fetch;
const be = new LLMBackbone({ provider: 'xai', model: 'grok-build-0.1', apiKey: 'xai-test-key-1234567890', baseUrl: 'https://api.x.ai/v1' } as never);
const res = await be.chatWithTools([{ role: 'user', content: 'scan' }], TOOLS as never);
expect(res.toolCalls?.[0]?.name).toBe('nmap_scan');
expect(spy.mock.calls[0][0]).toContain('api.x.ai');
expect((spy.mock.calls[0][1] as { headers: Record<string, string> }).headers.Authorization).toContain('xai-test-key');
});
});
+2 -2
View File
@@ -164,7 +164,7 @@ async function interactiveMode(): Promise<void> {
showBanner();
// Check for API keys
const providers = getConfiguredProviders().filter(p => p !== 'mock' && p !== 'local');
const providers = getConfiguredProviders().filter(p => p !== 'mock');
if (providers.length === 0) {
showBox(
@@ -527,7 +527,7 @@ async function openSettings(): Promise<void> {
break;
case 'provider':
const providers = getConfiguredProviders().filter(p => p !== 'mock' && p !== 'local');
const providers = getConfiguredProviders().filter(p => p !== 'mock');
if (providers.length === 0) {
showWarning('No API keys configured');
break;
+60 -12
View File
@@ -22,6 +22,7 @@ export interface TempestSettings {
venice?: string;
anthropic?: string;
openai?: string;
xai?: string;
};
// Default LLM settings
@@ -54,6 +55,12 @@ export interface TempestSettings {
defaultModel: string;
};
// xAI — Grok Build / Grok models (OpenAI-compatible API)
xai: {
baseUrl: string;
defaultModel: string;
};
// Codex CLI/account subscription backend
codex: {
command: string;
@@ -112,6 +119,11 @@ const DEFAULT_SETTINGS: TempestSettings = {
defaultModel: 'gpt-4-turbo-preview',
},
xai: {
baseUrl: 'https://api.x.ai/v1',
defaultModel: 'grok-build-0.1',
},
codex: {
command: 'codex',
defaultModel: 'codex-default',
@@ -389,14 +401,35 @@ export const AVAILABLE_MODELS: Record<LLMProvider, ModelInfo[]> = {
capabilities: ['testing'],
},
],
// Model IDs verified against xAI's published model list (docs.x.ai/docs/models, 2026-07-05):
// grok-build-0.1 = coding model (256K ctx); grok-4.3 = general (1M ctx). Any current xAI
// model id can be passed via config/CLI/model arg — these are just the curated defaults.
xai: [
{
id: 'grok-build-0.1',
name: 'Grok Build (grok-build-0.1, 256K)',
provider: 'xAI',
contextWindow: 256000,
maxOutput: 8192,
capabilities: ['reasoning', 'code', 'analysis', 'agents', 'tools'],
},
{
id: 'grok-4.3',
name: 'Grok 4.3 (general, 1M)',
provider: 'xAI',
contextWindow: 1000000,
maxOutput: 8192,
capabilities: ['reasoning', 'code', 'analysis', 'agents', 'tools'],
},
],
local: [
{
id: 'local-model',
name: 'Local Model',
name: 'Local model (Ollama / LM Studio / vLLM — set TEMPEST_LOCAL_MODEL)',
provider: 'Local',
contextWindow: 32000,
maxOutput: 4096,
capabilities: ['reasoning', 'code'],
capabilities: ['reasoning', 'code', 'tools'],
},
],
'local-agent': [
@@ -486,6 +519,7 @@ class ConfigManager {
// Environment variables (including values loaded from ~/.t3mp3st/.env above)
// are intentionally process-local. getApiKey() already gives env vars highest
// priority, so do not persist them into Conf's apiKeys store as a side effect.
// (XAI_API_KEY / Grok Build follows the same safe path — see the envVarMap in getApiKey.)
this.envLoaded = true;
}
@@ -513,7 +547,7 @@ class ConfigManager {
/**
* Set an API key for a provider
*/
setApiKey(provider: 'openrouter' | 'venice' | 'anthropic' | 'openai', key: string): void {
setApiKey(provider: 'openrouter' | 'venice' | 'anthropic' | 'openai' | 'xai', key: string): void {
const apiKeys = this.config.get('apiKeys');
apiKeys[provider] = key;
this.config.set('apiKeys', apiKeys);
@@ -522,13 +556,14 @@ class ConfigManager {
/**
* Get an API key for a provider
*/
getApiKey(provider: 'openrouter' | 'venice' | 'anthropic' | 'openai'): string | undefined {
getApiKey(provider: 'openrouter' | 'venice' | 'anthropic' | 'openai' | 'xai'): string | undefined {
// First check environment variables (highest priority)
const envVarMap = {
openrouter: 'OPENROUTER_API_KEY',
venice: 'VENICE_API_KEY',
anthropic: 'ANTHROPIC_API_KEY',
openai: 'OPENAI_API_KEY',
xai: 'XAI_API_KEY',
};
// Force a fully UNCONFIGURED server (no key from env OR the saved store) — used by
@@ -547,7 +582,7 @@ class ConfigManager {
/**
* Check if a provider has a valid API key configured
*/
hasApiKey(provider: 'openrouter' | 'venice' | 'anthropic' | 'openai'): boolean {
hasApiKey(provider: 'openrouter' | 'venice' | 'anthropic' | 'openai' | 'xai'): boolean {
const key = this.getApiKey(provider);
return !!key && key.length > 10;
}
@@ -555,7 +590,7 @@ class ConfigManager {
/**
* Remove an API key
*/
removeApiKey(provider: 'openrouter' | 'venice' | 'anthropic' | 'openai'): void {
removeApiKey(provider: 'openrouter' | 'venice' | 'anthropic' | 'openai' | 'xai'): void {
const apiKeys = this.config.get('apiKeys');
delete apiKeys[provider];
this.config.set('apiKeys', apiKeys);
@@ -571,6 +606,7 @@ class ConfigManager {
if (this.hasApiKey('venice')) providers.push('venice');
if (this.hasApiKey('anthropic')) providers.push('anthropic');
if (this.hasApiKey('openai')) providers.push('openai');
if (this.hasApiKey('xai')) providers.push('xai');
// Codex uses the local Codex CLI/account auth instead of API-key storage.
providers.push('codex');
@@ -612,6 +648,12 @@ class ConfigManager {
baseUrl = this.config.get('openai').baseUrl;
actualModel = model || this.config.get('openai').defaultModel;
break;
case 'xai':
// Grok Build / Grok — xAI's OpenAI-compatible API (native tool-calling).
apiKey = this.getApiKey('xai');
baseUrl = this.config.get('xai').baseUrl;
actualModel = model || this.config.get('xai').defaultModel;
break;
case 'codex':
actualModel = model || this.config.get('codex').defaultModel;
break;
@@ -619,8 +661,13 @@ class ConfigManager {
actualModel = 'mock-model';
break;
case 'local':
baseUrl = 'http://localhost:11434/api';
actualModel = model || 'llama3';
// Self-hosted, keyless. Defaults to Ollama; point TEMPEST_LOCAL_BASE_URL at
// any OpenAI-compatible server (LM Studio :1234/v1, vLLM :8000/v1, llama.cpp)
// and TEMPEST_LOCAL_MODEL at the model tag you're serving.
baseUrl = process.env.TEMPEST_LOCAL_BASE_URL?.trim() || 'http://localhost:11434/api';
// 'local-model' is the placeholder id from AVAILABLE_MODELS — treat it as unset so
// TEMPEST_LOCAL_MODEL (or the llama3 default) wins; a real tag passed in still takes priority.
actualModel = (model && model !== 'local-model' ? model : undefined) || process.env.TEMPEST_LOCAL_MODEL?.trim() || 'llama3';
break;
default:
throw new Error(`Unknown provider: ${actualProvider}`);
@@ -652,7 +699,7 @@ class ConfigManager {
const flag = (process.env.TEMPEST_MODEL_FALLBACK || '').trim().toLowerCase();
if (!flag || ['0', 'false', 'off', 'no'].includes(flag)) return [];
const chain: FallbackEntry[] = [];
const add = (p: 'openrouter' | 'venice' | 'anthropic' | 'openai') => {
const add = (p: 'openrouter' | 'venice' | 'anthropic' | 'openai' | 'xai') => {
if (p === primary || !this.hasApiKey(p)) return;
chain.push({
provider: p,
@@ -665,6 +712,7 @@ class ConfigManager {
add('venice');
add('anthropic');
add('openai');
add('xai');
return chain;
}
@@ -782,8 +830,8 @@ OPENAI_API_KEY=
export const config = new ConfigManager();
// Helper functions for quick access
export const getApiKey = (provider: 'openrouter' | 'venice' | 'anthropic' | 'openai') => config.getApiKey(provider);
export const setApiKey = (provider: 'openrouter' | 'venice' | 'anthropic' | 'openai', key: string) => config.setApiKey(provider, key);
export const hasApiKey = (provider: 'openrouter' | 'venice' | 'anthropic' | 'openai') => config.hasApiKey(provider);
export const getApiKey = (provider: 'openrouter' | 'venice' | 'anthropic' | 'openai' | 'xai') => config.getApiKey(provider);
export const setApiKey = (provider: 'openrouter' | 'venice' | 'anthropic' | 'openai' | 'xai', key: string) => config.setApiKey(provider, key);
export const hasApiKey = (provider: 'openrouter' | 'venice' | 'anthropic' | 'openai' | 'xai') => config.hasApiKey(provider);
export const getLLMConfig = (provider?: LLMProvider, model?: string) => config.getLLMConfig(provider, model);
export const getConfiguredProviders = () => config.getConfiguredProviders();
+74 -26
View File
@@ -705,6 +705,14 @@ class MockAdapter implements LLMProviderAdapter {
// LOCAL ADAPTER (Ollama, etc.)
// =============================================================================
// Local model over HTTP — a fully self-hosted, keyless backbone. Talks to Ollama's
// native /api/chat by default, or any OpenAI-compatible local server (LM Studio,
// vLLM, llama.cpp, Ollama's own /v1) when TEMPEST_LOCAL_BASE_URL is a /v1 endpoint.
// Tool-calling is done over TEXT (renderToolContract + parseTextToolCalls, defined
// below and shared with the CLI-agent adapters): it works on ANY local model,
// whether or not it supports native function-calling — without it a local model
// hits the keyless-path abstain bug (no toolCalls → ReAct bails turn 0 → Arsenal
// never runs).
class LocalAdapter implements LLMProviderAdapter {
name = 'local';
private config: LLMConfig;
@@ -717,28 +725,55 @@ class LocalAdapter implements LLMProviderAdapter {
return { valid: true };
}
async chat(messages: LLMMessage[], options?: ChatOptions): Promise<LLMResponse> {
const baseUrl = this.config.baseUrl || 'http://localhost:11434/api';
const url = `${baseUrl}/chat`;
// A /v1 base URL means an OpenAI-compatible server (/chat/completions, choices[]).
private isOpenAIWire(baseUrl: string): boolean {
return /\/v1(\/|$)/.test(baseUrl);
}
const requestBody = {
model: this.config.model,
messages: messages.map(m => ({
role: m.role,
content: m.content,
})),
stream: false,
options: {
num_predict: options?.maxTokens || this.config.maxTokens || 4096,
temperature: options?.temperature ?? this.config.temperature ?? 0.7,
},
};
// Inject the Arsenal contract as a system turn when tools are offered, and
// sanitize prior tool-request / tool-result turns — a plain local model doesn't
// understand role:'tool', and re-emitting a prior ```json``` block would get
// re-parsed as a fresh live call (defeats termination). Its TEXT reply is parsed
// back into tool calls after the round-trip.
private buildMessages(messages: LLMMessage[], options?: ChatOptions): { role: string; content: string }[] {
const contract = renderToolContract(options?.tools);
const preamble = 'You are the planning brain for T3MP3ST, an authorized offensive-security harness.'
+ (contract
? ' You drive a ReAct loop: REQUEST tools, the harness runs them and returns results, you reason until the surface is exhausted.' + contract
: ' Operate in planning and analysis mode; when JSON is requested, return ONLY the requested block — no preamble.');
const out: { role: string; content: string }[] = [{ role: 'system', content: preamble }];
for (const m of messages) {
if (m.role === 'assistant' && m.toolCalls?.length) {
out.push({ role: 'assistant', content: `[requested tools: ${m.toolCalls.map(t => t.name).join(', ')}]` });
} else if (m.role === 'tool') {
out.push({ role: 'user', content: `TOOL RESULT (${m.name || 'tool'}):\n${m.content}` });
} else {
out.push({ role: m.role, content: m.content });
}
}
return out;
}
async chat(messages: LLMMessage[], options?: ChatOptions): Promise<LLMResponse> {
const baseUrl = (this.config.baseUrl || 'http://localhost:11434/api').replace(/\/$/, '');
const openaiWire = this.isOpenAIWire(baseUrl);
const url = `${baseUrl}/${openaiWire ? 'chat/completions' : 'chat'}`;
const wireMessages = this.buildMessages(messages, options);
const maxTokens = options?.maxTokens || this.config.maxTokens || 4096;
const temperature = options?.temperature ?? this.config.temperature ?? 0.7;
const requestBody = openaiWire
? { model: this.config.model, messages: wireMessages, max_tokens: maxTokens, temperature, stream: false }
: { model: this.config.model, messages: wireMessages, stream: false, options: { num_predict: maxTokens, temperature } };
try {
const response = await fetch(url, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
// Local OpenAI-compatible servers usually ignore auth, but LM Studio & co.
// expect *some* bearer — send a dummy unless the operator set a real key.
...(openaiWire ? { Authorization: `Bearer ${this.config.apiKey || 'local'}` } : {}),
},
body: JSON.stringify(requestBody),
signal: AbortSignal.timeout(this.config.timeout || 120000),
@@ -748,24 +783,35 @@ class LocalAdapter implements LLMProviderAdapter {
throw new Error(`Local LLM error: ${response.status}`);
}
const data = await response.json() as OllamaResponse;
const data = await response.json() as OllamaResponse & {
choices?: Array<{ message?: { content?: string } }>;
usage?: { prompt_tokens: number; completion_tokens: number; total_tokens: number };
};
const content = openaiWire ? (data.choices?.[0]?.message?.content || '') : (data.message?.content || '');
// Tool-calling over text: if the Arsenal was offered, parse the model's tool
// requests so the ReAct loop EXECUTES them instead of abstaining on turn 0.
const toolCalls = options?.tools?.length ? parseTextToolCalls(content) : undefined;
const usage = openaiWire
? (data.usage
? { promptTokens: data.usage.prompt_tokens, completionTokens: data.usage.completion_tokens, totalTokens: data.usage.total_tokens }
: undefined)
: (data.eval_count
? { promptTokens: data.prompt_eval_count || 0, completionTokens: data.eval_count || 0, totalTokens: (data.prompt_eval_count || 0) + (data.eval_count || 0) }
: undefined);
return {
content: data.message?.content || '',
content,
model: data.model || this.config.model,
usage: data.eval_count
? {
promptTokens: data.prompt_eval_count || 0,
completionTokens: data.eval_count || 0,
totalTokens: (data.prompt_eval_count || 0) + (data.eval_count || 0),
}
: undefined,
finishReason: 'stop',
usage,
finishReason: toolCalls?.length ? 'tool_calls' : 'stop',
toolCalls,
};
} catch (error) {
if (error instanceof TypeError && error.message.includes('fetch')) {
throw new Error(
'Could not connect to local LLM. Make sure Ollama is running (ollama serve)'
'Could not connect to local LLM. Start Ollama (`ollama serve`), or point TEMPEST_LOCAL_BASE_URL at your OpenAI-compatible server (LM Studio / vLLM / llama.cpp).'
);
}
throw error;
@@ -1155,6 +1201,8 @@ export class LLMBackbone extends EventEmitter<LLMEvents> {
return new AnthropicAdapter(config);
case 'openai':
return new OpenAIAdapter(config);
case 'xai':
return new OpenAIAdapter(config); // xAI (Grok Build / grok-*) is OpenAI-compatible
case 'codex':
return new CodexAdapter(config);
case 'mock':
+1 -1
View File
@@ -6,7 +6,7 @@
// LLM CONFIGURATION
// =============================================================================
export type LLMProvider = 'openrouter' | 'venice' | 'anthropic' | 'openai' | 'codex' | 'mock' | 'local' | 'local-agent';
export type LLMProvider = 'openrouter' | 'venice' | 'anthropic' | 'openai' | 'xai' | 'codex' | 'mock' | 'local' | 'local-agent';
export interface LLMConfig {
provider: LLMProvider;