Building a 10-Plane Local AI Agent Harness at $0 Cost: How We Made 0.8B–8B SLMs Reliable on Pure CPU

Why you do not need a GPU rig, an API key, or a $200/month cloud subscription to run a production-grade agent harness — a complete systems breakdown of the open-source SLMAgents Pure-CPU Harness (./slm).

---

Every week, a new cloud coding assistant or autonomous agent framework launches with the same underlying assumption: to get reliable agentic behavior, you must rent a massive frontier LLM over an API and pay per token.

If you have ever hooked an agent loop up to a commercial API, you already know how fast the meter spins. A single multi-step task—reading a local SQLite schema, writing a query, fixing a column error, running a Python script, and formatting the result as JSON—can easily consume 25,000 to 50,000 prompt tokens across five turns. Multiply that across daily engineering workflows, and "accessible AI" quickly turns into a recurring $20 to $200/month per-seat tax, while your private code, financial spreadsheets, and database schemas are shipped to someone else's cloud.

On the flip side, developers who try to escape the cloud bill by downloading a 1.5B, 3B, or 4B parameter Small Language Model (SLM) and plugging it into a standard cloud-oriented agent framework almost always hit a wall:

1. JSON & Tool-Call Collapse: Prompting a 3B model with "Please respond ONLY with valid JSON matching one of these 25 tool schemas" results in markdown chatter, unescaped quotes, or hallucinated tool arguments.
2. CPU Prefill Starvation: Dumping 4,000 tokens of raw CSV rows, tool definitions, and conversation history into every turn makes a laptop CPU freeze for 10+ seconds before generating a single token.
3. Blind Hallucination Loops: Without deterministic verification, a small model that writes a slightly wrong SQL column name or Python import will either return broken code or get stuck repeating the exact same mistake until the context window overflows.
4. RAM Thrashing & OOM Crashes: Loading an LLM alongside an embedding model, a reranker, an OCR engine, and a speech-to-text model naively blows past 16 GB of system memory.

Small models do not fail because they lack reasoning capacity for everyday tasks; they fail because we wrap them in lazy harnesses designed for 200B-parameter cloud models.

When you move determinism out of the prompt and into the systems harness—using grammar-constrained decoding, <35ms cascade routing, content-addressed artifact handles, watermarked RAM arenas, and deterministic self-healing verifiers—quantized 0.8B to 8B models running on 100% standard CPU hardware (n_gpu_layers=0) become fast, trustworthy, and completely free of cost.

01 / The 10-Plane Pure-CPU Harness Architecture

Rather than treating an agent as a while True loop around a single chat prompt, slm_harness separates responsibilities across 10 orthogonal systems planes:

Plane	Subsystem (slm_harness/)	What It Solves on CPU at $0 Cost
Plane 1	Agentic Kernel (kernel/)	Typed EventBus, 4-mode TurnStateMachine, SQLite WAL CheckpointStore, /fork, /rewind, /undo via ShadowSnapshotVFS, and mid-flight SteerQueue.
Plane 2	Constrained Decoding & Tools (runtime/constrained.py, tools/)	Compiles JSON Schemas into LlamaGrammar (GBNF) and ONNX token-trie masks (ONNXPrefixEnumMasker) + RFC 8259 JSON repair; selects only Top-K tools per turn.
Plane 3	Context Engineering (context/)	Preserves static-prefix KV caches; replaces massive raw files with compact artifact://<sha12> descriptors; 256-token chunked prefill.
Plane 4	4-Tier Cognitive Memory (memory/)	T1 TurnBlackboard, T2 SQLite Session Tree, T3 HybridVectorFTS (FTS5 BM25 + INT8 ONNX Embeddings + RRF + bge-reranker-base), T4 ProceduralMemory (SLM.md).
Plane 5	Pure-CPU Compute & Arena (runtime/)	P-core/SIMD thread auto-tuner, LRU MemoryArena RAM Governor with automatic OOM tier step-down (8B -> 4B -> 3B -> 0.8B), <40ms CancellationToken.
Plane 6	Cascade Router & DAG (orchestrator/)	<35ms 4-stage CascadeRouter dispatching across 26 specialist agents and topological DAGScheduler with HandoffEnvelope contracts.
Plane 7	Deterministic Verifiers (verifiers/)	Live read-only SQLite verifier with Levenshtein column auto-healing, sandboxed Python AST/exec verifier, SymPy symbolic solver, and [1] RAG citation grounder.
Plane 8	Governance & HITL (governance/)	L0_PURE–L4_SYSTEM_OR_NET risk lattice, WorkspaceJail, POSIX rlimit subprocess sandbox, secret/PII scrubber, and interactive ApprovalBroker.
Plane 9	Unified Surfaces (surfaces/, website/)	Rich TUI (./slm chat), Unix CLI (./slm ask), 100% offline Web AI Studio (./slm serve), OpenAI /v1 API, and stdio MCP server (./slm mcp-serve).
Plane 10	Telemetry & Ops (telemetry/)	Hierarchical Waterfall Span Tracer (/trace), hardware diagnostics (./slm doctor), empirical benchmark suite (./slm bench), and FP32 duplicate pruner.

02 / The <35ms 4-Stage Cascade Router & Token-Trie Masking

In slm_harness/orchestrator/cascade_router.py, our 4-Stage Calibrated Cascade Router resolves over 90% of routing decisions across all 26 specialists in under 12 milliseconds on CPU (via MIME inspection, regex/AST signatures, and 1024-d INT8 ONNX embedding margins). When semantic margins are ambiguous, it invokes qwen3.5-0.8b-onnx constrained by ONNXPrefixEnumMasker, achieving 267.56 tokens/sec on CPU with 0% label hallucination:

class ONNXPrefixEnumMasker:
    def mask_logits(self, logits: np.ndarray, generated_ids: Sequence[int]) -> np.ndarray:
        valid_next = self.allowed_next_tokens(generated_ids)
        if not valid_next:
            return logits
        masked = np.full_like(logits, -1e9)
        idx = list(valid_next)
        masked[idx] = logits[idx]
        return masked

03 / Eliminating Syntax Failures with GBNF & Self-Healing Verifiers

Whenever structured output is requested, Plane 2 compiles the JSON Schema into a LlamaGrammar GBNF state machine, while Plane 7 verifies code, SQL, symbolic math, and citations deterministically. For SQL queries, verify_and_execute_sql runs against a read-only SQLite URI and uses Levenshtein fuzzy matching across PRAGMA table_info to auto-heal column typos in a single retry.

04 / Live Pure-CPU Benchmark Scorecard ($0 Cost)

Measured on standard CPU hardware (Profile=balanced, Threads=7, Peak RSS=3,272.1 / 4,700 MB):

Stage	Model	Backend	Latency / TTFT	Throughput	Verification Outcome
Embeddings (1024-d)	mxbai-embed-large	onnx_embed	0.03 ms (warm)	dim = 1024	PASS (0ms cold start after arena pin)
Primary LLM + GBNF JSON	qwen3-4b-gguf (Q4_K_M)	gguf	1143.0 ms TTFT	24.57 tok/s	JSON Schema Valid: {"engine":"slm_harness","cpu_only":true}
ONNX Enum Trie Router	qwen3.5-0.8b-onnx	onnx (INT4)	1249.0 ms TTFT	267.56 tok/s	Label = 'sql' (100% constrained token trie)

05 / Six Zero-Cost Workflows You Can Run Today

# 1. Inspect CPU topology, SIMD flags, RAM watermarks & 13 quantized models
./slm doctor

# 2. Run empirical Pure-CPU tok/s, TTFT, GBNF JSON Schema & MemoryArena benchmark
./slm bench

# 3. Launch Interactive Rich Terminal TUI / REPL (18 slash commands & @file attachments)
./slm chat

# 4. Headless single-shot & Unix stdin pipe with GBNF JSON-Schema constrained decoding
./slm ask "Solve 3*x + 9 = 30"
cat broken.json | ./slm ask "Extract clean record" \
  --schema '{"type":"object","properties":{"id":{"type":"integer"}},"required":["id"]}'

# 5. Launch 100% Offline Web AI Studio + OpenAI /v1 API (http://127.0.0.1:7860/chat.html)
./slm serve --port 7860

# 6. Expose SLMAgents Pure-CPU Harness as a stdio MCP Server
./slm mcp-serve

06 / Standalone OS Harness Installers (macOS, Windows, Linux)

In addition to running ./slm from source, SLMAgents 2.0 ships standalone self-contained installers for every operating system—both for the full 26-agent studio and as lightweight per-agent harness installers: