Files as Memory, Modes as Principals: Lessons from Building a Local-First Personal Agent Without a Framework

Cipher is a single-user, voice-first personal assistant that runs as a local desktop application and stores everything it knows as plain markdown in an Obsidian vault its owner can read, edit, and delete. It is built without an agent framework: a single hand-rolled agent loop in plain Python, lexical-semantic retrieval fused by reciprocal rank fusion, an intent classifier that files structured notes, a mode system that treats the active mode as a security principal for tool access, and a human-in-the-loop oversight surface for every autonomous write. This report contributes (1) a reference architecture for that design point; (2) an empirical characterization against the system's own stated targets, on the live vault and on schema-matched synthetic vaults up to 1,000 notes: lenses render in ~4.5 ms live and hold the <200 ms target at 500 notes (worst-lens p95 ~110 ms, synthetic) with a measured ceiling near 1,000; recall p95 stays under 41 ms at every size (target: <1 s); a 5,390-token cached prompt prefix verified live; and a USD 50/month inference budget enforced by per-category caps, under which the system's entire build-and-use history to date has cost USD 1.43, all at near-zero infrastructure; (3) a documented design rationale, including why agent-memory frameworks were rejected, why markdown is canonical, why a fanless laptop makes cloud inference the architecturally correct choice, and four observed failure modes of AI-assisted solo development with the multi-chat methodology used to mitigate them; and (4) a security case study of two real authorization-boundary incidents, an injection-shaped tool description shipped by an official MCP server and a laundered-authorization injection against the auto-approving development tooling, together with their shared mitigation. All results are from one user on one machine; synthetic-scale measurements are labeled as such.