# PXOS Complete LLM Reference (v2.3.1) > Aggregated documentation for direct AI context injection, indexing, and RAG pipelines. Generated automatically from repository sources (https://github.com/madebypx/PXOS). --- ## DOCUMENT: README.md (Executive & Core Architecture) # PXOS — The AI Operating System for Product & Software Engineering > **Transforming AI agents from erratic, chatty coders into disciplined, product-aware senior engineering partners.** **PXOS** is a framework-agnostic, document-driven operating system for AI-assisted development. It establishes an operational contract between human product builders and AI agents—standardizing reasoning, enforcing **Product Design and UX heuristics**, eliminating architectural amnesia, and enabling scalable **multi-agent parallel execution** without merge collisions or token waste. --- ## Table of Contents - [Executive Summary](#executive-summary) - [The Concrete Difference (Empirical Benchmarks)](#the-concrete-difference-empirical-benchmarks) - [Product Design & UX as First-Class Citizens](#product-design--ux-as-first-class-citizens) - [The 6-Phase Disciplined Development Lifecycle](#the-6-phase-disciplined-development-lifecycle) - [Multi-Agent Git Worktree Engine](#multi-agent-git-worktree-engine) - [Architecture & Workspace Structure](#architecture--workspace-structure) - [Workflows & Slash Commands Directory](#workflows--slash-commands-directory) - [Quick Installation & IDE Integration](#quick-installation--ide-integration) - [Downstream Adoption Badges](#downstream-adoption-badges) - [Standard LLM Ingestion (llms.txt)](#standard-llm-ingestion-llmstxt) - [The PXOS Engineering Manifesto](#the-pxos-engineering-manifesto) - [Author & Portfolio Attribution](#author--portfolio-attribution) - [Architectural Whitepaper](./docs/WHITEPAPER.md) - [Changelog](./CHANGELOG.md) - [License](#license) --- ## Executive Summary ### Why Modern AI Development Breaks Down LLMs are extraordinary code generators, but they are fundamentally unconstrained. When developers use AI assistants via raw chat interfaces or naive autonomous agent swarms, projects suffer from six chronic failure modes: 1. **Architectural Amnesia:** Every new session loses context. The AI casually swaps libraries, duplicates utilities, and re-introduces previously solved bugs. 2. **Blind Implementation (Zero UX Awareness):** Agents write functional code in a vacuum, ignoring user mental models, loading states, empty states, error recovery, and design token consistency. 3. **Chat-Loop Token Hemorrhage:** Unstructured chat loops burn hundreds of thousands of tokens re-explaining the project, degrading attention windows and inflating API bills. 4. **Premature Abstraction & Overengineering:** Left unguided, agents write speculative abstractions for non-existent future requirements rather than the simplest working solution. 5. **Multi-Agent Collision Chaos:** Multiple agents operating on the same branch inevitably overwrite shared files, break each other's assumptions, and create unresolvable merge conflicts. 6. **No Risk Boundaries:** Agents casually perform high-risk operations (rewriting schemas, replacing dependencies) without explicit human consent. ### The PXOS Solution PXOS solves this by providing a **minimal, reusable operational layer** (`.ai/`) stored directly inside the repository. Rather than relying on volatile conversational memory, PXOS uses **version-controlled, structured Markdown artifacts** as the asynchronous communication medium. Humans retain strategic control (Product Vision, UX Architecture, High-Risk Approvals), while AI agents execute with surgical precision across specialized roles (Architect, Executor, Auditor). --- ## The Concrete Difference (Empirical Benchmarks) PXOS was engineered to replace chaotic conversational loops with deterministic engineering rigor. The quantitative impact across real-world codebases demonstrates radical efficiency gains: ### Benchmark Comparison Matrix | Dimension | Vanilla AI Chat (ChatGPT / Claude / Copilot Chat) | Autonomous Agent Swarms (AutoGPT / CrewAI / Multi-Agent Chat) | **PXOS Document-Driven Framework** | | :--- | :--- | :--- | :--- | | **Context Token Waste** | **High** (Re-ingests entire chat history every turn) | **Extreme** (100k+ tokens burned in inter-agent chat banter) | **Minimal (72% – 84% reduction)** via isolated Markdown artifacts | | **API Cost Per Feature** | Baseline ($$$) | 3x – 5x Baseline ($$$$$) | **65% – 78% Cost Reduction** via 80/20 Model Specialization | | **Architectural Drift / Amnesia** | Constant drift across sessions | Frequent hallucinated architecture | **0% Drift** (100% continuity via persistent `DECISION_LOG.md`) | | **Parallel Feature Delivery** | Impossible (Single thread) | Unstable (File clobbering & race conditions) | **3.8x Velocity** via zero-collision Git Worktrees | | **Code Rework & Regressions** | 45% – 60% of code rewritten | 50% – 70% discarded due to drift | **< 12% Rework** via mandatory Discover → Plan gates | | **UX & Interaction Coverage** | Almost zero (only happy paths) | Patchy & uncoordinated | **100% Coverage** (States, edge cases & Nielsen heuristics enforced) | | **Human Strategic Control** | Micromanagement required | Black-box loss of control | **Precision Gates** (Low/Med/High autonomy boundaries) | ### Key Performance Telemetry ``` [Token Consumption per Complex Feature] Vanilla Chat: ████████████████████████████ 320k tokens Agent Swarms: ████████████████████████████████████████ 480k tokens PXOS Framework: ██████ 62k tokens (-81% reduction) [Rework / Regression Rate] Vanilla Chat: ██████████████████ 52% rework PXOS Framework: ████ 11% rework (4.7x accuracy improvement) [Context Continuity Across Sprints] Vanilla Chat: 0% (Session memory resets every prompt) PXOS Framework: 100% (Durable ADRs, Spec History & Subsystem Audits) ``` 1. **72% – 84% Token Waste Reduction:** By replacing chat banter with structured artifact handoffs (`SPEC-*.md` and `AI_BASE.md`), agents read only the concise context required for their specific task. 2. **The 80/20 Model Economy:** PXOS routes 90% of raw code execution to fast, cost-effective models (e.g. Gemini Flash / Claude Haiku) operating in isolated worktrees, reserving expensive high-reasoning models (e.g. Claude Sonnet / Gemini Pro) exclusively for initial architecture (`/spec`) and gatekeeper code audits (`/review`). 3. **Zero Git Merge Collisions:** Worktrees physically decouple concurrent agents into isolated filesystem trees (`../trees/feat-*`), guaranteeing that parallel agents never touch each other's uncommitted work. ### Empirical Validation Suite & Reproducibility PXOS does not rely on hand-waved claims or marketing hyperbole. Every efficiency metric, cost formula, and UX heuristic is evaluated via our open-source empirical benchmark suite: - **[Benchmark Methodology](./benchmarks/METHODOLOGY.md):** Formal research questions (RQs), control vs. treatment experimental design, mathematical cost functions, and bias mitigation protocols. - **[Agent Interview & Elicitation Protocol](./benchmarks/AGENT_INTERVIEW_PROTOCOL.md):** Adversarial, anti-sycophantic elicitation prompts and strict JSON schemas to extract unvarnished telemetry from working AI agents. - **[Automated Analysis Engine (`analyze.py`)](./benchmarks/analyze.py):** Zero-dependency Python 3 engine that parses agent telemetry, computes statistical distributions, and generates audited markdown reports. - **[Telemetry Server (`server.py`)](./benchmarks/server.py):** Lightweight, zero-dependency ingestion server running at `https://telemetry.madebypx.com` to aggregate anonymous community benchmarks. - **[Live Telemetry Monitor (`pxos-telemetry-monitor.py`)](./scripts/pxos-telemetry-monitor.py):** Terminal dashboard and CI utility for real-time daemon stability checks and public submission monitoring. - **[Latest Benchmark Report](./benchmarks/REPORT.md):** Real-world audited telemetry across Tier 1 (micro), Tier 2 (medium), and Tier 3 (complex) engineering tasks. --- ## Product Design & UX as First-Class Citizens Unlike traditional developer frameworks that treat user interfaces merely as frontend code syntax, PXOS embeds **Product Design, Human-Computer Interaction (HCI), and UX Strategy** directly into the core operating loop. ``` ┌─────────────────────────────────────────────────────────┐ │ PRODUCT DESIGN & UX IN PXOS │ └────────────────────────────┬────────────────────────────┘ │ ┌─────────────────────────────────┼─────────────────────────────────┐ ▼ ▼ ▼ ┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐ │ 1. DISCOVER │ │ 2. SPEC │ │ 5. REVIEW │ │ Macro Research │ │ User Journey │ │ Nielsen Heuristics│ ├───────────────────┤ ├───────────────────┤ ├───────────────────┤ │ • User Personas │ │ • Core User Value │ │ • System Status │ │ • Mental Models │ │ • Interactive Flow│ │ • Error Shielding │ │ • Design Tokens │ │ • Edge & Empty UI │ │ • Cognitive Load │ │ • Benchmark Spikes│ │ • Accessibility │ │ • Micro-Feedback │ └───────────────────┘ └───────────────────┘ └───────────────────┘ ``` ### 1. Macro UX Grounding (`.ai/research/` & `docs/DESIGN.md`) Before an agent touches a single line of code, the **Discover phase** mandates reading project research indexed in `.ai/research/INDEX.md` and design language rules in `docs/DESIGN.md`. - Prevents arbitrary color palettes, random spacings, and ad-hoc CSS utility classes. - Grounds the agent in the established design system tokens: **Primitive → Semantic → Component**. - Maintains brand tone of voice and target user mental models across all product screens. ### 2. User-Value & State-Driven Specifications (`/spec`) In PXOS, an AI is forbidden from writing technical implementation plans until the user experience is formalized. Every task specification (`.ai/specs/SPEC-*.md`) explicitly mandates: - **Core User Value:** Clear statement of what user friction is eliminated. - **Interaction & Step-by-Step Flow:** The complete human journey from trigger to resolution. - **Total State Coverage:** Mandatory modeling of **Empty States**, **Loading Skeletons**, **Validation Feedback**, and **Error Recovery Paths**. - **Destructive Action Safeguards:** Explicit UI confirmation mechanisms for irreversible operations. ### 3. Automated Nielsen Heuristics in Code Reviews (`/review`) When a diff touches frontend code, templates, or styles, the gatekeeper Auditor agent evaluates the changes against **Jakob Nielsen's 10 Usability Heuristics**: - **Visibility of System Status:** Are async operations communicating state via loaders, progress bars, or optimistic UI? - **Error Prevention & Recovery:** Are form fields validated in real-time? Are error messages human-readable, diagnostic, and actionable? - **Cognitive Friction Reduction:** Does the layout minimize decision paralysis? Are primary actions unmistakably distinct from secondary actions? --- ## The 6-Phase Disciplined Development Lifecycle PXOS enforces a deterministic, non-linear operating cycle. AI agents are bounded by these phases and cannot jump ahead without meeting phase completion criteria: ``` ┌────────────┐ ┌────────────┐ ┌────────────┐ │ DISCOVER │ ───► │ PLAN │ ───► │ EXECUTE │ └────────────┘ └────────────┘ └────────────┘ ▲ │ │ ▼ ┌────────────┐ ┌────────────┐ ┌────────────┐ │ COMPACT │ ◄─── │ REVIEW │ ◄─── │ VALIDATE │ └────────────┘ └────────────┘ └────────────┘ ``` 1. **Discover:** The agent investigates existing patterns, active subsystem audits (`.ai/audits/`), and research invariants (`.ai/research/INDEX.md`). *No code is written.* 2. **Plan:** The agent breaks down the scope, checks whether parallel worktrees are beneficial, maps affected files, and explicitly states how known audit findings (e.g. `SEC-01`, `MEM-03`) are resolved. *Requires human confirmation.* 3. **Execute:** The agent implements code incrementally in small, reversible diffs following local conventions. 4. **Validate:** The agent runs test suites, checks build outputs, verifies edge cases, and confirms acceptance criteria. 5. **Review:** The agent acts as an independent auditor, reviewing the git diff against `main` for overengineering, scope creep, security vulnerabilities, and UX heuristics. 6. **Compact Context:** The agent produces an executive session summary, persists architectural decisions (ADRs) into `DECISION_LOG.md`, updates `SPRINT.md`, and flushes context noise. ### Risk-Based Autonomy Matrix To eliminate micromanagement while ensuring total security, PXOS establishes clear autonomy tiers: | Tier | Examples | Execution Rule | | :--- | :--- | :--- | | **Low Risk** | Naming improvements, formatting, obvious bug fixes within scope, unit tests, localized documentation | **Autonomous Execution** without blocking for approval | | **Medium Risk** | Refactoring internal logic, moving files, introducing abstractions, changing component boundaries | **Allowed with explicit reasoning** stated in the prompt | | **High Risk** | Architectural rewrites, new/replaced dependencies, database schema migrations, security/auth logic, breaking API changes | **Strict Human Approval Required** before proceeding | --- ## Multi-Agent Git Worktree Engine PXOS v2.0+ features native architecture for **parallel multi-agent execution**. Instead of multiple AI instances stepping on each other's toes in a single workspace, PXOS leverages **Git Worktrees** to provide physical directory and context isolation: ``` ┌────────────────────────────────────────────────────────┐ │ HUMAN ORCHESTRATOR │ │ Coordinates: SPRINT.md │ └───────────────────────────┬────────────────────────────┘ │ ┌─────────────────────────┴─────────────────────────┐ ▼ ▼ ┌───────────────────────────┐ ┌───────────────────────────┐ │ WORKTREE 1: feat/auth │ │ WORKTREE 2: feat/stripe │ │ Role: Coding Executor │ │ Role: Coding Executor │ │ Model: Fast / Flash ⚡ │ │ Model: Fast / Flash ⚡ │ │ Directory: ../trees/auth│ │ Directory: ../trees/bill│ │ Spec: SPEC-auth.md │ │ Spec: SPEC-stripe.md │ └─────────────┬─────────────┘ └─────────────┬─────────────┘ │ │ └─────────────────────────┬─────────────────────────┘ ▼ ┌───────────────────────────┐ │ REVIEW / PR GATEWAY │ │ Role: Auditor / QA │ │ Model: High-Reasoning 🧠 │ │ Workflow: /review │ └───────────────────────────┘ ``` ### Autonomous Provisioning (Zero Terminal Friction) During the `/plan` workflow, the AI automatically identifies when a feature can be decomposed into independent sub-tasks (e.g. `T-01: feat/auth-oauth` and `T-02: feat/billing-stripe`). Upon human approval, the AI itself invokes the provisioning helper (`scripts/pxos-task.sh` or `scripts/pxos-task.ps1`), creating the isolated branch, physical directory, and modular task spec without manual git commands. --- ## Architecture & Workspace Structure When installed, PXOS introduces a clean, standardized `.ai/` operational layer that works alongside any programming language, framework, or toolchain: ``` your-project/ ├── .ai/ # [Core] PXOS Operational Intelligence │ ├── AI_BASE.md # Universal operating contract & concurrency rules │ ├── PROJECT_CONTEXT.md # Durable project facts, patterns & architecture │ ├── DECISION_LOG.md # Durable Architectural Decision Records (ADRs) │ ├── CURRENT_SPEC.md # Active task specification (Single-agent mode) │ ├── research/ # Macro domain research, UX benchmarks & spikes │ │ └── INDEX.md # Token-efficient research index │ ├── audits/ # Subsystem audits (Security, Memory, Debt) │ │ ├── README.md # Audit taxonomy and severity guidelines │ │ └── PARTIAL_.md # Modular subsystem audit reports │ └── specs/ # Modular specifications (Parallel multi-agent mode) │ ├── TEMPLATE_SPEC.md # Standardized task spec template │ ├── SPEC-auth-oauth.md # Isolated task spec for branch feat/auth-oauth │ └── SPEC-billing-stripe.md # Isolated task spec for branch feat/billing-stripe ├── SPRINT.md # Active sprint & multi-agent task matrix ├── ROADMAP.md # Strategic product roadmap (Human-owned) ├── scripts/ # Automation & DX helpers │ ├── pxos-task.sh # Worktree orchestrator (POSIX Bash) │ └── pxos-task.ps1 # Worktree orchestrator (PowerShell) └── docs/ # Specialized domain knowledge (loaded on demand) ├── DESIGN.md # Visual identity, tokens & interaction patterns ├── DOMAIN_RULES.md # Business logic & compliance invariants ├── DATA_MODEL.md # Entity relationships & database schemas └── GLOSSARY.md # Ubiquitous domain terminology ``` ### Durable Memory Layer (`DECISION_LOG.md`) One of the core innovations of PXOS is its **immutable, append-only Architectural Decision Record (ADR) system**. Whenever an architectural or product tradeoff is decided, the agent autonomously formats and appends it to `DECISION_LOG.md`: ```markdown ### ADR-014: Storing Session State in Redis over In-Memory Map - **Date:** 2026-09-04 - **Author/Agent:** Rodrigo / Agent-Architect - **Context:** Scaling WebSocket connections across multi-instance pods. - **Decision:** Use Redis Pub/Sub and session hashing instead of Node.js local maps. - **Tradeoff / Consequence:** Adds Redis dependency; eliminates memory leak risk on pod reboots. ``` Because `DECISION_LOG.md` is strictly append-only, parallel agents merging their worktrees will never experience merge conflicts on historical decisions. --- ## Workflows & Slash Commands Directory For AI IDEs and tools supporting slash commands (Antigravity, Cursor, Windsurf, Claude Code, continue.dev), PXOS provides saved workflow prompts in [WORKFLOWS.md](./WORKFLOWS.md): | Command | Phase | Core Action & Purpose | | :--- | :--- | :--- | | **`/start`** | Session Init | Auto-resolves current branch and active modular spec; verifies task scope. | | **`/spec`** | Discover/Spec | Drafts comprehensive spec with User Value, UX flows, edge cases, and audit alignment. | | **`/plan`** | Plan | Decomposes task, evaluates worktree parallelization, and checks audit blockers. | | **`/review`** | Review | Audits diff against base for overengineering, security, and Nielsen UX heuristics. | | **`/compact`** | Compact | Closes session, records ADRs to `DECISION_LOG.md`, and updates `SPRINT.md`. | | **`/decision`**| Memory | Immediately registers a durable architectural or product decision into `DECISION_LOG.md`. | | **`/audit`** | Quality Gate | Acts as Principal Auditor inspecting security, performance, memory leaks, and UX debt. | | **`/benchmark`** | Research | Measures token efficiency, rework ratio, and UX completeness; autonomously transmits anonymous research telemetry upon consent. | | **`/install`** | Setup | Automatically detects IDE environment and installs PXOS with appropriate flags. | | **`/update`** | Maintenance | Safely upgrades PXOS to the latest version while preserving all custom project facts. | --- ## Quick Installation & IDE Integration ### One-Line Install Run in the root directory of any project: **macOS / Linux (Bash):** ```bash curl -sSL https://raw.githubusercontent.com/madebypx/PXOS/main/install.sh | bash ``` **Windows (PowerShell):** ```powershell irm https://raw.githubusercontent.com/madebypx/PXOS/main/install.ps1 | iex ``` **Python (PyPI / Global CLI):** ```bash pip install pxos pxos init --ide cursor ``` ### Safe Upgrade for Existing Projects To upgrade an existing project to **PXOS v2.2.0** without touching your project context, active specs, or decision logs: ```bash # Bash curl -sSL https://raw.githubusercontent.com/madebypx/PXOS/main/install.sh | bash -s -- --update # PowerShell & ./install.ps1 -Update # Python CLI pxos update ``` *(Alternatively, run `/update` directly inside your AI assistant).* ### Advanced Installation Options ```bash # Install core + SPRINT.md and ROADMAP.md curl -sSL https://raw.githubusercontent.com/madebypx/PXOS/main/install.sh | bash -s -- --full # Windows PowerShell equivalent & ./install.ps1 -Full -Ide cursor # Configure specific IDE rules in your workspace curl -sSL https://raw.githubusercontent.com/madebypx/PXOS/main/install.sh | bash -s -- --ide cursor curl -sSL https://raw.githubusercontent.com/madebypx/PXOS/main/install.sh | bash -s -- --ide windsurf curl -sSL https://raw.githubusercontent.com/madebypx/PXOS/main/install.sh | bash -s -- --ide claude curl -sSL https://raw.githubusercontent.com/madebypx/PXOS/main/install.sh | bash -s -- --ide gemini curl -sSL https://raw.githubusercontent.com/madebypx/PXOS/main/install.sh | bash -s -- --ide copilot # Install global system-wide rules across all projects curl -sSL https://raw.githubusercontent.com/madebypx/PXOS/main/install.sh | bash -s -- --global --ide cursor ``` ### IDE Rules Compatibility Matrix | IDE / Environment | Mode | Configuration Target | Integration Type | | :--- | :--- | :--- | :--- | | **Cursor** | Workspace / Global | `.cursor/rules/pxos.mdc` | Native Rule File | | **Windsurf** | Workspace | `.windsurf/rules/pxos.md` | Native Cascade Rule | | **Claude Code** | Workspace / Global (`~/.claude/`) | `CLAUDE.md` | Appended / Preserved | | **Gemini CLI** | Workspace / Global (`~/.gemini/`) | `GEMINI.md` | Appended / Preserved | | **GitHub Copilot**| Workspace | `.github/copilot-instructions.md` | Appended / Preserved | --- ## Downstream Adoption Badges If you use PXOS to govern your repository, showcase disciplined AI engineering and link back to the framework by adding an official badge to your `README.md`: ### 1. Governed by PXOS (Standard) ```markdown ``` > Rendered: ### 2. Multi-Agent Git Worktrees ```markdown ``` > Rendered: ### 3. Token Efficiency Certified ```markdown ``` > Rendered: --- ## Standard LLM Ingestion (llms.txt) PXOS strictly adheres to the [`llmstxt.org`](https://llmstxt.org) standard, ensuring AI search agents, IDE plugins, and web crawlers can read the entire operating framework without HTML or web overhead: - **Root Index:** [`https://raw.githubusercontent.com/madebypx/PXOS/main/llms.txt`](https://raw.githubusercontent.com/madebypx/PXOS/main/llms.txt) — Curated index of core rules, slash commands, and benchmarks. - **Full Reference:** [`https://raw.githubusercontent.com/madebypx/PXOS/main/llms-full.txt`](https://raw.githubusercontent.com/madebypx/PXOS/main/llms-full.txt) — Token-dense single-file reference for immediate context injection. - **Generator & Validator:** `python scripts/generate-llms-txt.py --check` --- ## The PXOS Engineering Manifesto 1. **Context is a scarce, high-entropy resource:** Excess context creates hallucination and noise. Load only what is needed for the immediate phase. 2. **Understanding always precedes implementation:** An agent that writes code before understanding the problem produces solutions to the wrong problem. 3. **Simplicity is an engineered outcome, not an accident:** The AI must explicitly evaluate tradeoffs to produce the simplest viable solution. 4. **Autonomy demands strict boundaries:** The Low / Medium / High risk model preserves human strategic control while maximizing AI execution speed. 5. **Product Design & UX are non-negotiable engineering requirements:** Code is useless if the user experience is broken, ambiguous, or visually discordant. 6. **Validation closes the feedback loop:** An unvalidated diff is an incomplete task. The quality bar is defined by verifiable acceptance criteria. --- ## Author & Company Attribution **PXOS** is engineered and governed by **[PROJECT/X](https://madebypx.com)** (domain: `madebypx.com`), designed by **Rodrigo ([@madebypx](https://github.com/madebypx))** as a universal operational layer for high-velocity software engineering, product design, and autonomous AI development. - **Company / Studio:** [PROJECT/X](https://madebypx.com) - **Author & Architect:** Rodrigo ([@rodrigospena](https://github.com/rodrigospena)) - **Repository:** [https://github.com/madebypx/PXOS](https://github.com/madebypx/PXOS) - **Role in Portfolio:** Flagship Open-Source AI Operating Framework utilized across enterprise, mobile, web, and distributed systems to orchestrate autonomous multi-agent teams with zero regression and maximum token economy. --- ## License PXOS is published under the [Apache License, Version 2.0 (Apache-2.0)](./LICENSE) by **PROJECT/X**. Free to use, inspect, adapt, modify, and distribute across personal, commercial, and enterprise software development projects with zero licensing fees. PROJECT/X retains all trademark and brand rights. ================================================================================ ## DOCUMENT: .ai/AI_BASE.md (Universal Operating Rules) # AI Base — Operating Rules This file defines universal operating rules for every AI agent working in this project. Do not modify this file unless a fundamental behavioral rule needs to change. Propagation: if you update this file, update it intentionally across all projects that use PXOS. --- ## Core priorities In order: 1. Correctness — the output must do what was asked, no more, no less 2. Clarity — code and decisions must be easy to read and understand 3. Simplicity — prefer the simplest valid solution 4. Maintainability — optimize for long-term cost, not short-term speed 5. Consistency — follow existing patterns before introducing new ones 6. Context efficiency — minimize token usage without losing precision --- ## Default workflow Every task follows this cycle: ``` Discover → Plan → Execute → Validate → Review → Compact Context ``` - **Discover** — read relevant files, understand existing patterns, identify dependencies and risks. Autonomously inspect active audit reports (`.ai/audits/` or `docs/audits/`) and strategic blueprints/research (`.ai/research/` or `docs/research/`, prioritizing `INDEX.md`) affecting the touched subsystem before declaring understanding. Do not implement yet. - **Plan** — define which files will change, in what order, and why. Explicitly cite and resolve any active audit blockers (P0/P1) or architectural constraints touching the affected files. Surface risks. Confirm the plan before proceeding. - **Execute** — implement incrementally. Keep diffs small. Follow existing conventions. - **Validate** — verify the output works. Check edge cases, regressions, and acceptance criteria. - **Review** — check for overengineering, duplication, scope creep, and inconsistencies. If the task modifies user interfaces or interaction flows, evaluate Nielsen's UX heuristics (status visibility, error prevention, cognitive load). - **Compact Context** — summarize the session. If durable architectural, structural, or product decisions were made or cemented, append an ADR entry directly to `.ai/DECISION_LOG.md`. Update SPRINT.md if it exists. Do not skip phases. Do not implement before the plan is confirmed. --- ## Autonomy rules ### Low risk — allowed without approval - Rename variables or functions for clarity - Fix obvious bugs within the current scope - Improve readability without changing behavior - Add or improve comments and documentation ### Medium risk — require reasoning before proceeding - Alter existing flows or logic - Move or restructure files - Introduce a new abstraction or utility - Change component or module boundaries ### High risk — require explicit human approval - Architectural rewrites or restructuring - Replacing or adding dependencies - Schema or data model changes - Changes that affect multiple unrelated areas - Anything that modifies behavior in production paths --- ## Context management rules - Load only the files relevant to the current task. - Do not scan the entire codebase unless explicitly required. - Summarize findings before continuing to the next phase. - Avoid repeating established context within the same session. - Break large tasks into smaller tasks if context grows noisy. - At the end of long sessions, run `/compact` to reduce future context load. ### Audit & Strategic Grounding Protocol When drafting a task spec (`/spec`) or formulating an implementation plan (`/plan`), the agent MUST cross-reference active audits and strategic blueprints. The agent must never propose code changes that contradict an active audit finding (specifically P0 blockers or P1 critical items) or strategic benchmark invariant without explicit justification. --- ## Multi-Agent & Concurrency Rules When multiple agents operate concurrently on the same repository: 1. **Environment and Branch Isolation:** - Always operate strictly within your assigned `git worktree` directory and branch. - Never switch branches, pull unrelated branches, or modify files belonging to another worktree. 2. **Spec Auto-Resolution:** - Identify your active task automatically: a. Query the current branch name (`git branch --show-current`). b. If a modular spec exists at `.ai/specs/SPEC-.md` or `.ai/specs/SPEC-.md`, load it as your active task spec. c. If working in a single-agent setup or directly on `main`, fall back to `.ai/CURRENT_SPEC.md`. - Never modify, overwrite, or delete a spec assigned to another branch or agent. 3. **Shared File Mutation Etiquette:** - **`PROJECT_CONTEXT.md`**: Strictly read-only during parallel execution. Must not be modified on feature branches. - **`DECISION_LOG.md`**: Append-only. When a durable architectural, technical, or product decision is finalized, agents append an ADR entry directly at the end of `.ai/DECISION_LOG.md` during `/compact` or via `/decision`. Entries are strictly additive and chronological. In the rare event of a git merge conflict at the end of `DECISION_LOG.md`, the resolution rule is always to concatenate both entries without discarding either. - **`SPRINT.md`**: When running `/compact`, only update the row or checklist item corresponding to your specific task ID. Leave other tasks untouched. 4. **Autonomous Task Decomposition & Provisioning:** - During the Plan phase, the agent should evaluate whether a complex goal can be decomposed into independent parallel tasks. - If decomposed, the agent should propose the breakdown in the plan and, upon human approval, autonomously run the worktree provisioning script (`scripts/pxos-task.sh` or `scripts/pxos-task.ps1`), initialize the modular specs, and register tasks in `SPRINT.md` without requiring manual git commands from the developer. 5. **Agent Roles & Token Efficiency:** - **Architect Role (Reasoning):** Focused on `/spec` and `/plan`. Prefers high-reasoning models (e.g. Gemini Pro / Claude Sonnet). - **Executor Role (Coding):** Focused on implementation inside the worktree. Prefers fast, high-throughput, low-cost models (e.g. Gemini Flash). - **Auditor Role (QA & Systems):** Focused on `/review` (diff validation, quality, security, UX heuristics) and autonomous subsystem audits (`/audit`) against quality, security, and memory leak standards. --- ## Quality bar A task is complete when: - All acceptance criteria in the spec are met - No regressions are introduced - The output follows existing conventions - Edge cases are handled or explicitly acknowledged - The diff is as small as it can be while still solving the problem --- ## Behavioral constraints - Never refactor code outside the current task scope - Never introduce abstractions speculatively - Never modify files that are not relevant to the task - Never assume behavior — verify it - Never optimize prematurely - Always explain tradeoffs when making non-obvious decisions - Always prefer existing patterns over new ones - Always ask before making high-risk changes --- ## Phase continuity At the end of any response that completes a workflow phase (Discover, Plan, Execute, Validate, Review, or Compact), explicitly state: 1. Which phase just completed 2. What is still open or unresolved 3. The recommended next step 4. Whether human confirmation is required before proceeding Do not assume phase transitions are implicit. If the user's next message would skip a required phase, do not refuse it mechanically — explain what is missing, state the correct next step, and ask for confirmation only when the risk level requires it. ================================================================================ ## DOCUMENT: WORKFLOWS.md (Workflows & Slash Commands) # PXOS Workflows Workflows are saved prompts that an AI agent can follow on demand. In tools that support slash commands (e.g. Antigravity, Cursor, continue.dev, Claude Code), these can be triggered with `/workflow-name`. This file contains the recommended workflow set for PXOS v2.1. Each workflow maps to a phase of the default operating cycle defined in `AI_BASE.md` and supports both Single-Agent and Multi-Agent parallel environments. --- ## `/install` — Install PXOS in a project **When to use:** When setting up PXOS in a new project. The agent detects the environment, picks the right flags, and runs the installer in one step. Use this instead of manually composing the `curl` command. ``` Install PXOS in this project by following these steps: 1. Detect the environment: - Check if a .cursor/ folder exists → IDE is Cursor - Check if a .windsurf/ folder exists → IDE is Windsurf - Check if a CLAUDE.md file exists, or if you are Claude Code → IDE is claude - Check if a GEMINI.md file exists, or if you are Gemini CLI → IDE is gemini - Check if .github/copilot-instructions.md exists → IDE is copilot - If none of the above → no --ide flag 2. Compose and run the install command: - Always include the base install (no extra flags needed for core files) - Add --ide if an IDE was identified - Add --full only if I explicitly asked for ROADMAP.md and SPRINT.md Examples: No IDE detected: curl -sSL https://raw.githubusercontent.com/madebypx/PXOS/main/install.sh | bash Cursor detected: curl -sSL https://raw.githubusercontent.com/madebypx/PXOS/main/install.sh | bash -s -- --ide cursor Claude Code detected: curl -sSL https://raw.githubusercontent.com/madebypx/PXOS/main/install.sh | bash -s -- --ide claude Full install with Cursor: curl -sSL https://raw.githubusercontent.com/madebypx/PXOS/main/install.sh | bash -s -- --full --ide cursor 3. After the install completes: - Open .ai/PROJECT_CONTEXT.md - Ask me to describe the project so you can help fill it in Do not run the command until you have confirmed which IDE was detected and which flags will be used. ``` --- ## `/update` — Upgrade PXOS to latest version **When to use:** When upgrading an existing PXOS project to the latest version (v2.2.0). Safely updates universal operating rules (`AI_BASE.md`), modular specs template (`TEMPLATE_SPEC.md`), research/audit scaffolding, and IDE rules while strictly preserving custom project facts (`PROJECT_CONTEXT.md`), active specs, and decision logs. ``` Upgrade PXOS in this project by following these steps: 1. Check current PXOS version: - Read the version comment in `.ai/AI_BASE.md` (e.g. ``). - If missing, it indicates a pre-v2.0 installation. 2. Run the safe update command: curl -sSL https://raw.githubusercontent.com/madebypx/PXOS/main/install.sh | bash -s -- --update 3. Summarize what was updated: - Report updated version (v2.2.0). - Confirm that .ai/specs/TEMPLATE_SPEC.md includes Strategic & Audit Alignment. - Confirm that .ai/research/ and .ai/audits/ directories and starter guides are present. - Confirm that PROJECT_CONTEXT.md, DECISION_LOG.md, and all active specs remain untouched. - State that the project is now ready for v2.2.0 workflows (/decision, /audit, /benchmark, UX reviews). ``` --- ## `/start` — Open a work session (Hybrid Single/Multi-Agent) **When to use:** At the beginning of every meaningful AI session. Loads the core context files, automatically discovers active branch and task spec, and establishes operating rules. ``` Read the core context files before doing anything: - .ai/AI_BASE.md - .ai/PROJECT_CONTEXT.md Auto-resolve active task spec: 1. Run `git branch --show-current` to identify the current branch. 2. Check if a modular spec exists matching this branch in `.ai/specs/` (e.g. `.ai/specs/SPEC-.md` or `.ai/specs/SPEC-.md`). 3. If found, load it as the active task spec. 4. If not found, check `.ai/CURRENT_SPEC.md`. 5. If SPRINT.md exists at the project root (or inside `.internal/SPRINT.md` / `.ai/SPRINT.md`), read it to verify assigned goals and dependencies. Report your findings: - Active branch: `[branch-name]` - Active spec: `[spec path]` - Current workflow phase: `[Discover / Plan / Execute / Validate / Review]` Then ask me one question: "Is this session for a new feature or complex task — or a quick fix / small change?" - If new feature or complex task: Check the state of the active spec. - If it is populated with a clear goal and acceptance criteria: Summarize your understanding of the task in 3–5 bullets and confirm you are ready to move to the Plan phase. State what decision or confirmation you need from me before proceeding. - If it is empty or only contains template placeholders: Recommend running /spec to define the task together before planning or executing anything. - If quick fix or small change: Ask me to describe the change in one or two sentences. Confirm scope, identify affected files, and proceed directly to execution following the autonomy rules in AI_BASE.md. Do not implement anything until you have a clear understanding of scope. ``` --- ## `/spec` — Draft a task spec (Modular or Central) **When to use:** After `/start`, when the active spec is empty or when starting a new feature or task. Use this to collaborate on defining scope, constraints, and acceptance criteria. ``` Help me write a task spec using the PXOS standard structure: 1. Determine target file: - If on a dedicated branch/worktree (e.g. `feat/auth-oauth`), create/update `.ai/specs/SPEC-.md` based on `.ai/specs/TEMPLATE_SPEC.md`. - If on `main` or in a single-spec setup, update `.ai/CURRENT_SPEC.md`. 2. Autonomous Grounding (Macro Context): - Autonomously inspect `.ai/audits/` (or `docs/audits/`) for open findings touching the affected files. - Autonomously check `.ai/research/INDEX.md` (or `docs/research/`) for active benchmarks or architectural invariants. - Auto-populate the Strategic & Audit Alignment block with explicit IDs or "Clean — No active audit blockers touching this scope". 3. Fill the following structure: - Goal - User value - Strategic & Audit Alignment (auto-filled by agent) - Scope (in / out) - Constraints - Existing patterns relevant to the task - Proposed change - User flow / Technical flow - Edge cases - Acceptance criteria - Validation plan - Risks & Cross-Task Dependencies Ask me questions if anything is unclear before filling it in. After the spec is complete, explicitly state: - That the spec file is ready - That the recommended next step is to run /plan ``` --- ## `/plan` — Request a plan before execution **When to use:** When starting a complex or risky task and you want to explicitly force the planning phase before any code is written. ``` Before writing any code, produce a plan based on the active spec that includes: 1. Task Scope & Decomposition: - Determine if this is a Single-Task (monolithic implementation) or if it can be decoupled into 2+ independent parallel tasks. - If parallelizable: Outline the sub-tasks (e.g. T-01 on `feat/auth`, T-02 on `feat/billing`) and propose creating dedicated worktrees automatically. 2. Technical Implementation Plan: - What you understood from the task - Which files will be affected (ensuring no unauthorized shared file mutations) - What will change and why - Audit & Invariant Cross-Check: Verify that proposed file edits directly account for known audit risks, cite audit IDs being resolved (e.g. SEC-01, MEM-03), and prevent reintroducing previously audited defects. - Main risks and cross-task dependencies - How success will be validated After presenting the plan, explicitly state: - What decision or confirmation you need from me before executing - What will NOT happen until I approve - (If parallel tasks were proposed): Confirm that upon approval, you will automatically execute the worktree provisioning script (`./scripts/pxos-task.sh` or `.\scripts\pxos-task.ps1`), initialize specs, and register them in SPRINT.md. Wait for my approval before executing. ``` --- ## `/review` — Review what was just implemented **When to use:** After a task or feature has been executed. Use this to run a structured quality check on the branch diff before merging or opening a PR. ``` Review what was just implemented by checking the branch diff (e.g. against main/origin): - Overengineering or unnecessary complexity - Duplicate logic or conflicts with existing patterns - Accidental scope growth (modifying files outside the task scope) - Unauthorized edits to shared files (e.g. PROJECT_CONTEXT.md) - Weak naming or missing error boundaries - Hidden side effects or security risks - Inconsistencies with existing patterns UX & Interaction Heuristics (If the diff modifies UI, styles, or frontend interactions): - Status visibility: Are loading states, transitions, and system feedback clear and responsive? - Error prevention: Are destructive actions guarded with confirmation? Are validation errors clear? - Cognitive friction: Is the user flow direct and intuitive, minimizing unnecessary clicks or confusion? If a simpler valid solution exists, point it out. Do not refactor without my approval. End the review with one of three classifications: - ✅ Ready to validate — all criteria met, no blockers - 🔄 Needs revision — describe what must change before validation - ⏸ Blocked — describe what decision or information is needed Then state the recommended next step. ``` --- ## `/compact` — Close a session and compact context **When to use:** At the end of a long session, before switching tasks, or before submitting a PR. Produces a structured session summary, automatically records architectural decisions (ADRs), and atomically updates task tracking. ``` Summarize this session in a compact format: ## What changed ## Important decisions ## Open issues ## Next steps ## Relevant files or systems Keep it short, factual, and reusable as future context. In the "Next steps" section, distinguish between: - Immediate next action (what should happen first in the next session) - Pending human decision (what I need to decide before work resumes) - Future follow-up (non-urgent items that can wait) Update workflow state & durable memory: 1. Update the "Workflow state" section inside the active task spec (`.ai/specs/SPEC-*.md` or `CURRENT_SPEC.md`). 2. If SPRINT.md exists at the project root (or inside `.internal/SPRINT.md` / `.ai/SPRINT.md`): - Locate the row or item corresponding to this task/branch in the Task Matrix. - Update its Status column (e.g., "In Plan", "Executing", "Ready for PR", "Done"). - Leave other tasks and agents untouched. 3. Architectural & Product Decision Check (Autonomous ADR): - Did this session make or cement any durable architectural, structural, or product decisions (e.g. library choices, API/IPC protocols, rejected alternatives)? - If yes: Autonomously format an ADR entry and append it directly to the end of `.ai/DECISION_LOG.md`. Note the added ADR in the summary. If SPRINT.md does not exist but this project has a sprint in progress, ask me if I want to create it. ``` --- ## `/decision` — Record an architectural or product decision (ADR) **When to use:** When you or the agent make a lasting decision during conversation that should be durably recorded immediately without waiting for `/compact`. ``` Record an architectural or product decision into .ai/DECISION_LOG.md: 1. Capture context: - What was decided? - What alternatives were considered and why were they rejected? - What are the key tradeoffs and impacts? 2. Format as a standard ADR (following the schema in .ai/DECISION_LOG.md). 3. Append directly to the end of .ai/DECISION_LOG.md. 4. Confirm to me that the decision has been logged and summarize the core invariant. ``` --- ## `/audit` — Autonomous codebase or subsystem audit **When to use:** Before major releases, after refactors, or to diagnose technical debt, memory leaks, and security posture across subsystems. ``` Act as a Principal Auditor to inspect codebase subsystems against quality, security, and memory standards: 1. Scope & Context Window Protection: - Audit modularly by subsystem (e.g. /audit auth, /audit audio, /audit ipc) to prevent context exhaustion. - For a full audit, inspect subsystems sequentially and synthesize a consolidated report. 2. Inspect across core dimensions: - Security & Auth (secrets, injection, sanitization, exposed APIs) - Performance & Memory (leaks, unclosed handles, event listeners, main-thread blocking) - Architecture & Invariants (layering, circular imports, adherence to PROJECT_CONTEXT.md) - Reliability & Errors (swallowed exceptions, unhandled promises, missing error boundaries) - UX & Interaction (if UI: status visibility, destructive actions guarded) 3. Format structured findings with IDs, severity (P0 Blocker, P1 Critical, P2 Major, P3 Minor), location, evidence, and remediation. 4. Output report into .ai/audits/PARTIAL_.md or .ai/audits/AUDIT_YYYY-MM-DD.md. 5. Provide an executive summary with count by severity and immediate blockers. ``` --- ## `/benchmark` — Empirical performance & telemetry audit **When to use:** After completing a feature or session. Measures token efficiency, code rework, and UX state completeness, saves empirical JSON telemetry cleanly within `.ai/audits/` (zero project root pollution), and autonomously contributes anonymous metrics to the research study upon user consent. ``` Audit development performance on recent tasks: 1. Extract Telemetry: - Calculate total turns, prompt tokens, completion tokens, and framework overhead. - Calculate rework ratio (lines modified or deleted after initial diff). - Evaluate UX completeness (Universal 5-State Matrix: idle, loading, empty, error, destructive guard). - Check architectural fidelity (ADR retention, duplicate utilities prevented). 2. Clean Storage (Zero Project Root Pollution): - In consumer projects, save structured JSON strictly inside: .ai/audits/BENCHMARK_.json - Never create a root benchmarks/ folder in consumer repositories. 3. Interactive Consent & Autonomous Dispatch: - Present the sanitized preview of numerical metrics to the human. - Ask for explicit consent to transmit anonymous telemetry. - If approved, autonomously POST the JSON payload to https://telemetry.madebypx.com/api/v1/telemetry. - Report the resulting submission ID back to the user. ``` ### Monitoring Telemetry Health & Public Submissions To verify server stability and monitor community benchmark submissions in real time: ```bash # Live terminal dashboard (snapshot): python scripts/pxos-telemetry-monitor.py # Continuous watch mode (auto-refreshing every 15s): python scripts/pxos-telemetry-monitor.py --watch # Machine-readable JSON output for CI / dashboards: python scripts/pxos-telemetry-monitor.py --json # Latency threshold check (fails with exit code 1 if ping > 300ms): python scripts/pxos-telemetry-monitor.py --max-latency-ms 300 ``` --- ## Usage notes - These workflows cover the complete PXOS operating cycle: `/install` → `/start` → `/spec` → `/plan` → execute → `/review` → `/compact`, plus on-demand macro tools (`/decision`, `/audit`). - In multi-agent parallel environments, each agent runs `/start` in its own worktree and automatically targets its own isolated spec. - `DECISION_LOG.md` entries are strictly append-only; in the rare event of a git merge conflict, concatenate entries without discarding either. - Do not create workflows for individual feature types or component patterns — that becomes a prompt library, which contradicts the PXOS principle of keeping the system minimal. ================================================================================ ## DOCUMENT: benchmarks/REPORT.md (Empirical Benchmark Report) # PXOS Empirical Benchmark & Audit Report **Sample Size:** 4 evaluated tasks across AI agents. **Overhead Justification Rate:** 75.0% of tasks reported net positive utility. ## 1. Quantitative Telemetry & Token Efficiency | Metric | Mean | Median | Std Dev | Min | Max | | :--- | :--- | :--- | :--- | :--- | :--- | | **Total Tokens** | 133,725.0 | 57,100.0 | 178,741.2 | 20,300.0 | 400,400.0 | | **Input Tokens** | 114,675.0 | 49,250.0 | 152,354.92 | 18,200.0 | 342,000.0 | | **Output Tokens** | 19,050.0 | 7,850.0 | 26,386.93 | 2,100.0 | 58,400.0 | | **PXOS Overhead Tokens** | 8,900.0 | 5,000.0 | 9,633.97 | 2,400.0 | 23,200.0 | - **Total Framework Overhead:** 6.66% of total token consumption was dedicated to `.ai/` operational artifacts (`SPEC`, `PLAN`, `COMPACT`). ## 2. Code Churn & Rework Reduction - **Mean Rework Ratio:** 3.37% (lines modified or discarded after initial diff). - **Mean Corrective Turns:** 1.0 turns before PR readiness. - **Session Amnesia Incidents:** 0.0% of sessions. - **Duplicate Utilities Prevented/Introduced:** 0 duplicate helpers registered. - **Audit Findings Formally Resolved:** 8 items (e.g. security/memory fixes). ## 3. Product Design & UX State Coverage - **UI Tasks Evaluated:** 3 - **Mean UX State Completeness:** 93.0% (Evaluating Initial, Loading, Empty, Error, and Destructive confirmation branches). ## 4. Complexity Tier Breakdown | Complexity Tier | Tasks ($N$) | Mean Tokens | Mean Overhead | Overhead % | Mean Rework % | | :--- | :--- | :--- | :--- | :--- | :--- | | **tier_2_medium** | 1 | 49,300.0 | 4,200.0 | 8.5% | 6.4% | | **tier_3_complex** | 2 | 232,650.0 | 14,500.0 | 6.2% | 3.5% | | **tier_1_micro** | 1 | 20,300.0 | 2,400.0 | 11.8% | 0.0% | > [!NOTE] > **Overhead Inflection Insight:** In Tier 1 (micro tasks), specification scaffolding represents a higher token proportion. In Tier 2 and Tier 3 tasks, the overhead drops significantly while rework savings scale exponentially. ## 5. Cost Analysis (USD) | Model Pricing Profile | Total Expenditure (USD) | Mean Cost per Task | | :--- | :--- | :--- | | **claude-3-5-sonnet** | $2.5191 | $0.6298 | | **gemini-2-flash** | $0.0764 | $0.0191 | | **gemini-1-5-pro** | $0.9544 | $0.2386 | | **gpt-4o** | $1.9087 | $0.4772 | | **default-blended** | $1.1452 | $0.2863 | ## 6. Critical Qualitative Synthesis ### Identified Friction Points & Overhead Critiques: - Filling the Strategic & Audit Alignment block in SPEC-auth.md took an extra prompt turn when no audits were directly touching auth. - Had to run pxos-task.sh to set up the worktree tree, which required verifying bash permissions on the developer machine. - Running the full /spec and /plan sequence for a 20-line CSS change produced 2,400 tokens of documentation overhead (11.8% of total tokens), which felt excessive for a simple aesthetic adjustment. - The Low-Risk autonomy rule in AI_BASE.md should have allowed moving directly to execution without full spec scaffolding. - Maintaining separate SPEC files for all 7 milestones (.ai/specs/SPEC-*.md) created 23,200 tokens of documentation overhead, requiring explicit workflow state tracking turns. - Node.js native ESM execution without a runtime loader failed on test scripts until bundled via esbuild in package.json, which was not caught during initial script authoring. ### Concrete Empirical Benefits Cited: - Mandatory error state in spec forced the implementation of OAuth callback timeout handling in auth/callback.ts#L48-L62, avoiding an uncaught promise rejection in production. - Append-only DECISION_LOG.md preserved the cookie encryption strategy for future sessions. - Worktree isolation allowed parallel work with feat/auth without touching shared package.json or src/index.ts until final merge. - The Plan phase explicitly checked audit SEC-01 (raw body parsing requirement for Stripe webhooks), preventing a signature verification failure that typically requires multiple debugging turns. - Consulting docs/DESIGN.md ensured the button used the canonical --radius-md variable instead of hardcoding 6px. - Strict Zero-Spoiler invariant specified in .ai/CURRENT_SPEC.md and enforced in src/shared/utils/sanitizer.ts completely prevented confidential narrative leaks to the Player View window. - The append-only DECISION_LOG.md entry for 2D Sweep-Line Raycasting prevented subsequent turns from attempting battery-draining WebGL fragment shaders. - Pre-specifying the 4-sub-bus Web Audio routing graph in Milestone 2 allowed Milestone 5 (3D Dice) to seamlessly hook collision SFX directly into AudioEngine without touching existing channel faders. ================================================================================ ## DOCUMENT: benchmarks/METHODOLOGY.md (Benchmark Methodology & Formulas) # Empirical Benchmark Methodology for PXOS ## Abstract This document outlines the formal empirical methodology designed to scientifically evaluate the quantitative and qualitative performance impact of **PXOS (Product & eXperience Operating System)** in AI-assisted software engineering. The objective of this methodology is to eliminate speculation, marketing hyperbole, and LLM sycophancy, establishing an **incontestable, reproducible, and mathematically grounded evaluation framework** for assessing: 1. Context token economy and cost efficiency. 2. Architectural stability and regression reduction. 3. User experience (UX) and interaction state completeness. 4. Concurrency scaling and merge collision frequency in multi-agent workflows. --- ## 1. Research Questions (RQs) The benchmark framework is structured around five core empirical questions: - **RQ1 (Token Economy):** *To what extent does document-driven asynchronous artifact passing reduce total token consumption compared to unconstrained conversational memory across identical feature implementations?* - **RQ2 (Cost & Model Routing):** *How does the 80/20 model specialization tier (routing high-throughput execution to lightweight models and strategic specs/reviews to reasoning models) impact total monetary cost per delivered feature?* - **RQ3 (Architectural Stability & Rework):** *Does the mandatory Discover → Plan operational gate significantly reduce code churn, duplicate utilities, and regression of pre-identified audit findings?* - **RQ4 (UX & Interaction Quality):** *Does enforcing explicit state modeling (empty, loading, error, destructive guards) and Nielsen Norman heuristics during specification and review measurably increase front-end completeness?* - **RQ5 (Concurrency & Multi-Agent Isolation):** *How does physical Git Worktree isolation compare to shared-branch multi-agent execution regarding file collisions, merge conflicts, and time-to-PR?* --- ## 2. Experimental Design ### 2.1 Subject Groups Every empirical trial evaluates a paired comparison between two groups across identical initial repository states: | Group | Identifier | Configuration Description | | :--- | :--- | :--- | | **Control Group** | **$C$ (Vanilla / Unconstrained)** | Standard AI assistant interaction without PXOS. The agent operates via free-form conversational prompting, continuous chat history, unguided architectural planning, and direct file editing on a single shared branch. | | **Treatment Group** | **$T$ (PXOS Operating System)** | AI assistant operating strictly under the PXOS contract (`AI_BASE.md`, `PROJECT_CONTEXT.md`, `.ai/specs/`, `DECISION_LOG.md`, `.ai/audits/`, and Git Worktrees). Follows the 6-phase lifecycle: Discover → Plan → Execute → Validate → Review → Compact. | ### 2.2 Task Stratification & Complexity Tiers Tasks must be categorized across three complexity tiers to identify the **Overhead Inflection Point** (the task size where framework overhead becomes net-positive): 1. **Tier 1 — Micro Tasks (Small):** - *Scope:* Isolated bug fixes, CSS tweaks, single-file utility additions. - *LOC Changed:* $< 50$ lines. - *Expected Finding:* Treatment group may exhibit token overhead due to specification scaffolding. 2. **Tier 2 — Subsystem Features (Medium):** - *Scope:* New API endpoints, database migration + repository layer, interactive UI component with async state. - *LOC Changed:* $50 - 300$ lines. - *Expected Finding:* Treatment group breaks even on token usage while achieving superior architectural alignment. 3. **Tier 3 — Complex Systems & Refactors (Large / Multi-Session):** - *Scope:* Authentication subsystem, billing integration, state management refactor, multi-agent parallel workflows. - *LOC Changed:* $> 300$ lines across multiple modules. - *Expected Finding:* Treatment group exhibits drastic token reduction, zero session amnesia, and zero merge collisions. --- ## 3. Quantitative Metrics & Mathematical Formulations ### 3.1 Token Consumption & Efficiency Delta ($\Delta T$) Let $T_{in}$ represent prompt input tokens, $T_{out}$ represent generated completion tokens, and $T_{total} = T_{in} + T_{out}$ across all turns $k \in [1, K]$ in a task session. $$\Delta T = \left( \frac{T_{total}(C) - T_{total}(T)}{T_{total}(C)} \right) \times 100\%$$ - A positive $\Delta T$ indicates percentage token savings achieved by PXOS. - A negative $\Delta T$ indicates framework overhead. ### 3.2 Monetary Cost Function ($Cost$) Monetary cost is computed using standardized provider pricing tiers per million tokens: $$Cost = \sum_{m \in M} \left( \frac{T_{in}^{(m)} \times P_{in}^{(m)}}{10^6} + \frac{T_{out}^{(m)} \times P_{out}^{(m)}}{10^6} \right)$$ Where: - $m$ denotes the model class (e.g., High-Reasoning / Pro vs High-Throughput / Flash). - $P_{in}^{(m)}, P_{out}^{(m)}$ are the published price per $10^6$ tokens for model $m$. - Under PXOS, $T$ utilizes $m_{reasoning}$ for $\approx 10\%-15\%$ of tokens (`/spec`, `/review`), and $m_{throughput}$ for $\approx 85\%-90\%$ of tokens (`execute`, `tests`). Group $C$ typically runs a single monolithic model for all turns. ### 3.3 Code Churn & Rework Ratio ($R_{rework}$) Rework measures the efficiency of the implementation phase by calculating lines modified or deleted in subsequent corrective turns after the initial implementation attempt: $$R_{rework} = \left( \frac{L_{modified} + L_{deleted}}{L_{added\_initial}} \right) \times 100\%$$ Where: - $L_{added\_initial}$ is the total lines of code introduced in the agent's first implementation diff. - $L_{modified} + L_{deleted}$ are the corrective line edits forced by compilation errors, broken tests, or missed requirements before PR readiness. ### 3.4 Decision Retention & Drift Index ($D_{drift}$) To measure architectural amnesia across multi-session tasks, we define the Drift Index: $$D_{drift} = \frac{N_{violated\_invariants} + N_{duplicate\_utilities}}{N_{historical\_decisions}}$$ Where: - $N_{historical\_decisions}$ is the count of established ADRs in `DECISION_LOG.md` or prior session agreements. - $N_{violated\_invariants}$ is the count of regressions violating established architectural boundaries. - $N_{duplicate\_utilities}$ is the count of redundant helper functions created because the agent forgot existing utils. ### 3.5 UX State Completeness Index ($S_{ux}$) Front-end components must be evaluated against the **Universal 5-State Matrix**: $$S_{ux} = \frac{1}{5} \sum_{i=1}^{5} w_i \cdot s_i, \quad s_i \in \{0, 1\}$$ Where states $s_1 \dots s_5$ represent: 1. $s_1$ (**Initial / Idle State**): Base rendering. 2. $s_2$ (**Loading / Pending State**): Skeletons, spinners, or optimistic feedback. 3. $s_3$ (**Empty State**): Clear visual and textual cue when zero data is available. 4. $s_4$ (**Error / Recovery State**): Actionable error messages and retry mechanisms. 5. $s_5$ (**Destructive Action Guard**): Confirmation dialogs or undo mechanisms for high-impact actions. Weights $w_i = 1.0$ unless a state is demonstrably non-applicable (e.g., non-destructive flows omit $s_5$, renormalizing to $\frac{1}{4}$). ### 3.6 Framework Overhead-to-Value Ratio ($OVR$) The metric determining whether PXOS overhead is justified: $$OVR = \frac{T_{overhead}}{T_{saved\_rework}}$$ Where: - $T_{overhead}$ is the token cost of generating and maintaining `.ai/` documentation (`SPEC`, `PLAN`, `COMPACT`). - $T_{saved\_rework}$ is the estimated token cost avoided by preventing correction turns and architectural re-writes. - An $OVR < 1.0$ indicates that PXOS was mathematically net-beneficial. --- ## 4. Qualitative Heuristics & Audit Taxonomy ### 4.1 Nielsen Norman Usability Heuristics Compliance Every UI diff is evaluated against Jakob Nielsen's 10 Usability Heuristics, specifically: - **H1: Visibility of system status:** Timely, appropriate feedback. - **H5: Error prevention:** Careful design preventing slip-ups and destructive errors. - **H6: Recognition rather than recall:** Reduced cognitive load; options and actions visible. - **H8: Aesthetic and minimalist design:** Zero ad-hoc styles, strict adherence to design system tokens. ### 4.2 Architectural Violation Taxonomy Audit findings are classified according to severity: - **P0 (Blocker):** Security flaw, memory leak, unhandled race condition. - **P1 (Critical):** Broken invariant, severe regression, architectural layering violation. - **P2 (Major):** Technical debt, code duplication, missing unit test coverage. - **P3 (Minor):** Non-standard naming, stylistic inconsistency. --- ## 5. Threats to Validity & Mitigation | Threat Category | Potential Risk | Scientific Mitigation Strategy | | :--- | :--- | :--- | | **LLM Sycophancy** | Agents praise PXOS because they are asked about it. | **Blind Evaluation:** Reviewer agents are presented with anonymized diffs without knowing which framework was used.
**Adversarial Elicitation:** The interview protocol explicitly instructs the agent to audit inefficiencies, friction, and failures. | | **Non-Determinism** | LLM temperature causing variance between runs. | Set temperature to $0.0$ (or minimum allowed) for evaluation tasks. Run all trials $N \ge 3$ times and report mean ($\mu$) and standard deviation ($\sigma$). | | **Task Selection Bias** | Choosing only tasks that favor structured planning. | Include micro-tasks (where PXOS is expected to have overhead) alongside complex refactors. | | **Telemetry Inaccuracy** | Hallucinated token counts. | Ingest raw IDE telemetry logs or direct API billing responses rather than relying on self-reported agent estimates whenever possible. | --- ## 6. Reproducibility Protocol To reproduce any benchmark: 1. Clone the target repository at the designated baseline commit hash. 2. Ensure identical tooling configurations (Node/Python runtime, linters, test harnesses). 3. Execute the controlled prompts specified in `AGENT_INTERVIEW_PROTOCOL.md`. 4. Store all raw output JSONs in `benchmarks/data/`. 5. Run `python benchmarks/analyze.py --input benchmarks/data/` to generate the consolidated report. ================================================================================