{ "fixture_version": 2, "slug": "local-agent-turn-time-budget-2026", "purpose": "Three prompts of different length so prefill cost is visible separately from decode cost. Each prompt asks for a short, deterministic reply (num_predict 48) so decode work is held roughly constant across fixtures.", "notes": "target_words is the whitespace-split word count of the prompt string exactly as published here, measured 2026-11-08; it is metadata for a human reader only. The harness never reads it: prompt tokens come from the server's prompt_eval_count field, so a different tokenizer changes the ratio but not the method. An earlier draft of this file carried planning targets (35/260/900) that the prompts never met; those were replaced with the realized counts and the three distinct lengths were kept.", "fixtures": [ { "name": "short", "target_words": 23, "prompt": "Name the three fields in an Ollama /api/generate response that report prompt processing and token generation timing. Answer in one sentence per field." }, { "name": "medium", "target_words": 142, "prompt": "You are reviewing a Python worker that calls a local model over HTTP. It reads a JSON job file, validates four required keys (role, slug, tree, stage), rejects unknown keys with a message naming the offending key, writes a run manifest with a sha256 of each artifact, and retries a failed HTTP POST twice with a one second pause. Read the following requirements and list, as a numbered list, every place where the described worker would fail a reviewer who checks exact evidence binding. Requirements: (1) the manifest must bind each artifact by hash; (2) a failed POST must not silently downgrade to success; (3) unknown metadata keys must be reported, not dropped; (4) the tree path must never be hard-coded; (5) the worker must stop after two attempts and record the exact error text. Keep each numbered item under twenty words." }, { "name": "long", "target_words": 236, "prompt": "You are auditing a coding-agent turn budget for a local model served on CPU. The client sends one request per tool-call turn: a growing conversation prompt, a bounded output, then a tool result appended for the next turn. The server reports prompt_eval_count, prompt_eval_duration, eval_count and eval_duration per request, all in nanoseconds for the durations. Assume the client has a fixed per-request timeout and retries twice on timeout. Audit the following claims and answer each with AGREE or DISAGREE plus one sentence: (1) a single headline tokens-per-second figure is sufficient to predict turn time; (2) prompt processing and token generation can be estimated from one measured rate because both scale linearly with token count; (3) a thirty-turn task costs thirty times the first turn, regardless of how the prompt grows; (4) a decode rate measured once on a shared host is a stable property of that host; (5) if prefill alone approaches the client timeout, shortening the output is the first fix to try; (6) the minimum decode rate that fits a timeout depends on the prompt length as well as the output length; (7) a run whose repeats disagree by more than ten percent should still be reported as a baseline if the median looks plausible. After the seven answers, state which two measurements a reader must take before any of these numbers mean anything for their own machine. Keep the whole answer under 250 words." } ] }