--- name: agentsop-code-execution-decision version: 0.1.0 description: >- Decision rubric for when an LM agent should write-and-run code (Program-of-Thought / code interpreter) versus reason in natural language: classify each step as deterministic- computable (emit + execute code, feed the result back) vs judgment (stay in prose). Use when designing or debugging an agent step that does arithmetic/parsing/data transforms, when prose reasoning hallucinates a computation (under-coding), or when a sandbox round- trip is wasted on a judgment task (over-coding). Search keywords: code interpreter, agent does math wrong, calculator hallucination, when to run code vs reason, program of thought, PoT, tool vs reasoning. domain: | Deciding when an LM agent should write-and-run code (Program-of-Thought / code interpreter) versus reason in natural language. An enhancement overlay on top of DSPy's ProgramOfThought module, which ships the mechanism but not the decision rubric. type: enhancement-overlay source: | DSPy ProgramOfThought module + docs [dspy.ai/learn/programming/modules/]; "Program of Thoughts" (Chen et al., arXiv 2211.12588); PAL: Program-aided Language Models (Gao et al., arXiv 2211.10435); OpenAI Code Interpreter / Assistants code-tool docs; Anthropic code-execution tool docs; LangChain PythonREPLTool; sibling skill output-format-by-model (PoT for math/parse content shape). audience: | Coder-agent authors, tool-use harness designers, RAG/eval pipeline builders deciding whether a step should emit executable code or stay in prose. status: tool-skill --- # Code-Execution Decision — emit-code-vs-reason > **One-liner**: LMs are unreliable calculators but reliable coders. When the answer needs > determinism and precision — arithmetic, exact data manipulation, deterministic transforms — > emit code and run it. When the answer needs judgment, taste, or open-ended synthesis, reason > in natural language. The cost of getting this gate wrong is silent: prose arithmetic > *hallucinates a plausible-looking wrong number*, and over-coding a judgment task burns a > sandbox round-trip for nothing. This is an **enhancement overlay**. DSPy already gives you `dspy.ProgramOfThought` (PoT) — the *mechanism* for write-then-execute. What it does not give you is the *decision rubric* for when to reach for it. That rubric is this skill. Cross-link the sibling [[agentsop-output-format-by-model]] (which decides *how* code-shaped content should be serialized) and [[agentsop-test-fix-loop]] (which closes the execute → error → retry loop). --- ## 1. 何时激活 (When to activate) Activate this skill **before** committing a step to a reasoning strategy whenever the task has a **verifiable, deterministic core** — or whenever you catch an agent doing arithmetic in prose. | Trigger | Signal | |---|---| | Arithmetic / math | "compute the compound interest", "what's 17.5% of $4,392.18", multi-step word problems, unit conversions, date deltas | | Precise data manipulation | "sort these 240 rows by the third column", "dedupe and count", "join these two lists on id", "parse this CSV and sum column B" | | Deterministic transforms | regex extraction, string reformatting, base conversion, hashing, sorting, set operations | | Symbolic / combinatorial | "how many distinct permutations", "solve this system of equations", calendar/scheduling math | | You see a model doing math in prose | "Let me add: 1,204 + 8,991 + ... = 10,195" — almost always worth a code check | | Choosing a DSPy module | deciding between `ChainOfThought` and `ProgramOfThought` for a signature [dspy.ai/learn/programming/modules/] | **Anti-triggers** (do NOT reach for code execution): - The task is **judgment**: tone, summarization, ranking by quality, "is this reply empathetic?", design tradeoffs, open-ended explanation. - The "computation" is trivial and within the model's reliable range (single-digit arithmetic, a 3-item count) — the sandbox round-trip costs more than it saves. - No sandbox is available and the determinism requirement is soft. - The output's *consumer* is a human reading prose, and an approximate answer is acceptable. --- ## 2. 核心心智模型 (Core mental model) > **LMs are unreliable calculators but reliable coders.** A language model predicts the *next token*, not the *correct value*. When you ask it to add `48,217 + 9,884` in prose, it emits the most *plausible-looking* digit sequence — which is frequently wrong, and wrong in a way that looks right. The same model can write `48217 + 9884` as a Python expression flawlessly, because emitting the *program* is a pattern-matching task it is genuinely good at, and the *Python interpreter* is a deterministic oracle. This decoupling — model writes the recipe, interpreter computes the result — is the entire thesis of Program-of-Thought (PoT) [arXiv 2211.12588] and PAL [arXiv 2211.10435]. ``` ┌────────────────────────────────────────────────────────┐ │ THE GATE: does this answer need determinism/precision? │ └────────────────────────────────────────────────────────┘ │ │ YES (computable) NO (judgment) │ │ ▼ ▼ ┌─────────────────────┐ ┌─────────────────────┐ │ EMIT CODE │ │ REASON IN PROSE │ │ model writes recipe │ │ model is the engine │ │ interpreter = oracle│ │ no oracle exists │ └─────────────────────┘ └─────────────────────┘ │ ▼ sandbox → run → feed result back into LM → LM narrates/uses it ``` **Two failure modes the gate prevents:** | Failure | Mechanism | Symptom | |---|---|---| | **Under-coding** (reason when you should compute) | LM hallucinates a calculation it cannot reliably perform | A confident, wrong, plausible-looking number; off-by-one counts; arithmetic that "looks" right | | **Over-coding** (compute when you should reason) | LM wraps a judgment task in a sandbox round-trip that adds no determinism | Wasted latency + cost; brittle code that encodes a subjective rubric as if it were a formula; `print("the tone is friendly")` | **Why the asymmetry matters.** Under-coding fails *silently* — the wrong number propagates downstream and no exception fires. Over-coding fails *loudly and cheaply* — you notice the useless sandbox call. So the default lean, when genuinely uncertain and a verifiable core exists, is **toward code**. But "genuinely uncertain" is the operative phrase: most judgment tasks are not close calls. The **format corollary** (from [[agentsop-output-format-by-model]]): once you decide to emit code, emit it as *code in a fenced block / single string field* — never nested inside JSON sub-structure. Code-in-JSON measurably degrades the code itself (Aider: 61%→20% on GPT-4 Turbo). The execution decision and the serialization decision are two separate gates; pass both. --- ## 3. SOP 工作流 (Decision workflow) A three-step gate. Run it per *step*, not per *task* — one task can have computable steps and judgment steps interleaved. ### Step 1 — Classify the step: deterministic-computable vs judgment Ask: **"Is there a single correct answer that a program could verify?"** - **Yes → computable.** Arithmetic, sorting, parsing, counting, set/regex ops, symbolic math, anything with a checkable ground truth. → Step 2. - **No → judgment.** Quality ranking, tone, summarization, design choice, open synthesis. → reason in prose. Stop here. - **Mixed → decompose.** "Summarize these sales and tell me the total" = a judgment summary step + a computable total step. Route each independently. The total goes to code; the summary stays prose. A useful tiebreaker for the "trivial computable" gray zone: **if the model would be embarrassed to get it wrong and you'd reach for a calculator yourself, emit code.** If you'd do it in your head without a second thought, prose is fine. ### Step 2 — If computable, emit + run code 1. Emit the computation as a real program (Python is the PoT default; the interpreter is the oracle). 2. Keep the code **minimal and side-effect-free** for the computation — read inputs, compute, `print`/return the result. No network, no filesystem unless the task *is* I/O. 3. Serialize the code per [[agentsop-output-format-by-model]]: fenced block or single string field, **not** JSON-nested. ### Step 3 — Sandbox → run → feed result back into the LM 1. **Sandbox choice** (see §4 op #3, cross-link [[agentsop-output-format-by-model]] for the wire shape): DSPy `ProgramOfThought` uses a Python interpreter (Deno/PythonInterpreter sandbox in recent versions); OpenAI Code Interpreter runs in a managed container; Anthropic code-execution tool runs in a sandboxed VM; LangChain `PythonREPLTool` runs **in-process and is unsandboxed** — treat as untrusted-input-hostile. 2. **Run** the code in the sandbox; capture stdout / return value / traceback. 3. **Feed the result back into the LM** — this is the step under-coders forget. The code produces a *value*; the LM must consume it to narrate, format, or chain into the next step. PoT is "code computes, LM contextualizes," not "code replaces the LM." 4. **Error → retry** (cross-link [[agentsop-test-fix-loop]]): on traceback, feed the error back to the LM, regenerate the code, re-run. Bound the retries (DSPy `ProgramOfThought` defaults to `max_iters` ≈ 3). After the bound, fall back to prose reasoning or surface the failure — do not loop forever. **Exit criterion:** the step produces either (a) a code-derived value the LM has consumed, or (b) a prose judgment, with the gate decision recorded so a reviewer can audit *why* code was or wasn't used. --- ## 4. 操作模型 (Operations: Trigger → Action → Output → Evidence) Seven operations. Each row is a reusable move. | # | Op | Trigger | Action | Output | Evidence | |---|---|---|---|---|---| | 1 | **Computable-vs-judgment gate** | Any step about to be reasoned | Apply §3 Step 1: single verifiable answer? | Route to code or prose | PoT premise: decouple compute from reasoning [arXiv 2211.12588] | | 2 | **Decompose mixed steps** | Step has both a number and a narrative | Split into computable sub-step (code) + judgment sub-step (prose) | Two routed sub-steps | Mirrors mixed-content split in [[agentsop-output-format-by-model]] §5 Case B | | 3 | **Sandbox choice** | Decided to emit code | Pick interpreter by trust + capability: DSPy PoT (Python sandbox), OpenAI Code Interpreter (managed container), Anthropic code-exec (VM), LangChain `PythonREPLTool` (**unsandboxed, in-process**) | Chosen runtime | DSPy modules [dspy.ai/learn/programming/modules/]; LangChain PythonREPLTool docs | | 4 | **Result-back-into-LM** | Code produced a value | Inject stdout/return value into the next LM turn so the model narrates/uses it | LM-consumed result | PoT design: code computes, LM contextualizes [arXiv 2211.12588] | | 5 | **Error retry (bounded)** | Code raised a traceback | Feed error to LM, regenerate, re-run; cap at `max_iters` (~3) then fall back | Fixed code or graceful fallback | DSPy `ProgramOfThought` `max_iters`; see [[agentsop-test-fix-loop]] | | 6 | **Precision escalation** | Prose answer involves multi-step arithmetic | Re-route the arithmetic to code even if prose started it | Code-verified number | "LMs are unreliable calculators" — PAL [arXiv 2211.10435] | | 7 | **Over-coding veto** | About to sandbox a judgment task | Stop: no deterministic core → reasoning, not code | Prose reasoning, no sandbox call | §5 Case B; avoids wasted round-trip | In DSPy terms, op #1 is exactly the choice between `dspy.ChainOfThought` (prose reasoning) and `dspy.ProgramOfThought` (emit+run) for a signature — the dspy skill lists the modules but this overlay supplies the *when*. --- ## 5. 困境决策案例 (Dilemma cases) ### Case A — Arithmetic in prose hallucinates → PoT fixes it (under-coding) **Trigger**: A finance-summary agent step: "Given these 14 line items, compute the total, the 8.25% tax, and the grand total." The agent is a `dspy.ChainOfThought` module emitting prose. **Constraints**: - 14 multi-digit addends + a percentage + a sum-of-sums. Well outside reliable mental-math range. - The number flows into an invoice — wrong is *expensive* and *silent*. - A Python sandbox is available. **Decision steps**: 1. **Gate (§3 Step 1):** "Is there a single verifiable answer?" Yes — the total is a fact, not a judgment. → computable. 2. **Recognize the under-coding failure mode.** Prose chain-of-thought will emit a plausible running sum that is frequently off by some digits, and *no exception will fire*. This is the silent failure the gate exists to prevent. 3. **Switch the module from `ChainOfThought` to `ProgramOfThought`.** The model now emits `subtotal = sum([...]); tax = round(subtotal * 0.0825, 2); total = subtotal + tax` and the interpreter computes it exactly. 4. **Feed the result back (op #4):** the LM receives `total = 4,217.93` and narrates the invoice line. Code computed; LM contextualized. 5. **Serialize per [[agentsop-output-format-by-model]]:** the generated code goes in a fenced block, not a JSON `program` field — code-in-JSON would degrade it. **Outcome**: The arithmetic is now deterministic and auditable. PoT-style code execution is the documented fix for exactly this class of error [arXiv 2211.12588, arXiv 2211.10435]. **Extractable operation**: **Multi-step arithmetic in prose is a smell. Re-route it to code (op #6).** --- ### Case B — Over-coding a judgment task wastes a sandbox round-trip (over-coding) **Trigger**: A support-triage agent step: "Read this customer message and decide whether the tone is hostile, neutral, or warm." An over-eager engineer wires it through `ProgramOfThought` because "code is more reliable." **Constraints**: - "Tone" has no single verifiable answer — it is a judgment. - The sandbox round-trip adds latency + cost. - Forcing it into code produces something like `if "!!!" in msg: tone = "hostile"` — a brittle rule that *encodes a subjective rubric as if it were a formula*, and is worse than the model's native judgment. **Decision steps**: 1. **Gate (§3 Step 1):** "Single verifiable answer a program could check?" No — tone is judgment. → reason in prose. Stop. 2. **Invoke the over-coding veto (op #7).** There is no deterministic core to delegate to an oracle. Code adds *no determinism* — it just relocates the same judgment into worse, hard-coded heuristics. 3. **Cost it out.** The sandbox call adds a full round-trip (and a code-gen + parse step) for zero precision gain. This is pure waste — the loud, cheap failure mode. 4. **Keep it as `ChainOfThought`** (or `Predict`). The LM *is* the right engine for judgment. 5. **Counter-check:** if a later spec adds "and count how many exclamation marks" — *that* sub-step is computable; decompose (op #2) and route the count to code, the tone to prose. **Outcome**: No sandbox call. The judgment stays where judgment belongs. Over-coding is a real and common anti-pattern: not everything benefits from an interpreter, only things with a verifiable deterministic core. **Extractable operation**: **No verifiable answer → no code. Veto the sandbox round-trip (op #7).** --- ### Case C — Mixed step: summary + total (decompose) **Trigger**: "Summarize this quarter's sales narrative and give me the exact total revenue." **Constraints**: One sentence contains both a judgment (summary) and a computation (total). **Decision steps**: 1. **Gate → mixed.** Decompose (op #2). 2. Route the **total** to `ProgramOfThought`: code sums the figures, interpreter verifies. 3. Route the **summary** to `ChainOfThought`: prose synthesis, no oracle exists. 4. **Feed the code result back** into the prose step so the narrative cites the exact, verified total. **Outcome**: Each sub-step uses its correct engine. This is the execution-decision analogue of the mixed-content two-pass pattern in [[agentsop-output-format-by-model]] §5 Case B. **Extractable operation**: **One task ≠ one strategy. Gate per step, decompose mixed steps.** --- ## 6. 反模式与边界 (Anti-patterns & boundaries) ### Anti-patterns 1. **Code for everything ("code is always more reliable").** False. Code is more reliable only where a *deterministic, verifiable* answer exists. For judgment tasks, code just hard-codes a subjective rubric and adds a wasted round-trip (Case B). The interpreter is an oracle only for computable questions. 2. **Reasoning for arithmetic ("the model can just add it").** The headline failure. LMs are unreliable calculators; multi-step prose arithmetic hallucinates plausible wrong numbers, *silently* (Case A). Any non-trivial computation → code [arXiv 2211.12588]. 3. **Emitting code but never feeding the result back.** PoT is "code computes, LM contextualizes." If the value never re-enters the LM turn, you have a dangling computation the agent can't use or narrate (op #4). 4. **Unbounded error-retry loops.** On a traceback, regenerate-and-rerun is correct — but cap it (`max_iters` ≈ 3) and fall back to prose or surface the failure. Infinite code-repair loops burn cost. See [[agentsop-test-fix-loop]]. 5. **Nesting generated code inside JSON sub-structure.** The execution decision says *emit code*; [[agentsop-output-format-by-model]] says emit it as a fenced block / single string, never as nested JSON — code-in-JSON degrades the code (Aider 61%→20%). Pass both gates. 6. **Treating LangChain `PythonREPLTool` as a safe sandbox.** It runs **in-process, unsandboxed**. Fine for trusted self-authored code; hostile to untrusted input. Use a real sandbox (OpenAI Code Interpreter container, Anthropic code-exec VM, DSPy's interpreter) when inputs are untrusted. 7. **Over-decomposing trivial computation.** A 3-item count or single-digit add does not need a sandbox round-trip; the tax exceeds the benefit. The gate is for *non-trivial* computation. ### Boundaries (when this gate doesn't fire) - **No sandbox available and determinism is soft.** Sometimes you must reason in prose because there is no interpreter; accept the precision risk and flag it. - **The model is overwhelmingly capable for a tiny computation.** Frontier models reliably handle small arithmetic; the round-trip isn't worth it below a complexity threshold. - **The task is one-shot exploration** with no downstream consumer of the precise value. - **Latency-critical paths** where a sandbox round-trip violates a hard SLA and an approximate answer is acceptable — a deliberate, documented tradeoff, not a default. --- ## 7. 跨框架对照 (Ecosystem cross-reference) How major frameworks expose the emit-code-vs-reason mechanism. This skill operates at the **decision** layer; each framework supplies the **mechanism**. ### DSPy `ProgramOfThought` [dspy.ai/learn/programming/modules/] The canonical declarative version. A signature compiled with `dspy.ProgramOfThought(Sig)` makes the LM emit Python, runs it in an interpreter sandbox, and feeds the result back — with bounded retry on error (`max_iters`). The sibling **dspy skill** lists `ChainOfThought` vs `ProgramOfThought` as module choices but does **not** give the decision rubric; *this overlay is that rubric*. Use `ProgramOfThought` exactly when §3 Step 1 returns "computable." ### OpenAI Code Interpreter / Assistants code tool A managed container that the model can write Python into and execute, with files and state persisted across turns. Heavier and stateful — good for data-analysis sessions (load CSV, compute, plot). Same gate applies: route computable steps in, keep judgment in chat. ### Anthropic code-execution tool A sandboxed VM exposed as a tool; the model emits code, it runs, results return to the conversation. First-party, sandboxed — safe for untrusted inputs. The execution-decision gate maps directly: offer the tool, but the *model/agent* should only invoke it for computable steps. ### LangChain `PythonREPLTool` A tool wrapping a Python REPL. **Runs in-process and is unsandboxed** — powerful and dangerous. Use only for trusted, self-authored computation; never expose it to untrusted input without an external sandbox. The decision rubric is identical; the *safety* profile is worst-in-class. ### Cross-framework summary ``` Framework | Mechanism | Sandbox | Result-back ---------------------|--------------------------|-------------|------------ DSPy PoT | ProgramOfThought module | Python intp | automatic (max_iters) OpenAI Code Interp. | Assistants code tool | container | persisted state Anthropic code-exec | code-execution tool | VM (safe) | into conversation LangChain | PythonREPLTool | NONE (proc) | manual wiring ``` Every framework can be *mis-invoked* — pointed at a judgment task (over-code) or skipped for a computation (under-code). This overlay is the gate that decides invocation, regardless of which mechanism is underneath. --- ## Quick reference card ``` ┌──────────────────────────────────────────────────────────────────────┐ │ EMIT-CODE-VS-REASON DECISION CARD │ ├──────────────────────────────────────────────────────────────────────┤ │ Single verifiable answer a program could check? │ │ YES → EMIT CODE → sandbox → run → feed result back into LM │ │ NO → REASON IN PROSE (no oracle exists for judgment) │ │ MIXED → decompose; route each sub-step independently │ ├──────────────────────────────────────────────────────────────────────┤ │ Arithmetic / parse / sort / count / regex / symbolic → CODE │ │ Tone / quality / summary / design / synthesis → PROSE │ │ "summary AND total" → SPLIT │ ├──────────────────────────────────────────────────────────────────────┤ │ LMs are unreliable calculators but reliable coders. │ │ Under-coding fails SILENTLY (wrong plausible number). │ │ Over-coding fails LOUDLY+CHEAPLY (wasted sandbox round-trip). │ │ When genuinely uncertain AND a verifiable core exists → lean CODE. │ ├──────────────────────────────────────────────────────────────────────┤ │ NEVER: │ │ • code a judgment task (over-coding veto) │ │ • do multi-step arithmetic in prose (under-coding) │ │ • forget to feed the code result back into the LM │ │ • nest generated code in JSON (see [[agentsop-output-format-by-model]]) │ │ • loop code-repair unbounded (see [[agentsop-test-fix-loop]]) │ │ • trust LangChain PythonREPLTool on untrusted input (unsandboxed) │ └──────────────────────────────────────────────────────────────────────┘ ``` --- ## 引用源 (Citations) **Primary anchors:** - Program of Thoughts (PoT) — Chen et al., arXiv 2211.12588 — decouple computation from reasoning; LM writes code, interpreter computes. - PAL: Program-aided Language Models — Gao et al., arXiv 2211.10435 — LM as coder, runtime as deterministic solver. - DSPy `ProgramOfThought` module — [dspy.ai/learn/programming/modules/] — emit+run+retry mechanism. **Framework / API docs:** - OpenAI Code Interpreter / Assistants code tool — [platform.openai.com/docs/assistants/tools/code-interpreter] - Anthropic code-execution tool — [docs.anthropic.com/en/docs/agents-and-tools/tool-use/code-execution-tool] - LangChain `PythonREPLTool` — [python.langchain.com/docs/integrations/tools/python] (unsandboxed; in-process). **Companion / overlaid skills:** - `dspy-sop-skill/SKILL.md` — ships `ProgramOfThought` but not this decision rubric (the gap this overlay fills). - `d-output-format-by-model-skill/SKILL.md` — sibling: once you emit code, how to serialize it (PoT for math/parse; code never nested in JSON). Cross-linked as [[agentsop-output-format-by-model]]. - `test-fix-loop` — the execute → error → retry loop this skill defers to for bounded code repair. Cross-linked as [[agentsop-test-fix-loop]].