# Protocol v3: final experimental design The committed E1-E7 tables were generated by `notebooks/LLMstrategicplanning_protocol_v3_executed.ipynb` with protocol identifier `3.0-final-cache-crn`. The protocol is designed to isolate the effect of planning architecture while preventing prompt-cache and random-number artefacts from creating spurious differences among strategies. ## Experimental object A **planning-induced control trajectory** is the ordered sequence of planning operations and node-level curtailment directives produced during a 24-hour episode. It is distinct from the electrical trajectory that emerges after prosumers respond. The causal chain is: ```text fixed scenario and objective -> planning strategy and typed policy -> planning-induced control trajectory -> strategic prosumer response -> feeder trajectory and physical objective ``` The benchmark reports three distinct evaluation dimensions: planning-strategy heterogeneity, execution fidelity, and adaptive selection. The protocol below is designed so that evidence for one dimension cannot be mistaken for evidence about another. In particular, mode agreement is not used as a substitute for physical intent, and declaration frequency is not used as a substitute for per-scenario routing quality. ## LLM interfaces and non-LLM components The LLM is bounded to structured policy declaration/advice and language-mediated communication. The natural declaration prompt receives a compact state and returns a typed `DispatchPolicy` JSON object. The hybrid prompt receives the same state plus deterministic mode scores and returns only bounded per-mode shifts. At dispatch time, one prompt generates the customer-facing message; a second prompt for a common-random-number sample of targeted prosumers returns a bounded willingness shift. All outputs are parsed, range-clipped, and type-checked. The LLM does **not** solve power flow, create the numerical allocation, choose the prosumer's action, or score physical performance. Strategy executors, game-theoretic base probabilities, random compliance and realisation, and DistFlow are explicit code. The optional LLM critic is disabled in the reported causal banks; E4 uses only the deterministic deadline gate. Exact system prompts and dynamic user-payload schemas are in `prompts/` and `docs/LLM_INTERFACES.md`. The four executors are operationally distinct: - **PREDEFINED:** compiles the remaining-horizon schedule once and does not revise it. - **SEQUENTIAL:** executes the current step and reconstructs the remaining suffix after a violation, using recent compliance as an updated prior. - **HIERARCHICAL:** decomposes the reduction request across laterals and, at depth three, feeder segments; it may revise after violations. - **SEARCH:** generates multiple targeting-intensity schedules, rolls them out against forecast power flow, and commits to the lowest predicted-cost candidate. ## Isolation in forced-mode comparisons Within a paired forced-mode scenario, all four strategies receive the same: - feeder topology and physical parameters; - load-PV realisation and observed forecast; - 40-prosumer population and archetype assignment; - system objective, targeting rule, message theme, and stress-calibrated intensity; - tool-failure stream and latent prosumer-response draws. Only the mode-defining semantics change: one-shot commitment, replanning, hierarchical decomposition, or candidate search. Natural and bounded-hybrid selectors are analysed separately because they may also change non-mode policy fields. ## Protocol corrections ### Exact-prompt cache discipline Every LLM cache entry is keyed by the endpoint/model namespace, system message, and full prompt. A collision guard raises an error if a key is ever reused for a different prompt. The final run recorded 21,961 uncached API calls and 72,146 exact-prompt cache hits. ### Independent random streams and common response draws Forecast error, planning-internal randomness, tool failures, adversarial substitution, and prosumer response use separate deterministic streams. For paired strategies, latent LLM sampling, compliance, and realisation draws are keyed by `(seed, hour, node)`. Planning search or hierarchy therefore cannot change later prosumer randomness merely by consuming additional random numbers. ### Correct final-step handling A new plan is declared only after the existing plan has no step for the current hour. The final scheduled step is executed rather than replaced by an artificial last-hour declaration. E1 consequently contains one declaration per episode. ### Event-level deadline accounting Every executor invocation, including a revision, incurs a stipulated nominal decision cost. A deadline miss is recorded when that invocation exceeds the control interval. Feasibility is therefore defined by `deadline_misses == 0`, not by comparing accumulated episode latency with a per-decision deadline. ### Critic isolation The LLM critic is disabled in the causal forced-mode banks. E4 activates only the deterministic deadline gate because the gate itself is the experimental factor. The gate constructs type-valid fallback policies. ## Model and common settings - Model: `meta-llama/Llama-3.3-70B-Instruct` - Endpoint: Nebius Token Factory, OpenAI-compatible API - Temperature: 0.2 - Maximum completion tokens: 400 - Horizon: 24 hours - Feeder: four laterals, ten prosumer nodes per lateral - Default forecast noise: 0.10 - Default prosumer resistance mean: 0.30 - LLM-evaluated targeted agents: 40% - Bounded narrative shift: [-0.30, +0.30], attenuated by resistance - Tool-failure probability per dispatch: 0.02 E1, E2, and E5 use seeds `{7, 13, 42, 101, 202}`. E3, E4, E6, and each E7 factor combination use `{7, 13, 42}`. ## E7 cost-sensitive routing E7 crosses four stress groups, three forecast-noise levels, two deadlines, two resistance levels, and three seeds, yielding 144 scenarios. Every mode is forced in every scenario, for 576 paired episodes. A mode is feasible only when it has no event-level deadline miss. The selection cost is ```text selection_cost = physical objective + 10,000 * I[any deadline miss] ``` One ridge cost model is fitted per mode. Cross-fitting holds out an entire feeder-stress group, so no scenario from the held-out stress regime is used for training. The feature set contains only information available before routing: cap tightness, line resistance, forecast noise, decision deadline, and mean prosumer resistance. Regularisation is fixed at 10; no hyperparameter selection is performed on test groups. Uncertainty is computed by a 5,000-draw scenario-level paired bootstrap over the routed scenario table. The forced-mode counterfactuals remain joined within each scenario before routing and regret are computed. ## Secondary E7 analysis layer Protocol-v3 generation and the prespecified one-stage estimator remain unchanged. The repository adds a post-hoc diagnostic over the same forced bank. For the committed E7 policies, nominal event costs are 1.0, 1.6, 4.2, and 5.5 seconds. The constraint-aware analysis filters modes whose known cost exceeds the scenario deadline, fits the same per-mode ridge on feasible training rows, and predicts physical objective. It retains the same features, `alpha = 10`, stress-group folds, paired scenarios, and bootstrap. It adds no episode or LLM call. The all-feasible 12-second subset isolates quality ranking. See `docs/E7_DIAGNOSTICS.md`; this is an analysis-layer addition, not a change to the committed protocol. ## Interface-level multimodel extension The M0-M5 extension is additive to protocol v3. It reuses the exact bounded interfaces but does not alter the E1-E7 causal bank. Its canonical post-processing separates declaration latency from other LLM interfaces and reports the three-band state-dependence analysis as secondary to the original 20-state permutation test. See `MULTIMODEL_EXTENSION.md`.