# Benchmark config JSON schema

The live-validation scripts (`${SKILL_DIR}/scripts/deploy_model.py`, `${SKILL_DIR}/scripts/benchmark_model.py`, `${SKILL_DIR}/scripts/generate_perf_report.py`) read `dynamodb_data_model.json` — the same design JSON consumed by `${SKILL_DIR}/scripts/calculate_costs.py` (see `${SKILL_DIR}/references/cost-model-schema.md`) — plus a **separate** `benchmark_config.json` that carries per-run knobs. Keeping them separate preserves the design JSON as the single source of truth for the design itself; the benchmark config is short-lived and run-specific. (`${SKILL_DIR}` is defined in SKILL.md "Resolving the skill's own paths" — substitute the absolute path of the directory containing SKILL.md.)

This document specifies the JSON shape of `benchmark_config.json`, what each knob does, and how the defaults relate to measurement quality.

## Minimal working example

```json
{
  "aws_profile": "dev-sandbox",
  "region": "us-east-1",
  "resource_prefix": "ddb-skill-bench-20260512-abc12345",
  "_comment_resource_prefix": "optional — omit it and deploy_model.py generates ddb-skill-bench-<date>-<uuid8> for you",
  "tags": { "owner": "me@example.com", "purpose": "ddb-skill-bench", "ttl_hours": "4" },
  "mode": "standard",
  "table_settle_seconds": 30,
  "warmup_seconds": 10,
  "duration_seconds": 90,
  "ramp_seconds": 10,
  "scale_factor": 0.01,
  "max_rps_per_pattern": 50,
  "min_rps_per_pattern": 1,
  "concurrency_per_pattern": 32,
  "seed_items_per_table": 500,
  "abort_on_throttle_rate": 0.2,
  "read_pattern_key_sampling": "uniform",
  "dry_run": false
}
```

## Mode presets

`mode` (optional, default `"standard"`) picks a preset for the timing/seed knobs. Explicit values in the config always win; the preset only fills fields the user did not set.

| `mode` | Behavior |
|---|---|
| `"standard"` | Use the defaults documented in the tables below. The benchmark's statistical claims (p99 stability, extrapolated cost within tolerance) assume this mode. |
| `"quick"` | Short per-pattern window for a cheap smoke test — `table_settle_seconds: 10`, `warmup_seconds: 3`, `duration_seconds: 15`, `ramp_seconds: 3`, `seed_items_per_table: 100`. Useful when the user wants to confirm "does this design deploy and run at all?" without committing to a full validation. Percentiles are less stable and extrapolated cost is less trustworthy than `"standard"` — the report header labels this a smoke test. Per-pattern `scale_factor`/`min_rps`/`max_rps`/`concurrency_per_pattern` are unchanged by the preset, so driven RPS still reflects the design. |
| `"representative"` | **Scale-risk mode for the iterative design loop.** Drives proportionally higher load with hot-key skew and realistic item-collection cardinality to surface what `quick`/`standard` cannot: hot-partition throttling, throttle-under-load, GSI amplification at volume, and Query-at-realistic-cardinality. Preset: `scale_factor: 0.15` (in the 0.10–0.25 band), `max_rps_per_pattern: 1800` (the honest ceiling — one in-region Lambda with `concurrency_per_pattern: 32` threads tops out near 1,500–2,000 RPS/pattern), `min_rps_per_pattern: 5`, `concurrency_per_pattern: 32`, `seed_items_per_table: 2000`, `items_per_partition: 40`, `duration_seconds: 120`, `warmup_seconds: 15`, `table_settle_seconds: 45`, `ramp_seconds: 20`, `read_pattern_key_sampling: "zipf"`, `zipf_s: 1.1`. **Honest scope:** representative mode is BOUNDED — it surfaces hot-partition/throttle risk at bounded cost; it does NOT prove the design sustains declared peak. Cost extrapolation stays linear and valid (per-op CU is scale-invariant, extrapolated against *declared* peak, never the driven bench RPS); throttle/latency are load-risk signals and must not be linearly extrapolated. This is the mode `iterate_design.py` runs by default. |

Unknown `mode` values are rejected at script startup (so a typo like `"quik"` can't silently fall back to defaults).

## Fields

### Required

| Field | Purpose |
|---|---|
| `aws_profile` | AWS CLI profile name for all boto3 clients. Must resolve to valid credentials before any script runs; the preflight in `deploy_model.py` prints caller identity and refuses if the resolved account alias or ARN contains `prod`, `production`, `prd`, or `live`. |
| `region` | AWS region. Used consistently for DynamoDB and for the `sts` identity check. Single-region deploy — `global_tables: true` on a table is noted but not exercised. |

### Resource naming

| Field | Default | Purpose |
|---|---|---|
| `resource_prefix` | *(auto-generated)* | String that every created resource gets prefixed with. **Optional** — omit it and `deploy_model.py` generates `ddb-skill-bench-<YYYYMMDD>-<uuid8>` from the run's clock, prints the value it chose, and writes it into the manifest. If you DO supply one it must start with `ddb-skill-bench-` (so teardown can scope deletions safely); any other value is refused. Supply your own only when you need a recognizable, stable prefix; otherwise leave it out and let the deploy name the run. |

### Tagging

| Field | Default | Purpose |
|---|---|---|
| `tags` | `{}` | Extra tags merged onto every `CreateTable`. Always joined with the built-in tags `{purpose: ddb-skill-bench, run_id: <uuid>, created_at: <iso>, prefix: <prefix>}`. Use this to attach an owner, a cost-allocation tag, or a TTL expectation for cleanup tooling. |

### Cold-start discipline

These three knobs control how bias from fresh-table capacity, client warmup, and partition priming is excluded from steady-state numbers. Tightening them trades measurement quality for total runtime. See SKILL.md `## Live validation` — "Cold-start discipline" for the rationale.

| Field | Default | Effect |
|---|---|---|
| `table_settle_seconds` | `30` | Time to wait after `table_exists` returns before any measurement or seed call. On-demand tables start at baseline capacity (~2,000 WCU / ~4,000 RCU) and need a short window for adaptive capacity to stabilize. Setting to `0` will show throttles on fresh-table traffic that steady-state would not produce — useful as a one-off diagnostic, never trustworthy as a default. |
| `warmup_seconds` | `10` | Per-pattern warmup window run at measurement RPS *before* the measurement window. Rows are tagged `phase: "warmup"` and excluded from percentiles and the extrapolation. Reported separately as a cold-start snapshot: patterns where `warmup_p99 > 2 × steady_p99` are flagged `cold_start_elevated: true` in the report. |
| `seed_items_per_table` | `500` | Synthetic items written via `BatchWriteItem` (25-item batches) before measurement starts, for each table targeted by a read pattern. Seeds warm partitions the benchmark will later hit. Rows tagged `phase: "seed"` and excluded from measurement aggregation. Set to `0` to skip seeding entirely (reads will hit missing keys, flagging GetItem `non_existent_rate` as 100% — diagnostic but not representative). |

### Measurement window

| Field | Default | Effect |
|---|---|---|
| `duration_seconds` | `90` | Measurement-phase length per pattern. Short windows (under 60s) give statistically unstable percentiles on low-RPS patterns. Longer windows (over 180s) cost more and rarely move the numbers materially for a unit-cost benchmark. |
| `ramp_seconds` | `10` | Portion of the measurement window during which the abort guard is armed. Throttles concentrated in the first `ramp_seconds` often mean the warmup window was insufficient, not that the design is bad. After `ramp_seconds`, throttles feed `measurement_tainted: true` and stop the pattern. |
| `scale_factor` | `0.01` | Multiplier applied to each pattern's declared `peak_rps` before clamping. `0.01` means benchmark at 1% of declared load — cheap, single-digit cents for a typical run, still adequate for per-op unit-cost validation. The report extrapolates linearly using the declared `peak_rps`. |
| `min_rps_per_pattern` | `1` | Lower clamp on post-scale RPS. Ensures every pattern produces at least a handful of samples even when `peak_rps × scale_factor` would round to zero. Raise this if low-traffic patterns are producing percentile columns marked "insufficient samples." |
| `max_rps_per_pattern` | `50` | Upper clamp. Caps the bill on extremely-high-RPS patterns and keeps the run inside shared-account throttle budgets. Patterns that hit this cap are flagged in the report so the user knows the extrapolation assumes linear scaling. |
| `concurrency_per_pattern` | `32` | Threads per pattern in the `ThreadPoolExecutor`. Each thread issues one `DescribeTable` on startup to force TLS handshake + credential refresh off the critical path. At the unit-cost `scale_factor 0.01` default the pool is far larger than the driven RPS needs, so most threads idle — it matters in `"representative"` mode, where 32 threads are what let one in-region Lambda sustain ~1,500–2,000 RPS/pattern (the honest single-Lambda ceiling). Drop to 4 or 2 for patterns that are bursty under contention. |

### Safety

| Field | Default | Effect |
|---|---|---|
| `abort_on_throttle_rate` | `0.2` | Fraction of calls throttled at which a pattern's measurement is aborted and flagged `measurement_tainted: true`. Armed through warmup + the first `ramp_seconds` of measurement. Throttles confined to warmup do not taint the run (`cold_start_elevated: true` instead). Lower this value on a shared account; raise it (up to `0.5`) only as a deliberate probe. |
| `dry_run` | `false` | When `true`, `deploy_model.py` prints intended `CreateTable` kwargs and exits without creating any resources or writing a manifest. Useful for reviewing what would be deployed before committing. |
| `cost_guardrail_usd` | `0.50` | **Pre-spend gate.** Before invoking the Lambda, `benchmark_model.py` estimates the actual AWS charge (driven-load CU + seeding writes, priced with the calculator's own per-op CU). If the estimate exceeds this value the run is REFUSED unless `--allow-spend` is passed (the orchestrator forwards it after user consent). Distinct from `abort_on_throttle_rate`, which is a *runtime* guard once the run is underway. Raise it for deliberately larger representative runs. **Vector designs:** the estimate also covers vector search and write bytes, which consume no CU and would otherwise be invisible to the gate — a `SearchVectors` pattern would estimate at $0. Search is priced there with a deliberately high bound (a gate may be loose, never low), so the gate's figure is *not* a cost estimate; the report's measured figure is. |
| `provisioned_capacity` | *(unset → on-demand)* | Optional `{"read": N, "write": M}`. When set, `deploy_model.py` creates tables (and their GSIs) as **PROVISIONED** at that capacity instead of PAY_PER_REQUEST. Bench-only — production designs default to on-demand (Mechanics #19). Purpose: a low provisioned ceiling is a HARD per-table limit with no adaptive-capacity absorption, so hot-partition **throttling** (Mechanics #3) is observable — on on-demand, adaptive capacity absorbs a hot key as latency and a single load generator rarely throttles. Also the way to validate a *planned* provisioned capacity. A per-table `table.provisioned_capacity` overrides this global value. Note a fresh provisioned table carries ~5 min of burst capacity; use a longer `duration_seconds` or lower capacity so the burst drains and the sustained window throttles. |

### Read-pattern sampling

| Field | Default | Effect |
|---|---|---|
| `read_pattern_key_sampling` | `"uniform"` | How reads pick a target partition. `"uniform"` round-robins over the seeded partitions (even distribution). `"zipf"` (implemented) draws partition ranks with P(rank r) ∝ 1/r^`zipf_s`, concentrating load on a few hot partitions so one partition approaches the per-partition ceiling (Mechanics #3) and hot-partition throttling becomes observable. Auto-enabled by `"representative"` mode. Use `"zipf"` when probing hot-partition behavior; leave `"uniform"` for unit-cost runs. |
| `zipf_s` | `1.1` | Zipf skew exponent (higher = more concentrated). At `1.1` over ~50 partitions the hottest partition absorbs ~25% of traffic and dwarfs the tail ~70×. |
| `items_per_partition` | `1` | Collection cardinality per partition. `1` (quick/standard) means singleton partitions — every partition holds one item, so a Query reads one item and is indistinguishable from a GetItem. `> 1` (representative sets `40`) seeds that many distinct sort keys under each partition, so Query patterns read realistic multi-item pages and GSI collections are observable. The seeded partition count is `seed_items_per_table // items_per_partition`, and that partition count is the space the read-key sampler draws over. A table with no sort key cannot hold a collection and falls back to one item per partition. The size of the distinct-key sample pool is `seed_items_per_table` itself — there is no separate key-space knob. |

## How the knobs interact with measurement quality

- **Bias removal is cumulative.** The settle → seed → warmup sequence each removes a different class of bias. Skipping any one of them does not sum to "just a bit noisier" — the result is that the corresponding class of bias silently contaminates the steady-state numbers. Cold-start discipline is strongest when all three run.
- **Sample size matters for percentiles.** `p99` over 60 samples is not a `p99`. If `clamp(peak_rps × scale_factor, min, max) × duration_seconds < 100`, the report flags that pattern as "percentiles are not statistically meaningful" and widens to `p50`/`p95` only. Raise `min_rps_per_pattern` or widen `duration_seconds` to address.
- **Extrapolation assumes linearity.** Extrapolated monthly cost = mean observed CU × declared `peak_rps` × seconds-per-month × unit price. This is valid for per-op unit cost and GSI amplification; it is **not** valid for throttle behavior (throttles scale nonlinearly with partition-key skew — Mechanics #3). Patterns that hit `max_rps_per_pattern` and had throttles are flagged so the user does not read the extrapolation as a capacity-sustain claim. **`"representative"` mode deliberately surfaces this nonlinearity:** it drives hot-key skew to make throttles appear, and the report quarantines those throttle/latency numbers in a "Load-risk signals" section with an explicit "do not extrapolate" caption. The cost figure stays linear and valid because it uses *declared* peak and scale-invariant per-op CU — never the elevated driven RPS. Vector bytes extrapolate on the same basis and are similarly scale-invariant per call: metered search bytes measured **exactly** linear in `TopK` within a configuration, and index population is not a linear driver between 10 and 200 vectors in the searched partition because approximate-nearest-neighbour search prunes. One caveat on that last point: an **empty** 1024-dimension index measured roughly **twice** a populated one, so a benchmark run against a barely-seeded vector index can report *higher* per-search bytes than production will — raise `seed_items_per_table` before treating a vector search figure as representative.
- **Shared-account throttle sensitivity.** If the sandbox account has other workloads running, raise `abort_on_throttle_rate` only if you've verified there's no external contention. The abort guard exists so a bad run stops instead of silently producing corrupted numbers.

## Environment preconditions (not config knobs, but scripts will refuse without them)

- `boto3 >= 1.34` installed. `deploy_model.py` and `benchmark_model.py` refuse with `pip install boto3>=1.34` on import failure.
- **Vector designs additionally need `botocore`/`boto3 >= 1.43.64`** (AWS CLI v2 `>= 2.36.16`). Below that floor there is no `search_vectors` operation and no `VectorIndexes` parameter at all — the failure looks like the feature does not exist rather than like a stale SDK. `deploy_model.py` checks the floor and refuses before creating anything. The benchmark Lambda gets a vetted SDK vendored into its bundle from the local install (only for designs that declare a vector index, so every other design keeps its small package), because the Lambda runtime's built-in boto3 is older; the driver also fails loudly if `search_vectors` is missing rather than letting the pattern report zero consumed capacity and read as free.
- Valid AWS credentials resolvable via `aws_profile`. Expired SSO / missing token produces a specific remediation message (`aws sso login --profile <p>`) before any resource creation.
- Account identity does not contain prod markers. Even with `--yes-deploy`, `deploy_model.py` aborts if `sts get-caller-identity` returns an ARN or alias containing `prod`, `production`, `prd`, or `live`.
- `dynamodb_data_model.json` has non-empty `access_patterns` and every pattern has `peak_rps > 0`. Unknown RPS is a design gap (Mechanics #2), not a benchmark input.

## Tuning guide

| Goal | Adjust |
|---|---|
| Cheaper, quicker smoke test | `duration_seconds: 30`, `warmup_seconds: 5`, `seed_items_per_table: 100` |
| Sharper percentiles on low-RPS patterns | Raise `min_rps_per_pattern` or `duration_seconds` |
| Probe fresh-table cold-start behavior | `table_settle_seconds: 0`, `warmup_seconds: 0` — then read the Cold Start section of the report as the primary artifact |
| Avoid blowing the shared-account throttle budget | Lower `max_rps_per_pattern` and `concurrency_per_pattern`; raise `abort_on_throttle_rate` only if you've verified no external contention |
| Dry-run review before committing | `dry_run: true` on a first pass; then set `false` for the real run |