--- name: netlify-deploy-traps description: "Find and prevent the class of Netlify failure where the deploy succeeds and the site is wrong. Load this skill before running netlify deploy by hand, when one repository builds more than one site, when a site returns 404 on every page or redirects to itself, when a scheduled function looks dead in the logs, when a cron job runs far later than its schedule says, when a form on a live site has never captured a submission, when a Netlify Blobs read or lock behaves as though it lost a race it should have won, and when an environment variable reads as asterisks in a local script. Also load it when deciding how to roll back a bad deploy, because restoring the previous deploy and rebuilding from git are not the same move and only one of them is instant. The deploy log saying Published is not evidence that the right files or the right rules went out." license: MIT metadata: author: Anthony Limpert organization: Clickflame version: "1.0.0" date: September 2026 abstract: A field guide to Netlify deploys that report success and serve the wrong thing. Written from production incidents across a network of static sites and functions. Covers the --dir flag overriding netlify.toml and shipping from disk, redirect rules that stay behind when the files move, sampled function logs that make a live job look dead, scheduled functions running half an hour late under contention, form detection defaulting to off, eventually consistent blob reads that break a lock, conditional writes whose failure is a return value, and secret env vars masked into your own shell. --- # Netlify Deploy Traps A Netlify deploy tells you it published. That statement is true and it is not the question. The question is which files went out, which rules went with them, and whether the thing you changed is the thing now serving. Those can all be wrong while the deploy log is green. Every incident below is from a network of static sites and functions in production. None of them threw. Each was found by asking the platform what it is actually serving, rather than by reading the command that was typed. ## Start here Three probes answer almost everything in this file. Reach for them before theorizing. **Ask what files are live.** `listSiteFiles` on the deploy returns the manifest. A healthy static site is rooted at `/index.html`. If the manifest shows `/scripts/...`, `/data/...` or `/exports/...`, you published a repository instead of a site. **Ask what a URL actually answers.** `curl -sI https:///` and read the status, the `Location` header and the `Server` header together. `Server: Netlify` with a self-referential `Location` and a body around 42 bytes of `text/plain` is Netlify's redirect engine firing. A rule is running. Go find the rule and stop checking DNS. **Ask the function to speak for itself.** Netlify function logs are sampled. Print your own line on every invocation and read that, or plant a row in a database that the function must consume and watch the row. The deploy summary is not one of these probes. It reads like "4 new files uploaded, 73 functions deployed" because Netlify only uploads changed assets. A small upload count says nothing at all about what the manifest declares. ## The --dir flag overrides netlify.toml, silently `netlify deploy --dir ` ignores the `publish` value in `netlify.toml`. No warning, no diff, no confirmation of which one won. The danger is that the correct value differs per site and the flag does not care. A site whose repo root is its web root genuinely publishes `.`. A site whose publish directory is build output must never receive that value. Nothing in the CLI knows which one you are pointed at, so the same command is right on one site and catastrophic on the next. Measured cost of getting it the wrong way round once: **44 minutes down, and the entire working tree served.** Two failures land at once, and the first hides the second: 1. **Every page 404s**, homepage included, because `index.html` is now at `/public/index.html`. That path serving 200 is the tell. 2. **The whole working tree becomes downloadable.** Exported CSVs of contact records, JSON data files and deploy scripts all served 200. The second one is the one that matters and the first one is the one that gets reported. Fix in that order anyway, because the rollback closes both. **`--dir` deploys from disk, not from git.** Files that are gitignored ship. The CLI does skip dotfiles, so `.git`, `.env` and similar stayed private, but that is a property of the CLI and not a guard you designed. Do not build a threat model on it. **Build from git for any site whose publish directory is build output.** A local `public/` goes stale within days, so `--dir public --no-build` puts an old site up and reports success. ## Rolling back and rebuilding are different moves `netlify api restoreSiteDeploy` to the last good deploy is **instant** and it closes an exposure now. It changes only what is served. It never touches files on disk, so uncommitted work is not at risk. Rebuilding from the main branch is the other move, and for a site with a real build step that ingest can run for minutes. Going straight to a rebuild leaves the bad files served for the whole of it. **Restore first, then rebuild** to pick up any commits the restore stepped back past. Afterward, be honest about what you cannot know. A static site with analytics disabled and no log drain keeps **no access logs for static assets**, and analytics cannot be enabled retroactively. An exposure window with no logs is unknown, not empty. Do not write "nobody downloaded it" in the incident note. Write that it cannot be determined, and turn analytics on before the next one. ## The same flag moves files and leaves the rules behind One repository, two brands, two Netlify sites. `netlify.toml` can name only one `publish` directory, and it named the first brand. The second brand built into its own directory and was deployed with `--dir`. `--dir` moved the files. It did not move the redirect rules. The first brand's `_redirects` held a catch-all forward: ``` /* https://otherbrand.example/:splat 301! ``` That rule shipped attached to the second brand's pages. Every URL on the site answered `301` to itself, homepage included. `curl -L` burned four hops and never landed. It ran about 20 minutes. **What made it expensive to find:** - `listSiteFiles` was **clean**. 205 files, a real `/index.html` at 31,927 bytes, no `_redirects` in the manifest at all. The files were right. Only the rules were wrong, and the manifest does not describe rules. - `--no-build` did not help. Redeploying the same correct `--dir` did not help. - The TLS certificate was issued, the DNS zone was on the right account, and the records pointed at the right site. Every usual suspect checked out clean, which is the part that costs the hours. - The deploy preview URL also redirected to the apex domain, which is **normal** primary-domain behavior and reads exactly like a second symptom. **The fix that proved it:** move the other brand's `_redirects` out of the way, redeploy, site returns 200. Nothing else changed. **If one repo builds more than one site, do not deploy it by hand.** Write a deploy script that rewrites the `publish` line for the brand being deployed, prints the `_redirects` that will actually ship, deploys, restores the line, then checks the live status code against what that brand expects. A brand whose correct answer is 200 and a brand whose correct answer is 301 cannot share one unchecked command. ## A blob read can return what was there before your own write Netlify Blobs reads are eventually consistent by default. `getStore(...).get(key)` is served from a cache and can hand back the value from before your own write. This broke a publisher that used a claim, pause, read back lock. It wrote its own id, read back a stale value, concluded that another process had won the claim, and returned a 409 saying nothing had been double posted. Nothing had been posted at all. Three approved items sat in the queue displaying as published. It reproduced twice in a row. **Any read whose answer decides something must pass `{ consistency: 'strong' }`.** That means every lock or claim readback, and every "have we already done this" index check. `list()` is **always** eventually consistent and takes no consistency option at all. A listing can name a key that was deleted minutes ago. The way around it is to re-read every key from the listing with a strong read and drop the ones that come back empty. The lock pattern in this codebase was correct in one file and carried a comment explaining why. The bug arrived when a second feature copied the pattern and not the flag. **A correctness flag with a comment next to it is still going to be dropped by the next copy.** Put it in the store constructor where it cannot be separated from the calls. ## A conditional write that did not happen returns normally `@netlify/blobs` v11 gives you a real compare and swap with no database. `setJSON(key, data, opts)` accepts `onlyIfMatch: etag` and `onlyIfNew: true`, and `getWithMetadata(key, { type: 'json' })` hands back `{ data, etag }`. ```js const db = getStore({ name: 'bookings', consistency: 'strong' }) for (let i = 0; i < 6; i++) { const cur = await db.getWithMetadata(key, { type: 'json' }) if (clashes(cur?.data)) return { ok: false, reason: 'taken' } const res = await db.setJSON(key, next, cur ? { onlyIfMatch: cur.etag } : { onlyIfNew: true }) if (res.modified) return { ok: true } // somebody wrote between our read and our write, go around again } ``` Proved live: three simultaneous requests for the same resource and the same dates gave one success and two rejections, every time, and the losers were handed the next open slot. **The trap is `res.modified`.** A conditional write that was refused does not throw. It returns an object with `modified: false`, and code that ignores the return value reports success and double books. This is the same shape as a PostgREST update that matches zero rows and returns 200. The call came back fine and nothing happened. **Put the contested state in one blob per contested thing**, never one blob per record, or two callers are not competing for the same key and the compare and swap protects nothing. ## No log line is not the same as no run Netlify function logs are **sampled**. The platform's own `Duration:` report line is not emitted for every invocation. So "no log entry" and "did not run" are indistinguishable from outside. Half an hour went into a wrong conclusion here once, reporting an entire cohort of scheduled functions as dead when they were merely under-logged. **Make the function print its own line.** A line your code emits is carried reliably in a way the platform's report line is not. One `tick` log with something identifying in it is enough. **Or watch a side effect instead.** Park a row the function must consume and watch the row. The database is not sampled. ## A five minute schedule is not every five minutes Measured on a busy site: a row planted at 15:06:57 was not picked up until 15:35:09. Twenty nine minutes, about six missed tick boundaries, by a function scheduled `*/5 * * * *`. It then behaved perfectly. This is scheduling latency, not a code fault. Runs actually observed in one 18 minute window: | schedule | runs seen | |---|---| | `* * * * *` | 35 | | `*/2 * * * *` | 20 | | `* * * * *` (second function) | 15 | | `*/3 * * * *` | 4 | | `*/5 * * * *` (five functions) | 1 to 2 each | The likely cause was contention. One function scheduled `* * * * *` was firing about three times a minute at 15 to 20 seconds a run, which is roughly 48 seconds of execution per minute from a single function before anything else is counted. Under that load the `*/5` cohort was delayed. **Two things follow from this.** First, anything with a promise attached to it should not be on a schedule at all. A message that has to reach a customer quickly belongs in the request that created it, not in a sweep. Second, a watchdog on a `*/5` schedule checks far less often than it reads like it does, which means the alarm for a dead pipeline is itself slower than the page implies. **Before blaming the schedule, count the runs.** A function firing three times a minute on a one minute cron is its own bug and it starves everything else. ## Form detection defaults to off A Netlify site can default to `ignore_html_forms: true`. Every form on it can carry `data-netlify="true"` and the site will register no form and store no submission. An audit of 21 sites found that **every site with a Netlify form had detection off and had recorded zero submissions, ever.** Submitting fails as a 404, so visitors saw an error rather than a false thank you, but the lead is gone either way. The reason it survived a QA gate is worth more than the fix. The gate said "form submits and the lead notification arrives", which somebody ticked by looking at the page. **Fixing one, in order:** 1. `PATCH /sites/{id}` with `{"processing_settings":{"ignore_html_forms":false,"html":{"pretty_urls":true}}}`. A top-level `ignore_html_forms` is **silently ignored** on git-linked sites. 2. `POST /sites/{id}/builds` with a JSON content type header, because detection runs only at build time. An empty body without the header returns no deploy id. 3. `listSiteForms` to get the form id. 4. `POST /hooks?site_id=...` for a `submission_created` notification, with the fields at **top level**. Nesting them returns a 422 complaining about a URL. 5. **POST the live form and confirm a 200 and a row.** Steps 1 to 4 are reading. This step is the audit. `netlify link --id` wants the site UUID, not the subdomain name. ## A secret environment variable is masked into your own shell A Netlify env var marked **secret** is masked in `env:get`, which is expected, and it is also **injected masked into `dev:exec`**, which is not. Measured: the child process received a 20 character string, 16 of them asterisks. A request built from it returned 401. There is no local route to a secret's value. Only the deployed function sees the real one. **What makes this easy to miss** is that non-secret variables come through `dev:exec` intact. The probe looks like it worked, because most of the variables did. If a local script needs such a value, it has to arrive out of band into a gitignored file. Check what `.gitignore` already covers before choosing the filename, especially on a site whose repository is also its web root, where a stray file is a published file. ## The verification moves, collected - **After any hand deploy, read the manifest.** `listSiteFiles` rooted anywhere but `/index.html` is a wrong publish directory. - **After any deploy that changes routing, curl the apex and read three headers.** Status, `Location`, `Server`. A self-referential `Location` is a rule, not DNS. - **If one repo builds two sites, deploy through a script that checks the live status code**, and give each brand its expected code. - **Restore the last good deploy before rebuilding.** One is instant, the other is as slow as your build. - **Set `consistency: 'strong'` on the store**, not on the call site. - **Check `res.modified` on every conditional write** and treat false as a failed attempt, not as success. - **Make every scheduled function print its own line**, and reason from that line or from a database side effect, never from the absence of a platform log. - **Count actual runs before trusting a cron expression.** - **Submit every live form once and confirm the row.** A form that renders is not a form that captures. - **Print the length of a secret you loaded locally before using it.** Twenty characters of mostly asterisks is the answer. Every one of these asks the platform what is true right now. The deploy command you typed, the config file you read, and the schedule you wrote are all statements of intent. None of them is evidence. --- *Written from production incidents across a network of Netlify sites and functions, by [Clickflame](https://clickflame.com). Companion skills: `supabase-rls-audit` and `postgrest-silent-failures`, for the same class of quiet failure in the database and in the client.*