[^<]*

--- name: crawlability-audit description: Find pages search engines cannot reach or will not trust — robots.txt rules that block real content, sitemaps contradicting robots, error pages served as HTTP 200, missing canonicals, and duplicate metadata. Use before a launch, when organic traffic is flat, after changing robots or routing, or when the user asks why a page is not indexed. --- # Crawlability Audit ## Install Save this file as `~/.claude/skills/crawlability-audit/SKILL.md`, or `.claude/skills/crawlability-audit/SKILL.md` to scope it to one repo. Claude Code auto-discovers it. Invoke with `/crawlability-audit` or by asking "why isn't this page getting indexed?". ## Why this exists SEO advice is mostly about content. The failures that actually cost the most are mechanical, silent, and take minutes to fix once seen: a prefix rule that blocks more than anyone intended, a sitemap advertising URLs that robots forbids, an error page returning 200 so crawlers file it as thin content instead of retrying. None of these announce themselves. Traffic is simply flat, and everyone blames the copy. Rank by what it costs. A rule blocking your signup or seller page outranks a missing meta description on a blog post, always. ## Step 1 — Read robots.txt as a machine would ``` curl -s https:///robots.txt ``` **Every `Disallow` is a PREFIX match, not a path match.** This is the single most expensive misunderstanding in the file: - `Disallow: /sell` blocks `/sell`, and also `/sellers`, `/selling`, `/seller/anyone`, `/sell-your-thing`. Whole sections vanish. - `Disallow: /sell/` blocks only children, leaving `/sell` itself crawlable. For each `Disallow`, enumerate the real routes it matches. Read the router or route directory; do not guess: ``` rg --files -g '*/page.tsx' -g '*/route.ts' -g 'pages/**' | sort ``` Then ask, per matched route: **is this genuinely private?** A rule written for an authenticated dashboard very often also covers the public marketing page that shares its prefix. That is the bug, and it is usually invisible because the public page still works perfectly for humans. ## Step 2 — Cross-check robots against every sitemap Fetch every sitemap, including nested ones: ``` curl -s https:///sitemap.xml curl -s https:///robots.txt | grep -i sitemap ``` Two contradictions to hunt: 1. **URLs in a sitemap that robots disallows.** You are asking a crawler to index pages you have forbidden it to read. Search consoles report this as a warning most people never open. 2. **Sitemaps that exist but are never declared.** Generated and unlinked means undiscoverable. Compare generated sitemap routes against the `Sitemap:` lines. ``` comm -12 <(sitemap paths, sorted) <(disallowed prefixes expanded, sorted) ``` Any overlap is a finding. ## Step 3 — Check what status codes crawlers actually receive A page that renders an error but returns 200 is worse than one returning 503. On 200 the crawler stores an empty page as your content; on 5xx it retains what it knows and retries. ``` curl -s -o /dev/null -w "%{http_code}\n" https:/// curl -s https:/// | head -c 2000 ``` Compare status to CONTENT, not to whether the request succeeded. Flag: - 200 responses whose body is an error state, a spinner, or empty. - 200 responses containing a client-side render abort with no server content. - Sitemaps returning 200 while listing zero real URLs. This actively tells a crawler your catalog was withdrawn, every time it is fetched. - Soft 404s: "not found" text served with a 200. ## Step 4 — Metadata on the pages that earn money For each high-intent route (catalog, pricing, signup, top products): ``` curl -s https:/// | grep -oE '[^<]*|]*>|]*>' ``` Flag, in priority order: 1. **Duplicate title or description across routes.** Two pages competing for one query with identical words. Extremely common on catalog pages that inherit a site-wide default and get missed because they "look fine". 2. **Missing canonical on anything reachable by more than one URL.** Query parameters for filtering and sorting mint unbounded near-duplicates. Pin the canonical to the bare path. 3. **Missing structured data on commercial pages.** Product, Offer, and price markup is how both rich results and AI answer engines read a listing. Check whether a helper already exists and is simply not wired to the page it describes; that is a frequent and cheap win. ## Step 5 — Internal links, since that is how discovery actually works Sitemaps are a hint. Internal links are the signal. ``` curl -s https:/// | grep -oE 'href="/[^"]*"' | sort -u ``` Flag redirect hops on primary navigation. If every catalog card points at a tracking route that 302s to the real page, no page has direct internal links. Tracking is a fair reason; just do not let the pages built to RANK inherit it. ## Step 6 — Report Ranked by revenue impact, not by count: ``` BLOCKED FROM CRAWLERS <- matched by "Disallow: ", is actually CONTRADICTIONS URLs are in a sitemap AND disallowed sitemaps generated but never declared in robots.txt WRONG STATUS CODES 200 but body is METADATA duplicate title with no canonical, reachable via query permutations ``` Lead with the single most expensive line and say what it costs in plain words: "your seller signup page has been invisible to Google" beats "robots.txt misconfiguration". ## Rules - Verify against the LIVE site. The deployed robots.txt is frequently older than the source file, and the live one is the only one crawlers obey. - Never recommend blocking a page to "save crawl budget" unless it is genuinely private or genuinely duplicate. Crawl budget is not a real constraint for most sites and this advice deletes traffic. - Never recommend adding structured data describing things that are not on the page. Invented ratings and fake availability get manual actions. - Prefer one fixed prefix rule over twenty new pages of content. --- From [Toolbay](https://toolbay.ai/product/crawlability-audit). Free to use, modify, and share. Keep this line and others can find it too.