geolint
ESLint for AI search. Lint your website for AI-search readiness β
AI crawler access, llms.txt, structured data and citability.
π©πͺ Deutsch Β·
πͺπΈ EspaΓ±ol Β·
π―π΅ ζ₯ζ¬θͺ
## 30-second quickstart
No install, no config:
```bash
npx @iliasabk/geolint check yoursite.com
```
geolint fetches the page, its robots.txt and llms.txt, evaluates **51 known AI
crawler tokens** against your robots.txt, runs **52 audit rules**, and prints a
scored report with a concrete fix for every finding.
## Why
- **AI answers are the new front page.** ChatGPT, Perplexity, Claude, Copilot and
Google AI Overviews send traffic β or don't β based on whether their crawlers
can fetch and quote your pages.
- **Most sites accidentally block or confuse AI crawlers.** A stale
`Disallow: /`, a `noindex` left over from staging, a client-rendered page that
looks empty to a bot that doesn't run JavaScript.
- **Existing tools are blocklists or score-only web apps.** They tell you to
block everything, or give you a number with no path to improve it. geolint is
the *linter*: concrete findings, concrete fixes, runnable in CI on every PR.
## What it checks
52 rules across 5 categories β `geolint rules` lists them all, and
[docs/rules.md](docs/rules.md) documents what each rule checks, why it matters
and how to fix violations.
| Category | Rules | Examples |
| --- | ---: | --- |
| AI Crawler Access | 10 | `ai-crawler/search-bots-blocked`, `ai-crawler/wildcard-block-all`, `ai-crawler/user-fetch-bypass`, `ai-crawler/stale-tokens` |
| llms.txt | 12 | `llms-txt/missing`, `llms-txt/invalid-structure`, `llms-txt/broken-links`, `llms-txt/relative-links` |
| Structured Data | 7 | `schema/no-jsonld`, `schema/invalid-jsonld`, `schema/missing-article-fields` |
| Citability | 12 | `content/thin-content`, `content/no-h1`, `content/missing-dates`, `content/no-question-headings` |
| Technical Foundation | 11 | `technical/client-rendered`, `technical/https`, `technical/slow-response`, `technical/sitemap-missing` |
## What a report looks like
Real output, auditing the bundled demo site (`examples/demo-site`, which
deliberately blocks two bots) β trimmed for width:
```text
$ geolint check localhost:4173 --ignore technical/https
geolint v0.2.1 β AI-search readiness
http://localhost:4173/
200 OK Β· text/html Β· TTFB 113ms Β· robots 200 Β· llms.txt 404
ββββββββββββββββββββββββββββββ 86/100 Grade B
CATEGORIES
AI Crawler Access ββββββββββ 70 β 2 errors
llms.txt ββββββββββ 92 β 1 warning Β· 1 hint
Structured Data ββββββββββ 88 β 1 warning Β· 3 hints
Citability ββββββββββ 82 β 2 warnings Β· 3 hints
Technical Foundation ββββββββββ 100 β clean
AI CRAWLER ACCESS β 49/51 allowed Β· 2 blocked
OpenAI
GPTBot β training
OAI-SearchBot β search
ChatGPT-User β user-fetch
Perplexity
PerplexityBot β search
Perplexity-User β user-fetch
Google
Googlebot β search
Google-Extended β training
β¦ 51 tokens total, grouped by vendor β¦
FINDINGS
AI Crawler Access
β ai-crawler/search-bots-blocked PerplexityBot is blocked by robots.txt β Perplexity cannot use your pages as AI answer sources
fix: Remove the Disallow covering PerplexityBot in robots.txt, or add an explicit "Allow: /" for it.
evidence: Disallow: / (matched by PerplexityBot)
llms.txt
β llms-txt/missing No llms.txt found
fix: Create /llms.txt at the site root: an H1 title, a short blockquote summary, and ## sections linking to your key content.
evidence: http://localhost:4173/llms.txt β HTTP 404
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
2 errors Β· 4 warnings Β· 7 hints Β· 32/44 checks passed
```
Every finding carries a rule id, a severity, the evidence geolint matched, and a
fix. Compare two pages or two competitors head-to-head:
```bash
geolint check a.com --compare b.com
```
## Commands
| Command | What it does | Key flags |
| --- | --- | --- |
| `geolint check ` | Audit a single URL | `--format`, `--fail-under`, `--only`/`--ignore`/`--category`, `--compare`, `--baseline`, `--badge`, `--verbose` |
| `geolint crawl ` | Crawl same-origin pages and audit the whole site | `--max-pages`, `--max-depth`, `--concurrency`, `--fail-under` |
| `geolint init ` | Crawl the site and generate a `llms.txt` | `-o`, `--max-pages` |
| `geolint diff ` | Compare two JSON reports: score delta, added/resolved findings | β |
| `geolint rules` | List the 52 audit rules | `--category`, `--format table\|json\|markdown` |
| `geolint bots` | List the 51 known AI crawlers and the impact of blocking each | `--format table\|json` |
| `geolint mcp` | Run an MCP server on stdio for AI assistants | `--timeout` |
Full flag reference: [docs/configuration.md](docs/configuration.md).
## Run it in CI
### GitHub Action
```yaml
- uses: iliasabk/geolint@v1
id: geolint
with:
url: https://example.com
fail-under: 80
- uses: github/codeql-action/upload-sarif@v3
if: always()
with:
sarif_file: ${{ steps.geolint.outputs.sarif-file }}
```
The action produces score/grade step outputs, a SARIF report for GitHub code
scanning, and a markdown report for job summaries and PR comments. Full recipes
β SARIF upload, updating a single PR comment, baseline drift detection β in
[docs/github-action.md](docs/github-action.md).
### Any other CI
```bash
npx @iliasabk/geolint check https://example.com --fail-under 80
```
Exit code is `1` when the score drops below the gate (or findings regress
against `--baseline`), `0` otherwise β works in GitLab CI, CircleCI, npm
scripts, pre-deploy hooks.
### Show your score as a README badge
```bash
npx @iliasabk/geolint check https://example.com --badge
# β writes geolint-badge.svg + prints the markdown snippet to paste
```
Commit the SVG, or regenerate a [shields endpoint JSON](docs/badges.md) in CI
(`--badge-endpoint`) for a badge that never goes stale.
## Output formats
`-f pretty` (default) renders the terminal report above. The machine formats:
- `-f json` β the full `ScanReport`: findings, per-category scores, bot access matrix
- `-f sarif` β SARIF 2.1.0, upload straight to GitHub code scanning
- `-f markdown` β PR-comment/job-summary-ready tables
- `-f html` β a self-contained interactive report (score ring, findings filter,
bot matrix) you can share or host anywhere
Add `-o report.json` to write to a file; stdout stays clean for piping.
## geolint on the real web
The repo dogfoods itself: a [nightly workflow](metrics/) re-audits eight
well-known sites and commits the scores back, and the [showcase
site](https://iliasabk.github.io/geolint/) publishes the full interactive
reports β github.com, anthropic.com, stripe.com and more, regenerated on every
push to `main`.
## Programmatic API
```ts
import { scan } from '@iliasabk/geolint';
const report = await scan('https://example.com', {
ignore: ['technical/https'],
timeout: 10_000,
});
console.log(report.score, report.grade); // e.g. 86 'B'
for (const f of report.findings) {
console.log(f.severity, f.ruleId, f.message, f.fix);
}
```
`scan(url, options)` returns a typed `ScanReport`. Also exported: the bot
registry (`AI_BOTS`, `botsByPurpose`), the rule registry (`allRules`,
`ruleById`), robots.txt/llms.txt parsers, badge generators, scorers and all
four reporters.
## Use it from AI assistants (MCP)
`geolint mcp` speaks the [Model Context Protocol](https://modelcontextprotocol.io)
over stdio β Claude Desktop, Cursor, VS Code and Windsurf can audit sites,
generate `llms.txt` and compare URLs as native tools:
```jsonc
// claude_desktop_config.json / ~/.cursor/mcp.json
{
"mcpServers": {
"geolint": {
"command": "npx",
"args": ["-y", "@iliasabk/geolint", "mcp"]
}
}
}
```
Five tools: `audit_url`, `generate_llms_txt`, `compare_urls`, `list_rules`,
`list_ai_bots` β all read-only, with structured output and per-call timeouts.
Setup for every client: [docs/mcp.md](docs/mcp.md).
## The bot registry is the point
`geolint bots` lists 51 AI crawler tokens with a **purpose-aware** impact
assessment β because "should I block this bot?" has a different answer for each:
| Purpose | Examples | If you block it |
| --- | --- | --- |
| `training` | GPTBot, ClaudeBot, CCBot | absent from *future* training data |
| `search` | OAI-SearchBot, PerplexityBot, Claude-SearchBot | invisible in AI answers *now* |
| `user-fetch` | ChatGPT-User, Claude-User | invisible in AI answers *now* |
| `mixed` | Bytespider, Amazonbot, Diffbot | both |
And two nuances other tools miss:
- **Some fetchers ignore robots.txt.** OpenAI, Perplexity and Meta document that
their user-triggered fetchers (ChatGPT-User, Perplexity-User,
Meta-ExternalFetcher) may not honor robots.txt. `ai-crawler/user-fetch-bypass`
tells you when a `Disallow` won't work β enforce at the WAF/auth layer instead.
- **Stale tokens.** `anthropic-ai`, `Claude-Web`, `FacebookBot` are retired.
`ai-crawler/stale-tokens` flags them and names the replacement token β a
`User-agent: anthropic-ai` rule does nothing today.
Control-only tokens like `Google-Extended` and `Applebot-Extended` never fetch
at all β they only set a preference β and geolint treats them accordingly.
## What geolint is honest about
- **llms.txt is a proposal, not a standard.** No major AI vendor has committed
to reading it β so `llms-txt/*` findings are weighted as warnings and hints,
not errors. geolint still checks it (and `geolint init` generates it) because
adoption is growing and the cost is one file.
- **Correlation β causation.** The citability rules are grounded in published
GEO research (quotations/statistics/citations measurably lift share-of-answer;
AI crawlers other than Googlebot and Applebot don't execute JavaScript), but
signals like question-shaped headings are hints, not facts β they're `info`
severity and geolint says so.
- **Every rule shows its reasoning.** [docs/rules.md](docs/rules.md) documents
why each rule exists; the research sources are in
[docs/research-notes.md](docs/research-notes.md), including the vendor docs
behind every bot's robots.txt posture.
- **The bot registry is a standalone reference.**
[docs/ai-crawlers.md](docs/ai-crawlers.md) lists every tracked token with
purpose, per-vendor robots.txt posture and vendor docs β the same data
`geolint bots` and the `list_ai_bots` MCP tool expose.
## Compared to the alternatives
| | Purpose-aware bot registry | Per-vendor robots.txt posture | Runs in CI | Fix per finding | Generates llms.txt | Free / OSS |
| --- | :-: | :-: | :-: | :-: | :-: | :-: |
| **geolint** | β
| β
| β
| β
| β
| β
|
| ai.robots.txt-style blocklists | β | β | n/a | β | β | β
|
| GEO-optimizer skills / prompt packs | β | β | β | β | β | varies |
| llms.txt validators | β | β | some | partial | some | β
|
| Hosted GEO audit web apps | partial | β | β | partial | β | β |
Details and the reasoning behind each column: [docs/comparison.md](docs/comparison.md).
geolint also ships an MCP server, a score badge and regression baselines.
## Roadmap
Planned for v0.4+:
- `geolint watch` β re-audit on deploys/file changes
- Custom rule API for project-specific checks
- Deeper schema coverage (more `@type` validators)
- Homebrew formula
- Report localization beyond English
## Contributing
Issues and PRs welcome β see [CONTRIBUTING.md](CONTRIBUTING.md). New rules are
the best contribution: each needs a `check(ctx)`, findings with `fix`, a test
and a docs entry.
## License
[MIT](LICENSE) Β· [changelog](CHANGELOG.md) Β· [security](SECURITY.md)
---
If geolint helped, a β helps others find it.