## Token Savings: LSP vs Grep/Read

How much context does an AI agent consume when navigating code with grep/read
versus structured LSP calls? This experiment measures the bytes flowing through
each approach on real codebases across three languages and four projects.

### Key findings

| Codebase | Language | Lines | Overall | `/lsp-rename` | Precision | Tokens saved |
|----------|----------|------:|--------:|--------------:|----------:|-------------:|
| agent-lsp | Go | 15K | **5x** | **96x** | 92% noise | ~1.3M |
| Hono | TypeScript | 24K | **13x** | **1,441x** | 93% noise | ~1.2M |
| FastAPI | Python | 33K | **2x** | **116x** | 97% noise | ~693K |
| Next.js | TypeScript | 196K | **5x** | **52x** | 98% noise | ~1.1M |
| HashiCorp Consul | Go | 319K | **34x** | **97x** | 99% noise | ~13.1M |

- **`/lsp-rename` saves 92-1,441x consistently.** The grep agent must read every file
  containing the symbol to safely rename it; LSP does it atomically in 3 calls.
- **92-99% of grep results are false positives.** Grep matches strings; LSP returns
  only actual symbol references. On consul (319K lines), grep for `Close` returned
  1,156 matches; only 12 are real references.
- **Savings scale with codebase size.** 5x at 15K lines, 21x at 319K lines.
- **Speculative execution (`/lsp-safe-edit`): 60x.** Preview an edit without touching
  disk. The grep agent must edit, build (1.3s), revert, re-build. LSP answers in 2ms.
- **Code Map (`/lsp-understand`): 6x, 936 grep calls reduced to 162.** Full file
  analysis with hover, references, and call hierarchy for every exported symbol.
- **Edit safety: 23-656x.** Structured diagnostics vs raw compiler output.
- **Interface implementations: 1,002-1,813x** (Go). Grep cannot determine which types
  satisfy an interface without reading every file.
- **Multi-hop call chains: 9-12x, 87-311x faster** (Go).

**Methodology:**
- **Grep/Read** = total bytes of grep output + file content read + build/test output.
- **LSP** = JSON response bytes from each LSP call (normalized to compact relative
  paths, matching what agent-lsp returns to clients).
- Token estimate: 1 token per 4 bytes (standard approximation for code).
- LSP startup cost excluded (amortizes over a session).
- Go: 13 tasks covering 7 agent skills. Python/TypeScript: 11 tasks.
- Skill workflows test the full multi-step sequence (e.g. `/lsp-refactor` = blast-radius +
  rename + verify), not individual tool calls. The grep side models the equivalent
  multi-step agent workflow for the same task.
- Generated by `go run ./experiments/token-savings`.

---

### agent-lsp (15,850 lines, 82 files)

**Simple tasks**

| Task | Grep/Read | LSP | Ratio | Round trips | Time |
|------|----------:|----:|------:|------------:|-----:|
| Find callers of `Shutdown` | 3,115 | 1,133 | **3x** | 1 vs 1 | 79ms vs 2ms |
| Type signature of `Shutdown` | 4,872 | 393 | **12x** | 1 vs 1 | 54ms vs 0ms |
| Edit safety check (break build, measure output) | 75,740 | 1,171 | **65x** | 3 vs 3 | 0ms vs 0ms |

**Skill workflows and advanced tasks (10 tasks)**

| Task | Grep/Read | LSP | Ratio | Round trips | Time |
|------|----------:|----:|------:|------------:|-----:|
| Skill: `/lsp-refactor` rename `Shutdown` | 28,304 | 30,733 | **1x** | 17 vs 5 | 0ms vs 0ms |
| Skill: `/lsp-impact` on `client.go` (52 exports) | 2,082,537 | 419,978 | **5x** | 795 vs 109 | 0ms vs 0ms |
| Skill: `/lsp-rename` `Shutdown` (14 files) | 220,455 | 2,285 | **96x** | 16 vs 3 | 0ms vs 0ms |
| Skill: `/lsp-dead-code` on `client.go` (52 exports, 5 dead) | 679,099 | 239,761 | **3x** | 57 vs 57 | 0ms vs 0ms |
| Skill: `/lsp-understand` Code Map of `client.go` (52 exports) | 3,250,695 | 554,820 | **6x** | 936 vs 162 | 924ms vs 30ms |
| Skill: `/lsp-safe-edit` (speculative edit + verify + revert) | 75,740 | 1,269 | **60x** | 4 vs 3 | 1241ms vs 2ms |
| Skill: `/lsp-verify` (diagnostics + build + tests) | 75,804 | 27,406 | **3x** | 3 vs 3 | 5157ms vs 0ms |
| Precision: `Close` (61 grep vs 5 LSP refs, 56 false+) | 4,958 | 517 | **10x** | 1 vs 1 | 59ms vs 22ms |
| Multi-hop: callers of callers of `Shutdown` | 41,680 | 4,637 | **9x** | 25 vs 2 | 1097ms vs 2ms |
| Interface: implementations of `logSender` (1 found) | 98,221 | 98 | **1002x** | 4 vs 1 | 171ms vs 27ms |

**Total: 6,641,220 grep/read vs 1,284,201 LSP = 5x savings (~1,339,254 tokens saved)**

### Hono (24,178 lines TypeScript, 185 files)

**Simple tasks**

| Task | Grep/Read | LSP | Ratio | Round trips | Time |
|------|----------:|----:|------:|------------:|-----:|
| Find callers of `Variables` | 7,205 | 84 | **86x** | 1 vs 1 | 59ms vs 1ms |
| Type signature of `Variables` | 0 | 168 | **0x** | 1 vs 1 | 37ms vs 1ms |
| Edit safety check (break build, measure output) | 79,359 | 121 | **656x** | 3 vs 3 | 0ms vs 0ms |

**Skill workflows and advanced tasks (8 tasks)**

| Task | Grep/Read | LSP | Ratio | Round trips | Time |
|------|----------:|----:|------:|------------:|-----:|
| Skill: `/lsp-refactor` rename `Variables` | 49,578 | 447 | **111x** | 27 vs 5 | 0ms vs 0ms |
| Skill: `/lsp-impact` on `types.ts` (41 exports) | 1,625,162 | 129,096 | **13x** | 610 vs 86 | 0ms vs 0ms |
| Skill: `/lsp-rename` `Variables` (24 files) | 492,954 | 342 | **1441x** | 26 vs 3 | 0ms vs 0ms |
| Skill: `/lsp-dead-code` on `types.ts` (41 exports, 8 dead) | 694,757 | 129,014 | **5x** | 45 vs 45 | 0ms vs 0ms |
| Skill: `/lsp-understand` Code Map of `types.ts` (41 exports) | 1,922,818 | 142,249 | **14x** | 748 vs 128 | 475ms vs 14ms |
| Skill: `/lsp-safe-edit` (speculative edit + verify + revert) | 79,359 | 218 | **364x** | 4 vs 3 | 201ms vs 1ms |
| Skill: `/lsp-verify` (diagnostics + build + tests) | 86,227 | 68 | **1268x** | 3 vs 3 | 930ms vs 0ms |
| Precision: `Close` (15 grep vs 1 LSP refs, 14 false+) | 1,051 | 91 | **12x** | 1 vs 1 | 33ms vs 6ms |

**Total: 5,038,470 grep/read vs 401,898 LSP = 13x savings (~1,159,143 tokens saved)**

### fastapi-bench (32,564 lines, 621 files)

**Simple tasks**

| Task | Grep/Read | LSP | Ratio | Round trips | Time |
|------|----------:|----:|------:|------------:|-----:|
| Find callers of `middleware` | 7,928 | 572 | **14x** | 1 vs 1 | 91ms vs 53ms |
| Type signature of `middleware` | 0 | 156 | **0x** | 1 vs 1 | 91ms vs 0ms |
| Edit safety check (break build, measure output) | 181,470 | 4,200 | **43x** | 3 vs 3 | 0ms vs 0ms |

**Skill workflows and advanced tasks (8 tasks)**

| Task | Grep/Read | LSP | Ratio | Round trips | Time |
|------|----------:|----:|------:|------------:|-----:|
| Skill: `/lsp-refactor` rename `middleware` | 24,665 | 4,202 | **6x** | 25 vs 5 | 0ms vs 0ms |
| Skill: `/lsp-impact` on `routing.py` (38 exports) | 1,460,499 | 802,062 | **2x** | 418 vs 79 | 0ms vs 0ms |
| Skill: `/lsp-rename` `middleware` (22 files) | 416,042 | 3,601 | **116x** | 24 vs 3 | 0ms vs 0ms |
| Skill: `/lsp-dead-code` on `routing.py` (38 exports, 22 dead) | 904,821 | 626,555 | **1x** | 41 vs 41 | 0ms vs 0ms |
| Skill: `/lsp-understand` Code Map of `routing.py` (38 exports) | 1,482,337 | 825,217 | **2x** | 346 vs 118 | 1454ms vs 146ms |
| Skill: `/lsp-safe-edit` (speculative edit + verify + revert) | 181,470 | 4,297 | **42x** | 4 vs 3 | 0ms vs 3ms |
| Skill: `/lsp-verify` (diagnostics + build + tests) | 379,102 | 3,659 | **104x** | 4 vs 3 | 0ms vs 0ms |
| Precision: `validate` (64 grep vs 2 LSP refs, 62 false+) | 6,465 | 204 | **32x** | 1 vs 1 | 86ms vs 24ms |

**Total: 5,044,799 grep/read vs 2,274,725 LSP = 2x savings (~692,518 tokens saved)**

### consul-bench (319,072 lines, 1468 files)

**Simple tasks**

| Task | Grep/Read | LSP | Ratio | Round trips | Time |
|------|----------:|----:|------:|------------:|-----:|
| Find callers of `GetNode` | 10,815 | 5,956 | **2x** | 1 vs 1 | 315ms vs 27ms |
| Type signature of `GetNode` | 20,817 | 464 | **45x** | 1 vs 1 | 381ms vs 0ms |
| Edit safety check (break build, measure output) | 173,498 | 7,477 | **23x** | 3 vs 3 | 0ms vs 0ms |

**Skill workflows and advanced tasks (10 tasks)**

| Task | Grep/Read | LSP | Ratio | Round trips | Time |
|------|----------:|----:|------:|------------:|-----:|
| Skill: `/lsp-refactor` rename `GetNode` | 46,874 | 14,055 | **3x** | 21 vs 5 | 0ms vs 0ms |
| Skill: `/lsp-impact` on `catalog.go` (57 exports) | 17,745,627 | 841,371 | **21x** | 5534 vs 119 | 0ms vs 0ms |
| Skill: `/lsp-rename` `GetNode` (18 files) | 802,815 | 8,286 | **97x** | 20 vs 3 | 0ms vs 0ms |
| Skill: `/lsp-dead-code` on `catalog.go` (57 exports, 22 dead) | 7,133,669 | 348,452 | **20x** | 62 vs 62 | 0ms vs 0ms |
| Skill: `/lsp-understand` Code Map of `catalog.go` (57 exports) | 27,288,671 | 315,737 | **86x** | 6661 vs 178 | 5424ms vs 473ms |
| Skill: `/lsp-safe-edit` (speculative edit + verify + revert) | 173,498 | 7,575 | **23x** | 4 vs 3 | 25829ms vs 28ms |
| Skill: `/lsp-verify` (diagnostics + build + tests) | 173,557 | 1,551 | **112x** | 3 vs 3 | 17149ms vs 0ms |
| Precision: `Close` (1156 grep vs 12 LSP refs, 1144 false+) | 83,702 | 1,337 | **63x** | 1 vs 1 | 281ms vs 223ms |
| Multi-hop: callers of callers of `GetNode` | 134,480 | 15,901 | **8x** | 29 vs 2 | 3687ms vs 33ms |
| Interface: implementations of `ExportFetcher` (1 found) | 177,668 | 98 | **1813x** | 4 vs 1 | 312ms vs 134ms |

**Total: 53,965,691 grep/read vs 1,568,260 LSP = 34x savings (~13,099,357 tokens saved)**

---

### Reproduce

```bash
# Run on any Go project (13 tasks, 7 skills)
go run ./experiments/token-savings --root /path/to/go/project

# Run on any Python project (11 tasks, 7 skills)
go run ./experiments/token-savings --root /path/to/python/project --language python

# Run on any TypeScript project (11 tasks, 7 skills)
go run ./experiments/token-savings --root /path/to/ts/project/src --language typescript

# Append results to the doc
go run ./experiments/token-savings --root /path/to/project --output docs/token-savings.md
```

Prerequisites: language server on PATH (`gopls`, `pyright-langserver`, or `typescript-language-server`).

Source: [`experiments/token-savings/main.go`](../experiments/token-savings/main.go)
### src (196,523 lines, 1201 files)

**Simple tasks**

| Task | Grep/Read | LSP | Ratio | Round trips | Time |
|------|----------:|----:|------:|------------:|-----:|
| Find callers of `RouteCacheEntry` | 9,014 | 352 | **26x** | 1 vs 1 | 138ms vs 1ms |
| Type signature of `RouteCacheEntry` | 7,896 | 310 | **25x** | 1 vs 1 | 114ms vs 1ms |
| Edit safety check (break build, measure output) | 110,830 | 6,386 | **17x** | 3 vs 3 | 0ms vs 0ms |

**Skill workflows and advanced tasks (8 tasks)**

| Task | Grep/Read | LSP | Ratio | Round trips | Time |
|------|----------:|----:|------:|------------:|-----:|
| Skill: `/lsp-refactor` rename `RouteCacheEntry` | 22,660 | 7,000 | **3x** | 9 vs 5 | 0ms vs 0ms |
| Skill: `/lsp-impact` on `cache.ts` (43 exports) | 1,493,510 | 649,755 | **2x** | 477 vs 90 | 0ms vs 0ms |
| Skill: `/lsp-rename` `RouteCacheEntry` (6 files) | 345,596 | 6,635 | **52x** | 8 vs 3 | 0ms vs 0ms |
| Skill: `/lsp-dead-code` on `cache.ts` (43 exports, 0 dead) | 698,926 | 641,104 | **1x** | 47 vs 47 | 0ms vs 0ms |
| Skill: `/lsp-understand` Code Map of `cache.ts` (43 exports) | 1,958,789 | 803,213 | **2x** | 612 vs 134 | 1871ms vs 55ms |
| Skill: `/lsp-safe-edit` (speculative edit + verify + revert) | 111,343 | 6,483 | **17x** | 4 vs 3 | 1778ms vs 1ms |
| Skill: `/lsp-verify` (diagnostics + build + tests) | 111,963 | 6,066 | **18x** | 3 vs 3 | 1847ms vs 0ms |
| Precision: `Close` (158 grep vs 3 LSP refs, 155 false+) | 13,279 | 357 | **37x** | 1 vs 1 | 103ms vs 93ms |

**Total: 4,883,806 grep/read vs 2,127,661 LSP = 2x savings (~689,036 tokens saved)**