# tgrep Trigram-indexed grep with a client/server architecture for fast regex search in large codebases. **tgrep is integrated into [GitHub Copilot CLI](https://github.com/github/copilot-cli) to power fast grep searches across large repositories.** ## Why? Tools like `grep` and `ripgrep` scan every file on every search — O(total bytes) per query. In a 100k+ file monorepo, that's painfully slow. tgrep pre-builds a trigram index so searches only touch the small set of files that could match. **Start a server once, search instantly forever.** ```bash tgrep index . # build the trigram index tgrep serve . # start server (watches for file changes) tgrep "fn main" . # instant — auto-connects to running server ``` Using tgrep from an AI coding agent? See [AGENTS.md](AGENTS.md). See [full benchmark results](BENCHMARKS.md) — up to **52x faster** than ripgrep on large repos. ### Benchmark highlights (avg latency per query, index pre-built) | Repo | Files | Platform | ripgrep | tgrep | Speedup | | --- | ---: | --- | ---: | ---: | ---: | | gecko-dev | 388K | macOS arm64 | 33,402ms | 643ms | **51.9x** | | gecko-dev | 388K | Windows | 17,841ms | 463ms | **38.6x** | | gecko-dev | 388K | Linux | 1,195ms | 162ms | **7.36x** | | chromium | 504K | macOS arm64 | 41,806ms | 2,643ms | **15.8x** | | chromium | 504K | Windows | 24,576ms | 1,396ms | **17.6x** | | chromium | 504K | Linux | 2,404ms | 631ms | **3.81x** | | go | 16K | Windows | 592ms | 79ms | **7.53x** | | rust | 62K | Windows | 1,489ms | 194ms | **7.69x** | | kubernetes | 31K | Windows | 1,342ms | 190ms | **7.08x** | | linux | 96K | macOS arm64 | 5,390ms | 256ms | **21.0x** | | linux | 96K | Windows | 3,280ms | 94ms | **34.8x** | | linux | 96K | Linux | 427ms | 46ms | **9.38x** | tgrep wins 17 of the 18 measured cells; the exception is Kubernetes on Linux, a near-tie at 0.93x. The margin depends on repo size and on how many matches a query returns — a search that returns tens of thousands of matches spends more on delivering them than the index saves on finding them. See [What decides the margin](BENCHMARKS.md#what-decides-the-margin). ## Architecture ``` tgrep ---TCP---> tgrep serve (multi-client) (client) | HybridIndex / \ IndexReader LiveIndex (mmap disk) (in-memory overlay) ^ ^ | | Periodic Flush File Watcher (notify) (50K files / Background Indexer 5 min) (rayon parallel) ``` - **IndexReader** — mmap'd on-disk index (zero-copy, binary search on sorted trigram lookup table) - **LiveIndex** — in-memory overlay for files modified after server start, or being built by the background indexer - **HybridIndex** — merges both layers; overlay takes precedence - **Background Indexer** — builds the index in parallel batches of 1,024 files (with additional byte-based splitting); a cold start serves an empty index until the first build is published, while a resumed partial index is processed at 500 files - **Periodic Flush** — every 50K files or 5 minutes, the in-memory index is flushed to disk and the reader is swapped, keeping memory bounded - **Automatic refresh** — native `notify` subscriptions update LiveIndex in real time when available; budget or registration failures switch to polling - **Filename Index** — `--files` unions content-index paths with a compact sidecar containing only admitted paths that have no searchable content - **TCP Server** — JSON-RPC 2.0 over newline-delimited TCP; each connection handled in a separate thread; multiple clients can connect simultaneously - **File Cache** — 50K-entry content cache with RwLock for lock-free reads ## Performance tgrep is designed to be significantly faster than ripgrep on large repos: - **Parallel search** — candidate files are searched in parallel using rayon - **Fast query planning** — sorted posting lists are intersected/unioned without unnecessary resorting, and on-disk posting lists skip redundant deduplication - **Memory-efficient full builds** — index builds batch extraction and stream sorted postings, file entries, and lookup entries instead of retaining the full inverted index in memory - **Smart file walking** — extension-based binary rejection (50+ formats) and an 8KB content check, with a 64 MiB size cap on both indexing and searching (`--no-max-filesize` removes it) - **Lock-free reads** — `RwLock` cache allows concurrent reads without contention - **Hot serving** — queries work immediately during background index building; no need to wait for full index See [BENCHMARKS.md](BENCHMARKS.md) for end-to-end large-repo benchmarks and Criterion microbenchmarks for query execution, trigram extraction, and index building. ## Usage ### Build the index ```bash tgrep index . # index current directory tgrep index /path/to/repo # index a specific repo tgrep index . --index-path /tmp/idx # custom index location tgrep index . --exclude vendor --exclude third_party # skip directories ``` Each build reports its elapsed time and peak memory when it finishes: ``` Index built successfully at /tmp/idx Indexed in 22.6s using external strategy (peak memory 160.1 MiB) ``` The peak is the memory the process itself holds — private/committed bytes, not resident set. The two differ once large files are memory-mapped: mapped pages are file-backed and reclaimable, so counting them would report the size of the files being indexed rather than tgrep's own use. Indexing a single 2 GiB file holds 77.8 MiB while its working set reaches 1.99 GiB. When the working set is substantially larger it is named alongside, so nothing is hidden: ``` Indexed in 46.1s using external strategy (peak memory 77.8 MiB private, 1.99 GiB working set incl. memory-mapped files) ``` #### Repositories without a `.git` directory `.gitignore` files only take effect inside a git repository. This matches ripgrep, which gates them the same way, but it surprises people indexing a Perforce, Source Depot, or plain-directory enlistment: the root `.gitignore` is read by nothing, and the only symptom is an index far larger than expected. tgrep says so rather than leaving you to guess: ``` Walking /src/enlistment... warning: /src/enlistment has a .gitignore but is not a git repository, so it is not applied (this matches ripgrep). Pass --no-require-git to apply it. Found 290018 text files (2893 binary skipped, 0 too large, 0 errors) ``` `--no-require-git` applies the rules anyway, and works on `index`, `serve`, and search alike, so the index and your queries agree on which files exist: ```bash tgrep index . --no-require-git tgrep serve --no-require-git ``` #### Case-insensitive repositories When git clones onto a filesystem that does not distinguish case — which on Windows is every clone — it sets `core.ignorecase` and stops distinguishing case when it matches ignore rules. A rule spelled `QLogs` then hides a directory named `qlogs`. Most tools, ripgrep included, always match ignore rules case-sensitively, so that directory is walked, read and indexed even though `git status` never mentions it. On one Windows enlistment that was a single 13.4 GiB build artifact, 71% of the corpus, adding about 16 seconds to *every* query. tgrep reads `core.ignorecase` and matches the way the repository itself does. Files git **tracks** are exempt, which is git's own rule — ignore rules only decide the fate of files git does not already know about. Without that exemption the same change would have hidden 273 tracked `.JPG`, `.PNG` and `.RLL` files caught by rules written in lower case. On that enlistment the walk went from listing one file more than `git ls-files --cached --others --exclude-standard` to matching it exactly, at a cost of roughly 0.4 s on a 293k-file walk. `--no-ignore` turns it off along with every other ignore source, and repositories that distinguish case are unaffected and pay nothing. Only the repository's own root `.gitignore` and `.git/info/exclude` are matched this way. Rules in nested `.gitignore` files are not, because the walk does not know they exist until it reaches their directory. Missing one only leaves a file visible that git would hide, which is what every other tool does anyway. #### Keep `index` and `serve` flags in step Flags that decide *which files belong in the index* — `--no-require-git`, `--no-ignore`, `--max-filesize`, `--exclude` — must match between the `tgrep index` that built an index and the `tgrep serve` that serves it. The server compares the index against the filesystem at startup and treats an indexed file it cannot see as deleted. So serving an index built without a cap under a `--max-filesize 8M` server drops every file above 8 MiB from that index, permanently. Both sides default to 64 MiB, so this only bites when one side names a limit; pass the same flags to both: ```bash tgrep index . --max-filesize 8M tgrep serve --max-filesize 8M ``` #### Memory use on very large repos Builds default to `--index-strategy=external`, which bounds peak memory with an external merge sort: postings accumulate in a fixed-size arena that spills sorted, compact segments to disk when full, and the segments are k-way merged straight into the index. Peak memory is roughly flat in repo size rather than linear. If the arena never fills, nothing is spilled and the build takes exactly the in-memory path, so small and mid-size repos pay nothing for this default. ```bash tgrep index . # external, 64 MB arena tgrep index . --index-buffer 16 # smaller arena, lower peak tgrep index . --index-strategy=memory # opt out: sort entirely in RAM ``` On the Linux kernel (94,634 files, 990 MiB index), measured under the 1 MiB indexing cap that was the default at the time, the default strategy is a **~17x reduction in peak memory, and no slower**: | Strategy | Spill segments | Peak working set | Build | | --- | ---: | ---: | ---: | | `external` (default, 64 MiB arena) | 31 | **160.1 MiB** | 22.6 s | | `external --index-buffer 16` | 122 | 109.6 MiB | ~23 s | | `memory` | - | 2.20 - 3.76 GiB | 23 - 32 s | `--index-buffer` trades peak memory against merge fan-in. Bounded memory is also *predictable* memory — the `memory` row varied by over a gigabyte across identical runs because `Vec` growth doubles and both buffers are briefly resident during the final reallocation, while `external` varied by 8 MiB. The 1 MiB indexing cap those rows were measured under is now 64 MiB, and files past 1 MiB are memory-mapped rather than read onto the heap. On the same repo the `external` build now settles at roughly **152 MiB of private memory in 27 s**, against 197-200 MiB and 41-42 s when every admitted file was read onto the heap. The arena bound is unchanged; what changed is that a handful of 20 MB generated headers no longer cost their full size in heap in every worker that touches one. The 152 MiB figure was taken with no cap at all, and the 64 MiB default excludes only files *above* 64 MiB, so it leaves these numbers alone here: the largest file in the kernel tree is a 22.9 MiB generated AMD register header, and nothing in its 95,862 files reaches the cap. Two caveats on reading those numbers. The peak tgrep prints is *private bytes*, which excludes mapped file pages, so it reports what the process actually holds rather than the size of the files it is reading — the same build is 152 MiB private against ~192 MiB resident, and the working set is named alongside only when it is substantially larger, as in [Build the index](#build-the-index). macOS is the exception: `libc` does not surface the Mach counter that separates the two, so it still reports resident set there. And a very large file that is neither valid UTF-8 nor detectably binary still costs about its own size, because the index has to hold the same repaired bytes a search will match against; a 135 MB Latin-1 file indexes at roughly 205 MiB. Pass `--max-filesize` if a build has to fit a tighter budget, or `--no-max-filesize` to lift the 64 MiB default entirely. `--index-strategy=memory` remains available as an escape hatch for environments where spilling is undesirable or impossible, such as a read-only or full index volume. Both strategies produce byte-identical indexes from the same walk — note that file IDs follow walk order, which the parallel walker does not fix between runs, so two builds of the same tree need not be byte-identical to each other. See [BENCHMARKS.md](BENCHMARKS.md#index-build-strategies) for full numbers. `tgrep serve` uses the same bounded builder when it has to create an index from scratch, so starting a server on an unindexed repo costs the same memory as `tgrep index` (**148.6 MiB** rather than 1.6 GiB on the Linux kernel, and 2.6x faster). While that first build runs the server answers from an empty index rather than a partial one; incremental updates after it completes are unaffected. ### Start the server ```bash tgrep serve . # start server (auto-builds index if missing) tgrep serve . --index-path /tmp/idx # custom index location tgrep serve . --watch-mode poll # poll without any native subscriptions tgrep serve . --poll-interval 60 # polling cadence after fallback tgrep serve . --watch-budget 4096 # lower this process's native watch ceiling tgrep serve . --no-watch # disable all automatic refresh tgrep serve . --exclude node_modules # exclude directories from indexing ``` The server builds the index in the background if none exists. During that first build, queries are answered from an empty index and return nothing; `tgrep status` reports that indexing is in progress. When the server resumes a partial index instead, queries are answered from the files already indexed. Multiple clients can connect simultaneously. Resource use during that initial build can be tuned. These apply to both `tgrep serve` and `tgrep index`: | Flag | Default | Effect | |------|---------|--------| | `--max-memory ` | 50% of RAM (512 MB–16 GB) | Flush to disk once the in-memory index exceeds this, bounding peak memory | | `--max-cpu ` | `50` | Confine parallel reading and trigram extraction to this share of logical cores | | `--auto-save-mutations ` | `5000` | Accumulated index changes that trigger a background save; higher means fewer pauses but more to redo if killed | | `--watcher-queue-cap ` | `16384` | Filesystem events buffered between the OS watcher and the indexing worker; raise it if bulk changes log watcher queue overflows, since each overflow forces a full stale check | #### Staying in step with the filesystem Automatic refresh has two modes, configured on `tgrep serve`: | Flag | Default | Effect | |------|---------|--------| | `--watch-mode ` | `auto` | Prefer native notifications with polling fallback, or use polling only | | `--poll-interval ` | `120` | Wait after each polling reconciliation completes; range 1-86400 | | `--watch-budget ` | `8192` | Conservative process-local native watch ceiling; range 1-4294967295 | | `--no-watch` | off | Disable all automatic refresh: native watching, polling, and periodic reconciliation | In `auto` mode, exceeding the watch budget, exhausting OS watch capacity, or another error preventing complete native coverage switches the **whole process** to polling. Status retains the specific fallback reason. This fallback is sticky until the server restarts: it releases only this process's native watches and does not keep retrying native registration. `poll` mode starts with **zero native subscriptions**, including on Linux. It uses tgrep's metadata reconciliation, not a second native watcher or `notify::PollWatcher`. The default budget of 8192 is a conservative ceiling, **not an estimate of free per-user capacity**. On Linux the inotify quota is shared with other processes running as the same user; they may consume capacity before this server reaches its budget. Raising the budget does not raise that shared quota. tgrep does not probe the configured Linux quota: it uses this fixed ceiling and authoritative OS registration errors. The budget counts one subscription per admitted directory on Linux/Android; recursive backends count their single root subscription as one. The event buffer controlled by `--watcher-queue-cap` is separate: queue overflow triggers recovery reconciliation, not watch-budget fallback. Polling walks the admitted tree and checks metadata for additions, changes, and deletions. Its default cadence is completion-based: the next poll waits 120 seconds after the previous reconciliation finishes. Actual freshness also includes scan/update time (and any in-progress indexing or save); it is not a 120-second hard freshness guarantee. Searches do not defer polling, and slow scans do not cause overlapping polls or catch-up storms. Fallback does not hide failures: a failed polling reconciliation or unreadable input remains unhealthy and is reported in status. Native notifications can also go missing, for example on a network or virtualised filesystem. Native mode therefore retains its safety reconciliation: about once an hour it walks the tree and compares it against the index, waiting for a two-minute gap in queries and deferring no longer than four hours. `--poll-interval` sets the polling cadence, not this native safety cadence. No-change metadata scans leave the index untouched; they do not rewrite it. A changed delta merge may still rewrite the whole on-disk index. On Linux, nanosecond mtime and ctime plus file identity detect ordinary writes even when the modification timestamp is restored. Other OS/filesystem metadata limitations remain: changes that preserve all available evidence may be missed. A scan and its updates do not provide an atomic filesystem snapshot. `--no-watch` still permits the initial build/startup reconciliation, but turns off all subsequent automatic refresh. It cannot be combined explicitly with `--watch-mode`, `--poll-interval`, or `--watch-budget`. Explicit `--watch-mode poll` also rejects `--watch-budget`, since polling uses no native watches. Inherited default values do not cause conflicts. On Linux and Android, tgrep registers only the non-ignored directories with inotify, avoiding watch-descriptor growth beneath ignored trees. This guarantee is backend-specific: the implementation intentionally keeps one recursive `ReadDirectoryChangesW` root subscription on Windows and one root FSEvents stream on macOS, where ignored events are filtered after delivery and ignored descendants remain watched. kqueue and `PollWatcher` are not covered. ### Search ```bash tgrep "pattern" . # basic regex search tgrep "pattern" file1.rs file2.rs # search multiple files/paths tgrep "TODO|FIXME" . # alternations tgrep '\w+(?!_test)' . # PCRE-style lookahead fallback tgrep "error" . -i # case-insensitive tgrep "error" . -S # smart-case (auto if all lowercase) tgrep -F "Vec" . # literal string tgrep "MyStruct" . -l # filenames only tgrep "pattern" . -c # count per file tgrep "pattern" . -o # only matching text tgrep "pattern" . -w # whole word tgrep "pattern" . -v # invert match tgrep "pattern" . -m 5 # max 5 matches per file tgrep "pattern" . -g "*.rs" # glob filter tgrep "pattern" . -g "*.rs" -g "*.toml" # multiple globs (OR) tgrep "pattern" . -t rust # type filter tgrep "pattern" . -e "also_this" # multiple patterns tgrep "pattern" . -A 3 # 3 lines after match tgrep "pattern" . -B 2 # 2 lines before match tgrep "pattern" . -C 3 # 3 lines before & after tgrep "pattern" . --json # ripgrep-compatible JSON stream tgrep "pattern" . --vimgrep # vim-compatible output tgrep "pattern" . --stats # show query plan & timing tgrep "pattern" . --no-index # brute-force (skip index) tgrep "pattern" . -U # multiline matching tgrep "pattern" . -q # quiet: exit code only tgrep "pattern" . --files-without-match # files that DON'T match tgrep "pattern" . --no-filename # suppress filenames tgrep "pattern" . -N # suppress line numbers tgrep --files . # list searchable files tgrep --files src/main.rs # list a single file if searchable tgrep --files -t rust . # list Rust files only tgrep --type-list # show all file types ``` With the default traversal rules, `--files` reads the live server or the local index instead of walking the repository. The local result is an index snapshot; use `--no-index` to inspect the filesystem as it exists now. Flags that change traversal membership or the file-size policy also fall back to a walk. ### Check status ```bash tgrep status . ``` ``` Server status for /src/my-monorepo PID: 37980 Port: 51043 Files: 152 Trigrams: 12265 Cache: 2/50000 Watcher: active Watch mode: native (requested: auto) Watch budget: 8192 Poll interval: 120s (after completion) Reconcile: idle Last successful reconcile: 2m ago Reconcile pending: no Reconcile overdue: no Last reconcile duration: 42ms Indexing: complete ``` Status retains the native `Watcher` indicator and reports the requested mode (`auto`, `poll`, or `disabled`) and active mode (`native`, `poll`, `disabled`, or `starting`). A polling server normally shows an inactive native watcher. Fallback reasons, the last successful reconciliation, the latest attempt's duration/error, and whether reconciliation is running, pending, or overdue help distinguish a healthy polling server from one that has stopped refreshing. Pending means catch-up work is requested; overdue means that work is pending or the polling interval/native safety deadline has elapsed since completion. Older servers without these fields still display their existing status. ### Count files ```bash tgrep count-files . # count text files (no server needed) tgrep count-files /path/to/repo # scan a specific repo ``` Prints the count to stdout (scriptable) and details to stderr: ``` 284957 284957 text files (47516 binary skipped, 0 errors) in 1200ms ``` ## CLI Flags | Flag | Description | |------|-------------| | `-i, --ignore-case` | Case-insensitive matching | | `-s, --case-sensitive` | Force case-sensitive matching (overrides `-S`) | | `-S, --smart-case` | Case-insensitive if pattern is all lowercase | | `-F, --fixed-strings` | Treat pattern as a literal string | | `-w, --word-regexp` | Match whole words only | | `-v, --invert-match` | Show lines that do NOT match | | `-o, --only-matching` | Print only the matched parts | | `-e, --regexp ` | Additional pattern (repeatable for OR) | | `-f, --file ` | Read patterns from file (one per line) | | `-U, --multiline` | Enable multiline matching (`.` still excludes `\n`) | | `--multiline-dotall` | Make `.` match `\n`; implies `-U` | | `-n, --line-number` | Show line numbers (default: on when stdout is a terminal) | | `-N, --no-line-number` | Suppress line numbers | | `-c, --count` | Print match count per file | | `-l, --files-with-matches` | Print only filenames | | `--files-without-match` | Print files that do NOT match | | `-q, --quiet` | Suppress output; exit code only | | `-m, --max-count ` | Limit matches per file | | `-g, --glob ` | Filter files by glob pattern, case-sensitive (repeatable) | | `--iglob ` | Case-insensitive glob filter (repeatable) | | `--glob-case-insensitive` | Treat all `-g` globs as case-insensitive | | `-t, --type ` | Filter by file type (`rust`, `py`, `js`, …; repeatable) | | `-T, --type-not ` | Exclude a file type (repeatable) | | `--type-add ` | Add/extend a type, e.g. `--type-add 'web:*.html'` | | `--type-clear ` | Remove a type's definitions | | `--type-list` | Print all supported file types (reflects `--type-add`/`--type-clear`) | | `--files` | List files that would be searched | | `-A, --after-context ` | Lines of context after match | | `-B, --before-context ` | Lines of context before match | | `-C, --context ` | Lines of context before and after | | `--heading / --no-heading` | Grouped vs flat output | | `-H, --with-filename` | Show filenames (default: on unless a single file was named) | | `-I, --no-filename` | Suppress filenames in output | | `--json` | ripgrep-compatible JSON stream (one object per line) | | `--vimgrep` | Vim-compatible `file:line:col:content`, one row per match | | `--color auto/always/never` | Color mode control | | `-0, --null` | NUL byte filename separator (for xargs) | | `--trim` | Trim leading/trailing whitespace | | `-., --hidden` | Include hidden files and directories | | `--no-ignore` | Don't respect `.gitignore` or `p4ignore.ini` files | | `-a, --text` | Search binary files as if they were text | | `--binary` | Search binary files, reporting a note instead of their contents | | `-u, --unrestricted` | Unrestricted: `-u` = no-ignore, `-uu` = +hidden, `-uuu` = +binary | | `--max-filesize ` | Skip files larger than `SIZE` (`K`/`M`/`G` suffixes); default 64M | | `--no-max-filesize` | Apply no size limit, as ripgrep does | | `-L, --follow` | Follow symbolic links | | `--no-messages` | Suppress error messages about unreadable/missing paths | | `--no-index` | Skip index, grep all files | | `--exclude ` | Exclude directory from indexing (repeatable); `index` and `serve` only, not accepted by a search | | `--stats` | Print query plan and candidate stats | | `--index-path ` | Custom index directory | **Pattern matching** | Flag | Description | |------|-------------| | `-x, --line-regexp` | The pattern must match a whole line (beats `-w`) | | `-P, --pcre2` | Use the backtracking engine (lookaround, backreferences) | | `--engine ` | Pick the regex engine explicitly; `auto` falls back to `pcre2` | | `--pcre2-version` | Print the backtracking engine in use and exit | | `--no-unicode` | Disable Unicode-aware character classes | | `--regex-size-limit ` | Cap the compiled regex size (`K`/`M`/`G` suffixes) | | `--dfa-size-limit ` | Cap the regex DFA cache size | | `-r, --replace ` | Replace each match; `$1`/`${name}` expand capture groups | | `--passthru` | Print every line, matching or not | | `--stop-on-nonmatch` | Stop searching a file at its first non-matching line | **Output formatting** | Flag | Description | |------|-------------| | `--column` / `--no-column` | Show the 1-based column of the first match | | `-b, --byte-offset` | Show the byte offset of the line (or match, with `-o`) | | `-M, --max-columns ` | Omit lines longer than `N` bytes | | `--max-columns-preview` | Show a truncated preview instead of omitting | | `--count-matches` | Count matches rather than matching lines | | `--include-zero` | With `-c`, also print files with a count of `0` | | `-p, --pretty` | Alias for `--color always --heading -n` | | `--context-separator ` | Separator between context groups (default `--`) | | `--no-context-separator` | Print no separator between context groups | | `--field-match-separator ` | Separator between match fields (default `:`) | | `--field-context-separator ` | Separator between context fields (default `-`) | | `--path-separator ` | Rewrite the separator in printed paths | | `--sort ` / `--sortr ` | Sort by `path`/`modified`/`accessed`/`created`/`none` | | `--sort-files` | Shorthand for `--sort path` | | `--line-buffered` / `--block-buffered` | Force line- or block-buffered stdout | **Encoding** | Flag | Description | |------|-------------| | `-E, --encoding