# bigfiles [![CI](https://img.shields.io/github/actions/workflow/status/Par-python/bigfiles/ci.yml?branch=main&style=flat-square&logo=githubactions&logoColor=white&label=CI&labelColor=24292e)](https://github.com/Par-python/bigfiles/actions/workflows/ci.yml) [![crates.io](https://img.shields.io/crates/v/bigfiles.svg?style=flat-square&logo=rust&logoColor=white&labelColor=24292e&color=orange)](https://crates.io/crates/bigfiles) [![Downloads](https://img.shields.io/crates/d/bigfiles.svg?style=flat-square&logo=docsdotrs&logoColor=white&labelColor=24292e&color=blue)](https://crates.io/crates/bigfiles) [![Stars](https://img.shields.io/github/stars/Par-python/bigfiles.svg?style=flat-square&logo=github&logoColor=white&labelColor=24292e&color=yellow)](https://github.com/Par-python/bigfiles/stargazers) [![License](https://img.shields.io/badge/license-AGPL--3.0-blue.svg?style=flat-square&logo=gnu&logoColor=white&labelColor=24292e)](LICENSE) A small Rust CLI that walks a directory in parallel, groups files by type, flags stale ones, finds duplicates (hardlink-aware), and renders a color-coded summary in the terminal. Cross-platform: Linux, macOS, Windows. https://github.com/user-attachments/assets/0f4b9c85-b7b4-4a77-8871-b17900c02dfc ## What it does - **Interactive TUI** (`bigfiles tui`) — ncdu-style directory browser with arrow-key navigation, `/` filter, `o` to reveal in OS file manager, `d` to send to Trash, `D` for dupes in the current subtree, `r` to re-scan - **Quick audit** (`bigfiles audit`) — severity-coded "what's eating your disk" insights in one screen - **`bigfiles top`** — flat list of the N largest files, no category grouping - **Safe by default**: `delete` and `dupes --delete` send to Trash by default; `--force` opts into permanent deletion - Walks a directory tree **in parallel** and collects file sizes, extensions, and modified timestamps - Respects `.gitignore` and `.ignore` files by default (use `--no-ignore` to disable) - Skips symlinks (no double-counting, no follow-link footguns) - Groups files into categories: video, images, archives, audio, documents, code, junk, other - Flags files not modified in the last N years as stale - Renders a color-coded table with size bars, optionally with the largest files per category - **Sortable** category table (`--sort size|count|stale-size|stale-count|name`, `--reverse`) - Finds duplicate files by content hash with **parallel BLAKE3 hashing**, **hardlink awareness**, and a **persistent on-disk cache** so re-scans are near-instant - Interactively deletes stale files **or** duplicate copies with explicit confirmation - Emits JSON for piping into other tools - Colorized `--help` output via clap styles ## Install ### crates.io (requires Rust via [rustup](https://rustup.rs)) ```bash cargo install bigfiles ``` To upgrade: ```bash cargo install bigfiles --force ``` ### Pre-built binaries Download from the [releases page](https://github.com/Par-python/bigfiles/releases) for Linux (x86_64, aarch64), macOS (Intel, Apple Silicon), and Windows (x86_64). Extract and move `bigfiles` (or `bigfiles.exe`) onto your `$PATH`. **From source:** ```bash git clone https://github.com/Par-python/bigfiles cd bigfiles cargo install --path . ``` ## Usage ```bash # Scan current directory bigfiles # Scan a specific path bigfiles ~/Downloads # Skip hidden files and dirs, only descend 3 levels bigfiles ~ --skip-hidden --depth 3 # Show the 5 largest files per category alongside the summary bigfiles ~/Downloads --top 5 # Exclude paths via glob (repeatable) bigfiles ~ --exclude 'node_modules' --exclude '*.log' --exclude 'target' # Don't respect .gitignore / .ignore bigfiles ~/some-project --no-ignore # Treat anything not modified in 5+ years as stale (default: 2) bigfiles ~/Documents --stale-years 5 # Pipe JSON into jq (envelope: { version, root, total_size, skipped, categories }) bigfiles ~/Movies --json | jq '.categories[] | select(.stale_size > 1000000000)' # Sort the breakdown by file count instead of size; reverse for smallest-first bigfiles ~ --sort count bigfiles ~ --sort size --reverse # Quick "what's eating my disk" insights view bigfiles audit ~ # Show just the 10 largest files anywhere under the path bigfiles top ~/Downloads -n 10 # Send stale files to Trash (default), or permanently delete with --force bigfiles delete ~/Downloads bigfiles delete ~/Downloads --force ``` ### .gitignore awareness By default bigfiles uses [the same `ignore` crate that ripgrep uses](https://crates.io/crates/ignore), so `.gitignore`, `.ignore`, and global git excludes are respected automatically. Scanning a Rust project? `target/` is skipped. Node project? `node_modules` is skipped. No flag needed. Use `--no-ignore` to walk everything regardless. ### Interactive TUI `bigfiles tui ` opens a full-screen ncdu-style directory browser. Sizes are aggregated per directory; the largest entries float to the top. ```bash bigfiles tui ~ ``` Keys: - `↑/↓` or `j/k` — move - `Enter` or `→` — descend into directory - `←` or `Backspace` — go up - `/` — filter children by substring (`Esc` cancels, `Enter` keeps) - `o` — reveal selected entry in your OS file manager (`open -R` on macOS, `explorer /select,` on Windows, `xdg-open` on Linux) - `d` — send the selected file or directory to Trash (yellow confirm bar appears; `y`/Enter confirms, any other key cancels). Trash-only in the TUI for safety. For permanent delete, use `bigfiles delete --force` or `bigfiles dupes --delete --force` from the CLI. - `D` — open a duplicate-detection popup scoped to the currently-highlighted directory. Uses the persistent hash cache, so repeat runs over the same subtree are near-instant. `j/k` or PgUp/PgDn to scroll, `Esc`/`q` to close. - `r` — re-run the scan from disk (the TUI exits briefly, shows the spinner, then re-enters with fresh data) - `q`/`Esc` — quit - `?` — toggle help ### Quick audit `bigfiles audit ` runs a normal scan, then prints a short list of severity-coded insights about what's eating your disk — heaviest category, top extensions, installer-junk total (`.dmg`/`.pkg`/`.iso`/`.exe`/`.msi`/`.deb`/`.rpm`), top-N-file concentration, and the share of stale data. Useful as a first-run "where do I start?" view. ```bash bigfiles audit ~ ``` Insights are bulleted by severity: red `!` for heavy (≥40% of total), yellow `•` for notable (≥20%), dimmed `·` for informational. Respects `--stale-years` and all global filters (`--skip-hidden`, `--exclude`, `--depth`, etc.). ### Top N largest files `bigfiles top ` prints a flat list of the N largest files under the given path, sorted by size descending. No category grouping, no bars, no stale flags — just the biggest files. Pairs well with `| head`, `| grep`, or pipelines. ```bash # Default: top 20 bigfiles top ~/Downloads # Top 5 bigfiles top ~/Movies -n 5 # Pipe into other tools bigfiles top ~ -n 100 | grep '\.mp4' ``` Respects all global filters (`--skip-hidden`, `--exclude`, `--depth`, `--no-ignore`). ### Find duplicate files `bigfiles dupes` finds files with identical content. It uses a fast three-stage check, parallelized with `rayon`: 1. Group by size 2. Hash first/last 4 KB (`partial_hash`) 3. Full BLAKE3 hash on remaining candidates Hardlinks are collapsed by inode before hashing, so multiple paths pointing to the same on-disk file are reported as a single entry (and don't inflate "reclaimable" numbers). When a duplicate group includes hardlinks, the additional paths are shown indented under the primary path. ```bash # Find dupes >= 1 MB in Downloads bigfiles dupes ~/Downloads --min-size 1048576 # Default min-size is 1 KB; tune as needed bigfiles dupes ~/Documents --min-size 1 ``` #### Remove duplicate copies (interactive) `bigfiles dupes --delete` walks each duplicate group and lets you pick which copy to **keep**; the rest are queued for removal. After all groups are processed, you get a summary and a `y/N` confirm before any file is touched. By default, removed copies are **moved to your OS Trash** (recoverable). Pair with `--force` to delete permanently. ```bash # Default: moves duplicate copies to Trash bigfiles dupes ~/Downloads --delete # Permanent removal (not recoverable) bigfiles dupes ~/Downloads --delete --force ``` Safety guarantees: - Per-group single-choice picker — you can only remove by *not picking* one to keep - Every group offers a "skip — keep all" option; `Esc` also skips - Always keeps ≥1 copy per group (it's structurally impossible to empty a group) - No removal happens until the final `y/N` confirm; default is **No** - Files are re-stat'd immediately before removal; non-regular files (symlinks, sockets, devices) are refused - **Trash by default**: moved copies can be restored from your OS Trash unless you pass `--force` - If Trash is unavailable (e.g. on some network mounts), the operation refuses and points you at `--force` Note that dupes are only ever paired *within* the scan root. If two copies live in separate trees, scan a common parent. #### Persistent hash cache `bigfiles dupes` caches full-file BLAKE3 hashes to your OS cache directory so subsequent runs over the same tree are near-instant. - **Location**: `~/Library/Caches/bigfiles/hashes.json` (macOS), `$XDG_CACHE_HOME/bigfiles/hashes.json` (Linux), `%LOCALAPPDATA%\bigfiles\Cache\hashes.json` (Windows) - **Cache key**: `(path, mtime, size)` — any change invalidates the entry, forcing a re-hash. Tagged with the hash algorithm so the cache survives version bumps. - **Pruning**: entries for paths that no longer exist are dropped on the next save. - `--no-cache` runs without touching the cache (no read, no write). - `--clear-cache` deletes the cache file before running. In local testing, the warm cache is roughly 40× faster than a cold first run on the same tree. ### Remove stale files (interactive) `bigfiles delete` shows you every file older than `--stale-years` (default 2) in an interactive checklist. You tick which ones to remove, see a confirmation summary, and only then are files touched. By default, selected files are **moved to your OS Trash** (recoverable). Pair with `--force` to delete permanently. ```bash # Default: moves stale files to Trash bigfiles delete ~/Downloads --stale-years 3 # Permanent removal bigfiles delete ~/Downloads --stale-years 3 --force ``` The flow: list → tick boxes (Space) → Enter → review summary → type `y` to confirm. Hit Ctrl-C any time to bail. If Trash is unavailable, the operation refuses and points you at `--force`. ### Flags (global) | Flag | Default | Description | |---|---|---| | `` | `.` | Directory to scan | | `-s, --stale-years ` | `2` | Flag files not modified in this many years as stale | | `-H, --skip-hidden` | off | Skip dotfiles and dot-directories | | `-d, --depth ` | unlimited | Limit traversal depth (1 = only files directly in root) | | `--no-ignore` | off | Do not respect `.gitignore` / `.ignore` files | | `--no-pager` | off | Don't auto-page output through `$PAGER` | | `-e, --exclude ` | none | Skip files/dirs matching this glob; repeatable | | `--units