# Architecture ## Tech Stack - **Language**: Go 1.26+ (module declares `go 1.26`; the toolchain auto-fetches a matching version on build if the installed `go` is older but supports toolchain switching, i.e. Go 1.21+) - **CLI Framework**: Cobra (`github.com/spf13/cobra`) - **Email**: SMTP only (`internal/email/smtp.go`). SendGrid and Resend were removed - see [auditing.md](auditing.md#dead-code--removed). - **Database**: SQLite (for history tracking via `modernc.org/sqlite`) - **Config**: YAML (`gopkg.in/yaml.v3`) - **Browser automation**: chromedp, for `fill`/`confirm` commands and CAPTCHA detection - **Local agent connection**: official Go MCP SDK, STDIO transport (`delistd mcp`) ## Project Structure ``` delistd/ ├── cmd/delistd/ │ ├── main.go # Root cobra.Command wiring + shared helpers (config/profile/ │ │ # broker path resolution) - each command's own flags and Run │ │ # logic live in their own cmd_*.go file, one per command: │ │ # cmd_init.go, cmd_send.go, cmd_brokers.go (list-brokers/ │ │ # add-broker), cmd_status.go, cmd_bounces.go (mark-bounced/ │ │ # cleanup-bounces), cmd_monitor.go, cmd_pipeline.go, │ │ # cmd_fill.go, cmd_confirm.go, cmd_serve.go, cmd_profile.go ├── internal/ │ ├── broker/broker.go # Broker struct, YAML loading, filtering, add/remove │ ├── browser/ # chromedp automation: form filling, CAPTCHA detection, │ │ # confirmation-link clicking, shared broker-domain allowlist │ │ # validation (domain.go) │ ├── config/config.go # User configuration (profile(s), email, options, inbox, pipeline) │ ├── evidence/evidence.go # evidence report (requests, replies, deadlines, DPA) for `export` and the web download │ ├── email/ │ │ ├── sender.go # SMTP sender constructor, address validation │ │ ├── removal.go # SendRemoval: render + send + history record, shared by CLI and web │ │ └── smtp.go # SMTP implementation │ ├── history/history.go # SQLite history tracking, pipeline status, per-profile scoping │ ├── inbox/ # IMAP scan (scan.go: ScanAndStore/RecordReply) + reply classification (success/form-required/ │ │ # confirmation/rejection/pending/bounced) │ ├── schedule/ # unattended cycles: shared lock + state file, launchd/systemd install │ ├── mcpserver/ # profile-scoped MCP tools, reviewed batches, persistent browser sessions │ ├── template/ │ │ ├── template.go # Template rendering engine │ │ └── templates/ # Embedded: gdpr.tmpl, ccpa.tmpl, generic.tmpl │ └── web/ │ ├── server.go # Server struct, NewServer, router setup, core render/helper │ │ # methods - handlers themselves live in handlers_*.go, grouped │ │ # by resource: handlers_pages.go (dashboard/brokers/history/ │ │ # pipeline/tasks), handlers_api.go (HTMX JSON/fragment │ │ # endpoints), handlers_jobs.go (send-job API + background │ │ # send processing), handlers_settings.go, handlers_setup.go; │ │ # scheduler.go runs `delistd auto --once` for schedule.enabled │ │ # (setup wizard), handlers_profile.go (profile switching) │ ├── job.go # Job/JobManager - background send-job state, mutex-protected │ └── session.go # Setup-wizard session store ├── data/brokers.yaml # 700+ data broker database (embedded via data/embed.go) ├── data/brokers-verified.yaml # smaller registry-sourced list; `send --list verified` ├── scripts/import-registries/ # helper: grow either list from state registry CSVs ├── docs/ # Granular reference docs (this directory) └── EU-NOTES.md # GDPR/EU-specific setup and customization notes ``` CI (`.github/workflows/ci.yml`) runs `go build`/`go vet`/`go test -race` and `golangci-lint` (config: `.golangci.yml`) on every push/PR, plus `zizmor` and `actionlint` (config: `.github/actionlint.yaml`) when workflow files change. `.github/workflows/release-please.yml` watches Conventional Commits on `main` and keeps an open release PR with the version bump and `CHANGELOG.md`; merging it tags `vX.Y.Z` and publishes a GitHub Release. That tag push fires `.github/workflows/release.yml`, which runs GoReleaser (`.goreleaser.yaml`) to cross-compile `linux/darwin/windows × amd64/arm64` (`CGO_ENABLED=0`, pure-Go SQLite) and append archives + checksums to the release release-please already created. A parallel `docker` job in the same workflow builds the distroless `Dockerfile` for `linux/amd64` and `linux/arm64` and pushes it to `ghcr.io/drumandbytes/delistd` (`vX.Y.Z`, `X.Y`, `latest`, with provenance and SBOM; no build cache), and `docker-smoke` pulls and runs it per architecture. Both smoke jobs alarm without gating the release. `main.version` is set via `-ldflags` at that point; it's `"dev"` otherwise. ## Key Concepts ### MCP automation `cmd_mcp.go` exposes `internal/mcpserver` through the official Go MCP SDK. The server is local and serves one client session. The client supplies the AI. `begin_batch` and `retry_action` work only after a person approves a stored preview (`batch_approvals`: single use, expiring, bound to the preview's digest) on the web UI's Automation page or with `delistd mcp approve`; the server then enforces the resulting profile/task/broker scope. Changing settings invalidates that scope. Browser tools use `internal/browser/automation.go`, retaining Chrome between calls and referencing controls from fresh snapshots in an isolated JavaScript world. Destinations are the selected broker's catalog hosts and their `www.` form, nothing broader; on shared platforms (`sharedPlatforms` in `automation.go`) page loads also stay under the catalog entry's form path, and a response from a non-public address stops the session. Request interception checks redirects and form traffic; a cumulative document CSP blocks popups, workers, and out-of-scope connections. This deliberately limits compatibility with embedded forms and external dependencies. `internal/history/automation.go` adds an action journal. An action is claimed before external work; interrupted submissions remain unknown and cannot be automatically replayed. Notes, screenshots, and completion use `pending_tasks`, which gains `actor` (`human` or `mcp:`) and `review_reason`; `removal_requests` gains `pipeline_actor`. Accepted requests remain awaiting a response in the pipeline. Email uses `email.SendRemoval` and the existing profile-scoped limits; inbox replies use the existing classifier with an explicitly resolved owner. See [mcp.md](mcp.md) for operational limits. ### Broker `data/brokers.yaml` is compiled into the binary (`data/embed.go`, `//go:embed`). `broker.Load(override)` resolves it from `--brokers`, then `~/.eraser/brokers.yaml` (written by `delistd update-brokers`, a conditional download), then the embedded copy - no implicit `./data` scanning. `add-broker`/`cleanup-bounces` write to a real path via `resolveBrokerWritePath()`. `data/brokers-verified.yaml` is a second embedded list: a smaller set built from public registries and authoritative privacy notices, where every entry is a confirmed data broker with a current first-party privacy email or rights portal. The send-family commands (`send`, `draft`, `mark-sent`, `serve`) call `broker.LoadList(flag, cfg.Options.BrokerFile, cfg.Options.BrokerList)` instead of `broker.Load`, adding two steps ahead of the `~/.eraser` fallback: an `options.broker_file` path, then the verified list when `options.broker_list` (or `send --list`) is `verified`. Everything else stays on the full list. Grow the verified list with `scripts/import-registries` - see [auditing.md](auditing.md). Each broker in `data/brokers.yaml` (top-level key `brokers:`) has: - `id`: Unique lowercase hyphenated identifier (e.g., `spokeo`, `been-verified`) - `name`: Display name - `email`: Privacy/removal contact email (may be empty string - see below) - `website`: Company website (optional) - `opt_out_url`: Direct opt-out link (optional) - `region`: `us`, `eu`, or `global` - `category`: `people-search`, `marketing`, `background-check`, `financial-b2b`, `data-intermediary`, `device-id-only` (tracks by cookie/device ID, not name/email - can't be reached via this tool's standard profile-based request), or `requires-id` (won't act on a request without a government-issued ID document or similar heavyweight identity verification - this tool won't supply one on your behalf) - `countries`: Optional list of upper-case ISO 3166-1 alpha-2 codes (e.g. `[DE]`). When set, the default selection in `send`, `draft`, `mark-sent`, the web UI (list, send, email page) and the MCP server only includes the broker for profiles whose `country` is one of them. An empty or unrecognised profile country never hides a broker, and naming a broker with `send --broker` bypasses the scope (for example when you have a past address in that country). `list-brokers` shows the scope and the web list marks scoped brokers and says how many are hidden. For brokers that only hold one market's data and ask not to be sent requests from elsewhere - `notes`: Free-text, optional - used to record why an entry looks unusual (e.g. "use the form, not email") **Brokers with `email: ""`** can't be reached by `send` at all - they only take requests through a web form/DSR portal, or the address bounced and was cleared by `cleanup-bounces`/`mark-bounced`. Use `delistd list-brokers --missing-email` to see them; they need manual follow-up outside the tool. ### Templates Three email templates (`internal/template/templates/`): - **GDPR**: Invokes EU Article 17 "Right to Erasure" - the correct default for EU users - **CCPA**: Invokes California Consumer Privacy Act - **Generic**: General privacy request referencing multiple laws ### Web UI static assets `internal/web/static/` (embedded via `//go:embed static/*`) holds: - `js/htmx.min.js` - vendored as-is. - `css/tokens.css` - design tokens (colour / elevation / type custom properties for both themes). The only place raw hex values are named. Linked first in `layout.html`. - `css/utilities.css` - a hand-maintained stylesheet: a preflight reset plus the layout / spacing / sizing / flex / grid utilities the templates use. It began as Tailwind v3.4.17 output, but there is **no Tailwind toolchain** any more - no config, no CLI, no build step. If a template needs a utility class that isn't in the file, add the rule by hand (the spacing scale is 0.25rem steps: `.p-2` = 0.5rem, `.p-4` = 1rem, ...). - Component classes (`.btn`, `.card`, `.badge`, `.nav-link`, ...) live in the `