# Migrating from BBOT 2.x to 3.0 BBOT 3.0 is a major release with a wide-ranging cleanup of the CLI, preset syntax, event API, and module ecosystem. This page enumerates everything that changed in a backwards-incompatible way so existing scans, custom modules, and integrations can be updated. If a change is missing here, please open an issue. --- ## CLI / Targeting The `--whitelist` concept was retired. BBOT now distinguishes between **target** (what defines scope, i.e. what `in_target()` checks) and **seeds** (the events that actually drive passive modules at scan start). | 2.x | 3.0 | |-----|-----| | `-t / --targets` (seed + scope) | `-t / --targets` (scope only) | | `-w / --whitelist` | removed (positional `-t` now plays this role) | | *(no equivalent)* | `-s / --seeds` (optional override of seed events) | | `-s / --silent` | `-S / --silent` (capitalized; `-s` reassigned to `--seeds`) | | `--allow-deadly` | removed (see flag changes below) | !!! warning "The `-s` flag changed meaning" In 2.x `-s` was `--silent`; in 3.0 it's `--seeds`, and silent moved to `-S`. A script still passing `-s` for quiet output will now be adding a seed instead, with no error. Behavior: - If `--seeds` is omitted, seeds default to whatever was passed to `--targets`, so the simple case (`bbot -t evilcorp.com -p subdomain-enum`) is unchanged. - `--strict-scope` now means "only this exact host, no subdomains" against the target, not the whitelist. - The `Preset(...)` Python API mirrors the CLI: positional args are the **target**, and `seeds=` is a keyword. `whitelist=` is gone. ### Python API ```python # 2.x preset = Preset("evilcorp.com", whitelist=["evilcorp.com", "evilcorp.net"]) scan.whitelist # Target object scan.whitelisted(host) # 3.0 preset = Preset("evilcorp.com", "evilcorp.net", seeds=["evilcorp.com"]) scan.target # BBOTTarget wrapper scan.target.target # the radix-backed scope scan.target.seeds # seed events scan.in_target(host) ``` `Scanner.whitelist` and `Scanner.whitelisted()` have been removed. `BBOTTarget.whitelist` is now `BBOTTarget.target`. Pickling of `BBOTTarget` was also removed. --- ## Flags ### Renamed | 2.x | 3.0 | |----------------------|--------------------| | `web-basic` | `web` | | `web-thorough` | `web-heavy` | ### Removed | Removed | Replacement / notes | |---------------------|---------------------| | `aggressive` | use `loud` (network volume) and/or `invasive` (destructive) | | `deadly` | dropped along with `--allow-deadly`; affected modules now live under regular flags | ### Added | Added | Meaning | |-------------|---------| | `invasive` | Intrusive or potentially destructive | | `safe` | Non-intrusive and non-destructive (now enforced on every module) | | `download` | Modules that download files, apps, or repositories | Every module must now declare at least one of `passive` / `active` **and** at least one of `safe`, `loud`, or `invasive`. Custom modules carrying the removed flags will fail validation. --- ## Presets ### Renamed | 2.x | 3.0 | |----------------------|--------------------| | `web-basic` | `web` | | `web-thorough` | `web-heavy` | | `nuclei-intense` | `nuclei-heavy` | | `spider-intense` | `spider-heavy` | | `baddns-intense` | `baddns-heavy` | | `dirbust-light` | `webbrute` | | `dirbust-heavy` | `webbrute-heavy` | | `lightfuzz-medium` | `lightfuzz` | | `lightfuzz-superheavy` | `lightfuzz-max` | ### Added `baddns`, `waf-bypass`, `wayback`, `wayback-heavy`, `web/paramminer-heavy`, `web/virtualhost`, `web/virtualhost-heavy`. ### Syntax Preset YAML files now accept the singular key `target:` in addition to `targets:` (both are merged). The old `whitelist:` key is gone; use `seeds:` if you need seeds that differ from the target. Lines beginning with `#` inside target / seed / blacklist lists are stripped as comments. File paths in those lists are resolved relative to the preset file via the new `PresetPath` mechanism. The top-level scan options that used to be implicit are now required to live under `config:` inside a preset (this was previously documented behavior, now enforced and called out in `defaults.yml`). Presets and `-c key=value` options are now **validated when loaded**: unknown keys, typos, and wrong value types are rejected up front (usually with a closest-match hint, e.g. `Did you mean "scope.strict"?`) instead of being silently ignored. The old `${env:VAR}` interpolation inside config values is gone too; inject secrets with shell expansion (`-c modules.shodan.api_key="$SHODAN_KEY"`) or keep them in `secrets.yml`. --- ## Modules ### Removed (no direct replacement) - `smuggler` - `wpscan` ### Removed and replaced | 2.x | 3.0 replacement | |------------------|-----------------| | `httpx` | `http` | | `ffuf` | `webbrute` | | `ffuf_shortnames`| `webbrute_shortnames` | | `extractous` | `kreuzberg` | | `output.http` | `output.webhook` | | `vhost` | `virtualhost` (rebuilt; emits the new `VIRTUAL_HOST` event) | If you used any of these in a custom preset or `-m` / `-em` invocation, update the module name accordingly. The `--list-modules` output is the source of truth. !!! warning "The `http` name was reassigned" The module called `http` in 3.0 is **not** the same as the output module called `http` in 2.x. The name was reassigned: - **`http` (3.0, scan module)** -- the replacement for the old `httpx` scan module. Probes URLs over the shared in-process blasthttp client. - **`output.http` (2.x, output module)** -- the webhook-style output module that POSTed events to an arbitrary HTTP endpoint. This is now `output.webhook` in 3.0. Old scans that ran `bbot -om http ...` were emitting events to a webhook; in 3.0 the equivalent is `bbot -om webhook ...`. By contrast, `bbot -m http ...` enables the scan module, which probes URLs and is unrelated to the former output module. ### Late 2.x removals These were dropped during the final 2.x releases, so they're already gone if you're on the latest 2.x, but may still surprise you if you're upgrading from an earlier 2.x version: - `digitorus`, `passivetotal`, `sitedossier`, `wappalyzer` (removed, no replacement) - `azure_realm` (functionality merged into `azure_tenant`) - `bucket_azure` (now `bucket_microsoft`) - `censys` (split into `censys_dns` and `censys_ip`) ### New modules - Scan: `bucket_hetzner`, `shodan_enterprise`, `trajan`, `waf_bypass` - Output: `elastic`, `kafka`, `mongo`, `nats`, `rabbitmq`, `zeromq` ### Module API - `BaseModule` no longer has `_event_handler_watchdog_task` as a class attribute and the watchdog is owned by `_setup()`. - New `BaseModule.update_event(event, **kwargs)` and `emit_event(existing_event, ...)` flow. Passing an existing event to `make_event()` now raises -- call `update_event()` instead. - New `BaseModule.setup_deps()` lifecycle hook (runs alongside `setup()` for pure dependency installation like AI models or wordlists). - New `_disable_auto_module_deps = True` opt-out so a module that watches URLs doesn't automatically pull in `http`/`blasthttp`. - New `accept_seeds` attribute. Defaults to `True` for passive modules, `False` otherwise. Override explicitly if you want different behavior. - `default_discovery_context` now uses `{event.pretty_string}` instead of `{event.data}` -- the latter is now a dict for URL-like events (see below). - New `BaseModule._is_http_wildcard_host(event)` helper. Returns `True` when the target responds identically to two random paths (catch-all / SPA router). Used by `webbrute`, `lightfuzz`, and `paramminer` to skip hosts that would produce only false positives. --- ## Events ### Removed event types - `VULNERABILITY` is gone. Emit a `FINDING` with `severity` set to `"CRITICAL"`, `"HIGH"`, `"MEDIUM"`, `"LOW"`, or `"INFO"` instead. The pseudo event type `TARGET` was also retired in favor of `SEED`. - `VHOST` was renamed to `VIRTUAL_HOST`, emitted by the rebuilt `virtualhost` module (formerly `vhost`). ### Class hierarchy | 2.x | 3.0 | |-----|-----| | `URL_UNVERIFIED(BaseEvent)` -- data is a string | `URL_UNVERIFIED(DictHostEvent)` -- data is a dict with `url`, `path`, etc. | | `URL(URL_UNVERIFIED)` -- string data | `URL(URL_UNVERIFIED)` -- dict data | | `STORAGE_BUCKET(DictEvent, URL_UNVERIFIED)` | `STORAGE_BUCKET(URL_UNVERIFIED)` | | `HTTP_RESPONSE(URL_UNVERIFIED, DictEvent)` | `HTTP_RESPONSE(URL_UNVERIFIED)` | | `DictPathEvent(DictEvent)` | `DictPathEvent(DictHostEvent)` | | *(no UDP event)* | `OPEN_UDP_PORT(OPEN_TCP_PORT)` | URL events are now dict-backed. Reading `.data` on a `URL` / `URL_UNVERIFIED` / `HTTP_RESPONSE` returns a dict, not a string. Use the new `event.url` property to get the URL string, or `event.pretty_string` for a human-readable form. Many built-in modules and most tests were updated accordingly; custom modules that compared or formatted `event.data` for URL events must be updated. ### Removed attributes - `event.confidence` and `event.cumulative_confidence` no longer exist on `BaseEvent`. Confidence is now a per-finding value carried inside `FINDING.data["confidence"]` and validated against `("UNKNOWN", "LOW", "MEDIUM", "HIGH", "CONFIRMED")`. - `source_domain` has been removed from events. - `_always_emit_tags`: the literal tag `"target"` was renamed to `"seed"`. - Tags `ip-
`, `http-title-`, and `cloud-<type>` are no longer emitted as event tags by the `http` / cloudcheck pipelines. Use `event.resolved_hosts`, `event.host_metadata`, and the simplified cloud tags (`cloud`, plus a single provider tag) instead. ### New event surface - `event.url` -- string URL property (works on URL-like events, returns `""` otherwise). - `event.pretty_string` -- human-readable representation, used in logs and discovery context. - `event.host_metadata` -- dict of structured per-host metadata (cloud providers, ASN info, etc.). Replaces the long-tail of `cloud-*` tags. - Mutation helpers `add_resolved_host()`, `update_resolved_hosts()`, `add_dns_child()`, `set_raw_dns_record()`. Direct assignment to the backing slots (`event._resolved_hosts = ...`, `event.dns_children["A"].add(...)`) is no longer the public contract: the slots are lazily initialized to `None` for memory reasons and exposed read-only via properties. ### FINDING severity / confidence `FINDING.data` is now validated against a fixed allowlist: - `severity`: `"INFO" | "LOW" | "MEDIUM" | "HIGH" | "CRITICAL"` (the old `"INFORMATIONAL"` and `"MODERATE"` values have been renamed to `INFO` and `MEDIUM`) - `confidence`: `"UNKNOWN" | "LOW" | "MEDIUM" | "HIGH" | "CONFIRMED"` Severity and confidence are recorded on the event as `severity-<level>` and `confidence-<level>` tags. Lowercase tags like `high` or `medium` are no longer added directly. ### ASN as a target type `ASN:12345` (or `AS12345`) can now be used as a scan target -- the seed will be expanded to its registered CIDRs via the `asndb` library at scan start. If the lookup fails (network error, API unavailable), BBOT retries 3 times with a 3-second delay. If all retries fail, the scan aborts with a message suggesting you pass CIDR ranges directly instead. ASN lookups use the BLS API at `api.bbot.io`. No API key is required -- unauthenticated users can operate at a reasonable pace. If you want higher rate limits, set the `bbot_io_api_key` config value (or `BBOT_IO_API_KEY` env var). Currently the ASN API is the only `api.bbot.io` service BBOT uses, but any future BLS APIs will work with the same key. ASN *enrichment* (the ASN/subnet data attached to in-scope IPs via `host_metadata`) degrades gracefully rather than aborting: after several consecutive lookup failures, a circuit breaker disables ASN enrichment for the rest of the scan and the scan continues without it. This is separate from the ASN-as-target expansion above, which must abort because there would be nothing to scan. ### JSON output: before and after If you have downstream tooling that parses BBOT's NDJSON output, the schema has changed in several ways. Here is a representative event from each version: **2.x -- URL event (string data)** ```json { "type": "URL", "id": "URL:ab12cd34...", "data": "https://www.evilcorp.com/login", "host": "www.evilcorp.com", "port": 443, "resolved_hosts": ["1.2.3.4"], "dns_children": {}, "scope_distance": 0, "scan": "SCAN:deadbeef...", "timestamp": 1719792000.0, "parent": "DNS_NAME:ef56ab78...", "tags": ["in-scope", "status-200", "ip-1.2.3.4", "http-title-Login"], "module": "httpx", "module_sequence": "httpx" } ``` **3.0 -- same URL event (dict data)** ```json { "type": "URL", "id": "URL:ab12cd34...", "uuid": "f47ac10b-58cc-4372-a567-0e02b2c3d479", "data_json": { "url": "https://www.evilcorp.com/login", "status_code": 200, "http_title": "Login" }, "host": "www.evilcorp.com", "port": 443, "resolved_hosts": ["1.2.3.4"], "dns_children": {}, "scope_distance": 0, "scan": "SCAN:deadbeef...", "timestamp": 1719792000.0, "parent": "DNS_NAME:ef56ab78...", "parent_uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", "tags": ["in-scope", "status-200"], "module": "http", "module_sequence": "http", "discovery_context": "http probed www.evilcorp.com and found URL: https://www.evilcorp.com/login", "discovery_path": ["SCAN:deadbeef", "DNS_NAME:www.evilcorp.com", "URL:https://www.evilcorp.com/login"], "parent_chain": ["SCAN:deadbeef", "DNS_NAME:www.evilcorp.com"], "host_metadata": {"cloud_provider": "amazon", "asn": 16509} } ``` Key differences for parsers: | What changed | 2.x | 3.0 | |---|---|---| | URL event data | `"data": "https://..."` (string) | `"data_json": {"url": "https://...", ...}` (dict) | | Non-URL event data | `"data": "..."` (string) | `"data": "..."` (still a string) | | Module name for HTTP probing | `"module": "httpx"` | `"module": "http"` | | Per-host cloud/IP tags | `"ip-1.2.3.4"`, `"http-title-Login"`, `"cloud-amazon"` in tags | Moved to `host_metadata` dict and `data_json` fields; tags simplified | | Event ID | `"id"` only | `"id"` + new `"uuid"` (globally unique) | | Parent reference | `"parent"` only | `"parent"` + new `"parent_uuid"` | | Discovery chain | not present | `"discovery_context"`, `"discovery_path"`, `"parent_chain"` | !!! warning "The `data` vs `data_json` split" When an event's `.data` is a **string** (DNS_NAME, IP_ADDRESS, EMAIL_ADDRESS, OPEN_TCP_PORT, etc.), the JSON key is `"data"`. When `.data` is a **dict** (URL, URL_UNVERIFIED, HTTP_RESPONSE, FINDING, STORAGE_BUCKET, etc.), the JSON key is `"data_json"`. In 2.x, URL events used `"data"` with a string value. In 3.0, they use `"data_json"` with a dict. Parsers that key on `"data"` for URL events will silently get nothing back. **2.x -- VULNERABILITY event** ```json { "type": "VULNERABILITY", "data": { "host": "www.evilcorp.com", "severity": "HIGH", "description": "SQL injection in login form", "url": "https://www.evilcorp.com/login" }, "tags": ["in-scope", "high"] } ``` **3.0 -- equivalent FINDING event** ```json { "type": "FINDING", "data_json": { "host": "www.evilcorp.com", "severity": "HIGH", "confidence": "HIGH", "name": "SQL Injection", "description": "SQL injection in login form", "url": "https://www.evilcorp.com/login" }, "tags": ["in-scope", "severity-high", "confidence-high"] } ``` Changes: - `VULNERABILITY` type is gone; use `FINDING` instead. - `confidence` is now **required** on every FINDING (one of `UNKNOWN`, `LOW`, `MEDIUM`, `HIGH`, `CONFIRMED`). - `name` is now **required** (short label for the finding). - Severity strings changed: `INFORMATIONAL` is now `INFO`, `MODERATE` is now `MEDIUM`. - Tags changed from bare lowercase (`high`) to prefixed (`severity-high`, `confidence-high`). --- ## Config `bbot/defaults.yml` saw a number of breaking renames and additions: ### Renamed | 2.x | 3.0 | |---------------------------|---------------------------| | `web.httpx_timeout` | `web.http_timeout` (target-directed traffic, default `10`) | | *(no equivalent)* | `web.http_timeout_infrastructure` (API calls, wordlist downloads, etc., default `10`) | | `web.httpx_retries` | `web.http_retries` | | `dns.threads` (global) | `dns.threads` (now per-resolver; default lowered from 25 -> 10) | | `web.ssl_verify` | `web.ssl_verify_target` (target-directed traffic, default `false`) | | *(no equivalent)* | `web.ssl_verify_infrastructure` (API calls, wordlist downloads, etc., default `true`) | ### Removed - `deps.ffuf.version` (ffuf is gone). - The top-level `modules.json.siem_friendly` flag (and the `siem_friendly` output module). ### Added - `max_mem_percent` -- global ingress throttle when RSS exceeds the threshold. - `redact_secrets` (default `true`) -- redacts secret values (API keys, tokens) when a resolved preset is serialized to YAML (e.g. the generated `bbot.yml`). Set to `false` to include them. - `web.user_agent_suffix` -- appended to the user agent (previously buried as a hidden CLI flag). - `web.http_rate_limit` -- global rps cap across the shared blasthttp client. - `web.http_proxy_exclude` -- hosts/CIDRs to exclude from the HTTP proxy (`NO_PROXY` equivalent). - `web.body_spill.{enabled,cache_mb,compress}` -- disk-spill HTTP response bodies to keep them off the Python heap. - `dns.cache_size` -- DNS LRU size. - `bbot_io_api_key` -- optional API key for `api.bbot.io` services. Currently only used by ASN lookups; no key is required for basic use. Future BLS APIs will share this key. - `dns.abort_threshold` default lowered from `50` -> `10`. - `dns.filter_ptrs` semantics: PTR-derived hostnames are now treated as affiliates by default rather than being injected as in-scope DNS_NAME events during IP-range scans. ### Stale config files and `--reset-config` / `--reset-secrets` BBOT generates two config files the first time it runs and then never overwrites them: - `~/.config/bbot/bbot.yml` -- a fully-commented snapshot of all default options - `~/.config/bbot/secrets.yml` -- the secret-bearing subset (API keys, etc.), written owner-only (`0600`) Because they're written once and left alone, they drift out of sync with the defaults as you upgrade. A `bbot.yml` generated under 2.x can still mention `web.httpx_timeout`, `web.ssl_verify`, `modules.json.siem_friendly`, and other keys that were renamed or removed in 3.0. In 2.x this was harmless: unknown keys were silently ignored. In 3.0, config is **validated on load**, so a leftover key in your generated file now produces a validation error before the scan starts. When the offending key lives in one of these generated files (rather than being a `-c` typo on the command line), BBOT tells you which file it came from and points you at the matching reset flag. Two new CLI flags regenerate the files from current defaults: ```bash bbot --reset-config # regenerate bbot.yml bbot --reset-secrets # regenerate secrets.yml ``` - **Destructive.** A regenerated file is a fresh, fully-commented template, so any options you had *uncommented* are wiped. The existing file is backed up first to `<name>.bak` (then `.bak.1`, `.bak.2`, ... so earlier backups aren't clobbered). - **Confirmation required.** You're prompted before anything is overwritten; pass `-y` / `--yes` to skip the prompt. Non-interactive runs without `--yes` refuse and exit rather than overwrite silently. - **Independent.** The two files are reset separately: `--reset-config` never touches the API keys in `secrets.yml`, and `--reset-secrets` never touches the tuned options in `bbot.yml`. The regenerated `secrets.yml` is written atomically and stays owner-only. The usual fix after an in-place upgrade is to copy any customizations out of the old file, run the matching reset flag, then re-apply your changes to the fresh template (or move them into a preset). --- ## DNS and HTTP internals Both the DNS and HTTP engines were rewritten and the old subprocess architecture was deleted. - **DNS**: `bbot/core/helpers/dns/engine.py` and `dns/mock.py` were removed. The `DNSHelper` no longer inherits from `EngineClient`; resolution now goes through the native [blastdns](https://github.com/blacklanternsecurity/blastdns) Rust client. The `self.helpers.dns.resolver` `dnspython` resolver is gone. - **HTTP**: `bbot/core/helpers/web/client.py` and `web/engine.py` were removed. `WebHelper` no longer inherits from `EngineClient`. All HTTP goes through the shared [blasthttp](https://github.com/blacklanternsecurity/blasthttp) client (`self.helpers.blasthttp`). The httpx-based `request_batch` / `request_custom_batch` / `curl` methods were replaced by `request()`, `request_batch_stream(urls, threads=10, **kwargs)`, and `download()`. - **Engine framework removed**: the `bbot.core.engine` module (`EngineBase` / `EngineClient` / `EngineServer`) and the `BBOTEngineError` exception were deleted along with the subprocess architecture. `WebError` and `DNSError` now subclass `BBOTError` directly, and `CurlError` was removed. External code importing `bbot.core.engine`, `BBOTEngineError`, or `CurlError` must be updated. - The blasthttp dependency line is `blasthttp>=0.9.0`. Modules that previously instantiated their own `httpx.AsyncClient` or built custom curl invocations must switch to `self.helpers.request(...)` / `self.helpers.blasthttp`. --- ## Output and DB models - The pydantic / SQLModel models moved from `bbot.db.sql.models` to `bbot.models.sql` (plus a new `bbot.models.pydantic` / `bbot.models.helpers` split). External importers must update. - `output.http` was renamed to `output.webhook`. - `output.json` lost the `siem_friendly` option. - New output modules: `elastic`, `kafka`, `mongo`, `nats`, `rabbitmq`, `zeromq`. - Neo4j output now also serializes `host_metadata`. ### Output modules are now purely additive In 2.x, specifying `-om` could silently replace the default output modules depending on whether any of the names overlapped with the defaults. This was confusing and inconsistent. In 3.0, `-om` is always **additive**: it adds modules on top of the defaults (`csv`, `txt`, `json`, plus `stdout` from the CLI). To remove a default, use the new `-eom` / `--exclude-output-modules` flag: ```bash # defaults (csv, txt, json, stdout) are all enabled bbot -t evilcorp.com -p subdomain-enum # neo4j is added alongside all defaults bbot -t evilcorp.com -p subdomain-enum -om neo4j # json only -- explicitly exclude the other defaults bbot -t evilcorp.com -p subdomain-enum -eom csv txt stdout ``` The same applies in preset YAML (`exclude_output_modules:`) and the Python API (`Preset(exclude_output_modules=[...])`). The `python` output module was also moved from `output/` to `internal/` -- it was never a real output module, it is the Python API event bridge. ### Omitted event types Certain event types are excluded from output by default via the `omit_event_types` config. These events are still processed by modules internally, but they do not appear in JSON, CSV, or stdout output unless you remove them from the list: ```yaml omit_event_types: - HTTP_RESPONSE - RAW_TEXT - URL_UNVERIFIED - DNS_NAME_UNRESOLVED - FILESYSTEM - WEB_PARAMETER - RAW_DNS_RECORD ``` If your downstream tooling relied on seeing `HTTP_RESPONSE` or `URL_UNVERIFIED` events in the output, you will need to remove them from `omit_event_types` in your preset or config. --- ## Dependencies and tooling - **Build system**: migrated from Poetry to [uv](https://docs.astral.sh/uv/) with a hatchling build backend. `pyproject.toml` now follows PEP 621, and `uv.lock` replaces `poetry.lock`; install dev dependencies with `uv sync --group dev`. Versioning is static (`version = "3.0.0"` in `pyproject.toml`) rather than derived from git tags by `poetry-dynamic-versioning`. - **License**: changed from `GPL-3.0` to `AGPL-3.0`. - **Python**: minimum bumped from `3.9` to `3.10`. Upper bound is now `<3.15`. - **Lockstep deps**: - `radixtarget >=4.0.1,<5` (composition pattern, no longer subclassed) - `cloudcheck >=11.1.0,<12` - `blasthttp >=0.9.0` (new) - `blastdns >=1.9.0,<2` (new) - `asndb >=1.1.0` (new) - `zstandard` (new; used by HTTP body spill) - `httpx` **removed** as a runtime dep. Custom modules that imported `httpx`, `dns.asyncresolver`, `radixtarget` subclasses, or `bbot.db.sql.models` will need to be ported.