v2.3.1 (2026-10-27) + Fixed --phishing returning no matches for every keyword while phish.co.za is down: an HTTP error page (503/403) was saved as the phishing database and the keywords were matched against it. Only a 200 response with a non-empty text body is accepted now. Thanks to Slava Olesik (PR #119). + New flag --phishing-fallback URL: an optional https list of phishing domains/URLs (one per line) used only when the Phishing Database cannot be downloaded. There is no default fallback. + Phishing lists are treated as untrusted input: - Every line is validated; only hostnames and http(s) URLs are kept, so control characters and ANSI escape sequences can no longer reach the terminal or the --phishing output file. - A source is rejected when it is HTML/XML/JSON, redirects to plain http, is larger than 256 MB, or when less than half of its lines are valid. - Invalid UTF-8 no longer crashes the phishing check. - phishing.db is written atomically, so a failed update never leaves a partial file or replaces the previous database. + The run summary now shows which list the phishing results came from. + Lines starting with "#" in the phishing list are ignored. v2.3.0 (2026-04-08) + Added Premium Feed mode (--premium): downloads paid NRD feed from api.opensquat.com/v1/feeds/nrd-lite using X-API-Key header auth, then runs the same local Levenshtein pipeline. + Added Premium API mode (--api): queries the openSquat lookalike REST API per keyword instead of running local Levenshtein on a downloaded feed. Both Premium modes share a single openSquat API key. + New flags: --premium, --api, --api-key, --api-fuzziness, --api-history-days, --api-max-results. + API key resolution priority: --api-key > $OPENSQUAT_API_KEY > api_key.txt + New module opensquat/auth.py for API key resolution. + New module opensquat/api_client.py with typed exceptions (APIAuthError, APIPlanError, APIQuotaExhausted, APIBadRequest, APIError). + Premium API mode is parallelized with ThreadPoolExecutor (one worker per keyword, capped at 8) and reuses a single requests.Session for connection pooling. + Premium API mode honors --dns by annotating each returned domain via DNSValidator. + Quota exhaustion (HTTP 429) returns partial results gracefully. + Summary output now reports the active mode, API call count, and balance. + Default Community mode is unchanged - existing users need no flags. + Premium API mode preserves keyword file order in streaming output (keywords print in the same order as keywords.txt even though calls run in parallel). + CLI prints a masked confirmation line when an API key is loaded (e.g. "[*] API key loaded: os_gL...L5Mb") so users can verify which key was picked up without leaking the full value. + Usability hint: passing --api-key without --premium or --api prints a warning explaining the key will be ignored in Community mode. + Premium API mode summary now shows balance used vs initial (e.g. "4972 (used 4 of 4976 this run)") when both values are available. + Community feed downloads now set a 60s HTTP timeout (previously unbounded - a hung connection would hang the whole CLI). + Added tests for auth, api_client, feed_manager, and arg_parser. + Fixed homoglyphs dependency pin: >=2.1.0 did not exist on PyPI (max is 2.0.4). Lowered to >=2.0.0 on both requirements.txt and pyproject.toml so fresh installs resolve. + Premium API mode now distinguishes transient upstream rate limiting (HTTP 429 with Retry-After header) from permanent quota exhaustion. Rate limit hits print a yellow warning with the retry delay and preserve the real API balance in the summary; quota exhaustion keeps the existing red abort path that forces balance to 0. + New --api-rate-limit N flag for Premium API mode caps outbound request rate across all workers via a shared fixed-interval limiter. Default is unlimited (unchanged behavior for existing scripts). Set to a value at or below your backend's sustained rate policy to avoid triggering rate limits on large scans (e.g. 8 for a backend that allows 10 req/s). + api_client now surfaces plain-text error response bodies in error messages when the server returns a non-JSON 4xx/5xx, instead of falling back to a generic label. The actual upstream message now reaches the user. + Premium API summary now shows "calls made: N (of M)" when a run ends early, plus a yellow "Rate limit hit" or red "Quota exhausted" reason line so users can tell at a glance why the scan was cut short. + Premium API mode JSON output now preserves the full per-domain metadata the server returns: tld, NRD first-seen date, an IDN homograph flag, and the unicode rendering for homographs. Previously the tool parsed these fields from the response body and then discarded them, hiding the unicode characters behind bare punycode strings like "xn--mirosoft-hw7c.com". + CSV output is now a proper one-row-per-domain table with columns keyword,domain,tld,first_seen,is_idn,unicode. Fixes the pre-existing writerow-vs-writerows bug that was writing all domains as a single row. Written with a UTF-8 BOM so Excel on Windows correctly renders the unicode homograph column. + BREAKING (JSON consumers of v2.2.3 / v2.2.4): each entry in a keyword's "domains" array is now an object {"domain": "..."} instead of a bare string. Migration is one line of user code: OLD: for d in kw["domains"]: print(d) NEW: for d in kw["domains"]: print(d["domain"]) The top-level [{"keyword": ..., "domains": [...]}] shape is unchanged. + Community and Premium Feed JSON output uses the same per-domain- object shape for cross-mode consistency; the extra metadata fields are populated only in Premium API mode where the upstream source provides them. + --dns annotation is unchanged by the enrichment (still receives the punycode domain string as before); post-processing filters (--vt, --subdomains, --portcheck, --phishing) continue to operate on a flat list[str] contract and are not affected by the rich JSON/CSV output. + New LookalikeDomain dataclass in opensquat.api_client exposes the parsed per-domain fields for any downstream consumer using the module programmatically. + Pre-release hardening pass — bug fixes: - feed_manager: surface local feed read errors instead of silently re-downloading on every run (replaced bare except with targeted OSError/IOError + yellow warning). - vt.py: fixed --subdomains feature (issue #111). main() now takes a named mode argument ("report" default or "subdomains") with a new subdomains() method that calls the VT v3 /api/v3/domains/{domain}/subdomains endpoint. - vt.py: fixed domain_report() returning None on missing data (issue #112). Now returns [0, -1] which the caller's existing "VT throttling" branch already handles, eliminating the TypeError. - vt.py: fixed set_apikey() crash on empty lines (line[0] indexing bug, same pattern as #113) and last-line-wins behavior — now reads the first usable line and breaks. - phishing.py: fixed line[0] crash on whitespace-only keyword lines plus broken keyword counter; replaced with strip()+startswith("#"). - phishing.py: fixed case-sensitivity bug — phishing_site = site.lower() was being immediately overwritten by site.replace("\\n", ""), so mixed-case phishing entries (e.g. "Login.com") never matched lowercase keywords. - phishing.py update_db: fixed chunked-encoding false-negative (Content-Length=0 was treated as "file not found" even when the body had valid data — same bug already fixed in feed_manager.py). - phishing.py update_db: added 60s HTTP timeout (was unbounded). - feed_manager.py: collapsed two literally-identical if/else branches; premium downloads now actually use the longer 120s timeout the code comment promised. - check_update.py: parse_version() raised ValueError on non-semver server responses (HTML error pages, "2.3.0-rc1", blank bodies), crashing the CLI at the very end of an otherwise successful run. Now wrapped in try/except. Also added a 10s timeout to the version check fetch. - ct.py: added 10s timeout to requests.get() in CRTSH.check_certificate so a hung crt.sh response no longer freezes the CLI when --doppelganger iterates over many domains. - api_client.py: pass allow_redirects=False on the lookalike POST so the session's X-API-Key header cannot leak to a redirect target on a different host (requests strips Authorization on cross-host redirects but NOT custom headers). v2.2.4 (2026-03-30) + Raised minimum Python version from 3.8 to 3.10 (Python 3.8 and 3.9 are EOL). v2.2.3 (2026-03-29) + Fixed Unicode encoding error on Windows (issue #104). + JSON output now groups results by keyword (issue #105). + JSON output respects post-processing filters (--vt, --subdomains, --portcheck). + Fixed file_input.py crash on empty lines in keywords file. + Added encoding='utf-8' to all file open() calls. + Added pretty-print and Unicode support to JSON output. v2.2.2 (2026-03-28) + Fixed homoglyphs dependency version pin (2.1.0 doesn't exist on PyPI, corrected to 2.0.0) v2.2.1 (2026-03-28) + PyPI package support: pip install opensquat + Auto-publish to PyPI via GitHub Actions on push to master + Added opensquat CLI entry point for pip users + Backward compatible: python opensquat.py still works for repo users v2.2.0 (2026-03-28) + Major Code Refactoring: Split app.py into FeedManager, DNSValidator, and SquattingDetector modules. + Performance: Native Levenshtein with early exit optimization (~5.6x faster). + Removed strsimpy, numpy, and packaging dependencies (native Python replacements). + Removed Jaro-Winkler method (Levenshtein catches more squatting patterns). + Added -u/--url flag for custom feed URL. + Added --doppelganger flag (separated from --ct). + Added path traversal protection for feed URL filename extraction. + Fixed args.ct incorrectly mapped to doppelganger_only parameter. + Fixed ct.py not printing actual URL on errors. + Fixed read_files() appending raw lines instead of cleaned domains. + Fixed keyword enumeration starting at 0 instead of 1. + Removed deprecated Telegram Bot and RapidAPI integrations from README. + Updated Python version support from 3.6+ to 3.8+. + Pinned minimum dependency versions in requirements.txt. + Separated runtime and dev dependencies (requirements-dev.txt). + Renamed project to openSquat Core (open-core model). v2.1.2 (2026-01-08) + Fixed feed validation for chunked responses (feeds with missing Content-Length header) v2.1.1 (2026-01-04) + Added support for comments in the feeds file - Removed/Deprecated support for weekly and monthly feed files. v2.1.0 (2023-06-04) + openSquat now supports Multiprocessing and Multithreading (thanks to maaaaz) - major performance boost + dnsPython library has been updated (thanks to maaaaz) + Multiple fixes including DepreciationWarning (more details pull #85) v2.0.0 (2023-03-11) + Improved VirusTotal integration and support for VT API v3 + new configuration file "vt_key.txt" for VirusTotal API key + Fixed bug #78 on CT (thanks to jj98594 and TerminalFi) + Adopted Semantic Versioning (https://semver.org) v1.99 (2022-05-21) + Bug correction (#50) - Suspicious certificate results not showing in results.txt v1.98 (2022-04-17) + Bug correction (#63) - double DNS validation v1.97 (2020-10-18) + Domains can be verified is they are blacklisted to any VirusTotal engine + You can now filter results using different args + minor change, the number of domains showing with the thousand comma v1.96 (2020-10-11) + Bandwidth saving - only download the latest feeds if it has changed + added generic.txt containing generic keywords used for phishing v1.95 (2020-09-25) + Added --portcheck flag to validate if flagged domains have port 80/443 opened v1.94 (2020-09-19) + Run file is back again to "opensquat.py" + Checks for sudomains (virustotal) - Doppenganger options replaced by certificate transparency v1.93 (2020-09-07) + Checks for the latest version after each run v1.92 (2020-09-04) + Feeds URL changed from Github to feeds.opensquat.com (CDN for faster delivery) + Added the old feed (Github) as the backup URL case the primary fails + Default period is now day + default HTTP USER_AGENT is now openSquat+version v1.91 (2020-07-19) + Switched Quad9 from DoH to native DNS + It now shows squatting progress in percentage - Bug on phishing keyword counter v1.9 (2020-07-15) + Now it checks against active and known Phishing domains (Phishing Database) v1.8 (2020-07-12) + Add support for DNS validations (Quad9) v1.7 - (2020-06-03) + New searchable period by month + default search period changed from day to month v1.6 - (2020-05-26) + Support for certificate transparency (CT) search v1.5 - (2020-05-20) + Added doppelganger only feature v1.4 - (2020-05-17) + Jaro-Winkler algorithm support for string similarity v1.3 - (2020-05-15) + New searchable period options: day (default) and week + Added download size information for domains file + Travis-CI support (by mateuszz0000) + Code clean-up (by mateuszz0000) v1.2 - (2020-05-11) + Added support for IDN homographs attacks + New logo + New python library homoglyphs is required + New python library confusable_homoglyphs is required + Terminal colours added - Removed support for lxml v1.1 - (2020-05-08) + Major improvements for Levenshtein implementation (5 times faster) + New python library strsimpy is required - Removed numpy lib v1.0 - (2020-05-07) + Initial release