AI-Powered Automated Cybersecurity Threat Detection Platform
Watcher is a Django & React JS platform designed to discover and monitor emerging cybersecurity threats with **AI-powered threat intelligence analysis**. It can be deployed on webservers or quickly run via Docker.
## Watcher Capabilities
Watcher empowers your security operations with comprehensive threat detection and monitoring:
- **AI-Driven Threat Intelligence** - Transform raw threat data into actionable intelligence with automated weekly digests of top-5 trending cybersecurity topics, real-time breaking news alerts when threats emerge, on-demand summaries for any security keyword including related CVE and threat actor details.
- **CVE & Ransomware Intelligence** - Continuously fetch, correlate, and surface external threat data: CVEs from [cve.circl.lu](https://cve.circl.lu), ransomware victims and groups from [ransomware.live](https://ransomware.live) and [ransomlook.io](https://ransomlook.io). Define keyword-based Watch Rules to get alerted when specific threats match your organisation's context.
- **Emerging Threat Detection** - Monitor cybersecurity trends via RSS feeds from CERT-FR (www.cert.ssi.gouv.fr), CERT-EU (www.cert.europa.eu), US-CERT (www.us-cert.gov), Australian Cyber Security Centre (www.cyber.gov.au), and more. Track new vulnerabilities, malware campaigns, and threat advisories as they appear.
- **Legitimate Domain Management** - Centralized approved domains with expiry, repurchase status, registrar info, and contacts. Easily convert monitored malicious domains into legitimate ones. Automated UDRP case tracking.
- **Information Leak Monitoring** - Detect sensitive data exposure across the webs including Pastebin, StackOverflow, GitHub, GitLab, Bitbucket, APKMirror, npm registries, and other platforms. Catch leaked credentials, API keys, and confidential information early.
- **Malicious Domain Surveillance** - Monitor malicious domains for changes in IP addresses, mail/MX records, and web content. Use [TLSH](https://github.com/trendmicro/tlsh) fuzzy hashing to detect modifications. Automatic RDAP/WHOIS checks with registrar and expiry alerts.
- **Suspicious Domain Detection** - Identify potentially malicious domains targeting your organisation via:
- **Domain Generation Algorithm Detection** using [dnstwist](https://github.com/elceef/dnstwist) to find typosquatting, homograph attacks, and similar domain variants
- **Certificate Transparency Monitoring** via [certstream](https://github.com/CaliDog/certstream-python) to catch newly registered suspicious domains in real-time
## Additional Features
Extend Watcher's capabilities with powerful integrations and management tools:
- **TheHive Full Synchronization** - Integration with [TheHive](https://thehive-project.org/) featuring automated alert creation, smart case management, IOC enrichment, and ready-to-use Cortex Analyzers & Responders. Detailed configuration are provided in the documentation [here.](https://thalesgroup-cert.github.io/Watcher/README.html#thehive-export)
- **MISP Integration** - Seamlessly export Indicators of Compromise (IOCs) to [MISP](https://www.misp-project.org/) with smart UUID tracking, automatic object creation, and manual attribute updates for collaborative threat intelligence sharing
- **SSO / OpenID Connect** - Federated login via any OIDC provider (Keycloak, Azure AD, etc.) with PKCE and Knox token issuance. Configurable per-instance via `.env`
- **Bulk Add & Bulk Actions** - Add many items at once with the *One item per line* toggle (with a no-write Preview and a per-item created / skipped / error summary), or tick rows in any table to edit, delete or export them (to Legitimate Domains or MISP) in one go
- **Flexible Authentication** - Support for LDAP, local, and SSO/OIDC authentication systems
- **Connectors Dashboard** - A superuser-only `/connectors` page to view, edit, and test every external integration (SMTP, Slack, Citadel, TheHive, MISP, CyberWatch feeds, and more) from one place, with encrypted credential storage and per-connector health checks.
- **Smart Notifications** - Receive email, Slack, or Citadel alerts for critical findings and threshold violations across all modules including CyberWatch and UDRP decisions
- **Interactive API Documentation** - Auto-generated Swagger UI at `/api/docs/` and OpenAPI 3 schema at `/api/schema/` powered by drf-spectacular
- **Ticketing System Integration** - Automatically feed your ticketing system with security findings
- **Comprehensive Admin Interface** - Manage all aspects of Watcher through Django's powerful admin panel
- **Advanced Access Control** - Granular user permissions and group management for team collaboration
- **Modern UI Experience** - A modern interface with customizable themes, resizable dashboard panels, advanced filtering with saved filter sets, and persistent user preferences
## Involved dependencies
Watcher leverages open source tools and libraries:
- [**Hugging Face Transformers**](https://huggingface.co/docs/transformers) - AI/ML framework powering threat intelligence summarization and entity extraction
- [**google/flan-t5-base**](https://huggingface.co/google/flan-t5-base) - Text-to-text generation model for AI-powered threat summaries
- [**dslim/bert-base-NER**](https://huggingface.co/dslim/bert-base-NER) - Named Entity Recognition for automatic IOC extraction
- [**certstream**](https://github.com/CaliDog/certstream-python) - Certificate Transparency monitoring
- [**dnstwist**](https://github.com/elceef/dnstwist) - Domain name permutation engine
- [**SearxNG**](https://github.com/searxng/searxng) - Privacy-respecting metasearch engine
- [**PyMISP**](https://github.com/MISP/PyMISP) - MISP threat intelligence platform integration
- [**TLSH**](https://github.com/trendmicro/tlsh) - Fuzzy hashing for content similarity detection
- [**mozilla-django-oidc**](https://github.com/mozilla/mozilla-django-oidc) - OpenID Connect authentication backend
- [**drf-spectacular**](https://github.com/tfranzel/drf-spectacular) - OpenAPI 3 schema generation and Swagger UI
- [**deck.gl**](https://deck.gl/) - WebGL-powered geospatial visualization (WorldMap)
- [**MUI (Material UI)**](https://mui.com/) - React component library for KPI cards and charts
- [**shadow-useragent**](https://github.com/lobstrio/shadow-useragent) - User-Agent rotation library
- [**NLTK**](https://www.nltk.org/) - Natural Language Toolkit for text processing
## App Preview
All the screenshots below use fictional data (a made-up company, "Altair Dynamics").
### Overview
A tour of the platform: the AI weekly report and breaking news, the threat dashboard, legitimate and suspicious domains, DNS threats, data leaks and Cyber Watch.
### Threat Detection
### AI-Powered Weekly Summary & Breaking News
### Suspicious domain names detection
### Legitimate Domain List
Add several domains in one go with the *One item per line* toggle: defanged links are restored, a *Preview* shows the result before anything is saved, and the summary lists every domain as created, skipped or in error.
### Data Leak Detection
### Suspicious domain names monitoring
Select several domains and change legitimacy, takedown, legal and blocking flags together.
### Bulk actions and exports
False positives can be exported to Legitimate Domains in bulk: the exported sites disappear from Website Monitoring and appear in the legitimate list, with no page refresh.
### Cyber Watch
Watch rules, monitored keywords and RSS sources, with the CVE and ransomware matches they produce.
### History of every change
Every create, update and delete is recorded with its author, so you can see who changed what and when.
### Customisable dashboards
Show, hide, move and resize panels: the layout is saved in your profile.
### Theme Previews
Watcher offers multiple visual themes to match your preferences and working environment.
### Admin Interface
Django provides a ready-to-use user interface for administrative activities. We all know how an admin interface is important for a web project: Users management, user group management, Watcher configuration, usage logs...
## Installation
```bash
# 1. Clone the repo
git clone https://github.com/thalesgroup-cert/watcher.git
cd watcher/deployment
# 2. Initialize environment, configs & directory structure
make init
# 3. Start the stack
make up
# 4. On first run: run database migrations + create superuser
make migrate
make superuser
make populate-db
# 5. Open the web UI
# http://localhost:9002 (or your configured domain/port)
```
Get Watcher up and running in just **10 minutes** using Docker. **Detailed instructions available in our [Installation Guide](https://thalesgroup-cert.github.io/Watcher/README.html)**
## Platform Architecture
```mermaid
flowchart LR
subgraph SRC["External sources"]
RSS["RSS / CERT advisories"]
CVE["CVE & ransomware feeds"]
CT["Certificate Transparency"]
WEB["Paste sites & code registries"]
RSS ~~~ CVE ~~~ CT ~~~ WEB
end
subgraph HOST["Host server (Docker)"]
TRAEFIK["Traefik
TLS reverse proxy"]
subgraph WATCHER["Watcher (Django)"]
UI["React UI / Django admin"]
API["Django REST API"]
JOBS["Scheduled jobs (APScheduler)
Threats · CyberWatch · DNS Threats
Data Leak · Website Monitoring"]
CONN["Connectors"]
end
CERTSTREAM["CertStream"]
SEARX["SearXNG"]
DB[("MySQL")]
end
subgraph OUT["Outputs"]
HIVE["TheHive"]
MISP["MISP"]
CHAT["Slack / Citadel"]
MAIL["Email (SMTP)"]
end
ANALYST(["Analyst"]) --> TRAEFIK --> UI
UI <--> API
API <--> DB
JOBS <--> DB
RSS & CVE --> JOBS
CT --> CERTSTREAM --> JOBS
WEB --> SEARX --> JOBS
JOBS --> CONN
API --> CONN
CONN --> HIVE & MISP & CHAT & MAIL
```
Watcher's modular architecture ensures scalability, reliability, and easy integration with your existing security stack.
## Contributing
We welcome contributions from the security community!
To report bugs, request features, or submit code, please read our full [Contributing Guide](CONTRIBUTING.md).
## Pastebin Compliance
In order to use Watcher pastebin API feature, you need to subscribe to a pastebin pro account and whitelist Watcher public IP (see https://pastebin.com/doc_scraping_api).