Blackbox
An intelligent self-hosted forensic event timeline for homelabs and home servers.
Know what changed, when it changed, and why things broke.
---
---
## What is Blackbox?
Blackbox is a lightweight, self-hosted event correlation platform built for homelabbers who want to understand their infrastructure at a glance. It collects events from Docker, config file changes, selected systemd units, uptime monitors, and container update tools, correlates them into a single chronological timeline, and groups likely outages into incidents with scored cause candidates and optional local-AI analysis or AI-enhanced correlation.
When your homelab breaks, Blackbox tells you what happened. You don't need to lift a finger.
Full documentation: [docs.blackboxd.dev](https://docs.blackboxd.dev)
---
> [!CAUTION]
> This is beta software. I am working towards v1.0.0, but in the meantime, it is being actively developed. Expect changes. Possibly breaking ones. During beta development, I would much appreciate if you report issues when you find them and suggest features.
---
## Quick Start
> This gets a single-node setup running in minutes. For multi-node, see the [deployment docs](https://docs.blackboxd.dev/docs/deployment/multi-node).
> Both images run as non-root. The server uses UID 65532 (fixed). The agent defaults to UID/GID 65532 but can be overridden with `PUID`/`PGID` to match the owner of your watched host paths. The agent entrypoint auto-detects the GIDs of any mounted resources (Docker socket, systemd journal) at startup - no manual group configuration needed.
> The project is available for both `amd64` and `arm64` architectures.
**1. Create a `docker-compose.yml`:**
```yaml
services:
blackbox-server:
image: ghcr.io/maxjb-xyz/blackbox-server:latest
container_name: blackbox-server
restart: unless-stopped
ports:
- "8080:8080"
volumes:
- blackbox-data:/data
environment:
JWT_SECRET: "change-me-to-a-long-random-string"
AGENT_TOKENS: "homelab=change-me-to-a-secret-agent-token"
WEBHOOK_SECRET: "change-me-to-a-webhook-secret"
TZ: "America/New_York" # optional: set to your local timezone so container logs match your clock
networks:
- blackbox
blackbox-agent:
image: ghcr.io/maxjb-xyz/blackbox-agent:latest
container_name: blackbox-agent
restart: unless-stopped
cap_drop:
- ALL
cap_add:
- SETUID
- SETGID
security_opt:
- no-new-privileges:true
read_only: true
volumes:
- blackbox-agent-data:/data
- /var/run/docker.sock:/var/run/docker.sock:ro
- /etc:/watch/etc:ro
- /run/log/journal:/run/log/journal:ro
- /var/log/journal:/var/log/journal:ro
- /etc/machine-id:/etc/machine-id:ro
environment:
SERVER_URL: "http://blackbox-server:8080"
AGENT_TOKEN: "change-me-to-a-secret-agent-token"
NODE_NAME: "homelab"
WATCH_PATHS: "/watch/etc"
WATCH_SYSTEMD: "true"
TZ: "America/New_York" # optional: set to your local timezone so container logs match your clock
networks:
- blackbox
volumes:
blackbox-agent-data:
blackbox-data:
networks:
blackbox:
driver: bridge
```
**2. Start it:**
```bash
docker compose up -d
```
**3. Open `http://your-server:8080` and complete the setup wizard.**
**4. Optional:**
- Open `Admin > Data Sources` to manage per-node and server-wide data sources and set up capability-aware systemd, file watcher, webhook, and Docker exclusions.
- Open `Admin > System` to configure file-diff redaction and Ollama-based incident analysis.
---
## Features
### Event Ingestion
- **Docker events** - Automatic detection of container start, stop, die, create, pull, and delete events. Per-node, with logic for collapsing events like restarts.
- **Smarter service inference** - Docker and file events resolve service names from Compose labels, Swarm metadata, image/container lookups, and common homelab path layouts so correlation works with cleaner service names.
- **Crash log capture** - Collapsed Docker stop/restart events include a best-effort tail of recent container logs, which Blackbox can surface directly inside incidents.
- **Config file watching** - inotify-based watching of `.yaml`, `.yml`, `.conf`, `.env`, `.json`, and `.ini` files via configurable `WATCH_PATHS`.
- **Config diffs** - File change events include a bounded text diff in metadata for deeper timeline analysis, with optional secret redaction controlled from the Admin settings page.
- **Systemd unit watching** - Linux agents can watch selected systemd units from the journal and emit `started`, `stopped`, `restart`, and `failed` lifecycle entries.
- **OOM kill detection** - Kernel OOM events are ingested as `systemd` entries, and failed units include a recent journal log snippet for faster triage.
- **Uptime Kuma webhooks** - Ingest Down/Up state changes. Down events open confirmed incidents and score likely causes from recent Docker, file, and webhook activity.
- **Watchtower webhooks** - Ingest container image update events with version metadata and use them as incident evidence when a restart follows shortly after.
- **Komodo webhooks** - Ingest events from Komodo via wekbook.
- **Custom entries** - Post arbitrary events via the API.
- **Container & stack exclusions** - Exclude specific Docker containers or Compose stacks from event ingestion entirely. Configured in Admin > Data Sources. Excluded events are silently dropped server-side - agents do not need reconfiguring.
### Sources
- **Source catalog** - Manage supported event sources from Admin > Data Sources. The catalog is capability-aware per node, supports both server-scoped and agent-scoped sources, and preserves stored secrets when you edit an existing source.
### Notifications
- **Webhook notification support** - Send outbound notifications to Discord, Slack, ntfy, and other targets when Blackbox events occur. Configure destinations in Admin > Integrations > Notifications.
- **AI review notifications** - When Ollama completes an incident analysis, all subscribed notification destinations receive the event with the AI summary embedded in the payload.
- **Incident deep-links** - Configure a **Base URL** in Admin > Integrations > Notifications so notification payloads include a direct link to the incident in Blackbox.
### Incidents & Correlation
- **Incident lifecycle** - Blackbox opens confirmed incidents from monitor-down events and suspected incidents from crash loops, unexpected container exits, watched systemd failures/OOM kills/restart loops, and update-triggered restarts.
- **Weighted cause scoring** - Likely causes are ranked from recent Docker, file, systemd, and update entries using event-specific lookback windows, same-node bonuses, and log-snippet bonuses.
- **Event chain view** - The Incidents page shows open and resolved incidents, duration, linked trigger/cause/evidence/recovery events, AI-derived causes when enabled, and the chosen root-cause entry.
- **Optional AI modes** - If Ollama or other AI provider is configured, Blackbox can run either plain AI analysis or an AI-enhanced correlation mode from the same Admin settings page.
- **AI across suspected incidents too** - Suspected incidents get the same log-backed AI treatment as confirmed incidents, including crash-log context and the selected AI mode.
- **Postmortem PDF reports** - Download a structured PDF report from any resolved incident. The report includes the incident timeline, root cause, AI summary (if available), and a chronological event chain. Accessible via the Download Report button on resolved incident cards.
### Timeline
- Chronological, paginated event feed across all nodes
- Filter by node, event source, or full-text search
- **Full-text search** - SQLite FTS5 powered search across all entry content and service names with term highlighting
- ULID-based IDs for guaranteed sort-order correctness
- Annotate any event with notes - with user attribution
### Node Management
- Nodes auto-register on first agent heartbeat
- Per-node metadata: agent version, IP address, OS, last-seen timestamp
- Agent capability reporting powers source setup defaults and node-specific source availability in the admin UI
- Live node pulse indicator in the sidebar
### Authentication
- Local username/password with Argon2id hashing
- JWT session cookies (configurable TTL, default 24h)
- OIDC (OpenID Connect) support for Authentik, Authelia, Keycloak, etc.
- Admin bootstrap wizard on first launch
- Invite-code-based user registration
### MCP Server (optional)
- **Model Context Protocol server** - Expose your Blackbox data to AI assistants (Claude Desktop, claude.ai, and any MCP-compatible client). Toggle it on or off from Admin > System > MCP Server. When enabled, Blackbox mounts MCP at `/mcp` on the main server and protects it with a server-wide bearer token that you can regenerate from Admin. Older MCP clients configured for a separate port or `/sse` endpoint must be updated. Available tools: `list_incidents`, `get_incident`, `list_entries`, `search_entries`, and `list_nodes`.
### Admin & Observability
- **Audit log** - Every admin action (user management, invite creation, OIDC provider changes, notification destination changes) is recorded with actor identity, IP address, and timestamp. Viewable in Admin > Access > Audit Log. Last 10,000 entries retained.
- **Webhook delivery log** - Every inbound Uptime Kuma and Watchtower webhook is recorded with source, status (processed / ignored / error), payload snippet, and the correlated entry ID when applicable. Viewable in Admin > Integrations > Webhook Log. Last 1,000 entries retained.
### Security
- Non-root container images - server runs distroless (no shell, UID 65532), agent runs as configurable `PUID`/`PGID` (default 65532), drops all capabilities then adds only `SETUID`/`SETGID` for identity switching, with a read-only filesystem
- Constant-time token comparison for all shared secrets
- Rate limiting on auth endpoints
- Security headers middleware
---
## Documentation
Full docs: [docs.blackboxd.dev](https://docs.blackboxd.dev)
- [Quick Start](https://docs.blackboxd.dev/docs/getting-started/quick-start)
- [Single-Node Deployment](https://docs.blackboxd.dev/docs/deployment/single-node)
- [Multi-Node Deployment](https://docs.blackboxd.dev/docs/deployment/multi-node)
- [Server Environment](https://docs.blackboxd.dev/docs/configuration/server-environment)
- [Agent Environment](https://docs.blackboxd.dev/docs/configuration/agent-environment)
- [Data Sources Overview](https://docs.blackboxd.dev/docs/data-sources/overview)
- [Authentication](https://docs.blackboxd.dev/docs/operations/authentication)
- [MCP Server](https://docs.blackboxd.dev/docs/integrations/mcp-server)
- [API Reference](https://docs.blackboxd.dev/docs/integrations/api-reference)
---
## Building from Source
Development setup, local run instructions, and contributor docs live at:
- [Development Setup](https://docs.blackboxd.dev/docs/contributing/development-setup)
- [Testing](https://docs.blackboxd.dev/docs/contributing/testing)
- [Project Structure](https://docs.blackboxd.dev/docs/contributing/project-structure)
---
## License
[AGPL-3.0](LICENSE)
---
## AI Usage
A disclaimer - generative AI is used in the development of this repository. The agenda, features, roadmap, etc. are all set by me (a human), but a large portion of the code in this project is created by generative AI. I scan this code for issues and vulnerabilities the best I know how, but I'm not an experienced programmer.
If that makes you uncomfortable, please feel free to poke around the codebase and submit issues for anything out of place. I welcome feedback and suggestions from those more experienced than me. Please send me a private message if you find a security vulnerability that may affect other users, so I can fix it before informing everyone.
---
## Contributing
Contributions are welcome. Please review the contributing docs before doing so.
[Contributing](https://docs.blackboxd.dev/docs/contributing/overview/)
---
Built for homelabbers who want to know what broke their lab at 2 AM.