# FlyDefence ### Investigate security events. Explore real neural structure. Measure what works. A local blue-team research platform connecting evidence-based investigation,
controlled response workflows, and an interactive fruit-fly connectome laboratory. ![Python](https://img.shields.io/badge/Python-3.12%2B-3776AB?style=flat-square&logo=python&logoColor=white) ![Version](https://img.shields.io/badge/version-0.2.0-5be7df?style=flat-square) ![Status](https://img.shields.io/badge/status-active_research-d9b96c?style=flat-square) ![Response](https://img.shields.io/badge/response-sandbox_only-6b7280?style=flat-square) [Quick start](#quick-start) · [Use cases](#where-it-fits) · [Neural laboratory](#neural-laboratory) · [Results](#research-results) · [Roadmap](#roadmap) · [繁體中文](README.zh-TW.md)

![FlyDefence investigation workbench with synthetic demonstration data](docs/assets/overview.png) FlyDefence brings security telemetry, case evidence, human review, reversible sandbox actions, and connectome experiments into one workbench. It is built for people who want to investigate the engineering value of biological network structure while keeping operational decisions traceable. **The software runs today. Its biological detection advantage is not established.** Rules and evidence determine case priority; the fruit-fly model runs in a research lane and cannot authorize a response. Enterprise sensor deployment, external containment connectors, and production acceptance remain open work. The interface and detailed operational manuals currently use Traditional Chinese. This README provides an English entry point. ## What you can do - **Follow an investigation from event to decision.** Browse source-scoped telemetry, inspect a case timeline, request additional evidence from stored records, and review the reasons behind a rule result. - **Import supported sensor exports with provenance.** Parse Windows Security 4624, Sysmon 1/3, and Zeek connection records; retain parser versions, raw-record hashes, and identity-confidence information. - **Keep delivery recoverable.** Queue immutable JSONL snapshots in a SQLite collector outbox, retry interrupted sends, reconcile duplicate receipts, and explicitly quarantine blocking records without silently discarding them. - **Require another person to review consequential changes.** Case feedback and sandbox action authorization use separate actors, evidence references, and policy checks. - **Practice reversible response.** Propose a time-limited sandbox network restriction, authorize it, inspect its state, and verify expiry or rollback behavior. - **Explore a real neural subgraph.** Rotate, filter, inspect, and animate a MaleCNS connectome with traceable neuron IDs and clearly labeled coordinate sources. - **Compare the biological hypothesis with controls.** Run an offline evaluation against a linear baseline, an encoder-only model, and degree-preserving rewired graphs. The runtime uses CPU sparse-matrix computation and serves the interface locally. It does not require an LLM API key or an external inference service. ## Where it fits | Use case | Useful today | Work still needed | | --- | --- | --- | | **SOC investigation** | Event browsing, case timelines, rule explanations, evidence review, and audit records | Reliable cross-source identity resolution and acceptance using authorized enterprise data | | **Sensor onboarding** | Supported export parsing, source permissions, provenance, durable delivery, and failure inspection | Continuous collection, rotation handling, source health alerts, and deployment operations | | **Blue-team training** | Explicitly synthetic scenarios, two-person review, sandbox restrictions, and rollback exercises | Validated integrations with the organization's actual security controls | | **Connectome research** | Real topology, interactive activity traces, pinned inputs, and comparative offline experiments | Independent evaluation, stronger baselines, confidence intervals, and useful performance at low false-positive rates | FlyDefence is currently suitable for a local research or training environment. It has not been accepted as a replacement for an enterprise SIEM, EDR, or identity provider. ## Quick start ### Requirements - Python **3.12 or newer**; the recorded validation used Python 3.12. - Git and a modern browser. - Optional: Docker with Compose and PowerShell 7 for the PostgreSQL deployment helper. - Optional: approximately **1.07 GB** of source downloads, plus working disk and memory, to reconstruct the neural laboratory data. The commands below use PowerShell, matching the currently validated development environment. Run them from the repository root. On macOS or Linux, use `.venv/bin/python` in place of `& $FlyPython`; native startup does not depend on PowerShell helpers. For multiline commands, use your shell's continuation syntax instead of PowerShell backticks. ### 1. Install ```powershell git clone https://github.com/david93518/flydefence.git cd flydefence python -m venv .venv $FlyPython = ".\.venv\Scripts\python.exe" & $FlyPython -m pip install -r requirements.lock.txt & $FlyPython -m pip install -e . --no-deps ``` The lockfile includes development and research dependencies. Private runtime state, credentials, databases, sensor exports, and downloaded datasets are not part of this repository. ### 2. Optionally build the real connectome Skip this step to start the investigation workbench without neural data. To reproduce the neural laboratory shown below, complete it **before the first `init`** so initialization can pin the generated graph manifest.

Download verified source tables and compile the graph

```powershell & $FlyPython scripts/download-malecns.py & $FlyPython -m flydefence.cli compile-graph ` --annotations data/malecns/annotations.feather ` --connections data/malecns/connections.feather ` --output data/graphs/malecns-mb-v2 & $FlyPython scripts/compile-brain-view.py ` --graph data/graphs/malecns-mb-v2 ` --annotations data/malecns/annotations.feather ` --out data/brain/malecns-mb-v2-positions.json ``` The downloader fetches official MaleCNS v1.0 annotation and weighted-connection tables and checks their recorded SHA-256 hashes. It does not download electron-microscopy volumes. The graph compiler preserves neuron IDs and produces an immutable release directory with an artifact manifest; existing output directories are not overwritten. Source pins identify the files observed by this project, rather than publisher-signed attestations. A newly compiled manifest also includes its creation time, so its hash need not equal the historical experiment's manifest hash. If you already initialized the application, preserve your credentials: set `graph_path` in `.local/config.json` to the compiled directory and `graph_manifest_sha256` to that directory's verified `manifest.json` SHA-256, then restart. Do not delete the configuration or rerun initialization to switch graph versions. See the [neural laboratory manual](docs/NEURAL-LAB.zh-TW.md) for provenance and coordinate details.

### 3. Initialize and start ```powershell & $FlyPython -m flydefence.cli init & $FlyPython -m flydefence.cli serve ``` Open **[http://127.0.0.1:8787](http://127.0.0.1:8787)**. Initialization creates unique credentials in `.local/tokens.json`; copy the value of `analyst` into the login field. There is no shared default password. For a second-person review, log out and use the separately generated `approver` token. Native startup uses SQLite. Tokens remain in browser memory and must be entered again after a page refresh. Keep `.local/` private. Initialization refuses to replace an existing configuration. ### 4. Add a demonstration case In a second terminal, from the repository root: ```powershell .\.venv\Scripts\python.exe -m flydefence.cli seed ``` This submits **four explicitly synthetic events** through the authenticated collector APIs. After the worker processes them, open the case for `synthetic-finance-user`. It does not execute an attack or scan a network. The screenshots in this README show this local demonstration environment.

Run the workbench with PostgreSQL and Docker Compose

Initialize the local configuration first. Stop a running native server before using the same port for Compose. ```powershell pwsh -File scripts/prepare-compose.ps1 docker compose --env-file .local/compose.env up -d --build docker compose --env-file .local/compose.env ps Invoke-RestMethod http://127.0.0.1:8787/readyz ``` The helper creates a random PostgreSQL password in `.local/compose.env` without printing it. The Compose deployment includes PostgreSQL 17, the API, the in-process worker, and the static workbench. Both published ports bind to loopback: the application uses `8787`, and PostgreSQL uses `55438`. The application container runs without root privileges, drops Linux capabilities, uses a read-only root filesystem, and mounts configuration and graph data read-only. These controls do not constitute production deployment acceptance. ```powershell # Stop services while retaining the database volume. docker compose --env-file .local/compose.env stop ``` SQLite and Compose PostgreSQL are separate databases. A case seeded into one is not copied into the other. See the [runbook](docs/RUNBOOK.zh-TW.md) for backup, recovery, and configuration details.

## Neural laboratory ![Interactive MaleCNS neural laboratory showing a real connectome subgraph](docs/assets/brain.png) The laboratory shows a selected **MaleCNS v1.0 subgraph**, with populations drawn from ALPN, ALLN, Kenyon cells, MBON, and APL. It supports mouse and keyboard rotation, zoom, group filters, body-ID lookup, node inspection, and playback of synthetic inputs. | Component | What is actually represented | | --- | --- | | **5,269 neurons** | Real source body IDs in the selected subgraph | | **152,689 directed edges** | Full graph used by every simulation; source connections meet a five-synapse threshold | | **Up to 8,000 visible edges** | A deterministic sample of real graph edges for interactive rendering | | **5,244 source positions** | Soma coordinates from the matching annotation table | | **25 estimated positions** | Explicitly marked graph-derived fallback positions | | **16 update steps** | Engineered, dimensionless activity with an all-zero initial state | Points indicate soma locations, and straight lines indicate connectivity. They are not reconstructed axons or dendrites, and the selected subgraph is not a whole fly brain. Source coordinates retain relative proportions; the viewer does not assert unverified physical units or anatomical axis labels. The current dynamics use a seeded engineering encoder and an unsigned, normalized weight matrix: ```text h[0] = 0 drive[ALPN] = encoder × features h[t+1] = 0.5 × h[t] + 0.5 × tanh(0.8 × W × h[t] + drive) ``` `W[post, pre]` follows source-to-target direction. Source synapse counts undergo `log1p` transformation and input-row normalization; neurotransmitter signs are not inferred. Each request resets the state. Activity values are neither firing rates nor attack probabilities. The animation shows synthetic input propagation through this engineering model and does not subscribe to live enterprise telemetry. Tests verify that its final state matches the existing graph computation. See [NEURAL-LAB.zh-TW.md](docs/NEURAL-LAB.zh-TW.md). ## Architecture ```mermaid flowchart LR S[Supported sensor exports] --> O[Durable collector outbox] O --> A[Authenticated ingestion API] A --> V[Adapters and provenance validation] V --> DB[(SQLite or PostgreSQL)] DB --> W[In-process worker] W --> R[Rules and case evidence] W -. shadow features .-> G[Verified connectome] R --> UI[Investigation workbench] UI --> H[Independent human review] H --> M[Reviewed case memory] H --> P[Response policy gates] P --> X[Sandbox executor] X --> B[Expiry and rollback] R --> AU[Local audit chain] P --> AU G --> N[Neural laboratory] D[Pinned public dataset] --> E[Offline comparative evaluation] G --> E ``` **Operational path.** Source credentials fix tenant, source, trust group, accepted formats, and event kinds. Validated records are persisted before the worker creates or updates cases. Event IDs and content hashes support idempotent ingestion and reject conflicting reuse. Rules determine priority; connectome projections remain shadow outputs. **Review and response.** Policies check evidence freshness and revision, independent source support, allowed and protected assets, action limits, and different proposing and approving actors. Authorized proposals are signed, then rechecked before sandbox execution. Expiry and rollback state is persisted. There is no external EDR or firewall executor in this release. **Research path.** Offline experiments and neural visualization cannot change case scores or grant action permissions. Reviewed memories provide annotations rather than automatic suppression. The audit chain supports local integrity checks, but it is not externally anchored and cannot prevent an administrator from rewriting the entire database. ## Research results The current question is specific: **does the selected biological topology improve classification under the implemented feature and readout protocol?** The completed experiment does not establish that it does. The recorded UNSW-NB15 experiment used **175,341 training-pool rows** split into **140,273 fit / 35,068 validation**, followed by **82,332 test rows**. Preprocessing was fit on the fit partition. Readout regularization and classification thresholds were selected on validation and frozen before the full test was evaluated. | Representation | Test average precision | Test false-positive rate | | --- | ---: | ---: | | Logistic regression, 8 features | 75.44% | 53.15% | | **Encoder only** | **82.34%** | **51.36%** | | Real connectome | 76.77% | 51.98% | | Degree-preserving rewiring, seed 17 | 77.20% | 55.09% | | Degree-preserving rewiring, seed 29 | 77.28% | 54.15% | | Degree-preserving rewiring, seed 43 | 76.06% | 54.31% | These false-positive rates use each model's validation-F1-selected threshold. They are **not acceptable evidence for enterprise alert deployment**. The real graph did not consistently outperform rewired controls and trailed the encoder-only representation. Interpretation also depends on the experiment's limits: - The eight offline flow features differ from the workbench's eight session features. The trained offline readout is not connected to operational decisions. - A smaller sample from the same test file had already been examined. The full run is an expanded evaluation, not an independent replication. - **43.14%** of test rows shared the selected eight-feature profile with training-pool rows. A matching projection does not necessarily identify the same original flow, but it limits conclusions about generalization. - The data came from a pinned third-party mirror; content hashes do not authenticate equivalence to a publisher-certified copy. - Rewired seeds share the same test set. They do not provide independent test populations or confidence intervals. The repository retains the [full machine-readable result](docs/evidence/unsw-heldout-full.json), [evaluation protocol](docs/EVALUATION.zh-TW.md), [source manifest](docs/evidence/unsw-source-manifest.json), and [validation report](docs/VALIDATION.zh-TW.md). Dataset CSV files are not distributed here. Reproduction instructions, the common optimizer budget, source requirements, and usage restrictions are documented with the experiment. ## Telemetry and API ### Supported inputs | Input | Scope in this release | | --- | --- | | Windows Security | Event 4624, exported JSON or individual XML records | | Sysmon | Events 1 and 3, exported JSON or individual XML records | | Zeek | Connection records in JSON | | Canonical event API | Explicitly authorized sources providing normalized semantic facts | Newly registered collectors default to raw-record ingestion and cannot bypass the parser by asserting canonical facts. Raw network byte counts are not automatically labeled exfiltration; process creation is not automatically labeled sensitive-file access. Identity fields remain scoped to their sources until reliable cross-source mapping is available. The collector consumes immutable JSONL snapshots. It is not yet a Windows Event Log subscriber, an EVTX decoder, or a daemon that follows Zeek log rotation. Start with the [collector guide](docs/COLLECTOR.zh-TW.md) and [adapter contracts](docs/ADAPTERS.zh-TW.md). ### API map All `/v1/` endpoints require an appropriate Bearer credential. Collector and operator roles have different access; a collector token cannot open the neural laboratory. The authenticated schema is available at `/v1/openapi.json`. | Endpoint | Purpose | | --- | --- | | `GET /healthz`, `GET /readyz` | Process health; database and worker readiness | | `GET /v1/me`, `GET /v1/collector` | Operator identity or collector's own source configuration | | `POST /v1/events:raw` | Validate and ingest supported raw records | | `POST /v1/events:batch` | Ingest authorized canonical events | | `GET /v1/events`, `GET /v1/overview` | Filtered telemetry and workbench state | | `GET /v1/cases`, `GET /v1/cases/{case_id}` | Cases, evidence, and assessment details | | `POST /v1/cases/{case_id}/feedback` | Submit evidence-linked feedback for review | | `POST /v1/actions:propose` | Propose a bounded sandbox response | | `POST /v1/actions/{action_id}:authorize` | Authorize through a separate permitted actor | | `GET /v1/brain/graph`, `POST /v1/brain/simulate` | Traceable neural structure and synthetic-input dynamics | | `GET /v1/research/evaluation` | Recorded offline evaluation summary | | `GET /v1/audit`, `GET /v1/audit:verify` | Audit records and local chain verification | This is an orientation map, not the full contract. Request models live in [`flydefence/contracts.py`](flydefence/contracts.py); routes and role checks live in [`flydefence/api.py`](flydefence/api.py). ## Validation and development The recorded **2026-09-25** local validation passed **171 tests on SQLite** and **171 tests on an isolated PostgreSQL test database**. Browser checks covered the existing workflows, neural controls, login/logout cleanup, and desktop and 390px mobile layouts. These are saved verification results, not a hosted CI status or a production capacity claim. ```powershell & $FlyPython -m pytest -q & $FlyPython -m ruff check flydefence tests ``` Tests cover contracts, provenance, authorization boundaries, delivery recovery, policy enforcement, graph consistency, simulation inputs, and evaluation mechanics. PostgreSQL testing must use a dedicated test database, never the workbench or a production database. Optional browser scripts require Node.js, Playwright, and Edge; see the [interface validation report](docs/INTERFACE-VALIDATION.zh-TW.md) for their setup and evidence. | Evidence | Report | | --- | --- | | SQLite regression | [sqlite-neural-validation.json](docs/evidence/sqlite-neural-validation.json) | | PostgreSQL regression | [postgres-neural-validation.json](docs/evidence/postgres-neural-validation.json) | | Neural and guide interactions | [browser-neural-validation.json](docs/evidence/browser-neural-validation.json) | | Existing workbench workflows | [browser-redesign-validation.json](docs/evidence/browser-redesign-validation.json) | ### Technology | Layer | Implementation | | --- | --- | | API and contracts | Python, FastAPI, Pydantic, Uvicorn | | Persistence | SQLAlchemy, SQLite, PostgreSQL, Psycopg | | Research computation | NumPy, SciPy sparse matrices, PyArrow | | Workbench | HTML, CSS, vanilla JavaScript, Canvas 2D projection | | Deployment | Docker Compose, local Python environment | | Verification | pytest, Ruff, Playwright browser scripts |

Repository layout

```text flydefence/ ├── api.py API, authorization, and workbench delivery ├── adapters.py Raw sensor record parsers and provenance ├── collector.py Durable export outbox and receipt handling ├── service.py Investigation, review, and response policies ├── detection.py Deterministic evidence-based rules ├── connectome.py Graph compilation and shadow computation ├── brain.py Coordinate provenance and simulation traces ├── evaluation.py Offline training and comparative evaluation ├── static/ Workbench, neural viewer, and product guide └── evaluation_result.json Compact recorded experiment summary tests/ Functional and boundary tests scripts/ Setup, graph reconstruction, and verification docs/ Design, research, operations, and evidence compose.yaml Local PostgreSQL deployment requirements.lock.txt Recorded dependency versions ``` Generated `.local/` state and `data/` artifacts are intentionally excluded from version control.

## Roadmap The next milestone is a **verified first real source**: ingest authorized exports and reconcile source records, accepted counts, timestamps, identities, and delivery latency. Neural visualization does not substitute for this acceptance step. | Workstream | Current foundation | Completion criterion | | --- | --- | --- | | **Continuous collection** | Export adapters and durable outbox | A deployed source survives rotation, disconnects, retries, and restarts with reconciled counts and actionable health alerts | | **Identity correlation** | Source-scoped IDs and confidence metadata | Independently checked account, host, process, and connection mappings across sources | | **Detection quality** | Reproducible baselines and topology controls | A new authorized evaluation with time/entity separation, fair nonlinear baselines, uncertainty estimates, and an agreed low-FPR operating point | | **External response** | Two-person policy gates and sandbox rollback | A real EDR, IAM, or firewall connector with verified execution, failure handling, expiry, and recovery | | **Enterprise access** | Local role-scoped Bearer credentials | SSO/MFA, credential lifetime and rotation, and a validated TLS deployment | | **Operations and assurance** | Local persistence, audit checks, and backup tooling | External audit anchoring, explicit schema migration, tested capacity, and recovery objectives demonstrated in drills | There are no claimed delivery dates or invented completion percentages. The [uses and gaps guide](docs/USES-AND-GAPS.zh-TW.md) describes the acceptance boundaries; the [delivery plan](docs/DELIVERY.zh-TW.md) separates product goals from implemented behavior. ## Documentation Detailed documents are currently in Traditional Chinese. Start with the [中文導覽](README.zh-TW.md). | Read this when you need… | Document | | --- | --- | | The full system design and threat model | [系統設計](docs/SYSTEM-DESIGN.zh-TW.md) | | Current implemented behavior | [實作狀態](docs/IMPLEMENTATION.zh-TW.md) | | Practical uses and remaining enterprise gaps | [用途與待補齊](docs/USES-AND-GAPS.zh-TW.md) | | Startup, credentials, backups, and failure handling | [執行與維運](docs/RUNBOOK.zh-TW.md) | | Collector registration and durable delivery | [接入與重送](docs/COLLECTOR.zh-TW.md) | | Exact supported record formats | [感測器轉接器](docs/ADAPTERS.zh-TW.md) | | Neural coordinates, dynamics, and provenance | [神經實驗室](docs/NEURAL-LAB.zh-TW.md) | | Experimental methods and reproduction | [離線評估](docs/EVALUATION.zh-TW.md) | | Literature, prior work, and novelty boundaries | [研究查證](docs/RESEARCH.zh-TW.md) | | Recorded backend and research checks | [驗證報告](docs/VALIDATION.zh-TW.md) | | Recorded UI and neural checks | [介面驗證](docs/INTERFACE-VALIDATION.zh-TW.md) | | Concept-art provenance | [素材紀錄](docs/ARTWORK.zh-TW.md) | ## Contributing Useful contributions include reproducible adapter fixtures, failure-recovery tests, accessible interface improvements, better evaluation controls, and English translations of the operational documentation. Begin with [CONTRIBUTING.md](CONTRIBUTING.md). Report ordinary bugs with the expected behavior, reproduction steps, version, and a sanitized example. Do not upload credentials, enterprise logs, or personal data to issues. See [SECURITY.md](SECURITY.md) for security-reporting guidance. Changes that affect model claims should include the protocol and evidence needed to evaluate them. A compelling animation or a successful unit test is not evidence of improved intrusion detection. ## Attribution and licensing The MaleCNS source data is attributed in graph manifests to **HHMI Janelia FlyEM, the University of Cambridge, MRC LMB, and Google Research**. See the [MaleCNS download page](https://male-cns.janelia.org/download/) and the [research notes](docs/RESEARCH.zh-TW.md) for source context and references. The project records its selected MaleCNS source under CC BY 4.0; consult the source terms when obtaining or redistributing data. UNSW-NB15 remains subject to its authors' dataset terms, including the distinction between research and commercial use. See the [original dataset page](https://research.unsw.edu.au/projects/unsw-nb15-dataset) and the repository's [source manifest](docs/evidence/unsw-source-manifest.json). Dataset files are not bundled with the source code. The login and overview use conceptual artwork generated through Higgsfield. It is labeled as artwork and is separate from the data-driven neural renderer; provenance is recorded in [ARTWORK.zh-TW.md](docs/ARTWORK.zh-TW.md). **A software license has not yet been selected for this repository.** Public source availability does not itself grant a general license to reuse, modify, or redistribute the code, and does not change third-party data or asset terms.