# pianist-fly

## Project Overview

Google scientists recently mapped the [complete brain connectome of a male fruit fly (MaleCNS v1.0)](https://research.google/blog/a-connectomics-milestone-mapping-the-complete-male-fruit-fly-brain/)—a static anatomical blueprint of 166,700 neurons and 25.6M chemical synapses processed into `data/graph.npz`. Rather than a trained AI, this raw matrix has no learning rules; our engine (`src/native_brain.py`) supplies the physics via a real-time Leaky Integrate-and-Fire (LIF) simulation, driving a 3D MuJoCo fruit fly that senses spatial sugar plumes and aims leg strikes at piano keys.


## How It Works

The simulation operates as a closed loop between sensory chemistry, biological circuit dynamics, and physical robotics:

1. **Sensing & Olfaction (`src/brain_driver.py`):** Calculates bilateral concentration ($c_L, c_R$) from a spatial sugar plume ($1 / (1 + (r/\sigma)^2)$, $\sigma = 0.15\text{ m}$) placed over a keybed coordinate ($y$). Drives 23 antennal sensory neurons (`sugar` receptors) and updates motivation/hunger dynamics.
2. **Circuit Dynamics (`src/native_brain.py`):** Runs a vectorized LIF kernel over the 166,700-neuron, 25.58M-synapse connectome graph (`data/graph.npz`) at $dt = 0.1\text{ ms}$ with $1.8\text{ ms}$ synaptic delay queues. Motor pool rates ($T1L, T1R$) are calculated every $28.6\text{ ms}$ sim tick and published via atomic JSON IPC (`FLY_ACTIVITY`).
3. **Robotics & Striking (`src/play_piano.py`, `src/leg_ik.py`, `src/mirror.py`):** Maps leg choice ($T1_R / T1_L$) to reachable natural keys (15 of 25 keys). Executes Damped Least Squares IK ramps (raise $\rightarrow$ hover $\rightarrow$ descend $\rightarrow$ hold $\rightarrow$ release) on the MuJoCo fly body. Verified physical claw contact on key tops triggers synthesized audio (`src/key_sound.py`) and hops the sugar target.

---

## Demo

**Live brain window** — the 141k-neuron 3D window firing while the fly hunts for sugar.

https://github.com/user-attachments/assets/b2f60532-bbd5-4e43-bc9c-c196fea15598

**Key strikes** — the fly reaching for invisible sugar on the keyboard.

https://github.com/user-attachments/assets/69a055f9-12fc-4785-a6fd-5a18b2a301cd

---

## System Architecture

```
                    ┌───────────────────────────┐
                    │      brain_driver.py      │
                    │   Python, LIF Brain Sim   │
                    │ • 166,700 MaleCNS neurons │
                    │ • Plume + Hunger + Intent │
                    └─────┬───────────────▲─────┘
                          │               │
                   writes │               │ reads
                          ▼               │
                  ┌───────▼──────┐ ┌──────▲───────┐
                  │ FLY_ACTIVITY │ │   FLY_CMD    │
                  └──────┬───┬───┘ └──────▲───────┘
                         │   │            │
            ┌────────────┘   └──────┐     │ writes
            │                       │     │ (sugar pos)
            ▼                       ▼     │
  ┌───────────────────┐   ┌───────────────┴───┐
  │  gnat-brain-male  │   │   play_piano.py   │
  │  (Rust 3D Window) │   │ (MuJoCo 3D Body)  │
  │ • Polls activity  │   │ • Presses keys    │
  │ • Per-soma glow   │   │ • Verifies press  │
  └───────────────────┘   │ • Merges {piano}  │
                          └───────────────────┘
```

---

## Simulation Principles

To keep this simulation clear, reproducible, and scientifically grounded, we enforce a strict boundary between biological network dynamics and kinematic robotics:

* **Measured (Biological Dynamics):** Neural spike counts and firing rates generated by the 166.7k-neuron LIF kernel (`src/native_brain.py`, `data/graph.npz`). Scent inputs drive 23 antennal sugar cells ($12\text{L} / 11\text{R}$ split by postsynaptic partner majority in `src/brain_driver.py`). Synaptic weights are static float32 arrays with zero backpropagation, trained weights, or artificial overrides.
* **Predicted (Physical Kinematics):** Joint trajectories, stance pinning, and claw contact verification (`src/play_piano.py`, `src/leg_ik.py`, `src/mirror.py`). The connectome supplies lateral orientation; Damped Least Squares IK executes the 6-DOF leg reach across the 15 reachable natural keys.
* **Preserving Natural Constraints:** The connectome's genuine dynamics always gate the body — the fly's own L/R antennal firing determines *which side* the sugar is on, and a leg moves only when that read matches the placed sugar (`sugar_should_press`, the absolute gate; zero presses on a wrong aim is asserted by the 5-gate suite `scripts/scent_ab.py`). Centerline key $C_5$ ($y \approx -0.0001\text{ m}$) is the honest "ahead" case: its symmetric plume reads $c_L \approx c_R$, so the gate treats it as straight ahead and presses once the antennae read level — no lateral steering bias is ever injected.

---

## Installation & Setup [Linux only]

### Requirements & Tech Stack
* **Linux:** x86-64 Linux (tested on Arch Linux)
* **Python:** 3.10+ (tested on 3.12)
* **Python Packages (`requirements.txt`):**
  * `numpy` ($\ge 2.1$) — Vectorized LIF neural simulation kernel
  * `pandas` ($\ge 2.2$) & `pyarrow` ($\ge 17.0$) — Connectome annotation metadata
  * `mujoco` ($\ge 3.1.4$) & `dm_control` ($\ge 1.0.24$) — 3D physics engine and environment tasks
  * `flybody` — MuJoCo fruit fly body model (installed by `setup.sh` with `--no-deps`: its upstream metadata pins `numpy==1.26.4`, which conflicts with our verified numpy 2.x stack)
* **System Utilities:** `pulseaudio` / `pipewire` (`paplay` or `pw-play` for host audio), GLFW (`MUJOCO_GL=glfw` for rendering)
* **Rust / Cargo (Optional):** Required only to build the 3D Soma visualizer (`gnat-brain-male`)

### Setup Instructions

1. **Clone:**
   ```bash
   git clone https://github.com/Noir-infini/pianist-fly.git
   cd pianist-fly
   ```

2. **Run Interactive Setup (everything, verified):**
   ```bash
   ./setup.sh
   ```
   *A 7-step interactive install that creates `.venv`, installs packages (with
   progress), fetches the 3 canonical MaleCNS `.feather` tables (live download
   bars, SHA-256 pinned to Janelia), compiles `data/graph.npz`, builds the brain
   point cloud, then verifies the whole thing works headlessly — library
   versions, the MuJoCo fly + keyboard model, and a real connectome boot.
   The optional 3D brain window (`~/gnat`) is verified if present, skipped with
   guidance if not. Pip-installs into `.venv`, matching what `pianist.sh` runs.*
   ```bash
   # optional: the 3D brain window (fly + piano run fine without it)
   cd ~/gnat && cargo build --release -p gnat-brain-male
   ```

3. **Launch the Simulation:**
   ```bash
   ./pianist.sh
   ```
   *Launches the simulation in sugar-reach hunting mode (Brain Driver + 3D Visualizer + MuJoCo Fly).*

---

## Documentation

Comprehensive guides are available in the [`docs/`](docs/) directory:

* [`docs/architecture.md`](docs/architecture.md) — Technical specification for the LIF kernel, olfaction model, and leg IK kinematics.
* [`docs/verification.md`](docs/verification.md) — Automated 5-gate neural test suite, honesty ledger, and biological edge cases.
* [`docs/songs_guide.md`](docs/songs_guide.md) — Song file syntax, 15 natural key mapping, and custom score tutorial.
* [`docs/ipc_schema.md`](docs/ipc_schema.md) — Specification for `FLY_ACTIVITY` and `FLY_CMD` JSON message schemas.

---

## Credits

This project relies heavily on the work of the open-source fly community:

* **[doomfly](https://github.com/nftechie/doomfly)** (MIT) — We used it heavily and it helped this project a lot. The MaleCNS connectome pipeline (`connectome.py` / `prepare.py` → our `scripts/build_graph_npz.py`), the LIF brain kernel, and the retinal decode readouts all trace back to this project.
* **[awesome-fly](https://github.com/cobanov/awesome-fly)** (CC0) — A curated list of fruit fly connectome / neuroscience resources that guided our research and tooling choices.

---

## License

Licensed under the [MIT License](LICENSE).