# fly-experiment

**A living fly and a retained MaleCNS fly connectome, both driving Doom through
the same control contract — then the connectome alone at a chess board.**

```
CONNECTOME FLY   vs   LIVING FLY
        \              /
         \            /
           DOOM + CHESS
```

There are already mapped fly connectomes — no need to invent a fake fly
trajectory, or build a new game. Underneath it was a bigger curiosity: where AI
goes next probably isn't only bigger models — people are already exploring
neurons and cells as compute. This is a much smaller, desk-scale version of
that idea: physical, sophisticated hardware instead of only models. A small
budget, a desk, some binder clips, and the principle that if something looks
fun to try, I build it.

> This repo holds the experiment layer built for this project — the living-fly
> interface, chess adapter, benchmark tooling, media, and recorded evidence.
> It extends an upstream connectome–ViZDoom codebase; derived files keep their
> upstream licensing.

## The rig

| | |
|---|---|
| ![Arena close-up](media/setup-photo-1.jpg) | ![The bench](media/setup-photo-2.jpg) |
| The arena — transparent plates, circular enclosure, binder clips, and the fly as a speck in the disc | The bench — arena, a second dish with a tubing line, phone, cabling |

| | | |
|---|---|---|
| [![rig walkthrough](media/setup-1.jpg)](media/setup-1.mp4) | [![inside the arena](media/setup-2.jpg)](media/setup-2.mp4) | [![behind the scenes](media/setup-3.jpg)](media/setup-3.mp4) |
| the whole rig (35s) | inside the arena (38s) | behind the scenes (15s) |

- **Phone** — overhead camera, streamed over DroidCam.
- **Laptop** — turned tracked movement into the shared control fields, ran the
  game, and generated the FlyVision feedback signal shown back to the fly.
- **Flies** — borrowed from teaching practicals, moved in and out of the arena
  with a length of tubing as a handheld aspirator, and returned after sessions.
- **Nothing aversive** — no electrodes, no shocks, no sound or electrical
  punishment. Movement was the only signal in; light on a screen was the only
  feedback out; sucrose was the only reward.

![The real tracking view](media/tracker-view.jpg)

*The actual tracker view: arena bound, orientation marker, the fly a speck in
the disc. The marker stayed mostly on the fly with periodic shifts onto clips
and rim shadows — watched live and re-anchored by hand.*

## What ran

Four channels through the same control contract (ViZDoom `combat_survival`,
seed 41027):

| Channel | Observed episodes | Observed kills* |
|---|---:|---:|
| Living fly | 9,098 | 3,758 |
| Connectome under the fly's recorded envelope | 8,832 | 3,170 |
| Connectome, unconstrained | 563 | 6,527 |
| Random baseline | 8,468 | 6,820 |

\* Sum of available per-(run, episode) maximum kill counters, deduplicated —
coverage and protocols differ, so read these as observed counters inside this
setup, not a formal benchmark.

**The fair comparison is the middle pairing**: connectome vs connectome limited
to the fly's own recorded envelope. Under matching signal constraints the
living channel ran ahead (3,758 vs 3,170) — the fly did more with the same
budget, and the rig only ever captured a fraction of the signal it had.

## Then chess — connectome only

A separate experiment: board pixels and a sorted legal-move menu fed the
retained MaleCNS network; fixed decoded controls moved a menu cursor. Opponent:
a seeded random legal-move sampler. One completed game — **0–1, checkmate at
93...Rg1#**, 186 plies, replay-checked for legal moves and input-frame hashes.
An interface experiment, not evidence of chess understanding.

| | |
|---|---|
| ![The experiment page at game end](media/chess-interface.jpg) | ![Recorded neural selection replay](media/selection-replay.gif) |
| the page it actually ran on | recorded selections + descending-neuron readout rates |

The living fly never got a board — the rig captured enough signal to steer
Doom, not enough for 186 plies of menu navigation. That gap is itself part of
the finding.

Evidence: [`media/match.pgn`](media/match.pgn) ·
[`media/protocol.json`](media/protocol.json) ·
[`media/summary.json`](media/summary.json)

## Field notes

- **It kept apart from its mates.** Returned to the vial after sessions, the
  fly didn't go where the others were going — it stayed apart from the group.
  And inside the arena, stillness produced no motion signal: stillness was
  logged as stillness, never as a finished turn.
- **One substrate kept improving.** The fly's game-relevant signals got better
  across sessions; the connectome was static after setup.
- **Exploration before tailoring.** Before the feedback signal was tuned toward
  the game, the fly roamed more of the arena. My own thought, not a result:
  once an agent knows enough, a constrained channel may reveal intent better
  than an open world — worth remembering for how we constrain models, even in
  RL, and note what they do.

## The code

**Living side** — `real_fly_tracker.py` (DroidCam → OpenCV candidate blobs,
arena bound, drift re-anchoring, honest lost-state), `real_fly_server.py`
(action server on `:8780`, FlyVision), `fly_tracker.py`, `night_watch.py`
(keep-alive/relaunch), `run_fly.ps1`, `start_all.ps1`, `envelopes/` (recorded
fly walk/pause envelope used by the gated arm).

**Connectome side** — `doom/` modifications (retained MaleCNS v1.0 kernel
wiring, ViZDoom loop, fixed DN decoder), `doom_learning*/` variants,
`tools/chess_native.py` (board/menu RGB → neural controls, recorded evidence),
`tools/chess_match.py`, `tools/serve_chess_run.py` (read-only board for a
finished run), `tools/scoreboard.py` (deduplicated consolidation).

**Docs & evidence** — `BENCHMARKS.md`, `REALFLY.md`, `INNOVATIONS.md`,
`AGENTS.md` (project constraints), `RESULTS.ipynb`, `outputs/` (scoreboards,
chess run records, audits, consolidated telemetry), `tools/fly_frames/`
(tracker debug frames).

## Honest notes

- The retained graph (166,700 neurons / 25,582,938 edges) is the measured
  wiring map; retinal projection, dynamics, and the DN→button decoder are
  engineering choices. A full retained connectome is not a literal living
  brain, and a game score is not biological validation.
- Game-server counters are preserved as recorded; channels differ in coverage
  and protocol. The connectome graph data and large session recordings are not
  in this repo (size) — see `_fetch_malecns.py` for the upstream MaleCNS v1.0
  release.

*Fun project. September 2026.*