# fly-ears: a real fruit fly brain watches your videos and hears your music

**Project page: <https://arpanguria68-ui.github.io/fly-ears/>**

[![Walkthrough: a real fly brain watches a video (80 s, click to play)](docs/walkthrough.jpg)](docs/walkthrough.mp4)

**[Watch the 80-second walkthrough](docs/walkthrough.mp4)**: give it a video, light enters its eyes, its own
wiring sees an approaching disc coming, brain, body and emotions, every other sense, the music test, and
how every run is checked.

![The fly watching the test video: a dark disc approaches, its looming neurons (LPLC2) fill, the brain map lights up](docs/demo.gif)

| the Watch page | the Brain & Body viewer |
|---|---|
| ![Watch page: paste a link or pick a file, press Watch; the job log on the right](docs/watch.png) | ![Viewer: the video, the brain map, a 3D fly posed by its motor neurons, emotions](docs/viewer.jpg) |
| **Inside the brain:** the retina, both eyes' motion detectors at their places in the eye | |
| ![Inside the brain: retina, left and right eye T4/T5 maps, ears](docs/inside.jpg) | |


A video or a song goes into the eyes and ears of the complete fruit fly connectome (MaleCNS: 166,700
neurons, 25.6 M connections from an electron-microscope map, simulated by `flybrain` from
[fly.ai](https://github.com/alextitonis/fly.ai)), frame by frame, in the video's own time. Everything after
the receptors is the fly's own wiring: nothing is trained, nothing is drawn in.

* **Watch:** paste a link or pick a file. The light reaches 6,006 photoreceptors at their real eye
  columns, the soundtrack reaches the ears, and you get the video beside the brain: every neuron that
  fires, both eyes' motion detectors, the escape neuron, a 3D fly posed by its own motor neurons, and
  readouts of fear, appetite, compass heading and more.
* **Listen:** a test, decided before running, of whether the fly brain's response tells human-made
  music from AI-generated music, beyond what its ears received and beyond a scrambled brain.
* **Honest by design:** every watched video is re-simulated and checked step for step (`audit.py`);
  what is measured, what is modelled and what is only drawn is said on every page; the failures are
  written down below (the fly's motion detectors are far less direction-selective than a real fly's).

## Setup

Needs Python 3.10+, `ffmpeg` on the PATH, and an NVIDIA GPU for real time (a CPU works, slowly).

```bash
# 1. the brain: the flybrain code, next to this folder, named firefly
git clone https://github.com/arpanguria68-ui/flybrain-nes firefly
git clone https://github.com/arpanguria68-ui/fly-ears
cd firefly
python -m venv .venv
.venv/Scripts/pip install -r requirements.txt        # Linux/macOS: .venv/bin/pip
.venv/Scripts/pip install -r requirements-gpu.txt    # NVIDIA GPU (CuPy)
.venv/Scripts/python -m flybrain download            # the prebuilt brain, ~260 MB, into ~/fly-data
cd ../fly-ears

# 2. try it: make the test video and let the fly watch it (about 10 s)
../firefly/.venv/Scripts/python make_test_video.py
../firefly/.venv/Scripts/python watch.py out/testvideo/test_events.mp4 --seconds 7

# 3. the browser app: watch, results, brain & body viewer, library, music test
../firefly/.venv/Scripts/python ui.py                # http://127.0.0.1:8790/
```

The first run also fetches the MaleCNS optic-column table (~1 MB) to place every photoreceptor on its
eye column. Optional:

* **3D body** (NeuroMechFly, needs Python 3.12, ~680 MB): `py -3.12 -m venv .venv-body` then
  `.venv-body/Scripts/pip install flygym==2.1.0` in this folder; each watched video then gets `body3d.mp4`.
* **YouTube links:** `pip install yt-dlp` in the brain's environment. Use videos you have the right to
  download and use.
* **Checks:** `python -m pytest tests -q` (no GPU needed), `python physio.py` (the eye like a real fly's),
  `python looming.py` (an approaching disc, 5 seeds), `python audit.py out/watch/<name>`.

Your videos, songs and results stay on your PC (`out/` and `songs/` are ignored by git).

## Watch a video: the fly sees and hears it, frame by frame

```powershell
..\firefly\.venv\Scripts\python.exe watch.py "https://www.youtube.com/watch?v=..." --seconds 60
..\firefly\.venv\Scripts\python.exe watch.py my_video.mp4 --start 30 --seconds 60
..\firefly\.venv\Scripts\python.exe watch.py my_video.mp4 --rewired      # scrambled brain, to compare
```

Links need `yt-dlp` (`pip install yt-dlp`); a downloaded file works without it. Use only videos
you own or are licensed to use: downloading may be restricted by the site's terms, and other people's
videos are copyrighted. Don't publish results made from videos you have no rights to.

Every frame (25 a second = two 20 ms brain steps; 50 for 50 fps videos) reaches the brain the way a
fly would get it. The frame is spread over both eyes, its left half on the left eye (front of each eye
at the frame's centre):

| sense | neurons | from the video |
|---|---|---|
| light | 6,006 photoreceptors (R1-6, R7, R8) | the frame's physical light at each one's eye column, through the eye's optics and its rhodopsin (see Vision) |
| motion, looming | T4/T5, LPLC2 and everything after | nothing injected: the fly's own optic lobe computes them from the photoreceptors (fly-own vision, the default) |
| sound | Johnston's organ (138) | the soundtrack, as for songs |

Out (`out/watch/<name>/`): **`fly_watching.mp4`**, the video beside the fly's brain live (every
neuron that fires, both eyes' motion detectors, the ear bands, the response of each pathway),
with the original sound; `timeline.csv` (every 20 ms: each group's spike rate, looming, ear bands);
`summary.json` (each group while watching vs at rest). About 0.9 s of compute per video second (fly-own; 0.4 s assisted) on
a GPU.

Checked on a test video with known events (bar moving right, then left, then a disc approaching;
beeps over the first half):

| response (spikes/neuron/s, fly-own vision, seed 1) | bar moving right | bar moving left | approaching disc | still | rest |
|---|---|---|---|---|---|
| T4/T5 (motion) | 0.83 | 0.73 | 0.80 | 0.64 | 0.48 |
| LPLC2 (looming) | 1.36 | 1.88 | **2.92** | 0.53 | 0.97 |
| escape (giant fiber) | 1.11 | 1.39 | **2.22** | 0.00 | 0.50 |
| ear JO-A/B (beeps in the first two) | 5.53 | 4.47 | 0.33 | 0.33 | 0.32 |

(`make_test_video.py` makes this video; `python watch.py out/testvideo/test_events.mp4 --seconds 7`.)

## In the browser

```powershell
..\firefly\.venv\Scripts\python.exe ui.py          # opens http://127.0.0.1:8790/
```

Paste a link or pick a video file, choose where to start and how long, press **Watch**: the log
shows the fly at work, then the result plays on the page with the biggest responses. The
**Library** keeps everything watched; **Music test** runs the human-vs-AI test on a folder and
shows its report. One job at a time (the brain uses the GPU).

## Brain, body and emotions (the viewer)

Every watched video gets a viewer (**Brain, body & emotions** on the result, or
`/view?name=<video>`), synced to the clip:

* **Brain**: every neuron that fires, live.
* **Body**: a fly driven by its real motor neurons. The connectome includes the nerve cord, so
  the wings follow the flight power motor neurons (DLM, DVM) and steering ones (b1-3, hg1-4...),
  each of the 6 legs its own leg motor neurons (front/middle/hind told apart by their position
  along the nerve cord, side by side), plus neck and abdomen. Command neurons overlay what the fly
  "decides": walk (DNg100), turn (DNa02 left vs right), escape (DNp01, the giant fiber), back up
  (MDN), groom (DNg11), sing (pIP10).
* **3D body**: NeuroMechFly (EPFL, Apache-2.0; Lobato-Rios et al. 2022, Wang-Chen et al. 2024), an
  anatomically accurate fly from a micro-CT scan of a real fly (6 legs with all segments, wings,
  halteres, head, antennae, proboscis, abdomen), rendered with MuJoCo and posed frame by frame by the
  same motor neurons (each leg by its own, wings by flight power / steering, head by the neck motor
  neurons, abdomen, jump muscle on the middle legs; fixed scale: +10 spikes/neuron/s = full movement).
  No muscles, physics, gait or wingbeat is invented. It runs in its own environment because flygym needs
  Python 3.12: `.venv-body` in this folder (see Setup), or any Python with flygym, set in `FLY_BODY_PY`;
  `watch.py` renders `body3d.mp4` after each video when it is there, or run `body3d.py out\watch\<name>`
  with it.
* **Emotions**: populations that carry internal states in real flies, against the same brain at
  rest: fear (giant-fiber escape circuit), pleasure (PAM dopamine), distress (PPL1 dopamine),
  excitement (octopamine), desire (pC1), anger (aIPg), mood (serotonin), appetite (NPF). A readout
  of this model, not a measured feeling.
* **Emotions over time** (click to jump) and **strongest moments** (each state's biggest peaks).
* **Inside the brain**: the retina (the grey picture the eyes get), both eyes' 13,581 motion
  detectors at their places in the eye coloured by preferred direction, the ear's frequency
  bands with light / looming / motion per side, left vs right for every paired pathway (eyes,
  ears, hearing relay, descending neurons, turning, wings, each leg, neck), a spike raster of real
  neurons (4 per group, last 5 s) and a live table of every recorded group against rest.

On the test video, with the fly's own vision, the escape neuron (DNp01, the giant fiber) fires more
while the disc approaches than for the sliding bars in 5 of 5 noise seeds (2.39 vs 1.03 spikes/s, 0.35
at rest; `looming.py`): the approaching disc is detected by the fly's own wiring, from the
photoreceptors on. The jump motor neuron (TTMn) does not follow. The reward dopamine
neurons (PAM) fire on every step at rest and while watching (50 spikes/s, this model's ceiling) and
the punishment ones (PPL1) nearly so: saturated, they cannot report anything here, and the viewer
marks them so.

## Vision: the fly's own (default) or assisted

**Fly-own vision (default)** sees only through the fly: each photoreceptor gets the light at its own
eye column, and the connectome does everything after that. Nothing of this program detects motion or
looming. How the light gets in (`senses.ColumnEyes`), checked against the fly:

* **where each photoreceptor looks**: 5,968 of the 6,006 photoreceptors in the dataset sit on an eye
  column (MaleCNS optic-column table; R1-6 by the lamina cartridge they feed). The eye's front/up axes are
  set from the T4 cells' wiring, so they were checked independently against the 3-D anatomy (axes from
  landmarks: antennal lobe front, central complex back, VNC behind; dorsal = small y). In the lamina,
  "front" lies anterior (correlation +0.88 left, +0.72 right) and "up" dorsal (+0.94, +0.99); in the
  medulla front/back is reversed (-0.88, -0.91), as the real optic chiasm crosses it, and up is kept.
  Each eye's own photoreceptors are on its own side (all 5,968). The 38 without a column get their
  eye's mean light;
* **optics**: an ommatidium sees a blurred patch, not a point. Its acceptance angle (Gaussian, full width
  at half maximum) is set to the spacing between neighbouring columns, as in Drosophila (both ~5 deg);
  here 2.63 px, so the frame is blurred with sigma 1.12 px before sampling. Before this fix each
  photoreceptor read one pixel, about 1/7 of its patch;
* **light**: video pixels are gamma-encoded (sRGB); they are turned back into the screen's physical
  light before the eye. Photoreceptors adapt divisively, output light / (light + adapted light) over
  ~1 s, so the same contrast gives the same response in dim and bright scenes (Weber), as real ones do.
  Before this fix they subtracted gamma-encoded values;
* **colour**: each type weighs the screen's red, green and blue by its rhodopsin (rough reading of the
  curves in Salcedo et al. 1999): R1-6 (Rh1) nearly red-blind, R8 as Rh6 (the dataset does not say which
  R8s are the blue-sensitive "pale" ones, ~30%), R7 (UV) only the blue primary, since screens emit no UV.
  Before this fix R1-6 took the plain mean of red, green and blue;
* **what is not like a real fly**: the video is spread over each eye's whole field (front of the eye
  at the frame's centre, back of the eye at its edge), as if the fly were inside a panoramic screen, so
  a normal 60-90 deg camera view is magnified several times. The dataset has fewer R1-6 than a real eye
  (left 1,112, right 2,228, against ~4,500 per eye); R7/R8 (625 and 704 columns) are close to a real
  eye's ~750. Frames come at the video's rate (25 or 50 per second), which a fly sees as flicker.

To make that work like a real fly, `flyears/hybrid.py` simulates the optic lobe the way it works in a
fly: **graded** neurons (photoreceptors, lamina, medulla, lobula, T4/T5 signal with smooth voltage around
a resting level, so inhibition can lower their output), photoreceptors that **adapt** to the recent light
(each signals contrast against its own ~1 s average, as real ones do) with **type-specific speeds** from published
measurements (Mi1, Tm3, Tm1, Tm2, Tm4 and the lamina fast, ~10 ms; Mi4, Mi9, Tm9 slow, 150 ms; Behnia et
al. 2014, Arenz et al. 2017; see below for how 150 was chosen), and keeps the central brain, descending and motor neurons **spiking**
(fly.ai's model, at 5 ms steps). Resting levels are set label-free (every graded cell rests at the same
level under a plain grey view); a graded cell passes its change from its own rest on to spiking
partners. In rates, rasters and maps a graded cell counts while it is depolarised above its own rest, so they
show responses, not the resting level. With fly.ai's all-spiking model the signal stopped at the photoreceptors' first synapse: their
synapses onto the lamina are inhibitory, and spiking cells resting below threshold cannot carry that.

Checked like a real fly (`physio.py`, gratings and edges into the photoreceptors only):

* every stage has the real fly's sign for a light step: photoreceptors up; L1, L2, L3 down; the ON
  cells Mi1, Tm3, Mi4 up; the OFF cells Mi9, Tm1, Tm2, Tm4, Tm9 down;
* **T4 prefers bright edges and T5 dark edges** (T4 +0.040 vs -0.031; T5 -0.039 vs +0.030);
* **direction selectivity emerges from the wiring**: 13 of 16 T4/T5 subtype x eye groups respond most
  to the real fly's direction (a front-to-back, b back-to-front, c up, d down) at 1 Hz; **but weakly**:
  mean index +0.04 on response amplitude, +0.04 on depolarisation only (as calcium imaging measures it,
  positive in 10 of 16). (Before the optics fix above: 15 of 16, +0.04 and +0.06.) A real fly's is about 0.3-0.8. **This is not solved** (see below);
* on the test video, over 5 noise seeds (`looming.py`), **LPLC2 and the escape neuron (giant fiber)
  respond more to the approaching disc than to the sliding bars in 5 of 5** (LPLC2 2.98 vs 1.69, rest
  0.93; giant fiber 2.39 vs 1.03, rest 0.35 spikes/s).

What was tried for the weak direction selectivity, with the numbers:

* **timing** (`ds_timing.py`): T4/T5 compare a fast input with a delayed one, so the slow inputs' time
  constant matters. Slow 50 ms: 12/16 best at 1 Hz (16/16 at 3 Hz), mean index +0.02. Slow 150 ms:
  15/16 at 1 Hz, +0.04. Kept at 150 ms; measured slow medulla cells range up to a few hundred ms, but
  this was chosen by the test, so it is a fitted number;
* **shunting inhibition** (`tune_shunt.py`; inhibitory input also divides a cell's response, as
  conductance synapses do): no help (strength 0: +0.02, 12/16; strength 40: -0.00, 7/16). Kept at 0;
* the likely missing pieces, not modelled: real T4/T5 multiply/divide their inputs at the dendrite
  (nonlinear), synapses have their own delays and signs set by receptor type (here sign comes from the
  transmitter only), and the connectome's weights are normalised per cell, which flattens the
  spatial offsets the detector relies on. Matching a real fly's selectivity would need those, so
  this model's motion vision is direction-*biased*, not direction-*selective* like a fly's.

How the one free number was set, honestly: the strength of graded -> spiking transmission
(`calibrate_gs.py`). The rule written first (lobula visual projection neurons at 5-15 spikes/s during
gratings) picked 8, but then the giant fiber fired 5.5/s at rest, which a real one never does; no
setting met both. Since those lobula neurons are feature detectors that respond little to gratings,
the giant fiber's silence at rest was kept and the strength set to 4 (giant fiber 0.67/s at rest).
The looming test was never used to choose it.

Found on the way: the mapping of eye columns to the screen was mirrored in each eye (front and back of
the eye swapped) in all earlier runs; it is fixed, and all videos were watched again.

**Assisted vision** (`--vision assisted`, the checkbox) is fly.ai's all-spiking brain plus this
program's detectors: optical flow into each T4/T5 cell along its preferred direction, and a four-way
expansion detector into LPLC2. Its detections are ours; outputs are tagged `_assisted`. (`--vision
lamina`, the all-spiking brain with darkening into L2/L3, is kept for comparison.)

## What else a real fruit fly has

Beyond sight and hearing, the fly gets what a real fruit fly senses, through its own receptor
neurons, and the viewer's **What else a real fly has** section shows the systems that use them:

| feature | neurons | how it gets in |
|---|---|---|
| colour vision | R1-6 Rh1 (3,377), R8 as Rh6 (1,329), R7 UV (1,300) | the video's physical light weighed by each rhodopsin, through the eye's optics (screens emit no ultraviolet: only blue reaches R7) |
| smell | fruit (ORN_DM1/DM2), vinegar (VL2a), mould / geosmin (DA2), CO2 (V), male pheromone cVA (DA1, VA1d) | on a schedule you set |
| taste | sugar (LB3, LB2d), bitter (LB1) | schedule |
| wind, heat, cold, dry / moist air | Johnston's organ wind cells, TRN_VP2 / VP3, HRN_VP4 / VP5 | schedule |
| touch | body bristles (SNta, per side), head bristles | schedule |
| smell -> memory | projection neurons, mushroom body (4,064 Kenyon cells), lateral horn, memory output neurons | readout |
| compass | the 50 EPG neurons of the ellipsoid body ring (heading), PEN, PFL | readout |
| clock and hunger | clock neurons (LNv, LNd, DN1), insulin-producing cells | readout |

Schedule stimuli on the Watch page (*Also let it smell, taste, feel*) or on the command line:
`watch.py my_video.mp4 --stim "vinegar:5-15,heat:20-30"` (seconds into the clip; 0.6 V per step into
the receptors while on). Each stimulus drives exactly its receptors during its window (checked:
receptors ~0.3 -> 25 spikes/s and back), and `audit.py` replays the schedule too.

Honest finding: the mushroom body is **saturated** in this model: its Kenyon cells fire on every
step (100% active, at rest and with smells), where a real fly's are sparse (about 5-10% per odour),
and its memory output neurons are near the ceiling too. So the learning centre cannot show smell
responses here; the viewer says so. Smell projection neurons do respond a little (about +1
spike/s), and the insulin cells rose with sugar in one run (1.5 -> 2.6 spikes/s).

## The music test

Songs are played into the fly's ears and its responses recorded. One question: can you tell from **how
the brain responds** whether a song was made by people or generated by AI, and does the brain add
anything beyond what its ears received? The connectome is frozen; only a plain logistic readout is
fitted, on training songs only.

### How a song reaches the brain

```
audio file ──ffmpeg──> mono, 8 kHz ──> clips (fixed positions) ──> same loudness
   └─> 8 frequency bands, 80 Hz - 1.2 kHz (the fly hears low; most music detail is above)
         └─> loudness per band every 20 ms ──> voltage into Johnston's organ neurons
               (JO-B: low bands, JO-A: high bands; 138 sound-sensing neurons)
                 └─> 0.5 s of silence to settle, then the clip, through the whole connectome
                       └─> spike rates of fixed groups: the ear, AMMC, WED/WEDPN (auditory relay),
                           and every superclass (descending neurons, motor, ...), every 20 ms
```

### Questions, decided before running

1. **Reference:** plain full-band audio features separate human from AI (if not, the set is too hard).
2. **The fly hears it:** the brain's responses separate the classes above chance (permutation p < 0.05).
3. **The brain adds something:** the brain's responses beat *its own ear input*, summarised the same
   way (paired bootstrap 95% CI of the AUC difference above 0). If not, the brain only passes on what
   its ears got: an honest negative.
4. **The wiring matters:** the real connectome beats a degree-preserving rewired copy.

The ablation "flat ear" (every ear neuron hears all bands alike) tests the tonotopic assumption.

### Keeping it unbiased

| risk | what is done |
|---|---|
| file format / bitrate gives the class away | every file decoded the same way; `--same-codec` first pushes all through one MP3 codec |
| AI masters are louder | every clip set to the same loudness |
| choosing "good" parts | clips at fixed positions (1/4, 1/2, 3/4 of the song), never by content |
| clips of one song in training and test | splits by **song**; a song's score is the mean over its clips; statistics count songs |
| unequal classes | the larger class is randomly cut to the size of the smaller (seeded, reported) |
| tuning on the test set | fixed readout (L2 logistic, fixed strength), standardised on training songs only |
| chance looking like a result | label-permutation null (whole cross-validation rerun), bootstrap CIs over songs |
| the method itself leaking | `--synthetic`: a null set (both classes from one generator) must score ~0.5 everywhere |
| many groups compared | per-group differences Holm-corrected |

What it can't remove: if your human and AI folders differ in genre, era or production style,
that difference is what gets detected. Match them as well as you can (same genres, similar
tempos, the AI songs prompted to resemble the human ones).

### Run the music test

Needs the fly.ai code next to this folder (`../firefly`, or set `FLYBRAIN_SRC`) and its Python
environment (NumPy, SciPy; CuPy for the GPU), plus `ffmpeg` on the PATH.

```powershell
# 1. check the method (no songs needed, ~5 min on a GPU)
..\firefly\.venv\Scripts\python.exe run.py --synthetic

# 2. your songs: songs\human\*.mp3 and songs\ai\*.mp3 (at least 4 each; 20+ each for real conclusions)
..\firefly\.venv\Scripts\python.exe run.py songs --same-codec

# a quick first look (real brain only, fewer permutations)
..\firefly\.venv\Scripts\python.exe run.py songs --quick
```

Results: `out/<folder>/report.md` (tables, the four answers, how each neuron group responds) and
`results.json`. Brain responses are cached, so re-running the statistics is fast.

Tests (no brain, GPU or audio needed): `python -m pytest tests -q`.

### Method check (synthetic songs, 12 per class, real GPU run)

| feature set | **null** set: AUC (p) | **signal** set: AUC (p) |
|---|---|---|
| audio (full band) | 0.58 (0.37) | **0.94 (0.005)** |
| ear input | 0.46 (0.57) | 0.79 (0.050) |
| brain (real) | 0.57 (0.32) | 0.44 (0.68) |
| brain (rewired) | 0.52 (0.45) | 0.53 (0.44) |
| brain (flat ear) | 0.58 (0.33) | 0.74 (0.085) |

* **No leak:** on the null set (both classes from one generator) nothing beats chance.
* **The pipeline finds a real difference:** on the signal set (machine-tight vs played timing),
  plain audio finds it clearly and the ear input mostly does.
* **The brain, as wired here, does not pass it on.** The ear neurons respond strongly to sound
  (0.28 -> 12.3 spikes/s), but the auditory relay barely changes (WED 2.66 -> 2.83, WEDPN
  5.55 -> 5.69 spikes/s; AMMC slightly down), so the timing difference drowns in the brain's own
  background activity. This is a result about this model, not a bug, and the ear strength was
  NOT turned up to make the brain "work": that would be tuning on the answer. Any change to the
  ear gain should be decided without labels (for example: set it so the relay's response to
  sound, averaged over all songs, reaches a target), written down, then rerun.

## What is real, what is modelled (and how it is checked)

**Real (simulated, not drawn or edited):** every spike, rate, raster mark and bar comes from
stepping the MaleCNS connectome, 20 ms per step, in the video's own time. `audit.py` checks each
watched video:

```powershell
..\firefly\.venv\Scripts\python.exe audit.py out\watch\<name>
```

* **Timing:** brain steps x 20 ms = the clip's length (1 s of video = 1 s of brain time; picture,
  sound and spikes on one clock). Videos with 50+ frames a second give the brain a new frame every
  step; slower ones hold each frame for two steps.
* **Real data:** the first 2 s are simulated again from the same inputs and seed and must match the
  saved timeline step for step. Found on the way: the GPU library's sparse product sums in a varying
  order, and the spiking network turned that rounding into different spikes, so 2 of 6 audits failed
  on one try and passed on the next. The graded brain now uses its own product that sums every
  neuron's inputs in a fixed order, so a run repeats bit for bit on the same GPU (the 0.5% limit is
  only for another GPU or the CPU).
* **No phantom responses:** a plain grey view in silence must leave the senses at rest.

All watched videos pass all three.

**Modelled by this program, then injected** (transduction: the physical world into receptor
neurons): the light at each photoreceptor's eye column (optics, physical light, rhodopsin colour weights), 8 frequency bands into
the ears (JO-B low, JO-A high: an assumption), smells and tastes into their receptors. In assisted
vision only, also motion into T4/T5 and looming into LPLC2. The neuron model (graded optic lobe,
spiking brain) is described above. From the receptors on, everything is the connectome.

**Drawn, not simulated:** the fly's body is a picture posed by its motor neurons: no muscles, no
physics, no wingbeat (real wings beat ~200 times a second, far faster than the 20 ms steps).
Scales are fixed for every video (a full bar = +10 spikes/neuron/s above rest; emotion bars +-5;
a state is named only at +1 or more) so a small change never looks big. Emotion words name
populations, not feelings; populations under 10 neurons are marked noisy.

**Limits compared with a real fly:**
* 20 ms per step: each synapse takes one step in the model, much slower than a real synapse
  (about 1-5 ms), so responses several synapses deep come later than in a fly, and nothing
  faster than 25 Hz can be represented.
* Video is at most 50 frames a second; a fly's eye resolves flicker far faster (~200+ Hz).
* Hearing's first stage is our model (8 frequency bands into Johnston's organ); seeing starts at the
  photoreceptors (fly-own vision), with the optic lobe's direction selectivity much weaker than a
  real fly's (see Vision).
* The brain runs just faster than real time (fly-own vision: about 0.9 s per video second on this GPU;
  assisted: about 0.4 s) and the result is then played back in sync; it is not a live stream.

## Files

| file | what it does |
|---|---|
| `flyears/audio.py` | decode, fixed clips, loudness |
| `flyears/ear.py` | frequency bands -> Johnston's organ neurons |
| `flyears/listen.py` | play clips to the connectome (real or rewired), record group rates |
| `flyears/features.py` | the same summary for ear input and brain output; full-band audio baseline |
| `flyears/evaluate.py` | song-level cross-validation, permutation null, bootstrap |
| `flyears/synth.py` | synthetic null and signal sets |
| `flyears/vision.py` | the vision modes; for assisted vision, optical flow into T4/T5 and looming into LPLC2 |
| `watch.py` | a link or video file -> the fly watches it -> video, timeline, summary |
| `flyears/body.py` | body parts, behaviour commands and internal states, from real neurons |
| `flyears/hybrid.py` | the brain closer to a real fly: graded optic lobe with type speeds, spiking brain |
| `flyears/lamina.py` | eye-column positions of photoreceptors and optic-lobe cells; the lamina workaround |
| `physio.py`, `looming.py`, `calibrate_gs.py` | the real-fly checks (gratings, edges, approaching disc) and the one calibration |
| `ds_timing.py`, `tune_shunt.py` | the direction-selectivity experiments (timing, shunting inhibition) |
| `make_test_video.py` | the test video with known events (bars, approaching disc, beeps) |
| `flyears/senses.py` | colour vision, smell, taste, wind, temperature, humidity, touch; memory, compass, clock readouts |
| `ui.py`, `ui.html`, `view.html`, `retro.css` | the browser pages (port 8790) |
| `body3d.py` | the 3D body: NeuroMechFly posed by the motor neurons (run with the `.venv-body` environment) |
| `audit.py` | checks a watched video: timing, re-simulation, no phantom responses |
| `run.py` | everything, and the report |

Your audio files stay on your PC: `songs/` is ignored by git, and nothing is uploaded.

## Credits and licences

A non-commercial research project. The walkthrough and demo use a synthetic test video made for this
project; no third-party footage is included.

This code: MIT (`LICENSE`). The brain simulator `flybrain` is from [fly.ai](https://github.com/alextitonis/fly.ai)
(MIT). The connectome is MaleCNS v1.0 (Janelia FlyEM and collaborators), used under
[CC BY 4.0](https://male-cns.janelia.org/download/); it is not in this repository, `flybrain download`
fetches it. The 3D fly is NeuroMechFly / flygym (EPFL Neuroengineering Laboratory, Apache-2.0;
Lobato-Rios et al. 2022 Nat Methods, Wang-Chen et al. 2024 Nat Methods). Physiology the checks compare
with: Maisak et al. 2013 (T4/T5 directions), Behnia et al. 2014 and Arenz et al. 2017 (medulla timing),
Salcedo et al. 1999 (rhodopsin spectra).