# Gesture Media ### Infrared gestures → Bluetooth media control [简体中文](README.zh-CN.md) · [Build guide](docs/BUILD.md) · [Version history](CHANGELOG.md) · [Validation](validation/README.md) A working embedded media-controller prototype built with an **M5Stack AtomS3R** and **Unit Gesture / PAJ7620U2**. Hand movements become standard BLE HID Consumer Control commands for a connected computer, phone, or tablet. Normal use requires power and a compatible BLE host; a PC relay, cloud service, or project-specific host application is not required. **Published baseline: v24.** The main sketch is byte-for-byte identical to the archived v24 sources that were compiled, flashed, and user-tested. The later fist/slap filtering draft is retained separately as an **unverified v25 experiment**, not an implemented fix. ## Interaction | Input | Action | |---|---| | Open palm, then close fingers into a fist toward the sensor | Play / pause | | Left slap-style movement | Previous track | | Right slap-style movement | Next track | | Each completed clockwise circle | One volume-up key | | Each completed counterclockwise circle | One volume-down key | | Front screen button: one click | Pause / resume gesture control | | Front screen button: double-click | Disconnect the current host and wait for another | | Double-click again while waiting | Cancel host switching | | Hold front screen button for 3 seconds | Clear stored Bluetooth bonds | The front screen is pressed as a physical button. It is distinct from the side Reset button. A single click is decided about 350 ms after release so a double-click does not also pause control. Volume step size and previous-track behavior are determined by the host/player. ## Hardware - AtomS3R: ESP32-S3, 128 × 128 display, front button, USB-C power/programming. - Unit Gesture U127: PAJ7620U2 infrared optical sensor. - Grove cable for I²C and power; sensor address `0x73`. - Existing calibrated mounting: sensor face vertical, toward the user. Start around 8–10 cm away and keep the motion in front of the sensor. Official references: [AtomS3R](https://docs.m5stack.com/en/core/AtomS3R), [Unit Gesture](https://docs.m5stack.com/en/unit/Unit_Gesture). ## Architecture ```mermaid flowchart LR Hand[Hand movement] --> Sensor[PAJ7620U2] Sensor -->|I²C: center, area, presence, flags| Filter[Median filtering] Filter --> Gestures[Sweep / Grip / Circle trackers] Gestures --> Guard[Event arbitration and connection guards] Guard --> HID[BLE HID Consumer Control] HID --> Host[Host OS and media player] Gestures --> UI[Screen and diagnostic serial output] ``` The C++/Arduino firmware polls the sensor about every 4 ms and samples trajectories about every 16 ms. These are scheduling intervals, **not measured end-to-end latency**. - `SweepTracker` estimates horizontal motion and suppresses held-hand withdrawal. - `GripTracker` combines recent area contraction, centroid changes, endpoint settling, and a one-event latch. - `CircleTracker` integrates changes in motion heading, checks circular extent, and counts turns. Native sensor hints assist a clipped final arc. - `MediaHid` exposes a consumer-only, two-byte media input report, with encrypted report access and automatic key release. - Host switching preserves bonds, rejects the previous host during switching, and completes after a different host authenticates and subscribes. The firmware uses an existing BLE stack; it does not implement a new Bluetooth protocol. It transmits media commands, not audio. ## Build Pinned baseline: M5Stack ESP32 core **3.3.9**, board **M5AtomS3R**. ```powershell # Requires Arduino CLI in PATH, or -Cli with its executable path. ./scripts/setup.ps1 ./scripts/build.ps1 # Choose the actual USB serial port; COM4 is only an example. ./scripts/upload.ps1 -Port COM4 ``` The scripts also locate the bundled CLI in the standard Windows Arduino IDE installation. See [BUILD.md](docs/BUILD.md) for versions, IDE instructions, and configuration overrides. Uploading replaces the program on the board. No firmware upload or bond clearing is performed by the build script. ## Validation and known limits Historical hardware checks include Windows media control, Xiaomi phone volume control, iPad media control after HID permission fixes, v23 grasp/withdrawal/slap checks, and v24 switching to another host. They were performed at different development stages; they are not a full platform-by-function certification matrix. The public sensor fixtures are allowlisted numeric excerpts from those experiments. Run supplementary Python replay checks: ```text python -m unittest discover -s tests -v ``` The Python replay is a separate reference implementation, **not execution of the C++ binary**. One expected-failure case documents the unresolved fist-sweep false positive; it must not be read as a passing fist-rejection feature. See [validation details](validation/README.md). **Important limitations** - PAJ7620U2 cannot directly count fingers or classify a fist versus an open palm. Gesture recognition is movement-based approximation, without a camera or a trained ML model. - A clenched-fist lateral movement can still trigger a track command in v24. - Thresholds depend on the present mounting, distance, motion, and user. No general accuracy percentage or universal latency is claimed. - `BT: READY` means authentication and report subscription succeeded; it is not proof that a player acted. - One host is controlled at a time. Select/connect `Gesture Media` on the new host after double-clicking. - Xiaomi Home / Home Assistant integration was an initial goal and is **not implemented**. ## Repository ```text src/MediaGestureController/ v24 firmware scripts/ Arduino setup, build and optional upload tools/ Python trajectory replay reference tests/ Behavioral checks against recorded samples validation/ Numeric fixtures, source manifest and evidence summary history/ Available v13–v23 source snapshots experiments/v25/ Unverified slap-filter patch patches/ Historical BLE library compatibility fix docs/ Build and design explanations ``` Full flash/NVS backups, credentials, development-machine paths, installers and raw unfiltered logs are deliberately omitted. ## Development and attribution This is a student-led interaction-design and hardware-testing project developed with open-source libraries and substantial AI-assisted programming/debugging. The project includes a locally diagnosed BLE library compatibility fix; no upstream merge is claimed. See [third-party notices](THIRD_PARTY_NOTICES.md). No project-wide license has been selected. Dependencies retain their own licenses.