# Posture Pet An AI-powered posture monitoring system that physically mirrors your body alignment through a robotic companion. Built for [HackAI @ Purdue 2026](https://devpost.com/software/posture-pet-brh6vx). **[→ View Full Project on Devpost](https://devpost.com/software/posture-pet-brh6vx)** --- ## What It Does Most posture trackers fail in practice — camera-only systems lose tracking when you move, and simple IMU wristbands can't tell the difference between a relaxed lean and a harmful slouch. Posture Pet solves this by fusing two independent sensing modalities and routing decisions through a physically expressive robotic companion. When you sit up straight, the robot sits up straight. When you slouch, the robot droops — and its OLED eyes shift from happy to distressed. The feedback is immediate and impossible to ignore. --- ## Architecture ``` ┌─────────────────────────────────┐ │ ESP32-S3 (Wearable) │ │ │ │ 3x MPU-6500 IMUs │ │ ├─ Neck │ │ ├─ Left Shoulder │ │ └─ Right Shoulder │ │ │ │ Mahony Fusion Filter (42 Hz) │ │ → 3x Quaternions (48 bytes) │ │ → BLE Notify @ 20 Hz │ └──────────────┬──────────────────┘ │ BLE (NimBLE) ┌──────────────▼──────────────────┐ │ Arduino Uno Q (Hub Brain) │ │ │ │ ┌────────────────────────┐ │ │ │ Qualcomm QRB2210 (Linux)│ │ │ │ │ │ │ │ • BLE → Quaternion RX │ │ │ │ • 60-feature extractor│ │ │ │ (FFT + rolling stats)│ │ │ │ • TFLite Float16 MLP │ │ │ │ • UDP server for CV RX │ │ │ │ • Cross-validation & │ │ │ │ final decision logic │ │ │ └──────────┬─────────────┘ │ │ │ RouterBridge RPC │ │ ┌──────────▼─────────────┐ │ │ │ STM32U585 (MCU) │ │ │ │ │ │ │ │ • Servo control │ │ │ │ • OLED eye expressions│ │ │ └────────────────────────┘ │ └──────────────▲──────────────────┘ │ UDP over WiFi ┌──────────────┴──────────────────┐ │ PC + Webcam (Vision) │ │ │ │ MediaPipe Pose Landmarks │ │ + Selfie Segmentation Mask │ │ + One-Euro Filter smoothing │ │ → CV Posture Classification │ └─────────────────────────────────┘ ``` --- ## Hardware Components | Part | Role | |---|---| | **ESP32-S3** | Wearable compute — 3x MPU-6500 over shared I2C, NimBLE stack, Mahony filter | | **3x MPU-6500** | 6-DOF IMUs at neck and both shoulders. DLPF set to 42 Hz to reject vibration | | **Arduino Uno Q** | Dual-architecture hub — Qualcomm QRB2210 for AI inference + STM32U585 for real-time I/O | | **PC + Webcam** | MediaPipe skeletal tracking; sends UDP classification packets to the Uno Q | | **Servo Motors (x2)** | Drive the pet's physical posture along two axes | | **SSD1306 OLED** | Displays the pet's eye expressions (happy → neutral → distressed) | ### Wired Communication Protocol Map | Link | Protocol | Payload | |---|---|---| | ESP32 → Uno Q | BLE NimBLE Notify | 48 bytes — 3x Quaternion (float32 ×4) | | PC → Uno Q | UDP over WiFi | CV class label + confidence float | | Qualcomm → STM32 | RouterBridge (Serial RPC) | `set_posture(angle_x, angle_y)` commands | --- ## AI Pipeline ### Signal Processing — 60-Feature Extractor Before inference, a rolling window of 200 IMU samples is processed into 60 features per frame: 1. **Statistical features** — mean, standard deviation, min, max per sensor axis 2. **Frequency domain (FFT)** — spectral power per axis to detect micro-tremors and muscle fatigue invisible to threshold systems 3. **Coordinate normalization** — all joint angles are expressed relative to the neck sensor, making classification body-position independent ### Neural Network — Float16 MLP (TFLite) A 4-layer dense MLP trained on 60-feature vectors: ``` Input (60) → Dense(128, ReLU) → Dropout(0.3) → Dense(64, ReLU) → Dropout(0.3) → Dense(32, ReLU) → Output (N classes, Softmax) ``` - **Regularization**: L2 + Dropout to generalize across body types - **Training**: AMD Ryzen + ROCm acceleration with synthetic jitter augmentation (2× dataset) - **Deployment**: Converted to Float16 TFLite for Qualcomm QRB2210 ARM core - **Accuracy**: 98.18% cross-validation accuracy ### Cross-Validation Fusion The Qualcomm core receives two independent posture signals per cycle — one from the wearable (TFLite inference) and one from the PC (MediaPipe). It cross-validates both before issuing a physical feedback command to the STM32, reducing false positives from either source alone. --- ## Web Dashboard A FastAPI + BabylonJS real-time dashboard streams telemetry to the browser over WebSockets: - **3D Digital Twin**: A 3D spine/shoulder model that rotates in real time using the incoming quaternion data. Nodes for neck, mid-spine, and lower spine are individually animated. - **Color Feedback**: Model turns green when aligned, red when a posture violation is detected. - **One-Euro Filter**: Applied on all sensor and landmark streams to smooth jitter while remaining responsive during deliberate movement. - **Selfie Segmentation**: MediaPipe segmentation mask isolates the user from the background, preventing tracking noise from room objects. --- ## Repository Structure ``` . ├── mpu/ # ESP32-S3 firmware (ESP-IDF + C++) │ ├── main/ │ │ ├── main.cpp # Application entry point │ │ ├── imu_manager.cpp # MPU-6500 I2C polling, Mahony filter │ │ └── ble_server.cpp # NimBLE GATT server, quaternion notify │ └── components/ │ └── mpu_fusion/ # Custom quaternion math and IMU calibration │ ├── uno q/ # Arduino Uno Q hub software │ ├── app.py # Qualcomm Linux app — BLE RX, UDP server, TFLite inference │ ├── inference.py # 60-feature extractor + TFLite runtime wrapper │ └── sketch/sketch.ino # STM32 sketch — servo control, OLED expressions │ ├── cam/ # PC-side computer vision pipeline │ ├── app.py # FastAPI backend + MediaPipe pose + WebSocket stream │ └── frontend/index.html # BabylonJS 3D digital twin dashboard │ ├── training/ # ML training scripts and model artifacts │ ├── train.py # Main training loop (Keras + ROCm) │ ├── advanced_tune.py # Hyperparameter search │ └── models/ # Exported .h5 and .tflite files │ └── docs/ └── devpost/ # Full project documentation ├── OVERVIEW.md ├── HARDWARE.md ├── AI_PIPELINE.md └── SOFTWARE_FRONTEND.md ``` --- ## Building & Running ### Wearable Firmware (ESP32-S3) ```bash cd mpu idf.py set-target esp32s3 idf.py build idf.py flash monitor ``` ### Uno Q Hub (Qualcomm Linux) ```bash cd "uno q" pip install -r ../training/requirements.txt python app.py ``` Flash the STM32 sketch (`sketch/sketch.ino`) separately via the Arduino IDE with the Uno Q board package. ### PC Vision Pipeline ```bash cd cam pip install -r requirements.txt python app.py ``` Open `http://localhost:8000` to view the 3D digital twin dashboard. ### Training ```bash cd training bash setup.sh # Install dependencies (ROCm or CUDA) python train.py # Train base model python advanced_tune.py # Hyperparameter tuning ``` --- ## Acknowledgements Built by Aman Katyal and team at HackAI @ Purdue 2026.