# ๐๏ธ Gaze Control
**Hands-free scrolling for Android โ powered by real-time on-device eye tracking**
[](https://flutter.dev)
[](https://kotlinlang.org)
[](LICENSE)
[](CONTRIBUTING.md)
[](https://android.com)
[](https://developers.google.com/mediapipe)
[**Read the Full Blog Post โ**](https://www.deepskylabs.in/portfolio/were-open-sourcing-gaze-control-hands-free-accessibility-for-everyone) ยท [Report a Bug](https://github.com/Deep-SkyLabs/gaze-control/issues) ยท [Request a Feature](https://github.com/Deep-SkyLabs/gaze-control/discussions)
---
## Table of Contents
- [Overview](#overview)
- [Features](#features)
- [Architecture](#architecture)
- [Getting Started](#getting-started)
- [Prerequisites](#prerequisites)
- [Installation](#installation)
- [Running in Development](#running-in-development)
- [Building a Release APK](#building-a-release-apk)
- [Project Structure](#project-structure)
- [Configuration Reference](#configuration-reference)
- [Contributing](#contributing)
- [Roadmap](#roadmap)
- [Privacy](#privacy)
- [License](#license)
---
## Overview
Gaze Control is an Android accessibility utility that lets users scroll social media feeds โ Instagram, TikTok, YouTube Shorts, X/Twitter, Reddit โ using only their eyes, head movements, or palm gestures.
It runs as an Android Foreground Service, processes all camera frames locally on-device via a [Google MediaPipe](https://developers.google.com/mediapipe) + L2CS-Net ONNX pipeline, and dispatches physical scroll gestures through the Android Accessibility Service API. No data ever leaves the device.
Built for people who want hands-free mobile interaction: those recovering from surgery, living with motor disabilities, multitasking, or simply curious about what gaze-driven interfaces feel like.
---
## Features
| Capability | Description |
|:---|:---|
| ๐๏ธ **Eye Gaze Tracking** | Look up/down past a configurable dwell threshold to trigger scrolls |
| ๐ค **Head Nod State Machine** | Subtle vertical/horizontal nods fire instant scroll actions via gyroscope fusion |
| ๐๏ธ **Palm Gesture Control** | Open Palm = Play/Pause ยท Closed Fist = Stop ยท Thumbs Up = Select |
| ๐ **9-Stage Calibration** | Personalized neutral anchors, gaze extents, deadzones, and directional biases |
| ๐ **Full On-Device Privacy** | Camera frames processed in native memory, never stored or transmitted |
| ๐ **Thermal-Aware Performance** | Dynamic frame throttling based on device temperature and tracking confidence |
| โก **NNAPI Acceleration** | L2CS-Net inference runs with hardware NPU acceleration + math fallback |
| ๐ก๏ธ **Blink-Safe Tracking** | Eye Aspect Ratio + blendshape fusion prevents involuntary blinks from triggering actions |
---
## Architecture
The system is split into two primary layers communicating via Flutter platform method channels.
```
โโโโโโโโโโโโโโโโโโโโโโโโโโโโ Flutter (Dart) โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ โ
โ GoRouter โโโบ Dashboard โโโบ Calibration โโโบ Settings โโโบ Onboarding โ
โ โ โ โ โ
โ Riverpod NotifierProviders โโโ Platform Channel Adapters โ
โ โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
โ MethodChannel / EventChannel
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโ Kotlin (Android) โโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ โ
โ GazeForegroundService โ
โ โโโ Camera2 API โโโบ MediaPipe Face/Hand Landmarker โ
โ โ โโโโบ L2CS-Net ONNX (gaze pitch/yaw) โ
โ โโโ SensorEventListener โโโบ GravityRollDecoupler โ
โ โ (head rotation โ gaze intent) โ
โ โโโ CropStabilizer (EMA + affine alignment) โ
โ โโโ ConfidenceEngine (visibility + eyelid + gyro) โ
โ โโโ BlinkDetector (EAR + blendshapes) โ
โ โโโ DirectionHysteresis (entry/exit thresholds) โ
โ โโโ GazeStateMachine โ
โ โ NO_FACE โ SEARCHING โ TRACKING โ STABLE_LOCK โ ACTION_READY โ
โ โโโ InteractionProfileManager (per-app scroll tuning) โ
โ โโโ ThermalPerformanceManager (dynamic frame throttling) โ
โ โโโ AccessibilitySafetyEngine โ
โ โโโโบ GazeAccessibilityService โโโบ System Gestures โ
โ โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
```
### Key Subsystems
**`GazeForegroundService.kt`** โ The system heartbeat. Sticky foreground service that coordinates the full perception pipeline from frame capture to gesture dispatch.
**L2CS-Net ONNX** โ Predicts high-resolution pitch/yaw gaze angles from MediaPipe eye ROI crops. Runs via `OnnxReflectiveRunner` with NNAPI hardware acceleration.
**`GravityRollDecoupler`** โ Fuses accelerometer and gyroscope vectors to separate head rotation from genuine gaze intent. Essential for real-world usability when the device is not held perfectly upright.
**`GazeStateMachine`** โ Prevents false positive gesture dispatch by enforcing the full state cycle: `NO_FACE โ SEARCHING โ TRACKING โ UNSTABLE โ STABLE_LOCK โ ACTION_READY โ COOLDOWN`. Actions only fire from `ACTION_READY`.
**`ConfidenceEngine`** โ Combines landmark visibility scores, eyelid closure blendshapes, and device motion to generate a real-time 0.0โ1.0 tracking quality score. Suppresses the filter chain during low-confidence periods.
---
## Getting Started
### Prerequisites
| Requirement | Version |
|:---|:---|
| Flutter SDK | `^3.11.5` |
| Dart SDK | `^3.11.5` |
| Android Studio | Latest stable |
| Android SDK Platform Tools | `33+` |
| Physical Android Device | API 29+ with front camera |
> **โ ๏ธ Emulator Note**: Camera2 is unavailable on emulators. The app will launch but fall back to a degraded heuristic engine. Physical device testing is strongly recommended.
### Installation
```bash
# 1. Clone the repository
git clone (https://github.com/Deep-SkyLabs/gaze-control.git
cd gaze-control
# 2. Install Flutter dependencies
flutter pub get
# 3. Run code generators (Riverpod, GoRouter)
flutter pub run build_runner build --delete-conflicting-outputs
```
### Running in Development
```bash
# Connect your Android device and verify it's visible
flutter devices
# Run in debug mode (note: CV pipeline is noticeably slower in debug)
flutter run
# Run in profile mode for realistic performance with debugging
flutter run --profile
```
### Building a Release APK
```bash
# Build optimized release APK
flutter build apk --release
# The output APK will be at:
# build/app/outputs/flutter-apk/app-release.apk
```
For App Bundle (recommended for Play Store distribution):
```bash
flutter build appbundle --release
```
---
## Project Structure
```
gaze-control/
โโโ android/
โ โโโ app/src/main/kotlin/com/example/gaze/
โ โโโ MainActivity.kt # Flutter โ Native MethodChannel bindings
โ โโโ GazeAccessibilityService.kt # Accessibility Service gesture dispatcher
โ โโโ GazeForegroundService.kt # Full CV engine: camera, sensors, ML, state machine
โโโ lib/
โ โโโ main.dart # App entry point
โ โโโ src/
โ โโโ app/
โ โ โโโ app.dart # Root widget, theme injection
โ โโโ core/
โ โ โโโ platform/ # MethodChannel / EventChannel adapters
โ โ โโโ services/ # Foreground service lifecycle controllers
โ โ โโโ theme/ # Forui design token customization
โ โโโ routing/
โ โ โโโ app_router_config.dart # GoRouter declarations
โ โโโ features/
โ โโโ dashboard/ # Real-time telemetry & state visualization
โ โโโ gaze_tracking/ # Calibration wizard & diagnostic overlays
โ โโโ onboarding/ # First-launch welcome flow
โ โโโ permissions/ # Accessibility + overlay permission checks
โ โโโ settings/ # Threshold configuration & app whitelisting
โโโ docs/
โ โโโ assets/ # Banner, demo GIF, screenshots
โ โโโ ARCHITECTURE.md # Deep-dive architecture documentation
โ โโโ CALIBRATION.md # Calibration system explained
โโโ test/
โ โโโ unit/ # Dart unit tests
โ โโโ widget/ # Flutter widget tests
โโโ CHANGELOG.md
โโโ CONTRIBUTING.md
โโโ LICENSE
โโโ pubspec.yaml
```
---
## Configuration Reference
All user configuration persists in Android SharedPreferences:
| Key | Type | Default | Description |
|:---|:---|:---|:---|
| `flutter.user_calibration_profile` | `String (JSON)` | `null` | Neutral gaze vectors, directional bounds, deadzones, and biases from the 9-stage calibration flow |
| `is_first_launch` | `bool` | `true` | Controls whether the onboarding carousel renders on next launch |
| `sensitivity` | `double` | `0.5` | Global gesture sensitivity multiplier `[0.1 โ 1.0]` |
Calibration profiles use the following JSON schema:
```json
{
"neutralPitch": 0.0,
"neutralYaw": 0.0,
"topThreshold": 0.18,
"bottomThreshold": -0.18,
"leftThreshold": -0.22,
"rightThreshold": 0.22,
"deadzonePitch": 0.06,
"deadzoneYaw": 0.06,
"verticalBias": 1.0,
"horizontalBias": 1.0
}
```
---
## Contributing
Contributions are welcome. Please read [`CONTRIBUTING.md`](CONTRIBUTING.md) before opening a pull request.
### Quick Contribution Guidelines
- **Design tokens only**: All colors, spacing, and typography must use `context.theme` properties. No raw hex values.
- **Explicit state**: Use `NotifierProvider` implementations. Avoid implicit provider reads in business logic.
- **Lint compliance**: Run `flutter analyze` before submitting. The project uses strict linting rules in `analysis_options.yaml`.
- **Tests**: Widget tests for new feature screens; unit tests for any new business logic in the `core/` layer.
### Good First Issues
Look for issues tagged [`good first issue`](https://github.com/Deep-SkyLabs/gaze-control/labels/good%20first%20issue) โ these are scoped, well-documented, and don't require deep knowledge of the native CV pipeline.
---
## Roadmap
- [ ] **Smart App Detection** โ Auto-wake camera service when a supported app enters foreground
- [ ] **Context-Aware Smoothing** โ Dynamic One Euro Filter coefficients based on active scroll velocity
- [ ] **Ambient Dynamic Theming** โ Mirror the active app's color palette in the Gaze Control overlay
- [ ] **iOS Port** โ ARKit-based gaze engine for iPhone
- [ ] **Passive Calibration Refinement** โ Incremental profile tuning over time without explicit recalibration sessions
- [ ] **Additional Gesture Vocabulary** โ Tongue detection, expression triggers, breath sensing
Track progress and vote on priorities in [Discussions โ Roadmap](https://github.com/Deep-SkyLabs/gaze-control/discussions).
---
## Privacy
Gaze Control is designed privacy-first by architecture:
- Camera frames are processed in native memory and immediately discarded โ they are never written to disk
- No gaze coordinates, usage events, or identifiers are transmitted off-device
- The only persistent data is your calibration profile and settings, stored locally in Android SharedPreferences
- The app requests no internet permission
---
## License
Apache License 2.0 โ see [`LICENSE`](LICENSE) for full terms.
---
Built with care by **[Deep Skylabs](https://www.deepskylabs.in)**
[Website](https://www.deepskylabs.in) ยท [Blog](https://www.deepskylabs.in/blog)
*If Gaze Control has been useful to you, consider starring the repo โ it helps others find it.*
> โ ๏ธ **Early Development** โ This project is actively in development. Some features may be incomplete, unstable, or subject to breaking changes. Use in production at your own discretion.