[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)

# OmniBot VR Controller

This repository is the **Meta Quest headset client only**. It does not contain the OhhO website, the Android app, or the robot's ROS 2 workspace.

Unity application for controlling the OmniBot mecanum-wheel robot and SO-101 arm over ROSBridge (WebSocket, v2 JSON). Primary target: **Meta Quest 3 / 3S**. The camera feed prefers WebRTC (`com.unity.webrtc` 3.0.0, ohho_webrtc signaling) and falls back to MJPEG. Measuring Quest glass-to-glass latency is still roadmap.

The app is a **single unified world-space screen** ("OhhO Screen") that floats
in front of you in passthrough and always faces you. It is the headset-native
front end of **ohho-robotics.com**: the same Supabase account, the same garage
of robots, the same branding — sign in once and everything is already there.

Flow on this commit: **Login → Console (VR products) → Garage (your robots) → Teleop** —
all inside the one screen. `WorldSpaceUiPlacer` defaults to 1.2 m ahead
(`distance`) and billboards toward you; a scene value can override that.
Press **left-controller B** to hide/show it.
Simulate vs Drive real is not in this commit. It is
[OHH-212](https://linear.app/ohho-robotics/issue/OHH-212/ohho-vr-robot-card-simulate-drive-real-target-picker-local-desktop)
on `ohh-212-session-picker` ([PR #9](https://github.com/ohho-robotics/OhhO-VR/pull/9)).

Guides: [user guide](docs/user-guide.md), [local sim](docs/setup-local-sim.md),
[cloud sim](docs/setup-cloud.md) (not ready), [real OmniBot](docs/setup-real-omnibot.md),
[safety](docs/safety.md).

---

## 1. Prerequisites

| Tool | Version |
|---|---|
| Unity Editor | **6000.5.2f1** (exact; changeset `eb73d3b415a1` in `ProjectSettings/ProjectVersion.txt`) |
| Meta XR SDK Core | 203.0.0 (vendored under `Packages/com.meta.xr.sdk.core`) |
| Meta XR Interaction SDK | 203.0.0 (vendored under `Packages/com.meta.xr.sdk.interaction`) |
| Meta XR Interaction SDK OVR | 203.0.0 (Meta scoped registry) |
| OpenXR | 1.17.1 |
| NativeWebSocket | upm branch (git URL in `Packages/manifest.json`) |
| Newtonsoft JSON | 3.2.2 |
| Android Build Support | Installed via Unity Hub (required for a Quest APK; not bundled in this repo) |
| Android SDK / NDK | API level 32+ (included with Unity's Android module) |

The Meta XR packages keep their own Oculus SDK licence (`Packages/com.meta.xr.sdk.core/LICENSE.md` and `Packages/com.meta.xr.sdk.interaction/LICENSE.md`). They are not relicensed by this repo.

Unity packages are declared in `Packages/manifest.json` and resolve on first open.

---

## 2. Platform Targets

### Meta Quest 3 / 3S (primary)

- OpenXR backend via `com.unity.xr.openxr` 1.17.1
- Meta OpenXR feature set (hand tracking, passthrough)
- Android API target: 32+, ARM64

### Apple Vision Pro

Not in this repository. There is no visionOS target and no Swift project here. The Quest build is the headset client.

---

## 3. Build Steps for Quest

1. Open this repository root as a Unity project (Unity Hub → Add project from disk). The only enabled build scene is `Assets/scene1.unity`.
2. Wait for package resolution (first open takes ~2–5 minutes).
3. Android player settings are ARM64, IL2CPP, application id `com.ohhorobotics.ohhovr` (API 32–34). OpenXR for Android already enables **Meta Quest Support** and **Hand Tracking**. If you reconfigure in the GUI:
   - Go to **Edit → Project Settings → XR Plug-in Management**.
   - Enable **OpenXR** under the Android tab.
   - Under OpenXR features, enable **Meta Quest support**, **Hand Tracking**, and **Passthrough**.
4. Go to **File → Build Settings**.
   - Switch platform to **Android**.
   - Enable **Development Build** for first deployment (optional).
5. Connect Quest via USB, enable Developer Mode on the headset.
6. Click **Build and Run** (or **Build** to produce an `.apk`). The menu **OmniBot → Build Quest APK** calls `OmniBot.VR.Editor.BuildAutomator.BuildQuestApk` and writes `build/Android/OhhO-VR.apk`.

CI job `build-android` in `.github/workflows/unity-tests.yml` runs that same method through `game-ci/unity-builder` (Android ARM64 IL2CPP, image `unityci/editor:ubuntu-6000.5.2f1-android-3`). The job generates a debug keystore (`ci/debug.keystore`, alias `androiddebugkey`, the public Android debug passwords) and the build method refuses to continue in batch mode without it. That file is gitignored and is not a release key. Release signing is OHH-232. The job prints the APK size in the step summary and uploads artifact `OhhO-VR-<git sha>.apk`. The editor menu still builds when the debug keystore is absent and leaves signing unchanged.

**Human TODO:** sideload `OhhO-VR-<sha>.apk` onto a Quest once. CI does not install the headset build.

> **Rebuilding the UI:** the whole screen is generated by an editor script.
> If you ever need to regenerate it (or after pulling UI-breaking changes):
> **OmniBot → Rebuild Unified OhhO Screen**. It re-fixes the hand prefabs,
> re-wires every controller, rebuilds the card prefabs, and saves the scene.

---

## 4. Edit-mode tests

The NUnit fixtures are in `Assets/Scripts/Editor/Tests/` (59 tests in namespace `OmniBot.VR.Tests`). Of those, 39 are the earlier fixtures: 13 drive-spec, 11 control-scheme factory, 9 calibration, 6 Fabrik IK. Feature fixtures under `Assets/Scripts/Detection/Editor/` and `Assets/Scripts/AR/Editor/` use that same namespace and the same `OmniBot.VR.Tests` filter. CI skips the Unity steps until the Unity licence secrets are set. Run them locally in batch mode.

CI runs these edit-mode tests with GameCI (`.github/workflows/unity-tests.yml`) once three repository secrets are set at Settings → Secrets and variables → Actions (https://github.com/ohho-robotics/OhhO-VR/settings/secrets/actions): `UNITY_LICENSE` (the full contents of a Unity personal licence `.ulf` file, e.g. `C:\ProgramData\Unity\Unity_lic.ulf` on Windows), `UNITY_EMAIL` and `UNITY_PASSWORD` (the Unity ID account that owns that licence). Until they are set the workflow skips the Unity steps and passes. When the secrets are set, `scripts/check-editmode-results.py` reads `test-results/editmode-results.xml` (artifact name `editmode-test-results`) and fails the job if that file is missing or reports zero executed tests (passed + failed + inconclusive = 0). The counts are written to the job summary.

Confirmed from the latest green run before this gate, [Unity tests run 38548684961](https://github.com/ohho-robotics/OhhO-VR/actions/runs/38548684961) (pull request 6, commit `458d2a8`, 14 Oct 2026): GameCI's Test Results check parsed `editmode-results.xml` as **53 passed / 53 total** in 0.211s. That checkout added teleop fixtures on top of the 39 tests above. The artifact zip itself is not anonymously downloadable; the check is produced from that XML. A green run of the fixtures in this checkout must show an executed count greater than zero (expected 59 in `OmniBot.VR.Tests`, including the 39 drive, scheme, calibration, and IK fixtures above, plus the Detection and AR feature fixtures when the runner discovers them).

Windows:

```
"C:\Program Files\Unity\Hub\Editor\6000.5.2f1\Editor\Unity.exe" -batchmode -nographics -projectPath <clone> -runTests -testPlatform editmode -testFilter OmniBot.VR.Tests -testResults <xml> -logFile <log>
```

Linux:

```
unity -batchmode -nographics -projectPath <clone> -runTests -testPlatform editmode -testFilter OmniBot.VR.Tests -testResults <xml> -logFile <log>
```

---

## 5. Setup: Sign In & Connect

1. Put on the headset and launch the app.
2. The **OhhO Screen** appears in front of you with the **Login** page.
3. Enter the same email you use on ohho-robotics.com → **Send Code** → type
   the 6-digit code from your email → **Verify**. The Google button does not
   complete Google sign-in: `LoginPanelController.OnGoogleSignIn` skips to the
   garage with no token. Use the email code.
4. The **Console** shows VR products from `{site}/vr/manifest.json` (today the
   manifest is written for **OhhO Pilot**). Open it.
5. The **Garage** reads `user_robots` for that Supabase account
   (`GarageClient`). Empty means the account has no rows or the read failed.
   Add a robot on ohho-robotics.com and open the garage again. **Open Website**
   launches the site in the Quest browser.
6. **Pick a robot** → the **Teleop** page opens. Enter the IP
   (default `192.168.1.101`) and port (default `9090`) → **Connect**.
   The status dot turns green when the WebSocket connects. That is not a
   video check and not a stop check. A Simulate / Drive real page exists only
   on [PR #9](https://github.com/ohho-robotics/OhhO-VR/pull/9). See
   [docs/user-guide.md](docs/user-guide.md).

Connect expects a robot that is **not in this repository**. The headset opens a ROSBridge WebSocket (default port `9090`). Recording export expects an HTTP bridge on port `8765`. The camera view tries WebRTC first: with PlayerPrefs `ohho.webrtc.token` it opens `wss://<robot>:8443/webrtc/signaling`, and without a token it opens `ws://<robot>:8766/signaling`. MJPEG from `web_video_server` on port `8080` is the fallback. An explicit socket can be set in `ohho.webrtc.url`. Those processes live on the robot PC. This repo does not ship them.

### In-headset sim (no robot PC)

`RobotLinkFactory.Select(RobotLinkTarget.InHeadset)` returns a `SimulatedRobotLink`. It never opens a socket. The base follows the website deadman rule (`lib/connect/teleop.ts`): motion is commanded only while the deadman is held, and releasing it sends zero. E-stop latches zero until it is cleared. Each linear axis is capped at 0.2 m/s and yaw at 1 rad/s (`RobotConfig`). Arm joints slew at that same 1 rad/s and stay inside their position limits. Odometry, joint states, IMU, and `/sim/telemetry` are marked `kind=sim`.

Shared numeric vectors live in `Assets/Scripts/Control/Simulated/SimulatedModel.cs` (`SimulatedVectors`): the teleop deadman sample `{0.2, 0.05, -0.4}`, keyboard scales 0.2 m/s and 0.8 rad/s, battery 0.87, and the 100 ms holonomic step from `lib/connect/simulated.ts`.

OHH-212 session picker hookup, one line:

```
link = RobotLinkFactory.Select(RobotLinkTarget.InHeadset);
```

Assigning an `InHeadsetSimHost` to `TeleopController.linkProvider` is the same choice. That host ticks the sim. Deadman is the left index trigger, or Space in Play Mode, so the grip buttons stay turbo and e-stop. Until the OHH-211 Unity twin (`OmniBot.VR.Twin`) is in the project, the host builds a 4 m placeholder room instead of the twin mesh.

---

## 6. Controls Reference

### Base Movement

| Input | Action |
|---|---|
| Left thumbstick Y | Forward / backward (linear.x) |
| Left thumbstick X | Strafe left / right (linear.y) |
| Right thumbstick X | Rotate left / right (angular.z) |
| Right grip (hold) | Scales stick deflection by 2, then clamps it to ±1 before `MaxLinVel`. Full stick is not faster. OmniBot `MaxLinVel` is 0.2 m/s |
| **Both grips simultaneously** | Publishes `/emergency_stop` true and stops new twists. Does not publish a zero twist. Release publishes false. Not a hardware e-stop — [docs/safety.md](docs/safety.md) |
| Left controller B | Hide / show the OhhO Screen |
| Right thumbstick press | Cycle camera feed |

### Hand Tracking Arm Control

| Gesture | Action |
|---|---|
| Right index trigger (hold) | Arm clutch — the arm moves only while this is held |
| Right hand position (in workspace, clutch held, tracking good) | IK target, as motion relative to the pose where the clutch closed |
| Right hand rotation | Wrist orientation target |
| Right index–thumb pinch | Gripper close (pinch=1 → closed, pinch=0 → open) |
| Left controller X | Arm enable. On a real robot, press again to confirm. Press while armed to turn off |
| Left hand thumbs-up (hold 0.5 s) | Request arm enable, or turn it off. Does not confirm a real robot |
| Right stick Y while clutch held and hand tracking is poor | Jog the selected joint |
| Left stick click | Next jog joint |

---

## 7. Hand Tracking Arm Workspace

The arm workspace maps the right hand's position within a 0.40 m radius sphere
(centered on `handWorkspaceOrigin`) to the SO-101 arm's reachable space. A wire
sphere is drawn on that origin while teleop is running.

Enabling the arm does not move it. Motion starts when the right index trigger
closes, from the pose the arm already holds. A simulated garage robot enables
on one request. Any other status, including a live robot, waits for a second
Left X press before `/arm/enable` is published. Joint commands step at most
0.15 rad, the same cap as the robot's arm safety node.

SO-101 link lengths and kinematic zeros come from `RobotDescriptor`. Servo
home ticks are not calibrated in this client (OHH-94). Until that measurement
lands, joint homes stay at 0 rad. A video of this flow on the physical arm
is still outstanding — do not treat sim tests as that recording.

```
              [Headset view — looking down]

        ┌────────────────────────────────────┐
        │                                    │
        │       Hand workspace sphere        │
        │         radius = 0.40 m            │
        │                                    │
        │    [O] = workspace origin           │
        │       (arm base in VR scene)        │
        │                                    │
        │   Move right hand within sphere    │
        │   → IK drives arm to match         │
        │                                    │
        │   Arm reach: L1+L2+L3 = 0.33 m    │
        │     L1 = 0.117 m (upper arm)       │
        │     L2 = 0.133 m (forearm)         │
        │     L3 = 0.080 m (wrist)           │
        └────────────────────────────────────┘
```

Place the `handWorkspaceOrigin` transform in your Unity scene at the robot
arm's base position (0.35 m above the robot base by default).

CCD IK runs at 20 Hz with max 50 iterations and 1 mm tolerance.
All 6 joints are clamped to their configured limits at every step.

**Hand-tracking settings:** both hand prefabs run
with `_updateRootPose` + `_updateRootScale` + `_applyBoneTranslations` enabled
(hand mesh exactly follows the tracked wrist — no offset), the hand skinned
meshes use `updateWhenOffscreen` (prevents per-eye frustum culling that made
hands vanish in one eye), the tracking origin is **Stage**, and the Android
manifest requests **HIGH** hand-tracking frequency.

---

## 8. Display Quality

`VRGraphicsBoost` (on the bootstrap GameObject) applies at startup:

| Setting | Default | Effect |
|---|---|---|
| `eyeTextureResolutionScale` | **1.5×** | Sharper everything (the "old 3D game" fix) |
| MSAA | **4×** | Clean UI edges |
| Fixed Foveated Rendering | **Off** | No peripheral blur |
| Display refresh | **90 Hz** | Smoother hands + UI |
| Texture mipmap limit | **0** | Full-res textures |

Tweak the values in the inspector on `[OhhO VR App] → VRGraphicsBoost`.

---

## 9. Dataset Recording Workflow

1. Open the **Teleop** page (pick a robot in the garage).
2. Press **Start Recording**. The status shows the episode name + live timer.
3. Perform the demonstration — all arm states, odometry, and commands are
   captured at 30 Hz locally on the headset.
4. Press **Stop & Save** to finalize. The JSONL file is written to
   `<persistentDataPath>/recordings/<episodeName>.jsonl`.
5. Press **Discard** to throw away the current episode without saving.

### Exporting to the Robot

After recording, export episodes to the robot for training:

1. Ensure the robot VR bridge is running (see section 10).
2. Press **Export to Robot** in the Teleop page.
3. Each JSONL file is POSTed to `http://<robotIp>:8765/upload_episode` (multipart field `file`).
4. No handler for that path is in this repo. A search of OmniBot `ohh-205-wsl2-network` did not find one either. Foxglove on port 8765 is a different service. Expect export to fail; the JSONL stays on the headset.

---

## 10. Recording signals the headset sends

`EpisodeManager` publishes `/vr/record_start` (`std_msgs/String`, the episode name) and `/vr/record_stop` (`std_msgs/Bool`, true = save, false = discard). No subscriber for those topics is in this repo.

Export POSTs the JSONL to `http://<robot>:8765/upload_episode`. The headset does not call `/health`. No upload route was found in this repo or in OmniBot `ohh-205-wsl2-network`. Foxglove's bridge on port 8765 is a WebSocket and does not accept that POST. A `/vr/obs` mirror is not implemented here.

---

## 11. Not in this commit

Cloud sim ([OHH-216](https://linear.app/ohho-robotics/issue/OHH-216)), AprilTag tracking ([OHH-213](https://linear.app/ohho-robotics/issue/OHH-213)), and a grip-deadman drive loop (`ohh-59-gameci-quest-teleop`) are not in this tree. WebRTC remains roadmap. visionOS is not started. Details and the files those claims rest on are in [docs/user-guide.md](docs/user-guide.md) and [docs/safety.md](docs/safety.md).

---

## 12. Project Structure

```
./
├── Assets/
│   ├── scene1.unity            # the only enabled build scene
│   ├── Scripts/
│   │   ├── App/OhhoVrApp.cs    # login→console→garage→teleop routing (singleton)
│   │   ├── Core/
│   │   │   ├── Messages/       # ROSBridge JSON message types
│   │   │   ├── Platform/       # manifest/catalog/Supabase/garage clients
│   │   │   ├── ROSBridgeClient.cs  # WebSocket client (singleton)
│   │   │   ├── ConnectionManager.cs # Connection lifecycle (singleton)
│   │   │   ├── VRGraphicsBoost.cs   # 1.5× resolution + MSAA + FFR + 90 Hz
│   │   │   └── RobotConfig.cs      # All constants
│   │   ├── AR/                 # 1:1 twin overlay, robot-anchored HUD, manual placement
│   │   ├── Control/            # RobotProfile, TeleopController, drive/manip schemes
│   │   ├── Detection/          # AprilTag robot pose (see Assets/Scripts/AR/HOOKUP.md)
│   │   ├── Input/
│   │   │   ├── ArmIKSolver.cs      # CCD IK solver (pure C#)
│   │   │   ├── GestureDetector.cs  # OVR hand tracking
│   │   │   └── ...
│   │   ├── MR/
│   │   │   ├── PassthroughManager.cs
│   │   │   ├── WorldSpaceUiPlacer.cs  # parks the screen in front of you
│   │   │   └── ScreenToggle.cs        # B button hides/shows the screen
│   │   ├── UI/
│   │   │   ├── Auth/LoginPanelController.cs
│   │   │   ├── Console/ConsolePanelController.cs
│   │   │   ├── Garage/GaragePanelController.cs
│   │   │   ├── TeleopHudController.cs # connection+telemetry+camera+recording+export
│   │   │   └── ...
│   │   ├── Video/              # WebRTC receive (ohho_webrtc) with MJPEG fallback
│   │   └── Recording/          # EpisodeManager, ProfileDrivenRecorder
│   └── Editor/
│       └── OhhoUnifiedAppBuilder.cs  # OmniBot → Rebuild Unified OhhO Screen
├── Packages/manifest.json
└── ProjectSettings/ProjectVersion.txt
```

---

## 13. Troubleshooting

| Problem | Solution |
|---|---|
| Status indicator stays red | Check the IP and that rosbridge is listening on port 9090 on `0.0.0.0`, not only `127.0.0.1`. The headset uses a WebSocket, not DDS. `ROS_DOMAIN_ID=30` is the sim/compose setting so Gazebo and rosbridge share a graph |
| Hand tracking not working | Ensure Hand Tracking is enabled in Quest developer settings and in XR Plug-in Management; the manifest already requests the `HAND_TRACKING` permission (HIGH frequency) |
| Hand mesh offset from real hand | The hand prefabs update root pose, scale, and bone translations; if it regresses, re-run **OmniBot → Rebuild Unified OhhO Screen** |
| Hand visible in one eye only | The hand meshes use `updateWhenOffscreen` so per-eye frustum culling can't drop them |
| Camera feed shows "No Signal" | Ensure `web_video_server` is running: `ros2 run web_video_server web_video_server` |
| Export fails | Check that an HTTP service answers on `http://<robot>:8765/health` and that the firewall allows port 8765; the bridge is not in this repo. Until something serves `POST /upload_episode` on port 8765, the JSONL stays under `persistentDataPath/recordings/` on the headset. Foxglove on 8765 will not accept the upload |
| Arm not moving | Request enable with Left X or a thumbs-up. On a real robot press Left X again to confirm, then hold the right index trigger. The arm will not jump to the hand on enable. Nothing is published on `/arm/enable` before that confirm |
| UI looks low-res | `VRGraphicsBoost` sets 1.5× eye resolution at startup; check it exists on `[OhhO VR App]` |
| Text renders magenta | The TMP font materials use `TextMeshPro/Distance Field`; if they regress, re-assign `TextMeshPro/Distance Field` to the LiberationSans SDF materials |

---

## Demo clip

The 30–60 second Quest passthrough clip (driving OmniBot and moving the arm) is **pending**. There is no video URL until that file is on YouTube.

Shot list, Meta Quest Developer Hub / `adb` / scrcpy capture, captions, and the upload checklist: [docs/recording/quest-teleop-clip.md](docs/recording/quest-teleop-clip.md).

**Needs Varun: Quest + robot, record and upload.**

---

## 14. AR mode

AR mode draws a 1:1 twin on the detected robot and a HUD that stays on that robot. The HUD shows joint angles, planar speed, yaw rate, and tag confidence. When detection is not locked, `PlaceInFrontOf` or `PlaceManually` places the twin. The overlay only subscribes; it does not publish teleop commands. It is not in `Assets/scene1.unity`. Hookup is `Assets/Scripts/AR/HOOKUP.md`.

---

## Coordinated release (not yet tagged)

Nothing in this section is a published release. Git tags, GitHub Releases, and the sim container tag below have not been created. `ProjectSettings/ProjectSettings.asset` on this commit (`4ebc555a7495ee7a2cd0f62cbe85fbc201ad1d4b`, `origin/main`) still has `bundleVersion: 1.0`.

| Piece | Intended pin | On this commit? |
|---|---|---|
| OhhO-VR app | git tag `v1.0.0` | The Quest client is here. The tag is not yet tagged. |
| `protocolVersion` | `1.0.0` | No. See the matrix. |
| `ohho.telemetry.v1` | schema `1.0.0` | No. |
| `RobotDescriptor` | `ohho.descriptor.v1` (`1.0.x`) | No. |
| OmniBot VR profile | git tag `vr-profile-v1.0.0` | No. That tag is not yet tagged. |
| Sim container | `ghcr.io/ohho-robotics/omnibot-sim:vr-1.0.0` | Placeholder. Not published. |

Full table, branch SHAs, and the human tag commands: [docs/compatibility-matrix.md](docs/compatibility-matrix.md), [CHANGELOG.md](CHANGELOG.md), [docs/release-tags.md](docs/release-tags.md). `scripts/print-release-commands.sh` prints those commands and does not run them.

## Licence

First-party code in this repository is [Apache License 2.0](LICENSE).

The vendored Meta XR SDK is not Apache-2.0. It stays under the Oculus SDK License Agreement (`Packages/com.meta.xr.sdk.core/LICENSE.md`, `Packages/com.meta.xr.sdk.interaction/LICENSE.md`). Other third-party notices are in [NOTICE](NOTICE).