# ๐ค Corosuke Robot / ใณใญๅฉใญใใใ
> **C**ommunication-**O**riented **R**obot for **O**bserving, **S**upporting, and **U**nderstanding **K**ids' **E**veryday โ that's **COROSUKE**.
[](LICENSE)
[](https://developer.d-robotics.cc/)
[](STAGE3.md)
[](https://docs.ros.org/)
[](https://www.espressif.com/)
A ~46 cm open-source animatronic of **Korosuke (ใณใญๅฉ)** from *Kiteretsu Daihyakka* โ he
sees, listens, thinks, and talks in **Japanese or English**, emotes through his eyes, waves
his arms, and shuts himself down safely. Everything runs **100 % on-device on a D-Robotics
RDK X5**, with an ESP32-S3 driving the eyes and arms.
> ๐ **D-Robotics Robotics Dream Keeper Challenge โ Stage 3 (Launch)**
> ๐ [STAGE3.md](STAGE3.md) โ showcase & benchmarks ยท ๐งฉ [PROPOSAL.md](PROPOSAL.md) ยท ๐บ [ROADMAP.md](ROADMAP.md)
## โถ Watch the demo
---
## What it does
Powered on and left alone, Corosuke wakes and greets you, tracks you with BPU pose
detection, holds a conversation (on-device STT โ LLM โ TTS, with a "thinking" eye animation
while he ponders), reacts to gestures, waves his arms, and shuts down safely on a button
press (goodnight voice โ โโ "safe-to-unplug" eyes). A tabbed **web dashboard** at
`http://:8080` streams the live camera + skeleton and tunes every setting.
| Subsystem | Implementation | Measured |
|---|---|---|
| Vision | BPU YOLO11n-pose (Bayes-e `.bin`) | ~19.5 FPS |
| Speech-to-text | sherpa-onnx **SenseVoice** (JP/EN) + Silero VAD | real-time (CPU) |
| LLM dialogue | llama.cpp + TinySwallow-1.5B *(Qwen2.5-1.5B-based)* โ JP & EN personas | 1.5โ3.3 tok/s ยท ~5โ10 s/reply |
| Text-to-speech | **JP** Open JTalk / **EN** espeak-ng | instant |
| Language | **ๆฅๆฌ่ช / English** โ STT ยท LLM ยท TTS ยท Web UI, switchable on the fly | on-device |
| Audio out | **MAX98357A I2S amp** (custom kernel driver) โ ฯ50 speaker | DSP-tuned |
| Eyes | 2ร GC9A01 on ESP32-S3 โ 8 emotions (smile / thinking / โโ) | โฅ30 FPS |
| Arms | 2ร SG90 rope-pull bellows arms | auto-detach |
> ๐ **Bilingual & fully on-device.** One switch flips the whole robot โ speech recognition,
> LLM replies, voice, *and* the dashboard โ between **Japanese and English**, with **no cloud
> and no API keys**. The voice stack follows D-Robotics' own [RDK voice-interaction
> reference](https://developer.d-robotics.cc/magicbox_doc/en/algorithm-development/voice-interaction):
> **SenseVoice** ASR + a **Qwen2.5-1.5B**-family LLM (we run the Japanese-tuned
> **TinySwallow-1.5B** via llama.cpp on CPU).
### Build tools & jigs
To reproduce the body you'll also need:
| Tool | Purpose | Where to get |
|---|---|---|
| **Velcro tape** | attaching parts to the 3D-printed body | [Amazon](https://www.amazon.co.jp/dp/B0GJZJM4TG) |
| **Ultrasonic cutter** (or an alternative) | post-processing the 3D-printed parts โ anything that melts plastic works, even a heated metal needle | AliExpress |
| **3D printer** | printing the parts! (we used a **Bambu Lab A1**; parts are within 18 cm, so an **A1 mini** should *just* fit) | โ |
| **Double-sided tape** | bonding parts | โ |
| **String** | moving the hands (rope-pull arms) | โ |
| **Zip ties** | tuning the rope arms | [Daiso](https://jp.daisonet.com/products/4550480088891) |
| **Soldering iron** | a little soldering (power button, amp terminals, etc.) | โ |
| **Breadboard** (small) | ESP32-S3 wiring | โ |
| **Jumper wires** (M-M / M-F) | connecting the parts | โ |
## Challenge journey โ Stage 1 โ 2 โ 3
| Stage | Theme | What was proven | Doc |
|---|---|---|---|
| **1 โ Ideation** | On-device perception | YOLO11 vision on the **RDK X5 BPU** (+ open-source [RDK X5 case](https://github.com/gurimaruking/rdk-x5-modular-case)) | [STAGE1.md](STAGE1.md) |
| **2 โ Build** | System design | The **RDK X5 as the single cognitive core** under a **ROS 2 graph**; ESP32s become actuator sub-controllers | [PROPOSAL.md](PROPOSAL.md) |
| **3 โ Launch** | Shipped robot | The full interactive Corosuke โ sees / listens / thinks / talks / emotes, **bilingual**, 100 % on-device (this repo) | [STAGE3.md](STAGE3.md) |
Stage 1's BPU vision is still today's perception layer, and Stage 2's ROS 2 design ships as
[`ros2_ws/`](ros2_ws/) alongside the low-latency monolith.
## Quick start (on the RDK X5)
> All on-device. **No cloud, no API keys.** Perception on the BPU, dialogue on the CPU.
```bash
# 1) Clone
git clone https://github.com/gurimaruking/corosuke-robot.git && cd corosuke-robot
# 2) Flash the eye/arm co-MCU (ESP32-S3) โ from the RDK X5 (uses the CH343 UART port)
cd firmware/corosuke_eyes && pio run -t upload && cd ../..
# 3) One-time: MAX98357A I2S amp โ build the out-of-tree codec driver + device-tree overlay
# Follow docs/rdk_x5_40pin_i2s_max98357a.md (kernel headers are already on the board)
# 4a) Run the INTEGRATED demo (monolithic, low-latency) as a systemd service
sudo cp deploy/*.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now korosuke-monitor korosuke-shutdown-btn
# โ open http://:8080 (Monitor / Settings tabs)
# 4b) โ or โ run the MODULAR ROS 2 graph (same on-device stack: TinySwallow + Open JTalk, no cloud)
cd ros2_ws && colcon build && source install/setup.bash
ros2 launch korosuke_nodes korosuke.launch.py # vision โ brain โ eyes / dialogue โ voice
ros2 launch korosuke_nodes korosuke.launch.py with_web:=true # + rosbridge :9090 โ open ros2_ws/src/korosuke_nodes/web/console.html
# NB: eye path is HW-verified; full graph is code-complete but not yet run end-to-end (see STAGE3 ยง5.1)
# Safe shutdown: hold the GPIO button ~1 s โ goodnight voice โ โโ eyes โ OS halt โ cut power.
```
**Models** (STT / LLM / YOLO `.bin`) download on the board โ see
[stt_research.md](docs/stt_research.md) and [dialogue_stack_research.md](docs/dialogue_stack_research.md).
The LLM stays on the CPU **by design**: the X5 BPU cannot accelerate LLMs
([rdk_x5_bpu_llm.md](docs/rdk_x5_bpu_llm.md)), so replies take ~5โ10 s โ the "thinking" eyes
cover the wait.
## Repository layout
```
corosuke-robot/
โโโ STAGE3.md, PROPOSAL.md, ROADMAP.md # showcase / Stage-2 design / roadmap
โโโ scripts/ # korosuke_monitor.py (integrated demo) ยท shutdown_button.py ยท fix_max98357a.sh
โโโ ros2_ws/ # ROS 2 graph: korosuke_msgs + korosuke_nodes (vision/brain/dialogue/voice/serial_bridge)
โโโ firmware/ # ESP32-S3 (PlatformIO): corosuke_eyes (eyes + arms) ยท max98357a (I2S driver)
โโโ deploy/ # systemd units + sudoers (auto-start, safe shutdown, audio self-heal)
โโโ hardware/3d_models/ # OpenSCAD body (color-split) + RDK X5 modular case
โโโ docs/ # architecture, audio(I2S), BPU-LLM, network, power/USB, research, photos, diary
```
## Documentation
- **[STAGE3.md](STAGE3.md)** โ architecture (Mermaid), benchmarks, **known issues & limitations**, requirement coverage
- **Story / presentation** โ intro blog [ๆฅๆฌ่ช](docs/blog/intro_ja.md) ยท [English](docs/blog/intro_en.md) ยท 5-min talk: [slides PDF](docs/slides/talk_5min.pdf) ยท [deck (PNG)](docs/slides/talk_5min_deck.md)
- Audio (I2S amp): [docs/rdk_x5_40pin_i2s_max98357a.md](docs/rdk_x5_40pin_i2s_max98357a.md) ยท [firmware/max98357a/](firmware/max98357a/)
- Why the LLM stays on CPU: [docs/rdk_x5_bpu_llm.md](docs/rdk_x5_bpu_llm.md)
- Network / maintenance: [docs/network_setup.md](docs/network_setup.md) ยท Power/USB brown-out: [docs/power_usb_troubleshooting.md](docs/power_usb_troubleshooting.md)
- Testing: [docs/TESTING.md](docs/TESTING.md) ยท Build log: [docs/diary/](docs/diary/)
## Credits & inspiration
Corosuke's design borrows ideas from these open / animatronic robots:
- [Disney's Olaf robot](https://thewaltdisneycompany.com/olaf-robotic-character/) โ expressive animatronic eyes & face
- [Open Duck Mini (BDX)](https://github.com/apirrone/Open_Duck_Mini) โ compact bipedal droid
- [3D-printed animatronic dual-eye mechanism](https://www.instructables.com/Simplified-3D-Printed-Animatronic-Dual-Eye-Mechani/) โ the eye-rig approach
## License & disclaimer
MIT for the code/hardware โ see [LICENSE](LICENSE). Fan-made, **non-commercial** tribute:
"Korosuke" (ใณใญๅฉ) is a character from *Kiteretsu Daihyakka* by Fujiko F. Fujio; all
character rights belong to their respective owners.
**ใใฏใฌใใคใฏใณใญๅฉใใช๏ผใ** ๐ค