[简体中文](README.md) · **English**
# Screen Translator
**Real-time on-screen translator for Android · capture → read → translate → show / speak**
[](LICENSE)
[](../../releases)
[](../../releases)
[](https://github.com/ciddwd/overlay-translator)
[](../../issues)
[](https://qun.qq.com/universal-share/share?ac=1&authKey=%2Fs0%2FaO4mEHsgutzjUnhGIQEWLcAcGPXTefUY2YwdMkPdnHHuB%2FpLZm9hPjcrw6n5&busi_data=eyJncm91cENvZGUiOiIxMDU5NjU1OTI2IiwidG9rZW4iOiJ4b25nS0FvSFQyMko4WjJTMHhGRlIwSnppeVB2eGJCNjFua0FDTGZzNUhEWlY3VkdPcFVaOEdMams0aEY3aFBTIiwidWluIjoiNTcyMjQyOTk4In0%3D&data=j7H7DHUunIEqMXYLZxhTkx-K_LZTTs5aBJS95LT_Y50uQy37d5IiUU2y3gAPcy9CYRzRufvHuTCaSHOQsLTkTw&svctype=4&tempid=h5_group_info)
Capture via MediaProjection / Shizuku → on-device or cloud OCR → LLM / MT → floating overlay.
Translations can appear beside the original text or be read aloud with a phone voice, a self-hosted voice service, or an online voice.
No ROOT, with on-device options available, designed for visual novels, manga, game dialogue and any on-screen text.
[Install](#-install) · [Usage](#-usage) · [Configuration](#%EF%B8%8F-configuration) · [Contributing](#-contributing) · [Releases](../../releases) · [Issues](../../issues)
---
## ✨ Features
### 🎯 In one line
Game / manga / visual novel on screen → tap the floating ball → the translation appears over the original when recognition and translation finish, with optional read-aloud controls.
### 🖼️ Translate a group of saved images
- Tap **New task** on the home screen to choose several images at once, or share images and screenshots to **Screen Translator** to open a new task directly
- Choose a preset and start the task; processing continues after you leave the task page. The home screen shows the latest or active task, and **Task list** keeps the rest
- The result page shows the source and translation and lets you swipe between images. With **Source-aligned** or **Adapt to screen**, you can also save translated copies that keep the task's font, colors, layout, and text direction
### 🖱️ How to trigger
- **Tap the floating ball**: defaults to translating the whole screen; in **Word-pick mode**, draw around a word or phrase, then fine-tune the selection and tap **Translate** by default. Disable **Precise adjustment** in Settings to recognize and translate immediately on release
- **Long-press the floating ball**: opens an arc menu with common actions one finger can reach: loop / re-pick region / input translation / switch between **"Translate full screen"** and **"Translate a word"** / quick source-target language switch / preset switch / settings / back to main. **Order and page size are configurable** in settings; extra buttons paginate with a "Next page" button automatically
- **Input translation**: single-tap the floating ball to translate input-field text into the preset's source language; double-tap for your choice of full-screen or word-pick translation
- **Quick toggle**: start or stop the capture service directly from your phone's Quick Settings panel
- **Loop mode**: **Translate when settled** is the default, translating automatically when the screen stops changing. You can also translate at fixed intervals or wait until all the text has appeared.
- **Volume two-key**: hold **Vol+ and Vol−** together for 0.3 s to trigger; your hands stay on the game (requires enabling the bundled accessibility service)
- **From any other app**: long-press to select some text in any app → tap "Screen Translator" in the system menu → translation card pops up immediately, no need to switch back to this app first
### 🔍 OCR (read the text on screen)
- **On-device**: ML Kit (CJK + Latin) / PaddleOCR / manga OCR — your screenshots never leave the phone; works offline; PaddleOCR supports multiple v5/v6 tiers, with PP-OCRv5 Korean available as a separate choice
- **Local HTTP OCR**: connect to Umi-OCR / LunaTranslator services on your LAN or PC
- **Cloud**: PP-OCRv6 Online (PaddleOCR AI Studio) / Baidu / Tencent / Youdao — fall back to these when on-device OCR misreads; PP-OCRv6 Online requires an AI Studio Access Token
- When you switch source language, the app checks whether the current OCR engine can read it; if not, it suggests a better one
- First-run setup matches Japanese, Korean, Chinese, and English with the corresponding OCR; other source languages default to recommending PaddleOCR v5 mobile
- Optional orientation detection can route horizontal / vertical / rotated text to a better OCR path, or you can lock the direction manually
- PaddleOCR, manga OCR, and orientation models select 1 / 2 / 4 / 6 CPU inference threads from the device's available cores, capped at 6
### 🌐 Translation engines
- **LLMs**: DeepSeek / ChatGPT / Claude / Zhipu / self-hosted services using OpenAI- or Anthropic-compatible connections, with results shown as they arrive
- **Provider quick setup**: choose ModelScope, OpenRouter, AIHubMix, 302.AI, SCNet, or another listed provider; some services offer mainland China and international addresses
- **On-device translation**: Google ML Kit works offline after downloading its language packs; Sakura is tuned for Japanese manga and visual novels translated into Simplified Chinese, while Hy-MT2 supports more languages
- **Language-direction guard**: source and target cannot be set to the same language; conflicting choices are disabled with an explanation in Settings and the floating quick-switch menu
- **Consistent terminology**: save names, places, organizations, and domain terms globally or per app, either individually or through JSON batch import. Long-press an existing translation to correct the source or translation and keep the correction
- **More consistent over time**: when the same or a similar sentence appears later, the app can reuse a saved translation. Review, search, and manage these entries in the **Translation library**
- **DeepL**: free / Pro plan auto-detected, **supports self-hosted [deeplx](https://github.com/OwO-Network/DeepLX)** (open-source proxy, run on your own server, no key needed); pick between Official / deeplx / Auto-fallback
- **NiuTrans**: choose Flash or Pro; Pro can display translations as they arrive
- **Youdao Pic-Trans**: skips the OCR step entirely — send the screenshot, get back boxes with translations. Great for comics.
- **Google**: no key, free (proxy required inside mainland China)
- Online engines have a **Test connection** action; DeepL can also show the current quota
### 🔊 Read translations aloud (TTS)
- **Phone system voices**: no account is needed. Use voices already available on the phone and adjust their speed and pitch
- **More voice choices**: use your own compatible voice service, or an online service such as Volcengine, MiniMax, or MiMo; online services may charge
- **What it can read**: source text, translations, word cards, and dictionary details. You can also select part of a horizontal passage and read only that selection
- **Playback controls**: tap the same passage again to pause or resume; tapping another passage switches to it, and failures are explained on screen. Some phone TTS engines do not report the current reading position, so resuming with those voices starts the passage again
- TTS is off by default and does not affect recognition or translation when unused
### 🔤 Word lookup (one tap = mini dictionary)
Not just full-screen translation. Draw a rectangle around a single word or phrase and a card pops up with:
- **Translation** — every engine returns this
- **Pronunciation / part of speech / inflections / synonyms / multiple definitions / example sentence pairs / difficulty notes** — only when an **LLM engine** is selected; rare words, specialist terms, abbreviations, cultural references, and confusing usages are explained separately (Baidu / Tencent / DeepL etc. only return a plain translation)
- **Selection workflow** — fine-tune the box and tap **Translate** by default; disable **Precise adjustment** for release-to-translate. **Remember last selection** is off by default
- A "Copy source" and "Copy translation" button on the card; long-press any text inside to copy too
- In landscape, the close bar stays pinned at the top and both copy actions stay pinned at the bottom; only the content scrolls, clear of system bars and display cutouts
- Great for game / manga / VN vocabulary you don't recognize — one second per word, faster than switching to a dictionary app
**Text extraction**: enable **Extract text only**, then select an area to extract its text without translating it. You can copy the text or have it read aloud.
### 🎨 How the overlay looks
- **Two render modes**: glued to each source box (BLOCKS), or packed into a **draggable / resizable floating window** — perfect for games with on-screen joysticks and buttons; you can lock the window to prevent accidental touches
- **Adapt to screen**: when translations are shown over the original screen, it chooses colors, background, and text size for each area. In comics, the bundled bubble detector can erase source text, repair the local background, and fit horizontal or vertical translations to the detected shape. Low-confidence or failed cases fall back to the existing rectangular renderer, with no separate bubble-model download
- **Unified layout and reading order**: translated text follows OCR-detected horizontal / vertical layout and reading order by default; when disabled, both the layout and LTR / RTL direction must be selected manually, and OCR sorting uses the same resolved direction as rendering
- **Selectable translation blocks**: long-press a block for system text selection, or choose tap-to-open panel mode to select a range or copy the full source / translation
- **5 color themes + visual color picker**: choose background, text, border color, and opacity directly instead of typing ARGB; the picker scrolls on short landscape screens
- **Full text styling**: bold, italic, underline, letter spacing, line spacing, alignment, outline, and shadow can be tuned independently, with a live preview
- **Comic / subtitle optimizations**: sentences split across multiple OCR boxes get merged before translating; vertical Japanese is read right-to-left; tiny ruby-text columns (furigana) next to kanji are filtered out so you don't get duplicate translations; vertical scenes can use vertical translation layout
- **Marquee for long lines**: in single-line mode, long translations scroll horizontally instead of being truncated with "…"
- **Custom translation font**: import `.ttf` files for translated text only; the app UI keeps using the system font, and imported fonts stay available as selectable chips
- **Screen orientation correction**: for situations where the picture's orientation differs from the device's orientation
### 🛠️ Small conveniences
- **English / 简体中文 UI + Light / Dark / Follow-system theme**, applies instantly
- **Step-by-step first-run guide**: choose UI, source and target languages, display style, everyday / manga use, translation method, and TTS one page at a time; cloud LLM setup includes provider, URL, API key, and model fields, while offline choices list required downloads
- **Interactive floating-ball tour**: after Capture Service is first enabled, it teaches the long-press gesture, every arc-menu action, and menu pagination; **Help** in the home Capture Service card can rerun both setup and the tour
- **Share with friends**: About can share a feature summary, the official source repository, and the latest download page
- **Compact setting choices**: detection profile, merge strength, display mode, and translation-block copy behavior switch directly through button groups
- **In-settings search**: search in either language; new color, font, backup, import, and export settings are indexed too
- **System presets**: built-in bundles such as "offline Japanese manga OCR → Simplified Chinese" set up recognition, translation, and display in one step; the manga preset uses Adapt to screen by default, and lists any models you still need to download
- **API keys per preset**: each preset can store its own keys on this device and switch them along with its settings; keys are never exported
- **English offline dictionary**: import an English (ECDICT) dictionary pack and choose on-device word lookup
- **Background model downloads**: continue after leaving Settings; Android 13 and later requests notification permission before a download starts, and both the app and system notification show the current model and progress, with cancel, retry, and resume support
- **Per-app translation context**: detect the foreground app and apply its terminology; package names always stay on-device, and only the display name is sent when you explicitly enable that option for a context-aware translator
- **Portable settings bundle**: export / import non-sensitive settings, custom presets, and font files in one step; API keys and tokens remain on the current device
- **Translation cache**: during the current process, identical text with the same translation configuration reuses its result for cloud engines and on-device LLMs
- **Auto-saved crash reports**: Java crashes, native crashes, ANRs, low-memory exits, and related failures are surfaced on the next launch as an exportable sanitized report
- **Baseline TalkBack support**: the floating ball, arc menu, language / preset switchers, region picker, word card, and color picker expose labels, state, and action hints
- **Encrypted sensitive settings**: API keys, prompts, mirror URLs and presets are migrated into encrypted local storage to reduce plaintext leftovers
- **Update prompt**: checks for new versions when you open the home screen (at most once a day); offers a direct link if GitHub is unreachable
- **Chinese-ROM background guides**: shortcuts to auto-start / battery whitelist settings for Xiaomi / OPPO / VIVO / Huawei / Samsung devices that aggressively kill background services
- **Privacy default**: cleartext HTTP is only allowed inside your LAN by default; public hosts must use HTTPS
## 🧩 Project highlights
Screen Translator treats **capture, text recognition, translation, display, and speech** as independently configurable parts:
| Feature | Description |
|---|---|
| **Mix and match recognition and translation** | Read text on the phone or through your own computer or a cloud service, then choose translation separately. One unavailable provider does not disable the whole workflow |
| **Speech is independent from translation** | Any translation method can be paired with a phone voice, your own voice service, or an online voice without reconfiguring recognition and translation |
| **Live screens and saved images** | Use the floating ball to translate what is on screen immediately, or send a group of saved images to the background; both workflows can use your saved presets |
| **Input translation** | Single-tap the floating ball to translate input-field text into the preset's source language; double-tap for your choice of full-screen or word-pick translation |
| **Fully offline or fully self-hosted** | On-device recognition and translation plus an installed offline phone voice can keep screenshots and text on the phone; recognition, translation, and speech services can also run on your own computer or server |
| **Built for games, manga, and visual novels** | Wait for dialogue to finish typing, skip unchanged scenes, prioritize dialogue regions, handle vertical manga, preserve source position and reading direction, and keep per-game names and terms consistent |
| **Open source without configuration lock-in** | Apache-2.0 source; settings, presets, terminology, and fonts can be exported for migration, while API keys stay out of the export |
It is intended for users who want to choose their own recognition, translation, and speech methods while keeping offline use, vertical manga, and self-hosted services available.
## 📸 Screenshots
**Home screen** — Swipe vertically between **Presets / Current status** in the upper section, and horizontally between **Capture service / Batch translation** in the lower section.
| Presets and batch translation | Status and capture service |
|---|---|
| 
| 
|
**Live overlay** — Discord rules page, OCR boxes glued to the source text:
| Classic Dark | Paper Light | Adapt to screen |
|---|---|---|
| 
| 
| 
|
**Japanese manga — Adapt to screen comparison** — The same vertical Japanese page shown as the original, with Adapt to screen, and with the translated text changed to a horizontal left-to-right layout:
| Original | Adapt to screen | Adapt to screen (horizontal, left-to-right) |
|---|---|---|
| 
| 
| 
|
**In-game** — The same Sandship screen compared as the original, a source-aligned overlay, and Adapt to screen.
| Display | Preview |
|---|---|
| Original, no processing | 
|
| Source-aligned overlay | 
|
| Adapt to screen | 
|
**Comic / subtitle scene** — manga bubbles OCR'd by column, translation glued to source:
| Korean manga | Vertical Japanese manga |
|---|---|
| 
| 
|
**Floating-window mode** — collects all source / translated lines into one draggable, resizable overlay so translations don't cover game controls (joysticks, action buttons, dialog-advance keys). The window can be **locked**: unlocked shows a close button + bottom resize handle and accepts drag / pinch; locked strips them away, leaving only the text and ignoring every gesture — no more accidental drags during gameplay.
| Unlocked (drag / resize / close) | Locked (gesture-proof, content only) |
|---|---|
| 
| 
|
**Arc menu (long-press) & word-lookup card** — Left: the arc menu that fans out when you long-press the ball: loop / re-pick region / switch between "Translate full screen" and "Translate a word" / back to main app (the button order is freely reorderable in settings). Right: the card you get after circling a single word on screen — pronunciation, part of speech, multiple definitions and example sentences, plus separate read-aloud controls for the source, translation, and dictionary content (the full dictionary requires an LLM-based engine).
| Arc menu on long-press | Word-lookup card (dictionary + TTS) |
|---|---|
| 
| 
|
**Translation-block selection / copy / read aloud** — Set the translation-block tap action to **Open selection panel**, then tap any translated block to view its source and translation together. Drag to select a range and use the system **Copy / Select all** actions, or copy either full block with one tap. The speaker button beside each section uses the app's read-aloud feature to play the complete source or translation.
Note: Vertical translations cannot currently use long-press to select a short passage for speech. Switch to the tap-to-select mode to read a full source or translation with its speaker button.
**Settings**:
| Language / theme / system presets | Translation backend |
|---|---|
| 
| 
|
| **OCR engines** | **Orientation model settings** |
| 
| 
|
| **Overlay style and font** | **Adapt to screen** |
| 
| 
|
| **Arc-menu buttons** | **Floating-window mode options** |
| 
| 
|
| **Floating ball** | **Smart loop translation** |
| 
| 
|
| **Word-select translation / Capture region / Input translation** | **TTS speech settings** |
|---|---|
| 
| 
|
## 📦 Install
1. Grab the latest `ScreenTranslator-x.y.z.apk` from [Releases](../../releases)
2. Tap it on your Android device (first time you'll need to allow "Install unknown apps" in system settings)
3. Grant **Overlay** and **Notification** permissions on first launch
Only **`arm64-v8a` (64-bit ARM)** builds are currently published. 32-bit ARM and x86 devices are not supported. Check the processor architecture in your system information or device specifications before downloading.
Each APK ships with a `.sha256` so you can verify integrity against `Get-FileHash` / `sha256sum`.
## 🚀 Usage
1. Open the app, tap **Start capture service**, accept the system "Start recording?" dialog
2. Switch to any game / VN / manga app
3. Tap the floating button → the translation appears when recognition and translation finish; the wait depends on the phone, the screen, and the selected services
4. Long-press the floating button → open the arc menu to toggle loop mode, re-pick the capture region, switch source/target language, switch presets, or toggle between full-screen and word-pick translation (loop can use a fixed interval or translate as soon as text stabilizes)
5. By default, tap a translation block to hide it or long-press to select and copy; Settings can instead make taps open the source / translation copy panel
Optional:
- Enable the accessibility service so **holding Vol+ and Vol- together for 300 ms** acts as a global trigger. Screen Translator's own primary overlays expose TalkBack labels, state, and action hints.
- Install [Shizuku](https://github.com/RikkaApps/Shizuku) and grant permission; in settings, switch the capture path to Shizuku to skip the per-session system dialog. Capture prefers the raw-pixel path and falls back to PNG for compatibility.
- Pick a system preset at the top of settings, such as "Offline Japanese manga OCR → Simplified Chinese". If a required model is missing, the preset card lists it and offers a download action.
- For single-word lookup, switch the floating-ball action to **Word-pick mode** from the arc menu, then draw around the word / phrase. You can also select text in another app and invoke Screen Translator from the system selection menu.
- To hear source text or translations, enable TTS in Settings and choose a voice, then tap the speaker button beside the text.
- Add the Screen Translator tile to your phone's Quick Settings panel to start or stop the capture service directly.
### Input translation
Switch to **Input translation** in the arc menu, then tap the input field you want to translate and enter your text:
- **Single-tap the floating ball**: translate the input text into the preset's source language and replace the original content
- **Double-tap the floating ball**: run full-screen or word-pick translation, as selected in the **Input translation** card in Settings
Accessibility permission is required. Input text is read and replaced only when you tap the floating ball. Password-type fields are not read.
## ⚙️ Configuration
Open the app and tap **Settings**. The top of the settings page lets you switch **App language** and **Theme**; any section is reachable through the search icon, matching both Chinese and English keywords.
### OCR engines
| Engine | Good for | Notes |
|---|---|---|
| **On-device** ML Kit (auto / latin / ja / zh / ko) | Default; Japanese / Chinese / Korean / Latin | Offline, on-device |
| **On-device** PaddleOCR PP-OCRv5 / v6 | Multilingual dense text, UI buttons, horizontal Chinese / English | Supports v5 mobile and v6 tiny / small / medium; models must be ready — see below |
| **On-device** PP-OCRv5 Korean | Korean text | A separate Korean recognition model selectable in Settings; download the model before first use |
| **On-device** manga OCR | Japanese manga, vertical bubbles, hand-drawn fonts | Uses manga-ocr ONNX and reuses DBNet detection; ~140 MB, downloadable or importable |
| **Local HTTP** Umi-OCR | You already run [Umi-OCR](https://github.com/hiroi-sora/Umi-OCR) on a PC / LAN server | Point the app at `http://:/api/ocr`; screenshots stay on your LAN |
| **Local HTTP** LunaTranslator | You already run [LunaTranslator](https://github.com/HIllya51/LunaTranslator) OCR on a PC / LAN server | Fill in the LunaTranslator OCR HTTP endpoint |
| **Cloud** PP-OCRv6 Online | Chinese, Japanese, and Latin-script languages | Uses the PaddleOCR AI Studio asynchronous `PP-OCRv6` jobs API; requires an AI Studio Access Token |
| **Cloud** Baidu OCR | Fallback when ML Kit / Paddle miss | Needs API Key + Secret, pay-per-call; 5 endpoints (basic / with-position / accurate / accurate-with-position / web-image); image size / aspect-ratio limits |
| **Cloud** Tencent OCR | Same | Needs SecretId + SecretKey; 3 endpoints: GeneralBasic / GeneralAccurate / **RecognizeAgent** (LLM agent, integrated with ParagNo paragraph grouping) |
| **Cloud** Youdao OCR | Simple one-tap | Needs App ID (API Key) + App Secret; `langType` auto-derived from source language, no separate picker |
On-device OCR benchmark
Test conditions:
- Device: Qualcomm Snapdragon 8 Gen 3 with 16 GB RAM
- Test image: the same full-screen `1440×3200` Japanese manga screenshot for every model
- Preprocessing and text merging disabled; Debug build with detailed performance logging enabled
- “OCR time” is the engine's recognition time. “Complete pipeline” also includes capture, orientation detection, any required model switch, and rendering.
- These measurements come from one device and one image. They compare on-device models within this project and are not universal performance estimates.
#### Speed priority
| OCR model | OCR time | Complete pipeline | Result on this test image |
|---|---:|---:|---|
| PP-OCRv5 Mobile | 1.17 s average | 2.30 s average | 34 segments; reasonable speed, but many wrong, missing, or garbled characters |
| PP-OCRv6 Tiny | 0.65 s | 2.00 s¹ | 29 segments; fastest, but Japanese recognition was largely unusable |
| PP-OCRv6 Small | 0.84 s | 2.51 s¹ | 35 segments; best overall PaddleOCR result, with most body text recognized correctly |
| PP-OCRv6 Medium | 3.60 s | 6.36 s¹ | 35 segments; recovered some body text, but still had recognition and splitting errors |
| Manga OCR | 5.54 s | 6.45 s | 17 complete bubbles; best vertical grouping, sentence integrity, and overall accuracy |
¹ The complete pipeline includes switching from and loading the previous PaddleOCR model: about 0.44 s for Tiny, 0.77 s for Small, and 1.80 s for Medium.
#### Accuracy priority
| OCR model | OCR time | Complete pipeline | Compared with Speed priority |
|---|---:|---:|---|
| PP-OCRv5 Mobile | 3.08 s | 4.24 s | Found 5 more segments, but added false positives and garbled text |
| PP-OCRv6 Tiny | 1.81 s | 3.02 s | Found more boxes, but Japanese remained largely unusable |
| PP-OCRv6 Small | 4.01 s | 5.35 s | Recovered a little small text, while adding several false positives |
| PP-OCRv6 Medium | 16.85 s | 18.16 s | Kept only 2 more segments at a very high time cost |
| Manga OCR | 5.73 s average | 7.00 s average | 21 segments; recovered more small text and improved the difficult bottom area, with some added noise |
For this image, **PP-OCRv6 Small with Speed priority** offered the best balance of speed and Japanese recognition quality. Use **Manga OCR** when vertical manga layout and accuracy matter more. None of the PaddleOCR Accuracy profiles produced a consistent accuracy gain in this test.
On-device translation benchmark
Test conditions:
- Device: Qualcomm Snapdragon 8 Gen 3 with 16 GB RAM
- The same 18 Japanese OCR segments translated into Simplified Chinese
- “Disable translation cache” enabled in Developer options; both runs reported zero cache hits
- CPU inference at TG=6 / PP=6; native JNI B4 batching in 5 groups (4+4+4+4+2)
- The app was force-stopped before each model run. Opening floating translation automatically loaded and prewarmed the selected model.
- “First shown” and “All 18 complete” start at translation batch submission and exclude OCR time. Results were collected from a Debug build.
| On-device model | Model file | Cold prewarm | First shown | All 18 complete | Result on this test text |
|---|---:|---:|---:|---:|---|
| Sakura 1.5B (Q5_K_S) | ~1.20 GB | 2.99 s | 3.40 s | 10.30 s | More natural Chinese, but one negation was reversed and one time expression was mistranslated |
| Hy-MT2 1.8B (Q4_K_M) | ~1.08 GB | 2.38 s | 3.51 s | 10.84 s | Handled negation and the time expression better, but had weaker context, long-sentence, and Chinese phrasing quality |
Sakura showed its first batch about 0.11 s sooner and finished all segments about 0.54 s sooner. Hy-MT2 completed cold prewarming about 0.61 s sooner. Sakura produced more natural Chinese in this sample, while Hy-MT2 handled some semantic details more accurately. Hy-MT2 used a 512 MiB KV buffer versus Sakura's 224 MiB, which matters on devices with less memory.
Supplemental manga sample (Japanese → Simplified Chinese):
- The same screen and the same 13 Manga OCR segments were used. One digits-only segment was preserved directly, leaving 12 segments for translation.
- Timing starts when the translation batch begins and excludes OCR. The digits-only passthrough is not counted as the first real translation.
- Sakura was cold-started after force-stopping the app. Its log reported zero cache hits, TG=6 / PP=6, and three native JNI B4 groups.
- ML Kit was tested after switching engines in the same app process, with its Japanese and Chinese language models already downloaded. It has no separate model-prewarm stage; initialization overhead is included in the first real translation time.
| On-device translator | Model storage | Model prewarm | First real translation | All 12 translations complete | Result on this test text |
|---|---:|---:|---:|---:|---|
| Sakura 1.5B (Q5_K_S) | ~1.20 GB | 3.66 s | 2.06 s | 7.15 s | Better overall quality, with some semantic inaccuracies |
| Google ML Kit (ja → zh-CN) | Japanese and Chinese models, ~30 MB each | None | 0.32 s | 2.68 s | Much faster, but weaker overall quality |
On this sample, ML Kit produced the first real translation about 1.74 s sooner and completed all translations about 4.47 s sooner. Sakura's Chinese quality was clearly better for manga text that depends on context and character voice.
**PaddleOCR models**: the first use requires a download or local import. The default is **PP-OCRv5 mobile**. You can switch versions to suit the screen and language:
- **v5 mobile**: the default, supporting Simplified Chinese, Traditional Chinese, English, and Japanese
- **v6 tiny**: the smallest and fastest, with many Latin-script languages but no Japanese support
- **v6 small**: a balance of speed and accuracy, with Japanese support
- **v6 medium**: larger and slower, for faster devices where a longer wait is acceptable
**PP-OCRv6 Online**: this is a cloud recognition service. It needs an AI Studio Access Token but does not need the on-device models above. Screenshots are sent to PaddleOCR's online service when it is used.
**Orientation models**: these detect rotated screens and upside-down text. They are included with the app, so no extra download is needed when they show as ready.
**Manga OCR models**: the first use requires a download or local import. The main public source is Hugging Face; if it is unavailable on your network, use a proxy, custom mirror, or local import.
**Downloads and ready state**: PaddleOCR, manga OCR, orientation, and offline translation models can download in the background. Downloads continue after leaving Settings, and both the app and system notification show progress. You can cancel at any time; interrupted downloads retry and continue from where they stopped. A model can be used when it shows as ready. Delete the installed model first if you want to replace it.
**Automatic orientation detection**: enabled by default. It chooses horizontal or vertical text from the screen and language. If it gets a page wrong, select horizontal, vertical Japanese, or another direction manually.
**Detection speed and accuracy**: "Speed" is the default. Switch to "Accuracy" when tiny, faint, or narrow vertical text is missed, but expect a longer wait. Most users should leave the advanced OCR settings unchanged; adjust them gradually only if text is repeatedly missed or too many false detections appear.
**On-device performance**: local recognition and translation adjust automatically to the phone. Faster phones wait less, while entry-level devices may take longer; there is no need to set thread counts manually.
**Baidu OCR caveats**: image limits are *longest side ≤ 4096 px, shortest side ≥ 15 px, aspect ratio 1:4–4:1, base64 < 4 MB*. The first two are handled automatically (downscale + JPEG quality fallback); aspect ratio out of range can only be fixed by shrinking the capture region.
### Translation engines
Choose by use case first; you do not need to read every setting before getting started:
| Engine | Best for | What you need | Before you use it |
|---|---|---|---|
| **Sakura (on-device)** | Japanese manga, visual novels, and galgames translated into Simplified Chinese | Download the ~1.26 GB model | Japanese → Simplified Chinese only; works offline but depends on phone performance |
| **Hy-MT2 (on-device)** | Multilingual offline translation | Download the ~1.13 GB model | Source and target follow your settings; low-memory devices may fail to load it |
| **Google ML Kit (on-device)** | Fast offline translation without an account | Download the required language packs on first use | Usually fast, but game dialogue and manga context may be less natural than an LLM |
| **OpenAI-compatible** | Existing DeepSeek, OpenAI, Zhipu, SiliconFlow, Ollama, or compatible services | Base URL, API key, and model name | Supports streaming and custom prompts; pricing and rate limits depend on the provider |
| **Anthropic-compatible** | Existing Claude or another service with Anthropic support | Service URL, API key, and model name | Supports streaming and custom prompts; pricing and rate limits depend on the provider |
| **DeepL / DeepLX** | Conventional machine translation or an existing self-hosted DeepLX service | DeepL Auth Key or a DeepLX service URL | Official DeepL selects free / pro automatically; Test connection can show quota |
| **NiuTrans** | General text translation | API key; Flash also requires an App ID | Choose Flash or Pro; Pro can display translations as they arrive |
| **Youdao PicTrans** | Comics, full images, and screens with many text boxes | Youdao App ID + App Secret | Translates the screenshot directly and does not use the OCR engine selected above |
| **Google** | A quick trial without entering a key | Nothing | Uses an unofficial endpoint; usually needs a proxy in mainland China and may be rate-limited or stop working |
| **Volcengine** | Long loop-translation sessions | AK / SK with Machine Translation enabled | Keep the default region; this is an online service |
| **Baidu Translate** | General text translation | Baidu Translate APPID + secret | Separate product and account system from Baidu OCR |
| **Tencent Translate** | General translation or an existing Tencent OCR setup | Tencent SecretId + SecretKey | Reuses the same credentials as Tencent OCR |
Every online engine has a **Test connection** action. Use it before switching to your game.
How do I fill in an OpenAI-compatible service?
- **Quick setup**: choose ModelScope, OpenRouter, AIHubMix, 302.AI, SCNet, or another listed provider; some services offer mainland China and international addresses
- **Base URL**: use the provider's OpenAI-compatible endpoint, usually ending in `/v1/`
- **API key**: use the key issued by the provider; it may be left blank for a self-hosted service without authentication
- **Model name**: copy the exact name from the provider console or your self-hosted service instead of guessing from an example
- **Prompt template**: the default is written for conversational game translation. You may customize it; untouched prompts follow the UI language, while edited prompts are never overwritten automatically
On-device model downloads and performance
- Hugging Face, hf-mirror, custom mirrors, and local model import are supported; background downloads provide notification progress, cancellation, retry, and resume
- Large downloads warn before starting on cellular or an unknown network type
- After capture starts, an installed and selected manga OCR, Sakura, or Hy-MT2 model is prepared in advance to reduce the first wait
- When one screen contains several text blocks, Sakura / Hy-MT2 tries to handle them together and automatically falls back to one at a time when needed
- On-device translation currently relies mainly on the CPU. Sakura's Vulkan option remains an experimental, default-off feature and may not work or run faster on every device
- Sakura / Hy-MT2 are available only on supported 64-bit devices and OS versions
### Read translations aloud (TTS)
To hear translations aloud, enable TTS in Settings and choose a voice option:
| Option | Best for | What to know |
|---|---|---|
| **Phone system voice** | Using the voices already on your phone | No account or service charge is required. You can choose an installed voice and adjust its speed and pitch |
| **Self-hosted voice service** | People already running a compatible voice service on a PC or local network | Enter its address and keep the voice and service under your own control |
| **Volcengine** | People with an active Volcengine speech account | Enter your account details and use a preset or cloned voice. Provider usage may incur charges |
| **MiniMax** | Finding more voices or cloning and designing your own | Search, preview, and select voices; upload a recording to clone a voice, or describe the new voice you want |
| **MiMo** | People already using the MiMo speech service | Supports preset voices, voice design, and voice cloning, plus plain-language instructions for speaking style |
- **How to speak text**: source text, translations, and word cards have speaker buttons. You can select part of a horizontal passage and read only that selection; for vertical translations, use the tap-to-select mode to read a full block
- **Pause and resume**: tap the same text again to pause, then tap once more to continue. Some phone TTS engines do not provide reading-position updates, so they restart the passage instead; tapping different text always starts the new content
- **When voices are too quiet**: self-hosted services, Volcengine, MiniMax, and MiMo can be boosted in Settings. Very high values may sound distorted. Phone system voices do not support this setting
- **Charges and problems**: phone system voices are free. The app warns before using other services that may charge. Missing voices, incorrect account settings, and playback failures are explained on screen
- **MiniMax voices**: the voice page has Find, Clone, and Design tabs. Search by language or voice name; tapping a voice asks for confirmation before selecting it. Creating, using, and deleting voices also require confirmation
### Terminology and app context
- **Terminology library**: store a source term, preferred translation, language pair, category, case rule, and enabled state, scoped globally or to one app; filters cover source, translation, app, and category
- **Foreground app detection**: choose Auto, Accessibility, Usage access, or Global only. Package names are used only for on-device terminology matching and are never sent to translation services; "Send app name to the model" is a separate opt-in
- **Supported engines**: app names and terminology context currently apply only to OpenAI-compatible, Anthropic-compatible, and on-device LLM translation; plain machine-translation engines keep their existing request format
#### Import terms in bulk
Tap **＋** in the terminology list, choose **Batch import**, and select a JSON file. Each entry only needs a source term and its preferred translation:
```json
[
{"source": "梓ちゃん", "target": "梓酱"},
{"source": "先輩", "target": "前辈"}
]
```
Chinese field names are also supported:
```json
[
{"原文": "梓ちゃん", "译文": "梓酱"}
]
```
- Preview, filter, and deselect entries before importing; choose whether to skip or overwrite matching existing terms
- Different source terms may share the same translation. If one source term has different translations in the file, select which one to keep first
- Each file may contain up to 5,000 entries and must not exceed 2 MB
### Dictionary library
Manage lookup options in **Translation library → Dictionary library**:
- **Online lookup**: use the currently configured cloud LLM; an OpenAI-compatible or Anthropic-compatible translation engine is required
- **Offline lookup**: import an English (ECDICT) `.bin` dictionary pack made for Screen Translator to look up words on your phone
- **Show definitions when tapping source text**: turn this feature on or off here
### Loop translation
- **Trigger mode**: **Translate when settled** is the default; **Fixed interval** and **Wait for text completion** are also available
- **Translate when settled**: automatically translate each new page as you turn it, without repeatedly tapping the floating ball.
- **Animated backgrounds**: on-device OCR can verify that text remains unchanged even while the frame animates; cloud OCR uses frame stability only, avoiding extra paid verification requests
- **Dialogue region**: choose Auto, Lower screen first, or Anywhere, then decide whether to translate only that region or use it for stability while translating all text recognized in the first frame
- **Duplicate skipping**: exact frame matches skip both OCR and translation; similar frames still verify whether OCR text changed, with an adjustable sensitivity threshold
### Display
- **Render mode**: BLOCKS (per OCR box, glued to source) / **Floating window** (a draggable, resizable overlay — see the [screenshots section](#-screenshots))
- **Placement** (BLOCKS only): below / overlap / above + pixel-level x/y offset
- **Translation layout**: "Follow recognized text layout" is enabled by default, so horizontal / vertical layout and reading order jointly drive OCR-block sorting and translated-text rendering; turn it off to explicitly choose horizontal / vertical and LTR / RTL, including vertical manga and horizontal RTL
- **Block interaction**: the default is tap-to-close plus long-press range selection; an alternate mode makes taps open a panel where you can select or copy the full source / translation
- **Floating window**: switch between "source + translation" and "translation only" content modes; **lock** the window to disable dragging and resizing during gameplay; a "Reset position / size" button restores defaults in one tap
- **Screen orientation correction**: for situations where the picture's orientation differs from the device's orientation
- **Floating-ball opacity**: adjust it in the **Floating button** card in Settings
- **Theme**: 5 presets + a visual custom color picker for independent background / text / border color and opacity; borders support solid / dashed / dotted / double / groove styles
- **Text style**: bold / italic / underline, letter spacing, line spacing, left / center / right alignment, outline width and color, plus shadow radius / offset / color
- **Font**: import `.ttf` files for the translation layer and preview only; imported fonts are kept as horizontally selectable chips, and reselecting an existing font does not reorder them
- **Avoidance & merge**: collision detection clamps translation width so it doesn't bleed into neighbouring OCR boxes; OCR-side box merging offers **Conservative / Standard / Aggressive** strengths — Conservative suits VNs / dense passages, Standard fits most scenes, Aggressive suits comic bubbles split into many columns (may occasionally merge adjacent bubbles)
### Presets and Arc Menu
- **System presets**: built-in bundles such as "offline Japanese manga OCR → Simplified Chinese" set up recognition, translation, language, and display in one step. The manga preset uses Adapt to screen by default, and lists missing models for download.
- **Custom presets**: save the current settings as a preset. A preset only shows as applied when all key settings still match; changing any key field returns the card to the unsaved state. Each preset can store its own API keys on this device and switch them along with its settings; keys are never exported.
- **Arc-menu buttons**: drag to reorder and choose 2-6 visible buttons per page. If there are more actions, the last slot becomes "Next page". Actions can include loop, capture region, input translation, word/full-screen mode switch, language quick switch, preset quick switch, settings, and back to main app.
### Backup and migration
- **Complete settings bundle**: `.otsettings` exports all non-credential settings, custom presets, terminology entries, and imported `.ttf` fonts for device migration or configuration sharing
- **Secrets stay on-device**: API keys, tokens, secrets, access IDs, and app IDs are excluded; importing preserves credentials already stored on the destination device
- **Legacy preset support**: older `.otpresets` files remain importable; the new bundle validates imported data and de-duplicates preset names
- **Model choices, not binaries**: selected models, mirrors, and tuning values are exported, but multi-gigabyte model files are not; download or locally import them on the destination device
### Accessibility and diagnostics
- **TalkBack**: primary overlay entry points expose readable labels, toggle state, and double-tap / long-press actions; region selection announces state and dimensions, while complex menus are read row by row
- **Word card**: the top close bar and bottom copy bar stay fixed while the body scrolls; landscape sizing subtracts system bars and cutout insets so actions are not clipped
- **Crash logs**: in addition to Java exceptions, Android 11+ imports system-reported native crashes, ANRs, low-memory exits, and signal exits; settings snapshots are sanitized and system traces are read with a size limit
- **Developer diagnostics**: temporarily disable the translation cache so every test translates again; elapsed time shows per-item translation time by default, with an optional batch-cumulative completion time
## ⚠️ Known limitations
- Android 14+ shows a system permission dialog on every fresh capture session — that's by design. The Shizuku path avoids it.
- Some ROMs (Xiaomi / OPPO / VIVO) kill background services or block overlays by default. You need to whitelist battery + allow background start manually. The app provides in-product shortcuts.
- Anti-cheat networked games may flag MediaProjection as a screen-recorder cheat — this project is for single-player / VN / manga only.
- Screens marked `FLAG_SECURE` (some banking / video apps) capture as black; this project does not bypass it.
- On-device OCR time varies with the phone, selected model, and screen complexity. Region selection can help on slower devices.
- On-device LLM models are roughly 1 GB or larger, so first download should use Wi-Fi; low-memory devices may fail to load them reliably.
- Production on-device OCR and LLM inference is currently CPU-first and does not yet use Qualcomm QNN / NPU. Sakura's Vulkan path remains an experimental, default-off option and is not guaranteed to work or run faster on every device.
- The public manga-OCR source is mainly Hugging Face. If it is unreachable from your network, use a proxy, custom mirror, or local import.
- TalkBack covers the primary overlay workflow, but full-screen OCR text is not yet exposed as individually navigable system accessibility nodes.
## 🗺️ Roadmap
### ✅ Available now
| Feature | What users can do |
|---|---|
| **Floating translation** | Tap the floating ball over a game, manga, or visual novel and show translations beside the source or in a separate floating window |
| **Input translation** | Single-tap the floating ball to translate input-field text into the preset's source language; double-tap for your choice of full-screen or word-pick translation |
| **Quick toggle** | Start or stop the capture service directly from your phone's Quick Settings panel |
| **Batch image translation** | Select multiple saved images and let the task continue after leaving the page; review progress, source text, and translations, then save translated images with the task's text style |
| **Adapt to screen** | Choose colors, background, and text size automatically; comic scenes use bundled bubble detection, source-text erasure, local repair, and shape-aware layout with a safe fallback |
| **Automatic loop** | Translate at a fixed interval or wait for dialogue to finish typing; skip unchanged screens and optionally prioritize lower-screen dialogue boxes |
| **Multiple OCR choices** | Use on-device ML Kit / PaddleOCR / manga OCR, including a separate PP-OCRv5 Korean option; connect to OCR on your LAN, or use Baidu, Tencent, Youdao, and PP-OCRv6 Online |
| **Horizontal and vertical text** | Detect rotated, horizontal, and vertical manga text; follow the recognized layout or manually choose layout and reading direction |
| **Screen orientation correction** | For situations where the picture's orientation differs from the device's orientation |
| **Multiple translators** | Use Google ML Kit on-device translation, OpenAI-compatible services, DeepL, NiuTrans, Google, Volcengine, Baidu, Tencent, Youdao PicTrans, or offline Sakura / Hy-MT2 models |
| **Provider quick setup** | Choose ModelScope, OpenRouter, AIHubMix, 302.AI, SCNet, or another listed provider; some services offer mainland China and international addresses |
| **Copy translations** | Long-press to select a range, or tap a block to open a panel and copy part of the text, the full source, or the full translation |
| **Translation speech (TTS)** | Read source text, translations, and word-card content with phone system voices, a self-hosted service, Volcengine, MiniMax, or MiMo; pause / resume, speak selected text, and manage voices |
| **Word lookup** | Select a word or phrase for a translation; LLM engines can also return pronunciation, part of speech, inflections, synonyms, definitions, examples, and usage notes |
| **English offline dictionary** | Import an English (ECDICT) dictionary pack and choose on-device word lookup |
| **Consistent names and terms** | Save global or per-game names, places, and terminology so wording stays consistent across scenes; import JSON files with preview, filtering, and entry selection |
| **Model downloads** | Download on-device models in the background, view progress in the app and notification, cancel, retry, or resume interrupted files |
| **Faster local translation** | Prepare offline models after capture starts and handle several text blocks together when possible to reduce waiting |
| **Customization and migration** | Adjust floating-ball opacity and translation colors, fonts, size, outline, and shadow; export settings, presets, terminology, and fonts for another device |
| **Assistive features** | Trigger with both volume keys, use TalkBack action hints, export crash logs, and open background-running guides for aggressive Android ROMs |
| **Broader Shizuku compatibility** | Use Sui, official Shizuku, Nightzuku, thedjchi/Shizuku, or Shizuku+ for authorization and screen capture |
### 📋 Next directions
These are user-facing improvements we want to continue working on, not a fixed release order:
| Direction | Intended experience |
|---|---|
| **Floating translation history** | Revisit screens and dialogue translated with the floating ball instead of losing the last line after switching scenes |
| **Character and speaker recognition** | Better identify who is speaking, reduce mistakes in names, relationships, and pronouns, and make ongoing dialogue sound more natural |
| **Long-term on-device model improvements** | Keep improving downloads, startup time, memory use, heat, battery life, and device compatibility across all on-device models, while exploring speech and more features that can run locally on phones |
| **System assistant and agent integration** | Track [Android AppFunctions](https://developer.android.com/ai/appfunctions) and [A2A](https://a2a-protocol.org/latest/), then explore starting translation, speech, and common actions through system assistants or other agents as platform support matures |
| **Deeper accessibility** | Make full-screen OCR text and dynamic translations individually navigable with a screen reader |
On-device model improvements are an ongoing plan that will progress over time as models and phones change, rather than a one-release task.
## 🤝 Contributing
Contributions of any kind — bug fixes, features, UI polish, translations, doc tweaks — are welcome.
If you plan to work on an issue yourself, please leave a comment on it beforehand to let others know, so we can avoid duplicating effort. If no one has said they will take it on, I will generally work on it as my schedule allows.
### Branch & PR rules
> ⚠️ **Do not open PRs against `main`.**
>
> `main` is the stable trunk that drives releases; **only maintainer-reviewed merges land there**.
>
> External contributors:
1. **Open an issue first** to discuss direction — avoid wasted work / scope mismatches
2. **Fork** the repo, branch off the latest `main`:
```bash
git checkout -b feat/your-idea origin/main
```
3. **Develop and test locally** (at least `./gradlew installDebug` on a real device)
4. **Open the PR against `dev`** (if `dev` doesn't exist yet, ping the maintainer in the issue to create it). The maintainer aggregates multiple PRs on `dev`, retests, then merges into `main`
5. **Direct push / force-push to `main` is forbidden** (enforced by branch protection)
### Build locally
```bash
git clone https://github.com/ciddwd/overlay-translator.git
cd overlay-translator
cp local.properties.example local.properties
# Edit local.properties and point sdk.dir at your local Android SDK
./gradlew installDebug # installs onto the connected device
```
Requires JDK 17 + Android SDK 35. Opening the project in Android Studio auto-syncs and fills in the Gradle wrapper jar; on a bare CLI environment, run `gradle wrapper --gradle-version 8.10.2` first.
### Translation contributions
To add a new language or fix an existing one:
1. Copy `app/src/main/res/values/strings.xml` to `app/src/main/res/values-/strings.xml` (e.g. `values-zh-rTW`, `values-ja`)
2. Translate the **values** inside `` tags only. **Keep** the `name="..."` key and placeholders (`{source}`, `{target}`, `%1$s`, `\n`, etc.)
3. Add `` to `app/src/main/res/xml/locales_config.xml`
4. Append a row to `APP_LANGUAGE_OPTIONS` in `app/src/main/java/com/gameocr/app/ui/SettingsScreen.kt`
5. In the PR description, attach coverage (X / Y strings translated) and screenshots of the localized settings page
We may migrate to [Weblate](https://weblate.org/) when there are 10+ languages.
### Commit messages
Use a simplified [Conventional Commits](https://www.conventionalcommits.org/):
```
feat: new feature
fix: bug fix
docs: documentation
refactor: refactor (no behaviour change)
chore: build / CI / tooling
i18n: translations
```
Write the subject in either English or Chinese — **don't mix in one line**. Keep the first line ≤ 72 chars.
### Code of conduct
- Be civil, stay on topic
- No off-topic content, ads, or politics
- Maintainers reserve the right to close issues / PRs that don't match the project direction, with a brief reason
## 🙏 Acknowledgements
### Upstream
- [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR) · PP-OCRv5 mobile model
- [bukuroo/PPOCRv5-ONNX](https://huggingface.co/bukuroo/PPOCRv5-ONNX) · ONNX-converted v5 mobile mirror
- [l0wgear/manga-ocr-2025-onnx](https://huggingface.co/l0wgear/manga-ocr-2025-onnx) · manga-ocr ONNX model
- [kha-white/manga-ocr](https://github.com/kha-white/manga-ocr) · original manga-ocr project
- [ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp) · GGUF / on-device LLM inference
- [SakuraLLM/SakuraLLM](https://github.com/SakuraLLM/SakuraLLM) · Sakura Japanese-to-Chinese translation model
- [Tencent-Hunyuan/HY-MT](https://github.com/Tencent-Hunyuan/HY-MT) · Hy-MT translation model
- [Shizuku](https://github.com/RikkaApps/Shizuku) · privileged channel without ROOT
- [ML Kit](https://developers.google.com/ml-kit) · Google's on-device OCR and translation
- [ONNX Runtime](https://onnxruntime.ai/) · on-device inference engine
- [Jetpack Compose](https://developer.android.com/jetpack/compose) / [Material 3](https://m3.material.io/) · UI stack
- [Hilt](https://dagger.dev/hilt/) · dependency injection
- [Retrofit](https://github.com/square/retrofit) + [OkHttp](https://github.com/square/okhttp) · networking
- [kotlinx.serialization](https://github.com/Kotlin/kotlinx.serialization) · JSON
- [Timber](https://github.com/JakeWharton/timber) · logging
### Contributors
(Will appear automatically after the first PR is merged.)
### Inspiration
PC tools in the VNR / Visual Novel Reader family have served galgame / VN players for years. This project carries that pipeline to Android, redesigned for the phone (battery, ROM constraints, touch UI).
## 📄 License
Code is licensed under [Apache-2.0](LICENSE). Models and third-party dependencies retain their own licenses.