# Architecture Overview

> Last updated: **2026-10-21**. Covers bundled jniLibs binaries, `DownloadService`,
> Kotlin-callback event delivery, JS runtimes, persistent logging, and the
> crash-report pipeline.

Grablytic uses a layered architecture with a Flutter frontend and a Python
download engine. The frontend and engine communicate through a platform-specific
IPC layer. Android adds a Kotlin native layer (binary resolution + keep-alive +
event bridge) between Flutter and Python.

## Tech Stack

| Layer | Technology | Role |
|---|---|---|
| UI framework | Flutter with Impeller renderer | Cross-platform UI at 120fps. Single Dart codebase for Android, Windows, and Linux. |
| State management | Riverpod (StateNotifier + FutureProvider + Provider) | Download queue, history, settings, engine status, presets, playlists, resume, batch, logs. |
| Download engine | Python with yt-dlp (YoutubeDL class API) | Media extraction, format selection, downloading, and post-processing. Never subprocess. |
| Python bridge (Android) | Chaquopy (Gradle plugin) | Embeds CPython 3.11 into the APK. Flutter calls Python via Platform Channels. |
| Python bridge (Desktop) | JSON over stdin/stdout | Flutter spawns Python as a managed subprocess. Structured JSON-RPC messages over stdio. |
| Event bridge (Android) | Kotlin `EngineEventListener.onEvent` callback | yt-dlp hooks push JSON straight to Kotlin → EventChannel. Replaces queue polling (no GIL/polling lag). Public interface so R8 cannot strip it. |
| Native binaries (Android) | `BinaryPackageManager` + jniLibs | `libffmpeg.so` / `libdeno.so` shipped in our own jniLibs (SHA-256 pinned at build time). Executed **in place** in `nativeLibraryDir` (copies won't run on targetSdk > 28). Support trees extracted to `noBackupFilesDir/packages`. |
| Keep-alive (Android) | `DownloadService` (`dataSync` FGS) | Ongoing progress notification, `START_NOT_STICKY`, `onTimeout` stop. Covers process *survival*; process *death* still needs DB resume (follow-up). |
| JS runtime (Android) | Deno `.so` (primary) · QuickJS (`python-quickjs`) | Executes YouTube's EJS challenges + PO Token generation. Priority: deno > node > quickjs. |
| JS runtime (Desktop) | Deno (bootstrapped via GitHub releases) | V8-based runtime for YouTube JS decryption. Downloaded on first run, SHA-256 gated. |
| Challenge scripts | `yt-dlp-ejs` + `remote_components=ejs:github` (only when a JS runtime is configured; never without one) | Solver scripts auto-fetched; integrity gated by yt-dlp's own SHA3-512 + version pin (`ejs.py`), NOT by our SHA-256 stub allowlist (`po_token.verify_js_code` covers only the two inert local stubs). |
| Media processing | FFmpeg (static binary / jniLibs `.so`) | Muxing DASH streams, audio extraction, subtitle/thumbnail embedding, chapter splitting/cutting. |
| Download accelerator | aria2c (static binary) | Parallel fragments (`-x1..16`, validated). **Never** for DASH/HLS (native downloader instead, CVE-2026-50574). |
| Persistence | SharedPreferences + SQLite | Settings/presets/playlists/auth in prefs; download history in SQLite (`sqflite` + `sqflite_common_ffi` on desktop). |
| Logging | `AppLogger` (Dart) + `logger`/`persistent` (Python) | Buffered 30 s flush, `app_logs.txt` mirror, `RotatingFileHandler` (`server_logs.log`), traced IPC, global handlers, Riverpod/nav/engine observers. |
| Crash reporting | `CrashReports` + `GithubReporter` (Dart), `github_notifier` (Python) | Capture is local and unconditional (redacted draft on disk, no network on the fatal path). A three-report queue with fingerprint dedup and a three-day prompt cooldown feeds a prefilled GitHub form the user submits. The Python notifier is a developer tool that needs an explicit token. |
| Fonts | Bricolage Grotesque (headings) / Figtree (body) / Iosevka Charon Mono (mono) | Bundled in `assets/fonts/`, no runtime fetch. InstrumentSans remains only in legacy settings screens. |

## Layered Architecture

```
┌────────────────────────────────────────────────────────────────┐
│                     FLUTTER UI LAYER                            │
│                                                                │
│  ┌────────────┐  ┌────────────┐  ┌────────────┐               │
│  │Onboarding  │  │   Home     │  │  Library   │               │
│  │  Screen    │  │   Screen   │  │   Screen   │               │
│  └────────────┘  └─────┬──────┘  └──────┬─────┘               │
│                        │                │                      │
│  ┌────────────────────────────────────────────────┐            │
│  │           AppShell (BottomNav / NavRail)        │            │
│  │           Home · Library · Settings             │            │
│  └────────────────────────────────────────────────┘            │
│                        │                                        │
│  ┌────────────────────────────────────────────────┐            │
│  │  Format Picker · Batch · Playlist Details       │            │
│  │  Media Preview · Presets · Profile Editor       │            │
│  │  Cookie WebView · Subtitles · Schedule ·        │            │
│  │  SponsorBlock · Templates · Observed Sources ·  │            │
│  │  Download History · Diagnostics & Logs          │            │
│  └────────────────────────────────────────────────┘            │
├────────────────────────────────────────────────────────────────┤
│                     RIVERPOD STATE LAYER                       │
│                                                                │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐     │
│  │  settings    │  │  download    │  │   engine_status   │     │
│  │  Provider    │  │  Provider    │  │   Provider        │     │
│  │(StateNotifier│  │(StateNotifier│  │(FutureProvider)   │     │
│  │ + SharedPref)│  │ + Stream)    │  │                   │     │
│  └──────┬───────┘  └──────┬───────┘  └────────┬─────────┘     │
│         │                 │                   │                │
│  ┌──────┴───────┐  ┌──────┴───────┐  ┌────────┴─────────┐     │
│  │   preset     │  │  playlist    │  │    resume         │     │
│  │   Provider   │  │  Provider    │  │    Provider       │     │
│  │(StateNotifier│  │(StateNotifier│  │(StateNotifier     │     │
│  │ + SharedPref)│  │ + SharedPref)│  │ + FutureProvider) │     │
│  └──────────────┘  └──────────────┘  └──────────────────┘     │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐     │
│  │    batch     │  │   history    │  │   logBuffer /     │     │
│  │   Provider   │  │  (SQLite)    │  │   logIngester     │     │
│  └──────────────┘  └──────────────┘  └──────────────────┘     │
│                                                                │
├────────────────────────────────────────────────────────────────┤
│                  ENGINE SERVICE ABSTRACTION                     │
│                                                                │
│  ┌────────────────────────────────────────────────────────┐    │
│  │                EngineService (abstract)                 │    │
│  │  bootstrap · setPaths · startDownload · cancelDownload  │    │
│  │  progressStream · logStream · getFormats · getPlaylist  │    │
│  │  getSharedUrl · sharedUrlStream · scanResumeCandidates  │    │
│  │  updateCheck · setUpdateChannel                         │    │
│  └──────────────────────┬─────────────────────────────────┘    │
│                         │                                       │
│         ┌───────────────┼───────────────┐                      │
│         │               │               │                      │
│  ┌──────┴──────┐  ┌─────┴──────┐  ┌────┴──────┐               │
│  │ Platform    │  │  Desktop   │  │   Mock    │               │
│  │ Channel     │  │  Engine    │  │   Engine  │               │
│  │ Engine      │  │  Service   │  │   Service │               │
│  │ (Android)   │  │(Win/Lin)   │  │ (Test)    │               │
│  └──────┬──────┘  └─────┬──────┘  └───────────┘               │
│         │               │                                       │
├─────────┴───────────────┴───────────────────────────────────────┤
│                KOTLIN NATIVE LAYER (Android)                    │
│                                                                │
│  ┌──────────────────┐  ┌──────────────────┐  ┌──────────────┐ │
│  │  MainActivity     │  │ BinaryPackage    │  │ Download     │ │
│  │  MethodChannel +  │  │ Manager          │  │ Service      │ │
│  │  EventChannel +   │  │ jniLibs resolve  │  │ dataSync FGS │ │
│  │  onEvent callback │  │ + probe + extract│  │ + notif      │ │
│  └────────┬─────────┘  └────────┬─────────┘  └──────────────┘ │
│           │                     │                              │
│  ┌────────┴─────────┐  ┌────────┴─────────┐                   │
│  │  BootReceiver    │  │ ObservedSources  │                   │
│  │  reboot recovery │  │ PollWorker       │                   │
│  │  SQLite reset    │  │ WorkManager      │                   │
│  └──────────────────┘  └──────────────────┘                   │
│           │                     │                              │
├───────────┴─────────────────────┴──────────────────────────────┤
│                      IPC TRANSPORT LAYER                        │
│                                                                │
│  ┌─────────────────────┐    ┌──────────────────────────┐       │
│  │  Android:           │    │  Desktop:                │       │
│  │  MethodChannel      │    │  JSON-RPC over stdin/    │       │
│  │  "com.theonly.      │    │  stdout subprocess IPC   │       │
│  │  grablytic/engine"  │    │  (UTF-8, \n delimited)   │       │
│  │                     │    │                          │       │
│  │  EventChannel       │    │  Progress events:        │       │
│  │  "com.theonly.      │    │  {"type":"event",...}    │       │
│  │  grablytic/progress"│    │  multiplexed on stdout   │       │
│  │                     │    │                          │       │
│  │  Kotlin callback:   │    │  + {"type":"log",...}    │       │
│  │  onEvent(json)      │    │  filtered to logStream   │       │
│  └─────────────────────┘    └──────────────────────────┘       │
├────────────────────────────────────────────────────────────────┤
│                      PYTHON ENGINE LAYER                       │
│                                                                │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────────┐  │
│  │  paths   │  │  config  │  │  opts_   │  │  format_     │  │
│  │          │  │          │  │  builder │  │  selector    │  │
│  └──────────┘  └──────────┘  └──────────┘  └──────────────┘  │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────────┐  │
│  │  hooks   │  │  errors  │  │  formats │  │  playlist    │  │
│  │+callback │  │+vpn hints│  │          │  │  +guards     │  │
│  └──────────┘  └──────────┘  └──────────┘  └──────────────┘  │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────────┐  │
│  │downloader│  │ bootstrap│  │  resume  │  │  site_       │  │
│  │(Thread)  │  │ (CDN+jni)│  │ +sanitize│  │  profiles    │  │
│  └──────────┘  └──────────┘  └──────────┘  └──────────────┘  │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────────┐  │
│  │ po_token │  │  logger  │  │persistent│  │ github_      │  │
│  │ (JS gen) │  │(struct)  │  │ (files)  │  │ notifier     │  │
│  └──────────┘  └──────────┘  └──────────┘  └──────────────┘  │
└────────────────────────────────────────────────────────────────┘
```

## Flutter Layer: Screens

| Screen | File | Navigation |
|---|---|---|
| Onboarding | `lib/features/onboarding/screens/onboarding_screen.dart` | Shown before first launch, routes to AppShell |
| App Shell | `lib/features/shell/screens/app_shell.dart` | BottomNav (narrow) / NavigationRail (wide), IndexedStack |
| Home | `lib/features/home/screens/home_screen.dart` | Tab 0: URL input, active downloads, format picker |
| Format Picker | `lib/features/home/screens/format_picker_screen.dart` | Modal: format list, quality selection, container, muxed badges |
| Batch Download | `lib/features/home/screens/batch_download_screen.dart` | Multi-URL paste, list management |
| Media Preview | `lib/features/home/screens/media_preview_screen.dart` | Downloaded-file thumbnail + metadata, open in system player |
| Playlist Selection | `lib/features/home/screens/playlist_selection_screen.dart` | Playlist URL → entry multi-select, reverse/shuffle, unavailable marking |
| Share Sheet | `lib/features/home/widgets/share_intent_sheet.dart` | Bottom sheet: shared-URL choice UI, engine-ready gated |
| Download Log Sheet | `lib/features/home/widgets/download_log_sheet.dart` | Bottom sheet: one download's logs (live + history) |
| Overflow Menu | `lib/features/home/widgets/download_overflow_menu.dart` | Per-item actions: redownload, audio, delete, logs |
| Batch Import | `lib/features/home/screens/batch_import_dialog.dart` | Dialog: import from clipboard or file |
| Library | `lib/features/library/screens/library_screen.dart` | Tab 1: completed downloads, grid/list toggle |
| Download History | `lib/features/library/screens/download_history_screen.dart` | SQLite history, search & filters |
| Playlist Details | `lib/features/library/screens/playlist_details_screen.dart` | Playlist entry list, add/remove items |
| Settings | `lib/features/settings/screens/settings_screen.dart` | Tab 2: all settings grouped by section |
| About | `lib/features/settings/screens/about_screen.dart` | Version info, licenses, links |
| Presets | `lib/features/settings/screens/presets_screen.dart` | Download preset CRUD (7 predefined + custom) |
| Profile Editor | `lib/features/settings/screens/profile_editor_screen.dart` | Site profile configuration |
| Cookie WebView | `lib/features/settings/screens/cookie_webview_screen.dart` | In-app browser for cookie-based auth |
| Subtitle Settings | `lib/features/settings/screens/subtitle_settings_screen.dart` | Language selection, auto-subs toggle, embed switch |
| SponsorBlock | `lib/features/settings/screens/sponsorblock_settings_screen.dart` | Category selection for SponsorBlock |
| Schedule | `lib/features/settings/screens/schedule_settings_screen.dart` | Download scheduling config |
| Templates | `lib/features/settings/screens/command_templates_screen.dart` | Custom output template management |
| Observed Sources | `lib/features/settings/screens/observed_sources_screen.dart` | Channel/source monitoring |
| Diagnostics & Logs | `lib/features/settings/screens/log_viewer_screen.dart` | Troubleshooting flow + live log stream + export |

## Navigation

**AppShell** uses `IndexedStack` with 3 tabs:
- Narrow (<600px): BottomNavigationBar with custom-styled `_NavItem` widgets
- Wide (≥600px): `NavigationRail` with `VerticalDivider` + `Expanded`

The engine status banner and download progress are surfaced via Riverpod listeners in the shell. Share intent URLs arrive through `EngineService.sharedUrlStream` and are handled by the shell's `_initSharedUrlListening`; the default path shows a choice bottom sheet (`ShareIntentSheet`) gated on `engineStatusProvider.ready`, with an auto-start opt-in that skips it.

## Riverpod Provider Dependency Graph

```
sharedPreferencesProvider (Provider<SharedPreferences>)
  │
  ├── settingsProvider (StateNotifierProvider<SettingsNotifier, AppSettings>)
  │     └── engineProvider (Provider<EngineService>)  ← reads settings for paths
  │           ├── engineStatusProvider (FutureProvider<EngineStatus>)  ← calls engine.bootstrap()
  │           ├── downloadProvider (StateNotifierProvider<DownloadNotifier, List<DownloadItem>>)
  │           │     └── subscribes to engine.progressStream
  │           └── resumeProvider (StateNotifierProvider<ResumeNotifier, AsyncValue<List<ResumeCandidate>>>)
  │                 └── calls engine.scanResumeCandidates()
  │
  ├── playlistProvider (StateNotifierProvider<PlaylistNotifier, List<Playlist>>)
  ├── presetsProvider (StateNotifierProvider<PresetsNotifier, PresetsState>)
  ├── batchProvider (StateNotifierProvider, multi-URL queue)
  └── downloadHistoryProvider (SQLite-backed history)
```

Plus independent providers:
- `sharedUrlProvider (StateProvider<String?>)`, set by share intent
- `logBufferProvider` (5K circular buffer) → `logEntriesProvider` (filtered view) ← `logIngesterProvider` (subscribes `engine.logStream` + `AppLogger`)

All state that survives app restarts is persisted to `SharedPreferences` (settings, presets, playlists) or SQLite (history). Transient state (active downloads, progress) is held in memory by `DownloadNotifier`.

## Engine Service Abstraction

`EngineService` (`lib/core/engine/engine_service.dart`) defines the unified interface:

```dart
abstract class EngineService {
  Future<Map<String, dynamic>> bootstrap();
  Future<void> setPaths(Map<String, dynamic> paths);
  Future<Map<String, dynamic>> startDownload({String url, String downloadId, Map config, String networkType});
  Future<Map<String, dynamic>> cancelDownload(String downloadId);
  Stream<Map<String, dynamic>> get progressStream;
  Stream<Map<String, dynamic>> get logStream;
  Future<Map<String, dynamic>> getFormats({String url, Map config});
  Future<Map<String, dynamic>> getPlaylistInfo({String url, Map config});
  Future<Map<String, dynamic>> search({required String query, String site = 'youtube', int limit = 20, required Map<String, dynamic> config});
  Future<String?> getSharedUrl();
  Stream<String> get sharedUrlStream;
  Future<Map<String, dynamic>> scanResumeCandidates({required String cacheDir});
  Future<Map<String, dynamic>> queueStatus();
  Future<Map<String, dynamic>> setConcurrency(int maxConcurrent);
  Future<Map<String, dynamic>> clearArchive();
  Future<Map<String, dynamic>> openNotificationSettings();
  Future<Map<String, dynamic>> updateCheck();
  Future<Map<String, dynamic>> setUpdateChannel(String channel);
  Future<Map<String, dynamic>> syncSchedule({required int intervalMinutes, required bool wifiOnly, required bool requiresCharging});
}
```

### Platform Implementations

1. **`PlatformChannelEngineService`** (Android via Chaquopy):
   - MethodChannel `com.theonly.grablytic/engine`: request/response
   - EventChannel `com.theonly.grablytic/progress`: streaming progress events
   - Kotlin `EngineEventListener.onEvent(json)`: direct hook → Flutter path (no polling)
   - `MethodCallHandler` for `intent/shared_url`: receives share intents from Kotlin
   - All payloads serialized as JSON strings, decoded by Python via `jsonDecode`

2. **`DesktopEngineService`** (Windows/Linux via JSON stdin/stdout):
   - Spawns `python -m grablytic_engine` as a managed subprocess
   - `.venv` detection in `engine/` directory first, falls back to `python3`
   - JSON-RPC over line-delimited stdin/stdout with UUID request IDs
   - `_pending` map of `Completer`s for request/response correlation
   - Progress events identified by `{"type": "event", ...}`: broadcast via `StreamController`
   - Log lines identified by `{"type": "log", ...}`: filtered into `logStream`
   - Auto-restart on unexpected process exit (up to 3 attempts)
   - 30-second request timeout

3. **`MockEngineService`** (testing):
   - Returns realistic fake data for all methods
   - Simulates progress stream with timed delays
   - Used by widget tests and frame rate audits

Selection is automatic via `engineProvider`:
- Android → `PlatformChannelEngineService`
- Windows/Linux/macOS → `DesktopEngineService`
- Other (web, etc.) → `MockEngineService`

## Kotlin Native Layer (Android)

| Class | File | Responsibility |
|---|---|---|
| `MainActivity` | `MainActivity.kt` | Chaquopy startup, MethodChannel/EventChannel setup, `paths/set` (bundled jniLibs win over Dart `bin/` paths), `EngineEventListener` callback sink, share-intent receiver, `DownloadService` start/update/done wiring, `download/queue_status` + `download/set_concurrency` + `download/clear_archive` handlers, `schedule/sync` handler for WorkManager scheduling, terminal finished/error/cancelled alert routing |
| `BinaryPackageManager` | `BinaryPackageManager.kt` | Resolves `libffmpeg.so`/`libdeno.so` in `nativeLibraryDir`, extracts `lib*.zip.so` support trees to `noBackupFilesDir/packages`, `--version` probes, zip-slip guard, size-marker skip |
| `DownloadService` | `DownloadService.kt` | `dataSync` foreground service: progress notification, `START_NOT_STICKY`, `onTimeout` stop, swipe-away handling, HIGH-importance completion/error alerts with per-download IDs. No new runtime permissions. |
| `BootReceiver` | `BootReceiver.kt` | BroadcastReceiver for `ACTION_BOOT_COMPLETED` and `ACTION_MY_PACKAGE_REPLACED`. Sweeps SQLite `downloads` table to mark unfinished rows as `interrupted`. Never starts an FGS directly to comply with Android 14+ background execution limits. |
| `ObservedSourcesPollWorker` | `ObservedSourcesPollWorker.kt` | WorkManager CoroutineWorker for `observed-sources-poll`. Runs periodic inexact checks with battery/network constraints. Executes two-tier poll (Atom RSS feed first; Chaquopy `extract_flat` fallback), checks against schema v4 `seen_source_videos` table, and queues new videos directly into SQLite. |

> Binary updates ride **app updates**: downloaded files can't execute on
> targetSdk > 28, so silent in-app binary updating is impossible on Android.
> Desktop keeps the CDN bootstrap path.

## IPC Contract

The IPC payload shapes are defined below and are identical across platforms. Only the transport differs.

### Android (MethodChannel + EventChannel + callback)

```
Flutter Dart → MethodChannel.invokeMethod() → Kotlin → Python function call
Python return → Kotlin → MethodChannel → Flutter Future<Map>

Progress stream (new, Sept 2026):
Python hooks → EngineEventListener.onEvent(json) → EventChannel.Sink → Flutter Stream<Map>
(fallback: threading.Queue → queue_reader thread → EventChannel)

Terminal vs stream contract: streaming events are capped at 99%;
only the terminal `finished` event carries 100% + filesize_bytes.
```

### Desktop (JSON-RPC over stdin/stdout)

```
Flutter Dart → JSON encode → stdin write → Python reads line
Python → JSON encode → stdout write → Flutter reads line → JSON decode → Future<Map>

Request:  {"id": "uuid", "method": "download/start", "params": {...}}
Response: {"id": "uuid", "result": {...}}  or  {"id": "uuid", "error": {...}}

Progress events interleaved on stdout:
{"type": "event", "event": "downloading", "download_id": "...", ...}
Log lines interleaved on stdout:
{"type": "log", "level": "...", "message": "...", ...}
```

### Channels

| Channel | Direction | Purpose |
|---|---|---|
| `engine/bootstrap` | F → P | App startup: check binaries, manifest, yt-dlp version |
| `engine/contract` | F → P | Hand back the pinned method list and error codes, or refuse a mismatched major |
| `paths/set` | F → P | Inject data/output/cache dirs, binary paths, cookies, log level (8 keys on the desktop payload; Android passes eleven positional arguments to `set_paths()` and so also sets `po_token`, `ffmpeg_ld_path` and `nodejs_path`) |
| `download/start` | F → P | Start download in thread, returns immediately |
| `download/cancel` | F → P | Set threading.Event cancel flag |
| `download/queue_status` | F → P | Snapshot of the queue and in-flight downloads; runs inline, never queued behind an extraction |
| `download/set_concurrency` | F → P | Write the concurrency limit and pump the queue |
| `download/clear_archive` | F → P | Unlink one archive file strictly inside the data dir, so an archive-skipped URL can be re-downloaded; runs on the maintenance lane |
| `progress/stream` | P → F | Continuous progress/error/complete events (an EventChannel stream, not a method) |
| `formats/get` | F → P | List available streams for URL |
| `playlist/info` | F → P | List playlist entries with metadata |
| `search/query` | F → P | Search YouTube/SoundCloud queries via yt-dlp search extractors |
| `resume/scan` | F → P | Scan cache dir for .part files |
| `resume/report` | F → P | Record whether a resume attempt succeeded: success clears the strike count, failure increments a per-file strike count, and three strikes mark the candidate `exhausted` |
| `schedule/sync` | F → K | Synchronize WorkManager periodic polling constraints and interval |
| `engine/update_check` | F → P | Force re-check binaries from CDN |
| `engine/set_update_channel` | F → P | Set stable/nightly/master channel |

## Python Engine: 23 Modules + Entry Point

| Module | File | Responsibility |
|---|---|---|
| `paths` | `engine/grablytic_engine/paths.py` | Global path store. `set_paths()`, `get_paths()`, `is_initialized()`. Injects binary directories to PATH, prepends site-packages to sys.path for dynamic yt-dlp updates. |
| `config` | `engine/grablytic_engine/config.py` | `DEFAULT_CFG` dict: format, container, quality ceiling, subtitles, SponsorBlock, sections, retries, network, auth, archive, playlist, live. |
| `opts_builder` | `engine/grablytic_engine/opts_builder.py` | `build_ydl_opts()`, merges config with paths, validated aria2c opts (native for DASH/HLS), canonical PP chain (EmbedSubtitle → ModifyChapters → Metadata), `updatetime: False`, `writethumbnail`, sections via `download_range_func`, JS runtime via `_configure_js_runtime()`. |
| `format_selector` | `engine/grablytic_engine/format_selector.py` | `build_format_string()`, tiered format string: explicit ID > audio-only > quality ceiling cascade (AV1 → VP9 → H264). Muxed-stream support for non-YouTube. |
| `site_profiles` | `engine/grablytic_engine/site_profiles.py` | 6 built-in profiles (YouTube 1080p, YouTube 4K, Podcast Audio, Lossless FLAC, Opus Compact, Twitter/X Video). CDN-fetchable overrides. |
| `downloader` | `engine/grablytic_engine/downloader.py` | `start_download()` spawns `threading.Thread` → `download_thread()`. Creates `YoutubeDL` with built opts, progress hook + cancel check. `_active_downloads` updated in place; terminal `finished` carries `filesize_bytes`; errors carry `suggests_vpn`. Accepts `event_callback`. |
| `hooks` | `engine/grablytic_engine/hooks.py` | `build_progress_hook()`, yt-dlp progress hook → queue **or** `callback.onEvent(json)`. 99 % cap on streaming progress; `filesize_bytes` on finished. `pp_key`-keyed stage lookup with real PP names. |
| `errors` | `engine/grablytic_engine/errors.py` | `GrablyticError` with typed error codes (ERROR_GEO_BLOCKED, ERROR_RATE_LIMITED, …) + recoverable flag + suggests_vpn. `classify_error()` matches exception text against keyword map. `ERROR_CANCELLED` maps to `cancelled` event on desktop. |
| `formats` | `engine/grablytic_engine/formats.py` | `get_formats()`, extract info without downloading, parse format list into structured objects with codec/resolution/bitrate. Returns recommended video/audio format IDs. |
| `playlist` | `engine/grablytic_engine/playlist.py` | `get_playlist_info()`, flat-extract playlist entries with generator guards, expanded video IDs, deleted-entry marking, storage sanitization. |
| `search` | `engine/grablytic_engine/search.py` | `search()`, flat-extract YouTube (`ytsearch{N}:`) and SoundCloud (`scsearch{N}:`) results via yt-dlp search extractors with entry normalization and error classification. |
| `bootstrap` | `engine/grablytic_engine/bootstrap.py` | CDN manifest fetch, SHA-256-or-fail, Zip/Tar-Slip-hardened extraction, parallel ffmpeg/aria2c/deno fetch, uv venv + yt-dlp install (desktop), Android fail-closed + bundled-jniLibs fallback with live `--version` probes, QuickJS + Deno detection. |
| `resume` | `engine/grablytic_engine/resume.py` | `scan_resume_candidates()`, scans cache dir for .part files, checks age vs 24h expiry, recovers URL from .info.json metadata, sanitizes storage paths. |
| `scheduler_check` | `engine/grablytic_engine/scheduler_check.py` | Pure helper functions (`parse_atom_feed_entries`, `filter_unseen_videos`) for lightweight YouTube Atom RSS XML feed parsing and video deduplication without heavy extractor overhead. |
| `po_token` | `engine/grablytic_engine/po_token.py` | Allowlisted JS (`verify_js_code`), `detect_js_runtime()` (paths module first), QuickJS-context + Deno-subprocess generation. |
| `logger` | `engine/grablytic_engine/logger.py` | 5-level structured logger, daily rotation, IPC queue forwarding, thread-local context, trace timing, exception logging with tracebacks. |
| `persistent` | `engine/grablytic_engine/persistent.py` | `RotatingFileHandler` (`server_logs.log`), `traced_request` IPC middleware. |
| `github_notifier` | `engine/grablytic_engine/github_notifier.py` | Crash search-based dedup + auto-report pipeline. |
| `contract` | `engine/grablytic_engine/contract.py` | Pins the IPC shape (method names, field names, error codes) so a host and an engine from different builds refuse each other instead of half-understanding each other. |
| `filename_guard` | `engine/grablytic_engine/filename_guard.py` | Bounds every path component to `NAME_MAX` in **bytes**, split on both separators, so an emoji-laden title cannot fail with `[Errno 36]`. |
| `url_guard` | `engine/grablytic_engine/url_guard.py` | Input guards for every IPC-controlled network and file sink: scheme and host policy, private-address refusal, proxy validation. Fails closed. |
| `ytdlp_network` | `engine/grablytic_engine/ytdlp_network.py` | Pins yt-dlp's own HTTP(S) connections to the validated transport, closing the DNS time-of-check window that a URL-only check leaves open. |
| `wheel_pin` | `engine/grablytic_engine/wheel_pin.py` | SHA-256 for the yt-dlp wheel the bootstrap installs. A version pin says which release was tested; the digest says the bytes are that release. |

### `__main__.py`: Entry Point

The desktop IPC entry point (`python -m grablytic_engine`):
1. Starts a daemon thread (`poll_queues`) that drains download progress/result queues
2. Enters a persistent JSON-RPC stdin loop, dispatching to module functions by method name
3. Supports CLI mode for one-shot commands (e.g., `python -m grablytic_engine bootstrap`)

## Data Flow: Complete Download Lifecycle

```
┌──────────────────────────────────────────────────────────────────────┐
│ USER PASTES URL (or SHARE INTENT)                                    │
│   ↓                                                                   │
│ Flutter: HomeScreen captures URL (or shell receives sharedUrlStream)  │
│   ↓                                                                   │
│ Flutter: EngineService.getFormats(url, config)                        │
│   ├─ Android: MethodChannel('formats/get') → Python get_formats()     │
│   └─ Desktop: JSON-RPC stdin → formats/get → stdout                   │
│   ↓                                                                   │
│ Python: yt_dlp.YoutubeDL.extract_info(url, download=False)            │
│   (+ JS runtime resolves n-sig/SABR via deno/quickjs + ejs)           │
│   ↓                                                                   │
│ Flutter: FormatPickerScreen displays available streams                │
│   ↓                                                                   │
│ USER PICKS FORMAT (or uses preset/default)                            │
│   ↓                                                                   │
│ Flutter: DownloadService.start(id, title) [Android keep-alive]        │
│ Flutter: EngineService.startDownload(url, downloadId, config, net)    │
│   ├─ Android: MethodChannel('download/start') → Python                │
│   └─ Desktop: JSON-RPC stdin → download/start                         │
│   ↓                                                                   │
│ Python: start_download() spawns threading.Thread                      │
│   ↓                                                                   │
│ Python: build_ydl_opts() → format string + PP chain + JS runtime      │
│   ↓                                                                   │
│ Python: YoutubeDL(download_opts).download([url])                      │
│   ↓ concurrent fragment fetching (aria2c for progressive, native DASH)│
│ Python: progress_hook → event_callback.onEvent() [Android]            │
│                        → progress_queue → poll → stdout [Desktop]      │
│   ├─ Android: EventChannel → Flutter Stream<Map>                      │
│   └─ Desktop: poll_queues → stdout → Flutter line reader              │
│   ↓                                                                   │
│ Flutter: downloadProvider.handleProgressEvent() → state update        │
│ Flutter: DownloadService.update(id, percent) [Android notif]          │
│   ↓                                                                   │
│ Python: FFmpeg post-processing (merge, embed thumbnail/subs, tags)    │
│   ↓                                                                   │
│ Python: result_queue → terminal finished/error/cancelled event        │
│   (finished carries filesize_bytes; errors carry suggests_vpn)        │
│   ↓                                                                   │
│ Flutter: DownloadItem marked completed/error → UI update + history    │
│ Flutter: DownloadService.done(id) [Android stop when idle]            │
└──────────────────────────────────────────────────────────────────────┘
```

## JS Runtime & YouTube Bypass

YouTube requires a JS runtime (yt-dlp 2025.11.12+). Priority: **deno >
node > quickjs**, resolved by `_configure_js_runtime()` with absolute paths:

- Android: bundled `libdeno.so` (via `BinaryPackageManager` → `deno_path`)
  wins; QuickJS binding is the lightweight fallback.
- Desktop: bootstrapped Deno binary (SHA-256 gated) or system `deno`/`node`.
- `remote_components = ["ejs:github"]` set only when a JS runtime is
  actually configured (never on the no-runtime path), so challenge scripts
  stay fresh without app updates without pointless unfetchable requests.
- PO Tokens generated via `po_token.generate_po_token()` (inert allowlisted
  stubs returning null; real challenge scripts arrive via `yt-dlp-ejs` and
  are covered by yt-dlp's hash/version pin, not our stub list. See
  `po_token.py` module docstring for the trust boundary).
- Extractor args use `player_client = ["default", "mweb"]`, and never force a
  single client (multi-client returns 31+ formats incl. AV1/VP9).

Full research: `docs/JS_RUNTIMES_ANDROID_RESEARCH.md`.

## Logging & Diagnostics Pipeline

```
Flutter: AppLogger (buffered, 30 s worker, ERROR/FATAL instant flush)
   → app_logs.txt mirror + LogBuffer (5K circular) → LiveLogView
   → GithubReporter (search dedup) → auto-filed issues
   → observers: navigation / Riverpod / engine tracing

Python: EngineLogger (5 levels, rotation, thread-local ctx)
   → server_logs.log (RotatingFileHandler) + IPC queue ({"type":"log"})
   → traced_request middleware → github_notifier (dedup)
```

View live: **Settings → Diagnostics & Logs → Live Stream** (level/tag/search/
source filters, tap-to-expand, auto-scroll, export).

## Theme System

Colors follow the "Earth & Ethos" palette defined in DESIGN.md tokens.
`GrablyticColors` provides light and dark color constants used by `AppTheme`.

- `AppTheme.light()` and `AppTheme.dark()` build Material 3 `ThemeData` from `ColorScheme`
- Typography applies bundled Bricolage Grotesque to headings and Figtree to body text
- Monospace text uses `GrablyticTextStyles.mono` extension on `TextTheme` (Iosevka Charon Mono)

## Accessibility

All screens pass WCAG 2.2 compliance:
- Semantic labels via `Semantics` widget on all interactive elements
- 48x48 minimum touch targets via `ConstrainedBox(minWidth: 48, minHeight: 48)`
- Screen reader support with descriptive `label` properties
- Navigation labels on all BottomNav and NavigationRail items

The rules live in `AGENTS.md`. Touch-target sizes and semantic labels are
asserted in the widget tests under
[`test/features/`](https://github.com/OnlyXianzo/Grablytic/tree/main/test/features).
`test/performance/frame_rate_audit.dart` is a rendering check, not an
accessibility one: it asserts that screens build and paint without errors.

## Test Coverage

```
test/                              105 test files
├── core/                          engine bridge, logger, crash queue, utils
├── features/                      one folder per screen or widget
├── providers/                     queue, resume, settings, downloads
└── performance/                   frame-rate and platform matrix audits

engine/tests/                      62 files, 844 tests
├── test_config.py                defaults and merge behaviour
├── test_downloader.py            cancel, VPN hints, filesize, finished contract
├── test_errors.py                error classification and edge cases
├── test_format_selector.py       format strings and storage guards
├── test_hooks.py                 99% cap, pp_key stages, callback vs queue
├── test_main_ipc.py              JSON-RPC dispatch loop
├── test_opts_builder.py          PP order, sections, aria2c, JS runtime
├── test_paths.py                 path injection, PATH management
├── test_resume_hardening.py      .part/.info.json scans, bounds
├── test_bootstrap_extract.py     Zip-Slip and Tar-Slip hardening
├── test_url_guard.py             SSRF, traversal, proxy validation
├── test_filename_guard.py        NAME_MAX in bytes, both separators
└── …                             one file per engine module
```

Both suites are gated in CI rather than counted in docs: the Flutter job
enforces a 30% floor over `lib/`, the Python job a 75% floor over the engine,
and the broad-catch ratchet fails when either count grows without a recorded
reason. Run `pytest engine/tests/ -q` and `flutter test` locally for the
current numbers.

The gates are code too, so `tool/tests/` covers them and the Python job runs
`pytest tool/tests -q`. A gate that silently stops matching is worse than no
gate: the version-literal check only caught one of the six shapes that had
actually shipped until it was taught to look inside a quoted string rather
than requiring the string to be nothing but a version.

## Build & Deploy

### GitHub Actions

**Build workflow** (`build.yml`):
- Manual trigger (`workflow_dispatch`)
- **Android matrix** (parallel builds on `ubuntu-latest`):
  - ARM64-v8a + Deno APK (`grablytic-v<ver>-arm64-deno.apk`, recommended default for modern 64-bit phones)
  - ARM64-v8a + Node.js APK (`grablytic-v<ver>-arm64-node.apk`, smaller download for 64-bit phones)
  - ARMeabi-v7a + Node.js APK (`grablytic-v<ver>-armv7-node.apk`, budget & low-end 32-bit phones like Samsung Galaxy A04e)
  - x86_64 + Deno APK (`grablytic-v<ver>-x86_64-deno.apk`, 64-bit emulators & Chromebooks)
  - x86_64 + Node.js APK (`grablytic-v<ver>-x86_64-node.apk`, lightweight 64-bit emulator build)
  - Universal APK (`grablytic-v<ver>-universal.apk`, multi-ABI fallback covering ARMv7, ARM64, and x86_64 with both runtimes)
- **Linux matrix** (`ubuntu-latest` for x64, `ubuntu-24.04-arm` for arm64):
  - Debian packages (`.deb` via `dpkg-deb`)
  - RedHat / openSUSE packages (`.rpm` via `alien`)
  - Arch Linux packages (`.pkg.tar.zst` via `zstd`)
  - Portable release tarballs (`.tar.gz`)
  - Python engine bundled directly alongside each release
- **Windows** (`windows-latest`):
  - 64-bit portable release zip (`grablytic-windows-x64.zip`) + release bundle with Python engine included

**Verification workflow** (`verify.yml`): runs on every push and PR. The job
table lives in [building.md](building.md#verification-verifyyml) and
`tool/tests/test_docs_match_code.py` fails when the table drifts from the
workflow, so this page links to it instead of keeping a second copy. The same
test keeps the IPC table above in step with `contract.METHODS`.

**Publish** (`publish-release.yml`): attaches `SHA256SUMS` and keyless cosign
bundles to an existing release, verified before upload. `nightly.yml` builds
the rolling nightly tag, and the two `winget-*` workflows maintain the Windows
package manifests.

### Binary Distribution

- Desktop FFmpeg/aria2c/Deno + yt-dlp updates via CDN/GitHub releases (SHA-256-or-fail)
- Manifest JSON lists available versions per platform with SHA-256 checksums
- Bootstrap on first download, SHA-verified on every launch via `engine/bootstrap`
- yt-dlp updates applied via uv venv + site-packages injection (desktop)
- Android binaries ride app updates (jniLibs), so nothing is downloaded and executed at runtime
- Dynamic `PYTHONPATH` injection ensures updated modules load before system packages

### Desktop Virtual Environment

- `.venv` at `engine/.venv/` or project root `.venv/` auto-detected by `DesktopEngineService`
- Falls back to system `python3` if no venv found
- `PYTHONPATH` extended with `engine/` directory for module discovery in development
- Production bundles include the `grablytic_engine` directory as a resource

## Performance

- Impeller renderer enabled for 120fps UI
- Frame rate audit in `test/performance/frame_rate_audit.dart`
- All downloads run in `threading.Thread` (never main thread)
- Progress events stream at hook rate; Kotlin callback removes polling latency
- aria2c external downloader for progressive fragments (native for DASH/HLS)
- `continuedl=True` for resume support
- No `setState()` for shared state. All of it is Riverpod, which keeps rebuilds
  local