# VHS — Video Hoarding System ## 1. Context VHS is an open source project to download, archive, organize and play back videos from YouTube, RaiPlay and, in the future, other platforms. The primary goal is to build a **personal video library**. The project is not meant to redistribute the archived content. ### Vision VHS must be: - simple to install; - simple to understand; - modular; - API-first; - independent of the UI; - extensible to new platforms; - suitable for a future open source community. The Vue UI provided by the project will therefore be **a client of the API**, not the only way to use VHS. In the future there may be: - CLI; - personal scripts; - media server integrations; - plugins; - alternative clients; - external automations. --- # 2. Architectural principle The fundamental principle is: > **Every component must have a clear responsibility and must know as little as possible about the other components.** The main pipeline is: ```text URL │ ▼ API │ ▼ Service │ ├──► Database │ └──► Task │ ▼ Engine │ ▼ Storage │ ▼ Library ``` For playback: ```text Browser │ ├──► API ──► playback progress │ └──► stream endpoint │ ▼ Django │ │ authorization │ X-Accel-Redirect ▼ Nginx │ ▼ Video file ``` The responsibilities are therefore: | Component | Responsibility | | --------- | -------------- | | Vue | UI, player, local state | | API | public HTTP contract | | Services | application logic and use cases | | Models | persistence and domain | | Tasks | asynchronous execution | | Engine | download and metadata extraction | | Storage | physical file management | | Nginx | proxy, HTTPS and file transfer | | PostgreSQL | application data and job state | --- # 3. Technology stack | Component | Choice | Responsibility | | ---------- | ------ | -------------- | | Backend | Django 6.x | application, ORM, authentication, API | | API | Django Ninja | REST/OpenAPI API | | Frontend | Vue 3 + Vite + TypeScript | user interface | | State management | Pinia | frontend state | | Task queue | django-q2 | asynchronous execution and scheduling | | Task broker/backend | PostgreSQL | task persistence | | Database | PostgreSQL 16 | application data and search | | Download engine | yt-dlp + ffmpeg | download and processing | | Reverse proxy | Nginx | proxy, HTTPS, static files, protected media | | Package manager | uv | Python dependencies | | Container | Docker Compose | local and production environment | | Storage | configurable filesystem | video library | | Deployment | server or VM with Docker | production environment (for example a Proxmox VM) | | Network storage | NFS | post-MVP, not supported in the MVP (§15) | ### Intentionally reduced infrastructure dependencies VHS **does not use Redis**. PostgreSQL is already required for the application domain, and django-q2 also uses it for task persistence. This avoids introducing an additional infrastructure service without a concrete requirement. If a real need for Redis emerges in the future, it can be introduced later. --- # 4. Architecture ```text ┌──────────────────────┐ │ Nginx │ │ │ │ HTTPS │ │ Reverse proxy │ │ Vue static files │ │ Protected media │ └──────────┬───────────┘ │ ┌───────────────┴───────────────┐ │ │ ▼ ▼ ┌──────────────┐ ┌──────────────┐ │ Vue SPA │ │ Django API │ │ │ │ │ │ Player │ │ /api/v1/* │ │ Library UI │ │ Auth │ └──────────────┘ └──────┬───────┘ │ ┌─────────────────────┼──────────────────┐ │ │ │ ▼ ▼ ▼ ┌────────────┐ ┌──────────────┐ ┌────────────┐ │ Services │ │ PostgreSQL │ │ Task layer │ │ │ │ │ │ django-q2 │ └─────┬──────┘ └──────────────┘ └─────┬──────┘ │ │ ▼ │ ┌────────────┐ │ │ Engine │◄────────────────────────────────┘ │ │ │ yt-dlp │ │ ffmpeg │ └─────┬──────┘ │ ▼ ┌────────────┐ │ Storage │ │ │ │ local │ └────────────┘ ``` --- # 5. Architectural boundaries ## 5.1 Frontend Vue knows about: - the HTTP API; - the interface state; - the player; - routes; - components. Vue **does not know about**: - the Django ORM; - PostgreSQL; - the filesystem and storage; - yt-dlp; - django-q2. --- ## 5.2 API Django Ninja exposes VHS's public contract. The API: - validates input; - authenticates/authorizes; - serializes data; - invokes the services; - returns HTTP responses. The API **does not contain complex business logic**. Example: ```text POST /api/v1/downloads/ │ ▼ DownloadService ``` not: ```text API endpoint ├── yt-dlp ├── filesystem ├── ORM ├── retry └── business logic ``` --- ## 5.3 Services The services represent the **application use cases**. Examples: ```text DownloadService ChannelService LibraryService PlaybackService ``` They can be used by: - the API; - tasks; - management commands; - tests; - any future clients. The services must not depend on Vue. The services may use: - the ORM; - the engine; - storage; - other required application components. --- ## 5.4 Tasks django-q2 is an infrastructure detail. Tasks must be thin: ```text task ↓ service ``` and contain no business logic. Example: ```text download_task(download_id) ↓ DownloadService.execute(download_id) ``` The service must not know whether it was invoked: - by django-q2; - by a test; - by a CLI; - by another part of the application. --- ## 5.5 Engine The engine is pure Python and does not depend on Django. Responsibilities: - interact with yt-dlp; - extract metadata; - download; - report progress; - use ffmpeg when needed. The engine does not save Django models directly. --- ## 5.6 Storage Storage is the boundary between VHS and the filesystem. The domain should not depend directly on: ```python open("/srv/video-library/...") ``` but on a storage abstraction. Conceptual example: ```text Storage ├── save() ├── exists() ├── delete() ├── move() ├── get_path() └── available_space() ``` Initial implementation: ```text FilesystemStorage ``` which can point to: - a folder on the local disk (development without Docker); - a host folder mounted into the container (bind mount), with Docker. NFS, SMB and NAS are not supported in the MVP (§15 and [storage-decisions.md](storage-decisions.md)). This leaves open the future possibility of implementing other storage backends without changing the domain. --- # 6. Repository structure ```text vhs/ ├── backend/ │ ├── config/ │ │ ├── settings.py │ │ ├── urls.py │ │ └── asgi.py │ │ │ ├── core/ │ │ ├── models/ │ │ │ ├── video.py │ │ │ ├── channel.py │ │ │ ├── source_playlist.py # post-MVP (subscriptions) │ │ │ ├── download.py │ │ │ ├── tag.py │ │ │ ├── collection.py │ │ │ ├── playback.py │ │ │ └── cookies.py # deferred (§14) │ │ │ │ │ ├── admin/ │ │ └── migrations/ │ │ │ ├── api/ │ │ ├── __init__.py │ │ ├── videos.py │ │ ├── channels.py │ │ ├── source_playlists.py # post-MVP (subscriptions) │ │ ├── downloads.py │ │ ├── library.py │ │ ├── playback.py │ │ ├── cookies.py # deferred (§14) │ │ └── schemas/ │ │ │ ├── engine/ │ │ ├── downloader/ │ │ │ ├── base.py │ │ │ └── ytdlp.py │ │ │ │ │ ├── platforms/ # one class per supported platform, and the registry │ │ │ ├── base.py │ │ │ └── youtube.py │ │ │ │ │ ├── metadata.py # platform-neutral video metadata │ │ ├── urls.py │ │ └── naming.py │ │ │ ├── storage/ │ │ ├── base.py │ │ └── filesystem.py │ │ │ ├── services/ │ │ ├── download_service.py │ │ ├── channel_service.py # post-MVP (subscriptions) │ │ ├── source_playlist_service.py # post-MVP (subscriptions) │ │ ├── library_service.py │ │ └── playback_service.py │ │ │ ├── tasks/ │ │ ├── downloads.py │ │ ├── channels.py # post-MVP (polling) │ │ └── source_playlists.py # post-MVP (polling) │ │ │ └── manage.py │ ├── frontend/ │ ├── src/ │ │ ├── api/ │ │ ├── components/ │ │ ├── composables/ │ │ ├── layouts/ │ │ ├── pages/ │ │ ├── router/ │ │ ├── stores/ │ │ ├── App.vue │ │ └── main.ts │ │ │ ├── package.json │ ├── vite.config.ts │ └── tsconfig.json │ ├── docker/ │ ├── Dockerfile.backend │ └── nginx.conf │ ├── docs/ │ ├── installation.md # for those who install and use VHS │ ├── development.md # for developers │ └── architecture.md, conventions.md, storage-decisions.md │ ├── scripts/ # backup.sh, restore.sh, compare_reports.py, check-dev-config.sh │ ├── vhs # commands for an installation ├── dev # commands for the development environment │ ├── docker-compose.yml ├── docker-compose.dev.yml ├── docker-compose.prod.yml ├── pyproject.toml ├── uv.lock └── README.md ``` ### Principles - a single Django app, `core`; - `engine` completely independent of Django; - `storage` separate from the domain; - `services` as business logic; - `tasks` as asynchronous adapters; - `api` as the public interface; - frontend completely separate from the backend. --- # 7. Data model ## Platform A platform is not a table: it is a class in `engine/platforms/`, listed in the registry of that package. The `platform` field of `Video` and `Channel` stores its key (`youtube`), which is also the platform's folder in the library. The field has no `choices`: the services check the key against the registry, so adding a platform needs no migration. --- ## Channel In the MVP the channel is the **source channel** of the downloaded videos: metadata used to show where a video comes from and to filter the Library. There are no subscriptions, no polling and no automatic downloads. Implemented fields (MVP): ```text name platform platform_id url ``` Fields planned for subscriptions (post-MVP): ```text thumbnail_url description auto_download check_interval last_checked_at min_duration_seconds max_duration_seconds include_shorts max_initial_downloads platform_metadata ``` `platform_id` identifies the channel on the source platform. The channel name can change on the platform and is updated, but it is not part of the identity. Personal collections stay independent of channels: VHS does not automatically create a collection per channel. When subscriptions are introduced (post-MVP), the auto-download rules will apply to the videos discovered through the channel. `max_initial_downloads` limits only the first population; later checks follow the normal policy. Missing or uncertain metadata must not silently turn into an irreversible decision: the video stays discovered and can be chosen manually. Categorization coming from the platform stays separate from the tags and collections assigned locally in VHS. Source playlists (post-MVP) have dedicated models, described in section 10. --- # 8. Video `Video` represents **the content that VHS knows about and archives**. It does not represent a download attempt. Common fields: ```text title description duration thumbnail_url source_url platform platform_id upload_date channel platform_metadata ``` Fields about the local copy and provenance: ```text file_path # archived copy playback_path # optional: derived browser copy thumbnail_path # optional: archived thumbnail (sidecar) file_size container video_codec audio_codec resolution audio_language # BCP 47 language of the archived audio track, "" if unknown audio_kind # original | default | dubbed | description, "" if unknown downloaded_at source_metadata_snapshot # metadata collected at acquisition acquired_at checksum_sha256 ``` The state of the local copy and the state of the source are independent: ```text local_status: absent | available | missing source_status: unknown | available | unavailable | deleted ``` For example, a video can stay `available` in the library even after being `deleted` from the platform. A temporary extraction error is not enough to mark the source as deleted. `queued` and `downloading` belong to the active `Download`, not to the persistent state of the `Video`. `local_status = available` means that the archived file exists, is complete and passes the minimum integrity checks. Playability in the browser is a separate property: it depends on container and codecs, or on the presence of `playback_path`. ### Paths Every archived video has **one main file**, whose path relative to the library is stored in `file_path`. Next to it are the thumbnail (`thumbnail_path`) and the `.info.json` sidecar, with the same basename. The browser copy, if any (`playback_path`), lives instead in a separate tree, `.browser/`, that mirrors the original's path (§18): the channel folder holds a single video file per content item. - The path is computed only once, when the file is archived, and from then on the database is the source of truth: no part of the application recomputes it from the title or the channel name. - A change of title or channel name on the platform updates the metadata; it does not move the file. A **new download** of an already registered video also reuses the basename of the stored path. - If a new download produces a different extension, the old path is always recorded in `superseded_files` (with checksum and date), whether the file exists or not: the previous file is never left orphaned without a trace. - Tags and collections are relations of the `Video` in the database: they neither move nor duplicate files. - The API and the UI identify the video by its ID; no response exposes paths. ### Identity The video must have a unique constraint: ```text UNIQUE(platform, platform_id) ``` This prevents duplicates from being created when the same video is found through different URLs. Example: ```text youtube.com/watch?v=ABC youtu.be/ABC www.youtube.com/watch?v=ABC ``` must identify the same `Video`. --- # 9. Download `Download` represents **an operation/attempt to obtain a local copy of a Video**. Relation: ```text Video 1 ──── N Download ``` Not: ```text Video 1 ──── 1 Download ``` because the same video can be downloaded several times over its lifetime. Fields: ```text video status progress speed eta error_message error_code ytdlp_version started_at completed_at last_heartbeat_at last_progress_at ``` States: ```text queued downloading processing completed failed cancelled ``` This separation makes it possible to keep the history of attempts. `error_code` is a stable category the UI can use, for example `authentication`, `source_unavailable`, `network`, `storage` or `processing`. `error_message` holds the diagnostic detail, stripped of sensitive URLs and credentials. `last_progress_at` makes it possible to tell an active transfer from a job that looks stuck; a manual retry creates a new attempt. A video whose local copy is available is never downloaded again: the protection is based on the state of the file (`local_status`), not on the existence of a completed attempt. A request for an archived video with no completed attempt gets `409 already_archived` with the video ID. If, at the start of an attempt, the stored file is present again with the same checksum (for example after the storage comes back), the download is closed as `completed` without a transfer (`recovered_existing`). ### A single active attempt per video Creating a download must be idempotent: two concurrent requests for the same video either return the attempt that is already active or create only one. A database constraint prevents more than one `Download` in the `queued`, `downloading` or `processing` state at the same time for the same `Video`; finished attempts stay in the history. Enqueueing happens after the required records are committed, and a reconciliation check recovers attempts left in `queued` if enqueueing fails. The task receives only `download_id`, checks the current state and can be run again without corrupting files or creating a second attempt. django-q2's timeout, `retry` and attempt limit must be explicit and consistent with the maximum job duration: a task redelivered while the first one is still working must not start a concurrent download. The application-level retry after an error creates a new `Download` and keeps the failed one. --- # 10. Discovery vs Download > **Post-MVP.** Discovery, channel and playlist polling and automatic downloads are part > of subscriptions, outside the MVP (§39). In the MVP videos enter VHS only through a URL > requested by the user. Discovering a video and downloading it are two different operations. Pipeline: ```text Channel polling │ ▼ Discovery │ ▼ Video │ ├──── auto_download = false ──► end │ ▼ Download │ ▼ Worker ``` In the future this makes it possible to: - see new videos without downloading them; - choose manually what to archive; - implement auto-download rules; - change the policy without modifying the discovery mechanism. ## Source playlists `SourcePlaylist` represents a playlist created on the platform, independent of the local `Collection`s created by the user in VHS: ```text platform platform_id url title channel # optional: may be unknown or not relevant check_interval last_checked_at last_successful_check_at platform_metadata UNIQUE(platform, platform_id) ``` `SourcePlaylistVideo` keeps the membership and order of the playlist: ```text source_playlist video position last_seen_at UNIQUE(source_playlist, video) ``` A playlist can contain videos from different channels; a video can appear in several playlists. `position` is updated after a complete, successful check; a temporary disappearance deletes neither the archived file nor the video's record. A `last_seen_at` earlier than the last successful check makes it possible to flag a membership that is no longer present in the source, while keeping a trace of it. With subscriptions, discovery will be able to add a playlist by URL and check its content periodically. Automatic download based on a playlist, with its own rules separate from the channel's, remains a later choice: it must not be inferred from playlist membership alone. --- # 11. Tag ```text name slug color ``` M2M relation with `Video`. A `Tag` is assigned locally in VHS: any platform tags stay in `platform_metadata` and in the acquired snapshot. They are not automatically imported as local tags. Implementation (Phase 2b): - case-insensitive unique name; `slug` generated from the name, with a numeric suffix on collision; `#RRGGBB` color chosen from a palette that is readable on the dark theme; - in the interface local tags are called **"Personal tags"**, to set them apart from the platform metadata; - platform tags are shown on the video page as "Source tags", read-only and visually distinct; they are never converted into personal tags; - deleting a tag removes only the associations: the videos stay; - a tag can also be assigned to videos that are not archived; the counts distinguish `video_count` (all) and `archived_count` (with a local copy). --- # 12. Collection ```text name description cover_image ``` The relation with videos uses a through table with: ```text collection video position ``` The through table is the `CollectionVideo` model, with the constraint `UNIQUE(collection, video)`. `position` describes the order chosen by the user, not the order of a `SourcePlaylist`. A video can belong to several collections without duplicating the file. Local `Collection`s are not overwritten when the platform's playlists change. Implementation (Phase 2b): - `cover_image` is implemented as **`cover_video`**: the cover is the local thumbnail of a video in the collection. By default it is the thumbnail of the first video in order that has one (even if not archived, because the thumbnail is saved during the download); another video of the collection can be chosen. This way no image uploads are needed, nor a new kind of media to protect; an uploaded cover remains a possible evolution. - `CollectionVideo` also has the constraint `UNIQUE(collection, position)`, deferred to the end of the transaction: two videos cannot have the same position, and reordering renumbers everything in a single transaction. - Additions, removals and reorders take the same lock on the collection (`select_for_update`): concurrent operations are serialized and do not assign the same position. A video is added at the end; reordering requires exactly the set of current videos. - A collection can contain videos that are not archived: its page shows them with a badge; `video_count` and `archived_count` have the same definition as for tags. - Deleting a collection does not delete the videos. - Collections are independent of channels and source playlists: VHS does not create any automatically, neither per channel nor per playlist. - The folder layout does not represent collections: a video can belong to several collections and remains a single file, at the path decided at archiving time (§17). --- # 13. PlaybackProgress The playback position is application data, separate from the video file. Model: ```text PlaybackProgress user video position_seconds duration updated_at ``` Constraint: ```text UNIQUE(user, video) ``` The frontend saves the position periodically. Example: ```text PUT /api/v1/videos/123/progress { "position_seconds": 1815, "duration": 3600 } ``` On reopening: ```text GET /api/v1/videos/123/progress ``` The player sets: ```javascript video.currentTime = position ``` Playback persistence is therefore independent of the streaming system. Implemented behavior: - the position is saved every 10 seconds during playback and immediately on pause, after a seek, at the end of the video and when the page goes to the background (`fetch` with `keepalive`, which, unlike `sendBeacon`, allows the CSRF header); - on reopening, the player resumes from the saved position, except in the first 5 seconds and the last 10, where it starts over from the beginning; - the server clamps the position between 0 and the duration; a `GET` with no saved data returns `position_seconds: 0`. --- # 14. CookieConfig > **Status: deferred.** VHS currently downloads without platform cookies, and anonymous > downloads work. This section describes the design to adopt when a source requires login > or passes anti-bot checks only with cookies; no model, endpoint or page is implemented > yet. Meanwhile, errors of this kind are classified as `authentication` (§9) and shown on > the Downloads page. When it is introduced: - manual upload of `cookies.txt`; - association with the platform; - use by yt-dlp. Cookies must be treated as **credentials**, not as plain metadata. Their content must not be handled as normal application data. Goal: ```text CookieConfig ↓ encrypted at rest ``` The encryption key must be kept separate from the database. Conceptual fields: ```text platform encrypted_cookies last_updated_at last_validated_at validation_status # unknown | valid | invalid | error last_validation_error_code ``` The UI shows the status and the date of the last check, telling expired or invalid cookies apart from temporary source errors. For example: ```text Cookies valid ✓ Cookies expired ✗ Check failed — try again ``` Validity is checked periodically through a suitable metadata extraction operation. Only an administrator can upload or replace the cookies. The worker decrypts them only when they are needed and makes them available to yt-dlp in a temporary file with restrictive permissions, outside the library and the logs; the file is removed even in case of error. Key rotation, cookie deletion and the behavior when the key is missing are defined. Database encryption alone does not protect the temporary plaintext copy, if any. ### Alternative to evaluate: `cookiesfrombrowser` yt-dlp can read cookies directly from a browser profile (`cookiesfrombrowser`), without an upload. It is more convenient, but: - it works only where the worker has access to that profile, which is never the case in containers (neither in development nor in production); - it gives yt-dlp access to all the profile's cookies, not only the platform's, and falls outside the credential handling described above. The choice between the two solutions will be made when the feature becomes necessary. --- # 15. Storage and filesystem ## Media root The library is a **host folder**, set by the mandatory `VHS_HOST_LIBRARY` and mounted in the containers as `/srv/video-library` (`VHS_MEDIA_ROOT`, fixed in the containers). This applies to the installation (for example `/srv/vhs/library`) and to development (`./video-library`), which also runs in containers (§34). NFS, SMB, NAS and separately mounted disks **are not supported in the MVP**: a dedicated mode was implemented and then removed, and it remains post-MVP (storage-decisions.md, §5 and §6). The paths stored in the database (`file_path`, `playback_path`, `thumbnail_path`) are relative to `VHS_MEDIA_ROOT`: the library can be moved by changing only `VHS_HOST_LIBRARY`. ## Library availability VHS never writes where it does not find the library. The case to avoid is an empty folder in place of the library (folder renamed or moved, wrong `VHS_HOST_LIBRARY`, folder replaced): VHS would write work dirs, sidecars and video files into it. Reasoning, tests and limits are in [storage-decisions.md](storage-decisions.md). - **`.vhs-library` marker** in the root of the library: the storage writes only where it finds it (`work_dir`, `finalize`, `delete`) and never creates the root, except for a new library (no archived video in the DB and the folder missing or empty), which is initialized automatically on the first download. - **States:** `ok`, `new` (created on the first download), `not_initialized` (non-empty folder without a marker: `init_library` is needed once), `missing` (a populated library has disappeared). - With the library unavailable: downloads and preparations fail with `error_code=storage` without writing; stream and thumbnails respond 503; `/health` responds `degraded`; the `vhs.W002` system check reports it at startup; the UI shows a warning. - `manage.py storage_status` shows folder, marker and state; `manage.py init_library` explicitly initializes an existing library. - **Limits:** the check narrows but does not close the window between check and write; a transfer already in progress can leave temporary files where the library was, but they enter neither the library nor the database (storage-decisions.md, §4). --- # 16. Safe download to storage A download must not make an incomplete file visible right away. The principle is: ```text download │ ▼ temporary/incomplete file │ ▼ processing │ ▼ atomic finalize │ ▼ final filename │ ▼ Video.local_status = available ``` Example: ```text .incomplete/ abc123.part ``` then: ```text youtube/ Channel/ 2024-03-15 - Title [abc123].mp4 ``` The `Video` is considered available only once the file and the required operations are complete. The temporary directory for the final file is on the **same filesystem** as the destination. A rename across different filesystems turns into a copy and does not provide the intended atomic finalization. Renaming the file does not also make the database update atomic, nor the publication of several sidecars: the service checks all the required outputs, promotes the file, then sets `local_status = available`. The API exposes only files registered as available. At startup and periodically, a reconciliation process compares jobs, temporary files and final files. When the library is unavailable (§15), reconciliation is skipped: it does not mark attempts as interrupted, register files or clean up work dirs until the library comes back. It resumes interrupted jobs or marks them as failed, recognizes files that are already finalized but not registered, and never overwrites a complete copy because of a retry. This is especially important in case of: - a worker crash; - a container restart; - a temporarily unavailable library; - an ffmpeg error; - insufficient space. --- # 17. File naming Two built-in templates. ## Standard ```text video-library/ youtube/ Channel Name/ 2024-03-15 - Video title [dQw4w9WgXcQ].mp4 2024-03-15 - Video title [dQw4w9WgXcQ].info.json 2024-03-15 - Video title [dQw4w9WgXcQ].webp ``` ## TV Show ```text video-library/ youtube/ Channel Name/ Season 2024/ s2024.e0315 - Video title [dQw4w9WgXcQ].mp4 s2024.e0315 - Video title [dQw4w9WgXcQ].info.json s2024.e0315 - Video title [dQw4w9WgXcQ].nfo s2024.e0315 - Video title [dQw4w9WgXcQ].webp ``` ### Rules - the platform ID is always present; - sidecars share the same basename; - invalid characters are sanitized; - titles are truncated to a reasonable length; - the channel name is readable; - custom templates are post-v1; - the template is applied at archive time: changing the template, or the title and the channel name on the platform, does not move files that are already archived (the registered path stays valid); - folders represent only provenance (source platform and channel), never personal tags or collections; - browser copies are not stored in the channel folders but in `.browser/` (§18), so each folder contains only one video file per item. The readable basename can change, but the stable identity remains `(platform, platform_id)`. For preservation, VHS keeps the acquisition date, the original URL, a snapshot of the metadata and the SHA-256 checksum of the archived copy. A periodic check reports missing files or mismatched checksums without automatically deleting the video's record. `.info.json` files can contain personal data: they stay protected like the other media and are not published by default. The `.nfo` file in the TV Show example is **not generated in the MVP**: the Jellyfin integration is a dedicated phase, outside the MVP and independent of subscriptions (§39). The TV Show template already exists as a folder layout. In that phase the preset will be able to generate a basic `.nfo` using the fields already present in `Video` (title, description, channel, date and source ID), without a Django model for the sidecar; the layout and the files must be verified with Jellyfin. Plex compatibility requires tests and specific rules; the presence of an NFO does not guarantee it. The current choices keep that integration possible without implementing it ahead of time: each video has a main file with a registered path (and browser copies live outside the channel folders), identity does not depend on the path, metadata (`source_metadata_snapshot`, `.info.json`) and thumbnails remain available for an export, and folders do not try to represent collections. --- # 18. Download engine Abstract class: ```python class BaseDownloader: def extract_info(self, url): ... def download(self, url, output_path, options): ... def get_progress(self): ... ``` Initial implementation: ```text YTDLPDownloader ``` which uses the yt-dlp Python API. The engine does not know about the Django models. ### Format policy for v1 When the source offers one, the engine prefers a format that plays in HTML5 in the target browsers, for example MP4 with H.264 video and AAC audio. The downloaded copy always remains the main archive and is never modified. After the download is registered, `ffprobe` analyzes the **archived file**: the `.mp4` extension or the source metadata alone do not guarantee playback. The browser target is: MP4 container, 8-bit H.264 video (`yuv420p`), AAC or MP3 audio, or no audio. ```text ffprobe on the archived file │ ├── native → plays the original ├── remux → automatic derived copy (container change only) ├── transcode → "Prepare for playback", on request only └── unsupported → no video stream ``` - **Remux**: the codecs are already compatible but in another container (for example H.264/AAC in MKV). The derived copy is created automatically with `ffmpeg -c copy`: it is fast and loses no quality. - **Transcode**: video, audio or both must be re-encoded (VP9, AV1, HEVC, 10-bit H.264, Opus, Vorbis...). The conversion costs CPU, so it starts only when the user requests it from the video page; it re-encodes only the streams that need it (`libx264`/`aac`, configurable preset and quality). - The derived copy lives in a separate tree that mirrors the original's path, `.browser/.mp4`, and is registered in `playback_path`. This way the channel folder contains only one video file per item, and a media server, which transcodes on the fly on its own, does not need the derived copies. How to exclude `.browser/` from the scan must be validated in the phase dedicated to Jellyfin. Before registering it, `ffprobe` checks that it is actually playable. - Each preparation is a `PlaybackPreparation` (only one active per video, with history, heartbeat and reconciliation like downloads). A preparation or analysis error **does not invalidate** the archived video: it stays `available`, and the UI shows the reason and lets the user retry. - The UI shows the state of the browser copy: **Ready** (original or derived copy), **Preparing** (analysis, remux or conversion, with a percentage) or **Unavailable** (with the reason and, if a conversion is needed, the button). - Videos archived before analysis existed are analyzed by the periodic reconciliation or with `manage.py analyze_media`. ### Audio track Some videos have several audio tracks: YouTube's original language and its dubs, RaiPlay's Italian track, original version and audio description. VHS downloads **one** track: the one the platform plays by default, never an audio description (`format_selector()` in `engine/downloader/ytdlp.py`). - On YouTube yt-dlp marks the original track and sorts it first. - A platform that marks no track declares the language of its default one (`Platform.default_audio_language`; RaiPlay: Italian), and VHS asks for it first. - The archived track is recorded on the video: `audio_language` (BCP 47) and `audio_kind` (`original`, `default`, `dubbed`, `description`, or empty when unknown), from the format yt-dlp downloaded (`Platform.audio_track()`). Videos archived earlier have them empty; their `.info.json` holds what is needed to fill them. Planned: choosing the language per video (and per channel with subscriptions), adding other languages as audio-only files next to the video, and switching track during playback. Recording the track now means none of this will need a new download. --- # 19. Metadata extraction Metadata mapping is kept separate from the downloader. ```text Platform (engine/platforms/base.py) │ ├── YouTube └── RaiPlay ``` The downloader produces raw data; the platform decides how to map it. `Platform.map()` reads yt-dlp's common fields, the same for every extractor; a platform overrides only what differs: the channel, short videos, platform metadata, its audio tracks and its own errors. - **YouTube:** the channel is the uploader's channel; Shorts are recognized. - **RaiPlay:** the channel is the **series** ("Report", "Blanca"), falling back to the network when there is none. The **network** ("Rai 3") is kept in `platform_metadata` and shown on the video page: RaiPlay lists episodes of the same series under different networks (reruns, extras), so it would split a series. A 404 on the video's description means an expired video, not a network error. This makes it possible to add platforms without changing the downloader core. --- # 20. Download progress The yt-dlp callback produces, for each stream: ```text downloaded bytes total bytes speed eta ``` With separate video and audio formats yt-dlp downloads the streams one after the other and reports each one on its own, starting again from zero. The downloader sums them (`StreamProgress`), so the progress of the whole download never goes back. The total is the larger of the size estimated before starting (every selected format) and the reported one (only the streams started so far). The percentage stays below 100% until the file is archived. yt-dlp's `eta` covers only the current stream. The ETA VHS shows is an **estimate** for the whole download: the bytes still missing from the estimated total, divided by the current speed. It changes with the speed and is only as accurate as the size estimate; without an estimate made before starting it is unknown. When merging starts the download switches to `processing`, so a long merge is not taken for a stall. At completion the download records the size of the archived file. Progress does not have to be written to the database on every callback. The service/task updates the state with reasonable throttling. It updates `last_progress_at` when it receives actual progress and uses `last_heartbeat_at` to signal that the worker is still alive, even during extraction or processing. The UI shows the error category and detail and offers a manual retry for failed attempts. The v1 frontend uses polling: ```text GET /api/v1/downloads/{id} ``` In the future: ```text SSE ``` or possibly: ```text WebSocket ``` --- # 21. Video streaming Video files are **not streamed through Django**. Django handles: - authentication; - authorization; - locating the file; - deciding whether the video is playable. Nginx handles: - transferring the bytes; - HTTP Range Requests; - seeking; - long-lived connections; - serving the file. --- # 22. Protected media with Nginx Library files must not be exposed directly as a public directory. The intended pattern is: ```text Browser │ │ GET /api/v1/videos/123/stream ▼ Django │ ├── checks authorization │ └── X-Accel-Redirect │ ▼ Nginx │ ▼ /srv/video-library/... ``` The Nginx location that contains the media will be `internal`. So the client knows: ```text /api/v1/videos/123/stream ``` and not: ```text /srv/video-library/youtube/... ``` This keeps the filesystem an internal detail of the application. ### Thumbnails The archived thumbnail uses the same mechanism. The other sidecars, for example `.info.json`, are not exposed by default (§17). ```text GET /api/v1/videos/123/thumbnail │ ▼ Django ── authorization ── X-Accel-Redirect │ ▼ Nginx ── /media-internal/.../Title [id].webp ``` The UI uses the local copy when the video is archived: the thumbnail stays visible even after the video is removed from the platform, and browsing the library makes no requests to external services. `thumbnail_url` remains source metadata; it can be used as a fallback only for videos not yet archived, or replaced by a placeholder. The thumbnail is published in the library as soon as yt-dlp downloads and converts it, before the video transfer: the UI already shows it during the download, and also if the download fails. While the download is queued and the worker has not picked it up yet, the UI shows the placeholder. Django does not transfer media bytes even in development: streams and thumbnails always go through `X-Accel-Redirect`, and to work on the player you start the development Nginx proxy described in §34. The internal URI is made only of the fixed prefix `VHS_MEDIA_ACCEL_PREFIX` (`/media-internal/`) and the relative path registered on the `Video`, validated by the storage: absolute paths, `..`, the `.incomplete/` work area and symbolic links (on the file or on an intermediate directory) are rejected. Nginx also applies `disable_symlinks on` and rejects `/media-internal/.incomplete/`. --- # 23. Seeking The player uses HTML5 `