# Internationalization (i18n) The English lessons are canonical. Translated **lesson** markdown is machine-generated and **never committed to `main`** — it lives on a separate `translations` branch, and the site fetches it at runtime the same free way it fetches English (`raw.githubusercontent.com`). No Vercel functions, no KV, no per-view cost. The one exception is the per-language **README**, which is hand-authored for quality and committed to `main` (see below). This pipeline covers lessons under `phases/` only. Certification curricula remain English-only and are intentionally excluded from both machine translation and book generation. The default engine is **NLLB-200** ([No Language Left Behind, Meta AI, 2022](https://arxiv.org/abs/2207.04672)), an open translation model covering 200 languages, which runs **inside the GitHub Actions runner with no API key and no bill**. Compute is the free CI runner; there is no translation-API cost at all. Paid LLM providers (Anthropic/OpenAI) and DeepL remain available as optional higher-quality upgrades via `--provider`. ## Pieces The `languages.json` registry drives the machine-translation surfaces — lessons, the site switcher, and the book matrix. The per-language **README is separate**: its language set and translated blocks live in `scripts/readme_translations.py`, hand-authored for quality and independent of the registry. | Piece | Where | Committed to main? | |-------|-------|--------------------| | Language registry | `languages.json` | Yes (one small file) | | Lesson translation script | `scripts/translate_lessons.py` (prose-only walker) | Yes | | README generator + translations | `scripts/build_readme_i18n.py` + `scripts/readme_translations.py` (hand-authored) | Yes | | CI workflow | `.github/workflows/translate.yml` (sharded per language × phase) | Yes | | Translated lesson markdown | `translations` branch, `i18n//…/.md` | No (separate branch) | | Translated README | `i18n//README.md`, linked from the header of `README.md` | **Yes** | | Translated books | `scripts/build_book.py --lang

` reads the translated markdown and emits `aiefs-vol{n}-{slug}-.epub/.pdf` as release assets (English fallback per lesson) | No (release assets) |
| Switcher options | `site/langs.js` (generated by `build.js` from the `source`/`ci:true` set) | No (gitignored, like `data.js`) |
| Site switcher | `site/lesson.html` | Yes (the UI + fetch logic) |

## Add a language

Add one entry to `languages.json`, the single source of truth. Fields:

- `code` — the tag used in filenames, URLs, and the switcher value.
- `name` — English name, used in the LLM-provider prompt.
- `native` — label shown in the switcher.
- `nllb` — the FLORES-200 code (e.g. `zho_Hans`, `tur_Latn`); **required** for the default NLLB provider — `translate_lessons.py` exits if a requested language has none.
- `ci` — `true` opts the language into the auto push-triggered workflow's build matrix. `workflow_dispatch` can request any registered language regardless.

Both matrices read from `ci: true`, so the same flag opts a language into lesson translation **and** into the site switcher — `build.js` only emits `source`/`ci:true` languages into `site/langs.js`, so the switcher never offers a language the site can't serve. The README bar is independent: it can list any language `readme_translations.py` covers. Nothing else to touch.

## How it runs

1. A push to `main` that changes `phases/**/docs/en.md`, `languages.json`, or the translation workflow itself triggers `.github/workflows/translate.yml`. Certification lesson changes do not trigger it.
2. The matrix is **one job per (language, phase)**. A full 503-lesson language run is ~27h on a CPU runner, far past the 6-hour job limit, so a per-language job always timed out before publishing. Splitting by phase keeps the largest job (phase 19, 85 lessons) near ~4.5h, well under the limit. Each job restores its language's prior output from the branch, runs `translate_lessons.py --phase `, and pushes only its own `i18n//phases//` slice back with a fetch-and-retry loop, so disjoint slices merge without collisions.
3. The site's language switcher sets `?lang=`; `fetchLesson` pulls `…/translations/i18n//…/.md` and **falls back to canonical English** if that translation isn't published yet. English (`lang=en`) is byte-identical to the pre-i18n fetch path.

## Never wastes a run

Every lesson is keyed by the `sha256` of its English source in a per-(language, phase) cache (`i18n//.cache/.json`), written per lesson in the runner and **published to the `translations` branch when the job finishes**. So:

- The first run translates everything once. Every completed run after it re-translates **only** lessons whose English actually changed — everything else is a cache hit and is skipped instantly.
- A cancelled or interrupted phase job does not publish, so it loses that job's runner-local progress and the next run resumes that phase from the **last published** cache. Because each shard is a single phase, it comfortably finishes and publishes within the job limit.

## Per-language README

The README is the landing page, so its translations are **hand-authored** (highest quality) rather than machine-generated, and they live in `main` at `i18n//README.md`. `scripts/readme_translations.py` holds the translated strings keyed by the exact English block; `scripts/build_readme_i18n.py` rebuilds each file by replacing **only** the translated line-spans in a copy of the English README. Everything else — the banner, badges, the 584-row lesson table, all code and link *text* — is preserved verbatim (asserted by a round-trip identity check on every run). The one deliberate transform is on link and image *targets*: repo-root-relative paths get a `../../` prefix so they still resolve from two directories deep (fenced code is skipped, so code that looks like a link is left alone). Any block without a translation falls back to English, so coverage can grow language by language.

`python3 scripts/build_readme_i18n.py --check` runs in CI (`curriculum.yml`) and fails if any committed README is stale, and the README-counts bot regenerates them whenever it syncs the English stats block.

## What is protected during translation

NLLB is a sentence translator, not an instruction-following model, so the walker never lets it see anything but prose. It processes the lesson line by line: fenced code blocks, tables, image lines, and the metadata header (`**Type:** … **Time:**`) pass through verbatim; on every other line it protects inline `` `code` ``, `$math$`, image refs, full markdown links, `**bold**` terms, and URLs behind placeholders, translates only the plain-text fragments between them, then restores. Bold spans are kept because in this course they are almost always technical terms (Skip-gram, negative sampling) that should stay English.

This is verified mechanically: running the walker with an identity "translation" reproduces all **503 lessons byte-for-byte**, so a real run can only ever change prose. If a placeholder fails to round-trip, that lesson keeps its English (fail-safe).

## Cost

Default provider `nllb`: **free.** NLLB-200 runs in the GitHub Actions runner — no API key, no secret, no per-token bill. The only resource is free CI compute; the model is cached between runs and only changed lessons are re-translated (sha256 cache). Vercel cost does not change.

Optional upgrades (`--provider anthropic|openai|deepl`) give higher fluency at low-tens-of-dollars per full pass per language, if you later want to polish a high-traffic language. Those providers read the key from the `LLM_API_KEY` environment variable (DeepL uses the same variable for its auth key). NLLB stays the zero-cost default.

## Quality sample

Below is `05 · 03 Word Embeddings` translated through Step 1, showing that prose reads naturally while code, math, `king - man + woman`, technical terms, and the metadata header survive untouched. This sample is LLM-tier fluency (what `--provider anthropic` produces); the free NLLB-200 default is solid and fully faithful on the preservation guarantee, with slightly plainer phrasing. Upgrade a high-traffic language to an LLM provider later if you want this level of polish.

### 简体中文 (zh)

> # 词嵌入 —— 从零实现 Word2Vec
>
> > 词以其所处的语境为意。用一个浅层网络训练这个想法,几何结构便会自然浮现。
>
> **Type:** Build · **Languages:** Python · **Prerequisites:** 阶段 5 · 02(BoW + TF-IDF)、阶段 3 · 03(从零实现反向传播) · **Time:** 约 75 分钟
>
> **问题** —— TF-IDF 知道 `dog` 和 `puppy` 是不同的词,但不知道二者含义几乎相同。在 `dog` 上训练的分类器无法泛化到讨论 `puppy` 的评论。你可以靠列举同义词勉强弥补,但这在罕见词、领域术语,以及所有你未曾预料的语言上都会失效。
>
> 你想要的是这样一种表示:`dog` 和 `puppy` 在空间中彼此靠近;`king - man + woman` 落在 `queen` 附近;在 `dog` 上训练的模型能免费地把部分信号迁移到 `puppy`。
>
> **核心概念** —— **分布假说**(Firth,1957):"要了解一个词,就看它与哪些词为伴。" Word2Vec 有两种形式:**Skip-gram** 给定中心词预测周围词(窗口为 2 时 `cat -> (the, sat, on)`);**CBOW(连续词袋)** 给定周围词预测中心词。网络只有一个隐藏层且无非线性:输入是词表上的 one-hot 向量,输出是 softmax,训练后丢弃输出层,隐藏层权重即为嵌入。诀窍是用**负采样(negative sampling)**把对 10 万词的 softmax 转成二分类。

### Türkçe (tr)

> # Kelime Gömmeleri —— Word2Vec'i Sıfırdan
>
> > Bir kelime, birlikte bulunduğu kelimelerden ibarettir. Bu fikri sığ bir ağ ile eğitin, geometri kendiliğinden ortaya çıkar.
>
> **Type:** Build · **Languages:** Python · **Prerequisites:** Aşama 5 · 02 (BoW + TF-IDF), Aşama 3 · 03 (Sıfırdan Geri Yayılım) · **Time:** ~75 dakika
>
> **Problem** —— TF-IDF, `dog` ve `puppy`'nin farklı kelimeler olduğunu bilir ama neredeyse aynı anlama geldiklerini bilmez. `dog` üzerinde eğitilen bir sınıflandırıcı, `puppy` hakkındaki bir yoruma genelleme yapamaz. Eş anlamlıları listeleyerek geçiştirebilirsiniz, ama bu; nadir terimlerde, alana özgü jargonda ve öngöremediğiniz her dilde başarısız olur.
>
> `dog` ve `puppy`'nin uzayda birbirine yakın düştüğü, `king - man + woman`'ın `queen`'e yakın indiği bir temsil istiyorsunuz. Word2Vec bunu verdi: iki katmanlı bir ağ, trilyon token'lık eğitim, 2013'te yayımlandı.
>
> **Kavram** —— **Dağılımsal hipotez** (Firth, 1957): "Bir kelimeyi birlikte bulunduğu kelimelerden tanırsın." İki tür: **Skip-gram** merkez kelimeden çevredekileri tahmin eder (pencere 2 ile `cat -> (the, sat, on)`); **CBOW** çevredekilerden merkezi tahmin eder. Ağın doğrusal olmayan aktivasyonu olmayan tek bir gizli katmanı vardır: girdi one-hot vektör, çıktı softmax'tır; eğitimden sonra çıktı katmanı atılır, gizli katman ağırlıkları gömmelerdir. Püf nokta, 100 bin kelimelik softmax'ı **negatif örnekleme (negative sampling)** ile ikili sınıflandırmaya çevirmektir.

Note how `dog`, `puppy`, `king - man + woman`, `softmax`, `one-hot`, `Skip-gram`, `CBOW`, `TF-IDF`, `Word2Vec`, and the `**Type:** / **Time:**` labels stay verbatim in both languages while the surrounding prose is fully localized. That preservation is enforced mechanically by the placeholder round-trip, not left to the model.