# Methodology

This benchmark evaluates Korean source-side multi-turn context translation. Each sample contains one current Korean utterance and one to three prior context turns. Participant systems translate only the current utterance while receiving either the full context prompt or a context-blind prompt, depending on the participant configuration.

## Benchmark Scope

The benchmark focuses on context-sensitive translation phenomena that are common in Korean conversation:

- omitted subjects and objects that require referent recovery,
- ellipsis completion,
- register and politeness carryover,
- pragmatic intent resolution,
- addressivity and speaker relationship cues,
- false leads and stale context that should be ignored.

The benchmark does not claim to measure general translation quality across domains. It is designed for Korean conversational context handling.

## Primary Metric

The primary score is raw mean penalty from the GEMBA-MQM-based evaluation. Lower is better.

Severity weights are:

- minor: 1
- major: 5
- critical: 25

For a participant, `mean_penalty` is the mean total penalty over valid judged cells.

## Main Experiments

All three use `gemba-mqm-context-v1`: 216 Korean items × English, Japanese, and Simplified Chinese = 648 expected cells per system or condition.

| Experiment | Question | Design | Results |
| --- | --- | --- | --- |
| 1 — Integrated comparison | Which translation systems perform best? | 12 text-input systems + 3 direct speech translators | [15-system comparison](../experiments/2026-09-integrated-15-system/) |
| 2 — Full-stack translation | Which speech-input pipeline produces the best translation? | 6 ASR→LLM pipelines + the same 3 direct speech translators | [9-pipeline comparison](../experiments/2026-09-fullstack/) |
| 3 — Context effect | How much do history and policy help the same model? | Gemma 4 E4B QAT Q4: sentence-only vs. policy + full history | [642 paired cells](../experiments/2026-08-gemini35-live-10p-highjudge/ablation/) |

### Inputs and Comparisons

- Experiment 1 labels text and speech inputs; its 12-system text-only view excludes recognition effects.
- Experiment 2 includes ASR errors in the final translation score. Judges compare against canonical Korean text and context, not the recognized transcript. Gemini Transcribe and Gemini Live Translate are distinct systems.
- Soniox→LLM uses pure-STT transcripts. Direct speech results are shared between Experiments 1 and 2, not independent repetitions.
- Experiment 3 changes both policy and history. It measures their combined effect, with `use` and `ignore` slices reported separately.

### Reporting

- All-valid means retain each system's coverage; failed or missing judgments are excluded, never scored as zero.
- Common-cell rankings use the intersection of successful source-item/target-language keys within each comparison. Experiment 2 has 618 common cells; this is distinct from Experiment 1's intersection.
- Context effects use the 642 cells successfully judged in both conditions, with paired improvement, regression, and tie counts.
- Reports include language, context expectation, and context-length slices where available.

### Timing

Text translation times span request to full response. For ASR→LLM, ASR and translation were collected separately; their sum is not measured integrated end-to-end latency. Direct speech session times include realtime-paced context/current audio. These timing boundaries are reported separately.

## Historical Experiments

The earlier publications use the same dataset (fingerprint `9ab9e98752155a83cda100fc121f1b952474c82c1a20607887d9eb774855d110`). Their original configurations are preserved below.

| | 2026-04 (archived) | 2026-08 issue-1 (archived) | 2026-08 high judge |
| --- | --- | --- | --- |
| Run ID | `gemba-mqm-context-v1-gemini-context-v2-expanded-nodeepl-api-20260429-011514` | `issue1-milmmt-e4b-papago-deepseek-0731-integrated-20260815-01` | `unified-12arm-highjudge-20260822` |
| Translation prompt | `gemini-context-v2.md` | `puripuly-translation-latest.md` (MiLMMT arms use their own) | Live: no prompt (audio only, provenance marker file); Hy-MT2 7B and Gemma 4 12B QAT Q4 fresh; other rows carried over |
| Judge model | `gemini-3.1-pro-preview` | `google/gemini-3.7-flash:batch` | `google/gemini-3.7-flash:batch`, reasoning effort high |
| Judge backend | Vertex AI | OpenRouter Batch API | OpenRouter Batch API |
| Judge prompt format | JSON structured output | Upstream GEMBA-MQM text annotation | Upstream GEMBA-MQM text annotation |
| Participants | Gemini/Gemma/Qwen/DeepSeek + DeepL + Google | Gemma 31B/26B, DeepSeek 0731, local E4B ×3, MiLMMT ×2, Papago, DeepL, Google | Gemini 3.5 Live Translate (audio-native two voice), Gemma 31B/26B/12B QAT Q4, DeepSeek 0731, local E4B ×2, Hy-MT2 7B, MiLMMT X0, Papago, DeepL, Google (10 carried arms reused) |
| Extra arms | no-context baselines | quantization and prompt-regime arms | TTS two-voice audio pipeline, ASR CER analysis |
| Reports | CSV under `experiments/2026-04-…/reports/` | JSON under `experiments/2026-08-…-archived/reports/` | JSON + JSONL raw data under `experiments/2026-08-gemini35-live-10p-highjudge/` |

Because the prompt, judge, and judge prompt format differ, scores from different experiments are not directly comparable. Compare systems only within one experiment.

The 2026-08 issue-1 run was a fork-and-merge continuation: 5,828 translations were reused from an earlier run of the same benchmark config, 1,296 were imported (Papago, DeepSeek 0731), and only missing cells were judged fresh. See `experiments/2026-08-…-archived/fork-prepared.json`.

The 2026-08 high-judge lineage forks from the live-two-voice run (`gemini35-live-two-voice-20260817`): all 6,468 carried translations (10 participants, including the 640 Gemini 3.5 Live Translate cells) were reused and judged with the high-effort judge in `gemini35-live-10p-highjudge-20260820`. The unified collect `unified-12arm-highjudge-20260822` re-publishes those 10 arms byte-identically (same 5 batch jobs) and adds two fresh arms — Hy-MT2 7B and Gemma 4 12B QAT Q4 — translated and judged in their own high-judge sessions. The Live participant streams locally TTS-synthesized two-voice audio (`Qwen3-TTS-12Hz-0.6B-CustomVoice`, voices `sohee`/`uncle_fu`) into `gemini-3.5-live-translate-preview`; its streaming transcription is recorded in `translation-provider-details.jsonl` and analyzed in `reports/ASR-CER-analysis-high.md`. See `experiments/2026-08-gemini35-live-10p-highjudge/README.md`.