{ "cells": [ { "cell_type": "code", "execution_count": 5, "metadata": { "pycharm": { "name": "#%%\n" }, "slideshow": { "slide_type": "skip" } }, "outputs": [ { "data": { "text/html": [ "\n", "
\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "%%html\n", "\n", "
\n", "" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "pycharm": { "name": "#%%\n" }, "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "%%capture\n", "import sys\n", "sys.path.append(\"..\")\n", "import statnlpbook.util as util\n", "import matplotlib\n", "matplotlib.rcParams['figure.figsize'] = (10.0, 6.0)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "\n", "$$\n", "\\newcommand{\\Xs}{\\mathcal{X}}\n", "\\newcommand{\\Ys}{\\mathcal{Y}}\n", "\\newcommand{\\y}{\\mathbf{y}}\n", "\\newcommand{\\balpha}{\\boldsymbol{\\alpha}}\n", "\\newcommand{\\bbeta}{\\boldsymbol{\\beta}}\n", "\\newcommand{\\aligns}{\\mathbf{a}}\n", "\\newcommand{\\align}{a}\n", "\\newcommand{\\source}{\\mathbf{s}}\n", "\\newcommand{\\target}{\\mathbf{t}}\n", "\\newcommand{\\ssource}{s}\n", "\\newcommand{\\starget}{t}\n", "\\newcommand{\\repr}{\\mathbf{f}}\n", "\\newcommand{\\repry}{\\mathbf{g}}\n", "\\newcommand{\\x}{\\mathbf{x}}\n", "\\newcommand{\\prob}{p}\n", "\\newcommand{\\bar}{\\,|\\,}\n", "\\newcommand{\\vocab}{V}\n", "\\newcommand{\\params}{\\boldsymbol{\\theta}}\n", "\\newcommand{\\param}{\\theta}\n", "\\DeclareMathOperator{\\perplexity}{PP}\n", "\\DeclareMathOperator{\\argmax}{argmax}\n", "\\DeclareMathOperator{\\argmin}{argmin}\n", "\\newcommand{\\train}{\\mathcal{D}}\n", "\\newcommand{\\counts}[2]{\\#_{#1}(#2) }\n", "\\newcommand{\\length}[1]{\\text{length}(#1) }\n", "\\newcommand{\\indi}{\\mathbb{I}}\n", "$$" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "pycharm": { "name": "#%%\n" }, "slideshow": { "slide_type": "skip" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The tikzmagic extension is already loaded. To reload it, use:\n", " %reload_ext tikzmagic\n" ] } ], "source": [ "%load_ext tikzmagic" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Please complete the course evaluation\n", "\n", "### [evaluering.ku.dk](https://evaluering.ku.dk)\n", "\n", "Deadline: **November 2**" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# What languages do you speak?\n", "\n", "# [tinyurl.com/diku-nlp-lang](https://tinyurl.com/diku-nlp-lang)\n", "\n", "([Responses](https://www.mentimeter.com/app/presentation/389360b38fa508b4ffd4e40bf47003e4/edit?question=f43e4a2212e8))" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Machine Translation" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "+ Challenges\n", "+ History\n", "+ Statistical MT\n", "+ Neural MT\n", "+ Evaluation\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Languages are hard (even for humans)!\n", "\n", "
\n", " \n", " \n", " (Source: Flickr)\n", "
\n", "\n", "[随便](https://translate.google.com/#view=home&op=translate&sl=zh-CN&tl=en&text=%E9%9A%8F%E4%BE%BF)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Automatic machine translation is hard!\n", "\n", "
\n", "\n", "
\n", "\n", "[J'ai besoin d'un avocat pour mon procés de guacamole.](https://translate.google.com/?sl=fr&tl=en&text=J%27ai%20besoin%20d%27un%20avocat%20pour%20mon%20proc%C3%A9s%20de%20guacamole.&op=translate)\n", "\n", "[guacamole lawsuit](https://www.latimes.com/archives/la-xpm-2006-dec-10-fi-letters10.2-story.html)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Many things could go wrong." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "
\n", "\n", "# [tinyurl.com/diku-nlp-mt](https://tinyurl.com/diku-nlp-mt)\n", "\n", "([Responses](https://docs.google.com/forms/d/10UUpQqlduvYuz7-SXSvzMRDAs5qxdliKJmgnkWiHNK8/edit#responses))\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Challenges\n", "\n", "Divergences between languages include:\n", "\n", "* Word order\n", "* Grammatical (morphological) marking (e.g., gender)\n", "* Division of concept space (e.g., English \"wall\" vs. German \"Mauer\"/\"Wand\")\n", "\n", "Addressing them requires resolving ambiguities:\n", "\n", "* Word sense (e.g., \"bass\")\n", "* Attributes with grammatical marking (e.g., formality in Japanese)\n", "* Reference in pro-drop contexts (e.g., in Mandarin Chinese)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## History\n", "\n", "
\n", " \n", "
\n", "(Source: freeCodeCamp)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Direct Machine Translation\n", "\n", "Just use a **dictionary** and translate word-by-word.\n", "\n", "
\n", " \n", "
\n", "(Source: freeCodeCamp)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Transfer-based Machine Translation\n", "\n", "Add **rules** to inflect/join/split words based on source and target syntax.\n", "\n", "
\n", " \n", "
\n", "(Source: freeCodeCamp)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Interlingua\n", "\n", "The ideal: transform to and from a language-independent representation.\n", "\n", "
\n", " \n", "
\n", "(Source: freeCodeCamp)\n", "\n", "Too hard to achieve with rules!" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Example-based Machine Translation (EBMT)\n", "\n", "Retrieve a similar example from a translation database, and make adjustments as necessary.\n", "\n", "
\n", " \n", "
\n", "(Source: freeCodeCamp)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Statistical Machine Translation (SMT)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "## IBM Translation Models\n", "\n", "In the late 80s and early 90s, IBM researchers revolutionised MT using statistical approaches instead of rules.\n", "\n", "\n", "(Source: freeCodeCamp)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### IBM Model 1\n", "\n", "Simple word-by-word translation, but with **statistical** dictionaries.\n", "\n", "
\n", " \n", "
\n", "\n", "(Source: freeCodeCamp)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "
\n", " \n", "
\n", "\n", "(Source: freeCodeCamp)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### IBM Model 2\n", "Statistical translation and **reordering** model.\n", "\n", "
\n", " \n", "
\n", "\n", "(Source: freeCodeCamp)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### IBM Model 3\n", "Allows inserting new words.\n", "\n", "
\n", " \n", "
\n", "\n", "(Source: freeCodeCamp)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Word-based SMT\n", "\n", "Decompose source and target to words, with statistical **alignment**.\n", "\n", "
\n", " \n", "
\n", "\n", "\n", "(Source: freeCodeCamp)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Phrase-based SMT\n", "\n", "Decompose source and target to **phrases** and look them up in phrase tables.\n", "\n", "
\n", " \n", "
\n", "\n", "\n", "(Source: freeCodeCamp)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## MT as Structured Prediction\n", "\n", "**Model** \\\\(p(\\target,\\source)\\\\): how likely the target \\\\(\\target\\\\) is to be a translation of source $\\source$.\n", "\n", "$$p(\\textrm{I like music}, \\textrm{音楽 が 好き}) \\gg p(\\textrm{I like persimmons}, \\textrm{音楽 が 好き})$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "* How is the scoring function defined (**modeling**)?\n", "$$\\prob(\\target,\\source)\\approx s_\\params(\\target,\\source)$$\n", "\n", "* How are the parameters \\\\(\\params\\\\) learned (**training**)?\n", "\n", "$$\n", "\\argmax_\\params \\prod_{(\\target,\\source) \\in \\train} s_\\params(\\target, \\source)\n", "$$\n", "\n", "* How is translation \\\\(\\argmax\\\\) found (**decoding**)?\n", "\n", "$$\\argmax_\\target s_\\params(\\target,\\source)$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Training\n", "Learn the parameters \\\\(\\params\\\\) from data.\n", "\n", "
\n", " \n", "
\n", "\n", "(Source: freeCodeCamp)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Generative Models \n", "Estimate $\\prob(\\target,\\source)$: how is the $(\\target,\\source)$ data **generated**?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "### Noisy Channel\n", "\n", "* Imagine a message $\\target$ is sent through a noisy channel and $\\source$ is received at the end.\n", "* Can we recover what was $\\target$?\n", "* **Language model** $\\prob(\\target)$: does the target $\\target$ look like real language?\n", "* **Translation model**: $\\prob(\\source|\\target)$: does the source $\\source$ match the target $\\target$?\n", "\n", "This defines a **joint** distribution\n", "\n", "$$\\prob(\\target,\\source) = \\prob(\\target) \\prob(\\source|\\target)$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Neural Machine Translation (NMT)\n", "\n", "Model $s_\\params(\\target,\\source)$ directly using a neural network.\n", "\n", "
\n", " \n", "
\n", "\n", "\n", "(Source: freeCodeCamp)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Sequence-to-sequence models (seq2seq)\n", "\n", "Encoder–decoder architecture first \"read\" the input sequence and then generate an output sequence\n", "([Sutskever et al., 2014](https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf), [Cho et al., 2014](https://arxiv.org/abs/1406.1078)).\n", "\n", " \n", "\n", "
\n", " \n", "
\n", "\n", "(Examples are Basque–English)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### We can use RNNs for that!\n", "\n", "Example architecture:\n", "* Encoder: word embedding layer + Bi-LSTM to capture contextual information\n", "* Decoder: Uni-directional LSTM (because we need to *decode* word by word) + softmax layer on top\n", "\n", "
\n", " \n", "
\n", "\n", "The end-of-sequence symbol `` is necessary to know when to start and stop generating." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### But something's missing (again)...\n", "\n", " \n", "\n", "
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Output words **depend on each other**!" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Autoregressive MT\n", "\n", "At each step, feed the predicted word to the decoder as input for predicting the next word.\n", "\n", "
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "### Training\n", "\n", "+ Loss function: negative log-likelihood\n", "+ **Teacher forcing:** always feed the ground truth into the decoder.\n", "\n", "*Alternative:*\n", "\n", "+ **Scheduled sampling:** with a certain probability, use model predictions instead.\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "### Decoding\n", "\n", "+ Greedy decoding:\n", " + Always pick the **most likely word** (according to the softmax)\n", " + Continue generating more words **until the `` symbol is predicted**" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "+ Beam search:\n", " * In each step chooses **best next source word to translate**\n", " * **Append a target word** based on source word\n", " * Maintains a list of top-$k$ hypotheses in a **beam**" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## More problems with our approach\n", "\n", " \n", "\n", "
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Word alignment\n", "\n", "In rule-based and statistical MT, word alignments are crucial.\n", "\n", "
\n", " \n", "
\n", "\n", "Can we use them in neural MT?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Attention mechanism\n", "\n", "Jointly learning to align and to translate.\n", "\n", "
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Attention matrix\n", "\n", "
\n", " \n", "
\n", "\n", "
\n", " (from Bahdanau et al., 2014)\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Transformers\n", "\n", "Replace LSTMs by self-attention. Attend to encoded input *and* to partial output (autoregressive).\n", "\n", "
\n", " \n", "
\n", "\n", "
\n", " (from The Illustrated GPT-2)\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# MT evaluation\n", "\n", "We're training the model with *negative log-likelihood*, but that's not the best way to *evaluate* it." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Consider:\n", " \n", "+ After lunch, he went to the gym.\n", "+ After he had lunch, he went to the gym.\n", "+ He went to the gym after lunch.\n", "+ He went to the gym after lunchtime.\n", "\n", "In machine translation, there are often **several acceptable variations!**" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Human evaluation\n", "\n", "* **Accuracy** (or meaning preservation) to evaluate the \"translation model\"\n", "* **Fluency** to evaluate the \"target language model\"\n", "\n", "
\n", " \n", "
\n", "\n", "
\n", " (from Freitag et al., 2021)\n", "
\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "In general, manual evaluation is the best way, but it is not scalable. **Automatic** metrics are therefore often used." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## BLEU score\n", "\n", "A widely used *reference-based* metric ([Papineni et al., 2002](https://aclanthology.org/P02-1040/)):\n", "\n", "+ Compare the prediction to one or more reference translations.\n", "+ Count the number of matching $n$-grams between them.\n", " - It is common to consider $1 \\le n \\le 4$\n", "+ Divide by total number of $n$-grams.\n", "\n", "The BLEU score will range between 0 (*no match at all*) and 1.0 (*perfect match*, 100%)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Recommended library: [sacreBLEU](https://github.com/mjpost/sacrebleu)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### BLEU score examples" ] }, { "cell_type": "code", "execution_count": 39, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "!pip install -q sacrebleu" ] }, { "cell_type": "code", "execution_count": 40, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "data": { "text/plain": [ "100.00000000000004" ] }, "execution_count": 40, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sacrebleu.metrics import BLEU\n", "bleu = BLEU()\n", "\n", "refs = [[\"After lunch, he went to the gym.\"],\n", " [\"He went to the gym after lunch.\"]]\n", "bleu.corpus_score([\"After lunch, he went to the gym.\"], refs).score" ] }, { "cell_type": "code", "execution_count": 41, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "34.57207846419409" ] }, "execution_count": 41, "metadata": {}, "output_type": "execute_result" } ], "source": [ "bleu.corpus_score([\"Turtles are great animals to the gym.\"], refs).score" ] }, { "cell_type": "code", "execution_count": 42, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "69.89307622784945" ] }, "execution_count": 42, "metadata": {}, "output_type": "execute_result" } ], "source": [ "bleu.corpus_score([\"After he had lunch, he went to the gym.\"], refs).score" ] }, { "cell_type": "code", "execution_count": 43, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "86.33400213704509" ] }, "execution_count": 43, "metadata": {}, "output_type": "execute_result" } ], "source": [ "bleu.corpus_score([\"Before lunch, he went to the gym.\"], refs).score" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Beyond BLEU\n", "\n", "BLEU counts word n-gram overlap with a brevity penalty. That’s useful, but it can under-reward good translations when:\n", "\n", "* Morphology matters (rich inflection, compounding): small suffix/prefix changes break word overlap.\n", "\n", "* Tokenization is brittle (e.g., clitics, hyphenation, script variants).\n", "\n", "* Synonyms & paraphrases change surface forms while preserving meaning.\n", "\n", "* We have few references, so \"valid\" variants may be missed.\n", "\n", "Result: perfectly fluent, faithful outputs can receive lower BLEU than they deserve." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### chrF\n", "\n", "**chrF** measures **character n-gram** F-score between hypothesis and reference ([Popović, 2015](https://aclanthology.org/W15-3049/)).\n", "\n", "$\\text{chrF}_\\beta=(1+\\beta^2)\\frac{\\text{chrP}+\\text{chrR}}{\\beta^2 \\cdot \\text{chrP}+\\text{chrR}}$\n", "\n", "This makes it:\n", "\n", "* Robust to morphology/tokenization. Character n-grams still match across inflectional variants and token boundaries.\n", "\n", "* Language-agnostic & lightweight. No lemmatizer or tokenizer tuning needed.\n", "\n", "chrF++ extends chrF by also including **word n-grams** (set `word_order=2` in SacreBLEU), often improving correlation when phrasing matters." ] }, { "cell_type": "code", "execution_count": 44, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "data": { "text/plain": [ "100.0" ] }, "execution_count": 44, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sacrebleu.metrics import CHRF\n", "chrf = CHRF()\n", "\n", "chrf.corpus_score([\"After lunch, he went to the gym.\"], refs).score" ] }, { "cell_type": "code", "execution_count": 45, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "31.727051682113938" ] }, "execution_count": 45, "metadata": {}, "output_type": "execute_result" } ], "source": [ "chrf.corpus_score([\"Turtles are great animals to the gym.\"], refs).score" ] }, { "cell_type": "code", "execution_count": 46, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "46.78706825841291" ] }, "execution_count": 46, "metadata": {}, "output_type": "execute_result" } ], "source": [ "chrf.corpus_score([\"He went to luncheon, afterwards gymnastics.\"], refs).score" ] }, { "cell_type": "code", "execution_count": 47, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "28.006406701683282" ] }, "execution_count": 47, "metadata": {}, "output_type": "execute_result" } ], "source": [ "chrf.corpus_score([\"Before breakfast, he hit the gym.\"], refs).score" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### BERTScore: capturing meaning with embeddings\n", "BERTScore ([Zhang et al., 2020](https://arxiv.org/abs/1904.09675)) compares contextual embeddings of hypothesis and reference tokens to compute Precision/Recall/F1.\n", "\n", "Handles synonyms/paraphrases better than n-gram overlap." ] }, { "cell_type": "code", "execution_count": 48, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "!pip install -q bert-score" ] }, { "cell_type": "code", "execution_count": 57, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "a458212c56be45ac94fd8f5d0f58a6a7", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Downloading tokenizer_config.json: 0%| | 0.00/52.0 [00:00