{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "import sys\n",
    "sys.path.append(\"..\")\n",
    "import statnlpbook.util as util\n",
    "import statnlpbook.mt as mt\n",
    "util.execute_notebook('word_mt.ipynb')\n",
    "matplotlib.rcParams['figure.figsize'] = (10.0, 6.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<!---\n",
    "Latex Macros\n",
    "-->\n",
    "$$\n",
    "\\newcommand{\\Xs}{\\mathcal{X}}\n",
    "\\newcommand{\\Ys}{\\mathcal{Y}}\n",
    "\\newcommand{\\y}{\\mathbf{y}}\n",
    "\\newcommand{\\balpha}{\\boldsymbol{\\alpha}}\n",
    "\\newcommand{\\bbeta}{\\boldsymbol{\\beta}}\n",
    "\\newcommand{\\aligns}{\\mathbf{a}}\n",
    "\\newcommand{\\align}{a}\n",
    "\\newcommand{\\source}{\\mathbf{s}}\n",
    "\\newcommand{\\target}{\\mathbf{t}}\n",
    "\\newcommand{\\ssource}{s}\n",
    "\\newcommand{\\starget}{t}\n",
    "\\newcommand{\\repr}{\\mathbf{f}}\n",
    "\\newcommand{\\repry}{\\mathbf{g}}\n",
    "\\newcommand{\\x}{\\mathbf{x}}\n",
    "\\newcommand{\\prob}{p}\n",
    "\\newcommand{\\a}{\\alpha}\n",
    "\\newcommand{\\b}{\\beta}\n",
    "\\newcommand{\\vocab}{V}\n",
    "\\newcommand{\\params}{\\boldsymbol{\\theta}}\n",
    "\\newcommand{\\param}{\\theta}\n",
    "\\DeclareMathOperator{\\perplexity}{PP}\n",
    "\\DeclareMathOperator{\\argmax}{argmax}\n",
    "\\DeclareMathOperator{\\argmin}{argmin}\n",
    "\\newcommand{\\train}{\\mathcal{D}}\n",
    "\\newcommand{\\counts}[2]{\\#_{#1}(#2) }\n",
    "\\newcommand{\\length}[1]{\\text{length}(#1) }\n",
    "\\newcommand{\\indi}{\\mathbb{I}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# (Word-based) Machine Translation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "##  Machine Translation (MT)\n",
    "\n",
    "* Paradigms: \n",
    "  * Rule-based vs **statistical**\n",
    "  * **feature-engineering** vs neural\n",
    "  * **word** vs phrase vs syntax units "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##  Word Based MT\n",
    "\n",
    "Translate **word-by-word**\n",
    "* **foundational** to all current approaches (e.g. neural methods)\n",
    "* **subcomponent** in more complex systems (for alignments)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## MT as Structured Prediction\n",
    "\n",
    "* **source** sentence \\\\(\\source\\\\)\n",
    "    * aka $\\x$, usually tokenized\n",
    "    * \"音楽 が 好き\" or \"Ich mag Musik\"\n",
    "* **target** sentence \\\\(\\target\\\\)\n",
    "    * aka $\\y$, usually tokenized \n",
    "    * I like Music\n",
    "* a **model** \\\\(s_\\params(\\target,\\source)\\\\) to measure match of \\\\(\\target\\\\) to $\\source$\n",
    "    * $s($I like Music,音楽 が 好き$)=12$\n",
    "    * $s($I like StatNLP,音楽 が 好き$)=-20$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "For most of this lecture:\n",
    "* **Target**: English\n",
    "* **Source**: German"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Training\n",
    "learn the parameters \\\\(\\params\\\\) from data \n",
    "* usually from **parallel corpora** of **aligned source and target sentences**\n",
    "* Where can you get such data from?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Translation\n",
    "predict highest-scoring translation given source $\\source$:\n",
    "\n",
    "\\begin{equation}\n",
    "\\argmax_\\target s_\\params(\\target,\\source)\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "MT approaches differ primarily in \n",
    "* how \\\\(s\\\\) is defined, \n",
    "* how \\\\(\\params\\\\) are learned\n",
    "* how the translation \\\\(\\argmax\\\\) is found"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Modelling\n",
    "How to define $s_\\params(\\target,\\source)$?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Generative Models \n",
    "How is the $(\\target,\\source)$ data **generated**? \n",
    "\n",
    "Develop a **distribution** $s_\\params(\\target,\\source)=\\prob_\\params(\\target,\\source)$ that generates **faithful samples**  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Good** generative model: samples look like\n",
    "* I like music --- Ich mag Musik\n",
    "* I love music --- Ich mag Musik sehr gerne \n",
    "* etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Bad** generative model:\n",
    "* I don't like music --- Ich mag Musik\n",
    "* I love music --- Musik sehr gerne ich mag\n",
    "    * *example is a reasonable translation, but very unlikely German.* \n",
    "\n",
    "Generative models need to model both input and output! Contrast with **discriminative models**...\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "How can we decompose the problem?\n",
    "\n",
    "*in the same way we decomposed language models $\\prob(w_1,\\ldots,w_n)=\\prob(w_1)\\prob(w_2|w_1)\\ldots$*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Think about the problem **backwards**\n",
    "* Generate the target $\\target$ using $\\prob_{\\params_t}(\\cdot)$\n",
    "* Generate the source $\\source$ using $\\prob_{\\params_s}(\\cdot|\\target)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "This defines a **joint** distribution $\\prob_\\params(\\target,\\source) = \\prob_{\\params_t}(\\target) \\prob_{\\params_s}(\\source|\\target)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Training\n",
    "Find $\\params$ using **MLE** (or smoothed variants)\n",
    "\n",
    "$$\n",
    "\\argmax_\\params \\sum_{(\\target,\\source) \\in \\train} \\log \\prob_\\params(\\target, \\source)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Translation\n",
    "Operate **forwards**:\n",
    "\n",
    "$$\n",
    "\\argmax_\\target \\prob_\\params(\\target|\\source) = \\argmax_\\target \\prob_\\params(\\target,\\source)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Why this equality? $p(\\target|\\source)=\\frac{p(s,t)}{p(s)}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "How important would $\\prob(\\source)$ be if we had chosen $\\prob(\\target,\\source) = \\prob(\\source) \\prob(\\target|\\source)$?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Noisy Channel  \n",
    "$\\prob(\\target,\\source) = \\prob(\\target) \\prob(\\source|\\target)$ is often called a **noisy channel model**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAAnEAYAAABxl8L2AAAJJGlDQ1BpY2MAAHjalZVnUJNZF8fv\n8zzphUASQodQQ5EqJYCUEFoo0quoQOidUEVsiLgCK4qINEUQUUDBVSmyVkSxsCgoYkE3yCKgrBtX\nERWUF/Sd0Xnf2Q/7n7n3/OY/Z+4995wPFwCCOFgSvLQnJqULvJ3smIFBwUzwg8L4aSkcT0838I96\nPwyg5XhvBfj3IkREpvGX4sLSyuWnCNIBgLKXWDMrPWWZDy8xPTz+K59dZsFSgUt8Y5mjv/Ho15xv\nLPqa4+vNXXoVCgAcKfoHDv+B/3vvslQ4gvTYqMhspk9yVHpWmCCSmbbcCR6Xy/QUJEfFJkT+UPC/\nSv4HpUdmpy9HbnLKBkFsdEw68/8ONTIwNATfZ/HW62uPIUb//85nWd+95HoA2LMAIHu+e+GVAHTu\nAED68XdPbamvlHwAOu7wMwSZ3zzU8oYGBEABdCADFIEq0AS6wAiYAUtgCxyAC/AAviAIrAN8EAMS\ngQBkgVywDRSAIrAH7AdVoBY0gCbQCk6DTnAeXAHXwW1wFwyDJ0AIJsArIALvwTwEQViIDNEgGUgJ\nUod0ICOIDVlDDpAb5A0FQaFQNJQEZUC50HaoCCqFqqA6qAn6BToHXYFuQoPQI2gMmob+hj7BCEyC\n6bACrAHrw2yYA7vCvvBaOBpOhXPgfHg3XAHXwyfgDvgKfBsehoXwK3gWAQgRYSDKiC7CRriIBxKM\nRCECZDNSiJQj9Ugr0o30IfcQITKDfERhUDQUE6WLskQ5o/xQfFQqajOqGFWFOo7qQPWi7qHGUCLU\nFzQZLY/WQVugeehAdDQ6C12ALkc3otvR19DD6An0ewwGw8CwMGYYZ0wQJg6zEVOMOYhpw1zGDGLG\nMbNYLFYGq4O1wnpgw7Dp2AJsJfYE9hJ2CDuB/YAj4pRwRjhHXDAuCZeHK8c14y7ihnCTuHm8OF4d\nb4H3wEfgN+BL8A34bvwd/AR+niBBYBGsCL6EOMI2QgWhlXCNMEp4SyQSVYjmRC9iLHErsYJ4iniD\nOEb8SKKStElcUggpg7SbdIx0mfSI9JZMJmuQbcnB5HTybnIT+Sr5GfmDGE1MT4wnFiG2RaxarENs\nSOw1BU9Rp3Ao6yg5lHLKGcodyow4XlxDnCseJr5ZvFr8nPiI+KwETcJQwkMiUaJYolnipsQUFUvV\noDpQI6j51CPUq9RxGkJTpXFpfNp2WgPtGm2CjqGz6Dx6HL2IfpI+QBdJUiWNJf0lsyWrJS9IChkI\nQ4PBYyQwShinGQ8Yn6QUpDhSkVK7pFqlhqTmpOWkbaUjpQul26SHpT/JMGUcZOJl9sp0yjyVRclq\ny3rJZskekr0mOyNHl7OU48sVyp2WeywPy2vLe8tvlD8i3y8/q6Co4KSQolCpcFVhRpGhaKsYp1im\neFFxWommZK0Uq1SmdEnpJVOSyWEmMCuYvUyRsryys3KGcp3ygPK8CkvFTyVPpU3lqSpBla0apVqm\n2qMqUlNSc1fLVWtRe6yOV2erx6gfUO9Tn9NgaQRo7NTo1JhiSbN4rBxWC2tUk6xpo5mqWa95Xwuj\nxdaK1zqodVcb1jbRjtGu1r6jA+uY6sTqHNQZXIFeYb4iaUX9ihFdki5HN1O3RXdMj6Hnppen16n3\nWl9NP1h/r36f/hcDE4MEgwaDJ4ZUQxfDPMNuw7+NtI34RtVG91eSVzqu3LKya+UbYx3jSONDxg9N\naCbuJjtNekw+m5qZCkxbTafN1MxCzWrMRth0tie7mH3DHG1uZ77F/Lz5RwtTi3SL0xZ/Wepaxls2\nW06tYq2KXNWwatxKxSrMqs5KaM20DrU+bC20UbYJs6m3eW6rahth22g7ydHixHFOcF7bGdgJ7Nrt\n5rgW3E3cy/aIvZN9of2AA9XBz6HK4ZmjimO0Y4ujyMnEaaPTZWe0s6vzXucRngKPz2viiVzMXDa5\n9LqSXH1cq1yfu2m7Cdy63WF3F/d97qOr1Vcnre70AB48j30eTz1Znqmev3phvDy9qr1eeBt653r3\n+dB81vs0+7z3tfMt8X3ip+mX4dfjT/EP8W/ynwuwDygNEAbqB24KvB0kGxQb1BWMDfYPbgyeXeOw\nZv+aiRCTkIKQB2tZa7PX3lwnuy5h3YX1lPVh68+EokMDQptDF8I8wurDZsN54TXhIj6Xf4D/KsI2\noixiOtIqsjRyMsoqqjRqKtoqel/0dIxNTHnMTCw3tir2TZxzXG3cXLxH/LH4xYSAhLZEXGJo4rkk\nalJ8Um+yYnJ28mCKTkpBijDVInV/qkjgKmhMg9LWpnWl05c+xf4MzYwdGWOZ1pnVmR+y/LPOZEtk\nJ2X3b9DesGvDZI5jztGNqI38jT25yrnbcsc2cTbVbYY2h2/u2aK6JX/LxFanrce3EbbFb/stzyCv\nNO/d9oDt3fkK+Vvzx3c47WgpECsQFIzstNxZ+xPqp9ifBnat3FW560thROGtIoOi8qKFYn7xrZ8N\nf674eXF31O6BEtOSQ3swe5L2PNhrs/d4qURpTun4Pvd9HWXMssKyd/vX779Zblxee4BwIOOAsMKt\noqtSrXJP5UJVTNVwtV11W418za6auYMRB4cO2R5qrVWoLar9dDj28MM6p7qOeo368iOYI5lHXjT4\nN/QdZR9tapRtLGr8fCzpmPC49/HeJrOmpmb55pIWuCWjZfpEyIm7J+1PdrXqtta1MdqKToFTGade\n/hL6y4PTrqd7zrDPtJ5VP1vTTmsv7IA6NnSIOmM6hV1BXYPnXM71dFt2t/+q9+ux88rnqy9IXii5\nSLiYf3HxUs6l2cspl2euRF8Z71nf8+Rq4NX7vV69A9dcr9247nj9ah+n79INqxvnb1rcPHeLfavz\ntuntjn6T/vbfTH5rHzAd6Lhjdqfrrvnd7sFVgxeHbIau3LO/d/0+7/7t4dXDgw/8HjwcCRkRPox4\nOPUo4dGbx5mP559sHUWPFj4Vf1r+TP5Z/e9av7cJTYUXxuzH+p/7PH8yzh9/9UfaHwsT+S/IL8on\nlSabpoymzk87Tt99ueblxKuUV/MzBX9K/FnzWvP12b9s/+oXBYom3gjeLP5d/Fbm7bF3xu96Zj1n\nn71PfD8/V/hB5sPxj+yPfZ8CPk3OZy1gFyo+a33u/uL6ZXQxcXHxPy6ikLxyKdSVAAAAIGNIUk0A\nAHomAACAhAAA+gAAAIDoAAB1MAAA6mAAADqYAAAXcJy6UTwAAAAGYktHRP///////wlY99wAAAAJ\ncEhZcwAAASwAAAEsAHOI6VIAAAAHdElNRQfhChgWJSuSNfMzAAAv9klEQVR42u19B1zN+///+Zw6\np61FGlRKNIwoo2EmVMZFCNluVOaNZI/KRdkZyV7XioRsHVRmuvY1QkOD9jTC/36en7ffv+N7nFvJ\nveH9uo+H5+Wc8/7s1+v1fK0Pj0eFChUqVKhQoUKFChUqVKhQoUKFChUqVKhQoUKFChUqVKhQoUKF\nChUqVKhQoUKFChUqVKhQoUKFChUqVKhQoUKFChUqVKhQoUKlpgnTgvyPWxUXaEywhJ5LKlSo1DD9\nNoT8T5uvXEiG4AmCLapp/0zoNaJChQoVKj+RGLzjsNt2DgX+VVuHP4VDG1sOjZPpuaVChcp/KyYD\nOOwygvCHSV9JFFpzOHgkh3qvqmc/zVZwaL2X/MNZeu2oUKFChcoPKPU3c/gbyXgY7yQffJ4BecCB\n0hZCNOp8YUGSAalHCIzPew71Q+m5pkKFyr8rDUlGYcpzou9yyQdfmQFhLnC4ZA2HFg7Sv6+UwaHg\nH9ZVacmhVwjZzU30GlKhQoUKlR9IBIYcTl/HYZdw6d/XeMThWksOtdZWbDsORWQ7PTiUnU7PPRUq\nVL6tyGtwOLchh7YR1bs+c4rD3305NN8u+XsywRyu0iWE4tcKEid9sn46h5pCek2pUKFChUoNEuaX\nLxi+f4jwWZoSw5jEoeLcL3zxGgf91Di8I8uhLsmcMF7ke70k/1zpGCEupzlsOoReMypUqFRQv/l9\nQb9ZS/+dbTyHQbM4FG6o2Pb4iwkWVpKA7JD8vVYXOXxMsCPJJDNkv3i/fWE/BnM4y5Lo3zB6L1Ch\nQoUKlf9QZBQ4bEsibl4LiGEjBtf+DTFYpQTVOVS4IL6ONzHgv5VK3o6QrOtuwOF9EkksbMvhljgO\nF5OagvpK0vd7JsmYjLpFryEVKlS+4HiTUqXWDTj0jOCwiyYhFiSw0vcFh66JHCpbiK/jSxx9j3jp\n21NV43AYcfB993E4jei5QJL57XL2HwhIT7L/ZL97ET13iTSnvycBn0Ok92R5Iw6b/EOJVR9S6rp4\nG9H/7vQeoUKlmuXjR4oUqw9/XDEihrgnaX70IdOmjupxaDKKw0+9GivyicH+LEMRQgz2L4++QHQm\nc2ixnMOLSzjM0OKwW1MOm5NInqKB9P0eRAjL0nlUn1GkSPWkZNEnvWN9iX7yJkQjijR9m5YRfUP0\nyBLSezHoICEGTTjcTBz3Ll7Stzd8NIeHnxJCsIcEVFI49CO9a+7MPxAQUlrKeBA9TXCjPIfvAjgc\nQ4Z9tCD6WfWk9P2zNORwDyndUjKneo8ixWrWgx8/+ogoUqw6jrHmUO3cj+3aGpDmcEMS+Vs2jsOZ\nCp9FEsdwuOlTKr+h+Of7SMlVBw3p25MhBCXiHodJxHDW6Vq5/e5BHImtXXk/gXz8yMzgsGshfT4p\n1hwcPIdD+YY188nRI5kGI1LqGUgIycI/PyMAlhyuJhmGQEJMZK8QfZXGYcuZ0rfnTvRaOnHso1tx\nuP4lIQwko6E74bPtn/mMgOySvP6cbA7fkoyzXSUJhBE5/ijiKqluqNl6z+A+h+6hBGdRpFgD8QiH\nfS+TG5fjIhQpVg2TlDg0Wc/7KURlIIdHpnHo+Eb8cw0yjSr6KocDPjN8O4khcwiUvh0hiSQeeUsI\nCOkZ0SKZFSUXDuu+l75Ob+JYhHb/OQiITG8Ot7rR55NizcHrNhxqdKvZT5ASGeN9sDOHPVeKf16L\nEIzTJEM79AgJvJASrQNTOWxdJn07tYljP86VBGYIJpASqgxSOjWdnC9mKcH1nxGQLzShzyPE5C35\nvT3R26qRHGqekb5/Jvs5jCSErJZ7zdZ7A/w5fDeYYG+KFGsgjiZ+ozMlIBSrARM3EQIi+3MQEIu6\nHF4mU6wafBbp67+Iw70HiKH9rATLnxjS4WOlb0dAMiWHyRz9F6QUS5tkWLoTgzh3ifR1xhJCNPP3\nn4SAdOZwUyZ9PinWHLxiQAhIDX+CjGtzGOPIYZMk8c9dSI/H/m0kAHKcfED00jKi9/r9gz0YlsWh\nB9Fz8qQpXI2M7w0kBGMf0bMCMqacsa8YAZlN9HIZKclqTwjJMD6Hnv+QEWlHxvxuJKWucgdqtt5z\n60WfM4rfD6YJKQGhSAlIpcWNlFzFx3I4kBjgrmTc48zdHJqNkvz7Tz0h/r8Qg/qFJvJPb+YdtpHD\nW2ocfppqteZTrfUXInmMHIdLiAF3+YsSEIoUKQH5h4wpyUAkRJMMBymJ6kKGWfiS3rdm8l8gFrM5\n9PvUA6Iv+Xu/EX12lrxAddJwcUKynDj+PT6NKbcheu1YxQiILZkWeL0ZIUykd2Q9eR9JWxnp52FU\nFIdTH5J/yKvhBMSQPmcUvx98EUwJCEVKQCosDGm2DCbEYeEHkgEx5LCRNylhGCB9nbpk2ssq0jxZ\n/x/G48oYcWjYl0MrEjk0IE2UjJHk3+mTUq01Thxq/SRN6JSAUKQEpApC9IU/CagEEQJiSIZuNLrJ\noXK69GUMSOBkJdFTWvUkf6826SHRJxncJuR7Lcj4cV0SQPl8zHhF3wPCI/pOl2QyrEhJl8GnMenH\nJf9MkfTuBSmT3935PvQeJSAUKQGhSAnIjyZk+pQKKXmKIgasz5gqrkcyJb1JpsTjMIfV9aJAPik1\nGEGaL3vr8n4ioQSEIiUglRKSgVUi7984TKZgDe5QxfVIj4UbyVQMIxkO/rJqCgR9IiBkKId5ZPWe\njk4xHHqTDI1sISUgFCnWYAKiHaMdw+JQ3lD8fWW/lf1Y3Pq3sLj8w/IPLA72GezDos4rnVc/8wXQ\nT9RPZNGqk1UnFhXWK6ynBKSmipDURLvZcXikDykZIM2XqlU01EIy3rE/iQB2JVNh+FU0eAyJFLYi\nJQf9MzmU+ykyH9+SgChEKUSx2Nm0symLc7vO7cri5pDNISyGbg7dzOIU6ynWLDbzaObBIv8g/6C0\ndeuPqT+GReub1jdZlM+Sz6KGqeaghq2GLYuto1tHs6ipo6nz4xEQGdIT0Y9kJA6RDMe0+hxqzq/a\nugrEgR9Mpvl1JPqTuf+VBISURgWQzK9Zz+o5D4YkoDSc9LhoLvm+9F7VCYhKVxXos9bprdNZbLup\n7SZgbNtYadh6fWv4LcaDjAexKIgSRFG98f+Rmc/MZ9Ei2gL6w9zO3I5FGQUZBXp+qoGA2HnZebEY\nqxyrzGJR+6L2LD6a92gei9dGXBvB4oPCB4UsFpwrOMfi5aGXh7JoG2YbhvVkeDI/0wVYVLKohMXn\nxs+NWTTLM8ujBKTGymOSmielUoqEkCgRIsI3+brlBWR8sRGZHiNzsIqGeRuHemQ+v1ws7yeU6iQg\n+qb6IBxb3m55y2Ju09ymLCb/lvwbi9dfXn/J4p29d/bic5NcE3x+N/kui55XPa/CMG8WbJa0/vzf\n5//OYtrLNKzTWKaxDDVMNQd7H+99HNd1bO5YFvv59/P/ATMg44leMyRIMqdKpHSTf/7rlpcjhKMB\nCYjwnb5yf8n0QJMbZD+Dquc01CHj1VWDvk+9V3UCYqtsC/8tzzoPAZRi9WJ1FvPv5kOP5WfnZ4vh\np383zDdk8enup7tZ3LR602oWTfgmfKo/Pn4UHhQiAHXR96Ivi2fOnYH/qzJHZQ49P19BQOqY1TFj\n8ULkhUgWs1tkt4DB/fs/VgzqGNRhUW2W2ixE+lrXb83iyKYjYcDTLNMsWYyZGzOXRa1Yrdif6QIE\nywfLs5gRlBEEhuxv4U8JCBUqNYOAqF5QvcDiH+F/hLNY8kvJL3huRwePxvPKt4CBVc9Xz0cGeKX2\nShYd/v6PFVG8KB6O6+vc1yz2WNpjqaTtBA4KROQwd3UuDLfpXNO51DDVHOwb2Rf27fXI1yNZHLBr\nwK4fvwmdys9GQOxt7G1YLFtVtorFsxPPTsT9f7HvRRDvaf2mlcdP/z4iYUQCiwdnHpyJ398pu8Pi\n3oi9ESwqC5WFP7P+kD0te5rFJTeW3GAxIC4gDhl1XwVfql+/goB0edrlKRTznNdgchsbb2yMkoPz\n/PNSf6/CU2Fx1u1Zt1l82/4tMiYupi6mFUoVClQEiExO0Z8C/Ps//LuXitfXnBAZTxkwJ53lOstZ\nNFQ2RERAZb/K/kql3DozncsTKsMYQ5SmaTbRbFL+e0GOQY4gIO0y2sGh8bHwkcqkWwhB8HTX6K5h\nsYFCA6TwNAdpDvqa4+YX84tZ1PPRw/ZVbVRtKAGh8tMSED2eHoujFUbj+XrzyxsQj7Wz1iKQotBG\noU1F1ml9tPVRBGb0srFe5LDIYSwqrlFcI0ZAzAIRyMl1ynVCBkS2sSwMl4os9KROpk4m9Mh+Q+gh\nDZ5G5UoAFBlFFmsfq30M66QbosRCP1Yf+knNRM3kqwzsbVnocb3VeiBQBlEGKMFQuapy9WvW1XTT\ndMP+phimsFjXuC4yxYwao1aZdeTuyMEhqjeyHgiEoZMhzrO6q7prhQhISF+U2L2Wf42A0YDAAYGU\ngFD54QiIsT2er/eF71GpslF3o25lfq91TguR/dgLsQjc5DzOecyi5QHLAxX5vdoaNehFQ29Dbzyv\nQ+uhQkb+nPy5qhyPUoZSBvTcOP1xwJb6LfHv9ZTqVUp/+jDwi+rsrLMTfpdVAysEnN5pv4PfeEnm\nktR1THjQr7JtZdvi+/kyCFjxuvG6/RuOvmKCIghi/fX1USpncMbgDIu1xtQa8zXrChYKFuI6rai3\nAkTTS9nrXyUgrtauSNV9kqC3QW8r8/umU5tOZTGsUVgj1FbHdpaYAZEdLjucxT42feAYnxp4aiBK\nHGKT8f1U01QQl9NTToOQuAa7BuN3XrIST4hdvB0ik4tfLUYPipm9mT2L/o/98cDc176vjQxN/bT6\n2J7FKQsQrkldJuGG7MP0kdgDE6cNZrvg3QLcmLdVb6uymH4y/SSLV/dc3YMemX5D0RuzsvFKELb0\nW+m3pBEQy/6W/dFL47MVnz+a9ggRiIx+GVjnWvtrIHDTrKfheqjbqttKWqeRViMtXKfbQXAYWvZo\n2YNFr5Ne2L/EnMQcZLD+FkpAqHwfYkFKMJokf0ZAelSVgCg1VsJzeWbjmY14Pueno4a3mX4z/Uql\n3hcKoaDXXl57mcUo1yg4vLqtdFuJERCLQOiX7AfZD1hs/7Q9AjvBnsF4Du+/uP8C+zEtHc99nFKc\nEotDzIaAuMjWk5VoUHXW68DgLJy+cDqL8QPiB2CdtPQ0FlMCUgJQGrDu4joWxwWOg2OtMFthtqT1\nnHY6wQDPT52fCr3pbQZHYfnR5SBaj3Y9QmbgxbwXKL09aXwSDk0XQRcEjHiveGI9f9pztBG4WnJv\nyT0c97b221h083TDcV+ZfQX7kd41HbXpt61vQ79NV5kOYqZiqyJRz/Hm8LCubQtbBGz+mPcH9ufp\n6KfIXKVdSIODFLs0FhkprwAvnAelcKVwSkCo1GzRIFO8Oq3mkN/oMwLS4KsJSPL7ZPhlY8Kq5KCG\nvgl9w+IHhw9IBbd72O6hRMc4TBGl9x6vPKAXYmRjEHhJK0wDAXo64+kMFveb7IcDb//E/kn5gMr/\nBFJH8UdBTxU4FbB4bOsx9B4nb0uGXkn2SYb/dDT+KPy/7vHdgfzx/PES/TkrbRANf0V/bC9hRQIc\n7YxVGcgQPVB+gAD1mp1roBcbtWzUUmJg21YGemqYYBj04OCcwfCz5G7Jwe8bunsoStcWb16MEl3d\nEbojpAbKD8qgpGvsr2N/xf5N9YcfXdu6NvQjvwO/A4vdNnVDD0+kfST82+cvn6PE98UfL/5g8bz5\neXPYkalD8HvhYeFhSdvraNoR/vXvHX/viNK6YyYIZC3xWIIex2clz9BK0D+9f/q/SkDaCtrihOYE\n58Dhf2b4DLWAQy2HorRK66EWbjxmH7NP6gk9JwOGy9xgboh9vp+HiN+Qk0PgIGdnZaM58+aUmyAa\n8/3no2TJ38kfEa27qnfh8GenZ+NEDJ0zFIaImcHMKL+uJ88TWLy6GBG7Xdt3bWdxZ9xOEIhhl4fB\nYZifNx89GTkbc+CIJMxOgEGsp1ZPLAKnXKiMB2ab1jY4+K//FjwA9Y+BwPiZ+OEBWt91PWdIX93G\nA/dw1UPcyKlhqWGSCEiTpU1gIP90+dMF3xuSOgTERX0lajOnp06HI3Bi1Ak8eCVyJXIshpwOQcpP\ncYyimAJxUHfA74r+KvoLhKb21tos3nt9D/t7wvAErl+3h90eVoGAhHHY8Bg1ElT+PbHN5nDdOw7r\naHH3IcMnBORlZfVak3dNEEDIiMlA5vL4mON4jhRiFapUIipnJgeioLRJCQaBP5Q/tPznAWEBeP6L\nw4qBJ9+ffM/i9qDtKM0cNnnYZOiRX/yQiUm8mojMQoZsBgy2zVUbsUyD3AS5CXAEBoYiUFO6sBRE\n6OCBg4hETus0DUMv5i+cj39PGJOA4yv+vRi9KAOcBzhjPXmefPl1F+gtQCYnq3lWcxb3jd8Hw701\naCv2c2jPoT3R29Z/EQImuVtyt2D99ATo43rv670vv56FgYUBasjt8tGUeWTYEWSIIiMiUboxcdHE\nRTC0z8Y+Q6BlybUl0K/vXuP6uLdxl5iJsltqB735xPkJjiPRJBH6d/G9xSA6s1rMAjER5YsQiSx2\nKYZ+XSC/AMcraCJo8g0JiB6H6tvps0ul8qLZhMP1JOBiIy9OQAbmfjUByXoPP2tj2MawSum5pnIo\nrT+y7ggCGpkZmchAWDS0aCgWOQ8WwF+cO34u9EdB0wL87nThafhRfpf8kFFYFrsM+jbZNRmBm6eu\nT4HtNdprSNp+tznd4O+lXUrD7xOsEkAg5lnPg2O+yGAR9E2ibiIyO6l1U+tivSntp4hlYkRqIhb3\nnNwDvzNvaR70yZ7FexazODV9KvTZlnVbcJx58XkgMhdtL4JoGGYZig0REd4RIgMbtz0Ofmb05Wj4\nl0ojlEA0JttOxu8+eH5A4MV9pvtMaefZONc4l8WkV0nwI4+/Pw69qtBPAQHpHkN7wL6kN0mHHrvj\newelXgtDF4bifKjNg/96w/4GiEnBrAJk9sfHjYf/K2MsY1x+e5MPTwYxyR+TDzuxxXgLPn+09BHO\nS/i6cJwHW1Nb03+VgCikKsABDu4YDGZUalWKC/7O7h0Myi3FW2COoe6h7uj9UBiJkobml5vjAiib\nK5tLW19ntw6Y4Z/MnwyaPe9dhwFpeKfhHUnft7C0APFJeJqACOJtp9sgJrrZutnlv+fRy6MXLvik\nD8hohH9E4OujZqpmqljqLY1BpHBFyQowvCLjIpz4jq86ikXyeh7oCcNe/Lz4Oabi8DejNlytg1oH\nMaLVUQbnaZjXMGRmilOKUVqQKp8Kw2ahbgGCwA/gIyK3dtdaRBRz9HMQeXXe6gxGz9gz9uXXVf9V\nHUx4i9MWHG/RliIY/u4R3SPKf6/T+U4ojSuKK4orPyTAubkzHArhCiEYPu8Y71il74fMRIyRTQ0w\n8+NUoZC88E6YRJHit0Ml0jT6G5ketiKV/VOrtB5emPbx1SaDyuo1xwBHPH9vh7zlCD+zEvqHl8BL\n+BYp8gCDABjGj4kfMRVvp/JOrvSzk0onSd8f7jwcjvW7vu/6wkHXGSs2lanxhMYgIKm3UhFhi5CJ\nQFO7yhuVN5LWa+PcBuvlKedhuyudVkKP8N7wxL4/O2U29NUH2w8wmGGDwlD6qZSvlC8WiYzgQ+8s\nn70cAZvXKq+RsbAV2YrKf8+8i3kXbHdQHtb56+BfiOxZdbXqKmk/O0R2QC9GQdcCfL762uprYvYo\nTAEOU7hjOEpbU5+lgrjYnbA7ITHCKaON8xJhGQG78dLqJexXq/mt5n8zAhJ9BUMr4jTrkiERwiL6\nHFOsBJKpi13IlMNPb4i3fsv++e7t/70J/UWVCYjZewRM9i3btwwO71RjRMiN3xu/F8PxxiAQzXY1\ng5/il+4Hxzx9Vzr+vvgj/PWPcgI5QfntWP9mjeEdmfGZcNz3Dd0Hh1lDWUNZbL+SeEnIoI7vgu2k\nl6aXsnhI7RAcaIVFCghQ1LpWC3rgrNtZlGze63WvV/kM7efH6eLsAn1XHFKM5zpELgSBW8aMwXEP\nCRsCPVJypwR+5uxOs6GH5S7IXRAjUmECfM97ojd6ZV6bv4Y/O//yfPi3TBlTBr9qtRCB7rg+caic\nifaJRqBZqYFSAwS8/JsgkJ7ZPhOVLLsm7IL+FswQzJC0/yOnjsT1KL1XCn94eJ3h6LWu5VML615o\ndwEl/fcy76F01yLEIkTSOkbLjdBqEHM+Bn7hU/en8NNNzpmIlbxN8J+A/Xsf+h4E5lLWJRCsVidb\ngaDJashyhHApb+l/MgVLKVkJKTu3kW6osd3xZgcM1wO3B7ghioOLwXg/HPsAxzbzUeYjlDYNOgXD\n4zbKDRF8OUaOEZs+ktMbqaqSAyVw8D32eeyryP5MTp6M/SkuKEYqrtvYbmPFCMhWDzjyb4+9xf6M\njB8ZL229sVPH4oIXlRbhAXDs5dir/OchHUJANLJdshFJa5nUMklqc+szVRjGS/UuoXQirW8aHAmz\nK2ZXQJi66aI28EnDJ4gc7NXei5IwfkN+Q2nrWj+1BvHKdc1FpMD/vv99SQSkpG8Jtrdh0wZEZJlG\nTKOvdaQ+2DxDathx4OIDnEIMI+Nlw/ZQpPgN8RcOt/lwmIgXqcm2jcQc/+Q/D/et7P3cO713evnS\n0gC9AL1vWaMbcDoAGcvSm6UYw+uy2GWxtO934nUCFt4txBQa79neYiVT2hnaiDx65XohUtZGu422\ntPVaqrQEQcgalQU9vLb9WhhCpjfTu/z35rjNgT5/M+sNImbOV5yvSFvXa5YXvlf6aykCJO1C24WK\nERBHcxCF/Ix87O/WLVsROGESGIlEz7iHMUpGk24m4Txt6L1BbP8sjlqgFCxzfCYcllVDVg2pyPl3\nMXeB41B8rRiOjHeSt5j+7rus77LqIiBvZ99YxGKDzUvIe4U2mdLnmGIVkLzB/VoYh7cm4W4at2I3\nd799mFBVAvKm7xv4B0WJRQiIpPRJgeOcsjNlpxj2TEHGMzMtE4Ha9zfeo4JlV/NdCGiq/aX2l9h2\nLHkg+n5r/dbiud+djwBz+4z2GRXZv62OW7nAwvXU6+hlaGAAB76VVysEdPPm5mF4xzyFeVLH2ypP\nVYY/tyFnA/zLwDaByKQq71TGce2z2YdS/0dzH2G9erb1bKWtV0uhFrZ33fY6vnfuzjkQFyUfJRAC\nQb4AAZq4FnHIvEa7RsM/UxmsMhgEpaMQgemDPgfx/WcnniFgYpRkJKaHBDsEO1CS1m8/Mh2Pgh4h\n86xXpgeiY3/CHr8rVC9EINtvpN/IipzXEaNHoDS1+EjxEbQwHHE9IkZALk7AsIGy22Uo3fdq49Wm\n+uzfN3gRoWyuLAyfTqkO57D7O4JBzVw/EzXJZ26fwYGUZpciM1H4rhApde823jgwJprBvOQpF6aA\ncZZpl8GAhu0IwwXw7eOLB8J3l+8uMdT3RabgD/c/3Ms7ECMfjXwkRkCyPbDdorNFZ7F/Lo4uUgmI\nyViT8qVLjnxHbrxcBA+RvlM2p3DDxq+LRypKc6LmRKlN3z34MKTrE9bD0KYnpANNB5vihmxzuA1S\nXrn1ckFQojOi8YD6GvkaSTzukb640ZYWLAXhKoksQaRwT+c9nSVmQGoXofRqQsaEjGq77nWferA4\n3m8geVFfLzeCARQpfjvseZdkQMgbmq9gfKjw3lQhi3kDdm+t7P3c/UN3vK+oTFSGiP2ymcu4lPgM\n3oxvQUACTQKhX3LX5UJ/mF0yk9rU2KmsEwxO4fPC5+X15uff+1T62rhNY3zu+swVgQ+frT4IwITo\nhYBYxYfGgxh8CPwAx3rN0jWIZDHOjHP59eaazkWKPWdcDpo6rc9bSx024j3JGxnm0jal2H47r3Ze\nkghI3ug8GED/uf5Sp3817NwQ+ixZPhlEYH3Z+jKxHpX9TijZfb3pNQIrR2cdBQHy1fPVk6g3+/ui\nVGzd1nU4H2U7ymBfgnsE9xAjIMP7Dq8uAvLe8tpAFu2EvUgTSK8n9DmmWAl05HDwWg7PLuUwRJ/9\n84Khz1Vyv7WtKgF5F/gO9/fVS1ehh3wFvshg+PJ8eWIo9MV0q3mH5h1CJN0mBn5Qqk8qHOmBSwai\nZJLH58Ff4r/go5dt847NeM4K3QvhpwWPCkbgw3eB7wKx53OnLwiBr5svAh8XfS5i3eL4YgSM20S2\ngZ/jttMN3ysdWDqwfEXKF49XmYdMi6KJIvSuwhMF9JbU+a0OMjPXYq6h9Pa59XOUbs25OOeiRP2x\n3hf+7MzFMxEweqLzBJnoxEaJCOhqXtW8Wr5Z+38ISIxKTPn9GrV1FPRQqUopAkIDwwaKlcCZWpqC\nwCWPSsb5Wj1zNewSs5FBi8CYemPgL77r8Q76a/fZ3fBvfZv5NpO4/6180Yu49dDWQ+X95SnZU8Qq\nhiY6ToSeLupbBGLqEOsQ+58SEFl9WTj6jSIb4QZoaNQQjrGMUKZC49Zq1amFlJFnpCd+n1+aD6Ly\n58g/4UjXFtWG4Z+6bSqah95GvIWjH68TjwssWiRC6k20QbRBDOeIUAMo2irChTyvfx776fTc6bkY\nAcn3ACMt6lWETIbDfgepU67G2oy1ESMgjCMyNYwDgyYr0RsRMj4xejEwdKqZqpkVOQ9BzkEw8Bn2\nGSipMjU2hQKwb2APZp8nzMP5fLz2MSIGIh+Rj8TjXiLCgy7aJILhvSBzAaUFs0WzRZIISOHFQjxQ\nnrqeutV3IyWu4dAogtbpUvn3RGM5h2vIm5Fd67B/rp0pM570gDSq7P3c4lELBCyyhmfB8YxYHwFD\nI+8u716V58PFzgUlqduPbEdkyXaV7SqJU7CsclECZOpo6vg1BET5kjIch4B1ASA0KaYpIA5Z/bPg\ncD9JeIKAx/W51+Hwh38IB+Eq8ivyAzGxCbGRSEBazkWTZXZ8NhwA63bW7aqFgPTIg8FcuHLhyq8h\nIL079O5QfirjffP75lLtxe8i9LyIlopAuETPRTifnvM854kRkLF9x1ZjD4iAQ9W29NmlUgUh76Oa\nQF7oOJ8QWaVR3H01oOlX94CUvcdzFTomtFJN6OZLzOGHPBv0DJUtJ1uehL6QWysH/0XmsgxKk7a/\n3Y5hRSWaJZpwzKfEoQdDtFK0UuJzGiJCCZEoTASH/PjF4/BfLKZbYLiGRxsP6JfSG6XIwHTU6KhR\nleOvq1QXwz3iS+Phj2Y1zsIwElE/UT+J+7VWxPlloSIEcER3RMh87FqzC9O8VFupwsEX3BLckkhA\nzquIBXAaeTdCyVjSoCScv82vNqPUX2a6DI7T67UXenXz3PJAyBysHazFepuveSKDW1a3DL0tCcMT\nYL9EC0ULJe7/AtGC8v6yKE2ETNYAZoBYJdLEYRPRm1cwpgD3Q7t+7fr9pwSkVlQtjFk8tewUUtMn\nrp3Agav1UOtRmXXkneVh4KJbReNC5R7PxQufjG8bI0Myfut4nJiCeQUwCJ2VOuMGEdoL4bALRwlH\niWEnIVerpyqHZnRFdUWkomR2yOyQSEAGFoExO/zp8GelCIjAkatpdOPhRjjoehA31P1h93GhtMdq\nj5VK4Mxk4XBst9sOxyR9bjocAdMlpniArZKtUEKWk5wDnHd+Hm5UYW1hbYnHPVqICKJQQYhUoEKa\nAm4kYR9hH4kE5HIhFIGnh6cHfQ8Ile9bnIZzOGUCh8x+7j7kXyMEJKuy97O6hbpF+abBZL9kOOaN\nHzau1HAGRsQgABA2LQzTq96UvoFh67KoyyKJBMQ+lwtEeJt6V4qAWHuLGSLXGFdE1kpSS9DTFv5X\nOPRWl8NdkFn99IKwWnNqwVHXD9ZHiWza+TToh7XOa50lEpAWc2FAsx9lg6BZ97HuUy0EpFceAkEL\nDyw88DUExDHSEQGtEqUS2AmvUi+cb0GuIFei3hwqRO25cLEQEUyFPAUMHREMEYiVbvVd03cNnYJF\npWZIfVJ69TvpHaozuvynHz+6Naraffn1U7AUVBXgd0VZRSGQctPpJnrJNOZqwL/hy/IxNGPj+Y3Q\nM0kNkhBobdyvMRxa4RDhEInPKV8IfSWXLIf9UjiqgFJL5h6DHgi3N24IAJc+LsUUU9eBrgOl7q8v\nD03ZRu5GCCiZuZsBtXZroSTsqvZVVNycvXoWGQwVPRU9iftFUHBSgF4I+bfyIFbyi+W5EtqjPOyn\n8ILwQkUIiGyxLF6HsGPzDkzD+lvNIhNhKmOKgHJkVCT87kttL2Gcr2qcapxYb+CD4ZiiWFpYimb+\nX4x/wfUUWggtJO4/8Q+Fx4XwuxWbK6J0TtZFVqwiaOLQidCTBbMLUOrb7kC7A/8pAZHxkUEkfu/U\nvaily2FywJjsTtudrsw6dWbXwQEljE9Aze6D7Q9g8Osq1kXzusNuB9wQxQrFcKz9xvuNr8i6Qy4P\ngYO9+ulq9EToXNG5Uq0ERNVRtfznM4/NRC9J8dFi3HA9DXsaSltPb4Iemozu6N5BBuLFlRfYP3Mt\nc0zR0szSRJPP3Zd3MTYtSi0KTVeKrxRfSe0BMbOGIxPmH4aSt/bu7d0pAaHyY0stTxIBlBU3xDJd\nCAGp/HtAgniorZ20axJS1WXqZQhkBF8JxnMqaCFoUZF1Ggc3hmOfODcRBvim1U0YZq1srexqJSB/\n/6D858EXgrk5/CtyMFSiafOmzaWtZ3bPDIY8yyALzfAhq0PQNMlYMVbfEwFp2LghIpbJ/snQfzuO\n7EDGSWApsJS2bufmnXF+QvuFwhGy7GrZlY7hpVIzRRjKoXqepE+r5T0gVSQgcpFyCAAcERxBgPZ+\nyn0MrdDN0wWx5zE8+Inj14wHoS96XATC0Kd+n/pS3zfRSwD9ML3hdPTAztOch8yJwhUF6ONm4c0w\nRehV6CtkIpbrLOeGcuzk7ZTYA1JbGYHcY0nH0GMRqR0JwqF6WRV+0Za/BXokJhmBHLNCs0KpfmxU\nHRCD5Y7Loc/GNB8DfcJfzYceFQYIAypCQD7hIKtB0Lt5e/P2YnqXziIcT1JxEgjKlPgpEnuWWz1r\nhRLb3DO5eM/H4qzF3DSuZbxl0va/T3Yf2KN1Z9ehZMs43Ti9RhOQT+g02gmR91eFr3CB4mTjwHA7\nduiIVLhGngZuPPlAeShs5Uhl3KBGzY1wgYIGB6Hn4V2td7VgiDsFIoPB38wHA1RfqY6U3LnN5/D3\nlIcpD8szO9VEVTRJqcSpgAk6mznDkKfkpWC7p8NP48ZU9VD1qFYCouWoJfaejlJLRNpS41NxY8TG\nxOLGtVK3guOiFKCEG1D3ii4emGXrlq0r38yZ7JUMw2xhY8GVPuQz2L+F0QvRC1MiW4Lz6tfTD01f\nde3qInOieFMRzZjmOeZopjrlego3dsaADMz7t3KzcqMEhMrPKNXxJnRtB22UWJ4pO8M5/K0LW2PK\nSdJ8GC7jB8aIOCnOUsRzrBKrgtrYlqtaosQq4k0EInOvW75GKYLncU9EmhgdRqdaCUhT76blP58x\nfwamOBVdKYK+6f1bb9Q28y/z8dzzV/BBTBqmN4Sh2dZ7G5q5P4g+IGOz4coG/I7RZrS/JwLyKdOx\nPnY9rkP+83yupErWE/pTc4Qmxl4qtVNC6Zh1HWuUAl8ZcAX68uH1h2huNR5gPIASECrfp977DwkI\nTw542OUw99qAlanw30zSTNLEAjOTGkMvPN76GBUuVw9fRWa2jWsb+C/Kk5Uxdlx9hjp67sYUjoF/\nWahYiMD06rLVeO5lp8oiAK6wUAElRnuj9oIIZIRnwO/rd6rfKWR6B9VCSZNqiioI0bh+4xBoKHEs\ngf4JNAxEwJjpxUAPuUS7wO8qSCqAnt/dcjf0t8lKk5Xl31+i+7fAj50RhP18ve41/Lpf3//KjRuf\nxZtVPsNQUQJiYGKA3pR7+ffgB77Mewl/9umrpwhAm2803yiRWI1TRm/e4d6Hoc8zCzLREzxw/EAE\n7tV2qyGgr3xFGfq9o05H2KGH4Q9xvq4euopekLpudd2+CwIiayMLh3lE5gj0PDx9/BSMNvtZNpjY\nlfQr6eVr9s65ncOBPb78GIYwv0k+5hRvV9yOG0srRytH0nbar24PJnn3+V0YlNzCXNyQl6Iu4YaL\nXh2Nz7MistArcm/UPaSY7C/bX5a0nkcLD9wIBSsKYIgd+jhINaQefh4owShULUTmo8vaLmvFMkLd\nZbqjqXvmBDQFZe/PRk9J8uxkXLAzq89g/xLOJoBhPqv3DM1ChzsfhkF9FPIIBs5iqYXYGDPds7r4\nfkTnCHyvMK0QD/JNuZsYG3fa5DRu1MQNiajpezniJQzs+KDxiODKHJc5Lua4OHSCQ1XAK8Dfx20f\nt50SECqUgPzDe0GuNUGJ6WnV03j+S+VL4YgmNkxERO6U7ymk9KP3R+O5T3ufBgOUXS8bz3lgRiCG\nPSjvVt4tcQqWZQAi9NlzslESZWpoKjWD2mlBJ9TuFugUwIB4TfKaVP7zZt2aYYre/Yf3EbBJaZcC\nh3v/gv0LyhvqG043UCIRYRIBPZIiSEHk8kXIC+ijyaMnjxabguU4BwY7Sz8LvXVWuVa5Uqdg7fHC\ni1eLPYsx396+1L5ULPNSYobx5jnLchChW3BiwQmp8+8/GKNX5fmo59Dv656vey7peyadTaAvz10+\nB/1f6FAIvXc17Coch9MPT3PnpXUKCGVKXAoCWO5+7tDzTEemo1iE8G0frmbdpgT2rn9K/xRKQKj8\naATE7qgdKjjK5Mug3zZ23ti5UiWnIxn08K56vgrP5euE1+g1817vjR46/q98TMNjGjLQm+5N3OH/\npW5ORYA5bUsaMg/nnM5BL8WejkVFTb5yvnJ5/9HkickTidP8FrYEEbnZ4CZKu3IicuAPivaKkEm4\ndOASHOe8jXlw4KPVolFZYpRqJPb6BbmRcjiOAMMA6OHc1Fx8/njeY7QCnLp7CtMHE64lwC4UtiiE\nP7lx80Ych9oFNbFxvcKLQvi/MdExIDbncs9Bb6rsUtklcUhREh/EZ43rGhCVjx0+IqC/w28H9JOg\nraCttOtgFWOFAHh8UDz8wDzfPNinWO1YBJTOXTyH/ckIyEBg/HHPxwhsO/Vz4no7hvLE3lM1YcUE\n+Ml5J/Kgn9u1bde2Zk3BKuSBEFgUWgB/U/8Nkf8dH3bAYJwwP4FmwPAj4UiJL7VaihST0xMn3EjK\n+5X3V2Q7pjqmMLizd85Gai3CPQIlRoeXHYYBm2s2F5FEcxNzE2nr6Ofpg1E6TXTCtKraI2tLHVdm\nIDCAYXZa6wTiUftSbYlTamTXyuJzR0VHEKoQpxA8SFGOUTDc6zLWwRHpcLcDbmCDJwY4/s69OoN5\nq1irWEtat3aD2nigRpeOhgHffWw3Sr6OnzoOhr88fjkyL532dsKDJttLtpekdTQvauLGc3JwgkE2\nCDcIpwSECiUgFSwZja4DAzIkcwgCLqFTQtE8eZx3HJ8fMTkCvfP74d8R0Wuf3B6RROEc4Rypek1g\nCv3S7UU3TIlRbqQsdSy25l1N6A8nZyf0aBi2NRQ3CAG8gPLjdYPuBKE5Mqo4Cin87XrbUdPs5uGG\nDKj6MHX0rg35MAT6+lCLQzCoU/ZM2SPm2GeZIKXfbVY3RPZUT6mekrafhk6G0H/OLs6IiGpM05gm\nFrE7pgw91r1td+x/Y8fGUpvvFV8o4vw4ZDugZKBpeFOp+kvnFx28uNE73xuRxP299yMyeNT9KOzG\nkj+W4I3AbTPaQi/z9/L3SsyE3dJGE6nzUmcEiHR5ujxKQKj8aAREQ1MDpU09/hYEMo42O1qVdYzC\njED0e9r2xFja1g9bg/Dz3/HfSeqRa6PUBj1bSyyXIBBz7P4xvD5gb529yFBOrD0RJVO6YboVejGi\n0SUj+GezDs7Ce4UOPTqEjO2h+4ewrl9bP+gb/R36O6T2KDeShx528XFBy8H6mPVw7KPuRaFkdYv7\nFuiRQTmDEDiv1aRWE4mEQomP47NbaYcMis18G2SoZRvINpC2/QauDUBAeuT2AGEx0TOp1Dj4Bt4N\nkEn37eiLgEr4y3CU9B85eATnZYHVAvjhzZKaceN++/L6SlzHogF6SLo/647EgsZijcU1egzv/+EB\nHhgn85RBLwbTk+lZneszsxlkGJjxzPhvsv/VtZ93mbvVuu4jHh4o5hpzreYcJyUgVH5sAvI/aMxD\nyQJznkEqnVnELKrReug2g+EevKm8qVK/58sgYsbz5/nX5OOp9PEbMlypxW5m93+3H5SAUKmZBKTG\nPKfkNQy8t7y31bLeKAYZU2YMM+ar1pPjcS8sfMJwGZixvLHf1XkdwKC0lAlmgmvOfn1LAkLxJ0JK\nQKj8ZASEIkVKQKhQAkKRIiUgFCkBoUKFEhCKlIBQoQSEIkVKQChSAkKFCiUgFClSAkKFEhCKFCkB\noUgJCBVKQChSpASEyg9CQDzpc0bx+8E0F0pAKFICQoUSEIoUKQGh8l3rvd71SWQ5n+CfFCnWPExT\n4TA+7f8B+P15KNMCkjoAAAAldEVYdGRhdGU6Y3JlYXRlADIwMTctMTAtMjRUMjI6Mzc6NDMrMDA6\nMDA1ZJ5wAAAAJXRFWHRkYXRlOm1vZGlmeQAyMDE3LTEwLTI0VDIyOjM3OjQzKzAwOjAwRDkmzAAA\nABR0RVh0cGRmOlZlcnNpb24AUERGLTEuNSAFXAs5AAAAAElFTkSuQmCC\n"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%tikz\n",
    "\\tikzset{every node/.style={font=\\sffamily,white}} \n",
    "\\node[fill=blue] at (0,0) (a) {Sender}; \n",
    "\\node[fill=blue] at (3,0) (b) {Channel}; \n",
    "\\node[fill=blue] at (6,0) (c) {Receiver}; \n",
    "\\draw[->] (a) -- (b) node [midway,above,font=\\scriptsize,black]{$p(\\mathbf{t})$}; \n",
    "\\draw[->] (b) -- (c) node [midway,above,font=\\scriptsize,black]{$p(\\mathbf{s}|\\mathbf{t})$};"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* A message $\\target$ is sent through a noisy channel and $\\source$ is received\n",
    "* What was $\\target$?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAAnEAYAAABxl8L2AAAJJGlDQ1BpY2MAAHjalZVnUJNZF8fv\n8zzphUASQodQQ5EqJYCUEFoo0quoQOidUEVsiLgCK4qINEUQUUDBVSmyVkSxsCgoYkE3yCKgrBtX\nERWUF/Sd0Xnf2Q/7n7n3/OY/Z+4995wPFwCCOFgSvLQnJqULvJ3smIFBwUzwg8L4aSkcT0838I96\nPwyg5XhvBfj3IkREpvGX4sLSyuWnCNIBgLKXWDMrPWWZDy8xPTz+K59dZsFSgUt8Y5mjv/Ho15xv\nLPqa4+vNXXoVCgAcKfoHDv+B/3vvslQ4gvTYqMhspk9yVHpWmCCSmbbcCR6Xy/QUJEfFJkT+UPC/\nSv4HpUdmpy9HbnLKBkFsdEw68/8ONTIwNATfZ/HW62uPIUb//85nWd+95HoA2LMAIHu+e+GVAHTu\nAED68XdPbamvlHwAOu7wMwSZ3zzU8oYGBEABdCADFIEq0AS6wAiYAUtgCxyAC/AAviAIrAN8EAMS\ngQBkgVywDRSAIrAH7AdVoBY0gCbQCk6DTnAeXAHXwW1wFwyDJ0AIJsArIALvwTwEQViIDNEgGUgJ\nUod0ICOIDVlDDpAb5A0FQaFQNJQEZUC50HaoCCqFqqA6qAn6BToHXYFuQoPQI2gMmob+hj7BCEyC\n6bACrAHrw2yYA7vCvvBaOBpOhXPgfHg3XAHXwyfgDvgKfBsehoXwK3gWAQgRYSDKiC7CRriIBxKM\nRCECZDNSiJQj9Ugr0o30IfcQITKDfERhUDQUE6WLskQ5o/xQfFQqajOqGFWFOo7qQPWi7qHGUCLU\nFzQZLY/WQVugeehAdDQ6C12ALkc3otvR19DD6An0ewwGw8CwMGYYZ0wQJg6zEVOMOYhpw1zGDGLG\nMbNYLFYGq4O1wnpgw7Dp2AJsJfYE9hJ2CDuB/YAj4pRwRjhHXDAuCZeHK8c14y7ihnCTuHm8OF4d\nb4H3wEfgN+BL8A34bvwd/AR+niBBYBGsCL6EOMI2QgWhlXCNMEp4SyQSVYjmRC9iLHErsYJ4iniD\nOEb8SKKStElcUggpg7SbdIx0mfSI9JZMJmuQbcnB5HTybnIT+Sr5GfmDGE1MT4wnFiG2RaxarENs\nSOw1BU9Rp3Ao6yg5lHLKGcodyow4XlxDnCseJr5ZvFr8nPiI+KwETcJQwkMiUaJYolnipsQUFUvV\noDpQI6j51CPUq9RxGkJTpXFpfNp2WgPtGm2CjqGz6Dx6HL2IfpI+QBdJUiWNJf0lsyWrJS9IChkI\nQ4PBYyQwShinGQ8Yn6QUpDhSkVK7pFqlhqTmpOWkbaUjpQul26SHpT/JMGUcZOJl9sp0yjyVRclq\ny3rJZskekr0mOyNHl7OU48sVyp2WeywPy2vLe8tvlD8i3y8/q6Co4KSQolCpcFVhRpGhaKsYp1im\neFFxWommZK0Uq1SmdEnpJVOSyWEmMCuYvUyRsryys3KGcp3ygPK8CkvFTyVPpU3lqSpBla0apVqm\n2qMqUlNSc1fLVWtRe6yOV2erx6gfUO9Tn9NgaQRo7NTo1JhiSbN4rBxWC2tUk6xpo5mqWa95Xwuj\nxdaK1zqodVcb1jbRjtGu1r6jA+uY6sTqHNQZXIFeYb4iaUX9ihFdki5HN1O3RXdMj6Hnppen16n3\nWl9NP1h/r36f/hcDE4MEgwaDJ4ZUQxfDPMNuw7+NtI34RtVG91eSVzqu3LKya+UbYx3jSONDxg9N\naCbuJjtNekw+m5qZCkxbTafN1MxCzWrMRth0tie7mH3DHG1uZ77F/Lz5RwtTi3SL0xZ/Wepaxls2\nW06tYq2KXNWwatxKxSrMqs5KaM20DrU+bC20UbYJs6m3eW6rahth22g7ydHixHFOcF7bGdgJ7Nrt\n5rgW3E3cy/aIvZN9of2AA9XBz6HK4ZmjimO0Y4ujyMnEaaPTZWe0s6vzXucRngKPz2viiVzMXDa5\n9LqSXH1cq1yfu2m7Cdy63WF3F/d97qOr1Vcnre70AB48j30eTz1Znqmev3phvDy9qr1eeBt653r3\n+dB81vs0+7z3tfMt8X3ip+mX4dfjT/EP8W/ynwuwDygNEAbqB24KvB0kGxQb1BWMDfYPbgyeXeOw\nZv+aiRCTkIKQB2tZa7PX3lwnuy5h3YX1lPVh68+EokMDQptDF8I8wurDZsN54TXhIj6Xf4D/KsI2\noixiOtIqsjRyMsoqqjRqKtoqel/0dIxNTHnMTCw3tir2TZxzXG3cXLxH/LH4xYSAhLZEXGJo4rkk\nalJ8Um+yYnJ28mCKTkpBijDVInV/qkjgKmhMg9LWpnWl05c+xf4MzYwdGWOZ1pnVmR+y/LPOZEtk\nJ2X3b9DesGvDZI5jztGNqI38jT25yrnbcsc2cTbVbYY2h2/u2aK6JX/LxFanrce3EbbFb/stzyCv\nNO/d9oDt3fkK+Vvzx3c47WgpECsQFIzstNxZ+xPqp9ifBnat3FW560thROGtIoOi8qKFYn7xrZ8N\nf674eXF31O6BEtOSQ3swe5L2PNhrs/d4qURpTun4Pvd9HWXMssKyd/vX779Zblxee4BwIOOAsMKt\noqtSrXJP5UJVTNVwtV11W418za6auYMRB4cO2R5qrVWoLar9dDj28MM6p7qOeo368iOYI5lHXjT4\nN/QdZR9tapRtLGr8fCzpmPC49/HeJrOmpmb55pIWuCWjZfpEyIm7J+1PdrXqtta1MdqKToFTGade\n/hL6y4PTrqd7zrDPtJ5VP1vTTmsv7IA6NnSIOmM6hV1BXYPnXM71dFt2t/+q9+ux88rnqy9IXii5\nSLiYf3HxUs6l2cspl2euRF8Z71nf8+Rq4NX7vV69A9dcr9247nj9ah+n79INqxvnb1rcPHeLfavz\ntuntjn6T/vbfTH5rHzAd6Lhjdqfrrvnd7sFVgxeHbIau3LO/d/0+7/7t4dXDgw/8HjwcCRkRPox4\nOPUo4dGbx5mP559sHUWPFj4Vf1r+TP5Z/e9av7cJTYUXxuzH+p/7PH8yzh9/9UfaHwsT+S/IL8on\nlSabpoymzk87Tt99ueblxKuUV/MzBX9K/FnzWvP12b9s/+oXBYom3gjeLP5d/Fbm7bF3xu96Zj1n\nn71PfD8/V/hB5sPxj+yPfZ8CPk3OZy1gFyo+a33u/uL6ZXQxcXHxPy6ikLxyKdSVAAAAIGNIUk0A\nAHomAACAhAAA+gAAAIDoAAB1MAAA6mAAADqYAAAXcJy6UTwAAAAGYktHRP///////wlY99wAAAAJ\ncEhZcwAAASwAAAEsAHOI6VIAAAAHdElNRQfhChgWJSuSNfMzAAAv9klEQVR42u19B1zN+///+Zw6\np61FGlRKNIwoo2EmVMZFCNluVOaNZI/KRdkZyV7XioRsHVRmuvY1QkOD9jTC/36en7ffv+N7nFvJ\nveH9uo+H5+Wc8/7s1+v1fK0Pj0eFChUqVKhQoUKFChUqVKhQoUKFChUqVKhQoUKFChUqVKhQoUKF\nChUqVKhQoUKFChUqVKhQoUKFChUqVKhQoUKFChUqVKhQoUKlpgnTgvyPWxUXaEywhJ5LKlSo1DD9\nNoT8T5uvXEiG4AmCLapp/0zoNaJChQoVKj+RGLzjsNt2DgX+VVuHP4VDG1sOjZPpuaVChcp/KyYD\nOOwygvCHSV9JFFpzOHgkh3qvqmc/zVZwaL2X/MNZeu2oUKFChcoPKPU3c/gbyXgY7yQffJ4BecCB\n0hZCNOp8YUGSAalHCIzPew71Q+m5pkKFyr8rDUlGYcpzou9yyQdfmQFhLnC4ZA2HFg7Sv6+UwaHg\nH9ZVacmhVwjZzU30GlKhQoUKlR9IBIYcTl/HYZdw6d/XeMThWksOtdZWbDsORWQ7PTiUnU7PPRUq\nVL6tyGtwOLchh7YR1bs+c4rD3305NN8u+XsywRyu0iWE4tcKEid9sn46h5pCek2pUKFChUoNEuaX\nLxi+f4jwWZoSw5jEoeLcL3zxGgf91Di8I8uhLsmcMF7ke70k/1zpGCEupzlsOoReMypUqFRQv/l9\nQb9ZS/+dbTyHQbM4FG6o2Pb4iwkWVpKA7JD8vVYXOXxMsCPJJDNkv3i/fWE/BnM4y5Lo3zB6L1Ch\nQoUKlf9QZBQ4bEsibl4LiGEjBtf+DTFYpQTVOVS4IL6ONzHgv5VK3o6QrOtuwOF9EkksbMvhljgO\nF5OagvpK0vd7JsmYjLpFryEVKlS+4HiTUqXWDTj0jOCwiyYhFiSw0vcFh66JHCpbiK/jSxx9j3jp\n21NV43AYcfB993E4jei5QJL57XL2HwhIT7L/ZL97ET13iTSnvycBn0Ok92R5Iw6b/EOJVR9S6rp4\nG9H/7vQeoUKlmuXjR4oUqw9/XDEihrgnaX70IdOmjupxaDKKw0+9GivyicH+LEMRQgz2L4++QHQm\nc2ixnMOLSzjM0OKwW1MOm5NInqKB9P0eRAjL0nlUn1GkSPWkZNEnvWN9iX7yJkQjijR9m5YRfUP0\nyBLSezHoICEGTTjcTBz3Ll7Stzd8NIeHnxJCsIcEVFI49CO9a+7MPxAQUlrKeBA9TXCjPIfvAjgc\nQ4Z9tCD6WfWk9P2zNORwDyndUjKneo8ixWrWgx8/+ogoUqw6jrHmUO3cj+3aGpDmcEMS+Vs2jsOZ\nCp9FEsdwuOlTKr+h+Of7SMlVBw3p25MhBCXiHodJxHDW6Vq5/e5BHImtXXk/gXz8yMzgsGshfT4p\n1hwcPIdD+YY188nRI5kGI1LqGUgIycI/PyMAlhyuJhmGQEJMZK8QfZXGYcuZ0rfnTvRaOnHso1tx\nuP4lIQwko6E74bPtn/mMgOySvP6cbA7fkoyzXSUJhBE5/ijiKqluqNl6z+A+h+6hBGdRpFgD8QiH\nfS+TG5fjIhQpVg2TlDg0Wc/7KURlIIdHpnHo+Eb8cw0yjSr6KocDPjN8O4khcwiUvh0hiSQeeUsI\nCOkZ0SKZFSUXDuu+l75Ob+JYhHb/OQiITG8Ot7rR55NizcHrNhxqdKvZT5ASGeN9sDOHPVeKf16L\nEIzTJEM79AgJvJASrQNTOWxdJn07tYljP86VBGYIJpASqgxSOjWdnC9mKcH1nxGQLzShzyPE5C35\nvT3R26qRHGqekb5/Jvs5jCSErJZ7zdZ7A/w5fDeYYG+KFGsgjiZ+ozMlIBSrARM3EQIi+3MQEIu6\nHF4mU6wafBbp67+Iw70HiKH9rATLnxjS4WOlb0dAMiWHyRz9F6QUS5tkWLoTgzh3ifR1xhJCNPP3\nn4SAdOZwUyZ9PinWHLxiQAhIDX+CjGtzGOPIYZMk8c9dSI/H/m0kAHKcfED00jKi9/r9gz0YlsWh\nB9Fz8qQpXI2M7w0kBGMf0bMCMqacsa8YAZlN9HIZKclqTwjJMD6Hnv+QEWlHxvxuJKWucgdqtt5z\n60WfM4rfD6YJKQGhSAlIpcWNlFzFx3I4kBjgrmTc48zdHJqNkvz7Tz0h/r8Qg/qFJvJPb+YdtpHD\nW2ocfppqteZTrfUXInmMHIdLiAF3+YsSEIoUKQH5h4wpyUAkRJMMBymJ6kKGWfiS3rdm8l8gFrM5\n9PvUA6Iv+Xu/EX12lrxAddJwcUKynDj+PT6NKbcheu1YxQiILZkWeL0ZIUykd2Q9eR9JWxnp52FU\nFIdTH5J/yKvhBMSQPmcUvx98EUwJCEVKQCosDGm2DCbEYeEHkgEx5LCRNylhGCB9nbpk2ssq0jxZ\n/x/G48oYcWjYl0MrEjk0IE2UjJHk3+mTUq01Thxq/SRN6JSAUKQEpApC9IU/CagEEQJiSIZuNLrJ\noXK69GUMSOBkJdFTWvUkf6826SHRJxncJuR7Lcj4cV0SQPl8zHhF3wPCI/pOl2QyrEhJl8GnMenH\nJf9MkfTuBSmT3935PvQeJSAUKQGhSAnIjyZk+pQKKXmKIgasz5gqrkcyJb1JpsTjMIfV9aJAPik1\nGEGaL3vr8n4ioQSEIiUglRKSgVUi7984TKZgDe5QxfVIj4UbyVQMIxkO/rJqCgR9IiBkKId5ZPWe\njk4xHHqTDI1sISUgFCnWYAKiHaMdw+JQ3lD8fWW/lf1Y3Pq3sLj8w/IPLA72GezDos4rnVc/8wXQ\nT9RPZNGqk1UnFhXWK6ynBKSmipDURLvZcXikDykZIM2XqlU01EIy3rE/iQB2JVNh+FU0eAyJFLYi\nJQf9MzmU+ykyH9+SgChEKUSx2Nm0symLc7vO7cri5pDNISyGbg7dzOIU6ynWLDbzaObBIv8g/6C0\ndeuPqT+GReub1jdZlM+Sz6KGqeaghq2GLYuto1tHs6ipo6nz4xEQGdIT0Y9kJA6RDMe0+hxqzq/a\nugrEgR9Mpvl1JPqTuf+VBISURgWQzK9Zz+o5D4YkoDSc9LhoLvm+9F7VCYhKVxXos9bprdNZbLup\n7SZgbNtYadh6fWv4LcaDjAexKIgSRFG98f+Rmc/MZ9Ei2gL6w9zO3I5FGQUZBXp+qoGA2HnZebEY\nqxyrzGJR+6L2LD6a92gei9dGXBvB4oPCB4UsFpwrOMfi5aGXh7JoG2YbhvVkeDI/0wVYVLKohMXn\nxs+NWTTLM8ujBKTGymOSmielUoqEkCgRIsI3+brlBWR8sRGZHiNzsIqGeRuHemQ+v1ws7yeU6iQg\n+qb6IBxb3m55y2Ju09ymLCb/lvwbi9dfXn/J4p29d/bic5NcE3x+N/kui55XPa/CMG8WbJa0/vzf\n5//OYtrLNKzTWKaxDDVMNQd7H+99HNd1bO5YFvv59/P/ATMg44leMyRIMqdKpHSTf/7rlpcjhKMB\nCYjwnb5yf8n0QJMbZD+Dquc01CHj1VWDvk+9V3UCYqtsC/8tzzoPAZRi9WJ1FvPv5kOP5WfnZ4vh\np383zDdk8enup7tZ3LR602oWTfgmfKo/Pn4UHhQiAHXR96Ivi2fOnYH/qzJHZQ49P19BQOqY1TFj\n8ULkhUgWs1tkt4DB/fs/VgzqGNRhUW2W2ixE+lrXb83iyKYjYcDTLNMsWYyZGzOXRa1Yrdif6QIE\nywfLs5gRlBEEhuxv4U8JCBUqNYOAqF5QvcDiH+F/hLNY8kvJL3huRwePxvPKt4CBVc9Xz0cGeKX2\nShYd/v6PFVG8KB6O6+vc1yz2WNpjqaTtBA4KROQwd3UuDLfpXNO51DDVHOwb2Rf27fXI1yNZHLBr\nwK4fvwmdys9GQOxt7G1YLFtVtorFsxPPTsT9f7HvRRDvaf2mlcdP/z4iYUQCiwdnHpyJ398pu8Pi\n3oi9ESwqC5WFP7P+kD0te5rFJTeW3GAxIC4gDhl1XwVfql+/goB0edrlKRTznNdgchsbb2yMkoPz\n/PNSf6/CU2Fx1u1Zt1l82/4tMiYupi6mFUoVClQEiExO0Z8C/Ps//LuXitfXnBAZTxkwJ53lOstZ\nNFQ2RERAZb/K/kql3DozncsTKsMYQ5SmaTbRbFL+e0GOQY4gIO0y2sGh8bHwkcqkWwhB8HTX6K5h\nsYFCA6TwNAdpDvqa4+YX84tZ1PPRw/ZVbVRtKAGh8tMSED2eHoujFUbj+XrzyxsQj7Wz1iKQotBG\noU1F1ml9tPVRBGb0srFe5LDIYSwqrlFcI0ZAzAIRyMl1ynVCBkS2sSwMl4os9KROpk4m9Mh+Q+gh\nDZ5G5UoAFBlFFmsfq30M66QbosRCP1Yf+knNRM3kqwzsbVnocb3VeiBQBlEGKMFQuapy9WvW1XTT\ndMP+phimsFjXuC4yxYwao1aZdeTuyMEhqjeyHgiEoZMhzrO6q7prhQhISF+U2L2Wf42A0YDAAYGU\ngFD54QiIsT2er/eF71GpslF3o25lfq91TguR/dgLsQjc5DzOecyi5QHLAxX5vdoaNehFQ29Dbzyv\nQ+uhQkb+nPy5qhyPUoZSBvTcOP1xwJb6LfHv9ZTqVUp/+jDwi+rsrLMTfpdVAysEnN5pv4PfeEnm\nktR1THjQr7JtZdvi+/kyCFjxuvG6/RuOvmKCIghi/fX1USpncMbgDIu1xtQa8zXrChYKFuI6rai3\nAkTTS9nrXyUgrtauSNV9kqC3QW8r8/umU5tOZTGsUVgj1FbHdpaYAZEdLjucxT42feAYnxp4aiBK\nHGKT8f1U01QQl9NTToOQuAa7BuN3XrIST4hdvB0ik4tfLUYPipm9mT2L/o/98cDc176vjQxN/bT6\n2J7FKQsQrkldJuGG7MP0kdgDE6cNZrvg3QLcmLdVb6uymH4y/SSLV/dc3YMemX5D0RuzsvFKELb0\nW+m3pBEQy/6W/dFL47MVnz+a9ggRiIx+GVjnWvtrIHDTrKfheqjbqttKWqeRViMtXKfbQXAYWvZo\n2YNFr5Ne2L/EnMQcZLD+FkpAqHwfYkFKMJokf0ZAelSVgCg1VsJzeWbjmY14Pueno4a3mX4z/Uql\n3hcKoaDXXl57mcUo1yg4vLqtdFuJERCLQOiX7AfZD1hs/7Q9AjvBnsF4Du+/uP8C+zEtHc99nFKc\nEotDzIaAuMjWk5VoUHXW68DgLJy+cDqL8QPiB2CdtPQ0FlMCUgJQGrDu4joWxwWOg2OtMFthtqT1\nnHY6wQDPT52fCr3pbQZHYfnR5SBaj3Y9QmbgxbwXKL09aXwSDk0XQRcEjHiveGI9f9pztBG4WnJv\nyT0c97b221h083TDcV+ZfQX7kd41HbXpt61vQ79NV5kOYqZiqyJRz/Hm8LCubQtbBGz+mPcH9ufp\n6KfIXKVdSIODFLs0FhkprwAvnAelcKVwSkCo1GzRIFO8Oq3mkN/oMwLS4KsJSPL7ZPhlY8Kq5KCG\nvgl9w+IHhw9IBbd72O6hRMc4TBGl9x6vPKAXYmRjEHhJK0wDAXo64+kMFveb7IcDb//E/kn5gMr/\nBFJH8UdBTxU4FbB4bOsx9B4nb0uGXkn2SYb/dDT+KPy/7vHdgfzx/PES/TkrbRANf0V/bC9hRQIc\n7YxVGcgQPVB+gAD1mp1roBcbtWzUUmJg21YGemqYYBj04OCcwfCz5G7Jwe8bunsoStcWb16MEl3d\nEbojpAbKD8qgpGvsr2N/xf5N9YcfXdu6NvQjvwO/A4vdNnVDD0+kfST82+cvn6PE98UfL/5g8bz5\neXPYkalD8HvhYeFhSdvraNoR/vXvHX/viNK6YyYIZC3xWIIex2clz9BK0D+9f/q/SkDaCtrihOYE\n58Dhf2b4DLWAQy2HorRK66EWbjxmH7NP6gk9JwOGy9xgboh9vp+HiN+Qk0PgIGdnZaM58+aUmyAa\n8/3no2TJ38kfEa27qnfh8GenZ+NEDJ0zFIaImcHMKL+uJ88TWLy6GBG7Xdt3bWdxZ9xOEIhhl4fB\nYZifNx89GTkbc+CIJMxOgEGsp1ZPLAKnXKiMB2ab1jY4+K//FjwA9Y+BwPiZ+OEBWt91PWdIX93G\nA/dw1UPcyKlhqWGSCEiTpU1gIP90+dMF3xuSOgTERX0lajOnp06HI3Bi1Ak8eCVyJXIshpwOQcpP\ncYyimAJxUHfA74r+KvoLhKb21tos3nt9D/t7wvAErl+3h90eVoGAhHHY8Bg1ElT+PbHN5nDdOw7r\naHH3IcMnBORlZfVak3dNEEDIiMlA5vL4mON4jhRiFapUIipnJgeioLRJCQaBP5Q/tPznAWEBeP6L\nw4qBJ9+ffM/i9qDtKM0cNnnYZOiRX/yQiUm8mojMQoZsBgy2zVUbsUyD3AS5CXAEBoYiUFO6sBRE\n6OCBg4hETus0DUMv5i+cj39PGJOA4yv+vRi9KAOcBzhjPXmefPl1F+gtQCYnq3lWcxb3jd8Hw701\naCv2c2jPoT3R29Z/EQImuVtyt2D99ATo43rv670vv56FgYUBasjt8tGUeWTYEWSIIiMiUboxcdHE\nRTC0z8Y+Q6BlybUl0K/vXuP6uLdxl5iJsltqB735xPkJjiPRJBH6d/G9xSA6s1rMAjER5YsQiSx2\nKYZ+XSC/AMcraCJo8g0JiB6H6tvps0ul8qLZhMP1JOBiIy9OQAbmfjUByXoPP2tj2MawSum5pnIo\nrT+y7ggCGpkZmchAWDS0aCgWOQ8WwF+cO34u9EdB0wL87nThafhRfpf8kFFYFrsM+jbZNRmBm6eu\nT4HtNdprSNp+tznd4O+lXUrD7xOsEkAg5lnPg2O+yGAR9E2ibiIyO6l1U+tivSntp4hlYkRqIhb3\nnNwDvzNvaR70yZ7FexazODV9KvTZlnVbcJx58XkgMhdtL4JoGGYZig0REd4RIgMbtz0Ofmb05Wj4\nl0ojlEA0JttOxu8+eH5A4MV9pvtMaefZONc4l8WkV0nwI4+/Pw69qtBPAQHpHkN7wL6kN0mHHrvj\newelXgtDF4bifKjNg/96w/4GiEnBrAJk9sfHjYf/K2MsY1x+e5MPTwYxyR+TDzuxxXgLPn+09BHO\nS/i6cJwHW1Nb03+VgCikKsABDu4YDGZUalWKC/7O7h0Myi3FW2COoe6h7uj9UBiJkobml5vjAiib\nK5tLW19ntw6Y4Z/MnwyaPe9dhwFpeKfhHUnft7C0APFJeJqACOJtp9sgJrrZutnlv+fRy6MXLvik\nD8hohH9E4OujZqpmqljqLY1BpHBFyQowvCLjIpz4jq86ikXyeh7oCcNe/Lz4Oabi8DejNlytg1oH\nMaLVUQbnaZjXMGRmilOKUVqQKp8Kw2ahbgGCwA/gIyK3dtdaRBRz9HMQeXXe6gxGz9gz9uXXVf9V\nHUx4i9MWHG/RliIY/u4R3SPKf6/T+U4ojSuKK4orPyTAubkzHArhCiEYPu8Y71il74fMRIyRTQ0w\n8+NUoZC88E6YRJHit0Ml0jT6G5ketiKV/VOrtB5emPbx1SaDyuo1xwBHPH9vh7zlCD+zEvqHl8BL\n+BYp8gCDABjGj4kfMRVvp/JOrvSzk0onSd8f7jwcjvW7vu/6wkHXGSs2lanxhMYgIKm3UhFhi5CJ\nQFO7yhuVN5LWa+PcBuvlKedhuyudVkKP8N7wxL4/O2U29NUH2w8wmGGDwlD6qZSvlC8WiYzgQ+8s\nn70cAZvXKq+RsbAV2YrKf8+8i3kXbHdQHtb56+BfiOxZdbXqKmk/O0R2QC9GQdcCfL762uprYvYo\nTAEOU7hjOEpbU5+lgrjYnbA7ITHCKaON8xJhGQG78dLqJexXq/mt5n8zAhJ9BUMr4jTrkiERwiL6\nHFOsBJKpi13IlMNPb4i3fsv++e7t/70J/UWVCYjZewRM9i3btwwO71RjRMiN3xu/F8PxxiAQzXY1\ng5/il+4Hxzx9Vzr+vvgj/PWPcgI5QfntWP9mjeEdmfGZcNz3Dd0Hh1lDWUNZbL+SeEnIoI7vgu2k\nl6aXsnhI7RAcaIVFCghQ1LpWC3rgrNtZlGze63WvV/kM7efH6eLsAn1XHFKM5zpELgSBW8aMwXEP\nCRsCPVJypwR+5uxOs6GH5S7IXRAjUmECfM97ojd6ZV6bv4Y/O//yfPi3TBlTBr9qtRCB7rg+caic\nifaJRqBZqYFSAwS8/JsgkJ7ZPhOVLLsm7IL+FswQzJC0/yOnjsT1KL1XCn94eJ3h6LWu5VML615o\ndwEl/fcy76F01yLEIkTSOkbLjdBqEHM+Bn7hU/en8NNNzpmIlbxN8J+A/Xsf+h4E5lLWJRCsVidb\ngaDJashyhHApb+l/MgVLKVkJKTu3kW6osd3xZgcM1wO3B7ghioOLwXg/HPsAxzbzUeYjlDYNOgXD\n4zbKDRF8OUaOEZs+ktMbqaqSAyVw8D32eeyryP5MTp6M/SkuKEYqrtvYbmPFCMhWDzjyb4+9xf6M\njB8ZL229sVPH4oIXlRbhAXDs5dir/OchHUJANLJdshFJa5nUMklqc+szVRjGS/UuoXQirW8aHAmz\nK2ZXQJi66aI28EnDJ4gc7NXei5IwfkN+Q2nrWj+1BvHKdc1FpMD/vv99SQSkpG8Jtrdh0wZEZJlG\nTKOvdaQ+2DxDathx4OIDnEIMI+Nlw/ZQpPgN8RcOt/lwmIgXqcm2jcQc/+Q/D/et7P3cO713evnS\n0gC9AL1vWaMbcDoAGcvSm6UYw+uy2GWxtO934nUCFt4txBQa79neYiVT2hnaiDx65XohUtZGu422\ntPVaqrQEQcgalQU9vLb9WhhCpjfTu/z35rjNgT5/M+sNImbOV5yvSFvXa5YXvlf6aykCJO1C24WK\nERBHcxCF/Ix87O/WLVsROGESGIlEz7iHMUpGk24m4Txt6L1BbP8sjlqgFCxzfCYcllVDVg2pyPl3\nMXeB41B8rRiOjHeSt5j+7rus77LqIiBvZ99YxGKDzUvIe4U2mdLnmGIVkLzB/VoYh7cm4W4at2I3\nd799mFBVAvKm7xv4B0WJRQiIpPRJgeOcsjNlpxj2TEHGMzMtE4Ha9zfeo4JlV/NdCGiq/aX2l9h2\nLHkg+n5r/dbiud+djwBz+4z2GRXZv62OW7nAwvXU6+hlaGAAB76VVysEdPPm5mF4xzyFeVLH2ypP\nVYY/tyFnA/zLwDaByKQq71TGce2z2YdS/0dzH2G9erb1bKWtV0uhFrZ33fY6vnfuzjkQFyUfJRAC\nQb4AAZq4FnHIvEa7RsM/UxmsMhgEpaMQgemDPgfx/WcnniFgYpRkJKaHBDsEO1CS1m8/Mh2Pgh4h\n86xXpgeiY3/CHr8rVC9EINtvpN/IipzXEaNHoDS1+EjxEbQwHHE9IkZALk7AsIGy22Uo3fdq49Wm\n+uzfN3gRoWyuLAyfTqkO57D7O4JBzVw/EzXJZ26fwYGUZpciM1H4rhApde823jgwJprBvOQpF6aA\ncZZpl8GAhu0IwwXw7eOLB8J3l+8uMdT3RabgD/c/3Ms7ECMfjXwkRkCyPbDdorNFZ7F/Lo4uUgmI\nyViT8qVLjnxHbrxcBA+RvlM2p3DDxq+LRypKc6LmRKlN3z34MKTrE9bD0KYnpANNB5vihmxzuA1S\nXrn1ckFQojOi8YD6GvkaSTzukb640ZYWLAXhKoksQaRwT+c9nSVmQGoXofRqQsaEjGq77nWferA4\n3m8geVFfLzeCARQpfjvseZdkQMgbmq9gfKjw3lQhi3kDdm+t7P3c/UN3vK+oTFSGiP2ymcu4lPgM\n3oxvQUACTQKhX3LX5UJ/mF0yk9rU2KmsEwxO4fPC5+X15uff+1T62rhNY3zu+swVgQ+frT4IwITo\nhYBYxYfGgxh8CPwAx3rN0jWIZDHOjHP59eaazkWKPWdcDpo6rc9bSx024j3JGxnm0jal2H47r3Ze\nkghI3ug8GED/uf5Sp3817NwQ+ixZPhlEYH3Z+jKxHpX9TijZfb3pNQIrR2cdBQHy1fPVk6g3+/ui\nVGzd1nU4H2U7ymBfgnsE9xAjIMP7Dq8uAvLe8tpAFu2EvUgTSK8n9DmmWAl05HDwWg7PLuUwRJ/9\n84Khz1Vyv7WtKgF5F/gO9/fVS1ehh3wFvshg+PJ8eWIo9MV0q3mH5h1CJN0mBn5Qqk8qHOmBSwai\nZJLH58Ff4r/go5dt847NeM4K3QvhpwWPCkbgw3eB7wKx53OnLwiBr5svAh8XfS5i3eL4YgSM20S2\ngZ/jttMN3ysdWDqwfEXKF49XmYdMi6KJIvSuwhMF9JbU+a0OMjPXYq6h9Pa59XOUbs25OOeiRP2x\n3hf+7MzFMxEweqLzBJnoxEaJCOhqXtW8Wr5Z+38ISIxKTPn9GrV1FPRQqUopAkIDwwaKlcCZWpqC\nwCWPSsb5Wj1zNewSs5FBi8CYemPgL77r8Q76a/fZ3fBvfZv5NpO4/6180Yu49dDWQ+X95SnZU8Qq\nhiY6ToSeLupbBGLqEOsQ+58SEFl9WTj6jSIb4QZoaNQQjrGMUKZC49Zq1amFlJFnpCd+n1+aD6Ly\n58g/4UjXFtWG4Z+6bSqah95GvIWjH68TjwssWiRC6k20QbRBDOeIUAMo2irChTyvfx776fTc6bkY\nAcn3ACMt6lWETIbDfgepU67G2oy1ESMgjCMyNYwDgyYr0RsRMj4xejEwdKqZqpkVOQ9BzkEw8Bn2\nGSipMjU2hQKwb2APZp8nzMP5fLz2MSIGIh+Rj8TjXiLCgy7aJILhvSBzAaUFs0WzRZIISOHFQjxQ\nnrqeutV3IyWu4dAogtbpUvn3RGM5h2vIm5Fd67B/rp0pM570gDSq7P3c4lELBCyyhmfB8YxYHwFD\nI+8u716V58PFzgUlqduPbEdkyXaV7SqJU7CsclECZOpo6vg1BET5kjIch4B1ASA0KaYpIA5Z/bPg\ncD9JeIKAx/W51+Hwh38IB+Eq8ivyAzGxCbGRSEBazkWTZXZ8NhwA63bW7aqFgPTIg8FcuHLhyq8h\nIL079O5QfirjffP75lLtxe8i9LyIlopAuETPRTifnvM854kRkLF9x1ZjD4iAQ9W29NmlUgUh76Oa\nQF7oOJ8QWaVR3H01oOlX94CUvcdzFTomtFJN6OZLzOGHPBv0DJUtJ1uehL6QWysH/0XmsgxKk7a/\n3Y5hRSWaJZpwzKfEoQdDtFK0UuJzGiJCCZEoTASH/PjF4/BfLKZbYLiGRxsP6JfSG6XIwHTU6KhR\nleOvq1QXwz3iS+Phj2Y1zsIwElE/UT+J+7VWxPlloSIEcER3RMh87FqzC9O8VFupwsEX3BLckkhA\nzquIBXAaeTdCyVjSoCScv82vNqPUX2a6DI7T67UXenXz3PJAyBysHazFepuveSKDW1a3DL0tCcMT\nYL9EC0ULJe7/AtGC8v6yKE2ETNYAZoBYJdLEYRPRm1cwpgD3Q7t+7fr9pwSkVlQtjFk8tewUUtMn\nrp3Agav1UOtRmXXkneVh4KJbReNC5R7PxQufjG8bI0Myfut4nJiCeQUwCJ2VOuMGEdoL4bALRwlH\niWEnIVerpyqHZnRFdUWkomR2yOyQSEAGFoExO/zp8GelCIjAkatpdOPhRjjoehA31P1h93GhtMdq\nj5VK4Mxk4XBst9sOxyR9bjocAdMlpniArZKtUEKWk5wDnHd+Hm5UYW1hbYnHPVqICKJQQYhUoEKa\nAm4kYR9hH4kE5HIhFIGnh6cHfQ8Ile9bnIZzOGUCh8x+7j7kXyMEJKuy97O6hbpF+abBZL9kOOaN\nHzau1HAGRsQgABA2LQzTq96UvoFh67KoyyKJBMQ+lwtEeJt6V4qAWHuLGSLXGFdE1kpSS9DTFv5X\nOPRWl8NdkFn99IKwWnNqwVHXD9ZHiWza+TToh7XOa50lEpAWc2FAsx9lg6BZ97HuUy0EpFceAkEL\nDyw88DUExDHSEQGtEqUS2AmvUi+cb0GuIFei3hwqRO25cLEQEUyFPAUMHREMEYiVbvVd03cNnYJF\npWZIfVJ69TvpHaozuvynHz+6Naraffn1U7AUVBXgd0VZRSGQctPpJnrJNOZqwL/hy/IxNGPj+Y3Q\nM0kNkhBobdyvMRxa4RDhEInPKV8IfSWXLIf9UjiqgFJL5h6DHgi3N24IAJc+LsUUU9eBrgOl7q8v\nD03ZRu5GCCiZuZsBtXZroSTsqvZVVNycvXoWGQwVPRU9iftFUHBSgF4I+bfyIFbyi+W5EtqjPOyn\n8ILwQkUIiGyxLF6HsGPzDkzD+lvNIhNhKmOKgHJkVCT87kttL2Gcr2qcapxYb+CD4ZiiWFpYimb+\nX4x/wfUUWggtJO4/8Q+Fx4XwuxWbK6J0TtZFVqwiaOLQidCTBbMLUOrb7kC7A/8pAZHxkUEkfu/U\nvaily2FywJjsTtudrsw6dWbXwQEljE9Aze6D7Q9g8Osq1kXzusNuB9wQxQrFcKz9xvuNr8i6Qy4P\ngYO9+ulq9EToXNG5Uq0ERNVRtfznM4/NRC9J8dFi3HA9DXsaSltPb4Iemozu6N5BBuLFlRfYP3Mt\nc0zR0szSRJPP3Zd3MTYtSi0KTVeKrxRfSe0BMbOGIxPmH4aSt/bu7d0pAaHyY0stTxIBlBU3xDJd\nCAGp/HtAgniorZ20axJS1WXqZQhkBF8JxnMqaCFoUZF1Ggc3hmOfODcRBvim1U0YZq1srexqJSB/\n/6D858EXgrk5/CtyMFSiafOmzaWtZ3bPDIY8yyALzfAhq0PQNMlYMVbfEwFp2LghIpbJ/snQfzuO\n7EDGSWApsJS2bufmnXF+QvuFwhGy7GrZlY7hpVIzRRjKoXqepE+r5T0gVSQgcpFyCAAcERxBgPZ+\nyn0MrdDN0wWx5zE8+Inj14wHoS96XATC0Kd+n/pS3zfRSwD9ML3hdPTAztOch8yJwhUF6ONm4c0w\nRehV6CtkIpbrLOeGcuzk7ZTYA1JbGYHcY0nH0GMRqR0JwqF6WRV+0Za/BXokJhmBHLNCs0KpfmxU\nHRCD5Y7Loc/GNB8DfcJfzYceFQYIAypCQD7hIKtB0Lt5e/P2YnqXziIcT1JxEgjKlPgpEnuWWz1r\nhRLb3DO5eM/H4qzF3DSuZbxl0va/T3Yf2KN1Z9ehZMs43Ti9RhOQT+g02gmR91eFr3CB4mTjwHA7\nduiIVLhGngZuPPlAeShs5Uhl3KBGzY1wgYIGB6Hn4V2td7VgiDsFIoPB38wHA1RfqY6U3LnN5/D3\nlIcpD8szO9VEVTRJqcSpgAk6mznDkKfkpWC7p8NP48ZU9VD1qFYCouWoJfaejlJLRNpS41NxY8TG\nxOLGtVK3guOiFKCEG1D3ii4emGXrlq0r38yZ7JUMw2xhY8GVPuQz2L+F0QvRC1MiW4Lz6tfTD01f\nde3qInOieFMRzZjmOeZopjrlego3dsaADMz7t3KzcqMEhMrPKNXxJnRtB22UWJ4pO8M5/K0LW2PK\nSdJ8GC7jB8aIOCnOUsRzrBKrgtrYlqtaosQq4k0EInOvW75GKYLncU9EmhgdRqdaCUhT76blP58x\nfwamOBVdKYK+6f1bb9Q28y/z8dzzV/BBTBqmN4Sh2dZ7G5q5P4g+IGOz4coG/I7RZrS/JwLyKdOx\nPnY9rkP+83yupErWE/pTc4Qmxl4qtVNC6Zh1HWuUAl8ZcAX68uH1h2huNR5gPIASECrfp977DwkI\nTw542OUw99qAlanw30zSTNLEAjOTGkMvPN76GBUuVw9fRWa2jWsb+C/Kk5Uxdlx9hjp67sYUjoF/\nWahYiMD06rLVeO5lp8oiAK6wUAElRnuj9oIIZIRnwO/rd6rfKWR6B9VCSZNqiioI0bh+4xBoKHEs\ngf4JNAxEwJjpxUAPuUS7wO8qSCqAnt/dcjf0t8lKk5Xl31+i+7fAj50RhP18ve41/Lpf3//KjRuf\nxZtVPsNQUQJiYGKA3pR7+ffgB77Mewl/9umrpwhAm2803yiRWI1TRm/e4d6Hoc8zCzLREzxw/EAE\n7tV2qyGgr3xFGfq9o05H2KGH4Q9xvq4euopekLpudd2+CwIiayMLh3lE5gj0PDx9/BSMNvtZNpjY\nlfQr6eVr9s65ncOBPb78GIYwv0k+5hRvV9yOG0srRytH0nbar24PJnn3+V0YlNzCXNyQl6Iu4YaL\nXh2Nz7MistArcm/UPaSY7C/bX5a0nkcLD9wIBSsKYIgd+jhINaQefh4owShULUTmo8vaLmvFMkLd\nZbqjqXvmBDQFZe/PRk9J8uxkXLAzq89g/xLOJoBhPqv3DM1ChzsfhkF9FPIIBs5iqYXYGDPds7r4\nfkTnCHyvMK0QD/JNuZsYG3fa5DRu1MQNiajpezniJQzs+KDxiODKHJc5Lua4OHSCQ1XAK8Dfx20f\nt50SECqUgPzDe0GuNUGJ6WnV03j+S+VL4YgmNkxERO6U7ymk9KP3R+O5T3ufBgOUXS8bz3lgRiCG\nPSjvVt4tcQqWZQAi9NlzslESZWpoKjWD2mlBJ9TuFugUwIB4TfKaVP7zZt2aYYre/Yf3EbBJaZcC\nh3v/gv0LyhvqG043UCIRYRIBPZIiSEHk8kXIC+ijyaMnjxabguU4BwY7Sz8LvXVWuVa5Uqdg7fHC\ni1eLPYsx396+1L5ULPNSYobx5jnLchChW3BiwQmp8+8/GKNX5fmo59Dv656vey7peyadTaAvz10+\nB/1f6FAIvXc17Coch9MPT3PnpXUKCGVKXAoCWO5+7tDzTEemo1iE8G0frmbdpgT2rn9K/xRKQKj8\naATE7qgdKjjK5Mug3zZ23ti5UiWnIxn08K56vgrP5euE1+g1817vjR46/q98TMNjGjLQm+5N3OH/\npW5ORYA5bUsaMg/nnM5BL8WejkVFTb5yvnJ5/9HkickTidP8FrYEEbnZ4CZKu3IicuAPivaKkEm4\ndOASHOe8jXlw4KPVolFZYpRqJPb6BbmRcjiOAMMA6OHc1Fx8/njeY7QCnLp7CtMHE64lwC4UtiiE\nP7lx80Ych9oFNbFxvcKLQvi/MdExIDbncs9Bb6rsUtklcUhREh/EZ43rGhCVjx0+IqC/w28H9JOg\nraCttOtgFWOFAHh8UDz8wDzfPNinWO1YBJTOXTyH/ckIyEBg/HHPxwhsO/Vz4no7hvLE3lM1YcUE\n+Ml5J/Kgn9u1bde2Zk3BKuSBEFgUWgB/U/8Nkf8dH3bAYJwwP4FmwPAj4UiJL7VaihST0xMn3EjK\n+5X3V2Q7pjqmMLizd85Gai3CPQIlRoeXHYYBm2s2F5FEcxNzE2nr6Ofpg1E6TXTCtKraI2tLHVdm\nIDCAYXZa6wTiUftSbYlTamTXyuJzR0VHEKoQpxA8SFGOUTDc6zLWwRHpcLcDbmCDJwY4/s69OoN5\nq1irWEtat3aD2nigRpeOhgHffWw3Sr6OnzoOhr88fjkyL532dsKDJttLtpekdTQvauLGc3JwgkE2\nCDcIpwSECiUgFSwZja4DAzIkcwgCLqFTQtE8eZx3HJ8fMTkCvfP74d8R0Wuf3B6RROEc4Rypek1g\nCv3S7UU3TIlRbqQsdSy25l1N6A8nZyf0aBi2NRQ3CAG8gPLjdYPuBKE5Mqo4Cin87XrbUdPs5uGG\nDKj6MHX0rg35MAT6+lCLQzCoU/ZM2SPm2GeZIKXfbVY3RPZUT6mekrafhk6G0H/OLs6IiGpM05gm\nFrE7pgw91r1td+x/Y8fGUpvvFV8o4vw4ZDugZKBpeFOp+kvnFx28uNE73xuRxP299yMyeNT9KOzG\nkj+W4I3AbTPaQi/z9/L3SsyE3dJGE6nzUmcEiHR5ujxKQKj8aAREQ1MDpU09/hYEMo42O1qVdYzC\njED0e9r2xFja1g9bg/Dz3/HfSeqRa6PUBj1bSyyXIBBz7P4xvD5gb529yFBOrD0RJVO6YboVejGi\n0SUj+GezDs7Ce4UOPTqEjO2h+4ewrl9bP+gb/R36O6T2KDeShx528XFBy8H6mPVw7KPuRaFkdYv7\nFuiRQTmDEDiv1aRWE4mEQomP47NbaYcMis18G2SoZRvINpC2/QauDUBAeuT2AGEx0TOp1Dj4Bt4N\nkEn37eiLgEr4y3CU9B85eATnZYHVAvjhzZKaceN++/L6SlzHogF6SLo/647EgsZijcU1egzv/+EB\nHhgn85RBLwbTk+lZneszsxlkGJjxzPhvsv/VtZ93mbvVuu4jHh4o5hpzreYcJyUgVH5sAvI/aMxD\nyQJznkEqnVnELKrReug2g+EevKm8qVK/58sgYsbz5/nX5OOp9PEbMlypxW5m93+3H5SAUKmZBKTG\nPKfkNQy8t7y31bLeKAYZU2YMM+ar1pPjcS8sfMJwGZixvLHf1XkdwKC0lAlmgmvOfn1LAkLxJ0JK\nQKj8ZASEIkVKQKhQAkKRIiUgFCkBoUKFEhCKlIBQoQSEIkVKQChSAkKFCiUgFClSAkKFEhCKFCkB\noUgJCBVKQChSpASEyg9CQDzpc0bx+8E0F0pAKFICQoUSEIoUKQGh8l3rvd71SWQ5n+CfFCnWPExT\n4TA+7f8B+P15KNMCkjoAAAAldEVYdGRhdGU6Y3JlYXRlADIwMTctMTAtMjRUMjI6Mzc6NDMrMDA6\nMDA1ZJ5wAAAAJXRFWHRkYXRlOm1vZGlmeQAyMDE3LTEwLTI0VDIyOjM3OjQzKzAwOjAwRDkmzAAA\nABR0RVh0cGRmOlZlcnNpb24AUERGLTEuNSAFXAs5AAAAAElFTkSuQmCC\n"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%tikz\n",
    "\\tikzset{every node/.style={font=\\sffamily,white}} \n",
    "\\node[fill=blue] at (0,0) (a) {Sender}; \n",
    "\\node[fill=blue] at (3,0) (b) {Channel}; \n",
    "\\node[fill=blue] at (6,0) (c) {Receiver}; \n",
    "\\draw[->] (a) -- (b) node [midway,above,font=\\scriptsize,black]{$p(\\mathbf{t})$}; \n",
    "\\draw[->] (b) -- (c) node [midway,above,font=\\scriptsize,black]{$p(\\mathbf{s}|\\mathbf{t})$};"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Channel could be \n",
    "\n",
    "* a ethernet cable\n",
    "* an encryption algorithm\n",
    "* a speaker that thinks in English but speaks in German"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Noisy Channel in MT\n",
    "$$\n",
    "\\prob_\\params(\\target,\\source) = \\prob_{\\params_t}(\\target) \\prob_{\\params_s}(\\source|\\target)\n",
    "$$\n",
    "\n",
    "* $\\prob_{\\params_s}(\\source|\\target)$ is called the **translation model**\n",
    "    * Does source match the target?\n",
    "* $\\prob_{\\params_t}(\\target)$ is the **language model**\n",
    "    * Does target look like real language?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "MLE for $\\prob_\\params(\\target,\\source) = \\prob_{\\params_t}(\\target) \\prob_{\\params_s}(\\source|\\target)$ can be calculated in two **independent steps**:\n",
    "* Estimate $\\params_t$ for $\\prob_{\\params_t}(\\target)$\n",
    "* Estimate $\\params_s$ for $\\prob_{\\params_s}(\\source|\\target)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "What training data do you need for each? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Translation Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$\n",
    "\\prob^{\\text{Impossible}}_\\params(\\source|\\target) = \\params_{\\source,\\target}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "For example:\n",
    "$$\n",
    "\\prob^{\\text{Impossible}}_\\params(\\text{Ich mag Musik }|\\text{ I like music})=\\params_{\\text{Ich mag Musik},\\text{I like music}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Why impossible?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Generally we want models that **factorize** (break up into smaller parts) for dealing with \n",
    "\n",
    "* **data sparsity**\n",
    "* **memory limitations**\n",
    "* **runtime limitations**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "How did **language models** do this? \n",
    "\n",
    "$$\n",
    "\\prob(\\text{I like Music})=\\prob(\\text{I}) \\prob(\\text{like | I}) \\prob(\\text{Music | like})\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Look at example for inspiration:\n",
    "\n",
    "|Token| 1 | 2 | 3 | 4 | \n",
    "|-|---|---|---|---|\n",
    "|**Target**| the | house | is | small |\n",
    "|**Source**| das | Haus | ist | klein |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$\n",
    "\\prob_\\theta(\\source|\\target) = p(\\text{das | the}) * p(\\text{ Haus | house}) \\ldots\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Naive Translation Model\n",
    "\n",
    "$$\n",
    "\\prob_\\params^\\text{Naive}(\\source|\\target) = \\prod_i^{\\length{\\source}} \\prob_\\params(\\ssource_i|\\starget_i) = \\prod_i^{\\length{\\source}} \\param_{\\ssource_i,\\starget_i}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Why naive?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Training\n",
    "\n",
    "Use the **Maximum Likelihood Estimate**:\n",
    "\n",
    "$$\n",
    "\\params^* = \\argmax_\\params \\sum_{(\\target,\\source) \\in \\train} \\log \\prob_\\params(\\source|\\target)\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Amounts to **counting**: \n",
    "\n",
    "$$\n",
    "\\param^*_{\\ssource,\\starget} = \\frac{\\counts{\\train}{s,t}}{\\counts{\\train}{t}} \n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "In Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict \n",
    "def learn_naive_model(data):\n",
    "    norm = defaultdict(float)\n",
    "    counts = defaultdict(float) \n",
    "    for target, source in data:\n",
    "        for i in range(0, len(target)):\n",
    "            norm[target[i]] += 1.0\n",
    "            counts[(source[i],target[i])] += 1.0\n",
    "    result = {}\n",
    "    for (source,target),score in counts.items():\n",
    "        result[(source,target)] = score / norm[target]\n",
    "    return result\n",
    "# show defaultdict behaviour"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table align=\"left\" style=\"font-size:2em;\"><tr><td style='padding:20px; text-align:left'>('das', 'the')</td> <td style='padding:20px; text-align:left'>1.00</td><tr><tr><td style='padding:20px; text-align:left'>('Haus', 'house')</td> <td style='padding:20px; text-align:left'>0.50</td><tr><tr><td style='padding:20px; text-align:left'>('Gebauede', 'house')</td> <td style='padding:20px; text-align:left'>0.50</td><tr></table>"
      ],
      "text/plain": [
       "<statnlpbook.util.Table at 0x7fe45794f400>"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "naive_model = learn_naive_model([[('the','house'),('das','Haus')],\n",
    "                                 [('the','house'),('das','Gebauede')]])\n",
    "# try other genders or numbers \n",
    "util.Table(naive_model.items(), \"2em\", \"20px\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table align=\"left\" style=\"font-size:1.5em;\"><tr><td style='padding:20px; text-align:left'>['the', 'house', 'is', 'small']</td> <td style='padding:20px; text-align:left'>['das', 'Haus', 'ist', 'klein']</td><tr><tr><td style='padding:20px; text-align:left'>['the', 'house', 'is', 'small']</td> <td style='padding:20px; text-align:left'>['klein', 'ist', 'das', 'Haus']</td><tr><tr><td style='padding:20px; text-align:left'>['a', 'man', 'is', 'tall']</td> <td style='padding:20px; text-align:left'>['ein', 'Mann', 'ist', 'groß']</td><tr><tr><td style='padding:20px; text-align:left'>['my', 'house', 'is', 'small']</td> <td style='padding:20px; text-align:left'>['klein', 'ist', 'mein', 'Haus']</td><tr></table>"
      ],
      "text/plain": [
       "<statnlpbook.util.Table at 0x7fe457956e80>"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "util.Table(train, \"1.5em\", \"20px\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAGPCAYAAACefJ5lAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGwVJREFUeJzt3X+4pnVdJ/D3h0G0Ns0fTNYCBikbUWFrs2hdmGlZYhaR\nlpSbiRmNhVybaVJWu25l4lWrmeg0GqFtLltWNpcMkZlWrmvOkKai0k5oAZWMZCqm4uhn/3juY4+n\ngXOA78xznjOv13Vx+dz3/X3O+VzOfZ7nfX/v7/d7V3cHAIA776hFFwAAsFkIVgAAgwhWAACDCFYA\nAIMIVgAAgwhWAACDCFYAAIMIVgAAgwhWAACDHL2oX3zsscf2iSeeuKhfDwCwblddddUHu3vrWu0W\nFqxOPPHE7N27d1G/HgBg3arqb9fTzq1AAIBBBCsAgEEEKwCAQQQrAIBBBCsAgEEEKwCAQQQrAIBB\nBCsAgEEEKwCAQQQrAIBB1hWsqupRVXVNVe2rqgsPcvwbq+rDVfX26b+fHV8qAMDGtuazAqtqS5KL\nkzwyyfVJ9lTVru5+96qmf97djzkENQIALIX19FidnmRfd1/b3bckuSzJWYe2LACA5bOeYHVckuvm\ntq+f9q329VX1jqq6oqq+ckh1AABLZM1bgev0l0nu1903V9Wjk7wmycmrG1XVeUnOS5L73e9+g341\nHFlOvPDyRZfAknv/875t0SXAprWeHqsbkpwwt338tO+zuvsj3X3z9Hp3krtU1bGrf1B37+zubd29\nbevWrXeibACAjWc9wWpPkpOr6qSqOibJOUl2zTeoqi+uqppenz793JtGFwsAsJGteSuwuw9U1flJ\nrkyyJckl3X11VW2fju9I8rgkT62qA0k+nuSc7u5DWDcAwIazrjFW0+293av27Zh7/eIkLx5bGgDA\ncrHyOgDAIIIVAMAgghUAwCCCFQDAIIIVAMAgghUAwCCCFQDAIIIVAMAgghUAwCCCFQDAIIIVAMAg\nghUAwCCCFQDAIIIVAMAgghUAwCCCFQDAIIIVAMAgghUAwCCCFQDAIIIVAMAgghUAwCCCFQDAIIIV\nAMAgghUAwCCCFQDAIIIVAMAgghUAwCCCFQDAIIIVAMAgghUAwCCCFQDAIIIVAMAgghUAwCCCFQDA\nIIIVAMAgghUAwCCCFQDAIIIVAMAgghUAwCCCFQDAIIIVAMAgghUAwCCCFQDAIIIVAMAgghUAwCCC\nFQDAIIIVAMAgghUAwCCCFQDAIIIVAMAgghUAwCCCFQDAIIIVAMAg6wpWVfWoqrqmqvZV1YW30e4/\nVdWBqnrcuBIBAJbDmsGqqrYkuTjJmUlOTfK9VXXqrbS7KMkfjS4SAGAZrKfH6vQk+7r72u6+Jcll\nSc46SLunJfndJDcOrA8AYGmsJ1gdl+S6ue3rp32fVVXHJTk7yUvHlQYAsFxGDV5/YZJndfdnbqtR\nVZ1XVXurau/+/fsH/WoAgI3h6HW0uSHJCXPbx0/75m1LcllVJcmxSR5dVQe6+zXzjbp7Z5KdSbJt\n27a+o0UDAGxE6wlWe5KcXFUnZRaozknyffMNuvuklddVdWmS164OVQAAm92awaq7D1TV+UmuTLIl\nySXdfXVVbZ+O7zjENQIALIX19Filu3cn2b1q30EDVXc/6c6XBQCwfKy8DgAwiGAFADCIYAUAMIhg\nBQAwiGAFADCIYAUAMIhgBQAwiGAFADCIYAUAMIhgBQAwiGAFADCIYAUAMIhgBQAwiGAFADCIYAUA\nMIhgBQAwiGAFADCIYAUAMIhgBQAwiGAFADCIYAUAMIhgBQAwiGAFADCIYAUAMIhgBQAwiGAFADCI\nYAUAMIhgBQAwiGAFADCIYAUAMIhgBQAwiGAFADCIYAUAMIhgBQAwiGAFADCIYAUAMIhgBQAwiGAF\nADCIYAUAMIhgBQAwiGAFADCIYAUAMIhgBQAwiGAFADCIYAUAMIhgBQAwiGAFADCIYAUAMIhgBQAw\niGAFADCIYAUAMIhgBQAwiGAFADDIuoJVVT2qqq6pqn1VdeFBjp9VVe+oqrdX1d6qOmN8qQAAG9vR\nazWoqi1JLk7yyCTXJ9lTVbu6+91zzV6fZFd3d1WdluS3k5xyKAoGANio1tNjdXqSfd19bXffkuSy\nJGfNN+jum7u7p81/l6QDAHCEWU+wOi7JdXPb10/7PkdVnV1V701yeZInjykPAGB5DBu83t2/392n\nJPnOJD93sDZVdd40Bmvv/v37R/1qAIANYT3B6oYkJ8xtHz/tO6ju/rMkX1ZVxx7k2M7u3tbd27Zu\n3Xq7iwUA2MjWE6z2JDm5qk6qqmOSnJNk13yDqnpAVdX0+kFJ7prkptHFAgBsZGvOCuzuA1V1fpIr\nk2xJckl3X11V26fjO5I8NskTq+pTST6e5PFzg9kBAI4IawarJOnu3Ul2r9q3Y+71RUkuGlsaAMBy\nsfI6AMAgghUAwCCCFQDAIIIVAMAgghUAwCCCFQDAIIIVAMAgghUAwCCCFQDAIIIVAMAgghUAwCCC\nFQDAIIIVAMAgghUAwCCCFQDAIIIVAMAgghUAwCCCFQDAIIIVAMAgghUAwCCCFQDAIIIVAMAgghUA\nwCCCFQDAIIIVAMAgghUAwCBHL7qAQ+3ECy9fdAkssfc/79sWXQJsej6nubM20me1HisAgEEEKwCA\nQQQrAIBBBCsAgEEEKwCAQQQrAIBBBCsAgEEEKwCAQQQrAIBBBCsAgEEEKwCAQQQrAIBBBCsAgEEE\nKwCAQQQrAIBBBCsAgEEEKwCAQQQrAIBBBCsAgEEEKwCAQQQrAIBBBCsAgEEEKwCAQQQrAIBBBCsA\ngEEEKwCAQQQrAIBB1hWsqupRVXVNVe2rqgsPcvwJVfWOqnpnVb25qh44vlQAgI1tzWBVVVuSXJzk\nzCSnJvneqjp1VbP3JXlYd391kp9LsnN0oQAAG916eqxOT7Kvu6/t7luSXJbkrPkG3f3m7v7QtPmW\nJMePLRMAYONbT7A6Lsl1c9vXT/tuzQ8mueJgB6rqvKraW1V79+/fv/4qAQCWwNDB61X18MyC1bMO\ndry7d3b3tu7etnXr1pG/GgBg4Y5eR5sbkpwwt338tO9zVNVpSV6e5MzuvmlMeQAAy2M9PVZ7kpxc\nVSdV1TFJzkmya75BVd0vye8l+f7u/uvxZQIAbHxr9lh194GqOj/JlUm2JLmku6+uqu3T8R1JfjbJ\nfZK8pKqS5EB3bzt0ZQMAbDzruRWY7t6dZPeqfTvmXj8lyVPGlgYAsFysvA4AMIhgBQAwiGAFADCI\nYAUAMIhgBQAwiGAFADCIYAUAMIhgBQAwiGAFADCIYAUAMIhgBQAwiGAFADCIYAUAMIhgBQAwiGAF\nADCIYAUAMIhgBQAwiGAFADCIYAUAMIhgBQAwiGAFADCIYAUAMIhgBQAwiGAFADCIYAUAMIhgBQAw\niGAFADCIYAUAMIhgBQAwiGAFADCIYAUAMIhgBQAwiGAFADCIYAUAMIhgBQAwiGAFADCIYAUAMIhg\nBQAwiGAFADCIYAUAMIhgBQAwiGAFADCIYAUAMIhgBQAwiGAFADCIYAUAMIhgBQAwiGAFADCIYAUA\nMIhgBQAwiGAFADCIYAUAMIhgBQAwyLqCVVU9qqquqap9VXXhQY6fUlX/t6o+WVXPGF8mAMDGd/Ra\nDapqS5KLkzwyyfVJ9lTVru5+91yzf0pyQZLvPCRVAgAsgfX0WJ2eZF93X9vdtyS5LMlZ8w26+8bu\n3pPkU4egRgCApbCeYHVckuvmtq+f9t1uVXVeVe2tqr379++/Iz8CAGDDOqyD17t7Z3dv6+5tW7du\nPZy/GgDgkFtPsLohyQlz28dP+wAAmLOeYLUnyclVdVJVHZPknCS7Dm1ZAADLZ81Zgd19oKrOT3Jl\nki1JLunuq6tq+3R8R1V9cZK9Se6R5DNV9V+SnNrdHzmEtQMAbChrBqsk6e7dSXav2rdj7vU/ZnaL\nEADgiGXldQCAQQQrAIBBBCsAgEEEKwCAQQQrAIBBBCsAgEEEKwCAQQQrAIBBBCsAgEEEKwCAQQQr\nAIBBBCsAgEEEKwCAQQQrAIBBBCsAgEEEKwCAQQQrAIBBBCsAgEEEKwCAQQQrAIBBBCsAgEEEKwCA\nQQQrAIBBBCsAgEEEKwCAQQQrAIBBBCsAgEEEKwCAQQQrAIBBBCsAgEEEKwCAQQQrAIBBBCsAgEEE\nKwCAQQQrAIBBBCsAgEEEKwCAQQQrAIBBBCsAgEEEKwCAQQQrAIBBBCsAgEEEKwCAQQQrAIBBBCsA\ngEEEKwCAQQQrAIBBBCsAgEEEKwCAQQQrAIBBBCsAgEEEKwCAQQQrAIBB1hWsqupRVXVNVe2rqgsP\ncryq6kXT8XdU1YPGlwoAsLGtGayqakuSi5OcmeTUJN9bVaeuanZmkpOn/85L8tLBdQIAbHjr6bE6\nPcm+7r62u29JclmSs1a1OSvJK3vmLUnuWVVfMrhWAIAN7eh1tDkuyXVz29cnefA62hyX5B/mG1XV\neZn1aCXJzVV1ze2qlkPh2CQfXHQRG1VdtOgKuAOc02twXi8d5/QaDtM5/aXrabSeYDVMd+9MsvNw\n/k5uW1Xt7e5ti64DRnFOs9k4p5fLem4F3pDkhLnt46d9t7cNAMCmtp5gtSfJyVV1UlUdk+ScJLtW\ntdmV5InT7MCHJPlwd//D6h8EALCZrXkrsLsPVNX5Sa5MsiXJJd19dVVtn47vSLI7yaOT7EvyL0nO\nPXQlM5hbs2w2zmk2G+f0EqnuXnQNAACbgpXXAQAGEawAAAYRrIAjWlXVomsANg/Bitutqj5/+l9f\nSCy9NtCUTa6q7rXoGo4kghW3S1WdneQ5VfUIX0gss6p6fFW9sKruW1X3WHQ9cChU1SOSvKqqvqWq\nvnzR9RwJzApk3aYeqhMzexj3M5JcnuSvuvt1i6wL7oiqum+SZyapzJae2dndVy+2KhinqrZ096er\n6tuTnJHknpl9Zr9kwaVtaoIV61JV35XkAUle0t03V9VXJjkzyUlJ3tjdv7PQAmGdqurpSY7q7l+a\ntr86yTcn+Z4kP9Ldb1tkfTBCVT0+yROTPLa7P1FVX5TZRfF/TXJldz9voQVuYm4Fsl6fSnL3JE+p\nqmOmK/tXJHlbkgdX1VcttDpYvz9P8oiquiBJuvudSX4lyW8leW5VnbTI4mCQy5O8J8mlU8/Vjd39\nxiRPT3JGVT1uodVtYoIVt6mqVlbnf22SP05ydXffkiTdvT/JFZndRnnQYiqE9amqLUnS3XuSPDvJ\nP68c6+7PJHllkjcl+capvckZLJ2Vz+zuvjmz8/wV3f3puSbvyOwi4uSqOrqq5IDB/B/KrZqucg5M\nf6iP6+4/XRlPtfKl0903JHlVkgv0WrFRzY01Oaqqvq6739bdr5yOrZzLH0nyriQPm7aNk2CpzH1m\nb6mqp3X3J7v7iunYUUkyhay3JXloklOmiwoGEqy4VStfRJk9ZPvrVx2uZPal1N1vzexWyucd5hJh\nTVV11Ny5/IdJnjB/vLt77kvnD5J8vKq+aQGlwp0yd56/Jslpq459Zu4i4r2ZDeU4Tc/seAav829U\n1SnTH16q6qIkH+3un5/Wr/rW7v796VitXNVPvVXXdfeHF1Y4rFJVx029qqmqS5P8Q3f/5HRbcFuS\nd3f3R6fjR01fPickuam7/2VhhcPtUFUP7+43TK+fneTY7v6xqrp3krOS/J8k7+vuT8295x5JPt3d\nH1tI0ZuYYMXnqKrHJLlbd7962n5BZj2b/5xka2YzAfd19yMXVyWsrarOTHJid7902n5ekrcn+drM\nJmJ8e5JfS/Lz87dD5i8YYKObZmg/bGUJhao6L8n9k3xpkn9M8vDMbv09vbv/aWGFHkEEKz6rqr44\nyY3TVftTkrw3yTuTvDzJW5P8dnf/bVXtSnJBd79/cdXCrauqu3b3J6eeqZ9M8uokxyX5z0n+NsnO\nJPdO8otJfrC7b1xYsXAHVdVXdfe7ptc/k9kF8G8k2Z7kY5kNXP+Xqnpdkp/u7r9YXLVHjqPXbsKR\noKq+OcnZSV5RVW9PckqSr0jyse7+7qnNvafbKR8SqtiopjXXnlpVj+3uj0y3sH8hyY9397lTm6OS\nPDez29dCFUtnWn/tf1bVs7v7tZldBD8pyYfn1mi7V1W9LMnfCVWHj8HrrLgusx6q70pyTGaD0T+U\n5AlT6EqSn8qsl/MHEtPR2bB2J/mLJL8+zWh9UZK3JHlhVa0M6P3VJLd0948kzmWWz7T+2rOSPGNa\ne+3KJC9JcnZV/ejU7AeSfLC7fzBxnh8ubgXyWVV1/yRbu/st0/bxmf1hfmFm61i9tbs/MR07yjRd\nNqqquluSM7r7j6ft+2Y2G/AbkvxEkpu7+++nY85lllZVndzd/296fdfMzvEfSfK6JC+dm2DkPD9M\nBCsOatUMqacleXt3v2o6ZnAvS2cKVz+a5J+6+4XTPucym8LcZ/bdknxLkvt39wumY87zw0iwYk1V\ndW+zSdgMquoe00KgsGlV1dHdfWDRdRypBCvWzVUPG9EdOS+dyywj5+1yMHj9CFNVj6mqM+7Ie/1B\ns5FU1ZOr6mumldNv16Bc5zLLoqqetDIY3Xm7HASrI0hV3TPJGUkeVlUPvo12Zo6woU3n8n0ye0bl\nV88N0K1V7ZzLLK2qukuSDyT5j1X15Nto57t8A/GPcYSYHs75z5lNM6+seq5fVR1XVd9dVXdxVcRG\nN53L/yvJm5I8YG5/V9X9q+qsle0FlQh32vQImjdmdq5/aP5YVZ1QVf9tuj1ott8GIlgdAaZQ9elp\nFeqjkzyvu9+4qtlDMpum++8Pd32wXlV1WlU9bNp8TJI3rTy7cs6XJXlyVT3w8FYHY0yf1UmS7v54\nd7/+IOf5XTN7msDZh7U41iRYbXLT1czKE89/N8nj52eLzHUhvzazhUG/dQFlwpqmc/UTSZ5eVdck\n+aLu/uuDNH1rkquyqlcWlsH8Z3ZV7Zx6pc6ePz69fF+SPUk+fyGFcqs80maTm7sV8ltJ9nb385Ok\nqr68u6+Z1j3ZMj1X7elJ7rmwYuFWzPW6XpfZA5S3ZPbMv3+juz9cVX+S2RcPLJW5z+yXJrlbZufx\nL05Lhbxiut298vfwO5ldELOBCFab1Py03Kq6T5J7JNlbVU9M8nVJvqOqdnb3c1aujrr7Y5k9uBM2\njLkvkaOS/IckP5zkvkl+bPqy+dWp3fHdfX2SdPebFlcx3H5V9bXdfdX0+pcyy1hPnLbfn+Rl0+f6\npdPfQ3X3h27jR7IgbgVuQtMX0fxjDG5K8sokv5bZF9NvJnlkkhOnVXpj8CMb0apb2X+Y5KnT4zve\nmuTXk3xTVf1YVf1akkcssla4o6rqmzIbG7jiqCQPqqqHTIt9/mlmFxS/WlUPTUzM2MgsELrJrLq6\nf3GSTyX5w+6+oqru3t0frdmDaXcm+Vh3P22hBcOtqKqTuvt90+uXJbmpuy+sqmOSfG2S9yc5Kckz\nk9zY3T+8sGLhTqiqY7r7lqrantlDk19dVc9NcmKS/57kr6dhG1/R3e9ZaLGsSY/VJjM3+29Xkr/J\nbBzKi6vqgiT3mh7S+b+THLUSqqz1w0ZTVd+c5NFzuz6Q5L1V9cIkv5Lk1Um2d/ebkzx2JVQ5l1km\nK7P/plB1t8zGuD68qs7s7p9K8ndJ/luSU6fe2/dM7/PdvYHpsdokqurxmU09v6GqviPJqUlekOTy\nJNdmNiblz5L8dpJ7d/dfTe/zxHM2lKr6gu6+efry+OnMZqzeNcmTklyfWW/r3ZO8KLNw9XfT+zzu\ng6Wx6u7CtiQf7e73VNUPJfmaJK+d7jTsSPLO7r54oQWzboLVJlBVd0/ymiRvy2yNqg9Ot0t+Isln\nuvu5U7fymUme0d2vn97ni4gNZVrY88eTPK67b6yqn8nsS+fCuav1o5O8PMnHu/upi6sW7pwpVF2e\n5B1JHprkj5L8QmbjqU5J8ifd/XuLq5A7QnfikptWSv9oku/JbNzJc6rqXt19S5K7JDltavpFSS5b\nCVWJwY9sPN39B0lel+Tl062RHUn+OMnz618fw/TLST65Eqrc/mOJ/XJm5/fPTdsfmFZbf1lmwzju\nvtLQeb489FgtsZXbeFPv1G8k+WiS78qs9+rCJF+Q5CVJviTJu7r7B6b36aliQ6mq70/y9d391Gnc\nyUNXng5QVccmeXxmi9f+bJIbunv/dMytbJbG6s/eqnpakvdm1kv7hu6+qKpOTvKFSf7Sub2c9Fgt\nqWmG32emL6GfSLKlu7cn+aokW5M8L/8atLYLVWxwVyT5wLQu1aeT/OnKge7+YGYTLt6S5BFzocoz\n0lga88vgzLk5yZVJXt/dF037np/kYSvntp6q5WOB0CU0DU5/ZlU9dhqH8v4kW6vq86bt7Unendm/\n7wXdvWd6n1DFRvXpJF+Z5PuS7JhWl/7s+TqNG3xRd9+88gbnMstkbqD6FUn2Z/Y4mt/I7EL43Kq6\nKbML4f3d/ctz73OeLxk9Vkuou3dlNsjxkqr6vMymot83ySlVddfu/kBmiym+zxcRy2BaQfrnkzy7\nqr5v2tc1mbZvTlzBs1xq7oHKSbZnNsnot5Icn+RnMnt0zXMye3zN67v73Ol9vp+XlDFWS2SNcSgX\nZjZF98NJHpDk6u6+YDqmp4qlMK1f9aIkz+/uS+f2G0vF0pkbB3tUkscl+cYkL+3ud1bVQ5Kcldmd\nhRd099+vft9CiuZOE6yWyDSI9/wk/6O7P3KQgZBnJLlXkuO6e8e0T6hiqUzn8W9mtg7bvu7eveCS\n4HabHkVzYOphvTyzi97TMxuk/pSpzYOTfH+SN3f3qxZXLSMJVkukqu6V2eKIr58PTpn9O/6bqxtX\nPSyraWbUIzN7ftq75nuvYKOrz3182E8lOam7z62q+2Q2WP0N3f3Mqe39u/tvFlkvYwlWS6aqHpjZ\nStTPWrnC0SvFZjbNFPzIouuA9ViZXJTksUluSfKLSb4hyRO6++1Vdd/MBrBf1d0/NPc+n+ObhMFx\nS2Z6FM25SX66qp407evEYEc2J6GKZTI3uehlST6W2S3t303ypKo6bZpc9G1J/nHV+4SqTUKP1ZIy\nDgVg4zjI5KJv6O43TMcemOQ7MnsCxqXdfdXc+/RUbTJ6OJZUd78pybckOZDkESu9VwAsxOpFbt+4\ncmC60/AHST6R5NT5NwlVm48eq03COBSAxbm1yUWrZm6f0N3XLapGDg/BCgAGuLXJRcnn9ky5/be5\neaQNAAzQ3X9VVecmeVFVHdPdl85PLlpZ/kao2tz0WAHAQCYXHdkEKwAYzCK3Ry7BCgAOIZOLjiyC\nFQDAINaxAgAYRLACABhEsAIAGESwAgAYRLACABhEsAIAGOT/A/5BbmKeNwsZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe457956940>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "table = learn_naive_model(train)\n",
    "plot_table_for_target(table, \"is\") # try house"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Translation / Decoding\n",
    "\n",
    "Find the highest scoring target $\\target$ \n",
    "\n",
    "$$\n",
    "\\target^*(\\source) = \\argmax_\\target \\prod_i^{\\length{\\source}} \\prob_\\params(\\ssource_i|\\starget_i)\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "How about **brute-force** (like in our Structured Prediction example)?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Local Decoding \n",
    "\n",
    "How to find this $\\argmax$?\n",
    "\n",
    "$$\n",
    "\\argmax_{y_1,y_2} f_1(y_1) f_2(y_2) = \\ldots\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Local Decoding for the Naive Model\n",
    "\n",
    "$$\n",
    "\\target^*(\\source) = \\argmax_\\target \\prod_i^{\\length{\\source}} \\prob_\\params(\\ssource_i|\\starget_i) \\\\\n",
    "= (\\argmax_{\\starget_1}  \\prob_\\params(\\ssource_1|\\starget_1), \\ldots, \\argmax_{\\starget_{\\length{\\source}}}  \\prob_\\params(\\ssource_{\\length{\\source}}|\\starget_{\\length{\\source}}))\n",
    "$$\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "In Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a', 'man']"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def decode(source_sent, model, lm):\n",
    "    source_to_targets = defaultdict(list)\n",
    "    for (source,target),prob in model.items():\n",
    "        source_to_targets[source] += [(target,prob)]\n",
    "    result = []\n",
    "    for tok in source_sent:\n",
    "        candidates = source_to_targets[tok]\n",
    "        multiplied_with_lm = [(target,prob * lm.probability(target)) \n",
    "                              for target, prob in candidates]\n",
    "        target = max(multiplied_with_lm, key=lambda t: t[1])\n",
    "        result.append(target[0])\n",
    "    return result\n",
    "\n",
    "lm = UniformLM(set([target for _, target in table.keys()]))\n",
    "decode([\"ein\", \"Mann\"], table, lm) # try other inputs with given vocab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Alignments\n",
    "Naive Model assumes a sequential **alignment**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <svg id='f3ac2410-b90b-11e7-8cfc-0242ac110002' xmlns=\"http://www.w3.org/2000/svg\"\n",
       "             xmlns:xlink=\"http://www.w3.org/1999/xlink\"\n",
       "             viewBox=\"0 0 400 100\"\n",
       "             >\n",
       "\n",
       "            <text x=\"0\" y=\"20\" class=\"source\">\n",
       "                <tspan id='t0'>the</tspan> <tspan id='t1'>house</tspan> <tspan id='t2'>is</tspan> <tspan id='t3'>small</tspan>\n",
       "            </text>\n",
       "            <text x=\"0\" y=\"95\" class=\"target\">\n",
       "                <tspan id='t0'>das</tspan> <tspan id='t1'>Haus</tspan> <tspan id='t2'>ist</tspan> <tspan id='t3'>klein</tspan>\n",
       "            </text>\n",
       "            <g class='connections'></g>\n",
       "            <script>\n",
       "              $(function() {\n",
       "                  root = $(document.getElementById('f3ac2410-b90b-11e7-8cfc-0242ac110002'));\n",
       "                  root.find('.connections').empty();\n",
       "                  alignments = [['.source #t0','.target #t0',1.0],['.source #t1','.target #t1',1.0],['.source #t2','.target #t2',1.0],['.source #t3','.target #t3',1.0]];\n",
       "                  function appendLine(alignment) {\n",
       "                      s1 = root.find(alignment[0])[0];\n",
       "                      x1 = s1.getExtentOfChar(0).x + s1.getComputedTextLength() / 2.0;\n",
       "                      y1 = s1.getExtentOfChar(0).y + s1.getExtentOfChar(0).height;\n",
       "                      s2 = root.find(alignment[1])[0];\n",
       "                      x2 = s2.getExtentOfChar(0).x + s2.getComputedTextLength() / 2.0;\n",
       "                      y2 = s2.getExtentOfChar(0).y;\n",
       "                      var newLine = document.createElementNS('http://www.w3.org/2000/svg','line');\n",
       "                      var score = alignment[2];\n",
       "                      newLine.setAttribute('x1',x1.toString());\n",
       "                      newLine.setAttribute('y1',y1.toString());\n",
       "                      newLine.setAttribute('x2',x2.toString());\n",
       "                      newLine.setAttribute('y2',y2.toString());\n",
       "                      newLine.setAttribute('style',\"stroke:black;stroke-width:2;stroke-opacity:\" + score + \";\");\n",
       "                      root.find('.connections').append(newLine)\n",
       "                  };\n",
       "                  //console.log(alignments);\n",
       "                  for (var i = 0; i < alignments.length; i++) {\n",
       "                    appendLine(alignments[i]);\n",
       "                  }\n",
       "                  //console.log($(root).find('.connections'));\n",
       "              });\n",
       "            </script>\n",
       "        </svg>\n",
       "        "
      ],
      "text/plain": [
       "<statnlpbook.word_mt.Alignment at 0x7fe457a8c588>"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_mt.Alignment(\"the house is small\".split(\" \"),\n",
    "                  \"das Haus ist klein\".split(\" \"),\n",
    "                  [(0,0),(1,1),(2,2),(3,3)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "But word order can differ! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <svg id='f3aff590-b90b-11e7-8cfc-0242ac110002' xmlns=\"http://www.w3.org/2000/svg\"\n",
       "             xmlns:xlink=\"http://www.w3.org/1999/xlink\"\n",
       "             viewBox=\"0 0 400 100\"\n",
       "             >\n",
       "\n",
       "            <text x=\"0\" y=\"20\" class=\"source\">\n",
       "                <tspan id='t0'>the</tspan> <tspan id='t1'>house</tspan> <tspan id='t2'>is</tspan> <tspan id='t3'>small</tspan>\n",
       "            </text>\n",
       "            <text x=\"0\" y=\"95\" class=\"target\">\n",
       "                <tspan id='t0'>klein</tspan> <tspan id='t1'>ist</tspan> <tspan id='t2'>das</tspan> <tspan id='t3'>Haus</tspan>\n",
       "            </text>\n",
       "            <g class='connections'></g>\n",
       "            <script>\n",
       "              $(function() {\n",
       "                  root = $(document.getElementById('f3aff590-b90b-11e7-8cfc-0242ac110002'));\n",
       "                  root.find('.connections').empty();\n",
       "                  alignments = [['.source #t0','.target #t2',1.0],['.source #t1','.target #t3',1.0],['.source #t2','.target #t1',1.0],['.source #t3','.target #t0',1.0]];\n",
       "                  function appendLine(alignment) {\n",
       "                      s1 = root.find(alignment[0])[0];\n",
       "                      x1 = s1.getExtentOfChar(0).x + s1.getComputedTextLength() / 2.0;\n",
       "                      y1 = s1.getExtentOfChar(0).y + s1.getExtentOfChar(0).height;\n",
       "                      s2 = root.find(alignment[1])[0];\n",
       "                      x2 = s2.getExtentOfChar(0).x + s2.getComputedTextLength() / 2.0;\n",
       "                      y2 = s2.getExtentOfChar(0).y;\n",
       "                      var newLine = document.createElementNS('http://www.w3.org/2000/svg','line');\n",
       "                      var score = alignment[2];\n",
       "                      newLine.setAttribute('x1',x1.toString());\n",
       "                      newLine.setAttribute('y1',y1.toString());\n",
       "                      newLine.setAttribute('x2',x2.toString());\n",
       "                      newLine.setAttribute('y2',y2.toString());\n",
       "                      newLine.setAttribute('style',\"stroke:black;stroke-width:2;stroke-opacity:\" + score + \";\");\n",
       "                      root.find('.connections').append(newLine)\n",
       "                  };\n",
       "                  //console.log(alignments);\n",
       "                  for (var i = 0; i < alignments.length; i++) {\n",
       "                    appendLine(alignments[i]);\n",
       "                  }\n",
       "                  //console.log($(root).find('.connections'));\n",
       "              });\n",
       "            </script>\n",
       "        </svg>\n",
       "        "
      ],
      "text/plain": [
       "<statnlpbook.word_mt.Alignment at 0x7fe457956f98>"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_mt.Alignment(\"the house is small\".split(\" \"),\n",
    "                  \"klein ist das Haus\".split(\" \"),\n",
    "                  [(0,2),(1,3),(2,1),(3,0)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Formalise alignments:\n",
    "\n",
    "* for each source sentence index \\\\(i \\in [1 \\ldots \\length{\\source}]\\\\)\n",
    "    * \\\\(a_i \\in [0 \\ldots \\length{\\target}]\\\\): index of **aligned target word**\n",
    "* \\\\(\\align_i\\\\) can be \\\\(0\\\\)\n",
    "    * Corresponds to imaginary _NULL_ token \\\\(\\starget_0\\\\)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <svg id='f3b3d160-b90b-11e7-8cfc-0242ac110002' xmlns=\"http://www.w3.org/2000/svg\"\n",
       "             xmlns:xlink=\"http://www.w3.org/1999/xlink\"\n",
       "             viewBox=\"0 0 400 100\"\n",
       "             >\n",
       "\n",
       "            <text x=\"0\" y=\"20\" class=\"source\">\n",
       "                <tspan id='t0'>NULL</tspan> <tspan id='t1'>the</tspan> <tspan id='t2'>house</tspan> <tspan id='t3'>is</tspan> <tspan id='t4'>small</tspan>\n",
       "            </text>\n",
       "            <text x=\"0\" y=\"95\" class=\"target\">\n",
       "                <tspan id='t0'>klein</tspan> <tspan id='t1'>ist</tspan> <tspan id='t2'>das</tspan> <tspan id='t3'>Haus</tspan>\n",
       "            </text>\n",
       "            <g class='connections'></g>\n",
       "            <script>\n",
       "              $(function() {\n",
       "                  root = $(document.getElementById('f3b3d160-b90b-11e7-8cfc-0242ac110002'));\n",
       "                  root.find('.connections').empty();\n",
       "                  alignments = [['.source #t1','.target #t2',1.0],['.source #t2','.target #t3',1.0],['.source #t3','.target #t1',1.0],['.source #t4','.target #t0',1.0]];\n",
       "                  function appendLine(alignment) {\n",
       "                      s1 = root.find(alignment[0])[0];\n",
       "                      x1 = s1.getExtentOfChar(0).x + s1.getComputedTextLength() / 2.0;\n",
       "                      y1 = s1.getExtentOfChar(0).y + s1.getExtentOfChar(0).height;\n",
       "                      s2 = root.find(alignment[1])[0];\n",
       "                      x2 = s2.getExtentOfChar(0).x + s2.getComputedTextLength() / 2.0;\n",
       "                      y2 = s2.getExtentOfChar(0).y;\n",
       "                      var newLine = document.createElementNS('http://www.w3.org/2000/svg','line');\n",
       "                      var score = alignment[2];\n",
       "                      newLine.setAttribute('x1',x1.toString());\n",
       "                      newLine.setAttribute('y1',y1.toString());\n",
       "                      newLine.setAttribute('x2',x2.toString());\n",
       "                      newLine.setAttribute('y2',y2.toString());\n",
       "                      newLine.setAttribute('style',\"stroke:black;stroke-width:2;stroke-opacity:\" + score + \";\");\n",
       "                      root.find('.connections').append(newLine)\n",
       "                  };\n",
       "                  //console.log(alignments);\n",
       "                  for (var i = 0; i < alignments.length; i++) {\n",
       "                    appendLine(alignments[i]);\n",
       "                  }\n",
       "                  //console.log($(root).find('.connections'));\n",
       "              });\n",
       "            </script>\n",
       "        </svg>\n",
       "        "
      ],
      "text/plain": [
       "<statnlpbook.word_mt.Alignment at 0x7fe457956fd0>"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_mt.Alignment(\"NULL the house is small\".split(\" \"),\n",
    "                  \"klein ist das Haus\".split(\" \"),\n",
    "                  [(1,2),(2,3),(3,1),(4,0)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$a_1 = 4$ ??? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Why the **NULL** Token? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <svg id='f3b77158-b90b-11e7-8cfc-0242ac110002' xmlns=\"http://www.w3.org/2000/svg\"\n",
       "             xmlns:xlink=\"http://www.w3.org/1999/xlink\"\n",
       "             viewBox=\"0 0 400 100\"\n",
       "             >\n",
       "\n",
       "            <text x=\"0\" y=\"20\" class=\"source\">\n",
       "                <tspan id='t0'>NULL</tspan> <tspan id='t1'>I</tspan> <tspan id='t2'>like</tspan> <tspan id='t3'>music</tspan>\n",
       "            </text>\n",
       "            <text x=\"0\" y=\"95\" class=\"target\">\n",
       "                <tspan id='t0'>音楽</tspan> <tspan id='t1'>が</tspan> <tspan id='t2'>好き</tspan>\n",
       "            </text>\n",
       "            <g class='connections'></g>\n",
       "            <script>\n",
       "              $(function() {\n",
       "                  root = $(document.getElementById('f3b77158-b90b-11e7-8cfc-0242ac110002'));\n",
       "                  root.find('.connections').empty();\n",
       "                  alignments = [['.source #t0','.target #t1',1.0],['.source #t2','.target #t2',1.0],['.source #t3','.target #t0',1.0]];\n",
       "                  function appendLine(alignment) {\n",
       "                      s1 = root.find(alignment[0])[0];\n",
       "                      x1 = s1.getExtentOfChar(0).x + s1.getComputedTextLength() / 2.0;\n",
       "                      y1 = s1.getExtentOfChar(0).y + s1.getExtentOfChar(0).height;\n",
       "                      s2 = root.find(alignment[1])[0];\n",
       "                      x2 = s2.getExtentOfChar(0).x + s2.getComputedTextLength() / 2.0;\n",
       "                      y2 = s2.getExtentOfChar(0).y;\n",
       "                      var newLine = document.createElementNS('http://www.w3.org/2000/svg','line');\n",
       "                      var score = alignment[2];\n",
       "                      newLine.setAttribute('x1',x1.toString());\n",
       "                      newLine.setAttribute('y1',y1.toString());\n",
       "                      newLine.setAttribute('x2',x2.toString());\n",
       "                      newLine.setAttribute('y2',y2.toString());\n",
       "                      newLine.setAttribute('style',\"stroke:black;stroke-width:2;stroke-opacity:\" + score + \";\");\n",
       "                      root.find('.connections').append(newLine)\n",
       "                  };\n",
       "                  //console.log(alignments);\n",
       "                  for (var i = 0; i < alignments.length; i++) {\n",
       "                    appendLine(alignments[i]);\n",
       "                  }\n",
       "                  //console.log($(root).find('.connections'));\n",
       "              });\n",
       "            </script>\n",
       "        </svg>\n",
       "        "
      ],
      "text/plain": [
       "<statnlpbook.word_mt.Alignment at 0x7fe457956cf8>"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_mt.Alignment(\"NULL I like music\".split(\" \"),\n",
    "                  \"音楽 が 好き\".split(\" \"),\n",
    "                  [(0,1),(2,2),(3,0)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "One-to-many translations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <svg id='f3bb20be-b90b-11e7-8cfc-0242ac110002' xmlns=\"http://www.w3.org/2000/svg\"\n",
       "             xmlns:xlink=\"http://www.w3.org/1999/xlink\"\n",
       "             viewBox=\"0 0 400 100\"\n",
       "             >\n",
       "\n",
       "            <text x=\"0\" y=\"20\" class=\"source\">\n",
       "                <tspan id='t0'>NULL</tspan> <tspan id='t1'>the</tspan> <tspan id='t2'>house</tspan> <tspan id='t3'>is</tspan> <tspan id='t4'>tiny</tspan>\n",
       "            </text>\n",
       "            <text x=\"0\" y=\"95\" class=\"target\">\n",
       "                <tspan id='t0'>das</tspan> <tspan id='t1'>Haus</tspan> <tspan id='t2'>ist</tspan> <tspan id='t3'>sehr</tspan> <tspan id='t4'>klein</tspan>\n",
       "            </text>\n",
       "            <g class='connections'></g>\n",
       "            <script>\n",
       "              $(function() {\n",
       "                  root = $(document.getElementById('f3bb20be-b90b-11e7-8cfc-0242ac110002'));\n",
       "                  root.find('.connections').empty();\n",
       "                  alignments = [['.source #t1','.target #t0',1.0],['.source #t2','.target #t1',1.0],['.source #t3','.target #t2',1.0],['.source #t4','.target #t3',1.0],['.source #t4','.target #t4',1.0]];\n",
       "                  function appendLine(alignment) {\n",
       "                      s1 = root.find(alignment[0])[0];\n",
       "                      x1 = s1.getExtentOfChar(0).x + s1.getComputedTextLength() / 2.0;\n",
       "                      y1 = s1.getExtentOfChar(0).y + s1.getExtentOfChar(0).height;\n",
       "                      s2 = root.find(alignment[1])[0];\n",
       "                      x2 = s2.getExtentOfChar(0).x + s2.getComputedTextLength() / 2.0;\n",
       "                      y2 = s2.getExtentOfChar(0).y;\n",
       "                      var newLine = document.createElementNS('http://www.w3.org/2000/svg','line');\n",
       "                      var score = alignment[2];\n",
       "                      newLine.setAttribute('x1',x1.toString());\n",
       "                      newLine.setAttribute('y1',y1.toString());\n",
       "                      newLine.setAttribute('x2',x2.toString());\n",
       "                      newLine.setAttribute('y2',y2.toString());\n",
       "                      newLine.setAttribute('style',\"stroke:black;stroke-width:2;stroke-opacity:\" + score + \";\");\n",
       "                      root.find('.connections').append(newLine)\n",
       "                  };\n",
       "                  //console.log(alignments);\n",
       "                  for (var i = 0; i < alignments.length; i++) {\n",
       "                    appendLine(alignments[i]);\n",
       "                  }\n",
       "                  //console.log($(root).find('.connections'));\n",
       "              });\n",
       "            </script>\n",
       "        </svg>\n",
       "        "
      ],
      "text/plain": [
       "<statnlpbook.word_mt.Alignment at 0x7fe4578737f0>"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_mt.Alignment(\"NULL the house is tiny\".split(\" \"),\n",
    "                  \"das Haus ist sehr klein\".split(\" \"),\n",
    "                  [(1,0),(2,1),(3,2),(4,3),(4,4)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Many-to-one translations?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Alignments are **hidden** when given a source! \n",
    "\n",
    "Need to be \n",
    "* predicted or marginalised out\n",
    "* modelled"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## IBM Translation Models\n",
    "\n",
    "In the late 80s, early 90s\n",
    "* **IBM researchers revolutionised MT** (and Speech Recognition) \n",
    "* using statistical approaches instead of rules \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "A famous IBM quote from that time\n",
    "\n",
    "> Every time I fire a linguist, the performance of the speech recognizer goes up\n",
    "> \n",
    ">  -- <cite>Fred Jelinek</cite>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Group around **Robert Mercer** were extremely successful\n",
    "\n",
    "* then disappeared to create **Renaissance Technologies**\n",
    "\n",
    "![mercer](http://images.huffingtonpost.com/2014-10-23-RobertMercer-thumb.JPG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## IBM Model 2\n",
    "Translation and **alignment** model:\n",
    "\n",
    "$$\n",
    "p_\\params^\\text{IBM2}(\\source,\\aligns|\\target)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "*Generative story* with parameters $\\params = (\\alpha, \\beta)$\n",
    "\n",
    "Given target sentence $\\target$ (e.g. NULL I like music) with length $l_\\starget$ (e.g. 4):\n",
    "\n",
    "* Generate a **source sentence length** $l\n",
    "_\\ssource$ with uniform probability $\\epsilon$\n",
    "    * e.g. $l_\\ssource=3$\n",
    "* Generate a **target position** $a_i$ for each source as position $i$ with $\\beta(a_i|i,l_\\starget,l_\\ssource)$\n",
    "    * e.g. $(3,0,2)$\n",
    "* Generate the **source word** $\\ssource_{i}$ at $a_i$ with probability $\\alpha(\\ssource_i|\\starget_{a_i})$ \n",
    "    * e.g. (音楽, が, 好き)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\\begin{equation}\n",
    "  p_\\params^\\text{IBM2}(\\ssource_1 \\ldots \\ssource_{l_\\ssource},\\align_1 \\ldots \\align_{l_\\ssource}|\\starget_1 \\ldots \\starget_{l_\\starget}) = \\epsilon \\prod_i^{l_\\ssource} \\alpha(\\ssource_i|\\starget_{a_i}) \\beta(a_i|i,l_\\starget,l_\\ssource)\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Example:\n",
    "$$\n",
    "p_\\params^\\text{IBM2}(\\text{das, Haus},1, 2 | \\text{NULL, the, house}) = \\ldots\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Answer: \n",
    "$$\n",
    "\\a(das|the) \\b(1|1,3,2) \\ldots\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Training IBM Model 2\n",
    "\n",
    "If we had $\\train = ((\\source_i, \\aligns_i, \\target_i))_i$ we could optimise\n",
    "\n",
    "$$\n",
    "\\sum_{(\\source, \\aligns, \\target) \\in \\train} \\log p_\\params^\\text{IBM2}(\\source,\\aligns|\\target)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "But we have **no training alignments** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Instead use **sentence-aligned** data $\\train = ((\\source_i, \\target_i))_i$ \n",
    "\n",
    "and **marginal log-likelihood**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "$$\n",
    "\\sum_{(\\source, \\target) \\in \\train}   \\log p_\\params^\\text{IBM2}(\\source|\\target)\n",
    "$$\n",
    "\n",
    "where\n",
    "\n",
    "$$\n",
    "p_\\params^\\text{IBM2}(\\source|\\target) = \\sum_{\\aligns}p_\\params^\\text{IBM2}(\\source,\\aligns|\\target)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**No closed-form** solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Expectation Maximisation (EM) Algorithm\n",
    "maximises a [lower bound of marginal log-likelihood](http://localhost:8888/notebooks/chapters/em.ipynb) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Figures out (soft) alignments by iterating:\n",
    "\n",
    "* **E-step**: Infer alignments (or their **expectations**)\n",
    "    * using current parameters\n",
    "*  **M-step**: **Maximise** training objective to estimate parameters \n",
    "    * using current alignments  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "How is this possible **without good initial parameters**? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Can you find out which target words belong to which source words? \n",
    "\n",
    "Assume static word order (but possibly different in each language)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <svg id='f3bf0864-b90b-11e7-8cfc-0242ac110002' xmlns=\"http://www.w3.org/2000/svg\"\n",
       "             xmlns:xlink=\"http://www.w3.org/1999/xlink\"\n",
       "             viewBox=\"0 0 400 100\"\n",
       "             >\n",
       "\n",
       "            <text x=\"0\" y=\"20\" class=\"source\">\n",
       "                <tspan id='t0'>Bb</tspan> <tspan id='t1'>Aa</tspan> <tspan id='t2'>|</tspan> <tspan id='t3'>Aa</tspan> <tspan id='t4'>Cc</tspan> <tspan id='t5'>|</tspan> <tspan id='t6'>Qq</tspan> <tspan id='t7'>Pp</tspan>\n",
       "            </text>\n",
       "            <text x=\"0\" y=\"95\" class=\"target\">\n",
       "                <tspan id='t0'>Xx</tspan> <tspan id='t1'>Yy</tspan> <tspan id='t2'>|</tspan> <tspan id='t3'>Zz</tspan> <tspan id='t4'>Xx</tspan> <tspan id='t5'>|</tspan> <tspan id='t6'>Ss</tspan> <tspan id='t7'>Tt</tspan>\n",
       "            </text>\n",
       "            <g class='connections'></g>\n",
       "            <script>\n",
       "              $(function() {\n",
       "                  root = $(document.getElementById('f3bf0864-b90b-11e7-8cfc-0242ac110002'));\n",
       "                  root.find('.connections').empty();\n",
       "                  alignments = [];\n",
       "                  function appendLine(alignment) {\n",
       "                      s1 = root.find(alignment[0])[0];\n",
       "                      x1 = s1.getExtentOfChar(0).x + s1.getComputedTextLength() / 2.0;\n",
       "                      y1 = s1.getExtentOfChar(0).y + s1.getExtentOfChar(0).height;\n",
       "                      s2 = root.find(alignment[1])[0];\n",
       "                      x2 = s2.getExtentOfChar(0).x + s2.getComputedTextLength() / 2.0;\n",
       "                      y2 = s2.getExtentOfChar(0).y;\n",
       "                      var newLine = document.createElementNS('http://www.w3.org/2000/svg','line');\n",
       "                      var score = alignment[2];\n",
       "                      newLine.setAttribute('x1',x1.toString());\n",
       "                      newLine.setAttribute('y1',y1.toString());\n",
       "                      newLine.setAttribute('x2',x2.toString());\n",
       "                      newLine.setAttribute('y2',y2.toString());\n",
       "                      newLine.setAttribute('style',\"stroke:black;stroke-width:2;stroke-opacity:\" + score + \";\");\n",
       "                      root.find('.connections').append(newLine)\n",
       "                  };\n",
       "                  //console.log(alignments);\n",
       "                  for (var i = 0; i < alignments.length; i++) {\n",
       "                    appendLine(alignments[i]);\n",
       "                  }\n",
       "                  //console.log($(root).find('.connections'));\n",
       "              });\n",
       "            </script>\n",
       "        </svg>\n",
       "        "
      ],
      "text/plain": [
       "<statnlpbook.word_mt.Alignment at 0x7fe4576c13c8>"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_mt.Alignment(\"Bb Aa | Aa Cc | Qq Pp\".split(\" \"),\n",
    "                  \"Xx Yy | Zz Xx | Ss Tt\".split(\" \"),[])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$\\a(X|A)= 1/2 \\qquad \\a(Y|A)= 1/4 \\qquad \\a(X|B)= \\qquad \\a(Y|B)= \\qquad \\a(T|P)=$\n",
    "\n",
    "$\\b(1|1)= 1/2 \\qquad \\b(1|2)= 1/2$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Formalise and implement, but for dataset with `NULL` tokens and \n",
    "\n",
    "* **non-monotonic** and **consistent alignment**\n",
    "\n",
    "such that EM will \n",
    "* first learn about word-to-word correspondences\n",
    "* and then about distortion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "train_model_2_raw = [\n",
    "  (\"NULL the house is small\" , \"klein ist das Haus\"),\n",
    "  (\"NULL a man is tall\" , \"groß ist ein Mann\"),\n",
    "  (\"NULL my house is small\" , \"klein ist mein Haus\"),\n",
    "  (\"NULL the building is big\" , \"groß ist das Gebäude\"),\n",
    "  (\"NULL the building is long\" , \"lang ist das Gebäude\")\n",
    "]\n",
    "train_model_2 =  [(t.split(\" \"), s.split(\" \")) for t,s in train_model_2_raw]\n",
    "# What could \"house\" translate to? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table align=\"left\" style=\"font-size:x-large;\"><tr><td style='padding:5px; text-align:left'>NULL the house is small</td> <td style='padding:5px; text-align:left'>klein ist das Haus</td><tr><tr><td style='padding:5px; text-align:left'>NULL a man is tall</td> <td style='padding:5px; text-align:left'>groß ist ein Mann</td><tr><tr><td style='padding:5px; text-align:left'>NULL my house is small</td> <td style='padding:5px; text-align:left'>klein ist mein Haus</td><tr><tr><td style='padding:5px; text-align:left'>NULL the building is big</td> <td style='padding:5px; text-align:left'>groß ist das Gebäude</td><tr><tr><td style='padding:5px; text-align:left'>NULL the building is long</td> <td style='padding:5px; text-align:left'>lang ist das Gebäude</td><tr></table>"
      ],
      "text/plain": [
       "<statnlpbook.util.Table at 0x7fe4578b15c0>"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "util.Table(train_model_2_raw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### E-Step\n",
    "\n",
    "Calculate posterior distribution over alignments given source and target \n",
    "\n",
    "$$\n",
    "\\pi(\\aligns|\\source,\\target) = p_\\params^\\text{IBM2}(\\aligns|\\source,\\target)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Distribution **factorizes** for Model 2:\n",
    "$$\n",
    "\\pi(\\aligns|\\source,\\target) = \\prod_i^{l_{\\ssource}} \\pi(a_i|\\source,\\target,i) \n",
    "$$\n",
    "\n",
    "with\n",
    "\n",
    "$$\n",
    "\\pi(a_i|\\source,\\target,i) = \n",
    "  \\frac\n",
    "    {\\alpha(\\ssource_i|\\starget_{a_i}) \\beta(a_i|i,l_\\starget,l_\\ssource)}\n",
    "    {\\sum_j^{l_{\\starget}} \\alpha(\\ssource_i|\\starget_j) \\beta(j|i,l_\\starget,l_\\ssource) }\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "metadata": {
    "hideCode": false,
    "hidePrompt": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <svg id='08a74dce-b9a0-11e7-8cfc-0242ac110002' xmlns=\"http://www.w3.org/2000/svg\"\n",
       "             xmlns:xlink=\"http://www.w3.org/1999/xlink\"\n",
       "             viewBox=\"0 0 400 100\"\n",
       "             >\n",
       "\n",
       "            <text x=\"0\" y=\"20\" class=\"source\">\n",
       "                <tspan id='t0'>1:NULL</tspan> <tspan id='t1'>2:the</tspan> <tspan id='t2'>3:house</tspan>\n",
       "            </text>\n",
       "            <text x=\"0\" y=\"95\" class=\"target\">\n",
       "                <tspan id='t0'>1:das</tspan> <tspan id='t1'>2:Haus</tspan>\n",
       "            </text>\n",
       "            <g class='connections'></g>\n",
       "            <script>\n",
       "              $(function() {\n",
       "                  root = $(document.getElementById('08a74dce-b9a0-11e7-8cfc-0242ac110002'));\n",
       "                  root.find('.connections').empty();\n",
       "                  alignments = [['.source #t1','.target #t0',1.0]];\n",
       "                  function appendLine(alignment) {\n",
       "                      s1 = root.find(alignment[0])[0];\n",
       "                      x1 = s1.getExtentOfChar(0).x + s1.getComputedTextLength() / 2.0;\n",
       "                      y1 = s1.getExtentOfChar(0).y + s1.getExtentOfChar(0).height;\n",
       "                      s2 = root.find(alignment[1])[0];\n",
       "                      x2 = s2.getExtentOfChar(0).x + s2.getComputedTextLength() / 2.0;\n",
       "                      y2 = s2.getExtentOfChar(0).y;\n",
       "                      var newLine = document.createElementNS('http://www.w3.org/2000/svg','line');\n",
       "                      var score = alignment[2];\n",
       "                      newLine.setAttribute('x1',x1.toString());\n",
       "                      newLine.setAttribute('y1',y1.toString());\n",
       "                      newLine.setAttribute('x2',x2.toString());\n",
       "                      newLine.setAttribute('y2',y2.toString());\n",
       "                      newLine.setAttribute('style',\"stroke:black;stroke-width:2;stroke-opacity:\" + score + \";\");\n",
       "                      root.find('.connections').append(newLine)\n",
       "                  };\n",
       "                  //console.log(alignments);\n",
       "                  for (var i = 0; i < alignments.length; i++) {\n",
       "                    appendLine(alignments[i]);\n",
       "                  }\n",
       "                  //console.log($(root).find('.connections'));\n",
       "              });\n",
       "            </script>\n",
       "        </svg>\n",
       "        "
      ],
      "text/plain": [
       "<statnlpbook.word_mt.Alignment at 0x7fe4553a2c88>"
      ]
     },
     "execution_count": 207,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_mt.Alignment(\"1:NULL 2:the 3:house\".split(\" \"),\n",
    "                  \"1:das 2:Haus\".split(\" \"),[(1,0)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "$$\n",
    "\\pi(2|\\source,\\target,1) = \\frac{\\a(\\text{das|the}) \\b(2|1,3,2)}{\\a(\\text{das|the}) \\b(2|1,3,2) + \\a(\\text{das|NULL}) \\b(1|1,3,2) + \\ldots}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "For a uniform initial model: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "source_vocab = set([tok for _,s in train_model_2 for tok in s])\n",
    "target_vocab = set([tok for t,_ in train_model_2 for tok in t])\n",
    "\n",
    "alpha = mt.create_translation_table(source_vocab, target_vocab) \n",
    "#  {'is':{'ist':0.5, 'das':0.5}}\n",
    "beta = mt.create_distortion_table(5) #{1:1}\n",
    "init_model = IBMModel2(alpha,beta)\n",
    "align_matrices = e_step(init_model, train_model_2)\n",
    "\n",
    "def show_alignment_for_model(sent, fixed_alpha = {}, fixed_beta = {}):\n",
    "    alpha_1 = mt.create_translation_table(source_vocab, target_vocab, fixed=fixed_alpha) \n",
    "    #  {'is':{'ist':0.5, 'das':0.5}}\n",
    "    beta_1 = mt.create_distortion_table(5) #{1:1}\n",
    "    init_model_1 = IBMModel2(alpha_1,beta_1)\n",
    "    align_matrices = e_step(init_model_1, train_model_2)\n",
    "    return word_mt.Alignment.from_matrix(align_matrices[sent], \n",
    "                              train_model_2[sent][1], \n",
    "                              train_model_2[sent][0])\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <svg id='81ac279e-b9a0-11e7-8cfc-0242ac110002' xmlns=\"http://www.w3.org/2000/svg\"\n",
       "             xmlns:xlink=\"http://www.w3.org/1999/xlink\"\n",
       "             viewBox=\"0 0 400 100\"\n",
       "             >\n",
       "\n",
       "            <text x=\"0\" y=\"20\" class=\"source\">\n",
       "                <tspan id='t0'>NULL</tspan> <tspan id='t1'>the</tspan> <tspan id='t2'>building</tspan> <tspan id='t3'>is</tspan> <tspan id='t4'>big</tspan>\n",
       "            </text>\n",
       "            <text x=\"0\" y=\"95\" class=\"target\">\n",
       "                <tspan id='t0'>groß</tspan> <tspan id='t1'>ist</tspan> <tspan id='t2'>das</tspan> <tspan id='t3'>Gebäude</tspan>\n",
       "            </text>\n",
       "            <g class='connections'></g>\n",
       "            <script>\n",
       "              $(function() {\n",
       "                  root = $(document.getElementById('81ac279e-b9a0-11e7-8cfc-0242ac110002'));\n",
       "                  root.find('.connections').empty();\n",
       "                  alignments = [['.source #t0','.target #t0',0.25],['.source #t1','.target #t0',0.25],['.source #t2','.target #t0',0.25],['.source #t3','.target #t0',0.0],['.source #t4','.target #t0',0.25],['.source #t0','.target #t1',0.1111111111111111],['.source #t1','.target #t1',0.1111111111111111],['.source #t2','.target #t1',0.1111111111111111],['.source #t3','.target #t1',0.5555555555555555],['.source #t4','.target #t1',0.1111111111111111],['.source #t0','.target #t2',0.25],['.source #t1','.target #t2',0.25],['.source #t2','.target #t2',0.25],['.source #t3','.target #t2',0.0],['.source #t4','.target #t2',0.25],['.source #t0','.target #t3',0.25],['.source #t1','.target #t3',0.25],['.source #t2','.target #t3',0.25],['.source #t3','.target #t3',0.0],['.source #t4','.target #t3',0.25]];\n",
       "                  function appendLine(alignment) {\n",
       "                      s1 = root.find(alignment[0])[0];\n",
       "                      x1 = s1.getExtentOfChar(0).x + s1.getComputedTextLength() / 2.0;\n",
       "                      y1 = s1.getExtentOfChar(0).y + s1.getExtentOfChar(0).height;\n",
       "                      s2 = root.find(alignment[1])[0];\n",
       "                      x2 = s2.getExtentOfChar(0).x + s2.getComputedTextLength() / 2.0;\n",
       "                      y2 = s2.getExtentOfChar(0).y;\n",
       "                      var newLine = document.createElementNS('http://www.w3.org/2000/svg','line');\n",
       "                      var score = alignment[2];\n",
       "                      newLine.setAttribute('x1',x1.toString());\n",
       "                      newLine.setAttribute('y1',y1.toString());\n",
       "                      newLine.setAttribute('x2',x2.toString());\n",
       "                      newLine.setAttribute('y2',y2.toString());\n",
       "                      newLine.setAttribute('style',\"stroke:black;stroke-width:2;stroke-opacity:\" + score + \";\");\n",
       "                      root.find('.connections').append(newLine)\n",
       "                  };\n",
       "                  //console.log(alignments);\n",
       "                  for (var i = 0; i < alignments.length; i++) {\n",
       "                    appendLine(alignments[i]);\n",
       "                  }\n",
       "                  //console.log($(root).find('.connections'));\n",
       "              });\n",
       "            </script>\n",
       "        </svg>\n",
       "        "
      ],
      "text/plain": [
       "<statnlpbook.word_mt.Alignment at 0x7fe4553a2a58>"
      ]
     },
     "execution_count": 209,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# try  {'is':{'ist':0.5, 'Haus':0.5}}\n",
    "show_alignment_for_model(3,{'is':{'ist':0.5, 'Haus':0.5}})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### M-Step\n",
    "\n",
    "The M-Step optimizes a *weighted* or *expected* version of log-likelihood using distribution \\\\(\\pi\\\\) from last E-Step:\n",
    "\n",
    "$$\n",
    "  \\params^* = \\argmax_\\params \\sum_{(\\target,\\source) \\in \\train} \\sum_\\aligns \\pi(\\aligns|\\target,\\source) \\log \\prob _\\params^\\text{IBM2}(\\source,\\aligns|\\target)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Because \\\\(\\pi\\\\) factorizes we have a closed-form solution:\n",
    "\n",
    "$$\n",
    "  \\alpha(\\ssource|\\starget) = \\frac\n",
    "    {\\sum_{(\\target,\\source)}\\sum_i^{l_\\source} \\sum_j^{l_\\target} \\pi(j|i) \\delta(\\ssource,\\ssource_i) \\delta(\\starget,\\starget_j) }\n",
    "    {\\sum_{(\\target,\\source)} \\sum_j^{l_\\target} \\delta(\\starget,\\starget_j) }\n",
    "$$\n",
    "\n",
    "\\\\(\\delta(x,y)\\\\) is 1 if \\\\(x=y\\\\) and 0 otherwise\n",
    "\n",
    "$\\beta$ parameters estimated similarly ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {
    "hideCode": false,
    "hidePrompt": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <svg id='ff6c4838-b998-11e7-8cfc-0242ac110002' xmlns=\"http://www.w3.org/2000/svg\"\n",
       "             xmlns:xlink=\"http://www.w3.org/1999/xlink\"\n",
       "             viewBox=\"0 0 400 100\"\n",
       "             >\n",
       "\n",
       "            <text x=\"0\" y=\"20\" class=\"source\">\n",
       "                <tspan id='t0'>NULL</tspan> <tspan id='t1'>the</tspan> <tspan id='t2'>house</tspan> <tspan id='t3'>is</tspan> <tspan id='t4'>small</tspan>\n",
       "            </text>\n",
       "            <text x=\"0\" y=\"95\" class=\"target\">\n",
       "                <tspan id='t0'>klein</tspan> <tspan id='t1'>ist</tspan> <tspan id='t2'>das</tspan> <tspan id='t3'>Haus</tspan>\n",
       "            </text>\n",
       "            <g class='connections'></g>\n",
       "            <script>\n",
       "              $(function() {\n",
       "                  root = $(document.getElementById('ff6c4838-b998-11e7-8cfc-0242ac110002'));\n",
       "                  root.find('.connections').empty();\n",
       "                  alignments = [['.source #t0','.target #t0',0.2],['.source #t1','.target #t0',0.2],['.source #t2','.target #t0',0.2],['.source #t3','.target #t0',0.2],['.source #t4','.target #t0',0.2],['.source #t0','.target #t1',0.2],['.source #t1','.target #t1',0.2],['.source #t2','.target #t1',0.2],['.source #t3','.target #t1',0.2],['.source #t4','.target #t1',0.2],['.source #t0','.target #t2',0.2],['.source #t1','.target #t2',0.2],['.source #t2','.target #t2',0.2],['.source #t3','.target #t2',0.2],['.source #t4','.target #t2',0.2],['.source #t0','.target #t3',0.2],['.source #t1','.target #t3',0.2],['.source #t2','.target #t3',0.2],['.source #t3','.target #t3',0.2],['.source #t4','.target #t3',0.2]];\n",
       "                  function appendLine(alignment) {\n",
       "                      s1 = root.find(alignment[0])[0];\n",
       "                      x1 = s1.getExtentOfChar(0).x + s1.getComputedTextLength() / 2.0;\n",
       "                      y1 = s1.getExtentOfChar(0).y + s1.getExtentOfChar(0).height;\n",
       "                      s2 = root.find(alignment[1])[0];\n",
       "                      x2 = s2.getExtentOfChar(0).x + s2.getComputedTextLength() / 2.0;\n",
       "                      y2 = s2.getExtentOfChar(0).y;\n",
       "                      var newLine = document.createElementNS('http://www.w3.org/2000/svg','line');\n",
       "                      var score = alignment[2];\n",
       "                      newLine.setAttribute('x1',x1.toString());\n",
       "                      newLine.setAttribute('y1',y1.toString());\n",
       "                      newLine.setAttribute('x2',x2.toString());\n",
       "                      newLine.setAttribute('y2',y2.toString());\n",
       "                      newLine.setAttribute('style',\"stroke:black;stroke-width:2;stroke-opacity:\" + score + \";\");\n",
       "                      root.find('.connections').append(newLine)\n",
       "                  };\n",
       "                  //console.log(alignments);\n",
       "                  for (var i = 0; i < alignments.length; i++) {\n",
       "                    appendLine(alignments[i]);\n",
       "                  }\n",
       "                  //console.log($(root).find('.connections'));\n",
       "              });\n",
       "            </script>\n",
       "        </svg>\n",
       "        "
      ],
      "text/plain": [
       "<statnlpbook.word_mt.Alignment at 0x7fe4576c1160>"
      ]
     },
     "execution_count": 193,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_alignment_for_model(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "$$\n",
    "  \\alpha(\\text{das|the}) = \\frac\n",
    "    {\\sum_{(\\target,\\source)}\\sum_i^{l_\\source} \\sum_j^{l_\\target} \\pi(j|i) \\delta(\\text{das},\\ssource_i) \\delta(\\text{the},\\starget_j) }\n",
    "    {\\sum_{(\\target,\\source)} \\sum_j^{l_\\target} \\delta(\\text{the},\\starget_j) } \\\\ = \\frac{0.2 * 0 * 0 + \\ldots + 0.2 * 1 * 1 + \\ldots}{0 + 1 + 0 + 0 + 0} = \\frac{0.2}{1}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Implement M-Step, estimating parameters \\\\(\\params\\\\) from (soft) alignments \\\\(\\aligns\\\\)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "theta1 = m_step(align_matrices, train_model_2)    \n",
    "def show_initial_alpha(target):\n",
    "    plot_table_for_target(theta1.alpha, target) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl0AAAGdCAYAAAAogsYCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XfYJFWZ9/HvzZAUQRGGJIywiroYMCDoiiBmRMW4ggiC\nIouKYBZzXCMmlOCsIuqKGFFWUMQcWcGE4IqLiCu8KqAoGEn3+8d9mqlpH2Z6YKiufub7uS6u6VA1\nnJqqrvrVqRMiM5EkSdKNa7VpF0CSJGlVYOiSJEnqgaFLkiSpB4YuSZKkHhi6JEmSemDokiRJ6oGh\nS5IkqQeGLkmSpB4YuiRJknqw+rQLMJcNN9wwt9xyy2kXQ5Ikabm+973vXZKZC5e33CBD15ZbbskZ\nZ5wx7WJIkiQtV0T8cpLlfLwoSZLUA0OXJElSDwxdkiRJPTB0SZIk9cDQJUmS1ANDlyRJUg8MXZIk\nST0wdEmSJPXA0CVJktQDQ5ckSVIPJgpdEfHQiDgnIs6NiEPn+H6viDgzIn4cEd+OiG07353fPv9h\nRDi3jyRJWiUtd+7FiFgAHAE8CLgAOD0iTszMn3QW+wWwc2ZeGhG7AouBHTrf75KZl6zEckuSJM2U\nSWq6tgfOzczzMvMK4Hhg9+4CmfntzLy0vT0N2HzlFlOSJGm2LbemC7gV8KvO+wtYuhZr3FOBz3Xe\nJ/DFiLgaeE9mLp5rpYg4ADgAYNGiRRMUS5I0bstDT5p2EZbr/DfuNu0iSFMxSeiaWETsQoWuHTsf\n75iZF0bERsCpEfHTzPz6+LotjC0G2G677XJllkuSJGnaJnm8eCGwRef95u2zpUTEXYD3Artn5u9G\nn2fmhe3Pi4ATqMeVkiRJq5RJQtfpwNYRsVVErAnsAZzYXSAiFgGfAvbOzJ91Pl8nItYdvQYeDJy1\nsgovSZI0K5b7eDEzr4qIg4BTgAXAMZl5dkQc2L4/GngFsAFwZEQAXJWZ2wEbAye0z1YHjsvMz98o\nWyJJkjRgE7XpysyTgZPHPju683p/YP851jsP2Hb8c0mSpFWNI9JLkiT1wNAlSZLUA0OXJElSDwxd\nkiRJPTB0SZIk9cDQJUmS1ANDlyRJUg8MXZIkST0wdEmSJPXA0CVJktQDQ5ckSVIPDF2SJEk9MHRJ\nkiT1wNAlSZLUA0OXJElSDwxdkiRJPTB0SZIk9cDQJUmS1ANDlyRJUg8MXZIkST0wdEmSJPXA0CVJ\nktQDQ5ckSVIPDF2SJEk9MHRJkiT1wNAlSZLUA0OXJElSDwxdkiRJPTB0SZIk9cDQJUmS1ANDlyRJ\nUg8MXZIkST0wdEmSJPXA0CVJktQDQ5ckSVIPDF2SJEk9MHRJkiT1wNAlSZLUA0OXJElSDwxdkiRJ\nPTB0SZIk9cDQJUmS1ANDlyRJUg8MXZIkST0wdEmSJPXA0CVJktQDQ5ckSVIPDF2SJEk9MHRJkiT1\nwNAlSZLUA0OXJElSDwxdkiRJPTB0SZIk9WCi0BURD42IcyLi3Ig4dI7v94qIMyPixxHx7YjYdtJ1\nJUmSVgXLDV0RsQA4AtgV2AbYMyK2GVvsF8DOmXln4LXA4hVYV5Ikad6bpKZre+DczDwvM68Ajgd2\n7y6Qmd/OzEvb29OAzSddV5IkaVUwSei6FfCrzvsL2mfX5anA567nupIkSfPS6ivzL4uIXajQteP1\nWPcA4ACARYsWrcxiSZIkTd0kNV0XAlt03m/ePltKRNwFeC+we2b+bkXWBcjMxZm5XWZut3DhwknK\nLkmSNDMmCV2nA1tHxFYRsSawB3Bid4GIWAR8Ctg7M3+2IutKkiStCpb7eDEzr4qIg4BTgAXAMZl5\ndkQc2L4/GngFsAFwZEQAXNVqreZc90baFkmSpMGaqE1XZp4MnDz22dGd1/sD+0+6riRJ0qrGEekl\nSZJ6YOiSJEnqgaFLkiSpB4YuSZKkHhi6JEmSemDokiRJ6oGhS5IkqQeGLkmSpB4YuiRJknpg6JIk\nSeqBoUuSJKkHhi5JkqQeGLokSZJ6YOiSJEnqgaFLkiSpB4YuSZKkHhi6JEmSemDokiRJ6oGhS5Ik\nqQeGLkmSpB4YuiRJknpg6JIkSeqBoUuSJKkHhi5JkqQeGLokSZJ6YOiSJEnqgaFLkiSpB4YuSZKk\nHhi6JEmSemDokiRJ6oGhS5IkqQeGLkmSpB4YuiRJknpg6JIkSeqBoUuSJKkHhi5JkqQeGLokSZJ6\nYOiSJEnqgaFLkiSpB4YuSZKkHhi6JEmSemDokiRJ6oGhS5IkqQeGLkmSpB4YuiRJknpg6JIkSeqB\noUuSJKkHhi5JkqQeGLokSZJ6YOiSJEnqgaFLkiSpB4YuSZKkHhi6JEmSejBR6IqIh0bEORFxbkQc\nOsf3d4iI70TE3yPi+WPfnR8RP46IH0bEGSur4JIkSbNk9eUtEBELgCOABwEXAKdHxImZ+ZPOYr8H\nDgYedR1/zS6ZeckNLawkSdKsmqSma3vg3Mw8LzOvAI4Hdu8ukJkXZebpwJU3QhklSZJm3iSh61bA\nrzrvL2ifTSqBL0bE9yLigBUpnCRJ0nyx3MeLK8GOmXlhRGwEnBoRP83Mr48v1ALZAQCLFi3qoVhS\nP7Y89KRpF2Ei579xt2kXQZLmtUlqui4Etui837x9NpHMvLD9eRFwAvW4cq7lFmfmdpm53cKFCyf9\n6yVJkmbCJKHrdGDriNgqItYE9gBOnOQvj4h1ImLd0WvgwcBZ17ewkiRJs2q5jxcz86qIOAg4BVgA\nHJOZZ0fEge37oyNiE+AMYD3gmoh4NrANsCFwQkSM/l/HZebnb5xNkSRJGq6J2nRl5snAyWOfHd15\n/RvqseO4y4Btb0gBJUmS5gNHpJckSeqBoUuSJKkHhi5JkqQeGLokSZJ6YOiSJEnqgaFLkiSpB4Yu\nSZKkHhi6JEmSemDokiRJ6oGhS5IkqQeGLkmSpB4YuiRJknpg6JIkSeqBoUuSJKkHhi5JkqQeGLok\nSZJ6YOiSJEnqgaFLkiSpB4YuSZKkHhi6JEmSemDokiRJ6oGhS5IkqQeGLkmSpB4YuiRJknpg6JIk\nSeqBoUuSJKkHhi5JkqQeGLokSZJ6YOiSJEnqgaFLkiSpB4YuSZKkHhi6JEmSemDokiRJ6oGhS5Ik\nqQeGLkmSpB4YuiRJknpg6JIkSeqBoUuSJKkHhi5JkqQeGLokSZJ6YOiSJEnqgaFLkiSpB4YuSZKk\nHhi6JEmSemDokiRJ6oGhS5IkqQeGLkmSpB4YuiRJknpg6JIkSeqBoUuSJKkHhi5JkqQeGLokSZJ6\nYOiSJEnqgaFLkiSpBxOFroh4aEScExHnRsShc3x/h4j4TkT8PSKevyLrSpIkrQqWG7oiYgFwBLAr\nsA2wZ0RsM7bY74GDgcOux7qSJEnz3iQ1XdsD52bmeZl5BXA8sHt3gcy8KDNPB65c0XUlSZJWBatP\nsMytgF913l8A7DDh3z/xuhFxAHAAwKJFiyb866+/LQ896Ub/f9xQ579xt4mXnW/bI/XF385wzcK+\ngVV3/2jFDaYhfWYuzsztMnO7hQsXTrs4kiRJK9UkoetCYIvO+83bZ5O4IetKkiTNG5OErtOBrSNi\nq4hYE9gDOHHCv/+GrCtJkjRvLLdNV2ZeFREHAacAC4BjMvPsiDiwfX90RGwCnAGsB1wTEc8GtsnM\ny+Za98baGEmSpKGapCE9mXkycPLYZ0d3Xv+GenQ40bqSJEmrmsE0pJckSZrPDF2SJEk9MHRJkiT1\nwNAlSZLUA0OXJElSDwxdkiRJPTB0SZIk9cDQJUmS1ANDlyRJUg8MXZIkST0wdEmSJPXA0CVJktQD\nQ5ckSVIPDF2SJEk9MHRJkiT1wNAlSZLUA0OXJElSDwxdkiRJPTB0SZIk9cDQJUmS1ANDlyRJUg8M\nXZIkST0wdEmSJPXA0CVJktQDQ5ckSVIPDF2SJEk9MHRJkiT1wNAlSZLUA0OXJElSDwxdkiRJPTB0\nSZIk9cDQJUmS1ANDlyRJUg8MXZIkST0wdEmSJPXA0CVJktQDQ5ckSVIPDF2SJEk9MHRJkiT1wNAl\nSZLUA0OXJElSDwxdkiRJPTB0SZIk9cDQJUmS1ANDlyRJUg8MXZIkST0wdEmSJPXA0CVJktQDQ5ck\nSVIPDF2SJEk9MHRJkiT1wNAlSZLUA0OXJElSDyYKXRHx0Ig4JyLOjYhD5/g+IuLw9v2ZEXH3znfn\nR8SPI+KHEXHGyiy8JEnSrFh9eQtExALgCOBBwAXA6RFxYmb+pLPYrsDW7b8dgKPanyO7ZOYlK63U\nkiRJM2aSmq7tgXMz87zMvAI4Hth9bJndgQ9mOQ24RURsupLLKkmSNLOWW9MF3Ar4Vef9BSxdi3Vd\ny9wK+DWQwBcj4mrgPZm5eK7/SUQcABwAsGjRookKr/lpy0NPmnYRJnL+G3ebdhGmYhb2z6q6bzRs\n8+23M9+2pw99NKTfMTPvSj2CfGZE7DTXQpm5ODO3y8ztFi5c2EOxJEmS+jNJ6LoQ2KLzfvP22UTL\nZOboz4uAE6jHlZIkSauUSULX6cDWEbFVRKwJ7AGcOLbMicA+rRfjvYA/ZuavI2KdiFgXICLWAR4M\nnLUSyy9JkjQTltumKzOvioiDgFOABcAxmXl2RBzYvj8aOBl4GHAu8Bdgv7b6xsAJETH6fx2XmZ9f\n6VshSZI0cJM0pCczT6aCVfezozuvE3jmHOudB2x7A8soSZI08xyRXpIkqQeGLkmSpB4YuiRJknpg\n6JIkSeqBoUuSJKkHhi5JkqQeGLokSZJ6YOiSJEnqgaFLkiSpB4YuSZKkHhi6JEmSemDokiRJ6oGh\nS5IkqQeGLkmSpB4YuiRJknpg6JIkSeqBoUuSJKkHhi5JkqQeGLokSZJ6YOiSJEnqgaFLkiSpB4Yu\nSZKkHhi6JEmSemDokiRJ6oGhS5IkqQeGLkmSpB4YuiRJknpg6JIkSeqBoUuSJKkHhi5JkqQeGLok\nSZJ6YOiSJEnqgaFLkiSpB4YuSZKkHhi6JEmSemDokiRJ6oGhS5IkqQeGLkmSpB4YuiRJknpg6JIk\nSeqBoUuSJKkHhi5JkqQeGLokSZJ6YOiSJEnqgaFLkiSpB4YuSZKkHhi6JEmSemDokiRJ6oGhS5Ik\nqQeGLkmSpB4YuiRJknpg6JIkSeqBoUuSJKkHE4WuiHhoRJwTEedGxKFzfB8RcXj7/syIuPuk60qS\nJK0Klhu6ImIBcASwK7ANsGdEbDO22K7A1u2/A4CjVmBdSZKkeW+Smq7tgXMz87zMvAI4Hth9bJnd\ngQ9mOQ24RURsOuG6kiRJ815k5rIXiHgc8NDM3L+93xvYITMP6izzWeCNmfnN9v5LwIuALZe3bufv\nOICqJQO4PXDODdu0qdgQuGTahVhJ5tO2gNszdPNpe+bTtoDbM3RuzzDcOjMXLm+h1fsoySQyczGw\neNrluCEi4ozM3G7a5VgZ5tO2gNszdPNpe+bTtoDbM3Ruz2yZJHRdCGzReb95+2ySZdaYYF1JkqR5\nb5I2XacDW0fEVhGxJrAHcOLYMicC+7RejPcC/piZv55wXUmSpHlvuTVdmXlVRBwEnAIsAI7JzLMj\n4sD2/dHAycDDgHOBvwD7LWvdG2VLhmGmH4+OmU/bAm7P0M2n7ZlP2wJuz9C5PTNkuQ3pJUmSdMM5\nIr0kSVIPDF2SJEk9MHTpRhMRMe0yaP7y+BqmVWG/zKdtjAhzQI/8x54BEXGPaZdhRUXExtlpMDif\nTlKzKiJu2v6c+X0REevlPGuQGhE3jYiNpl2OG2q+7ZeuiLg/zP42RsT6EbFTRERmXjPt8qwsEbH+\ntMuwPIaugYuIfwLeHhEvaiP6D1obNuQ+wKci4pCI2BPmxUnqmRHx6Ii437TLcn1ExKOBV0fE/efB\nvngg8JGIeO3o+Jp1EXET4BnAf0TEEyNig2mXaUVFxBMi4h0RsXFErDft8qxsEXEr4CkR8R8R8biI\n2GK5Kw3XA4HHA6dGxH0jYrNpF+iGaoH4uIh4cETcftrluS72XhywiLhtZp7b5rG8L3A/YIvMfMR0\nS3bdImKNzLwyIrYFNgVeBnw/Mw9u38csXvQjYndgbWp73gd8JDN/O91STabVbG1JTTr/fOAk4EeZ\neeo0y3V9RMS6mXl5RNwG2Az4D+BY4PDM/MtUC3c9RcQtgL9m5t8j4u7AocBPgR9n5senW7rJRcTG\nwAuAoIYjWjxfhgiKiB0709ztAywCdgBenJlnTbVwKyAiFgJ3yMxvtPfPAG5HDfX0kcz88TTLd31F\nxILMvDoiHgHsCNyCOscdOeWi/QND10BFxHbAV4DdMvPr7bl7AB8CtszMf5lqAecQEXsArwG2zcy/\nts/Wo7bjO3PNuTl0rXbxXOC0zMyIuB3wBuAs4PjM/J+pFnA5IuIxwG2BIzPzTxFxR2BXYCvgqzN2\nUX8CsA/wlFHgbeHrPcDXMvO10yzf9dFquA6hxjFcnJkXR8Tm1LiH2wLfzMyPTLOMyxMRzwVWy8zD\n2vs7UzUp/wo8IzN/MM3y3VARsSXwcWqcyaPaZ7cEnkxt436Z+dOpFXBC7RryeOoG/suZ+an2+XbA\nQ4H1qPPE+VMr5PXQOS88NjP/1h7RbwO8EjglM9841QKO8fHiALXaoDOoQWY/EBG3BsjMqzPzicC5\nETG4BJ+Zx1O1KF+JiNVbrddlVA3dzhExcxdF4DbAk4DtI2LNzPwZdZHcFHjsVEs2mSuBdYH9W/nP\nBj4A/ADYISLuNNXSrYDM/ChVA/S2iLhZu7v9ObA/8MiIeM50S7ji2s3J96la1MdGxOqZeQHwsfb5\ntqPf/4B9A7h/RBwM0GpL3gl8GHh9RGw1zcLdUC2EPB14dETcs332e+Bw4FPAM1tt5aC1tltfA84A\nthm1f2rXmpPbYneEmWv3eRLwP8Cx7ZxwUWZ+FXgusGNEPG6qpRtj6BqYdtBkC16foGq6fpmZ13R6\nmbwC+NNQnltHxILR68x8DvCKzLyqPWZckJmXA7sBt2l3wYMXEasDZOargJ8BG2XmFW2/XAD8O3Wh\n33+KxbxOo/IDnwW+CJydmVcAZObFwOeoR0B3n04JJzd2fD0PeH9m/qk9TlitXRT3okLL9tMq54oa\n/Z4z8wvAJ4DPZuZV7bM/UBfCTaiblsEZ7ZfMPB14KfCH0XftAv9B4Ju08s/YhXz8uDuDqtE6vfPZ\n1dQ++jOwfltnkNvYKdfl1H45KjMvHX2emd8HzgYOiYi1Z6EJSOcc/Sfq+PtA2ycjZ1LBf+tWCTCI\nvDOIQqh0nkuvBixuoeqc9t1qnV4mv6HujHeaUlGv1S1zRLyxPe45dey7oMr8c2DjaZZ3Eq3cV7Uf\n6r0y852Z+V9QHQLavvgl8DTgIa2B7WB0yw88LjO/Nmq/1TnJXggcBxw85NqusePrDRFxx8z8Yue7\na9qfPwM+CtxyuiWeTAvv17RGv8/NzB9l5gXdi3bW/LWHA/tGxG2nV9p/NLZf7p2ZP8jMD7bvRsfY\nZdRj+J3b+8FfyEfGtu+EiLgr8P/ad6t1tvFsqtnHc9r7wW1jO19lROxK1aBulJm/63w/2pYPAD8C\nHjOdkk6uc45bEBHPysy/Z+bn2nejm5mrqRr9+1Lt2AbRS9PQNSCdwHUi8OfMPGeU3EcHTDtZ/w14\nK7BG1ETiUzNW5rUz8+ejE88ocGW5Avg6cNfuHeQQdbbp88ADuuXtXOhXo0LkmcAgfswjY/tkvO1f\nwLXH0Xepx0A36bmIE+tsy0nAGtlpmN2p6Rrd3f4c+OdplHNFtYvg7tTv+Oyxz1eDay+W3weOpmpT\nBmH0b975jezV/b67DZn5GeCvEfGAKRT1ehvbvvMy84ed89o13W0EXgx8KyLWmFZ55zIqTztf7UIF\n+Ddk5m8iYq32XY5ty/HUI/xB6+yfTwN3Gfvumk6Q/CnVnOIug6mFzEz/m/J/wLNY0qnhEVRvrJtQ\nF5DXA08F/qmz/Grtz7WnWOYnd17vR/V8Adia6sF0ALDBHOvdZNr/3svYpnU6r99LPSaFqlW8F3XR\nZ7Sv2uv7AhtOu+ytLHfovH4T8LL2+qbAozvfdct/J+Dm0y77HNvyiLHj62Od8j4feMlcxz9ws2mX\nfRnbtDFwn84++TjwT8CaVK31W0fH4Ng+2noovxvgVp3Xx1IXcaiOADsA63a+H52ntgBuOu2yT7h9\nzwQ2aa//hQq8awP3BN4NPG/sXBxU5cWgtq8da/uNjhvgRVRj83WpDgDfBF4+x3prDG1bxsq3S+f1\nS4G3t9e3bNt7u9F5urPcet1z+7T/s6Zrylr6/nPmtXcbP6NqTr5GPb7alPrBbzJaJ1utV1aNV+/l\nbbVrt+l8/H3gjxHxPeBA4G7UxfF24+tn69U4NFGNgO/f+ejPwFoR8SzgKOAw4IRRVf1oocz8RmZe\n0m9p/1FEPJz6Nx9ZE1gYEa+myv62iDgVln4EkplnZeYfey3sckSNUXW3zkenAVdExA9pNyDAranA\nu5Ss9h2D037nDwEujoh1soa3COrY+iDwaGD79np8H/3vEH437fHUIzsf/Qb4UUS8BTiCalT+nE4t\n16gG+IKcgeE82uP487NqgjYH/o/q5ft5qjPNH4F7A9e2pc1yzQC37xbAt4GbRY3B9d/AO4BPUoHs\nrcCjImKpmuHMvHKA2wJAVM/rbnkvps4Lx1PtnJ9N3Yyt210vMy/LzMHUFE899a3K/wF7d14/j+oF\nFMCdgYd1vvss1YhzCGXesfP6PSypgbg78ITOdycBj5l2eVdgu9Ztf+5D3TXdHTiFGgLjHsAGVO3X\n4GpSqEA+qlXYnxqn5uZUTcoLgFu3706khhuZepmXsS3bdV6/m7o4LKACZff4+gyw+7TLO+E2bQjc\npb3eAHgXFRjXoGrt7tm+uy3V5mbdaZV1GduwVvtzATVW3R2ABwDvB15FjZl2J+C/qDZDUy/zCm7f\nv9FqeIA9qJqghdTN5X07y32CGgZj6mW+ju3YgBpk95bt/eHAa9uxdhtg0/b55lQQ22raZZ5wu+7U\nef1y6unQzdrv5+mdfXcqsMO0y7vMbZl2AVbV/1q4+jE1LgrU48T3tx/1gvbZZu3C+f5pl7eV59bA\nZcCT2vt1gJ+MtqF9dkvqbmrxtMs74TY9FTioXRjXB05vF5Xuo8ag2gUcO+3yzlH+B1K1DNtTtVuH\ntaByt7F9cizVu2fqZV7GttyBaiP36Pb+tu39yzvL3Ixqx/HeaZd3wm1aQD32eBcVhm9GjR/0dmCn\nznKPpxoxP2raZZ5jGx7TLmbrtfevb7/xLTvLrNaOsSOnUcYbuH1rAEdS7Zlu0oLLoW0bb9+WuTUV\niD807fIuZ1vuTTUteFE7H+zSzgkvoT0WBZ5ADbHw6GmWdQW26c7tt/Hw9v5R7RywT2eZ9ameiu+b\ndnmX95+PF6dg1LgcuCtwx4h4RNbjg4OoLr2fao23NwN+kJn7tfWmtr9amX9J/YhfHRHbZVXZ3gPY\nJSIWt0XvSo2kfUBbb+jH2M+A7aiL3aXU2Fv3Al4UEQujxrI5jKpJ2hcG1y38V1TD18dQJ9l3ApcC\ne0VNlwN1wo3MfDIMrvzXymr0+myq2/pdMvNc6nHWEyLi5W2xewA/zMz9YfjHV1Yj/69QF7mdsx5/\nHg38Ftg9Ih7W9sfOVLj89AD3z8lUrcj72iO4w6lHvu+IiFEj5ncBV2TmM2C4x9hcMvNK6knD/1FD\n9PyOusk6nRpnbGsqVH4/M/eG4R53mfkd6inD1VRbza9QT0o2AB4fEYuoDieHZOYJs7CfssZ9exHw\n/Kgx306hQvKjI+KZbbEnA5dk5lNh2MefI9JPSadL8lLT4kTEOtQJbKvM3KXzeXfIiKkYlSEiNs3M\nX4/KHjWq9veogLjX+PLTK/FkIuJewGa5ZITmLagf9enUNDNrtsA5yG2KGqZjYWae1t5vTp2Ebk6d\ncL+brf3fEMs/LiK2ysxfdI6vW1Pb8dnMfHFnucFvy0jUsCJXZuZF7f1G1D7aihrq4hvttzXIabIi\nYm2qacFouI6NqV6LOwEvBP6UmdcOqTAr+6UramDaqzrvN6H20a5UrdCl7fNBbl/32ImIm2WnfWNE\n7EQ9Nv018PpcejyrmRARW2fm/7bXa1HH3jOoWtijOts+yP0zYuiaorEfybUHSkSsC+yZmYuX+RdM\nWbvbWy1rvJR1gFdm5gunXa4bYnTibXeEHwPemJmfbt8N8oLY1QnGW1DtHn6Ymce17wZf/q52fEW7\nObkNsH83dM2qTpjcmHq8/enM/Mm0y7WiWvmfCfw+M9/RPpupY2wuY+fiWwF3zzZO36wZu8bsAvwm\nBz512fJ0znFrAw8GbpOZb2/fDf74M3T15PoeDNM8iCb9f89xhzj4A39ZOj/qW2ZN9zGTZr38I6Na\n4c77mT6+YKngtUZ7vDWTImK9rEFQ571ZPe5mtdyTGL/2zAJDVw86tSdLXTxmwSyWeWUYu0Oc6ZPW\nkMs/9EcBKyJq+pTRY9zB/psvz/Up+6xsb0RskjUkxEyU97pExM2pzj7/b9pluTHM+v5ZlkE2BpxP\nIuJfgc9ExFq5ZEqc0XeDbOwXEYdGxIdgyajy0y7TDRURu8aS+QiXq/uDH8KPPyIeHhE7Xp91h1D+\nroh4TbTJz+dR4FofeFZ7hDO4f/NJRMRTIuKurQZuhX7zs7C9UaPivzsi7nBd5Z2Fc117rHYI1TB+\ni87nMbbc4LelKyL2jdYwfhaOp+vL0HXjO5nqufT+VmvUPZjWgkH2hHkrsCgijoRrp/UY/0FPHGCm\nLSJuQQ0D8dxYekqfmTgptfLvCOwcETssY7mZ2B5qvLNHRcShow9GZR/tnxnalpE1qFHmd4k5Jt2O\ngU0RM64dYxtQc3HeuVvLO7bcrO2XrnOoQadfGBEbdr9o2z8TF/tWm/odaqyt+8WS6X5G+2zL7vtZ\n0Lbht8DdIuIpy1huaNfKFTbzGzBUsWRU5j9Rg7l9aKxdyobAJ9oJbhB3+6MLXmtj8gDgjLHvbx0R\nn22PhK6ahRNwK+sfqHGQ7kdnLsIWJreINirzELenBfU/UD1ag7F5EiPiVhHx+NY2aNAn2SirZeb/\nAQ8Dru0EeGaPAAAZhklEQVRd1fbFIuCdo5uTIe6PcRGxcVRv3ouo4Tp+A1wwtsyDqEmrpzpP6rK0\nY+wj1KCgt+18nhFxm6h5ImfqQj7SORdfQI2D+N7szCLRHtW9NSIePaUiTqyzLadS4zp+rdsmMCLu\nDTw9xkaaH7q2DV+ljsFLu9+1c/Sr2iPHQVwrbwhD140glkyKHBGxWWb+Nf9xBvRLqPF79o0acmGq\nYskQFqtFxF6ZeVVmHtO+G01a/UtqcNR3wfBPwJ39sFpr+7BHZn5jbLEnUYNVDm57OvtkAbA61ZPy\nq2OL3YvqOr1Z3+VbEaMg1fbHBpn5q8x8d/tuNDntaNqVI9r7Qe2PcRGxDTX1zVsi4uktuLwnM//f\nWGDcmBqtfXATvUfEXSJi5/b24cA3M/OEscX+CXhKRGzbb+luuO45IKpX+MWZ+e323WgfXUGdi7cZ\ncg3+KHRExDYR8Wbgosz8v27tPVVbdFM6wXnIumVv18kvzXH8rQXcipoqa+YZulaydoEfzYB+EvC6\niPhYRNxzjqT+BWqguqk3VO+U+WRqVOPud9l5PHIocOYQguKyjO2H57bahjU7348u9G8AFkTEIVMq\n6pzasTIq/yepKXC6PURHv93PUtv1kCkUcyLdfRERH6NqeJ/SahhGx9fo5LsH8POIuNnUCjyBdvy/\nm6rdWkybj29Umz22TcdRc+HtMsdfNTXtGPob9fs4h5q+52dzLPpdahy+Qf/mx3VvJIEvU/vp46NH\nibTrX9bA1GdRF/fB1q62Y+rB1ETPu1HzXG7W2UYy8zzgBOaY93Zouue4iFjcarMe3f2+vfwFNWbi\nTadS0JXM3osrWTv4kxpU80zgQ8D5wOMz8wttmW7PuHUz8/IpFZdueSLirdQM7QdHxHrU47jzMvOs\nzrJrUoOFDnJiYVhqe4KaRmlN6qL3OeC/RtvTHsldGRE3peZUvGh6pZ5bRHwEODszX9fe3z4zz2mv\nRxeVdYBbZOaF0yzrsrR9cQR1J/5NapT8E4DjR4962jJBHYN/n1ZZJ9F+B6OJ0G9Pzf/2ZmpA2vUz\n831tuVHP5UH10uwcOzehbg4XAa/NzA9cx/I7Av+bmb/ts5wrQ0Q8j5qf9E1UU497AQ/JzD9EZ8iO\nGBtQdGgi4u5UzepewLbAltTv5S2ZedHo2jP0GuJxEfEeKtB/CXgx8IbRcdg5Ttenrjszd/yNs6Zr\nJYmIV0fEvduJdW3gQmq+qA9RJ7MvRLW/WartzbQDVyvDqDz/C1wUEccBb6AmSn1aRGzQWfaKIZ+Y\nYKnteTvwi8x8JDVQ6CbAIyPiTm25K9uffxlK4Oo+lmr/7usBZ0TEPhFxFPDliBg9Dr26Xcz/PMTA\nFRH7dd7+GzXJ81GZ+SXq4vcgYM+oQTYZPX4ceuCC+h0Al1DB6xvAL6ljbBvg36NNW9QC17U13GOP\nHadirAbodtS+2Zf6bTyrs9zmo9eZ+c1ZueBFxCOj9eprN5KPomYzuCQzDwG+DZwSNY7dte2hhnpe\n6xwzi4AvZea3MvNI6qnEXalpsxa2387gA1dE3KPz+jDqp79PC1r/Brw0IvaFJb3nM/PSWTn+lsfQ\ntfL8ETgqIu7eqqvXoGq7vpyZh7Vl3k/NJTcoEfHRVq17InAl8ANqrqsHUpMQbzTF4k1sLLAsonr3\n7NZqE39E/ftvBuzdvaAMRXR6t7Yw9Tvgg8B7qIvjh6igsmVUt/HBDrkQ1X5mrc5HZ1I3Ii+OiJu3\ndjVvBPYE7jLHXzF4mfki6rfyceAVmfmoViP5ZGCDVhs2qOFHxh5bfx54etbUKt8F3gc8ICKe02of\n7j/Nsl4f7Xe/IDN/1R4jHkl12NilHZNk5nOoHuUHT6+ky9c5n42u0z+hevftBpDVvvMX1M3kQ8bW\nGaSoYTv+qfPRasDdI+JerVb4a1TweldE3Bem/5tZ2Xy8eANFZ/DQiDgC2IGaxX3U5uMD1HxX+wGX\nZ5uod5rGH3VExB7A64BnZuYp7bM1gWOAv2SbvHrIxvbDepl5WVQP0TdRj3uf2z7bHrhDZn5wmuUd\nN1b78G4q/H4+Mz83egQd1ch3MfDnzHzWMv/CgYiIVwDbZuZjI+KeVMeFS4G3tf1x26yJrQev89h6\ne+CfgXMy87RW8/gI6sK3CTUh79OydZ4ZimhzWrbX/wH8LjMPbb/1e1DNILYCXkA10v63qRX2Bmq1\nrPcA/p1qC/QeqmbofZn5x2mWbUVExP2BhwLfosr/JKoH9ulU7/KjqfZq68zCOSEi1szMKyLiQGqC\n6k9ExOupR6WvAX6W1Vngn3PGpyu6LoaulSCWNJr/MXA3arybx7U/HwlsCvw6M18+Wn7aNRTtjmhU\nroyIRwFvA56dmSdGjaG0RWY+c7T8UO84Rv+ebT+cAlwO/C0znxgRm1Lth9YBntM94Q5tm6IaXn+G\n6kmV1Lx27wQ+TbWFOo4K7vu25QdVflg6/Lb3G1MXhosz84Coccb2ojqPvDhnbAT3iHgEdYNyBtVW\n8ErqzvzF1Ha9BfhJq8kbjIh4IHD7zDyivX8dcC71eGot6jz13sx8Zff8NEP7Zfy425baH1dQNzE3\nY8mj4Le0pxGD3r6IuA/VDvJT1I3826mwtSn1JOLPVKP6hcCBwD6Z+ZfplHbZxm6K1waeDWxBPfb9\nXES8kQper6PasM7E5NXXh6FrJYiIpwEPzMwntCrtPYGnUJNWn9v9YU/7IOrcrR9K1co9j2r3lG07\nXgYcAHyltVuZepkn1S4ka1Jt0T5HTcT7yBa8DqO26b3TLOO4iHgC1U3/woh4JNUm6O1UiD+PGm7g\n69Tk27dsj0kHuU/Gauv2oB7rnEI1Ll9MBfynR8ROwK0z80NTLO4Ka6H4KOA/M/Pr7VHW/sCVmfna\niHgbNXn116da0DHRGoi3/fIyqsfrWlQ7rguofbMucDhwYNbQHYMOJF1jx92LqbHSTgcuAp5L9dA8\nkupMs3tmvmlqhZ1Q1ATv7wA+0GqD7gO8kHokfCwV9lcDdm7L7ZmZZ06puMs0tn+2o24c/6ddb+7K\nkuB1NPDj0Y3BfGWbruthjufmfwH+2l5fTg3AdznwlWijA4/Wm9aFMpYMfJrtzzdSbWxeQd1hAJxK\ntb15aCdwzcSAdFENl3egeidenpk7AreIiBMz89fAMwYYuNalAu5zImLDzDyRqm18AdUW8ADgbOqR\nwu06gWtw+ySWHqLjy1SPvpdR7Rr/BjwN2CwiPpaZXx8FrqG3QRlpNXS3pxoz3619fCHw37QxkTLz\nuS2MDWabogY1PTkiNmrHTFLj0v0hM/8tM18L/I7aV78cBS6YnbY0nePuM1SY3JQKJ5cCx7fPXkqF\n/jfBcI+7Trm2ojpk7RnVBvJbVBvIx1G1WlD78k7AY4YauGCp/XMS8FjgfRHxKio8/g/V7vYxmXng\nfA9cYOi6Xjq1Vs+KatR4IrBtRLw0M6/OzIup8PK6zDy/E3SmchKLpQc+fXVEPKm1pTmICoevjmpI\n/07gpKyGpkyzzMsTSw8ICDUtxmXAjhFxa4DM3Am4fURc+1hxKCfbqF6slwP/SrU9eXVErN/C7hos\naVy+ETWswpdG6w5tn0S1ORs93j2QCoyvpO7Gv5WZl7XfxAHAad11h7Ytc4ka3fudVM3Ja4GHR8Rj\n2+OSPwCbRsTC0bE1pG3KzM9QN1PvbY91jga+CLw5lkwp9Vbg75n5dBjOb2R5IuLOnbePpIYieT01\nWPCbMvPvmfl9qjPKt7LTtGBI+wiW+jffKKpB+RepR4i/pm7K1svM71C9fr+RNXj1lcA7sg0hM3Bv\npY6717b3v23l/w+q5++6owVn5fi7vny8uALGnkuPnqMvpB79XEDdaf2Iuhv+eWY+tS079UdBLaSc\nSnWX3gJYHzgyMz8fEc+lRvy9WbbGs0N+tDBWXb038ENquIttgOdQJ9+Ts0bQH5xY0gZtTapH5eXA\nY6i2W4dS7U+OpO7Yz8rMJ7f1BrdP2iPRFwCPzRor6EHUFFI7A5/MzMOieorulJnHddYb3LbMJapt\n0OepYV+ObLWTD6BqJL9A9fA9JDNPmmIx/0FE7A38S3ucuwC4b7bZDKI6mDyBavj/CuDCFooHca6a\nRLvZfRTwqfZo6kHAE6lz239l5jvbvnoG8M4ccNvBzvngYcCrqEbzV2Tmi1ow3pOqLX59Zl42xaJO\nbPzfOWookp9SzVm+kplvioitqaYH35+FY25lMXStoJbCN8zMi9vF5PHU47njqEdBO1Lh5ROj5adY\nw3UvKvxdHBHPobpSHxYR36Ea0V5DzQn5xbH1Bn/ijSWj5/+KqrH9M3UXtQX1w/4B8P6sYRcGc7KN\nJT0RF1DtT+6UmXtExEZUD6uLWdJIdtvMPL2tN4jyz6XzaPfxwPZUY9iv5pKOI5+mLuzPnF4pJzf6\nt+6E+y8Dm2TmNp1ltqLa2/0lM88c2v5pweoglvQSHb8IbkjVPP4tM9/WPhvUNixLRGwG7ANsSJ0H\nvkk91o7MvE9b5pPABVljcw1ORKyVbUy6iLgfNb3aE4DdqbZoJ2XmvlHtufYC3p41vMegxVinhvbZ\nftSQJC/KzLe0z06g2rO+tb2fmePvhjB0TSBqEL3ft9f7UxfFXTLzgqjGtM+ixlE6OjvdxKcZXtod\n0supMHgC1Xvvb9QP+4ftTvCb1Phir89qMzDoAz8ibpOZP2+v30QNTvk2qnbxx1RQeSE1tti2mXnU\ntMo6lzlqhZ4E3BM4NDP/GtXT7ydUjenB2QZrHOI+maMmZcesMXaIiIOpBrPrUO1SLhnV1g1dJ3Bt\nT7U/eUVm/j0ivkI9lZqJsauiRvBeTA2meXT7bDx4DXoE9rnE0j0rN6Q6MmwKfJS6kfwvqq3dOsD5\nQ625jyVjiB2Zmd9sNXW/pp44vIbq5PABqqb7KVHtumZpqIvVqM5MF1OdGt5PPRnal+rU9BiqR/N+\n1/V3zFe26VqOiDiIujACkNUY++PARyNiUVbD04/Tpv/orjvN2qLM/G/gP6n53h4L/J1q8H9L6kcA\ndXL6+ChwtfUGc2LqiohjgI9ExGj+uo9SF5WTqAaZr6Qu9O+luhwf1dYbTPuArIbyXwCOiZp+5bdU\nbckd2l3vb6lHWb/oXgwHuk8+B/y2tTW5muphCUBmHk71qDqcGoF+9Hh08OebFrgeSDW8Phg4NiLW\nzsxdgL9FxGnL/huGITMvpWocXxoRT2yfZTTt/bWhfnolnVxEbNoew0VEfJjq2PABqrfi46nG5/eh\narxf0wlcqw3pN9TK8wdqvsfnRcR2mXkqFRr3oh5l/w/wVeA+EXHnWQhcsXQ72wOppw0fpgapfjnV\n8/fV1I3Yl0aBaxbOCyvTKrWxK6rVTHwtM0+JiL2juriSmS+h5ok6KarNx8FU4+HjlvHX9SIirh3t\nNzOPpwZpfRFLRsL/LPCp9sjk6sw8tq039BPvz4EFwP0j4qFZDWQDuDQzD8uaqPdnVBuPQTWYbcfO\nqNbt9cBhmfnXdqL9IbV/Do+IL1EDVr62rTfkfXI1cEeqHc1cF/TvZ+bXMvOzMMwelyPdk35E3IUa\n1+lQqo3gusDbogZ1fBjw11jSAH3Qsnq77ge8LJZMq5JtX63WWW7qv5HliRpH8M1Rg2Ym1Unjwqye\nye+ngtcTqZ7XPxiruR/McRc1tdcHoqbteT11s/LKiNghq93ZX4DNY8nwMQ/PzB9PscgTiaUntf9X\nqlflh9uTn08CV1HtB7+RmUdm5ts76w1m//TB0HUdoho5B3BWRJxIdQt/bER8MSI2ycxXUENDvIoa\nd+Q1bb2pXSijZqD/cHs0QtSov1tQtT8PoBpkvhd4GNW49IltuUFVvV+H06jj9Rrg3hGxK/Vo9M4R\n8eGIOBn4Y2b+JwwusIzXCn1t9EXW0B3vpsLwxzPzYBj+PrmumpRlLD/IbYmIWwIfjIhN2kdrUI94\nf541evveVGP59wNk5i6tFnkmZLXXPIC6sB8c1Vh7sNNHzSVqQvrvU+PWPSMi7kjVEl8BkJm/YUmH\nlHW76w7tuMtqY3ohVdu9PlXuU6hgfAfq6cQO1I3YsTkbbbhWH9VAUuexR1OdNA4ByMzTqE5CNwHu\n1113lo7DlcU2XXPoXvCixhNZnXp8dQo1wN7fqXY5v4nWMLotO802XDej2jQdQk0TcQE1ntDeWW3P\n9qKmk/g61cD8qmmXeXki4snUo7avt/cvonqG/ppqQPtBqvPC04Fr2mOtwQWWuI72NdTv7x/+7Ye8\nT8a1R3GHA28e1Zq2z2dpG94A3JmqFVqHejx1JDVQ45/azcuzqIFPXzq9kl5/UT3FHkTNe3dWd18N\nWdQ4Y8+hLuQ3odpwbUEFk1OpMPYHqhf5Sdl6YQ5NVEeZl2bmIe23fzA1KOjPo9qmPZGa6/IlmfmT\nWNLhZlDnsnGx9BRlLwG2ysz9Wo3eKVRPxRe0Za9tk7sqM3SNaan9qs77DamBT4+lXTQj4mxqtO2H\ntGfz0+6lOOrt8pjM/F3UEBAvAfbPzE+PyhYRTwHWHF34hyxq3LBPUne3H6HuALekTq5fYenu4ad2\n1hvkxb49hv4s1XvnuPbZoE+ok4qIHanJuN8OnJuZJ0+5SMs1x0XwWdRF8LyIeD7VjvNb1O98H2rb\n9geeOLrJmlWtxnUmhh4AiJrb8m5Uu607UW1UH0SNm3YqdQO2brbxBYf6u4qII4EjMvPs8fNUu87s\nSwWvJ1K19oPbhq7odAyiah3fQI2Rtldm/jCqY9DngO9l5tM66w1y//Rl9WkXYEgiYp3M/HNL7TsB\nf83M77ST8p+oho5QXZMvHQUumG41dmZ+pl3Uj4maVuZYqs3D3hHx26xB9cjMY6ZVxhUR1avqhKhe\ncA+hLoDnUyejbajRzRdTPRUXdtcdYuCCal8T1W368NY+6NhObeogg+KksnpfPZi6EN4/avTzY6dc\nrGXK6j26RkTcsV0E3z3aB1nDqjyRqhXambqhWY/qLHP1df+ts2EWAld0esdSj7HvmzWY5g8i4krq\nUeLmVO3WzzvrDe6C3q4fC4DfU6Hq7NHjuFFZM/OSiPgA1cTgD8v46wYja47ebakBTh9H3ZhcDOwb\nEcdkDaWyGzVWWne9Qe2fvlnT1bST7PpUjcoJ1A/kntTYIqdQvQBvR40U/q1sY79MuYZr0gEQ35yZ\n35xGGVdUq7V7HtWl+A/Ak6lBEN9CNcZ8IfCJzPzPmLFu1DCbtUIraug1KZ2L4KuokbHf1fl8tVx6\n4uQFwK7Amxjw/HbzTSx/nLFtqfZ2Z2bmB6dVzhURNbPB54GXj8o8xJC4PHNcd3bKzK+077alOm1t\nRLVJ+15nvZnb1huDDemXWJcKVa+m2jw8jmoDdXPgXlT7oWOAY4YQuJplNdC+hBpW4dvUgJUzIWva\nki9SYXd16pHcKVQ19iWZ+SgqFJMDm9pnEi38PpgKkPeP1qtsPhly4IJre/BdRXVnf35E7DP6nOqo\n0V32amoOv8cbuHo1Z+/Y0ZdZPTPfOSuBCyBrGIi9gRe24NKdUm6WrsXj152vjr5o++Uz1JiQ23RX\nMnCVVb6mK5ae2udJ1PPpm1In2csiYhvqINstM8/qrDf1R0LX1UB77I5wJgZAnKDWbk/q8dXM1NpN\nYui1QvNdROxENZp/U7ZJuNvn076hWuVdVztIWPoCPmv7KmqswQ9Sw8eclJ1JxmfBhNedLTLzV9Mq\n45DNUrpe6WLJNB8LALKGG3gX1R35ca2N10+o8Z9u0l132oGrlWE+DYC4vFq7jzBjtXaTMHBNV1bP\n2GcBr4+Ip0fNMOFd+QDkPBpnrKs9itsVuCvw7Ih48ZSLtEImvO78CmbiutO7VbamK5ZMMroadTf1\nM2BrqkHgaJyRzakAFpm559QKuxwxP7rtz5taO82eiLgTFb7+TE1P8oYpF0nNfG0HGTUrxQLgHsB3\nM/OvUy7SCpkP151pWGVDF1ybwj9GDbz5IWpU452yemM9kup18fHMfN9o+aHeVc2HE9N8fZyg2TDr\nF8H5LGZ0nLH5bj5cd/q2yoWuqNnpL8sa9HAtauC9D1MNtz+fmW+LiM2zBhS9Z2ae3tYb/IV+PpyY\nvHuStCy2gxyW+XDd6dMqFboi4h3AWlRvuC9QPRaPpab4eXnWXIVExCepQey+3N4PPnCNm+UTk3dP\nkjR7Zvm605dVJnRFxDFUY/jnAn8eHRgRsTNwEvAIamyul1CDou47paIK754kSfPPKhG6Ws+Xf83M\nh3U+24wafO9wat61p1OjHF+Smc9ry8xcDdd85N2TJGk+WFVC1yHA2pn5pjY8xAbAN6jJkv8OvII2\nxU93sDrbEEmSpJVlXo/TFRHrtpcXAXeJmjbmamq03Fdn5mOAM4F/gaUCVxi4JEnSyjRvQ1cb8uFz\nUTOdfwMI4L4teF02GpIA2A5Yo/sY0UeKkiRpZZu3oSszT6R6Kb4nMy8AvgvsCzwqIu4eETeJiE8B\nv8nM906xqJIkaRUw79p0zTGH386doR/2A+4APAH4DtVL8SntOxvNS5KkG818DF0bUr0S35Y1YfU/\nhKmI2BT4U2Ze3t7baF6SJN2o5uPjxauBOwJLTcTZXSAzf90JXDaalyRJN7p5V9MF1z2Hn48PJUnS\ntMzHmi4y80fAfsDL2sCoS42/NcWiSZKkVdS8rOkacQ4/SZI0FPM6dIFz+EmSpGGY96Gryzn8JEnS\ntKxSoUuSJGlabFQuSZLUA0OXJElSDwxdkiRJPTB0SZIk9cDQJUmS1ANDlyRJUg/+P4UlH9S+y7wJ\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe45539d588>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_initial_alpha(\"is\") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "align_matrices = e_step(theta1, train_model_2)\n",
    "def show_initial_alignment(sent):\n",
    "    return word_mt.Alignment.from_matrix(align_matrices[sent], \n",
    "                                  train_model_2[sent][1], \n",
    "                                  train_model_2[sent][0])    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "How do alignments look like with these parameters?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {
    "hideCode": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <svg id='2bd297d4-b9a2-11e7-8cfc-0242ac110002' xmlns=\"http://www.w3.org/2000/svg\"\n",
       "             xmlns:xlink=\"http://www.w3.org/1999/xlink\"\n",
       "             viewBox=\"0 0 400 100\"\n",
       "             >\n",
       "\n",
       "            <text x=\"0\" y=\"20\" class=\"source\">\n",
       "                <tspan id='t0'>NULL</tspan> <tspan id='t1'>the</tspan> <tspan id='t2'>house</tspan> <tspan id='t3'>is</tspan> <tspan id='t4'>small</tspan>\n",
       "            </text>\n",
       "            <text x=\"0\" y=\"95\" class=\"target\">\n",
       "                <tspan id='t0'>klein</tspan> <tspan id='t1'>ist</tspan> <tspan id='t2'>das</tspan> <tspan id='t3'>Haus</tspan>\n",
       "            </text>\n",
       "            <g class='connections'></g>\n",
       "            <script>\n",
       "              $(function() {\n",
       "                  root = $(document.getElementById('2bd297d4-b9a2-11e7-8cfc-0242ac110002'));\n",
       "                  root.find('.connections').empty();\n",
       "                  alignments = [['.source #t0','.target #t0',0.12765957446808507],['.source #t1','.target #t0',0.10638297872340426],['.source #t2','.target #t0',0.31914893617021284],['.source #t3','.target #t0',0.12765957446808507],['.source #t4','.target #t0',0.31914893617021284],['.source #t0','.target #t1',0.19999999999999993],['.source #t1','.target #t1',0.20000000000000004],['.source #t2','.target #t1',0.20000000000000004],['.source #t3','.target #t1',0.19999999999999993],['.source #t4','.target #t1',0.20000000000000004],['.source #t0','.target #t2',0.18749999999999994],['.source #t1','.target #t2',0.3125],['.source #t2','.target #t2',0.15625],['.source #t3','.target #t2',0.18749999999999994],['.source #t4','.target #t2',0.15625],['.source #t0','.target #t3',0.12765957446808507],['.source #t1','.target #t3',0.10638297872340426],['.source #t2','.target #t3',0.31914893617021284],['.source #t3','.target #t3',0.12765957446808507],['.source #t4','.target #t3',0.31914893617021284]];\n",
       "                  function appendLine(alignment) {\n",
       "                      s1 = root.find(alignment[0])[0];\n",
       "                      x1 = s1.getExtentOfChar(0).x + s1.getComputedTextLength() / 2.0;\n",
       "                      y1 = s1.getExtentOfChar(0).y + s1.getExtentOfChar(0).height;\n",
       "                      s2 = root.find(alignment[1])[0];\n",
       "                      x2 = s2.getExtentOfChar(0).x + s2.getComputedTextLength() / 2.0;\n",
       "                      y2 = s2.getExtentOfChar(0).y;\n",
       "                      var newLine = document.createElementNS('http://www.w3.org/2000/svg','line');\n",
       "                      var score = alignment[2];\n",
       "                      newLine.setAttribute('x1',x1.toString());\n",
       "                      newLine.setAttribute('y1',y1.toString());\n",
       "                      newLine.setAttribute('x2',x2.toString());\n",
       "                      newLine.setAttribute('y2',y2.toString());\n",
       "                      newLine.setAttribute('style',\"stroke:black;stroke-width:2;stroke-opacity:\" + score + \";\");\n",
       "                      root.find('.connections').append(newLine)\n",
       "                  };\n",
       "                  //console.log(alignments);\n",
       "                  for (var i = 0; i < alignments.length; i++) {\n",
       "                    appendLine(alignments[i]);\n",
       "                  }\n",
       "                  //console.log($(root).find('.connections'));\n",
       "              });\n",
       "            </script>\n",
       "        </svg>\n",
       "        "
      ],
      "text/plain": [
       "<statnlpbook.word_mt.Alignment at 0x7fe45513bc18>"
      ]
     },
     "execution_count": 211,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_initial_alignment(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Initialisation\n",
    "Good initialisation is **crucial for EM**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Baby Steps\n",
    "Initialise with the parameters of a simpler model with **fixed distortion** table:\n",
    "\n",
    "$$\n",
    "  \\beta(a_i|i,l_\\starget,l_\\ssource) = \\frac{1}{l_\\starget + 1}\n",
    "$$\n",
    "\n",
    "This is **IBM Model 1**!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Train IBM Model 1 and see if it **converges**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "ibm1_iterations = em_model1(init_model, train_model_2, 100)\n",
    "def plot_ibm1_change(end):\n",
    "    plt.plot(range(0,end), [change for _, _, change in ibm1_iterations[:end]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmMAAAFpCAYAAADQuy+GAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X2UXXV97/HP5zxNSDJAQoZJTIAECWAUeXAKVL29Ij4A\n9TZw24vQFqK1i9KCii33NrW9LV0ur1wXSnVdSoyaFm+tFB+QtE1LU2rVuxSbAXlKICRGMAl5GJ4T\nEjNP3/vH2ZMcDufMnEnO3nsmeb/WmnX2/u3f3ue3Zy/IZ/bvt3/bESEAAADko5B3AwAAAI5khDEA\nAIAcEcYAAAByRBgDAADIEWEMAAAgR4QxAACAHBHGAAAAckQYAwAAyBFhDAAAIEeEMQAAgByV8m7A\neMyaNSvmz5+fdzMAAADG9MADDzwbEV1j1ZtUYWz+/Pnq7e3NuxkAAABjsv10K/XopgQAAMgRYQwA\nACBHhDEAAIAcEcYAAAByRBgDAADIEWEMAAAgR4QxAACAHBHGAAAAckQYAwAAyFFLYcz2RbbX295o\ne2mD7b9h+xHbj9r+ge0zx9rX9kzbq21vSD5ntOeUAAAAJo8xw5jtoqTbJF0saZGkK20vqqv2U0n/\nOSLOkPQJSctb2HeppPsiYqGk+5J1AACAI0ord8bOlbQxIjZFRL+kOyUtrq0QET+IiBeS1fslzWth\n38WS7kiW75B06cGfRntsfn6P/u2JHRocGs67KQAA4AjRShibK2lzzfqWpKyZD0n6pxb27Y6Ibcny\ndkndjQ5m+xrbvbZ7+/r6WmjuwfuXdTv0W3/dq1f2DaX6PQAAACPaOoDf9gWqhrE/HM9+ERGSosm2\n5RHRExE9XV1dbWhlc5WiJUn93BkDAAAZaSWMbZV0Qs36vKTsVWy/WdKXJC2OiOda2HeH7TnJvnMk\n7Rxf09uvUqr+OgYIYwAAICOthLE1khbaXmC7IukKSStrK9g+UdK3JF0VEU+2uO9KSUuS5SWS7jn4\n02iPcrH66+gfJIwBAIBslMaqEBGDtq+XdK+koqQVEbHW9rXJ9mWS/lTScZL+0rYkDSZdiw33TQ59\ns6S7bH9I0tOSLm/zuY3bSBjjzhgAAMjKmGFMkiJilaRVdWXLapZ/W9Jvt7pvUv6cpAvH09i0jXRT\nMmYMAABkhRn4a1TopgQAABkjjNU40E3Z8MFOAACAtiOM1eBpSgAAkDXCWI3yyDxjdFMCAICMEMZq\n7J/agjtjAAAgI4SxGh10UwIAgIwRxmowzxgAAMgaYaxGucTUFgAAIFuEsRr7B/AztQUAAMgIYaxG\nR7EoSRrgzhgAAMgIYaxGuTRyZ4wwBgAAskEYq7F/AD93xgAAQEYIYzVKBcvmaUoAAJAdwlgN2yoX\nC9pHGAMAABkhjNWpFAsaGORpSgAAkA3CWJ1KqUA3JQAAyAxhrE65aCZ9BQAAmSGM1SkXuTMGAACy\nQxirUykVmGcMAABkhjBWp8KdMQAAkCHCWJ1yscCYMQAAkBnCWJ1y0RrgReEAACAjhLE6jBkDAABZ\nIozVoZsSAABkiTBWhwH8AAAgS4SxOszADwAAskQYq0M3JQAAyBJhrE51Bn6epgQAANloKYzZvsj2\netsbbS9tsP102z+0vc/2jTXlp9l+qObnZds3JNtusr21Ztsl7Tutg8fTlAAAIEulsSrYLkq6TdK7\nJW2RtMb2yohYV1PteUkfkXRp7b4RsV7SWTXH2Srp7poqt0bELYd0Bm1W4UXhAAAgQ63cGTtX0saI\n2BQR/ZLulLS4tkJE7IyINZIGRjnOhZJ+EhFPH3RrM8CLwgEAQJZaCWNzJW2uWd+SlI3XFZK+Vlf2\nYduP2F5he8ZBHLPteJoSAABkKZMB/LYrkn5F0tdrim+XdLKq3ZjbJH2myb7X2O613dvX15d6W0cG\n8EcwiB8AAKSvlTC2VdIJNevzkrLxuFjSgxGxY6QgInZExFBEDEv6oqrdoa8REcsjoicierq6usb5\nteNXKVV/JQziBwAAWWgljK2RtND2guQO1xWSVo7ze65UXRel7Tk1q5dJemycx0xFuWhJYnoLAACQ\niTGfpoyIQdvXS7pXUlHSiohYa/vaZPsy27Ml9Uo6WtJwMn3Fooh42fY0VZ/E/J26Q3/a9lmSQtJT\nDbbnolKs5tOBwWGpI+fGAACAw96YYUySImKVpFV1Zctqlrer2n3ZaN9XJB3XoPyqcbU0I2W6KQEA\nQIaYgb9OObkzxlxjAAAgC4SxOh3JnTGmtwAAAFkgjNXZf2eMMAYAADJAGKtT3j+An6cpAQBA+ghj\ndZhnDAAAZIkwVmdknjEG8AMAgCwQxursn2eMO2MAACADhLE6FZ6mBAAAGSKM1WGeMQAAkCXCWB2m\ntgAAAFkijNU5MGaMqS0AAED6CGN1GDMGAACyRBirw9QWAAAgS4SxOmXujAEAgAwRxupUGMAPAAAy\nRBirw9QWAAAgS4SxOsWCVSyYbkoAAJAJwlgDlWKBqS0AAEAmCGMNlIummxIAAGSCMNZApVRgAD8A\nAMgEYayBSrGgAe6MAQCADBDGGihzZwwAAGSEMNZAuVjgaUoAAJAJwlgDlWJB/YM8TQkAANJHGGug\nXOLOGAAAyAZhrIEKU1sAAICMEMYaYMwYAADICmGsgQrdlAAAICMthTHbF9leb3uj7aUNtp9u+4e2\n99m+sW7bU7Yftf2Q7d6a8pm2V9vekHzOOPTTaY9ysaB9dFMCAIAMjBnGbBcl3SbpYkmLJF1pe1Fd\nteclfUTSLU0Oc0FEnBURPTVlSyXdFxELJd2XrE8IFbopAQBARlq5M3aupI0RsSki+iXdKWlxbYWI\n2BkRayQNjOO7F0u6I1m+Q9Kl49g3VdVuSqa2AAAA6WsljM2VtLlmfUtS1qqQ9K+2H7B9TU15d0Rs\nS5a3S+oexzFTxYvCAQBAVkoZfMfbI2Kr7eMlrbb9RER8r7ZCRITthreikgB3jSSdeOKJ6bdWPE0J\nAACy08qdsa2STqhZn5eUtSQitiafOyXdrWq3pyTtsD1HkpLPnU32Xx4RPRHR09XV1erXHpIK76YE\nAAAZaSWMrZG00PYC2xVJV0ha2crBbU+z3TmyLOk9kh5LNq+UtCRZXiLpnvE0PE3V1yERxgAAQPrG\n7KaMiEHb10u6V1JR0oqIWGv72mT7MtuzJfVKOlrSsO0bVH3ycpaku22PfNffRsQ/J4e+WdJdtj8k\n6WlJl7f31A4e3ZQAACArLY0Zi4hVklbVlS2rWd6uavdlvZclndnkmM9JurDllmaoUipoOKSh4VCx\n4LybAwAADmPMwN9AuVj9tXB3DAAApI0w1kC5WL0bxiz8AAAgbYSxBiol7owBAIBsEMYaqNBNCQAA\nMkIYa2BkzBjTWwAAgLQRxhoo000JAAAyQhhroLL/zhgvCwcAAOkijDVQKVWfpuSVSAAAIG2EsQaY\nZwwAAGSFMNbA/qcpGcAPAABSRhhrYGQA/z7ujAEAgJQRxhrgzhgAAMgKYayBAzPw8zQlAABIF2Gs\ngf2Tvg4N5dwSAABwuCOMNTDyovAB5hkDAAApI4w1sH/SVwbwAwCAlBHGGqjwOiQAAJARwlgDvCgc\nAABkhTDWADPwAwCArBDGGhgZwN/P1BYAACBlhLEGbKtSLNBNCQAAUkcYa6JcNN2UAAAgdYSxJiql\nAmEMAACkjjDWRJluSgAAkAHCWBPlYoFJXwEAQOoIY010lAq8KBwAAKSOMNZEtZuSF4UDAIB0Ecaa\nKJfMnTEAAJC6lsKY7Ytsr7e90fbSBttPt/1D2/ts31hTfoLt79heZ3ut7Y/WbLvJ9lbbDyU/l7Tn\nlNqjXORpSgAAkL7SWBVsFyXdJundkrZIWmN7ZUSsq6n2vKSPSLq0bvdBSX8QEQ/a7pT0gO3VNfve\nGhG3HPJZpIBJXwEAQBZauTN2rqSNEbEpIvol3SlpcW2FiNgZEWskDdSVb4uIB5PlXZIelzS3LS1P\nWaXE05QAACB9rYSxuZI216xv0UEEKtvzJZ0t6Uc1xR+2/YjtFbZnjPeYaaKbEgAAZCGTAfy2p0v6\npqQbIuLlpPh2SSdLOkvSNkmfabLvNbZ7bff29fVl0VxJ1W7KgUEG8AMAgHS1Esa2SjqhZn1eUtYS\n22VVg9hXI+JbI+URsSMihiJiWNIXVe0OfY2IWB4RPRHR09XV1erXHrIy3ZQAACADrYSxNZIW2l5g\nuyLpCkkrWzm4bUv6sqTHI+Kzddvm1KxeJumx1pqcjXLRDOAHAACpG/NpyogYtH29pHslFSWtiIi1\ntq9Nti+zPVtSr6SjJQ3bvkHSIklvlnSVpEdtP5Qc8uMRsUrSp22fJSkkPSXpd9p7aoemgxeFAwCA\nDIwZxiQpCU+r6sqW1SxvV7X7st7/k+Qmx7yq9WZmj3dTAgCALDADfxPlYkEDdFMCAICUEcaaqPCi\ncAAAkAHCWBMj3ZQRBDIAAJAewlgTlWJ1qBt3xwAAQJoIY02Ui9VfDU9UAgCANBHGmqiUCGMAACB9\nhLEmRu6MMfErAABIE2GsicpIGOPOGAAASBFhrIkD3ZQM4AcAAOkhjDVBNyUAAMgCYayJ8v6pLQhj\nAAAgPYSxJka6KRkzBgAA0kQYa6JCNyUAAMgAYayJMvOMAQCADBDGmqgwAz8AAMgAYawJnqYEAABZ\nIIw1USlVn6bsZ54xAACQIsJYE/tfFM6dMQAAkCLCWBNMbQEAALJAGGuizAB+AACQAcJYEwzgBwAA\nWSCMNdHBi8IBAEAGCGNNcGcMAABkgTDWRLFgFcyYMQAAkC7C2CgqpQJhDAAApIowNopysaB9dFMC\nAIAUEcZGUSlyZwwAAKSLMDYKuikBAEDaWgpjti+yvd72RttLG2w/3fYPbe+zfWMr+9qeaXu17Q3J\n54xDP532KhcLPE0JAABSNWYYs12UdJukiyUtknSl7UV11Z6X9BFJt4xj36WS7ouIhZLuS9YnlHLR\nzDMGAABS1cqdsXMlbYyITRHRL+lOSYtrK0TEzohYI2lgHPsulnRHsnyHpEsP8hxSUy4WeDclAABI\nVSthbK6kzTXrW5KyVoy2b3dEbEuWt0vqbvGYmeko0U0JAADSNSEG8EdESGrYH2j7Gtu9tnv7+voy\nbVeZpykBAEDKWgljWyWdULM+LylrxWj77rA9R5KSz52NDhARyyOiJyJ6urq6Wvza9iCMAQCAtLUS\nxtZIWmh7ge2KpCskrWzx+KPtu1LSkmR5iaR7Wm92NiqlgvoZwA8AAFJUGqtCRAzavl7SvZKKklZE\nxFrb1ybbl9meLalX0tGShm3fIGlRRLzcaN/k0DdLusv2hyQ9Lenydp/coWJqCwAAkLYxw5gkRcQq\nSavqypbVLG9XtQuypX2T8uckXTiexmatUjLdlAAAIFUTYgD/RMXrkAAAQNoIY6OgmxIAAKSNMDaK\nMu+mBAAAKSOMjaLCnTEAAJAywtgoqlNbEMYAAEB6CGOj4EXhAAAgbYSxUZSLBQ0Nh4aGCWQAACAd\nhLFRVErVXw+D+AEAQFoIY6OoFKu/HsaNAQCAtBDGRlFOwtgAT1QCAICUEMZGcaCbkjFjAAAgHYSx\nUYzcGWOuMQAAkBbC2CjKRUtizBgAAEgPYWwUHTxNCQAAUkYYGwXdlAAAIG2EsVHsf5qSO2MAACAl\nhLFRjDxNyZgxAACQFsLYKOimBAAAaSOMjaJSZJ4xAACQLsLYKMql6tQWjBkDAABpIYyNokI3JQAA\nSBlhbBRlXhQOAABSRhgbRYVJXwEAQMoIY6OgmxIAAKSNMDaKMnfGAABAyghjoxh5UThTWwAAgLQQ\nxkZBNyUAAEgbYWwUtlUumqcpAQBAaloKY7Yvsr3e9kbbSxtst+3PJ9sfsX1OUn6a7Ydqfl62fUOy\n7SbbW2u2XdLeU2uPcrGgAe6MAQCAlJTGqmC7KOk2Se+WtEXSGtsrI2JdTbWLJS1Mfs6TdLuk8yJi\nvaSzao6zVdLdNfvdGhG3tONE0lIpFRjADwAAUtPKnbFzJW2MiE0R0S/pTkmL6+oslvSVqLpf0rG2\n59TVuVDSTyLi6UNudYbKxQLdlAAAIDWthLG5kjbXrG9JysZb5wpJX6sr+3DSrbnC9owW2pK5SrGg\n/kGepgQAAOnIZAC/7YqkX5H09Zri2yWdrGo35jZJn2my7zW2e2339vX1pd7WeuWi6aYEAACpaSWM\nbZV0Qs36vKRsPHUulvRgROwYKYiIHRExFBHDkr6oanfoa0TE8ojoiYierq6uFprbXpVSgaktAABA\naloJY2skLbS9ILnDdYWklXV1Vkq6Onmq8nxJL0XEtprtV6qui7JuTNllkh4bd+szUC4ygB8AAKRn\nzKcpI2LQ9vWS7pVUlLQiItbavjbZvkzSKkmXSNooaY+kD47sb3uaqk9i/k7doT9t+yxJIempBtsn\nBAbwAwCANI0ZxiQpIlapGrhqy5bVLIek65rs+4qk4xqUXzWuluaEbkoAAJAmZuAfQ4VuSgAAkCLC\n2BiqT1MytQUAAEgHYWwMzMAPAADSRBgbQ7nImDEAAJAewtgYKjxNCQAAUkQYGwPdlAAAIE2EsTHQ\nTQkAANJEGBtDdQZ+nqYEAADpIIyNoVwyY8YAAEBqCGNj6Ei6KasvGQAAAGgvwtgYysXqr2hwmDAG\nAADajzA2hnKp+iviiUoAAJAGwtgYKsmdMZ6oBAAAaSCMjWHkzhiD+AEAQBoIY2OoFC1JTG8BAABS\nQRgbQ2VkzBjdlAAAIAWEsTGMPE1JNyUAAEgDYWwM0zpKkqSX9g7k3BIAAHA4IoyN4ZSu6ZKkDTt2\n59wSAABwOCKMjWHusUdpWqWo9dtfzrspAADgMEQYG0OhYJ06u1Prd+zKuykAAOAwRBhrwWndnVq/\nfRfvpwQAAG1HGGvBqd2demHPgPp278u7KQAA4DBDGGvB6bM7JUlPbmcQPwAAaC/CWAtOTcIY48YA\nAEC7EcZaMGt6h46bVuGJSgAA0HaEsRadNrtT65lrDAAAtBlhrEWndndqw45dGh7miUoAANA+LYUx\n2xfZXm97o+2lDbbb9ueT7Y/YPqdm21O2H7X9kO3emvKZtlfb3pB8zmjPKaXj9Nmd2tM/pC0v7M27\nKQAA4DAyZhizXZR0m6SLJS2SdKXtRXXVLpa0MPm5RtLtddsviIizIqKnpmyppPsiYqGk+5L1CYtB\n/AAAIA2t3Bk7V9LGiNgUEf2S7pS0uK7OYklfiar7JR1re84Yx10s6Y5k+Q5Jl46j3Zk7tTuZ3oIw\nBgAA2qiVMDZX0uaa9S1JWat1QtK/2n7A9jU1dbojYluyvF1Sd8utzsH0jpLmzThK67cTxgAAQPuU\nMviOt0fEVtvHS1pt+4mI+F5thYgI2w1HxicB7hpJOvHEE9Nv7ShGXosEAADQLq3cGdsq6YSa9XlJ\nWUt1ImLkc6eku1Xt9pSkHSNdmcnnzkZfHhHLI6InInq6urpaaG56Tp3dqZ/07Vb/4HCu7QAAAIeP\nVsLYGkkLbS+wXZF0haSVdXVWSro6earyfEkvRcQ229Nsd0qS7WmS3iPpsZp9liTLSyTdc4jnkrrT\nZ3dqcDj01HOv5N0UAABwmBizmzIiBm1fL+leSUVJKyJire1rk+3LJK2SdImkjZL2SPpgsnu3pLtt\nj3zX30bEPyfbbpZ0l+0PSXpa0uVtO6uUjAzif2L7rv3LAAAAh6KlMWMRsUrVwFVbtqxmOSRd12C/\nTZLObHLM5yRdOJ7G5u3krmkqFqwnt+9qclYAAADjwwz849BRKurkWdOYawwAALQNYWycTp3NE5UA\nAKB9CGPjdFp3p372/B7t6R/MuykAAOAwQBgbp9OS1yJt2LE755YAAIDDAWFsnE5LnqKkqxIAALQD\nYWycTpg5VVPKBQbxAwCAtiCMjVOxYJ3a3ckLwwEAQFsQxg7Cqd2deoJuSgAA0AaEsYNwWnen+nbt\n0/Ov9OfdFAAAMMkRxg7CyBOVj2x5MeeWAACAyY4wdhDOOWmGuo/u0KdWPaF9g0N5NwcAAExihLGD\nML2jpE/91zO0fscuff6+DXk3BwAATGKEsYP0ztO79Wtvmadl392khzfTXQkAAA4OYewQ/M/3LVLX\n9A7d+PWH9fMBuisBAMD4EcYOwTFHlfWpXz1DG3bu1uforgQAAAeBMHaILjjteF3eM09f+O5P9OOf\nvZB3cwAAwCRDGGuDP3nfInUfPYXuSgAAMG6EsTY4ekpZ//tX36yf9L3C05UAAGBcCGNt8kundulX\nz5mn5d/bxHsrAQBAywhjbfTHv/wGdU4p6ePfelTDw5F3cwAAwCRAGGujmdMq+vglb1Dv0y/o73o3\n590cAAAwCRDG2uzX3jJP5y2YqU+telx9u/bl3RwAADDBEcbazLY+edkZ+vnAsD75j+vybg4AAJjg\nCGMpOOX46frdd7xe337oGX1/Q1/ezQEAABMYYSwlv/uO12vBrGn6k28/xtxjAACgKcJYSqaUi/rk\npW/S08/t0W3f2Zh3cwAAwARFGEvRW0+ZpcvOnqsvfHeTfvrsK3k3BwAATECEsZT90SWnq6NU0J+t\nXKsI5h4DAACv1lIYs32R7fW2N9pe2mC7bX8+2f6I7XOS8hNsf8f2OttrbX+0Zp+bbG+1/VDyc0n7\nTmviOL5zin7/Pafqe0/26d61O/JuDgAAmGDGDGO2i5Juk3SxpEWSrrS9qK7axZIWJj/XSLo9KR+U\n9AcRsUjS+ZKuq9v31og4K/lZdWinMnFddf5JOn12pz7xD+u0p38w7+YAAIAJpJU7Y+dK2hgRmyKi\nX9KdkhbX1Vks6StRdb+kY23PiYhtEfGgJEXELkmPS5rbxvZPCqViQZ+49E3a+uJeBvMDAIBXaSWM\nzZVU+26fLXptoBqzju35ks6W9KOa4g8n3ZorbM9osc2T0i/Mn7n/ReI/6dudd3MAAMAEkckAftvT\nJX1T0g0R8XJSfLukkyWdJWmbpM802fca2722e/v6JvcEqksvPl1TykXdxGB+AACQaCWMbZV0Qs36\nvKSspTq2y6oGsa9GxLdGKkTEjogYiohhSV9UtTv0NSJieUT0RERPV1dXC82duLo6O3Tje07T9zc8\nq1WPbs+7OQAAYAJoJYytkbTQ9gLbFUlXSFpZV2elpKuTpyrPl/RSRGyzbUlflvR4RHy2dgfbc2pW\nL5P02EGfxSTyG+edqDe+7mjd9Pdr9eKe/rybAwAAcjZmGIuIQUnXS7pX1QH4d0XEWtvX2r42qbZK\n0iZJG1W9y/V7SfnbJF0l6Z0NprD4tO1HbT8i6QJJH2vbWU1gpWJBn/61N+uFV/r153/Pi8QBADjS\neTKNXerp6Yne3t68m9EWt65+Up+7b4O+eHWP3r2oO+/mAACANrP9QET0jFWPGfhzct0Fp+gNc47W\nx+9+lO5KAACOYISxnFRKBd3y3+iuBADgSEcYy9EbX3eMfu+CU3T3j7dq9TpelQQAwJGIMJaz6y84\nRafP7qS7EgCAIxRhLGfV7soz9fwr/frTe5gMFgCAIw1hbAJ409xjdMOFC7Xy4Wd0xw+eyrs5AAAg\nQ4SxCeK6C07Ru95wvD7xj4/r/k3P5d0cAACQEcLYBFEoWJ99/1k6aeZUXffVB/XMi3vzbhIAAMgA\nYWwCOXpKWcuvfov2DQ7rd//mAf18YCjvJgEAgJQRxiaYU47v1GcuP1MPb3lJf/LtxxjQDwDAYY4w\nNgG9942z9ZF3nqJvPLBF//f+p/NuDgAASFEp7wagsRvedaoee+Zl3bRyrY45qqzFZ83Nu0kAACAF\n3BmboAoF6//8+tk6d8FMfezvHtK3f7w17yYBAIAUEMYmsKmVklZ84Bd03oLj9Pt3PaS7f7wl7yYB\nAIA2I4xNcCOB7PyTj9Pv3/WwvvkAgQwAgMMJYWwSOKpS1JeX/ILe+vrjdOM3HtbXezfn3SQAANAm\nhLFJYiSQve31s/Tfv/GIPvVPj2twaDjvZgEAgENEGJtEppSL+tKSHv36eSfqC9/dpF//4o+04+Wf\n590sAABwCAhjk8yUclH/67IzdOv7z9SjW1/SL3/++/rBxmfzbhYAADhIhLFJ6rKz52nl9W/TsVMr\n+s0v/0ifv28D3ZYAAExChLFJbGF3p+657m36L2e+Tp9d/aQu/tz39e/rd+bdLAAAMA6EsUluWkdJ\nf/H+s/SFq96i/qFhfeCv1ugDf/Uf2rhzV95NAwAALSCMHQZs671vnK1/+dgv6Y8veYMeeOoFvfcv\nvq+bVq7Vtpf25t08AAAwCkdE3m1oWU9PT/T29ubdjAnv2d37dOvqJ/W1//hZEtS6dfUvztd5C2bK\ndt7NAwDgiGD7gYjoGbMeYezwtfn5Pfqb+5/WnWs266W9Azqtu1NXv/Ukve+M1+mYqeW8mwcAwGGN\nMIb99vYP6e8ffkZ//YOntG7byyoVrHMXzNR7FnXrXYu6NW/G1LybCADAYYcwhteICD285SXdu3a7\nVq/boY07d0uSFs05Wv9p4Sydc9IMveWkGZo1vSPnlgIAMPkRxjCmnz77ilavqwazhze/pP5knrL5\nx03VOSfN0Blzj9Gp3Z1a2D1dXdM7GG8GAMA4tDWM2b5I0uckFSV9KSJurtvuZPslkvZI+kBEPDja\nvrZnSvo7SfMlPSXp8oh4YbR2EMbS8/OBIa195iX1PvWCHnj6BT34sxf07O7+/duPnVrWqcd36uSu\naZo34yjNmzF1/+fxnR0qFAhqAADUalsYs12U9KSkd0vaImmNpCsjYl1NnUskfVjVMHaepM9FxHmj\n7Wv705Kej4ibbS+VNCMi/nC0thDGshMR6tu1T0/u2K0NO3dVP3fs0lPP7dGzu/e9qm6xYM2aXlFX\nZ4e6pneoq7NDx03v0IypZR17VEXHTC3r2KPKOmZqWZ1TypreUdL0jpKKBDgAwGGs1TBWauFY50ra\nGBGbkgPfKWmxpHU1dRZL+kpUk939to+1PUfVu17N9l0s6R3J/ndI+ndJo4YxZMe2jj96io4/eore\nvnDWq7bt7R/S1hf3aPMLe7Xlhb3a9uJePbt7n/p27VPf7n1at+1lPbe7X4PDowf9o8pFTZ9S0tRK\nUUeVi9X9xM2AAAAG/0lEQVTPSlFHlUuaUi6oo1R81WelVFC5WFBH8ln98f7lUtEqF61SoaBSwSoW\nrFLxwPKrflz9LCTLhYKqn66WFVz9HRQsFWzZkvXqcrptAQDt0EoYmytpc836FlXvfo1VZ+4Y+3ZH\nxLZkebuk7hbbjJwdVSnqlOM7dcrxnU3rRIRe6R/SC6/066W9A3pxz4Be3NuvV/YNatfPB7V736B2\nJ597B4a0p39Ie/uHtKd/UM/t7lf/4LD2DQ5r3+CQ9g0M6+eDQxoYmpjjG6tBrRrOvH+9Wli7PlJP\nNXVHCl5V3qDegdzn/d9Zu9+ry9xk+2vD4/56NZtcs1ejvFnbtvqyhhWbFI3s32jfxt/bOPweSiZu\n+Dtped/xfE+L9Vr+9oP/jsbfm4Ic/1g5nP5M4m++9Nz4ntP0tlNmjV0xA62EsdRFRNhu+C+t7Wsk\nXSNJJ554YqbtwsGzvb878oQ2HXN4ODQwPKz+weRnaFiDQ6GBoWENJJ+Dw6HB5HNoOJLPar3hiP3l\nI9siQkPD0lCMLIeGoxomh6O6PByhSMoidKBMkpLPCCmU1KtZ1/716japbnuyPCJqjlet++r9Xl0/\navY78NnwuK/Z90A9Nag30pZ6jY/zWqPtW7sSDfZuNHKi2WiKRvs3rNfomC3Wa/bNLddsseqh/Klx\nKA9ipfEnTp7PhU3MP9kOzmR6wG4ymkhDZVoJY1ulV/17Oi8pa6VOeZR9d9ieExHbki7Nhm+4jojl\nkpZL1TFjLbQXh6lCweooFNVRKubdFAAA2qaVd1OukbTQ9gLbFUlXSFpZV2elpKtddb6kl5IuyNH2\nXSlpSbK8RNI9h3guAAAAk86Yd8YiYtD29ZLuVXV6ihURsdb2tcn2ZZJWqfok5UZVp7b44Gj7Joe+\nWdJdtj8k6WlJl7f1zAAAACYBJn0FAABIQatTW7TSTQkAAICUEMYAAAByRBgDAADIEWEMAAAgR4Qx\nAACAHBHGAAAAckQYAwAAyBFhDAAAIEeEMQAAgBwRxgAAAHI0qV6HZLtP1fdYpmmWpGdT/g4cHK7N\nxMR1mbi4NhMT12Xiave1OSkiusaqNKnCWBZs97byHilkj2szMXFdJi6uzcTEdZm48ro2dFMCAADk\niDAGAACQI8LYay3PuwFoimszMXFdJi6uzcTEdZm4crk2jBkDAADIEXfGAAAAckQYq2H7ItvrbW+0\nvTTv9hypbJ9g+zu219lea/ujSflM26ttb0g+Z+Td1iOR7aLtH9v+h2Sd6zIB2D7W9jdsP2H7cdu/\nyLXJn+2PJf8fe8z212xP4brkw/YK2zttP1ZT1vRa2P6jJA+st/3eNNtGGEvYLkq6TdLFkhZJutL2\nonxbdcQalPQHEbFI0vmSrkuuxVJJ90XEQkn3JevI3kclPV6zznWZGD4n6Z8j4nRJZ6p6jbg2ObI9\nV9JHJPVExJskFSVdIa5LXv5a0kV1ZQ2vRfJvzhWS3pjs85dJTkgFYeyAcyVtjIhNEdEv6U5Ji3Nu\n0xEpIrZFxIPJ8i5V/1GZq+r1uCOpdoekS/Np4ZHL9jxJvyzpSzXFXJec2T5G0i9J+rIkRUR/RLwo\nrs1EUJJ0lO2SpKmSnhHXJRcR8T1Jz9cVN7sWiyXdGRH7IuKnkjaqmhNSQRg7YK6kzTXrW5Iy5Mj2\nfElnS/qRpO6I2JZs2i6pO6dmHcn+QtL/kDRcU8Z1yd8CSX2S/irpQv6S7Wni2uQqIrZKukXSzyRt\nk/RSRPyLuC4TSbNrkWkmIIxhwrI9XdI3Jd0QES/XbovqY8A8Cpwh2++TtDMiHmhWh+uSm5KkcyTd\nHhFnS3pFdV1fXJvsJeOPFqsall8naZrt36ytw3WZOPK8FoSxA7ZKOqFmfV5ShhzYLqsaxL4aEd9K\ninfYnpNsnyNpZ17tO0K9TdKv2H5K1W78d9r+G3FdJoItkrZExI+S9W+oGs64Nvl6l6SfRkRfRAxI\n+pakt4rrMpE0uxaZZgLC2AFrJC20vcB2RdWBeytzbtMRybZVHfvyeER8tmbTSklLkuUlku7Jum1H\nsoj4o4iYFxHzVf3v498i4jfFdcldRGyXtNn2aUnRhZLWiWuTt59JOt/21OT/axeqOgaW6zJxNLsW\nKyVdYbvD9gJJCyX9R1qNYNLXGrYvUXVMTFHSioj4ZM5NOiLZfruk70t6VAfGJn1c1XFjd0k6UdLT\nki6PiPrBmMiA7XdIujEi3mf7OHFdcmf7LFUfrKhI2iTpg6r+wc21yZHtP5f0flWfEv+xpN+WNF1c\nl8zZ/pqkd0iaJWmHpD+T9G01uRa2/1jSb6l67W6IiH9KrW2EMQAAgPzQTQkAAJAjwhgAAECOCGMA\nAAA5IowBAADkiDAGAACQI8IYAABAjghjAAAAOSKMAQAA5Oj/A1vGPjlqjWLZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe4551d4fd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_ibm1_change(100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Translation Table?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def show_ibm1_alpha(target):\n",
    "    plot_table_for_target(ibm1_iterations[-1][1].alpha, target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAGfCAYAAACdqpz+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmUZVV99vHvQyOIA4rQCDIIGBRxwKHFISYmKhHUiDhE\nHFDRiKCCryNonM2bgEskDiiiIk4EBxwQWokah/CqkRYVRUNscQAHaFFBnBD9vX/sU3AoG7rAXXVv\ndX0/a/Wy7j37UtvaVec8Z589pKqQJEnSn2+DSVdAkiRpfWGwkiRJ6sRgJUmS1InBSpIkqRODlSRJ\nUicGK0mSpE4MVpIkSZ0YrCRJkjoxWEmSJHWy4aS+8RZbbFE77LDDpL69JEnSnH35y1/+aVUtX1e5\niQWrHXbYgVWrVk3q20uSJM1Zku/PpZyPAiVJkjoxWEmSJHVisJIkSerEYCVJktSJwUqSJKkTg5Uk\nSVInBitJkqRODFaSJEmdGKwkSZI6MVhJkiR1MqdglWTPJOckWZ3ksLUc/5skFyf56vDvJf2rKkmS\nNN3WuVdgkmXA0cAewPnAGUlOrqpvzir6X1X14HmooyRJ0qIwlx6r3YHVVXVuVV0GnAjsPb/VkiRJ\nWnzmEqy2Ac4bvT5/eG+2eyU5K8nHktyuS+0kSZIWkXU+CpyjM4Htq+rSJA8EPgzsPLtQkgOAAwC2\n3377Tt/6mu1w2KkL8n2Wiu8d/qBJV0HSNfCc15fnPF1bc+mx+iGw3ej1tsN7V6iqS6rq0uHrlcD1\nkmwx+z9UVcdW1YqqWrF8+fI/o9qSJEnTZy7B6gxg5yQ7JtkI2Bc4eVwgyVZJMny9+/Dfvah3ZSVJ\nkqbZOh8FVtXlSZ4BnAYsA46rqrOTHDgcPwZ4BHBQksuB3wD7VlXNY70lSZKmzpzGWA2P91bOeu+Y\n0ddvAN7Qt2qSJEmLiyuvS5IkdWKwkiRJ6sRgJUmS1InBSpIkqRODlSRJUicGK0mSpE4MVpIkSZ0Y\nrCRJkjoxWEmSJHVisJIkSerEYCVJktSJwUqSJKkTg5UkSVInBitJkqRODFaSJEmdGKwkSZI6MVhJ\nkiR1YrCSJEnqxGAlSZLUicFKkiSpE4OVJElSJwYrSZKkTgxWkiRJnRisJEmSOjFYSZIkdWKwkiRJ\n6sRgJUmS1InBSpIkqRODlSRJUicGK0mSpE4MVpIkSZ0YrCRJkjoxWEmSJHVisJIkSerEYCVJktSJ\nwUqSJKkTg5UkSVInBitJkqRODFaSJEmdGKwkSZI6MVhJkiR1YrCSJEnqxGAlSZLUicFKkiSpE4OV\nJElSJwYrSZKkTgxWkiRJnRisJEmSOjFYSZIkdWKwkiRJ6sRgJUmS1InBSpIkqRODlSRJUidzClZJ\n9kxyTpLVSQ67hnJ3S3J5kkf0q6IkSdLisM5glWQZcDSwF7Ar8Ogku15NuSOA/+hdSUmSpMVgLj1W\nuwOrq+rcqroMOBHYey3lDgZOAi7sWD9JkqRFYy7BahvgvNHr84f3rpBkG2Af4E39qiZJkrS49Bq8\n/m/AoVX1x2sqlOSAJKuSrFqzZk2nby1JkjQdNpxDmR8C241ebzu8N7YCODEJwBbAA5NcXlUfHheq\nqmOBYwFWrFhR17XSkiRJ02guweoMYOckO9IC1b7AY8YFqmrHma+THA+cMjtUSZIkre/WGayq6vIk\nzwBOA5YBx1XV2UkOHI4fM891lCRJWhTm0mNFVa0EVs56b62Bqqqe+OdXS5IkafFx5XVJkqRODFaS\nJEmdGKwkSZI6MVhJkiR1YrCSJEnqxGAlSZLUicFKkiSpE4OVJElSJwYrSZKkTgxWkiRJnRisJEmS\nOjFYSZIkdWKwkiRJ6sRgJUmS1InBSpIkqRODlSRJUicGK0mSpE4MVpIkSZ0YrCRJkjoxWEmSJHVi\nsJIkSerEYCVJktSJwUqSJKkTg5UkSVInBitJkqRODFaSJEmdGKwkSZI6MVhJkiR1YrCSJEnqxGAl\nSZLUicFKkiSpE4OVJElSJwYrSZKkTgxWkiRJnRisJEmSOjFYSZIkdWKwkiRJ6sRgJUmS1InBSpIk\nqRODlSRJUicGK0mSpE4MVpIkSZ0YrCRJkjoxWEmSJHVisJIkSerEYCVJktSJwUqSJKkTg5UkSVIn\nBitJkqRODFaSJEmdGKwkSZI6MVhJkiR1YrCSJEnqZE7BKsmeSc5JsjrJYWs5vneSs5J8NcmqJPfu\nX1VJkqTptuG6CiRZBhwN7AGcD5yR5OSq+uao2KeAk6uqktwReB+wy3xUWJIkaVrNpcdqd2B1VZ1b\nVZcBJwJ7jwtU1aVVVcPLGwKFJEnSEjOXYLUNcN7o9fnDe1eRZJ8k/wOcCjypT/UkSZIWj26D16vq\nQ1W1C/BQ4JVrK5PkgGEM1qo1a9b0+taSJElTYS7B6ofAdqPX2w7vrVVVfQ7YKckWazl2bFWtqKoV\ny5cvv9aVlSRJmmZzCVZnADsn2THJRsC+wMnjAkn+IkmGr+8CbAxc1LuykiRJ02ydswKr6vIkzwBO\nA5YBx1XV2UkOHI4fAzwceHyS3wO/AR41GswuSZK0JKwzWAFU1Upg5az3jhl9fQRwRN+qSZIkLS6u\nvC5JktSJwUqSJKkTg5UkSVInBitJkqRODFaSJEmdGKwkSZI6MVhJkiR1YrCSJEnqxGAlSZLUicFK\nkiSpE4OVJElSJwYrSZKkTgxWkiRJnRisJEmSOjFYSZIkdWKwkiRJ6sRgJUmS1InBSpIkqRODlSRJ\nUicGK0mSpE4MVpIkSZ0YrCRJkjoxWEmSJHVisJIkSerEYCVJktSJwUqSJKkTg5UkSVInBitJkqRO\nDFaSJEmdGKwkSZI6MVhJkiR1YrCSJEnqxGAlSZLUicFKkiSpE4OVJElSJwYrSZKkTgxWkiRJnRis\nJEmSOjFYSZIkdWKwkiRJ6sRgJUmS1InBSpIkqRODlSRJUicGK0mSpE4MVpIkSZ0YrCRJkjoxWEmS\nJHVisJIkSerEYCVJktSJwUqSJKkTg5UkSVInBitJkqRODFaSJEmdGKwkSZI6mVOwSrJnknOSrE5y\n2FqOPzbJWUm+nuTzSXbrX1VJkqTpts5glWQZcDSwF7Ar8Ogku84q9l3gPlV1B+CVwLG9KypJkjTt\n5tJjtTuwuqrOrarLgBOBvccFqurzVfXz4eUXgW37VlOSJGn6zSVYbQOcN3p9/vDe1Xky8LG1HUhy\nQJJVSVatWbNm7rWUJElaBLoOXk/yt7RgdejajlfVsVW1oqpWLF++vOe3liRJmrgN51Dmh8B2o9fb\nDu9dRZI7Am8F9qqqi/pUT5IkafGYS4/VGcDOSXZMshGwL3DyuECS7YEPAvtV1f/2r6YkSdL0W2eP\nVVVdnuQZwGnAMuC4qjo7yYHD8WOAlwCbA29MAnB5Va2Yv2pLkiRNn7k8CqSqVgIrZ713zOjrfwT+\nsW/VJEmSFhdXXpckSerEYCVJktSJwUqSJKkTg5UkSVInBitJkqRODFaSJEmdGKwkSZI6MVhJkiR1\nYrCSJEnqxGAlSZLUicFKkiSpE4OVJElSJwYrSZKkTgxWkiRJnRisJEmSOjFYSZIkdWKwkiRJ6sRg\nJUmS1InBSpIkqRODlSRJUicGK0mSpE4MVpIkSZ0YrCRJkjoxWEmSJHVisJIkSerEYCVJktSJwUqS\nJKkTg5UkSVInBitJkqRODFaSJEmdGKwkSZI6MVhJkiR1YrCSJEnqxGAlSZLUicFKkiSpE4OVJElS\nJwYrSZKkTgxWkiRJnRisJEmSOjFYSZIkdWKwkiRJ6sRgJUmS1InBSpIkqRODlSRJUicGK0mSpE4M\nVpIkSZ0YrCRJkjoxWEmSJHVisJIkSerEYCVJktSJwUqSJKkTg5UkSVInBitJkqRO5hSskuyZ5Jwk\nq5MctpbjuyT5QpLfJXlu/2pKkiRNvw3XVSDJMuBoYA/gfOCMJCdX1TdHxX4GHAI8dF5qKUmStAjM\npcdqd2B1VZ1bVZcBJwJ7jwtU1YVVdQbw+3mooyRJ0qIwl2C1DXDe6PX5w3vXWpIDkqxKsmrNmjXX\n5T8hSZI0tRZ08HpVHVtVK6pqxfLlyxfyW0uSJM27uQSrHwLbjV5vO7wnSZKkkbkEqzOAnZPsmGQj\nYF/g5PmtliRJ0uKzzlmBVXV5kmcApwHLgOOq6uwkBw7Hj0myFbAK2BT4Y5L/A+xaVZfMY90lSZKm\nyjqDFUBVrQRWznrvmNHXP6E9IpQkSVqyXHldkiSpE4OVJElSJwYrSZKkTgxWkiRJnRisJEmSOjFY\nSZIkdWKwkiRJ6sRgJUmS1InBSpIkqRODlSRJUicGK0mSpE4MVpIkSZ0YrCRJkjoxWEmSJHVisJIk\nSerEYCVJktSJwUqSJKkTg5UkSVInBitJkqRODFaSJEmdGKwkSZI6MVhJkiR1YrCSJEnqxGAlSZLU\nicFKkiSpE4OVJElSJwYrSZKkTgxWkiRJnRisJEmSOjFYSZIkdWKwkiRJ6sRgJUmS1InBSpIkqROD\nlSRJUicGK0mSpE4MVpIkSZ0YrCRJkjoxWEmSJHVisJIkSerEYCVJktSJwUqSJKkTg5UkSVInBitJ\nkqRODFaSJEmdGKwkSZI6MVhJkiR1YrCSJEnqxGAlSZLUicFKkiSpE4OVJElSJwYrSZKkTgxWkiRJ\nnRisJEmSOplTsEqyZ5JzkqxOcthajifJ64bjZyW5S/+qSpIkTbd1Bqsky4Cjgb2AXYFHJ9l1VrG9\ngJ2HfwcAb+pcT0mSpKk3lx6r3YHVVXVuVV0GnAjsPavM3sA7q/kicNMkW3euqyRJ0lSbS7DaBjhv\n9Pr84b1rW0aSJGm9tuFCfrMkB9AeFQJcmuSchfz+U24L4KeTrsS65IhJ12DiFkU7LXG20eKwKNrJ\nc97iaKcFcsu5FJpLsPohsN3o9bbDe9e2DFV1LHDsXCq21CRZVVUrJl0PXTPbafrZRouD7bQ42E7X\n3lweBZ4B7JxkxyQbAfsCJ88qczLw+GF24D2Ai6vqx53rKkmSNNXW2WNVVZcneQZwGrAMOK6qzk5y\n4HD8GGAl8EBgNfBrYP/5q7IkSdJ0mtMYq6paSQtP4/eOGX1dwNP7Vm3J8RHp4mA7TT/baHGwnRYH\n2+laSstEkiRJ+nO5pY0kSVInBitJugZJMuk66Nqz3TQpBqslxpPNdEmyIsmdJl0P/akkf59ko3K8\nxKJku2lSDFbruSQvTvLIJHeGdrIxXE2HJDsAfwO8NckzktxxohXSFZLcDHgA8O4kj7dtFo8kD0/y\nliS3TLL5pOujtRv+rv560vWYDw5eX88l2YO23+P2wGVVdfDwfryjm5wkewJfqKqLk9waOBD4I3B2\nVb19srVb2pI8qKpOHb7eG9iJFrKOrKpPTLRyWqckmwLPBjYFNgbeNexhqymSZD/gccBngB9V1Tsm\nW6N+DFbrqSSvBN5aVd9PckPg+sCHab/AjxrKGK4mIMnOtCB1IfCOqvrJ0EPyQOAewKer6qRJ1nGp\nSrIceB/w2ap62fDejYCHAM8CnlNVn5tcDXV1khwMbFlVLx5e3wa4D/CPwPOr6jMTrJ4GQzu9r6ou\nSHIr4A7AY4DvV9XzJlu7PnwUuP4KcEqSbarqV1V1UVX9FbA8yYngGIRJqapvAx8CNgF2Ht77GW0H\ng28Cd0xyvcnVcOmqqjXAk4Hdkvzt8N6lVXUC8CbgcUluMck66mp9ErhDkhcBVNU5wzZqRwP/lOT2\nE62dSLIBsA3wliQ3r6rv0M57LwR2TfKSiVawE4PVeibJMoCqehHwXmCf4f0Nh/fvC2yW5NkTq+QS\nNdM2AFV1OnB0Vf3XcCxVdQmtV/GewCMmU8ulaVbbnAs8qao+PavYZ4CLaJvSOhFkSsy0Q1V9C3g+\n8N1ZRT5A2znkL8fltfCq6o/Ai4FVwP1n3quq1cCLgG2T3HOCVezCR4HrkSTLquoPM/97DcfvBjyg\nqv55AtVckkY/+w2A/YDLad3hv0+yQVX9cebRbJJ7AU+gPb64eKIVXwJmtc3fA7+gjX+7bC1lnwvc\npaoes9D11J8an+uS3LCqfjU6tsFwISfJA4AnAo+xp37hXd01aVaZGwPPAH5cVccvSMXmiT1W64nh\nJDJzcXhDkpcmeUqSrWbKjH6xLwTulGSXiVR2CRq1zUeBWwKPBD47TOf/41Bm5oT/deBU4NKJVHaJ\nGbXNZ4C7A28AXpXkJjNlRr0irwbeN2xIrwkan/OSvBs4KskLkmwPV/SOMHx9GnA+sOeEqrtkzWqn\n5yQ5OMmOM09RZlTVL4FPAPsv9sftBqv1xEyPB63LezVwCW3/xj/5Ba2q79O6XS9c0EouUaNHD08D\nPg/8C3AD4L3jXpHRxfuXwCfXdYenP9+obZ4FfLyqXgj8Dvj6MGNz5tH6eJmS09bWm6WFNTrnnQB8\nCXgP8AJg63G5ITQDHAU48WCBjdrp48D1gHsB7wK2GpcbeuxX0drwZwte0Y4MVuuX2wBfrqojgb2A\nt1fVmUm2mT0Yuqr+ZxgwrXkyvigPb50H/IZ2gvlUVb12aJuDZpWjqn690PVdStbSNquBjZOcDpxY\nVW8bej6u6OGYKVtVv1no+upKo6AErff3B8ApwHOAl1fVfw+zO4Gr9Fz9ePyoUPNvdDPycOCMqjoc\n2Jz2N3Z+kuvPlB39La6qqt8ucFW7MlgtYuMBt4M1tEd83wVOHi7cy2gnnG0WvIJL2HjczvBYdlta\nD+FzgW9W1RFD0dcBO06sokvQrLZ5+fA3chktRH1ieNwH7ZHg/SdWUf2Joe3+mGYD4EfARrRH55+p\nqiOH9nxLkhXjzzq2auFczY3Lpkm+TOvxfUOSLYHnrOWmf9H3BhusFqlZz61PTVsI9FLgP4D/Bs4a\niv47cLOq+t6EqrokzRq3c4OqOr+qvkAblP7QJG9I8gngF1X1fHC20kIZtc2pwM2r6g9V9THaEhg7\nJHlPko8CP6+qZ020srrCrHGkHweeNlyEv0vrtfrK0Mv4HuCi4bGSFtisG5cT0tYT+xWwHfAF4I1D\n0aOBHarq95Oq63xxVuAiNlyIVwJfraoXDO/djjat+Cm0R08XV9X+M+W9a5tf459xkgOAO1TVwUlu\nS3s8u4q2VtWuwE2r6uSh7BUzmDQ/ZrXNIcBWVfXCJNvQ2uZTtJ7dmwEbVdUHhrK2zRRJ8ing9Kp6\n6fD6erTxi7cFbg78oKqeORzznDcBQ/j9JPC1mZuTJPcGDqKtiL8R7dHsE4dj61U7GawWmVx1evFt\naAP9Xk4ba/A44AJar9VXgE2q6oKhrBeHBZRkY9pWQu8AzgB+Ttuy5h+AParqK6Oy69VJZTFI8iRa\nmLp0+LeCdlf96GqLhM6Us20mLFddNmEz4PXAU4GHAvcDbg0cMownvdnM2FHPeQtrVjvtATy4qp6Z\n5C7AY2m9VWcBBWwx9OCvl+204bqLaJoMXawBduPKX9L3A/8FfAvYEtixqj5Lmxk4c3FYr35xp1mS\n59PG7LwZeCbtLvp9VXVJkpvSZsZcwQv3wklbh+oGVfWKoS1+AZw0zABcSZsAckWwsm0mbzSr7FFV\ndeJw0/JJ2pCHdwJ7AI8CzhyFKs95C2zUTregjX27X5JTgO/RZtoeCjyv2tZC34b1t50MVovTA4CT\naCt0PxW4TVV9HSDJe5m1/pEXh/m1ljuuC2iPYx9HW1LhkiQ3SXIS8Nuq+tJEKroEJdmwqi4fvt4Y\nOAd4SJJnVtVrhve3TnI0bVzO6ROsrmYZ9RjeDfiHJDeqqkcm2bqqfjyUeTzt4n0Fz3kTsx+wN7D/\n8O9mwH9WWwj5/cxaYmF9bScHry8Cs6YXU1Ufp61R9UHgHlX19SS3TPI+4HdV9cpJ1HOpGs1Seuzw\n+h20he7uSrsY3Jx2YVhTVY8FB6ovlKq6fBhEe6eq+h1tHNX7gV2SHDoUezBwSVXtB7bNNJhpg9GF\n91u0HuB7JnluVf04yU2HHpFU1SvGn9PCyJ/OTP868FXgSNrkj9OArZN8gHZtOnGh6zgJBqtFYHTh\nfmaSWw7vHQ+8DHhb2hY1F9GmsT4e/jSMqb9ZJ/E7Ak+euVhX1ftpY6ueDzwa+O+qOnD43Abr653a\ntJjVNnsCZya5T7X1wb4w/Pv7JIdW1Vuq6mnD52ybKZJkZZK/qbZo7um0Wc53SXJQVf0COH40Oce2\nW2AzQ1OSPGt4/RXa7NpzgUPTZmneBPhOVT0Olsa1ab3/P7iYzfoF3Jj2CPD5SbYbHnG8GzgT+DRt\nSYW3zXxufXxuPU2GSQQzM8w2rKqv0YLubkkOAxja44e0O7dfznzWtplf47YBqKqVtDE4HxldpE+l\njQO5ePS59XK8x2Iyc84btd9pwElJ/rLa4p6fBX4NPD7JfZ25ORmzrk03AF6e5E0AVfUNWjvdFngN\n8L2qOnTmc0uhnZwVOKVy5Vogof2Cfh/4LfAm2ga+h1fVD5K8Eji/qt48weouKTMnh+HkchywCW0J\nhZNoqwo/hzYA+ia0qd/Pnlhll5hZbXMUbWr3F2k9HX9N2/7kacC+tG1rZpYpcfbfhM06592XdkH+\nTpJHAm8BHlhVn09yHG3LpxMmWuElKledmb5lVV2Y5IYMPcFV9dTh2DtpO0y8Y4LVnQiD1RQbLg4n\nAdenbYXyjap6SZI3AjekzQw8s6qeNFN+KdwNTIskJ9B6pN5Fe+T3v8C7gd/TLt4XV9vCwQv3Akty\nGm28x9nAzsBy4P8Afwvcm9YpYqiaMsM57xO0XqlNaON1DqcFrcNpg9TXVNWjhvK23QKa+XkP7fRu\nYDPgO7Txvv+P9gTly8C2wLdHIWtJtZPBaooleQ3tEeAhwE7Aq2m/tK+kjenZZnjMseR+cSctyRa0\nQPXUoedwK9qKwl+ZPXnAwLuwhrZ4U1XtM7z+C9pkjzOr6l1JrlfDas+2zXRJ2zfz9lX19CS7A/cA\ntq6qF6St27dxVZ01lPWcNyFJjqf1yh9NWw/uTrTz4ZnAY2g3LjNDU5ZcOznGaoqsZUbLr2lTVf9A\nuyt4Ee2x4I2r6mujUOWgzXk2e8BlVf2UNkDzgUk2r6qfAEcAa9vw2gv3PEoye9mYC4HlaeuJUVWr\naevm3HV4fcUWGrbNZK1lIPNmwI0Bqi1Lsgq4W5KtquocQ9XkJbkBrTfxHdW2Snsfrbd+n6r6dVW9\nddZ43yXXTgarKbKWX8BzgFcl2X64APyA9khjs1mf8+Iwj5Lcoq5cUfiwJM8dTi4fBO4MPCHJrrQN\nli+u9XDvq2k1XHAvT7LhMGv2cUNbvZwWcl88FL0Po4U/NR3qyhnP2w5vnQBcL8l+w/HPA3+g7TM3\n/tySu1hPyuwb/mFm7bdps/62qqqLaHs37pRk01lll+S1yUeBUyDJw6vqpOHrk2njqf4DeC/weOB5\ntN6qR9L2VzpoUnVdapLsD+wCHA/8X1ov4h9ovR93Be5O21pjJ+C7deW+WN5Rz7MkDwceBPwLrbfw\nt7RH5B8CPkzr+XgxbTuhX9V6ui/ZYpTk0bTe+AvSNrxeTltj7G0Mi4HSdi+4Pq3tHjexyi5hSe5W\nVWcMXx9F2+Pv1bQeq0fQxvkeQbtGnedEncZgNWFJtgY+RwtRv+PKGWYza1O9Bvg7YAfa+IJ/HT7n\nxWGeJVlOa4+X0LYHun5dud7Rm2kr39+zqn6VZLOq+vlwzHE782x43HpL4EnA1rQV7Q9KsiXtb2Y1\ncAxtFfwbV9XM9k62zYQNbfRy2g3kDYHzaXtqHg18njZh55fAA2mh6sThc57zFlCS+9DG954A/D3t\nBmUT2rZprwV+AjwM2JG2pMzMkgpLvp0MVhOU5J5V9YUkt6WtVLtlVa0Yjt2fFqguBd5Sw/YNwzEv\nDvNs+Pk/jNYuvwFeQeuVOnR0B3cMbaDmzWkX9vKkMv+Gnqqn0NpnK+Bg2hpv+1TVOUluQburvhj4\np7rq/nG2zQQlOYC2v9+2tO1P9gQOrqovJdmBNvNvNfDGqvrR6HOe8xZQkttW1beSPIW2QflWVbX3\ncOwwWs/wMVX1ufHfle3UOMZqQtJWUD9tGBPyLeDZtAG3RwFU1SdpjwO3oT1uuoK/uAviPOB/gANo\nG/UeDnwXuG+S2wNUW0n9kKr6zcyJxQv3gjiVNvvo7bT13d4KnAw8Jcmthwvy82nrVP1s5kO2zWQN\nvYx3prXXebS1xb4JHJRkp2Eg9PNo57t7jT/rOW/hJLkD8N4k96uqt9Bmot8myRMAqi0hcxbwsiS3\nGYUqF9gd2GM1AaO1QO5K27dsn6r6WpJdgGOBz1XVi4ayt6uqsydZ36Uqya2A5VX1xeH1bWkB+HvA\nqVX11VFZ79QWUJLrA39VVZ8YXt8VeAhtUdZjq+qbo7L2VE2JJJvQen+/UFUfHNptb+BGwBuq6twk\nNxsHYi28JHsBh9LG+F4KPBy4C21c3PuHMg+uqlMmV8vpZbCakFy5QvQtZnV534a2yvDXqurg0fte\nHCYoV64KvQttsPTxVXXypOulKw0X6f2AM6rqPZOuj9YubQuoy0ev70YLxdvQHrWvGd73nDdBSXau\nqm8PX9+ctln53YHTq+qdo3K20ywGqykx7vEYHjU9tKr+ecLV0siop3Hr8Zg3TY8kOwyPlDTlctWt\nUf6SthDoByZcLc0y6gS4OW0rqF9W1XGTrtc0M1gtkOua6r0bmF62zcKYy895dhnbZvGy7aZXkk1n\nZtjq6jl4fYEMPR3Lrsvn5qM+apI8OMm9r8tnbZv5leRJSe40M9vymsrObgvbZrLSthVa224S62Tb\nLby5ttNo2ZJr3a5LicFqnqWt1P0ugLpy53ZNgSQ3pW3Ie58kd7+GcrbZAhvaZnPgkCR3GM88mlXO\ntpkySe4HvCHJLlcXkmy3yUvyxCRPh2sfZg2/18xgNf+OBLZP8ka4oudq9sVh9l5nmmfD+I5fAK8H\nQlv4bnx8mySPTNuw15PIAhva5t+B04G/GL1fSW6VZO+Z1xOqoq7eOcBngeenbVZ+hSEw224TNix9\ncQFw5yRPuoZyZoTrwB/aPJl57Fdt37j70TYTHR+/ZZJThoGBl3sHt3BGM/yWARsCh1fVZ2YVuwfw\n18AtFrorfFgcAAAL4UlEQVR+S1mSO6at+AxtFtLpVfWhWcV2Ap6UZLeFrZ2uycxFuKrOBz4AvLXa\nZuUzx28CHJlknwlVUYPhuvQZ2s3Lz8fHkmyX5GWuS3XdGazmwejCvUGSx1bV5TOzKGYGZlbV92nb\npLwevINbKMPP/w/DReAk4FGzpn7P/E2cQtsX6wETqOaSNPzsfws8O8k5tJ0I/nctRb9EW7Rwk7Uc\n0wQM57w/Due8GwNrqm2gPH7sdxnwaWBXe+knYzzOt9rCxp9ay43LxrSlLwzA15HBah6MLtwrafvJ\njY/V0A0LcBhwVtqieVoAowD7HmBVVb0Krlg/jOHisKyqfkdbDPTUydR0aZm5MNNW5L4xsIy2qvqf\nqKqLgf+krYSvCRvfSNLa5Vjg/TOP/RiuM1X1G+AbtAu3PfQLbHxTmeTYoVdqn/Hx4cvvAmcAN5hI\nRdcDLrfQ2WitoyOB61XVIUk2Bf4GOLeqvjEquxGwUVVdOqHqLhnjKdxJNqftV/Z62oai96QtUHhs\nVb18KONK6gtk1oX5DsCvafsvPgv4TFW9fii37fCYSVMoyXNoezceAbyY9jj9AVX1i2Gs4u+Hcjfy\nnDc5aRvIbwJ8CngB8K9V9Y7h2Mzf4ma0a9MFE6zqomWPVWejHpFvAxcmOQH4V+CVtL3MNh+VvcwT\nzPwbThbjTUIvogWrNwO3Bt4F7AHskLZVinuTLZBZj2Y/DhxUbbXnLwFvA+6X5FnDxeC+k6yrrirJ\nQ5JsN3x9JPBQ4JSq+mlVPRP4PG0/1JvNhCoAz3kLK21HgpmvX027TD1+CFNPBf4pyRPhypnrVfVz\nQ9V1Z7DqLMl7h+7Vk4HfA1+h7bl0f2AXWg+JFsis8W5vBI5KsldVvRe4fbU9Gb8EPBe4tKp+O9EK\nLyFJdhzdiLwZOLOqDhx6cu9K+9t5FW0SATXaRkOTlWR7YFlVnTc88nsjbU+5vx3GWFFVzwK+BRwy\nuZoubcPSFzuN3toAuEuSe6RtLfRZWrh6fZK/Asf79uCjwD/T7EdGSfYF/hl4elWdNry3EXAc8Ouq\nOmAyNV26hgGbH6ENnC3g6cBrgQ/TphyfQNum4YlDeVd+nmdJ7g/cpqqOHl7/M7AauBNtDM5DaLPK\nXpqrbvdk20yRJPvTQvD/pY3JeTNtbOnbhrFwmqAkG1XVZUkOBH5aVR9I8i/ADrTNsP93GFd626r6\n1kQrux6xx+rPNPxSJskthpP+ibTejzclechQ7NnAxTOhyqUV5l+SRyXZZnj5INp6SG8AHkgbW3A/\n4JG0HsRXjELVBl6459cwxuaTtL+RlyS5C22SwD2Bi4CXAfcB7pZke0PV9Mif7h5xJm1M3NOBXwEH\nAnsBzxxPyvGct7By5XI/lw3DG25K603cq6peCPyA9ne26/B39a3hc2aCDvwh/hlGJ4tDgaOBHYdf\n0g/TxlW9PskDgNdU1dOHz3jhnmfDo4gDgGcl2aKqTgZeAzwP+M8h4J4NPA64dVV9bfic67bMs7SF\nPVcm2XL4WRfwUuAXVfXUqnolLVy9CPh+Vf1g5rP+3UzWrMfq/5TkybT2ew1taZKn0R4HHgz8bpgF\nCNh2C2lWO+0O7FhVhwNfBR48hKvDgF8A9xm3jee/PgxW18HobqCG/z0c+CHwEloXK8AngLOAPavq\nsuFzXrjn2TD76JfAP9AeUbw8yWZDG1wPuONQdEvgxKr61MxnPfnPv6r6CO1v463DnfQxwCeBV+XK\nbYWOpF2YDwJ7O6bFaJLBR2iPa7emTTj4OXDi8N4/AT+uqiPAtpuEUTudCjwceFuSlwHH08a8PSjJ\nw6rqwJlH8erLMVbX0qyp4S+lzf77YlWtTvJ64CbAh4AnAh+rqmMmV9ulZWYszjCm7e3AL4GH0cZS\nHQbciDbIdmvgG1X1hOFzPmKaZ0n2A+5VVQcNNyZ/VcNq92nbnjyKthjrS4AfVtWa4ZjLXkxY2l6N\nXx++fihwG9oYxZOBU6vqtcOx2wF3GIZDaIKSHAWcTxvz9h/Au6rqTUk2pk0muHC0xILnv84MVtfB\ncGH4BG068XbAZsAbq+rjSZ5NW7X2RlX11KG8v7jzLMmNq+qXQ9u8gDbjb98kW9JOLmtoj2x/BexW\nVWcMn7NtFsAQnp5Beyx+yeyf+3D8AOC3VfWa4T3bZsKSPIi2jMIHq+pjSfYAHkM77320ql47PHp/\nGvDamVm1tt3CWsvf08HA/wDPAT5dVUck2Zl243+mNyvzy0eBc5Q2PXX58PIQYOUwVf/WwMXAo5Pc\nv6peU1XPGYUqx1TNs2GSwMy4nT8A3wMuSLJJVV1IG1D7cNojpo0MVRPxB+B2tIvyn2xGXm1PudfN\nhKqZMgteS832FeA7tPXE7kvbXHlnYJOZniraI6Zb1GipEttu4WS0Tt/IpcBpwKdmHsvSli65z3gy\nyAJWc0lxv6Y5GMZ+vAg4IcmHaAtK/jbJ22njdF6b5HTaYOnfVNX/Gz7nmKoFUFUnp23Ie1ySR9KW\nULg5sEuSb1bVBUk+Dny3RosTevJfOFX187QlFU5JcklVnTAOV9VcCgbeaTDzCLaqfpTkrcA/AnvT\n9nJ8GPDRJB8Ebgh8r9qCoLbdBIyGpnyM1jN/Bm0oxHJg/yQX0dpsTVUdOfqc7TRPfBQ4R2nrU92P\ndsf2EdojpQ8BR1TV55O8lzam6vjJ1XJpWce4ncNoayJdDPwFcHZVHTIc8+Q/IWnrV70OeNX4b8Wx\nVNMjydZV9eMh9L6bNi7xXNq40S1pA9W/TNt+6AajG0nbcAHNjPcdvn4asD3t+jSzQ8EraMvLbA5s\nXFVHDWVtp3lmsLoGSXaqqnNHr98N7AYcDvw78GTaVjXfBH5SVY8ZynnhXgBzGLdzb9r4t21mJhHY\nNpM3tMu7gKOA1VW1csJV0mAYnP5w4F+q6ltJjgdeVlXfS7IV8ATgFsAnquqU0ef8u1pAo4k6GwCP\noO1F+6aq+nqSe9B6FzcEjqqqH83+3EQqvYQ4xupqJPk74D1p64CQtnLtdsBbaT1Xjx6+fiBt0Kah\nauGtddzOcLKhqk6vqo+OQpXj3aZAVZ0O/B1wOXDfDPuUabKS3IC24Oe5wNOGWX4XAJcBVNVPuHK2\n7Y3Hn/XvauGkbUXzx6FH8RRgH9qM2mcCVNUXaTOhN6EFrisYqhaGPVZrkeRGtEd9zwTuRZu2emdg\nv6o6P8ljgT2BzwFvr6rLh895N7DAhrFVpwCHVtUJw3uG20UkyaZVdcmk67GUpS3c+izaRXoT2piq\n7YC702ZAn0lbUHI5bYmFNROq6pI2mv28IfBC2uKf+yfZnDZY/dNV9byh7K2q6juTrO9SZY/VLMMJ\n5lTgZlX1b8AXgcfTeqXOHy7a76HtO7dsJlSBdwOTUG3V9P2BF+XKHdoL3J5hsTBUTV61hVs/TeuR\nWgN8lNZb9RvazOfNaSFrt7pyjTFnlS2g8exn2pp8WwO7J7lTVV1E27rrfkneAjATqmynhWeP1Vok\neQlt1e5H0TYWfSytS/XVVfWFCVZNV8NxO9K1t44JILenXay3Bf5t3Pthr/BkJHkxsII2rmpH2rZc\nmwLHVdVZSbYGnlZVL55gNZc8g9VgHSeY8crQrxrGiGjKpC2AtwewE21l9eMnWyNpus1hAshuwH7A\nWVX1zknVcylby7Xpr6vq08Ox3YCH0GZrHl9VXx59zvA7Ia5jdaWPAbeaGe+R5LMzB6rqp8NyCjcG\ndgcMVlOoqr5N22KIJJtOuDrSYjCeAHLMzASQmQtyVX0tyc+q6ryJ1nJpm31t+szMgaF9ihZ+d6Ut\ngzFzzFA1IfZYDZJsBhxLW6l2rVPzk9yoRgtMStJid3UTQOCqF2d7QCZjjtem7Qy/08NgNeIJRtJS\n5MKt081r0+Lio8CRoVt1f+B1STaqquPHM8xmTjD+4kpan1TVJ5McALxreIy+uqpWGqqmg9emxcUe\nq7VwhpmkpcgJINPNa9PiYLC6Gp5gJC1lLtw6nbw2TT+D1Rx4gpEkTRuvTdPJYCVJktSJW35IkiR1\nYrCSJEnqxGAlSZLUicFKkiSpE4OVJElSJwYrSZKkTv4/xPDBxnlDgZMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe4550ebfd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_ibm1_alpha(\"house\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Alignments?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def show_ibm1_alignments(sent):\n",
    "    return word_mt.Alignment.from_matrix(ibm1_iterations[-1][0][sent],train_model_2[sent][1], train_model_2[sent][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <svg id='42399508-b9a3-11e7-8cfc-0242ac110002' xmlns=\"http://www.w3.org/2000/svg\"\n",
       "             xmlns:xlink=\"http://www.w3.org/1999/xlink\"\n",
       "             viewBox=\"0 0 400 100\"\n",
       "             >\n",
       "\n",
       "            <text x=\"0\" y=\"20\" class=\"source\">\n",
       "                <tspan id='t0'>NULL</tspan> <tspan id='t1'>the</tspan> <tspan id='t2'>house</tspan> <tspan id='t3'>is</tspan> <tspan id='t4'>small</tspan>\n",
       "            </text>\n",
       "            <text x=\"0\" y=\"95\" class=\"target\">\n",
       "                <tspan id='t0'>klein</tspan> <tspan id='t1'>ist</tspan> <tspan id='t2'>das</tspan> <tspan id='t3'>Haus</tspan>\n",
       "            </text>\n",
       "            <g class='connections'></g>\n",
       "            <script>\n",
       "              $(function() {\n",
       "                  root = $(document.getElementById('42399508-b9a3-11e7-8cfc-0242ac110002'));\n",
       "                  root.find('.connections').empty();\n",
       "                  alignments = [['.source #t0','.target #t0',2.6280259695372476e-14],['.source #t1','.target #t0',1.863439198452529e-42],['.source #t2','.target #t0',0.4999999999999737],['.source #t3','.target #t0',2.6280259695372476e-14],['.source #t4','.target #t0',0.4999999999999737],['.source #t0','.target #t1',0.5],['.source #t1','.target #t1',6.940522787929912e-22],['.source #t2','.target #t1',3.88664621703916e-27],['.source #t3','.target #t1',0.5],['.source #t4','.target #t1',3.88664621703916e-27],['.source #t0','.target #t2',0.055552866879861405],['.source #t1','.target #t2',0.8888942662402771],['.source #t2','.target #t2',4.152175776717941e-35],['.source #t3','.target #t2',0.055552866879861405],['.source #t4','.target #t2',4.152175776717941e-35],['.source #t0','.target #t3',2.6280259695372476e-14],['.source #t1','.target #t3',1.863439198452529e-42],['.source #t2','.target #t3',0.4999999999999737],['.source #t3','.target #t3',2.6280259695372476e-14],['.source #t4','.target #t3',0.4999999999999737]];\n",
       "                  function appendLine(alignment) {\n",
       "                      s1 = root.find(alignment[0])[0];\n",
       "                      x1 = s1.getExtentOfChar(0).x + s1.getComputedTextLength() / 2.0;\n",
       "                      y1 = s1.getExtentOfChar(0).y + s1.getExtentOfChar(0).height;\n",
       "                      s2 = root.find(alignment[1])[0];\n",
       "                      x2 = s2.getExtentOfChar(0).x + s2.getComputedTextLength() / 2.0;\n",
       "                      y2 = s2.getExtentOfChar(0).y;\n",
       "                      var newLine = document.createElementNS('http://www.w3.org/2000/svg','line');\n",
       "                      var score = alignment[2];\n",
       "                      newLine.setAttribute('x1',x1.toString());\n",
       "                      newLine.setAttribute('y1',y1.toString());\n",
       "                      newLine.setAttribute('x2',x2.toString());\n",
       "                      newLine.setAttribute('y2',y2.toString());\n",
       "                      newLine.setAttribute('style',\"stroke:black;stroke-width:2;stroke-opacity:\" + score + \";\");\n",
       "                      root.find('.connections').append(newLine)\n",
       "                  };\n",
       "                  //console.log(alignments);\n",
       "                  for (var i = 0; i < alignments.length; i++) {\n",
       "                    appendLine(alignments[i]);\n",
       "                  }\n",
       "                  //console.log($(root).find('.connections'));\n",
       "              });\n",
       "            </script>\n",
       "        </svg>\n",
       "        "
      ],
      "text/plain": [
       "<statnlpbook.word_mt.Alignment at 0x7fe455364860>"
      ]
     },
     "execution_count": 216,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_ibm1_alignments(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Distortions?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def show_ibm1_distortions(si):\n",
    "    util.plot_bar_graph([ibm1.beta[ti,si,5,4] for ti in range(0,5)],\n",
    "                        range(0,5)) # change source index != 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmMAAAFpCAYAAADQuy+GAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAF2NJREFUeJzt3X2opned3/HPtxNDiw8Y17NxyEMTYVCGdo0yZAWFsisu\nSbY48R9JaGMQlzFg1IBSpvtHsfSfVHwoQpoh7g6NdHeDi4pDO92QTYVlWbUzSdOYB1Nn04TMMMnM\namt2VzCO+faPcwVuj2dyrpNM5jf3yesFh3Ndv+vh/l3chHnnvu773NXdAQBgjH8wegIAAK9mYgwA\nYCAxBgAwkBgDABhIjAEADCTGAAAGEmMAAAOJMQCAgcQYAMBAYgwAYKDzRk9gM9785jf3ZZddNnoa\nAAAbuu+++/6mu1c22m+pYuyyyy7L4cOHR08DAGBDVfXknP3cpgQAGEiMAQAMJMYAAAYSYwAAA4kx\nAICBxBgAwEBiDABgIDEGADCQGAMAGGhWjFXVVVX1WFUdqaq962z/F1X1YFV9v6r+qqresdGxVfWm\nqrqnqn44/b7gzFwSAMDy2DDGqmpbktuSXJ1kZ5Lrq2rnmt3+T5J/1t3/NMm/S3LHjGP3Jrm3u3ck\nuXdaBwB4VZnzytiVSY509+Pd/VySu5LsXtyhu/+qu//vtPrdJBfPOHZ3kjun5TuTXPvSLwMAYDnN\nibGLkjy1sH50Gjudjyb5bzOOvbC7j0/LTye5cMZcAAC2lPPO5Mmq6reyGmPv3cxx3d1V1ac5554k\ne5Lk0ksvfdlz3Mhle//rK/4YrzZP3Pq7Z/ycnqczz/N07vMcLQfP03J4JZ6nl2rOK2PHklyysH7x\nNPZLquo3kvxBkt3d/aMZxz5TVdunY7cnObHeg3f3Hd29q7t3rayszJguAMDymBNjh5LsqKrLq+r8\nJNclObC4Q1VdmuQbSW7o7v8989gDSW6clm9M8q2XfhkAAMtpw9uU3X2qqm5OcneSbUn2d/fDVXXT\ntH1fkn+T5NeS/MeqSpJT06tZ6x47nfrWJF+rqo8meTLJh87wtQEAnPNmvWesuw8mObhmbN/C8u8l\n+b25x07jP0ryvs1MFgBgq/EX+AEABhJjAAADiTEAgIHEGADAQGIMAGAgMQYAMJAYAwAYSIwBAAwk\nxgAABhJjAAADiTEAgIHEGADAQGIMAGAgMQYAMJAYAwAYSIwBAAwkxgAABhJjAAADiTEAgIHEGADA\nQGIMAGAgMQYAMJAYAwAYSIwBAAwkxgAABhJjAAADiTEAgIHEGADAQGIMAGAgMQYAMNCsGKuqq6rq\nsao6UlV719n+9qr6TlX9rKo+szD+tqp6YOHn2aq6Zdr22ao6trDtmjN3WQAAy+G8jXaoqm1Jbkvy\n/iRHkxyqqgPd/cjCbj9O8skk1y4e292PJbli4TzHknxzYZcvdffnX9YVAAAssTmvjF2Z5Eh3P97d\nzyW5K8nuxR26+0R3H0ry8xc5z/uS/HV3P/mSZwsAsMXMibGLkjy1sH50Gtus65L8yZqxT1TVg1W1\nv6oueAnnBABYamflDfxVdX6SDyT504Xh25O8Nau3MY8n+cJpjt1TVYer6vDJkydf8bkCAJxNc2Ls\nWJJLFtYvnsY24+ok93f3My8MdPcz3f2L7n4+yVeyejv0V3T3Hd29q7t3raysbPJhAQDObXNi7FCS\nHVV1+fQK13VJDmzyca7PmluUVbV9YfWDSR7a5DkBAJbehp+m7O5TVXVzkruTbEuyv7sfrqqbpu37\nquotSQ4neUOS56c/X7Gzu5+tqtdm9ZOYH1tz6s9V1RVJOskT62wHANjyNoyxJOnug0kOrhnbt7D8\ndFZvX6537N8n+bV1xm/Y1EwBALYgf4EfAGAgMQYAMJAYAwAYSIwBAAwkxgAABhJjAAADiTEAgIHE\nGADAQGIMAGAgMQYAMJAYAwAYSIwBAAwkxgAABhJjAAADiTEAgIHEGADAQGIMAGAgMQYAMJAYAwAY\nSIwBAAwkxgAABhJjAAADiTEAgIHEGADAQGIMAGAgMQYAMJAYAwAYSIwBAAwkxgAABhJjAAADzYqx\nqrqqqh6rqiNVtXed7W+vqu9U1c+q6jNrtj1RVd+vqgeq6vDC+Juq6p6q+uH0+4KXfzkAAMtlwxir\nqm1JbktydZKdSa6vqp1rdvtxkk8m+fxpTvNb3X1Fd+9aGNub5N7u3pHk3mkdAOBVZc4rY1cmOdLd\nj3f3c0nuSrJ7cYfuPtHdh5L8fBOPvTvJndPynUmu3cSxAABbwpwYuyjJUwvrR6exuTrJn1fVfVW1\nZ2H8wu4+Pi0/neTCTZwTAGBLOO8sPMZ7u/tYVf16knuq6gfd/ReLO3R3V1Wvd/AUcHuS5NJLL33l\nZwsAcBbNeWXsWJJLFtYvnsZm6e5j0+8TSb6Z1dueSfJMVW1Pkun3idMcf0d37+ruXSsrK3MfFgBg\nKcyJsUNJdlTV5VV1fpLrkhyYc/Kqem1Vvf6F5SS/k+ShafOBJDdOyzcm+dZmJg4AsBVseJuyu09V\n1c1J7k6yLcn+7n64qm6atu+rqrckOZzkDUmer6pbsvrJyzcn+WZVvfBYf9zdfzad+tYkX6uqjyZ5\nMsmHzuylAQCc+2a9Z6y7DyY5uGZs38Ly01m9fbnWs0necZpz/ijJ+2bPFABgC/IX+AEABhJjAAAD\niTEAgIHEGADAQGIMAGAgMQYAMJAYAwAYSIwBAAwkxgAABhJjAAADiTEAgIHEGADAQGIMAGAgMQYA\nMJAYAwAYSIwBAAwkxgAABhJjAAADiTEAgIHEGADAQGIMAGAgMQYAMJAYAwAYSIwBAAwkxgAABhJj\nAAADiTEAgIHEGADAQGIMAGAgMQYAMNCsGKuqq6rqsao6UlV719n+9qr6TlX9rKo+szB+SVV9u6oe\nqaqHq+pTC9s+W1XHquqB6eeaM3NJAADL47yNdqiqbUluS/L+JEeTHKqqA939yMJuP07yySTXrjn8\nVJJPd/f9VfX6JPdV1T0Lx36puz//sq8CAGBJzXll7MokR7r78e5+LsldSXYv7tDdJ7r7UJKfrxk/\n3t33T8t/m+TRJBedkZkDAGwBc2LsoiRPLawfzUsIqqq6LMk7k3xvYfgTVfVgVe2vqgs2e04AgGV3\nVt7AX1WvS/L1JLd097PT8O1J3prkiiTHk3zhNMfuqarDVXX45MmTZ2O6AABnzZwYO5bkkoX1i6ex\nWarqNVkNsT/q7m+8MN7dz3T3L7r7+SRfyert0F/R3Xd0967u3rWysjL3YQEAlsKcGDuUZEdVXV5V\n5ye5LsmBOSevqkryh0ke7e4vrtm2fWH1g0kemjdlAICtY8NPU3b3qaq6OcndSbYl2d/dD1fVTdP2\nfVX1liSHk7whyfNVdUuSnUl+I8kNSb5fVQ9Mp/z97j6Y5HNVdUWSTvJEko+d2UsDADj3bRhjSTLF\n08E1Y/sWlp/O6u3Ltf4ySZ3mnDfMnyYAwNbkL/ADAAwkxgAABhJjAAADiTEAgIHEGADAQGIMAGAg\nMQYAMJAYAwAYSIwBAAwkxgAABhJjAAADiTEAgIHEGADAQGIMAGAgMQYAMJAYAwAYSIwBAAwkxgAA\nBhJjAAADiTEAgIHEGADAQGIMAGAgMQYAMJAYAwAYSIwBAAwkxgAABhJjAAADiTEAgIHEGADAQGIM\nAGCgWTFWVVdV1WNVdaSq9q6z/e1V9Z2q+llVfWbOsVX1pqq6p6p+OP2+4OVfDgDActkwxqpqW5Lb\nklydZGeS66tq55rdfpzkk0k+v4lj9ya5t7t3JLl3WgcAeFWZ88rYlUmOdPfj3f1ckruS7F7cobtP\ndPehJD/fxLG7k9w5Ld+Z5NqXeA0AAEtrToxdlOSphfWj09gcL3bshd19fFp+OsmF652gqvZU1eGq\nOnzy5MmZDwsAsBzOiTfwd3cn6dNsu6O7d3X3rpWVlbM8MwCAV9acGDuW5JKF9YunsTle7Nhnqmp7\nkky/T8w8JwDAljEnxg4l2VFVl1fV+UmuS3Jg5vlf7NgDSW6clm9M8q350wYA2BrO22iH7j5VVTcn\nuTvJtiT7u/vhqrpp2r6vqt6S5HCSNyR5vqpuSbKzu59d79jp1Lcm+VpVfTTJk0k+dKYvDgDgXLdh\njCVJdx9McnDN2L6F5aezegty1rHT+I+SvG8zkwUA2GrOiTfwAwC8WokxAICBxBgAwEBiDABgIDEG\nADCQGAMAGEiMAQAMJMYAAAYSYwAAA4kxAICBxBgAwEBiDABgIDEGADCQGAMAGEiMAQAMJMYAAAYS\nYwAAA4kxAICBxBgAwEBiDABgIDEGADCQGAMAGEiMAQAMJMYAAAYSYwAAA4kxAICBxBgAwEBiDABg\nIDEGADDQrBirqquq6rGqOlJVe9fZXlX15Wn7g1X1rmn8bVX1wMLPs1V1y7Tts1V1bGHbNWf20gAA\nzn3nbbRDVW1LcluS9yc5muRQVR3o7kcWdrs6yY7p5zeT3J7kN7v7sSRXLJznWJJvLhz3pe7+/Jm4\nEACAZTTnlbErkxzp7se7+7kkdyXZvWaf3Um+2qu+m+SNVbV9zT7vS/LX3f3ky541AMAWMSfGLkry\n1ML60Wlss/tcl+RP1ox9Yrqtub+qLpgxFwCALeWsvIG/qs5P8oEkf7owfHuSt2b1NubxJF84zbF7\nqupwVR0+efLkKz5XAICzaU6MHUtyycL6xdPYZva5Osn93f3MCwPd/Ux3/6K7n0/ylazeDv0V3X1H\nd+/q7l0rKyszpgsAsDzmxNihJDuq6vLpFa7rkhxYs8+BJB+ePlX57iQ/6e7jC9uvz5pblGveU/bB\nJA9tevYAAEtuw09Tdvepqro5yd1JtiXZ390PV9VN0/Z9SQ4muSbJkSQ/TfKRF46vqtdm9ZOYH1tz\n6s9V1RVJOskT62wHANjyNoyxJOnug1kNrsWxfQvLneTjpzn275P82jrjN2xqpgAAW5C/wA8AMJAY\nAwAYSIwBAAwkxgAABhJjAAADiTEAgIHEGADAQGIMAGAgMQYAMJAYAwAYSIwBAAwkxgAABhJjAAAD\niTEAgIHEGADAQGIMAGAgMQYAMJAYAwAYSIwBAAwkxgAABhJjAAADiTEAgIHEGADAQGIMAGAgMQYA\nMJAYAwAYSIwBAAwkxgAABhJjAAADiTEAgIFmxVhVXVVVj1XVkarau872qqovT9sfrKp3LWx7oqq+\nX1UPVNXhhfE3VdU9VfXD6fcFZ+aSAACWx4YxVlXbktyW5OokO5NcX1U71+x2dZId08+eJLev2f5b\n3X1Fd+9aGNub5N7u3pHk3mkdAOBVZc4rY1cmOdLdj3f3c0nuSrJ7zT67k3y1V303yRuravsG592d\n5M5p+c4k125i3gAAW8KcGLsoyVML60ensbn7dJI/r6r7qmrPwj4XdvfxafnpJBfOnjUAwBZx3ll4\njPd297Gq+vUk91TVD7r7LxZ36O6uql7v4Cng9iTJpZde+srPFgDgLJrzytixJJcsrF88jc3ap7tf\n+H0iyTezetszSZ554Vbm9PvEeg/e3Xd0967u3rWysjJjugAAy2NOjB1KsqOqLq+q85Ncl+TAmn0O\nJPnw9KnKdyf5SXcfr6rXVtXrk6SqXpvkd5I8tHDMjdPyjUm+9TKvBQBg6Wx4m7K7T1XVzUnuTrIt\nyf7ufriqbpq270tyMMk1SY4k+WmSj0yHX5jkm1X1wmP9cXf/2bTt1iRfq6qPJnkyyYfO2FUBACyJ\nWe8Z6+6DWQ2uxbF9C8ud5OPrHPd4knec5pw/SvK+zUwWAGCr8Rf4AQAGEmMAAAOJMQCAgcQYAMBA\nYgwAYCAxBgAwkBgDABhIjAEADCTGAAAGEmMAAAOJMQCAgcQYAMBAYgwAYCAxBgAwkBgDABhIjAEA\nDCTGAAAGEmMAAAOJMQCAgcQYAMBAYgwAYCAxBgAwkBgDABhIjAEADCTGAAAGEmMAAAOJMQCAgcQY\nAMBAYgwAYCAxBgAw0KwYq6qrquqxqjpSVXvX2V5V9eVp+4NV9a5p/JKq+nZVPVJVD1fVpxaO+WxV\nHauqB6afa87cZQEALIfzNtqhqrYluS3J+5McTXKoqg509yMLu12dZMf085tJbp9+n0ry6e6+v6pe\nn+S+qrpn4dgvdffnz9zlAAAslzmvjF2Z5Eh3P97dzyW5K8nuNfvsTvLVXvXdJG+squ3dfby770+S\n7v7bJI8muegMzh8AYKnNibGLkjy1sH40vxpUG+5TVZcleWeS7y0Mf2K6rbm/qi6YOWcAgC3jrLyB\nv6pel+TrSW7p7men4duTvDXJFUmOJ/nCaY7dU1WHq+rwyZMnz8Z0AQDOmjkxdizJJQvrF09js/ap\nqtdkNcT+qLu/8cIO3f1Md/+iu59P8pWs3g79Fd19R3fv6u5dKysrM6YLALA85sTYoSQ7quryqjo/\nyXVJDqzZ50CSD0+fqnx3kp909/GqqiR/mOTR7v7i4gFVtX1h9YNJHnrJVwEAsKQ2/DRld5+qqpuT\n3J1kW5L93f1wVd00bd+X5GCSa5IcSfLTJB+ZDn9PkhuSfL+qHpjGfr+7Dyb5XFVdkaSTPJHkY2fs\nqgAAlsSGMZYkUzwdXDO2b2G5k3x8neP+Mkmd5pw3bGqmAABbkL/ADwAwkBgDABhIjAEADCTGAAAG\nEmMAAAOJMQCAgcQYAMBAYgwAYCAxBgAwkBgDABhIjAEADCTGAAAGEmMAAAOJMQCAgcQYAMBAYgwA\nYCAxBgAwkBgDABhIjAEADCTGAAAGEmMAAAOJMQCAgcQYAMBAYgwAYCAxBgAwkBgDABhIjAEADCTG\nAAAGEmMAAAOJMQCAgWbFWFVdVVWPVdWRqtq7zvaqqi9P2x+sqndtdGxVvamq7qmqH06/LzgzlwQA\nsDw2jLGq2pbktiRXJ9mZ5Pqq2rlmt6uT7Jh+9iS5fcaxe5Pc2907ktw7rQMAvKrMeWXsyiRHuvvx\n7n4uyV1Jdq/ZZ3eSr/aq7yZ5Y1Vt3+DY3UnunJbvTHLty7wWAIClMyfGLkry1ML60Wlszj4vduyF\n3X18Wn46yYUz5wwAsGWcN3oCSdLdXVW93raq2pPVW59J8ndV9djZm9k5781J/mb0JDZS/370DIZa\niuco8TxlCZ6nV/lzlHielsFSPEfJWXue/vGcnebE2LEklyysXzyNzdnnNS9y7DNVtb27j0+3NE+s\n9+DdfUeSO2bM81Wnqg53967R8+D0PEfLwfO0HDxP5z7P0Usz5zbloSQ7quryqjo/yXVJDqzZ50CS\nD0+fqnx3kp9MtyBf7NgDSW6clm9M8q2XeS0AAEtnw1fGuvtUVd2c5O4k25Ls7+6Hq+qmafu+JAeT\nXJPkSJKfJvnIix07nfrWJF+rqo8meTLJh87olQEALIHqXvetWiyBqtoz3cblHOU5Wg6ep+XgeTr3\neY5eGjEGADCQr0MCABhIjC2hjb6eivGqan9Vnaiqh0bPhdOrqkuq6ttV9UhVPVxVnxo9J35ZVf3D\nqvofVfW/pufo346eE6dXVduq6n9W1X8ZPZdlIsaWzMyvp2K8/5TkqtGTYEOnkny6u3cmeXeSj/vv\n6ZzzsyS/3d3vSHJFkqumT+1zbvpUkkdHT2LZiLHlM+frqRisu/8iyY9Hz4MX193Hu/v+aflvs/qP\nyNpvGGGg6Wv2/m5afc30483O56CqujjJ7yb5g9FzWTZibPnM+XoqYJOq6rIk70zyvbEzYa3p1tcD\nWf3j4Pd0t+fo3PQfkvyrJM+PnsiyEWPAq15VvS7J15Pc0t3Pjp4Pv6y7f9HdV2T1W1yurKp/MnpO\n/LKq+udJTnT3faPnsozE2PKZ8/VUwExV9Zqshtgfdfc3Rs+H0+vu/5fk2/F+zHPRe5J8oKqeyOrb\nZ367qv7z2CktDzG2fOZ8PRUwQ1VVkj9M8mh3f3H0fPhVVbVSVW+clv9Rkvcn+cHYWbFWd//r7r64\nuy/L6r9L/727/+XgaS0NMbZkuvtUkhe+YurRJF9b+IopzhFV9SdJvpPkbVV1dPraL84970lyQ1b/\nL/6B6eea0ZPil2xP8u2qejCr/zN6T3f7swlsKf4CPwDAQF4ZAwAYSIwBAAwkxgAABhJjAAADiTEA\ngIHEGADAQGIMAGAgMQYAMND/Bxv1RCXyFXfpAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe455445cf8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_ibm1_distortions(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Now learn **distortion table** in Model 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "ibm1 = ibm1_iterations[-1][1] # model 1 of last iteration \n",
    "\n",
    "def show_ibm2_alpha(num_iterations, target):\n",
    "    ibm2_iterations = em_model2(ibm1, train_model_2, num_iterations)\n",
    "    ibm2 = ibm2_iterations[-1][1] # model 2 of last iteration\n",
    "    plot_table_for_target(ibm2.alpha, target)\n",
    "    \n",
    "def show_ibm2_alignments(num_iterations, sent):\n",
    "    ibm2_iterations = em_model2(ibm1, train_model_2, num_iterations)\n",
    "    ibm2 = ibm2_iterations[-1][1] # model 2 of last iteration\n",
    "    return word_mt.Alignment.from_matrix(ibm2_iterations[-1][0][sent],train_model_2[sent][1], train_model_2[sent][0])\n",
    "\n",
    "def show_ibm2_distortions(num_iterations, si):\n",
    "    ibm2_iterations = em_model2(ibm1, train_model_2, num_iterations)\n",
    "    ibm2 = ibm2_iterations[-1][1] # model 2 of last iteration\n",
    "    util.plot_bar_graph([ibm2.beta[ti,si,5,4] for ti in range(0,5)],\n",
    "                    range(0,5)) # change source index != 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAFpCAYAAABeYWb6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAD19JREFUeJzt3WGoZPdZx/Hf426DlaoRs0rNZt28iNVFTa1rLLRgbalu\norgIvkiqLYaWJZBIBcHGN4r0jaUopTTtstQQitIgNGisq7FgtS/aaDaapt3GlCWtzaaFJFartWBY\n8/jiTsvNbZI7SZ7NzKSfD1yYc86fOQ8clvvdM3NnqrsDAMBz922rHgAA4IVCWAEADBFWAABDhBUA\nwBBhBQAwRFgBAAwRVgAAQ4QVAMAQYQUAMERYAQAM2buqE1900UV98ODBVZ0eAGBpd99996PdvW+3\ndSsLq4MHD+bUqVOrOj0AwNKq6t+WWeelQACAIcIKAGCIsAIAGCKsAACGCCsAgCHCCgBgiLACABgi\nrAAAhggrAIAhu4ZVVd1cVQ9X1aef4nhV1bur6kxV3VtVr5gfEwBg/S1zx+qWJEee5viVSS5b/BxL\n8r7nPhYAwObZNay6+2NJvvw0S44m+UBvuTPJhVX10qkBAQA2xcR7rC5O8uC27bOLfQAA31L2Pp8n\nq6pj2Xq5MAcOHHg+Tw0Aa+HgjX+16hFecD7/B7+w6hG+YeKO1UNJLtm2vX+x75t094nuPtzdh/ft\n2zdwagCA9TERVrcnedPirwNfmeQr3f2lgecFANgou74UWFUfTPKaJBdV1dkkv5fkRUnS3ceTnExy\nVZIzSb6W5NrzNSwAwDrbNay6+5pdjneS68cmAgDYUD55HQBgiLACABgirAAAhggrAIAhwgoAYIiw\nAgAYIqwAAIYIKwCAIcIKAGCIsAIAGCKsAACGCCsAgCHCCgBgiLACABgirAAAhggrAIAhwgoAYIiw\nAgAYIqwAAIYIKwCAIcIKAGCIsAIAGCKsAACGCCsAgCHCCgBgiLACABgirAAAhggrAIAhwgoAYIiw\nAgAYIqwAAIYIKwCAIcIKAGCIsAIAGCKsAACGCCsAgCHCCgBgiLACABgirAAAhggrAIAhwgoAYIiw\nAgAYIqwAAIYIKwCAIcIKAGCIsAIAGCKsAACGCCsAgCHCCgBgiLACABgirAAAhggrAIAhwgoAYMhS\nYVVVR6rq/qo6U1U3Psnx766qv6yqT1bV6aq6dn5UAID1tmtYVdWeJDcluTLJoSTXVNWhHcuuT/KZ\n7r48yWuS/GFVXTA8KwDAWlvmjtUVSc509wPd/ViSW5Mc3bGmk3xnVVWSlyT5cpJzo5MCAKy5ZcLq\n4iQPbts+u9i33XuS/EiSLyb5VJK3dvfjIxMCAGyIqTev/3ySe5L8QJKXJ3lPVX3XzkVVdayqTlXV\nqUceeWTo1AAA62GZsHooySXbtvcv9m13bZLbesuZJJ9L8sM7n6i7T3T34e4+vG/fvmc7MwDAWlom\nrO5KcllVXbp4Q/rVSW7fseYLSV6XJFX1/UleluSByUEBANbd3t0WdPe5qrohyR1J9iS5ubtPV9V1\ni+PHk7w9yS1V9akkleRt3f3oeZwbAGDt7BpWSdLdJ5Oc3LHv+LbHX0zyc7OjAQBsFp+8DgAwRFgB\nAAwRVgAAQ4QVAMAQYQUAMERYAQAMEVYAAEOEFQDAEGEFADBEWAEADBFWAABDhBUAwBBhBQAwRFgB\nAAwRVgAAQ4QVAMAQYQUAMERYAQAMEVYAAEOEFQDAEGEFADBEWAEADBFWAABDhBUAwBBhBQAwRFgB\nAAwRVgAAQ4QVAMAQYQUAMERYAQAMEVYAAEOEFQDAEGEFADBEWAEADBFWAABDhBUAwBBhBQAwRFgB\nAAwRVgAAQ4QVAMAQYQUAMERYAQAMEVYAAEOEFQDAEGEFADBEWAEADBFWAABDhBUAwBBhBQAwRFgB\nAAwRVgAAQ4QVAMAQYQUAMGSpsKqqI1V1f1Wdqaobn2LNa6rqnqo6XVX/MDsmAMD627vbgqrak+Sm\nJK9PcjbJXVV1e3d/ZtuaC5O8N8mR7v5CVX3f+RoYAGBdLXPH6ookZ7r7ge5+LMmtSY7uWPOGJLd1\n9xeSpLsfnh0TAGD9LRNWFyd5cNv22cW+7X4oyfdU1d9X1d1V9aapAQEANsWuLwU+g+f5ySSvS/Li\nJJ+oqju7+7PbF1XVsSTHkuTAgQNDpwYAWA/L3LF6KMkl27b3L/ZtdzbJHd39P939aJKPJbl85xN1\n94nuPtzdh/ft2/dsZwYAWEvLhNVdSS6rqkur6oIkVye5fceav0jy6qraW1XfkeSnk9w3OyoAwHrb\n9aXA7j5XVTckuSPJniQ3d/fpqrpucfx4d99XVX+T5N4kjyd5f3d/+nwODgCwbpZ6j1V3n0xycse+\n4zu235nknXOjAQBsFp+8DgAwRFgBAAwRVgAAQ4QVAMAQYQUAMERYAQAMEVYAAEOEFQDAEGEFADBE\nWAEADBFWAABDhBUAwBBhBQAwRFgBAAwRVgAAQ4QVAMAQYQUAMERYAQAMEVYAAEOEFQDAEGEFADBE\nWAEADBFWAABDhBUAwBBhBQAwRFgBAAwRVgAAQ4QVAMAQYQUAMERYAQAMEVYAAEOEFQDAEGEFADBE\nWAEADBFWAABDhBUAwBBhBQAwRFgBAAwRVgAAQ4QVAMAQYQUAMERYAQAMEVYAAEOEFQDAEGEFADBE\nWAEADBFWAABDhBUAwBBhBQAwRFgBAAwRVgAAQ4QVAMAQYQUAMERYAQAMWSqsqupIVd1fVWeq6san\nWfdTVXWuqn5lbkQAgM2wa1hV1Z4kNyW5MsmhJNdU1aGnWPeOJH87PSQAwCZY5o7VFUnOdPcD3f1Y\nkluTHH2Sdb+R5ENJHh6cDwBgYywTVhcneXDb9tnFvm+oqouT/HKS982NBgCwWabevP6uJG/r7sef\nblFVHauqU1V16pFHHhk6NQDAeti7xJqHklyybXv/Yt92h5PcWlVJclGSq6rqXHf/+fZF3X0iyYkk\nOXz4cD/boQEA1tEyYXVXksuq6tJsBdXVSd6wfUF3X/r1x1V1S5IP74wqAIAXul3DqrvPVdUNSe5I\nsifJzd19uqquWxw/fp5nBADYCMvcsUp3n0xycse+Jw2q7v715z4WAMDm8cnrAABDhBUAwBBhBQAw\nRFgBAAwRVgAAQ4QVAMAQYQUAMERYAQAMEVYAAEOEFQDAEGEFADBEWAEADBFWAABDhBUAwBBhBQAw\nRFgBAAwRVgAAQ4QVAMAQYQUAMERYAQAMEVYAAEOEFQDAEGEFADBEWAEADBFWAABDhBUAwBBhBQAw\nRFgBAAwRVgAAQ4QVAMAQYQUAMERYAQAMEVYAAEOEFQDAEGEFADBEWAEADBFWAABDhBUAwBBhBQAw\nRFgBAAwRVgAAQ4QVAMAQYQUAMERYAQAMEVYAAEOEFQDAEGEFADBEWAEADBFWAABDhBUAwBBhBQAw\nRFgBAAwRVgAAQ5YKq6o6UlX3V9WZqrrxSY7/alXdW1WfqqqPV9Xl86MCAKy3XcOqqvYkuSnJlUkO\nJbmmqg7tWPa5JD/T3T+W5O1JTkwPCgCw7pa5Y3VFkjPd/UB3P5bk1iRHty/o7o93938sNu9Msn92\nTACA9bdMWF2c5MFt22cX+57Km5P89XMZCgBgE+2dfLKq+tlshdWrn+L4sSTHkuTAgQOTpwYAWLll\n7lg9lOSSbdv7F/ueoKp+PMn7kxzt7n9/sifq7hPdfbi7D+/bt+/ZzAsAsLaWCau7klxWVZdW1QVJ\nrk5y+/YFVXUgyW1J3tjdn50fEwBg/e36UmB3n6uqG5LckWRPkpu7+3RVXbc4fjzJ7yb53iTvraok\nOdfdh8/f2AAA62ep91h198kkJ3fsO77t8VuSvGV2NACAzeKT1wEAhggrAIAhwgoAYIiwAgAYIqwA\nAIYIKwCAIcIKAGCIsAIAGCKsAACGCCsAgCHCCgBgiLACABgirAAAhggrAIAhwgoAYIiwAgAYIqwA\nAIYIKwCAIcIKAGCIsAIAGCKsAACGCCsAgCHCCgBgiLACABgirAAAhggrAIAhwgoAYIiwAgAYIqwA\nAIYIKwCAIcIKAGCIsAIAGCKsAACGCCsAgCHCCgBgiLACABgirAAAhggrAIAhwgoAYIiwAgAYIqwA\nAIYIKwCAIcIKAGCIsAIAGCKsAACGCCsAgCHCCgBgiLACABgirAAAhggrAIAhwgoAYIiwAgAYIqwA\nAIYsFVZVdaSq7q+qM1V145Mcr6p69+L4vVX1ivlRAQDW265hVVV7ktyU5Mokh5JcU1WHdiy7Msll\ni59jSd43PCcAwNpb5o7VFUnOdPcD3f1YkluTHN2x5miSD/SWO5NcWFUvHZ4VAGCtLRNWFyd5cNv2\n2cW+Z7oGAOAFbe/zebKqOpatlwqT5KtVdf/zef41d1GSR1c9BLtyndafa7QZXKfNsBHXqd7xvJzm\nB5dZtExYPZTkkm3b+xf7numadPeJJCeWGexbTVWd6u7Dq56Dp+c6rT/XaDO4TpvBdXrmlnkp8K4k\nl1XVpVV1QZKrk9y+Y83tSd60+OvAVyb5Snd/aXhWAIC1tusdq+4+V1U3JLkjyZ4kN3f36aq6bnH8\neJKTSa5KcibJ15Jce/5GBgBYT0u9x6q7T2YrnrbvO77tcSe5fna0bzleIt0MrtP6c402g+u0GVyn\nZ6i2mggAgOfKV9oAAAwRViu229cFsXpVdXNVPVxVn171LDy1qrqkqj5aVZ+pqtNV9dZVz8QTVdW3\nV9U/VdUnF9fo91c9E0+tqvZU1b9U1YdXPcsmEVYrtOTXBbF6tyQ5suoh2NW5JL/V3YeSvDLJ9f49\nrZ3/TfLa7r48ycuTHFn8JTnr6a1J7lv1EJtGWK3WMl8XxIp198eSfHnVc/D0uvtL3f3Pi8f/na1f\nCL4BYo0svvbsq4vNFy1+vNF3DVXV/iS/kOT9q55l0wir1fJVQHAeVNXBJD+R5B9XOwk7LV5euifJ\nw0k+0t2u0Xp6V5LfTvL4qgfZNMIKeEGpqpck+VCS3+zu/1r1PDxRd/9fd788W9/QcUVV/eiqZ+KJ\nquoXkzzc3XevepZNJKxWa6mvAgKWU1UvylZU/Wl337bqeXhq3f2fST4a719cR69K8ktV9flsvUXl\ntVX1J6sdaXMIq9Va5uuCgCVUVSX54yT3dfcfrXoevllV7auqCxePX5zk9Un+dbVTsVN3/0537+/u\ng9n6vfR33f1rKx5rYwirFeruc0m+/nVB9yX5s+4+vdqp2KmqPpjkE0leVlVnq+rNq56JJ/WqJG/M\n1v+u71n8XLXqoXiClyb5aFXdm63/WH6ku/0pPy8oPnkdAGCIO1YAAEOEFQDAEGEFADBEWAEADBFW\nAABDhBUAwBBhBQAwRFgBAAz5fxEo2XyC39ByAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe45549fe80>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_ibm2_distortions(10,0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Alignments? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <svg id='9e145192-b9a3-11e7-8cfc-0242ac110002' xmlns=\"http://www.w3.org/2000/svg\"\n",
       "             xmlns:xlink=\"http://www.w3.org/1999/xlink\"\n",
       "             viewBox=\"0 0 400 100\"\n",
       "             >\n",
       "\n",
       "            <text x=\"0\" y=\"20\" class=\"source\">\n",
       "                <tspan id='t0'>NULL</tspan> <tspan id='t1'>the</tspan> <tspan id='t2'>house</tspan> <tspan id='t3'>is</tspan> <tspan id='t4'>small</tspan>\n",
       "            </text>\n",
       "            <text x=\"0\" y=\"95\" class=\"target\">\n",
       "                <tspan id='t0'>klein</tspan> <tspan id='t1'>ist</tspan> <tspan id='t2'>das</tspan> <tspan id='t3'>Haus</tspan>\n",
       "            </text>\n",
       "            <g class='connections'></g>\n",
       "            <script>\n",
       "              $(function() {\n",
       "                  root = $(document.getElementById('9e145192-b9a3-11e7-8cfc-0242ac110002'));\n",
       "                  root.find('.connections').empty();\n",
       "                  alignments = [['.source #t0','.target #t0',0.0],['.source #t1','.target #t0',0.0],['.source #t2','.target #t0',1.0728382442222892e-168],['.source #t3','.target #t0',0.0],['.source #t4','.target #t0',1.0],['.source #t0','.target #t1',0.5],['.source #t1','.target #t1',0.0],['.source #t2','.target #t1',0.0],['.source #t3','.target #t1',0.5],['.source #t4','.target #t1',0.0],['.source #t0','.target #t2',0.0],['.source #t1','.target #t2',1.0],['.source #t2','.target #t2',0.0],['.source #t3','.target #t2',0.0],['.source #t4','.target #t2',0.0],['.source #t0','.target #t3',0.0],['.source #t1','.target #t3',0.0],['.source #t2','.target #t3',1.0],['.source #t3','.target #t3',0.0],['.source #t4','.target #t3',1.4432122651435016e-167]];\n",
       "                  function appendLine(alignment) {\n",
       "                      s1 = root.find(alignment[0])[0];\n",
       "                      x1 = s1.getExtentOfChar(0).x + s1.getComputedTextLength() / 2.0;\n",
       "                      y1 = s1.getExtentOfChar(0).y + s1.getExtentOfChar(0).height;\n",
       "                      s2 = root.find(alignment[1])[0];\n",
       "                      x2 = s2.getExtentOfChar(0).x + s2.getComputedTextLength() / 2.0;\n",
       "                      y2 = s2.getExtentOfChar(0).y;\n",
       "                      var newLine = document.createElementNS('http://www.w3.org/2000/svg','line');\n",
       "                      var score = alignment[2];\n",
       "                      newLine.setAttribute('x1',x1.toString());\n",
       "                      newLine.setAttribute('y1',y1.toString());\n",
       "                      newLine.setAttribute('x2',x2.toString());\n",
       "                      newLine.setAttribute('y2',y2.toString());\n",
       "                      newLine.setAttribute('style',\"stroke:black;stroke-width:2;stroke-opacity:\" + score + \";\");\n",
       "                      root.find('.connections').append(newLine)\n",
       "                  };\n",
       "                  //console.log(alignments);\n",
       "                  for (var i = 0; i < alignments.length; i++) {\n",
       "                    appendLine(alignments[i]);\n",
       "                  }\n",
       "                  //console.log($(root).find('.connections'));\n",
       "              });\n",
       "            </script>\n",
       "        </svg>\n",
       "        "
      ],
      "text/plain": [
       "<statnlpbook.word_mt.Alignment at 0x7fe455576470>"
      ]
     },
     "execution_count": 222,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_ibm2_alignments(10, 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Translation Table? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAGfCAYAAACdqpz+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xu87uWc//HXu50IRbKjo2ISDXLYjmOGEaMwchhDKGIk\nh/RzLOfT/GbiJ41DSpKcmhxySG0ajPj1wyihpIkth3KoLUROTfr8/ri+d/vubtdecq1132uv1/Px\n6GHd9/e697qsa63v9/29vtchVYUkSZL+fBtMuwKSJEnrC4OVJElSJwYrSZKkTgxWkiRJnRisJEmS\nOjFYSZIkdWKwkiRJ6sRgJUmS1InBSpIkqZMNp/WNb37zm9f2228/rW8vSZI0Z1/96ld/VlXL11Vu\nasFq++235/TTT5/Wt5ckSZqzJD+YSzkfBUqSJHVisJIkSerEYCVJktSJwUqSJKkTg5UkSVInBitJ\nkqRODFaSJEmdGKwkSZI6MVhJkiR1ss5gleToJBcl+eY1HE+SNydZleTMJHftX01JkqTZN5ceq2OA\n3a7l+O7AjsN/+wKH//nVkiRJWnzWGayq6gvAz6+lyB7Ae6r5MnDTJFv2qqAkSdJi0WOM1dbA+WOv\nLxjekyRJWlIWdPB6kn2TnJ7k9NWrVy/kt5YkSZp3G3b4N34EbDv2epvhvaupqiOBIwFWrFhRHb73\nOm1/0EkL8W2WjO8f/NBpV0GSpJnVo8fqBGDvYXbgvYBLquonHf5dSZKkRWWdPVZJ/h24P3DzJBcA\nrwSuB1BVRwArgYcAq4DfAvvMV2UlSZJm2TqDVVXtuY7jBTyrW40kSZIWKVdelyRJ6sRgJUmS1InB\nSpIkqRODlSRJUicGK0mSpE4MVpIkSZ0YrCRJkjoxWEmSJHVisJIkSerEYCVJktSJwUqSJKkTg5Uk\nSVInBitJkqRODFaSJEmdGKwkSZI6MVhJkiR1YrCSJEnqxGAlSZLUicFKkiSpE4OVJElSJwYrSZKk\nTgxWkiRJnRisJEmSOjFYSZIkdWKwkiRJ6sRgJUmS1InBSpIkqRODlSRJUicGK0mSpE4MVpIkSZ0Y\nrCRJkjoxWEmSJHVisJIkSerEYCVJktSJwUqSJKkTg5UkSVInBitJkqRODFaSJEmdGKwkSZI6MVhJ\nkiR1YrCSJEnqxGAlSZLUicFKkiSpE4OVJElSJwYrSZKkTgxWkiRJnRisJEmSOjFYSZIkdWKwkiRJ\n6sRgJUmS1InBSpIkqRODlSRJUicGK0mSpE7mFKyS7Jbk3CSrkhy0luM3SfKJJN9IcnaSffpXVZIk\nabatM1glWQYcBuwO7AzsmWTniWLPAr5VVbsA9wcOSbJR57pKkiTNtLn0WN0DWFVV51XVZcBxwB4T\nZQrYJEmAGwM/By7vWlNJkqQZN5dgtTVw/tjrC4b3xr0VuD3wY+As4ICquqJLDSVJkhaJXoPXHwx8\nHdgKuDPw1iSbThZKsm+S05Ocvnr16k7fWpIkaTbMJVj9CNh27PU2w3vj9gE+Us0q4HvA7Sb/oao6\nsqpWVNWK5cuXX9c6S5IkzaS5BKvTgB2T7DAMSH8ccMJEmR8CuwIkuQWwE3Bez4pKkiTNug3XVaCq\nLk/ybOBkYBlwdFWdnWS/4fgRwGuBY5KcBQQ4sKp+No/1liRJmjnrDFYAVbUSWDnx3hFjX/8Y+Lu+\nVZMkSVpcXHldkiSpE4OVJElSJwYrSZKkTgxWkiRJnRisJEmSOjFYSZIkdWKwkiRJ6sRgJUmS1InB\nSpIkqRODlSRJUicGK0mSpE4MVpIkSZ0YrCRJkjoxWEmSJHVisJIkSerEYCVJktSJwUqSJKkTg5Uk\nSVInBitJkqRODFaSJEmdGKwkSZI6MVhJkiR1YrCSJEnqxGAlSZLUicFKkiSpE4OVJElSJwYrSZKk\nTgxWkiRJnRisJEmSOjFYSZIkdWKwkiRJ6sRgJUmS1InBSpIkqRODlSRJUicGK0mSpE4MVpIkSZ0Y\nrCRJkjoxWEmSJHVisJIkSerEYCVJktSJwUqSJKkTg5UkSVInBitJkqRODFaSJEmdGKwkSZI6MVhJ\nkiR1YrCSJEnqxGAlSZLUicFKkiSpE4OVJElSJwYrSZKkTgxWkiRJnRisJEmSOjFYSZIkdTKnYJVk\ntyTnJlmV5KBrKHP/JF9PcnaSz/etpiRJ0uzbcF0FkiwDDgMeBFwAnJbkhKr61liZmwJvA3arqh8m\n2WK+KixJkjSr5tJjdQ9gVVWdV1WXAccBe0yUeTzwkar6IUBVXdS3mpIkSbNvLsFqa+D8sdcXDO+N\nuy2wWZJTknw1yd5r+4eS7Jvk9CSnr169+rrVWJIkaUb1Gry+IXA34KHAg4GXJ7ntZKGqOrKqVlTV\niuXLl3f61pIkSbNhnWOsgB8B24693mZ4b9wFwMVV9RvgN0m+AOwCfLtLLSVJkhaBufRYnQbsmGSH\nJBsBjwNOmCjzceC+STZMckPgnsA5fasqSZI029bZY1VVlyd5NnAysAw4uqrOTrLfcPyIqjonyaeA\nM4ErgKOq6pvzWXFJkqRZM5dHgVTVSmDlxHtHTLz+P8D/6Vc1SZKkxcWV1yVJkjoxWEmSJHVisJIk\nSerEYCVJktSJwUqSJKkTg5UkSVInBitJkqRODFaSJEmdGKwkSZI6MVhJkiR1YrCSJEnqxGAlSZLU\nicFKkiSpE4OVJElSJwYrSZKkTgxWkiRJnRisJEmSOjFYSZIkdWKwkiRJ6sRgJUmS1InBSpIkqROD\nlSRJUicGK0mSpE4MVpIkSZ0YrCRJkjoxWEmSJHVisJIkSerEYCVJktSJwUqSJKkTg5UkSVInBitJ\nkqRODFaSJEmdGKwkSZI6MVhJkiR1YrCSJEnqxGAlSZLUicFKkiSpE4OVJElSJwYrSZKkTgxWkiRJ\nnRisJEmSOjFYSZIkdWKwkiRJ6sRgJUmS1InBSpIkqRODlSRJUicGK0mSpE4MVpIkSZ0YrCRJkjox\nWEmSJHVisJIkSerEYCVJktSJwUqSJKmTOQWrJLslOTfJqiQHXUu5uye5PMk/9KuiJEnS4rDOYJVk\nGXAYsDuwM7Bnkp2vodzrgP/oXUlJkqTFYC49VvcAVlXVeVV1GXAcsMdayu0PHA9c1LF+kiRJi8Zc\ngtXWwPljry8Y3rtSkq2BRwKH96uaJEnS4tJr8Pq/AQdW1RXXVijJvklOT3L66tWrO31rSZKk2bDh\nHMr8CNh27PU2w3vjVgDHJQG4OfCQJJdX1cfGC1XVkcCRACtWrKjrWmlJkqRZNJdgdRqwY5IdaIHq\nccDjxwtU1Q6jr5McA5w4GaokSZLWd+sMVlV1eZJnAycDy4Cjq+rsJPsNx4+Y5zpKkiQtCnPpsaKq\nVgIrJ95ba6Cqqif/+dWSJElafFx5XZIkqRODlSRJUicGK0mSpE4MVpIkSZ0YrCRJkjoxWEmSJHVi\nsJIkSerEYCVJktSJwUqSJKkTg5UkSVInBitJkqRODFaSJEmdGKwkSZI6MVhJkiR1YrCSJEnqxGAl\nSZLUicFKkiSpE4OVJElSJwYrSZKkTgxWkiRJnRisJEmSOjFYSZIkdWKwkiRJ6sRgJUmS1InBSpIk\nqRODlSRJUicGK0mSpE4MVpIkSZ0YrCRJkjoxWEmSJHVisJIkSerEYCVJktSJwUqSJKkTg5UkSVIn\nBitJkqRODFaSJEmdGKwkSZI6MVhJkiR1YrCSJEnqxGAlSZLUicFKkiSpE4OVJElSJwYrSZKkTgxW\nkiRJnRisJEmSOjFYSZIkdWKwkiRJ6sRgJUmS1InBSpIkqRODlSRJUicGK0mSpE4MVpIkSZ0YrCRJ\nkjqZU7BKsluSc5OsSnLQWo4/IcmZSc5K8sUku/SvqiRJ0mxbZ7BKsgw4DNgd2BnYM8nOE8W+B9yv\nqu4IvBY4sndFJUmSZt1ceqzuAayqqvOq6jLgOGCP8QJV9cWq+sXw8svANn2rKUmSNPvmEqy2Bs4f\ne33B8N41eSrwyT+nUpIkSYvRhj3/sSR/SwtW972G4/sC+wJst912Pb+1JEnS1M2lx+pHwLZjr7cZ\n3ruKJHcCjgL2qKqL1/YPVdWRVbWiqlYsX778utRXkiRpZs0lWJ0G7JhkhyQbAY8DThgvkGQ74CPA\nXlX17f7VlCRJmn3rfBRYVZcneTZwMrAMOLqqzk6y33D8COAVwObA25IAXF5VK+av2pIkSbNnTmOs\nqmolsHLivSPGvv4n4J/6Vk2SJGlxceV1SZKkTgxWkiRJnRisJEmSOjFYSZIkdWKwkiRJ6sRgJUmS\n1InBSpIkqRODlSRJUicGK0mSpE4MVpIkSZ0YrCRJkjoxWEmSJHVisJIkSerEYCVJktSJwUqSJKkT\ng5UkSVInBitJkqRODFaSJEmdGKwkSZI6MVhJkiR1YrCSJEnqxGAlSZLUicFKkiSpE4OVJElSJwYr\nSZKkTgxWkiRJnRisJEmSOjFYSZIkdWKwkiRJ6sRgJUmS1InBSpIkqRODlSRJUicGK0mSpE4MVpIk\nSZ0YrCRJkjoxWEmSJHVisJIkSerEYCVJktSJwUqSJKkTg5UkSVInBitJkqRODFaSJEmdGKwkSZI6\nMVhJkiR1YrCSJEnqxGAlSZLUicFKkiSpE4OVJElSJwYrSZKkTgxWkiRJnRisJEmSOjFYSZIkdWKw\nkiRJ6sRgJUmS1MmcglWS3ZKcm2RVkoPWcjxJ3jwcPzPJXftXVZIkabatM1glWQYcBuwO7AzsmWTn\niWK7AzsO/+0LHN65npIkSTNvwzmUuQewqqrOA0hyHLAH8K2xMnsA76mqAr6c5KZJtqyqn3SvsSTp\nGm1/0EnTrsJ65fsHP3TaVdAiM5dHgVsD54+9vmB4708tI0mStF6bS49VN0n2pT0qBLg0ybkL+f1n\n3M2Bn027EuuS1027BlO3KNppibONFodF0U6e8xZHOy2QW82l0FyC1Y+AbcdebzO896eWoaqOBI6c\nS8WWmiSnV9WKaddD1852mn220eJgOy0OttOfbi6PAk8DdkyyQ5KNgMcBJ0yUOQHYe5gdeC/gEsdX\nSZKkpWadPVZVdXmSZwMnA8uAo6vq7CT7DcePAFYCDwFWAb8F9pm/KkuSJM2mOY2xqqqVtPA0/t4R\nY18X8Ky+VVtyfES6ONhOs882Whxsp8XBdvoTpWUiSZIk/bnc0kaSJKkTg5UkXYskmXYd9Kez3TQt\nBqslxpPNbEmyIsmdp10PXV2Sv0+yUTleYlGy3TQtBqv1XJKXJ3lMkrtAO9kYrmZDku2B+wNHJXl2\nkjtNtUK6UpKbAQ8G3pdkb9tm8Ujy6CTvSHKrJJtPuz5au+Hv6m+mXY/54OD19VySB9H2e9wOuKyq\n9h/ej3d005NkN+BLVXVJktsC+wFXAGdX1bumW7ulLclDq+qk4es9gFvTQtYhVfXpqVZO65RkU+B5\nwKbA9YH3VtWXp1srTUqyF/BE4BTgx1X17unWqB+D1XoqyWuBo6rqB0luBNwA+BjtF/ixQxnD1RQk\n2ZEWpC4C3l1VPx16SB4C3Av4XFUdP806LlVJlgMfBD5fVa8a3rsx8HDgucDzq+oL06uhrkmS/YEt\nqurlw+udgPsB/wS8qKpOmWL1NBja6YNVdWGS2wB3BB4P/KCqXjjd2vXho8D1V4ATk2xdVb+pqour\n6q+B5UmOA8cgTEtVfQf4KLAxsOPw3s9pOxh8C7hTkutNr4ZLV1WtBp4K7JLkb4f3Lq2qY4HDgScm\n2WqaddQ1+gxwxyQvA6iqc4dt1A4DXprkDlOtnUiyAbA18I4kt6iq79LOey8Bdk7yiqlWsBOD1Xom\nyTKAqnoZ8AHgkcP7Gw7vPwDYLMnzplbJJWrUNgBVdSpwWFX93+FYqupXtF7FewP/MJ1aLk0TbXMe\n8JSq+txEsVOAi2mb0joRZEaM2qGqzgFeBHxvosiHaTuH/NV4eS28qroCeDlwOvDA0XtVtQp4GbBN\nkntPsYpd+ChwPZJkWVX9cfS/13L87sCDq+qfp1DNJWnsZ78BsBdwOa07/H+SbFBVV4wezSa5D/Ak\n2uOLS6Za8SVgom3+HvglbfzbZWsp+wLgrlX1+IWup65u/FyX5EZV9ZuxYxsMF3KSPBh4MvB4e+oX\n3jVdkybKbAI8G/hJVR2zIBWbJ/ZYrSeGk8jo4vDWJK9M8rQktxyVGfvFvgi4c5LbTaWyS9BY23wC\nuBXwGODzw3T+K4YyoxP+WcBJwKVTqewSM9Y2pwD3BN4KvD7JTUZlxnpF3gB8cNiQXlM0fs5L8j7g\n0CQvTrIdXNk7wvD1ycAFwG5Tqu6SNdFOz0+yf5IdRk9RRqrq18CngX0W++N2g9V6YtTjQevyXgX8\nirZ/49V+QavqB7Ru14sWtJJL1Nijh2cCXwT+Bbgh8IHxXpGxi/evgc+s6w5Pf76xtnku8Kmqegnw\nB+CsYcbm6NH6+DIlJ6+tN0sLa+ycdyzwFeD9wIuBLcfLDaEZ4FDAiQcLbKydPgVcD7gP8F7gluPl\nhh7702lt+PMFr2hHBqv1y07AV6vqEGB34F1VdUaSrScHQ1fVfw8DpjVPxi/Kw1vnA7+jnWA+W1Vv\nGtrmGRPlqKrfLnR9l5K1tM0q4PpJTgWOq6p3Dj0fV/ZwjMpW1e8Wur5aYywoQev9/SFwIvB84NVV\n9V/D7E7gKj1XPxl/VKj5N3Yz8mjgtKo6GNic9jd2QZIbjMqO/S2eXlW/X+CqdmWwWsTGB9wOVtMe\n8X0POGG4cC+jnXC2XvAKLmHj43aGx7Lb0HoIXwB8q6peNxR9M7DD1Cq6BE20zauHv5HLaCHq08Pj\nPmiPBB84tYrqaoa2uyLNBsCPgY1oj85PqapDhvZ8R5IV4591bNXCuYYbl02TfJXW4/vWJFsAz1/L\nTf+i7w02WC1SE8+tT0pbCPRS4D+A/wLOHIr+O3Czqvr+lKq6JE2M27lhVV1QVV+iDUp/RJK3Jvk0\n8MuqehE4W2mhjLXNScAtquqPVfVJ2hIY2yd5f5JPAL+oqudOtbK60sQ40k8Bzxwuwt+j9Vp9behl\nfD9w8fBYSQts4sbl2LT1xH4DbAt8CXjbUPQwYPuq+p9p1XW+OCtwERsuxCuBr1fVi4f3/pI2rfhp\ntEdPl1TVPqPy3rXNr/GfcZJ9gTtW1f5Jbk97PHs6ba2qnYGbVtUJQ9krZzBpfky0zXOAW1bVS5Js\nTWubz9J6dm8GbFRVHx7K2jYzJMlngVOr6pXD6+vRxi/eHrgF8MOqOmA45jlvCobw+xngG6ObkyT3\nBZ5BWxF/I9qj2ScPx9ardjJYLTK56vTinWgD/V5NG2vwROBCWq/V14CNq+rCoawXhwWU5Pq0rYTe\nDZwG/IK2Zc0/Ag+qqq+NlV2vTiqLQZKn0MLUpcN/K2h31XtWWyR0VM62mbJcddmEzYC3AE8HHgHs\nCtwWeM4wnvRmo7GjnvMW1kQ7PQh4WFUdkOSuwBNovVVnAgXcfOjBXy/bacN1F9EsGbpYA+zCml/S\nDwH/FzgH2ALYoao+T5sZOLo4rFe/uLMsyYtoY3beDhxAu4v+YFX9KslNaTNjruSFe+GkrUN1w6p6\nzdAWvwSOH2YArqRNALkyWNk20zc2q+yxVXXccNPyGdqQh/cADwIeC5wxFqo85y2wsXbaijb2bdck\nJwLfp820PRB4YbWthb4D6287GawWpwcDx9NW6H46sFNVnQWQ5ANMrH/kxWF+reWO60La49gn0pZU\n+FWSmyQ5Hvh9VX1lKhVdgpJsWFWXD19fHzgXeHiSA6rqjcP7WyY5jDYu59QpVlcTxnoM7w78Y5Ib\nV9VjkmxZVT8ZyuxNu3hfyXPe1OwF7AHsM/x3M+A/qy2E/CEmllhYX9vJweuLwMT0YqrqU7Q1qj4C\n3KuqzkpyqyQfBP5QVa+dRj2XqrFZSk8YXr+bttDd3WgXg1vQLgyrq+oJ4ED1hVJVlw+DaO9cVX+g\njaP6EHC7JAcOxR4G/Kqq9gLbZhaM2mDswnsOrQf43kleUFU/SXLToUckVfWa8c9pYeTqM9PPAr4O\nHEKb/HEysGWSD9OuTcctdB2nwWC1CIxduA9IcqvhvWOAVwHvTNui5mLaNNa94ephTP1NnMTvBDx1\ndLGuqg/Rxla9CNgT+K+q2m/43Abr653arJhom92AM5Lcr9r6YF8a/vv7JAdW1Tuq6pnD52ybGZJk\nZZL7V1s091TaLOe7JnlGVf0SOGZsco5tt8BGQ1OSPHd4/TXa7NrzgAPTZmneBPhuVT0Rlsa1ab3/\nP7iYTfwCXp/2CPBFSbYdHnG8DzgD+BxtSYV3jj63Pj63niXDJILRDLMNq+obtKC7S5KDAIb2+BHt\nzu3Xo8/aNvNrvG0AqmolbQzOx8cu0ifRxoFcMva59XK8x2IyOueNtd/JwPFJ/qra4p6fB34L7J3k\nAc7cnI6Ja9MNgVcnORygqr5Ja6fbA28Evl9VB44+txTayVmBMypr1gIJ7Rf0B8DvgcNpG/geXFU/\nTPJa4IKqevsUq7ukjE4Ow8nlaGBj2hIKx9NWFX4+bQD0TWhTv583tcouMRNtcyhtaveXaT0df0Pb\n/uSZwONo29aMlilx9t+UTZzzHkC7IH83yWOAdwAPqaovJjmatuXTsVOt8BKVq85M36KqLkpyI4ae\n4Kp6+nDsPbQdJt49xepOhcFqhg0Xh+OBG9C2QvlmVb0iyduAG9FmBp5RVU8ZlV8KdwOzIsmxtB6p\n99Ie+X0beB/wP7SL9yXVtnDwwr3AkpxMG+9xNrAjsBz4X8DfAveldYoYqmbMcM77NK1XamPaeJ2D\naUHrYNog9dVV9dihvG23gEY/76Gd3gdsBnyXNt73/9GeoHwV2Ab4zljIWlLtZLCaYUneSHsE+Bzg\n1sAbaL+0r6WN6dl6eMyx5H5xpy3JzWmB6ulDz+EtaSsKf21y8oCBd2ENbXF4VT1yeP0XtMkeZ1TV\ne5Ncr4bVnm2b2ZK2b+YdqupZSe4B3AvYsqpenLZu3/Wr6syhrOe8KUlyDK1X/jDaenB3pp0PzwAe\nT7txGQ1NWXLt5BirGbKWGS2/pU1V/SPtruBltMeCm1TVN8ZClYM259nkgMuq+hltgOZDkmxeVT8F\nXgesbcNrL9zzKMnksjEXAcvT1hOjqlbR1s252/D6yi00bJvpWstA5s2ATQCqLUtyOnD3JLesqnMN\nVdOX5Ia03sR3V9sq7YO03vpHVtVvq+qoifG+S66dDFYzZC2/gOcCr0+y3XAB+CHtkcZmE5/z4jCP\nkmxVa1YUPijJC4aTy0eAuwBPSrIzbYPlS2o93PtqVg0X3MuTbDjMmn3i0FavpoXclw9F78fYwp+a\nDbVmxvM2w1vHAtdLstdw/IvAH2n7zI1/bsldrKdl8oZ/mFn7Hdqsv1tW1cW0vRtvnWTTibJL8trk\no8AZkOTRVXX88PUJtPFU/wF8ANgbeCGtt+oxtP2VnjGtui41SfYBbgccA/xvWi/iH2m9H3cD7knb\nWuPWwPdqzb5Y3lHPsySPBh4K/Autt/D3tEfkHwU+Ruv5eDltO6Hf1Hq6L9lilGRPWm/8hWkbXi+n\nrTH2TobFQGm7F9yA1nZPnFpll7Akd6+q04avD6Xt8fcGWo/VP9DG+b6Odo0634k6jcFqypJsCXyB\nFqL+wJoZZqO1qd4I/B2wPW18wb8On/PiMM+SLKe1xyto2wPdoNasd/R22sr3966q3yTZrKp+MRxz\n3M48Gx633gp4CrAlbUX7ZyTZgvY3swo4grYK/iZVNdreybaZsqGNXk27gbwRcAFtT83DgC/SJuz8\nGngILVQdN3zOc94CSnI/2vjeY4G/p92gbEzbNu1NwE+BRwE70JaUGS2psOTbyWA1RUnuXVVfSnJ7\n2kq1W1TViuHYA2mB6lLgHTVs3zAc8+Iwz4af/6No7fI74DW0XqkDx+7gjqAN1LwF7cJenlTm39BT\n9TRa+9wS2J+2xtsjq+rcJFvR7qovAV5aV90/zraZoiT70vb324a2/cluwP5V9ZUk29Nm/q0C3lZV\nPx77nOe8BZTk9lV1TpKn0TYov2VV7TEcO4jWM3xEVX1h/O/KdmocYzUlaSuonzyMCTkHeB5twO2h\nAFX1GdrjwK1pj5uu5C/ugjgf+G9gX9pGvQcD3wMekOQOANVWUn9OVf1udGLxwr0gTqLNPnoXbX23\no4ATgKclue1wQX4RbZ2qn48+ZNtM19DLeBdae51PW1vsW8Azktx6GAj9Qtr57j7jn/Wct3CS3BH4\nQJJdq+odtJnoOyV5EkC1JWTOBF6VZKexUOUCuwN7rKZgbC2Qu9H2LXtkVX0jye2AI4EvVNXLhrJ/\nWVVnT7O+S1WS2wDLq+rLw+vb0wLw94GTqurrY2W9U1tASW4A/HVVfXp4fTfg4bRFWY+sqm+NlbWn\nakYk2ZjW+/ulqvrI0G57ADcG3lpV5yW52Xgg1sJLsjtwIG2M76XAo4G70sbFfWgo87CqOnF6tZxd\nBqspyZoVorea6PLeibbK8Deqav+x9704TFHWrAp9O9pg6WOq6oRp10trDBfpvYDTqur9066P1i5t\nC6jLx17fnRaKt6Y9al89vO85b4qS7FhV3xm+vgVts/J7AqdW1XvGytlOEwxWM2K8x2N41PSIqvrn\nKVdLY8Z6GrccH/Om2ZFk++GRkmZcrro1yl/RFgL98JSrpQljnQC3oG0F9euqOnra9ZplBqsFcl1T\nvXcDs8u2WRhz+TlPlrFtFi/bbnYl2XQ0w1bXzMHrC2To6Vh2XT43H/VRk+RhSe57XT5r28yvJE9J\ncufRbMtrKzvZFrbNdKVtK7S23STWybZbeHNtp7FlS/7kdl1KDFbzLG2l7vcC1Jqd2zUDktyUtiHv\n/ZLc81rK2WYLbGibzYHnJLnj+MyjiXK2zYxJsivw1iS3u6aQZLtNX5InJ3kW/Olh1vB77QxW8+8Q\nYLskb4Mre64mLw6Te51png3jO34JvAUIbeG78eNbJ3lM2oa9nkQW2NA2/w6cCvzF2PuV5DZJ9hi9\nnlIVdc3OBT4PvChts/IrDYHZdpuyYemLC4G7JHnKtZQzI1wH/tDmyeixX7V943albSY6fvxWSU4c\nBgZe7h2mA7i5AAAL90lEQVTcwhmb4bcM2BA4uKpOmSh2L+BvgK0Wun5LWZI7pa34DG0W0qlV9dGJ\nYrcGnpJkl4Wtna7N6CJcVRcAHwaOqrZZ+ej4TYBDkjxySlXUYLgunUK7efnF+LEk2yZ5letSXXcG\nq3kwduHeIMkTqury0SyK0cDMqvoBbZuUt4B3cAtl+Pn/cbgIHA88dmLq9+hv4kTavlgPnkI1l6Th\nZ/974HlJzqXtRPDttRT9Cm3Rwo3XckxTMJzzrhjOeZsAq6ttoDz+2O8y4HPAzvbST8f4ON9qCxt/\ndi03LtenLX1hAL6ODFbzYOzCvZK2n9z4sRq6YQEOAs5MWzRPC2AswL4fOL2qXg9Xrh/GcHFYVlV/\noC0GetJ0arq0jC7MtBW5NwGW0VZVv5qqugT4T9pK+Jqy8RtJWrscCXxo9NiP4TpTVb8Dvkm7cNtD\nv8DGbyqTHDn0Sj1y/Pjw5feA04AbTqWi6wGXW+hsbK2jQ4DrVdVzkmwK3B84r6q+OVZ2I2Cjqrp0\nStVdMsancCfZnLZf2VtoG4rem7ZA4ZFV9eqhjCupL5CJC/Mdgd/S9l98LnBKVb1lKLfN8JhJMyjJ\n82l7N74OeDntcfqDq+qXw1jF/xnK3dhz3vSkbSC/MfBZ4MXAv1bVu4djo7/FzWjXpgunWNVFyx6r\nzsZ6RL4DXJTkWOBfgdfS9jLbfKzsZZ5g5t9wshjfJPRiWrB6O3Bb4L3Ag4Dt07ZKcW+yBTLxaPZT\nwDOqrfb8FeCdwK5JnjtcDB4wzbrqqpI8PMm2w9eHAI8ATqyqn1XVAcAXafuh3mwUqgA85y2stB0J\nRl+/gXaZ2nsIU08HXprkybBm5npV/cJQdd0ZrDpL8oGhe/UE4H+Ar9H2XHogcDtaD4kWyMR4t7cB\nhybZvao+ANyh2p6MXwFeAFxaVb+faoWXkCQ7jN2IvB04o6r2G3py70b723k9bRIBNbaNhqYryXbA\nsqo6f3jk9zbannJ/O4yxoqqeC5wDPGd6NV3ahqUvbj321gbAXZPcK21roc/TwtVbkvw1ON63Bx8F\n/pkmHxkleRzwz8Czqurk4b2NgKOB31bVvtOp6dI1DNj8OG3gbAHPAt4EfIw25fhY2jYNTx7Ku/Lz\nPEvyQGCnqjpseP3PwCrgzrQxOA+nzSp7Za663ZNtM0OS7EMLwf+bNibn7bSxpe8cxsJpipJsVFWX\nJdkP+FlVfTjJvwDb0zbD/vYwrvT2VXXOVCu7HrHH6s80/FImyVbDSf84Wu/H4UkePhR7HnDJKFS5\ntML8S/LYJFsPLx9KWw/prcBDaGMLdgUeQ+tBfM1YqNrAC/f8GsbYfIb2N/KKJHelTRK4N3Ax8Crg\nfsDdk2xnqJodufruEWfQxsQ9C/gNsB+wO3DA+KQcz3kLK2uW+7lsGN5wU1pv4u5V9RLgh7S/s52H\nv6tzhs+ZCTrwh/hnGDtZHAgcBuww/JJ+jDau6i1JHgy8saqeNXzGC/c8Gx5F7As8N8nNq+oE4I3A\nC4H/HALu2cATgdtW1TeGz7luyzxLW9hzZZIthp91Aa8EfllVT6+q19LC1cuAH1TVD0ef9e9muiYe\nq780yVNp7fdG2tIkz6Q9Dtwf+MMwCxCw7RbSRDvdA9ihqg4Gvg48bAhXBwG/BO433jae//owWF0H\nY3cDNfzvwcCPgFfQulgBPg2cCexWVZcNn/PCPc+G2Ue/Bv6R9oji1Uk2G9rgesCdhqJbAMdV1WdH\nn/XkP/+q6uO0v42jhjvpI4DPAK/Pmm2FDqFdmJ8B9nbMirFJBh+nPa7dkjbh4BfAccN7LwV+UlWv\nA9tuGsba6STg0cA7k7wKOIY25u2hSR5VVfuNHsWrL8dY/Ykmpoa/kjb778tVtSrJW4CbAB8Fngx8\nsqqOmF5tl5bRWJxhTNu7gF8Dj6KNpToIuDFtkO2WwDer6knD53zENM+S7AXcp6qeMdyY/HUNq92n\nbXvyWNpirK8AflRVq4djLnsxZWl7NZ41fP0IYCfaGMUTgJOq6k3Dsb8E7jgMh9AUJTkUuIA25u0/\ngPdW1eFJrk+bTHDR2BILnv86M1hdB8OF4dO06cTbApsBb6uqTyV5Hm3V2htX1dOH8v7izrMkm1TV\nr4e2eTFtxt/jkmxBO7mspj2y/Q2wS1WdNnzOtlkAQ3h6Nu2x+K8mf+7D8X2B31fVG4f3bJspS/JQ\n2jIKH6mqTyZ5EPB42nnvE1X1puHR+zOBN41m1dp2C2stf0/7A/8NPB/4XFW9LsmOtBv/M7xZmV8+\nCpyjtOmpy4eXzwFWDlP1bwtcAuyZ5IFV9caqev5YqHJM1TwbJgmMxu38Efg+cGGSjavqItqA2kfT\nHjFtZKiaij8Cf0m7KF9tM/Jqe8q9eRSqRmUWvJaa9DXgu7T1xB5A21x5R2DjUU8V7RHTVjW2VIlt\nt3Aytk7fmEuBk4HPjh7L0pYuud/4ZJAFrOaS4n5NczCM/XgZcGySj9IWlPx9knfRxum8KcmptMHS\nv6uq/zd8zjFVC6CqTkjbkPfoJI+hLaFwC+B2Sb5VVRcm+RTwvRpbnNCT/8Kpql+kLalwYpJfVdWx\n4+GqmkvBwDsLRo9gq+rHSY4C/gnYg7aX46OATyT5CHAj4PvVFgS17aZgbGjKJ2k986fRhkIsB/ZJ\ncjGtzVZX1SFjn7Od5omPAucobX2qXWl3bB+nPVL6KPC6qvpikg/QxlQdM71aLi3rGLdzEG1NpEuA\nvwDOrqrnDMc8+U9J2vpVbwZeP/634liq2ZFky6r6yRB630cbl3gebdzoFrSB6l+lbT90w7EbSdtw\nAY3G+w5fPxPYjnZ9Gu1Q8Bra8jKbA9evqkOHsrbTPDNYXYskt66q88Zevw/YBTgY+HfgqbStar4F\n/LSqHj+U88K9AOYwbue+tPFvW48mEdg20ze0y3uBQ4FVVbVyylXSYBic/mjgX6rqnCTHAK+qqu8n\nuSXwJGAr4NNVdeLY5/y7WkBjE3U2AP6Bthft4VV1VpJ70XoXNwQOraofT35uKpVeQhxjdQ2S/B3w\n/rR1QEhbuXZb4Chaz9Wew9cPoQ3aNFQtvLWO2xlONlTVqVX1ibFQ5Xi3GVBVpwJ/B1wOPCDDPmWa\nriQ3pC34eR7wzGGW34XAZQBV9VPWzLbdZPyz/l0tnLStaK4YehRPBB5Jm1F7AEBVfZk2E3pjWuC6\nkqFqYdhjtRZJbkx71HcAcB/atNW7AHtV1QVJngDsBnwBeFdVXT58zruBBTaMrToROLCqjh3eM9wu\nIkk2rapfTbseS1nawq3PpV2kN6aNqdoWuCdtBvQZtAUll9OWWFg9paouaWOznzcEXkJb/HOfJJvT\nBqt/rqpeOJS9TVV9d5r1XarssZownGBOAm5WVf8GfBnYm9YrdcFw0X4/bd+5ZaNQBd4NTEO1VdP3\nAV6WNTu0F7g9w2JhqJq+agu3fo7WI7Ua+AStt+p3tJnPm9NC1i61Zo0xZ5UtoPHZz7Q1+bYE7pHk\nzlV1MW3rrl2TvANgFKpsp4Vnj9VaJHkFbdXux9I2Fn0CrUv1DVX1pSlWTdfAcTvSn24dE0DuQLtY\nbwP823jvh73C05Hk5cAK2riqHWjbcm0KHF1VZybZEnhmVb18itVc8gxWg3WcYMZXhn79MEZEMyZt\nAbwHAbemrax+zHRrJM22OUwA2QXYCzizqt4zrXouZWu5Nv1NVX1uOLYL8HDabM1jquqrY58z/E6J\n61it8UngNqPxHkk+PzpQVT8bllPYBLgHYLCaQVX1HdoWQyTZdMrVkRaD8QkgR4wmgIwuyFX1jSQ/\nr6rzp1rLpW3y2nTK6MDQPkULvzvTlsEYHTNUTYk9VoMkmwFH0laqXevU/CQ3rrEFJiVpsbumCSBw\n1YuzPSDTMcdr07aG39lhsBrjCUbSUuTCrbPNa9Pi4qPAMUO36j7Am5NsVFXHjM8wG51g/MWVtD6p\nqs8k2Rd47/AYfVVVrTRUzQavTYuLPVZr4QwzSUuRE0Bmm9emxcFgdQ08wUhayly4dTZ5bZp9Bqs5\n8AQjSZo1Xptmk8FKkiSpE7f8kCRJ6sRgJUmS1InBSpIkqRODlSRJUicGK0mSpE4MVpIkSZ38f75y\nowMcxC0gAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe45553e7b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_ibm2_alpha(4, \"house\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Decoding with Model 2\n",
    "\n",
    "Ideally **marginalise out alignments**\n",
    "\n",
    "$$\n",
    "\\argmax_{\\target} p_\\params^\\text{IBM2}(\\source|\\target) \\prob^\\text{LM}(\\target)\n",
    "$$\n",
    "\n",
    "with \n",
    "$$\n",
    "  p_\\params^\\text{IBM2}(\\source|\\target) =  \\sum_{\\aligns}  p_\\params^\\text{IBM2}(\\source,\\aligns|\\target) \n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Computationally **very expensive**, instead find both optimal alignment and translation:\n",
    "\n",
    "$$\n",
    "\\argmax_{\\target,\\aligns} p_\\params^\\text{IBM2}(\\source,\\aligns|\\target) \\prob^\\text{LM}(\\target)\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Choices:\n",
    "\n",
    "* Simple **approximate** method\n",
    "* Complex **exact** method\n",
    "\n",
    "Both presentend in [these slides](https://www.dropbox.com/s/p495n19h5rtk3uf/IBM-decoding.pdf?dl=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Beam Based Decoder\n",
    "* In each step chooses **best next source word to translate**\n",
    "* **Append a target word** based on source word\n",
    "* Maintains a list of top-$k$ hypotheses in a **beam**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "uni_lm = UniformLM({w for w in target_vocab if w != 'NULL'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def show_decode_history(begin, end, beam=4, lm=uni_lm, num_iterations=3):\n",
    "    ibm2_iterations = em_model2(ibm1, train_model_2, num_iterations)\n",
    "    ibm2 = ibm2_iterations[-1][1] # model 2 of last iteration\n",
    "    hist2 = decode_model_2(ibm2, lm, source, beam)\n",
    "    return mt.render_history(hist2[begin:end])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table align=\"left\" style=\"font-size:x-large;\"><tr><th>Target</th><th>Remaining</th><th>Len</th><th>Score</th></tr><tr><td style='padding:5px; text-align:left'>NULL</td> <td style='padding:5px; text-align:left'>groß ist ein Mann</td> <td style='padding:5px; text-align:left'>4</td> <td style='padding:5px; text-align:left'>0.00</td><tr><tr><td style='padding:5px; text-align:left'>NULL man</td> <td style='padding:5px; text-align:left'>groß ist _ Mann</td> <td style='padding:5px; text-align:left'>3</td> <td style='padding:5px; text-align:left'>-9.20</td><tr><tr><td style='padding:5px; text-align:left'>NULL a</td> <td style='padding:5px; text-align:left'>groß ist _ Mann</td> <td style='padding:5px; text-align:left'>3</td> <td style='padding:5px; text-align:left'>-2.49</td><tr><tr><td style='padding:5px; text-align:left'>NULL a tall</td> <td style='padding:5px; text-align:left'>groß ist _ _</td> <td style='padding:5px; text-align:left'>2</td> <td style='padding:5px; text-align:left'>-8.19</td><tr><tr><td style='padding:5px; text-align:left'>NULL a man</td> <td style='padding:5px; text-align:left'>groß ist _ _</td> <td style='padding:5px; text-align:left'>2</td> <td style='padding:5px; text-align:left'>-3.31</td><tr><tr><td style='padding:5px; text-align:left'>NULL a man NULL</td> <td style='padding:5px; text-align:left'>groß _ _ _</td> <td style='padding:5px; text-align:left'>1</td> <td style='padding:5px; text-align:left'>-7.09</td><tr><tr><td style='padding:5px; text-align:left'>NULL a man is</td> <td style='padding:5px; text-align:left'>groß _ _ _</td> <td style='padding:5px; text-align:left'>1</td> <td style='padding:5px; text-align:left'>-4.70</td><tr><tr><td style='padding:5px; text-align:left'>NULL a man is big</td> <td style='padding:5px; text-align:left'>_ _ _ _</td> <td style='padding:5px; text-align:left'>0</td> <td style='padding:5px; text-align:left'>-7.83</td><tr><tr><td style='padding:5px; text-align:left'>NULL a man is tall</td> <td style='padding:5px; text-align:left'>_ _ _ _</td> <td style='padding:5px; text-align:left'>0</td> <td style='padding:5px; text-align:left'>-5.52</td><tr></table>"
      ],
      "text/plain": [
       "<statnlpbook.util.Table at 0x7fe45498ada0>"
      ]
     },
     "execution_count": 230,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ngram_lm = LaplaceLM(NGramLM(lm_train, 3),0.1)\n",
    "show_decode_history(begin=0,end=5,beam=2, lm=ngram_lm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Summary \n",
    "\n",
    "* MT is an instance structured prediction recipe\n",
    "* The noisy channel is one modeling framework\n",
    "* word-based MT is foundation and blue print for more complex models\n",
    "* Training with EM\n",
    "* NLP Tricks: \n",
    "    * introducing latent alignment variables to simplify problem\n",
    "    * decoding with Beams"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Background Material\n",
    "* [Lecture notes on IBM Model 1 and 2](http://www.cs.columbia.edu/~mcollins/courses/nlp2011/notes/ibm12.pdf) of Mike Collins.  \n",
    "* Jurafsky & Martin, Speech and Language Processing: \n",
    "    * Chapter 26, Machine Translation.\n",
    "    * Chapter 6, EM Algorithm\n",
    "* Brown et al., [The Mathematics of Statistical Machine Translation: Parameter Estimation](http://www.aclweb.org/anthology/J93-2003)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}