{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"slideshow": {
"slide_type": "skip"
}
},
"outputs": [],
"source": [
"from traitlets.config.manager import BaseJSONConfigManager\n",
"import jupyter_core\n",
"#path = \"/home/damian/miniconda3/envs/rise_latest/etc/jupyter/nbconfig\"\n",
"path = \"/Users/i.oseledets/anaconda2/envs/teaching/etc/jupyter/nbconfig\"\n",
"cm = BaseJSONConfigManager(config_dir=path)\n",
"cm.update(\"livereveal\", {\n",
" \"theme\": \"sky\",\n",
" \"transition\": \"zoom\",\n",
" \"start_slideshow_at\": \"selected\",\n",
" \"scroll\": True\n",
"})"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Lecture 17: Tensors and tensor decompositions"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Recap of the previous lecture\n",
"- Matrix functions (matrix exponentials, matrix sign function, Schur-Parlett algorithm, ...)\n",
"- Matrix equations"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Today lecture \n",
"\n",
"- Tensor decompositions and applications\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Tensors\n",
"\n",
"By **tensor** we imply a **multidimensional array**:\n",
"\n",
"$$\n",
"A(i_1, \\dots, i_d), \\quad 1\\leq i_k\\leq n_k,\n",
"$$\n",
"\n",
"where $d$ is called dimension, $n_k$ - mode size.\n",
"This is standard definition in applied mathematics community. For details see [[1]](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.153.2059&rep=rep1&type=pdf), [[2]](http://arxiv.org/pdf/1302.7121.pdf), [[3]](http://epubs.siam.org/doi/abs/10.1137/090752286).\n",
"\n",
"* $d=2$ (matrices) $\\Rightarrow$ classic theory (SVD, LU, QR, $\\dots$)\n",
"\n",
"* $d\\geq 3$ (tensors) $\\Rightarrow$ under development. Generalization of standard matrix results is not **straightforward**.\n",
"\n",
"\n",
"Picture is taken from [this presentation](http://users.cecs.anu.edu.au/~koniusz/tensors-cvpr17/present/anandkumar_anima_tmcv2017.pdf)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## More formal definition\n",
"\n",
"Tensor is a multilinear form. When you fix the basis, you get a $d$-dimensional table."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Curse of dimensionality\n",
"\n",
"The problem with multidimensional data is that number of parameters grows exponentially with $d$:\n",
"\n",
"\n",
"$$\n",
" \\text{storage} = n^d.\n",
"$$\n",
"For instance, for $n=2$ and $d=500$\n",
"$$\n",
" n^d = 2^{500} \\gg 10^{83} - \\text{ number of atoms in the Universe}\n",
"$$\n",
"\n",
"Why do we care? It seems that we are living in the 3D World :)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Applications\n",
"\n",
"#### Quantum chemistry\n",
"\n",
"Stationary Schroedinger equation for system with $N_{el}$ electrons\n",
"\n",
"$$\n",
" \\hat H \\Psi = E \\Psi,\n",
"$$\n",
"\n",
"where\n",
"\n",
"$$\n",
"\\Psi = \\Psi(\\{{\\bf r_1},\\sigma_1\\},\\dots, \\{{\\bf r_{N_{el}}},\\sigma_{N_{el}}\\})\n",
"$$\n",
"\n",
"3$N_{el}$ spatial variables and $N_{el}$ spin variables. \n",
"\n",
"\n",
"* Drug and material design\n",
"* Predicting physical experiments"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"#### Uncertainty quantification\n",
"\n",
"Example: oil reservoir modeling. Model may depend on parameters $p_1,\\dots,p_d$ (like measured experimentally procity or temperature) with uncertainty\n",
"\n",
"$$\n",
"u = u(t,{\\bf r},\\,{p_1,\\dots,p_d})\n",
"$$\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"#### And many more\n",
"\n",
"* Signal processing\n",
"* Recommender systems\n",
"* Neural networks\n",
"* Language models\n",
"* Financial mathematics\n",
"* ..."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Working with many dimensions\n",
"\n",
"How to work with high-dimensional functions?\n",
"\n",
"- **Monte-Carlo**: class of methods based on random sampling. Convergence issues\n",
"- **Sparse grids**: special types of grids with small number of grid points. Strong regularity conditions\n",
"- **Best N-term approximation** : sparse expansions in certain basis.\n",
"- Promising approach based on tensor decompositions "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Tensor decompositions\n",
"\n",
"## 2D\n",
"\n",
"Skeleton decomposition:\n",
"$$\n",
"A = UV^T\n",
"$$\n",
"or elementwise:\n",
"$$\n",
"a_{ij} = \\sum_{\\alpha=1}^r u_{i\\alpha} v_{j\\alpha}\n",
"$$\n",
"leads us to the idea of **separation of variables.**\n",
"\n",
"**Properties:**\n",
"* Not unique: $A = U V^T = UBB^{-1}V^T = \\tilde U \\tilde V^T$\n",
"* Can be calculated in a stable way by **SVD**"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Canonical decomposition\n",
"\n",
"The most straightforward way to generize separation of variables to many dimensions is the **canonical decomposition**: (CP/CANDECOMP/PARAFAC)\n",
"\n",
"$$\n",
"a_{ijk} = \\sum_{\\alpha=1}^r u_{i\\alpha} v_{j\\alpha} w_{k\\alpha}\n",
"$$\n",
"\n",
"minimal possible $r$ is called the **canonical rank**. Matrices $U$, $V$ and $W$ are called **canonical factors**. This decomposition was proposed in 1927 by Hitchcock, [link](https://onlinelibrary.wiley.com/doi/abs/10.1002/sapm192761164)."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"**Properties**:\n",
"\n",
"* For a $d$ dimensional tensor memory is $nrd$\n",
"* Unique under mild conditions\n",
"* Set of tensors with rank$\\leq r$ is not closed (by contrast to matrices):
\n",
" $a_{ijk} = i+j+k$, $\\text{rank}(A) = 3$, but\n",
" \n",
" $$a^\\epsilon_{ijk} = \\frac{(1+\\epsilon i)(1+\\epsilon j)(1+\\epsilon k) - 1}{\\epsilon}\\to i+j+k=a_{ijk},\\quad \\epsilon\\to 0 $$\n",
" \n",
" and $\\text{rank}(A^{\\epsilon}) = 2$\n",
"* No stable algorithms to compute best rank-$r$ approximation"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Alternating Least Squares algorithm\n",
"\n",
"0. Intialize random $U,V,W$\n",
"1. fix $V,W$, solve least squares for $U$\n",
"2. fix $U,W$, solve least squares for $V$\n",
"3. fix $U,V$, solve least squares for $W$\n",
"4. go to 2."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### DNN compression ([Lebedev, et. al 2015](https://arxiv.org/pdf/1412.6553.pdf))\n",
"\n",
"- Convolution is represented as 4D tensor\n",
"- Apply CP decomposition to this tensor\n",
"- Apply one-by-one convolutions with smaller kernels given by CP decomposition\n",
"- Fine tune such model\n",
"\n",
""
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Using numpy backend.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"[[[ 0. 1.]\n",
" [ 2. 3.]\n",
" [ 4. 5.]\n",
" [ 6. 7.]]\n",
"\n",
" [[ 8. 9.]\n",
" [10. 11.]\n",
" [12. 13.]\n",
" [14. 15.]]\n",
"\n",
" [[16. 17.]\n",
" [18. 19.]\n",
" [20. 21.]\n",
" [22. 23.]]]\n"
]
},
{
"data": {
"text/plain": [
"Text(0,0.5,'Approximation error')"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8VPW9//HXJ4GwL0IWdghrWOMSd5FFRbQoVhGXLventmrVtlatRbvY2sVrFW17S1VUtLe3ahW14lZaZRWpNaCsAQxhC1vCvhMSPr8/ZuBOc5MwJJmcSeb9fDzOI8zJzJz3PB7G95zzPed7zN0RERGprqSgA4iISP2mIhERkRpRkYiISI2oSEREpEZUJCIiUiMqEhERqREViYiI1IiKREREakRFIiIiNdIo6AB1ITU11Xv06BF0DBGRemXBggXb3D3tRM9LiCLp0aMHubm5QccQEalXzGxdNM/ToS0REamRelkkZtbTzJ43s6lBZxERSXRxUyRmNsXMisxsabn1o81spZnlm9kEAHcvcPdbgkkqIiKR4qZIgBeB0ZErzCwZmARcBgwAbjCzAXUfTUREKhM3ReLuc4Ad5VafBeSH90BKgFeAsXUeTkREKhU3RVKJzsCGiMeFQGcza29mTwOnmdkDFb3QzG41s1wzyy0uLq6LrCIiCSneT/+1Cta5u28Hbq/qhe4+GZgMkJOTo9tAiojESLzvkRQCXSMedwE21dXGt+87zCPv5/GvNTsoLTtaV5sVEalX4n2P5FOgj5llAhuB64Eb62rjSzbuZspHa3hmdgFtmjVmeL80RmalM6xvGm2bp9RVDBGRuBY3RWJmLwPDgVQzKwQecvfnzewuYDqQDExx92V1lWl4v3QW/vgSPvpiGx+uKGLmiiLe+nwTSQY53dsxsn86F2Wl0zu9JWYVHYUTEWn4zL3hDx/k5OR4bUyRcvSos6hwFzNWFPFhXhHLN+8BoGu7ZlyUlcHIrHTO7tmOJo2Sa7wtEZGgmdkCd8854fNUJNW3efdBZqwoYkZeER/lb+Nw6VGapyQztE8qI7PSGdEvnfTWTWt9uyIidUFFEiFWRRLpYEkZ8wu2HS+WTbsPATCkSxtGZqVzUVYGAzu1JilJh8BEpH5QkUSoiyKJ5O6s2LI3fAhsK59t2IU7pLdqwsisdEZmpXNBn1Sap8TNEJWIyP+hIolQ10VS3vZ9h5m1spgZK4qYs6qYvYdLSWmUxLk92x8vlq7tmgeWT0SkIiqSCEEXSaSS0qPkrt1x/Cywgm37Aeib0ZKRWRlc1D+d07q2pVFyvF/iIyINnYokQjwVSXkFxftC4yorikIXPh512jZvzPC+aYzsn8GwPmm0ad446JgikoBUJBHiuUgi7Tl0hLmrtvHhiq3MWlnMjv0lJCcZOd1PCQ3Y90+nV5quWRGRuqEiiVBfiiRS2VHn8w27mLFiKzNWFJMXvmalW7vmx0vlrExdsyIisaMiiVAfi6S8jbsOMjN8CGxe+JqVFinJDO2Txsj+oWtW0lo1CTqmiDQgKpIIDaFIIh0sKePj1aFpW2bkFbFlT+ialeyubbkofBbYwE6tdQhMRGpERRKhoRVJJHdn+eY9zMgr4sMVRSwqDF2zktH62DUrGZzfu72uWRGRk6YiidCQi6S8bcevWdnKnFXb2Be+ZuW8Xu25KCudEVnpdDlF16yIyImpSCIkUpFEKik9yqdrd/BhXhEfrtjKuu0HAMjq0Or4gP2pXU8hWdO2iEgFVCQRErVIIrk7Bdv2hw+BbeXTtTspO+qc0rwxw/uFxlUu7JtGm2a6ZkVEQlQkEVQk/9fug0eY+0UxM/KKmLmyiJ0HjpCcZJzZ4xS+NKQTXz27mwbrRRJctEWiEdgE1aZZY8YM6cSYIZ3C16zs5MO8Ij7I28qP/7oUA756TvegY4pIPaAJnYTkJOOM7u24f3QW0+++kKF9UvnVe3msD4+piIhURUUi/8bMePSaISSbcd9rizh6tOEf+hSRmqmXRWJmPc3seTObGnSWhqhT22b85IoB/GvtDqbMWxN0HBGJc3VeJGY2xcyKzGxpufWjzWylmeWb2YSq3sPdC9z9ltgmTWzjzujCxf3TeWz6SvKL9gUdR0TiWBB7JC8CoyNXmFkyMAm4DBgA3GBmA8xssJm9U25Jr/vIicfM+NXVg2mWksy9ry2itOxo0JFEJE7VeZG4+xxgR7nVZwH54T2NEuAVYKy7L3H3MeWWomi2Y2a3mlmumeUWFxfX8qdIDOmtmvLzsYNYtGEXz8wpCDqOiMSpeBkj6QxsiHhcGF5XITNrb2ZPA6eZ2QMVPcfdJ7t7jrvnpKWl1W7aBHJFdie+NLgjv/lg1fGp7EVEIsVLkVR05Vulpwu5+3Z3v93de7n7IzHMJcDPrxpEm2aNuffVRZSU6hCXiPy7eCmSQqBrxOMuwKaAskg57Vqk8KsvD2b55j38fsYXQccRkTgTL0XyKdDHzDLNLAW4HpgWcCaJMGpgB64+vTOTZq1mceGuoOOISBwJ4vTfl4H5QD8zKzSzW9y9FLgLmA7kAa+6+7K6ziZVe+iKgaS1bMI9ry7i0JGyoOOISJwI4qytG9y9o7s3dvcu7v58eP177t43PO7xy7rOJSfWplljHh03hPyifTzxj1VBxxGROBEvh7aknhjWN40bz+7Gs3MLyF1b/ixuEUlEKhI5aQ9e3p/ObZtx72uLOFBSGnQcEQmYikROWssmjXj82mzWbT/Ao++vCDqOiARMRSLVck7P9tx0fg/+OH8d8/K3BR1HRAKkIpFqu//SLHqmtuD+qYvZe+hI0HFEJCAqEqm2ZinJPD4+m827D/KLd/KCjiMiAVGRSI2c3u0UbhvWi7/kbmDGiq1BxxGRAKhIpMbuvrgP/TJaMeH1Jew6UBJ0HBGpYyoSqbEmjZKZOD6bHftLeGiaJiQQSTQqEqkVgzq34dsj+/DW55t4f8nmoOOISB1SkUituWNELwZ3bsMP/7qUbfsOBx1HROqIikRqTePkJCaOz2bfoVJ+9OZS3Cu9pYyINCAqEqlVfTNacc+ovvxt2Rbe+ly3lBFJBCoSqXXfHNqT07u15SdvLWXrnkNBxxGRGFORSK1LTjImjj+VkrKj/OD1xTrEJdLAqUgkJjJTWzBhdBazVhbzl083BB1HRGJIRSIx8/Vze3Buz/b8/J3lbNhxIOg4IhIj9bJIzKy/mT1tZlPN7FtB55GKJSUZvx43BID7py7m6FEd4hJpiIK4Z/sUMysys6Xl1o82s5Vmlm9mE6p6D3fPc/fbgfFATizzSs10bdecH48ZwPyC7fzpn+uCjiMiMVBlkZhZspn9Ty1v80VgdPntAJOAy4ABwA1mNsDMBpvZO+WW9PBrrgQ+Aj6s5XxSy647syvD+6XxyPt5rNm2P+g4IlLLqiwSdy8D0swspbY26O5zgPI3+z4LyHf3AncvAV4Bxrr7EncfU24pCr/PNHc/D/hKbWWT2DAz/vPqIaQkJ3Hfa4so0yEukQalURTPWQvMM7NpwPGvk+7+RC3m6AxEntpTCJxd2ZPNbDhwNdAEeK+S59wK3ArQrVu32sop1dShTVMeHjuIu//yOc/NLeC2Yb2CjiQitSSaItkUXpKAVjHKYRWsq/Rrq7vPAmZV9YbuPhmYDJCTk6OvwHFg7KmdeH/pZib+fRUjstLpmxGr/5xEpC6dsEjc/WcAZtYq9ND3xSBHIdA14nEXQuUlDYiZ8csvD2bUk3O499VFvHHHeTROrpcnDopIhBP+FZvZIDP7DFgKLDOzBWY2sJZzfAr0MbPM8HjM9cC0Wt6GxIHUlk345VWDWLJxN3+YuTroOCJSC6L5OjgZuMfdu7t7d+Be4NnqbtDMXgbmA/3MrNDMbnH3UuAuYDqQB7zq7rpDUgN12eCOjD21E/814wuWbtwddBwRqSE70TxIZrbI3bNPtC6e5eTkeG5ubtAxJMKuAyWMenIOpzRPYdq3z6dJo+SgI4lIOWa2wN1PeK1eNHskBWb2YzPrEV5+BKypeURJZG2bp/Cf1wxm5da9/PaDL4KOIyI1EE2R3AykAW+El1TgpliGksQwMiuD63K68vTs1SxcvzPoOCJSTSe8sh140N2/4+6nh5e73V1/9VIrfjSmPx3bNOO+VxdxsKQs6DgiUg3RXNl+Rh1lkQTUqmljfj1uCAXb9vPY9JVBxxGRaojmgsTPwle1v8a/X9n+RsxSSUI5v3cqXz+3O1PmrWHUwAzO6dk+6EgichKiGSNpB2wHRgJXhJcxsQwliWfCZVl0b9+c709dxL7DpUHHEZGTEM0YyWJ3v6nccnMd5ZME0TylEROvzaZw50F+9V5e0HFE5CREM0ZyZR1lkQSX06Md3xzak5c+Wc/sVcVBxxGRKEVzaOtjM/u9mQ01s9OPLTFPJgnpnkv60ju9JT+YupjdB48EHUdEohBNkZwHDAQeBiaGl8djGUoSV9PGyTwxPpvifYd5+O3lQccRkShEM/vviLoIInLMkC5tuXN4L343I59LB2YwamCHoCOJSBWimf03w8yeN7P3w48HmNktsY8mieyukX0Y0LE1D765hB37S4KOIyJViObQ1ouEZuXtFH68Crg7VoFEAFIaJTFxfDa7Dx7hx28tDTqOiFQhmiJJdfdXgaMA4SnfNZeFxFz/jq25++K+vLt4M28v0n3OROJVNEWy38zaE771rZmdA+gmElInbruwJ9ld2/Ljt5ZStPdQ0HFEpALRFMk9hO5W2MvM5gH/DXw7pqlEwholJzHx2mwOlpTx4BtLONH9c0Sk7p2wSNx9ITCM0GnAtwED3X1xrIOJHNM7vSXfv7QfH+QV8frCjUHHEZFyotkjwd1L3X2Zuy91d10lJnXu5vMzOSuzHT+btoxNuw4GHUdEIkRVJPHGzIab2Vwze9rMhgedR2IvKcl4fFw2Ze784PXFOsQlEkfqvEjMbIqZFZnZ0nLrR5vZSjPLN7MJJ3gbB/YBTYHCWGWV+NKtfXMevLw/c7/Yxp8/WR90HBEJi+Z+JJhZZ6B75PPdfU41t/ki8HtCg/bH3j8ZmARcQqgYPg3fAyUZeKTc628G5rr7bDPLAJ4AvlLNLFLPfOXsbkxftoVfvZfHhX3S6Na+edCRRBLeCYvEzB4FrgOW87/XjzhQrSJx9zlm1qPc6rOAfHcvCG/zFWCsuz9C1fc+2Qk0qU4OqZ/MjEevGcKlT87hvtcW8cqt55CUZEHHEklo0eyRXAX0c/fDMczRGdgQ8bgQOLuyJ5vZ1cClQFtCezcVPedW4FaAbt261VpQCV6nts34yRUD+P7UxUyZt4ZvDO0ZdCSRhBbNGEkB0DjGOSr6SlnpaKq7v+Hut7n7de4+q5LnTHb3HHfPSUtLq62cEifGndGFi/un89j0leQX7Qs6jkhCi6ZIDgCfm9kzZva7Y0st5ygEukY87gJoTgyplJnxq6sH0ywlmXtfW0Rp2dGgI4kkrGiKZBrwc+BjYEHEUps+BfqYWaaZpQDXh7crUqn0Vk35+dhBLNqwi2fmFAQdRyRhRXM/kj+G/+feN7xqZU0uSjSzl4HhQKqZFQIPufvzZnYXoVmGk4Ep7r6sutuQxHFFdif+tmwLv/lgFSOz0unfsXXQkUQSjp3owq7wBX9/BNYSGsvoCvxHDU7/rXM5OTmem5sbdAyJkR37Sxj15GzSWjXlrTvPJ6VRvbzOViTumNkCd8850fOi+YubCIxy92HufiGhs6WerGlAkdrSrkUKj1w9hLzNe/j9jC+CjiOScKIpksbuvvLYA3dfRezP4hI5KZcMyODq0zszadZqFm3YFXQckYQSTZHkhm+1Ozy8PEvtD7aL1NhDVwwkrWUT7n1tEYeO6N5rInUlmiL5FrAM+A7wXUJXuN8ey1Ai1dGmWWN+PW4I+UX7eOIfq4KOI5Iwojlr6zCh+ayeiH0ckZq5sG8aN57djWfnFjBqQAY5PdoFHUmkwat0j8TMXg3/XGJmi8svdRdR5OQ8eHl/upzSjHtfW8SBktKg44g0eFUd2vpu+OcY4IoKFpG41LJJIx4bl8267Qd49P0VQccRafAqLRJ33xz+5x3uvi5yAe6om3gi1XNOz/bcfH4mf5y/jnn524KOI9KgRTPYfkkF6y6r7SAite3+0f3omdqC+6cuZu8h3SFaJFaqGiP5lpktAfqVGx9ZA2iMROJe08bJPD4+m827D/KLd/KCjiPSYFV11tZLwPuE7lAYeevbve6+I6apRGrJ6d1O4bZhvXhq1mouHZTByKyMoCOJNDhVjZHsdve17n5DeFzkIKF7hLQ0M90pSuqNuy/uQ7+MVkx4fQm7DpQEHUekwTnhGImZXWFmXwBrgNmEJm98P8a5RGpNk0bJTByfzY79JTw0TZNKi9S2aAbbfwGcA6xy90zgImBeTFOJ1LJBndvw7ZF9eOvzTby/ZPOJXyAiUYumSI64+3YgycyS3H0mcGqMc4nUujtG9GJw5zb88K9L2bbvcNBxRBqMaIpkl5m1BOYAfzaz3wK6XFjqncbJSUwcn82+Q6X88M0lnOhePCISnWiKZCyhgfbvAX8DVqMr26We6pvRintH9WX6sq289fmmoOOINAgnLBJ33+/uZUBz4G3gfwidvSVSL31jaE/O6H4KP3lrKVt2Hwo6jki9F81ZW7eZ2VZCFyHmEroXSaD3rTWzoWb2tJk9Z2YfB5lF6p/kJOPxa7MpKTvKhDcW6xCXSA1Fc2jrPmCgu/dw957ununuPau7QTObYmZFZra03PrRZrbSzPLNbEJlrwdw97nufjvwDqH7yYuclMzUFjxwWX9mrSzmL59uCDqOSL0WTZGsBg7U4jZfBEZHrjCzZGASoTm8BgA3mNkAMxtsZu+UW9IjXnoj8HItZpME8rVzunNuz/b8/J3lbNhRm/+JiySWaIrkAeBjM3vGzH53bKnuBt19DlB+ipWzgHx3L3D3EuAVYKy7L3H3MeWWIoDw1fW73X1PdbNIYktKMn49bghmxv1TF3P0qA5xiVRHNEXyDDAD+Ceh8ZFjS23qDEQeXygMr6vKLcALlf3SzG41s1wzyy0uLq6FiNIQdW3XnB99qT/zC7bzp3+uCzqOSL10wlvtAqXufk+Mc1gF66r8eujuD53g95OByQA5OTn6qimVuu7Mrvxt2RYeeT+PC/umkZnaIuhIIvVKNHskM8Pf7juaWbtjSy3nKAS6RjzuAugkf6kTZsaj1wwhJTmJ+15bRJkOcYmclGiK5EbC4yT872Gt2j7991Ogj5llmlkKcD0wrZa3IVKpjNZNeXjsIBas28lzcwuCjiNSr0RzQWJmBUtNTv99GZhP6IZZhWZ2i7uXAncB04E84FV31zStUqfGntqJSwdmMPHvq1i1dW/QcUTqDavsYiwzG+nuM8zs6op+7+5vxDRZLcrJyfHc3ECvoZR6Ytu+w4x6cg6d2zbjjTvOo3FyNDvtIg2TmS1w95wTPa+qv5Jh4Z9XVLCMqXFCkTiU2rIJv7xqEEs27uapWauDjiNSL1R61taxs6Lc/aa6iyMSvMsGd+TK7E787sMvGJmVzqDObYKOJBLXoplr609m1ibicXcz+zC2sUSC9fDYgZzSIoX7XlvE4dKyoOOIxLVoDgB/BHxiZpeb2TeBfwC/iW0skWC1bZ7Co9cMZsWWvfz2gy+CjiMS1054QaK7P2Nmy4CZwDbgNHffEvNkIgEbmZXB+JwuPD17NZcMyOC0bqcEHUkkLkVzaOtrwBTg64QmXHzPzLJjnEskLvxozAA6tG7Kva8t4tARHeISqUg0h7auAS5w95fd/QHgdkKFItLgtW7amF+Py6ageD+PTV8ZdByRuBTNBYlXHZtxN/z4X8DZMU0lEkcu6JPK187pzpR5a/ikYHvQcUTiTjSHtrqY2ZtmVmxmW83sdSD9RK8TaUgmXJZFt3bNuW/qIvYfLg06jkhciebQ1guE5r3qSGhq97epYvp2kYaoRZNGPH5tNoU7D/Kr9/KCjiMSV6IpkjR3f8HdS8PLi0BajHOJxJ0ze7TjGxdk8udP1jP3C93jRuSYaIpkm5l91cySw8tXAR0oloR076h+9Eprwf1TF7Pn0JGg44jEhWiK5GZgPLAF2AyMC68TSThNGyczcfypFO09zMNvLw86jkhcqLJIzCwZuMbdr3T3NHdPD5/FpXuSSsI6tWtbvjWsF1MXFPLB8q1BxxEJXJVF4u5lwNg6yiJSb3znoj5kdWjFhDeWsHN/SdBxRAIVzaGteWb2ezMbamanH1tinkwkjqU0SuKJ8aey+2AJP5mme7BJYjvhXFvAeeGfD0esc2Bk7ccRqT8GdGrNd0b2YeI/VjF6YAe+NKRj0JFEAhHNpI0j6iKISH30reG9+CBvKz/66xLOymxHWqsmQUcSqXPRXNne3sx+Z2YLzWyBmf3WzNrXRbgqMg0ws1fN7CkzGxdkFklsjZKTmDg+m/0lZTz45hIqu3W1SEMWzRjJK0Axockbx4X//ZfqbtDMpphZkZktLbd+tJmtNLN8M5twgre5DPgvd/8WoVmJRQLTO70V3x/Vj38s38qbn20MOo5InYumSNq5+8/dfU14+QXQtgbbfBEYHbkifJrxJEIFMQC4IbzXMdjM3im3pAN/Aq43s8eAQPeORABuviCTnO6n8NC0ZWzefTDoOCJ1KpoimWlm15tZUngZD7xb3Q26+xxgR7nVZwH57l7g7iWE9oLGuvsSdx9TbikKL3cCEwjdbEskUMlJxuPXZlNa5vzgdR3iksQSTZHcBrwElISXV4B7zGyvme2ppRydgQ0RjwvD6ypkZj3MbDLw38BjlTznVjPLNbPc4mLNiySx1yO1BQ9cnsWcVcW8/K8NJ36BSAMRzVlbreogh1W06cqe7O5rgVurekN3nwxMBsjJydHXQ6kTXz27O9OXbeGX7y5naJ9UurZrHnQkkZiLZo8EM7vazJ4ws4lmdlUMchQCXSMedwE2xWA7IjGVlGT8elw2ZsZ9ry3i6FF9h5GGL5rTf/9A6Pa6S4ClwO1mNqmWc3wK9DGzTDNLAa4ndA8UkXqnc9tm/GTMAD5Zs4M/zl8bdByRmItmj2QYcGn4niQvAJcDw6u7QTN7GZgP9DOzQjO7xd1LgbuA6UAe8Kq7a94JqbeuzenCiH5pPPq3FRQU7ws6jkhMRVMkK4FuEY+7Aouru0F3v8HdO7p7Y3fv4u7Ph9e/5+593b2Xu/+yuu8vEg/MjP+8ZghNGiVz32uLKNMhLmnAoimS9kCemc0ys1nAciDdzKaZmQ4/iVQio3VTHh47kIXrd/Hs3IKg44jETDSTNv4k5ilEGqgrszvx/pItPPH3VYzol06/DnVxEqRI3TrhHom7z45cgFJgfMRjEamEmfGLLw+iVdNG3Pva5xwpOxp0JJFaF+3pv6ea2a/NbC3wC0ID4iIShdSWTfjllwexdOMeJs3MDzqOSK2r9NCWmfUldBruDcB2QhM1mqaVFzl5owd15KpTO/H7Gflc3D+DQZ3bBB1JpNZUtUeyArgIuMLdL3D3/wLK6iaWSMPzsysH0a5FCve8+jmHS/WnJA1HVUVyDbCF0KSNz5rZRVQ8lYmIRKFN88Y8es0QVm3dx28++CLoOCK1ptIicfc33f06IAuYBXwPyAjfTGpUHeUTaVBGZKVzXU5Xnpm9moXrdwYdR6RWRHPW1n53/7O7jyE0B9bnhKZvF5Fq+NGY/nRs04z7Xl3EwRId4pL6L6qzto5x9x3u/oy7j4xVIJGGrlXTxjw2bggF2/bz6+krgo4jUmMnVSQiUjvO653K18/tzgvz1jJ/9fag44jUiIpEJCATLsuie/vmfH/qIvYdLg06jki1qUhEAtI8pRETr81m466D/Oo9XeMr9ZeKRCRAOT3a8c2hPXnpk/XMXqVbQkv9pCIRCdg9l/Sld3pLfjB1MbsPHgk6jshJU5GIBKxp42QmXptN8b7DPPz28qDjiJw0FYlIHMju2pY7hvfi9YWF/GP51qDjiJwUFYlInPj2yD7079iaB95Ywo79JUHHEYla3BeJmfU0s+fNbGpV60Tqu5RGSTwxPpvdB0v48VtLg44jErWYFomZTTGzIjNbWm79aDNbaWb5ZlbldCvuXuDut5xonUhD0L9ja+6+uC/vLt7M24s2BR1HJCqx3iN5ERgducLMkoFJwGXAAOAGMxtgZoPN7J1yS3qM84nEndsu7El217b8+K2lFO09FHQckROKaZG4+xxgR7nVZwH54b2KEuAVYKy7L3H3MeWWoupu28xuNbNcM8stLtb5+VJ/NEpOYuK12RwsKePBN5bi7kFHEqlSEGMknYENEY8Lw+sqZGbtzexp4DQze6CydeW5+2R3z3H3nLS0tFqMLxJ7vdNb8v1L+/FB3lZeX7gx6DgiVar0VrsxVNHNsSr9yuXu24HbT7ROpKG56fxMpi/bws/eXsZ5vdrTqW2zoCOJVCiIPZJCoGvE4y6ARhVFyklOMh6/NpvSMucHry/WIS6JW0EUyadAHzPLNLMU4HpgWgA5ROJe9/YtePDyLOZ+sY2X/rU+6DgiFYrpoS0zexkYDqSaWSHwkLs/b2Z3AdOBZGCKuy+LZQ6R+uwrZ3dn+rKt/PLdPIb2TqNb++ZBR6pXSsuO8u6SzTwzu4CDR8oY3i+NkVnpnJXZjiaNkoOO1yBYIuwu5+TkeG5ubtAxRKpt466DjH5yDv07teaVb55DUlJFQ40S6XBpGW8s3MjTs1ezbvsB+ma0pFPbZsxfvZ3DpUdpkZLMBX1SGZmVzoh+6aS3bhp05LhjZgvcPedEzwtisF1ETlLnts348RUDuH/qYl74eC23XJAZdKS4daCklJf/tYFn5xSwZc8hhnRpwzNfO4NL+meQlGQcLCnj49XbmLGiiBkripi+LDS32eDObRiRlc7IrHSGdG6jsj4J2iMRqSfcnW/8MZeP8rfx3neH0iutZdCR4srug0f40/y1TJm3lh37SzinZzvuHNGbC3qnYlZxKbg7K7bsPV4qn63fyVGH1JZNjh8CG9onlVZNG9fth4kT0e6RqEhE6pGiPYe45Mk5ZKa2YOrt59IoOe6ny4u5bfsOM+WjNfxp/jr2Hi5lRL807hrZmzOAdpXMAAANU0lEQVS6tzvp99q5v4TZq4r5cEURs1cWsedQKY2TjTN7tGNkeG+lZwIVuIokgopEGpK3Pt/Id1/5nPtH9+OO4b2DjhOYTbsOMnlOAa98up7DpUe5fFBH7hjRi4Gd2tTK+5eWHWXh+l18uGIrM1cUsWrrPgB6tG/OiKx0LsrK4KzMdqQ0arhlriKJoCKRhsTdufOlhXywvIhp3z6frA6tg45Up9Zu289Ts1bzxmeFuMNVp3XmW8N7xfxQ34YdB5i5MnQI7OPV2ymJGLC/KCuD4VlppLdqWAP2KpIIKhJpaLbvO8yoJ+fQoU1T/nrn+TROgENcK7bs4Q8zV/PO4k00Sk7i+jO7cuuFPelySt2fDn2gpJSP87czY2URM1cUsXl3aHLNwZ3bHD8ENrgBDNirSCKoSKQhmr5sC7f9aQHfvagP37ukb9BxYuaz9TuZNHM1H+RtpUVKMl89tzu3XJAZN9/+3Z28zXuP760sXL8TDw/YjwgP2F9QTwfsVSQRVCTSUH3vL58zbdEm/nrH+QzuUjtjA/HA3Zm/ejuTZuUzL387bZs35qbzMvmP87rTtnlK0PGqtGN/CbNXFTFjRfG/DdifldmOEf3q14C9iiSCikQaqt0HjjDqN7Np06wx0+66gKaN6/eV2u7Oh3lFTJqVz2frd5HWqgm3Du3JjWd3o0WT+nfZW2nZURas28mMlUXMyCvii6LQgH1magtG9Evnov7pnNkjfgfsVSQRVCTSkM1cWcRNL3zKbcN68sBl/YOOUy1lR533lmxm0sx8VmzZS5dTmnH7sF6MO6NLvS/HSMcG7D/MK2J+QWjAvmWTRlzQO5WR/dMZ3i++BuxVJBFUJNLQTXh9Ma/mbuC128+t1vUTQSkpPcpfP9vIU7NXs2bbfnqlteCO4b258tRODf4EgmMD9h+uCA3Yb9kTGrAf0uV/B+wHdQp2wF5FEkFFIg3d3kNHGP2buaQ0SuK97wylWUp8f4s/dKSMV/61nslzCti0+xCDOrfmzuG9uXRgh3p/plN1HBuwn7Fia+gK+w27cIe0VpED9mm0rOPDeyqSCCoSSQQfr97Gjc9+wv87rwc/vXJg0HEqtPfQEf70z3VM+WgN2/aVcGaPU7hzRG+G9U2rdBqTRLR932Fmrypmxooi5qwqPj5gf3Zm++PzgWWmtoh5DhVJBBWJJIqfTlvGix+v5aVvns15vVKDjnPcjv0lvDBvDS9+vJa9h0oZ1jeNO0f05qzM+nMYLihHwgP2M8PzgR0bsO+Z2uJ4qcRqwF5FEkFFIoniQEkpl/92LkfKnOnfu7DOD4WUt2X3IZ6dW8BLn6znUGkZowd24I7hvRvUqcp1bcOOA8cnmYwcsB/aJ5UR4Snx01o1qZVtqUgiqEgkkSxYt4Nrn57PdWd25ZGrhwSSYf32Azw1ezWvLyikzJ2xp3biW8N60SejVSB5GqoDJaXMy9/OjHID9tld2hyfD2xgp9bVHndSkURQkUiieeS9PJ6ZU8CLN53J8H7pdbbdVVv38tSs1UxbtIlkM67N6cLtw3rRtZ3u6hhr7s7yzXuOHwI7NmB/9WmdeeK6U6v1ng2mSMysJ/BDoI27jwuv6w98F0gFPnT3p6p6DxWJJJpDR8q44r8+Ys+hI/z97mG0aR7b6TkWF+5i0sx8pi/bSvOUZL5ydje+MbQnGbrrYGCODdh3aN2U83pXb7wsLorEzKYAY4Aidx8UsX408FtC92x/zt3/M4r3mnqsSCLWJQHPuvstVb1WRSKJaEnhbq76wzzGZneq9jfSqrg7n6zZwaSZ+cz9Yhutmzbi/52fyU3n9eCUFvE9jYlEJ15utfsi8Hvgv4+tMLNkYBJwCVAIfGpm0wiVyiPlXn+zuxdV9MZmdiUwIfz+IlLO4C5tuHNEb3734RdcOqgDlw7sUCvv6+7MWlnMpJn55K7bSWrLFCZclsVXzu5WLycmlJqLaZG4+xwz61Fu9VlAvrsXAJjZK8BYd3+E0N5LtO89DZhmZu8CL9VOYpGG5a4Rvflg+VZ++OYSzuzRjnY12FMoO+pMX7aFSTPzWbZpD53bNuPhsQMZn9O1QU1jIicviDkIOgMbIh4XhtdVyMzam9nTwGlm9kB43XAz+52ZPQO8V8nrbjWzXDPLLS4ursX4IvVHSqMknrgum90Hj/Cjvy6hOoeyj5QdZeqCQi55cjZ3/HkhB0vKeGzcEGbeN5yvn9tDJSIxP7RVkYrOQ6v0v2533w7cXm7dLGBWVRtx98nAZAiNkZxsSJGGIqtDa+6+uC+PTV/J24s3c2V2p6hed+hIGa/lbuDp2QVs3HWQ/h1b8/sbT+OyQR1JTsBpTKRyQRRJIdA14nEXYFMAOUQSxm0X9uTvy7fyk7eWck5mO9KrOJtq3+FS/vzPdTw7dw3b9h3m9G5t+flVAxnRL13TmEiFgiiST4E+ZpYJbASuB24MIIdIwmiUnMTEa7P50u/m8sAbS3juP3L+Tyns3F/Cix+v5cWP17L74BGG9knljuGncU7PdioQqVJMi8TMXgaGA6lmVgg85O7Pm9ldwHRCZ2pNcfdlscwhItA7vSXfv7Qfv3g3j6kLCrk2J3RgoGjPIZ77aA3/8891HCgpY9SADO4Y0ZtTu7YNOLHUF7E+a+uGSta/RyWD5CISOzefn8nfl2/l4beX0719C6Yt2siruYWUlh3liuxO3DG8N/06aBoTOTn1796VIlJtSUnG4+OyGf3bOYx/Zj6Nk41xZ4SmMenePvbTkkvDpCIRSTDd2jfnt9efxmfrd/K1c7vTsU2zoCNJPaciEUlAlwzI4JIBGUHHkAaiYd8UWUREYk5FIiIiNaIiERGRGlGRiIhIjahIRESkRlQkIiJSIyoSERGpERWJiIjUSEzv2R4vzKwYWFfNl6cC22oxTn2gz5wY9JkTQ00+c3d3TzvRkxKiSGrCzHLdPSfoHHVJnzkx6DMnhrr4zDq0JSIiNaIiERGRGlGRnNjkoAMEQJ85MegzJ4aYf2aNkYiISI1oj0RERGpERVIJM5tiZkVmtjToLHXFzLqa2UwzyzOzZWb23aAzxZqZNTWzf5nZovBn/lnQmeqCmSWb2Wdm9k7QWeqKma01syVm9rmZ5QadJ9bMrK2ZTTWzFeG/6XNjti0d2qqYmV0I7AP+290HBZ2nLphZR6Cjuy80s1bAAuAqd18ecLSYMTMDWrj7PjNrDHwEfNfd/xlwtJgys3uAHKC1u48JOk9dMLO1QI67J8R1JGb2R2Cuuz9nZilAc3ffFYttaY+kEu4+B9gRdI665O6b3X1h+N97gTygc7CpYstD9oUfNg4vDfrblZl1Ab4EPBd0FokNM2sNXAg8D+DuJbEqEVCRSCXMrAdwGvBJsEliL3yY53OgCPiHuzf0z/wb4H7gaNBB6pgDfzezBWZ2a9BhYqwnUAy8ED6E+ZyZtYjVxlQk8n+YWUvgdeBud98TdJ5Yc/cydz8V6AKcZWYN9lCmmY0Bitx9QdBZAnC+u58OXAbcGT583VA1Ak4HnnL304D9wIRYbUxFIv8mPE7wOvBnd38j6Dx1KbzrPwsYHXCUWDofuDI8XvAKMNLM/ifYSHXD3TeFfxYBbwJnBZsopgqBwoi966mEiiUmVCRyXHjg+Xkgz92fCDpPXTCzNDNrG/53M+BiYEWwqWLH3R9w9y7u3gO4Hpjh7l8NOFbMmVmL8AkkhA/xjAIa7BmZ7r4F2GBm/cKrLgJidtJMo1i9cX1nZi8Dw4FUMysEHnL354NNFXPnA18DloTHDAAedPf3AswUax2BP5pZMqEvVq+6e8KcEptAMoA3Q9+VaAS85O5/CzZSzH0b+HP4jK0C4KZYbUin/4qISI3o0JaIiNSIikRERGpERSIiIjWiIhERkRpRkYiISI2oSESqwcw6mNkrZrbazJab2Xtm1tfMepjZwfAMs8vN7Gkzq/W/MzP7qZndV9vvK1IdKhKRkxS+cPNNYJa793L3AcCDhK5VAFgdnnJlCDAAuKqq94pF0YjUJf0HLHLyRgBH3P3pYyvc/XN3nxv5JHcvBT4GekeuD++15JnZH4CFQFcze8rMcsvfEyV8D42fmdnC8L00ssqHMbNvmtn74SvzReqcikTk5A0idK+WKplZc0JTUyyp4Nf9CN3r5jR3Xwf80N1zCO3FDDOzIRHP3RaebPAp4N8OZ5nZXcAVhO4bc7Ban0akhjRFikjt6xWeYsaBt9z9/Qqes67czbPGh6c2b0Ro2pYBwOLw745NnrkAuDriNV8jNDnfVe5+pDY/gMjJUJGInLxlwLgqfn9sjKQq+4/9w8wyCe1pnOnuO83sRaBpxHMPh3+W8e9/s0uBY9Pfr4kuukjt06EtkZM3A2hiZt88tsLMzjSzYdV8v9aEimW3mWUQul9GND4DbgOmmVmnam5bpMZUJCInyUMznX4ZuCR8+u8y4KfApmq+3yJCpbAMmALMO4nXfkRob+ZdM0utzvZFakqz/4qISI1oj0RERGpERSIiIjWiIhERkRpRkYiISI2oSEREpEZUJCIiUiMqEhERqREViYiI1Mj/BwLSxm3duzlnAAAAAElFTkSuQmCC\n",
"text/plain": [
"