{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"2022-01-19-prod2vec-beh.ipynb","provenance":[{"file_id":"https://github.com/recohut/nbs/blob/main/raw/T061971%20%7C%20Behavioral%20testing%20and%20evaluation%20of%20the%20Prod2vec%20model%20on%20Coveo%20dataset.ipynb","timestamp":1644651036429}],"collapsed_sections":[],"toc_visible":true,"mount_file_id":"1uBu1EqYzY7kxrZSQvH6GvMNHs6tT_LT3","authorship_tag":"ABX9TyP6ReEOr8rqdFhNOgErwCM0"},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"}},"cells":[{"cell_type":"markdown","metadata":{"id":"3rq9HpuMhtIR"},"source":["# Behavioral testing and evaluation of the Prod2vec model on Coveo dataset"]},{"cell_type":"markdown","metadata":{"id":"XQ37oedJhL1c"},"source":["Metrics like Hit Rate provide only a partial picture of the expected behavior of recommenders in the wild: two models with very similar accuracy can have very different behavior on, say, the long-tail, or model A can be better than model B overall, but at the expense of providing disastrous performance on a set of inputs that are particularly important in production. Metrics such as coverage, serendipity, and bias have been therefore proposed to capture other aspects of the behaviors of RSs, but they still fall short of what is needed to debug RSs in production, or provide any guarantee that a model will be reliable when released.\n","\n","**Dataset** - This tutorial shows how easy it is to run behavioral tests on a target dataset, in this case a wrapper around a large e-commerce dataset (the [Coveo Data Challenge dataset](https://github.com/coveooss/SIGIR-ecom-data-challenge)).\n","\n","**Use case** - Complementary items. We are targeting a \"complementary items\" use cases, such as for example a cart recommender. *if a shopper added item X to the cart, what is she likely to add next?*\n","\n","**Model** - We train a simple, yet effective prod2vec baseline [model](https://arxiv.org/abs/2007.14906), re-using for convenience a \"training embedding\" function. Word2vec algorithm is used for embedding over product SKU sequences."]},{"cell_type":"markdown","metadata":{"id":"3dhtPqgRhPLC"},"source":["### Flow diagram\n","![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAmMAAAChCAYAAACLd7jrAAAgAElEQVR4nO3de1zUVf748RcgwnARZMYrhQgW6pa4hiuGaelmlmllpauobVmireU3Xe2Ku18pK13rp99KMa0tJVcrK80blZaKQLEkZggmhKh5CRSUYYDh8vvjMzNcHG4zA8PA+/l4fB4wn8/5nHNgDs7bc87nHBBCCCGEEEIIIYQQQgghhBBCCCGEEEIIIYQQQgghhBBCCCGEEEIIIYQQQgghhBBCCCGEEEIIIYQQQgghhBBCCCGEEEIIIYQQQgghhBBCCCGEEEIIIYQQHVcIcASoqueIaGZ+EcBaQGXjtEIIIYQQ7VII8IzhezWw2XAOlGCpucGYEKKdc7Z3BYQQoh3aX8/5DMCjNSsihBBCCNGR1e0ZM4pGGU7cYzj8Da+NQ5nRNdJG1nitNqS/s0b6yHrShgB7UXri9nDtEGkE1w6h1sxLCCGEEMLhmQvGIlECnz2G61B7rldEjWvGgCnacM0YgB0x5FlfWmPQVrOcyBplGAO1kBr5SiAmhJ10sncFhBCig4kDggzf5xu+JhgOgDzACdAYzk03pNcBLwKBQAyQaUh/uUYexrT5wHxgteFrPpAD3GZIqwF+BXIN+R6sUSchRCuTOWNCtF/TvHw1ac4uLnrqf7Kvwx/OLi56L19NGjDN8l+1zUSizCvr0cLlZAA+QIDhdRCQbeMypP05XvsTdiI9Y0K0Q24qj9U+3fxnPjR/hc/A8LG4uslKB/XRl+o6pSfFD/pk1aJ3Cn8/G16qK37aDtWIAA6h9Gz1R+nRakn5wFsoQRnAEpQeO5uQ9td0baT9CTuTnjEh2p9pPt38Zy7ZnOYTOuo++SBshKubitBR97Fkc5qPTzf/mdi2h0ID+DUh3WiUQMxmAVEjQoA/oQyHOqEMe9qKtL9maOH2JxyEi70rIISwLS9fzaapi9/q0zv4JntXxaG4dHJF3TPA/fj3X99QVlK81gZZRgCpQDAwDzgJ/IQyFPl/wB01zlUAXwL/BG5EGaa8DfAE3jOkdQEmAlOAW4BzwNvAn4HQOmndgUeBu4AbgKtAPBCGEiB+DSwANhrK/CfwAMqSHMZ5bBaR9meZFmh/woE42bsCQgjbcnZx0a8+cLWT9Eg0n75Ux9MjvcsrKypc7V2XFmZ8ivJIjXNqYAjwlTUZS/uzXAdqf6IOGaYUop2prKiQD0ILubqpqKyoaO9zaVUoOwR41jk/AuXpSqtI+7NcB2l/wgwJxoQQomPRAW+iDHHWfLLPi+rlMoQQrUgicCGE6HgygcH2roQQQiE9Y0IIIYQQdiTBmBCi2ZJ3xxEV5mT2WP3UOLSFzXsgLystgbhlc9CX6lqoxsKRaQvzWf3UOKLCnBpsJzvXxxAV5kTMtMFcONX0EVd9qY64ZXPISktoMJ22MJ/1L0xtVt5CNIUEY0IIiyzecIjYlCoWbzjEyElRvJVQTGxKFX8c/WCz8woOjSDyhbWyJpUwy9NHzdP/t4eJc5ZyYFssuRmp16TRFuaTlZbAyElRPPd+Ij361N2b3Tx9qY6tK5/hwLbYRtN9/vaLnMs5btHPIERDJBgTwvEsBLztWYEuft3pGdjf7LXAPwylqCCvlWskWpHd2l8XdU9GTooiaefGa3rHTh45hP8Ng5qdp6ubiskL32TkpKhG093/t1foFTig2WUI0RgJxoRwPC+jLMz5Knb6UBww7E48fdRmr11/42B69Alh5/oY4pbNYfVT41j91DgKLp4lbtkc03DmzvXVi74n744zvTYOSR1P/sqUPnl3ay1ML5rAru0vfPwM8n77lUvnq1fh0JfquJJ/ge7X9bsmvXHo0lw7Ml7bHvuPWueNw5aWDHkKYQkJxoRwPM+hrNi+APgdOwZl9UneHcf2tUvIP5fDrJfjePr/9pB/LgeAtxKKWbzhEFlpCaahpfeipwPVQ0E/J+7lk1WL+HPkMyzecIjkXRubPQ9NtBi7tj+/Xn3Q9O5LyldbTedyM1LpHfwHOqtqL522c30MGv8gYlOqWPppBvEbV5jmhRkDs9iUKgaPuq/WMOV3n67lz5HPEJtSxdgZi/g67k2ZzyhalARjQjieVUAx0Blwow0GZcPujmTinKUEh0aYetBqzgvz8tVQVVVFUUEewaERPBazCageCvrD8LuY9uzb9OgTgpevBg/vrvb8cURtdm1/nd1UhI+fYQrmAX7LTr9m2FxbmM+57HQCB4YB0KNPCGNnLCJp50YKLp7lp4NfEnbnZAAC+g8xDVNqC/NJT9zLkgf7ExXmxHvR08k+llSrJ04IW5N1xoRovv3A7fauRA2dDV8XouxD2KYl747jvejpXHdjqL2r0pAqe1fAgbR6+wvoPwR1r0COHd5D4MAwuvh1v2bYvKggD+2VS7XOaXoHcvLHgxQVKnMavXw11+RdVJCHh3dX3vgm75o8pXdWtBTpGROi+W5H2dfVnkfNT4UyoBRYibI5dJuUlZZAVJiyHe7STzPo4tfDzjVqkL3f37Z+2LX9ubqp6PfH20jetZET//3O7MMkXr4anJycrnmYxLe7P66d3dFeuVTvgyYNXROiJUgwJoTjmQ94UP0h+AbQDXgeuNralcn7LadJ6TJ+2MdjMZsYdndkC9dItLA20f5uunUcoMwX8+sZcM11Tx81waER7Fwfg7YwH32pjqSdG+k/dDR+PQPQ9O5rmgt26Xwu2ceS+Oj1v6EvK6l1DeB48lfSKyZalAxTCuF4XgNcUHoilmGHAMxo5/oYtq9dAkD2sSRmv7qFHn1CTBP4ATT+QQy7O5L+Q0ezfNYI3ouezsDwsVy5dIEt/5rP4DseIG7ZHAAqyvVcvXSRnxP3Uph/nolR/8v22H9w5kQaxVcvM+vluHqf4hStptXbn7Ywnw0vRfJz4l42vTKbxRsOERwawbB7ZqDpHYirm8psWxw74+9svXiWBWOU4cjHYjYRHBoBwP1/e4UNL0UyL8KDoWP/Qq/AAdwxZR7X3RDK5IVvsnXlM8yL8DDdB5jqcC7nuKmtC2ELTvaugBAOqAr7/u0sBNZR/4dgVWyKTHmylGEoVf5trJ+0vxYk7a9jkp4xIRzPSntXQHRo0v6EsDGZMyaEEEIIYUcSjAkhhBBC2JEEY0IIIYQQdiTBmBBCCCGEHVk6gX+qn5/fQ5WVlX/SarXd9Xp958ZvER2Zq6trmaen50VnZ+fvL1269Amw2d51EkIIIdqC5gZjz6lUqheHDRtWMXXqVJ/w8HCCgoLw8vJqkcqJ9qOoqKhzdnb2dUlJSddt3rx5THJy8jqdTvcKyppFwoacXVzK9aW6Tq5uKntXxeHoS3U4u7iUV1ZU2LsqDkvan+Wk/XVcTR2m/KNarT5x7733Pp+UlOS1f/9+n9mzZzNo0CAJxESTeHl5MWjQIGbPns3+/ft9kpKSvO69997n1Wr1CeCP9q5fe+Lh3TU9PSne3tVwSOlJ8Xh4d023dz0cmbQ/y0n767iaEozd6erqmrRs2bIbduzY0WXQoEEtXinR/g0aNIgdO3Z0WbZs2Q2urq5JwJ32rlN7UVSQ9/onqxYVGrdyEU2jL9XxyapFhUUFea/buy6OTNqfZaT9dWwujVz/o6ur676tW7d2joyU/eSE7d1yyy3cfPPNLtu2bZtSWVm5Ezhv7zo1wT+B/7V3JRrwU3lZSe/k3XH91T0D3NW9+uDSydXedWqz9KU6jiXsYs2iSYWFv5/9sKJcv9zedXJw0v6aQdqfaJRarT4RGxtbJURLi42NrTIMWToCR9nrZZqXrybN2cVFj1JnOcwczi4uei9fTRowzfJftTBD2p+0P9FEDe1/9dy99977/I4dO7pYknF61inWbdnJof8eo+BKEb5dvBhxy03MnjKegcF9LKyuaM8mTJhw5csvv3yVtj+pvwrZO04IRyV/v6LNqbdBqlSqq0lJSV6WzBE78MNRHn9xJQ+PH8OQm0Lo4uXJlSItqccy+XjnN6x/ZSEjh8rcM1Hb0aNHCQ8PL9LpdN72rksj5B9zIRyX/P2KNqe+Bjn19ttvX7N//36f5mZYWVnFHTMXMuHPtxE64IZrrqcd/4UdXx9k/4crcXaWvwdR2x133FH47bffzqVtr0Mm/5gL4bjk71e0OWafpvTz83to6tSpzQ7EAD74bC8B/j3NBmIAoQNuIMC/Jx98tteS7EU7N3XqVB8/P7+H7F0PK0VQPSck2vA6ogn3qVGC0BDDaxWwton3CiGEcFBmg7HKyso/hYeHW5ThrgM/EHbzgAbThN08gN0Hf7Aof9G+hYeHU1lZ+Sd718MKamAe0B/lf9/7gENNuE8FvALU/OPRAXOAhBp5P2GzmrYf+2kDE7HlcJjjW4RoY8wGY1qttntQUJBFGZ749TR9r+/dYJq+1/cmM/u0RfmL9i0oKAitVtvd3vWwQn+gEMg1vE4ARjThPh3wInC8gTRPAj2tql37dDtK4CuHHE057kCINsbsdkh6vb6zJSvrl1dUUFxSisrdrcF0Knc3iktKKa+ooJNLY0ud2VZu7ml27YnnQGIKZ3NzKNFpcVN5cl1AICOHh3HPuLEEBFwvZdiJl5cXDr7XaR4QDvwdiDGcS6hxPRK4DUgFYoG9hnP5ZvJSA2+hrGs2GVhqOB/RwD1CCCEcjKUbhZvPzMUFD3c3dI0EZLqSUjzc3Vo9EFv+xmp2fLkb9eAJeIZFccM9/XBx96KipAjdxZPsyThM3JbZTBh/N4sXPN2hyxAWywT+hjI0uRSYDsQZrkUAmwzfTwc8gDdRerxiamdzzbBlzet10wohhD095u6jeaS8RDe4vFRr0XJY7UknN88rndxVR0oK8z4A3mvKPU71nK+qqqqyqBIPz49hxNDBDLyhb71p0n/5lYSUI2z9f9EWldFceXn5PL3oJYq9gtCMnIuLe/29fhUlReQdWINHUTarV7yMRqPuUGW0BU5OTlB/22wLqmha/aJRArKavV/GnrFnUIYmIwzpjFtcGHvCMqndM5ZpSAcSjNXV1PejNUVgfq7gCGr3lFoiBKVNzENpUxHADKrblBCt5d7Onr6x3frd4tl/7BM+PQeOwNOvNzi1tT/HVlRVhfbSb5xPP0RG/LuFv5/8r7ZMWxAFfNnQbU3dKLzJ7hk5lJSfGpr2Aik/Hefu24bauuh6Pb3oJYp7DKfH2EUNBjAALu5e9Bi7iOIew3l60UsdrgxhUzGAxvD9k/WkyQMut051RCtKQHnv/0P1wxzTUQI0a56ODQG2AF3rlDWHxgMxeQBE2NJcV5X31jv+59+9x8d87RN82xQ81f4dOxADcHLCU+1P8G1TGB/ztc8d//Pv3q4q763A3IZus3kw9sgDd5F79jxpx38xez3t+C/knj3PIw/cZeuizVr+xmqKvYLoETGzWff1iJhJsVcQy99Y3WHKEDYRAtxX43U+SlDmjzL0KDquPSi9pKOtyCMTmIJlAbw8ACJs5V5XlffK+5YnqPoMu6/x1B1Yn2H3cd/yBJWrynslcG996WwejDk7OxEz/6+s++hzvk1K5UqRFoArRVq+TUpl3UefEzP/r62y4Gtu7ml27NyNZmSDAWm9NCPnsuPL3eTm1v/kZ3spQ9jUk1SvFQbKh+9BqnsuAlHmi4EyMf9LZDJ+cyxE+V3Or3N+vuH8wlavUdMUAzk1XkejrCO3x3CoUdrNEZSh17VUB/BqQ5oqYFSdfCOpHsKmTh7GfI1D5kvrnDuC0lN3pw1+PtFBdPb0jR29YKPKr8/N9q6KQ/DrczOjF2xUdfb0ja0vjc2Dsd0HvmfvoRT6Xt+Lz/Z+x7Ovvc3cl5bz7Gtv89neb+l7fS/2Hkph94HvbV30NXbtiUcdOqHRIb36uLh7oR48gV174tt9GcKmlqLM3zGuawTVk/hBGbL6ps41teHrX1CGoYbVeR2CsmbZUpQP0JrBXkezDnABXkUZ5gUlmH3VcH6dnerVmElAFMr7GInyXgYavh9nSDMZGE51sD6J6oc5YlDaTg+qhylrPhQCSjv6J0rvmQdK8DfOcO8SwzEOZeg021DWDMCRl5MRreuxbv1u8ZQesebpM+w+uvW7xRN4zNx1mz1NuX1fIm+8/wl+Pl3of0NfHh4/hm7qrni4u6PXl+Pq2onikhJ+z79M9ulzvLt1N6+/u4UFjz7ExNHDbVWNWg4kpuAZFmVVHp7Bt3IgMZY5s2e16zKEzWQaDuM8HnN+Be6n9hyffKo/kI3qvs6k7U1Ut4erwEpgAUrwAeAHlBnOX7VTvczpCmTUeD0dpW0kAMbFHI29ov2p7r2q6Zzha6rh61ZgoOH7BEOexrz6A+kobQXqb4OgBGF7GkkjRC3uPppH+o99wqIdejq6/mOf8MnPSXukpDDvmicsbRKMbfhkNx98/hWTx48hJKjPNdddXZViPNzd6ePfiz7+vbgjfAiZ2adY+d7H/H6pgFkP3W2LqtRyJjeHG+/pZ1Uequ79+CU3p97r7aUMIRzMMpTex5qqDOfbkssoAVJmYwlResmWcO3TspHAWZr2pGRgE+tlDAjzaHitOyFqKS/RDe45sCnrWIu6eg4cQXmJbrC5a1YPUx7NzOatTV8wJ3KS2UCsISFBfYiKnMRbm77gaGa2tVW5RqlOa/HQnpGLuxclOm27L0O0CuOQUhTKorDCcldR1mgrM7wuM7xuS71ilqjvQY+mPgCS04y0xmHPBJRhUHnARDSqvFTbxdOv4V12hHmefr2pbx02q4Oxj3bsY9yocLr5+Vp0fzc/X8aNCuejHfusrco13FWeVJQUWZVHRUkRbirPdl+GaBUJVG/JImuFWW8Z1fPu2mKvWHOloOzeMMnwWoXyZG7d82EocwnNBVAZKL1jQ+rkUZdxFwdQhj2vWF990WF09OUrLNXA783qYOxAylGGhg5sPGEDhoYO5EDKT9ZW5Rr+AYHoLp60Kg/dxZNcF1B/z397KUMIB2TsHdPT9nrFIlCGAP+CEiBF1rlunMC/tMY147IVi1CCy0TDvcZdHTYZzoeirF/2IkqAtsmQTzTKUON84O06eUDtB0ACgAmGNBnAF8iCsULYjVVzxk78eoby8kq8PT0aT9wAb08PyssrOPHrGW7se51VedU0cngYezIO4xVgdoi2SbRZhxk3PKzdlyGEg1oG3EDb6xUz9oLWJ47aT9gaZQLm/tDry89cPk3NYzNKsCiEsDOresayTv9G757dbFIR/17dyDr9m03yMrpn3Fjyj+yweIivoqSI/LQd3DNubLsvQ7RL07x8NWnOLi56qpfZaG/HFeBBw1eL8nB2cdF7+WrSgGmW/6qFGR2h/Vl9SPsTYGXPWNbpc/TqZps9D3tq1GSdPtd4wmYICLieCffezeEDa+gxdlGz7887sIYJ4+8mIOD6dl+GaF/cVB6rfbr5z3xo/gqfgeFjcXWTudn10ZfqOqUnxQ/6ZNWidwp/Pxteqit+2t51cnTS/ppO2p8AK3vGzpzPQ+Nnm+VGNH4+nDmf13jCZlq84Gk8irK5kPBhs+67kPAhHkXZLF7Q+N9FeylDtBvTfLr5z1yyOc0ndNR98kHYCFc3FaGj7mPJ5jQfn27+M5EeCmtJ+2uG9tj+Tn4Xx7qJTtccn84fTMHZxld5KTibya5/3NWktE11/ngCB9+ZQ3mZbadGpm6J4eR35mYcNI9VwVje5UK6+ph9SrPZuvp0Ie9yoU3yqmv1ipfxuJDIhfgVjQ71VZQUcSF+BR4XElm94uUOV4ZwfF6+mmcfmr/CRz4Em8fVTcVD81f4ePlqnrV3XRyZtD/LtKf2129UJJPXZBB821+YGZfH7O1VzN5eReikRXyzfEqDQVbJ1XwOvzsfXcEFm9ap54AIbntyLZ06265dnvwujpS4JTbJy6pgLP/yFbp42Wa5hC5enuRfbpmnqzUaNR+9v4Zb+7qRufZhLux/h6LcI6aApqKkiKLcI1zY/w6Zax7m1r5ufPT+GjSapg/BtpcyhOMrvnp54MBwmR9oiYHhYym+etm6x8M7OGl/lmvv7S9w+CS6h4RTcqX+UTB3bzW3PrEKlW+PVqyZZfqNiiQssu6GGZaxas7YFa0WLw/bRJleHiquaFt2UdLFC57mLw89wK498RxIjOVEbg6lOi3uKk/8AwIZNzyMe/5nnVVzq9pLGcJxVVZUdJJeCcu4uqmorKiw2TZxHZG0P8u19/aXk7iNqxdz8L2uv+lc6pYYU+/SxNcP0XNAhOlayZU8dr07nzOpe03Xyst0JK5/huN7lD23wyKXMmRKtCmf64bcxeiFcZRcyeOb5VPoHhLO8MffJCdxG1fOZzNkSrSS99V89q2M5EzqXtR9QxmzeAu+/iGm8z0HRJASt4SwyKUEjZjMN8unkP9rmtl62oJVb3qxrhSVu5tNKqJyd6NYV2qTvBoSEHA9c2bPatE9GttLGUIIIYSlCs4c58NIjen1gHFRjH3hM9NQ4cnv4ug9aDSzp0Rz/ngCqVtiGL1QmX+lK7hgen0mdQ+/7N+IJngIeVnKFq2PfVJMXlYqqVtiKLmabwqyeg8ajbu3GndvNbfP/zcubiryslLZt3K6qRer5Go+CWvncesTq/D1D+Hkd3Ecfnc+o57aQOqWGM6k7qXngAhmb68ClIAxdNIi+o2KJHVLDL8d3de2grGS0jLc3DrbpCJubp0pKS1rPKEQQggh2jzf6wYw/uVvcPdWpsqkbonhvYc8mPj6ITTBQzj380GOr5xe656CMxm4d9Gg8u3BrU+swt1bjVf3QPj5IKDM/TIGQu5dNFBVRcmVPNy91fQeNLpWoKQv1aIOGgz+IYxeuIkr57NNZXT29MGrWwCgDJ+e+/kg5459y9AZr3D1Yg69B4021ckY6AF06RnEuZ8PUl6ms+n8M4uDsYqKSsr0elycrV7EHwAXZ2fK9Hqb5CWEEEKItmXQA39Hm3+WX/ZvxLt7IGXaQiavycDXP6RWuqY8RXnyuzj2rZyOum+o6Zzvdf35Zf9GSq4qe96ruphfB7XoYk6t1506q/BU+zdYXs3h0QHjohqtX3NZHEmVV1RQXlFhy7rYPD8hRMvTFuaz+qlxRIU5mT2Sd1v/2LfouLLSEhptSzXbYNyyOehLdWSlJZi+F21TqfZyg5P5zTl/PIF1E5WNJCavyag10d/dW42n2p+CMxnkZaXi1sX8w2te3QO5ejGH8tLia86bk7olhi8WDefm+55h9MJNzapvU1kcjFVWVlJZWWXLutg8PyFE6wgOjeCthGLeSihm5KQoFm84RGxKFUs/zcDdw6vZ+aV99wUXTtlujSHhuIJDI3jjmzz+MPwukndtRFuYf02a8zkZ/Jy4l8diNhH5wlpc3VQEh0aYvhdtQ07iNo7viaXXH27DU+1PzwERpjlfAPnZRxrtFfvt6D5GL9xEv1F1t3tV9B40mlPJX1BeWmwaHq3L+ABB+q53AKUn7nJueq0HC4xKruZzOTfdNMG/pVg8TFlVBVVVVcx9abkt6yOEaNxCYB1taGPsoJvDcXVTXdML4dczgPM5GfXcZd6FU5l8+/E7/GXRaltWUdiOXdpfcGgEWWkJHDu8h2F3V38Q60t1/Prz9wwdK9ts2kgIyv6mFjMOIQJkHfxPrWs1AynjU5DGSf5hkUsZeM+TylOOP8Zz+N35DLx7LvGv3G+6P2jEZHa+NIZ9K6dz3R/Hoiu4wOF35zN6YRzu3mp8r+tP6pYY+gy7z2x9jOXe+sQqvlk+hZS4JaanKTu5eZC4/hnOpO5Fd/m8KQDrGjCQrXOVQC34tilkHdxiGtasuc5YfQFiU9S3kW1VVZX0Ugn7cHJygoY3Wba3KuxbPx3gAqxE2SC77odiVWyKff5+9aU6tq58hvDxMwgOrX7a6MKpTNY9P4UzJ9IYOSmKyQvfxNVNRVZaAstnjeCxmE24e3ih7h3I+/94hDMnlEfIH4vZVOuDtzVEhbX59mdvrd7+tIX5pO7bRmd3D07+eNDUfkBpW6eO/5dfUg/Q74+3mdpL8u448s5mM/5xZfL1zvUxpO77lGnPvk1ZSTEDht15Tfszbt1kPA8wcc7SBvOwNTu3Pw8gC8gG/g4kmklTZXzKUDSfYYj1mvfXNrPvhRCt6TmgAlgA/A68CnjbtUYN0Bbmk/LVVp57P5G3EpQ5Gqn7tqEtzCfjh32mc8cSdtMj4EZmv7qFgeFjWfppRqsHYqJJ7Nb+AgeGkX0sidyMVNO5E6kHuGHIyFrpstISeC96eq3XGv8gnns/kaSdG7ly6aLZ9gdKcPfrz98Tm1LFG9/kkZWWYDrq5tEOFQNxwDDga2Ab8Ae71qiDkGBMCMezCuUfzc6AG208KDufk8H2tUuYF+HBvAgPDmyL5eSPB9GXlpCVlkBuRirD7o6U+T2Ow27tr0efEIaMfpCMH/YBmOaPefnUnhsUHBrBYzG1J1on79pIWUkxkS+sNQX55tpfTnoKH7+xgKgwJxaM0fBz4l5TeebyaIf+hdLz6QE8AKQC7wKyingLarcr/QqH19b7wdtS/YyL/S0EbrBnRczJ+y2n1lBPTeMfjzYNBy3ecKjW0KadtaX3t61r1fYXdudk/rPiaS6cyuR8TgY31ukVMyc4NILg0AgWjNHwh+F3MevlODx91GbbX97Z7HqHx83l0UJs2f6+BW4FSoEyw2Hu+5rnzgM9Dfd3BmYAj6P0mokWID1joq1yasNHW6hfzUfKjP+QrgQebcbvuNUUXDxrdomB4NAIYlOqWLzhEB+9/re29ASlvd/ftn7Yrf359QxA07svh77YwJX8C/j1DGjSfeMfjyY2pYrg0Ag+f/tF9KW6ettf3tnsJufRQmz5Xt0BdAH8USbnhxnOTQCmAVEogfT/oryHsSjD0DVVAIXAuRb4WQUSjAnhiOajDCEYPwTfALoBz2PnJyzLSorJP1d7QUXjPJ/UfdsAZZJ/2ndfoC3MZ+f6GPSlOgL6D0E2l3YYdm1/rm4qwsfPIP7DFXRWeTZpaDsrLUVU2A4AAB1jSURBVMG0RlnYnZNx9+qCtvCS2fbXf+hotq9dQlZaAqAMhR5P/spsHg6kFOW9yQd+A3KAE8BPwH9RJup/C8QDKqqHmy+jBGCLAF/DV9ECJBgTwvG8hjLFoM0EYaBMfH5j7hh+TtzL8lkj2Lk+BlDm+cx+dQvxG1cQFebEa48Op2eg8ph4wcWzvPbocOZFeODu4U2PPiGmno8lD/Y35SHalFZtf1lpCSwYo2HTK7NZ/dQ4tIX5BPQfwsQ5S7npVuX1mr8/wIFtsbwXPZ24ZXM4vON93ouezva1S0xt6OjBHUSFObHkwf4MHnUfrm7uZttfcGgEizccYvmsEUSFObHhpUgC+g+pJ492OcdxNeAKHAeeBnoD79i1Rh2AUz3nrV7aIj3rFOu27OTQf49RcKUI3y5ejLjlJmZPGc/A4D5W5S3aN1naolGNrfNkt6Ut2gNZ2qJR0v5aUBtof4eB14Ev6rne4ktbFJzN5JvlU8j/NY0B46IYOG4OV38/RWCNtcMcVX1LW7TIBP4DPxzl8RdX8vD4MTz/5CN08fLkSpGW1GOZ3D93CetfWcjIoYNaomghOoKV9q6A6NCk/bVvt9qz8PIyHT998SYRc96m54AIU2AWOql9j5DafJiysrKK6FX/Zva0+7k9fAhdvDwB6OLlye3hQ5g97X6iV/1btj4SQgghRC1Fv+dSpi00bU3k6x+irI7v3vxt1RyJzYOxDz7bS4B/T0IHmH/COXTADQT49+SDz/baumghhBBCODD3LhpKtZf5YeOLlJcpT6v6+ofUGqI0bha+bqITB9+ZY0pXcjWfXf8cx7qJTnw6f7Bpn0vj+dQtMayb6ETqlphr8jGesxebB2O7DvxA2M0DGkwTdvMAdh/8wdZFCyGEEMKBuXurufWJVVzMTOK9hzyuCZIKzmby85dvMTMuj5lxeVy9mENeViolV/NJWDuPW59YxeztVYROWsThd+ejzT/LDxtf5Eyq0gE0e3sVQ6ZEU3A2k99PfM/s7VXMjMvj/PEEzh9PsMePDLRAMHbi19P0vb53g2n6Xt+bzOzTti5aCCGEEA7O1z+EB1cdYfTCTaTELanVy5V3MoU+f7oXd2817t5q7vnnHmVu2ZkMOnv64NVNWXcucPgkvLsHcu7Ytwyd8QrXDbmL3oNGm8rIO5lC4oYFrJvoxIeRGs6k7uW3o/vs8vOCjSfwl1dUUFxSisrdrcF0Knc3iktKKa+ooJOLiy2r0Kjc3NPs2hPPgcQUzubmUKLT4qby5LqAQEYOD+OecWMJCLBu1wcpQwghhLBOv1GRBA6fROL6Z/jpizcZ/vibXDmfTZeeQdekLbpYe33DTp1VeKr96837yvlsRi/cRL9RbWNbK5sGY51cXPBwd0PXSECmKynFw92t1QOx5W+sZseXu1EPnoBnWBQ33NMPF3cvKkqK0F08yZ6Mw8Rtmc2E8XezeMHTUkYLlyFahrOLS7m+VNepna6B1KL0pTqcXVzKKyvqLkAumkran+Wk/Snzu3ISt9F/7BOAElTdfN8z/Pejf1JeWkyXnkFcOX/tDgle3QO5un8j5aXFdOqsqnW+PubysRebD1Pe2Pd6fj39W4Npfj39GyFBrdejkpeXz7RH53L411JC5n5MjzuexCtgMC6GpzNc3L3wChhMjzueJGTOxxz+tZRpj84lLy+/kZylDEvKEC3Lw7trenpSvL2r4ZDSk+Lx8O6abu96ODJpf5aT9qfIPvxprflbeSdT6BowEHdvNZp+YZxPP1Rrcv6ZI1+Znr5M36WsT1twNpPLuemm83X1HjSalLglpnKM+diLzRd9ff/TPez7/igzJ91Tb5oPt+1i9J8G8eiD4ywqo7mmPTqX4h7D6RExs8n3XEj4EI8LiXz0/hopw8ZlNEYWfbXatO4BN7yzZHOaj/RONJ2+VMfSqaGFF3N/eRL4yN71cWDS/izgQO2vRRd9LbmaT15WKhczk0iJWwLAgHFRDH/8TVOP1/njCWx/Vtng/bohdzF6YRzu3upai8Wq+4YyZvEWvLoFkLj+GY7viTWd8/UPaTCfllTfoq82D8YqK6u4Y+ZCJvz5NrPLW6Qd/4UdXx9k/4crcXZu+c+z5W+s5vCvpfQY2/wF4y7Er+DWvm6NDsNJGU0voykkGLOem8pjtU83/5kPzV/hMzB8bHvdtsUm9KU60pPi+WTVosLC389+WKorlnF3K0n7azoHbH8tvgJ/e9ZqwRjUXoF/yE0htVbg/3jnN622An9u7mkiH51NyJyPTcNszVFRUkTmmoeJ+/e6eieqSxlNL6OpJBizmWlevppni69eHlhZUdEiu220B84uLuUe3l3TiwryXqdt90g4Gml/TeCA7U+CMSu02nZIuw98z6H/HqPv9b34bO93bPnya9O1zq6d6Ht9L/YeSkGrK+HukX+ydfG17NoTjzp0gkXBBSjzo9SDJ7BrTzxzZs+SMqwsQ7S6j4oK8hzhH3e7qqyooKggz97VaI/aYvvzBt4HHqUFNzdvDml/Amw4gX/7vkRun7GQd7fupsq5Ew+PH8Orz85lzcuLWf2PBax5eTGvPvskD48fQ5VzJ97dupvbZyxk+75EW1XhGgcSU/AMtm6bLc/gWzmQmCJl2KAMIYSwsxeAiYavQrQZNukZ2/DJbj74/Csmjx9DSFCfa667uirFeLi708e/F338e3FH+BAys0+x8r2P+f1SAbMeutsWVanlTG4ON97Tz6o8VN378UtuTr3XpYymlyGEEHbkDTwDuBq+LqON9I4JYXXP2NHMbN7a9AVzIieZDcQaEhLUh6jISby16QuOZtp+vY9SndbiYTcjF3cvSnRaKcMGZQghhB29QPVcHSekd0y0IVYHYx/t2Me4UeF08/O16P5ufr6MGxXORztsvw2Bu8qTipIiq/KoKCnCTeUpZdigDCGEsBNjr1hnw+vOhtfeFuSlAtaiPMgT0cj1trG8u61Z8YBfh9bA783qYOxAylGGhg60Ko+hoQM5kPKTtVW5hn9AILqLJ63KQ3fxJNcF1L+Cr5TR9DKEEMJOavaKGVnaO6ZDCeRigRkowVdNAYav04E4C/Jv0zq5eV7RXmp4YXdhnvbSb3Ry87xi7ppVwdiJX89QXl6Jt6eHNdng7elBeXkFJ349Y1U+dY0cHoY267BVeWizDjNyeJiUYYMyhBDCDryBhUAlcMlwLh+oMJy3pHcMINPwNaDO+YAa19qdTu6qI+fTD9m7Gg7pfPohOrmrjpi7ZlUwlnX6N3r37GZNFib+vbqR1cg2Ss11z7ix5B/ZYfHwW0VJEflpO7hn3FgpwwZlCCGEHcxGCbyeA4zLq2uA5w3nZ1uYrxb4FJhc45waCAQu1kkbjTJsuceQpuZQ5hEgpEa6PcB9wGhDGuM90Ya0oTXO17we2Uh+5oZUm62kMO+DjPh3C22RV0eTEf9uYUlh3gfmrlkZjJ2jVzfbbB3QU6Mm6/Q5m+RlFBBwPRPuvZu8A5Zt05N3YA0Txt/d4CKmUkbTyxBCCDtYiRL8rKpzfpXh/Eor8j4O+FMd5F0PHKiTJhLIRhkW3Qg8CQwBzhrOrQDCDHmcN6SfC5QCbwKXDfn8C4gHSoAXDec2AuOA/kCQIb+/oQyj+tfJz1be+/3kf7Wnkr+wYZbt36nkL/j95H+1wHvmrlsVjJ05n4fGz8eaLEw0fj6cOW/7he8WL3gaj6JsLiR82Kz7LiR8iEdRdpO295Eyml6GEEK0I/nAQZSACKAPkFsnTRCwCaXXahNKkJQKxKD0XG0ypCsGbgFeAR4AEoA8oL5eqBwgw/B9ILDUUMYhw2vM5GcTZdqCqH1vzNBdOmX7ud7t0aVTP7HvjRm6Mm1BVH1prArG8i4X0tWnizVZmHT16ULe5Zbp+Vy94mU8LiRyIX5Fo8NwFSVFXIhfgceFRFaveFnKaIEyhBCiHdmDEnANNrzWmUkzAqXXygmYgzKv7AiwD2Wiv/G+OSi9YYk0f1hxeo0yxqH0vFmTX0O+1OuuLvxicYROesgadir5C75YHKHT664uBL6sL51VwVj+5St08bLNUgZdvDzJv2z2IQOraTRqPnp/Dbf2dSNz7cNc2P8ORblHTMFGRUkRRblHuLD/HTLXPMytfd346P01aDRNH4KVMlp2p3shhGij8g1fp6H0StWVTfVTlyqUOWq3ogxP1uytMs4LywWmAHX3CwxAmS9mTo6hDOM/xE8ANzSSn7XW6HVXJ+//f3/9bWf0nwuzDm5Bm39Wlr2oqkKbf5asg1vYGf3nwv3/76+/6XVXJwMNzgGyaqPwMX9dxBN/uR91V+uHKvMvF/Lufz7nm3+vsDqvhuTmnmbXnngOJKZwJjeHUp0Wd5Un/gGBjBwexj3jxlo970nKsI5sFC6EaEHW/v2qUHqbolCWt3gGJVC6x3A+GmXIEKqXt6h5boTha93AbTzKxPtHDK+noDyVabz3P4bzGpRer0dqlK9DmRtmHPKcjtJjN9NMfi3hMXcfzSPlJbrB5aVa2wyXObBObp5XOrmrjhgm65udI1aX2Qbp6upaeunSpc5eXg2vyD58ytM8N3cmHir35te2jmJdCa+t+ZDELautzks4rqKiIvz8/Mr0er2bvevSAAnGRGNCgC1U92RMR+m9ABvO3REWkb9f0eaYHab09PS8mJ3d+PZEJaVluLl1bjRdU7i5daaktMwmeQnHlZ2djaenZ93HwoVwJBEoE6v/RvUcHjA/hGUPapRhLCFEG2E2GHN2dv4+KSmpwRsrKiop0+txcbZ6EX8AXJydKdPrbZKXcFxJSUk4Ozt/b+96CGEh47yf6dTuAYtDGZ5qC1tUPAn0tHclhBDVzEZSly5d+mTz5s0NPtpYXlFBeUWFTStj6/yE49m8eXPhpUuXPrF3PYSwkHGJgz1mriVQe3ucCJQhsyqUAA6UYG4PcCfm9zcMQXkKr8pw3bgVTzTVC4DWXCS0qk5a4/yjpTXS1ZenEKKV1NettTk5Odnl6NGj9d5YWVlJZaVtn5qwdX7CsRw9epTk5GQXYLO96yKEhYJQ5oYVN5IuBOXpNieUCdkRKCuuvwLchfKk3ZsovWnGp+TUKKu9DweMe9BNQgnWlqL0ukWiBIQaYKDha3+gL8ok8xhgieEwBo7m8hRCtKJO9V3Q6XSvvPjii8/v2LHD7JMRVVVQVVXF3JeWt1ztRIfy4osvXtHpdK/aux5CWCkQJbAxrjelRukRu8vwegnKcgdvGA6jBJSV1QNRgibjk2/GFdj7U92rVdMzhjyhepmFTGBqjTRVKIFZ3afp6stzG+bXyxJCtDa1Wn0iNja2SoiWFhsbW6VWq0/Yu803kaN04U7z8tWkObu46KkerpKjzuHs4qL38tWkoawTZa1Iau8NWFM01cOR0dQefjQyDlMaF+gMQekpNu49GG3mnrp511RzKDTCTNqG8rSWtL/Wb3/CQdXbMwaQn58/Zd68eUndu3fvfP/997dWnUQH8/nnnzNv3rwyvV4/xd51aS/cVB6rfbr5z3xo/gqfgeFjcXWTaUD10ZfqOqUnxQ/6ZNWidwp/Pxteqiu2Zl+vFMPXySi9Ww0JsiB/f5Q5XY31WhmX1vgUpUcsroG0Tc2zyaT9NZ2N259wUI09CvmjXq+/d/LkyWXr1q1rlQqJjmXdunVMnjy5TK/X3wv8aO/6tBPTfLr5z1yyOc0ndNR98kHYCFc3FaGj7mPJ5jQfn27+M7GuhyITZUmLpTTc47TPkMbYW6VGmbTfkBQgnOo5XSrgvnrShqEEYo0FhM3Js6mk/TWDjdufcFBNWZfiK71eH/7CCy/8MmHChCsNTeoXoqmOHj3KhAkTrrzwwgu/6PX6cOAre9epvfDy1Tz70PwVPvIh2Dyubioemr/Cx8tX86yVWSVQPSm/5pDUg8DWGmlGoKw9VoXSc5VO9QT+t1GCoi3AXwzX81BWUV9kuCcRZT0z4wT+pVQPfaYYyqsCvkF5UOBtlB4zYyB4xJDWXJ4Wk/ZnGRu2P+GAmrsK8XMqlerFYcOGVUydOtUnPDycoKAgGlupX4iioiKys7NJSkpi8+bNhcnJyS46ne4V4DV7180CVbThFbydXVz0qw9c7SQfhs2nL9Xx9Ejv8sqKCld718VRSfuznLS/jsvSD5Spfn5+D1VWVv5Jq9V21+v1tlmGX7Rbrq6uZZ6enhednZ2/N6wj5sjLV7TpYAyoik2psncdHFZUWJvfG7Wtk/ZnBWl/HVODE/gbsPnSpUuO/GEqWpler6egoMDe1RBCCCHaHNvsZSSEEEIIISwiwZgQQgghhB1JMCZEO6cv1RG3bA5RYU5mj6y0hMYzqSN5dxxxy+agL7V8aaqstIR686h7LXl3HDvXN7ZKg3BUF05lEjNtMFFhTsQtm8PpE0dI++4L03VL339j2ze2cW1hPutfmMqFU3U3ImicNfcK0RhL54wJIRyEq5uKyBfW0u+Pt5F3Npvxj1cvf3XhVCbnc5q3kkFWWgLvRU9n5KQoi+uUlZbA8lkjzOZh7tqwu80tVi/aA32pjq/j3mTas28THBrBhVOZrHt+CmNnLDKlsfT9N7Z9Yzmfv/0i53KOW5SXp4+ax5dVT5U2BnjBoRH13SJEk0nPmBCOZyHgbYuMevQJIXRU89b4DA6N4LGYTVaVGxwaweINh5p9TbQJNmt/AJfO56IrKqRnYH9AaZOzX92Cu4dtl0xydVNx/99eoVfgAKvz0hbmS0+tsCkJxoRwPC+jbAj9KlZ8KGalJdQaojQOPR5P/oqoMCdWPzXO9KFjHD6qO6SYtHNjrbRGxnvqDoMah6Nipg3mt6yfa+XV0LWd62NI3h3XYD3NlR0V5kTMtMFcOJVpyn/n+hiOJ39V6x7RLDZpf0ZevhqKr17m87dfNLWvuv9JqPn+XziVyap5d5GVlsDqp8aZ2lhWWkKt9xsaH1o03lP3vuTdcax/YSrrX5haq/2sf2EqF0//woaXIvk5cS/LZ43gs7eeN9XDGKAl744jKszJVGchGiPBmBCO5zmgAlgA/E4zPhS3r11i+vBZPmuE6bxx6PHAtliyf0rirYRi1L0C+ej1v9F/6Gje+CYPXVEhl87nmu45sC2WzipPU9pvP34HUD6I+g8dTWxKFYs3HGLn+hi0hfloC/PZEftPZr+6hefeTyQ3I9WUV0PXknfHsX3tkgbreezwHlPagotneSuhmKWfZjAwfCyzX92CX88AUr7ayoI139B/6Gi+2rTSst+8ACvanzmePmqm/H0V2ceSmBfhcU2PU833X1uYz5Z/zSc9KZ6d62OY9XIcj8Vs4st3l/Lrz98Tm1LFkNEPkpOe0uiwpL5UR9LOjSzecIi3EooJuimcnPQUUxs7l3OcCVH/JPqjI3j5atjyr/kUX72MZxc/Zr0cxx+G38XiDYd4YN6rzP3XZ0ycs5TbH34SgJtuHcf/vB0vw+uiyTp6MKYC1lJ7yxLjcQRl6xAh2ppVQDHQGXCjGR+KE+csJTalyhQoGRmHHkdOimLsjL/j6qbCt7s/g26bYJoTU3z1MkUFeaZ7Rk6KYsjoSbi6qfhz5DOcy06n4OJZTv54kOWzRpgCvp8T93I+J4Njh/fQK2ggPfqE4OqmInz8DFNeDV0bdnckE+csbbCeRnlns+n3x9twdVPh1zMATe++pjoXXDzLscN7CA6N4On/24Onj9qiX76wvP3Vp0efEKI/OsJjMZvYvnZJrV6qmu+/MXAbGD6WKX9fhaePGk3vQDS9+zLqwTmm/PLOZjc6LGmcTxYcGmFqR3lns01tLOimcPx6BtQq18O7a715afyDTP8puHThtOleIZqiowdjOmAOMB1YgrLqsRPgASRRve9bY9TAEy1QvwiqNxIWbce3mA/gW/Pwq1Ef44fiQuD9pv4QwaERNp98XFZajK6okKWfZpiCvtiUKoJDI8g7m43GP8jsfQ1da47+Q0dz8seD6Et1lJUodfHy1ZgCxviNK5o6fGTv97etH1a3P3OG3R1p6qX6Ou5Nq57WbSptYT6rnxpn6n2z1E23juPkjwfRFuZz6dwpCcZEs3T0YKw+xiBtCcoGuo31kD0J9LRxHdRAdKOphD3cQXXgbq+j5oSnMqAUWAk82iI/cSOKCvJQefnQ2c3jmh60mvLOZtebR0PXmio4NALf7v7Mi/BgwRgNd0yZR48+yp+vsfdl6acZxG9c0diSHvZ+f9v6YbP2py3M5+Bn75peGwNnXVEhZSXFzc2uyYzLXmx4KZJZL8eZet8s5emjxre7PyePHKKzuweyN6doDgnGGrbV8DXM8DWa6v8ZrkUZ5owGlhqOPShBVESNdDWHO0MMr6OBOw1pa5435usPxAF3AYeQoEzUNh+l99b4IfgG0A14HrjanIz0pTo+Xb3YorWT8s/lUFZSbJp70++Pt+Hb3Z/g0AjTPDGA0yeOcOFUJv2Hjmb72iWmICjjh30c2BZL/MZ/NXitOWrOVzP2yBl/TmOdevQJYcjoB5v98woTm7U/ox/3fVorOM5JT6FX0MAWHUo2PsU56+U4m5UTdudkvv34HekVE80m64w1LA84DwShBEwDAY3hWA0EADVnm8agBGgzgBFAKvAmSjCXC0wGxgD9UQKsSJSAbDIw3JDHm8DthmtxhjybvyqnaM9eA1xQeiKW0cgHoL5Ux9aVz3BgWyzANcMxE+cspaggj/eipwOY5mAZ05XptORmpPJz4l4K888z+9UtponJC8ZoAHgsZpPp3PjHo9m5PsZ0beKcpYx/PJoefUJ4LGaT6cGBiXOW1pr7Vd+1netjTHW5kHuCne8uNVtPAE3vwFoPJoAyt+2BecsAeGPuGM6cSGPkpCgC+g9p8Jcs6tWs9tcUd05fSMYP+0zv3chJUUxe+CZArfe/tLiII99+RnpSPFv+NZ9RD83lnYX3m/IJ6D/ElNa7a3eOfPsZPyfu5VzOcaa/EMuO2H+YXj/6vx+g8vIxtdOwsVNIid9CRUV5rTY2/vFo09pnZ06kUXz1MrNejiM4NMK0Ht7khW+a5ikOGf2gBGOi2WRneEUkSsBVd+EYNUpAlFDnmvG8MVAy9lyZW3jGeO1fKIHWQcO9RhEovV81xRry2oAEY+JaC4F11P8hWBWbUtWK1Wk7jid/RUD/IbV6Ok6fOEJnN5VpuLIxUWFOIP82NkTaXz20hfnkZqQyYNidFuch7a9jkmHKhmlQ5oLVnMwSgdJjdlcj96pRhi2NExF0KMHYIpThSOODAYHUfnjACWW+Won11Rft1Eps0BvR3lw4lUnCF+/VOqcv1ZFz7HvpqbAtaX/1OHnkkLQ1YREJxho2GWWYcg/V87pGowRpe+u5x7hcRhxKwFVzTCgTGIwyTLmI6icl/Q33CSEs1KNPCDffdi8LxmhMa6m99uhwbrxllEymFi2m5t6vJcVFTe6BFaImmTNmngqlFysKZe5XPjAO+BRl2LCh2Z4BgA9KIFbziSMV8HfgHZSg7FPD+RSUwGwSSgCnAsZy7dClEKIRw+6OlIU2Rasyrldm3ANTCEt09J4xYy/WJpThROMTkMUow4caqudrpQAPGq5/gzKc+DZKj9k+w/1HAHegEGUoswr4g+HaXwz5fGM4748ywT8TmEL18GUiyvpm+YayD1H95KYQQggh2hmZJChE+9NhJ1Dbgkygtpq0PytI++uYOnrPmBBCCCGEXUkwJkQ74+ziUt4a28i0R/pSHc4uLuX2rocjk/ZnOWl/HZcEY0K0Mx7eXdPTk+LtXQ2HlJ4Uj4d313R718ORSfuznLS/jkuCMSHamaKCvNc/WbWoUHonmkdfquOTVYsKiwryXrd3XRyZtD/LSPvr2FzsXQEhhM39VF5W0jt5d1x/dc8Ad3WvPrh0crV3ndosfamOYwm7WLNoUmHh72c/rCjXL7d3nRyctL9mkPYnhBDt2zQvX02as4uLnuplW+Soczi7uOi9fDVpwDTLf9XCDGl/0v6EEEIIIYQQQgghhBBCCCGEEEIIIYQQQgghhBBCCCGEEEIIIYQQQgghhBBCCCGEEEIIIYQQQgghhBBCCNGh/H/+apn9xRZVwgAAAABJRU5ErkJggg==)\n","\n","*Sample workflow for behavioral tests. Starting with shopping data (left), the dataset split and model training mimic the usual training loop. We also create a latent space, which is used to measure the relationships between inputs, ground truths and predictions, such as how far misses are from ground truths. Since a session can be viewed as a sequence of items or features (brands), we can use the same method to create embeddings for different tests.*"]},{"cell_type":"markdown","metadata":{"id":"O8ZUJHMrho_A"},"source":["### Inputs and outputs\n","\n","**Inputs:**\n","\n","1. Model `RecModel`\n","2. Dataset `RecDataset`\n","    - Public datasets - Coveo, MSD, MovieLens\n","3. Use Case `RecTask`\n","    - Similar items - {GT:👞} → {Pred:👟} is ✅, but {GT:👞} → {Pred:🩳} is 🟥\n","    - Complementary items - {GT:💻} → {Pred:⌨️} is ✅, but {GT:⌨️} → {Pred:💻} is 🟥\n","    - Session-based recommendations\n","4. Behavioral Tests `RecTest`\n","5. List of behavioral tests `RecList`\n","\n","**Outputs:**\n","\n","- Results of behavioral tests.\n","\n","### Modules\n","\n","- **P2VRecModel**: is a very basic recommender that builds a space of products and use Knn to run predictions\n","- **CoveoDataset**: is the dataset we are going to play with\n","- **train_embeddings**: is a function that allows us to train prod2vec embeddings\n","- **CoveoCartRecList**: is the RecList! this is a pre-made RecList for you to evaluate your models (or our P2VRecModel, in this tutorial) on the Coveo Dataset."]},{"cell_type":"markdown","metadata":{"id":"-NfELzGNIo2U"},"source":["## Setup"]},{"cell_type":"code","metadata":{"id":"7pTDbLiLPu_w"},"source":["!pip install gensim==4.0.1\n","!pip install jinja2==3.0.2\n","!pip install algoliasearch==2.6.0\n","!pip install appdirs==1.4.4\n","!pip install wget==3.2\n","!pip install pytest==6.2.5\n","!pip install requests==2.22.0\n","!pip install tqdm==4.62.3\n","!pip install matplotlib==3.4.3\n","!pip install numpy==1.21.2\n","!pip install pathos==0.2.8\n","!pip install flask==2.0.2\n","!pip install networkx==2.6.3\n","!pip install python-Levenshtein==0.12.2"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"4kCfrlxSIpU1"},"source":["import gensim\n","from abc import ABC, abstractmethod\n","import ast\n","from datetime import datetime\n","import inspect\n","import os\n","from abc import ABC, abstractmethod\n","from functools import wraps\n","from pathlib import Path\n","import time\n","import json\n","import tempfile\n","import zipfile\n","import random\n","import requests\n","from tqdm import tqdm\n","from enum import Enum\n","from typing import List\n","import itertools\n","import numpy as np\n","import collections\n","from scipy.spatial.distance import cosine\n","import matplotlib.pyplot as plt\n","from statistics import mean\n","import json\n","import networkx as nx\n","from networkx.algorithms.shortest_paths.generic import shortest_path\n","from collections import Counter, defaultdict\n","import math"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"fKz7N6DDKSAD"},"source":["## Dataset"]},{"cell_type":"code","metadata":{"id":"EXtGJMySIpXi"},"source":["COVEO_INTERACTION_DATASET_S3_URL = 'https://reclist-datasets-6d3c836d-6djh887d.s3.us-west-2.amazonaws.com/coveo_sigir.zip'"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"qo2bINDhKE2o"},"source":["def download_with_progress(url, destination):\n","    \"\"\"\n","    Downloads a file with a progress bar\n","    :param url: url from which to download from\n","    :destination: file path for saving data\n","    \"\"\"\n","    try:\n","        response = requests.get(url, stream=True)\n","        response.raise_for_status()\n","    except requests.exceptions.RequestException as e:\n","        raise SystemExit(e)\n","    with tqdm.wrapattr(open(destination, \"wb\"), \"write\",\n","                       miniters=1, desc=url.split('/')[-1],\n","                       total=int(response.headers.get('content-length', 0))) as fout:\n","        for chunk in response.iter_content(chunk_size=4096):\n","            fout.write(chunk)\n","\n","\n","def get_cache_directory():\n","    \"\"\"\n","    Returns the cache directory on the system\n","    \"\"\"\n","    cache_dir = '/content/coveo_reclist'\n","\n","    Path(cache_dir).mkdir(parents=True, exist_ok=True)\n","\n","    return cache_dir\n","\n","\n","class Dataset(Enum):\n","    COVEO = 'coveo'\n","    COVEO_INTERNAL = 'coveo-internal'"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"uteUWIghIpcg"},"source":["class RecDataset(ABC):\n","    \"\"\"\n","    Implements an abstract class for the dataset\n","    \"\"\"\n","    def __init__(self, force_download=False):\n","        \"\"\"\n","        :param force_download: allows to force the download of the dataset in case it is needed.\n","        :type: force_download: bool, optional\n","        \"\"\"\n","        self._x_train = None\n","        self._y_train = None\n","        self._x_test = None\n","        self._y_test = None\n","        self._catalog = None\n","        self.force_download = force_download\n","        self.load()\n","\n","    @abstractmethod\n","    def load(self):\n","        \"\"\"\n","        Abstract method that should implement dataset loading\n","        @return:\n","        \"\"\"\n","        return\n","\n","    @property\n","    def x_train(self):\n","        return self._x_train\n","\n","    @property\n","    def y_train(self):\n","        return self._y_train\n","\n","    @property\n","    def x_test(self):\n","        return self._x_test\n","\n","    @property\n","    def y_test(self):\n","        return self._y_test\n","\n","    @property\n","    def catalog(self):\n","        return self._catalog"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"iPzJOfxPIpg4"},"source":["class CoveoDataset(RecDataset):\n","    \"\"\"\n","    Coveo SIGIR data challenge dataset\n","    \"\"\"\n","    def __init__(self, **kwargs):\n","        super().__init__(**kwargs)\n","\n","    def load(self):\n","        cache_directory = get_cache_directory()\n","        filename = os.path.join(cache_directory, \"coveo_sigir.zip\")  # TODO: make var somewhere\n","\n","        if not os.path.exists(filename) or self.force_download:\n","            download_with_progress(COVEO_INTERACTION_DATASET_S3_URL, filename)\n","\n","        with tempfile.TemporaryDirectory() as temp_dir:\n","            with zipfile.ZipFile(filename, 'r') as zip_ref:\n","                zip_ref.extractall(temp_dir)\n","            with open(os.path.join(temp_dir, 'dataset.json')) as f:\n","                data = json.load(f)\n","\n","        self._x_train = data[\"x_train\"]\n","        self._y_train = None\n","        self._x_test = data[\"x_test\"]\n","        self._y_test = data[\"y_test\"]\n","        self._catalog = data[\"catalog\"]"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"yp337k5NKXr8"},"source":["## Model"]},{"cell_type":"code","metadata":{"id":"WxUtXx9YIpaF"},"source":["def train_embeddings(\n","    sessions: list,\n","    min_c: int = 3,\n","    size: int = 48,\n","    window: int = 5,\n","    iterations: int = 15,\n","    ns_exponent: float = 0.75,\n","    is_debug: bool = True):\n","    \"\"\"\n","    Train CBOW to get product embeddings with sensible defaults (https://arxiv.org/abs/2007.14906).\n","    :param sessions: list of lists, as user sessions are list of interactions\n","    :param min_c: minimum frequency of an event for it to be calculated for product embeddings\n","    :param size: output dimension\n","    :param window: window parameter for gensim word2vec\n","    :param iterations: number of training iterations\n","    :param ns_exponent: ns_exponent parameter for gensim word2vec\n","    :param is_debug: if true, be more verbose when training\n","    :return: trained product embedding model\n","    \"\"\"\n","    model = gensim.models.Word2Vec(sentences=sessions,\n","                                   min_count=min_c,\n","                                   vector_size=size,\n","                                   window=window,\n","                                   epochs=iterations,\n","                                   ns_exponent=ns_exponent)\n","\n","    if is_debug:\n","        print(\"# products in the space: {}\".format(len(model.wv.index_to_key)))\n","\n","    return model.wv"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"DpKCy3pkJKdS"},"source":["class RecModel(ABC):\n","    \"\"\"\n","    Abstract class for recommendation model\n","    \"\"\"\n","\n","    def __init__(self, model=None):\n","        \"\"\"\n","        :param model: a model that can be used in the predict function\n","        \"\"\"\n","        self._model = model\n","\n","    @abstractmethod\n","    def predict(self, prediction_input: list, *args, **kwargs):\n","        \"\"\"\n","        The predict function should implement the behaviour of the model at inference time.\n","        :param prediction_input: the input that is used to to do the prediction\n","        :param args:\n","        :param kwargs:\n","        :return:\n","        \"\"\"\n","        return NotImplementedError\n","\n","    @property\n","    def model(self):\n","        return self._model"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"2d4nsWPtK5_e"},"source":["class P2VRecModel(RecModel):\n","    \"\"\"\n","    Implement of the prod2vec model through the standard RecModel interface.\n","    Since init is ok, we just need to overwrite the prediction methods to get predictions\n","    out of it.\n","    \"\"\"\n","    def __init__(self, **kwargs):\n","        super().__init__(**kwargs)\n","\n","    model_name = \"prod2vec\"\n","    def predict(self, prediction_input: list, *args, **kwargs):\n","        \"\"\"\n","        Implement the abstract method, accepting a list of lists, each list being\n","        the content of a cart: the predictions returned by the model are the top K\n","        items suggested to complete the cart.\n","        \"\"\"\n","        predictions = []\n","        for _x in prediction_input:\n","            # we assume here that every X is a list of one-element, the product already in the cart\n","            # i.e. our prediction_input list is [[sku_1], [sku_3], ...]\n","            key_item = _x[0]['product_sku']\n","            nn_products = self._model.most_similar(key_item, topn=10) if key_item in self._model else None\n","            if nn_products:\n","                predictions.append([{'product_sku':_[0]} for _ in nn_products])\n","            else:\n","                predictions.append([])\n","\n","        return predictions\n","\n","    def get_vector(self, product_sku):\n","        try:\n","            return list(self._model.get_vector(product_sku))\n","        except Exception as e:\n","            return []"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"1PdrG6g5LON2"},"source":["## Metrics"]},{"cell_type":"markdown","metadata":{"id":"a4jTjpDbVPxJ"},"source":["### Standard Metrics"]},{"cell_type":"code","metadata":{"id":"mY8Fb_W4LN9x"},"source":["def statistics(x_train, y_train, x_test, y_test, y_pred):\n","    train_size = len(x_train)\n","    test_size = len(x_test)\n","    # num non-zero preds\n","    num_preds = len([p for p in y_pred if p])\n","    return {\n","        'training_set__size': train_size,\n","        'test_set_size': test_size,\n","        'num_non_null_predictions': num_preds\n","    }\n","\n","\n","def sample_hits_at_k(y_preds, y_test, x_test=None, k=3, size=3):\n","    hits = []\n","    for idx, (_p, _y) in enumerate(zip(y_preds, y_test)):\n","        if _y[0] in _p[:k]:\n","            hit_info = {\n","                'Y_TEST': [_y[0]],\n","                'Y_PRED': _p[:k],\n","            }\n","            if x_test:\n","                hit_info['X_TEST'] = [x_test[idx][0]]\n","            hits.append(hit_info)\n","\n","    if len(hits) < size or size == -1:\n","        return hits\n","    return random.sample(hits, k=size)\n","\n","\n","def sample_misses_at_k(y_preds, y_test, x_test=None, k=3, size=3):\n","    misses = []\n","    for idx, (_p, _y) in enumerate(zip(y_preds, y_test)):\n","        if _y[0] not in _p[:k]:\n","            miss_info =  {\n","                'Y_TEST': [_y[0]],\n","                'Y_PRED': _p[:k],\n","            }\n","            if x_test:\n","                miss_info['X_TEST'] = [x_test[idx][0]]\n","            misses.append(miss_info)\n","\n","    if len(misses) < size or size == -1:\n","        return misses\n","    return random.sample(misses, k=size)\n","\n","\n","def hit_rate_at_k(y_preds, y_test, k=3):\n","    hits = 0\n","    for _p, _y in zip(y_preds, y_test):\n","        if _y[0] in _p[:k]:\n","            hits += 1\n","    return hits / len(y_test)\n","\n","\n","def mrr_at_k(y_preds, y_test, k=3):\n","    \"\"\"\n","    Computes MRR\n","    :param y_preds: predictions, as lists of lists\n","    :param y_test: target data, as lists of lists (eventually [[sku1], [sku2],...]\n","    :param k: top-k\n","    \"\"\"\n","    rr = []\n","    y_test = [k[0] for k in y_test]\n","    for _p, _y in zip(y_preds, y_test):\n","        if _y in _p[:k+1]:\n","            rank = _p[:k+1].index(_y) + 1\n","            rr.append(1/rank)\n","        else:\n","            rr.append(0)\n","    return np.mean(rr)\n","\n","\n","def coverage_at_k(y_preds, product_data, k=3):\n","    pred_skus = set(itertools.chain.from_iterable(y_preds[:k]))\n","    all_skus = set(product_data.keys())\n","    nb_overlap_skus = len(pred_skus.intersection(all_skus))\n","\n","    return nb_overlap_skus / len(all_skus)\n","\n","\n","def popularity_bias_at_k(y_preds, x_train, k=3):\n","    # estimate popularity from training data\n","    pop_map = collections.defaultdict(lambda : 0)\n","    num_interactions = 0\n","    for session in x_train:\n","        for event in session:\n","            pop_map[event] += 1\n","            num_interactions += 1\n","    # normalize popularity\n","    pop_map = {k:v/num_interactions for k,v in pop_map.items()}\n","    all_popularity = []\n","    for p in y_preds:\n","        average_pop = sum(pop_map.get(_, 0.0) for _ in p[:k]) / len(p) if len(p) > 0 else 0\n","        all_popularity.append(average_pop)\n","    return sum(all_popularity) / len(y_preds)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"Ydqnq1OIVUIF"},"source":["### Cosine Distance Metrics"]},{"cell_type":"code","metadata":{"id":"yaCxpR7bLlWf"},"source":["def error_by_cosine_distance(model, y_test, y_preds, k=3, bins=25, debug=False):\n","    if not(hasattr(model.__class__, 'get_vector') and callable(getattr(model.__class__, 'get_vector'))):\n","        error_msg = \"Error : Model {} does not support retrieval of vector embeddings\".format(model.__class__)\n","        print(error_msg)\n","        return error_msg\n","    misses = sample_misses_at_k(y_preds, y_test, k=k, size=-1)\n","    cos_distances = []\n","    for m in misses:\n","        if m['Y_PRED']:\n","            vector_test = model.get_vector(m['Y_TEST'][0])\n","            vector_pred = model.get_vector(m['Y_PRED'][0])\n","            if vector_pred and vector_test:\n","                cos_dist = cosine(vector_pred, vector_test)\n","                cos_distances.append(cos_dist)\n","\n","    histogram = np.histogram(cos_distances, bins=bins, density=False)\n","    # cast to list\n","    histogram = (histogram[0].tolist(), histogram[1].tolist())\n","    # debug / viz\n","    if debug:\n","        plt.hist(cos_distances, bins=bins)\n","        plt.title('dist over cosine distance prod space')\n","        plt.show()\n","\n","    return {'mean': np.mean(cos_distances), 'histogram': histogram}\n","\n","def distance_to_query(model, x_test, y_test, y_preds, k=3, bins=25, debug=False):\n","    if not(hasattr(model.__class__, 'get_vector') and callable(getattr(model.__class__, 'get_vector'))):\n","        error_msg = \"Error : Model {} does not support retrieval of vector embeddings\".format(model.__class__)\n","        print(error_msg)\n","        return error_msg\n","    misses = sample_misses_at_k(y_preds, y_test, x_test=x_test, k=k, size=-1)\n","    x_to_y_cos = []\n","    x_to_p_cos = []\n","    for m in misses:\n","        if m['Y_PRED']:\n","            vector_x = model.get_vector(m['X_TEST'][0])\n","            vector_y = model.get_vector(m['Y_TEST'][0])\n","            vectors_p = [model.get_vector(_) for _ in m['Y_PRED']]\n","            c_dists =[]\n","            if not vector_x or not vector_y:\n","                continue\n","            x_to_y_cos.append(cosine(vector_x, vector_y))\n","            for v_p in vectors_p:\n","                if not v_p:\n","                    continue\n","                cos_dist = cosine(v_p, vector_x)\n","                if cos_dist:\n","                    c_dists.append(cos_dist)\n","            if c_dists:\n","                x_to_p_cos.append(mean(c_dists))\n","\n","    h_xy = np.histogram(x_to_y_cos, bins=bins, density=False)\n","    h_xp = np.histogram(x_to_p_cos, bins=bins, density=False)\n","\n","    h_xy = (h_xy[0].tolist(), h_xy[1].tolist())\n","    h_xp = (h_xp[0].tolist(), h_xp[1].tolist())\n","\n","    # debug / viz\n","    if debug:\n","        plt.hist(x_to_y_cos, bins=bins, alpha=0.5)\n","        plt.hist(x_to_p_cos, bins=bins, alpha=0.5)\n","        plt.title('distribution of distance to input')\n","        plt.legend(['Distance from Input to Label',\n","                    'Distance from Input to Label'],\n","                   loc='upper right')\n","        plt.show()\n","\n","    return {\n","        'histogram_x_to_y': h_xy,\n","        'histogram_x_to_p': h_xp,\n","        'raw_distances_x_to_y': x_to_y_cos,\n","        'raw_distances_x_to_p': x_to_p_cos,\n","    }"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"GhZatSAhVWq_"},"source":["### Generic Cosine Distance"]},{"cell_type":"code","metadata":{"id":"5DOZxRQyLqRk"},"source":["def generic_cosine_distance(embeddings: dict,\n","                            type_fn,\n","                            y_test,\n","                            y_preds,\n","                            k=10,\n","                            bins=25,\n","                            debug=False):\n","\n","    misses = sample_misses_at_k(y_preds, y_test, k=k, size=-1)\n","    cos_distances = []\n","    for m in misses:\n","        if m['Y_TEST'] and m['Y_PRED'] and type_fn(m['Y_TEST'][0]) and type_fn(m['Y_PRED'][0]):\n","            vector_test = embeddings.get(type_fn(m['Y_TEST'][0]), None)\n","            vector_pred = embeddings.get(type_fn(m['Y_PRED'][0]), None)\n","            if vector_pred and vector_test:\n","                cos_dist = cosine(vector_pred, vector_test)\n","                cos_distances.append(cos_dist)\n","\n","    # TODO: Maybe sample some examples from the bins\n","    histogram = np.histogram(cos_distances, bins=bins, density=False)\n","    # cast to list\n","    histogram = (histogram[0].tolist(), histogram[1].tolist())\n","    # debug / viz\n","    if debug:\n","        plt.hist(cos_distances, bins=bins)\n","        plt.title('cosine distance misses')\n","        plt.show()\n","    return {'mean': np.mean(cos_distances), 'histogram': histogram}"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"ujXEtRR9VddC"},"source":["### Graph Distance Test"]},{"cell_type":"code","metadata":{"id":"6evcPhDjLzBL"},"source":["def shortest_path_length():\n","    pass\n","\n","\n","get_nodes = lambda nodes, ancestors=\"\": [] if not nodes else ['_'.join([ancestors, nodes[0]])] + \\\n","                                                             get_nodes(nodes[1:], '_'.join([ancestors, nodes[0]]))\n","\n","\n","def graph_distance_test(y_test, y_preds, product_data, k=3):\n","    path_lengths = []\n","    misses = sample_misses_at_k(y_preds, y_test, k=k, size=-1)\n","    for _y, _y_p in zip([_['Y_TEST'] for _ in misses],\n","                        [_['Y_PRED'] for _ in misses]):\n","        if not _y_p:\n","            continue\n","        _y_sku = _y[0]\n","        _y_p_sku = _y_p[0]\n","\n","        if _y_sku not in product_data or _y_p_sku not in product_data:\n","            continue\n","        if not product_data[_y_sku]['CATEGORIES'] or not product_data[_y_p_sku]['CATEGORIES']:\n","            continue\n","        # extract graph information\n","        catA = json.loads(product_data[_y_sku]['CATEGORIES'])\n","        catB = json.loads(product_data[_y_p_sku]['CATEGORIES'])\n","        catA_nodes = [get_nodes(c) for c in catA]\n","        catB_nodes = [get_nodes(c) for c in catB]\n","        all_nodes = list(set([n for c in catA_nodes + catB_nodes for n in c]))\n","        all_edges = [(n1, n2) for c in catA_nodes + catB_nodes for n1, n2 in zip(c[:-1], c[1:])]\n","        all_edges = list(set(all_edges))\n","\n","        # build graph\n","        G = nx.Graph()\n","        G.add_nodes_from(all_nodes)\n","        G.add_edges_from(all_edges)\n","\n","        # get leaves\n","        cat1_leaves = [c[-1] for c in catA_nodes]\n","        cat2_leaves = [c[-1] for c in catB_nodes]\n","\n","        all_paths = [shortest_path(G, c1_l, c2_l) for c1_l in cat1_leaves for c2_l in cat2_leaves]\n","        s_path = min(all_paths, key=len)\n","        s_path_len = len(s_path) - 1\n","        path_lengths.append(s_path_len)\n","\n","    histogram = np.histogram(path_lengths, bins=np.arange(0, max(path_lengths)))\n","    histogram = (histogram[0].tolist(), histogram[1].tolist())\n","    return {'mean': mean(path_lengths), 'hist': histogram}"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"ScJfIPh6VfsX"},"source":["### Hits Distribution"]},{"cell_type":"code","metadata":{"id":"jzUPryhgLy_E"},"source":["def roundup(x: int):\n","    div = 10.0 ** (len(str(x)))\n","    return int(math.ceil(x / div)) * div\n","\n","\n","def hits_distribution(x_train, x_test, y_test, y_preds, k=3, debug=False):\n","    # get product interaction frequency\n","    prod_interaction_cnt = Counter([_ for x in x_train for _ in x])\n","    hit_per_interaction_cnt = defaultdict(list)\n","    for _x, _y_test, _y_pred in zip(x_test, y_test, y_preds):\n","        _x_cnt = prod_interaction_cnt[_x[0]]\n","        # TODO: allow for generic metric\n","        hit_per_interaction_cnt[_x_cnt].append(hit_rate_at_k([_y_pred], [_y_test], k=k))\n","    # get max product frequency\n","    max_cnt = prod_interaction_cnt.most_common(1)[0][1]\n","    # round up to nearest place\n","    max_cnt = int(roundup(max_cnt))\n","    # generate log-bins\n","    indices = np.logspace(1, np.log10(max_cnt), num=int(np.log10(max_cnt))).astype(np.int64)\n","    indices = np.concatenate(([0], indices))\n","    counts_per_bin = [[_ for i in range(low, high) for _ in hit_per_interaction_cnt[i]]\n","                      for low, high in zip(indices[:-1], indices[1:])]\n","\n","    histogram = [np.mean(counts) if counts else 0 for counts in counts_per_bin]\n","    count = [len(counts) for counts in counts_per_bin]\n","\n","    if debug:\n","        # debug / visualization\n","        plt.bar(indices[1:], histogram, width=-np.diff(indices)/1.05, align='edge')\n","        plt.xscale('log', base=10)\n","        plt.title('HIT Distribution Across Product Frequency')\n","        plt.show()\n","\n","    return {\n","             'histogram': {int(k): v for k, v in zip(indices[1:], histogram)},\n","             'counts':  {int(k): v for k, v in zip(indices[1:], count)}\n","           }"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"mKFUHI6rVj11"},"source":["### Hits Distribution by Data slice"]},{"cell_type":"code","metadata":{"id":"uXAIa7lkLy82"},"source":["def hits_distribution_by_slice(slice_fns: dict,\n","                               y_test,\n","                               y_preds,\n","                               product_data,\n","                               k=3,\n","                               sample_size=3,\n","                               debug=False):\n","\n","    hit_rate_per_slice = defaultdict(dict)\n","    for slice_name, filter_fn in slice_fns.items():\n","        # get indices for slice\n","        slice_idx = [idx for idx,_y in enumerate(y_test) if _y[0] in product_data and filter_fn(product_data[_y[0]])]\n","        # get predictions for slice\n","        slice_y_preds = [y_preds[_] for _ in slice_idx]\n","        # get labels for slice\n","        slice_y_test = [y_test[_] for _ in slice_idx]\n","        # TODO: We may want to allow for generic metric to be used here\n","        slice_hr = hit_rate_at_k(slice_y_preds, slice_y_test,k=3)\n","        # store results\n","        hit_rate_per_slice[slice_name]['hit_rate'] = slice_hr\n","        hit_rate_per_slice[slice_name]['hits'] = sample_hits_at_k(slice_y_preds, slice_y_test, k=k, size=sample_size)\n","        hit_rate_per_slice[slice_name]['misses'] = sample_misses_at_k(slice_y_preds, slice_y_test, k=k, size=sample_size)\n","\n","    # debug / visualization\n","    if debug:\n","        x_tick_names = list(hit_rate_per_slice.keys())\n","        x_tick_idx = list(range(len(x_tick_names)))\n","        plt.bar(x_tick_idx, hit_rate_per_slice.values(), align='center')\n","        plt.xticks(list(range(len(hit_rate_per_slice))), x_tick_names)\n","        plt.show()\n","\n","    # cast to normal dict\n","    return dict(hit_rate_per_slice)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"_DX1-DeLVqoJ"},"source":["### Perturbations"]},{"cell_type":"code","metadata":{"id":"55hBX3UkL_ZI"},"source":["# TODO: We might need to enforce some standardization of how CATEGORY is represented\n","def get_item_with_category(product_data: dict, category: set, to_ignore=None):\n","    to_ignore = [] if to_ignore is None else to_ignore\n","    skus = [_ for _ in product_data if product_data[_]['category_hash'] == category and _ not in to_ignore]\n","    if skus:\n","        return random.choice(skus)\n","    return []\n","\n","\n","def perturb_session(session, product_data):\n","    last_item = session[-1]\n","    if last_item not in product_data:\n","        return []\n","    last_item_category = product_data[last_item]['category_hash']\n","    similar_item = get_item_with_category(product_data, last_item_category, to_ignore=[last_item])\n","    if similar_item:\n","        new_session = session[:-1] + [similar_item]\n","        return new_session\n","    return []\n","\n","\n","def session_perturbation_test(model, x_test, y_preds, product_data):\n","    overlap_ratios = []\n","    # print(product_data)\n","    y_p = []\n","    s_perturbs = []\n","\n","    # generate a batch of perturbations\n","    for idx, (s, _y_p) in enumerate(tqdm(zip(x_test,y_preds))):\n","        # perturb last item in session\n","        s = [ _.split('_')[0] for _ in s]\n","        s_perturb = perturb_session(s, product_data)\n","        if not s_perturb:\n","            continue\n","\n","        s_perturb = ['_'.join([_,'add']) for _ in s_perturb]\n","\n","        s_perturbs.append(s_perturb)\n","        y_p.append(_y_p)\n","\n","    y_perturbs = model.predict(s_perturbs)\n","\n","    for _y_p, _y_perturb in zip(y_p, y_perturbs):\n","        if _y_p and _y_perturb:\n","            # compute prediction intersection\n","            intersection = set(_y_perturb).intersection(_y_p)\n","            overlap_ratio = len(intersection)/len(_y_p)\n","            overlap_ratios.append(overlap_ratio)\n","\n","    return np.mean(overlap_ratios)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"_FM1vcOUVu4I"},"source":["### Price Homogeneity"]},{"cell_type":"code","metadata":{"id":"yg06OFvWL_Wk"},"source":["def price_homogeneity_test(y_test, y_preds, product_data, bins=25, debug=True, key='PRICE'):\n","    abs_log_price_diff = []\n","    for idx, (_y, _y_pred) in enumerate(zip(y_test, y_preds)):\n","        # need >=1 predictions\n","        if not _y_pred:\n","            continue\n","        # item must be in product data\n","        if _y[0] not in product_data or _y_pred[0] not in product_data:\n","            continue\n","        if product_data[_y[0]][key] and product_data[_y_pred[0]][key]:\n","            pred_item_price = float(product_data[_y_pred[0]][key])\n","            y_item_price = float(product_data[_y[0]][key])\n","            if pred_item_price and y_item_price:\n","                abs_log_price_diff.append(np.abs(np.log10(pred_item_price)-(np.log10(y_item_price))))\n","\n","    histogram = np.histogram(abs_log_price_diff, bins=bins, density=False)\n","    histogram = (histogram[0].tolist(), histogram[1].tolist())\n","    if debug:\n","        # debug / viz\n","        plt.hist(abs_log_price_diff, bins=25)\n","        plt.show()\n","    return {\n","        'mean': np.mean(abs_log_price_diff),\n","        'histogram': histogram\n","    }"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"uG7H_aLoK0eA"},"source":["## RecList"]},{"cell_type":"code","metadata":{"id":"BcSsb7gEJJUO"},"source":["def rec_test(test_type: str):\n","    \"\"\"\n","    Rec test decorator\n","    \"\"\"\n","    def decorator(f):\n","        @wraps(f)\n","        def w(*args, **kwargs):\n","            return f(*args, **kwargs)\n","\n","        # add attributes to f\n","        w.is_test = True\n","        w.test_type = test_type\n","        try:\n","            w.test_desc = f.__doc__.lstrip().rstrip()\n","        except:\n","            w.test_desc = \"\"\n","        try:\n","            # python 3\n","            w.name = w.__name__\n","        except:\n","            # python 2\n","            w.name = w.__func__.func_name\n","        return w\n","\n","    return decorator"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"DrtqbN0XJGA8"},"source":["class RecList(ABC):\n","    \n","    META_DATA_FOLDER = '.reclist'\n","\n","    def __init__(self, model: RecModel, dataset: RecDataset, y_preds: list = None):\n","        \"\"\"\n","        :param model:\n","        :param dataset:\n","        :param y_preds:\n","        \"\"\"\n","        self.name = self.__class__.__name__\n","        self._rec_tests = self.get_tests()\n","        self._x_train = dataset.x_train\n","        self._y_train = dataset.y_train\n","        self._x_test = dataset.x_test\n","        self._y_test = dataset.y_test\n","        self._y_preds = y_preds if y_preds else model.predict(dataset.x_test)\n","        self.rec_model = model\n","        self.product_data = dataset.catalog\n","        self._test_results = []\n","        self._test_data = {}\n","        self._dense_repr = {}\n","\n","        assert len(self._y_test) == len(self._y_preds)\n","\n","    def train_dense_repr(self, type_name: str, type_fn):\n","        \"\"\"\n","        Train a dense representation over a type of meta-data & store into object\n","        \"\"\"\n","        # type_fn: given a SKU returns some type i.e. brand\n","        x_train_transformed = [[type_fn(e) for e in session if type_fn(e)] for session in self._x_train]\n","        wv = train_embeddings(x_train_transformed)\n","        # store a dict\n","        self._dense_repr[type_name] = {word: list(wv.get_vector(word)) for word in wv.key_to_index}\n","\n","    def get_tests(self):\n","        \"\"\"\n","        Helper to extract methods decorated with rec_test\n","        \"\"\"\n","\n","        nodes = {}\n","        for _ in self.__dir__():\n","            if not hasattr(self,_):\n","                continue\n","            func = getattr(self, _)\n","            if hasattr(func, 'is_test'):\n","                nodes[func.name] = func\n","\n","        return nodes\n","\n","    def __call__(self, verbose=True, *args, **kwargs):\n","        run_epoch_time_ms = round(time.time() * 1000)\n","        # iterate through tests\n","        for test_func_name, test in self._rec_tests.items():\n","            test_result = test(*args, **kwargs)\n","            # we could store the results in the test function itself\n","            # test.__func__.test_result = test_result\n","            self._test_results.append({\n","                'test_name': test.test_type,\n","                'description': test.test_desc,\n","                'test_result': test_result}\n","            )\n","            if verbose:\n","                print(\"============= TEST RESULTS ===============\")\n","                print(\"Test Type        : {}\".format(test.test_type))\n","                print(\"Test Description : {}\".format(test.test_desc))\n","                print(\"Test Result      : {}\\n\".format(test_result))\n","        # at the end, we dump it locally\n","        if verbose:\n","            print(\"Generating reports at {}\".format(datetime.utcnow()))\n","        self.generate_report(run_epoch_time_ms)\n","\n","    def generate_report(self, epoch_time_ms: int):\n","        # create path first: META_DATA_FOLDER / RecList / Model / Run Time\n","        report_path = os.path.join(\n","            self.META_DATA_FOLDER,\n","            self.name,\n","            self.rec_model.__class__.__name__,\n","            str(epoch_time_ms)\n","        )\n","        # now, dump results\n","        self.dump_results_to_json(self._test_results, report_path, epoch_time_ms)\n","        # now, store artifacts\n","        self.store_artifacts(report_path)\n","\n","    def store_artifacts(self, report_path: str):\n","        target_path = os.path.join(report_path, 'artifacts')\n","        # make sure the folder is there, with all intermediate parents\n","        Path(target_path).mkdir(parents=True, exist_ok=True)\n","        # store predictions\n","        with open(os.path.join(target_path, 'model_predictions.json'), 'w') as f:\n","            json.dump({\n","                'x_test': self._x_test,\n","                'y_test': self._y_test,\n","                'y_preds': self._y_preds\n","            }, f)\n","\n","    def dump_results_to_json(self, test_results: list, report_path: str, epoch_time_ms: int):\n","        target_path = os.path.join(report_path, 'results')\n","        # make sure the folder is there, with all intermediate parents\n","        Path(target_path).mkdir(parents=True, exist_ok=True)\n","        report = {\n","            'metadata': {\n","                'run_time': epoch_time_ms,\n","                'model_name': self.rec_model.__class__.__name__,\n","                'reclist': self.name,\n","                'tests': list(self._rec_tests.keys())\n","            },\n","            'data': test_results\n","        }\n","        with open(os.path.join(target_path, 'report.json'), 'w') as f:\n","            json.dump(report, f)\n","\n","    @property\n","    def test_results(self):\n","        return self._test_results\n","\n","    @property\n","    def test_data(self):\n","        return self._test_data\n","\n","    @property\n","    def rec_tests(self):\n","        return self._rec_tests"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"lmIgV9TgIpjb"},"source":["class CoveoCartRecList(RecList):\n","\n","    @rec_test(test_type='stats')\n","    def basic_stats(self):\n","        \"\"\"\n","        Basic statistics on training, test and prediction data\n","        \"\"\"\n","        return statistics(self._x_train,\n","                          self._y_train,\n","                          self._x_test,\n","                          self._y_test,\n","                          self._y_preds)\n","\n","    @rec_test(test_type='Coverage@10')\n","    def coverage_at_k(self):\n","        \"\"\"\n","        Coverage is the proportion of all possible products which the RS\n","        recommends based on a set of sessions\n","        \"\"\"\n","        return coverage_at_k(self.sku_only(self._y_preds),\n","                             self.product_data,\n","                             k=10)\n","\n","    @rec_test(test_type='HR@10')\n","    def hit_rate_at_k(self):\n","        \"\"\"\n","        Compute the rate in which the top-k predictions contain the item to be predicted\n","        \"\"\"\n","        return hit_rate_at_k(self.sku_only(self._y_preds),\n","                             self.sku_only(self._y_test),\n","                             k=10)\n","\n","    @rec_test(test_type='hits_distribution')\n","    def hits_distribution(self):\n","        \"\"\"\n","        Compute the distribution of hit-rate across product frequency in training data\n","        \"\"\"\n","        return hits_distribution(self.sku_only(self._x_train),\n","                                 self.sku_only(self._x_test),\n","                                 self.sku_only(self._y_test),\n","                                 self.sku_only(self._y_preds),\n","                                 k=10,\n","                                 debug=True)\n","\n","    @rec_test(test_type='distance_to_query')\n","    def dist_to_query(self):\n","        \"\"\"\n","        Compute the distribution of distance from query to label and query to prediction\n","        \"\"\"\n","        return distance_to_query(self.rec_model,\n","                                 self.sku_only(self._x_test),\n","                                 self.sku_only(self._y_test),\n","                                 self.sku_only(self._y_preds), k=10, bins=25, debug=True)\n","\n","    def sku_only(self, l:List[List]):\n","        return [[e['product_sku'] for e in s] for s in l]"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"3InDare_Ipl_"},"source":["## Runs"]},{"cell_type":"markdown","metadata":{"id":"pYAZTgU3Ti5N"},"source":["Let's download the data, and load it. If dataset is already downloaded, it will automatically skip the downloading."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"oFAH7Bm-Ipoq","executionInfo":{"status":"ok","timestamp":1637594039574,"user_tz":-330,"elapsed":165506,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"070269cb-e712-474e-8736-638cac5732a3"},"source":["coveo_dataset = CoveoDataset()"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stderr","text":["coveo_sigir.zip: 100%|██████████| 1.91G/1.91G [02:04<00:00, 16.5MB/s]\n"]}]},{"cell_type":"markdown","metadata":{"id":"PHUhFcAJT4p_"},"source":["Let's see what's inside the dataset, there is a lot of information here. We are seeing list of sessions; each element in the list is a user session, we know what the user was doing (e.g., \"detail\") and which product he/she/they were looking at."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"e-co-IK3Ipq-","executionInfo":{"status":"ok","timestamp":1637594845203,"user_tz":-330,"elapsed":20,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"c27cb12c-fc83-47e2-df06-76461e8c5ee7"},"source":["coveo_dataset.x_train[0]"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["[{'event_type': 'event_product',\n","  'hashed_url': '803f6c2d4202e39d6d7fdb232d69366b86bc843869c809f1e1954465bfc6e17f',\n","  'product_action': 'detail',\n","  'product_sku': '624bc145579b67b608e6a7b0d0516cc75e0ec4cbe44ec42c6ac53cc83925bc3e',\n","  'server_timestamp_epoch_ms': '1547528580651',\n","  'session_id': '0f1416c8c68bb9209c1bbc4576386df5480e9757f55ce9cb0d4d4017cf14fc1c'},\n"," {'event_type': 'event_product',\n","  'hashed_url': '80ee4de500233dc623dde6a2730d4bb61e651e5ae3fe33abd985598551de253e',\n","  'product_action': 'detail',\n","  'product_sku': '3b59c7591dd71a2ade8df707e4b1d4e65ef6d1b165aabf54191feccf056b93df',\n","  'server_timestamp_epoch_ms': '1547528590750',\n","  'session_id': '0f1416c8c68bb9209c1bbc4576386df5480e9757f55ce9cb0d4d4017cf14fc1c'},\n"," {'event_type': 'event_product',\n","  'hashed_url': 'ca21b70cf35319e289af06659c222a0fa24b6e9414f1d3d32a3ec86a1ecc0a44',\n","  'product_action': 'detail',\n","  'product_sku': '983d92391f7db7d45005e0d0c61387dea6a05116ae5c5260b90c82a55dc15121',\n","  'server_timestamp_epoch_ms': '1547528595759',\n","  'session_id': '0f1416c8c68bb9209c1bbc4576386df5480e9757f55ce9cb0d4d4017cf14fc1c'}]"]},"metadata":{},"execution_count":10}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"HfUmzC40Ipte","executionInfo":{"status":"ok","timestamp":1637595735164,"user_tz":-330,"elapsed":637,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"82b62cd6-e874-4aad-ebd9-9fff49da81fd"},"source":["x_train_skus = [[e['product_sku'] for e in s] for s in coveo_dataset.x_train]\n","x_train_skus[0:3]"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["[['624bc145579b67b608e6a7b0d0516cc75e0ec4cbe44ec42c6ac53cc83925bc3e',\n","  '3b59c7591dd71a2ade8df707e4b1d4e65ef6d1b165aabf54191feccf056b93df',\n","  '983d92391f7db7d45005e0d0c61387dea6a05116ae5c5260b90c82a55dc15121'],\n"," ['d6392b60c1bc118a06cdb12caacbc45513cd5d9d9d4aef0826c4b9015cf70d1d',\n","  '605d79343281f976747693e6bbb5757947d9842e02d21bcaa6a02edee3ecd9a9',\n","  'd6392b60c1bc118a06cdb12caacbc45513cd5d9d9d4aef0826c4b9015cf70d1d'],\n"," ['c3b0916415e119da78c0746cfaeaa6dc806f3076718c9c703337d75d9d61eb4f',\n","  'c17daa37dc5a0199940118e7edb402526fbca70929086f2e7ba042c5af2299b3',\n","  '859e3db8924990d6fb811e67a2f88f4711b6b9407b271c39f6a16838dcf22dcf']]"]},"metadata":{},"execution_count":29}]},{"cell_type":"markdown","metadata":{"id":"arJE3tvJUGKS"},"source":["Let's build a vanilla recommender with a KNN product based model!"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Mfl5t16eOUo0","executionInfo":{"status":"ok","timestamp":1637594975286,"user_tz":-330,"elapsed":115592,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"51f7a225-c139-4fda-e228-2055439e4f56"},"source":["embeddings = train_embeddings(sessions=x_train_skus)"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["# products in the space: 27562\n"]}]},{"cell_type":"code","metadata":{"id":"mYJ6YEolPEBh"},"source":["model = P2VRecModel(model=embeddings)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"vIkT1nQPUJHh"},"source":["Let's see if our model is working, we will try to do a simple prediction starting from a single product."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"mF1O-aF-Q2HU","executionInfo":{"status":"ok","timestamp":1637595138472,"user_tz":-330,"elapsed":643,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"621102b7-6f29-4940-be18-320c453d6efe"},"source":["test = [[{'event_type': 'event_product',\n","  'hashed_url': '803f6c2d4202e39d6d7fdb232d69366b86bc843869c809f1e1954465bfc6e17f',\n","  'product_action': 'detail',\n","  'product_sku': '624bc145579b67b608e6a7b0d0516cc75e0ec4cbe44ec42c6ac53cc83925bc3e',\n","  'server_timestamp_epoch_ms': '1547528580651',\n","  'session_id': '0f1416c8c68bb9209c1bbc4576386df5480e9757f55ce9cb0d4d4017cf14fc1c'}]]\n","\n","\n","model.predict(test)"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["[[{'product_sku': '064ccca34647e8c65f7421a22cc520e40646657322b2196218a99b9dc54417d9'},\n","  {'product_sku': '52c7818bf0ce31793a43319fac4136404c7df939152f3d91e9b193e86d20ef64'},\n","  {'product_sku': '39cd17df8567532b9db06983e6d5312c70f61b1d9276deef0343514cd83809ab'},\n","  {'product_sku': '2a112f0436d6d8b8cbecb61acca3fa88383ff76b05053d975caa40482e9736ed'},\n","  {'product_sku': '704aac67713b93c893912c3d10d9751af53de51c5f05f24afb702080545212b5'},\n","  {'product_sku': '1e7a9c74d8769294783c0198d649ecba4caea98c53de129c51a38f7566d1b09c'},\n","  {'product_sku': 'b42787bd0decd82b42906b2488a8dfb99ba9bbd717113c3c5d808eeb0265bec2'},\n","  {'product_sku': 'e5a80ede4427db027b53a73ed6943c1e9f378ddbbfe49e523dfaefaca660785e'},\n","  {'product_sku': 'eb083190e2fd4e8ea272786338092674430d8e4b51621e102816ad757aecee5f'},\n","  {'product_sku': '962122e7a9ded48c578e7f0f167d61c94d5bfc7e077c32ee533b42b7b4fb5833'}]]"]},"metadata":{},"execution_count":14}]},{"cell_type":"markdown","metadata":{"id":"CRQF4bYLUNDX"},"source":["Instantiate rec_list object, prepared with standard quantitative tests and sensible behavioral tests. Then, invoke rec_list to run tests."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":1000},"id":"t_6fqcguSEUG","executionInfo":{"status":"ok","timestamp":1637595329897,"user_tz":-330,"elapsed":10310,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"31e53315-0364-4f41-8759-db41e6a4ecc1"},"source":["# instantiate rec_list object\n","rec_list = CoveoCartRecList(\n","    model=model,\n","    dataset=coveo_dataset\n",")\n","# invoke rec_list to run tests\n","rec_list(verbose=True)"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["============= TEST RESULTS ===============\n","Test Type        : stats\n","Test Description : Basic statistics on training, test and prediction data\n","Test Result      : {'training_set__size': 927357, 'test_set_size': 790, 'num_non_null_predictions': 762}\n","\n","============= TEST RESULTS ===============\n","Test Type        : Coverage@10\n","Test Description : Coverage is the proportion of all possible products which the RS\n","        recommends based on a set of sessions\n","Test Result      : 0.0015063416985508992\n","\n","============= TEST RESULTS ===============\n","Test Type        : HR@10\n","Test Description : Compute the rate in which the top-k predictions contain the item to be predicted\n","Test Result      : 0.12151898734177215\n","\n"]},{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAXoAAAEMCAYAAADK231MAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAbHklEQVR4nO3df5xcdX3v8debxEQECQpbrvkBCQ22BlHEJdjbitxSIREklAZMQCGWa6Q2aq+KhnofSKPca6yKIvGWWCgRxCRNUaPEi1x5WCsiZqH8cMXoEgJJgLIkIRpoTEI+94/zjZwMsztnd2d3st+8n4/HPjLne77nzOd7ZvY9Z75nZ6KIwMzM8nVAqwswM7PB5aA3M8ucg97MLHMOejOzzDnozcwy56A3M8ucg36Yk9Qp6ZQm7esCSd8rLYekyc3Yd9rfNklHN2t/tjdJN0j6VKvrsH2Pg34AJK2T9Gc1bXMk/ai2j6S/TUG3TdJ2Sc+Xljvr7HtiCto9ff5D0nckvbXcLyKOjYgfNKhzz75G9tYvIr4WEadVGnwDkn4g6b/X7P/giFjbjP33cp9bJI0erPsYqFTj9vSYPi3pFkmvanVde1R5rki6QtLO0nNzm6SPDmWd1jcO+iESEf8rBd3BwCXAXXuWI+LYXjY9NG3zeuB24BuS5jS7vkYvAvs6SROBNwMBnNXPfQzVMZiXHtNXA4cCV7Wwlv5aVnr+HhwRn6ntIGlEKwqzF3PQDxMR8WREfBG4Algo6QDY+12FpKmSOiT9Or0D+Hza/Ifp32fS2dcfpXced0q6StIm4IradyPJ2yStTWeff1+63ysk3bSnU/lMUNKVFKF7Tbq/a1Kf300FSRoj6auSuiU9Kul/lvY9R9KPJH02naE/Iml6g0N0IfAT4AbgovIKSRPSmXO3pE2leuodg97qmizpXyVtTcdjWWpX2sdT6dg/KOm1FR7TzcC/AK8tPZYfk/QA8Gw6lmepmJ57Jr0beE1pXG+QdK+k36RaXlpa96LHsub4Hyjpc2mMW9PxPpA6z5VG4yjt/wZJ/0fSKknPAv9N0lhJ/5KO5yOSPlDqf2DaZoukn0u6VNKGevWW9v+p0vKZku5Lx+bHkl5XWrdO0kckPZDGt0xS+fjMSNv+WtLDkqZJOlfSPTVj+pCkb1U9BvsqB/3wcwvwe8Af1Fn3ReCLEXEI8PvA8tR+cvr30HT2dVdaPglYCxwBXNnD/f050A6cAMwA/rJRgRHxceDfSGeuETGvTrcvAWOAo4G3UAT1u0vrTwLWAIcDnwGuk6Re7vZC4Gvp53RJR8Dvziq/AzwKTATGAUtr7qd8DHqr65PA94BXAONTX4DTKI7xq9O25wGbeqmVVNvhwF8A/15qng2cQXGmfzTwdeBvgDZgFfBtSaMkjQK+CdwIvBL457Svqj4LvBH4r2n7jwK76fm5UtX5FMfx5cCPgW8D91Mc91OBv5F0eur7CYrn6e8Dp1PzAt0bSW8ArgfeCxwGXAus1N7TducB04BJwOuAOWnbqcBXgUspjvPJwDpgJTCp/GIKvCv1HdYc9AP3zXRG8YykZ4AvD/L9PZ7+fWWddTuByZIOj4htEfGTRvuKiC9FxK6I+M8e+iyMiM0R8RjwBYogGpAUvrOAyyLiNxGxDvgcxS/VHo9GxFci4nlgCfAqijCut78/AY4ClkfEPcDDFIEDMBUYC1waEc9GxPaIKJ/p/u4YADsa1LUz3c/Ymv3spAi2PwQUEQ9FxBO9HIKr03PlfuAJ4EPldRGxPj0e7wBujYjbI2InRTgfSBHObwJeAnwhInZGxApgdS/3WT5eB1C8YH8wIjZGxPMR8eOI+G2V7ZPzys97SWNT+7ci4s6I2A0cB7RFxIKI2JGuz3yF4hhDEcRXpufXeuDqPtz/XODaiLg71b8E+C3Fcdnj6oh4PL1z+jZwfGq/GLg+Hdfd6Rj8Io1/GfDOdJyOpTg5+E4f6tonOegH7uyIOHTPD/C+Qb6/cenfzXXWXUxxVvkLSaslndlgX+sr3F+5z6MUoTlQh1OE1KM1+x5XWn5yz42IeC7dPLiH/V0EfC8ink7LN/PC2eEEiheNXT1sWx5fo7o+Cgj4aZpO+ctU3x3ANcAi4ClJiyUd0sP9AXwgPV/GRcQFEdHdQz1jy7Wk8Fyf6hkLbIy9v5WwXHdvDqeY5nm4Yv96lpef9xGx5wSkXP9RwNiaE6G/5YUX7LG8+PlV1VHAh2v2PYG9n59Plm4/xwvPnwn0PPYlwPnp3eO70jj78gK4T3LQDz9/DjxFMa2xl4j4VUTMppjaWQiskHQQxQXKeqp8demE0u0jeeEdxbPAy0rr/ksf9v00L5wdl/e9sUI9e0nzyucBb5H0pKQngf8BvF7S6ymC5Ej1fHGzXGevdaXrJO+JiLEUUwZf3jOHHBFXR8QbgSkUL7aX9nUsdep5vFxLCp8JqZ4ngHE101lHlm7v9fhIKj8+TwPbKaZMerv//ihvvx54pOYF4eUR8ba0/gle/Pwqe46en2PrKd4NlPf9soj4eoUa11N/7KR3wTsorjGdTzE1Nuw56IcJSUdImkcxr3lZOrur7fNOSW1p3TOpeTfQnf7tz9+wXyrpFZImAB+keGsLcB9wsqQjJY0BLqvZ7j96ur80HbMcuFLSyyUdRTF9cVO9/g2cDTxPEbDHp5/XUFwjuBD4KUWgfFrSQZJeKumP+1NXulg3PnXfQhFquyWdKOkkSS+hCNjtFMd7oJYDZ0g6Ne37wxTTEz8G7gJ2AR+Q9BJJ51BMU+1xP3CspOPTRcgrSuPcTTG//fl0sXSEigv0oxnYc6XWT4HfqLjAfGC6n9dKOrE0vsvS82s88P6a7e+jOLseIWkaxTWTPb4CXJKOu9Jje4akl1eo6zrg3em4HiBpnKQ/LK3/KsU7tJ0103zDloN+3/eMir9geBB4G3BuRFzfQ99pQKekbRQXZmdFxH+mqY8rgTvT29w39bB9Pd8C7qH4pbuV4peEiLidIvQfSOtr5zG/CMxU8RcV9eZe308RimuBH1FMt/Q0rt5cBPxTRDyWzrifjIgnKX5RL6CYank7MBl4DNhAMffdk97qOhG4Ox3flRRz3GuBQyiCZwvF9MMm4O/7MZa9RMQaivniL1Gchb8deHua794BnENxgXFzGtMtpW1/CSwA/h/wqzSWso9QPKdWp+0XAgcM8LlSW//zwJkUL76PpDH8I8UFa4C/ozhej1Bc5K49e/5gGvMzFI/lN0v77gDeQ/E4bwG6SBdbK9T1U4oL7FcBW4F/Ze93cTdS/CVUf0489kkK/8cjZrYPUPEJ75siYnyDroNdx4EU06MnRMSvWllLs/iM3sxsb38FrM4l5AH29U/fmZkNGUnrKKb7zm5tJc3lqRszs8x56sbMLHMOejOzzO1zc/SHH354TJw4sdVlmJkNK/fcc8/TEdFWb90+F/QTJ06ko6Oj1WWYmQ0rknr8CglP3ZiZZc5Bb2aWOQe9mVnmHPRmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZpnb5z4wZfuWifNvbXUJA7Lu02e0ugSzlvMZvZlZ5ioFvaRpktZI6pI0v876kyXdK2mXpJl11h8iaYOka5pRtJmZVdcw6CWNABYB0yn+A+bZkqbUdHuM4v9rvLmH3XwS+GH/yzQzs/6qckY/FeiKiLXpPyReCswod4iIdRHxAMX/Hr8XSW8EjqD4z3/NzGyIVQn6ccD60vKG1NaQpAOAz1H8j/NmZtYCg30x9n3AqojY0FsnSXMldUjq6O7uHuSSzMz2L1X+vHIjMKG0PD61VfFHwJslvQ84GBglaVtE7HVBNyIWA4sB2tvb/Z/Ympk1UZWgXw0cI2kSRcDPAs6vsvOIuGDPbUlzgPbakDczs8HVcOomInYB84DbgIeA5RHRKWmBpLMAJJ0oaQNwLnCtpM7BLNrMzKqr9MnYiFgFrKppu7x0ezXFlE5v+7gBuKHPFZqZ2YD4k7FmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZplz0JuZZc5Bb2aWOQe9mVnmKv3HI2Y5mzj/1laXMGDrPn1Gwz7DfZxVxmj1+YzezCxzDnozs8w56M3MMlcp6CVNk7RGUpek+XXWnyzpXkm7JM0stR8v6S5JnZIekPSOZhZvZmaNNQx6SSOARcB0YAowW9KUmm6PAXOAm2vanwMujIhjgWnAFyQdOsCazcysD6r81c1UoCsi1gJIWgrMAH6+p0NErEvrdpc3jIhflm4/LukpoA14ZqCFm5lZNVWmbsYB60vLG1Jbn0iaCowCHu7rtmZm1n9DcjFW0quAG4F3R8TuOuvnSuqQ1NHd3T0UJZmZ7TeqBP1GYEJpeXxqq0TSIcCtwMcj4if1+kTE4ohoj4j2tra2qrs2M7MKqgT9auAYSZMkjQJmASur7Dz1/wbw1YhY0f8yzcysvxoGfUTsAuYBtwEPAcsjolPSAklnAUg6UdIG4FzgWkmdafPzgJOBOZLuSz/HD8ZAzMysvkrfdRMRq4BVNW2Xl26vppjSqd3uJuCmAdZoZmYD4E/GmpllzkFvZpY5B72ZWeYc9GZmmXPQm5llzkFvZpY5B72ZWeYc9GZmmXPQm5llzkFvZpY5B72ZWeYc9GZmmXPQm5llzkFvZpY5B72ZWeYc9GZmmXPQm5llzkFvZpY5B72ZWeYc9GZmmasU9JKmSVojqUvS/DrrT5Z0r6RdkmbWrLtI0q/Sz0XNKtzMzKppGPSSRgCLgOnAFGC2pCk13R4D5gA312z7SuATwEnAVOATkl4x8LLNzKyqKmf0U4GuiFgbETuApcCMcoeIWBcRDwC7a7Y9Hbg9IjZHxBbgdmBaE+o2M7OKqgT9OGB9aXlDaqui0raS5krqkNTR3d1dcddmZlbFPnExNiIWR0R7RLS3tbW1uhwzs6xUCfqNwITS8vjUVsVAtjUzsyaoEvSrgWMkTZI0CpgFrKy4/9uA0yS9Il2EPS21mZnZEGkY9BGxC5hHEdAPAcsjolPSAklnAUg6UdIG4FzgWkmdadvNwCcpXixWAwtSm5mZDZGRVTpFxCpgVU3b5aXbqymmZeptez1w/QBqNDOzAdgnLsaamdngcdCbmWXOQW9mljkHvZlZ5hz0ZmaZc9CbmWXOQW9mljkHvZlZ5hz0ZmaZc9CbmWXOQW9mljkHvZlZ5hz0ZmaZc9CbmWXOQW9mljkHvZlZ5hz0ZmaZc9CbmWXOQW9mljkHvZlZ5ioFvaRpktZI6pI0v8760ZKWpfV3S5qY2l8iaYmkByU9JOmyJtdvZmYNNAx6SSOARcB0YAowW9KUmm4XA1siYjJwFbAwtZ8LjI6I44A3Au/d8yJgZmZDo8oZ/VSgKyLWRsQOYCkwo6bPDGBJur0COFWSgAAOkjQSOBDYAfy6KZWbmVklVYJ+HLC+tLwhtdXtExG7gK3AYRSh/yzwBPAY8NmI2Fx7B5LmSuqQ1NHd3d3nQZiZWc8G+2LsVOB5YCwwCfiwpKNrO0XE4ohoj4j2tra2QS7JzGz/UiXoNwITSsvjU1vdPmmaZgywCTgf+L8RsTMingLuBNoHWrSZmVVXJehXA8dImiRpFDALWFnTZyVwUbo9E7gjIoJiuuZPASQdBLwJ+EUzCjczs2oaBn2ac58H3AY8BCyPiE5JCySdlbpdBxwmqQv4ELDnTzAXAQdL6qR4wfiniHig2YMwM7OejazSKSJWAatq2i4v3d5O8aeUtdttq9duZmZDx5+MNTPLnIPezCxzDnozs8w56M3MMuegNzPLnIPezCxzDnozs8w56M3MMuegNzPLnIPezCxzDnozs8w56M3MMuegNzPLnIPezCxzDnozs8w56M3MMuegNzPLnIPezCxzDnozs8xVCnpJ0yStkdQlaX6d9aMlLUvr75Y0sbTudZLuktQp6UFJL21i/WZm1kDDoJc0AlgETAemALMlTanpdjGwJSImA1cBC9O2I4GbgEsi4ljgFGBn06o3M7OGqpzRTwW6ImJtROwAlgIzavrMAJak2yuAUyUJOA14ICLuB4iITRHxfHNKNzOzKqoE/ThgfWl5Q2qr2ycidgFbgcOAVwMh6TZJ90r66MBLNjOzvhg5BPv/E+BE4Dng+5LuiYjvlztJmgvMBTjyyCMHuSQzs/1LlTP6jcCE0vL41Fa3T5qXHwNsojj7/2FEPB0RzwGrgBNq7yAiFkdEe0S0t7W19X0UZmbWoypBvxo4RtIkSaOAWcDKmj4rgYvS7ZnAHRERwG3AcZJell4A3gL8vDmlm5lZFQ2nbiJil6R5FKE9Arg+IjolLQA6ImIlcB1wo6QuYDPFiwERsUXS5yleLAJYFRG3DtJYzMysjkpz9BGximLapdx2een2duDcHra9ieJPLM3MrAX8yVgzs8w56M3MMuegNzPLnIPezCxzDnozs8w56M3MMuegNzPLnIPezCxzDnozs8w56M3MMuegNzPLnIPezCxzDnozs8w56M3MMuegNzPLnIPezCxzDnozs8w56M3MMuegNzPLnIPezCxzlYJe0jRJayR1SZpfZ/1oScvS+rslTaxZf6SkbZI+0qS6zcysooZBL2kEsAiYDkwBZkuaUtPtYmBLREwGrgIW1qz/PPDdgZdrZmZ9VeWMfirQFRFrI2IHsBSYUdNnBrAk3V4BnCpJAJLOBh4BOptSsZmZ9UmVoB8HrC8tb0htdftExC5gK3CYpIOBjwF/N/BSzcysPwb7YuwVwFURsa23TpLmSuqQ1NHd3T3IJZmZ7V9GVuizEZhQWh6f2ur12SBpJDAG2AScBMyU9BngUGC3pO0RcU1544hYDCwGaG9vj36Mw8zMelAl6FcDx0iaRBHos4Dza/qsBC4C7gJmAndERABv3tNB0hXAttqQNzOzwdUw6CNil6R5wG3ACOD6iOiUtADoiIiVwHXAjZK6gM0ULwZmZrYPqHJGT0SsAlbVtF1eur0dOLfBPq7oR31mZjZA/mSsmVnmHPRmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZplz0JuZZc5Bb2aWOQe9mVnmKn175XAycf6trS5h2Fj36TNaXYKZDQGf0ZuZZc5Bb2aWOQe9mVnmHPRmZplz0JuZZc5Bb2aWuUpBL2mapDWSuiTNr7N+tKRlaf3dkiam9rdKukfSg+nfP21y/WZm1kDDoJc0AlgETAemALMlTanpdjGwJSImA1cBC1P708DbI+I44CLgxmYVbmZm1VQ5o58KdEXE2ojYASwFZtT0mQEsSbdXAKdKUkT8e0Q8nto7gQMljW5G4WZmVk2VoB8HrC8tb0htdftExC5gK3BYTZ+/AO6NiN/W3oGkuZI6JHV0d3dXrd3MzCoYkouxko6lmM55b731EbE4Itojor2trW0oSjIz229UCfqNwITS8vjUVrePpJHAGGBTWh4PfAO4MCIeHmjBZmbWN1WCfjVwjKRJkkYBs4CVNX1WUlxsBZgJ3BERIelQ4FZgfkTc2aSazcysDxoGfZpznwfcBjwELI+ITkkLJJ2Vul0HHCapC/gQsOdPMOcBk4HLJd2Xfn6v6aMwM7MeVfqa4ohYBayqabu8dHs7cG6d7T4FfGqANZqZ2QD4k7FmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZplz0JuZZc5Bb2aWOQe9mVnmHPRmZplz0JuZZc5Bb2aWOQe9mVnmKgW9pGmS1kjqkjS/zvrRkpal9XdLmlhad1lqXyPp9CbWbmZmFTQMekkjgEXAdGAKMFvSlJpuFwNbImIycBWwMG07BZgFHAtMA76c9mdmZkOkyhn9VKArItZGxA5gKTCjps8MYEm6vQI4VZJS+9KI+G1EPAJ0pf2ZmdkQGVmhzzhgfWl5A3BST30iYpekrcBhqf0nNduOq70DSXOBuWlxm6Q1pdVjgK091NbTutr2qsvl9sOBp3u436p6q70v/eqtb9TW25jHAFu1cEjH2Khvf8ZYuzxYY+yplr72G5QxAgzxYznYv5P77RgZWO4c1eOaiOj1B5gJ/GNp+V3ANTV9fgaMLy0/nAq+Bnhnqf06YGaj+6zZ9+K+rqttr7pcbgc6+lJnX2vvS7966xu19Tbm0niHbIz9eSyrtPU05maOsVmP5WCNcagfy8H+nfQYB/58rf2pMnWzEZhQWh6f2ur2kTSS4hVqU8VtG/l2P9bVtldd7u2++qPq/hr1q7e+UVtvY27mOPuyr74+llXaehrXvvhY7q9jrNfuMfa9pn5TehXpuUMR3L8ETqUI6dXA+RHRWerz18BxEXGJpFnAORFxnqRjgZsp5uXHAt8HjomI5wdlNE0kqSMi2ltdx2DyGPOxP4zTY+y/hnP0Ucy5zwNuA0YA10dEp6QFFG8zVlJMydwoqQvYTPGXNqR+y4GfA7uAvx4OIZ8sbnUBQ8BjzMf+ME6PsZ8antGbmdnw5k/GmpllzkFvZpY5B72ZWeYc9BVJOkjSEklfkXRBq+sZDJKOlnSdpBWtrmWwSDo7PYbLJJ3W6noGg6TXSPoHSSsk/VWr6xks6XeyQ9KZra5lsEg6RdK/pcfzlP7uZ78OeknXS3pK0s9q2ut9ids5wIqIeA9w1pAX2099GWMUX3NxcWsq7b8+jvGb6TG8BHhHK+rtjz6O8aGIuAQ4D/jjVtTbH338fQT4GLB8aKscuD6OM4BtwEspvlmgfwbjU1jD5Qc4GTgB+FmpbQTFJ3uPBkYB91N8mdtlwPGpz82trn0wxlhav6LVdQ/BGD8HnNDq2gdrjBQnI9+l+MxLy+tv9hiBt1L8Gfcc4MxW1z6I4zwgrT8C+Fp/73O/PqOPiB9S/N1/WU9f4raB4pO9MIzeCfVxjMNSX8aowkLguxFx71DX2l99fRwjYmVETAeGzTRjH8d4CvAm4HzgPZKy/J2MiN1p/RZgdH/vs8qXmu1vevoSt6uBaySdwRB8ZHmQ1R2jpMOAK4E3SLosIv53S6prjp4ex/cDfwaMkTQ5Iv6hFcU1SU+P4ykUU42jgVVDX1ZT1R1jRMwDkDQHeLoUiMNVT4/lOcDpwKEU3x3WLw76iiLiWeDdra5jMEXEJoq562xFxNUUL9rZiogfAD9ocRlDIiJuaHUNgykibgFuGeh+hs3bnSHUjC9i29d5jHnwGPMxqON00L/YauAYSZMkjaK44LOyxTU1m8eYB48xH4M7zlZfgW7x1e+vA08AOynmxC5O7W+j+MbOh4GPt7pOj9Fj9BjzGGOrxukvNTMzy5ynbszMMuegNzPLnIPezCxzDnozs8w56M3MMuegNzPLnIPezCxzDnozs8w56M3MMvf/ATBwZzWKFG2DAAAAAElFTkSuQmCC\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"needs_background":"light"}},{"output_type":"stream","name":"stdout","text":["============= TEST RESULTS ===============\n","Test Type        : hits_distribution\n","Test Description : Compute the distribution of hit-rate across product frequency in training data\n","Test Result      : {'histogram': {10: 0.02702702702702703, 100: 0.14410480349344978, 1000: 0.12534818941504178, 10000: 0.125, 100000: 0}, 'counts': {10: 74, 100: 229, 1000: 359, 10000: 128, 100000: 0}}\n","\n"]},{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAXAAAAEICAYAAABGaK+TAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAes0lEQVR4nO3de3wV5b3v8c8vIQjba4XARjBGWkBQINCYg1DcKPVCxUuVjW2RBqub0m61G9tabE9fx7qt4qtW8EK1WCt4R7Eip9pdg4dURG5BAhZRkDRKLAoCUkSsQH7nj5nERUhYs5KVrEz4vl+v9crMmmdmfs+stX551rNmnjF3R0RE4icr0wGIiEjjKIGLiMSUEriISEwpgYuIxJQSuIhITCmBi4jElBJ4K2dms8zslnB6uJm9lcZt/8nMisPpCWb2Shq3Pc7MXkzX9lLY7zAz22BmH5vZJRHKN9vxjZvwmPXMdBwSnRJ4jLj7Infvk6ycmd1kZo9G2N4od5/d1LjMLN/M3MzaJWz7MXc/t6nbboSbgXvd/Sh3n5fKiuk+vi3NzCrN7KuNXT88ZhXpjKk+ZlZqZlc3934OB0rghyELtNXX/iRgbaaDEGkJbfVDHFtmNsjMXjOzXWY2B+iQsGyEmVUlzP/EzN4Ly75lZiPN7Hzgp8Dl4Vfi1WHZUjP7pZktBj4BetbTEjIzu9fMdprZm2Y2MmHBAa27Oq3Ql8O/H4X7PKNul4yZDTWzFeG2V5jZ0IRlpWb232a2OKzLi2bW+RDH6D/M7G0z225m883shPD5jUBP4P+GcRzRgsf3SjNbF5atMLPv1t2umf3QzLaY2WYzuzJheUcz+7WZvRMen1fMrGO4bIiZvWpmH5nZajMb0cAxeQTIS6j7DeHzF5nZ2nD9UjPre4jj6mb2pXB6lpnNMLPnwzotM7Mv1il7XVjXD83sVzWNgrrfUCzhG5qZ/RIYDtwbxnlvQ/FIBO6uRyt5AO2Bd4DJQA4wBtgL3BIuHwFUhdN9gE3ACeF8PvDFcPom4NE62y4F3gVOBdqF2y8Frg6XTwD2Jez7cmAncHy4vBL4asL2avcR7tuBdgnLJwCvhNPHAzuA8eG+vxnOd0qIbSPQG+gYzk9t4BidDXwIDAaOAO4BXk5YfkCcLXh8LwC+CBjwbwT/JAcnbHcfQfdODvC1cPkXwuUzwjp3B7KBoWHdugPbwvJZwDnhfG4D9av7GvUGdofr5QA3AG8D7RtY34EvhdOzwn0Vha/ZY8CTdcouDF/bPGA9n7+XDjg+dd8fJLzv9GjaQy3w1mUIwQdturvvdfe5wIoGyu4n+JD3M7Mcd690941Jtj/L3de6+z5331vP8i0J+54DvEWQmJrqAmCDuz8S7vsJ4E3gwoQyD7n7enffAzwFFDSwrXHA7939NXf/J3AjcIaZ5UeIo9mOr7s/7+4bPfAX4EWClmaNvcDN4X5fAD4G+oSt1u8AP3D399x9v7u/GtbtCuAFd3/B3avdvQQoI0joUVwOPO/uJeHrfQfBP8ihh16t1rPuvtzd9xEk8II6y2939+3u/i4wneAfs7QgJfDW5QTgPQ+bKaF36ivo7m8D/0XQ2tliZk/WdCUcwqYky+vbd7JtRnECB9fjHYIWZo33E6Y/AY6Ksi13/5igpdi9gfJ1122W42tmo8xsadit8xFBkk3sBtoWJsIaNXXsTNCNU98/h5OAfw+7Pz4Kt/sVoNsha/m5useqmuA9EOVYQfLXJPH9lK73iqRACbx12Qx0NzNLeC6vocLu/ri7f4Xgg+7A7TWLGlolyf7r2/ffw+ndwL8kLPvXFLb79zDGRHnAe0nWS7otMzsS6BRxW81yfMO+9mcIWrhd3f044AWC7pRkPgQ+Jeh+qWsT8Ii7H5fwONLdpzYUcp35usfKgBNp3HGvz4kJ01HfK5D8/SIRKYG3LksI+kqvM7McM7uUoA/yIGbWx8zODpPHp8AeoDpc/AGQb6mfadIlYd//DvQlSEQA5cA3wmWFBP3HNbaG+27oHOIXgN5m9q3wh6zLgX7AH1OMD+AJ4EozKwjrfiuwzN0rI6zbXMe3PUF3y1Zgn5mNAiKdQhm2in8P3GlmJ5hZtgU/Ah8BPApcaGbnhc93CH8Q7dHA5j7gwNfgKeCC8MfXHOCHwD+BV6PEFsGPzewLZnYi8ANgTvh8OXCmmeWZ2bEE3VyHilMaSQm8FXH3z4BLCX4A3E7Qh/mHBoofAUwlaMG9T5B8az4oT4d/t5nZaymEsAzoFW7zl8AYd98WLvs5QStxB/AL4PGEuD8Jyy8Ov+oPqVOvbcBoggSyjeDHtNHu/mEKsdVsa0EYyzMELeovAt+IuG6zHF933wVcR5AwdwDfAuanUK0fAa8T9MdvJ2jpZ7n7JuBigrNethK0yH9Mw5/b24D/Hb4GP3L3twj60e8J63EhcGF4HNLhOWAlQcJ+HngQIOyrnwOsCZfX/Ud9FzDGzHaY2d1piuWwZAd2B4qIJGdmDvQKfyuQDFELXEQkppTARURiSl0oIiIxpRa4iEhMtUteJH06d+7s+fn5LblLEZHYW7ly5Yfunlv3+RZN4Pn5+ZSVlbXkLkVEYs/M6r1iWF0oIiIxpQQuIhJTSuAiIjHVon3gIs1p7969VFVV8emnn2Y6FJFG6dChAz169CAnJydSeSVwaTOqqqo4+uijyc/P58ABB0VaP3dn27ZtVFVVcfLJJ0daR10o0mZ8+umndOrUSclbYsnM6NSpU0rfIJXApU1R8pY4S/X9qwQuIhJT6gOXNmtayfq0bm/yOb2TlsnOzqZ///7s3buXdu3a8e1vf5vJkyeTlZVFWVkZDz/8MHffXf8Q2JWVlbz66qt861vfSmvcUd19993cd999DB48mMcee6zZ9jNixAjuuOMOCgsL07rdW2+9lZ/+9KcpL2tIzYWHnTt3Tlp21qxZlJWVce+99zbL9hsSqQVuZseZ2Vwze9PM1oV3DDnezErMbEP49wuNjkIOtvC2zx8SGx07dqS8vJy1a9dSUlLCn/70J37xi18AUFhY2GDyhiCBP/744w0ub26/+c1vKCkpOSh579u3r4E1Wpdbb721UcviLGoXyl3A/7j7KcBAYB0wBXjJ3XsBL4XzIhLq0qULM2fO5N5778XdKS0tZfTo0QD85S9/oaCggIKCAgYNGsSuXbuYMmUKixYtoqCggGnTplFZWcnw4cMZPHgwgwcP5tVXgzuhlZaWMmLECMaMGcMpp5zCuHHjqBlVdMWKFQwdOpSBAwdSVFTErl272L9/Pz/+8Y85/fTTGTBgAL/97W8PinXSpElUVFQwatQopk2bxk033cT48eMZNmwY48ePp7KykrPPPpsBAwYwcuRI3n33XQAmTJjA9773PYYMGULPnj0pLS3lO9/5Dn379mXChAlJj9FRRx3Fz372MwYOHMiQIUP44IMParc7adIkCgsL6d27N3/8Y3BTn1mzZnHNNdfUrj969GhKS0uZMmUKe/bsoaCggHHjxh2wj/qW3XnnnZx22mmcdtppTJ8+PfJrunz5cs444wwGDRrE0KFDeeutt2qXbdq0iREjRtCrV6/af9oAjz76KEVFRRQUFPDd736X/fv3R95fMkkTeHhPuzP5/HZJn7n7RwS3epodFpsNXJK2qETaiJ49e7J//362bNlywPN33HEHM2bMoLy8nEWLFtGxY0emTp3K8OHDKS8vZ/LkyXTp0oWSkhJee+015syZw3XXXVe7/qpVq5g+fTpvvPEGFRUVLF68mM8++4zLL7+cu+66i9WrV7NgwQI6duzIgw8+yLHHHsuKFStYsWIFDzzwAH/7298OiOf+++/nhBNOYOHChUyePBmAN954gwULFvDEE09w7bXXUlxczJo1axg3btwBsezYsYMlS5Ywbdo0LrroIiZPnszatWt5/fXXKS8vP+Tx2b17N0OGDGH16tWceeaZPPDAA7XLKisrWb58Oc8//zyTJk065NkZU6dOrf32U/cbRN1lK1eu5KGHHmLZsmUsXbqUBx54gFWrVh0yzhqnnHIKixYtYtWqVdx8880HdMssX76cZ555hjVr1vD0009TVlbGunXrmDNnDosXL6a8vJzs7Oy0dk9F6QM/meB+fA+Z2UCCe9z9gODu25vDMu8DXetb2cwmAhMB8vIavAG4yGFl2LBhXH/99YwbN45LL72UHj0Ovk/x3r17ueaaa2o/+OvXf96nX1RUVLtOQUEBlZWVHHvssXTr1o3TTz8dgGOOOQaAF198kTVr1jB37lwAdu7cyYYNG5Kea3zRRRfRsWNHAJYsWcIf/hDcPnT8+PHccMMNteUuvPBCzIz+/fvTtWtX+vfvD8Cpp55KZWUlBQUFDe6jffv2td9KvvzlL1NSUlK7bOzYsWRlZdGrVy969uzJm2++ech4o3rllVf4+te/zpFHHgnApZdeyqJFixg0aFDSdXfu3ElxcTEbNmzAzNi7d2/tsnPOOYdOnTrVbvOVV16hXbt2rFy5svY12bNnD126dElLPSBaAm8HDAaudfdlZnYXdbpL3N3De+QdxN1nAjMBCgsLdfcIOaxUVFSQnZ1Nly5dWLduXe3zU6ZM4YILLuCFF15g2LBh/PnPfz5o3WnTptG1a1dWr15NdXU1HTp0qF12xBFH1E5nZ2cfsp/a3bnnnns477zzUoq9JsElUxNLVlbWAXFlZWUl7T/PycmpPXWubj3qnlJnZrRr147q6ura51r6qtuf//znnHXWWTz77LNUVlYyYsSIA+JLZGa4O8XFxdx2W/P8lhWlD7wKqHL3ZeH8XIKE/oGZdQsD7QZsaWB9kcPS1q1bmTRpEtdcc81BH+6NGzfSv39/fvKTn3D66afz5ptvcvTRR7Nr167aMjt37qRbt25kZWXxyCOPJO077dOnD5s3b2bFihUA7Nq1i3379nHeeedx33331bYW169fz+7du1Oqy9ChQ3nyyScBeOyxxxg+fHhK6zfG008/TXV1NRs3bqSiooI+ffqQn59PeXk51dXVbNq0ieXLl9eWz8nJOaBFnChx2fDhw5k3bx6ffPIJu3fv5tlnn41cn507d9K9e3cg6I9PVFJSwvbt29mzZw/z5s1j2LBhjBw5krlz59Z2oW3fvp133ql3ZNhGSdoCd/f3zWyTmfVx97eAkcAb4aMYmBr+fS5tUYmkQZTT/tKt5seymtMIx48fz/XXX39QuenTp7Nw4UKysrI49dRTGTVqFFlZWWRnZzNw4EAmTJjA97//fS677DIefvhhzj///KQt4vbt2zNnzhyuvfZa9uzZQ8eOHVmwYAFXX301lZWVDB48GHcnNzeXefPmpVSve+65hyuvvJJf/epX5Obm8tBDD6W0fmPk5eVRVFTEP/7xD+6//346dOjAsGHDOPnkk+nXrx99+/Zl8ODBteUnTpzIgAED6j0Nsu6yCRMmUFRUBMDVV1/dYPfJgAEDyMoK2rljx47lhhtuoLi4mFtuuYULLrjggLJFRUVcdtllVFVVccUVV9SeJnnLLbdw7rnnUl1dTU5ODjNmzOCkk05KyzGKdE9MMysAfge0ByqAKwla708BecA7wFh3336o7RQWFrpu6BBR4umDZ92YuThiZN26dfTt2zfTYUgaTJgwgdGjRzNmzJhMh9Li6nsfm9lKdz/oxPlIF/K4ezlQ31n3IxsToIiINJ2uxBSRVqdu/7LUTwlcWkSql7Vnov9aJG40mJWISEwpgYuIxJQSuIhITKkPXNqudI/kGOF0Tg0nm5yGk019+w1RC1wkjTScbOZoOFkRSRsNJzsh6THScLJNowQu0ow0nGz5IY+PhpNtGvWBi2SAhpMNaDjZplECF2lGGk5Ww8lmejhZEWkEDSfbNBpONjm1wKXtysAojhpONn00nGxykYaTTRcNJ5uCNjacbEuMhaLhZNsODScbbThZdaGIiMSUulBEpNXRcLLRqAUubUpLdgmKpFuq718lcGkzOnTowLZt25TEJZbcnW3bth1wumgy6kKRNmFayXpyrJq+x3zMUe/+PWn5YzrmtEBUIqnp0KFDvRd1NUQJXNqMvZ7Fmp1HJC+I7vgjbYO6UEREYkoJXEQkppTARURiSglcRCSmlMBFRGIq0lkoZlYJ7AL2A/vcvdDMjgfmAPlAJTDW3Xc0T5giIlJXKi3ws9y9IGFAlSnAS+7eC3gpnBcRkRbSlC6Ui4HZ4fRs4JImRyMiIpFFTeAOvGhmK81sYvhcV3ffHE6/D3Stb0Uzm2hmZWZWtnXr1iaGKyIiNaJeifkVd3/PzLoAJWZ2wM3p3N3NrN4BKNx9JjATgvHAmxStiIjUitQCd/f3wr9bgGeBIuADM+sGEP7d0vAWREQk3ZImcDM70syOrpkGzgX+CswHisNixcBzzRWkiIgcLEoXSlfg2fCmrO2Ax939f8xsBfCUmV0FvAOMbb4wRUSkrqQJ3N0rgIH1PL8NGNkcQYmISHK6ElNEJKaUwEVEYkoJXEQkppTARURiSglcRCSmlMBFRGJKCVxEJKaUwEVEYkoJXEQkppTARURiSglcRCSmlMBFRGJKCVxEJKaUwEVEYkoJXEQkppTARURiSglcRCSmlMBFRGJKCVxEJKaUwEVEYkoJXEQkppTARURiSglcRCSmlMBFRGJKCVxEJKbaRS1oZtlAGfCeu482s5OBJ4FOwEpgvLt/1jxhiqTXtJL1KZWffE7vZopEpPFSaYH/AFiXMH87MM3dvwTsAK5KZ2AiInJokRK4mfUALgB+F84bcDYwNywyG7ikGeITEZEGRG2BTwduAKrD+U7AR+6+L5yvArrXt6KZTTSzMjMr27p1a1NiFRGRBEkTuJmNBra4+8rG7MDdZ7p7obsX5ubmNmYTIiJSjyg/Yg4DLjKzrwEdgGOAu4DjzKxd2ArvAbzXfGGKiEhdSRO4u98I3AhgZiOAH7n7ODN7GhhDcCZKMfBc84UpqUr1LItUNfdZGc0dv0hb0JTzwH8CXG9mbxP0iT+YnpBERCSKyOeBA7h7KVAaTlcARekPSUREotCVmCIiMaUELiISU0rgIiIxpQQuIhJTKf2IKUksvO3z6bNuzFwcInJYUAtcRCSmlMBFRGJKCVxEJKbUBx4TurRcROpSC1xEJKaUwEVEYurw7ELR6X5Ndrh16cR9dEdpm9QCFxGJKSVwEZGYOjy7UKJorm4Wdd+ISJqoBS4iElNK4CIiMaUELiISU+oDj9InnVjmUOVaYrsiIiG1wEVEYkoJXEQkptSF0hLqdpWIiKSBWuAiIjGlBC4iElNK4CIiMZW0D9zMOgAvA0eE5ee6+/8xs5OBJ4FOwEpgvLt/1pzBikgg1dERNdph2xSlBf5P4Gx3HwgUAOeb2RDgdmCau38J2AFc1WxRiojIQZImcA98HM7mhA8Hzgbmhs/PBi5pjgBFRKR+kU4jNLNsgm6SLwEzgI3AR+6+LyxSBXRvYN2JwESAvLy8psbbOmhEQRFpBSL9iOnu+929AOgBFAGnRN2Bu89090J3L8zNzW1clCIicpCUzkJx94+AhcAZwHFmVtOC7wG8l97QRETkUKKchZIL7HX3j8ysI3AOwQ+YC4ExBGeiFAPPNWegjaKuDhFpw6L0gXcDZof94FnAU+7+RzN7A3jSzG4BVgEPNmOcIiJSR9IE7u5rgEH1PF9B0B8uIiIZoCsxRURi6vAZjbC5RgTUSIMikiFqgYuIxJQSuIhITCmBi4jElBK4iEhMKYGLiMSUEriISEwdPqcRihzGdAOItkktcBGRmFICFxGJKSVwEZGYUgIXEYkpJXARkZjSWSiJNDCViMSIWuAiIjGlBC4iElNK4CIiMaU+cBE5iK7cjAe1wEVEYkoJXEQkptSFkiZLKrY1eRtL933+tXXIu9vqfV5EpIZa4CIiMaUELiISU0rgIiIxlbQP3MxOBB4GugIOzHT3u8zseGAOkA9UAmPdfUfzhdqAxMvfz7qxwWKp9lGf0bNTYyMSSVmqp+2JQLQW+D7gh+7eDxgC/KeZ9QOmAC+5ey/gpXBeRERaSNIE7u6b3f21cHoXsA7oDlwMzA6LzQYuaaYYRUSkHimdRmhm+cAgYBnQ1d03h4veJ+hiqW+dicBEgLy8vEYHGkkaRxNMx2mBqRry7syUyizNm9ic4YhIKxf5R0wzOwp4Bvgvd/9H4jJ3d4L+8YO4+0x3L3T3wtzc3CYFKyIin4uUwM0shyB5P+bufwif/sDMuoXLuwFbmidEERGpT9IEbmYGPAisc/c7ExbNB4rD6WLgufSHJyIiDYnSBz4MGA+8bmbl4XM/BaYCT5nZVcA7wNhmiVBEROqVNIG7+yuANbB4ZHrDERGRqHQlpohITCmBi4jElBK4iEhMKYGLiMSUbugQA1Gu0BSRw49a4CIiMaUELiISU0rgIiIxpQQuIhJTSuAiIjGlBC4iElNK4CIiMaUELiISU0rgIiIxpQQuIhJTSuAiIjGlBC4iElNK4CIiMaXRCGMscZTCpXkTMxiJiGSCWuAiIjGlBC4iElPqQhGRJptWsj6l8pPP6d1MkRxe1AIXEYkpJXARkZiKZxfKwtsyHYGISMYlbYGb2e/NbIuZ/TXhuePNrMTMNoR/v9C8YYqISF1RulBmAefXeW4K8JK79wJeCudFRKQFJU3g7v4ysL3O0xcDs8Pp2cAl6Q1LRESSaWwfeFd33xxOvw90baigmU0EJgLk5eU1cncHW1KxLW3bEhGJoyafheLuDvghls9090J3L8zNzW3q7kREJNTYBP6BmXUDCP9uSV9IIiISRWMT+HygOJwuBp5LTzgiIhJVlNMInwCWAH3MrMrMrgKmAueY2Qbgq+G8iIi0oKQ/Yrr7NxtYNDLNsYiISAp0Kb2ISEwpgYuIxJQSuIhITCmBi4jEVDxHIxSRw4puGFE/tcBFRGJKCVxEJKbUhSIiLS7VLhGpn1rgIiIxpQQuIhJTSuAiIjGlPvA2aMi7Mw+YX5o3MUORiEhzUgtcRCSmlMBFRGIqPl0oC2/LdAStWt1uExGJLq5XeqoFLiISU0rgIiIxpQQuIhJT8ekDFxGJ6HC5VF8tcBGRmFICFxGJKSVwEZGYUgIXEYkpJXARkZjSWSgiIilqLVduNqkFbmbnm9lbZva2mU1JV1AiIpJcoxO4mWUDM4BRQD/gm2bWL12BiYjIoTWlBV4EvO3uFe7+GfAkcHF6whIRkWSa0gfeHdiUMF8F/K+6hcxsIlBzR4GPzeytFPbRGfiw0RG2Dq2gDr9u6gZaQR2aTHVoHQ7LOlzf9H2eVN+Tzf4jprvPBBo11qmZlbl7YZpDalGqQ+ugOrQOqkN6NaUL5T3gxIT5HuFzIiLSApqSwFcAvczsZDNrD3wDmJ+esEREJJlGd6G4+z4zuwb4M5AN/N7d16YtskBbuM2M6tA6qA6tg+qQRubumY5BREQaQZfSi4jElBK4iEhMtYoEnuySfDM7wszmhMuXmVl+BsI8pAh1uN7M3jCzNWb2kpnVe15nJkUdGsHMLjMzN7NWcSpVoih1MLOx4Wux1sweb+kYk4nwXsozs4Vmtip8P30tE3E2xMx+b2ZbzOyvDSw3M7s7rN8aMxvc0jEmE6EO48LYXzezV81sYEvHCIC7Z/RB8APoRqAn0B5YDfSrU+b7wP3h9DeAOZmOuxF1OAv4l3D6e3GsQ1juaOBlYClQmOm4G/E69AJWAV8I57tkOu5G1GEm8L1wuh9Qmem468R3JjAY+GsDy78G/AkwYAiwLNMxN6IOQxPeQ6MyVYfW0AKPckn+xcDscHouMNLMrAVjTCZpHdx9obt/Es4uJThvvjWJOjTCfwO3A5+2ZHARRanDfwAz3H0HgLtvaeEYk4lSBweOCaePBf7egvEl5e4vA9sPUeRi4GEPLAWOM7NuLRNdNMnq4O6v1ryHyODnuTUk8Pouye/eUBl33wfsBDq1SHTRRKlDoqsIWiCtSdI6hF91T3T351sysBREeR16A73NbLGZLTWz81ssumii1OEm4AozqwJeAK5tmdDSJtXPS2uXsc+zxgNvYWZ2BVAI/FumY0mFmWUBdwITMhxKU7Uj6EYZQdBqetnM+rv7R5kMKkXfBGa5+6/N7AzgETM7zd2rMx3Y4cbMziJI4F/JxP5bQws8yiX5tWXMrB3B18ZtLRJdNJGGFTCzrwI/Ay5y93+2UGxRJavD0cBpQKmZVRL0Xc5vZT9kRnkdqoD57r7X3f8GrCdI6K1FlDpcBTwF4O5LgA4EAyzFRZsYhsPMBgC/Ay5294zko9aQwKNckj8fKA6nxwD/z8NfD1qJpHUws0HAbwmSd2vrd4UkdXD3ne7e2d3z3T2foN/vIncvy0y49YryXppH0PrGzDoTdKlUtGCMyUSpw7vASAAz60uQwLe2aJRNMx/4dng2yhBgp7tvznRQqTCzPOAPwHh3T+32POmU6V97/fNfpdcT/Pr+s/C5mwkSBARv0KeBt4HlQM9Mx9yIOiwAPgDKw8f8TMecah3qlC2llZ2FEvF1MIKuoDeA14FvZDrmRtShH7CY4AyVcuDcTMdcJ/4ngM3AXoJvPFcBk4BJCa/BjLB+r7fS91GyOvwO2JHweS7LRJy6lF5EJKZaQxeKiIg0ghK4iEhMKYGLiMSUEriISEwpgYuIxJQSuIhITCmBi4jE1P8HXMtnAXDgb+UAAAAASUVORK5CYII=\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"needs_background":"light"}},{"output_type":"stream","name":"stdout","text":["============= TEST RESULTS ===============\n","Test Type        : distance_to_query\n","Test Description : Compute the distribution of distance from query to label and query to prediction\n","Test Result      : {'histogram_x_to_y': ([10, 21, 20, 22, 22, 36, 39, 39, 35, 58, 39, 53, 41, 35, 35, 21, 31, 26, 23, 12, 6, 7, 3, 4, 1], [0.027085185050964355, 0.07621452689170838, 0.1253438687324524, 0.1744732105731964, 0.22360255241394042, 0.27273189425468447, 0.32186123609542844, 0.37099057793617246, 0.4201199197769165, 0.4692492616176605, 0.5183786034584046, 0.5675079452991485, 0.6166372871398925, 0.6657666289806365, 0.7148959708213806, 0.7640253126621246, 0.8131546545028686, 0.8622839963436126, 0.9114133381843567, 0.9605426800251007, 1.0096720218658448, 1.0588013637065887, 1.1079307055473326, 1.1570600473880768, 1.2061893892288207, 1.2553187310695648]), 'histogram_x_to_p': ([4, 11, 21, 20, 29, 28, 35, 33, 24, 34, 33, 30, 28, 37, 33, 32, 34, 33, 20, 61, 17, 20, 13, 4, 5], [0.012268495559692384, 0.025094011306762698, 0.03791952705383301, 0.05074504280090332, 0.06357055854797364, 0.07639607429504394, 0.08922159004211426, 0.10204710578918456, 0.11487262153625488, 0.1276981372833252, 0.14052365303039552, 0.15334916877746582, 0.16617468452453615, 0.17900020027160646, 0.19182571601867676, 0.2046512317657471, 0.2174767475128174, 0.2303022632598877, 0.24312777900695803, 0.25595329475402834, 0.26877881050109864, 0.28160432624816895, 0.29442984199523925, 0.3072553577423096, 0.3200808734893799, 0.3329063892364502]), 'raw_distances_x_to_y': [0.9254719093441963, 0.462533175945282, 0.21085858345031738, 0.47345417737960815, 0.8189148157835007, 0.6212723255157471, 0.1114012598991394, 0.4277552366256714, 0.5421712100505829, 0.3438114523887634, 0.6834848821163177, 0.12182509899139404, 0.5115478038787842, 0.2190667986869812, 0.7817028611898422, 0.7468498349189758, 0.49416977167129517, 0.2725176215171814, 0.5640368163585663, 0.10806989669799805, 0.5597535669803619, 0.26346278190612793, 0.2727212905883789, 0.31810325384140015, 0.7297700345516205, 0.5911677479743958, 0.5790992975234985, 1.0076064178720117, 0.430014431476593, 0.2532806396484375, 0.4981846809387207, 0.6517561078071594, 0.08466029167175293, 0.15237146615982056, 0.781215712428093, 1.0076064178720117, 0.23168128728866577, 0.2835843563079834, 0.34047961235046387, 0.7128640711307526, 0.6921305954456329, 0.4090891480445862, 0.45411020517349243, 0.308832049369812, 0.8578503280878067, 0.8280059248209, 0.09990346431732178, 0.36499106884002686, 0.47029995918273926, 0.9402659200131893, 0.26203757524490356, 0.6552892327308655, 0.40001380443573, 0.27151793241500854, 0.8445519506931305, 0.28339993953704834, 0.17017924785614014, 0.38485950231552124, 0.9698969088494778, 0.2542409896850586, 0.30253440141677856, 0.5792183578014374, 0.5336647927761078, 0.6775735020637512, 0.6186142265796661, 0.7454920709133148, 0.076199471950531, 0.2535223960876465, 0.5798035562038422, 0.6477236449718475, 0.7253351509571075, 0.36245912313461304, 0.5802899301052094, 0.6831025183200836, 0.3974495530128479, 0.4377900958061218, 0.874511793255806, 0.09468638896942139, 0.30603235960006714, 0.6420265138149261, 0.8917598351836205, 0.8514892011880875, 0.28252047300338745, 0.3410959243774414, 0.7218420505523682, 0.49533963203430176, 0.7329410910606384, 0.4549601674079895, 0.8185496628284454, 0.5602165162563324, 0.935164324939251, 0.6192016303539276, 0.6965530812740326, 0.506960391998291, 0.6797125935554504, 0.972739702090621, 0.5587949752807617, 0.9648317396640778, 0.7918451577425003, 0.10904902219772339, 0.26039278507232666, 0.48779743909835815, 0.32050520181655884, 0.9108971506357193, 0.17316967248916626, 0.1562144160270691, 0.5707772672176361, 0.7172189950942993, 0.4743512272834778, 0.5884027183055878, 0.39823293685913086, 0.37635380029678345, 0.48903268575668335, 0.2427654266357422, 0.740830808877945, 0.3790507912635803, 0.48203784227371216, 0.26257091760635376, 0.714726060628891, 0.3886526823043823, 0.506866067647934, 0.38689154386520386, 0.8665386885404587, 0.4252302050590515, 0.3790507912635803, 0.2864288091659546, 0.5251165926456451, 0.2038726806640625, 0.6813859939575195, 0.6797683835029602, 0.13988149166107178, 0.4257371425628662, 0.6832488477230072, 0.4939197897911072, 0.7525335550308228, 0.6219100654125214, 0.15661191940307617, 0.639555811882019, 0.6219100654125214, 0.6514264643192291, 0.35110509395599365, 0.6271197497844696, 0.462327778339386, 0.6513311266899109, 0.10752040147781372, 0.5508485436439514, 0.3116947412490845, 0.7269882261753082, 0.6206513047218323, 0.5326894521713257, 0.8309376835823059, 0.2881903052330017, 0.3581002354621887, 0.16245216131210327, 0.5602153539657593, 0.5890812277793884, 0.1208798885345459, 0.4772169589996338, 0.44260478019714355, 0.6125787794589996, 0.09781920909881592, 0.7173270881175995, 0.9057733789086342, 0.21883291006088257, 0.2283496856689453, 0.3400246500968933, 0.6160574853420258, 0.46860283613204956, 0.4285128712654114, 0.4254186749458313, 0.4115585684776306, 0.8667544275522232, 0.06605154275894165, 0.44577592611312866, 0.4536110758781433, 1.0924406200647354, 0.5168035328388214, 0.5836308300495148, 0.4283052682876587, 0.22816848754882812, 0.3787039518356323, 0.07026445865631104, 0.8491666465997696, 0.3641747832298279, 1.1299540549516678, 0.44231313467025757, 1.167671948671341, 0.918891005218029, 0.7797763794660568, 0.8064161539077759, 0.7030901312828064, 0.1738971471786499, 0.11099112033843994, 0.4285128712654114, 0.6690216064453125, 0.5043551623821259, 0.5595879852771759, 0.40289217233657837, 0.06950515508651733, 0.46268099546432495, 0.39849114418029785, 0.4093968868255615, 0.7422437965869904, 0.5640649199485779, 0.5044651627540588, 0.3718870282173157, 0.4949328303337097, 0.3384582996368408, 0.9164132475852966, 0.8826794549822807, 0.4781320095062256, 0.39250558614730835, 0.47071176767349243, 0.2293713092803955, 0.6476076543331146, 0.8838279992341995, 0.27570104598999023, 0.19004637002944946, 0.6612381637096405, 0.6677241921424866, 0.19149482250213623, 0.7657949030399323, 0.5640649199485779, 0.3024331331253052, 0.1256752610206604, 0.36499106884002686, 0.4845578074455261, 0.5405918657779694, 0.5955899655818939, 0.9245310798287392, 0.9297516494989395, 0.21759074926376343, 0.3373871445655823, 0.3699283003807068, 0.6013206541538239, 0.9102961793541908, 0.843553215265274, 0.5168035328388214, 0.3229179382324219, 0.8645953387022018, 0.572632223367691, 0.4692692160606384, 0.9606809169054031, 0.5453396439552307, 0.5945159792900085, 0.6038006544113159, 0.85569629073143, 0.565295547246933, 0.9472744166851044, 0.49883782863616943, 0.32196158170700073, 0.6548276245594025, 0.38367241621017456, 0.3617027997970581, 0.6122760474681854, 1.0495793037116528, 0.12808352708816528, 0.7142658829689026, 0.8562994003295898, 0.2721226215362549, 0.3397049307823181, 0.027085185050964355, 0.6539434194564819, 0.5633589923381805, 0.3535528779029846, 0.18204158544540405, 0.2661483883857727, 0.8781896978616714, 0.6539434194564819, 0.3516688942909241, 0.9167362451553345, 0.12582182884216309, 0.19273149967193604, 0.6312002241611481, 0.49662697315216064, 0.7021820545196533, 0.2805701494216919, 0.39922797679901123, 0.6158891320228577, 0.07739633321762085, 0.617974579334259, 0.32775402069091797, 0.5247384607791901, 0.2149120569229126, 0.4774479866027832, 0.5334309637546539, 0.7717662453651428, 0.6299411654472351, 0.9769610203802586, 0.3056491017341614, 0.5658897757530212, 0.4421624541282654, 0.5553690791130066, 0.9007061719894409, 0.4115585684776306, 0.6690648198127747, 0.3697749972343445, 0.17395007610321045, 0.4646461009979248, 1.0299112629145384, 0.600534588098526, 0.4646461009979248, 0.9810393769294024, 0.8647336810827255, 0.34212249517440796, 0.5590666830539703, 0.3464195132255554, 0.05532979965209961, 0.20188939571380615, 0.4105381965637207, 0.4939197897911072, 0.2629045248031616, 0.28857362270355225, 0.29965150356292725, 0.4188130497932434, 0.2244529128074646, 0.7472360134124756, 0.6975664496421814, 0.49244630336761475, 0.08120405673980713, 0.48203784227371216, 0.4968187212944031, 0.6568638682365417, 0.5499407947063446, 0.4152105450630188, 0.8767637833952904, 0.6606419384479523, 0.5982445478439331, 0.8854007720947266, 0.451798677444458, 0.7668242305517197, 0.29245537519454956, 0.5700768232345581, 0.27996474504470825, 0.39198386669158936, 0.2561491131782532, 0.2906454801559448, 0.8172992914915085, 0.36499106884002686, 0.839078962802887, 0.871750995516777, 0.7232131659984589, 0.3438114523887634, 0.3022156357765198, 0.6184073686599731, 0.5854481160640717, 1.01153430249542, 0.7308022677898407, 0.21456676721572876, 0.7636695057153702, 0.8693548738956451, 0.42182499170303345, 0.7092864513397217, 1.2553187310695648, 0.5536989271640778, 0.8955832570791245, 0.4093968868255615, 0.5419392883777618, 0.505172461271286, 0.3697749972343445, 0.9452919512987137, 0.763588935136795, 0.6174618601799011, 0.5677748918533325, 0.5108366012573242, 0.2942226529121399, 0.8166680783033371, 0.6539860665798187, 0.6094558238983154, 0.3326358199119568, 0.30736833810806274, 0.4509674310684204, 0.40113013982772827, 0.5168035328388214, 0.20486664772033691, 0.4109269976615906, 0.03789621591567993, 0.7438116669654846, 0.7488087117671967, 1.0868622660636902, 0.7574658691883087, 0.5947867333889008, 0.20400893688201904, 0.5677686929702759, 0.49960315227508545, 0.4939197897911072, 0.6324397325515747, 0.4222393035888672, 0.8293663859367371, 0.9296053647994995, 0.5602153539657593, 0.11834901571273804, 0.7321200668811798, 1.1602952182292938, 0.5965181291103363, 0.9149350449442863, 0.506960391998291, 0.5186827778816223, 0.6399234533309937, 0.4691590666770935, 0.7098327279090881, 0.1822097897529602, 0.5572097897529602, 0.40265655517578125, 0.5968579947948456, 0.48597395420074463, 0.8904133811593056, 0.13772159814834595, 0.7772166877985001, 0.40333008766174316, 0.08821582794189453, 0.21322739124298096, 0.8235199153423309, 0.3984646201133728, 0.46459639072418213, 0.5436182022094727, 0.9407618939876556, 0.5690764486789703, 0.8153036832809448, 0.5076813995838165, 0.8492564111948013, 0.15788805484771729, 0.11221820116043091, 0.5724060535430908, 0.7176460325717926, 0.8440407365560532, 0.7636695057153702, 0.5076813995838165, 0.6323907971382141, 1.0598904006183147, 0.6086966693401337, 0.5732938051223755, 0.6028461754322052, 0.4580917954444885, 0.7809314131736755, 0.7321200668811798, 0.7066938281059265, 0.36259204149246216, 0.9492336809635162, 1.0921175256371498, 0.6050473153591156, 0.3663545846939087, 0.4949040412902832, 0.15298473834991455, 0.9291635751724243, 0.6140021979808807, 0.2852240204811096, 0.4260064363479614, 0.1934911012649536, 0.7881776839494705, 0.35110509395599365, 0.3110712170600891, 0.9965950401965529, 0.5273209810256958, 0.08015191555023193, 0.4719691276550293, 0.9257553964853287, 0.08657711744308472, 1.061875719577074, 0.7328509390354156, 0.41553622484207153, 0.4125799536705017, 0.6805701553821564, 0.4923962950706482, 0.06813162565231323, 0.5629296898841858, 0.4759763479232788, 0.5833230018615723, 0.36499106884002686, 0.9472270794212818, 0.5061180889606476, 0.9592425897717476, 0.5083636045455933, 0.14995676279067993, 0.5998384058475494, 1.0139076802879572, 0.49391037225723267, 0.7350744903087616, 0.8142646700143814, 0.6496772468090057, 0.7049342393875122, 0.7604426890611649, 0.7480517625808716, 0.7623411267995834, 0.6347159445285797, 0.5602153539657593, 0.1489863395690918, 0.18149614334106445, 0.3795756697654724, 0.6545434892177582, 0.5564406514167786, 0.6680121421813965, 0.9342424422502518, 0.657894492149353, 0.7992750257253647, 0.09565281867980957, 0.8649559319019318, 0.6687523424625397, 0.2839009165763855, 0.6356145441532135, 0.5763238072395325, 0.41780197620391846, 0.33846670389175415, 0.8244993835687637, 0.2919861078262329, 0.5979242324829102, 0.16055721044540405, 0.44577592611312866, 0.43739110231399536, 0.8500117361545563, 0.8017996996641159, 0.22006440162658691, 0.8103550374507904, 0.2716165781021118, 0.4894411563873291, 0.4743186831474304, 1.1532704830169678, 0.6796385049819946, 0.5788311660289764, 0.5738734304904938, 1.0360383354127407, 0.8758786469697952, 0.7802753150463104, 0.6680121421813965, 0.32175880670547485, 0.7403334379196167, 0.1184920072555542, 0.985954599454999, 0.5621178150177002, 0.05862969160079956, 0.9962523609865457, 0.3015354871749878, 0.6796385049819946, 0.8925864547491074, 0.28270334005355835, 0.6035843193531036, 0.5169978737831116, 0.2768561840057373, 0.4702553153038025, 0.5790992975234985, 1.1766748428344727, 0.47123968601226807, 0.5027961134910583, 0.8790005818009377, 0.7171919941902161, 0.6880250871181488, 0.8500117361545563, 1.069704793393612, 0.49865972995758057, 0.6096971333026886, 0.8759694769978523, 0.44577592611312866, 0.2793188691139221, 0.715606302022934, 0.3277524709701538, 0.8630160391330719, 0.6029638350009918, 0.672376275062561, 0.7672243714332581, 0.6264006495475769, 0.477161705493927, 0.8390759825706482, 0.41296738386154175, 1.1259145587682724, 0.595622718334198, 0.41994398832321167, 0.8203410059213638, 0.5836825668811798, 0.9287489503622055, 0.6963227391242981, 0.5261703729629517, 0.5353045463562012, 0.5869246125221252, 0.5790992975234985, 0.5850421190261841, 0.36374711990356445, 0.5985998511314392, 0.6088151931762695, 0.747675210237503, 0.06854182481765747, 0.6931126117706299, 0.645240068435669, 0.7922555953264236, 0.5883113443851471, 1.0752593576908112, 0.285663902759552, 0.6201569736003876, 0.56195929646492, 0.4090891480445862, 0.8667544275522232, 0.2510431408882141, 0.5058979690074921, 0.4431813359260559, 0.8103550374507904, 0.3243250250816345, 0.30453455448150635, 0.8527398705482483, 0.3313218951225281, 0.19052201509475708, 0.5382198095321655, 0.7340660691261292, 0.20157545804977417, 0.8104113787412643, 0.39328664541244507, 0.4090891480445862, 0.8347247093915939, 1.1893530488014221, 0.4659144878387451, 0.3405866026878357, 0.504460483789444, 0.3602169156074524, 0.6858391165733337, 1.0038859038613737, 0.5214316546916962, 0.8468800038099289, 0.47100013494491577, 0.8355625718832016, 0.9142830222845078, 0.7842075973749161, 0.48353803157806396, 0.6460709273815155, 0.9214013367891312, 0.16748452186584473, 0.7851828187704086, 0.9456217810511589, 1.0546553991734982, 0.6465227603912354, 0.7309760749340057, 0.7143962383270264, 0.6868995428085327, 0.29914140701293945, 0.8468800038099289, 0.1323196291923523, 0.6218132972717285, 0.36877548694610596, 0.8355625718832016, 0.29121387004852295, 0.22311073541641235, 0.10930991172790527], 'raw_distances_x_to_p': [0.07491215467453002, 0.1258436381816864, 0.07867926359176636, 0.2686879813671112, 0.2801799297332764, 0.0717882513999939, 0.0725929856300354, 0.23633030652999878, 0.10049613714218139, 0.2687165439128876, 0.21981453895568848, 0.060776901245117185, 0.20193864703178405, 0.17324100732803344, 0.2267237663269043, 0.2679001986980438, 0.2943339288234711, 0.09887873530387878, 0.18990106582641603, 0.013069415092468261, 0.22808496952056884, 0.177504962682724, 0.19701056480407714, 0.10048171281814575, 0.22985101938247682, 0.18766284584999085, 0.2594358205795288, 0.22143737077713013, 0.09927035570144653, 0.06417487263679504, 0.2943339288234711, 0.1856810450553894, 0.048224085569381715, 0.07560800909996032, 0.1954294800758362, 0.22143737077713013, 0.11648967862129211, 0.1531326115131378, 0.2031690001487732, 0.14105207920074464, 0.20138368010520935, 0.26318231225013733, 0.09289433360099793, 0.16757089495658875, 0.18792030811309815, 0.20214383006095887, 0.054546850919723514, 0.21114518642425537, 0.10422782301902771, 0.1210227906703949, 0.1002508282661438, 0.19361129999160767, 0.3329063892364502, 0.17842383980751036, 0.26788932681083677, 0.20689859390258789, 0.1385556697845459, 0.09504943490028381, 0.0713767647743225, 0.09155805706977845, 0.20662015676498413, 0.09082162380218506, 0.2263636529445648, 0.2943339288234711, 0.12961583137512206, 0.32856560349464414, 0.049546295404434205, 0.09032933712005616, 0.30195106863975524, 0.24248597025871277, 0.25832881331443786, 0.15774248242378236, 0.26330329179763795, 0.10972844958305358, 0.2620315611362457, 0.12930552363395692, 0.23038687109947203, 0.026624709367752075, 0.14155877828598024, 0.20760102868080138, 0.18119154572486879, 0.09862520694732665, 0.18418713212013244, 0.031426644325256346, 0.1537289261817932, 0.23018404245376586, 0.1488548696041107, 0.14751260876655578, 0.24583064913749694, 0.2943339288234711, 0.21902262568473815, 0.14626285433769226, 0.15618699193000793, 0.21385310888290404, 0.07923253774642944, 0.05288533568382263, 0.1357314705848694, 0.08316851258277894, 0.17194377779960632, 0.051333832740783694, 0.16315961480140687, 0.11301332712173462, 0.13821958899497985, 0.04227041006088257, 0.13023970127105713, 0.04402931928634644, 0.0550550639629364, 0.30488539934158326, 0.12046017050743103, 0.20294120907783508, 0.2607896029949188, 0.24519628882408143, 0.2092129111289978, 0.14207690358161926, 0.2014531373977661, 0.1417696475982666, 0.23628079891204834, 0.15885791778564454, 0.05424953699111938, 0.15618699193000793, 0.026917821168899535, 0.12801412343978882, 0.19133716821670532, 0.19039098024368287, 0.12497220635414123, 0.08388009071350097, 0.2687165439128876, 0.15618699193000793, 0.1058310627937317, 0.18474913239479065, 0.06041879653930664, 0.2943339288234711, 0.13736398816108703, 0.2687165439128876, 0.050558412075042726, 0.17581526637077333, 0.09195072650909424, 0.19391862154006959, 0.17581526637077333, 0.2424494206905365, 0.30195106863975524, 0.14327287077903747, 0.22033972144126893, 0.14627864956855774, 0.028694087266922, 0.31550863981246946, 0.18408670425415039, 0.2152866005897522, 0.10920886993408203, 0.1462748944759369, 0.265926456451416, 0.041348469257354734, 0.18667010664939881, 0.05466875433921814, 0.06425246000289916, 0.26812600493431094, 0.09431666135787964, 0.1754552900791168, 0.19701056480407714, 0.27325833439826963, 0.0812505841255188, 0.08923358917236328, 0.08142293095588685, 0.08049794435501098, 0.08388690948486328, 0.11004357933998107, 0.16105413436889648, 0.20935619473457337, 0.2286216974258423, 0.2687165439128876, 0.2943339288234711, 0.29934104084968566, 0.04505055546760559, 0.2687165439128876, 0.2687165439128876, 0.165482234954834, 0.2505683660507202, 0.2572839021682739, 0.13336470127105712, 0.10528640747070313, 0.18572738766670227, 0.02762399911880493, 0.28054004311561587, 0.2687165439128876, 0.2402168333530426, 0.21964013576507568, 0.20647518634796141, 0.12037063241004944, 0.23799507617950438, 0.18990106582641603, 0.23378283381462098, 0.0682909369468689, 0.09713414311408997, 0.2286216974258423, 0.313013356924057, 0.2574847638607025, 0.09879412055015564, 0.16634994745254517, 0.051359379291534425, 0.2650827825069427, 0.19282710552215576, 0.2687165439128876, 0.22142802476882933, 0.27086228132247925, 0.10801171064376831, 0.1440779447555542, 0.20884934067726135, 0.13752301335334777, 0.2809159100055695, 0.07321414351463318, 0.12935298681259155, 0.09392074346542359, 0.09345549941062928, 0.1448982834815979, 0.2368825912475586, 0.20223453640937805, 0.1077257513999939, 0.06970394849777221, 0.24059501886367798, 0.1531326115131378, 0.07047860026359558, 0.05411232113838196, 0.21195716857910157, 0.1774573504924774, 0.07035811543464661, 0.21114518642425537, 0.2943339288234711, 0.24689266681671143, 0.13023970127105713, 0.1622855007648468, 0.18095166683197023, 0.12736454010009765, 0.2687165439128876, 0.11606188416481018, 0.07301830649375915, 0.20024213194847107, 0.1914510667324066, 0.2687165439128876, 0.14800957441329957, 0.07394496202468873, 0.2943339288234711, 0.3224632441997528, 0.10174304246902466, 0.24306625127792358, 0.06830078363418579, 0.2943339288234711, 0.10397533774375915, 0.09609824419021606, 0.2799139261245728, 0.30195106863975524, 0.24489012360572815, 0.2943339288234711, 0.2669981956481934, 0.19226622581481934, 0.23796676993370056, 0.12403554916381836, 0.07518824338912963, 0.24027977585792543, 0.2129749596118927, 0.19538822770118713, 0.07956457734107972, 0.021501171588897704, 0.2117869734764099, 0.1697279155254364, 0.13646966218948364, 0.04563586115837097, 0.13630935549736023, 0.2325596272945404, 0.27604236602783205, 0.0848526418209076, 0.12355437874794006, 0.08359813094139099, 0.07769800424575805, 0.19930691719055177, 0.017387312650680543, 0.25326520800590513, 0.14934924840927125, 0.05194996595382691, 0.22447794675827026, 0.050048929452896115, 0.1993529200553894, 0.10217436552047729, 0.11040244102478028, 0.13426822423934937, 0.24725902676582337, 0.0681905210018158, 0.22447794675827026, 0.2687165439128876, 0.1155886709690094, 0.2221108555793762, 0.29718998670578, 0.14218377470970153, 0.14780757427215577, 0.24868152141571045, 0.15012385249137877, 0.2957450091838837, 0.23570191264152526, 0.07039734721183777, 0.20243200063705444, 0.2450391411781311, 0.2610181331634521, 0.2943339288234711, 0.21168712973594667, 0.23322834372520446, 0.22336766123771667, 0.14155877828598024, 0.03834769129753113, 0.04359009861946106, 0.17089332342147828, 0.23494293093681334, 0.22307827472686767, 0.10335022807121277, 0.0901472270488739, 0.1275405168533325, 0.2651544511318207, 0.10194430351257325, 0.0823241114616394, 0.2671786665916443, 0.05228369235992432, 0.05673014521598816, 0.23628079891204834, 0.137525075674057, 0.13061318397521973, 0.24059501886367798, 0.1940344214439392, 0.18790475130081177, 0.0804525375366211, 0.1972542643547058, 0.13630935549736023, 0.04387810826301575, 0.12807478308677672, 0.16763864159584047, 0.27681214213371275, 0.16889337301254273, 0.10969911217689514, 0.1264793336391449, 0.2058880627155304, 0.13305887579917908, 0.21114518642425537, 0.11689011454582214, 0.13438963890075684, 0.25695239901542666, 0.1478006660938263, 0.10499387383460998, 0.18095166683197023, 0.07256574034690857, 0.2998500466346741, 0.0661353349685669, 0.15724987983703614, 0.16255935430526733, 0.10478451251983642, 0.16101192235946654, 0.2666783809661865, 0.20674362182617187, 0.2074684500694275, 0.2637452960014343, 0.2687165439128876, 0.2943339288234711, 0.10662007927894593, 0.15771008729934693, 0.2045162320137024, 0.13224366307258606, 0.2799139261245728, 0.15469701290130616, 0.05253932476043701, 0.08710484504699707, 0.15890161395072938, 0.2323542058467865, 0.14316514134407043, 0.16188710927963257, 0.2687165439128876, 0.05047763586044311, 0.16098361611366271, 0.2687165439128876, 0.11001648306846619, 0.03917397260665893, 0.012268495559692384, 0.16453346014022827, 0.21224514842033387, 0.22161090970039368, 0.14690780639648438, 0.10857421159744263, 0.08916388154029846, 0.2687165439128876, 0.20792827010154724, 0.22307827472686767, 0.14967355728149415, 0.25636852383613584, 0.13464637994766235, 0.10542458891868592, 0.06425246000289916, 0.0630178153514862, 0.19448707699775697, 0.2340227782726288, 0.2687165439128876, 0.24868152141571045, 0.21385310888290404, 0.25854902267456054, 0.1818527340888977, 0.05183754563331604, 0.28751025199890134, 0.12577881813049316, 0.05676699876785278, 0.1897931694984436, 0.17065418362617493, 0.182553231716156, 0.028694087266922, 0.041348469257354734, 0.25440205335617067, 0.07136039137840271, 0.032577937841415404, 0.12833439707756042, 0.11628175973892212, 0.16251060962677003, 0.2943339288234711, 0.2687165439128876, 0.2554169952869415, 0.2056412637233734, 0.16429876685142517, 0.18444325923919677, 0.13646966218948364, 0.11304506659507751, 0.04368300437927246, 0.13392633199691772, 0.29578139781951907, 0.16492363810539246, 0.16255935430526733, 0.18444325923919677, 0.2693161189556122, 0.1712073028087616, 0.20294120907783508, 0.19065025448799133, 0.17955344915390015, 0.2943339288234711, 0.18558666110038757, 0.20904414653778075, 0.13144407868385316, 0.1312577486038208, 0.20102456212043762, 0.22140125632286073, 0.2943339288234711, 0.2687165439128876, 0.2687165439128876, 0.05089426636695862, 0.2081002414226532, 0.09939942359924317, 0.17711467146873475, 0.2813377916812897, 0.1275405168533325, 0.044525742530822754, 0.30195106863975524, 0.04257720708847046, 0.14626285433769226, 0.21185248494148254, 0.026778030395507812, 0.21441231966018676, 0.1540526568889618, 0.03564638495445251, 0.13489956259727479, 0.19494746923446654, 0.1498823344707489, 0.1713216245174408, 0.2957450091838837, 0.1910253345966339, 0.047457408905029294, 0.2687165439128876, 0.0846400499343872, 0.24712690114974975, 0.21114518642425537, 0.22140125632286073, 0.18413348197937013, 0.17220293879508972, 0.08344451785087585, 0.10791766047477722, 0.2943339288234711, 0.11980748176574707, 0.17051582932472228, 0.29050675630569456, 0.07664370536804199, 0.14097718000411988, 0.1090990126132965, 0.20662015676498413, 0.1486163854598999, 0.24198567271232604, 0.22140125632286073, 0.06425246000289916, 0.06679590940475463, 0.08926873207092285, 0.20193864703178405, 0.165316379070282, 0.2243583619594574, 0.23749199509620667, 0.10233506560325623, 0.22188621163368225, 0.24299496412277222, 0.07004004716873169, 0.11663140654563904, 0.2263636529445648, 0.11847202777862549, 0.27681214213371275, 0.2652188241481781, 0.1246204912662506, 0.24221355319023133, 0.18734511733055115, 0.0993520438671112, 0.18324695229530336, 0.08314414024353027, 0.2687165439128876, 0.3097450017929077, 0.17438538670539855, 0.15710639357566833, 0.16711366772651673, 0.2575950860977173, 0.1641591429710388, 0.2686879813671112, 0.2435591459274292, 0.30788477063179015, 0.2328833520412445, 0.2943339288234711, 0.20351722836494446, 0.1339774787425995, 0.1500688910484314, 0.1537289261817932, 0.2687165439128876, 0.19867417812347413, 0.24798166155815124, 0.03163349628448486, 0.1450836956501007, 0.2454463243484497, 0.043240517377853394, 0.27681214213371275, 0.07196561098098755, 0.2328833520412445, 0.2183497130870819, 0.2235489785671234, 0.0924670159816742, 0.18189479112625123, 0.09548647999763489, 0.27252782583236695, 0.2687165439128876, 0.17735087275505065, 0.17673290371894837, 0.23877571821212767, 0.22149975895881652, 0.06357990503311158, 0.2586200892925262, 0.17438538670539855, 0.12340834140777587, 0.19728720784187317, 0.09735530018806457, 0.2269219696521759, 0.2687165439128876, 0.15445377230644225, 0.22122191190719603, 0.07904585003852845, 0.265926456451416, 0.10367761254310608, 0.07003435492515564, 0.20137229561805725, 0.1289765179157257, 0.111567223072052, 0.0603985071182251, 0.2092129111289978, 0.25832881331443786, 0.2558863937854767, 0.10555083751678467, 0.2173720419406891, 0.0378653347492218, 0.11084820032119751, 0.20214383006095887, 0.08438714146614075, 0.2687165439128876, 0.11780235171318054, 0.2687165439128876, 0.19899271726608275, 0.09077228307723999, 0.1399840772151947, 0.14687377214431763, 0.105167555809021, 0.04305406212806702, 0.1743747115135193, 0.09195072650909424, 0.24299496412277222, 0.10525094270706177, 0.2801799297332764, 0.09630033373832703, 0.29934104084968566, 0.22234933376312255, 0.2687165439128876, 0.29934104084968566, 0.11002368330955506, 0.13696004152297975, 0.2943339288234711, 0.2575950860977173, 0.10142452120780945, 0.18189479112625123, 0.3224554419517517, 0.17229073643684387, 0.0851608693599701, 0.19139978289604187, 0.25695239901542666, 0.1909913182258606, 0.3245618581771851, 0.17286092638969422, 0.2687165439128876, 0.12159132361412048, 0.23253061175346373, 0.24027977585792543, 0.06982409954071045, 0.09837161302566529, 0.08640698194503785, 0.24048447608947754, 0.13646966218948364, 0.08163856863975524, 0.2243583619594574, 0.2687165439128876, 0.2476891040802002, 0.25287011861801145, 0.15283629298210144, 0.18016136884689332, 0.27681214213371275, 0.1574077844619751, 0.09922032356262207, 0.18095166683197023, 0.09856984615325928, 0.1839461386203766, 0.07881054282188416, 0.08569443821907044, 0.1804223418235779, 0.2763674437999725, 0.20935619473457337, 0.15054181814193726, 0.10751851201057434, 0.05906050801277161, 0.2687165439128876, 0.2476891040802002, 0.23040683269500734, 0.10335022807121277, 0.07769800424575805]}\n","\n","Generating reports at 2021-11-22 15:35:29.586247\n"]}]},{"cell_type":"markdown","metadata":{"id":"kelwzaKBfxcg"},"source":["---"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"olY--uGRV-uy","executionInfo":{"status":"ok","timestamp":1637596476870,"user_tz":-330,"elapsed":13,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"9b61ad4c-ea52-4dd1-cf4b-530c15b29ecc"},"source":["!apt-get install tree\n","!tree -a --du -h . -L 3"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":[".\n","├── [ 16K]  .config\n","│   ├── [   7]  active_config\n","│   ├── [   0]  config_sentinel\n","│   ├── [4.1K]  configurations\n","│   │   └── [  94]  config_default\n","│   ├── [   5]  gce\n","│   ├── [   3]  .last_opt_in_prompt.yaml\n","│   ├── [  37]  .last_survey_prompt.yaml\n","│   ├── [ 135]  .last_update_check.json\n","│   └── [8.0K]  logs\n","│       └── [4.0K]  2021.11.18\n","├── [1.9G]  coveo_reclist\n","│   └── [1.9G]  coveo_sigir.zip\n","├── [ 19M]  coveo_sigir.zip\n","└── [ 12K]  .reclist\n","    └── [8.0K]  CoveoCartRecList\n","        └── [4.0K]  P2VRecModel\n","\n"," 1.9G used in 8 directories, 9 files\n"]}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"DvXBahfQfxch","executionInfo":{"status":"ok","timestamp":1637598908502,"user_tz":-330,"elapsed":2859,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"46b34c4c-3c15-4846-965c-9cec16e7f277"},"source":["!pip install -q watermark\n","%reload_ext watermark\n","%watermark -a \"Sparsh A.\" -m -iv -u -t -d"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Author: Sparsh A.\n","\n","Last updated: 2021-11-22 16:35:08\n","\n","Compiler    : GCC 7.5.0\n","OS          : Linux\n","Release     : 5.4.104+\n","Machine     : x86_64\n","Processor   : x86_64\n","CPU cores   : 2\n","Architecture: 64bit\n","\n","json      : 2.0.9\n","numpy     : 1.21.2\n","IPython   : 5.5.0\n","gensim    : 4.0.1\n","matplotlib: 3.4.3\n","networkx  : 2.6.3\n","requests  : 2.22.0\n","\n"]}]},{"cell_type":"markdown","metadata":{"id":"5XlMYZkxfxci"},"source":["---"]},{"cell_type":"markdown","metadata":{"id":"4JwYKmJ5fxci"},"source":["**END**"]}]}