{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Learning in LTN\n", "\n", "This tutorial explains how to learn some language symbols (predicates, functions, constants) using the satisfaction of a knowledgebase as an objective. It expects basic familiarity of the first two turoials on LTN (grounding symbols and connectives)." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Init Plugin\n", "Init Graph Optimizer\n", "Init Kernel\n" ] } ], "source": [ "import ltn\n", "import tensorflow as tf\n", "import numpy as np\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We use the following simple example to illustrate learning in LTN.\n", "\n", "The domain is the square $[0,4] \\times [0,4]$. We have one example of the class $A$ and one example of the class $B$. The rest of the individuals are not labelled, but there are two assumptions:\n", "- $A$ and $B$ are mutually exclusive,\n", "- any two close points should share the same label." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEICAYAAABcVE8dAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAfU0lEQVR4nO3df5BU9bnn8feHYSIZJYuBCSoDM3pjjEAUYYJghFBuKqvGvWpianXHmJjdzApaSSp3vfFmsl7N3tmbpKyUMYm4U1dvtJgYrYgpFjX3akVLuPEXGkEBf+DdGRnlyogRxVE2wLN/9AHbpme6Z7pnupvzeVV1zTnf8z3nPH5tnj59fjytiMDMzA594yodgJmZjQ0nfDOzlHDCNzNLCSd8M7OUcMI3M0sJJ3wzs5RwwrdUk7RU0muSdkmanLNsRtJeN8Jt90j6XDL9PUn/UOR6uyQdN8iyr0laO5J4horP0mF8pQOw6iOpB5gK7AH2ApuA24CuiNhXxPotwP8F6iNizyjGWdJ+JNUDPwEWRMT63OUR8TJwRKlxJtv6X8PoW5Z9muXyEb4N5j9GxESgGfgh8F3g5sqGVHZTgQnAxkoHYjYWnPBtSBGxMyJWAf8J+Kqk2QCSviDpj5LekrRV0jVZqz2c/H0zOT2xUNJfSPq9pB2SXpfULWnS/hUkfVfSK5LelvS8pH+ftI+TdJWkl5J175T00cH2kxu/pMMkXS/p1eR1fdL2CeD5rPV/n2fdFkkhaXwy/5Ck/ynpX5I4/1nSlKz+X5HUm8TZkbOtayStSKZ/J+mKnOXrJX0xmQ5JH0+mJ0talYzz48BfDBZfVoz/NZkecsxz9j9f0rpkP69J+km+flbbnPCtKBHxONAHLEqa3gEuASYBXwCWSjovWbY4+TspIo6IiEcAAX8PHAOcCEwHrgGQdAJwBfDp5FvFfwB6km18EzgP+Gyy7p+AXwyxn1wdwAJgDnAyMB/4fkS8AMzKWv+MIofiPwOXAh8DPgT89+S/YSawHPhKEudkoGmQbfwKuGj/TLJuM3BPnr6/AN4Djga+nryKNeiY5/FT4KcR8REyHyp3DmM/ViOc8G04XgU+ChARD0XEMxGxLyI2ALeTScp5RcSWiLg/InZHRD+Zc+f7++8FDgNmSqqPiJ6IeClZ9t+Ajojoi4jdZBLWBdlHtQW0AT+IiO3Jfq8lk5RH6h8j4oWIeJdMUpyTtF8ArI6Ih5M4/wcw2PWOu4E5kpqzYlyZrHdAcrH4S8DVEfFORDwL3FpsoAXGPNefgY9LmhIRuyLi0WL3Y7XDCd+GYxrwBoCkUyU9KKlf0k7gMmDKYCtK+pikXyenbd4CVuzvHxFbgG+TSebbk37HJKs2A3dLelPSm8BmMh8QU4uM+RigN2u+N2kbqX/Lmh7g/Yu6xwBb9y+IiHeAHfk2EBFvkzmavzBpuhDoztO1kcyNFVuz2nrz9MtrqDHP478AnwCek/SEpHOK3Y/VDid8K4qkT5NJ+PtvCfwVsAqYHhH/DriJzCkEgHwlWP8+aT8pOW1wcVZ/IuJXEXE6mQQfwI+SRVuBsyJiUtZrQkS8Msh+cr2abHO/GUlbuW0jc8oEAEkNZE7rDOZ24KLkusOHgQfz9Oknc6fU9Ky2GVnT7yR/G7LajsqaHnLMs0XEixFxEZlTVT8CfiPp8CHitxrkhG9DkvSR5Gjv18CKiHgmWTQReCMi3pM0n8y57f36yZzOyL6XfCKwi8wF0mnAlVn7OEHSGZIOI3O++l0yR/GQ+SDp3H/6Q1KjpHOH2E+u24HvJ+tNAa4mc6Rbbr8BzpF0uqQPAT9g6H9f95L5IPoBcEe+210jYi+wErhGUkNyrv+rWcv7gVeAiyXVSfo6WRd1GWLMc0m6WFJjEsebSfPewfpbbXLCt8H8H0lvkznC7iBz/vfSrOXLgB8kfa4m6yJfRAwAncC/JKdiFpA5dz4X2EnmdMbKrG0dRubWz9fJnDL5GPC9ZNlPyXyT+OdkX48Cpw6xn1x/B6wDNgDPAE8lbWUVERuBy8l889lG5uJy3xD9d5MZg88l6wzmCjKnjf4N+CXwjznLv0Emke8gcxH6D1nLhhrzXGcCGyXtIjPmF0bEe0P0txok/wCKmVk6+AjfzCwlik74yTnCP0panWeZJN0gaYukDZLmljdMMzMr1XCO8L9F5pa4fM4Cjk9e7WQeQDEzsypSVMKX1ETmacrBqv2dC9wWGY8CkyQdXaYYzcysDIp9WvF64K/J3OaVzzQ++HBIX9K2LbuTpHYy3wA4/PDD533yk58cTqxmZqn35JNPvh4RjSNZt2DCT+7B3h4RT0paMli3PG0H3f4TEV1AF0Bra2usW7eu+EjNzAxJRT9tnauYUzqfAf5SmRrpvwbO2F/1L0sfH3wasInReZrRzMxGqGDCj4i/iYimiGghU/Pj9xFxcU63VcAlyd06C4CdEbEtd1tmZlY5I/7FK0mXAUTETWQeEz8b2EKmoNSlQ6xqZmYVMKyEHxEPAQ8l0zdltQeZx8pL8uc//5m+vj7ee89PdI/UhAkTaGpqor6+vtKhmFmVqarftO3r62PixIm0tLQg5S3qZ0OICHbs2EFfXx/HHntspcMxsypTVaUV3nvvPSZPnuxkP0KSmDx5sr8hmVleVZXwASf7Enn8zGwwVZfwzcxsdDjhl+iXv/wlV1xxRaXDMDMryAnfzCwlajrhd3d309LSwrhx42hpaaG7O9/vQA9PT08Ps2fPPjB/3XXXcc0117BkyRK++93vMn/+fD7xiU+wZs2ag9a95557WLhwIa+//jpf+9rX+OY3v8lpp53Gcccdx29+8xsgcyfNlVdeyezZs/nUpz7FHXfcAcCyZctYtWoVAOeffz5f//rXAbj55pv5/ve/T09PDyeeeCLf+MY3mDVrFp///Od59913S/7vNbP0qNmE393dTXt7O729vUQEvb29tLe3lyXpD2bPnj08/vjjXH/99Vx77bUfWHb33Xfzwx/+kHvvvZcpU6YAsG3bNtauXcvq1au56qqrAFi5ciVPP/0069ev54EHHuDKK69k27ZtLF68+MCHyCuvvMKmTZsAWLt2LYsWLQLgxRdf5PLLL2fjxo1MmjSJu+66a9T+W83s0FOzCb+jo4OBgYEPtA0MDNDR0TFq+/ziF78IwLx58+jp6TnQ/uCDD/KjH/2Ie+65hyOPPPJA+3nnnce4ceOYOXMmr732GpBJ4BdddBF1dXVMnTqVz372szzxxBMsWrSINWvWsGnTJmbOnMnUqVPZtm0bjzzyCKeddhoAxx57LHPmzMkbgxU2Gt8IzcplLN6fNZvwX3755WG1F2v8+PHs27fvwHz2Pe2HHXYYAHV1dezZs+dA+3HHHcfbb7/NCy+88IFt7e8PmVM52X9zTZs2jT/96U/87ne/Y/HixSxatIg777yTI444gokTJx60vdwYbGiV+EZoVqyxen/WbMKfMWPGsNqLNXXqVLZv386OHTvYvXs3q1cf9IuOB2lubmblypVccsklbNy4cci+ixcv5o477mDv3r309/fz8MMPM3/+fAAWLlzI9ddffyDhX3fddQdO51hpKvGN0KxYY/X+rNmE39nZSUNDwwfaGhoa6OzsLGm79fX1XH311Zx66qmcc845FPsjLSeccALd3d18+ctf5qWXXhq03/nnn89JJ53EySefzBlnnMGPf/xjjjrqKAAWLVrEnj17+PjHP87cuXN54403nPDLZLS+EZqVw5i9PyOiIq958+ZFrk2bNh3UNpQVK1ZEc3NzSIrm5uZYsWLFsNY/VA13HNOgubk5yPwozwdezc3NlQ7NbFjvT2BdjDDv1uwRPkBbWxs9PT3s27ePnp4e2traKh2SVanR+kZoVg5j9f6s6YRvVqy2tja6urpobm5GEs3NzXR1dfkgwarCWL0/FYPcNTLa8v2m7ebNmznxxBMrEs+hxONoduiS9GREtI5kXR/hm5mlhBO+mVlKFEz4kiZIelzSekkbJV2bp88SSTslPZ28rh6dcM3MbKSKOcLfDZwREScDc4AzJS3I029NRMxJXj8oZ5BWWS5JYIeyNL2/C/6mbXLf565ktj55VeZKr425/Y98738KcP8j34DvcLGal7b3d1Hn8CXVSXoa2A7cHxGP5em2MDntc5+kWeUMcjDd3dDSAuPGZf6W64P5vPPOY968ecyaNYuurq7ybLRGuSSBHcrS9v4ueIQPEBF7gTmSJgF3S5odEc9mdXkKaI6IXZLOBn4LHJ+7HUntQDuUXvOmuxva22H//6ve3sw8QKkfzLfccgsf/ehHeffdd/n0pz/Nl770JSZPnlzaRmuUSxLYoSxt7+9h3aUTEW8CDwFn5rS/FRG7kul7gXpJU/Ks3xURrRHR2tjYOOKgATo63k/2+w0MZNpLdcMNN3DyySezYMECtm7dyosvvlj6RmvUaBWpM6sGaXt/F3OXTmNyZI+kDwOfA57L6XOUJCXT85Pt7ih7tFkG+wAu9YP5oYce4oEHHuCRRx5h/fr1nHLKKR8okZw2Lklgh7K0vb+LOcI/GnhQ0gbgCTLn8FdLukzSZUmfC4BnJa0HbgAujFF+hHewD+BSP5h37tzJkUceSUNDA8899xyPPvpoaRuscS5JYIeytL2/a7a0Qu45fICGBujqKu0c/u7duznvvPN45ZVXOOGEE+jv7z/wm7a1wqUVzA5dpZRWKOqibTXan9Q7OjKncWbMgM7O0i/YHnbYYdx3332lB2hmVmVqNuFDJrkfot+8zMzKzrV0zKxqleMp2DQ9SVtITR/hm9mhqxxPwabtSdpCfIRvZlWpHE/Bpu1J2kKc8M2sKpXjKdi0PUlbiBO+mVWlcjwFm7YnaQtxws/R09PD7NmzKx2GWeqV4ynYtD1JW4gTvplVpXI8BZu2J2kLqe2EP0r1kffs2cNXv/pVTjrpJC644IKDLvqY2dhoa2ujp6eHffv20dPTM6JEXY5tHCpqN+Hvr63Q2wsR79dHLkPSf/7552lvb2fDhg185CMf4cYbbyxDwGZmlVW7CX8U6yNPnz6dz3zmMwBcfPHFrF27tuRtmplVWu0m/NGqjwwklZ4HnTczq0W1m/BHqz4ymXt0H3nkEQBuv/12Tj/99JK3aWZWabWb8Ds7M/WQszU0ZNpLdOKJJ3Lrrbdy0kkn8cYbb7B06dKSt2lmVmm1W0tnlOojt7S0sGnTpjIEaGZWXWo34YPrI5uZDUPtntIxM7NhqbqEX6mfXDxUePzMbDAFE76kCZIel7Re0kZJ1+bpI0k3SNoiaYOkuSMJZsKECezYscNJa4Qigh07djBhwoRKh2JmVaiYc/i7gTMiYpekemCtpPsi4tGsPmcBxyevU4Hlyd9haWpqoq+vj/7+/uGuaokJEybQ1NRU6TDMrAoVTPiROdzelczWJ6/cQ/BzgduSvo9KmiTp6IjYNpxg6uvrOfbYY4ezipmZFamoc/iS6iQ9DWwH7o+Ix3K6TAO2Zs33JW2522mXtE7SOh/Fm5mNraISfkTsjYg5QBMwX1Juwfh8tQcOOhEfEV0R0RoRrY2NjcMO1szMRm5Yd+lExJvAQ8CZOYv6gOlZ803Aq6UEZmZm5VXMXTqNkiYl0x8GPgc8l9NtFXBJcrfOAmDncM/fm5nZ6CrmLp2jgVsl1ZH5gLgzIlZLugwgIm4C7gXOBrYAA8CloxSvmZmNUDF36WwATsnTflPWdACXlzc0MzMrp6p70tbMzEaHE76ZWUo44ZuZpYQTvplZSjjhm5mlhBO+mVlKOOGbmaWEE76ZWUo44ZuZpYQTvplZSjjhm5mlhBO+mVlKOOGbmaWEE76ZWUo44ZuZpYQTvplZSjjhm5mlhBO+mVlKFPMj5tMlPShps6SNkr6Vp88SSTslPZ28rh6dcM3M3rds2VrGj+9D2sf48X0sW7a20iFVtWJ+xHwP8FcR8ZSkicCTku6PiE05/dZExDnlD9HM7GDLlq1l+fJTgMMB2Lu3ieXLjwTWcuONp1c0tmpV8Ag/IrZFxFPJ9NvAZmDaaAdmZjaUrq4W9if79x2etFs+wzqHL6kFOAV4LM/ihZLWS7pP0qxB1m+XtE7Suv7+/uFHa2aW2Lv3mGG12zASvqQjgLuAb0fEWzmLnwKaI+Jk4GfAb/NtIyK6IqI1IlobGxtHGLKZGdTVvTqsdisy4UuqJ5PsuyNiZe7yiHgrInYl0/cC9ZKmlDVSM7Ms7e09wDs5re8k7ZZPMXfpCLgZ2BwRPxmkz1FJPyTNT7a7o5yBmpllu/HG01m69I/U1fUB+6ir62Pp0j/6gu0QFBFDd5BOB9YAzwD7kubvATMAIuImSVcAS8nc0fMu8J2I+MNQ221tbY1169aVFr2ZWcpIejIiWkeybsHbMiNiLaACfX4O/HwkAZiZ2djwk7ZmZinhhG9mlhJO+GZWtbq7u2lpaWHcuHG0tLTQ3d1d6ZBqWjGlFczMxlx3dzft7e0MDAwA0NvbS3t7OwBtbW2VDK1m+QjfzKpSR0fHgWS/38DAAB0dHRWKqPY54ZtZVXr55ZeH1W6FOeGbWVWaMWPGsNqtMCd8M6tKnZ2dNDQ0fKCtoaGBzs7OCkVU+5zwzawqtbW10dXVRXNzM5Jobm6mq6vLF2xLULC0wmhxaQUzs+ErpbSCj/DNzFLCCd/MLCWc8M3MUsIJ38wsJZzwzcxSwgnfzCwlnPDNzFLCCd/MLCWK+RHz6ZIelLRZ0kZJ38rTR5JukLRF0gZJc0cnXDMrB9eZT6di6uHvAf4qIp6SNBF4UtL9EbEpq89ZwPHJ61RgefLXzKqM68ynV8Ej/IjYFhFPJdNvA5uBaTndzgVui4xHgUmSji57tGZWMteZT69hncOX1AKcAjyWs2gasDVrvo+DPxSQ1C5pnaR1/f39wwzVzMrBdebTq+iEL+kI4C7g2xHxVu7iPKscVJUtIroiojUiWhsbG4cXqZmVhevMp1dRCV9SPZlk3x0RK/N06QOmZ803Aa+WHp6ZlZvrzKdXMXfpCLgZ2BwRPxmk2yrgkuRunQXAzojYVsY4zaxMXGc+vQrWw5d0OrAGeAbYlzR/D5gBEBE3JR8KPwfOBAaASyNiyGL3rodvZjZ8pdTDL3hbZkSsJf85+uw+AVw+kgDMzGxs+ElbM7OUcMI3M0sJJ3wzs5RwwjczSwknfDOzlHDCNzNLCSd8M7OUcMKvcq5bbmblUkw9fKsQ1y03s3LyEX4Vc91yMysnJ/wq5rrlZlZOTvhVzHXLzaycnPCrmOuWm1k5OeFXMdctN7NyKlgPf7S4Hr6Z2fCVUg/fR/hmZinhhG9mlhJO+GZmKVHMj5jfImm7pGcHWb5E0k5JTyevq8sf5vC4HIGZ2cGKKa3wSzI/UH7bEH3WRMQ5ZYmoRC5HYGaWX8Ej/Ih4GHhjDGIpC5cjMDPLr1zn8BdKWi/pPkmzBuskqV3SOknr+vv7y7TrD3I5AjOz/MqR8J8CmiPiZOBnwG8H6xgRXRHRGhGtjY2NZdj1wVyOwMwsv5ITfkS8FRG7kul7gXpJU0qObIRcjsDMLL+SE76koyQpmZ6fbHNHqdsdKZcjMDPLr2BpBUm3A0uAKcBrwN8C9QARcZOkK4ClwB7gXeA7EfGHQjt2aQUzs+ErpbRCwdsyI+KiAst/Tua2TTMzq2J+0tbMLCWc8M3MUsIJ36yMXNbDqlkxpRXMrAgu62HVzkf4ZmXish5W7ZzwzcrEZT2s2jnhm5WJy3pYtXPCNysTl/WwaueEb1YmLuth1a5gaYXR4tIKZmbDV0ppBR/hm5mlhBO+mVlKOOGbmaWEE74VtHbZMvrGj2efRN/48axdtqzSIZkBsGzZMsaPH48kxo8fzzK/N4fk0go2pLXLlnHK8uUcnsw37d3LkcuXsxY4/cYbKxmapdyyZctYvnz5gfm9e/cemL/R7828fJeODalv/Hia9u49uL2ujqY9eyoQkVnG+PHj2ZvnvVlXV8eeQ/i96bt0bNQck+cf1FDtZmMlX7Ifqt2c8K2AV+vqhtVuNlbqBnkPDtZuRSR8SbdI2i7p2UGWS9INkrZI2iBpbvnDtErpaW/nnZy2d5L2seQ685arfZD34GDtBkTEkC9gMTAXeHaQ5WcD9wECFgCPFdpmRDBv3ryw2rBm6dLYWlcXeyG21tXFmqVLx3T/K1asiIaGhgAOvBoaGmLFihVjGodVn6VLl0ZdXV0AUVdXF0vH+L1ZCcC6KCLH5nsVddFWUguwOiJm51n2v4GHIuL2ZP55YElEbBtqm75oa8VqaWmht7f3oPbm5mZ6enrGPiCzCqr0RdtpwNas+b6k7SCS2iWtk7Suv7+/DLu2NHCdebPyKEfCV562vF8bIqIrIlojorWxsbEMu7Y0cJ15s/IoR8LvA6ZnzTcBr5Zhu2aA68yblUs5Ev4q4JLkbp0FwM5C5+/NhsN15s3Ko+BFW0m3A0uAKcBrwN8C9QARcZMkAT8HzgQGgEsjouDVWF+0NTMbvlIu2haspRMRFxVYHsDlI9m5mZmNHT9pa2aWEk74ZmYp4YRvZpYSTvhmZinhhG9mlhJO+GZmKeGEb2aWEk74ZmYp4YRvZpYSTvhmZinhhG9mlhJO+GZmKeGEb2aWEk74ZmYp4YRvZpYSTvhmZinhhG9mlhJO+GZmKVFUwpd0pqTnJW2RdFWe5Usk7ZT0dPK6uvyhmplZKQomfEl1wC+As4CZwEWSZubpuiYi5iSvH5Q5zjHX3d1NS0sL48aNo6Wlhe7u7kqHZGZWkoI/Yg7MB7ZExL8CSPo1cC6waTQDq6Tu7m7a29sZGBgAoLe3l/b2dgDa2toqGZqZ2YgVc0pnGrA1a74vacu1UNJ6SfdJmlWW6Cqko6PjQLLfb2BggI6OjgpFZGZWumKO8JWnLXLmnwKaI2KXpLOB3wLHH7QhqR1oB5gxY8bwIh1DL7/88rDazcxqQTFH+H3A9Kz5JuDV7A4R8VZE7Eqm7wXqJU3J3VBEdEVEa0S0NjY2lhD26Brsw6iaP6TMzAopJuE/ARwv6VhJHwIuBFZld5B0lCQl0/OT7e4od7BjpbOzk4aGhg+0NTQ00NnZWaGIzMxKVzDhR8Qe4Argn4DNwJ0RsVHSZZIuS7pdADwraT1wA3BhROSe9qkZbW1tdHV10dzcjCSam5vp6uryBVszq2mqVF5ubW2NdevWVWTfZma1StKTEdE6knX9pK2ZWUo44ZuZpURqE76fpH2fx8IsHYq5D/+Q4ydp3+exMEuPVF60bWlpobe396D25uZmenp6xj6gCvJYmNUWX7QdJj9J+z6PhVl6pDLh+0na93kszNIjlQnfT9K+z2Nhlh6pTPh+kvZ9Hguz9EjlRVszs1rli7ZmZlaQE76ZWUo44ZuZpYQTvplZSjjhm5mlhBO+mVlKOOGbmaWEE76ZWUo44ZuZpURRCV/SmZKel7RF0lV5lkvSDcnyDZLmlj9UMzMrRcGEL6kO+AVwFjATuEjSzJxuZwHHJ692YHmZ4zQzsxIVc4Q/H9gSEf8aEf8P+DVwbk6fc4HbIuNRYJKko8scq5mZlaCYnzicBmzNmu8DTi2izzRgW3YnSe1kvgEA7Jb07LCirYwpwOuVDqIIjrO8aiHOWogRHGe5nTDSFYtJ+MrTlltis5g+REQX0AUgad1IK76NJcdZXo6zfGohRnCc5SZpxGWGizml0wdMz5pvAl4dQR8zM6ugYhL+E8Dxko6V9CHgQmBVTp9VwCXJ3ToLgJ0RsS13Q2ZmVjkFT+lExB5JVwD/BNQBt0TERkmXJctvAu4Fzga2AAPApUXsu2vEUY8tx1lejrN8aiFGcJzlNuI4K/aLV2ZmNrb8pK2ZWUo44ZuZpcSoJ/xaKctQRJxLJO2U9HTyuroCMd4iaftgzy9U0VgWirMaxnK6pAclbZa0UdK38vSp+HgWGWc1jOcESY9LWp/EeW2ePtUwnsXEWfHxTOKok/RHSavzLBvZWEbEqL3IXOR9CTgO+BCwHpiZ0+ds4D4y9/IvAB4bzZhKiHMJsHqsY8uJYTEwF3h2kOUVH8si46yGsTwamJtMTwReqNL3ZjFxVsN4Cjgima4HHgMWVOF4FhNnxcczieM7wK/yxTLSsRztI/xaKctQTJwVFxEPA28M0aUaxrKYOCsuIrZFxFPJ9NvAZjJPh2er+HgWGWfFJWO0K5mtT165d4RUw3gWE2fFSWoCvgD8wyBdRjSWo53wByu5MNw+o63YGBYmXwXvkzRrbEIblmoYy2JVzVhKagFOIXO0l62qxnOIOKEKxjM5BfE0sB24PyKqcjyLiBMqP57XA38N7Btk+YjGcrQTftnKMoyyYmJ4CmiOiJOBnwG/He2gRqAaxrIYVTOWko4A7gK+HRFv5S7Os0pFxrNAnFUxnhGxNyLmkHnSfr6k2TldqmI8i4izouMp6Rxge0Q8OVS3PG0Fx3K0E36tlGUoGENEvLX/q2BE3AvUS5oydiEWpRrGsqBqGUtJ9WSSaHdErMzTpSrGs1Cc1TKeWfG8CTwEnJmzqCrGc7/B4qyC8fwM8JeSesicXj5D0oqcPiMay9FO+LVSlqFgnJKOkqRkej6ZsdsxxnEWUg1jWVA1jGWy/5uBzRHxk0G6VXw8i4mzSsazUdKkZPrDwOeA53K6VcN4Foyz0uMZEX8TEU0R0UImF/0+Ii7O6TaisSymWuaIxeiVZahEnBcASyXtAd4FLozkcvlYkXQ7mTsIpkjqA/6WzEWnqhnLIuOs+FiSOYr6CvBMcj4X4HvAjKw4q2E8i4mzGsbzaOBWZX4waRxwZ0SsrrZ/60XGWQ3jeZByjKVLK5iZpYSftDUzSwknfDOzlHDCNzNLCSd8M7OUcMI3M0sJJ3wzs5RwwjczS4n/DwqnXn6OpdOlAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "points = np.array(\n", " [[0.4,0.3],[1.2,0.3],[2.2,1.3],[1.7,1.0],[0.5,0.5],[0.3, 1.5],[1.3, 1.1],[0.9, 1.7],\n", " [3.4,3.3],[3.2,3.3],[3.2,2.3],[2.7,2.0],[3.5,3.5],[3.3, 2.5],[3.3, 1.1],[1.9, 3.7],[1.3, 3.5],[3.3, 1.1],[3.9, 3.7]])\n", "point_a = [3.3,2.5]\n", "point_b = [1.3,1.1]\n", "fig, ax = plt.subplots()\n", "ax.set_xlim(0,4)\n", "ax.set_ylim(0,4)\n", "ax.scatter(points[:,0],points[:,1],color=\"black\",label=\"unknown\")\n", "ax.scatter(point_a[0],point_a[1],color=\"blue\",label=\"a\")\n", "ax.scatter(point_b[0],point_b[1],color=\"red\",label=\"b\")\n", "ax.set_title(\"Dataset of individuals\")\n", "plt.legend();" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We define the membership predicate $C(x,l)$, where $x$ is an individual and $l$ is a onehot label to denote the two classes. $C$ is approximated by a simple MLP. The last layer, that computes probabilities per class, uses a `softmax` activation, ensuring that the classes are mutually-exclusive.\n", "\n", "We define the knowledgebase $\\mathcal{K}$ composed of the following rules:\n", "\\begin{align}\n", "& C(a,l_a)\\\\\n", "& C(b,l_b)\\\\\n", "\\forall x_1,x_2,l\\ \\big(\\mathrm{Sim}(x_1,x_2) & \\rightarrow \\big(C(x_1,l)\\leftrightarrow C(x_2,l)\\big)\\big)\n", "\\end{align}\n", "where $a$ and $b$ the two individuals already classified; $x_1$,$x_2$ are variables ranging over all individuals; $l_a$, $l_b$ are the one-hot labels for $A$ and $B$; $l$ is a variable ranging over the labels. $\\mathrm{Sim}$ is a predicate measuring similarity between two points. $\\mathcal{G}(\\mathrm{Sim}):\\vec{u},\\vec{v}\\mapsto \\exp(-\\|\\vec{u}-\\vec{v} \\|^2)$.\n", "\n", "The objective is to learn the predicate $C$ to maximize the satisfaction of $\\mathcal{K}$. If $\\theta$ denotes the set of trainable parameters, the task is :\n", "$$\n", "\\theta^\\ast = \\mathrm{argmax}_{\\theta\\in\\Theta}\\ \\mathrm{SatAgg}_{\\phi\\in\\mathcal{K}}\\mathcal{G}_{\\theta}(\\phi)\n", "$$\n", "where $\\mathrm{SatAgg}$ is an operator that aggregates the truth values of the formulas in $\\mathcal{K}$ (if there are more than one formula).\n", "\n", "To evaluate the grounding of each formula, one has to define the grounding of the non-logical symbols and of the operators." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Metal device set to: Apple M1\n", "\n", "systemMemory: 16.00 GB\n", "maxCacheSize: 5.33 GB\n", "\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2021-09-24 17:12:13.181397: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.\n", "2021-09-24 17:12:13.181501: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: )\n" ] } ], "source": [ "class ModelC(tf.keras.Model):\n", " def __init__(self):\n", " super(ModelC, self).__init__()\n", " self.dense1 = tf.keras.layers.Dense(5, activation=tf.nn.elu)\n", " self.dense2 = tf.keras.layers.Dense(5, activation=tf.nn.elu)\n", " self.dense3 = tf.keras.layers.Dense(2, activation=tf.nn.softmax)\n", "\n", " def call(self, inputs):\n", " \"\"\"inputs[0]: point, inputs[1]: onehot label\"\"\"\n", " x, label = inputs[0], inputs[1]\n", " x = self.dense1(x)\n", " x = self.dense2(x)\n", " prob = self.dense3(x)\n", " return tf.math.reduce_sum(prob*label,axis=1)\n", "\n", "C = ltn.Predicate(ModelC())" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "x1 = ltn.Variable(\"x1\",points)\n", "x2 = ltn.Variable(\"x2\",points)\n", "a = ltn.Constant([3.3,2.5], trainable=False)\n", "b = ltn.Constant([1.3,1.1], trainable=False)\n", "l_a = ltn.Constant([1,0], trainable=False)\n", "l_b = ltn.Constant([0,1], trainable=False)\n", "l = ltn.Variable(\"l\",[[1,0],[0,1]])\n", "\n", "Sim = ltn.Predicate.Lambda(\n", " lambda args: tf.exp(-1.*tf.sqrt(tf.reduce_sum(tf.square(args[0]-args[1]),axis=1)))\n", ")" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEICAYAAABcVE8dAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABAGklEQVR4nO29eXxkV3Xv+10qzT0PUne7uyV5nsCxTWNMSHi+hHsDDoT7EhJMOkyfXCu2w81zAi8JOGDjZ2V4uSSBMJgOIWBbmEDIdcDPfgn38giYPExsxzY2xsZuWoN7kHpQd2uWqtb9Y53tc1RdkkpSlWpa38+nPqo659Q562yd+p191lp7bVFVHMdxnOqnrtQGOI7jOKuDC77jOE6N4ILvOI5TI7jgO47j1Agu+I7jODWCC77jOE6N4IKfQETeLSIPJT6riJxXSpvmQ0TuFJEPrdKx5rRLkY7RISKjIpJa5vdHReSc6P3nReSOFdjyoIi8a4H1q9b2iWPeISJHReTwKh7zgIi8fhWO80ER+Wyxj+NAfakNKDUicgD4L6r6P4q0/88Dg6r6hyvYx7sxG38mLFPVG1ZuXc5jdQE/ARpUdbYYx8iFqvYDa1fw/WV/N8e+3hjer2bbz4eI7AbeB3Sq6tBqHns1UNU/yndbEbkNOE9Vf32BbQ5QxN90JeM9/BIjIjV/0y0XxCjH30QncKwaxd5ZZVS1pl/AAeD10ft3Aw8l1inWmwD4FtZrIHtbQIC/AIaAk8CTwMuAbmAGmAZGga8njvn70XZT2JPWHwAvAKeBHwL/e7TtxcAkkI72MRIt/zxwR/T+GmAQ6wUOAYeA9yRs3QJ8HTgF/BtwR/I8s9qjPzrv0ej16nCuwH8DTmBPAG9MfGcD8DfRcV+M9p+aZ/9XAY9EthwB/jxa3hUdtz7R3ncA/xraLjqP3sR5dM3zv0q2zSbgfmA4sv1+YFfie98CeoDvAhPAeeF/nU/bR5/fBDwOjET2XpZY9/tRm5wGngV+bp522QDcFdnZB/wh1iF7fWRXJrLh8/N8fyEbcl5bifXXA88k1l+ZuE7fj12nJ4G/A5rnOf67ozb8q2jbHyXPFTgL+BpwHHgeuD6x7jbgnqzr4F3YtXgUuCVa9wbstzQTtcUTOey4O2qriWib34uW/yLwdNQ+3wIuXkATPgYMYNfZo8DPllqnCqZ3pTag1C8KI/g/H10YGzHxvxjYEa37PAlxSBzzcWA30BIt+5XoR1EHvA0YS+xjjl3Z+8UEfxa4HWgArgXGgU3R+i9Fr1bgkuhink/www+uPutcZzBhSAE3AgcBidbfB3wGWAO0A98HfnOe/f//wDui92uBq3MdN2rv54FzMTH8IfAcJoD1mDj+7Tz/q2TbbAF+OTr3dcBXgPsS3/sWJiyXRvttSP6v82j7K7Gb7KuitnlX9P9tAi6M2vqsxDmeO0+73AX8Y2RjV3Suv5H4/w4ucA3Pa0Me19avYDekV2LX7nmY64hoH9+PvrsZuyncMI8N78auwd+J2vBtmPBvjtb/C/ApoBm4HLux/Vy07jbOFPy/BlqAn8I6RRdnb5vPbzr6fEF0zv8xsu33sGurcZ7v/zp23dRjnajDzHOjq7RXOT6+ViIz2A/1IkwEn1HVQ4t85+OqOqCqEwCq+hVVPaiqGVX9O+DHWG94KTbcrqozqvoA1ru5MAqC/jJwq6qOq+oPgS8s8fwA+lT1r1U1HX1/B7BNRLYBbwRuVtUxNbfDXwDXLWDneSKyVVVHVfV7Cxzzb1X1BVU9CTwIvKCq/0MttvAV4IrFjFbVY6r61ejcT2O9+f8ta7PPq+rTqjqrqjOL7TOL64HPqOrDqppW1S9gAnU19mTQBFwiIg2qekBVX8jeQfQ/ehvwAVU9raoHgI8C7yiADYtdW/8F+L9V9d/UeF5V+xL7/nj03ePYU9blC9gxBPxldA3+HfZE8wtRDOJngN9X1UlVfRz47CLn9xFVnVDVJ4AnMOFfLm8D/h9V/Ub0//1v2M3kp3NtrKr3RNfNrKp+lPjmXfG44BcAVf0m8Angk8AREdknIusX+dpA8oOIvFNEHheREREZwVxCW5dgxjGdG2Qdx3rQbVhPJXm8OcfOk5eyQ1R1PHq7FvMvNwCHErZ/Buvp5+I3sB7Xj0Tk30TkTQsc80ji/USOz4sGakWkVUQ+IyJ9InIK+DawMSsbaDntEegE3hfOPTr/3Viv/nngZqxXOiQiXxKRs3LsYyvQiLlyAn3AzpXaAIteW7sxd898JLOCwjU1Hy9q1EVOnMNZ0et4dMNNrlvo/JZy3MU4i0TbqmoG+5/nPL6IvE9EnhGRk1F7bWBpv8WyxQU/f8Ywt0Bge3Klqn5cVV+BuQYuAP7PsGqe/b20XEQ6sUfY9wJbVHUj8BT2iL3QPvJhGHvU3pVYtnuB7Zd6rAGsN7lVVTdGr/WqemnOnav+WFXfjt0Q/hT4exFZs8RjLoX3Yb2zV6nqeuC10XJJbLPQOS/WHgNAT+LcN6pqq6reC6CqX1TL8OmM9vWnOfZxFHvy6Uws68BcLfkwrw15XFsDmNusEOwUkWS7dmCuv4PAZhFZl7Uu3/NLks/1mb3NQRJtG9m4O9fxReRnsbjLr2Iu0Y2Ya0qyt61EXPDz53Hgl6Ie43lYTxUAEXmliLxKRBqwG0MI9IH1Ss9ZZN9rsIt0ONrfe7BeWOAIsEtEGpdqdOSC+Qfgtsj2i4B3LvCVYSzotZjNYf+HgH8GPioi60WkTkTOFZFstwkAIvLrItIW9bJGosXpXNsWiHXY08CIiGwGbl3i9xdr+78Gboj+/yIia0TkF0RknYhcKCKvE5Em7JqYIMe5Rv+jLwM90fc6gd8F7snTxnltYPFr67PA+0XkFdF3z4uOvxzagd8WkQYR+RUslvWAqg5ggeQ/FpFmEbkM+/30LuMYR4CuRbKpsn9zX8ZcSz8X/Ubfh3VS/jXHd9dhHaRhoF5EPgws9rReMbjg589fYBkCRzAfdvJiXY/96E5gj47HMD8hWPbKJdHj9H25dhz51T+KBTSPAC/HMh4C38QyDA6LyNFl2P5e7LH0MJbFcC92weeyZZwoayWy+eo89v9OzCXxQ6wN/h7z8efiDcDTIjKKZUNcp6qTSziXpfKXmL/2KPA94P9d4vcXbHtVfQTzoX8CO/fnsQAmmO/3T6JjH8YE8YPzHOe/Yp2F/VhG1BeBz+Vj4EI2LHZtqepXsP/3F7EsnfuwAO1yeBg4HzvfHuCtqnosWvd2LCB7EPjvWEzpG8s4xleiv8dE5LF5tvlj4A+j6/f9qvosFoj9q8i2NwNvVtXpHN/9Jyxe9Bz2W55kZS6/siJkWTg1hIj8KbBdVd9Valuc6iDXADWn/PAefg0gIheJyGXRI/tV2OP0fy+1XY7jrC55C76IpETk30Xk/hzrREQ+LiLPi8iTInJlYc10Vsg6zI8/hvkzP4rlfDuOU0Pk7dIRkd8F9gDrVfVNWeuuxXyQ12KDPz6mqq8qsK2O4zjOCsirhy8iu4BfwCL6uXgLcFc0cON7WJ7zfEE7x3EcpwTkW7jrL7HhyOvmWb+TuZHswWjZnNGmItKN1ZdhzZo1r7jooouWYqvjOE7N8+ijjx5V1bblfHdRwY9GQg6p6qMics18m+VYdoavSFX3AfsA9uzZo4888kj+ljqO4ziISN/iW+UmH5fOa4BfFKsx/SXgdSKSPSBkkLmjN3dh+baO4zhOmbCo4KvqB1R1l6p2YQWxvqlnTj7wNeCdUbbO1cBJXbx4mOM4jrOKLHvyDRG5AUBV7wQewDJ0nscKHb2nINY5juM4BWNJgq+q38JqhQehD8sV+K1CGuY4juMUFh9p6ziOUyO44DuO49QILviO4zg1ggu+4zhOjeCC7ziOUyO44DuO49QILviO4zg1ggu+4zhOjeCC79QMvb29dHV1UVdXR1dXF729y5lD23GKw2pcn8sureA4lURvby/d3d2Mj08AdfT19dHd3Q3A3r17S2ucU/PE16cCWrTrs2STmHt5ZGc16erqoq+vD9gKbAKOASfo7OzgwIEDJbXNcez6PAG8AngOeBGAzs7OM65PEXlUVfcs5zjew3dqgv7+/ujdSaARaAM20dd3tHRGOU5EX99B4JXAFHD4peXxdVsY3Ifv1AQdHR3RuxlsqoY+YJazznoFBw7A+HjJTHNqHFXYseM/Yh2RJ4H0S+vi67YwuOA7NUFPTw+tra2JJZO0tg5z223dpNPQ3w+DgzA9XTITnRrlJz+Bd73rd2huHgBGX1re2tpKT09PQY/lgu/UBHv37mXfvn10dnYiInR2drJv3z6uv/5tnHMOtLdbL3//fjh8GNLpxffpOCvl2DET/He84/V89rMfOeP6LHRCgQdtHScinYajR2FkBERgyxbYvNneO06hmZyE738fmppgzx5IpfL7ngdtHacApFKwbRts2gRDQzA8bOK/dSts2FBq65xqIpOBH/zA/r785fmL/Upxl47jZNHYCLt2QUeH/RAPHcIDu05B+fGP4dQpuOQSmBNaKjKLCr6INIvI90XkCRF5WkQ+kmOba0TkpIg8Hr0+XBxzHWf1aG2Fri446yw8sOsUjMOH7Trq6LDY0WqSTw9/Cnidqv4UcDnwBhG5Osd231HVy6PX7YU00ikttV6SYP16OOccaGuLA7tHjnhgt1pYzet7bAx+9CPYuBHOO69oh5mXRX340QTlIVeoIXqVJtLrrDrxkO9pQGq2JEEI4m7caIHdEyfg5EkP7FY68fWdAtYU9fpOp+HJJ81N+LKXleaayStLR0RSwKPAecAnVfX3s9ZfA3wVGMRGtbxfVZ9eaJ+epVMZzC1J0AhMAxN0dLTT17e/tMaVkOlpC+yOjkJDg/X+168vtVXOUomv71cDu4CvALlLGqyUp56ya+aKKywxYLmsJEsnr6CtqqZV9XKsRa4SkZdlbfIY0Bm5ff4KuG8eQ7tF5BEReWR4eHg59jqrTDy0+wRwChBgA/390wwPw+nTMDtbOvtKRXZg9+BBD+xWIvH1vYtQv2bu8sIwMGBuwHPOWZnYr5QlZemo6gjwLeANWctPqepo9P4BoEFEtub4/j5V3aOqe9ra2pZttLN6xEO705hnbxg4wu7dlqd46pRdyENDJv4zMyUytESEwO6OHXbj6++HF1/0wG6lYNd3G/b0OpC1vDCcPGlZOVu3QmdnwXa7LPLJ0mkTkY3R+xbg9cCPsrbZLmIeKRG5KtrvsYJb66w6Z5YkgNbWJv74jz9IWxts32456iIm/kNDdgM4daq2xH/DhjiwOzZmoyc9sFv+9PT00Nx8PpABDgGFLWkwPW359s3NcOmlpY/15DPwagfwhciPXwd8WVXvF5EbAFT1TuCtwI0iMgtMANdpqYbwOgUlBK5uueUW+vv76ejooKen56XlqRSsXWuvdNpGD05MWG//9Glb39Jir8bGUp5J8amrsyDuhg1zA7tbt9pjfKl/7M6Z7N27l0cf3UVv750MD6fp6Oicc32vBFV4+mnr+OzZA/VlMMzVSys4RSGTicV/asou/lTKejotLTacvNqZmrLRuh7YLV9GRuD+++Gqq+CCCwq77/377Unv4ottLEeh8NIKTtlRV2f+7dbWueI/Pm4uj7q6uT3/auz9NjVZYHdszFxdBw9ar7+93c7bKT2Dg/Z39+7C7jcURduxo7Biv1Jc8J2ikxR/1dzin+z5V5v4r1kDZ59t7p3hYejrg3XrrMdf7W6ucmdgwNxwhbwBT06aK2ftWrjwwsLttxB4LR1nVRGxH9fmzdb72bLFxH5y0npFhw7B8eN2Q6i2KFAI7G7dGgd2h4Y8sLsQhRgFO98+Jibsmitk7z4URVOFyy5bvaJo+eI9fKdkiJjYNzfbD2Rqyn6E4Qkgub6lpTp6/nV1JvhhxO7x4/GIXQ/sziUeBWuDG5YzCnahfVx1le1j167C2fzcc5ahdtll5em286CtU5YkxT+dNiFsarIfUXOzCWc1MDVlvfyxMQvstrebu8dJjoLdDjRh01IubRRsvI8LgKPA8Zf28bd/e4CTJ+E//+fC2Hv4sLlyOjuLWyfHg7ZO1dHUFGfyTE+b+IcbQFgfgr6VLP5NTeZSCIHdF1+0c/LAbnK069XAWoLgL2UUrG1bB7wReAT4LgB9fS9y+DCcf35hbB0dhWeesSe3c88tzD6LQQX/VJxaobHR/N/bt1ugc9066/WPjJjP/+hR+8FVsi88BHa3b7e87b4+E/9aGryWTTzadQir5ZTKWp7vPrZiUjf00vKdO19JOl0Y//3srPnt6+tLVxQtX1zwnYqisdFy2bdti90fmYz5wQ8fjvPeK1X8N26cG9jdv792A7vxKO9hrIbTliWPgrWRtKGegdXvam1t5Td+41YaGwtTj/6ZZ+zp82UvK//xJe7ScSqWhgZ7rV9vvazg9jl50l4NDbHbpxxGOeZLMrA7PBwHdsOycu5BFpIQmP3AB/6EgQFhx47L+bM/e+eSRsHu3buXp57axmc+821OnDhJZ2cnd9zRQ2Pjz7Nr18rbcmDAbsjnnVfaomj54kFbp+qYnY0zfUIRs/r6WPwbGkpr31JJBnYbG2O3Vi3xyU9aTvvrX7/07957r91E3/Y2+3zkCHzjG/Da11q10+UyMgKPPWY34ssuW/5+looHbR0nQX393Po+Idgb6vsE8W9uroyBT/MFdrdts3OoBdra7Glnqaja917+8njZ4KDdAHbsWL4909NW37652ealrRRc8J2qJlncLZOJ3T6VWNxtzRorxXzypAWqDxwwd1ZbW+U9tSyVbdvgiSdMwJfihjlxwp74tm2Llw0MWHB8uW1WjkXR8qWCTHWclVFXZ6K5Zs3c+j5jYxborYTibiLmx1+/3kaJHj9uN65Nm2zwVrmN7CwUbW0m3MeP23nmy9BQ/H0wN8zoqJUqXi7795sdF19cea41F3ynJqn0+j51dSZimzbVRmA3ZNMMDS1N8I8csZtg+E4olrbc0bXhyeqss8qrKFq+uOA7NU+o79PSMlf8ww2gnMW/vt580Zs2xZPPnDhRfYHdzZvtXIeGrGedL8PDdhMMg/NWUixtYsJcOevWlV9RtHxxwXecBNninyzxMD4e1/cJQd9yEf/mZss4GR2NA7utrdYzrobAbkhVHRpafNskQ0PxaNpQLO3yy5d+/FAUDSwAXKmju13wHWcesou7JUs8hOJu5VbfZ+1ai1GEUszVFNhtb4dnn81/+1On7EYd3EErcec895zFSn7qpyq75IULvuPkQRD3pibzkU9Nxa6fycnyEv/5ArubN5s7oxxuTMuhvR2efNJuZhs2LL59eBoIgj8wYDfEjRuXdtxDh+yJqbPTnjIqGRd8x1kGQfw3bFi4uFtzc+kyZ0JgN5RiPnbMslQqNbCbDNzmK/gi1gYzM1Z6Y6m+99FR+NGPLEZSzkXR8mXRe72INIvI90XkCRF5WkQ+kmMbEZGPi8jzIvKkiFxZHHMdp/xIFncL9X1CcbdyqO/T0GCB3a4uuxEdOWKTr4yOlsae5bJ1qwl4vn78oaE42HvokPnhl+LOCUXRGhrKvyhavuTTw58CXqeqoyLSADwkIg+q6vcS27wROD96vQr4dPTXcWqKZH2fmZnY7RPq+zQ2xkHf1R6wkx3YHRysrMBufb25pJYi+KEa5uAgSy6WFoqiXXll+Q/Ky5dFLzm1YjuhL9AQvbIL8LwFuCva9nsislFEdqjqoYJa6zgVRBD/devi4m6TkxZMPHWqdMXdQmB3ZCTOK9+wwXrQ5R7YbW+30tGLMT5uN7b2dgu4Dw6ypGJp/f1xhs9Sff7lTF7hGxFJicjjWEHpb6jqw1mb7AQGEp8Ho2XZ++kWkUdE5JHh5RTGcJwKpb4+nrh8+3YTWBET/iNH7HXq1OrVvxeJ/dJbttix9+8391Mmszo2LIf2dhsYNza28HbJgO3QkMVZ8nXnjIzA88/b/2olxdXKkbwEX1XTqno5sAu4SkRelrVJrvvmGWU4VXWfqu5R1T1tYayz49QYob5PEP+NG23Z6dMmTocPm/snVPosJiGwe845dkM6dgxeeMFErxwnkU8GbhciWVJhYMDaN5+RsZVaFC1flpSgpaojwLeAN2StGgSSc8fsAg6uxDDHqQVSKXOvbN1qgdWNG82tMjZmve0g/lNTxbWjocEEMQR2Dx8uz8Bu6CfmI/jr15twDw7ajXUxt5mqif3MjJU7rqSiaPmST5ZOm4hsjN63AK8HfpS12deAd0bZOlcDJ91/7zhLIxR327LFBGrTplj8jx61TJORERP/YvW+Q2B3Z+SQHRy0HnJINy01YRzEYh7hoSGrkBmKpeXjztm/38pSXHSRPYFVI/ncw3YAXxCRFHaD+LKq3i8iNwCo6p3AA8C1wPPAOPCeItnrODVBqYu7rVtnopcd2G1rK33PN/jl52N62uy+9NL8R9eGc9y5c2V18sudfLJ0ngSuyLH8zsR7BX6rsKY5jgP51fcJg7wKWd8nBHbDiN0TJyy4u2WL5beXasRuW5uVOpiayl3GOvT+29qsFMNixdKSRdEuuKA4NpcLVeilcpzqJbu+z2oUd0ulrFcdSjEfPTp3xO5qEwK3w8O5e+6h9x+C0AsVS6uWomj54oLvOBVKUvxhrvgXo7hbCOyGUsyHD1uvv73dYg+rRZi9amgot+AfOWKusBMn7PNC7pxnn62Oomj54oLvOFVCqO8DC9f3aWlZmfi3tFghsZBGOjBggt/evjozhbW22vHm8+MPD5stg4PWy5/vKeTQITh40DKTKr0oWr644DtOFdLYGNf4mZ6Oe/0jI/ZqaopdP8st7hYCuydOmOvkJz8xcd26tfiB3fkCt7OzZsvu3Sbm8xVLSxZFO+ec4tpaTrjgO06VE8Q/1PcJPf9kfZ/Q81+q+ItYAHfDhrmB3c2bixvYbW+3rJrZ2bk3l2PH4pHCmUxcSyfJ7KyVWa6momj54oLvODVEsrhbqO+TFP/l1vdJBnaHhuLAbltbfqWMl0qokXP0qI1ZCIRe/+Sk3chyDej/4Q9tfTUVRcsXF3zHqVFCfZ9Q3C24fZLF3YLbJ9+iag0Nlss+MWHie+iQTcBS6MBussRCtuDX19vNa/fuM3vv/f3m46+2omj54oLvOA719eaPX7vW6vaHYO/p0/aqr4+zffLpFRc7sLthg+0n248/NGQ2zsycmZ0TiqK1t1dfUbR8ccF3HGcOobjb2rXmBw9unyD+qVTs9llM/IsZ2M0O3Kpa772p6cxiadPTlm/f0gIXX7yy41YyLviO48xLqO+zZo2Jf3D7jI1ZpksqNbfEQy6Sgd3g2y9EYLe9HZ54wuyqqzPX0eysHS9ZLE3VxH52Fq64ovSlIUpJDZ+64zhLIVnfJ4h/GOEb6vsEt0+u+j6plA2ayh6xu9zAbnu7ifjx4/bEMDRkN6OWlrnZOaHc86WXVm9RtHxxwXccZ8mspLhbY6MFdsfH48DuiRMm/EsJ7CYDt0HwT582sQ/VPoeHbYasnTvnBndrFRd8x3FWRD7F3XLV92lttVGup06ZMA8MWA+8vT2/wHCYoDwUSxsasieP9nY71sSEpWDWQlG0fHHBdxynYGQXd0uWeJivvs/69SbKx49bYHf//vwCuyL2VHDkiH0eHLT97dplwv/kk7b8ssuqvyhavrjgO45TFIK4h0lLkj3/yckzxX/LFtsuGdgNpZjnGw0bSiCfOhXPcrV7ty0bHbWiaKG4nOOC7zjOKpFvcbetW+MRu8PDsX8/V2B32zbryYfA7IUXWgyh1oqi5YsLvuM4q06yuFuyvs/ISLx+48a4Rk8I7La3m+8/EAK3Tz1lAduODuvdb95cW0XR8sUF33GckpKs7zMzE2f8nDxp69essRvA6dNWGiEZ2N2yxdw9YVDVxITdJC69tLaKouXLooIvIruBu4DtQAbYp6ofy9rmGuAfgZ9Ei/5BVW8vqKWO41Q9QfxDfZ/g8lE14R8dNVdPyN+/9daH2LfvElQ3AmMcOXKAu+9+ec0VRcuXfHr4s8D7VPUxEVkHPCoi31DVH2Zt9x1VfVPhTXQcpxZJFncL9X2amsylc/w43HDDE9x335VA8PGs4+tfP5cPfOAhPvWpnyml6WXLoslKqnpIVR+L3p8GngF2Ftswx3GcQKjv09Zmg6jOOw/uu6+NWOwDrezb11UCCyuDJfnwRaQLuAJ4OMfqV4vIE8BB4P2q+nSO73cD3QAdtVquznGcJTMzY2mdU1OW4TM+DuZlPpN0+qycy50lCL6IrAW+CtysqqeyVj8GdKrqqIhcC9wHnJ+9D1XdB+wD2LNnjy7XaMdxqpd02kQ9KfBhFquZGZua8J//GWACOLMWQyp1EFhg5vIaJi/BF5EGTOx7VfUfstcnbwCq+oCIfEpEtqrq0cKZ6jhONRLm3J2ettfMTLyusdF89um0lVb+5jetXEJ9PZx77igvvNAMJOdlHKO7+wAu+LnJJ0tHgL8BnlHVP59nm+3AEVVVEbkKiw0cK6iljuNUPLOzca99ctLEXaNn/VTKgrJr18Z5+mNjNnftv/wLPP20fe/KKy318gc/2MbVV/+Ehx9ej+omUqmDdHcf8IDtAuTTw38N8A7gByLyeLTsg0AHgKreCbwVuFFEZrHnrOtU1V02jlPDZDJnumbSaVsXyiqsW2fC3twcT6CeyVjO/f798PDDNqhqctJGzl55pVW9fO97Ldf+r//6bO67z773q7+6C+/ZL8yigq+qDwELDmFQ1U8AnyiUUY7jVB7BJRMEPumaCfPjhikSc+XJz85aTZyjR+Gxx2ww1fS0lU84/3x7dXZCd7fdHP7kT2pvEvKV4iNtHcdZMiGwGnzvU1Oxa6auznrva9bY38bGhatVTk+b0I+MmNvmiSdsv+3tsGOHvS680D7ffrvNS/uxj9lyZ2m44DuOsyChxn3owU9Oxq4ZMEFfuzYW94aG/PY7OWnlE8bGTMSfeMJG0m7dahUvW1utd3/hhfZkcN998PWvw/XXw0//dFFOtepxwXccZw4zM2f63gP19bFbprnZxH0pNWtULYf+5Enb94svwuOPW2G0devg1a+Oyyt0dcG3v93LW95yC319LTQ13cNrX7uW66+/sNCnXDO44DtODZPJzBX2qak4572uzoR9/fpY5FOphfe30HHGxsx1Mz1tpRGefNKqYDY1mdCvW2c3g/XrzV//9a/38pu/2c34eAq4h6mpIzz00Ju5994/Y+/evQVrg1rCBd9xaojsnnuunPdQtz5f18xCpNOWcXP6dFwG+emnLQMnlYLLL7ee/IkTtn7XLjj7bHuSuOWWWxgfHwc+CrQD1zMxcYhbbrnFBX+ZuOA7TpWSTs8d0JQMrKZSse89ZM0UchrAmRnrzY+NmR3ptI2QfeYZs+GiiyytcmrKevsNDXDuuXMnLOnv74/efQ34DvBU1nJnqbjgO04VEHLek5kz+eS8F5rJSevNT0yYTamUDZz693+3tMuzz4aXv9zsGRkxF86mTVYMLcyGFejo6KCvrw/4lzOWO8vDBd9xKpDsYmLJwGrIeQ+umWLnqquawJ86FT9FtLSY0H//+3YT2LnTJhPfsMGEf2jIbghnn23plbkCvz09PXR3d0duHaO1tZWenp7inlAV44LvOGXOQsXEQs77hg2xwBfSNbMQIRA7Omp21dWZi+jQIfjGN6wH394OP/MzVta4rs5uCsePW6zgvPNs+/kIfvpbbrmF/v5+Ojo66Onpcf/9CnDBd5wyQjWe5i+I/OxsvL6xce6ApkIEVpdKOm0iPzpqttbX2/yzx4/Dgw+a4G/cCP/hP1gZhLo668EfOmTntWOHzT2bj1tp7969LvAFxAXfcUrIQjnvqZS5ZoLvvamptPO0zsyYf3583ES/sdEmCx8bg//5Py3zZs0a+NmftWwbMPvHxuDgQbs5XXyx+eyd0uCC7zirRHYxsWTOewisFiLnvdBMTcWBWFWzc+NGO5eHHrJyxQ0Nlkt/9tm2TUODncOLL5obZ/NmOOccr31TalzwHadIZGfNZBcTCznv8xUTKyUhEHv6dPzU0dJir0wGHnnEMm8yGbjiCrjkEuv1i5hffmICnnvOPp97rvnyS/l04hgu+I5TAPLJec+3mFgpyWTMZRMGSqVSdmNqjaaOffLJOPPm4ovhFa+w5el0fAN78UWreLl+vfXqw3ed0uOC7zhLJFlMLLhmsouJJf3u9RXwK0un44yb2Vl7AgnuJRHrrf/rv5p7prMTXvMaO7fJSdt2wwbr1T/zjO1r1y57leuNrVapgEvRcUrLQjnvKy0mVmpmZkzkx8bsRpYcfVtXB/398N3vWt58ezv8p/8EW7bErp4NG2zbgwfh8GFz+Zx/vi13yg8XfMdJsFjOe2NjnPNeToHVpRICsZOT9jlMTlJfb+d59KgFZPv67Hzf9CareXPypPXyW1vtxjA+Ds8+a737tjbr/ZciVdTJDxd8p2bJznlfjWJipUQ1Ln0QBkq1tsalFkSst//d71rdm5YWeN3rrBTC6dNw7JjdELZutW2Hh2Fw0L57zjkm+E55k88k5ruBu4DtQAbYp6ofy9pGgI8B1wLjwLtV9bHCm+s4yyd7Au3snPfkBNqlznkvJJlMnHETBko9+OCXueOODzEw0EdHx24+9KH/i7PPvo7HH7cbwdVXwytfadsPD9t+Nmywm8DkpAn9yIgtO/tsu2k45U8+PfxZ4H2q+piIrAMeFZFvqOoPE9u8ETg/er0K+HT013FKQj4TaK9fX/xiYqUkBGLHxky4QwmGr371Xm6++SYmJsYA6O/fyk03Pcpb33oB7373lfz0T9vTzMiIfa+lxfLu02krY9zfb+27a5fVyKmWG2MtkM8k5oeAQ9H70yLyDLATSAr+W4C7VFWB74nIRhHZEX3XcYrOSifQriZmZmKhz2TsvNevj902t932YSYmTgIXAdcAa5iZ+THf+c57uPvuJzh50sQ+uG8aG61Xf/iw9fZbWqxXv25dSU/TWQZL8uGLSBdwBfBw1qqdwEDi82C0bI7gi0g30A1e4tRZPtkTaOcqJlYJOe+FZmrKfPCTk3HFypaW+OlFxER8YOAFQIFWYAT4R+Agg4NrOXLEvrt+vQn6zIwFavv7rZ3b2qwOTiWkmjpnkve/TUTWAl8FblbVU9mrc3xFz1igug/YB7Bnz54z1jtONvlMoF3qYmKlJARiQ8VKEWuPkD8PdsNLpeIbX1xn/tHo1QC0s3Pn2TQ0WK2bujrz+x87ZkXPGhosMLtlS2nO0ykMeQm+iDRgYt+rqv+QY5NBYHfi8y7g4MrNc2qNfIqJVWrOeyEJgdjRUWuj+nrrlYdgs+qZQh+I68xPABuANbS0NHH77TfT1hYXSTt40P5u2GApmdkTlDiVRz5ZOgL8DfCMqv75PJt9DXiviHwJC9aedP+9sxirNYF2NZFOW+57CMQ2NlqPPMQlVGPXzXyurL179zI5meLWW/+cF188xO7dm+jp+X327r3upYlMXnzR9rV7t5U4rtUba7WRTw//NcA7gB+IyOPRsg8CHQCqeifwAJaS+TyWlvmeglvqVDyrPYF2NRECsePjdlMMGTf19SbMoUe/kNCH/YyMwM///HW8+c3XsWmTtXXY/9CQ1bVvabFe/UITlDiVRz5ZOg+R20ef3EaB3yqUUU7lU8oJtKuJqSkT4lCauKXFfPSplAl/JpOf0Ktaz/30adtu0ybbT6iKOT5uufWzsxaY3b3bn6iqEY+1OyumXCbQrhZCIHZszAQf7OlnzRoT63R6rtAv5m6ZmLBefTpt+9iwwb4bBqIdO2Y9+xCY9QlKqhcXfGfJlNME2tVECMSOjVmbplL2FBR64pmMiXa+Qj87a0IfKlpu2WL/j3BDmZqywOzYmMVKurr8/1XtuOA7C5JPznspJtCuJpKB2FCaeOPGOCsmPC3lK/Sq5ro5fdq23bgx9sWHXv2pU5ZuKWIjZrdt88BsLeCC77xEJUygXU3MzJjQhzlik+UeYOlCD/a/Gxmx/1trq92MU6l4PMPMDBw5YtuEEbM+QUnt4IJfw1TSBNrVRAjEhhGxzc12Iw2jV5cj9Om0ifjEhH2nrS1+Qgi9+okJc+HMzlpte5+gpPZwwa8RKnUC7WohGYidnrbPYerAkFoZhD6Viuve5LPf0VFz0YD16NeujQdfhfTXY8fi8sbnnusTlNQqLvhVSiVPoF1NhEDs+Lj9H+rqrDff2mrvVWO32VKEHuymnaxoGfLyIU6LnZkxX/3EhN3QfYKS2sYf6Mqc3t5eurq6qKuro6uri97e3jO2CWVwT5ww/2x/v/3Ijx2zH319vQXutm2z/OqzzrKMjZAH7xSedNqCpsPD9n/JZExwt241N5mICX06HY9LWIr75vhx27eq7XPLlljsg/vm1Cn4yU/sGti1C847z8W+1vEefhnT29sb1TwZB6Cvr4/rr+9merqOX/7lt1fNBNrVRAjETkzY/6WxETZvjgeXZTLx09ZSe/QQu2+SFS3D99PpONA+NGTbhRGzLS0FP1WnAnE5KGNuueUWxscnsTK2TUAjExON/OEffpzXve7tFT+BdjUxNWVCPzkZ16DfsMH+J6mUifFKhH562tw309N2I9+0ae7NPOnCO3TIjtXebk9zHph1Ai74ZURycE0mA319w8AaYCM2u+Q0cJKDB4fZtcsDq6UmBGLHx+MRsaEGfUNDPJo1ZD8tR+gzGatHPzZm3928eW4aZSYT32SOH7fJxxsazH2zfn3hztWpDlzwS0hS3MMrUFcHHR076e/fj9Wji/02nZ2dLvYlJBmInZmJA7EtLbEffnZ2+cHYwPi49eozGYu3rF8/t7ceevWzs3FgduNGn6DEmR+/LFaJkHaXFPmAiAlCKIAVxPyP/ujWOT58gNbWVnp6elbZegfiEbETE/Fk4OvWxfWBgtCH/+1yhT5UtJyaissfJ4OtyV796KhNPQiWgeMTlDgL4YJfBIJrJinuoVIkmAiER/6FBGHv3r2A+fL7+/vp6Oigp6fnpeXO6pAMxGYycemDkFkDhRH6+SpaZtsSxlAMD5u7p7XVRsz6BCXOYrjgF4Bscc92zQQBqKtbegBt7969LvAlIgRig8A2NZnbJjkwLSn09fXLj6vMV9EyECaLCZk4hw6ZO2f7dtixw4P1Tn644C+R7MBqsvcuEs/UFMTdf4iVRTIQG4Ktzc2xfz7UpSmU0M9X0TJJKIGhajn9ITB7wQU+QYmzNFzwFyEp7On0XNdMcuKJ5fTenfIhBGKDf17EXCXNzbGgZzKx0IdpBJcr9MmKljC3omVymzBp++ys+erHxy1TxycocZaDC36CZLbMfIHV+SaGdiqTZCB2dtZEfO1ac98EV1wYLFUIoYf5K1omCQXPVC0lMwRmzz7bJyhxls+isiUinxORIRF5ap7114jISRF5PHp9uPBmLo18yhGErJkwWCX86MNIRZF4Mo9Q5CrMtepiX/nMzFjA8+hRy3QRMeHduNF86I2Ntmxmxl6qJvQrKSyXTlu5i6NH7XNbm/XWk/sLvfpQSfPIEZtQvKUFLr7Yxd5ZGfn08D8PfAK4a4FtvqOqbyqIRSskVzmC7u5uMhm47rq98+a8rySw6lQOIRAbJnJpbIwHSoUc+kL36BeqaJkk2aufnjahn5210bI+QYlTCPKZxPzbItK1CrYUBCtHMI7Nu14P1DE+Drfccge/9Et7PbBagyQDsaG33twc++ezhV61MEIPC1e0TNqXfLI8ccJSLpua4MILfYISp3AUyof/ahF5AjgIvF9Vn861kYh0A90AHR0dBTr0XPr7+6N3KaABG6E6w8DA87S2urjXEslAbBDTlpZ4oFTIlS+G0KfT5jIaH7d9bdmSu4BZKGMcsr8OHrTvbN3qE5Q4hacQgv8Y0KmqoyJyLXAfcH6uDVV1H7APYM+ePZprm5XS0dFBX18fMBu9jM7OThf7GiEEYicnTehTqdgvnxTzYgg9LFzRMpCcnKSuzgKzBw/ae5+gxCkWK+4/qOopVR2N3j8ANIjI1hVbtkx6enpozXoG9nIEtUEIxB47Fgdi16838QxB95B1k5wUZqXB2MD0tJUlHhmxmMC2bXb8bLFPp+P0z7o6G0Q1OGh+/YsvdrF3iseKe/gish04oqoqIldhN5FjK7ZsmXg5gtojGYhVNbFduzb2zwe3SMijDz36QmVcZTLWox8dzV3RMtvWIPQzMzAwYDeAXbusnLHjFJNFBV9E7gWuAbaKyCBwK+YcR1XvBN4K3Cgis8AEcJ2qFsVdky9ejqD6SQZig4g3Nc0dKBXEPJ2OB80VUuhh8YqWgVASIZMx24aHLT2zpQXOP98nKHFWh3yydN6+yPpPYGmbjlN0sgOxdXUm8sFdk5wmsJhCv1hFyyShjHGwK0w76BOUOKuNj7R1KoJkIDbMA7tmTZw/n6xOWUyhz6eiZSBZxrihwdItDx0ye32CEqcUuOA7ZU0oTRzcIQ0N5h8PIp/MaS+m0MPiFS2TJHv1qZRNLH/6tE9Q4pQWf5h0ypKpKesRnzhhveT6eusRr10bu3CCaIYSGcmSGGFgXSGYnTV/+7Fjts/2duvZ59r/3Xf30tFxIU1N67j00gu5554v8+MfW9plRwecc46LvVM6/NJzyobsQCyYsCcLmSVTJ5M9+mTl0kLas1hFyyRf+MIXufHG32FiYgyYpr9/kptv/hi3397Cb//2m32CEqfkeA/fKTmZjPWAjx0z/3g6bb34ZP58Mk8+V4++0O6byUkrXHbqlGXQbN8+v9iHQPKHPnQHExNRwRwuALYwOfkTPvGJ/+pi75QF3sN3SkYyEBvSFVtbTbyzM27C9sXs0YdjjIyYgNfXW0XLhcQ6TE4CMDDwPDADbMBqOT0HjDEw4EO8nfLABd9ZdZKB2FB2uLU1Fvns+WDDBCBQPKHPt6JlcvtkxlBzM3R2nhWV9TgJnAJsOEqx6kY5zlJxl46zamQHYhsarM5MmHAkBGKDyIZywSHfvrGxOPMRTE1ZSYSTJ82Gbdty178JzM7aDSudjue5Fcku62Fi72U9nHLCe/hOUUkGYtNpW9bYaEIZeuvZE4AE1w3E2xSj8F0mY+6bxSpaJm0LN6BUKj6HgJf1cModF3ynKCRHxIaJRJqb43TJ7Iyb1RR6mFvRct263EXOkiQnJ2lsPHOi8YCX9XDKGRd8p6BkB2Lr6+MZpXL531db6KenrVc/PW099E2bFs6LT/bq6+rsXLwUglOpuOA7BSEZiIW5gdjk9JGBbKFPTkhSDJZS0TKQnJxkoV6941QKLvjOigiliUNt+SCMwW2T3VtfbaGH/CtaJm1MTk7ivXqnWvDL2FmU3t5eurq6qKuro6uri3vu6WViwgZKnTxp7o6mJvOFt7TEQdmGhljIVW276ek4lTHMQFUssZ+ZsTLEx4/bcbZts9GyC4l3cnKSULfHxb58uemmm7juurfxtrf9CvX19dx0002lNqms8R6+syC9vb10d3e/NDF8X98w3d0f4NSpJn71V9/6UiA2FAnLNUH3avfol1LRMklycpKWlsJMd+gUj5tuuolPf/rT2JQckE6no8/wqU99qoSWlS9SqrlK9uzZo4888khJju3kT1dXVzSYaA3Qgo0gnWH37h0899wP550LthRCD0uraBnILmMcbmBOeVNfX086nSYIPvw9AKlUitkwUq8KEZFHVXXPcr7rPXxnQfr7+xOfZoApQBkYeCFnRcpSCf3srAl9GNC1ZUt+QdZkGWPv1VcW6XCRcXye5U42LvjOguze3UF//4vAbPTKALN0dnackXUzO2u9ZFg9oV9qRctAsldfX28xB+/VVxapVCoS92+esdzJzaLhKBH5nIgMichT86wXEfm4iDwvIk+KyJWFN9NZbVTNn33bbX9MS8s6II317mdobW15qVxA2G56Oi6AVoxgbHbguLe3d0kVLZNMT1vmjqoNBmtudrGvRLq7u5e03AFUdcEX8FrgSuCpedZfCzyIOXevBh5ebJ+qyite8Qp1yo90WnVqSnViwl4zM6p3332PdnZ2qohoZ2en3nPPPZrJqE5Pq05O2mtmRjWTKY5N99xzj7a2tipWoEahTltadunHPvZVPXTIjp/vuY2Pq54+bedWLHud1ePGG2/UVCqlgKZSKb3xxhtLbVLRAR7RPDQ21yuvoK2IdAH3q+rLcqz7DPAtVb03+vwscI2qHlponx60LS+C3z2UQZjPJZPLdVPsGZziwDHAWsAmg929exN9fU/l1TufmbEsHJG5s2U5TqVR6qDtTmAg8XkwWnaG4ItIN9ANXjK2XAilh5PzwOZygWYLfa7MnGIRB46bsVrzk8AIg4OHFhX7XGWM3X3j1CqFGFKS6+eT87FBVfep6h5V3dPW1laAQzvLIYh3yDsPQh+mEkySycz10YcA52rGxeLOwSQwDBwD0ot2GkK5h0xmbhljx6lVCiH4g8DuxOddwMEC7NcpMEmhD9MDhlII8wn9zEw8SclqC31gbp15m15qoTrzoVc/NRUPompoWCVjHaeMKYTgfw14Z5StczVwcjH/vbO6hEyaZC33ZM2bJLmEPtcNYTXZu3cv+/bto7OzExGhs7OTffv25SxDHCYnCeUevA6O48QsGrQVkXuBa4CtwBHgVqABQFXvFBEBPgG8ARgH3qOqi0ZjPWhbfDKZ3AHWXG6NTCZ30LZSWGxyEsepFooatFXVty+yXoHfWs7BneKQFPpk6YN8hH41g7GFIt/JSRyn1vHktCoi34wbiG8KYdtKFHovY+w4S8MFvwqYnTWxrxWhB5+cxHGWgwt+hZIsUqZqPdswjWAuqkXos3v1zc2VeR6OUwpc8CuMkFqZXY2y2oUe7JynpuIyxk1NpbbIcSoLF/wKIQRXk0K/UIGybKFfqPdfCYQyxj45ieMsHxf8Mic7tXKhjBuY6+apBqH3yUkcp3C44Jcp2cXM6usXLvhVbUIPPjmJ4xQaF/wyYymplWH7ahN6n5zEcYqDC34ZkCvjZrHgaq7tK13oYW6vvrnZyxg7TiHxn1MJCUIf5lteLLUSqlfoMxnLwEmnvVfvOMXCBb8E5EqtXGxKwGoVepg7OYn36h2nePhPaxVZSjGzQDULvU9O4jiriwv+KrCUYmaBahZ6iCdVAXPfeL16xyk+LvhFZKkZN8nvQHUKvZcxdpzS4YJfBJZSzCz7OxALfbW5N5JljL1X7zirjwt+gVhOxk0yHTN8pxqFPtmr9zLGjlM6XPBXyFKLmYXv1ILQg09O4jjlhAv+MllqMTOoLaH3yUkcp/zI6ycoIm8QkWdF5HkR+YMc668RkZMi8nj0+nDhTS0PMpl4NGhykFBDw/zCHZ4CwnfCJOILfaeSSadtIvGZGTvP1lYXe8cpBxb9GYpICvgk8EbgEuDtInJJjk2/o6qXR6/bC2znqtPb20tXVxd1dXV0dXVx991ffEnoVU3owyChpQp9tfbqwdw3ExNxwTN34ThO+ZCPS+cq4HlV3Q8gIl8C3gL8sJiGlZLe3l66u7sZHx8H6ujrO8Rv/uZ7SaeFd7zj7Ytm3GS7boJfv1pFHnxyEsepBPJ50N4JDCQ+D0bLsnm1iDwhIg+KyKUFsa5E3HLLLZHYN0QvZWJihNtu+8CCYl+LPXqw852YsPNvaXGxd5xyJZ8efi6p0qzPjwGdqjoqItcC9wHnn7EjkW6gG6Cjo2Nplq4i/f390bsoxzI63Xj5XGqxRw8+OYnjVBr59PAHgd2Jz7uAg8kNVPWUqo5G7x8AGkRka/aOVHWfqu5R1T1tbW0rMLu4xDcjJXlvy75J1WqPHuycx8fn9uqr/Zwdp9LJR/D/DThfRM4WkUbgOuBryQ1EZLuI/dxF5Kpov8cKbexq0dPTQ2tr65xlra2t9PT0ACZyoRZMrQl9JmNCPz1t59va6jNROU6lsKhLR1VnReS9wD8BKeBzqvq0iNwQrb8TeCtwo4jMAhPAdaqa7fapGPbu3QuYL7+/v5+Ojg56enr4tV/by8zM3GqXteC6CXgZY8epbKRUurxnzx595JFHSnLspRJcN7Uq9D45ieOUDyLyqKruWc53vY+2ALmEvtZ6tckyxt6rd5zKxn++OQhlE4LQLza/bDXik5M4TvVRswPes0fS9vb2kslYj3ZmJh5N29RU/WKf3RZ33fVFxsdN7JuaLAvHxd5xKp+a7OHPHUkLfX39XH/9TczM1PFrv/b2murRn9kWh7nhhptJp+t417uu8xo4jlNF1GTQtquri76+PmxMWQp70FE6OnbR17e/JDaVirgtUkAT1iZTdHbu5MCBAyW1zXGcM/Gg7RKJR8xa2QQbUZthYOBAyWwqFdYWAjQDGSyrVucdVew4TuVSkw/s8YjZmeiVyVpeO9g5KzBJEPt4ueM41URNCn48kjZ2ZyVH0tYScVukX1pWq23hONVOTQr+3r172bdvH52dnYgInZ2d7Nu376URtrWEt4Xj1A41GbR1HMepVFYStK3JHr7jOE4t4oLvOI5TI7jgO47j1Agu+I7jODWCC77jOE6N4ILvOI5TI7jgO47j1Agu+I7jODWCC77jOE6NkJfgi8gbRORZEXleRP4gx3oRkY9H658UkSsLb6rjOI6zEhYVfBFJAZ8E3ghcArxdRC7J2uyNwPnRqxv4dIHtdBzHcVZIPj38q4DnVXW/qk4DXwLekrXNW4C71PgesFFEdhTYVsdxHGcF5DMByk5gIPF5EHhVHtvsBA4lNxKRbuwJAGBKRJ5akrWlYStwtNRG5IHbWVgqwc5KsBHczkJz4XK/mI/g55q+OrvEZj7boKr7gH0AIvLIciu+rSZuZ2FxOwtHJdgIbmehEZFllxnOx6UzCOxOfN4FHFzGNo7jOE4JyUfw/w04X0TOFpFG4Drga1nbfA14Z5StczVwUlUPZe/IcRzHKR2LunRUdVZE3gv8E5ACPqeqT4vIDdH6O4EHgGuB54Fx4D15HHvfsq1eXdzOwuJ2Fo5KsBHczkKzbDtLNuOV4ziOs7r4SFvHcZwawQXfcRynRii64FdKWYY87LxGRE6KyOPR68MlsPFzIjI03/iFMmrLxewsh7bcLSL/n4g8IyJPi8j/kWObkrdnnnaWQ3s2i8j3ReSJyM6P5NimHNozHztL3p6RHSkR+XcRuT/HuuW1paoW7YUFeV8AzgEagSeAS7K2uRZ4EMvlvxp4uJg2rcDOa4D7V9u2LBteC1wJPDXP+pK3ZZ52lkNb7gCujN6vA54r02szHzvLoT0FWBu9bwAeBq4uw/bMx86St2dkx+8CX8xly3Lbstg9/Eopy5CPnSVHVb8NHF9gk3Joy3zsLDmqekhVH4venwaewUaHJyl5e+ZpZ8mJ2mg0+tgQvbIzQsqhPfOxs+SIyC7gF4DPzrPJstqy2II/X8mFpW5TbPK14dXRo+CDInLp6pi2JMqhLfOlbNpSRLqAK7DeXpKyas8F7IQyaM/IBfE4MAR8Q1XLsj3zsBNK355/CfwekJln/bLastiCX7CyDEUmHxseAzpV9aeAvwLuK7ZRy6Ac2jIfyqYtRWQt8FXgZlU9lb06x1dK0p6L2FkW7amqaVW9HBtpf5WIvCxrk7JozzzsLGl7isibgCFVfXShzXIsW7Qtiy34lVKWYVEbVPVUeBRU1QeABhHZunom5kU5tOWilEtbikgDJqK9qvoPOTYpi/ZczM5yac+EPSPAt4A3ZK0qi/YMzGdnGbTna4BfFJEDmHv5dSJyT9Y2y2rLYgt+pZRlWNROEdkuIhK9vwpru2OrbOdilENbLko5tGV0/L8BnlHVP59ns5K3Zz52lkl7tonIxuh9C/B64EdZm5VDey5qZ6nbU1U/oKq7VLUL06JvquqvZ222rLbMp1rmstHilWUohZ1vBW4UkVlgArhOo3D5aiEi92IZBFtFZBC4FQs6lU1b5mlnydsS60W9A/hB5M8F+CDQkbCzHNozHzvLoT13AF8QmzCpDviyqt5fbr/1PO0sh/Y8g0K0pZdWcBzHqRF8pK3jOE6N4ILvOI5TI7jgO47j1Agu+I7jODWCC77jOE6N4ILvOI5TI7jgO47j1Aj/CxT2D+XMLo8BAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "similarities_to_a = Sim([x1,a]).tensor\n", "fig, ax = plt.subplots()\n", "ax.set_xlim(0,4)\n", "ax.set_ylim(0,4)\n", "ax.scatter(points[:,0],points[:,1],color=\"black\")\n", "ax.scatter(a.tensor[0],a.tensor[1],color=\"blue\")\n", "ax.set_title(\"Illustrating the similarities of each point to a\")\n", "for i, sim_to_a in enumerate(similarities_to_a):\n", " plt.plot([points[i,0],a.tensor[0]],[points[i,1],a.tensor[1]], alpha=sim_to_a.numpy(),color=\"blue\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Notice the operator for equivalence $p \\leftrightarrow q$; in LTN, it is simply implemented as $(p \\rightarrow q)\\land(p \\leftarrow q)$ using one operator for conjunction and one operator for implication." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "Not = ltn.Wrapper_Connective(ltn.fuzzy_ops.Not_Std())\n", "And = ltn.Wrapper_Connective(ltn.fuzzy_ops.And_Prod())\n", "Or = ltn.Wrapper_Connective(ltn.fuzzy_ops.Or_ProbSum())\n", "Implies = ltn.Wrapper_Connective(ltn.fuzzy_ops.Implies_Reichenbach())\n", "Equiv = ltn.Wrapper_Connective(ltn.fuzzy_ops.Equiv(ltn.fuzzy_ops.And_Prod(),ltn.fuzzy_ops.Implies_Reichenbach()))\n", "Forall = ltn.Wrapper_Quantifier(ltn.fuzzy_ops.Aggreg_pMeanError(p=2),semantics=\"forall\")\n", "Exists = ltn.Wrapper_Quantifier(ltn.fuzzy_ops.Aggreg_pMean(p=6),semantics=\"exists\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If there are several closed formulas in $\\mathcal{K}$, their truth values need to be aggregated.\n", "We recommend to use the generalized mean inspired operator `pMeanError`, already used to implement $\\forall$. \n", "The hyperparameter again allows flexibility in how strict the formula aggregation is ($p = 1$ corresponds to `mean`; $p \\to +\\inf$ corresponds to `min`).\n", "\n", "The knowledgebase should be written inside of a function that is decorated with `tf.function`. This Tensorflow decorator compiles the function into a callable TensorFlow graph (static)." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "formula_aggregator = ltn.Wrapper_Formula_Aggregator(ltn.fuzzy_ops.Aggreg_pMeanError(p=2))\n", "\n", "@tf.function\n", "def axioms():\n", " axioms = [\n", " C([a,l_a]),\n", " C([b,l_b]),\n", " Forall(\n", " [x1,x2,l],\n", " Implies( Sim([x1,x2]),\n", " Equiv(C([x1,l]),C([x2,l]))\n", " )\n", " )\n", " ]\n", " kb = formula_aggregator(axioms)\n", " sat = kb.tensor\n", " return sat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**It is important to always run (forward pass) the knowledgebase once before training, as Tensorflow initializes weights and compiles the graph during the first call.**" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2021-09-24 17:15:46.561522: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)\n", "2021-09-24 17:15:46.566398: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz\n", "2021-09-24 17:15:46.567351: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "axioms()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Eventually, one can write a custom training loop in Tensorflow." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2021-09-24 17:16:07.536407: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.\n", "2021-09-24 17:16:07.619984: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 0: Sat Level 0.471\n", "Epoch 200: Sat Level 0.678\n", "Epoch 400: Sat Level 0.931\n", "Epoch 600: Sat Level 0.954\n", "Epoch 800: Sat Level 0.958\n", "Epoch 1000: Sat Level 0.959\n", "Epoch 1200: Sat Level 0.959\n", "Epoch 1400: Sat Level 0.959\n", "Epoch 1600: Sat Level 0.959\n", "Epoch 1800: Sat Level 0.959\n", "Training finished at Epoch 1999 with Sat Level 0.959\n" ] } ], "source": [ "trainable_variables = C.trainable_variables\n", "optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)\n", "\n", "for epoch in range(2000):\n", " with tf.GradientTape() as tape:\n", " loss = 1. - axioms()\n", " grads = tape.gradient(loss, trainable_variables)\n", " optimizer.apply_gradients(zip(grads, trainable_variables))\n", " if epoch%200 == 0:\n", " print(\"Epoch %d: Sat Level %.3f\"%(epoch, axioms()))\n", "print(\"Training finished at Epoch %d with Sat Level %.3f\"%(epoch, axioms()))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "After a few epochs, the system has learned to identify samples close to the point $a$ (resp. $b$) as belonging to class $A$ (resp. $B$) based on the rules of the knowledgebase." ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkwAAADSCAYAAAC4l5/ZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAA1IUlEQVR4nO3deZgU1fX/8ffpdWZgYIDBDVCMGhV3RdS44RZBjahR4xLXGMRoYmJiYoxxTX5ZNEYjKhI1qDEa40oUY8xXDRpXREQRNYhGEWRfhll6prvP749uoGemm+6Z7pm6XX1ez1NPpqtqqk4b+HDr1q1boqoYY4wxxpjcAl4XYIwxxhjjOmswGWOMMcbkYQ0mY4wxxpg8rMFkjDHGGJOHNZiMMcYYY/KwBpMxxhhjTB7WYDLGGGOMycMaTD4mIiNEZEY3f1dFZNsizr2piMwVkWh3j2GMqVw9nV8iMkVEfpFjW1RE3heRTbpzfuNP1mAqcyJymojMEJG1IrJIRJ4WkQPSm68DbvCiLlVdDDwPjPfi/MYY9zmcXzHgbuAnXpzfuMkaTGVMRC4BbgL+H7ApsCVwGzBORDYHDgEe96o+4H7gfA/Pb4xxVBnk11+As6yX3KxjDaYyJSL9gWuBC1X1UVVtVNU2Vf27ql4KHAHMVNWW9P7biMgKEdkz/XkLEVkmIqOLqOFoEXlLRNaIyGcicnWHXV4DviQiW3X3HMYY/3Ehv9LqReRZEWkQkX9nZpWqLgBWAvsWeQ7jE9ZgKl/7AVXAYzm27wJ8sO6Dqn5Eqnv5fhGpAf4ETFHVF4qooRE4E6gDjgYuEJHjMs4ZB+YBuxVxDmOM/7iQXwCnk7r1Vw/MItUrnmkull8mLeR1AabbBgHL0o2SbOqA5ZkrVPWPIvI1Uj0/ChxbTAEdwmq2iDwAHEz7bvSGdC3GGLOO5/mV9pSqTgcQkZ8Bq0VkmKp+lt5u+WXWsx6m8rWcVHdyrkbvSqA2y/o/AjsDt6QHNnabiOwjIs+LyFIRWQ1MIHWllqkWWFXMeYwxvuN5fqWtaxihqmuBFcAWGdstv8x61mAqX68ALcBxObbPBr6cuUJE+pIaZHkXcLWIDCyyhr8AU4FhqtofmARIxvlCwLbA20WexxjjLy7kF8CwDscfCCzM2L4jll8mzRpMZUpVVwNXAreKyHEiUiMiYREZKyK/BZ4F9hSRqoxfuxl4U1XPA54i1cABQESuFpEXulhGLbBCVVtEZBRwWofto4BPVPV/XTyuMcbHHMkvgKNE5AARiZAay/TauttxIjKEVAPq1e58R+M/1mAqY6p6I3AJcAWwlFT38kXA4+l5kJ4DxgGIyDhgDKnbZqR/b08ROT39eRjwny6W8B3gWhFpIBV+D3XYfjoZoWaMMes4kF+Q6iW/itStuL1IZdY6pwH3lOjWn/EBUVWvazA9RERGAPcAozTP/9EiMgs4TFWXb2y/Lpx7E+DfwB7rHg02xphCeZxfUVK34g5S1SWlOKYpf9ZgMsZ0IiJ3A8cAS1R15yzbhdQtkqOAJuBsVZ3Zu1UaY0xnPZVfdkvObJSIzEm/tqDjcnr+3zZlbAqpWyC5jAW2Sy/jgdt7oSZjusTyq2JNoQfyy+ZhMhulqjt5XYPpfao6XUSGb2SXccC96Vslr4pInYhsrqqLeqdCY/Kz/KpMPZVf1sNkjOmOIWTMYQMsSK8zxhjXdSu/POthqq+v1+HDh3t1emPKxptvvrlMVQd35XeOPKRGl61I5tw+c3ZsDql5cNaZrKqTu3AKybKuogZEWoYZk5+f8suzBtPw4cOZMWOGV6c3pmyISJfnsVq2IsHL/8h9wVS1xcctqjqyiLIWkDHpHzCU9hP++Z5lmDH5+Sm/7JacMT6kQJxEzqUEpgJnSsq+wGobv2SMKQVX88sGffuYqkLbW5BYAOERSGhbr0syaZpcAbFXQKohuj+paV9KeHyURBFThqRfpDya1Pu+FpCa3C8MoKqTgGmkHsmdR+qx3HOKLNmYTpZ9vpzZ0+fSb1Atexy6M8FQ0OuSDJBIJJj13LusXrqGnQ/ckU2GdXyFaHFczS9rMPmUJlegK86CxGeAgCbQ6P5I3R8QCXtdXkVLNv4ZGn4DEiJ1K11gwB1IpJge5vYUaCP3GIC8v696ap7tClzY7RMYsxGqyt2X/4VHbn6KUDiIIERrolz/f1ey1Yhh+Q9gesxnH3zOjw69hpa1LShKvDXBsRceyfnXn0lqeqPiuZpfdkvOp3TVTyH+EWgTaCPQArH/oI1/9Lq0iqZt70HDb4FY6v8XXQvagK4cTyknRFegTZM5F2Nc9vq0mTw+8WnaWtpobmihqaGZVUtW8bOjf4VNtuwdVeVnx/yKlV+spKmhmeaGFtpibTx1x7O8/MQbpTsPbuaXNZh8SJNN0PoSEO+wpQWaHvSiJJOmzY8Crdm2QOzFkp4ruZHFGJdNvf0ZWhrbv8JNFdYsb+C/M+d7VJX5+J1PWfnFKjq2WVsaY0y9/ZmSnsvF/LJbcr7URs4nJO21bt7SRrL+lVdN9QaW6jSqtNqVuClTTWuas66XgHRqSJne09LYQiCYvZ8l1/9n3eFqflkPkw9JoD8Et86yJQRVh/Z6PWYDiX4VpCbLljhEvlKy8yhuXqEZU4jRp+xPtKbzgxCaVHYYZQ+veGXbPb+UdX20OsIhp+xfsvO4ml/WYPIpqfs1SB8gkl5TDYGBSN9LvCzLRA+GyL7AukZTAKiCvj9Agl2a222jFKFNcy/GuGzsuYey1YihVPVJNZqCoQDR6gg/mDyBSFUkz2+bnhKJhvnRXd8hWhMhGEo1H6r6RBm2wxCOHn94yc7jan7ZLTmfkvAuUP8PtOmvkPgIwnsi1V9HAn29Lq2iiQSg7jaIPY+2PA1Sg9SciIR3Lfm5ElknszXGfZGqCDe9dB3T//Yqrz45gwGb1XH0+CPYasehXpdW8Q78+r4M33kYT97xLCsWrWTUUXty8MlfIRIt7dPXLuaXNZh8TIKbIrXf87oM04FIAKoOQ6oO67FzpJ4ysQ5kU77CkTCHnX4gh51+oNelmA6GbT+EC248u8eO72p+5W0wiUgVMB2Ipvd/WFWv6rDPaOAJ4OP0qkdV9dqSVlqmkslGGpseIhZ7iWBwK2r7nkkoNNzrskwXaGIh2vQAxOdDeC+k5iQkUOt1WRuVRGjFJvmz/CqOqjJr+vv868GXSSaVQ0/ch5GH71yy+XZMz2ttaeW5B/7D69NmMnDzOo45/6sM38ntuaxcza9CephiwKGqulZSMx6+JCJPq+qrHfZ7UVWPKX2J5SuRXMniJV8lkVgONANhGpumUD/wHqqq7KqpHGjrTHTlOaBxoA1iL6JNd8Ggx0s65qgnJG2sElh+FWXS5Q/yzH0v0dKUmgrjladmceC4vbhk4tnWaCoDLU0xLt7/Zyyc9wUtjTECwQD/uOs5Lv3ThRx8cukeMukJLuZX3j4vTVmb/hhOL+497+egNWv+QCKxhFRjCaAN1WZWrPyeTb5WBlQVXX0ZaDOpqRoAWiC5El17k4eV5acIrRrMuVQKy6/u+2Tu5zx9z4vrG0uQ+gd4+hMz+HDmJ94VZgr21B3/5PMPF62fiiGZSBJrbuXGb0+iNdaW57e942p+FXSTUESCIjILWAI8q6qvZdltPxF5W0SeFpGdSllkuWpumUa2SQqTuppEossvcDa9TVdCItsLrOMQe67Xy+mK1GO5gZxLJbH86p43n5uDJju3LVub23jtn7M9qMh01b//9gqx5iwT5Qr89013JwB1Nb8KGvStqglgdxGpAx4TkZ1V9d2MXWYCW6W7vY8CHge263gcERkPjAfYcsstiyzdfQHpk/W9yqoJJOtcPMYtEXJ3RlT3ZiFdpioV1ZO0MaXKL6isDKvuU0UgFOh0zRcKB+lT6/aff5PSp1/2f2eSiSTVfat6uZrCuZpfXWqqqeoq4AVgTIf1a9Z1e6vqNCAsIp1eX6yqk1V1pKqOHDzY7fEfpdCnz7mIdAyWIJHIbgSDm3hSkymcBPpCZD86X1dUQc1pXpTUJUkk51KJis2v9PaKybADjt0z6/WCBISDji/di6JNzzn2wjHr57JaR0QYtMUAtt7F7Qa/i/mVt8EkIoPTV2ZI6l//w4H3O+yzmaRHAIrIqPRxl5e82jLTt89pVFePA6KI9EWkD6HgcOoHTva6NFMgqfsthLZJzc4tfYAoRA9F+pztdWkblRoDEMq5VArLr+7rN7AvV0yZQFWfKDW1VdTUVhGtjnDp7ecyeMhAr8szBdj3mL0Yd9FYwtEwNbXVVNdWMWjIAH7x5OVOD9p3Nb8KOfPmwD0iEiQVJA+p6pMiMgFAVScBJwIXiEic1AjnU9RGNSMSYNCA39O/9ge0ts4iGNyMSGRvp/+gmvYkMBAGTYW22ZD4HMIjkDKYFiI1j4l7XdoesPwqwt5H7MKDH/yOWdPfJ5lIssfBOzp9K8e0JyKc96vTOe6iMcx5+UP619eyy0E7Egy6nQ2u5lfeBpOqzgb2yLJ+UsbPE4GJpS3NP0KhLQmF3O7+NLmJCER2A3bzupSCKUKiwgZ3Z2P5Vbyqmij7jimfP/ums/ohgzj4pP28LqNgruaXexUZU2KaXEtyzXUkF48iuXhvkquvRpOrO++nSZKN95JcMprk4j1Irvg2Gp/nQcXFS12hhXIuxpjy8dxfXuScHS7m2H5n8IODfs57r36Ydb+5r/2XH46+imP7ncE5O1zMv/48vZcrLQ1X88saTMbXVJPoitOg6UHQVaCroflv6PJvoNp+HhJt+DU0/A6SC0EboXU6uvwkNP6pN8UXQRESmnsxxpSHx26Zxo3j72DBhwtpXtvCuy+9z48Pv5YP3mh/Mffhmx9x6WFXM3v6ezSvbWHBhwu5acJkHv79kx5V3n2u5pc1mIy/tb4EiU/ZMPEkqZ+Ti9vNpaTJVdD0FzZMMgqgoDG08c7eqbWEVN28QjPGFC4RT3DPlX8l1hRrtz7WFOPunz3Qbt2frniQWFNrp/3uu+Yh4m3xHq+1lFzNL2swGX9rmwsa67xeG9G2DzZ8jn8CEslygDi0zeqh4npS7kdyK3VaAWPKzcrFq4i3Zm/szH/7k3afP5r1cdb9EvEkK75YVeLKepqb+WWXmsbfQluCVKVusbVTg4QyXkAZ3AI0y4y4BCD0pZ6ssEcoVNT0Acb4Ub9BteRqH2y29SYdPm/KysWdx2aiSv96t18W3pGr+WU9TMbfooel50/K/KMeSDWiqjbMXyjBTSB6CKmX2meKIH3G90KhpaUIbRrMuRhj3BepinDsd44kWtM+l6I1Ec68+uR26868+mSiNZEO+0U55vwjiFZ3zDW3uZpf1mAyviYSQQY9BJFRpDpUQxDeCxn0UKdZ2KXuBqg+jlSjKQTBYciA25HwiN4vvEgKJDWQczHGlIdv/ep0Trj4KKr6VhEKBxm4+QAuufMC9h7TfraMkV/djR/d/R0GbTGAUDhIVZ8ox393LN++/gyPKu8+V/PLvT4vY0pMglsgA+9FNTWgu/PratL7SRTpfx3a70rQFpC+ZTvJ6LorNGNMeQsGg5z7y9M465pv0Ly2hT79a3Lm0uiT9+fgk75C05omqvpUEQyVZwa4ml92qWkqhkh1zsZS+/3CSKC2bBtL6ySQnEshRGSMiHwgIvNE5LIs2/uLyN9F5G0RmSMi55T8SxhjAAiGgvSt65M3l0SEPv37lG1jaR0X88t6mIzxIVWhLdn9v97pV4ncChwBLADeEJGpqvpexm4XAu+p6tdEZDDwgYjcr5p19LwxxhTE1fyyHiZjfEgp+m3fo4B5qjo/HSAPAuOynKY2/eLavsAKoLwmfDHGOMfV/LIeJmN8SBHakhvtkq8XkRkZnyer6uSMz0OAzzI+LwD26XCMicBUYCFQC3xDVZPdr9oYY9zNL2swGeNDBQyaXKaqIzeyPdtlnHb4fCQwCzgU2AZ4VkReVNU1XanVGGMyuZpfeW/JiUiViLyeMTDqmiz7iIj8IT24araI7JnvuMaYnpUkkHMpwAIgY2ZPhpK6Est0DvCopswDPgZ2KEnxJWL5ZUx5cjG/CjlzDDhUVXcDdgfGiMi+HfYZC2yXXsYDtxdwXGNMD1GFtmQg51KAN4DtRGRrEYkAp5Dqvs70KXAYgIhsCmwPzC/h1ygFyy9jyoyr+ZX3lpyqKrA2/TGcXjp2bY0D7k3v+6qI1InI5qq6KN/xjTGlp0hRE7ypalxELgKeAYLA3ao6R0QmpLdPAq4DpojIO6S6wH+iqsuKr750LL+MKT+u5ldBY5jSj+i9CWwL3Kqqr3XYJdsAqyGABY4xHlCgrcgZcVV1GjCtw7pJGT8vBL5a1El6geWXMeXF1fwqqCJVTajq7qTuA44SkZ077FLIACtEZLyIzBCRGUuXLu1KncaYLhEnXy3ghVLlF1iGGdM73MyvLp1ZVVcBLwBjOmwqZIAVqjpZVUeq6sjBgwd3rVJjTMFUU1douZZKVGx+pY9hGWZMD3M1vwp5Sm6wiNSlf64GDgfe77DbVODM9NMm+wKr7f6/Md5RhHgymHOpFJZfxpQfV/OrkDFMmwP3pMcBBICHVPXJDoOnpgFHAfOAJlKP6xljPFTgjLh+Z/llTBlyMb8KeUpuNrBHlvWZg6eU1HtZjDEOUKionqRcLL+MKT+u5pfN9G2MH6mQVPeu0IwxJi9H88saTMb4kALxCh3cbYwpb67mlzWYjPEhBSev0IwxJh9X88saTMb4UOopE/eu0IwxJh9X88saTMb4kbrZpW2MMXk5ml/WYDLGh1zt0jbGmHxczS9rMBnjQ652aRtjTD6u5pc1mIzxKXXwCs0YYwrhYn5Zg8kYH1JHxwAYY0w+ruaXNZiM8SkXr9CMMaYQLuaXNZiM8SUh4eAYAGOMyc/N/LIGkzE+5OpTJsYYk4+r+WUNJlPRVBNAApGI16WUlkLCwcAxxpSOqtIWayMcDSPio7/vjuZX3j4vERkmIs+LyFwRmSMiF2fZZ7SIrBaRWenlyp4p15jS0GQDyVU/Qhfvii7eleTyk9C2D7wuq2Q03aWda6kUll/Gr57647OcvPm3+Vrfb3LSZufx5B3/9LqkknE1vwrpYYoDP1TVmSJSC7wpIs+q6nsd9ntRVY8pfYnGlJaqoivPhba5QFtqZdvb6IpTof4ZJDjY0/pKRdXrCpxg+WV855kpz3P7D+4h1hQDYPXSNUz64b0EggGOOu9wj6srDRfzK29TTVUXqerM9M8NwFxgSE8XZkyPic+B+IdAa/v12oo2PehJSaWmCslkIOdSKSy/jB/dc9Vf1zeW1ok1xbj36oc8qqi0XM2vLp1ZRIYDewCvZdm8n4i8LSJPi8hOOX5/vIjMEJEZS5cu7Xq1xpRC/BOy/9Fvhbh/bsslVXIulajY/EofwzLMeG7Z5yuyrl+xaBXqYtdMN7iYXwU3mESkL/AI8H1VXdNh80xgK1XdDbgFeDzbMVR1sqqOVNWRgwf747aHKUOhL4MmsmyogvCuvV5OT0kmJedSaUqRX2AZZtyw+dabZF2/6fDBvhn87WJ+FdRgEpEwqbC5X1Uf7bhdVdeo6tr0z9OAsIjUl7RSY0pEwl+GyN5ANGNtAKQKqTnZq7JKShFUcy+FEJExIvKBiMwTkcty7DM6PVB6joj8u6RfokQsv4zfnPebM4jWtH+yN1oT4bxff9OjikrL1fwq5Ck5Ae4C5qrqjTn22Sy9HyIyKn3c5fmObYxXZMBt0OdMkDqgCqKHIYMeQQJ1HldWIlpcl7aIBIFbgbHACOBUERnRYZ864DbgWFXdCTip5N+jSJZfxo8OPGEfLr//+2y541DC0TDDdhjCT/98MQeftJ/XpZWGo/lVyFNy+wNnAO+IyKz0usuBLQFUdRJwInCBiMSBZuAU9cuNVONLIlGk9lKovdTrUnqMFtd1PQqYp6rzAUTkQWAckPl02WnAo6r6KYCqLinmhD3E8sv40lfG7c1Xxu3tdRk9xsX8yttgUtWXgI1WrqoTgYn5jmWMV1RbofV1QCEyCpFo3t8pd0X+kz8E+Czj8wJgnw77fJnU7asXgFrgZlW9t6izlpjll/GLebM+Zsn/lrHtHsPZZEv/j59zMb9spm/jexr7D7rqu5lroP+NSNUhntXU01RBN/74bb2IzMj4PFlVJ2d8ztbI6BhhIWAv4DCgGnhFRF5V1Q+7U7MxprM1yxv46dhf8r/3FhAMBWhrjXPoKftzyZ0XEAj4c4oQV/PLGkzG1zS5El35HVJ3WjLWr7oYBv+fbyapzCbPFdoyVR25ke0LgGEZn4cCC7Pss0xVG4FGEZkO7AZYg8mYEvnt2ROZP/sT4q0bnux94aFX2Gb34Rz/vaM9rKxnuZhf/myeGrNOyzM5Nii0PNWrpfQuQZO5lwK8AWwnIltL6kV7pwBTO+zzBHCgiIREpIZUl/fckn4NYypY45om3nx2drvGEqQmqXx84j88qqo3uJlf1sNk/E3Xsv71J+20osmGjQ9uKXdFjAFQ1biIXAQ8AwSBu1V1johMSG+fpKpzReQfwGwgCdypqu8WX7gxBlINo1zzKjU3NGdd7xsO5pc1mIy/RfYH/kDqlWKZqpDoQR4U1Eu06KdM1s1JNK3DukkdPl8PXF/UiYwxWQ3YtI5BWwzgi4/bP8AVDAXZ55i9PKqqFziaX3ZLzviahHeE6q+B1GSsrIGqw301q3dWKrkXY4zzRIQf3fUdojVRgqEgAJHqCP0G9eWsa77hcXU9zMH8sh4m43vS7xcQPRxtfhRIItXHQ/RQ37xCICebSciYsrfb6J24Y9b1PH7L0yz4cCG7HjSCo88/gn4Da70urWc5mF/WYDK+JyJQdYivpxHopARd2sYYNwzZdnMuvPlcr8voPY7mlzWYjPErB6/QjDGmIA7mlzWYjPEpcfAKzRhjCuFiflmDyRg/Upy8QjPGmLwczS9rMBnjSwIOXqEZY0x+buaXTStgPJdMLCLeOgNNrvK6FH9JbmQxxpRES1OMua/N4/N5X3hdir84mF95e5hEZBhwL7AZqVInq+rNHfYR4GbgKKAJOFtVZ5a+XOMnmmyieeWFxGMvgURAW4n0OYtov5/5/5H/nqbYfEtYfpme9dSdzzH5sr8QCAZItCXYcsctuObhHzJo8zqvSytvjuZXIT1MceCHqrojsC9woYiM6LDPWGC79DIeuL2kVRpfal59eaqxRAy0AYjR2nQfrU0PeF2aL0gy91JBLL9Mj3jnpQ+44yf309IYo2lNM7HmVubP/pSfn3CD16X5gov5lbfBpKqL1l1tqWoDqZfTDemw2zjgXk15FagTkc1LXq3xDU02E29+Eoh12NBM29o7PKnJ+I/ll+kpj93yNLGm1nbrEvEkCz5cxKfvf+5RVaYndWkMk4gMB/YAXuuwaQjwWcbnBXQOJURkvIjMEJEZS5cu7WKpxk9UG3NuS9pYppKQpORcKlGx+ZU+hmWYAWD5olVZ1wdDQVYva+jdYnzIxfwquMEkIn2BR4Dvq+qajpuz/EqnhwJVdbKqjlTVkYMHD+5apcZXJDAICdRn2RIgFN231+vxHc2zVJhS5BdYhpkNRo3dnUhVuNP6RFuCbXbbyoOKfMTR/CqowSQiYVJhc7+qPppllwXAsIzPQ4GFxZdn/EpEqKr7FVDNhn+vwiB9iPa7zMPK/MPFMQBesPwyPWHcBUfQf3A/wtENz05FayKcfc1J1NRWe1iZP7iYX4U8JSfAXcBcVb0xx25TgYtE5EFgH2C1qi4qXZnGj8JVhxCof5jY2kkk4/MJRkYS7TuBQCjr3RDTVRXWMMrG8sv0lL51fbj9tV/y2K3P8Pq0t6jbpB/Hf3cMex22i9el+YOD+VXIxJX7A2cA74jIrPS6y4EtAVR1EjCN1CO580g9lntOySs1vhSM7ELNwFu9LsN3RFOLsfwyPad2QB/OvOIEzrziBK9L8RVX8ytvg0lVXyL7Pf7MfRS4sFRFGWNKoEIHd2ey/DKmTDmYX/ZqFGN8ysUrNGOMKYSL+WUNJmP8SCtvcLcxxicczS9rMBnjVw5eoRljTEEczC97+W4WK5esZtH8xSSTDjZxjSmQi4/lmp6n2kwy/jGazD05rDGuczG/rIcpw4ovVvHL027m/df/SyAYoG//PvzwrgmMPGI3r0szpuscvEIzPUdVSTTcRKLpTkBAEwRqTifU76eIBL0uz5iucTC/rIcpTVW5bMwvee+VD2iLxYk1tbJ80UquOfFGFnxoc9iZMqMbHs3Nthj/STROSTWWtBm0CYiRbHqAxNqJXpdmTNc4ml/WYEr74I2P+OKTJSTi7fv74q1xpt7+T4+qMqYIyY0sxncSjXekGkvtNJNovIvUzAnGlBEH88saTGnLFq4gEOz8nyMRT7Bo/mIPKjKm+4Tir9BEZIyIfCAi80Qk5/tqRGRvEUmIyIklKt90R3Jl9vXaCCR6tRRjiuFqflmDKe3Le32JttZ4p/XRmgh7HLqzBxUZUwQtbtCkpAa93AqMBUYAp4rIiBz7/QZ4prRfwHSVhHfMvj44HBEbrmrKiKP5ZQ2mtE2G1XPENw8iWhNdvy4UCdFvYC1HnnOIh5UZ003Fve17FDBPVeeraivwIDAuy37fJfVi2yUlqNgUIdTvCpDMl1kDVBHqf5VXJRnTfQ7ml112ZPjerd9i+7234fGJ/6BpTTP7j9ubUy47jj79arwuzZguK/Lx2yHAZxmfF5B6Me2G44sMAY4HDgX2LupspmiByEjCgx4k0XAzyba5SGgbQrUXE4js5XVpxnSZi/llDaYMgUCAseceythzD/W6FGOKo+QbHFkvIjMyPk9W1ckZn7O9yKnjtd1NwE9UNSHi3nufKlEgvAuBgXd6XYYxxXE0v5xsMCUSCWY88zaLPlrMNrsPZ+cDdsAC2ZiuyTM4cpmqjtzI9gXAsIzPQ4GO82uMBB5M/92sB44SkbiqPt7lYn1G4/Mg9ioE6iB6KBKwXmpjusLF/MrbYBKRu4FjgCWq2mn0s4iMBp4APk6velRVr8133FyWL1rJ9w+4gtXL1pBoSxAIBdl6ly357bNXUpUxvsgYs3FFdmm/AWwnIlsDnwOnAKdl7qCqW68/l8gU4EkXG0u9mWGqiq65CpofSx88BFwJA/6ERGwCXGMK5WJ+FTLoewowJs8+L6rq7uml240lgBvOuZUlny6juaGF1pY2Wta28NFbH3Pv1Q8Vc1hjKk8RgyZVNQ5cROrpkbnAQ6o6R0QmiMiEniq5h0yhtzIs9i9oeQKIpRZtBF2LrjwfVXu035iCOZhfeXuYVHW6iAzv7gm6ItYc463n3iWZaN+0bG1p49l7/834357RG2UYU/ZKMSOuqk4DpnVYNynHvmcXd7ae05sZpk1/yzJ5JEAM2maBDcA2Ji9X86tU0wrsJyJvi8jTIrJTrp1EZLyIzBCRGUuXLu20PZnM/V8o0WZXZ8Z0SXGP5VaakmQYtOb6TdDO87wZY3JwML9K0WCaCWylqrsBtwCP59pRVSer6khVHTl48OBO26v7VLHdXl+i4/juUDjIAV/fp9P+xpjcXHzbt6NKlmFSPQ6ozv7LkT1KUasxFcHF/Cq6waSqa1R1bfrnaUBYROq7e7xL/3QhfQf0XT/Au7pvFfVDB/Gt/3dant80xqxX5Ey5laSkGVb1tdRtN1n3VFwEqEL634BIpDQFG+N3juZX0dMKiMhmwGJVVREZRaoRtry7x9tyhyHc99FE/nX/iyz4YCHb770tB524L5EqCxtjusRuvRWklBkmEoIBd0Lry2js3xAYiFQfhwQ3L2nNxvieg/lVyLQCDwCjSU0UtQC4CgjD+gFUJwIXiEgcaAZO0SJfjd2nfx/GfSffQy3GmI2xnqSU3s4wkQBED0CiBxRduzGVysX8KuQpuVPzbJ8ITCxZRcYJTbFWHn75HV545yMG1tZw2kG7s+c2Q70uy3RBsU+Z+IVlWGX6z5L/8tAnb9AQb+HILXbmuGF7EA2GvS7LFMjF/HJypm/jraZYK6fc8Be+WNlArC2OAC++9zHf/9oBnHqQDVwtC/lfLWCMb018//+4b/7LNCfaAHh31ec88dlb3LP/twgH7J895zmaX6WaVsD4yMMvv7O+sQSpP7strXFumvoSjS25Hps2LhE2zGWSbTHGr5a2NDDlo/+sbywBtCTa+KhhCc8snONhZaZQruaXNZhMJ8+/89H6xlKmUDDAu59+4UFFpjskqTkXY/xq5or/EQ50/qetOdHGC1+870FFpjtczC/rmzSdDOyb/UWhiaTSv6aql6sx3aJuDpo0pqf1D1eT7WX1AYSB0T69X5DpOkfzy3qYTCenHbQ7VZH2bemACJvW9WX7IZ0n6zOOcnCmXGN62shBw6nOMrg7Eghx4lZ7e1CR6RYH88saTKaTvbYdyveO3p9oOETfqgjVkTBbDq7jtgnHIx2nYTfOcnHiN2N6WigQZPJ+Z7NZVX9qghH6hqJUB8P8fNev8eV+m3pdnimQi/llt+RMVqeP3pNx++7EnP8tpl+fKnYYMtgaS+XEBnebCrZt7SY8c/glzFm9kKZ4jF3qhlIdssmPy4aj+WUNJpNT36oo+2y/pddlmG4QrCfJVDYRYee6IV6XYbrB1fyyW3I97O0X5nD+Hj/mq+FTOHHT8/jrb58gmXTwT0KFScSXsXTxIXzx+RZ88fkWLF44gljz816XVVqquRdjCqDJFSRW/YD4FyOIf7EjiZUXooklXpdlgNt+fD9HDTyPMf3P5aiB5zHxkvu8Lqm0HMwv62HqQe+/Po+ffe3XxJpScxetXtbAfdc9zJoVa/n2r0/3uLrKtmzJAaiuWf9ZdRUrV5xO/Sb/JhTezsPKSsTRp0xM+VCNk1h+MiQWAOk52WL/IrF8NsHB/0Ik6m2BFezWS//M3yc/t/5zMpHkybueR1X57u/P9LCyEnE0v6yHqQfdd+3f1jeW1ok1tfLExH/Q3NjiUVWmuWlqu8ZSptWrrujlanqOJHIvxuSjsX9DcinrGkspCdA1aMszXpVlgKfueiHr+qfvmd67hfQgF/PLGkw96JN3P8u6PhAMsPzzFb1cjVmntfX1nNsS8f/2YiU9y8WZck0ZiX8EGuu8XhvR+Lzer8esl0xk735JJpK+GfLhYn7lbTCJyN0iskRE3s2xXUTkDyIyT0Rmi8iepS+zPA3feVjW9YlEkkFDBnb7uP996xNu+f4UfnX2bbz42Osk4tZl0BWRSO65WIIhH9yOg3SXtnsz5XrBMqybQttAtttuUoOEtu32YZPJlbQ03Ebjigk0N/yBZGJZEUVWpkAw+z/dgWCAQJZZzsuOo/lVyH/ZKcCYjWwfC2yXXsYDtxdflj988+cnEq1u/yhrtCbKuAuPpLpP92bM/vvkf/HDI37BtLue54W/vcoN5/+Ry8ddb42mLqiuGYdIbdZt/fpf28vV9CAHJ37zyBQsw7pMogdDoJ72Q12DIP2QqiO7dcxE/BMaFh9MS8PvaWt5iljDH2hYcjCJNv/07PaGo84+OOv6MWce2MuV9CAH8ytvg0lVpwMbu380DrhXU14F6kRk81IVWM523Gc7fvH3n7D1zsMQEfoNquX0K07gvF+d1q3jNaxsZPJPHyDW3Eoy3cpuaYzx/hsf8eLjb5SydN+r3+Q/BENfXv9ZpD91A/9MOLK9h1WVjmjuq7NK62GyDOsekRDBQQ+lG0dhIIREDyM46JFuD/huXn01qquBdWM4Y6g20LT68hJVXRkuuvEMjvnWIet7mgLBAEedczDfu+ksjysrDVfzqxRPyQ0BMgfrLEivW1SCY5e93Q/Zmclv34CqFj3x4zsvvU8oHKK1pa3d+pbGGNMffZ3RJ+5b1PErSTBUz+BNXwAgmUz6oxu7AxurVDDLsBwkMIhg3c1o+lHuYjMsHnsR6DjGRkm0voZqEhH//T3sKRfdeAYX3XiG5VcvKsV/5Wx/g7J+VREZLyIzRGTG0qVLS3Dq8lGKWbKjNVGy/acVEfrUVhd9/Erlx7ABN18t4CjLsDxEpDQz/Uuu2bYjZP+/weRj+dV7SvFfegGQObp5KLAw246qOllVR6rqyMGD7SWuXbXbQTsQCnfuFIxUhRl77ujeL8i4S4GE5l5MJsuwXhKpORnoeDsvQrj6OHv1ktnA0fwqRYNpKnBm+kmTfYHVqlrxXdk9IRQO8YvHf0TtgD7U1FZRXVtFpCrMN392PCP28cnTXaZkin0sV0TGiMgH6afHLsuy/fT0U2WzReRlEdmt1N+hl1iG9ZLq2ssIRvYCqQbpC1JNMLwLNf2v9ro04xgX8yvvGCYReQAYDdSLyALgKlIjAFHVScA04ChgHtAEnFPY1zHdsf1eX+KB+bcw6/k5NDY0s/vBI6gb3M/rsoyDihkcKSJB4FbgCFI9MG+IyFRVfS9jt4+Bg1V1pYiMBSYD+xRRco+wDHOHBKqprf8ribb3SLR9QCC0LaHILl6XZRzkYn7lbTCp6ql5titwYb7jmNIJR0LsfWS5XsybXlH847ejgHmqOh9ARB4k9TTZ+sBR1Zcz9n+V1K0s51iGuScYHkEwPMLrMoyrHM0ve5ecMT4kgGz8Xn+9iMzI+DxZVSdnfM725NjGrr6+BTzd1TqNMaYjV/PLGkzG+JRs/K3ey1R15MZ+Pcu6XE+OHUIqcA4ovDpjjMnNxfyyBpMxfqQKxU3wVtCTYyKyK3AnMFZVlxdzQmOMAZzNL39O4GCMKXam3DeA7URkaxGJAKeQeppsw/FFtgQeBc5Q1Q9L/gWMMRXLxfyyHiZj/EiLm+BNVeMichHwDBAE7lbVOSIyIb19EnAlMAi4LT2HTjxPN7kxxuTnaH5Zg8kYvyrynUuqOo3UI/eZ6yZl/HwecF5RJzHGmGwczC9rMBnjU3kGTRpjjLNczC9rMBnjR+teLWCMMeXG0fyyBpMxPiSok1doxhiTj6v5ZQ0mY/wq6eFrvY0xphgO5pc1mIzxIwXcyxtjjMnP0fyyBpMxPiUOXqEZY0whXMwvazAZ40eqTnZpG2NMXo7mV0EzfYvIGBH5QETmichlWbaPFpHVIjIrvVxZ+lI7W/jRF/zmrFv45tbf4fsHXMFrT73ZG6c1pjwkN7JUEFfzS7WV5Nq7SC4dQ3LpV0munYhqc2+c2hj3OZhfeXuYRCQI3AocQer9LG+IyFRVfa/Dri+q6jE9UGNWi+Yv5oK9fkzL2haSSWXx/5Zy3Td+z/jrz+DYC47srTKMcZaLXdq9zdX8UlV05XhonQm0pFauvQNteR4GPUSqbGMql4v5VUgP0yhgnqrOV9VW4EFgXM+Wld991/6NlsYYyYzZQGNNMe766f20tbZ5WJkxDlBSM+XmWiqHk/lF20xoe4v1jSUAYpD4CGLTvarKGDc4ml+FNJiGAJ9lfF6QXtfRfiLytog8LSI7laS6jXhn+lySic4tUE0qi+Yv6enTG+O49BiAXEvlcDK/aJsFmuXCTpvQNhtaYCqdm/lVSINJsqzr2MSbCWylqrsBtwCPZz2QyHgRmSEiM5YuXdqlQjuqHzYw6/p4W4K6TfoVdWxjfEE191I5SpZfUMIMC2wKEs2yoRoJbtH94xrjFw7mVyENpgXAsIzPQ4GFmTuo6hpVXZv+eRoQFpH6jgdS1cmqOlJVRw4ePLiIsuHUy04gWtM+cCJVYb5y7Ej6Dawt6tjGlD1VSCRyL5WjZPmV3l6aDKs6AojQqT0nQajqtaFUxrjJ0fwqpMH0BrCdiGwtIhHgFGBq5g4ispmISPrnUenjLi91sZlGjd2DCTeeRU2/aqr6VhGOhtnv2JH86E8X9uRpjSkPCiSSuZfK4WR+iUSRQX+B4LZAFKiC4FbIwPuQgPWQmwrnaH7lfUpOVeMichHwDBAE7lbVOSIyIb19EnAicIGIxIFm4BTVnu83O2b8ERx59mgWf7KUfvW11rNkTKbKuvWWlcv5JaFtkMFPoYlFoAkIDiHdbjPGOJhfBU1cme6mntZh3aSMnycCE0tbWmfxtjirlq6hf30t4UgYgHAkzNAvu3PPP1uNxvQ+Nyd+84Ir+QWgyRVAGAlsuLiT4Oa9ceqCZavRmN7lZn6VxUzfqsojNz3Ffdc8RKItgQSEEy4+mrOu/QaBQEFzb/Y4VeXhG//On697eH2NX7/kGM686mRnajQVRHEycCqVtr2HrroUEp+kPof3ROquR4KbeVtYBm17F131Y0j8L/U5shfS/3okuKnHlZmK42h+lUWD6Z/3vMCUnz9IrCm2ft0jNz1FOBrimz8/ycPKNvjH3c9x71UP0ZJZ4++eJFoV4dSfnuBhZaZiORg4lUiTK9AVp4M2bljZNgNdcRrUP+vEJJWaWI6uOKN9ja1vpOqu/ycidtFnepmD+VUWfwv+fN3D7RpLkJqk8uEbn6QXhhoU5P5fPNKusQTQ0hTjr9c/4UyNppJsZNK3ypq40nPa9AhovMPaBCRXQusrntTUkTY/nKPG5c7UaCqJm/lVFj1MKxatzLq+eW0LrS2tRKuzzWfSu1Z8kb3GpjXNtLXGiURtPJPpRQpaWdMHuCvxCRDrvF4TkFjQ29Vkl/iY7DUmIfF5r5djKpyj+VUWPUxb77Jl1vX1QwYSqYr0cjXZDd8pe42bDKu3xpLpfY7OY1KJJLwXSE22LRDepdfryUbCI4HqLFsUwjv3djmm0jmaX2XRYBp//ZlEq9s3jKLVESb87ixnHsM9/4YsNdZEmHDjWR5VZCqegzPlVqTqoyAwEMi8cKqCyEgk3PNvYSlI9TEQHET7mw5VENkHCY/wqipTyRzMr7JoMO160Ah+8+yV7DZ6J/oNqmXHfbbj6sd+zIFf39fr0tbbbfRO/PqfP2fXg0esr/Gax37MAcfv43VppiIpmkjkXEzvEalCBj0C1d+AQD0EtoC+5yMDbve6tPVSNT7cocYJyIBbvS7NVCQ386ssxjAB7PSV7bnhuau9LmOjdt5/B373/DVel2HMhrd9F0FExgA3k5rw8U5V/XWH7ZLefhTQBJytqjOLOqlPSWAA0v9K4EqvS8lJAgOR/lcBV3ldiql0juZX2TSYjDGFU4obNCmpZ91vBY4g9T62N0Rkqqq+l7HbWGC79LIPcHv6f40xpttcza+yuCVnjOki1dQTTrmW/EYB81R1vqq2Ag8C4zrsMw64V1NeBepExK1pq40x5cfR/LIGkzE+VeQYgCHAZxmfF6TXdXUfY4zpMhfzy7Nbcm+++eYyEfmfV+fvpnpgmddFlJCfvo+fv8tWXT1AAyuf+Zc+XL+RXapEZEbG58mqOjnjc7bHTzsOKihkH9+yDPOcfRc3+Ta/PGswqepgr87dXSIyQ1VHel1Hqfjp+9h3aU9VxxRZxgJgWMbnocDCbuzjW5Zh3rLv4iY/55fdkjPGZPMGsJ2IbC0iEeAUYGqHfaYCZ0rKvsBqVV3U24UaY0wHPZJf9pScMaYTVY2LyEXAM6Qey71bVeeIyIT09knANFKP5M4j9VjuOV7Va4wx6/RUflmDqWsm59+lrPjp+9h3KTFVnUYqVDLXTcr4WYELe7suUxQn/myViH0XNznxXXoiv0TtNQnGGGOMMRtlY5iMMcYYY/KwBlMBRORuEVkiIu96XUuxRGSYiDwvInNFZI6IXOx1Td0lIlUi8rqIvJ3+LmX/XhoRCYrIWyLypNe1GP/wS4b5Kb/AMqzcWIOpMFOAYh9zdEUc+KGq7gjsC1woIuX6OvIYcKiq7gbsDoxJP+1Qzi4G5npdhPGdKfgjw/yUX2AZVlaswVQAVZ0OrPC6jlJQ1UXrXjCoqg2k/mCX5ezM6Snt16Y/htNL2Q7KE5GhwNHAnV7XYvzFLxnmp/wCy7ByYw2mCiYiw4E9gNc8LqXb0t2/s4AlwLOqWrbfBbgJ+DFQ0MuSjKlkfsgvsAwrJ9ZgqlAi0hd4BPi+qq7xup7uUtWEqu5OapbWUSKys8cldYuIHAMsUdU3va7FGNf5Jb/AMqycWIOpAolImFTY3K+qj3pdTymo6irgBcp3nMb+wLEi8gmpN2sfKiJ/9rYkY9zjx/wCy7ByYA2mCiMiAtwFzFXVG72upxgiMlhE6tI/VwOHA+97WlQ3qepPVXWoqg4nNY3/c6r6TY/LMsYpfsovsAwrN9ZgKoCIPAC8AmwvIgtE5Fte11SE/YEzSLX+Z6WXo7wuqps2B54Xkdmk3h30rKr67lFWY4rlowzzU36BZVhZsZm+jTHGGGPysB4mY4wxxpg8rMFkjDHGGJOHNZiMMcYYY/KwBpMxxhhjTB7WYDLGGGOMycMaTMYYY4wxeViDyRhjjDEmD2swGWOMMcbk8f8BBimsZnM22WwAAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "fig = plt.figure(figsize=(10,3))\n", "fig.add_subplot(1,2,1)\n", "is_a = C([x1,l_a])\n", "plt.scatter(x1.tensor[:,0],x1.tensor[:,1],c=is_a.tensor.numpy(),vmin=0,vmax=1)\n", "plt.title(\"C(x,l_a)\")\n", "plt.colorbar()\n", "fig.add_subplot(1,2,2)\n", "is_b = C([x1,l_b])\n", "plt.scatter(x1.tensor[:,0],x1.tensor[:,1],c=is_b.tensor.numpy(),vmin=0,vmax=1)\n", "plt.title(\"C(x,l_b)\")\n", "plt.colorbar()\n", "plt.show();" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Special Cases\n", "\n", "### Variables grounded by batch\n", "\n", "When working with batches of data, grounding the variables with different values at each step:\n", "1. Pass the values in arguments to the knowledgebase function,\n", "2. Create the ltn variables within the function. \n", "\n", "```python\n", "@tf.function\n", "def axioms(data_x, data_y):\n", " x = ltn.Variable(\"x\", data_x)\n", " y = ltn.Variable(\"y\", data_y)\n", " return Forall([x,y],P([x,y]))\n", "\n", "...\n", "for epoch in range(epochs):\n", " for batch_x, batch_y in dataset:\n", " with tf.GradientTape() as tape:\n", " loss_value = 1. - axioms(batch_x, batch_y)\n", " grads = tape.gradient(loss_value, trainable_variables)\n", " optimizer.apply_gradients(zip(grads, trainable_variables))\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Variables denoting a sequence of trainable constants\n", "\n", "When a variable denotes a sequence of trainable constants (embeddings):\n", "1. Do not create the variable outside the scope of `tf.GradientTape()`,\n", "2. Create the variable within the training step function.\n", "\n", "```python\n", "c1 = ltn.Constant([2.1,3], trainable=True)\n", "c2 = ltn.Constant([4.5,0.8], trainable=True)\n", "# Do not assign the variable here. Tensorflow would not keep track of the \n", "# gradients between c1/c2 and x during training.\n", "...\n", "@tf.function\n", "def axioms():\n", " # The assignation must be done within the tf.GradientTape, \n", " # inside of the training step function.\n", " x = ltn.Variable.from_constants(\"x\", [c1,c2])\n", " return Forall(x,P(x))\n", "...\n", "for epoch in range(epochs):\n", " with tf.GradientTape() as tape:\n", " loss_value = 1. - axioms() # the training step function is called within the tape.\n", " grads = tape.gradient(loss_value, trainable_variables)\n", " optimizer.apply_gradients(zip(grads, trainable_variables))\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [] } ], "metadata": { "interpreter": { "hash": "889985fd10eb245a43f2ae5f5aa0c555254f5b898fe16071f1c89d06fa8d76a2" }, "kernelspec": { "display_name": "Python 3.9.6 64-bit ('tf-py39': conda)", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.6" } }, "nbformat": 4, "nbformat_minor": 4 }