![](\"lecture_memory/point_attractor.png\")
\n",
"\n",
"- In neural network research, attractor networks (networks with dynamical attractors) have long been thought relevant for various behaviours \n",
" - e.g., memory, integration, off-line updating of representations, repetitive pattern generation, noise reduction, etc."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"\n",
" \n",
"- The neural integrator is an example of an attractor network\n",
"\n",
"- The neural integrator can be thought of as a line attractor as in a) \n",
" - Actually an approximate one as in b)\n",
"\n",
"![](\"lecture_memory/line_attractor.png\")
\n",
"\n",
"- Attractor networks were extensively examined in the ANN community (e.g. hopfield nets). Amit suggested that persistent activity could be associated with recurrent biological networks \n",
"- Persistent activity is found in motor, premotor, parietal, prefrontal, frontal, hippocampal, and inferotemporal cortex; and basal ganglia, midbrain, superior colliculus, and brainstem\n",
"- Focus to date is often on simple attractors:\n",
" - The NEF lets us easily generalize."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Generalizing dynamics\n",
"\n",
"- A cyclic attractor (i.e. a set of attractive points that the system moves through)\n",
"\n",
"![](\"lecture_memory/cyclic_attractor.png\")
\n",
"\n",
"- Note: The hill and ball analogy doesn't work anymore.\n",
"\n",
"- This is an oscillator. Technically a nonlinear one, since it should be stable (the Simple Harmonic Oscillator is linear). Let's build a stable oscillator."
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: pylab import has clobbered these variables: ['seed']\n",
"`%matplotlib` prevents importing * from pylab and numpy\n"
]
}
],
"source": [
"%pylab inline\n",
"import nengo\n",
"from nengo.utils.ensemble import response_curves\n",
"from nengo.dists import Uniform\n",
"\n",
"model = nengo.Network('Oscillator')\n",
"\n",
"freq = .25\n",
"scale = 1.1\n",
"N=300\n",
"\n",
"with model:\n",
" stim = nengo.Node(lambda t: [.5,.5] if .1![](\"lecture_memory/plane_attractor.png\")
\n",
"\n",
"- The attractor space is much more complicated, so you need a lot more neurons to do as good a job ($N^D$)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Populating the interactive namespace from numpy and matplotlib\n",
"\r",
"Building finished in 0:00:01. \n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: pylab import has clobbered these variables: ['piecewise']\n",
"`%matplotlib` prevents importing * from pylab and numpy\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\r",
"Simulating finished in 0:00:01. \n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAEACAYAAABVtcpZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8FfX1//HXAWVfZBEUkIgguIOKqFU0glhUFGtVFhVL\nbV2xVmvFndBiEbVWrPxEFBVwx/5UVFRQCIqKgoqIgOxbwiIghCQkIcn5/nHDzUL2e8kNzPv5eOTB\n3JnPzJz7IbnvO/OZO9fcHRERCaYasS5ARERiRyEgIhJgCgERkQBTCIiIBJhCQEQkwBQCIiIBFpUQ\nMLPxZrbJzBaU0uYpM1tmZvPNrEs09isiIpGJ1pHAi8BvS1poZhcC7d39aOBGYGyU9isiIhGISgi4\n+2zg11Ka9AUm5rX9GmhsZi2jsW8REam8qhoTaA2sK/A4KW+eiIjEkAaGRUQC7KAq2k8ScESBx23y\n5u3FzHQzIxGRCnJ3q8x60TwSsLyf4kwBBgGY2RnAdnffVNKG3H2f/fTs6UyfXvn1Fy1ymjUbtk9r\njNbPsGGqU3VW3x/VGb2fSETlSMDMXgXigWZmthYYBtQC3N3HuftUM7vIzJYDacDgaOxXREQiE5UQ\ncPeB5WgzJBr7EhGR6AncwHCER04A1KsXH/lGqkB8fHysSygX1RldqjO69pc6K8siPZ8UbWbm+7Km\nnj3hvvtC/1bGkiVw2WWhf0VEqgMzwys5MFxVVweJyAHsyCOPZM2aNbEu44AXFxfH6tWro7pNhYCI\nRGzNmjURX6UiZTOr1Jv9UmlMIEbbEBGpDgIXAgCRhOk+CGIRkZgJZAiIiEiIQkBEJM9FF13EpEmT\nYl1GlQrcwLDGBEQEYPjw4axYsYKJEyeG502dOjWGFcWGjgRERAIskCGggWGRYBk1ahRt2rShUaNG\nHHvssUydOpV//etfvPHGGzRs2JCTTz4ZgPPOO48XXngBgAkTJnD22Wdz55130qRJEzp06MBXX33F\nhAkTaNu2LYcddliho4j9VSBDQESCY+nSpYwZM4Zvv/2WlJQUPv74Y4499ljuu+8++vXrx86dO/n+\n+++LXfebb76hS5cubNu2jQEDBtC/f3/mzZvHihUrmDRpEkOGDCE9Pb2Kn1F0KQREZJ8zi85PZdSs\nWZOsrCwWLlxIdnY2bdu2pV27duVat127dgwaNAgzo1+/fqxfv55hw4Zx8MEH06tXL2rVqsXy5csr\nV1g1EbgQ0MCwSNVzj85PZbRv354nn3yShIQEWrRowcCBA9mwYUO51m3ZMv+r0OvWrQtA8+bNC81L\nTU2tXGHVROBCADQmIBI0/fv35/PPP2ft2rUADB06dJ/cgmF/FMgQEJHgWLp0KTNnziQrK4tatWpR\nt25datasyWGHHcbq1asrdM+jA/H+SAoBETmgZWZmcs8993DooYfSqlUrfvnlF0aOHMkVV1yBu9Os\nWTO6du0KlH2DtqLLD4SjicB9n8C558Lw4VDZ74lYtgwuvBD287EgkajKu599rMs44JXUz5F8n0Ag\njwQ0JiAiEhLIEBARkRCFgIhIgAUuBPQ5ARGRfIELAdCYgIjIHoEMARERCVEIiIgEmEJARCTAAhcC\nGhgWkUjcfPPNPPzwwyUuHzlyJDfccEMVVhSZwH29JGhgWEQq75lnnglPz5o1i2uuuYZ169aF5917\n772xKKvSonIkYGa9zWyJmS01s6HFLG9kZlPMbL6Z/Whmf4jGfkVEYsnd9/v7B0UcAmZWA3ga+C1w\nPDDAzI4p0uxW4Cd37wKcB/zbzAJ5FCIiVatdu3Y88sgjHH/88TRr1ozrr7+erKwsAJ577jmOPvpo\nmjdvzmWXXVboewbuuOMOWrZsSePGjencuTOLFi0CYPDgwTz00EOkp6dz0UUXkZycTMOGDWnUqBEb\nN25k+PDhXHvtteHtTJkyhRNOOIGmTZvSo0cPlixZUqi2f//733Tu3JkmTZowYMCAcG1VJRpHAt2A\nZe6+xt13A68DfYu0caBh3nRDYKu7Z0dh3xWmMQGR4Hn11VeZPn06K1as4Oeff2bEiBHMnDmT++67\nj7feeosNGzbQtm1b+vfvD8C0adOYPXs2y5cvZ8eOHbz55ps0a9as0Dbr1avHhx9+SKtWrdi5cycp\nKSkcdthhQP7dRZcuXcrAgQN56qmn+OWXX7jwwgu55JJLyM7Of/mbPHky06ZNY9WqVfzwww+89NJL\nVdMpeaLxbrw1sK7A4/WEgqGgp4EpZpYMNAD6RWG/laYxAZGqZcOj84fjwyr3Duy2226jVatWANx/\n//3cdtttJCcnc/3119O5c2cgNKDbtGlT1q5dy8EHH8zOnTtZtGgR3bp1o1OnTpXa75tvvkmfPn3o\n0aMHAHfddRejR4/myy+/5JxzzgHg9ttvD3+D2SWXXML8+fMrta/KqqpTMr8Fvnf3HmbWHphuZie5\n+/79vWwiUi6VffGOljZt2oSn4+LiSE5OZsOGDZx66qnh+fXr16dp06YkJSVx3nnnMWTIEG699VbW\nrl3L5ZdfzuOPP06DBg0qtN/k5GTi4uLCj82MI444gqSkpPC8gl9hWa9evXJ/9WW0RCMEkoC2BR63\nyZtX0GBgJIC7rzCzVcAxwLziNpiQkBCejo+PJ76yN/8XEYFCV++sXbuW1q1b06pVK1avXh2en5aW\nxtatW2ndujUAQ4YMYciQIWzZsoUrr7ySxx57jOHDhxfablmDwq1atWLhwoV71VIwlCojMTGRxMTE\niLaxRzRCYC7QwczigA1Af2BAkTZrgPOBL8ysJdARWFnSBguGQLRpTEAkeMaMGcPFF19M3bp1efjh\nh+nfvz/x8fEMHDiQq6++mk6dOnHfffdx5pln0rZtW+bNm0dubi6nnHIKdevWpU6dOtSosfcQasuW\nLdm6dSspKSk0atRor+VXXXUVo0aNYubMmXTv3p0nn3ySOnXqcOaZZ0b0fIq+OS4aThUR8cCwu+cA\nQ4BpwE/A6+6+2MxuNLM9n5gYAfzGzBYA04G73X1bpPuuLI0JiATLwIEDueCCC+jQoQNHH300999/\nPz179uSf//wnl19+Oa1bt2bVqlW89tprAKSkpPDnP/+Zpk2b0q5dO5o3b87f//73vbbbqVMnBgwY\nwFFHHUXTpk3ZuHFjoeUdO3bk5ZdfZsiQIRx66KF88MEHvPfeexx0UOj9d3W4vDRwXy951lnw6KOh\nfytj9erQV1MWOIoUCbzq/PWS7dq1Y/z48eHB2f2Zvl5SRESiSiEgIge06nDKpToL3Kd2NTAsEiwr\nV5Z4DYoQ0CMBDQyLiIQEMgRERCREISAiEmAaE4jRNkQOJHFxcRqArQIFb0ERLYELAdCYgEi0rdYH\nZ/ZbOh0kIhJgCgERkQALXAhoTEBEJF/gQgA0JiAiskcgQ0BEREIUAiIiARa4ENCYgIhIvsCFgIiI\n5AtkCGhgWEQkJJAhICIiIQoBEZEAC1wIaGBYRCRf4EIANCYgIrJHIENARERCFAIiIgEWuBDQmICI\nSL7AhQBoTEBEZI9AhoCIiIQoBEREAixwIaAxARGRfIELARERyReVEDCz3ma2xMyWmtnQEtrEm9n3\nZrbQzGZGY7+VpYFhEZGQgyLdgJnVAJ4GegLJwFwze9fdlxRo0xgYA1zg7klm1jzS/YqISOSicSTQ\nDVjm7mvcfTfwOtC3SJuBwP/cPQnA3bdEYb8iIhKhaIRAa2Bdgcfr8+YV1BFoamYzzWyumV0bhf1W\nigaGRUTyRXw6qAL7OQXoAdQHvjKzr9x9eXGNExISwtPx8fHEx8dHtRiNCYjI/iwxMZHExMSobCsa\nIZAEtC3wuE3evILWA1vcPQPIMLPPgM5AmSEgIiKFFX1zPHz48EpvKxqng+YCHcwszsxqAf2BKUXa\nvAucbWY1zawecDqwOAr7FhGRCER8JODuOWY2BJhGKFTGu/tiM7sxtNjHufsSM/sYWADkAOPcfVGk\n+65cvdVjGyIi1UFUxgTc/SOgU5F5zxZ5/DjweDT2FymNCYiIhOgTwyIiAaYQEBEJsMCFgMYERETy\nBS4EREQkXyBDQAPDIiIhgQwBEREJUQiIiARY4EJAA8MiIvkCFwKgMQERkT0CGQIiIhKiEBARCbDA\nhYDGBERE8gUuBEBjAiIiewQyBEREJEQhICISYIELAY0JiIjkC1wIiIhIvkCGgAaGRURCAhkCIiIS\nohAQEQmwwIWABoZFRPIFLgRAYwIiInsEMgRERCREISAiEmCBCwGNCYiI5AtcCIDGBERE9ghkCIiI\nSIhCQEQkwKISAmbW28yWmNlSMxtaSrvTzGy3mV0ejf1WhsYERETyRRwCZlYDeBr4LXA8MMDMjimh\n3SPAx5HuM1IaExARCYnGkUA3YJm7r3H33cDrQN9i2t0GvAVsjsI+RUQkCqIRAq2BdQUer8+bF2Zm\nrYDL3P0ZQO+lRUSqiYOqaD9PAgXHCkoNgoSEhPB0fHw88fHxUStEYwIisr9LTEwkMTExKtsyj/AV\nzczOABLcvXfe43sAd/dRBdqs3DMJNAfSgBvcfUox2/NIayrNccfB5Mlw/PGVW3/7doiLgx07oluX\niEhlmRnuXqmzLNE4EpgLdDCzOGAD0B8YULCBux+1Z9rMXgTeKy4AqooGhkVEQiIOAXfPMbMhwDRC\nYwzj3X2xmd0YWuzjiq4S6T5FRCQ6ojIm4O4fAZ2KzHu2hLZ/jMY+RUQkcoH7xLAGhkVE8gUuBEBj\nAiIiewQyBEREJEQhICISYIELAY0JiIjkC1wIgMYERET2CGQIiIhIiEJARCTAAhcCGhMQEckXuBAQ\nEZF8gQwBDQyLiIQEMgRERCREISAiEmCBCwENDIuI5AtcCIDGBERE9ghkCIiI7A/SstLYl1+3CwqB\nSrPhxvcbvmde8jwAUrNSseFGalbqPt/3ql9XMX/jfADWbF/Dtl3b9vk+pbCt6VurdH99Xu3DrNWz\n9vl+3J2JP0zc5/spr63pW0nemRzrMsolOzeb7NzsqG6zwcgGTPhhQlS3WVTgQiAaoZrZ43YAThl3\nCqc9dxo23HhwxoMANBzZMNzuktcuYUv6FrqO60qTUU14ZcEre20rIzuD3Tm7K7T/C1+5kJOfPRmA\nI0cfye/e+B0LNy/k0tcuZcw3Y3B3Rnw2olIvGu7O7LWzmfTDJACWbFnCtW9fC8BHyz9i8k+TAfhy\n3ZccN+Y4Br87OLzu5rTN4Xctryx4pdQ/iBmrZnDl5CuLXTZ/43ye/+75YmvbnbObxNWJTFsxjayc\nrL3eJeXk5pCdm81X677i6W+eBmDFthXh5Q/MeCD8ONdzmZs0d6/1k1KSOObpY0qsPTs3m+aPNafb\nc91Iy0pjc9pmbvngFjKyMwq125K+hY2pG0vcTkV8sOwDXl7wcqlt/jHrH4z4bER4398kfRNeNi95\nHjbcCv1/FeftJW9z3TvX4e7syNhBSmZKmbVNXTaVWz64pRzPYm+j54zm8jcuZ1PqJiD05mrX7l0A\nfL/he3pO7EnrJ1oXWmdL+haycrLYmbmTzOzMMvdxzyf3lPvN2bKty5ibNJe0rLRC8/fUt0dWThY2\nvPC54e4vduf8ieeXuO0dGTvIysnaa9vZudmc8uwp4cdJKUnsyNgRfrzy15XhdgXrzPVccnJzyvW8\nSmP7+lCjoszM92VNRx8NU6eG/q2MtDSnwePlz87p106n16Reheb97cy/8dX6r/hy3ZfhefFHxjNj\n0AxSMlOYvnI6Fx19EfM3zmdL+hb6vt6XT679hJYNWnJCixPo9HQnlm5dys57dxYKHYDGtRvTuE5j\n1u5YC8C7/d8l/sh4HpzxIH069qFdk3a0b9IeM2PW6lmc1fYs5ibNZcGmBSz6ZRFmxuivRwPw6aBP\neWPhG4z7bhwZ92fQ+onWbN21lTnXz2HM3DFMWhAKijV/XcPhDQ6n1ohaAEz63SSufftaZg+ezd+n\n/53WjVpzRusz+O83/6VHux48dO5DtBvdDoBFtyzi2EOPpeu4roztM5YTW5zIFZOv4P2l7wMw/tLx\nNKvbjKZ1m3LOS+cU28ejzh/F3WfdTfrudOr/qz71D65P2u7QH9pprU5jbvLccJ1xT8Zx/KHH89Mv\nPzHxsokMemcQK/6ygnaHtKPGP2pw3KHHseiXRQD4MGfX7l1M/GEiN31wE11bdWVe8jw+uvojer/S\nG4Alty7hmDH5gTFj0Ax6TOxB0p1J4RevP3b5I5k5mbzy4ytsu3sbabvTqF2zNldOvpI7z7yTxrUb\nM+qLUYzoMYL7Z9zPbd1uo2e7ntQ+qDbpu9M5+dmTWbp1KQDPXPwMn6z8hH7H9+OcuHNo2aAlAAs3\nL+TEZ04EYPbg2Zw/6fxwKC0dspSOT3cM1zg8fjgPnfsQ3yR9w8sLXmbZtmWkZaWxcPNCfs34FYAv\n/vgFvSb1ot0h7Xin/zvUsBoc1eQoZq+dzXXvXMdnf/iMNv9pwwuXvsCUpVN4Z8k7bPn7FurXqs+K\nbSs4aexJZD6QybzkeeHf/9SsVBbctIA56+dww/s3sH3odg4ZdUi4rvT70qn3r3pMHTiVRrUbcfaL\nZ9Oifgs2p23m5yE/s2TLEuodXI9ek3px15l38fhXj9OkThOS7kwieWcyLeq3oGHthuzI2EHjOo1Z\nsmUJx445FoB+x/dj6FlDObzh4WxJ30Jc4zg2pW1i7Y61LNi0gE9XfUrj2o155cfQG7VmdZux+q+r\naTiyIWv/upa2T7Zl7MVjOaLxEXRu2Zk6B9Wh+WPN2XTXJlrUbwEQDoU5189h265ttGnUhp1ZO7nx\n/RtZuHlhod/Z/if057Xfv8bX67/mjPFnhJ9/3YPrYsON4w49jh0ZO0jamcQD3R/g4o4Xc+b4M/l8\n8Oec3fZsbLgx+crJXDn5Sto3ac+K21fg7pUasVQIVND/fnyfK/7/JdEtah9rWb8lm9I2ld2wFM3r\nNWdL+pYKrVP3oLrsyt5VZrs2jdqwPmV9ZUs7ID3S8xGGzxpeZv89ev6j9O7Qm5PGnlRFlZXf1Sde\nHX5RlfJ5vNfj3DX9roqvmIBCoLwiDYE/vn0DLy54LrpFiYhEIqHyIaAxgQoY9+04BYCIHFACFwKR\n+MuHfwlPd2zWke5tu/PCpS+w+a7NfDro0/CyaddMY8AJAyLe36WdLi1x2emtT494+1WpZf2WsS5B\nRIoRyBCozAe+lm1dRmZO/pUI39/4PZ8N/ozBJw/m0PqH0qNdDwBe+/1r9Grfi1d//yqv//51bjr1\nJgBS7klh293b+Ojqj7j1tFtL3M9/L/xvePqoQ44qsd0Vx11R8ScBPHHBE8z981yW37ac+CPji23z\n2u9fA0IDiD/d8lN4/p7nsvy25cWuN3vwbNo3aV/ssqFnDQVCA71JdybRsFb+gPbFR19M83rNC7Xv\n3rY7Vx1/FTvu2UGTOk1KfD4ntDiBPh37lLi8vGZeN7NQ30NoQLUk9Q+uz6mHn1rssp+H/MyxzY8t\ndln3tt33mtexWf6gbeq9qbz2+9d4b8B79O3Ud6+27Zu057rO14Ufl9Q3W+8OXcI6uMtgrjnpGhrW\nasjM62YChQO5rL4r+OZmj2ObH8thDQ7jus6hAeKrjr+KkT1Hhpef1DI0PtGhaQeeufgZGtZqyM1d\nbwbg9tNvp3vb7qX+DUBorKNgm59u+Yk+HfuQcG5CeN7ngz8PT5/RJjS4+nCPh0vc5gktTmBK/ynU\nOahOof+7m7vezEktT2LJrUuYc/0cGtVuBMDiWxfvtY3//PY/dG3Vlfu738/iWxeTcX/GXm326Na6\nGzWtZqnPszgl/V7976r/AaG//dR7U2latykAt3W7rcL7KMTdq9VPqKR9p31792XLKr4eCRT6yczO\n3KvN9BXTPSc3p1zbm79hvru7P/zZw04C/tZPb/mGnRs8JzfH56yb4w/OeNBTM1N92vJp/reP/+Yk\nhPrlvZ/f8+SUZM/NzfVb3r/Fzxp/lt/x0R2+YecGn7Fyhm9L3+bu7j9s/MGf//Z5f2rOU+7uvjtn\nd7F1HP3U0YWeV48JPdzdPTkl2bNzst3d/Yu1X/gL373gCTMTwnW8+P2LTgKenpXuN713U3j+Hqc+\ne6qTgP+S9ou7u+fm5npWdlZ4eWpmqj//7fPhfaRmpnpSSpIPmznM1+1Yt1eda7av8ZXbVvqUJVOc\nBPyBTx8o1NdZ2Vk+YtYIn71mtm/YucEbj2zsJOBDpw8ttI3lW5f7im0rPDM7099e/La//uPr4Rrc\nC/8/F3zs7v7fr/8bfvzpyk/d3T07J9tr/7O2Z+zO8K3pW8PbWfXrKv/d67/zF757wXNyc3zhpoU+\ndu5Yz8nN8ePGHOefrvzUz594vp/30nmenZPtG3Zu2Ot3atHmRV5zeE2/46M7PC0rzUnAt6RtcXf3\nxb8sDvdtwTr7Te7nz857Nvx807LSCm0zJSPFd2bu9FmrZ/mPm350d/fVv672Pq/28eVbl/vsNbPd\n3X3SD5PCz/v5b58Pb3/k5yP3+r/ZIyc3J/wc0rLSPDc3N7xswcYF/qd3/1TseiTgL//wsjd/tLmT\ngPd/q/9ez6ugS169xEnAs3Oyvf3o9t7miTa+5Jcl4f+T13983V/6/qW9/mYLeuLLJ5wEfPuu7Xv9\nbdw29TYnAc/NzfXP13zuQ6cP9UYjGzkJ+KsLXt2r/lW/rvKUjBTPzM703NxcP+XZUzwpJSm8/O5p\nd/uizYvc3X3ErBHeb3I/HzdvnOfk5nirf7fyXbt3hdumZ6X7rt27/K6P7/J7P7nX5ybNLVR/l7Fd\nfOzcsXv3Yeh1s3KvuZVdcV/97C8hUPCFoypk7M7YZ9uelzTPf97yc5kBlpWd5Rt2bnD30Iv6nhck\n99CLUkE7M3cWepGKpoIBVZaCL0TlsW7HOt+Uuin8YvjgjAcLvYBsTd/q36z/psLbjVRObk74Rask\nJOAvfPdCVPa3cNNCb/ivhuHHKRkphV6sookEfML8Cb4jY4eTgL+x8I3wsm3p2zw1M3Wvdb5a95W7\nh/4/CoZvUT0n9PSLXrnI646oW2h+dk62b07dXOw6aVlpvnLbyr3mr/p1Vbnf5EVTdk62p2ell9om\nkhAI3NVB7dvDxx9Dhw7lX2ftjrXEPRkHwJQrP+aKs7uQua3FPqpQqpP03ems3r6a4w49LtalhD5I\ndf8u6hxUp9jlW9O30rRuU2w/u8HVo188yp9O+RNN6zYlKSWJVg1b7XfPIdbMTJeIllf79jBtWujf\n8uo1qRefrPyEO8+4k4fP/TeHHAIZJZ8KFBGpUpGEQFQGhs2st5ktMbOlZja0mOUDzeyHvJ/ZZnZi\nNPZbVWpYqJuuPunqGFciIhJdEYeAmdUAngZ+CxwPDDCzojdeWQmc4+6dgRHAfnWx/eJfQlcJnHzY\nyTGuREQkuqJxJNANWObua9x9N/A6UOjaNnef4+577og0B2hNjFT0TFOu57IuZR1A+DxlNTuDJiJS\nadEIgdbAugKP11P6i/yfgA+jsN9Kq8iY0547fF7S8ZIKrysiUt0dVJU7M7PzgMHA2aW1S0hICE/H\nx8cTHx+/T+sqzQfLPgCgfq36MatBRKSgxMREEhMTo7KtiK8OMrMzgAR37533+B5C16yOKtLuJOB/\nQG93X7H3lsLt9unVQUcdBZ98Evq3LLmeS6t/t2JT2ia2D91O4zqNycyERo0gs+zbmIuIVIlYXx00\nF+hgZnFmVgvoD0wpUmBbQgFwbWkBUBUqki9dxnYJ34K5cZ3GldqGiEh1FvHpIHfPMbMhwDRCoTLe\n3Reb2Y2hxT4OeBBoCvw/C42u7nb3bpHue1/7cfOPAOH7AomIHGiiMibg7h8BnYrMe7bA9J+BP0dj\nX9FQnsHdG9+7MTw97NxhFVpXRGR/Eci7iJbHuO/GhaeLu/OjiMiBIHAhUNHz+X079d3rPiYaExCR\nA0XgQqA80rLSwtN/6PKH2BUiIrKPBTIEyjqv/8cpfwxP9zqqV4XWFRHZnwQyBMry5k9vAjBj0Ax9\nSExEDmgKgVKU9PWLIiIHisCFQFmDujm5OeHpkr7YQgPDInKgCFwIQOnn9edvnF/pdUVE9jeBDIHS\ndH2uKwATLpsQ40pERPY9hUABHyz9IDxdHb5TVkRkXwtcCJR2Pn99yvrwdNdWXSu1DRGR/UngQqA0\no78eHesSRESqVCBDoLjB3fHfjWfxltB3CT9xwRMVWldEZH8VyBAozp/e+1N4elDnQTGsRESk6gQu\nBMo6nz/24rE0q9csom2IiOwvAhcCxen4347h6d8c8ZsYViIiUrUCGQJFz+sv27YMCN0r6MSWJ1Zo\nXRGR/VkgQ6CgtTvWhqfPa3deDCsREal6gQ4BdyfuyTgAMu7PiHE1IiJVL3AhUHBQt9EjjcLTtQ+q\nXaltiIjszwIXApB/Xj81KxWAdoe0q/C6IiIHgkCGQFG6T5CIBFVgQyA7Nzs8/dJlL8WuEBGRGApc\nCLhD2u5U/vrRX8PzmtdrHsOKRERi56BYF1DV3HI4ZmLDSq+vMQEROZAckEcC7k5mdia9JvVi1OxR\nZOVkhU//bLvwwkJtbzjlhliUKCJSLZhXs+sdzcwrUtP2jO0s3LyQ3xzxG3Zk7KDpo033anNInUPY\nnrGdSztdypSfpxRa5sMq/vzNdJmoiFQfZoa7V+o8RbU/EtiesX2veWt3rOXat69lU+om/jHrH3R/\nsTsXTLqg2AAouI2iAbDxbxsrXZdCQEQOBFE5EjCz3sCThEJlvLuPKqbNU8CFQBrwB3cv9hvdzcxJ\ngMd6PcbfzvwbNf5Rg9R7U6l7cF1qWA3OeP4Mvk76OuKacx7KoYZVLgPNIDdX4wMiUj1EciQQ8cCw\nmdUAngZ6AsnAXDN7192XFGhzIdDe3Y82s9OBscAZpW139NejObzB4QA0GNmAs444iy/WfRFpuWGV\nDQARkQNJNF4JuwHL3H2Nu+8GXgf6FmnTF5gI4O5fA43NrGVpG12fsp5r3r4m/DhaAWBZDbjppLui\nsi0Rkf1dNEKgNbCuwOP1efNKa5NUTJuouq7zdbzT753w4+nXTufDqz/ksBd38uDpj+3LXYuI7DcO\nmM8JtHgf0xDmAAAGXklEQVRlDZuvjqPOiqto8O2DTHv2BKYBhxMa8xj0bKjd5s1Qu/z3iitW3brQ\nqpXGBERk/xeNEEgC2hZ43CZvXtE2R5TRJt/MAtNHAnn3d+vT7kreXzUZgN/G9eXjNe8y68rFdDjk\nGPgztH4Obu/Xmb88ckKJm65dG5qV/u2RZUpOhvT0yLYhIlJZX36ZyFdfJYYfP/FE5bcV8dVBZlYT\n+JnQwPAG4BtggLsvLtDmIuBWd7/YzM4AnnT3YgeGzcwbj2zMsHOHAdCtdTdOa30atWrWwt1J351O\n/Vr1ycrJ4rsN33FGm/zNpGWlha8iEhEJikiuDormJaKjyb9E9BEzuxFwdx+X1+ZpoDehS0QHu/t3\nJWyrQh8WExEJupiHQDQpBEREKuaA/sSwiIjsOwoBEZEAUwiIiASYQkBEJMAUAiIiAaYQEBEJMIWA\niEiAKQRERAJMISAiEmAKARGRAFMIiIgEmEJARCTAFAIiIgGmEBARCTCFgIhIgCkEREQCTCEgIhJg\nCgERkQBTCIiIBJhCQEQkwBQCIiIBphAQEQkwhYCISIApBEREAkwhICISYAoBEZEAUwiIiASYQkBE\nJMAiCgEza2Jm08zsZzP72MwaF9OmjZnNMLOfzOxHM/tLJPsUEZHoifRI4B7gE3fvBMwA7i2mTTZw\np7sfD5wJ3Gpmx0S435hKTEyMdQnlojqjS3VGl+qsHiINgb7AhLzpCcBlRRu4+0Z3n583nQosBlpH\nuN+Y2l9+KVRndKnO6FKd1UOkIdDC3TdB6MUeaFFaYzM7EugCfB3hfkVEJAoOKquBmU0HWhacBTjw\nQDHNvZTtNADeAm7POyIQEZEYM/cSX7fLXtlsMRDv7pvM7DBgprsfW0y7g4D3gQ/dfXQZ26x8QSIi\nAeXuVpn1yjwSKMMU4A/AKOA64N0S2r0ALCorAKDyT0RERCou0iOBpsCbwBHAGuAqd99uZocDz7l7\nHzM7C/gM+JHQ6SIH7nP3jyKuXkREIhJRCIiIyP4tJp8YNrPeZrbEzJaa2dAS2jxlZsvMbL6Zdanq\nGvNqKLVOMzvXzLab2Xd5P8UNlu/rGseb2SYzW1BKm+rQl6XWWR36Mq+Ocn24MdZ9Wp46Y92nZlbb\nzL42s+/zahxWQrtY92WZdca6L4vUUiOvhiklLK9Yf7p7lf4QCp7lQBxwMDAfOKZImwuBD/KmTwfm\nVNM6zwWmVHVtRWo4m9BltwtKWB7zvixnnTHvy7w6DgO65E03AH6upr+f5akz5n0K1Mv7tyYwB+hW\n3fqynHXGvC8L1HIH8HJx9VSmP2NxJNANWObua9x9N/A6oQ+dFdQXmAjg7l8Djc2sJVWrPHVC6JLZ\nmHH32cCvpTSpDn1Znjohxn0J5f5wY8z7tJx1Qux/P9PzJmsTuhCl6PnnmPdl3r7LqhOqwe+nmbUB\nLgKeL6FJhfszFiHQGlhX4PF69v7lLdomqZg2+1p56gQ4M++w6wMzO65qSquQ6tCX5VWt+rKUDzdW\nqz4t40OYMe3TvFMX3wMbgenuPrdIk2rRl+WoE6rH7+d/gL9T8meyKtyfuotoZL4F2rp7F+Bp4J0Y\n17M/q1Z9ub98uLGMOmPep+6e6+4nA22A06tDuBenHHXGvC/N7GJgU94RoBGlI5NYhEAS0LbA4zZ5\n84q2OaKMNvtamXW6e+qew0h3/xA4OO+y2eqkOvRlmapTX+Z9uPEtYJK7F/fZl2rRp2XVWZ361N1T\ngJlA7yKLqkVf7lFSndWkL88CLjWzlcBrwHlmNrFImwr3ZyxCYC7QwczizKwW0J/Qh84KmgIMAjCz\nM4DtnnePoipUZp0Fz7WZWTdCl9xuq9oyQ7un5HcF1aEv9yixzmrUl1D2hxurS5+WWmes+9TMmlve\n7eXNrC7QC1hSpFnM+7I8dca6LwHc/T53b+vuRxF6PZrh7oOKNKtwf0b6ieEKc/ccMxsCTCMUQuPd\nfbGZ3Rha7OPcfaqZXWRmy4E0YHB1rBO4wsxuBnYDu4B+VV2nmb0KxAPNzGwtMAyoRTXqy/LUSTXo\ny7w6zwKuBn7MO0fswH2ErhKrNn1anjqJfZ8eDkwwsxqE/obeyOu7avW3Xp46iX1flijS/tSHxURE\nAkwDwyIiAaYQEBEJMIWAiEiAKQRERAJMISAiEmAKARGRAFMIiIgEmEJARCTA/g+ZxHQ3JlML3wAA\nAABJRU5ErkJggg==\n",
"text/plain": [
"\n",
" \n",
"
\n",
" "
],
"text/plain": [
"\n",
" \n",
"
\n",
" "
],
"text/plain": [
"\n",
" \n",
"
\n",
" "
],
"text/plain": [
"![](\"lecture_memory/human_wm.png\")
\n",
"\n",
"- So far, the networks we've seen would work well for free recall presumably.\n",
" - How to include order information?"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Serial Working Memory\n",
"\n",
"- An ordered list is a very simple kind of structure\n",
"- To represent structures, we can use what we learned last time: vector binding\n",
" - Specifically, Semantic Pointers\n",
" \n",
"- In this case, we can do something like:\n",
"\n",
"$Pos_0\\circledast Item_0 + Pos_1\\circledast Item_1 + ...$\n",
"\n",
"- The $Item$ can just whatever vector we want to remember\n",
"- How should we generate $Pos$? What features does it need?\n",
" - It would be good if we could generate them on the fly, forever\n",
" - It would be good if $Pos_0$ \"came before\" $Pos_1$ in some sense\n",
"\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"- There is a kind of vector that's ideal for this called a 'unitary vector'\n",
" - A unitary vector does not change length when convolved\n",
" - So, you can convolve forever\n",
" - The convolution of a unitary vector with itself is 'unlike' the original vector\n",
" - So, you can go fwd/backward with convolution/correlation\n",
"- This gives a way of 'counting' positions:\n",
" - $Pos_0 \\circledast PlusOne = Pos_1$\n",
" - $Pos_1 \\circledast PlusOne = Pos_2$\n",
" - $Pos_2 \\circledast PlusOne = Pos_3$\n",
" - etc.\n",
" - And, $Pos_3 \\circledast PlusOne' = Pos_2$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"\n",
"- We can put this representation together with primacy and recency, and we get something like this\n",
"\n",
"![](\"lecture_memory/wm_model.png\")
\n",
"\n",
"- To do 'recall' from this representation, we want to remember what we've already recalled, so we can add a circuit like this\n",
"\n",
"![](\"lecture_memory/wm_recall.png\")
\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"- This does a pretty good job of capturing lots of working memory data\n",
"\n",
"![](\"lecture_memory/model_data1.png\")
\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"- Order effects\n",
"\n",
"![](\"lecture_memory/model_data2.png\")
\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"- Error probability of items in the list (closer items are more likely to be incorrect, unless you change similarity between items)\n",
"\n",
"![](\"lecture_memory/model_data3.png\")
\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"- Item similarity effects\n",
"\n",
"![](\"lecture_memory/model_data4.png\")
\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"- Arbitrary list lengths\n",
"\n",
"- This model can actually do lots more as well\n",
" - backwards recall, etc.\n",
"- A slight variation does free recall\n",
" - you have to include non-bound item representations as well\n",
"- You'll notice that it relies on a 'clean-up' \n",
" - This is a kind of long term memory\n",
" - We saw this show up before when talking about symbols\n",
" - How can we build such a thing in spiking neurons?"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Long term memory\n",
"\n",
"\n",
"- In order for memories to be stable over the long term, they must be encoded in connection weights\n",
"- These kind of memories are often called 'associative memories'\n",
" - Auto-associative: recall the same thing you're shown\n",
" - e.g. for noise reduction\n",
" - pattern completion\n",
" - Hetero-associative: recall some arbitrary association\n",
" - e.g. for capturing domain relationships\n",
" - reasoning\n",
" \n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"- Typical solutions in ANNs are \n",
" - Hopfield networks: A many-point attractor network\n",
" - Linear associators: Do SVD on the input/output to get a weight matrix\n",
" - Multi-layer Perceptron (MLP): Use backprop to train a network to compute the feedfwd mapping\n",
"\n",
"![](\"lecture_memory/hopfield.png\")
\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"- Often worried about how to learn these\n",
" - We'll talk about learning in a later lecture\n",
" - Often the learning is so un-'biologically plausible' that it's best thought of as an optimization (like least squares in the NEF).\n",
" \n",
"- How to compute these kinds of mappings with the NEF/SPA?\n",
" - We want to 'recognize' specific vectors (in a vocabulary)\n",
" - We want only some neurons to fire when a given vector is present (sparse)\n",
" - We want to activate a given output vector if the input is recognized\n",
" \n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"- Can do it in one layer with a feedforward approach\n",
" - Set the encoding vectors of some small set of neurons to be a vocabulary item\n",
" - That will pick out specific vectors\n",
" - Set the intercepts of neurons to be high in the positive direction\n",
" - So they will only fire if there is 'enough' of that vector in the input\n",
" - Set the linear transformation on the output of those neurons to be the desired output vector\n",
" - Since these are in the weights, the output will perfectly represent the desired output\n",
" \n",
"- We've seen several techniques that will make this easy\n",
" - Ensemble arrays: For collecting lots of small populations together\n",
" - SPA module: For defining vocabs, etc."
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"import nengo\n",
"from nengo import spa\n",
"\n",
"seed=1\n",
"np.random.seed(seed)\n",
"model = spa.SPA(\"Associative Memory\", seed=seed)\n",
"\n",
"D = 32\n",
"vocab = spa.Vocabulary(D)\n",
"vocab.parse('BLUE+GREEN+RED')\n",
"\n",
"noise_RED = vocab.parse(\"RED\").v + .2*np.random.randn(D)\n",
"noise_RED = noise_RED/np.linalg.norm(noise_RED)\n",
"\n",
"noise_GREEN = vocab.parse(\"GREEN\").v + .2*np.random.randn(D)\n",
"noise_GREEN = noise_GREEN/np.linalg.norm(noise_GREEN)\n",
"\n",
"def memory_input(t):\n",
" if t < 0.2:\n",
" return vocab.parse(\"BLUE\").v\n",
" elif .2 < t < .5:\n",
" return noise_RED\n",
" elif .5 < t < .8:\n",
" return vocab.parse(\"RED\").v\n",
" elif .8 < t < 1:\n",
" return noise_GREEN\n",
" else:\n",
" return vocab.parse(\"0\").v\n",
"\n",
"with model:\n",
" stim = nengo.Node(output=memory_input, label='input')\n",
" model.am = spa.AssociativeMemory(vocab)\n",
" nengo.Connection(stim, model.am.input)\n",
"\n",
" in_p = nengo.Probe(stim)\n",
" out_p = nengo.Probe(model.am.output, synapse=0.03)\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
"
\n",
" "
],
"text/plain": [
"![](\"lecture_memory/cleanup.png\")
\n",
"\n",
"- Scaling properties of a neural clean-up memory. \n",
" - A SP is formed by binding $k$ pairs of random SPs and adding \n",
" - Binding to random probes from that set. \n",
" - Figure shows the minimum number of dimensions required to recover a lexical item from the input 99% of the time. \n",
" - Averages over 200 simulations for each combination of $k$, $M$, and $D$ values. \n",
" - Vertical dashed line is approximate size of an adult lexicon. \n",
" - Horizontal dashed lines show the performance of a non-neural clean-up directly implementing the algorithm."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## SPA Example\n",
"\n",
"- This is the same example as for the symbols lecture, but with a working memory\n",
" - The WM is just a single attractor network\n",
" - The model is supposed to memorize the bound inputs and then answer questions about what is bound to what"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: pylab import has clobbered these variables: ['seed', 'piecewise']\n",
"`%matplotlib` prevents importing * from pylab and numpy\n"
]
}
],
"source": [
"%pylab inline\n",
"import nengo\n",
"from nengo import spa\n",
"\n",
"def color_input(t):\n",
" if t < 0.25:\n",
" return 'RED'\n",
" elif t < 0.5:\n",
" return 'BLUE'\n",
" else:\n",
" return '0'\n",
"\n",
"def shape_input(t):\n",
" if t < 0.25:\n",
" return 'CIRCLE'\n",
" elif t < 0.5:\n",
" return 'SQUARE'\n",
" else:\n",
" return '0'\n",
"\n",
"def cue_input(t):\n",
" if t < 0.5:\n",
" return '0'\n",
" sequence = ['0', 'CIRCLE', 'RED', '0', 'SQUARE', 'BLUE']\n",
" idx = int(((t - 0.5) // (1. / len(sequence))) % len(sequence))\n",
" return sequence[idx]\n",
"\n",
"seed=1\n",
"model = spa.SPA(label=\"Simple question answering\", seed=seed)\n",
"\n",
"dimensions = 32\n",
"vocab = model.get_default_vocab(dimensions)\n",
"vocab.parse('BLUE+RED+CIRCLE+SQUARE')\n",
"\n",
"with model: \n",
" model.color_in = spa.Buffer(dimensions=dimensions)\n",
" model.shape_in = spa.Buffer(dimensions=dimensions)\n",
" model.conv = spa.Memory(dimensions=dimensions, subdimensions=4, synapse=0.4)\n",
" model.cue = spa.Buffer(dimensions=dimensions)\n",
" model.out = spa.Buffer(dimensions=dimensions)\n",
" model.am = spa.AssociativeMemory(vocab, threshold=0.1)\n",
"\n",
" model.inp = spa.Input(color_in=color_input, shape_in=shape_input, cue=cue_input)\n",
" \n",
" # Connect the buffers\n",
" cortical_actions = spa.Actions(\n",
" 'conv = color_in * shape_in',\n",
" 'out = conv * ~cue',\n",
" 'am = out'\n",
" )\n",
" model.cortical = spa.Cortical(cortical_actions) \n",
" \n",
" model.config[nengo.Probe].synapse = nengo.Lowpass(0.03)\n",
" color_in = nengo.Probe(model.color_in.state.output)\n",
" shape_in = nengo.Probe(model.shape_in.state.output)\n",
" cue = nengo.Probe(model.cue.state.output)\n",
" conv = nengo.Probe(model.conv.state.output)\n",
" out = nengo.Probe(model.out.state.output)\n",
" clean = nengo.Probe(model.am.output)\n",
" \n"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
"
\n",
" "
],
"text/plain": [
"