{ "metadata": { "name": "" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## Action Selection and the Basal Ganglia\n", "\n", "- Here is the creature we built that has two modes:\n", " - move in the direction it was told to move in\n", " - or go back to where it started from\n", "- Which one of those things it did would depend on a separate input ('mode')" ] }, { "cell_type": "code", "collapsed": false, "input": [ "import nengo\n", "\n", "model = nengo.Network(label='critter')\n", "with model:\n", " # velocity input\n", " velocity = nengo.Node([0, 0], label='velocity')\n", " \n", " # motor output\n", " motor = nengo.Ensemble(200, dimensions=2, label='motor')\n", " nengo.Probe(motor)\n", "\n", " # make the position memory\n", " position = nengo.Ensemble(500, dimensions=2, label='position',\n", " radius=3)\n", " nengo.Connection(position, position, synapse=0.1)\n", " nengo.Connection(motor, position, transform=0.1)\n", " nengo.Probe(position)\n", "\n", " # figure out which way is home\n", " home_location = [0.5, 0.5]\n", " home_dir = nengo.Ensemble(200, dimensions=2, label='home_dir')\n", " def compute_home(x):\n", " return (home_location - x) * 10\n", " nengo.Connection(position, home_dir, function=compute_home)\n", " nengo.Probe(home_dir)\n", "\n", " # what mode am I in?\n", " mode = nengo.Node(1, label='mode')\n", "\n", " # should I go where I'm told?\n", " d_velocity = nengo.Ensemble(300, dimensions=3, label='d_velocity',\n", " radius=2)\n", " nengo.Connection(velocity, d_velocity[[0,1]])\n", " nengo.Connection(mode, d_velocity[2])\n", " def velocity_func(x):\n", " a, b, mode = x\n", " if mode > 0.5:\n", " return a, b\n", " else:\n", " return 0, 0\n", "\n", " # should I go home?\n", " nengo.Connection(d_velocity, motor, function=velocity_func)\n", " d_home = nengo.Ensemble(300, dimensions=3, label='d_home',\n", " radius=2)\n", " nengo.Connection(home_dir, d_home[[0,1]])\n", " nengo.Connection(mode, d_home[2])\n", " def home_func(x):\n", " a, b, mode = x\n", " if mode < -0.5:\n", " return a, b\n", " else:\n", " return 0, 0\n", " nengo.Connection(d_home, motor, function=home_func)\n" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- This sort of system comes up a lot in cognitive models\n", " - A bunch of different possible actions\n", " - Pick one of them to do\n", "- How can we do this?\n", "\n", "- In the above example, we did it like this:\n", " - $m = (mode>0.5)(v_x, v_y) + (mode<-0.5)(h_x, h_y)$\n", "- What about more complex situations?\n", " - What if there are three possible actions?\n", " - What if the actions involve different outputs?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Action Selection and Execution\n", "\n", "- This is known as action selection\n", "- Often divided into two parts:\n", " - Action selection (identifying the action to perform)\n", " - Action execution (actually performing the action)\n", "- Actions can be many different things\n", " - physical movements\n", " - moving attention\n", " - changing contents of working memory (\"1, 2, 3, 4, ...\")\n", " - recalling items from long-term memory\n", "\n", "\n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Action Selection\n", "\n", "- How can we do this?\n", "- We have a bunch of different possible actions\n", " - go to where we started from\n", " - go in the direction we are told to go in\n", " - move randomly\n", " - go towards food\n", " - go away from predators\n", "- Which one do we pick?\n", " - Ideas?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Reinforcement Learning\n", "- Let's steal an idea from reinforcement learning\n", "- Lots of different actions, learn to pick one\n", "- Each action has a utility $Q$ that depends on the current state $s$\n", " - $Q(s, a)$\n", "- Pick the action that has the largest $Q$\n", "- Note\n", " - Lots of different variations on this\n", " - $V(s)$\n", " - softmax: $p(a_i) = e^{Q(s, a_i)/T} / \\sum_i e^{Q(s, a_i)/T}$ \n", "- In Reinforcement Learning research, people come up with learning algorithms for adjusting $Q$ based on rewards\n", "- Let's not worry about that for now and just use the basic idea\n", " - There's some sort of state $s$\n", " - For each action $a_i$, compute $Q(s, a_i)$ which is a function that we can define\n", " - Take the biggest\n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Implementation\n", "\n", "- One group of neurons to represent the state $s$\n", " - Or many different groups of neurons representing different parts of $s$\n", "- One group of neurons for each action's utility $Q(s, a_i)$\n", " - Or one large group of neurons for all the $Q$ values\n", "- Connect the $s$ neurons to the $Q$ neurons with functions that compute $Q(s, a_i)$\n", "\n", "- What should the output be?\n", " - We could have $index$, which is the index $i$ of the action with the largest $Q$ value\n", " - Or we could have something like $[0,0,1,0]$, indicating which action is selected\n", " - Advantages and disadvantages?\n", "- The second option seems easier if we consider that we have to do action execution next...\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### A Simple Example\n", "\n", "- State $s$ is a semantic pointer\n", " - 16-dimensional\n", "- Four actions (A, B, C, D)\n", "- Do action A if $s$ is near `CAT`, B if near `DOG`, C if near `RAT`, D if near `COW`\n", " - $Q(s, a_A)=s \\cdot$ `CAT`\n", " - $Q(s, a_B)=s \\cdot$ `DOG`\n", " - $Q(s, a_C)=s \\cdot$ `RAT`\n", " - $Q(s, a_D)=s \\cdot$ `COW`\n", "\n", "\n" ] }, { "cell_type": "code", "collapsed": false, "input": [ "import nengo\n", "import nengo.spa as spa\n", "\n", "model = nengo.Network(label=\"Action1\")\n", "with model:\n", " state = nengo.Ensemble(500, 16, label='state')\n", " Q_A = nengo.Ensemble(50, 1, label='Q_A')\n", " Q_B = nengo.Ensemble(50, 1, label='Q_B')\n", " Q_C = nengo.Ensemble(50, 1, label='Q_C')\n", " Q_D = nengo.Ensemble(50, 1, label='Q_D')\n", "\n", " vocab = spa.Vocabulary(16) \n", " state.vocab = vocab\n", " nengo.Connection(state, Q_A, transform=[vocab.parse('DOG').v])\n", " nengo.Connection(state, Q_B, transform=[vocab.parse('CAT').v])\n", " nengo.Connection(state, Q_C, transform=[vocab.parse('RAT').v])\n", " nengo.Connection(state, Q_D, transform=[vocab.parse('COW').v])\n", " \n", " nengo.Probe(state)\n", " nengo.Probe(Q_A)\n", " nengo.Probe(Q_B)\n", " nengo.Probe(Q_C)\n", " nengo.Probe(Q_D)\n" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- It's annoying to have all those separate $Q$ Ensembles\n", "- Nengo has an ```EnsembleArray``` capability to help with this\n", " - Doesn't change the model at all\n", " - It just groups things together for you" ] }, { "cell_type": "code", "collapsed": false, "input": [ "import nengo\n", "import nengo.spa as spa\n", "\n", "model = nengo.Network(label=\"Action2\")\n", "with model:\n", " state = nengo.Ensemble(500, 16, label='state') \n", " Q = nengo.networks.EnsembleArray(50, 4, label='Q')\n", "\n", " vocab = spa.Vocabulary(16) \n", " state.vocab = vocab\n", " nengo.Connection(state, Q.input, transform=[vocab.parse('DOG').v,\n", " vocab.parse('CAT').v,\n", " vocab.parse('RAT').v,\n", " vocab.parse('COW').v])\n", "\n", " nengo.Probe(state)\n", " nengo.Probe(Q.output)" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- How do we implement the $max$ function?\n", "- Well, it's just a function, so let's implement it\n", " - Need to combine all the $Q$ values into one 4-dimensional ensemble\n", " - Why?" ] }, { "cell_type": "code", "collapsed": false, "input": [ "import nengo\n", "import nengo.spa as spa\n", "\n", "model = nengo.Network(label=\"Action3\")\n", "with model:\n", " state = nengo.Ensemble(500, 16, label='state')\n", " \n", " Q = nengo.networks.EnsembleArray(50, 4, label='Q')\n", "\n", " vocab = spa.Vocabulary(16) \n", " state.vocab = vocab\n", " nengo.Connection(state, Q.input, transform=[vocab.parse('DOG').v,\n", " vocab.parse('CAT').v,\n", " vocab.parse('RAT').v,\n", " vocab.parse('COW').v])\n", " nengo.Probe(state)\n", " nengo.Probe(Q.output)\n", " \n", " Q_together = nengo.Ensemble(200, 4, label='Q_together')\n", " nengo.Probe(Q_together)\n", " nengo.Connection(Q.output, Q_together)\n", " \n", " def maximum(x):\n", " result = [0, 0, 0, 0]\n", " result[x.argmax()] = 1\n", " return result\n", " \n", " R = nengo.networks.EnsembleArray(50, 4, label='R')\n", " nengo.Connection(Q_together, R.input, function=maximum)\n", " nengo.Probe(R.output)\n" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- That doesn't seem to work very well\n", "- Very nonlinear function, so hard for neurons to approximate it well\n", " - Need to have all the $Q$ values represented in one ensemble\n", "- Other options?\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The Standard Neural Network Approach (modified)\n", "\n", "- If you give this problem to a standard neural networks person, what would they do?\n", "- They'll say this is exactly what neural networks are great at\n", " - Implement this with mutual inhibition and self-excitation\n", "- Neural competition\n", " - 4 \"neurons\"\n", " - have excitation from each neuron back to themselves\n", " - have inhibition from each neuron to all the others\n", "- Now just put in the input and wait for a while and it'll stablize to one option\n", "- Can we do that?\n", "- Sure! Just replace each \"neuron\" with a group of neurons, and compute the desired function on those connections\n", " - note that this is a very general method of converting any non-realistic neural model into a biologically realistic spiking neuron model\n", " " ] }, { "cell_type": "code", "collapsed": false, "input": [ "import nengo\n", "import nengo.spa as spa\n", "\n", "model = nengo.Network(label=\"Action4\")\n", "with model:\n", " state = nengo.Ensemble(500, 16, label='state')\n", " \n", " Q = nengo.networks.EnsembleArray(50, 4, label='Q')\n", "\n", " vocab = spa.Vocabulary(16) \n", " state.vocab = vocab\n", " nengo.Connection(state, Q.input, transform=[vocab.parse('DOG').v,\n", " vocab.parse('CAT').v,\n", " vocab.parse('RAT').v,\n", " vocab.parse('COW').v])\n", " nengo.Probe(state)\n", " nengo.Probe(Q.output)\n", "\n", " e = 0.1\n", " i = -1\n", "\n", " transform = [[e, i, i, i], [i, e, i, i], [i, i, e, i], [i, i, i, e]]\n", " nengo.Connection(Q.output, Q.input, transform=transform)\n" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Oops, that's not quite right\n", "- Why is it selecting more than one action?" ] }, { "cell_type": "code", "collapsed": false, "input": [ "import nengo\n", "import nengo.spa as spa\n", "\n", "model = nengo.Network(label=\"Action5\")\n", "with model:\n", " state = nengo.Ensemble(500, 16, label='state')\n", " \n", " Q = nengo.networks.EnsembleArray(50, 4, label='Q')\n", "\n", " vocab = spa.Vocabulary(16) \n", " state.vocab = vocab\n", " nengo.Connection(state, Q.input, transform=[vocab.parse('DOG').v,\n", " vocab.parse('CAT').v,\n", " vocab.parse('RAT').v,\n", " vocab.parse('COW').v])\n", " nengo.Probe(state)\n", " nengo.Probe(Q.output)\n", "\n", " e = 0.1\n", " i = -1\n", "\n", " def positive(x):\n", " if x[0]<0: return [0]\n", " else: return x\n", " Q.add_output('positive', positive)\n", " transform = [[e, i, i, i], [i, e, i, i], [i, i, e, i], [i, i, i, e]]\n", " nengo.Connection(Q.positive, Q.input, transform=transform)" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Now we only influence other Actions when we have a positive value\n", " - Note: Is there a more neurally efficient way to do this?\n", "- Much better\n", "- Selects one action reliably\n", "- But still gives values smaller than 1.0 for the output a lot\n", " - Can we fix that?\n", " - What if we adjust ```e```?" ] }, { "cell_type": "code", "collapsed": false, "input": [ "import nengo\n", "import nengo.spa as spa\n", "\n", "model = nengo.Network(label=\"Action5\")\n", "with model:\n", " state = nengo.Ensemble(500, 16, label='state')\n", " \n", " Q = nengo.networks.EnsembleArray(50, 4, label='Q')\n", "\n", " vocab = spa.Vocabulary(16) \n", " state.vocab = vocab\n", " nengo.Connection(state, Q.input, transform=[vocab.parse('DOG').v,\n", " vocab.parse('CAT').v,\n", " vocab.parse('RAT').v,\n", " vocab.parse('COW').v])\n", " nengo.Probe(state)\n", " nengo.Probe(Q.output)\n", "\n", " e = 1\n", " i = -1\n", "\n", " def positive(x):\n", " if x[0]<0: return [0]\n", " else: return x\n", " Q.add_output('positive', positive)\n", " transform = [[e, i, i, i], [i, e, i, i], [i, i, e, i], [i, i, i, e]]\n", " nengo.Connection(Q.positive, Q.input, transform=transform)" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Hmm, that seems to introduce a new problem\n", "- The self-excitation is so strong that it can't respond to changes in the input\n", " - Indeed, any method like this is going to have some form of memory effects\n", "- Could we do anything to help without increasing ```e``` too much? " ] }, { "cell_type": "code", "collapsed": false, "input": [ "import nengo\n", "import nengo.spa as spa\n", "\n", "model = nengo.Network(label=\"Action6\")\n", "with model:\n", " state = nengo.Ensemble(500, 16, label='state')\n", " \n", " Q = nengo.networks.EnsembleArray(50, 4, label='Q')\n", "\n", " vocab = spa.Vocabulary(16) \n", " state.vocab = vocab\n", " nengo.Connection(state, Q.input, transform=[vocab.parse('DOG').v,\n", " vocab.parse('CAT').v,\n", " vocab.parse('RAT').v,\n", " vocab.parse('COW').v])\n", " nengo.Probe(state)\n", " nengo.Probe(Q.output)\n", "\n", " e = 0.5\n", " i = -1\n", "\n", " def positive(x):\n", " if x[0]<0: return [0]\n", " else: return x\n", " Q.add_output('positive', positive)\n", " transform = [[e, i, i, i], [i, e, i, i], [i, i, e, i], [i, i, i, e]]\n", " nengo.Connection(Q.positive, Q.input, transform=transform)\n", "\n", " def threshold(x):\n", " if x[0]<0: return [0]\n", " else: return 1\n", " Q.add_output('threshold', threshold)\n", "\n", " R = nengo.networks.EnsembleArray(50, 4, label='R')\n", " nengo.Probe(R.output)\n", " nengo.Connection(Q.threshold, R.input)\n" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Better behaviour\n", "- But there's still situations where there's too much memory\n", "- We can reduce this by reducing ```e```" ] }, { "cell_type": "code", "collapsed": false, "input": [ "import nengo\n", "import nengo.spa as spa\n", "\n", "model = nengo.Network(label=\"Action6\")\n", "with model:\n", " state = nengo.Ensemble(500, 16, label='state')\n", " \n", " Q = nengo.networks.EnsembleArray(50, 4, label='Q')\n", "\n", " vocab = spa.Vocabulary(16) \n", " state.vocab = vocab\n", " nengo.Connection(state, Q.input, transform=[vocab.parse('DOG').v,\n", " vocab.parse('CAT').v,\n", " vocab.parse('RAT').v,\n", " vocab.parse('COW').v])\n", " nengo.Probe(state)\n", " nengo.Probe(Q.output)\n", "\n", " e = 0.2\n", " i = -1\n", "\n", " def positive(x):\n", " if x[0]<0: return [0]\n", " else: return x\n", " Q.add_output('positive', positive)\n", " transform = [[e, i, i, i], [i, e, i, i], [i, i, e, i], [i, i, i, e]]\n", " nengo.Connection(Q.positive, Q.input, transform=transform)\n", "\n", " def threshold(x):\n", " if x[0]<0: return [0]\n", " else: return 1\n", " Q.add_output('threshold', threshold)\n", "\n", " R = nengo.networks.EnsembleArray(50, 4, label='R')\n", " nengo.Probe(R.output)\n", " nengo.Connection(Q.threshold, R.input)\n" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Much less memory, but it's still there\n", "- And much slower to respond to changes\n", "- Note that this speed is dependent on $e$, $i$, and the time constant of the neurotransmitter used\n", "- Can be hard to find good values\n", "- And this gets harder to balance as the number of actions increases\n", " - Also hard to balance for a wide range of $Q$ values\n", " - (Does it work for $Q$=[0.9, 0.9, 0.95, 0.9] and $Q$=[0.2, 0.2, 0.25, 0.2]?)\n", "\n", "- But this is still a pretty standard approach\n", " - Nice and easy to get working for special cases\n", " - Don't really need the NEF (if you're willing to assume non-realistic non-spiking neurons)\n", " - (Although really, if you're not looking for biological realism, why not just compute the max function?)\n", " \n", "- Example: [OReilly, R.C. (2006). Biologically Based Computational Models of High-Level Cognition. Science, 314, 91-94.](http://grey.colorado.edu/mediawiki/sites/CompCogNeuro/images/a/a5/OReilly06.pdf)\n", " - [Leabra](http://grey.colorado.edu/CompCogNeuro/index.php/CCNBook/Main)\n", "- They tend to use a \"kWTA\" (k-Winners Take All) approach in their models\n", " - Set up inhibition so that only $k$ neurons will be active\n", " - But since that's complex to do, just [do the math instead of doing the inhibition](http://grey.colorado.edu/CompCogNeuro/index.php/CCNBook/Networks#kWTA_Approximation_for_Inhibition)\n", " - We think that doing it their way means that the dynamics of the model will be wrong (i.e. all the effects we saw above are being ignored).\n", " \n", "- Any other options?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Biology\n", "\n", "- Let's look at the biology\n", "- Where is this action selection in the brain?\n", "- General consensus: the basal ganglia\n", "\n", "\n", "\n", "- Pretty much all of cortex connects in to this area (via the striatum)\n", "- Output goes to the thalamus, the central routing system of the brain\n", "- Disorders of this area of the brain cause problems controlling actions:\n", " - Parkinson's disease\n", " - Neurons in the substancia nigra die off\n", " - Extremely difficult to trigger actions to start\n", " - Usually physical actions; as disease progresses and more of the SNc is gone, can get cognitive effects too\n", " - Huntington's disease\n", " - Neurons in the striatum die off\n", " - Actions are triggered inappropriately (disinhibition)\n", " - Small uncontrollable movements\n", " - Trouble sequencing cognitive actions too\n", "- Also heavily implicated in reinforcement learning\n", " - The dopamine levels seem to map onto reward prediction error\n", " - High levels when get an unexpected reward, low levels when didn't get a reward that was expected\n", " \n", "\n", "\n", "- Connectivity diagram:\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "- Old terminology:\n", " - \"direct\" pathway: cortex -> striatum -> GPi -> thalamus\n", " - \"indirect\" pathway: cortex -> striatum -> GPe -> STN -> GPi -> thalamus\n", "- Then they found:\n", " - \"hyperdirect\" pathway: cortex -> STN -> SPi -> thalamus\n", " - and lots of other connections\n", "\n", "- Activity in the GPi (output)\n", " - generally always active\n", " - neurons stop firing when corresponding action is chosen\n", " - representing [1, 1, 0, 1] instead of [0, 0, 1, 0]\n", "\n", "- Common approach (e.g. Leabra)\n", " - Each action has two groups of neurons in the striatum representing $Q(s, a_i)$ and $1-Q(s, a_i)$ (\"go\" and \"no go\")\n", " - Mutual inhibition causes only one of the \"go\" and one of the \"no go\" groups to fire\n", " - GPi neuron get connections from \"go\" neurons, with value multiplied by -1 (direct pathway)\n", " - GPi also gets connections from \"no go\" neurons, but multiplied by -1 (striatum->GPe), then -1 again (GPe->STN), then +1 (STN->GPi)\n", " - Result in GPi is close to [1, 1, 0, 1] form\n", "- Seems to match onto the biology okay\n", " - But why the weird double-inverting thing? Why not skip the GPe and STN entirely?\n", " - And why split into \"go\" and \"no-go\"? Just the direct pathway on its own would be fine\n", " - Maybe it's useful for some aspect of the learning...\n", " - What about all those other connections?\n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### An alternate model of the Basal Ganglia\n", "\n", "- Maybe the weird structure of the basal ganglia is an attempt to do action selection without doing mutual inhibition\n", "- Needs to select from a large number of actions\n", "- Needs to do so quickly, and without strong memory effects\n", "\n", "- [Gurney, Prescott, and Redgrave, 2001](http://neuroinformatics.usc.edu/mediawiki/images/3/37/Gurney_etal_01_A_computational_model_of_action_selection_in_the_basal_ganglia_-_II.pdf)\n", "\n", "- Let's start with a very simple version\n", "\n", "\n", "\n", "- Sort of like an \"unrolled\" version of one step of mutual inhibition\n", "\n", "- Now let's map that onto the basal ganglia\n", "\n", "\n", "\n", "- But that's only going to work for very specific $Q$ values.\n", "- Need to dynamically adjust the amount of +ve and -ve weighting\n", "\n", "\n", "\n", "- This turns out to work surprisingly well\n", "- But extremely hard to analyze its behaviour\n", "- They showed that it qualitatively matches pretty well\n", "\n", "- So what happens if we convert this into realistic spiking neurons?\n", "- Use the same approach where one \"neuron\" in their model is a pool of neurons in the NEF\n", "- The \"neuron model\" they use was rectified linear\n", " - That becomes the function the decoders are computing\n", "- Neurotransmitter time constant are all known\n", "- $Q$ values are between 0 and 1\n", "- Firing rates max out around 50-100Hz\n", "- Encoders are all positive and thresholds are chosen for efficiency\n", "\n" ] }, { "cell_type": "code", "collapsed": false, "input": [ "import nengo\n", "\n", "model = nengo.Network('Basal Ganglia')\n", "with model:\n", " mm = 1\n", " mp = 1\n", " me = 1\n", " mg = 1\n", " ws = 1\n", " wt = 1\n", " wm = 1\n", " wg = 1\n", " wp = 0.9\n", " we = 0.3\n", " e = 0.2\n", " ep = -0.25\n", " ee = -0.2\n", " eg = -0.2\n", " le = 0.2\n", " lg = 0.2\n", " tau_ampa = 0.002\n", " tau_gaba = 0.008\n", " N = 50\n", " D = 4\n", " model.config[nengo.Ensemble].radius = 1.5\n", " model.config[nengo.Ensemble].encoders = [[1]]*N\n", " \n", " strD1 = nengo.networks.EnsembleArray(N, D, label=\"StrD1\", \n", " intercepts=nengo.objects.Uniform(e, 1))\n", " strD2 = nengo.networks.EnsembleArray(N, D, label=\"StrD2\", \n", " intercepts=nengo.objects.Uniform(e, 1))\n", " stn = nengo.networks.EnsembleArray(N, D, label=\"STN\", \n", " intercepts=nengo.objects.Uniform(ep, 1))\n", " gpi = nengo.networks.EnsembleArray(N, D, label=\"GPi\", \n", " intercepts=nengo.objects.Uniform(eg, 1))\n", " gpe = nengo.networks.EnsembleArray(N, D, label=\"GPe\", \n", " intercepts=nengo.objects.Uniform(ee, 1))\n", "\n", " input = nengo.Node([0]*D, label=\"input\")\n", " output = nengo.Node(label=\"output\", size_in=D)\n", "\n", " # spread the input to StrD1, StrD2, and STN\n", " nengo.Connection(input, strD1.input, synapse=None,\n", " transform=ws * (1 + lg))\n", " nengo.Connection(input, strD2.input, synapse=None,\n", " transform=ws * (1 - le))\n", " nengo.Connection(input, stn.input, synapse=None,\n", " transform=wt) \n", " \n", " # connect the striatum to the GPi and GPe (inhibitory)\n", " def func_str(x):\n", " if x < e:\n", " return 0\n", " return mm * (x - e)\n", " strD1.add_output('func', func_str)\n", " import numpy\n", " nengo.Connection(strD1.func,\n", " gpi.input, synapse=tau_gaba,\n", " transform=-numpy.eye(D) * wm)\n", " strD2.add_output('func', func_str)\n", " nengo.Connection(strD2.func,\n", " gpe.input, synapse=tau_gaba,\n", " transform=-numpy.eye(D) * wm) \n", " def func_stn(x):\n", " if x < ep:\n", " return 0\n", " return mp * (x - ep)\n", " \n", " # connect the STN to GPi and GPe (broad and excitatory)\n", " tr = wp * numpy.ones((D, D))\n", " stn.add_output('func', func_stn)\n", " nengo.Connection(stn.func, gpi.input,\n", " transform=tr, synapse=tau_ampa)\n", " nengo.Connection(stn.func, gpe.input,\n", " transform=tr, synapse=tau_ampa)\n", "\n", " def func_gpe(x):\n", " if x < ee:\n", " return 0\n", " return me * (x - ee)\n", "\n", " # connect the GPe to GPi and STN (inhibitory)\n", " gpe.add_output('func', func_gpe)\n", " nengo.Connection(gpe.func, gpi.input, synapse=tau_gaba,\n", " transform=-we)\n", " nengo.Connection(gpe.func, stn.input, synapse=tau_gaba,\n", " transform=-wg)\n", "\n", " def func_gpi(x):\n", " if x < eg:\n", " return 0\n", " return mg * (x - eg)\n", " # connect GPi to output (inhibitory)\n", " gpi.add_output('func', func_gpi)\n", " nengo.Connection(gpi.func, output)\n", " \n", " nengo.Probe(output)" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "- Works pretty well\n", "- Scales up to many actions\n", "- Selects quickly\n", "- Gets behavioural match to empirical data, including timing predictions (!)\n", " - Also shows interesting oscillations not seen in the original GPR model\n", " - But these are seen in the real basal ganglia\n", "\n", "\n", "\n", "- [Dynamic Behaviour of a Spiking Model of Action Selection in the Basal Ganglia](http://compneuro.uwaterloo.ca/files/publications/stewart.2010.pdf)\n", "\n", "- Let's make sure this works with our original system" ] }, { "cell_type": "code", "collapsed": false, "input": [ "import nengo\n", "import nengo.spa as spa\n", "\n", "model = nengo.Network(label=\"BG1\")\n", "with model:\n", " state = nengo.Ensemble(500, 16, label='state')\n", " \n", " Q = nengo.networks.EnsembleArray(50, 4, label='Q')\n", "\n", " vocab = spa.Vocabulary(16) \n", " state.vocab = vocab\n", " nengo.Connection(state, Q.input, transform=[vocab.parse('DOG').v,\n", " vocab.parse('CAT').v,\n", " vocab.parse('RAT').v,\n", " vocab.parse('COW').v])\n", "\n", " nengo.Probe(state)\n", " nengo.Probe(Q.output)\n", " \n", " bg = nengo.networks.BasalGanglia(4)\n", " nengo.Connection(Q.output, bg.input)\n", " \n", " R = nengo.networks.EnsembleArray(50, 4, label='R',\n", " encoders=[[1]]*50, intercepts=nengo.objects.Uniform(0.2, 1))\n", " nengo.Connection(bg.output, R.input)\n", " nengo.Probe(R.output) \n", " \n", " import numpy\n", " bias = nengo.Node([1], label='bias')\n", " nengo.Connection(bias, R.input, transform=numpy.ones((4, 1)), synapse=None)\n", "\n", " nengo.Connection(R.output, R.input, transform=(numpy.eye(4)-1), synapse=0.008)\n" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- This system seems to work well\n", "- Still not perfect\n", "- Matches biology nicely\n", "- Some notes on the basal ganglia implementation:\n", "\n", "\n", "\n", "- Each action has a single \"neuron\" in each area that responds like this:\n", "\n", "$$\n", "y = \\begin{cases}\n", " 0 &\\mbox{if } x < \\epsilon \\\\ \n", " m(x- \\epsilon) &\\mbox{otherwise} \n", " \\end{cases}\n", "$$\n", "\n", "- These need to get turned into groups of neurons\n", " - What is the best way to do this?\n", "\n", "\n", "\n", "- encoders are all +1\n", "- intercepts are chosen to be $> \\epsilon$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Action Execution\n", "\n", "- Now that we can select an action, how do we perform it?\n", "- Depends on what the action is\n", "- Let's start with simple actions\n", " - Move in a given direction\n", " - Remember a specific vector\n", " - Send a particular value as input into a particular cognitive system\n", "\n", "- Example:\n", " - State $s$ is 16-dimensional\n", " - Four actions (A, B, C, D)\n", " - Do action A if $s$ is near `CAT`, B if near `DOG`, C if near `RAT`, D if near `COW`\n", " - $Q(s, a_A)=s \\cdot$ `CAT`\n", " - $Q(s, a_B)=s \\cdot$ `DOG`\n", " - $Q(s, a_C)=s \\cdot$ `RAT`\n", " - $Q(s, a_D)=s \\cdot$ `COW`\n", " - To do Action A, set $m=$ `MEOW` \n", " - To do Action B, set $m=$ `DOG`\n", " - To do Action C, set $m=$ `SQUEAK`\n", " - To do Action D, set $m=$ `MOO`\n", " " ] }, { "cell_type": "code", "collapsed": false, "input": [ "import nengo\n", "import nengo.spa as spa\n", "\n", "model = nengo.Network(label=\"BG4\")\n", "with model:\n", " state = nengo.Ensemble(500, 16, label='state')\n", " \n", " Q = nengo.networks.EnsembleArray(50, 4, label='Q')\n", "\n", " vocab = spa.Vocabulary(16) \n", " state.vocab = vocab\n", " nengo.Connection(state, Q.input, transform=[vocab.parse('DOG').v,\n", " vocab.parse('CAT').v,\n", " vocab.parse('RAT').v,\n", " vocab.parse('COW').v])\n", "\n", " nengo.Probe(state)\n", " nengo.Probe(Q.output)\n", " \n", " bg = nengo.networks.BasalGanglia(4)\n", " nengo.Connection(Q.output, bg.input)\n", " \n", " R = nengo.networks.EnsembleArray(50, 4, label='R',\n", " encoders=[[1]]*50, intercepts=nengo.objects.Uniform(0.2, 1))\n", " nengo.Connection(bg.output, R.input)\n", " nengo.Probe(R.output) \n", " \n", " import numpy\n", " bias = nengo.Node([1], label='bias')\n", " nengo.Connection(bias, R.input, transform=numpy.ones((4, 1)), synapse=None)\n", "\n", " # mutual inhibition on the actions\n", " nengo.Connection(R.output, R.input, transform=(numpy.eye(4)-1), synapse=0.008)\n", " \n", " motor = nengo.Ensemble(500, 16, label='motor')\n", " \n", " nengo.Connection(R.output, motor, \n", " transform=numpy.array([vocab.parse('BARK').v,\n", " vocab.parse('MEOW').v,\n", " vocab.parse('SQUEAK').v,\n", " vocab.parse('MOO').v]).T)\n", " nengo.Probe(motor)\n" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- What about more complex actions?\n", "- Consider the creature we were making in class\n", " - Action 1: set $m$ to the direction we're told to do\n", " - Action 2: set $m$ to the direction we started from\n", "- Need to pass information from one group of neurons to another\n", " - But only do this when the action is chosen\n", " - How?\n", "- Well, let's use a function\n", " - $m = a \\times d$\n", " - where $a$ is the action selection (0 for not selected, 1 for selected)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- There's also another way to do this\n", "- A special case for forcing a function to go to zero when a particular group of neurons is active\n", "\n", "- Build a communication channel with an intermediate group of neurons\n", " - When the action is not selected, inhibit that intermediate group of neurons\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- This is a situation where it makes sense to ignore the NEF!\n", " - All we want to do is shut down the neural activity\n", " - So just do a very inhibitory connection" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The Cortex-Basal Ganglia-Thalamus loop\n", "\n", "- We now have everything we need for a model of one of the primary structures in the mammalian brain\n", " - Basal ganglia: action selection\n", " - Thalamus: action execution\n", " - Cortex: everything else\n", " \n", "- We build systems in cortex that give some input-output functionality\n", " - We set up the basal ganglia and thalamus to make use of that functionality appropriately\n", " \n", "- This sort of model is going to get complicated, so we've added a wrapper to help build everything:\n", " " ] }, { "cell_type": "code", "collapsed": false, "input": [ "import nengo\n", "import nengo.spa as spa\n", "\n", "model = spa.SPA(label=\"SPA1\")\n", "with model:\n", " model.state = spa.Buffer(16)\n", " model.motor = spa.Buffer(16)\n", " actions = spa.Actions(\n", " 'dot(state, DOG) --> motor=BARK',\n", " 'dot(state, CAT) --> motor=MEOW',\n", " 'dot(state, RAT) --> motor=SQUEAK',\n", " 'dot(state, COW) --> motor=MOO',\n", " ) \n", " model.bg = spa.BasalGanglia(actions)\n", " model.thalamus = spa.Thalamus(model.bg)\n", " \n", " nengo.Probe(model.state.state.output)\n", " nengo.Probe(model.motor.state.output)" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Another Example\n", " - Cortex stores some state (integrator)\n", " - Add some state transition rules\n", " - If in state A, go to state B\n", " - If in state B, go to state C\n", " - If in state C, go to state D\n", " - ...\n", " - $Q(s, a_i) = s \\cdot a_i$\n", " - The effect of each action is to input the corresponding vector into the integrator\n", " " ] }, { "cell_type": "code", "collapsed": false, "input": [ "import nengo\n", "import nengo.spa as spa\n", "\n", "model = spa.SPA(label=\"SPA1\")\n", "with model:\n", " model.state = spa.Buffer(16)\n", " actions = spa.Actions(\n", " 'dot(state, A) --> state=B',\n", " 'dot(state, B) --> state=C',\n", " 'dot(state, C) --> state=D',\n", " 'dot(state, D) --> state=E',\n", " 'dot(state, E) --> state=A',\n", " ) \n", " model.bg = spa.BasalGanglia(actions)\n", " model.thalamus = spa.Thalamus(model.bg)\n", " \n", " nengo.Probe(model.state.state.output)\n" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Can also provide sort of input to start things off" ] }, { "cell_type": "code", "collapsed": false, "input": [ " nengo.Probe(model.state.state.output)\n" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- But that's all using the simple actions\n", "- What about an action that involves taking information from one neural system and sending it to another?\n", "- Let's have a separate visual state and use if to put information into the changing state" ] }, { "cell_type": "code", "collapsed": false, "input": [ "import nengo\n", "import nengo.spa as spa\n", "\n", "model = spa.SPA(label=\"SPA1\")\n", "with model:\n", " model.state = spa.Buffer(16)\n", " model.vision = spa.Buffer(16)\n", " actions = spa.Actions(\n", " 'dot(vision, LETTER) --> state=vision',\n", " 'dot(state, A) --> state=B',\n", " 'dot(state, B) --> state=C',\n", " 'dot(state, C) --> state=D',\n", " 'dot(state, D) --> state=E',\n", " 'dot(state, E) --> state=A',\n", " ) \n", " model.bg = spa.BasalGanglia(actions)\n", " model.thalamus = spa.Thalamus(model.bg)\n", " \n", " def my_input(t):\n", " if t < 0.1:\n", " return 'LETTER+D'\n", " else:\n", " return '0'\n", " model.input = spa.Input(vision=my_input)\n", " \n", " nengo.Probe(model.state.state.output)" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Behavioural Evidence\n", "\n", "- So this lets us build more complex models\n", "- Is there any evidence that this is the way it works in brains?\n", " - Consistent with anatomy/connectivity\n", "- What about behavioural?\n", "- Sort of\n", "- Timing data\n", " - How long does it take to do an action?\n", " - There are lots of existing computational (non-neural) cognitive models that have something like this action selection loop\n", " - Usually all-symbolic\n", " - A set of IF-THEN rules\n", " - e.g. [ACT-R](http://act-r.psy.cmu.edu/)\n", " - Used to model mental arithmetic, driving a car, using a GUI, air-traffic control, staffing a battleship, etc etc\n", " - Best fit across all these situations is to set the loop time to 50ms\n", "- how long does this model take?\n", " - Notice that all the timing is based on neural properties, not the algorithm\n", " - Dominated by the longer neurotransmitter time constants in the basal ganglia\n", "\n", "\n", "\n", "\n", "\n", "- This is in the right ballpark\n", "- But what about this distinction between the two types of actions?\n", " - Not a distinction made in the literature\n", " - But once we start looking for it, lots of evidence\n", " - [Resolves an outstanding weirdness where some actions seem to take twice as long as others](http://www.cogsci.northwestern.edu/cogsci2004/papers/paper425.pdf)\n", " - Starting to be lots of citations for 40ms for simple tasks \n", " - [Task artifacts and strategic adaptation in the change signal task](http://act-r.psy.cmu.edu/wordpress/wp-content/uploads/2014/01/2013-moore_gunzelmann-CSR.pdf)\n", "- This is a nice example of the usefulness of making neural models!\n", " - This distinction wasn't obvious from computational implementations\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### More complex tasks\n", "\n", "- Lots of complex tasks can be modelled this way\n", " - Some basic cognitive components (cortex)\n", " - action selection system (basal ganglia and thalamus)\n", "- The tricky part is figuring out the actions\n", "- Example: the Tower of Hanoi task\n", " - 3 pegs\n", " - N disks of different sizes on the pegs\n", " - move from one configuration to another\n", " - can only move one disk at a time\n", " - no larger disk can be on a smaller disk\n", " \n", "\n", "\n", "- can we build rules to do this?\n", "\n", "\n", "\n", "\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- How do people do this task?\n", " - Studied extensively by cognitive scientists\n", " - Simon (1975):\n", " 1. Find the largest disk not in its goal position and make the goal to get it in that position. This is the initial \u201cgoal move\u201d for purposes of the next two steps. If all disks are in their goal positions, the problem is solved\n", " 2. If there are any disks blocking the goal move, find the largest blocking disk (either on top of the disk to be moved or at the destination peg) and make the new goal move to move this blocking disk to the other peg (i.e., the peg that is neither the source nor destination of this disk). The previous goal move is stored as the parent goal of the new goal move. Repeat this step with the new goal move.\n", " 3. If there are no disks blocking the goal move perform the goal move and (a) If the goal move had a parent goal retrieve that parent goal, make it the goal move, and go back to step 2. (b) If the goal had no parent goal, go back to step 1." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- What do the actions look like?\n", "- State:\n", " - `goal`: what disk am I trying to move (D0, D1, D2)\n", " - `focus`: what disk am I looking at (D0, D1, D2)\n", " - `goal_peg`: where is the disk I am trying to move (A, B, C)\n", " - `focus_peg`: where is the disk I am looking at (A, B, C)\n", " - `target_peg`: where am I trying to move a disk to (A, B, C)\n", " - `goal_final`: what is the overall final desired location of the disk I'm trying to move (A, B, C)\n", "\n", "- Note: we're not yet modelling all the sensory and memory stuff here, so we manually set things like `goal_final`.\n", "- Action effects: when an action is selected, it could do the following\n", " - set `focus`\n", " - set `goal`\n", " - set `goal_peg`\n", " - actually try to move a disk to a given location by setting `move` and `move_peg`\n", " - Note: we're also not modelling the full motor system, so we fake this too\n", " \n", "- Is this sufficient to implement the algorithm described above?\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- What do the action rules look like?\n", " - if `focus`=NONE then `focus`=D2, `goal`=D2, `goal_peg`=`goal_final`\n", " - $Q$=`focus` $\\cdot$ NONE \n", " - if `focus`=D2 and `goal`=D2 and `goal_peg`!=`target_peg` then `focus`=D1\n", " - $Q$=`focus` $\\cdot$ D2 + `goal` $\\cdot$ D2 - `goal_peg` $\\cdot$ `target_peg`\n", " - if `focus`=D2 and `goal`=D2 and `goal_peg`==`target_peg` then `focus`=D1, `goal`=D1, `goal_peg`=`goal_final`\n", " - if `focus`=D1 and `goal`=D1 and `goal_peg`!=`target_peg` then `focus`=D0\n", " - if `focus`=D1 and `goal`=D1 and `goal_peg`==`target_peg` then `focus`=D0, `goal`=D0, `goal_peg`=`goal_final`\n", " - if `focus`=D0 and `goal_peg`==`target_peg` then `focus`=NONE\n", " - if `focus`=D0 and `goal`=D0 and `goal_peg`!=`target_peg` then `focus`=NONE, `move`=D0, `move_peg`=`target_peg`\n", " - if `focus`!=`goal` and `focus_peg`==`goal_peg` and `target_peg!=focus_peg` then `goal`=`focus`, `goal_peg`=A+B+C-`target_peg`-`focus_peg`\n", " - trying to move something, but smaller disk is on top of this one\n", " - if `focus`!=`goal` and `focus_peg`!=`goal_peg` and `target_peg==focus_peg` then `goal`=`focus`, `goal_peg`=A+B+C-`target_peg`-`goal_peg`\n", " - trying to move something, but smaller disk is on top of target peg\n", " - if `focus`=D0 and `goal`!=D0 and `target_peg`!=`focus_peg` and `target_peg`!=`goal_peg` and `focus_peg`!=`goal_peg` then `move`=`goal`, `move_peg`=`target_peg`\n", " - move the disk, since there's nothing in the way\n", " - if `focus`=D1 and `goal`!=D1 and `target_peg`!=`focus_peg` and `target_peg`!=`goal_peg` and `focus_peg`!=`goal_peg` then `focus`=D0\n", " - check the next disk\n", " \n", " \n", " \n", " \n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Sufficient to solve any version of the problem\n", "- Is it what people do?\n", "- How can we tell?\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Do science\n", " - What predictions does the theory make\n", " - Errors?\n", " - Reaction times?\n", " - Neural activity?\n", " - fMRI?\n", "\n", "- Timing:\n", "\n", "\n", "\n", "- not quite\n", "- much longer pauses in some situations than there should be\n", "- At those stages, it is recomputing plans that it has made previously\n", " - People probably remember those, and don't restart from scratch\n", " - Need to add that into the model\n", "- [Neural Cognitive Modelling: A Biologically Constrained Spiking Neuron Model of the Tower of Hanoi Task](http://compneuro.uwaterloo.ca/files/publications/stewart.2011a.pdf)" ] } ], "metadata": {} } ] }