{
"metadata": {
"name": ""
},
"nbformat": 3,
"nbformat_minor": 0,
"worksheets": [
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# SYDE 556/750: Simulating Neurobiological Systems\n",
"\n",
"Terry Stewart\n",
"\n",
"## Online Learning\n",
"\n",
"- What do we mean by learning?\n",
" - When we use an integrator to keep track of location, is that learning?\n",
" - Probably not\n",
" - What about the learning used to complete a pattern in the Raven's Progressive Matrices task?\n",
" - Less clear\n",
"- We'll stick with a simple definition of learning\n",
" - Changing connection weights between groups of neurons"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- Why might we want to change connection weights?\n",
"- This is what traditional neural network approaches do\n",
" - Change connection weights until it performs the desired task\n",
" - Once it's doing the task, stop changing the weights\n",
"- But we have a method for just solving for the optimal connection weights\n",
" - So why bother learning?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Why learning might be useful\n",
"\n",
"- We might not know the function at the beginning of the task\n",
" - Example: a creature explores its environment and learns that eating red objects is bad, but eating green objects is good\n",
" - what are the inputs and outputs here?\n",
"- The desired function might change\n",
" - Example: an ensemble whose input is a desired hand position, but the output is the muscle tension (or joint angles) needed to get there\n",
" - why would this change?\n",
"- The optimal weights we solve for might not be optimal\n",
" - How could they not be optimal?\n",
" - What assumptions are we making?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### The simplest approach\n",
"\n",
"- What's the easiest way to deal with this, given what we know?\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- If we need new decoders\n",
" - Let's solve for them while the model's running\n",
" - Gather data to build up our $\\Gamma$ and $\\Upsilon$ matrices\n",
"- Example: eating red but not green objects\n",
" - Decoder from state to $Q$ value (utility of action) for eating\n",
" - State is some high-dimensional vector that includes the colour of what we're looking for\n",
" - And probably some other things, like whether it's small enough to be eaten\n",
" - Initially doesn't use colour to get output\n",
" - But we might experience a few bad outcomes after red, and good after green\n",
" - These become new $x$ samples, with corresponding $f(x)$ outputs\n",
" - Gather a few, recompute decoder\n",
" - Could even do this after every timestep\n",
"- Example: converting hand position to muscle commands\n",
" - Send random signals to muscles\n",
" - Observe hand position\n",
" - Use that to train decoders\n",
"- Example: going from optimal to even more optimal\n",
" - As the model runs, we gather $x$ values \n",
" - Recompute decoder for those $x$ values\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### What's wrong with this approach\n",
"\n",
"- Feels like cheating\n",
"- Why?\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- Two kinds of problems:\n",
" - Not biologically realistic\n",
" - How are neurons supposed to do all this?\n",
" - store data\n",
" - solve decoders\n",
" - timing\n",
" - Computationally expensive\n",
" - Even if we're not worried about realism\n",
"- Note: these may be related points...."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Traditional neural networks\n",
"\n",
"- What do they do?\n",
"- Incremental learning\n",
" - as you get examples, shift the connection weights slightly based on that example\n",
" - don't have to consider all the data when making an update\n",
"- Example: Perceptron learning (1957)\n",
" - $\\Delta w_j = \\alpha(y_d - y)x_i$\n",
"\n",
"\n",
"\n",
"- Problems with perceptron\n",
" - can't do all possible functions\n",
" - Just linear functions of $x$\n",
" - Is that a problem?\n",
" \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Backprop and the NEF\n",
"\n",
"- How is this problem normally solved?\n",
" - Multiple layers\n",
" \n",
"\n",
" \n",
"- But now a new rule is needed\n",
" - Standard answer: backprop\n",
" - Same as perceptron for first layer\n",
" - Estimate correct \"hidden layer\" input, and repeat\n",
"- What would this be in NEF terms?\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- Remember that we're already fine with linear decoding\n",
" - encoders (and $\\alpha$ and $J^{bias}$) are first layer of weights, decoders are second layer\n",
" - Note that in the NEF, we combine many of these together\n",
"- We can just use the standard perceptron rule\n",
" - as long as there's lots of neurons, and we've initialized them well with the desired intercepts and maximum rates, we should be able to decode\n",
" - but, what might backprop do?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Biologically realistic perceptron learning\n",
"\n",
"- Simple learning rule: $\\Delta d_i = \\kappa (y_d - y)a_i$\n",
"- How do we make it realistic?\n",
"- Decoders don't exist in the brain\n",
" - Need weights\n",
"- $\\omega_{ij} = \\alpha_j d_i \\cdot e_j$\n",
"- $\\Delta \\omega_{ij} = \\alpha_j \\kappa (y_d - y)a_i \\cdot e_j$\n",
"- let's write $(y_d - y)$ as $E$\n",
"- $\\Delta \\omega_{ij} = \\alpha_j \\kappa a_i E \\cdot e_j$\n",
"- $\\Delta \\omega_{ij} = \\kappa a_i (\\alpha_j E \\cdot e_j)$\n",
" - What's $\\alpha_j E \\cdot e_j$\n",
" - That's the current that this neuron would get if it had $E$ as an input\n",
" - but we don't want this current to drive the neuron\n",
" - rather, we want it to change the weight\n",
" - a *modulatory* input\n",
"- This is the \"Prescribed Error Sensitivity\" PES rule (MacNeil & Eliasmith, 2011) \n",
" - Any model in the NEF could use this instead of computing decoders\n",
" - Requires some other neural group computing the error $E$\n",
" - Used in Spaun for Q-value learning (reinforcement task)\n",
" - Can even be used to learn circular convolution\n",
" - only demonstrated up to 3 dimensions\n",
" - why not more?\n",
" \n",
"- Nengo Examples:\n",
" - learn_communicate.py\n",
" - learn_square.py\n",
" - learn_product.py\n",
" \n",
"- Is this realistic?\n",
" - local information\n",
" - Does it look like anything like this happens in the brain?\n",
" - Dopamine seems to act as a gain on weight changes (maybe)\n",
" - But are weight changes proportional to pre-synaptic activity?\n",
" - Sort of\n",
" \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### More complex learning\n",
"\n",
"- Hebbian learning\n",
" - completely unsupervised\n",
" - Neurons that fire together, wire together\n",
" - $\\Delta \\omega_{ij} = \\kappa a_i a_j$\n",
" - just that would be unstable\n",
" - Why?\n",
"\n",
"- BCM rule (Bienenstock, Cooper, & Munro, 1982)\n",
" - $\\Delta \\omega_{ij} = \\kappa a_i a_j (a_j-\\theta)$\n",
" - $\\theta$ is an activity threshold\n",
" - if post-synaptic neuron is more active than this threshold, increase strength\n",
" - otherwise decrease it\n",
" - Other than that, it's a standard Hebbian rule\n",
" - Where would we get $\\theta$?\n",
" - need to store something about the overall recent activity of neuron $j$ so it can be compared to its current activity\n",
" - Just have $\\theta$ be a pstc-filtered spiking of $a_j$\n",
" - Result: only a few neurons will fire\n",
" - sparsification\n",
" - What would this do in NEF terms? \n",
" - Still represent $x$, but with very sparse encoders\n",
" - This is still a rule on the weight matrix, but functionally seems to be more about encoders than decoders\n",
" - What could we do, given that? "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The homeostatic Prescribed Error Sensitivity (hPES) rule\n",
"\n",
"- just do them both\n",
"- and have a parameter $S$ to adjust how much of each\n",
"- $\\Delta \\omega_{ij} = \\kappa \\alpha_j a_j (S e_j \\cdot E + (1-S) a_j (a_j-\\theta))$\n",
"\n",
"- http://mindmodeling.org/cogsci2013/papers/0058/paper0058.pdf\n",
"\n",
"\n",
"- Works as well (or better) than PES\n",
" - Seems to be a bit more stable, but analysis is ongoing\n",
"- Biological evidence?\n",
" - Spike-Timing Dependent Plasticity\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"- Still work to do for comparison, but seems promising\n",
"- Error-driven for improving decoders\n",
"- Hebbian sparsification to improve encoders\n",
" - or perhaps to sparsify connections (energy savings in the brain, but not necessarily in simulation)\n",
" \n"
]
}
],
"metadata": {}
}
]
}