{ "metadata": { "name": "" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# SYDE 556/750: Simulating Neurobiological Systems\n", "\n", "Terry Stewart\n", "\n", "## Online Learning\n", "\n", "- What do we mean by learning?\n", " - When we use an integrator to keep track of location, is that learning?\n", " - Probably not\n", " - What about the learning used to complete a pattern in the Raven's Progressive Matrices task?\n", " - Less clear\n", "- We'll stick with a simple definition of learning\n", " - Changing connection weights between groups of neurons" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Why might we want to change connection weights?\n", "- This is what traditional neural network approaches do\n", " - Change connection weights until it performs the desired task\n", " - Once it's doing the task, stop changing the weights\n", "- But we have a method for just solving for the optimal connection weights\n", " - So why bother learning?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Why learning might be useful\n", "\n", "- We might not know the function at the beginning of the task\n", " - Example: a creature explores its environment and learns that eating red objects is bad, but eating green objects is good\n", " - what are the inputs and outputs here?\n", "- The desired function might change\n", " - Example: an ensemble whose input is a desired hand position, but the output is the muscle tension (or joint angles) needed to get there\n", " - why would this change?\n", "- The optimal weights we solve for might not be optimal\n", " - How could they not be optimal?\n", " - What assumptions are we making?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The simplest approach\n", "\n", "- What's the easiest way to deal with this, given what we know?\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- If we need new decoders\n", " - Let's solve for them while the model's running\n", " - Gather data to build up our $\\Gamma$ and $\\Upsilon$ matrices\n", "- Example: eating red but not green objects\n", " - Decoder from state to $Q$ value (utility of action) for eating\n", " - State is some high-dimensional vector that includes the colour of what we're looking for\n", " - And probably some other things, like whether it's small enough to be eaten\n", " - Initially doesn't use colour to get output\n", " - But we might experience a few bad outcomes after red, and good after green\n", " - These become new $x$ samples, with corresponding $f(x)$ outputs\n", " - Gather a few, recompute decoder\n", " - Could even do this after every timestep\n", "- Example: converting hand position to muscle commands\n", " - Send random signals to muscles\n", " - Observe hand position\n", " - Use that to train decoders\n", "- Example: going from optimal to even more optimal\n", " - As the model runs, we gather $x$ values \n", " - Recompute decoder for those $x$ values\n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### What's wrong with this approach\n", "\n", "- Feels like cheating\n", "- Why?\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Two kinds of problems:\n", " - Not biologically realistic\n", " - How are neurons supposed to do all this?\n", " - store data\n", " - solve decoders\n", " - timing\n", " - Computationally expensive\n", " - Even if we're not worried about realism\n", "- Note: these may be related points...." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Traditional neural networks\n", "\n", "- What do they do?\n", "- Incremental learning\n", " - as you get examples, shift the connection weights slightly based on that example\n", " - don't have to consider all the data when making an update\n", "- Example: Perceptron learning (1957)\n", " - $\\Delta w_j = \\alpha(y_d - y)x_i$\n", "\n", "\n", "\n", "- Problems with perceptron\n", " - can't do all possible functions\n", " - Just linear functions of $x$\n", " - Is that a problem?\n", " \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Backprop and the NEF\n", "\n", "- How is this problem normally solved?\n", " - Multiple layers\n", " \n", "\n", " \n", "- But now a new rule is needed\n", " - Standard answer: backprop\n", " - Same as perceptron for first layer\n", " - Estimate correct \"hidden layer\" input, and repeat\n", "- What would this be in NEF terms?\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Remember that we're already fine with linear decoding\n", " - encoders (and $\\alpha$ and $J^{bias}$) are first layer of weights, decoders are second layer\n", " - Note that in the NEF, we combine many of these together\n", "- We can just use the standard perceptron rule\n", " - as long as there's lots of neurons, and we've initialized them well with the desired intercepts and maximum rates, we should be able to decode\n", " - but, what might backprop do?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Biologically realistic perceptron learning\n", "\n", "- Simple learning rule: $\\Delta d_i = \\kappa (y_d - y)a_i$\n", "- How do we make it realistic?\n", "- Decoders don't exist in the brain\n", " - Need weights\n", "- $\\omega_{ij} = \\alpha_j d_i \\cdot e_j$\n", "- $\\Delta \\omega_{ij} = \\alpha_j \\kappa (y_d - y)a_i \\cdot e_j$\n", "- let's write $(y_d - y)$ as $E$\n", "- $\\Delta \\omega_{ij} = \\alpha_j \\kappa a_i E \\cdot e_j$\n", "- $\\Delta \\omega_{ij} = \\kappa a_i (\\alpha_j E \\cdot e_j)$\n", " - What's $\\alpha_j E \\cdot e_j$\n", " - That's the current that this neuron would get if it had $E$ as an input\n", " - but we don't want this current to drive the neuron\n", " - rather, we want it to change the weight\n", " - a *modulatory* input\n", "- This is the \"Prescribed Error Sensitivity\" PES rule (MacNeil & Eliasmith, 2011) \n", " - Any model in the NEF could use this instead of computing decoders\n", " - Requires some other neural group computing the error $E$\n", " - Used in Spaun for Q-value learning (reinforcement task)\n", " - Can even be used to learn circular convolution\n", " - only demonstrated up to 3 dimensions\n", " - why not more?\n", " \n", "- Nengo Examples:\n", " - learn_communicate.py\n", " - learn_square.py\n", " - learn_product.py\n", " \n", "- Is this realistic?\n", " - local information\n", " - Does it look like anything like this happens in the brain?\n", " - Dopamine seems to act as a gain on weight changes (maybe)\n", " - But are weight changes proportional to pre-synaptic activity?\n", " - Sort of\n", " \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### More complex learning\n", "\n", "- Hebbian learning\n", " - completely unsupervised\n", " - Neurons that fire together, wire together\n", " - $\\Delta \\omega_{ij} = \\kappa a_i a_j$\n", " - just that would be unstable\n", " - Why?\n", "\n", "- BCM rule (Bienenstock, Cooper, & Munro, 1982)\n", " - $\\Delta \\omega_{ij} = \\kappa a_i a_j (a_j-\\theta)$\n", " - $\\theta$ is an activity threshold\n", " - if post-synaptic neuron is more active than this threshold, increase strength\n", " - otherwise decrease it\n", " - Other than that, it's a standard Hebbian rule\n", " - Where would we get $\\theta$?\n", " - need to store something about the overall recent activity of neuron $j$ so it can be compared to its current activity\n", " - Just have $\\theta$ be a pstc-filtered spiking of $a_j$\n", " - Result: only a few neurons will fire\n", " - sparsification\n", " - What would this do in NEF terms? \n", " - Still represent $x$, but with very sparse encoders\n", " - This is still a rule on the weight matrix, but functionally seems to be more about encoders than decoders\n", " - What could we do, given that? " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The homeostatic Prescribed Error Sensitivity (hPES) rule\n", "\n", "- just do them both\n", "- and have a parameter $S$ to adjust how much of each\n", "- $\\Delta \\omega_{ij} = \\kappa \\alpha_j a_j (S e_j \\cdot E + (1-S) a_j (a_j-\\theta))$\n", "\n", "- http://mindmodeling.org/cogsci2013/papers/0058/paper0058.pdf\n", "\n", "\n", "- Works as well (or better) than PES\n", " - Seems to be a bit more stable, but analysis is ongoing\n", "- Biological evidence?\n", " - Spike-Timing Dependent Plasticity\n", "\n", "\n", "\n", "\n", "\n", "- Still work to do for comparison, but seems promising\n", "- Error-driven for improving decoders\n", "- Hebbian sparsification to improve encoders\n", " - or perhaps to sparsify connections (energy savings in the brain, but not necessarily in simulation)\n", " \n" ] } ], "metadata": {} } ] }