\n",
" ![]()
\n",
"
\n",
"\n",
"\n",
"\n",
" \n",
" \n",
" \n",
"
\n",
"![](\"../.images/demos/SF_development.gif\")
\n",
" \n",
"In this tutorial will will show how `RatInABox`, in particular the `SuccessorFeatures` class can be used to model successor features (also called \"successor representations\"). \n",
"\n",
"Successor representations encode the expected future occupancy of a given location. As pointed out by Stachenfeld et al. (2017) such representations bear high resemblence to place cells in the hippocampal formation. A natural extension of successor representations is to successor _features_ $\\{\\psi_i\\}$: these encode the expected future activity of a given feature $\\{\\phi_i\\}$. If a feature is, for example, a place cell (encoding location) then the successor feature will be a successor representations for the location encoded by that place cells but in general succeessor features can encode anything. \n",
"\n",
"\n",
"$$\\hat{\\psi_i}(t) = \\sigma^{\\textrm{non-lin}}\\bigg(\\sum_j w_{ij} f_j(t)\\bigg) \\approx \\psi_i^{\\pi}(t) = \\mathbb{E} \\bigg[ \\int_{t}^{\\infty} e^{-\\frac{t^{\\prime}-t}{\\tau}} \\phi_i(t^{\\prime}) dt^{\\prime}\\bigg] $$\n",
"\n",
"Given successor features are a subset of value functions (a successor feature _is_ a value function where the reward is the feature activity) `SuccessorFeatures` are a subclass of `ValueNeuron`s (for more about this class see `reinforment_learning_example.ipynb`). The only substantial difference is that `SuccessorFeatures` insist on you passing in at initialisation a set of features which are then used on each `update_weights()` step as the multidimensional reward signal. Temporal difference learning is handled by the `ValueNeuron` parent class. "
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: ratinabox in /Users/tomgeorge/miniconda3/envs/riab/lib/python3.10/site-packages (1.10.1)\n",
"Requirement already satisfied: scipy in /Users/tomgeorge/miniconda3/envs/riab/lib/python3.10/site-packages (from ratinabox) (1.9.3)\n",
"Requirement already satisfied: matplotlib in /Users/tomgeorge/miniconda3/envs/riab/lib/python3.10/site-packages (from ratinabox) (3.5.3)\n",
"Requirement already satisfied: numpy in /Users/tomgeorge/miniconda3/envs/riab/lib/python3.10/site-packages (from ratinabox) (1.24.2)\n",
"Requirement already satisfied: shapely in /Users/tomgeorge/miniconda3/envs/riab/lib/python3.10/site-packages (from ratinabox) (2.0.1)\n",
"Requirement already satisfied: python-dateutil>=2.7 in /Users/tomgeorge/miniconda3/envs/riab/lib/python3.10/site-packages (from matplotlib->ratinabox) (2.8.2)\n",
"Requirement already satisfied: fonttools>=4.22.0 in /Users/tomgeorge/miniconda3/envs/riab/lib/python3.10/site-packages (from matplotlib->ratinabox) (4.38.0)\n",
"Requirement already satisfied: kiwisolver>=1.0.1 in /Users/tomgeorge/miniconda3/envs/riab/lib/python3.10/site-packages (from matplotlib->ratinabox) (1.4.4)\n",
"Requirement already satisfied: pyparsing>=2.2.1 in /Users/tomgeorge/miniconda3/envs/riab/lib/python3.10/site-packages (from matplotlib->ratinabox) (3.0.9)\n",
"Requirement already satisfied: packaging>=20.0 in /Users/tomgeorge/miniconda3/envs/riab/lib/python3.10/site-packages (from matplotlib->ratinabox) (23.0)\n",
"Requirement already satisfied: cycler>=0.10 in /Users/tomgeorge/miniconda3/envs/riab/lib/python3.10/site-packages (from matplotlib->ratinabox) (0.11.0)\n",
"Requirement already satisfied: pillow>=6.2.0 in /Users/tomgeorge/miniconda3/envs/riab/lib/python3.10/site-packages (from matplotlib->ratinabox) (9.4.0)\n",
"Requirement already satisfied: six>=1.5 in /Users/tomgeorge/miniconda3/envs/riab/lib/python3.10/site-packages (from python-dateutil>=2.7->matplotlib->ratinabox) (1.16.0)\n",
"Requirement already satisfied: tqdm in /Users/tomgeorge/miniconda3/envs/riab/lib/python3.10/site-packages (4.62.3)\n"
]
}
],
"source": [
"!pip install ratinabox\n",
"!pip install tqdm"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"from ratinabox.Environment import Environment\n",
"from ratinabox.Agent import Agent\n",
"from ratinabox.Neurons import *\n",
"from ratinabox.contribs.SuccessorFeatures import SuccessorFeatures\n",
"from tqdm.auto import tqdm # progress bar\n",
"ratinabox.stylize_plots(); ratinabox.autosave_plots=True; ratinabox.figure_directory = \"../figures/\"\n",
"\n",
"%load_ext autoreload\n",
"%autoreload 2"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Lets remind ourselves of the default params for successor features"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'n': 1,\n",
" 'name': 'FeedForwardLayer',\n",
" 'color': None,\n",
" 'noise_std': 0,\n",
" 'noise_coherence_time': 0.5,\n",
" 'min_fr': 0.0,\n",
" 'max_fr': 1.0,\n",
" 'save_history': True,\n",
" 'input_layers': [],\n",
" 'activation_function': {'activation': 'relu'},\n",
" 'biases': None,\n",
" 'tau': 2,\n",
" 'tau_e': None,\n",
" 'eta': 0.001,\n",
" 'L2': 0.001,\n",
" 'features': None}"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"SuccessorFeatures.get_all_default_params()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Open field foraging\n",
"\n",
"100 small place cells are initialised in a 1 x 1 m environment containing a wall. SuccessorFeatures wuth a predictive time horizon of 10 seconds are trained on these features. (The bassi set is taken to be the place cells as well, this double use is optional and alternatively a seperate set of neurons caould be used as the basis features i.e. `params['input_layers'])"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"Env = Environment()\n",
"Env.add_wall([[0.5,0],[0.5,0.5]])\n",
"Ag = Agent(Env,params={'dt':0.1})\n",
"features = PlaceCells(Ag, params={\"n\": 100, \"widths\": 0.1}) \n",
"SFs = SuccessorFeatures(Ag, params={\"features\": features,\n",
" \"input_layers\": [features],\n",
" 'tau':10, #preditive time horizon\n",
" 'eta':0.01,\n",
" })\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Train for 1000 seconds"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████| 10000/10000 [00:20<00:00, 491.60it/s]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Figure saved to /Users/tomgeorge/Documents/RatInABox/figures/03_11_23/trajectory_2118.png & .svg\n"
]
},
{
"data": {
"text/plain": [
"(