{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Experiments\n",
"\n",
"This notebook documents the use of the Experiment class for running many experiments, and comparing the results."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Using TensorFlow backend.\n",
"ConX, version 3.7.5\n"
]
}
],
"source": [
"import conx as cx"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First, we create an experiment object:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"exp = cx.Experiment(\"XOR-Test1\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Experiment takes a function, and any number of variations of values.\n",
"\n",
"The function takes whatever argumnets you wish, creates a network, trains it, and returns the network and a category describing the run:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"def function(optimizer, activation, **options):\n",
" category = \"%s-%s\" % (optimizer, activation)\n",
" print(\"category %s running...\" % category)\n",
" net = cx.Network(\"XOR\", 2, 2, 1, activation=activation)\n",
" net.compile(error=\"mse\", optimizer=optimizer)\n",
" net.dataset.append_by_function(2, (0, 4), \"binary\", lambda i,v: [int(sum(v) == len(v))])\n",
" net.train(report_rate=10000, verbose=0, plot=False, **options)\n",
" return category, net"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then we run a number of trials (perhaps just 1). Notice that each argument should be given as a list. The total number of runs per trial is the product of the lengths of the arguments."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"category adam-sigmoid running...\n",
"category sgd-sigmoid running...\n",
"category adam-relu running...\n",
"category sgd-relu running...\n",
"category adam-sigmoid running...\n",
"category sgd-sigmoid running...\n",
"category adam-relu running...\n",
"category sgd-relu running...\n"
]
}
],
"source": [
"exp.run(function,\n",
" trials=2,\n",
" epochs=[1000],\n",
" accuracy=[0.8],\n",
" tolerance=[0.2],\n",
" optimizer=[\"adam\", \"sgd\"],\n",
" activation=[\"sigmoid\", \"relu\"],\n",
" dir=\"/tmp/\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The results is a list of (category, network-name) pairs:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[('adam-sigmoid', '/tmp/XOR-Test1-00001-00001'),\n",
" ('sgd-sigmoid', '/tmp/XOR-Test1-00001-00002'),\n",
" ('adam-relu', '/tmp/XOR-Test1-00001-00003'),\n",
" ('sgd-relu', '/tmp/XOR-Test1-00001-00004'),\n",
" ('adam-sigmoid', '/tmp/XOR-Test1-00002-00001'),\n",
" ('sgd-sigmoid', '/tmp/XOR-Test1-00002-00002'),\n",
" ('adam-relu', '/tmp/XOR-Test1-00002-00003'),\n",
" ('sgd-relu', '/tmp/XOR-Test1-00002-00004')]"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"exp.results"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Often, you may wish to plot the results of learning. This may take some time, as the function will re-load each network:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
""
],
"text/plain": [
""
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"exp.plot(\"loss\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Notice that each category has its own color.\n",
"\n",
"There is also a generic apply method for calling a function with each of the (category, network-names). Ususally, you would probably want to re-load the network, and perform some operation in the function."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[('adam-sigmoid', '/tmp/XOR-Test1-00001-00001'),\n",
" ('sgd-sigmoid', '/tmp/XOR-Test1-00001-00002'),\n",
" ('adam-relu', '/tmp/XOR-Test1-00001-00003'),\n",
" ('sgd-relu', '/tmp/XOR-Test1-00001-00004'),\n",
" ('adam-sigmoid', '/tmp/XOR-Test1-00002-00001'),\n",
" ('sgd-sigmoid', '/tmp/XOR-Test1-00002-00002'),\n",
" ('adam-relu', '/tmp/XOR-Test1-00002-00003'),\n",
" ('sgd-relu', '/tmp/XOR-Test1-00002-00004')]"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"exp.apply(lambda category, exp_name: (category, exp_name))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.5"
}
},
"nbformat": 4,
"nbformat_minor": 2
}