{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c2cff14e-0b6f-42e0-8de8-e080085234a7",
   "metadata": {},
   "source": [
    "# Calysto Scheme Tensorflow Example"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "221e6fca-413f-4701-b04c-189b11ffe9be",
   "metadata": {},
   "source": [
    "First we install the Python modules we will need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "06f84fc2-24a8-4fe7-a72e-cffc634c6b8f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! pip install comet_ml tensorflow numpy --upgrade --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f05259b1-603b-4347-ba9e-382d6b5afb8f",
   "metadata": {},
   "source": [
    "And import them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c9b9167e-c057-45e7-9534-8a8f7ffc5ce3",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(IPython)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(import-as \"tensorflow\" \"tf\")\n",
    "(import-as \"numpy\" \"np\")\n",
    "(import \"comet_ml\")\n",
    "(import \"PIL.Image\")\n",
    "(import \"base64\")\n",
    "(import \"IPython\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81c8846c-f9ef-4f70-bd81-5ea96a7835ed",
   "metadata": {},
   "source": [
    "To use Comet to track your experiments, copy your API key from:\n",
    "https://www.comet.com/account-settings/apiKeys\n",
    "\n",
    "and paste it below between the quotes. (Otherwise, skip this cell)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e785e5e8-5683-468e-9e90-df792bfe5fda",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "(import \"os\")\n",
    "(set-item! os.environ \"COMET_API_KEY\" \"<PASTE-COMET-API-KEY-HERE>\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e0ace8a-021a-41ef-9faa-22172d7407ff",
   "metadata": {},
   "source": [
    "Next, we get the MNIST dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1b8fc415-360f-46c1-9594-fc8e187ce2de",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "(define mnist tf.keras.datasets.mnist)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ba35a3bb-b9e3-40ed-bc78-77370287b8db",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "(define dataset (mnist.load_data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c6261e3c-7272-4d2e-95f0-32d475c7f296",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "(define x_train (get-item (get-item dataset 0) 0))\n",
    "(define y_train (get-item (get-item dataset 0) 1))\n",
    "(define x_test (get-item (get-item dataset 1) 0))\n",
    "(define y_test (get-item (get-item dataset 1) 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75ef43d2-c2f3-464b-8f68-77de5fc7f4a1",
   "metadata": {},
   "source": [
    "*Hack to allow Python 3.9 to work:*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "3b52d66c-bd39-4360-a070-e443e66fb368",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "(set-attr! base64 \"encodestring\" base64.encodebytes)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3192f73d-1054-4367-860d-232106bdfce7",
   "metadata": {},
   "source": [
    "Let's take a look at an input matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "a9389c60-4b83-4a97-9aed-fed6486fdca3",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAABAElEQVR4nGNgGMyAWUhIqK5jvdSy\n/9/rGRgYGFhgEnJsVjYCwQwMDAxPJgV+vniQgYGBgREqZ7iXH8r6l/SV4dn7m8gmCt3++/fv37/H\ntn3/iMW+gDnZf/+e5WbQnoXNNXyMs/5GoQoxwVmf/n9kSGFiwAW49/11wynJoPzx4YIcRlyygR/+\n/i2XxCWru+vv32nSuGQFYv/83Y3b4p9/fzpAmSyoMnohpiwM1w5h06Q+5enfv39/bcMiJVF09+/f\nv39P+mFKiTtd/fv3799jgZiBJLT69t+/f/8eDuDEkDJf8+jv379/v7Ryo4qzMDAwMAQGMjBc3/y3\n5wM2V1IfAABFF16Aa0wAOwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<PIL.Image.Image image mode=L size=28x28 at 0x7F11330E3BE0>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(PIL.Image.fromarray (get-item x_train 0))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b4e6d07-00c5-430d-9e62-d148aeace321",
   "metadata": {},
   "source": [
    "What is the target for the above?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "b92d4b5c-01bb-4c72-b136-245a1bd27dd6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(get-item y_train 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "493fe6d5-e1d8-4e08-9cd8-765746d1b52a",
   "metadata": {},
   "source": [
    "## Prepare the dataset\n",
    "\n",
    "For this network, we'll scale the inputs to be between 0 and 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "342b9709-9041-44a5-9bf8-403f4d2a3154",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "(set! x_train (/ x_train 255.0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "fc5ab463-9b3f-4e89-a07c-77a8cc72d856",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "(set! x_test (/ x_test 255.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b611aa55-09ca-4189-9dd5-06570cf3db58",
   "metadata": {},
   "source": [
    "## Customize the Loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "c1f13fe0-302c-4350-8c93-6d6f2812d05c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "(define loss_fn (tf.keras.losses.SparseCategoricalCrossentropy (dict '((from_logits : #t)))))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9337fbfb-7aac-4a95-a624-a8263ea1f57f",
   "metadata": {},
   "source": [
    "Ok, let's train a model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "590499d0-333c-4f33-ad86-078f1f5505bf",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"https://www.comet.com/dsblank/calysto-scheme/5ab05a53032f4438a5aa27c97f461bbf\"\n",
      "Model: \"sequential_4\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " flatten_4 (Flatten)         (None, 784)               0         \n",
      "                                                                 \n",
      " dense_8 (Dense)             (None, 10)                7850      \n",
      "                                                                 \n",
      " dropout_4 (Dropout)         (None, 10)                0         \n",
      "                                                                 \n",
      " dense_9 (Dense)             (None, 10)                110       \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 7,960\n",
      "Trainable params: 7,960\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/5\n",
      "1875/1875 [==============================] - 3s 1ms/step - loss: 0.5386 - accuracy: 0.8441\n",
      "Epoch 2/5\n",
      "1875/1875 [==============================] - 3s 1ms/step - loss: 0.2936 - accuracy: 0.9161\n",
      "Epoch 3/5\n",
      "1875/1875 [==============================] - 2s 1ms/step - loss: 0.2651 - accuracy: 0.9243\n",
      "Epoch 4/5\n",
      "1875/1875 [==============================] - 3s 2ms/step - loss: 0.2498 - accuracy: 0.9290\n",
      "Epoch 5/5\n",
      "1875/1875 [==============================] - 3s 1ms/step - loss: 0.2391 - accuracy: 0.9322\n"
     ]
    }
   ],
   "source": [
    "(let* ((optimizer (choose \"adam\" \"rmsprop\" \"sgd\"))\n",
    "       (dropout_rate (choose 0.0 0.1 0.2 0.4))\n",
    "       (activation (choose \"relu\" \"sigmoid\"))\n",
    "       (hidden_layer_size (choose 10 20 30))\n",
    "       (options (dict `((optimizer : ,optimizer)(loss : ,loss_fn)(metrics : ,(vector \"accuracy\")))))\n",
    "       (epochs 5)\n",
    "       (experiment (comet_ml.Experiment (dict '((project_name : \"calysto-scheme\")))))\n",
    "       (model (tf.keras.models.Sequential \n",
    "          (vector\n",
    "            (tf.keras.layers.Flatten (dict '((input_shape : (28 28)))))\n",
    "            (tf.keras.layers.Dense hidden_layer_size (dict `((activation : ,activation))))\n",
    "            (tf.keras.layers.Dropout dropout_rate)\n",
    "            (tf.keras.layers.Dense 10)\n",
    "        ))))\n",
    "  (print experiment.url)\n",
    "  (model.compile options)\n",
    "  (model.summary)\n",
    "  (experiment.log_parameters (dict `((optimizer : ,optimizer) \n",
    "                                     (dropout_rate : ,dropout_rate) \n",
    "                                     (activation : ,activation) \n",
    "                                     (hidden_layer_size : ,hidden_layer_size)\n",
    "                                     (epochs : ,epochs)\n",
    "                                    )) (dict))\n",
    "  (experiment.set_model_graph model)\n",
    "  (let ((history (model.fit x_train y_train (dict `((epochs : ,epochs)))))\n",
    "        (step 0))\n",
    "      (map (lambda (key)\n",
    "          (set! step 0)\n",
    "          (map (lambda (v) (experiment.log_metric key v step) (set! step (+ step 1)))\n",
    "               (get-item history.history key)))\n",
    "       history.history))\n",
    "  (experiment.end)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a132729-95c0-4d74-9f1d-8874c200f8e4",
   "metadata": {},
   "source": [
    "Nice! Now let's do it again, picking different hyperparameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "f96e8822-9c56-4196-b404-ebb76f4056d1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"https://www.comet.com/dsblank/calysto-scheme/dc1f0e7f33844b89827218c9d099d78d\"\n",
      "Model: \"sequential_5\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " flatten_5 (Flatten)         (None, 784)               0         \n",
      "                                                                 \n",
      " dense_10 (Dense)            (None, 20)                15700     \n",
      "                                                                 \n",
      " dropout_5 (Dropout)         (None, 20)                0         \n",
      "                                                                 \n",
      " dense_11 (Dense)            (None, 10)                210       \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 15,910\n",
      "Trainable params: 15,910\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/5\n",
      "1875/1875 [==============================] - 3s 2ms/step - loss: 0.3972 - accuracy: 0.8883\n",
      "Epoch 2/5\n",
      "1875/1875 [==============================] - 3s 2ms/step - loss: 0.2279 - accuracy: 0.9354\n",
      "Epoch 3/5\n",
      "1875/1875 [==============================] - 3s 2ms/step - loss: 0.1949 - accuracy: 0.9438\n",
      "Epoch 4/5\n",
      "1875/1875 [==============================] - 3s 1ms/step - loss: 0.1764 - accuracy: 0.9487\n",
      "Epoch 5/5\n",
      "1875/1875 [==============================] - 3s 1ms/step - loss: 0.1609 - accuracy: 0.9540\n"
     ]
    }
   ],
   "source": [
    "(choose)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "747a0bde-2a27-40c7-ab8b-63d627d71a08",
   "metadata": {},
   "source": [
    "Continue until there are no more options."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Calysto Scheme 3",
   "language": "scheme",
   "name": "calysto_scheme"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "scheme"
   },
   "mimetype": "text/x-scheme",
   "name": "scheme",
   "pygments_lexer": "scheme"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}