{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5f81c4d3",
   "metadata": {},
   "source": [
    "# Organize your machine learning experiments with ScalarStop\n",
    "\n",
    "### What is ScalarStop?\n",
    "\n",
    "\n",
    "ScalarStop helps you train machine learning models by:\n",
    "\n",
    "* creating a system to uniquely name datasets, model\n",
    "  architectures, trained models, and their\n",
    "  hyperparameters.\n",
    "* saving and loading datasets and models to/from the\n",
    "  filesystem in a consistent way.\n",
    "* recording dataset and model names, hyperparameters, and\n",
    "  training metrics to a SQLite or PostgreSQL database.\n",
    "\n",
    "### Installing ScalarStop\n",
    "\n",
    "\n",
    "ScalarStop is [available on PyPI](https://pypi.org/project/scalarstop/). You can install it from\n",
    "the command line using::\n",
    "\n",
    "    pip3 install scalarstop\n",
    "\n",
    "### Getting started\n",
    "\n",
    "First, we will organize your training, validation, and test sets with subclasses of a `DataBlob` objects.\n",
    "\n",
    "Second, we will describe the architecture of your machine learning models with subclasses of `ModelTemplate` objects.\n",
    "\n",
    "Third, we'll create a `Model` subclass instance that initializes a model with a `ModelTemplate` and trains it on a `DataBlob`'s training and validation sets.\n",
    "\n",
    "Finally, we will save the hyperparameters and training metrics from many `DataBlob`s, `ModelTemplate`s, and `Model`s into a SQLite or PostgreSQL database using the `TrainStore` client.\n",
    "\n",
    "But first, let's import the modules we'll need for this demo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "25a07758",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "import scalarstop as sp\n",
    "\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73fc96fa",
   "metadata": {},
   "source": [
    "### Table of Contents\n",
    "\n",
    "#### 1. [**DataBlob**: Keeping your training dataset organized](#DataBlob:-Keeping-your-training-dataset-organized)\n",
    "#### 2. [**ModelTemplate**: Parameterizing your model creation](#ModelTemplate:-Parameterizing-your-model-creation)\n",
    "#### 3. [**Model**: Combine your ModelTemplate with your DataBlob](#Model:Combine-your-ModelTemplate-with-your-DataBlob)\n",
    "#### 4. [**TrainStore**: Save and query your training metrics in a database](#TrainStore:-Save-and-query-your-machine-learning-metrics-in-a-database)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "272e92db",
   "metadata": {},
   "source": [
    "### `DataBlob`: Keeping your training dataset organized\n",
    "\n",
    "The first step to training machine learning models with ScalarStop is to encase your dataset into a `DataBlob`.\n",
    "\n",
    "A `DataBlob` is a set of three `tf.data.Dataset` pipelines--representing your training, validation, and test sets.\n",
    "\n",
    "When you create a `DataBlob`, variables that affect the creation of the `tf.data.Dataset` pipeline are are stored in a nested Python dataclass named `Hyperparams`. Only store simple JSON-serializable types in the `Hyperparams` dataclass.\n",
    "\n",
    "Creating a new `DataBlob` with `Hyperparams` looks roughly like this:\n",
    "\n",
    "```python\n",
    "from typing import List, Dict\n",
    "import scalarstop as sp\n",
    "\n",
    "class my_datablob_group_name(sp.DataBlob):\n",
    "\n",
    "    @sp.dataclass\n",
    "    class Hyperparams(sp.HyperparamsType):\n",
    "        a: int\n",
    "        b: str\n",
    "        c: Dict[str, float]\n",
    "        d = List[int]\n",
    "\n",
    "    # ... more setup below ...\n",
    "```\n",
    "\n",
    "Then, we define three methods on our `DataBlob` subclass:\n",
    " - `set_training()`\n",
    " - `set_validation()`\n",
    " - `set_test()`\n",
    "\n",
    "Each one of them has to create a *new* instance of a `tf.data.Dataset` pipeline with data samples and labels zipped together. Typically that looks like:\n",
    "\n",
    "```python\n",
    "# Create a tf.data.Dataset for your training samples.\n",
    "samples = tf.data.Dataset.from_tensor_slices([1, 2, 3])\n",
    "\n",
    "# And another tf.data.Dataset for your training labels.\n",
    "labels = tf.data.Dataset.from_tensor_slices([0, 1, 0])\n",
    "\n",
    "# And zip them together.\n",
    "tf.data.Dataset.zip((samples, labels))\n",
    "```\n",
    "\n",
    "Do not apply any batching at this stage. We will do that later.\n",
    "\n",
    "Now we'll create a `DataBlob` that contains the Fashion MNIST dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "63f7404a",
   "metadata": {},
   "outputs": [],
   "source": [
    "class fashion_mnist_v1(sp.DataBlob):\n",
    "\n",
    "    @sp.dataclass\n",
    "    class Hyperparams(sp.HyperparamsType):\n",
    "        num_training_samples: int\n",
    "    \n",
    "    def __init__(self, hyperparams):\n",
    "        \"\"\"\n",
    "        You only need to override __init__ if you want to validate\n",
    "        your hyperparameters or add arguments that are not hyperparameters.\n",
    "\n",
    "        One example of a non-hyperparameter argument would be a\n",
    "        database connection URL.\n",
    "        \"\"\"\n",
    "        if hyperparams[\"num_training_samples\"] > 50_000:\n",
    "            raise ValueError(\"num_training_samples should be <= 50_000\")\n",
    "        super().__init__(hyperparams=hyperparams)\n",
    "        (self._train_images, self._train_labels), \\\n",
    "            (self._test_images, self._test_labels) = \\\n",
    "            tf.keras.datasets.fashion_mnist.load_data()\n",
    "\n",
    "    def set_training(self) -> tf.data.Dataset:\n",
    "        \"\"\"The training set.\"\"\"\n",
    "        samples = tf.data.Dataset.from_tensor_slices(\n",
    "            self._train_images[:self.hyperparams.num_training_samples]\n",
    "        )\n",
    "        labels = tf.data.Dataset.from_tensor_slices(\n",
    "            self._train_labels[:self.hyperparams.num_training_samples]\n",
    "        )\n",
    "        return tf.data.Dataset.zip((samples, labels))\n",
    "\n",
    "    def set_validation(self) -> tf.data.Dataset:\n",
    "        \"\"\"\n",
    "        The validation set.\n",
    "\n",
    "        In this example, the validation set does not change with the\n",
    "        hyperparameters. This allows us to compare results with\n",
    "        different training sets to the same validation set.\n",
    "\n",
    "        However, if your hyperparameters specify how to engineer\n",
    "        features, then you might wnat the validation set and\n",
    "        training set to rely on the same hyperparameters.\n",
    "        \"\"\"\n",
    "        samples = tf.data.Dataset.from_tensor_slices(\n",
    "            self._train_images[50_000:]\n",
    "        )\n",
    "        labels = tf.data.Dataset.from_tensor_slices(\n",
    "            self._train_labels[50_000:]\n",
    "        )\n",
    "        return tf.data.Dataset.zip((samples, labels))\n",
    "\n",
    "    def set_test(self) -> tf.data.Dataset:\n",
    "        \"\"\"The test set. Used to evaluate models but not train them.\"\"\"\n",
    "        samples = tf.data.Dataset.from_tensor_slices(\n",
    "            self._test_images\n",
    "        )\n",
    "        labels = tf.data.Dataset.from_tensor_slices(\n",
    "            self._test_labels\n",
    "        )\n",
    "        return tf.data.Dataset.zip((samples, labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b4c4547",
   "metadata": {},
   "source": [
    "Here we create a `DataBlob` instance with a dictionary to set our `Hyperparams`.\n",
    "\n",
    "The `DataBlob` name is computed by hashing your `DataBlob` subclass class name and the names and values of your `Hyperparams`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ba69ea4d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'fashion_mnist_v1-p166sf7xz19hg8n3mj8f93m8'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datablob1 = fashion_mnist_v1(hyperparams=dict(num_training_samples=10))\n",
    "datablob1.name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f2646d1",
   "metadata": {},
   "source": [
    "The `DataBlob` group name is by default the `DataBlob` subclass name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9f52dd66",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'fashion_mnist_v1'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datablob1.group_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3f374179",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fashion_mnist_v1.Hyperparams(num_training_samples=10)\n"
     ]
    }
   ],
   "source": [
    "print(datablob1.hyperparams)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2ac71f4",
   "metadata": {},
   "source": [
    "Now we create another `DataBlob` instance with a different value for `Hyperparams`.\n",
    "\n",
    "Note that it has a different automatically-generated `name`, but it'll have the same `group_name`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0c2e653a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('fashion_mnist_v1-3wzktz1cmz86vs1r7rbmdoza', 'fashion_mnist_v1')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datablob2 = fashion_mnist_v1(hyperparams=dict(num_training_samples=50))\n",
    "datablob2.name, datablob2.group_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "662725e0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<TakeDataset shapes: ((28, 28), ()), types: (tf.uint8, tf.uint8)>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datablob1.training.take(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0baf8a5",
   "metadata": {},
   "source": [
    "We can save a DataBlob to the filesystem and load it back later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "3c0141d2",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sp.DataBlob fashion_mnist_v1-p166sf7xz19hg8n3mj8f93m8>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.makedirs(\"datablobs_directory\", exist_ok=True)\n",
    "datablob1.save(\"datablobs_directory\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd445224",
   "metadata": {},
   "source": [
    "Here, we use the classmethod `from_filesystem()` to calculate the exact path of our saved `DataBlob` using a copy of the `DataBlob`'s hyperparameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "42eed9c2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sp.DataBlob fashion_mnist_v1-p166sf7xz19hg8n3mj8f93m8>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loaded_datablob1 = fashion_mnist_v1.from_filesystem(\n",
    "    hyperparams=dict(num_training_samples=10),\n",
    "    datablobs_directory=\"datablobs_directory\",\n",
    ")\n",
    "loaded_datablob1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a865dc0",
   "metadata": {},
   "source": [
    "Alternatiely, if we know the exact directory name of our saved `DataBlob`1, we can load it with `with_exact_path()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c1a622dc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sp.DataBlob fashion_mnist_v1-p166sf7xz19hg8n3mj8f93m8>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loaded_datablob2 = fashion_mnist_v1.from_exact_path(\n",
    "    os.path.join(\"datablobs_directory\", datablob1.name)\n",
    ")\n",
    "loaded_datablob2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30c6a680",
   "metadata": {},
   "source": [
    "---\n",
    "### `ModelTemplate`: Parameterizing your model creation\n",
    "\n",
    "The `ModelTemplate` is the same concept as the `DataBlob`, but instead of three `tf.data.Dataset` s, the `ModelTemplate` creates a machine learning framework model object.\n",
    "\n",
    "Here is an example of creating a Keras model. Building and compiling the model is parameterized by values in the `Hyperparams` dataclass."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "caa8cb5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "class small_dense_10_way_classifier_v1(sp.ModelTemplate):\n",
    "\n",
    "    @sp.dataclass\n",
    "    class Hyperparams(sp.HyperparamsType):\n",
    "        hidden_units: int\n",
    "        optimizer: str = \"adam\"\n",
    "\n",
    "    def new_model(self):\n",
    "        model = tf.keras.Sequential(\n",
    "            layers=[\n",
    "                tf.keras.layers.Flatten(input_shape=(28, 28)),\n",
    "                tf.keras.layers.Dense(\n",
    "                    units=self.hyperparams.hidden_units,\n",
    "                    activation=\"relu\",\n",
    "                ),\n",
    "                tf.keras.layers.Dense(units=10)\n",
    "            ],\n",
    "            name=self.name,\n",
    "        )\n",
    "        model.compile(\n",
    "            optimizer=self.hyperparams.optimizer,\n",
    "            loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
    "            metrics=[\"accuracy\"],\n",
    "        )\n",
    "        return model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b9f7fb4",
   "metadata": {},
   "source": [
    "Once again, the `ModelTemplate` has a unique name generated by hashing your subclass and the `Hyperparams`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "52f1a641",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'small_dense_10_way_classifier_v1-uptyfbjofo7rqv8antxrwhjs'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_template = small_dense_10_way_classifier_v1(hyperparams=dict(hidden_units=3))\n",
    "model_template.name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a5e7b9d",
   "metadata": {},
   "source": [
    "---\n",
    "### `Model`:Combine your `ModelTemplate` with your `DataBlob`\n",
    "\n",
    "`DataBlob`s and `ModelTemplate`s are not very useful until you bring them together with a `Model`.\n",
    "\n",
    "A `Model` is an object created by pairing together a `ModelTemplate` instance and a `DataBlob` instance, for the purpose of training the machine learning model created by the `ModelTemplate` on the `DataBlob`'s training and validation sets.\n",
    "\n",
    "Make sure to batch your `DataBlob` before using it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "a60e15fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "datablob = datablob2.batch(2)\n",
    "\n",
    "model = sp.KerasModel(\n",
    "    datablob=datablob,\n",
    "    model_template=model_template,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd0eeac7",
   "metadata": {},
   "source": [
    "Once again, the `Model` has a unique name. But this time it is just a concatenation of the `DataBlob` and `ModelTemplate` names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "709423c6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'mt_small_dense_10_way_classifier_v1-uptyfbjofo7rqv8antxrwhjs__d_fashion_mnist_v1-3wzktz1cmz86vs1r7rbmdoza'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "1944ca0b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "25/25 [==============================] - 3s 115ms/step - loss: 39.2720 - accuracy: 0.3199 - val_loss: 2.6354 - val_accuracy: 0.1039\n",
      "Epoch 2/2\n",
      "25/25 [==============================] - 2s 99ms/step - loss: 2.3040 - accuracy: 0.1014 - val_loss: 2.5192 - val_accuracy: 0.1040\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'loss': [23.980653762817383, 2.3024940490722656],\n",
       " 'accuracy': [0.18000000715255737, 0.11999999731779099],\n",
       " 'val_loss': [2.635443925857544, 2.5192012786865234],\n",
       " 'val_accuracy': [0.1039000004529953, 0.10400000214576721]}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(final_epoch=2, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "403f2137",
   "metadata": {},
   "source": [
    "In ScalarStop, training a machine learning model is an idempotent operation. Instead of saying, \"Train for $n$ **more** epochs,\" we say, \"Train until the model has been trained for $n$ epochs **total**.\"\n",
    "\n",
    "If we call `model.fit()` again with `final_epoch()` still at 2, we get the same metrics but no training happened."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "bf6c2cf7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'loss': [23.980653762817383, 2.3024940490722656],\n",
       " 'accuracy': [0.18000000715255737, 0.11999999731779099],\n",
       " 'val_loss': [2.635443925857544, 2.5192012786865234],\n",
       " 'val_accuracy': [0.1039000004529953, 0.10400000214576721]}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(final_epoch=2, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2eed1a70",
   "metadata": {},
   "source": [
    "Training ScalarStop `Model`s are idempotent because they keep track of how many epochs they have been trained for and the generated training metrics (e.g. loss, accuracy, etc.). This information is saved to the filesystem if you call `model.save()` and is loaded back from disk if you create a new `Model` object with `Model.from_filesystem()` or  `Model.from_filesystem_or_new()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "bf63ccc6",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: models_directory/mt_small_dense_10_way_classifier_v1-uptyfbjofo7rqv8antxrwhjs__d_fashion_mnist_v1-3wzktz1cmz86vs1r7rbmdoza/assets\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['mt_small_dense_10_way_classifier_v1-uptyfbjofo7rqv8antxrwhjs__d_fashion_mnist_v1-3wzktz1cmz86vs1r7rbmdoza']"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.makedirs(\"models_directory\", exist_ok=True)\n",
    "\n",
    "model.save(\"models_directory\")\n",
    "\n",
    "os.listdir(\"models_directory\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26c640bd",
   "metadata": {},
   "source": [
    "This is an example of us loading the model back, calculating the exact filename based on the hyperparameters of both the `DataBlob` and `ModelTemplate`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "0c870fdd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mt_small_dense_10_way_classifier_v1-uptyfbjofo7rqv8antxrwhjs__d_fashion_mnist_v1-3wzktz1cmz86vs1r7rbmdoza\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'accuracy': [0.18000000715255737, 0.11999999731779099],\n",
       " 'loss': [23.980653762817383, 2.3024940490722656],\n",
       " 'val_accuracy': [0.1039000004529953, 0.10400000214576721],\n",
       " 'val_loss': [2.635443925857544, 2.5192012786865234]}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model2 = sp.KerasModel.from_filesystem(\n",
    "    datablob=datablob,\n",
    "    model_template=model_template,\n",
    "    models_directory=\"models_directory\",\n",
    ")\n",
    "print(model2.name)\n",
    "model2.history"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9def4702",
   "metadata": {},
   "source": [
    "If you provide `models_directory` as an argument to `fit()`, ScalarStop will save the model to the filesystem after every epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "912305c2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 3/5\n",
      "25/25 [==============================] - 3s 103ms/step - loss: 2.3016 - accuracy: 0.1200 - val_loss: 2.5192 - val_accuracy: 0.1040\n",
      "INFO:tensorflow:Assets written to: models_directory/mt_small_dense_10_way_classifier_v1-uptyfbjofo7rqv8antxrwhjs__d_fashion_mnist_v1-3wzktz1cmz86vs1r7rbmdoza/assets\n",
      "Epoch 4/5\n",
      "25/25 [==============================] - 2s 88ms/step - loss: 2.2994 - accuracy: 0.1000 - val_loss: 2.5192 - val_accuracy: 0.1060\n",
      "INFO:tensorflow:Assets written to: models_directory/mt_small_dense_10_way_classifier_v1-uptyfbjofo7rqv8antxrwhjs__d_fashion_mnist_v1-3wzktz1cmz86vs1r7rbmdoza/assets\n",
      "Epoch 5/5\n",
      "25/25 [==============================] - 2s 90ms/step - loss: 2.2980 - accuracy: 0.1600 - val_loss: 2.5192 - val_accuracy: 0.1060\n",
      "INFO:tensorflow:Assets written to: models_directory/mt_small_dense_10_way_classifier_v1-uptyfbjofo7rqv8antxrwhjs__d_fashion_mnist_v1-3wzktz1cmz86vs1r7rbmdoza/assets\n"
     ]
    }
   ],
   "source": [
    "_ = model2.fit(final_epoch=5, verbose=1, models_directory=\"models_directory\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b7c2c2e",
   "metadata": {},
   "source": [
    "Once again, ScalarStop saves the model's trainining history alongside the model's weights, but this is not very convenient if you want to do large-scale analysis on the training metrics of many models at once.\n",
    "\n",
    "A better way of storing the training metrics is to use the `TrainStore`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33f358cf",
   "metadata": {},
   "source": [
    "---\n",
    "### `TrainStore`: Save and query your machine learning metrics in a database\n",
    "\n",
    "The `TrainStore` is a client that saves hyperparameters and training metrics to a SQLite or PostgreSQL database. Let's create a new `TrainStore` instance that will save data to a file named `train_store.sqlite3`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "97ab233b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sp.TrainStore sqlite:///train_store.sqlite3>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_store = sp.TrainStore.from_filesystem(filename=\"train_store.sqlite3\")\n",
    "train_store"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36c0deda",
   "metadata": {},
   "source": [
    "The `TrainStore` is also available as a Python context manager.\n",
    "\n",
    "```python\n",
    "with sp.TrainStore.from_filesystem(filename=\"train_store.sqlite3\") as train_store:\n",
    "    # use the TrainStore here\n",
    "\n",
    "# here the TrainStore database connection is automatically closed for you.\n",
    "```\n",
    "\n",
    "We don't use it that way in this example because we want to use the TrainStore across multiple Jupyter notebook cells.\n",
    "\n",
    "And if we want to connect to a PostgreSQL database, the syntax looks like:\n",
    "\n",
    "```python\n",
    "connection_string = \"postgresql://username:password@hostname:port/database\"\n",
    "with sp.TrainStore(connection_string=connection_string) as train_store:\n",
    "    # ...\n",
    "```\n",
    "\n",
    "The `TrainStore` will automatically save your `DataBlob` and `ModelTemplate` name, group name, and hyperparameters to the database. And when you train a `Model`, the `TrainStore` will persist the model name and the epoch training metrics.\n",
    "\n",
    "All of this happens automatically if you pass the `TrainStore` instance to `Model.fit()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "fe73a61f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 3/5\n",
      "25/25 [==============================] - 2s 87ms/step - loss: 2.3012 - accuracy: 0.0800 - val_loss: 2.5129 - val_accuracy: 0.1039\n",
      "Epoch 4/5\n",
      "25/25 [==============================] - 2s 88ms/step - loss: 2.2999 - accuracy: 0.0800 - val_loss: 2.5124 - val_accuracy: 0.0959\n",
      "Epoch 5/5\n",
      "25/25 [==============================] - 2s 90ms/step - loss: 2.2985 - accuracy: 0.0600 - val_loss: 2.5124 - val_accuracy: 0.0959\n"
     ]
    }
   ],
   "source": [
    "_ = model.fit(final_epoch=5, train_store=train_store)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8a285b8",
   "metadata": {},
   "source": [
    "Once you have some information in the `TrainStore`, you can query it for information and receive results as a Pandas `DataFrame`.\n",
    "\n",
    "First, let's list the `DataBlob`s that we have saved:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "99694036",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>group_name</th>\n",
       "      <th>hyperparams</th>\n",
       "      <th>last_modified</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>fashion_mnist_v1-3wzktz1cmz86vs1r7rbmdoza</td>\n",
       "      <td>fashion_mnist_v1</td>\n",
       "      <td>{'num_training_samples': 50}</td>\n",
       "      <td>2021-04-11 20:40:11.891137</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        name        group_name  \\\n",
       "0  fashion_mnist_v1-3wzktz1cmz86vs1r7rbmdoza  fashion_mnist_v1   \n",
       "\n",
       "                    hyperparams              last_modified  \n",
       "0  {'num_training_samples': 50} 2021-04-11 20:40:11.891137  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_store.list_datablobs()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d38a4ee",
   "metadata": {},
   "source": [
    "...and the `ModelTemplate`s that we have saved:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "962a2102",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>group_name</th>\n",
       "      <th>hyperparams</th>\n",
       "      <th>last_modified</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>small_dense_10_way_classifier_v1-uptyfbjofo7rq...</td>\n",
       "      <td>small_dense_10_way_classifier_v1</td>\n",
       "      <td>{'hidden_units': 3, 'optimizer': 'adam'}</td>\n",
       "      <td>2021-04-11 20:40:11.892808</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                name  \\\n",
       "0  small_dense_10_way_classifier_v1-uptyfbjofo7rq...   \n",
       "\n",
       "                         group_name                               hyperparams  \\\n",
       "0  small_dense_10_way_classifier_v1  {'hidden_units': 3, 'optimizer': 'adam'}   \n",
       "\n",
       "               last_modified  \n",
       "0 2021-04-11 20:40:11.892808  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_store.list_model_templates()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9222567c",
   "metadata": {},
   "source": [
    "...and the models that we have trained:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "11a3a94c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model_name</th>\n",
       "      <th>model_class_name</th>\n",
       "      <th>model_last_modified</th>\n",
       "      <th>datablob_name</th>\n",
       "      <th>datablob_group_name</th>\n",
       "      <th>model_template_name</th>\n",
       "      <th>model_template_group_name</th>\n",
       "      <th>dbh__num_training_samples</th>\n",
       "      <th>mth__hidden_units</th>\n",
       "      <th>mth__optimizer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mt_small_dense_10_way_classifier_v1-uptyfbjofo...</td>\n",
       "      <td>KerasModel</td>\n",
       "      <td>2021-04-11 20:40:11.894643</td>\n",
       "      <td>fashion_mnist_v1-3wzktz1cmz86vs1r7rbmdoza</td>\n",
       "      <td>fashion_mnist_v1</td>\n",
       "      <td>small_dense_10_way_classifier_v1-uptyfbjofo7rq...</td>\n",
       "      <td>small_dense_10_way_classifier_v1</td>\n",
       "      <td>50</td>\n",
       "      <td>3</td>\n",
       "      <td>adam</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          model_name model_class_name  \\\n",
       "0  mt_small_dense_10_way_classifier_v1-uptyfbjofo...       KerasModel   \n",
       "\n",
       "         model_last_modified                              datablob_name  \\\n",
       "0 2021-04-11 20:40:11.894643  fashion_mnist_v1-3wzktz1cmz86vs1r7rbmdoza   \n",
       "\n",
       "  datablob_group_name                                model_template_name  \\\n",
       "0    fashion_mnist_v1  small_dense_10_way_classifier_v1-uptyfbjofo7rq...   \n",
       "\n",
       "          model_template_group_name  dbh__num_training_samples  \\\n",
       "0  small_dense_10_way_classifier_v1                         50   \n",
       "\n",
       "   mth__hidden_units mth__optimizer  \n",
       "0                  3           adam  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_store.list_models()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5828394",
   "metadata": {},
   "source": [
    "...and this is how we query for the training history for a given model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "99d69c49",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>epoch_num</th>\n",
       "      <th>model_name</th>\n",
       "      <th>last_modified</th>\n",
       "      <th>metric__loss</th>\n",
       "      <th>metric__accuracy</th>\n",
       "      <th>metric__val_loss</th>\n",
       "      <th>metric__val_accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>mt_small_dense_10_way_classifier_v1-uptyfbjofo...</td>\n",
       "      <td>2021-04-11 20:40:13.981798</td>\n",
       "      <td>2.301222</td>\n",
       "      <td>0.08</td>\n",
       "      <td>2.512903</td>\n",
       "      <td>0.1039</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>mt_small_dense_10_way_classifier_v1-uptyfbjofo...</td>\n",
       "      <td>2021-04-11 20:40:16.110798</td>\n",
       "      <td>2.299875</td>\n",
       "      <td>0.08</td>\n",
       "      <td>2.512378</td>\n",
       "      <td>0.0959</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>mt_small_dense_10_way_classifier_v1-uptyfbjofo...</td>\n",
       "      <td>2021-04-11 20:40:18.271750</td>\n",
       "      <td>2.298548</td>\n",
       "      <td>0.06</td>\n",
       "      <td>2.512365</td>\n",
       "      <td>0.0959</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   epoch_num                                         model_name  \\\n",
       "0          3  mt_small_dense_10_way_classifier_v1-uptyfbjofo...   \n",
       "1          4  mt_small_dense_10_way_classifier_v1-uptyfbjofo...   \n",
       "2          5  mt_small_dense_10_way_classifier_v1-uptyfbjofo...   \n",
       "\n",
       "               last_modified  metric__loss  metric__accuracy  \\\n",
       "0 2021-04-11 20:40:13.981798      2.301222              0.08   \n",
       "1 2021-04-11 20:40:16.110798      2.299875              0.08   \n",
       "2 2021-04-11 20:40:18.271750      2.298548              0.06   \n",
       "\n",
       "   metric__val_loss  metric__val_accuracy  \n",
       "0          2.512903                0.1039  \n",
       "1          2.512378                0.0959  \n",
       "2          2.512365                0.0959  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_store.list_model_epochs(model_name=model.name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "f82edcf5",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_template_2 = small_dense_10_way_classifier_v1(hyperparams=dict(hidden_units=5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "fea5f567",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_2 = sp.KerasModel(datablob=datablob, model_template=model_template_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "5b007d96",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "25/25 [==============================] - 2s 91ms/step - loss: 28.2529 - accuracy: 0.1749 - val_loss: 2.5577 - val_accuracy: 0.1217\n",
      "Epoch 2/10\n",
      "25/25 [==============================] - 2s 92ms/step - loss: 2.8903 - accuracy: 0.2733 - val_loss: 2.4093 - val_accuracy: 0.1032\n",
      "Epoch 3/10\n",
      "25/25 [==============================] - 2s 89ms/step - loss: 2.2542 - accuracy: 0.2733 - val_loss: 2.3985 - val_accuracy: 0.1081\n",
      "Epoch 4/10\n",
      "25/25 [==============================] - 2s 95ms/step - loss: 2.2393 - accuracy: 0.2733 - val_loss: 2.3969 - val_accuracy: 0.1096\n",
      "Epoch 5/10\n",
      "25/25 [==============================] - 2s 89ms/step - loss: 2.2373 - accuracy: 0.2733 - val_loss: 2.3966 - val_accuracy: 0.1097\n",
      "Epoch 6/10\n",
      "25/25 [==============================] - 2s 89ms/step - loss: 2.2353 - accuracy: 0.2733 - val_loss: 2.3966 - val_accuracy: 0.1097\n",
      "Epoch 7/10\n",
      "25/25 [==============================] - 2s 88ms/step - loss: 2.2333 - accuracy: 0.2733 - val_loss: 2.3966 - val_accuracy: 0.1097\n",
      "Epoch 8/10\n",
      "25/25 [==============================] - 2s 93ms/step - loss: 2.2313 - accuracy: 0.2733 - val_loss: 2.3966 - val_accuracy: 0.1097\n",
      "Epoch 9/10\n",
      "25/25 [==============================] - 2s 91ms/step - loss: 2.2293 - accuracy: 0.2733 - val_loss: 2.3967 - val_accuracy: 0.1097\n",
      "Epoch 10/10\n",
      "25/25 [==============================] - 2s 88ms/step - loss: 2.2273 - accuracy: 0.2733 - val_loss: 2.3967 - val_accuracy: 0.1097\n"
     ]
    }
   ],
   "source": [
    "_ = model_2.fit(final_epoch=10, train_store=train_store)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "ace44f43",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>epoch_num</th>\n",
       "      <th>model_name</th>\n",
       "      <th>last_modified</th>\n",
       "      <th>metric__loss</th>\n",
       "      <th>metric__accuracy</th>\n",
       "      <th>metric__val_loss</th>\n",
       "      <th>metric__val_accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>mt_small_dense_10_way_classifier_v1-axos7t2rck...</td>\n",
       "      <td>2021-04-11 20:40:20.782506</td>\n",
       "      <td>13.858220</td>\n",
       "      <td>0.14</td>\n",
       "      <td>2.557676</td>\n",
       "      <td>0.1217</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>mt_small_dense_10_way_classifier_v1-axos7t2rck...</td>\n",
       "      <td>2021-04-11 20:40:22.989173</td>\n",
       "      <td>2.634313</td>\n",
       "      <td>0.18</td>\n",
       "      <td>2.409259</td>\n",
       "      <td>0.1032</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>mt_small_dense_10_way_classifier_v1-axos7t2rck...</td>\n",
       "      <td>2021-04-11 20:40:25.121682</td>\n",
       "      <td>2.264078</td>\n",
       "      <td>0.18</td>\n",
       "      <td>2.398494</td>\n",
       "      <td>0.1081</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>mt_small_dense_10_way_classifier_v1-axos7t2rck...</td>\n",
       "      <td>2021-04-11 20:40:27.412649</td>\n",
       "      <td>2.252162</td>\n",
       "      <td>0.18</td>\n",
       "      <td>2.396904</td>\n",
       "      <td>0.1096</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>mt_small_dense_10_way_classifier_v1-axos7t2rck...</td>\n",
       "      <td>2021-04-11 20:40:29.547414</td>\n",
       "      <td>2.250889</td>\n",
       "      <td>0.18</td>\n",
       "      <td>2.396623</td>\n",
       "      <td>0.1097</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>mt_small_dense_10_way_classifier_v1-axos7t2rck...</td>\n",
       "      <td>2021-04-11 20:40:31.679167</td>\n",
       "      <td>2.249614</td>\n",
       "      <td>0.18</td>\n",
       "      <td>2.396614</td>\n",
       "      <td>0.1097</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>mt_small_dense_10_way_classifier_v1-axos7t2rck...</td>\n",
       "      <td>2021-04-11 20:40:33.794024</td>\n",
       "      <td>2.248346</td>\n",
       "      <td>0.18</td>\n",
       "      <td>2.396612</td>\n",
       "      <td>0.1097</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>mt_small_dense_10_way_classifier_v1-axos7t2rck...</td>\n",
       "      <td>2021-04-11 20:40:36.031745</td>\n",
       "      <td>2.247089</td>\n",
       "      <td>0.18</td>\n",
       "      <td>2.396631</td>\n",
       "      <td>0.1097</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>mt_small_dense_10_way_classifier_v1-axos7t2rck...</td>\n",
       "      <td>2021-04-11 20:40:38.216244</td>\n",
       "      <td>2.245847</td>\n",
       "      <td>0.18</td>\n",
       "      <td>2.396651</td>\n",
       "      <td>0.1097</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>mt_small_dense_10_way_classifier_v1-axos7t2rck...</td>\n",
       "      <td>2021-04-11 20:40:40.344820</td>\n",
       "      <td>2.244625</td>\n",
       "      <td>0.18</td>\n",
       "      <td>2.396712</td>\n",
       "      <td>0.1097</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   epoch_num                                         model_name  \\\n",
       "0          1  mt_small_dense_10_way_classifier_v1-axos7t2rck...   \n",
       "1          2  mt_small_dense_10_way_classifier_v1-axos7t2rck...   \n",
       "2          3  mt_small_dense_10_way_classifier_v1-axos7t2rck...   \n",
       "3          4  mt_small_dense_10_way_classifier_v1-axos7t2rck...   \n",
       "4          5  mt_small_dense_10_way_classifier_v1-axos7t2rck...   \n",
       "5          6  mt_small_dense_10_way_classifier_v1-axos7t2rck...   \n",
       "6          7  mt_small_dense_10_way_classifier_v1-axos7t2rck...   \n",
       "7          8  mt_small_dense_10_way_classifier_v1-axos7t2rck...   \n",
       "8          9  mt_small_dense_10_way_classifier_v1-axos7t2rck...   \n",
       "9         10  mt_small_dense_10_way_classifier_v1-axos7t2rck...   \n",
       "\n",
       "               last_modified  metric__loss  metric__accuracy  \\\n",
       "0 2021-04-11 20:40:20.782506     13.858220              0.14   \n",
       "1 2021-04-11 20:40:22.989173      2.634313              0.18   \n",
       "2 2021-04-11 20:40:25.121682      2.264078              0.18   \n",
       "3 2021-04-11 20:40:27.412649      2.252162              0.18   \n",
       "4 2021-04-11 20:40:29.547414      2.250889              0.18   \n",
       "5 2021-04-11 20:40:31.679167      2.249614              0.18   \n",
       "6 2021-04-11 20:40:33.794024      2.248346              0.18   \n",
       "7 2021-04-11 20:40:36.031745      2.247089              0.18   \n",
       "8 2021-04-11 20:40:38.216244      2.245847              0.18   \n",
       "9 2021-04-11 20:40:40.344820      2.244625              0.18   \n",
       "\n",
       "   metric__val_loss  metric__val_accuracy  \n",
       "0          2.557676                0.1217  \n",
       "1          2.409259                0.1032  \n",
       "2          2.398494                0.1081  \n",
       "3          2.396904                0.1096  \n",
       "4          2.396623                0.1097  \n",
       "5          2.396614                0.1097  \n",
       "6          2.396612                0.1097  \n",
       "7          2.396631                0.1097  \n",
       "8          2.396651                0.1097  \n",
       "9          2.396712                0.1097  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_store.list_model_epochs(model_name=model_2.name)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}