{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://www.comet.ml/images/logo_comet_light.png\" width=\"200px\"/>\n",
    "\n",
    "# MNIST Example in keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This Jupyter notebook demonstrates using the **keras** deep learning framework with [comet.ml](https://www.comet.ml).\n",
    "\n",
    "In this example, we build a keras model, and train it on the MNIST dataset.\n",
    "\n",
    "keras is a framework built on top of lower level libraries, such as TensorFlow, or the Cognitive Toolkit. \n",
    "\n",
    "To find out more, you might find these links helpful:\n",
    "\n",
    "* https://keras.io/\n",
    "* https://en.wikipedia.org/wiki/MNIST_database\n",
    "* http://jupyter.org/\n",
    "\n",
    "Let's get started!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Imports\n",
    "\n",
    "First, we import the comet_ml library, followed by the keras library, and others if needed. The only requirement here is that **comet_ml be imported first**. If you forget, just restart the kernel, and import them in the proper order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "## Import this first:\n",
    "from comet_ml import Experiment\n",
    "\n",
    "## Import the deep learning framework:\n",
    "import keras\n",
    "from keras.datasets import mnist\n",
    "from keras.layers import Dense, Dropout\n",
    "from keras.models import Sequential, Model\n",
    "from keras.optimizers import RMSprop\n",
    "from keras.callbacks import Callback\n",
    "from keras.preprocessing import image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a simple demo, we'll start with the the MNIST dataset. In keras, we use the `load_data` method to download and load the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the data, shuffled and split between train and test sets\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what we have here. \n",
    "\n",
    "* x_train are the training inputs\n",
    "* y_train are the training targets\n",
    "* x_test are the test/validation inputs\n",
    "* y_test are the test/validation targets\n",
    "\n",
    "These are numpy tensors, so we can get the shape of each:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 28, 28)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That is, there are 60,000 training inputs, each 28 x 28. These are pictures of numbers.\n",
    "\n",
    "To visualize the patterns, we write a little function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def array_to_image(array, shape, scale):\n",
    "    img = image.array_to_img(array.reshape([int(s) for s in shape]))\n",
    "    x, y = img.size\n",
    "    img = img.resize((x * scale, y * scale))\n",
    "    return img"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We call it by providing a vector, a shape (rows, cols, color depth), and a scaling factor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAIwAAACMCAAAAACLqx7iAAACe0lEQVR4nO3bS6iNURiH8Z9LBuTSmVBKYkCJGKCkJEmKgZhQJpghIxMzA1IYIAMjZSBTjCjXgTollwmZu8xc00EOJqt9nOzV+fb+9uBV7zM47b6z1urp39vXWu9emyRJkiRJkiRJkuS/YVKbyVNg9vhnh2A6LIGDcAZ2wzc4Bce7rDe5jcygSZkaoWSmNh24AKbBOlgPc2BnbcZrOA874As8hwe1GaGSSZkaoWSavIFXwR3+edtW+QX74Gvn2Vv4AK9q00IlkzI1Qsk0KeAhGIZFtSHD8BE2wg+aV/sYoZJJmRqhZJpsId7DUdgGTylbA/AMNlPetsvgSH8yoZJJmRqhZHo7xM2ibGYvwX7YC1cHIhMqmZSpEUqm8SEOfO58+tT5dACuUXa+bQiVTMrUCCXTbxttBtyEDbAVbreVCZVMytQIJdOqD7wYnlDOb/fgMVyE3z2vFyqZlKkRSqZVAaM0eS/DzM6zY3AF3vWyVKhkUqZGKJn2BQyWw1nY1Hl2CU7Am4arhEomZWqEkhlQAaN8Mbed8kKeBHcpTbYGhEomZWqEkhlkAY/xndLg+Alb4P7E00IlkzI1Qsn01karsgJ2werxq76Ahw1XCZVMytQIJdOqgJfAYcpJbt74/45SDnFNG8ShkkmZGqFk+thCzIM9lLuSC7sMeUw5v93oZeVQyaRMjVAyjQt4LuWiwwVY2mXIMJyG6/TzxVyoZFKmRiiZCQp4iNIPW0n322iPKB20WzDSQiZUMilTI5RM1wJeS7l+tgbmdxkyAufgJH9f+21DqGRSpkYoma6HuB2dP2O8pFx+GKX8SujjgGVCJZMyNULJJEmSJEmS9MEfGk5MiN8lROoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<PIL.Image.Image image mode=L size=140x140 at 0x7FB6A68D8278>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "array_to_image(x_train[0], (28, 28, 1), 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often, we need to do a little preparation of the data to get it ready for the learning model. Here, we flatten the inputs, and put input values in the range 0 - 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "60000 train samples\n",
      "10000 test samples\n"
     ]
    }
   ],
   "source": [
    "x_train = x_train.reshape(60000, 784)\n",
    "x_test = x_test.reshape(10000, 784)\n",
    "x_train = x_train.astype(\"float32\")\n",
    "x_test = x_test.astype(\"float32\")\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "print(x_train.shape[0], \"train samples\")\n",
    "print(x_test.shape[0], \"test samples\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, we examine the targets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000,)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that they are just 60,000 values. These are the integer representation of the picture. In this example, we wish to have 10 outputs representing, in a way, the probability of what the picture represents. \n",
    "\n",
    "To turn each number 0-9 into a 10-output vector for training, we use the `keras.utils.to_categorical` function to turn it into a so-called \"one hot\" representation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert class vectors to binary class matrices\n",
    "y_train = keras.utils.to_categorical(y_train, 10)\n",
    "y_test = keras.utils.to_categorical(y_test, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then can check to see if the picture above is labeled correctly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 0., 0., 0., 1., 0., 0., 0., 0.], dtype=float32)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed, the first pattern is a 5. We can also visualize this vector like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAMgAAAAUCAAAAADtzy0WAAAAPElEQVR4nO3PsREAMAjDQMz+OycLpOSICn1JgU+pBedxy/BGD//7xhAaQ2gMoTGExhAaQ2gMoTGExhCaC5mgASiEiqdjAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<PIL.Image.Image image mode=L size=200x20 at 0x7FB6A33AE0B8>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "array_to_image(y_train[0], (1, 10, 1), 20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the \"one hot\" representation showing that y_train[0][5] is 1.0, and all of the rest are zeros."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can create a model, and train the network:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Model\n",
    "\n",
    "In this example, we will build a 5-layer (counting input and output layers), fully-connected neural network. We create a function to make the model for us:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_model_graph(input_shape=(784,)):\n",
    "    model = Sequential()\n",
    "    model.add(Dense(128, activation=\"sigmoid\", input_shape=(784,)))\n",
    "    model.add(Dense(128, activation=\"sigmoid\"))\n",
    "    model.add(Dense(128, activation=\"sigmoid\"))\n",
    "    model.add(Dense(10, activation=\"softmax\"))\n",
    "    model.compile(\n",
    "        loss=\"categorical_crossentropy\", optimizer=RMSprop(), \n",
    "        metrics=[\"accuracy\"]\n",
    "    )\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And call it to create the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = build_model_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the summary method to check the details:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_1 (Dense)              (None, 128)               100480    \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 128)               16512     \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 128)               16512     \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 10)                1290      \n",
      "=================================================================\n",
      "Total params: 134,794\n",
      "Trainable params: 134,794\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Experiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order for comet.ml to log your experiment and results, you need to create an Experiment instance. To do this, you'll need two items:\n",
    "\n",
    "* a Comet `api_key`\n",
    "* a `project_name`\n",
    "\n",
    "You can find your Comet api_key when you log in to https://www.comet.ml and click on your project. You should see a screen that looks similar to:\n",
    "\n",
    "![comet login screen](comet-key.png)\n",
    "\n",
    "Click on the API key to copy the key to your clipboard. \n",
    "\n",
    "It is recommended that you put your COMET_API_KEY in a `.env` key in the current directory. You can do that using the following code. Put it in a cell, replace the `...` with your key, and then delete the cell. That way your key stays private.\n",
    "\n",
    "```ipython\n",
    "%%writefile .env\n",
    "\n",
    "COMET_API_KEY=...\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also recommended that you use your project_name in the cell, so you can match the results with this code. You can make up a new name, or add this experiment to a project that already exists."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "COMET INFO: Experiment is live on comet.ml https://www.comet.ml/cometpublic/comet-notebooks/7092a5e4c362453fb0b3f06785a1d30c\n",
      "\n"
     ]
    }
   ],
   "source": [
    "experiment = Experiment(project_name=\"comet-notebooks\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you get the error that ends with:\n",
    "\n",
    "<pre style=\"background-color:#FDD;\">\n",
    "ValueError: Comet.ml requires an API key. Please provide as the first argument to Experiment(api_key) or as an environment variable named COMET_API_KEY \n",
    "</pre>\n",
    "\n",
    "then that means that either you don't have an `.env` file in this directory, or the key is invalid.\n",
    "\n",
    "Otherwise, you should see the message:\n",
    "\n",
    "<pre style=\"background-color:#FDD;\">\n",
    "COMET INFO: Experiment is live on comet.ml https://www.comet.ml/...\n",
    "</pre>\n",
    "\n",
    "If you click the URL, then a new page will open up. But, even better, you can execute the following line to see the experiment in the current notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"100%\"\n",
       "            height=\"800px\"\n",
       "            src=\"https://www.comet.ml/cometpublic/comet-notebooks/7092a5e4c362453fb0b3f06785a1d30c\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7fb6a36bd828>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "experiment.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By the way, the line `experiment.display()` works when you are at the console too. It will open up a window in your browser.\n",
    "\n",
    "By default, the above display shows the Charts tab, but says \"No plotable data points were found.\" Indeed, we haven't logged any data yet. Let's log some data!\n",
    "\n",
    "Comet.ml has a method to log a hash of the dataset, so that we can see if it changes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment.log_dataset_hash(x_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you view the \"Hyper parameters\" tab in the display, you should now see \"dataset_hash\" and a value in the table.\n",
    "\n",
    "Now, we are ready for training!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Training\n",
    "\n",
    "For this example, we are going to use our `array_to_image` function to watch the hidden layer representations for a particular input change over time. In addition, we will log these images to Comet.ml.\n",
    "\n",
    "First, we construct a keras callback that will call our array_to_image and log it to Comet.ml."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "class VizCallback(Callback):\n",
    "    def __init__(self, model, tensor, filename, experiment, shape):\n",
    "        self.mymodel = model\n",
    "        self.tensor = tensor\n",
    "        self.filename = filename\n",
    "        self.experiment = experiment\n",
    "        self.shape = shape\n",
    "\n",
    "    def on_epoch_end(self, epoch, logs=None):\n",
    "        if \"%s\" in self.filename:\n",
    "            filename = self.filename % (epoch,)\n",
    "        else:\n",
    "            filename = self.filename\n",
    "        log_image(self.mymodel, self.tensor, filename,\n",
    "                  self.experiment, self.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Comet.ml requires that we give the `experiment.log_image()` a filename, so we wrap our function in a function that will save the image to a file, and call log_image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def log_image(model, tensor, filename, experiment, shape):\n",
    "    output = model.predict(tensor)\n",
    "    img = array_to_image(output[0], shape, 10)\n",
    "    img.save(filename)\n",
    "    experiment.log_image(filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because we want to see the hidden layer activations, we need to build a model between the input and that hidden layer. We can do that with this function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_viz_model(model, visualize_layer):\n",
    "    viz_model = Model(inputs=[model.input],\n",
    "                      outputs=[model.layers[visualize_layer].output])\n",
    "    return viz_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can define the models and callbacks for the hidden layers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_model1 = build_viz_model(model, 1)\n",
    "viz_model3 = build_viz_model(model, 3)\n",
    "\n",
    "callbacks = [\n",
    "    VizCallback(viz_model1, x_train[0:1], \"hidden-epoch-%s.gif\",\n",
    "                experiment, (8,16,1)),\n",
    "    VizCallback(viz_model3, x_train[0:1], \"output-epoch-%s.gif\",\n",
    "                experiment, (1,10,1)),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we merely need to call `model.fit()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "60000/60000 [==============================] - 2s 36us/step - loss: 0.7472 - acc: 0.7903 - val_loss: 0.3090 - val_acc: 0.9089\n",
      "Epoch 2/10\n",
      "60000/60000 [==============================] - 2s 33us/step - loss: 0.2542 - acc: 0.9244 - val_loss: 0.2167 - val_acc: 0.9324\n",
      "Epoch 3/10\n",
      "60000/60000 [==============================] - 2s 28us/step - loss: 0.1844 - acc: 0.9451 - val_loss: 0.1719 - val_acc: 0.9489\n",
      "Epoch 4/10\n",
      "60000/60000 [==============================] - 2s 28us/step - loss: 0.1448 - acc: 0.9569 - val_loss: 0.1383 - val_acc: 0.9595\n",
      "Epoch 5/10\n",
      "60000/60000 [==============================] - 2s 30us/step - loss: 0.1192 - acc: 0.9644 - val_loss: 0.1308 - val_acc: 0.9603\n",
      "Epoch 6/10\n",
      "60000/60000 [==============================] - 2s 28us/step - loss: 0.0998 - acc: 0.9701 - val_loss: 0.1149 - val_acc: 0.9637\n",
      "Epoch 7/10\n",
      "60000/60000 [==============================] - 2s 29us/step - loss: 0.0864 - acc: 0.9742 - val_loss: 0.1009 - val_acc: 0.9694\n",
      "Epoch 8/10\n",
      "60000/60000 [==============================] - 2s 29us/step - loss: 0.0752 - acc: 0.9774 - val_loss: 0.1045 - val_acc: 0.9694\n",
      "Epoch 9/10\n",
      "60000/60000 [==============================] - 2s 29us/step - loss: 0.0656 - acc: 0.9798 - val_loss: 0.0904 - val_acc: 0.9735\n",
      "Epoch 10/10\n",
      "60000/60000 [==============================] - 2s 27us/step - loss: 0.0586 - acc: 0.9826 - val_loss: 0.0953 - val_acc: 0.9712\n"
     ]
    }
   ],
   "source": [
    "model.fit(\n",
    "    x_train, y_train,\n",
    "    batch_size=120,\n",
    "    epochs=10,\n",
    "    validation_data=(x_test, y_test),\n",
    "    callbacks=callbacks);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, if you visit the \"Charts\" tab in the display, you should see plots of the accuracy (acc), loss, validation accuracy (val_acc), and validation loss (val_loss).\n",
    "\n",
    "You'll also see information on many of the tabs, including images on the \"Graphics\" tab. You won't see anything on the \"Code\" tab, yet. That will be updated last when we are in a Jupyter environment (like this notebook)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Logging\n",
    "\n",
    "In keras, Comet will automatically log:\n",
    "\n",
    "* the model description\n",
    "* the training loss\n",
    "* the training accuracy\n",
    "* the training validation loss\n",
    "* the training validation accuracy\n",
    "* the source code\n",
    "\n",
    "To log other items manually, you can use any of the following:\n",
    "\n",
    "* `experiment.log_html(HTML_STRING)`\n",
    "* `experiment.html_log_url(URL_STRING)`\n",
    "* `experiment.image(FILENAME)`\n",
    "* `experiment.log_dataset_hash(DATASET)`\n",
    "* `experiment.log_other(KEY, VALUE)`\n",
    "* `experiment.log_metric(NAME, VALUE)`\n",
    "* `experiment.log_parameter(PARAMETER, VALUE)`\n",
    "* `experiment.log_figure(NAME, FIGURE)`\n",
    "\n",
    "For complete details, please see: \n",
    "\n",
    "https://www.comet.ml/docs/python-sdk/Experiment/#experiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Finish\n",
    "\n",
    "Finall, we are ready to tell Comet that our experiment is complete. You don't need to do this is a script that ends. But in Jupyter, we need to indicate that the experiment is finished. We do that with the `experiment.end()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "COMET INFO: Uploading stats to Comet before program termination (may take several seconds)\n",
      "COMET INFO: Experiment is live on comet.ml https://www.comet.ml/cometpublic/comet-notebooks/7092a5e4c362453fb0b3f06785a1d30c\n",
      "\n"
     ]
    }
   ],
   "source": [
    "experiment.end()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it! If you have any comments or questions, please visit us on https://cometml.slack.com"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}