{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "colab": { "name": "PytorchConfusionMatrixSimple.ipynb", "provenance": [], "collapsed_sections": [] }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" } }, "cells": [ { "cell_type": "markdown", "metadata": { "id": "0ca5f47a-43c1-47ba-9322-c48f645b0908" }, "source": [ "# Pytorch Example with Confusion Matrix with Comet ML\n", "\n", "For this example, we will use Pytorch and create a interactive Confusion Matrix\n", "in Comet ML. You'll need a Comet API key to log the Confusion Matrix, which is free for anyone.\n", "\n", "## Overview\n", "\n", "Our goal in this demonstration is to train a Pytorch model to categorize images of digits from the MNIST dataset, being able to see examples of each cell in a confusion matrix, like this:\n", "\n", "\n", "\n", "Comet provides a very easy way to make such confusion matrices. You can do that with a single command:\n", "\n", "```python\n", "experiment.log_confusion_matrix(actual, predicted, images=images)\n", "```\n", "\n", "where `actual` is the ground truth (given as vectors or labels), `predicted` is the ML's prediction (given as vectors or labels), and `images` is a list of image data.\n", "\n", "## End-to-End Example\n", "\n", "Let's explore a complete example from start to finish. \n", "\n", "First, we install the needed Python libraries:" ], "id": "0ca5f47a-43c1-47ba-9322-c48f645b0908" }, { "cell_type": "code", "metadata": { "id": "dec1NN24O_MH" }, "source": [ "%pip install comet_ml>=3.10.0 torch torchvision --quiet" ], "id": "dec1NN24O_MH", "execution_count": 1, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "9aY1iWM88iUa" }, "source": [ "Now we import Comet:" ], "id": "9aY1iWM88iUa" }, { "cell_type": "code", "metadata": { "id": "82048786-8d61-44c7-a6d2-0484a69779d0" }, "source": [ "import comet_ml" ], "id": "82048786-8d61-44c7-a6d2-0484a69779d0", "execution_count": 2, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "65e82fc4-adda-4a7b-a84b-91d7cce363c4" }, "source": [ "We can then make sure that our Comet API key is properly configured. The following command will give instructions if not:" ], "id": "65e82fc4-adda-4a7b-a84b-91d7cce363c4" }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "406fa63d-8854-4812-a082-e3ad011ed3fc", "outputId": "44a0adc9-12c7-4251-80ac-34bc8318644c" }, "source": [ "comet_ml.init()" ], "id": "406fa63d-8854-4812-a082-e3ad011ed3fc", "execution_count": 3, "outputs": [ { "output_type": "stream", "text": [ "COMET INFO: Comet API key is valid\n" ], "name": "stderr" } ] }, { "cell_type": "markdown", "metadata": { "id": "6cc97390-6fb2-45eb-9b81-47f25ae73cc7" }, "source": [ "Now, we import the rest of the Python libraries that we will need:" ], "id": "6cc97390-6fb2-45eb-9b81-47f25ae73cc7" }, { "cell_type": "code", "metadata": { "id": "f11f4599-c2d6-4356-9936-aab28ade436b" }, "source": [ "import torch\n", "import torch.nn as nn\n", "import torchvision.datasets as dsets\n", "import torchvision.transforms as transforms\n", "from torch.utils.data import SubsetRandomSampler\n", "from torch.autograd import Variable" ], "id": "f11f4599-c2d6-4356-9936-aab28ade436b", "execution_count": 4, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "32a480fb-f40e-48ec-b27b-12a0cbe72efb" }, "source": [ "## MNIST Dataset\n", "\n" ], "id": "32a480fb-f40e-48ec-b27b-12a0cbe72efb" }, { "cell_type": "markdown", "metadata": { "id": "b5b7c632-760c-490b-9d14-8e0e7dcbd277" }, "source": [ "The first time this runs may take a few minutes to download, and then a couple more minutes to process:" ], "id": "b5b7c632-760c-490b-9d14-8e0e7dcbd277" }, { "cell_type": "code", "metadata": { "id": "0cbb42ca-f73e-4de5-bb4a-d05dc45d6b47" }, "source": [ "train_dataset = dsets.MNIST(\n", " root='./data/',\n", " train=True,\n", " transform=transforms.ToTensor(),\n", " download=True)\n", "\n", "test_dataset = dsets.MNIST(\n", " root='./data/',\n", " train=False,\n", " transform=transforms.ToTensor())" ], "id": "0cbb42ca-f73e-4de5-bb4a-d05dc45d6b47", "execution_count": 5, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "563109fd-e169-4238-ac70-519a1c00c633" }, "source": [ "## Create the Model\n", "\n", "We'll now write a function that will create the model.\n", "\n", "In this example, we'll take advantage of Comet's `Experiment` to get access to the hyperparameters via `experiment.get_parameter()`. This will be very handy when we later use Comet's Hyperparameter Optimizer to generate the Experiments.\n", "\n", "This function will actually return the three components of the model: the rnn, the criterion, and the optimizer." ], "id": "563109fd-e169-4238-ac70-519a1c00c633" }, { "cell_type": "code", "metadata": { "id": "71cc8afc-0b6e-42e4-a0aa-f66a5b2fe006" }, "source": [ "def build_model(experiment):\n", " input_size = experiment.get_parameter(\"input_size\")\n", " hidden_size = experiment.get_parameter(\"hidden_size\")\n", " num_layers = experiment.get_parameter(\"num_layers\")\n", " num_classes = experiment.get_parameter(\"num_classes\")\n", " learning_rate = experiment.get_parameter(\"learning_rate\")\n", "\n", " class RNN(nn.Module):\n", " def __init__(self, input_size, hidden_size, num_layers, num_classes):\n", " super(RNN, self).__init__()\n", " self.hidden_size = hidden_size\n", " self.num_layers = num_layers\n", " self.lstm = nn.LSTM(\n", " input_size, \n", " hidden_size, \n", " num_layers, \n", " batch_first=True)\n", " self.fc = nn.Linear(hidden_size, num_classes)\n", "\n", " def forward(self, x):\n", " # Set initial states\n", " h0 = Variable(torch.zeros(self.num_layers, x.size(0), \n", " self.hidden_size))\n", " c0 = Variable(torch.zeros(self.num_layers, x.size(0), \n", " self.hidden_size))\n", "\n", " # Forward propagate RNN\n", " self.out, _ = self.lstm(x, (h0, c0))\n", "\n", " # Decode hidden state of last time step\n", " out = self.fc(self.out[:, -1, :])\n", " return out\n", "\n", " rnn = RNN(\n", " input_size,\n", " hidden_size,\n", " num_layers,\n", " num_classes,\n", " )\n", "\n", " # Loss and Optimizer\n", " criterion = nn.CrossEntropyLoss()\n", " optimizer = torch.optim.Adam(rnn.parameters(), lr=learning_rate)\n", "\n", " return (rnn, criterion, optimizer)" ], "id": "71cc8afc-0b6e-42e4-a0aa-f66a5b2fe006", "execution_count": 6, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "53886701-7210-4895-b946-cfd7d788662b" }, "source": [ "We'll call this function below, once we create an `Experiment`." ], "id": "53886701-7210-4895-b946-cfd7d788662b" }, { "cell_type": "markdown", "metadata": { "id": "8114f369-7ba3-4783-93f4-c13ac1c70a4b" }, "source": [ "## Train the Dataset on the Model\n", "\n", "Now we are ready to set up a Comet Experiment, and train the model.\n", "\n", "First, we can set all of the Hyperparameters of the model:" ], "id": "8114f369-7ba3-4783-93f4-c13ac1c70a4b" }, { "cell_type": "code", "metadata": { "id": "c8b50453-3097-4a0b-9cf3-9b69f8b51df7" }, "source": [ "hyper_params = {\n", " \"epochs\": 10,\n", " \"batch_size\": 120,\n", " \"first_layer_units\": 128,\n", " \"sequence_length\": 28,\n", " \"input_size\": 28,\n", " \"hidden_size\": 128,\n", " \"num_layers\": 2,\n", " \"num_classes\": 10,\n", " \"learning_rate\": 0.01\n", "}" ], "id": "c8b50453-3097-4a0b-9cf3-9b69f8b51df7", "execution_count": 7, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "062c6192-5c29-41ca-8841-5ceb82b7d978" }, "source": [ "Next we create the experiment, and log the Hyperparameters:" ], "id": "062c6192-5c29-41ca-8841-5ceb82b7d978" }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "174f3481-0798-4a11-acda-e651753a1caa", "outputId": "b9038054-a7b9-483a-8a19-11eb7c2f114c" }, "source": [ "experiment = comet_ml.Experiment(project_name=\"pytorch-confusion-matrix\")\n", "experiment.log_parameters(hyper_params)" ], "id": "174f3481-0798-4a11-acda-e651753a1caa", "execution_count": 8, "outputs": [ { "output_type": "stream", "text": [ "COMET WARNING: As you are running in a Jupyter environment, you will need to call `experiment.end()` when finished to ensure all metrics and code are logged before exiting.\n", "COMET INFO: Experiment is live on comet.ml https://www.comet.ml/dsblank/pytorch-confusion-matrix/819f19ee68ba4b91bab88421b795451d\n", "\n" ], "name": "stderr" } ] }, { "cell_type": "markdown", "metadata": { "id": "4a5976cc-79e6-43ae-8301-8c39abf302a0" }, "source": [ "We can now construct the model components:" ], "id": "4a5976cc-79e6-43ae-8301-8c39abf302a0" }, { "cell_type": "code", "metadata": { "id": "8778542d-da44-4f7e-a01e-2e24d2dfec3e" }, "source": [ "rnn, criterion, optimizer = build_model(experiment)" ], "id": "8778542d-da44-4f7e-a01e-2e24d2dfec3e", "execution_count": 9, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "423e17c2-e9ef-473c-b2ee-055669033984" }, "source": [ "To make this demonstration go a little faster, we'll just use a sample of the items from the training set:" ], "id": "423e17c2-e9ef-473c-b2ee-055669033984" }, { "cell_type": "code", "metadata": { "id": "56724a82-9a93-4af8-b0d2-6492364d4be9" }, "source": [ "SAMPLE_SIZE = 1000" ], "id": "56724a82-9a93-4af8-b0d2-6492364d4be9", "execution_count": 10, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "acczEZl36Bml" }, "source": [ "Now we can construct the loader:" ], "id": "acczEZl36Bml" }, { "cell_type": "code", "metadata": { "id": "761fe054-c539-4883-a58f-ad46a0108fcb" }, "source": [ "sampler = SubsetRandomSampler(list(range(SAMPLE_SIZE)))\n", "train_loader = torch.utils.data.DataLoader(\n", " dataset=train_dataset,\n", " batch_size=experiment.get_parameter('batch_size'),\n", " sampler=sampler,\n", " #shuffle=True, # can't use shuffle with sampler\n", ")" ], "id": "761fe054-c539-4883-a58f-ad46a0108fcb", "execution_count": 11, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "EaVEhHpJC2nY" }, "source": [ "Instead, if you would rather train on the entire dataset, you can:\n", "\n", "```python\n", "train_loader = torch.utils.data.DataLoader(\n", " dataset=train_dataset,\n", " batch_size=experiment.get_parameter('batch_size'),\n", " shuffle=True,\n", ")\n", "```" ], "id": "EaVEhHpJC2nY" }, { "cell_type": "markdown", "metadata": { "id": "RZvVJzzE6FgW" }, "source": [ "Now we can train the model. Some items to note:\n", "\n", "1. We use `experiment.train()` to provide the context for logged metrics\n", "2. We collect the actual, predicted, and images for each batch\n", "3. At the end of the epoch, compute and log the confusion matrix" ], "id": "RZvVJzzE6FgW" }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "8548f3f5-e1ef-4c35-b383-3dbe01941025", "tags": [], "outputId": "5df2bd81-5202-46b4-bb09-d3af5f07d5d3" }, "source": [ "with experiment.train():\n", " step = 0\n", " for epoch in range(experiment.get_parameter('epochs')):\n", " print(\"\\nepoch:\", epoch)\n", " correct = 0\n", " total = 0\n", " for batch_step, (images, labels) in enumerate(train_loader):\n", " print(\".\", end=\"\")\n", " images = Variable(images.view(\n", " -1,\n", " experiment.get_parameter('sequence_length'),\n", " experiment.get_parameter(\"input_size\")))\n", "\n", " labels = Variable(labels)\n", "\n", " # Forward + Backward + Optimize\n", " optimizer.zero_grad()\n", " outputs = rnn(images)\n", " loss = criterion(outputs, labels)\n", " loss.backward()\n", " optimizer.step()\n", "\n", " # Compute train accuracy\n", " _, predicted = torch.max(outputs.data, 1)\n", " batch_total = labels.size(0)\n", " total += batch_total\n", "\n", " batch_correct = (predicted == labels.data).sum()\n", " correct += batch_correct\n", "\n", " # Log batch_accuracy to Comet.ml; step is each batch\n", " step += 1\n", " experiment.log_metric(\"batch_accuracy\", \n", " batch_correct / batch_total, step=step)\n", "\n", " if (batch_step + 1) % 100 == 0:\n", " print('Epoch [%d/%d], Step [%d/%d], Loss: %.4f' % (\n", " epoch + 1,\n", " experiment.get_parameter('epochs'),\n", " batch_step + 1,\n", " len(train_dataset) // experiment.get_parameter('batch_size'),\n", " loss.item()))\n", "\n", " # Log epoch accuracy to Comet.ml; step is each epoch\n", " experiment.log_metric(\"batch_accuracy\", correct / total, \n", " step=epoch, epoch=epoch)" ], "id": "8548f3f5-e1ef-4c35-b383-3dbe01941025", "execution_count": 12, "outputs": [ { "output_type": "stream", "text": [ "\n", "epoch: 0\n", ".........\n", "epoch: 1\n", ".........\n", "epoch: 2\n", ".........\n", "epoch: 3\n", ".........\n", "epoch: 4\n", ".........\n", "epoch: 5\n", ".........\n", "epoch: 6\n", ".........\n", "epoch: 7\n", ".........\n", "epoch: 8\n", ".........\n", "epoch: 9\n", "........." ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "92436469-0313-46ca-817e-6f0ed4c6a663" }, "source": [ "### Comet Confusion Matrix\n", "\n", "After the training loop, we can then test the test dataset with:\n", "\n", "```python\n", "confusion_matrix = experiment.create_confusion_matrix()\n", "for batch in batches:\n", " ...\n", " confusion_matrix.compute_matrix(actual, predicted, images=images)\n", "experiment.log_confusion_matrix(matrix=confusion_matrix)\n", "```\n", "and that will create a nice Confusion Matrix visualization in Comet with image examples.\n", "\n", "Here is the actual code:" ], "id": "92436469-0313-46ca-817e-6f0ed4c6a663" }, { "cell_type": "code", "metadata": { "id": "9d04c26b-94ad-4869-bbf7-7a817bad7dcb", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "9a3c00d9-ef1d-4b8a-9d36-137c834415ca" }, "source": [ "test_loader = torch.utils.data.DataLoader(\n", " dataset=test_dataset,\n", " batch_size=32,\n", " shuffle=False,\n", ")\n", "\n", "confusion_matrix = experiment.create_confusion_matrix()\n", "\n", "for batch_step, (images, labels) in enumerate(test_loader):\n", " print(\".\", end=\"\")\n", " images = Variable(images.view(\n", " -1,\n", " experiment.get_parameter('sequence_length'),\n", " experiment.get_parameter(\"input_size\")))\n", " labels = Variable(labels)\n", "\n", " outputs = rnn(images)\n", " _, predicted = torch.max(outputs.data, 1)\n", " \n", " confusion_matrix.compute_matrix(\n", " labels.data, \n", " predicted, \n", " images=images)\n", "\n", "experiment.log_confusion_matrix(\n", " matrix=confusion_matrix,\n", " title=\"MNIST Confusion Matrix, Epoch #%d\" % (epoch + 1),\n", " file_name=\"confusion-matrix-%03d.json\" % (epoch + 1),\n", ");\n" ], "id": "9d04c26b-94ad-4869-bbf7-7a817bad7dcb", "execution_count": 13, "outputs": [ { "output_type": "stream", "text": [ "........................................................................................................................................................................................................................................................................................................................." ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "B9Rl3sOHJt8u" }, "source": [ "Now, because we are in a Jupyter Notebook, we signal that the experiment has completed:" ], "id": "B9Rl3sOHJt8u" }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "0d993454-6ea1-4b61-9cba-8425ba2c7eba", "outputId": "5145f3dd-9e41-466c-ea56-923ea67306cd" }, "source": [ "experiment.end()" ], "id": "0d993454-6ea1-4b61-9cba-8425ba2c7eba", "execution_count": 14, "outputs": [ { "output_type": "stream", "text": [ "COMET INFO: ---------------------------\n", "COMET INFO: Comet.ml Experiment Summary\n", "COMET INFO: ---------------------------\n", "COMET INFO: Data:\n", "COMET INFO: display_summary_level : 1\n", "COMET INFO: url : https://www.comet.ml/dsblank/pytorch-confusion-matrix/819f19ee68ba4b91bab88421b795451d\n", "COMET INFO: Metrics [count] (min, max):\n", "COMET INFO: train_batch_accuracy [100] : (0.10000000149011612, 0.925000011920929)\n", "COMET INFO: train_loss [9] : (0.4030756652355194, 2.309687614440918)\n", "COMET INFO: Parameters:\n", "COMET INFO: batch_size : 120\n", "COMET INFO: epochs : 10\n", "COMET INFO: first_layer_units : 128\n", "COMET INFO: hidden_size : 128\n", "COMET INFO: input_size : 28\n", "COMET INFO: learning_rate : 0.01\n", "COMET INFO: num_classes : 10\n", "COMET INFO: num_layers : 2\n", "COMET INFO: sequence_length : 28\n", "COMET INFO: Uploads [count]:\n", "COMET INFO: confusion-matrix : 1\n", "COMET INFO: environment details : 1\n", "COMET INFO: filename : 1\n", "COMET INFO: images [1258] : 1258\n", "COMET INFO: installed packages : 1\n", "COMET INFO: model graph : 1\n", "COMET INFO: notebook : 1\n", "COMET INFO: os packages : 1\n", "COMET INFO: source_code : 1\n", "COMET INFO: ---------------------------\n", "COMET INFO: Uploading metrics, params, and assets to Comet before program termination (may take several seconds)\n", "COMET INFO: The Python SDK has 3600 seconds to finish before aborting...\n" ], "name": "stderr" } ] }, { "cell_type": "markdown", "metadata": { "id": "4JAptoP87RW1" }, "source": [ "Finally, we can explore the Confusion Matrix in the Comet UI. You can select the epoch by selecting the \"Confusion Matrix Name\" and click on a cell to see examples of that type." ], "id": "4JAptoP87RW1" }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 821 }, "id": "6af03e15-bc30-4d26-b743-ff425d789137", "outputId": "4f40a1ae-fd2f-4cba-f09f-6709216086b2" }, "source": [ "experiment.display(tab=\"confusion-matrix\")" ], "id": "6af03e15-bc30-4d26-b743-ff425d789137", "execution_count": 15, "outputs": [ { "output_type": "display_data", "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": { "tags": [] } } ] }, { "cell_type": "markdown", "metadata": { "id": "fkZRIi8CJ7-G" }, "source": [ "Clicking on a cell in the matrix should show up to 25 examples of that type of confusion or correct classification.\n", "\n", "For more information about Comet ML, please see:\n", "\n", "1. [Getting started in 30 seconds](https://www.comet.ml/docs/python-sdk/getting-started/)\n", "2. [Experiments](https://www.comet.ml/docs/python-sdk/experiment-overview/)\n", "3. [Working with Jupyter Notebooks](https://www.comet.ml/docs/python-sdk/JupyterNotebooks/)\n", "4. [Confusion Matrix](https://www.comet.ml/docs/python-sdk/Comet-Confusion-Matrix/)" ], "id": "fkZRIi8CJ7-G" } ] }