{
"nbformat": 4,
"nbformat_minor": 5,
"metadata": {
"colab": {
"name": "PytorchConfusionMatrixSimple.ipynb",
"provenance": [],
"collapsed_sections": []
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.5"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "0ca5f47a-43c1-47ba-9322-c48f645b0908"
},
"source": [
"# Pytorch Example with Confusion Matrix with Comet ML\n",
"\n",
"For this example, we will use Pytorch and create a interactive Confusion Matrix\n",
"in Comet ML. You'll need a Comet API key to log the Confusion Matrix, which is free for anyone.\n",
"\n",
"## Overview\n",
"\n",
"Our goal in this demonstration is to train a Pytorch model to categorize images of digits from the MNIST dataset, being able to see examples of each cell in a confusion matrix, like this:\n",
"\n",
"\n",
"\n",
"Comet provides a very easy way to make such confusion matrices. You can do that with a single command:\n",
"\n",
"```python\n",
"experiment.log_confusion_matrix(actual, predicted, images=images)\n",
"```\n",
"\n",
"where `actual` is the ground truth (given as vectors or labels), `predicted` is the ML's prediction (given as vectors or labels), and `images` is a list of image data.\n",
"\n",
"## End-to-End Example\n",
"\n",
"Let's explore a complete example from start to finish. \n",
"\n",
"First, we install the needed Python libraries:"
],
"id": "0ca5f47a-43c1-47ba-9322-c48f645b0908"
},
{
"cell_type": "code",
"metadata": {
"id": "dec1NN24O_MH"
},
"source": [
"%pip install comet_ml>=3.10.0 torch torchvision --quiet"
],
"id": "dec1NN24O_MH",
"execution_count": 1,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "9aY1iWM88iUa"
},
"source": [
"Now we import Comet:"
],
"id": "9aY1iWM88iUa"
},
{
"cell_type": "code",
"metadata": {
"id": "82048786-8d61-44c7-a6d2-0484a69779d0"
},
"source": [
"import comet_ml"
],
"id": "82048786-8d61-44c7-a6d2-0484a69779d0",
"execution_count": 2,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "65e82fc4-adda-4a7b-a84b-91d7cce363c4"
},
"source": [
"We can then make sure that our Comet API key is properly configured. The following command will give instructions if not:"
],
"id": "65e82fc4-adda-4a7b-a84b-91d7cce363c4"
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "406fa63d-8854-4812-a082-e3ad011ed3fc",
"outputId": "44a0adc9-12c7-4251-80ac-34bc8318644c"
},
"source": [
"comet_ml.init()"
],
"id": "406fa63d-8854-4812-a082-e3ad011ed3fc",
"execution_count": 3,
"outputs": [
{
"output_type": "stream",
"text": [
"COMET INFO: Comet API key is valid\n"
],
"name": "stderr"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6cc97390-6fb2-45eb-9b81-47f25ae73cc7"
},
"source": [
"Now, we import the rest of the Python libraries that we will need:"
],
"id": "6cc97390-6fb2-45eb-9b81-47f25ae73cc7"
},
{
"cell_type": "code",
"metadata": {
"id": "f11f4599-c2d6-4356-9936-aab28ade436b"
},
"source": [
"import torch\n",
"import torch.nn as nn\n",
"import torchvision.datasets as dsets\n",
"import torchvision.transforms as transforms\n",
"from torch.utils.data import SubsetRandomSampler\n",
"from torch.autograd import Variable"
],
"id": "f11f4599-c2d6-4356-9936-aab28ade436b",
"execution_count": 4,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "32a480fb-f40e-48ec-b27b-12a0cbe72efb"
},
"source": [
"## MNIST Dataset\n",
"\n"
],
"id": "32a480fb-f40e-48ec-b27b-12a0cbe72efb"
},
{
"cell_type": "markdown",
"metadata": {
"id": "b5b7c632-760c-490b-9d14-8e0e7dcbd277"
},
"source": [
"The first time this runs may take a few minutes to download, and then a couple more minutes to process:"
],
"id": "b5b7c632-760c-490b-9d14-8e0e7dcbd277"
},
{
"cell_type": "code",
"metadata": {
"id": "0cbb42ca-f73e-4de5-bb4a-d05dc45d6b47"
},
"source": [
"train_dataset = dsets.MNIST(\n",
" root='./data/',\n",
" train=True,\n",
" transform=transforms.ToTensor(),\n",
" download=True)\n",
"\n",
"test_dataset = dsets.MNIST(\n",
" root='./data/',\n",
" train=False,\n",
" transform=transforms.ToTensor())"
],
"id": "0cbb42ca-f73e-4de5-bb4a-d05dc45d6b47",
"execution_count": 5,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "563109fd-e169-4238-ac70-519a1c00c633"
},
"source": [
"## Create the Model\n",
"\n",
"We'll now write a function that will create the model.\n",
"\n",
"In this example, we'll take advantage of Comet's `Experiment` to get access to the hyperparameters via `experiment.get_parameter()`. This will be very handy when we later use Comet's Hyperparameter Optimizer to generate the Experiments.\n",
"\n",
"This function will actually return the three components of the model: the rnn, the criterion, and the optimizer."
],
"id": "563109fd-e169-4238-ac70-519a1c00c633"
},
{
"cell_type": "code",
"metadata": {
"id": "71cc8afc-0b6e-42e4-a0aa-f66a5b2fe006"
},
"source": [
"def build_model(experiment):\n",
" input_size = experiment.get_parameter(\"input_size\")\n",
" hidden_size = experiment.get_parameter(\"hidden_size\")\n",
" num_layers = experiment.get_parameter(\"num_layers\")\n",
" num_classes = experiment.get_parameter(\"num_classes\")\n",
" learning_rate = experiment.get_parameter(\"learning_rate\")\n",
"\n",
" class RNN(nn.Module):\n",
" def __init__(self, input_size, hidden_size, num_layers, num_classes):\n",
" super(RNN, self).__init__()\n",
" self.hidden_size = hidden_size\n",
" self.num_layers = num_layers\n",
" self.lstm = nn.LSTM(\n",
" input_size, \n",
" hidden_size, \n",
" num_layers, \n",
" batch_first=True)\n",
" self.fc = nn.Linear(hidden_size, num_classes)\n",
"\n",
" def forward(self, x):\n",
" # Set initial states\n",
" h0 = Variable(torch.zeros(self.num_layers, x.size(0), \n",
" self.hidden_size))\n",
" c0 = Variable(torch.zeros(self.num_layers, x.size(0), \n",
" self.hidden_size))\n",
"\n",
" # Forward propagate RNN\n",
" self.out, _ = self.lstm(x, (h0, c0))\n",
"\n",
" # Decode hidden state of last time step\n",
" out = self.fc(self.out[:, -1, :])\n",
" return out\n",
"\n",
" rnn = RNN(\n",
" input_size,\n",
" hidden_size,\n",
" num_layers,\n",
" num_classes,\n",
" )\n",
"\n",
" # Loss and Optimizer\n",
" criterion = nn.CrossEntropyLoss()\n",
" optimizer = torch.optim.Adam(rnn.parameters(), lr=learning_rate)\n",
"\n",
" return (rnn, criterion, optimizer)"
],
"id": "71cc8afc-0b6e-42e4-a0aa-f66a5b2fe006",
"execution_count": 6,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "53886701-7210-4895-b946-cfd7d788662b"
},
"source": [
"We'll call this function below, once we create an `Experiment`."
],
"id": "53886701-7210-4895-b946-cfd7d788662b"
},
{
"cell_type": "markdown",
"metadata": {
"id": "8114f369-7ba3-4783-93f4-c13ac1c70a4b"
},
"source": [
"## Train the Dataset on the Model\n",
"\n",
"Now we are ready to set up a Comet Experiment, and train the model.\n",
"\n",
"First, we can set all of the Hyperparameters of the model:"
],
"id": "8114f369-7ba3-4783-93f4-c13ac1c70a4b"
},
{
"cell_type": "code",
"metadata": {
"id": "c8b50453-3097-4a0b-9cf3-9b69f8b51df7"
},
"source": [
"hyper_params = {\n",
" \"epochs\": 10,\n",
" \"batch_size\": 120,\n",
" \"first_layer_units\": 128,\n",
" \"sequence_length\": 28,\n",
" \"input_size\": 28,\n",
" \"hidden_size\": 128,\n",
" \"num_layers\": 2,\n",
" \"num_classes\": 10,\n",
" \"learning_rate\": 0.01\n",
"}"
],
"id": "c8b50453-3097-4a0b-9cf3-9b69f8b51df7",
"execution_count": 7,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "062c6192-5c29-41ca-8841-5ceb82b7d978"
},
"source": [
"Next we create the experiment, and log the Hyperparameters:"
],
"id": "062c6192-5c29-41ca-8841-5ceb82b7d978"
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "174f3481-0798-4a11-acda-e651753a1caa",
"outputId": "b9038054-a7b9-483a-8a19-11eb7c2f114c"
},
"source": [
"experiment = comet_ml.Experiment(project_name=\"pytorch-confusion-matrix\")\n",
"experiment.log_parameters(hyper_params)"
],
"id": "174f3481-0798-4a11-acda-e651753a1caa",
"execution_count": 8,
"outputs": [
{
"output_type": "stream",
"text": [
"COMET WARNING: As you are running in a Jupyter environment, you will need to call `experiment.end()` when finished to ensure all metrics and code are logged before exiting.\n",
"COMET INFO: Experiment is live on comet.ml https://www.comet.ml/dsblank/pytorch-confusion-matrix/819f19ee68ba4b91bab88421b795451d\n",
"\n"
],
"name": "stderr"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4a5976cc-79e6-43ae-8301-8c39abf302a0"
},
"source": [
"We can now construct the model components:"
],
"id": "4a5976cc-79e6-43ae-8301-8c39abf302a0"
},
{
"cell_type": "code",
"metadata": {
"id": "8778542d-da44-4f7e-a01e-2e24d2dfec3e"
},
"source": [
"rnn, criterion, optimizer = build_model(experiment)"
],
"id": "8778542d-da44-4f7e-a01e-2e24d2dfec3e",
"execution_count": 9,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "423e17c2-e9ef-473c-b2ee-055669033984"
},
"source": [
"To make this demonstration go a little faster, we'll just use a sample of the items from the training set:"
],
"id": "423e17c2-e9ef-473c-b2ee-055669033984"
},
{
"cell_type": "code",
"metadata": {
"id": "56724a82-9a93-4af8-b0d2-6492364d4be9"
},
"source": [
"SAMPLE_SIZE = 1000"
],
"id": "56724a82-9a93-4af8-b0d2-6492364d4be9",
"execution_count": 10,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "acczEZl36Bml"
},
"source": [
"Now we can construct the loader:"
],
"id": "acczEZl36Bml"
},
{
"cell_type": "code",
"metadata": {
"id": "761fe054-c539-4883-a58f-ad46a0108fcb"
},
"source": [
"sampler = SubsetRandomSampler(list(range(SAMPLE_SIZE)))\n",
"train_loader = torch.utils.data.DataLoader(\n",
" dataset=train_dataset,\n",
" batch_size=experiment.get_parameter('batch_size'),\n",
" sampler=sampler,\n",
" #shuffle=True, # can't use shuffle with sampler\n",
")"
],
"id": "761fe054-c539-4883-a58f-ad46a0108fcb",
"execution_count": 11,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "EaVEhHpJC2nY"
},
"source": [
"Instead, if you would rather train on the entire dataset, you can:\n",
"\n",
"```python\n",
"train_loader = torch.utils.data.DataLoader(\n",
" dataset=train_dataset,\n",
" batch_size=experiment.get_parameter('batch_size'),\n",
" shuffle=True,\n",
")\n",
"```"
],
"id": "EaVEhHpJC2nY"
},
{
"cell_type": "markdown",
"metadata": {
"id": "RZvVJzzE6FgW"
},
"source": [
"Now we can train the model. Some items to note:\n",
"\n",
"1. We use `experiment.train()` to provide the context for logged metrics\n",
"2. We collect the actual, predicted, and images for each batch\n",
"3. At the end of the epoch, compute and log the confusion matrix"
],
"id": "RZvVJzzE6FgW"
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "8548f3f5-e1ef-4c35-b383-3dbe01941025",
"tags": [],
"outputId": "5df2bd81-5202-46b4-bb09-d3af5f07d5d3"
},
"source": [
"with experiment.train():\n",
" step = 0\n",
" for epoch in range(experiment.get_parameter('epochs')):\n",
" print(\"\\nepoch:\", epoch)\n",
" correct = 0\n",
" total = 0\n",
" for batch_step, (images, labels) in enumerate(train_loader):\n",
" print(\".\", end=\"\")\n",
" images = Variable(images.view(\n",
" -1,\n",
" experiment.get_parameter('sequence_length'),\n",
" experiment.get_parameter(\"input_size\")))\n",
"\n",
" labels = Variable(labels)\n",
"\n",
" # Forward + Backward + Optimize\n",
" optimizer.zero_grad()\n",
" outputs = rnn(images)\n",
" loss = criterion(outputs, labels)\n",
" loss.backward()\n",
" optimizer.step()\n",
"\n",
" # Compute train accuracy\n",
" _, predicted = torch.max(outputs.data, 1)\n",
" batch_total = labels.size(0)\n",
" total += batch_total\n",
"\n",
" batch_correct = (predicted == labels.data).sum()\n",
" correct += batch_correct\n",
"\n",
" # Log batch_accuracy to Comet.ml; step is each batch\n",
" step += 1\n",
" experiment.log_metric(\"batch_accuracy\", \n",
" batch_correct / batch_total, step=step)\n",
"\n",
" if (batch_step + 1) % 100 == 0:\n",
" print('Epoch [%d/%d], Step [%d/%d], Loss: %.4f' % (\n",
" epoch + 1,\n",
" experiment.get_parameter('epochs'),\n",
" batch_step + 1,\n",
" len(train_dataset) // experiment.get_parameter('batch_size'),\n",
" loss.item()))\n",
"\n",
" # Log epoch accuracy to Comet.ml; step is each epoch\n",
" experiment.log_metric(\"batch_accuracy\", correct / total, \n",
" step=epoch, epoch=epoch)"
],
"id": "8548f3f5-e1ef-4c35-b383-3dbe01941025",
"execution_count": 12,
"outputs": [
{
"output_type": "stream",
"text": [
"\n",
"epoch: 0\n",
".........\n",
"epoch: 1\n",
".........\n",
"epoch: 2\n",
".........\n",
"epoch: 3\n",
".........\n",
"epoch: 4\n",
".........\n",
"epoch: 5\n",
".........\n",
"epoch: 6\n",
".........\n",
"epoch: 7\n",
".........\n",
"epoch: 8\n",
".........\n",
"epoch: 9\n",
"........."
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "92436469-0313-46ca-817e-6f0ed4c6a663"
},
"source": [
"### Comet Confusion Matrix\n",
"\n",
"After the training loop, we can then test the test dataset with:\n",
"\n",
"```python\n",
"confusion_matrix = experiment.create_confusion_matrix()\n",
"for batch in batches:\n",
" ...\n",
" confusion_matrix.compute_matrix(actual, predicted, images=images)\n",
"experiment.log_confusion_matrix(matrix=confusion_matrix)\n",
"```\n",
"and that will create a nice Confusion Matrix visualization in Comet with image examples.\n",
"\n",
"Here is the actual code:"
],
"id": "92436469-0313-46ca-817e-6f0ed4c6a663"
},
{
"cell_type": "code",
"metadata": {
"id": "9d04c26b-94ad-4869-bbf7-7a817bad7dcb",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "9a3c00d9-ef1d-4b8a-9d36-137c834415ca"
},
"source": [
"test_loader = torch.utils.data.DataLoader(\n",
" dataset=test_dataset,\n",
" batch_size=32,\n",
" shuffle=False,\n",
")\n",
"\n",
"confusion_matrix = experiment.create_confusion_matrix()\n",
"\n",
"for batch_step, (images, labels) in enumerate(test_loader):\n",
" print(\".\", end=\"\")\n",
" images = Variable(images.view(\n",
" -1,\n",
" experiment.get_parameter('sequence_length'),\n",
" experiment.get_parameter(\"input_size\")))\n",
" labels = Variable(labels)\n",
"\n",
" outputs = rnn(images)\n",
" _, predicted = torch.max(outputs.data, 1)\n",
" \n",
" confusion_matrix.compute_matrix(\n",
" labels.data, \n",
" predicted, \n",
" images=images)\n",
"\n",
"experiment.log_confusion_matrix(\n",
" matrix=confusion_matrix,\n",
" title=\"MNIST Confusion Matrix, Epoch #%d\" % (epoch + 1),\n",
" file_name=\"confusion-matrix-%03d.json\" % (epoch + 1),\n",
");\n"
],
"id": "9d04c26b-94ad-4869-bbf7-7a817bad7dcb",
"execution_count": 13,
"outputs": [
{
"output_type": "stream",
"text": [
"........................................................................................................................................................................................................................................................................................................................."
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "B9Rl3sOHJt8u"
},
"source": [
"Now, because we are in a Jupyter Notebook, we signal that the experiment has completed:"
],
"id": "B9Rl3sOHJt8u"
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "0d993454-6ea1-4b61-9cba-8425ba2c7eba",
"outputId": "5145f3dd-9e41-466c-ea56-923ea67306cd"
},
"source": [
"experiment.end()"
],
"id": "0d993454-6ea1-4b61-9cba-8425ba2c7eba",
"execution_count": 14,
"outputs": [
{
"output_type": "stream",
"text": [
"COMET INFO: ---------------------------\n",
"COMET INFO: Comet.ml Experiment Summary\n",
"COMET INFO: ---------------------------\n",
"COMET INFO: Data:\n",
"COMET INFO: display_summary_level : 1\n",
"COMET INFO: url : https://www.comet.ml/dsblank/pytorch-confusion-matrix/819f19ee68ba4b91bab88421b795451d\n",
"COMET INFO: Metrics [count] (min, max):\n",
"COMET INFO: train_batch_accuracy [100] : (0.10000000149011612, 0.925000011920929)\n",
"COMET INFO: train_loss [9] : (0.4030756652355194, 2.309687614440918)\n",
"COMET INFO: Parameters:\n",
"COMET INFO: batch_size : 120\n",
"COMET INFO: epochs : 10\n",
"COMET INFO: first_layer_units : 128\n",
"COMET INFO: hidden_size : 128\n",
"COMET INFO: input_size : 28\n",
"COMET INFO: learning_rate : 0.01\n",
"COMET INFO: num_classes : 10\n",
"COMET INFO: num_layers : 2\n",
"COMET INFO: sequence_length : 28\n",
"COMET INFO: Uploads [count]:\n",
"COMET INFO: confusion-matrix : 1\n",
"COMET INFO: environment details : 1\n",
"COMET INFO: filename : 1\n",
"COMET INFO: images [1258] : 1258\n",
"COMET INFO: installed packages : 1\n",
"COMET INFO: model graph : 1\n",
"COMET INFO: notebook : 1\n",
"COMET INFO: os packages : 1\n",
"COMET INFO: source_code : 1\n",
"COMET INFO: ---------------------------\n",
"COMET INFO: Uploading metrics, params, and assets to Comet before program termination (may take several seconds)\n",
"COMET INFO: The Python SDK has 3600 seconds to finish before aborting...\n"
],
"name": "stderr"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4JAptoP87RW1"
},
"source": [
"Finally, we can explore the Confusion Matrix in the Comet UI. You can select the epoch by selecting the \"Confusion Matrix Name\" and click on a cell to see examples of that type."
],
"id": "4JAptoP87RW1"
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 821
},
"id": "6af03e15-bc30-4d26-b743-ff425d789137",
"outputId": "4f40a1ae-fd2f-4cba-f09f-6709216086b2"
},
"source": [
"experiment.display(tab=\"confusion-matrix\")"
],
"id": "6af03e15-bc30-4d26-b743-ff425d789137",
"execution_count": 15,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/html": [
"\n",
" \n",
" "
],
"text/plain": [
""
]
},
"metadata": {
"tags": []
}
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fkZRIi8CJ7-G"
},
"source": [
"Clicking on a cell in the matrix should show up to 25 examples of that type of confusion or correct classification.\n",
"\n",
"For more information about Comet ML, please see:\n",
"\n",
"1. [Getting started in 30 seconds](https://www.comet.ml/docs/python-sdk/getting-started/)\n",
"2. [Experiments](https://www.comet.ml/docs/python-sdk/experiment-overview/)\n",
"3. [Working with Jupyter Notebooks](https://www.comet.ml/docs/python-sdk/JupyterNotebooks/)\n",
"4. [Confusion Matrix](https://www.comet.ml/docs/python-sdk/Comet-Confusion-Matrix/)"
],
"id": "fkZRIi8CJ7-G"
}
]
}