{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Deep Learning Models -- A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.\n",
"- Author: Sebastian Raschka\n",
"- GitHub Repository: https://github.com/rasbt/deeplearning-models"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Custom DataLoader Example for PNG files"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Illustration of how we can efficiently iterate through custom (image) datasets. For this, suppose \n",
"- mnist_train, mnist_valid, and mnist_test are image folders you created with your own custom images\n",
"- mnist_train.csv, mnist_valid.csv, and mnist_test.csv are tables that store the image names with their associated class labels"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Sebastian Raschka \n",
"\n",
"CPython 3.7.1\n",
"IPython 7.2.0\n",
"\n",
"torch 1.0.1\n",
"pandas 0.24.0\n",
"numpy 1.15.4\n",
"matplotlib 3.0.2\n"
]
}
],
"source": [
"%load_ext watermark\n",
"%watermark -a 'Sebastian Raschka' -v -p torch,pandas,numpy,matplotlib"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 1) Inspecting the Dataset"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"%matplotlib inline\n",
"import matplotlib.pyplot as plt\n",
"from PIL import Image"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<matplotlib.image.AxesImage at 0x11ea383c8>"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADidJREFUeJzt3X+MVfWZx/HPI1J/UBJRIhBBKajrbgixOjGbYAiTRqKmEWtSUyQrK6SDscRtsn8sEqUjpGo2trv9qzqNhMG0tjXKiqRuqcZUqxvjiFhtWUCbWcoyGTQ2AdRYfjz7xxw2I8z9npl7z487PO9XQu6957nnnic3fOacc8+Pr7m7AMRzVt0NAKgH4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/ENTZVS7MzDidECiZu9to3tfSmt/MbjSz3Wb2vpmtaeWzAFTLmj2338wmSNoj6QZJ+yW9KWmpu/8xMQ9rfqBkVaz5r5P0vrv/yd3/Kunnkpa08HkAKtRK+C+R9Odhr/dn077AzLrMrM/M+lpYFoCCtfKD30ibFqdt1rt7j6Qeic1+oJ20subfL2nWsNczJR1orR0AVWkl/G9KusLMvmJmX5L0LUlbi2kLQNma3ux392NmtlrSryVNkLTR3f9QWGcAStX0ob6mFsY+P1C6Sk7yATB+EX4gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxBU00N0S5KZ9Us6LOm4pGPu3lFEUyiOWXrA1ilTpiTrU6dOTdZXrFiRrC9btqxhbebMmcl580aQfuihh5L1+++/v2HtjjvuSM47f/78ZH3Pnj3Jem9vb7J+/PjxZL0KLYU/0+nuHxXwOQAqxGY/EFSr4XdJ283sLTPrKqIhANVodbN/gbsfMLOLJf3GzP7b3V8Z/obsjwJ/GIA209Ka390PZI8HJW2RdN0I7+lx9w5+DATaS9PhN7NJZjb55HNJiyW9V1RjAMrVymb/NElbskNJZ0v6mbv/ZyFdASid5R1LLXRhZtUtLJDbbrutYe3mm29OznvXXXcV3U5ljh07lqynjrV3dnYm550zZ05TPZ105ZVXJusffPBBS5+f4u7pkzsyHOoDgiL8QFCEHwiK8ANBEX4gKMIPBMWhvjZw1VVXJev33ntvsn7nnXc2rJ133nlN9YTWcKgPQNsi/EBQhB8IivADQRF+ICjCDwRF+IGgirh7L3LkXVb75JNPJusXXHBBke0UaseOHcn6hx9+2LB2/fXXJ+edNGlSUz1VYXBwMFn/9NNPK+qkeaz5gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAojvMXYPLkycn6unXrkvUyj+M//fTTyXre7a9feOGFZP35559P1q+55pqGtUWLFiXnLdPrr7+erD/66KPJ+jvvvJOsDwwMjLmnqrHmB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgco/zm9lGSV+XdNDd52XTLpT0C0mzJfVLut3d/1Jem+3t3HPPTdYvuuiiUpe/efPmhrWVK1cm5z1x4kRLy161alWyvn79+oa1c845p6Vl59m2bVvD2tKlS5Pzjofr8Vs1mjX/Jkk3njJtjaSX3P0KSS9lrwGMI7nhd/dXJH18yuQlknqz572Sbi24LwAla3aff5q7D0hS9nhxcS0BqELp5/abWZekrrKXA2Bsml3zD5rZDEnKHg82eqO797h7h7t3NLksACVoNvxbJS3Pni+X9Fwx7QCoSm74zewpSf8l6W/MbL+ZrZT0iKQbzGyvpBuy1wDGkdx9fndvdED0awX3Mm6l7k0vSTt37kzW58yZ09Lyjxw50rCWdxzfLD2Ue09PT7K+fPnyZH3ChAnJekpe7xs2bEjWH3744Ya1o0ePNtXTmYQz/ICgCD8QFOEHgiL8QFCEHwiK8ANBcevuCmzfvj1Z7+zsTNanTJmSrC9YsKBhbfbs2cl516xJX5C5YsWKZL1MqUN1UvpyYeRjzQ8ERfiBoAg/EBThB4Ii/EBQhB8IivADQZm7V7cws+oWNo4sW7YsWU/dmrvdpS437u7uTs772GOPJeufffZZMy2d8dw9fZ12hjU/EBThB4Ii/EBQhB8IivADQRF+ICjCDwTF9fzjQN5w0eeff35FnZzu0KFDyXrqfgBbtmwpuh2MAWt+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwgq93p+M9so6euSDrr7vGxat6RvSzo5NvVad/9V7sK4nr8pb7/9drI+f/78ijo5XW9vb7Je533/oyryev5Nkm4cYfq/ufvV2b/c4ANoL7nhd/dXJH1cQS8AKtTKPv9qM/u9mW00s/R4UgDaTrPh/7GkuZKuljQg6QeN3mhmXWbWZ2Z9TS4LQAmaCr+7D7r7cXc/Ieknkq5LvLfH3TvcvaPZJgEUr6nwm9mMYS+/Iem9YtoBUJXcS3rN7ClJiyRNNbP9kr4naZGZXS3JJfVLWlVijwBKkBt+d186wuQnSugF49DevXvrbgFN4gw/ICjCDwRF+IGgCD8QFOEHgiL8QFAM0V2B6dOnJ+v33Xdfsn733Xcn62efXd8d2F977bVkfeHChRV1gpMYohtAEuEHgiL8QFCEHwiK8ANBEX4gKMIPBMUQ3RWYO3dusr569eqKOinepZde2nR93759RbeDMWDNDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBcZy/ABMnTkzW165dW+ryjxw50rD24IMPJuddt25dsj558uRkfdasWcn6Pffc07D2wAMPJOc9evRoso7WsOYHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaBy79tvZrMkbZY0XdIJST3u/iMzu1DSLyTNltQv6XZ3/0vOZ52R9+2//PLLk/Xdu3eXuvzu7u6GtQ0bNiTn7ezsTNZffPHFZloalXnz5iXru3btKm3ZZ7Ii79t/TNI/u/vfSvp7Sd8xs7+TtEbSS+5+haSXstcAxonc8Lv7gLvvyJ4flrRL0iWSlkjqzd7WK+nWspoEULwx7fOb2WxJX5X0hqRp7j4gDf2BkHRx0c0BKM+oz+03sy9LekbSd939kNmoditkZl2SupprD0BZRrXmN7OJGgr+T9392WzyoJnNyOozJB0caV5373H3DnfvKKJhAMXIDb8NreKfkLTL3X84rLRV0vLs+XJJzxXfHoCyjGazf4Gkf5D0rpntzKatlfSIpF+a2UpJ+yR9s5wW299ll11W6/I3bdrUsHbWWem/7zfddFPB3WC8yA2/u/9OUqMd/K8V2w6AqnCGHxAU4QeCIvxAUIQfCIrwA0ERfiAobt1dgM8//7zW5W/durVhLe/219dee23R7XzBq6++2rDW399f6rKRxpofCIrwA0ERfiAowg8ERfiBoAg/EBThB4LKvXV3oQs7Q2/dnTdEd+o4vCQtXry4yHYqNTg4mKzfcsstDWt9fX1FtwMVe+tuAGcgwg8ERfiBoAg/EBThB4Ii/EBQhB8IiuP8FVi4cGGy/vLLL1fUyek++eSTZH39+vXJ+uOPP56sHz58eMw9oTUc5weQRPiBoAg/EBThB4Ii/EBQhB8IivADQeUe5zezWZI2S5ou6YSkHnf/kZl1S/q2pA+zt65191/lfFbI4/xAlUZ7nH804Z8haYa77zCzyZLeknSrpNslHXH3R0fbFOEHyjfa8OeO2OPuA5IGsueHzWyXpEtaaw9A3ca0z29msyV9VdIb2aTVZvZ7M9toZlMazNNlZn1mxj2bgDYy6nP7zezLkn4r6fvu/qyZTZP0kSSXtEFDuwYrcj6DzX6gZIXt80uSmU2UtE3Sr939hyPUZ0va5u7zcj6H8AMlK+zCHjMzSU9I2jU8+NkPgSd9Q9J7Y20SQH1G82v/9ZJelfSuhg71SdJaSUslXa2hzf5+SauyHwdTn8WaHyhZoZv9RSH8QPm4nh9AEuEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiCo3Bt4FuwjSf8z7PXUbFo7atfe2rUvid6aVWRvl432jZVez3/aws363L2jtgYS2rW3du1Lordm1dUbm/1AUIQfCKru8PfUvPyUdu2tXfuS6K1ZtfRW6z4/gPrUveYHUJNawm9mN5rZbjN738zW1NFDI2bWb2bvmtnOuocYy4ZBO2hm7w2bdqGZ/cbM9maPIw6TVlNv3Wb2v9l3t9PMbq6pt1lm9rKZ7TKzP5jZP2XTa/3uEn3V8r1VvtlvZhMk7ZF0g6T9kt6UtNTd/1hpIw2YWb+kDnev/ZiwmS2UdETS5pOjIZnZv0r62N0fyf5wTnH3f2mT3ro1xpGbS+qt0cjS/6gav7siR7wuQh1r/uskve/uf3L3v0r6uaQlNfTR9tz9FUkfnzJ5iaTe7Hmvhv7zVK5Bb23B3QfcfUf2/LCkkyNL1/rdJfqqRR3hv0TSn4e93q/2GvLbJW03s7fMrKvuZkYw7eTISNnjxTX3c6rckZurdMrI0m3z3TUz4nXR6gj/SKOJtNMhhwXufo2kmyR9J9u8xej8WNJcDQ3jNiDpB3U2k40s/Yyk77r7oTp7GW6Evmr53uoI/35Js4a9ninpQA19jMjdD2SPByVt0dBuSjsZPDlIavZ4sOZ+/p+7D7r7cXc/IeknqvG7y0aWfkbST9392Wxy7d/dSH3V9b3VEf43JV1hZl8xsy9J+pakrTX0cRozm5T9ECMzmyRpsdpv9OGtkpZnz5dLeq7GXr6gXUZubjSytGr+7tptxOtaTvLJDmX8u6QJkja6+/crb2IEZjZHQ2t7aeiKx5/V2ZuZPSVpkYau+hqU9D1J/yHpl5IulbRP0jfdvfIf3hr0tkhjHLm5pN4ajSz9hmr87ooc8bqQfjjDD4iJM/yAoAg/EBThB4Ii/EBQhB8IivADQRF+ICjCDwT1f7GaMHSyoBEtAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"im = Image.open('mnist_train/1.png')\n",
"plt.imshow(im)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Array Dimensions (28, 28)\n",
"\n",
"[[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
" 0 0 0 0 0 0 0 0 0 0]\n",
" [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
" 0 0 0 0 0 0 0 0 0 0]\n",
" [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
" 0 0 0 0 0 0 0 0 0 0]\n",
" [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
" 0 0 0 0 0 0 0 0 0 0]\n",
" [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
" 0 0 0 0 0 0 0 0 0 0]\n",
" [ 0 0 0 0 0 0 0 0 0 0 0 0 1 18 38 136 227 255\n",
" 254 132 0 90 136 98 3 0 0 0]\n",
" [ 0 0 0 0 0 0 0 0 0 0 0 82 156 253 253 253 253 253\n",
" 253 249 154 219 253 253 35 0 0 0]\n",
" [ 0 0 0 0 0 0 0 0 0 40 150 244 253 253 253 253 253 253\n",
" 253 253 253 253 253 253 35 0 0 0]\n",
" [ 0 0 0 0 0 0 0 0 74 237 253 253 253 253 253 203 182 242\n",
" 253 253 253 253 253 230 25 0 0 0]\n",
" [ 0 0 0 0 0 0 0 13 200 253 253 253 168 164 91 14 64 246\n",
" 253 253 253 195 79 32 0 0 0 0]\n",
" [ 0 0 0 0 0 0 0 21 219 253 253 159 2 0 0 103 233 253\n",
" 253 253 177 10 0 0 0 0 0 0]\n",
" [ 0 0 0 0 0 0 0 0 171 253 253 147 0 1 155 250 253 253\n",
" 251 126 5 0 0 0 0 0 0 0]\n",
" [ 0 0 0 0 0 0 0 0 101 236 253 206 32 152 253 253 253 253\n",
" 130 0 0 0 0 0 0 0 0 0]\n",
" [ 0 0 0 0 0 0 0 0 0 91 253 253 253 253 253 253 241 113\n",
" 9 0 0 0 0 0 0 0 0 0]\n",
" [ 0 0 0 0 0 0 0 0 0 91 243 253 253 253 253 239 81 0\n",
" 0 0 0 0 0 0 0 0 0 0]\n",
" [ 0 0 0 0 0 0 0 0 0 0 207 253 253 253 253 158 0 0\n",
" 0 0 0 0 0 0 0 0 0 0]\n",
" [ 0 0 0 0 0 0 0 0 0 0 207 253 253 253 253 121 0 0\n",
" 0 0 0 0 0 0 0 0 0 0]\n",
" [ 0 0 0 0 0 0 0 0 24 145 249 253 253 253 253 194 0 0\n",
" 0 0 0 0 0 0 0 0 0 0]\n",
" [ 0 0 0 0 0 0 0 0 59 253 253 253 253 253 253 224 30 0\n",
" 0 0 0 0 0 0 0 0 0 0]\n",
" [ 0 0 0 0 0 0 0 5 181 253 253 241 114 240 253 253 136 5\n",
" 0 0 0 0 0 0 0 0 0 0]\n",
" [ 0 0 0 0 0 0 0 36 253 253 253 125 0 65 253 253 253 41\n",
" 0 0 0 0 0 0 0 0 0 0]\n",
" [ 0 0 0 0 0 0 0 67 253 253 253 29 2 138 253 253 253 41\n",
" 0 0 0 0 0 0 0 0 0 0]\n",
" [ 0 0 0 0 0 0 0 60 253 253 253 207 202 253 253 253 192 9\n",
" 0 0 0 0 0 0 0 0 0 0]\n",
" [ 0 0 0 0 0 0 0 5 183 253 253 253 253 253 253 230 52 0\n",
" 0 0 0 0 0 0 0 0 0 0]\n",
" [ 0 0 0 0 0 0 0 0 62 253 253 253 253 242 116 13 0 0\n",
" 0 0 0 0 0 0 0 0 0 0]\n",
" [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
" 0 0 0 0 0 0 0 0 0 0]\n",
" [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
" 0 0 0 0 0 0 0 0 0 0]\n",
" [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
" 0 0 0 0 0 0 0 0 0 0]]\n"
]
}
],
"source": [
"import numpy as np\n",
"\n",
"im_array = np.array(im)\n",
"print('Array Dimensions', im_array.shape)\n",
"print()\n",
"print(im_array)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>\n",
"<br>"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(256, 2)\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Class Label</th>\n",
" <th>File Name</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>5</td>\n",
" <td>0.png</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>8</td>\n",
" <td>1.png</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>8</td>\n",
" <td>2.png</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>0</td>\n",
" <td>3.png</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>9</td>\n",
" <td>4.png</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Class Label File Name\n",
"0 5 0.png\n",
"1 8 1.png\n",
"2 8 2.png\n",
"3 0 3.png\n",
"4 9 4.png"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_train = pd.read_csv('mnist_train.csv')\n",
"print(df_train.shape)\n",
"df_train.head()"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(256, 2)\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Class Label</th>\n",
" <th>File Name</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0</td>\n",
" <td>256.png</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>8</td>\n",
" <td>257.png</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>7</td>\n",
" <td>258.png</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>4</td>\n",
" <td>259.png</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>7</td>\n",
" <td>260.png</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Class Label File Name\n",
"0 0 256.png\n",
"1 8 257.png\n",
"2 7 258.png\n",
"3 4 259.png\n",
"4 7 260.png"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_valid = pd.read_csv('mnist_valid.csv')\n",
"print(df_valid.shape)\n",
"df_valid.head()"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(256, 2)\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Class Label</th>\n",
" <th>File Name</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>4</td>\n",
" <td>512.png</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>0</td>\n",
" <td>513.png</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>6</td>\n",
" <td>514.png</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>8</td>\n",
" <td>515.png</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>4</td>\n",
" <td>516.png</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Class Label File Name\n",
"0 4 512.png\n",
"1 0 513.png\n",
"2 6 514.png\n",
"3 8 515.png\n",
"4 4 516.png"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_test = pd.read_csv('mnist_test.csv')\n",
"print(df_test.shape)\n",
"df_test.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 2) Custom Dataset Class"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"import torch\n",
"from PIL import Image\n",
"from torch.utils.data import Dataset\n",
"import os\n",
"\n",
"\n",
"\n",
"class MyDataset(Dataset):\n",
"\n",
" def __init__(self, csv_path, img_dir, transform=None):\n",
" \n",
" df = pd.read_csv(csv_path)\n",
" self.img_dir = img_dir\n",
" self.img_names = df['File Name']\n",
" self.y = df['Class Label']\n",
" self.transform = transform\n",
"\n",
" def __getitem__(self, index):\n",
" img = Image.open(os.path.join(self.img_dir,\n",
" self.img_names[index]))\n",
" \n",
" if self.transform is not None:\n",
" img = self.transform(img)\n",
" \n",
" label = self.y[index]\n",
" return img, label\n",
"\n",
" def __len__(self):\n",
" return self.y.shape[0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 3) Custom Dataloader"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"from torchvision import transforms\n",
"from torch.utils.data import DataLoader\n",
"\n",
"\n",
"# Note that transforms.ToTensor()\n",
"# already divides pixels by 255. internally\n",
"\n",
"custom_transform = transforms.Compose([#transforms.Lambda(lambda x: x/255.), # not necessary\n",
" transforms.ToTensor()\n",
" ])\n",
"\n",
"train_dataset = MyDataset(csv_path='mnist_train.csv',\n",
" img_dir='mnist_train',\n",
" transform=custom_transform)\n",
"\n",
"train_loader = DataLoader(dataset=train_dataset,\n",
" batch_size=32,\n",
" shuffle=True, # want to shuffle the dataset\n",
" num_workers=4) # number processes/CPUs to use"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"<br>\n",
"<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4) Iterating Through the Dataset"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch: 1 | Batch index: 0 | Batch size: 32\n",
"Epoch: 1 | Batch index: 1 | Batch size: 32\n",
"Epoch: 1 | Batch index: 2 | Batch size: 32\n",
"Epoch: 1 | Batch index: 3 | Batch size: 32\n",
"Epoch: 1 | Batch index: 4 | Batch size: 32\n",
"Epoch: 1 | Batch index: 5 | Batch size: 32\n",
"Epoch: 1 | Batch index: 6 | Batch size: 32\n",
"Epoch: 1 | Batch index: 7 | Batch size: 32\n",
"Epoch: 2 | Batch index: 0 | Batch size: 32\n",
"Epoch: 2 | Batch index: 1 | Batch size: 32\n",
"Epoch: 2 | Batch index: 2 | Batch size: 32\n",
"Epoch: 2 | Batch index: 3 | Batch size: 32\n",
"Epoch: 2 | Batch index: 4 | Batch size: 32\n",
"Epoch: 2 | Batch index: 5 | Batch size: 32\n",
"Epoch: 2 | Batch index: 6 | Batch size: 32\n",
"Epoch: 2 | Batch index: 7 | Batch size: 32\n"
]
}
],
"source": [
"device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
"torch.manual_seed(0)\n",
"\n",
"num_epochs = 2\n",
"for epoch in range(num_epochs):\n",
"\n",
" for batch_idx, (x, y) in enumerate(train_loader):\n",
" \n",
" print('Epoch:', epoch+1, end='')\n",
" print(' | Batch index:', batch_idx, end='')\n",
" print(' | Batch size:', y.size()[0])\n",
" \n",
" x = x.to(device)\n",
" y = y.to(device)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"torch.Size([32, 1, 28, 28])\n"
]
}
],
"source": [
"print(x.shape)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"torch.Size([32, 784])\n"
]
}
],
"source": [
"x_image_as_vector = x.view(-1, 28*28)\n",
"print(x_image_as_vector.shape)"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([[[[0., 0., 0., ..., 0., 0., 0.],\n",
" [0., 0., 0., ..., 0., 0., 0.],\n",
" [0., 0., 0., ..., 0., 0., 0.],\n",
" ...,\n",
" [0., 0., 0., ..., 0., 0., 0.],\n",
" [0., 0., 0., ..., 0., 0., 0.],\n",
" [0., 0., 0., ..., 0., 0., 0.]]],\n",
"\n",
"\n",
" [[[0., 0., 0., ..., 0., 0., 0.],\n",
" [0., 0., 0., ..., 0., 0., 0.],\n",
" [0., 0., 0., ..., 0., 0., 0.],\n",
" ...,\n",
" [0., 0., 0., ..., 0., 0., 0.],\n",
" [0., 0., 0., ..., 0., 0., 0.],\n",
" [0., 0., 0., ..., 0., 0., 0.]]],\n",
"\n",
"\n",
" [[[0., 0., 0., ..., 0., 0., 0.],\n",
" [0., 0., 0., ..., 0., 0., 0.],\n",
" [0., 0., 0., ..., 0., 0., 0.],\n",
" ...,\n",
" [0., 0., 0., ..., 0., 0., 0.],\n",
" [0., 0., 0., ..., 0., 0., 0.],\n",
" [0., 0., 0., ..., 0., 0., 0.]]],\n",
"\n",
"\n",
" ...,\n",
"\n",
"\n",
" [[[0., 0., 0., ..., 0., 0., 0.],\n",
" [0., 0., 0., ..., 0., 0., 0.],\n",
" [0., 0., 0., ..., 0., 0., 0.],\n",
" ...,\n",
" [0., 0., 0., ..., 0., 0., 0.],\n",
" [0., 0., 0., ..., 0., 0., 0.],\n",
" [0., 0., 0., ..., 0., 0., 0.]]],\n",
"\n",
"\n",
" [[[0., 0., 0., ..., 0., 0., 0.],\n",
" [0., 0., 0., ..., 0., 0., 0.],\n",
" [0., 0., 0., ..., 0., 0., 0.],\n",
" ...,\n",
" [0., 0., 0., ..., 0., 0., 0.],\n",
" [0., 0., 0., ..., 0., 0., 0.],\n",
" [0., 0., 0., ..., 0., 0., 0.]]],\n",
"\n",
"\n",
" [[[0., 0., 0., ..., 0., 0., 0.],\n",
" [0., 0., 0., ..., 0., 0., 0.],\n",
" [0., 0., 0., ..., 0., 0., 0.],\n",
" ...,\n",
" [0., 0., 0., ..., 0., 0., 0.],\n",
" [0., 0., 0., ..., 0., 0., 0.],\n",
" [0., 0., 0., ..., 0., 0., 0.]]]])"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"x"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.1"
}
},
"nbformat": 4,
"nbformat_minor": 4
}