{
"cells": [
{
"cell_type": "code",
"execution_count": 59,
"metadata": {
"slideshow": {
"slide_type": "skip"
}
},
"outputs": [],
"source": [
"import matplotlib as mpl\n",
"import matplotlib.pyplot as plot\n",
"\n",
"%matplotlib inline\n",
"mpl.rcParams['figure.dpi'] = 125"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"Machine Learning 101\n",
"===========\n",
"\n",
"KharkivPy #17
\n",
"November 25th, 2017\n",
"\n",
"by Roman Podoliaka, Software Engineer at DataRobot\n",
"\n",
"twitter: @rpodoliaka
\n",
"email: roman.podoliaka@gmail.com
\n",
"blog: http://podoliaka.org
\n",
"slides: http://podoliaka.org/talks/"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Goal\n",
"\n",
"Take a close look at one of the simplest machine learning algorithms - *logistic regression* - to understand how it works internally and how it can be applied for the task of image classification.\n",
"\n",
"... and, hopefully, dispel at least some hype around machine learning. "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Problem\n",
"\n",
"Create an algorithm (also called a **model**) to classify whether images contain either a *dog* or a *cat*.\n",
"\n",
"![](\"catsvsdogs.jpg\")
\n",
"\n",
"Kaggle competition: https://www.kaggle.com/c/dogs-vs-cats\n",
"\n",
"There are much better algorithms for this task, but we'll stick to logistic regression *for the sake of simplicity*."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"## Input data\n",
"\n",
"25000 **labeled** examples of JPEG images (of **different sizes**) of cats and dogs, e.g.: *cat.0.jpg*, *cat.1.jpg*, ..., *dog.1.jpg*, *dog.2.jpg*...\n",
"\n",
"\n",
"
![](\"train/cat.0.jpg\")
\n",
"
![](\"train/dog.2.jpg\")
\n",
"
![](\"gradient.svg\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"## Logistic regression: gradient descent\n",
"\n",
"$$ \n",
" \\frac{\\partial J}{\\partial z} = \\frac{\\partial J}{\\partial \\hat{y}} \\frac{\\partial \\hat{y}}{\\partial z} =\n",
" \\hat{y} - y \n",
"$$\n",
"\n",
"$$ \n",
" \\frac{\\partial J}{\\partial w} =\n",
" \\frac{\\partial J}{\\partial \\hat{y}} \\frac{\\partial \\hat{y}}{\\partial z} \\frac{\\partial z}{\\partial w} =\n",
" \\frac{1}{m} X (\\hat{y} - y)^T\n",
"$$\n",
"\n",
"$$\n",
" \\frac{\\partial J}{\\partial b} =\n",
" \\frac{\\partial J}{\\partial \\hat{y}} \\frac{\\partial \\hat{y}}{\\partial z} \\frac{\\partial z}{\\partial b} =\n",
" \\frac{1}{m} \\sum_{i=1}^{m} (\\hat{y}_i - y_i)\n",
"$$\n",
"\n",
"The process of changing the weights is sometimes called **backward propagation of errors** (as we use the difference between predictions $ \\hat{y} $ and true labels $ y $ in order to modify the weights of inputs)."
]
},
{
"cell_type": "code",
"execution_count": 75,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"def update_weights(w, b, x, y_h, y, learning_rate):\n",
" m = x.shape[1]\n",
"\n",
" # calculate the values of partial derivatives\n",
" dz = y_h - y\n",
" dw = np.dot(x, dz.T) / m\n",
" db = np.sum(dz) / m\n",
"\n",
" # update the weights for the next iteration\n",
" w = w - learning_rate * dw\n",
" b = b - learning_rate * db\n",
"\n",
" return w, b"
]
},
{
"cell_type": "code",
"execution_count": 76,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"def logistic_regression(train_set, learning_rate=0.001, iterations=100, batch_size=64, callback=None):\n",
" # stack training examples as columns into a (n, m) matrix\n",
" x = np.column_stack(x[0] for x in train_set)\n",
" y = np.array([x[1] for x in train_set], dtype=np.float64).reshape(1, len(train_set))\n",
"\n",
" # split the whole training set into batches of equal size\n",
" n, m = x.shape\n",
" num_batches = m // batch_size + (1 if m % batch_size > 0 else 0)\n",
" x_batches = np.array_split(x, num_batches, axis=1)\n",
" y_batches = np.array_split(y, num_batches, axis=1)\n",
"\n",
" # run the gradient descent to learn w and b\n",
" w, b = initialize_weights(n)\n",
" for iteration in range(iterations):\n",
" j = 0\n",
"\n",
" for x_batch, y_batch in zip(x_batches, y_batches):\n",
" y_hat = hypothesis(w, b, x_batch)\n",
" w, b = update_weights(w, b, x_batch, y_hat, y_batch, learning_rate)\n",
"\n",
" j += cost(w, b, x_batch, y_batch) / num_batches\n",
"\n",
" if callback is not None:\n",
" callback(iteration=iteration, w=w, b=b, cost=j)\n",
"\n",
" return w, b"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Model performance evaluation\n",
"\n",
"It's important that we test the trained model on a separate set of examples it **has not seen before**, so that we ensure it generalizes well and does not **overfit** the training set.\n",
"\n",
"Training and test set examples should come from the same distribution, so that the model is trained on the same kind of data it will be used to make predictions on later.\n",
"\n",
"The performance metric we will use is **accuracy**, i.e. the percentage of correct predictions on a given data set."
]
},
{
"cell_type": "code",
"execution_count": 77,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"def predict(w, b, x, threshold=0.5):\n",
" y_h = hypothesis(w, b, x)\n",
"\n",
" return y_h >= threshold"
]
},
{
"cell_type": "code",
"execution_count": 78,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"def accuracy(w, b, data):\n",
" # stack examples as columns into a (n, m) matrix\n",
" x = np.column_stack(x[0] for x in data)\n",
" y = np.array([x[1] for x in data]).reshape(1, x.shape[1])\n",
"\n",
" # calculate the accuracy value as a percentage of correct predictions\n",
" correct_predictions = np.count_nonzero((y == predict(w, b, x)))\n",
" total_predictions = x.shape[1]\n",
"\n",
" return correct_predictions / total_predictions"
]
},
{
"cell_type": "code",
"execution_count": 85,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"def main(path, iterations=100, max_examples=25000, train_ratio=0.9):\n",
" # load examples and make sure they are uniformly distributed\n",
" examples = load_examples(path)\n",
" random.shuffle(examples)\n",
"\n",
" # split all the examples into train and test sets\n",
" m_train = int(max_examples * train_ratio)\n",
" train_set = examples[:m_train]\n",
" test_set = examples[m_train:]\n",
" \n",
" # monitor the progress of training\n",
" def progress(iteration, cost, w, b):\n",
" print('Iteration %d' % iteration)\n",
" print('\\tCost: %f' % cost)\n",
" if iteration % 10 == 0:\n",
" print('\\tTrain set accuracy: %s' % accuracy(w, b, train_set))\n",
" print('\\tTest set accuracy: %s' % accuracy(w, b, test_set)) \n",
"\n",
" # run the training process to learn model parameters\n",
" w, b = logistic_regression(train_set, iterations=iterations, callback=progress)\n",
" print('\\tFinal train set accuracy: %s' % accuracy(w, b, train_set))\n",
" print('\\tFinal test set accuracy: %s' % accuracy(w, b, test_set)) "
]
},
{
"cell_type": "code",
"execution_count": 86,
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Iteration 0\n",
"\tCost: 0.671012\n",
"\tTrain set accuracy: 0.5730222222222222\n",
"\tTest set accuracy: 0.5704\n",
"Iteration 1\n",
"\tCost: 0.661474\n",
"Iteration 2\n",
"\tCost: 0.657053\n",
"Iteration 3\n",
"\tCost: 0.654155\n",
"Iteration 4\n",
"\tCost: 0.651971\n",
"Iteration 5\n",
"\tCost: 0.650178\n",
"Iteration 6\n",
"\tCost: 0.648624\n",
"Iteration 7\n",
"\tCost: 0.647228\n",
"Iteration 8\n",
"\tCost: 0.645945\n",
"Iteration 9\n",
"\tCost: 0.644749\n",
"Iteration 10\n",
"\tCost: 0.643624\n",
"\tTrain set accuracy: 0.6156444444444444\n",
"\tTest set accuracy: 0.6032\n",
"Iteration 11\n",
"\tCost: 0.642558\n",
"Iteration 12\n",
"\tCost: 0.641543\n",
"Iteration 13\n",
"\tCost: 0.640573\n",
"Iteration 14\n",
"\tCost: 0.639644\n",
"Iteration 15\n",
"\tCost: 0.638753\n",
"Iteration 16\n",
"\tCost: 0.637895\n",
"Iteration 17\n",
"\tCost: 0.637070\n",
"Iteration 18\n",
"\tCost: 0.636274\n",
"Iteration 19\n",
"\tCost: 0.635505\n",
"Iteration 20\n",
"\tCost: 0.634762\n",
"\tTrain set accuracy: 0.6285333333333334\n",
"\tTest set accuracy: 0.6108\n",
"Iteration 21\n",
"\tCost: 0.634043\n",
"Iteration 22\n",
"\tCost: 0.633347\n",
"Iteration 23\n",
"\tCost: 0.632671\n",
"Iteration 24\n",
"\tCost: 0.632016\n",
"Iteration 25\n",
"\tCost: 0.631379\n",
"Iteration 26\n",
"\tCost: 0.630760\n",
"Iteration 27\n",
"\tCost: 0.630158\n",
"Iteration 28\n",
"\tCost: 0.629572\n",
"Iteration 29\n",
"\tCost: 0.629001\n",
"Iteration 30\n",
"\tCost: 0.628444\n",
"\tTrain set accuracy: 0.6368444444444444\n",
"\tTest set accuracy: 0.6152\n",
"Iteration 31\n",
"\tCost: 0.627900\n",
"Iteration 32\n",
"\tCost: 0.627369\n",
"Iteration 33\n",
"\tCost: 0.626851\n",
"Iteration 34\n",
"\tCost: 0.626344\n",
"Iteration 35\n",
"\tCost: 0.625848\n",
"Iteration 36\n",
"\tCost: 0.625363\n",
"Iteration 37\n",
"\tCost: 0.624888\n",
"Iteration 38\n",
"\tCost: 0.624422\n",
"Iteration 39\n",
"\tCost: 0.623966\n",
"Iteration 40\n",
"\tCost: 0.623518\n",
"\tTrain set accuracy: 0.6437777777777778\n",
"\tTest set accuracy: 0.6204\n",
"Iteration 41\n",
"\tCost: 0.623079\n",
"Iteration 42\n",
"\tCost: 0.622647\n",
"Iteration 43\n",
"\tCost: 0.622224\n",
"Iteration 44\n",
"\tCost: 0.621808\n",
"Iteration 45\n",
"\tCost: 0.621398\n",
"Iteration 46\n",
"\tCost: 0.620996\n",
"Iteration 47\n",
"\tCost: 0.620600\n",
"Iteration 48\n",
"\tCost: 0.620211\n",
"Iteration 49\n",
"\tCost: 0.619827\n",
"Iteration 50\n",
"\tCost: 0.619450\n",
"\tTrain set accuracy: 0.6501777777777777\n",
"\tTest set accuracy: 0.6184\n",
"Iteration 51\n",
"\tCost: 0.619078\n",
"Iteration 52\n",
"\tCost: 0.618711\n",
"Iteration 53\n",
"\tCost: 0.618350\n",
"Iteration 54\n",
"\tCost: 0.617993\n",
"Iteration 55\n",
"\tCost: 0.617642\n",
"Iteration 56\n",
"\tCost: 0.617295\n",
"Iteration 57\n",
"\tCost: 0.616953\n",
"Iteration 58\n",
"\tCost: 0.616615\n",
"Iteration 59\n",
"\tCost: 0.616281\n",
"Iteration 60\n",
"\tCost: 0.615951\n",
"\tTrain set accuracy: 0.6545777777777778\n",
"\tTest set accuracy: 0.6216\n",
"Iteration 61\n",
"\tCost: 0.615625\n",
"Iteration 62\n",
"\tCost: 0.615304\n",
"Iteration 63\n",
"\tCost: 0.614986\n",
"Iteration 64\n",
"\tCost: 0.614671\n",
"Iteration 65\n",
"\tCost: 0.614360\n",
"Iteration 66\n",
"\tCost: 0.614052\n",
"Iteration 67\n",
"\tCost: 0.613748\n",
"Iteration 68\n",
"\tCost: 0.613447\n",
"Iteration 69\n",
"\tCost: 0.613149\n",
"Iteration 70\n",
"\tCost: 0.612854\n",
"\tTrain set accuracy: 0.6580444444444444\n",
"\tTest set accuracy: 0.6224\n",
"Iteration 71\n",
"\tCost: 0.612562\n",
"Iteration 72\n",
"\tCost: 0.612273\n",
"Iteration 73\n",
"\tCost: 0.611986\n",
"Iteration 74\n",
"\tCost: 0.611703\n",
"Iteration 75\n",
"\tCost: 0.611422\n",
"Iteration 76\n",
"\tCost: 0.611143\n",
"Iteration 77\n",
"\tCost: 0.610867\n",
"Iteration 78\n",
"\tCost: 0.610594\n",
"Iteration 79\n",
"\tCost: 0.610323\n",
"Iteration 80\n",
"\tCost: 0.610054\n",
"\tTrain set accuracy: 0.6606666666666666\n",
"\tTest set accuracy: 0.624\n",
"Iteration 81\n",
"\tCost: 0.609788\n",
"Iteration 82\n",
"\tCost: 0.609524\n",
"Iteration 83\n",
"\tCost: 0.609262\n",
"Iteration 84\n",
"\tCost: 0.609002\n",
"Iteration 85\n",
"\tCost: 0.608744\n",
"Iteration 86\n",
"\tCost: 0.608489\n",
"Iteration 87\n",
"\tCost: 0.608235\n",
"Iteration 88\n",
"\tCost: 0.607983\n",
"Iteration 89\n",
"\tCost: 0.607733\n",
"Iteration 90\n",
"\tCost: 0.607485\n",
"\tTrain set accuracy: 0.6634222222222222\n",
"\tTest set accuracy: 0.6228\n",
"Iteration 91\n",
"\tCost: 0.607239\n",
"Iteration 92\n",
"\tCost: 0.606995\n",
"Iteration 93\n",
"\tCost: 0.606752\n",
"Iteration 94\n",
"\tCost: 0.606512\n",
"Iteration 95\n",
"\tCost: 0.606272\n",
"Iteration 96\n",
"\tCost: 0.606035\n",
"Iteration 97\n",
"\tCost: 0.605799\n",
"Iteration 98\n",
"\tCost: 0.605565\n",
"Iteration 99\n",
"\tCost: 0.605332\n",
"\tFinal train set accuracy: 0.6658222222222222\n",
"\tFinal test set accuracy: 0.6244\n"
]
}
],
"source": [
"main(\"train\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Summary\n",
"\n",
"* logistic regression is a basic model, that does not generalize well for this task and only achieves ~63% accuracy on the test set (e.g. the leader in this Kaggle competition achieved ~98.6%)\n",
"\n",
"* there are more sophisticated ML algorithms like **convolutional neural networks**, that perform better at image classification tasks\n",
"\n",
"* concepts like backward propagation of errors and gradient descent are used in other algorithms as well"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"## Summary (continued)\n",
"\n",
"* we did not have to explain a computer how to distinguish a cat from a dog - we just had to show it a lot of labeled examples\n",
"\n",
"* ML algorithms are generic - you can use the very same logistic regression implementation for many different problems - just provide a different data set!"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Links\n",
"\n",
"Andrew Ng:\n",
"\n",
"* Introduction to ML: https://www.coursera.org/learn/machine-learning\n",
"\n",
"* Deep Learning specialization: https://www.coursera.org/specializations/deep-learning\n",
"\n",
"Machine Learning and Data Analysis specialization (Yandex):\n",
"\n",
"* https://www.coursera.org/specializations/machine-learning-data-analysis"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Thanks!\n",
"\n",
"![](\"questions.jpg\")
\n",
"\n",
"twitter: @rpodoliaka
\n",
"email: roman.podoliaka@gmail.com
\n",
"blog: http://podoliaka.org
\n",
"slides: http://podoliaka.org/talks/"
]
}
],
"metadata": {
"celltoolbar": "Slideshow",
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}