{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Classification metrics\n",
"Author: Geraldine Klarenberg\n",
"\n",
"Based on the Google Machine Learning Crash Course"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"## Tresholds\n",
"In previous lessons, we have talked about using regression models to predict values. But sometimes we are interested in **classifying** things: \"spam\" vs \"not spam\", \"bark\" vs \"not barking\", etc. \n",
"\n",
"Logistic regression is a great tool to use in ML classification models. We can use the outputs from these models by defining **classification thresholds**. For instance, if our model tells us there's a probability of 0.8 that an email is spam (based on some characteristics), the model classifies it as such. If the probability estimate is less than 0.8, the model classifies it as \"not spam\". The threshold allows us to map a logistic regression value to a binary category (the prediction).\n",
"\n",
"Tresholds are problem-dependent, so they will have to be tuned for the specific problem you are dealing with.\n",
"\n",
"In this lesson we will look at metrics you can use to evaluate a classification model's predictions, and what changing the threshold does to your model and predictions."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## True, false, positive, negative...\n",
"\n",
"Now, we could simply look at \"accuracy\": the ratio of all correct predictions to all predictions. This is simple, intuitive and straightfoward. \n",
"\n",
"But there are some problems with this approach:\n",
"* This approach does not work well if there is (class) imbalance; situations where certain negative or positive values or outcomes are rare; \n",
"* and, most importantly: different kind of mistakes can have different costs..."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### The boy who cried wolf...\n",
"\n",
"We all know the story!\n",
"\n",
"\n",
"\n",
"For this example, we define \"there actually is a wolf\" as a positive class, and \"there is no wolf\" as a negative class. The predictions that a model makes can be true or false for both classes, generating 4 outcomes:\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This table is also called a *confusion matrix*.\n",
"\n",
"There are 2 metrics we can derive from these outcomes: precision and recall.\n",
"\n",
"## Precision\n",
"Precision asks the question what proportion of the positive predictions was actually correct?\n",
"\n",
"To calculate the precision of your model, take all true positives divided by *all* positive predictions:\n",
"$$\\text{Precision} = \\frac{TP}{TP+FP}$$\n",
"\n",
"Basically: **did the model cry 'wolf' too often or too little?**\n",
"\n",
"**NB** If your model produces no negative positives, the value of the precision is 1.0. Too many negative positives gives values greater than 1, too few gives values less than 1."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise\n",
"Calculate the precision of a model with the following outcomes\n",
"\n",
"true positives (TP): 1 | false positives (FP): 1 \n",
"-------|--------\n",
"**false negatives (FN): 8** | **true negatives (TN): 90** "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Recall\n",
"Recall tries to answer the question what proportion of actual positives was answered correctly?\n",
"\n",
"To calculate recall, divide all true positives by the true positives plus the false negatives:\n",
"$$\\text{Recall} = \\frac{TP}{TP+FN}$$\n",
"\n",
"Basically: **how many wolves that tried to get into the village did the model actually get?**\n",
"\n",
"**NB** If the model produces no false negative, recall equals 1.0"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise\n",
"For the same confusion matrix as above, calculate the recall."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Balancing precision and recall\n",
"To evaluate your model, should look at **both** precision and recall. They are often in tension though: improving one reduces the other.\n",
"Lowering the classification treshold improves recall (your model will call wolf at every little sound it hears) but will negatively affect precision (it will call wolf too often)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise\n",
"#### Part 1\n",
"Look at the outputs of a model that classifies incoming emails as \"spam\" or \"not spam\".\n",
"\n",
"\n",
"\n",
"The confusion matrix looks as follows\n",
"\n",
"true positives (TP): 8 | false positives (FP): 2 \n",
"-------|--------\n",
"**false negatives (FN): 3** | **true negatives (TN): 17** \n",
"\n",
"Calculate the precision and recall for this model.\n",
"\n",
"#### Part 2\n",
"Now see what happens to the outcomes (below) if we increase the threshold\n",
"\n",
"\n",
"\n",
"The confusion matrix looks as follows\n",
"\n",
"true positives (TP): 7 | false positives (FP): 4 \n",
"-------|--------\n",
"**false negatives (FN): 1** | **true negatives (TN): 18** \n",
"\n",
"Calculate the precision and recall again.\n",
"\n",
"**Compare the precision and recall from the first and second model. What do you notice?** "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Evaluate model performance\n",
"We can evaluate the performance of a classification model at all classification thresholds. For all different thresholds, calculate the *true positive rate* and the *false positive rate*. The true positive rate is synonymous with recall (and sometimes called *sensitivity*) and is thus calculated as\n",
"\n",
"$ TPR = \\frac{TP} {TP + FN} $\n",
"\n",
"False positive rate (sometimes called *specificity*) is:\n",
"\n",
"$ FPR = \\frac{FP} {FP + TN} $\n",
"\n",
"When you plot the pairs of TPR and FPR for all the different thresholds, you get a Receiver Operating Characteristics (ROC) curve. Below is a typical ROC curve.\n",
"\n",
"\n",
"\n",
"To evaluate the model, we look at the area under the curve (AUC). The AUC has a probabilistic interpretation: it represents the probability that a random positive (green) example is positioned to the right of a random negative (red) example.\n",
"\n",
"\n",
"\n",
"So if that AUC is 0.9, that's the probability the pair-wise prediction is correct. Below are a few visualizations of AUC results. On top are the distributions of the outcomes of the negative and positive outcomes at various thresholds. Below is the corresponding ROC.\n",
"\n",
" \n",
"\n",
"**This AUC suggests a perfect model** (which is suspicious!)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"**This is what most AUCs look like**. In this case, AUC = 0.7 means that there is 70% chance the model will be able to distinguish between positive and negative classes."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"**This is actually the worst case scenario.** This model has no discrimination capacity at all... "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Prediction bias\n",
"Logistic regression should be unbiased, meaning that the average of the predictions should be more or less equal to the average of the observations. **Prediction bias** is the difference between the average of the predictions and the average of the labels in a data set.\n",
"\n",
"This approach is not perfect, e.g. if your model almost always predicts the average there will not be much bias. However, if there **is** bias (\"significant nonzero bias\"), that means there is something something going on that needs to be checked, specifically that the model is wrong about the frequency of positive labels.\n",
"\n",
"Possible root causes of prediction bias are:\n",
"* Incomplete feature set\n",
"* Noisy data set\n",
"* Buggy pipeline\n",
"* Biased training sample\n",
"* Overly strong regularization\n",
"\n",
"### Buckets and prediction bias\n",
"For logistic regression, this process is a bit more involved, as the labels assigned to an examples are either 0 or 1. So you cannot accurately predict the prediction bias based on one example. You need to group data in \"buckets\" and examine the prediction bias on that. Prediction bias for logistic regression only makes sense when grouping enough examples together to be able to compare a predicted value (for example, 0.392) to observed values (for example, 0.394). \n",
"\n",
"You can create buckets by linearly breaking up the target predictions, or create quantiles. \n",
"\n",
"The plot below is a calibration plot. Each dot represents a bucket with 1000 values. On the x-axis we have the average value of the predictions for that bucket and on the y-axis the average of the actual observations. Note that the axes are on logarithmic scales.\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Coding\n",
"Recall the logistic regression model we made in the previous lesson. That was a perfect fit, so not that useful when we look at the metrics we just discussed.\n",
"\n",
"In the cloud plot with the sepal length and petal width plotted against each other, it is clear that the other two iris species are less separated. Let's use one of these as an example. We'll rework the example so we're classifying irises for being \"virginica\" or \"not virginica\". "
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" sepal_length(cm) | \n",
" sepal_width(cm) | \n",
" petal_length(cm) | \n",
" petal_width(cm) | \n",
" species_id | \n",
" species_name | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" 5.1 | \n",
" 3.5 | \n",
" 1.4 | \n",
" 0.2 | \n",
" 0 | \n",
" setosa | \n",
"
\n",
" \n",
" | 1 | \n",
" 4.9 | \n",
" 3.0 | \n",
" 1.4 | \n",
" 0.2 | \n",
" 0 | \n",
" setosa | \n",
"
\n",
" \n",
" | 2 | \n",
" 4.7 | \n",
" 3.2 | \n",
" 1.3 | \n",
" 0.2 | \n",
" 0 | \n",
" setosa | \n",
"
\n",
" \n",
" | 3 | \n",
" 4.6 | \n",
" 3.1 | \n",
" 1.5 | \n",
" 0.2 | \n",
" 0 | \n",
" setosa | \n",
"
\n",
" \n",
" | 4 | \n",
" 5.0 | \n",
" 3.6 | \n",
" 1.4 | \n",
" 0.2 | \n",
" 0 | \n",
" setosa | \n",
"
\n",
" \n",
"
\n",
"