Measure with yardstick

In Lesson 4 you handed the risk committee a number: the loan-default model scores 0.648 accuracy, cross-validated, give or take a little. It felt solid. Then someone at the table tries something lazy: a "model" that simply stamps no default on every single applicant. On this loan book that scores about 63% too, higher than a fair few real models, and it catches exactly zero defaulters. If a do-nothing rule can match your model on accuracy, then accuracy was never the yardstick that mattered.

This lesson is about choosing the yardstick on purpose. The yardstick package gives you a whole toolbox of metrics; the skill is knowing which one answers your question.

By the end you will be able to:

• Read a confusion matrix and say why accuracy alone can flatter a useless model
• Compute precision, recall and specificity, and pick the one that matches the business cost
• Tell a hard-class metric from a probability metric, and read an ROC curve and its AUC
• Bundle your chosen metrics into a metric set and read them across every resample

Prerequisites: you can run R and use the |> pipe, and you can bundle a recipe and a model into a workflow and resample it (Lesson 4).

When accuracy flatters a useless model

Here is the lender's loan book again, rebuilt right here so this page runs on its own. Each row is one applicant; defaulted is what we are trying to predict. We set the factor levels so that yes (a default) comes first, because that is the outcome the bank cares about catching.

Of 240 applicants, 89 defaulted and 151 did not. The classes are lopsided: only about 37% of applicants are defaulters. That imbalance is exactly what lets accuracy lie. Watch what the do-nothing rule scores:

Predicting "no default" for everybody is right 62.9% of the time, purely because most applicants really do repay. It is a model with zero skill and a respectable-looking score. Now fit a genuine model, a logistic regression, on a training split and measure its accuracy on held-out applicants:

The real model scores 0.656. It beat the do-nothing rule (0.629) by barely two points. If you reported only that number, the committee would think the model is roughly as good as guessing "no" every time. Accuracy is not wrong here, it is just answering the wrong question. To see what the model is really doing, we have to look at which applicants it gets right and wrong.

The confusion matrix

A classifier can be right or wrong in two different ways each, so there are four possible outcomes. Laying them in a 2-by-2 table is called a confusion matrix. yardstick::conf_mat() builds it from the truth column and the predicted-class column:

Read it in plain loan terms. Down the rows is what the model predicted; across the columns is the truth:

• 7 applicants it flagged as "yes" who really did default. A caught default. We call this a true positive (TP).
• 5 it flagged as "yes" who actually repaid. A false alarm, a false positive (FP): the bank hassles or declines a good customer.
• 16 it cleared as "no" who went on to default. A missed default, a false negative (FN): the bank makes a loan it never gets back.
• 33 it cleared as "no" who really repaid. A correct approval, a true negative (TN).

Those two mistakes are not equal. A false positive costs a little goodwill and a lost loan. A false negative, the 16 missed defaults, costs real money on loans that go bad. The whole reason accuracy misled us is that it treats all four cells as interchangeable and adds up only the diagonal.

The widget below is a small generic classifier so you can feel how the four cells work. Slide the threshold and watch every count re-tally. Your loan model's own matrix above is just one snapshot of this, taken at the 0.5 cutoff.

Which mistake costs the bank?

The loan model made two kinds of error: 5 false positives (good applicants it flagged) and 16 false negatives (defaulters it cleared). Which one is the expensive mistake, the one that actually loses money on a bad loan?

Precision and recall

Once you have the four cells, you can ask two very different questions, and each has its own metric. Let \(TP\), \(FP\) and \(FN\) be the true-positive, false-positive and false-negative counts from the confusion matrix.

Precision answers "when the model flags an applicant as a default, how often is it right?"

\[ \text{precision} = \frac{TP}{TP + FP} = \frac{7}{7 + 5} = 0.583 \]

Recall (also called sensitivity) answers the opposite: "of all the applicants who really defaulted, how many did the model catch?"

\[ \text{recall} = \frac{TP}{TP + FN} = \frac{7}{7 + 16} = 0.304 \]

There is a third that watches the other class. Specificity answers "of all the applicants who really repaid, how many did the model correctly clear?", using \(TN\) (true negatives) and \(FP\):

\[ \text{specificity} = \frac{TN}{TN + FP} = \frac{33}{33 + 5} = 0.868 \]

yardstick has a function for each, and they read the same two columns conf_mat did:

Now the model's real behaviour is visible. Its specificity is 0.868, so it clears good applicants well, and its precision is 0.583, so a bit more than half its flags are genuine. But its recall is only 0.304: it catches under a third of the applicants who actually default. That failure was completely hidden inside the 0.656 accuracy. The chart makes the gap obvious, accuracy and specificity stand tall while recall sits on the floor.

Compute the metric that matters

For a lender, the number that keeps the risk team awake is recall: the share of real defaulters the model actually catches. Complete the call so it computes recall on the test predictions.

recall(preds, truth = defaulted, estimate = .pred_class)
#> # A tibble: 1 x 3
#>   .metric .estimator .estimate
#>   <chr>   <chr>          <dbl>
#> 1 recall  binary         0.304

F1: balancing precision and recall

Sometimes you want a single score that rewards a model only when precision and recall are both decent. Averaging them the usual way is too forgiving: a model with precision 0.95 and recall 0.05 would average to a cosy 0.50 while catching almost nothing. The F1 score fixes this by taking the harmonic mean, which is dragged down hard by the smaller of the two:

\[ F_1 = \frac{2 \cdot \text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}} = \frac{2 \cdot 0.583 \cdot 0.304}{0.583 + 0.304} = 0.400 \]

Because recall (0.304) is so weak, F1 lands at 0.400, well below the halfway point between the two. yardstick computes it with f_meas():

F1 is the go-to single number when the positive class is rare and both error types matter, which is precisely the loan-default situation. It refuses to be fooled by a high score on one side alone.

Reading precision, recall and F1

The loan model scores precision 0.58, recall 0.30, and F1 0.40. In plain terms, what is it doing?

Every hard-class metric hides a threshold

Here is something the metrics so far quietly assumed. Accuracy, precision, recall and F1 all read .pred_class, the predicted label. But that label was not handed down from on high: it was made by cutting the predicted probability at 0.5. Score above 0.5, call it "yes"; below, call it "no". Move that cutoff and every one of those metrics changes.

Slide the threshold in the widget below. Push it down and the model flags more applicants: recall climbs (you catch more defaulters) but precision falls (more false alarms). Push it up and the trade reverses. The single operating point traces out a curve as you sweep, and that curve is the subject of the next step.

The ROC curve and AUC

If the "best" threshold depends on the business, it would be useful to have a metric that does not commit to any single one. That is the ROC curve. As you sweep the threshold from high to low, you plot the true-positive rate (recall, the defaulters you catch) against the false-positive rate (the good applicants you wrongly flag). Each threshold is one point; the whole sweep is the curve you just watched form.

The single number that summarizes the whole curve is the AUC, the area under it. It has a clean interpretation. If \(\hat{p}_+\) is the model's predicted default-probability for a randomly chosen real defaulter and \(\hat{p}_-\) the same for a randomly chosen non-defaulter, then

\[ \text{AUC} = P(\hat{p}_+ > \hat{p}_-) \]

the probability that the model scores a true defaulter higher than a true non-defaulter. An AUC of 1.0 is a perfect ranker; 0.5 is a coin flip. Crucially, it never fixes a threshold, so it measures how well the model ranks applicants by risk, not how it labels them.

Here is the vital yardstick detail. A hard-class metric like accuracy reads the label column .pred_class. A probability metric like roc_auc must read the probability column, .pred_yes, because it needs a score to sweep, not a pre-made label:

The loan model's AUC is 0.625: better than a coin flip, but only modestly. Notice we passed .pred_yes, not .pred_class. Hand roc_auc the label column instead and it errors, because a fixed 0/1 label has no ranking left to sweep.

Which column does AUC need?

You want to score the loan model with roc_auc. Which column does yardstick need, and why?

Bundle metrics into a metric set

You rarely want one metric in isolation; you want a small dashboard of them. metric_set() bundles any mix of metrics into a single function you can call once. Because our set mixes hard-class metrics (accuracy, sensitivity, specificity) with a probability metric (roc_auc), we hand the resulting function both the class column and the probability column, and it routes each metric to the one it needs:

One call, four numbers, chosen on purpose: overall accuracy, how many defaulters we catch (sens), how many good applicants we clear (spec), and how well we rank risk (roc_auc). That loan_metrics object is now a reusable yardstick you can point at any set of predictions.

Read the metrics across every resample

A metric set truly earns its keep when you combine it with the resampling from Lesson 4. One test split gave the numbers above, but you already know one split is a roll of the dice. So hand your metric set to fit_resamples, which scores the workflow on every fold, and collect_metrics averages each metric with its standard error. This is the same loan workflow you built in Lesson 4, now measured with the yardstick we chose:

Now the committee gets the honest story, all four metrics at once, each with a mean and a spread. Accuracy holds at 0.648 (exactly the figure from Lesson 4). But look at sensitivity: 0.258. Averaged across five folds, the model catches barely a quarter of defaulters. That is the finding accuracy was hiding all along, and now it comes with a standard error so you know how firm it is.

The standard error matters because the per-fold scores swing. Here are the AUC values fold by fold; the 0.0481 standard error is summarizing that spread, from a weak 0.52 up to a much healthier 0.78:

Build the report's metric set

The risk team has decided the report will lead with recall (catching defaulters) and back it with AUC (how well the model ranks risk). Complete the metric set so it bundles exactly those two, then it is scored on the test predictions.

report_metrics <- metric_set(recall, roc_auc)
report_metrics(preds, truth = defaulted, estimate = .pred_class, .pred_yes)
#> # A tibble: 2 x 3
#>   .metric .estimator .estimate
#>   <chr>   <chr>          <dbl>
#> 1 recall  binary         0.304
#> 2 roc_auc binary         0.625

And it measures regression too

Everything so far has been classification, but yardstick uses the exact same grammar for a numeric outcome. Predicting a house price instead of a yes/no label? The metrics change to rmse (root mean squared error, in the units of the outcome), mae (mean absolute error) and rsq (R-squared, the share of variance explained), but metric_set and the truth-and-estimate call are identical:

Same idea, different yardsticks: on average the price predictions are off by about 19,400 dollars (mae), and the model explains 84% of the variation in price (rsq). You will lean on these in the regression course; the point here is that the yardstick workflow you just learned, choose, bundle, resample, carries straight over.

References

A few authoritative places to take this further:

• yardstick package documentation (tidymodels) - the reference for every metric used here, with its formula and required columns.
• yardstick: Metric types - the class-metric vs probability-metric distinction, and why a mixed metric set needs both columns.
• Tidy Modeling with R, ch. 9: Judging model effectiveness - Kuhn and Silge on choosing metrics and reading them across resamples, end to end.
• Fawcett (2006), An introduction to ROC analysis - the canonical, readable explanation of ROC curves and AUC.
• An Introduction to Statistical Learning, ch. 4 (free PDF) - sensitivity, specificity and the ROC curve in the wider context of classification.

Lesson 5 complete

You no longer report a single accuracy number and hope. You read the confusion matrix, choose precision, recall, specificity or F1 to match the actual cost of each mistake, tell a hard-class metric from a threshold-free probability metric like AUC, and bundle your choices into a metric set you can read across every resample, with an honest spread.

The loan model told you as much: 0.65 accuracy looked fine until recall of 0.26 revealed it catches barely a quarter of defaulters. Knowing that is the difference between a model that scores well and one that does its job.

Next, Lesson 6: Tune with the tune package. A weak recall is often a tuning problem. You will search over a model's hyperparameters with resampling, use a metric set exactly like this one to score every candidate, and let the data pick the settings that lift the number you actually care about.