Tune with the tune package
In Lesson 5 the loan-default model left us with an uncomfortable number: a cross-validated recall of about 0.26, meaning it caught barely a quarter of the applicants who actually defaulted. One honest response is to reach for a more flexible model. A decision tree can carve the loan book into regions of high and low risk that a straight-line logistic regression cannot.
But a tree comes with settings you have to choose: how deep it may grow, how hard it gets pruned back. Set them wrong and the tree either memorizes the training rows (and stumbles on new applicants) or collapses to a useless stump. The picture below is that whole problem in one shape. As a model grows more complex, error on the training data keeps falling, but error on fresh data traces a U: too simple on the left, too flexible on the right, with a sweet spot in between. Drag the slider and watch the two curves separate.
Tuning is how you find that sweet spot on purpose instead of guessing. The tune package searches a menu of candidate settings, scores each one honestly with the resampling from Lesson 4, and hands you the best.
By the end of this lesson you will be able to:
- Tell a parameter the model learns from a hyperparameter you set, and name a tree's two knobs
- Mark hyperparameters with
tune()and lay out a grid of candidate settings - Run a grid search across cross-validation folds with
tune_grid(), scored by a metric set - Read the results, select the best settings, finalize the workflow, and confirm it once on a sealed test set
Prerequisites: you can run R and use the |> pipe, and from earlier lessons you can bundle a recipe and model into a workflow (Lesson 3), score across cross-validation folds (Lesson 4), and [measure with roc_auc and a metric set](Measure-with-yardstick.html) (Lesson 5). We work from that same lender's loan book here.
A model with knobs to turn
Logistic regression had nothing to tune: you hand it the data, it estimates one slope per predictor, and that is that. A decision tree is different. It learns where to split (is income under 45k? is the job under two years old?), but you must tell it in advance how far it is allowed to go. Here is a small loan tree so the picture is concrete: each box asks one yes-or-no question, and each leaf at the bottom is a final verdict, default or repay.
Those settings you fix in advance are the tree's hyperparameters, and a tree has two that matter most:
- **
tree_depth**: the most questions the tree may ask along any path from top to bottom. A depth of 2 asks at most two questions before deciding; a depth of 12 can ask twelve, carving the loan book into far finer regions. - **
cost_complexity**: how much the tree is punished for having many leaves, which prunes it back. rpart grows a tree by minimizing
\[ R_\alpha(T) = R(T) + \alpha \lvert T \rvert \]
where \(R(T)\) is how often tree \(T\) misclassifies a training applicant, \(\lvert T \rvert\) is the number of leaves (the final risk buckets), and \(\alpha\) is cost_complexity. A big \(\alpha\) makes every extra leaf expensive, so the tree stays small; an \(\alpha\) near zero lets it grow almost without limit.
The distinction is not academic; the knob genuinely changes the tree. Rebuild the lender's loan book here so the page runs on its own. Each row is one applicant, and defaulted (yes or no) is what we predict, with yes as the first factor level so it stays the class we care about catching. We also split off a quarter of the book as a test set right now, and do not touch it again until the final step.
Of 500 applicants, 202 defaulted and 298 repaid. Now grow two trees on the training applicants, one with pruning switched off and one with it turned up, and count how many leaves each ends up with:
Same data, same algorithm, one setting changed. The overgrown tree splits the 374 training applicants into 93 tiny buckets (it has all but memorized them); the pruned tree keeps just 2. Neither is likely the right tree for judging a brand-new applicant. That single hyperparameter, cost_complexity, swung the model from one extreme to the other. Choosing well between those extremes is exactly what tuning does.
Parameter or hyperparameter?
The loan tree learned, from the data, that its first split should be "is income under 45k?". Separately, before fitting, you decided the tree may grow to a depth of 7. Which of those two is the hyperparameter?
Six moves, start to finish
Before the details, here is the shape of the entire lesson. Tuning is always the same six moves: mark the knobs you want searched, list the candidate settings, cut the training data into folds, score every candidate on every fold, keep the best, and confirm the winner once on the sealed test set. Everything that follows just fills in one box at a time.
Mark the knobs with tune()
To search a hyperparameter instead of fixing it, you write tune() in its place. It is a placeholder that says "leave this blank for now; we will fill it from the grid later." Here we mark both tree knobs as tunable, then wrap the spec in a workflow exactly as in Lesson 3, pairing it with a recipe. A tree needs no normalizing or dummy variables (rpart handles raw numbers and factors itself), so the recipe is just the formula.
Printing the spec shows both arguments held open at tune(). The workflow is now a template with two blanks in it; the search will stamp out a filled-in copy for every candidate setting.
Lay out the grid of settings
A grid is just the list of candidate settings to try. grid_regular() builds a regular one: hand it the parameter objects from the dials package and how many levels of each to space out, and it returns every combination. With two knobs at three levels each, that is \(3^2 = 9\) candidates.
Each parameter object knows its own sensible range. cost_complexity() sweeps from almost no pruning (0.0000000001) up to heavy pruning (0.1) on a log scale; tree_depth(range = c(2, 12)) spaces depths from a shallow 2 to a deep 12. Nine rows, nine trees to try.
Make the search finer
Three levels per knob is coarse. Rebuild the grid with 5 levels of each, so the search tries more candidate depths and penalties. How many candidates does that make?
Show answer
finer_grid <- grid_regular(
cost_complexity(),
tree_depth(range = c(2, 12)),
levels = 5
)
nrow(finer_grid)
#> [1] 25Search across the folds with tune_grid()
Now the honest part. We do not score the nine candidates on the training data as a whole (the overgrown tree would win by memorizing it), and we certainly do not peek at the sealed test set. Instead we reuse the cross-validation from Lesson 4: cut the training applicants into 5 folds, and score every candidate on every fold. tune_grid() does all of it, taking the workflow, the folds, the grid, and the metric set from Lesson 5.
That one call fitted a lot of trees. Nine candidate settings, each trained and scored on all 5 folds, is \(9 \times 5 = 45\) tree fits. Each row above is one fold; its .metrics holds an 18-row tibble (9 candidates times 2 metrics). tune keeps every score so we can average them next.
test and kept the best, that final score would be optimistic: you would have fitted your choice of settings to the very data meant to give an unbiased estimate. The test set earns its keep only by staying sealed until the end. Candidates are judged on the training folds, full stop.Where do candidates get scored?
You have nine candidate settings for the loan tree. To decide which is best, where should each one be scored?
Collect and rank the results
collect_metrics() averages each candidate's score across the 5 folds and pairs it with a standard error, so you see both the typical performance and how much it wobbled fold to fold.
Eighteen rows: one per candidate per metric. Rather than squint at all of them, ask show_best() for the top candidates by the metric you care about, roc_auc:
The best cross-validated roc_auc is 0.723, reached at depth 7 (the tiny cost_complexity values barely prune, so depth is doing the work here). Growing to depth 12 scores identically: on just 374 training rows the tree runs out of useful splits well before 12, so the extra allowance is never used. And you can see the whole search surface at a glance:
Select the best, then finalize
The search is done; now collect its verdict. select_best() pulls the single top-scoring candidate out of the results as a one-row tibble of settings. Fill in the function, keyed to roc_auc:
Show answer
best_tree <- select_best(tree_res, metric = "roc_auc")
best_tree
#> # A tibble: 1 x 3
#> cost_complexity tree_depth .config
#> <dbl> <int> <chr>
#> 1 0.0000000001 7 pre0_mod2_post0Was tuning worth it?
Fair question. Compare three trees on the exact same 5 folds: the careless deep tree (the worst setting in our grid, roc_auc 0.651), the rpart default (what you get with no tuning at all), and the tuned winner (0.723). We already have the first and third; here is the untouched default for comparison:
Now put all three side by side:
Tuning lifted roc_auc from the default's 0.707 to 0.723, a real if modest gain, and it steered you well clear of the careless deep tree at 0.651. That 0.072 spread between the best and worst settings is the whole point: the choice of hyperparameters swings performance by seven points of AUC, and tuning is how you land on the good end instead of gambling.
Confirm once on the sealed test set
The 0.723 came from cross-validation on the training folds. It is a good estimate, but it is the number the search optimized, so it leans a touch optimistic. For the figure you report to the risk committee, you want a score from data that played no part in choosing anything, and that is the test set you sealed away in step 2. last_fit() takes the finalized workflow, trains it on the full training set, and scores it once on that held-out test set:
The honest test-set roc_auc is 0.695, a little below the cross-validated 0.723, which is exactly the small optimism we expected. This 0.695 is the number to report: it is the only score computed on applicants that had no hand in picking the model or its settings.
Which number do you report?
Your tuned loan tree scored 0.723 roc_auc in cross-validation and 0.695 on the sealed test set. Which do you put in the report to the risk committee, and why?
References
A few authoritative places to take this further:
- tune package documentation (tidymodels) - the reference for tune_grid, collect_metrics, select_best and finalize_workflow, each with runnable examples.
- Get Started: Tune model parameters (tidymodels.org) - the official hands-on walkthrough, tuning a decision tree from grid to final fit.
- Tidy Modeling with R, ch. 13: Grid search - Kuhn and Silge on regular vs space-filling grids and how tune_grid works under the hood.
- dials package documentation (tidymodels) - the parameter objects (cost_complexity, tree_depth and the rest) and the grid builders that lay out a search.
- An Introduction to Statistical Learning, ch. 8 (free PDF) - decision trees and cost-complexity pruning, the theory behind the cost_complexity knob.
Lesson 6 complete
You can now tune a model instead of guessing at its settings. You marked a tree's cost_complexity and tree_depth with tune(), laid out a grid of candidates, and ran tune_grid() to score every one across cross-validation folds without once touching the test set. You read the leaderboard with collect_metrics() and show_best(), selected the winner, finalized the workflow, and confirmed it a single time on the sealed test set for an honest 0.695 roc_auc.
The deeper habit is the honest one: candidates are judged by resampling, the test set is spent only at the very end, and you tune for the metric that matches your decision, not whichever number happens to look best.
Next, Lesson 7: Compare many models with workflowsets. Tuning found the best tree, but why assume a tree is the right model at all? You will line up a logistic regression, this tuned tree, and a random forest, and race them on the exact same folds to crown a winner fairly.