Missing Value Treatment

In Lesson 3 you met one blank cell, the unlogged Bagel sale, and patched it with na.rm = TRUE. Real data is never that tidy. Maya ran a quick exit survey one busy morning, twelve customers, five questions each, and came back with a table full of holes: a spend she never recorded, a rating the tablet ate, tips people simply would not say.

The wrong move is to delete every row with a blank and carry on. As you will see, that throws away two thirds of Maya's survey and quietly biases what is left. This lesson is about treating those holes honestly.

By the end you will be able to:

• Find and count missing values, and see how far the damage spreads
• Name why data goes missing (the three mechanisms: MCAR, MAR, MNAR)
• Choose between dropping and filling, and say how each choice bends the answer

Prerequisites: Lessons 1 to 3 of this course, so you know tidy data, the dplyr verbs and the pipe, and group_by / summarise with na.rm. Everything new is defined as it appears.

Find the holes before you fix them

Here is Maya's survey. Each lesson starts in a fresh R session, so we build the table right here (run this once). A blank answer is stored as NA, which R reads as "a value exists but I do not know it". One column carries each kind of hole, and we will return to that on purpose.

is.na(x) is the workhorse: it returns TRUE wherever a value is missing. Wrap it in sum() to count blanks, and colSums() to count them per column:

So nine cells are blank: three spends, two ratings, four tips. The dplyr way says the same thing, one verb per column:

Counting per column hides something, though. The real question is how many whole rows are usable. complete.cases() flags the rows with no blank anywhere:

Only four of the twelve customers, Ana, Finn, Ivy and Kim, answered every question. Nine scattered blanks have spoiled two thirds of the rows.

Where did the rows go?

Only 3 of the 12 spend values are missing, yet complete.cases() says only 4 rows are fully complete. Why are eight rows incomplete when one column lost just three values?

Three reasons data goes missing

Before you touch a blank, ask why it is blank, because the answer decides whether any fix is safe. Statisticians sort missingness into three mechanisms, and Maya's three leaky columns are one clean example of each.

The names are slippery, so anchor them to the story. The dead tablet did not care who Cara and Hana were or how they would have rated, so rating is MCAR, the benign case. Cash customers were likelier to have an unrecorded spend, and payment is right there in the table, so spend is MAR. The customers who left no tip are exactly the ones who declined to report it, and we never see those values, so tip is MNAR, the dangerous case.

The tip column

The customers who left little or no tip are the ones who skipped the tip question, so tip is missing exactly for the low tippers. Maya thinks of filling every blank tip with the average of the tips she did collect ($4.4). Which mechanism is this, and why is that fill dangerous?

Drop: simple, and quietly expensive

The bluntest fix is to throw away rows with blanks. drop_na() from tidyr keeps only the complete rows; with no arguments it drops a row that is blank in any column.

Pressing Run below shows the damage from the full drop_na(): eight of twelve customers vanish.

Dropping rows like this (called listwise deletion) is honest only when the blanks are MCAR. Here they are not, and there is a second cost: the four survivors are all card customers who spent a lot, so the average spend over the complete rows is badly skewed.

When too many rows would die for one bad column, the alternative is to drop that column instead with select(-tip), keeping every customer. Both moves trade information for tidiness; the skill is knowing which you can afford to lose.

Impute: fill the blank with a stand-in

The other option is to impute: replace each blank with a plausible value so the row survives. The four common stand-ins:

The mean is the arithmetic average you met in Lesson 3: for values \(x_1, \dots, x_n\), where \(x_i\) is one observed spend and \(n\) the number observed,

\[ \bar{x} = \frac{1}{n}\sum_{i=1}^{n} x_i. \]

coalesce() fills a blank with the first non-missing value you give it, so it makes mean or median imputation a one-liner. Maya's spend is right-skewed (one $30 sale), so watch how the two stand-ins differ:

The mean fills each blank with 15.9, the median with the sturdier 14. For a category you would use the mode instead, the most frequent level:

Rating 4 is the mode (it appears four times), so it is the natural fill for a missing rating. And for time-ordered data, last observation carried forward repeats the previous reading, which only makes sense in row order:

# LOCF: only for time-ordered data, e.g. Maya's daily sales from Lesson 3
daily %>% arrange(date) %>% fill(temperature, .direction = "down")

Same data, three different answers

Here is why the choice matters. Below are three honest ways to estimate Maya's average spend from the same survey: drop the incomplete rows, keep all observed spends with na.rm, or impute each missing cash spend with the average spend of cash customers. Every bar is a real number computed from the data, yet they disagree by six dollars.

Dropping rows gives $19.5, because the survivors happen to be big-spending card customers. Averaging the observed spends gives $15.9. But the blanks are MAR, missing mostly for cash customers, who spent far less. Filling each blank with the average of the cash spends Maya did record pulls the estimate down to $13.9, the most defensible of the three. The lesson is not "imputing is best"; it is that your treatment of the holes can move the headline number more than the real data does, so the choice must be deliberate and disclosed.

Impute the honest way

Mean-imputing every blank spend with the overall average ($15.9) overstates Maya's takings, because the missing spends belong to cheaper cash customers. Fix it: group the survey by payment first, so each blank is filled with the mean spend of its own payment type. Fill in the blank.

survey %>%
  group_by(payment) %>%
  mutate(spend_filled = coalesce(spend, mean(spend, na.rm = TRUE))) %>%
  ungroup()

References

A few authoritative places to take this further:

• R for Data Science (2e), Missing values - the canonical, free chapter on explicit and implicit NAs and how to handle them in the tidyverse.
• tidyr: replace_na() and fill() reference - the official docs for the imputation verbs you used here.
• van Buuren, Flexible Imputation of Missing Data (free online) - the standard modern text; chapter 1 explains MCAR/MAR/MNAR carefully and why single-value imputation understates uncertainty.
• naniar: tidy tools for missing data - an R package for visualising and exploring missingness patterns before you treat them.
• Rubin (1976), Inference and missing data, Biometrika 63(3) - the paper that defined the three mechanisms you used to reason about Maya's columns.

Lesson 4 complete, and the course with it

You can now treat missing data honestly. You found it with is.na, colSums and complete.cases; you asked why it was missing (MCAR, MAR, MNAR) before touching it; you weighed dropping rows or columns against imputing with the mean, median, mode or LOCF; and you saw how the same survey yields three different averages depending on that one choice, so you now know to make it deliberately and write it down.

That closes Data Wrangling with dplyr. Across four lessons you went from a raw CSV to tidy data, learned the one-table verbs and the pipe, grouped and summarised, and now clean missing values with judgement. Two natural next courses build straight on this: Joining and Reshaping (combining several tables and pivoting between long and wide), and Exploratory Data Analysis (turning these skills loose on a brand-new dataset). Maya can run her bakery on numbers she trusts, and so can you.