6.3 Measures of Fit in Multiple Regression
In multiple regression, common summary statistics are \(SER\), \(R^2\) and the adjusted \(R^2\).
Taking the code from Section 6.2, simply use summary(mult.mod) to obtain the \(SER\), \(R^2\) and adjusted-\(R^2\). For multiple regression models the \(SER\) is computed as
\[ SER = s_{\hat u} = \sqrt{s_{\hat u}^2}, \] where modify the denominator of the premultiplied factor in \(s_{\hat u}^2\) in order to accommodate for additional regressors. Thus,
\[ s_{\hat u}^2 = \frac{1}{n-k-1} \, SSR \]
with \(k\) denoting the number of regressors excluding the intercept.
While summary() computes the \(R^2\) just as in the case of a single regressor, it is no reliable measure for multiple regression models. This is due to \(R^2\) increasing whenever an additional regressor is added to the model. Adding a regressor decreases the \(SSR\) — at least unless the respective estimated coefficient is exactly zero, which practically never happens (see Chapter 6.4 of the book for a detailed argument). The adjusted \(R^2\) takes this into consideration by “punishing” the addition of regressors using a correction factor. So the adjusted \(R^2\), or simply \(\bar{R}^2\), is a modified version of \(R^2\). It is defined as
\[ \bar{R}^2 = 1-\frac{n-1}{n-k-1} \, \frac{SSR}{TSS}. \]
As you may have already suspected, summary() adjusts the formula for \(SER\) and it computes \(\bar{R}^2\) and of course \(R^2\) by default, thereby leaving the decision of which measure to rely on to the user.
You can find both measures at the bottom of the output produced by calling summary(mult.mod).
summary(mult.mod)
#>
#> Call:
#> lm(formula = score ~ STR + english, data = CASchools)
#>
#> Residuals:
#> Min 1Q Median 3Q Max
#> -48.845 -10.240 -0.308 9.815 43.461
#>
#> Coefficients:
#> Estimate Std. Error t value Pr(>|t|)
#> (Intercept) 686.03224 7.41131 92.566 < 2e-16 ***
#> STR -1.10130 0.38028 -2.896 0.00398 **
#> english -0.64978 0.03934 -16.516 < 2e-16 ***
#> ---
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#>
#> Residual standard error: 14.46 on 417 degrees of freedom
#> Multiple R-squared: 0.4264, Adjusted R-squared: 0.4237
#> F-statistic: 155 on 2 and 417 DF, p-value: < 2.2e-16We can also compute the measures by hand using the formulas above. Let us check that the results coincide with the values provided by summary().
# define the components
n <- nrow(CASchools) # number of observations (rows)
k <- 2 # number of regressors
y_mean <- mean(CASchools$score) # mean of avg. test-scores
SSR <- sum(residuals(mult.mod)^2) # sum of squared residuals
TSS <- sum((CASchools$score - y_mean )^2) # total sum of squares
ESS <- sum((fitted(mult.mod) - y_mean)^2) # explained sum of squares
# compute the measures
SER <- sqrt(1/(n-k-1) * SSR) # standard error of the regression
Rsq <- 1 - (SSR / TSS) # R^2
adj_Rsq <- 1 - (n-1)/(n-k-1) * SSR/TSS # adj. R^2
# print the measures to the console
c("SER" = SER, "R2" = Rsq, "Adj.R2" = adj_Rsq)
#> SER R2 Adj.R2
#> 14.4644831 0.4264315 0.4236805Now, what can we say about the fit of our multiple regression model for test scores with the percentage of English learners as an additional regressor? Does it improve on the simple model including only an intercept and a measure of class size? The answer is yes: compare \(\bar{R}^2\) with that obtained for the simple regression model mod.
Including \(PctEL\) as a regressor improves the \(\bar{R}^2\), which we deem to be more reliable in view of the above discussion. Notice that the difference between \(R^2\) and \(\bar{R}^2\) is small since \(k=2\) and \(n\) is large. In short, the fit of (6.6) improves vastly on the fit of the simple regression model with \(STR\) as the only regressor.
Comparing regression errors we find that the precision of the multiple regression model (6.6) improves upon the simple model as adding \(PctEL\) lowers the \(SER\) from \(18.6\) to \(14.5\) units of test score.
As already mentioned, \(\bar{R}^2\) may be used to quantify how good a model fits the data. However, it is rarely a good idea to maximize these measures by stuffing the model with regressors. You will not find any serious study that does so. Instead, it is more useful to include regressors that improve the estimation of the causal effect of interest which is not assessed well by \(R^2\). The issue of variable selection is covered in Chapter 8.