---
Suppose we do not observe the population.
We have observed a sample of size 200.
Here it is:
```{r}
observed_data <- SF %>% sample_n(size = 200, replace = FALSE)
```
We're still interested in the mean of `arr_delay`.
Here is the mean of `arr_delay` for our observed data:
```{r}
observed_mean <- as.numeric(observed_data %>% summarize(mean(arr_delay)))
observed_mean
```
Is this a *parameter* or a *statistic*?
## A bootstrap sample from our data
```{r}
boot_samp <- observed_data %>% sample_n(size = 200, replace=TRUE)
boot_samp %>% summarize(mean_delay=mean(arr_delay))
```
## Another bootstrap sample
```{r}
boot_samp <- observed_data %>% sample_n(size = 200, replace=TRUE)
boot_samp %>% summarize(mean_delay=mean(arr_delay))
```
## 5000 bootstrap samples
Typically need lots of replications when bootstrapping.
*How many?*
Typically at least 1000.
As with all simulations, results vary.
You can experiment with how many replications are needed to give stable estimates to the desired accuracy.
```{r}
boot_means <- rep(NA, 5000) # where we'll store the means
for (i in 1:5000)
{
boot_samp <- observed_data %>% sample_n(size = 200, replace=TRUE)
boot_means[i] <- as.numeric(boot_samp %>% summarize(mean(arr_delay)))
}
boot_means <- data_frame(mean_delay=boot_means)
```
---
```{r, fig.height=3.5}
ggplot(boot_means, aes(x=mean_delay)) + geom_histogram(binwidth=2) +
labs(x="Means from bootstrap samples",
title="Bootstrap distribution for the mean of arr_delay")
```
*Where is the centre of the distribution? Does this make sense?*
## Do the bootstrap estimates capture the population parameter?
Remember the value of the population mean:
```{r}
population_mean <- SF %>% summarize(population_mean_delay=mean(arr_delay))
population_mean
```
---
```{r, warning=F, message=F}
ggplot(boot_means, aes(x=mean_delay)) + geom_histogram(binwidth=2) +
geom_dotplot(data=population_mean, aes(x=population_mean_delay), fill="red") +
labs(x="Means from bootstrap samples",
title="Bootstrap distribution for the mean of arr_delay")
```
## Based on the bootstrap distribution, what other population values of the mean might be plausible?
- In the real world we observe our sample of data, we can construct the bootstrap sampling distribution of the mean, and we don't know the population mean.
- We'd like to make inferences about the population, such as *what other population values of the mean might be plausible?*
- To answer this, look at the range of values that the bootstrap distibution covers, but exclude the values way out in the tails.
- Typically, we take the middle 95% of the bootstrap distribution of resampled means.
- These are values from the 2.5th percentile to the 97.5th percentile.
---
```{r, fig.height=3.5, echo=F}
ggplot(boot_means, aes(x=mean_delay)) + geom_histogram(binwidth=2) +
labs(x="Means from bootstrap samples",
title="Bootstrap distribution for the mean of arr_delay")
```
2.5th and 97.5th percentiles:
```{r}
quantile(boot_means$mean_delay, c(0.025, 0.975))
```
## Will this procedure always give an interval that captures the population mean?
Our interval that is the middle 95% of our bootstrap distribution is (-5.14, 5.99).
It includes the population mean (2.673).
To see how often an interval calculated this way from a sample of size 200 would capture the population mean, we can take advantage of the situtation here where we have the population and repeat this procedure many times. We can:
1. Randomly draw another data sample of size 200 from the population.
2. Find the bootstrap sampling distribution of the mean from 5000 replications of bootstrap samples of this new data.
3. Find the interval that is the middle 95% of the bootstrap distribution.
4. Repeat 1. to 3. 100 times.
---
Statistical theory says that these intervals should capture the population mean 95% of the time.
These are called 95% **confidence intervals** for the mean.
To see if this holds:
- Need to know the population mean
- Need to take a number of random samples, each representing a possible dataset
- Need to calculate bootstrap intervals for the mean for each dataset
- Need to check how many of these confidence intervals contain the population mean
Code to calculate 100 bootstrap confidence intervals for the mean of `arr_delay`, each calculated from a random sample from the population of size 200, is in the R markdown document for this lecture. Note that it takes a while to run.
*Results are on the next slide...*
```{r, eval=F, echo=F}
n_intervals <- 100
perc025 <- rep(NA, n_intervals) # where we'll store the lower limit of the intervals
perc975 <- rep(NA, n_intervals) # where we'll store the upper limit of the intervals
sample_size <- 200
replicates <- 5000
for (i in i:n_intervals)
{
# randomly sample a data set
observed_data <- SF %>% sample_n(size = sample_size, replace = FALSE)
# get the bootstrap means
boot_means <- rep(NA, replicates) # where we'll store the bootstrap means
for (j in 1:replicates)
{
boot_samp <- observed_data %>% sample_n(size = sample_size, replace=TRUE)
boot_means[j] <- as.numeric(boot_samp %>% summarize(mean(arr_delay)))
}
# get the 95% interval for this set of bootstrap means
perc025[i] <- quantile(boot_means, 0.025)
perc975[i] <- quantile(boot_means, 0.975)
}
# write results to a file
bootstrapCIs <- data_frame(perc025, perc975)
write.csv(bootstrapCIs, file = "bootstrapCIs.csv",row.names=FALSE)
```
---
100 bootstrap confidence intervals for the mean, each calculated from a random sample from the population of size 200
```{r, echo=F}
manyCIs <- read.csv("bootstrapCIs.csv")
manyCIs <- manyCIs %>% mutate(capture = ifelse(perc025 <= as.numeric(population_mean) & perc975 >= as.numeric(population_mean), "yes", "no"))
manyCIsforplot <- data_frame(number=c(1:100, 1:100),
limits=c(manyCIs$perc025, manyCIs$perc975),
capture=c(manyCIs$capture, manyCIs$capture))
ggplot(manyCIsforplot, aes(x=limits, y=number, group=number, color=capture)) +
geom_point(size=2) +
geom_line() +
geom_vline(xintercept=as.numeric(population_mean), colour="black") +
labs(x="Confidence interval limits", y="") +
theme_bw()
```
*How many of the confidence intervals capture the population mean?*
---
- Each of these confidence intervals gives a range of plausible values for what our parameter might be. This range is based on the incomplete and imperfect information we have in each set of data.
- A "good" interval captures the population mean.
- Since our intervals are the middle 95% of the bootstrap sampling distribution of the mean, we expect that 95% will be "good".
- Sometimes, because of chance, our randomly sampled data leads to a confidence interval that does not capture the population mean. We expect that this will happen for 95% of datasets.
# Confidence Intervals
## What is a confidence interval?
A 95% **confidence interval** for a population parameter is calculated from sample data in such a way that the interval will include the parameter for 95% of possible samples.
95% is the **confidence level**.
90% and 99% confidence intervals are also common.
[**Note:** A comment in your textbook near the bottom of page 153 that you can ignore, along with all references to standard error and standard deviation:
*As taught in introductory statistics courses, often a 95% confidence interval is calculated from the mean and standard error of the sampling distribution.*
You'll learn the theory behind this statement in your second year statistics courses.]
## How to calculate a bootstrap confidence interval
1. Take a bootstrap sample of the data by sampling with replacement, the same number of observations as the original data.
2. For the bootstrap sample, calculate the statistic that estimates the parameter you are interested in.
3. Repeat steps 1. and 2. many times to get a distribution of bootstrap statistics.
4. A 95% confidence interval for the parameter is the middle 95% of values of the bootstrap statistics.
## Example from Week 4:
### Kissing the Right Way
- Güntürkün (2003) recorded the direction kissing couples tilted their heads.
- Of the 124 couples he observed, 80 turned their heads to the right.
- 64.5% of couples in Güntürkün's sample tilted their heads to the right.
- *Today:* Find a 95% confidence interval for the proportion of all couples who tilt their heads to the right when they kiss.
```{r}
# Create a data frame
direction <- c( rep("right", 80), rep("left", 124-80) )
kissdata <- data_frame(direction)
```
---
### Bootstrap distribution for the proportion of couples who tilt their heads to the right when they kiss
```{r, echo=F}
set.seed(130)
```
```{r}
boot_p <- rep(NA, 5000) # where we'll store the bootstrap proportions
for (i in 1:5000)
{
boot_samp <- kissdata %>% sample_n(size = 124, replace=TRUE)
boot_p[i] <- as.numeric(boot_samp %>% filter(direction == "right") %>%
summarize(n()))/124
}
boot_p <- data_frame(boot_p)
```
---
```{r}
ggplot(boot_p, aes(x=boot_p)) + geom_histogram(binwidth=0.02) +
labs(x="Proportions from bootstrap samples",
title="Bootstrap distribution of proportion who kiss right")
```
---
```{r}
quantile(boot_p$boot_p, c(0.025, 0.975))
```
A 95% confidence interval for the proportion of couples who tilt their heads to the right when they kiss is:
---
```{r}
quantile(boot_p$boot_p, c(0.005, 0.995))
```
A _____% confidence interval for the proportion of couples who tilt their heads to the right when they kiss is:
---
```{r}
quantile(boot_p$boot_p, c(0.05, 0.95))
```
A _____% confidence interval for the proportion of couples who tilt their heads to the right when they kiss is:
---
90% confidence interval:
95% confidence interval:
99% confidence interval:
As the confidence level increases, it is more likely that our confidence interval captures the population median.
*What's the downside to having a higher confidence level?*
## Be careful interpreting confidence intervals
A 95% confidence interval for the proportion of couples who tilt their heads to the right when they kiss is (0.56, 0.73).
*Incorrect interpretation #1:*
I am 95% confident that the proportion for my sample data will be in my confidence interval.
*Incorrect interpretation #2:*
The probability that the proportion for the population is in my confidence interval is 95%.
## Be careful interpreting confidence intervals
A 95% confidence interval for the proportion of couples who tilt their heads to the right when they kiss is (0.56, 0.73).
*Incorrect interpretation #3:*
95% of my data is in the interval (0.56, 0.73).
*Incorrect interpretation #4:*
In 95% of samples I'd get a proportion in the interval (0.56, 0.73).
## A few notes about the bootstrap
- The bootstrap re-uses our data.
- Typically larger samples reflect the population better. The bootstrap may woork poorly when the sample has a small number of observations.
- If the sample is biased, the bootstrap confidence interval will also be biased.
- Using the bootstrap doesn't give us better estimates than the orginal data of the parameter of interest.
- It does give us an indication of the accuracy of our estimate.
## A few notes about the bootstrap
- The confidence interval method we've used is the *percentile bootstrap method*.
- There are other bootstrap methods that are more robust, that is they are better at capturing the parameter the correct percentage of the time.
- The percentile bootstrap method works best for large samples and when the bootstrap distribution is approximately symmetric and continuous.
## Reminder of today's concepts:
1. Percentiles
2. Population parameters and sample statistics to estimate them
2. Sampling distribution
3. Bootstrap sampling distribution
4. Confidence intervals