{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Homework 7: Testing Hypotheses" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Reading**: \n", "* [Testing Hypotheses](https://www.inferentialthinking.com/chapters/11/testing-hypotheses.html)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Please complete this notebook by filling in the cells provided.\n", "\n", "Directly sharing answers is not okay, but discussing problems with the course staff or with other students is encouraged. Refer to the policies page to learn more about how to learn cooperatively.\n", "\n", "For all problems that you must write our explanations and sentences for, you **must** provide your answer in the designated space. Moreover, throughout this homework and all future ones, please be sure to not re-assign variables throughout the notebook! For example, if you use `max_temperature` in your answer to one question, do not reassign it later on." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Don't change this cell; just run it. \n", "\n", "import numpy as np\n", "from datascience import *\n", "\n", "# These lines do some fancy plotting magic.\n", "import matplotlib\n", "%matplotlib inline\n", "import matplotlib.pyplot as plt\n", "plt.style.use('fivethirtyeight')\n", "import warnings\n", "warnings.simplefilter('ignore', FutureWarning)\n", "import scipy.stats\n", "\n", "\n", "#import otter\n", "#grader = otter.Notebook()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Spam Calls\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 1: 781 Fun" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Yanay gets a lot of spam calls. An area code is defined to be a three digit number from 200-999 inclusive. In reality, many of these area codes are not in use, but for this question we'll simplify things and assume they all are. **Throughout these questions, you should assume that Yanay's area code is 781.**" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false }, "source": [ "**Question 1.** Assuming each area code is just as likely as any other, what's the probability that the area code of two back to back spam calls are 781?\n", "\n", "\n", "" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "manual_grade": true, "manual_problem_id": "catching_cheaters_1" }, "outputs": [], "source": [ "prob_781 = ...\n", "prob_781" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false }, "source": [ "**Question 2.** Rohan already knows that Yanay's area code is 781. Rohan randomly guesses the last 7 digits (0-9 inclusive) of his phone number. What's the probability that Rohan correctly guesses Yanay's number, assuming he’s equally likely to choose any digit?\n", "\n", "*Note: A phone number contains an area code and 7 additional digits, i.e. xxx-xxx-xxxx*\n", "\n", "" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "prob_yanay_num = ...\n", "prob_yanay_num" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Yanay suspects that there's a higher chance that the spammers are using his area code (781) to trick him into thinking it's someone from his area calling him. Ashley thinks that this is not the case, and that spammers are just choosing area codes of the spam calls at random from all possible area codes (*Remember, for this question we’re assuming the possible area codes are 200-999, inclusive*). Yanay wants to test his claim using the 50 spam calls he received in the past month.\n", "\n", "Here's a dataset of the area codes of the 50 spam calls he received in the past month." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Just run this cell\n", "spam = Table().read_table('spam.csv')\n", "spam" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false }, "source": [ "**Question 3.** Define the null hypothesis and alternative hypothesis for this investigation. \n", "\n", "*Hint: Don’t forget that your null hypothesis should fully describe a probability model that we can use for simulation later.*\n", "\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "$H_o: \\pi = \\frac{1}{800}$\n", "\n", "H0: pi = 1/800\n", "\n", "\n", "\n", "$H_a: \\pi > \\frac{1}{800}$\n", "\n", "Ha: pi > 1/800" ] }, { "cell_type": "markdown", "metadata": { "export_pdf": true, "for_assignment_type": "solution" }, "source": [ "### Binomial Test\n", "\n", "Below is the code to run a binomial test, along with the results." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "num_781 = spam.where(\"Area Code\", 781).num_rows\n", "\n", "total_calls = spam.num_rows\n", "\n", "pval = scipy.stats.binom_test(num_781, total_calls, 1/800, alternative = \"greater\")\n", "\n", "print(f\"Binomial Test Results: The p-value = {pval}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 4.** Suppose you use a p-value cutoff of 5%. What do you conclude from the hypothesis test? Why?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Write your answer here, replacing this text.*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Suppose now that we want to re-run this hypothesis test using the simulation approach. \n", "\n", "\n", "**Question 5.** Which of the following test statistics would be a reasonable choice to help differentiate between the two hypotheses?\n", "\n", "*Hint*: For a refresher on choosing test statistics, check out the textbook section on [Test Statistics](https://www.inferentialthinking.com/chapters/11/3/decisions-and-uncertainty.html#Step-2:-The-Test-Statistic).\n", "\n", "1. The proportion of area codes that are 781 in 50 random spam calls\n", "2. The probability of getting an area code of 781 out of all the possible area codes.\n", "3. The proportion of area codes that are 781 in 50 random spam calls divided by 2\n", "4. The number of times you see the area code 781 in 50 random spam calls\n", "\n", "Assign `reasonable_test_statistics` to an array of numbers corresponding to these test statistics.\n", "\n", "" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "reasonable_test_statistics = make_array(1,4)" ] }, { "cell_type": "markdown", "metadata": { "deletable": false }, "source": [ "
\\pagebreak
\n", "\n", "**For the rest of this question, suppose you decide to use the number of times you see the area code 781 in 50 spam calls as your test statistic.**\n", "\n", "**Question 6.** \n", "Write a function called `simulate` that generates exactly one simulated value of your test statistic under the null hypothesis. It should take no arguments and simulate 50 area codes under the assumption that the result of each area is sampled from the range 200-999 inclusive with equal probability. Your function should return the number of times you saw the 781 area code in those 50 random spam calls.\n", "\n", "\n", "" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "possible_area_codes = np.arange(200,1000)\n", "\n", "\n", "def simulate():\n", " sample = np.random.choice(possible_area_codes, 50, replace=True)\n", " return sum(sample == 781)\n", " #return Table().with_column(\"Sample\", sample).where(\"Sample\", 781).num_rows\n", "# Call your function to make sure it works\n", "simulate()" ] }, { "cell_type": "markdown", "metadata": { "deletable": false }, "source": [ "**Question 7.** Generate 20,000 simulated values of the number of times you see the area code 781 in 50 random spam calls. Assign `test_statistics_under_null` to an array that stores the result of each of these trials. \n", "\n", "*Hint*: Use the function you defined in Question 6.\n", "\n", "" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "for_assignment_type": "solution" }, "outputs": [], "source": [ "test_statistics_under_null = make_array()\n", "repetitions = 20000\n", "\n", "for i in np.arange(repetitions):\n", " test_statistics_under_null = np.append(test_statistics_under_null, \n", " simulate())\n", " \n", "test_statistics_under_null" ] }, { "cell_type": "markdown", "metadata": { "deletable": false }, "source": [ "**Question 8.** Using the results from Question 7, generate a histogram of the empirical distribution of the number of times you saw the area code 781 in your simulation. **NOTE: Use the provided bins when making the histogram**\n", "\n", "\n", "" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "export_pdf": true, "for_assignment_type": "solution" }, "outputs": [], "source": [ "bins = np.arange(0,5,1) # Use these provided bins\n", "Table().with_column(\"Test Statistics Under Null\", \n", " test_statistics_under_null).hist(bins=bins)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": { "deletable": false }, "source": [ "
\\pagebreak
\n", "\n", "**Question 9.** Compute an empirical p-value for this test.\n", "\n", "\n", "" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "for_assignment_type": "solution" }, "outputs": [], "source": [ "# First calculate the observed value of the test statistic from the `spam` table.\n", "observed_val = spam.where(\"Area Code\", 781).num_rows\n", "print(observed_val)\n", "\n", "p_value = ...\n", "p_value" ] }, { "cell_type": "markdown", "metadata": { "deletable": false }, "source": [ "**Question 10.** Suppose you use a P-value cutoff of 1%. What do you conclude from the hypothesis test? Why?\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": { "export_pdf": true, "manual_grade": true, "manual_problem_id": "catching_cheaters_8" }, "source": [ "*Write your answer here, replacing this text.*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 2: Comparing two categorical variables " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Yanay collects information about this month's spam calls. The table `with_labels` is a sampled table, where the `Area Code Visited` column contains either `\"Yes\"` or `\"No\"` which represents whether or not Yanay has visited the location of the area code. The `Picked Up` column is `1` if Yanay picked up and `0` if he did not pick up." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false }, "outputs": [], "source": [ "# Just run this cell\n", "with_labels = Table().read_table(\"spam_picked_up.csv\")\n", "with_labels.pivot(\"Picked Up\", \"Area Code Visited\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Yanay is going to perform a test to see whether or not he is more likely to pick up a call from an area code he has visited. Specifically, his null hypothesis is that there is no difference in the distribution of calls he picked up between visited and not visited area codes, with any difference due to chance. His alternative hypothesis is that there is a difference between the two categories, specifically that he thinks that he is more likely to pick up if he has visited the area code. We are going to perform a $\\chi^2$ test for independence. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Chi-squared test for independence\n", "import numpy as np\n", "obs = np.array([[17,18],[4,11]]) # Put data into array structure\n", "\n", "import scipy.stats\n", "Results = scipy.stats.chi2_contingency(obs)\n", "\n", "## Don't change anything below this line\n", "print(obs)\n", "print(\"Chi-Square = \", Results[0])\n", "print(\"p-value = \", Results[1])\n", "print(\"degrees of freedom = \", Results[2])\n", "print(\"Expected Table = \")\n", "print(Results[3])\n", "\n", "print(\"Reminder: No zero cells, and not more than 20% with counts below 5 in Expected Table.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 11** Using a level of significance of 0.05, compare the p-value from the above and interpret the results in context. Is he more likely to pick up the phone if it comes from an area code he has visited?" ] }, { "cell_type": "markdown", "metadata": { "deletable": false }, "source": [ "*Write your answer here, replacing this text.*" ] }, { "cell_type": "markdown", "metadata": { "deletable": false }, "source": [ "## Part 3: Are elephants afraid of bees? \n", "\n", "[Researcher's](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0155690) have been studying whether elephants are afraid of bees. If they are, this knowledge could be applied to keep elephants from damaging crops or from wandering into areas where the elephants could get hurt, using a so-called [beehive fence](https://elephantsandbees.com/beehive-fence/). \n", "\n", "[Click here for a related video](https://www.youtube.com/embed/6mA1Yl7LEZ4)\n", "\n", "Using a [dataset](https://datadryad.org/stash/dataset/doi:10.5061/dryad.rj592) provided via the internet by some researchers in Gabon, this table was created, showing whether an elephant visited a site containing fruit trees Never (0 times), Sometimes (1-5 times) or Often (6-11 times) and whether a beehive was present during the visit. \n", "\n", "\n", "| | Never | Sometimes | Often |\n", "|-: | -: | :-: | :- |\n", "|Hives Absent |50 |16 |4 |\n", "|Hives Present|755 |77 |8 |\n", "\n", "We want to run a $\\chi^2$ test for independence to see if there's a relationship here that might imply that elephants are deterred by the presence of beehives. \n", "\n", "\n", "**Question 12.** What are the null and alternative hypotheses in this situation?\n", "\n", "\n", "\n", "\n", "\n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Write your answer here, replacing this text.*\n", "\n", "Null is that there is no relationship, elephants are just as likely to visit sites with bees as to not.\n", "\n", "Alternative is that there is some relationship, elephants avoid bees." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 13.** Copy and adapt the code used to run the $\\chi^2$ test above so as to run the test to determine if this data supports the claim that elephants are deterred by bees. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Chi-squared test for independence\n", "import numpy as np\n", "obs = ... # Put data into array structure\n", "\n", "import scipy.stats\n", "Results = scipy.stats.chi2_contingency(obs)\n", "\n", "## Don't change anything below this line\n", "print(obs)\n", "print(\"Chi-Square = \", Results[0])\n", "print(\"p-value = \", Results[1])\n", "print(\"degrees of freedom = \", Results[2])\n", "print(\"Expected Table = \")\n", "print(Results[3])\n", "\n", "print(\"Reminder: No zero cells, and not more than 20% with counts below 5 in Expected Table.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 14.** Using a p-value cut-off of 0.05, what conclusion do you draw from this data?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Write your answer here, replacing this text.*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Congratulations!\n", "\n", "You finished. Download it as HTML and submit it to the assignment folder in D2L. " ] } ], "metadata": { "celltoolbar": "Edit Metadata", "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.13" } }, "nbformat": 4, "nbformat_minor": 1 }