{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "#Statistical Inference for Everyone: Technical Supplement\n", "\n", "\n", "\n", "This document is the technical supplement, for instructors, for [Statistical Inference for Everyone], the introductory statistical inference textbook from the perspective of \"probability theory as logic\".\n", "\n", "\n", "\n", "[Statistical Inference for Everyone]: http://web.bryant.edu/~bblais/statistical-inference-for-everyone-sie.html\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Estimating a Proportion\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "$$\\newcommand{\\twocvec}[2]{\\left(\\begin{array}{c}\n", " #1 \\\\\\\\ #2\n", " \\end{array}\\right)}\n", "\\newcommand{\\nchoosek}[2]{\\twocvec{#1}{#2}}\n", "$$\n", "\n", "If $\\theta$ is the model representing the probability, $\\theta$, of the coin\n", "landing on heads (and $1-\\theta$ is the probability of landing on tails), we\n", "need to make an estimate of probability of model $\\theta$ being true given the\n", "data, which will consist of $N$ flips of which $h$ are heads. \n", "\n", "Bayes rule is:\n", "\\begin{eqnarray}\n", "p(\\theta|D,I) &=& \\frac{p(D|\\theta,I)p(\\theta|I)}{p(D|I)} = \n", "\\frac{p(D|\\theta,I)p(\\theta,I)}{\\sum_\\theta p(D|\\theta,I)p(\\theta|I)}\n", "\\end{eqnarray}\n", "\n", "Thus, the probability of a particular model $\\theta$ being true is the product\n", "of the probability of the observed data ($h$ heads in $N$ flips) given the\n", "model $\\theta$ and the prior probability of the model $\\theta$ being true\n", "before we even look at the data, divided by the probability of the data itself\n", "over all models.\n", "\n", "The prior probability of model $\\theta$ will be assumed to be uniform (from\n", "maximum entropy considerations). The probability, $\\theta$, ranges from 0 to\n", "1, to the prior is\n", "\\begin{eqnarray}\n", "p(\\theta|I) = 1\n", "\\end{eqnarray}\n", "\n", "The probability of the data given the random model, is just the binomial\n", "distribution:\n", "\n", "\\begin{eqnarray}\n", "p(D|\\theta)=\\nchoosek{N}{h} \\theta^h (1-\\theta)^{N-h}\n", "\\end{eqnarray}\n", "\n", "The probability of the data, $p(D|I)$, is found by summing (or in this case\n", "integrating) $p(D|\\theta,I)p(\\theta|I)$ for all $\\theta$:\n", "\n", "\\begin{eqnarray}\n", "p(D|I) &=& \\int_0^1 \\nchoosek{N}{h} \\theta^h (1-\\theta)^{N-h} \\cdot 1 d\\theta\n", "\\\\\\\\\n", "&=&\\frac{N!}{h!(N-h)!} \\frac{h!(N-h)!}{(N+1)!} = \\frac{1}{N+1}\n", "\\end{eqnarray}\n", "\n", "Now the probability of model $\\theta$ being true, given the data, is just\n", "\n", "\\begin{eqnarray}\n", "p(\\theta|D,I)&=& (N+1) \\cdot \\nchoosek{N}{h} \\theta^h (1-\\theta)^{N-h} \\\\\n", "&=& \\frac{(N+1)!}{h!(N-h)!} \\theta^h (1-\\theta)^{N-h} \n", "\\end{eqnarray}\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Max, Mean, Variance\n", "\n", "The model with the maximum probability is found by maximizing $p(\\theta|D,I)$\n", "w.r.t. $\\theta$:\n", "\n", "\\begin{eqnarray}\n", "\\frac{dP(\\theta|D,I)}{d\\theta} &=& 0 = \\frac{(N+1)!}{h!(N-h)!} \\left( \n", " -(N-h) \\theta^h (1-\\theta)^{N-h-1} + h \\theta^{h-1} (1-\\theta)^{N-h} \\right) \\\\\\\\\n", "(N-h) \\theta^h (1-\\theta)^{N-h-1} &=& h \\theta^{h-1} (1-\\theta)^{N-h} \\\\\\\\\n", "\\theta(N-h) &=& (1-\\theta) h = h-\\theta h = N\\theta-\\theta h \\\\\\\\\n", "\\theta&=&\\frac{h}{N} \\;\\;\\;\\;\\;\\surd\n", "\\end{eqnarray}\n", "\n", "The average and the standard deviation is also straightforward.\n", "\n", "\n", "\\begin{eqnarray}\n", "\\bar{\\theta} &=& \\int_0^1 \\theta \\cdot \\frac{(N+1)!}{h!(N-h)!} \\theta^h (1-\\theta)^{N-h} \\\\\\\\\n", " &=& \\frac{(N+1)!}{h!(N-h)!} \\int_0^1 \\theta^{h+1} (1-\\theta)^{N-h} \\\\\\\\\n", " &=&\\frac{(N+1)!}{h!(N-h)!} \\frac{(h+1)!(N-h)!}{(N+2)!} \\\\\\\\\n", " &=&\\frac{h+1}{N+2} \\\\\\\\\n", "\\bar{\\theta^2} &=& \\int_0^1 \\theta^2 \\cdot \\frac{(N+1)!}{h!(N-h)!} \\theta^h (1-\\theta)^{N-h} \\\\\\\\\n", " &=&\\frac{(N+1)!}{h!(N-h)!} \\frac{(h+2)!(N-h)!}{(N+3)!} \\\\\\\\\n", " &=&\\frac{(h+1)(h+2)}{(N+2)(N+3)} \\\\\\\\\n", "\\sigma^2 &=& \\bar{\\theta^2} - \\bar{\\theta}^2 = \\frac{(h+1)(h+2)}{(N+2)(N+3)} -\n", " \\frac{(h+1)(h+1)}{(N+2)(N+2)} \\\\\\\\\n", "&=&\\frac{(h+1)(N-h+1)}{(N+2)^2(N+3)} \\\\\\\\\n", "&=& \\frac{(h+1)}{(N+2)}\\left( \\frac{n+2}{n+2} - \\frac{h+1}{N+2}\\right)\n", "\\frac{1}{N+3} \\\\\\\\\n", "&=& \\bar{\\theta}(1-\\bar{\\theta})\\frac{1}{N+3}\n", "\\end{eqnarray}\n", "\n", "### An Approximation for the Variance\n", "\n", "If $f=h/N$ is the actual fraction of heads observed, then the variance above\n", "can be written as\n", "\\begin{eqnarray}\n", "\\sigma^2 &=&\\frac{(fN+1)(N-fN+1)}{(N+2)^2(N+3)} \\\\\\\\\n", "\\mbox{(for large $N$)}&\\approx& \\frac{(fN+1)(N-fN)}{N^3}\n", " =\\frac{(fN+1)(1-f)}{N^2} \\\\\\\\\n", "\\mbox{(for large $fN$)}&\\approx& \\frac{(fN)(N-fN)}{N^2} = \\frac{f(1-f)}{N} \\\\\\\\\n", "\\sigma^2&\\approx& \\frac{f(1-f)}{N}\n", "\\end{eqnarray}\n", "\n", "In this limit, the distribution (beta distribution) can be approximated with a\n", "Gaussian.\n" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "collapsed": false }, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---------------------" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "\n", "\n" ], "text/plain": [ "" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from IPython.core.display import HTML\n", "\n", "\n", "def css_styling():\n", " styles = open(\"../styles/custom.css\", \"r\").read()\n", " return HTML(styles)\n", "css_styling()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 2", "language": "python", "name": "python2" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.9" } }, "nbformat": 4, "nbformat_minor": 0 }