{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "57649ae7-96b1-45ed-b781-bf94e921580d",
   "metadata": {},
   "source": [
    "# Assignments\n",
    "\n",
    "## Statistics 240, spring 2023\n",
    "\n",
    "There will be approximately five problem sets, two coding projects involving open-source code contributions, and a longer term project involving data analysis.\n",
    "\n",
    "The longer project involves re-analyzing data in a published paper that inappropriately used parametric methods\n",
    "or used inappropriate nonparametric methods,\n",
    "using appropriate nonparametric methods instead. \n",
    "Please start looking for a paper that interests you right away.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfbe930e-6146-40d5-ad6f-07a12d46b25c",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Schedule\n",
    "\n",
    "All assignments are due at 11:59pm Pacific Time unless indicated otherwise.\n",
    "\n",
    "| Assignment    | due date |\n",
    "|--------------|-----------:|\n",
    "|[Problem set 1](./Hw/ps01-background.ipynb): math background | 1/30 (Monday)|\n",
    "|[Problem set 2](./Hw/ps02-binary-experiments.ipynb): confidence bounds for the average treatment effect in binary experiments | 2/12 (Sunday) |\n",
    "| [Problem set 3](./Hw/ps03-permute-rank.ipynb): permutation tests, rank-based tests, and simulating $P$-values | 2/26 (Sunday) |\n",
    "|[Computational project 1](./Hw/cp1-tests.ipynb): unit tests for `permute` | 3/12 (Sunday)| \n",
    "| [Problem set 4](./Hw/ps-04-permute-distance.ipynb): permutation tests using the Kolmogorov statistic and its generalization to arbitrary VC classes | 3/19 (Sunday) |\n",
    "|[Computational project 2](./Hw/cp2-function.ipynb): new functionality for `permute` | 4/16 (Sunday) |\n",
    "| [Term project](./Hw/tp01.ipynb) | 5/5 (Friday) | \n",
    "\n",
    "\n",
    "1. Weeks 1-3: Review and inference about binary populations\n",
    "    + If you have never logged into https://github.berkeley.edu, please do so asap so I can add you to the class organization.\n",
    "    + If you do not already have an ORCID, sign up for one at https://orcid.org/. \n",
    "    + review: \n",
    "        - [mathematical foundations](../math-foundations.ipynb)\n",
    "        - [mathematical inequalities](../math-inequalities.ipynb)\n",
    "        - [discrete probability](../prob.ipynb)\n",
    "    + Binomial and hypergeometric tests and confidence intervals\n",
    "        - [testing](../tests.ipynb)\n",
    "        - [duality between tests and confidence sets](../duality.ipynb)\n",
    "        - [confidence sets, tailoring the rejection region, bootstrap percentile CIs](../confidence-sets.ipynb)\n",
    "        - [binomial and hypergeometric simulations; comparison with the normal approximation](../binom.ipynb)\n",
    "    + Randomized, controlled experiments\n",
    "        - [Fisher's exact test](../fisher-exact.ipynb)\n",
    "        - [Causal inference, Neyman's potential outcomes model, experiments with binary outcomes](../causal-inference.ipynb)\n",
    "        - Li, X. and P. Ding, 2016. Exact confidence intervals for the average causal effect on a binary outcome, _Statistics in Medicine_, _35_, 6, 957-960.  10.1002/sim.6764 \n",
    "        - Aronow, P.M., H. Chang, and P. Lopatto, 2022?. Fast computation of exact confidence intervals for randomized experiments with binary outcomes. https://lopat.to/permutation.pdf\n",
    "     + [Inference about binary populations from stratified samples: Wright's method, Wendell & Schmee's method, and union-intersection tests using greedy optimization](../strat-binary.ipynb)\n",
    "     + [The SPRT for Bernoulli trials](../sprt.ipynb)\n",
    "        \n",
    "\n",
    "1. Weeks 4-6: Permutation tests\n",
    "    + [introduction to permutation tests](../permute-intro)\n",
    "    + [PRNGs and simulations](../pseudo-random.ipynb)\n",
    "    + [simulation and computational permutation tests](../permute-sample.ipynb)\n",
    "    + [classical permutation tests based on ranks](../permute-classical.ipynb)\n",
    "    + [tests of randomness and independence](../permute-independence.ipynb)\n",
    "    + Stark, P.B., and K. Ottoboni, 2018. Random Sampling: Practice Makes Imperfect.\n",
    "    + Ivanova, A., S. Lederman, P.B. Stark, G. Sullivan, and B. Vaughn, 2022. Randomization tests in clinical trials with multiple imputation for handling missing data, _Journal of Biopharmaceutical Statistics_, _32_, 441-449. https://doi.org/10.1080/10543406.2022.2080695\n",
    "    + [permutation tests based on the distance between the empirical and the null](../permute-distance.ipynb)\n",
    "    + [combining tests](../tests-combo.ipynb)\n",
    "    + [E-values](../E-values.ipynb)\n",
    "    \n",
    "1. Weeks 7-8: Supermartingale-based tests\n",
    "    + Reading:\n",
    "        - [martingales](../martingales.ipynb)\n",
    "        - nonnegative supermartingales and Ville's inequality\n",
    "        - examples: additive, multiplicative, likelihood ratios, prior-posterior ratio, betting\n",
    "        - the SPRT\n",
    "    + Inference from stratified samples\n",
    "        - combining tests using $E$-values: multiplication and averaging\n",
    "        - stratified inference using union-intersection tests\n",
    "        - product supermartingales: ALPHA and Sweeter than SUITE \n",
    "\n",
    "1. Weeks 9-10: Inference about bounded populations.\n",
    "        - Kaplan, H., 1987. A Method of One-Sided Nonparametric Inference for the Mean of a Nonnegative Population, _The Amer. Statistician_, _41_, 157-158. https://www.tandfonline.com/doi/abs/10.1080/00031305.1987.10475470?journalCode=utas20\n",
    "        - Stark, P.B., 2023. ALPHA: Audit that Learns from Previously Hand-Audited ballots, _Ann. Appl. Stat._, https://www.e-publications.org/ims/submission/AOAS/user/submissionFile/54812?confirm=3a9dc0d4\n",
    "        - Vovk, V., and R. Wang, 2021. $E$-values: Calibration, combination, and applications, http://alrw.net/e/02.pdf\n",
    "        - Waudby-Smith, I. and A. Ramdas, 2022. Estimating means of bounded random variables by betting, https://arxiv.org/abs/2010.09686\n",
    "\n",
    "1. Weeks 11-13: Betting and $E$-values, more nonnegative martingales. Combining $E$-values. Multiple testing. The False Discovery Rate. Controlling FWER and FDR using $E$-values\n",
    "\n",
    "1. Weeks 14-15: Conformal prediction \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab2542a5-132e-4349-be14-e5e0923759fe",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}