| Software | Version |
|---|
| Python | 3.6.8 64bit [GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] |
| IPython | 7.2.0 |
| OS | Darwin 18.2.0 x86_64 i386 64bit |
| pandas | 0.23.4 |
| numpy | 1.15.4 |
| Tue Mar 19 21:16:05 2019 PDT |
"
],
"text/latex": [
"\\begin{tabular}{|l|l|}\\hline\n",
"{\\bf Software} & {\\bf Version} \\\\ \\hline\\hline\n",
"Python & 3.6.8 64bit [GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE\\_401/final)] \\\\ \\hline\n",
"IPython & 7.2.0 \\\\ \\hline\n",
"OS & Darwin 18.2.0 x86\\_64 i386 64bit \\\\ \\hline\n",
"pandas & 0.23.4 \\\\ \\hline\n",
"numpy & 1.15.4 \\\\ \\hline\n",
"\\hline \\multicolumn{2}{|l|}{Tue Mar 19 21:16:05 2019 PDT} \\\\ \\hline\n",
"\\end{tabular}\n"
],
"text/plain": [
"Software versions\n",
"Python 3.6.8 64bit [GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]\n",
"IPython 7.2.0\n",
"OS Darwin 18.2.0 x86_64 i386 64bit\n",
"pandas 0.23.4\n",
"numpy 1.15.4\n",
"Tue Mar 19 21:16:05 2019 PDT"
]
},
"execution_count": 79,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# %load jupyter_default.py\n",
"import pandas as pd\n",
"import numpy as np\n",
"import os\n",
"import re\n",
"import datetime\n",
"import time\n",
"import glob\n",
"from tqdm import tqdm_notebook\n",
"from colorama import Fore, Style\n",
"\n",
"%matplotlib inline\n",
"import matplotlib.pyplot as plt\n",
"import matplotlib.colors\n",
"import seaborn as sns\n",
"\n",
"%config InlineBackend.figure_format='retina'\n",
"sns.set() # Revert to matplotlib defaults\n",
"plt.rcParams['figure.figsize'] = (12, 8)\n",
"plt.rcParams['axes.labelpad'] = 20\n",
"plt.rcParams['legend.fancybox'] = True\n",
"plt.style.use('ggplot')\n",
"\n",
"SMALL_SIZE, MEDIUM_SIZE, BIGGER_SIZE = 14, 16, 20\n",
"plt.rc('font', size=SMALL_SIZE)\n",
"plt.rc('axes', titlesize=SMALL_SIZE)\n",
"plt.rc('axes', labelsize=MEDIUM_SIZE)\n",
"plt.rc('xtick', labelsize=SMALL_SIZE)\n",
"plt.rc('ytick', labelsize=SMALL_SIZE)\n",
"plt.rc('legend', fontsize=MEDIUM_SIZE)\n",
"plt.rc('axes', titlesize=BIGGER_SIZE)\n",
"\n",
"def savefig(plt, name):\n",
" plt.savefig(f'../../figures/{name}.png', bbox_inches='tight', dpi=300)\n",
"\n",
"%load_ext version_information\n",
"%version_information pandas, numpy"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Bayesian Modeling Discussion\n",
"\n",
"We can model the probability of an outcome $y$ as $P_t(y)$ using a discrete **Poisson distribution** i.e. if discretizing the time $t$ in seconds.\n",
"\n",
"$$\n",
"P_t(\\mu) = \\frac{\\mu^te^{-\\mu}}{k!}\n",
"$$\n",
"\n",
"Instead we could also assume a Gamma posterior, which has the advantage of being continuous and has more parameters than can be optimized. For now we'll stick with using the simpler Poisson distribution.\n",
"\n",
"Based on a set of goalie pull observations $X$ from 2003-2007 NHL games, we'll solve for the posterior distribution $P_t(y|X)$, the probability of the outcome $y$, given the observations. This is done computationally using markov chain monte carlo and the `pymc3` library.\n",
"\n",
"The outcomes we're interested in are $y = \\big\\{\\mathrm{goal\\;for}, \\mathrm{goal\\;against}, \\mathrm{no\\;goal}\\big\\}$. \n",
"\n",
"We'll use a **uniform prior** over the domain of times (last 5mins). Note: when gathering the observations, we throw out goalie pulls greater than 5 minutes from the end of the game (due to high likelihood of false positives when parsing goalie pulls from the raw game table).\n",
"\n",
"Once we find the posteriors discussed above, we can study the risk reward of pulling a goalie. We'll compare posteriors to find the odds of scoring a goal (and the odds of getting scored on) over time $t$ where:\n",
" - **t = Time elapsed** e.g. if there's 3 minutes left, what is the chance that pulling the goalie will result in a goal for?\n",
" - **t = Time since goalie pull** e.g. after the goalie has been pulled for 1 minute, what is the chance of getting a goal?"
]
},
{
"cell_type": "code",
"execution_count": 80,
"metadata": {},
"outputs": [],
"source": [
"import pymc3 as pm"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Load the training data"
]
},
{
"cell_type": "code",
"execution_count": 81,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"20032004_goalie_pulls_2019-03-01.pkl 20062007_goalie_pulls_2019-03-01.pkl\n",
"20052006_goalie_pulls_2019-03-01.pkl\n"
]
}
],
"source": [
"ls ../../data/processed/pkl/"
]
},
{
"cell_type": "code",
"execution_count": 82,
"metadata": {},
"outputs": [],
"source": [
"def load_data():\n",
" files = glob.glob('../../data/processed/pkl/*.pkl')\n",
" files = sorted(files)\n",
" print(files)\n",
" return pd.concat((pd.read_pickle(f) for f in files))\n",
"\n",
"def clean_df(df):\n",
" _df = df.copy()\n",
" \n",
" len_0 = _df.shape[0]\n",
" print('Removing goal_for_time < 15 mins')\n",
" _df = _df[~(_df.goal_for_time < datetime.timedelta(seconds=15*60))]\n",
" print(f'Removed {len_0 - _df.shape[0]} total rows')\n",
" \n",
" if 'game_end_time' in df.columns:\n",
" len_0 = _df.shape[0]\n",
" print('Removing game_end_time < 15 mins')\n",
" _df = _df[~(_df.game_end_time < datetime.timedelta(seconds=60*15))]\n",
" print(f'Removed {len_0 - _df.shape[0]} total rows')\n",
"\n",
" return _df"
]
},
{
"cell_type": "code",
"execution_count": 83,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['../../data/processed/pkl/20032004_goalie_pulls_2019-03-01.pkl', '../../data/processed/pkl/20052006_goalie_pulls_2019-03-01.pkl', '../../data/processed/pkl/20062007_goalie_pulls_2019-03-01.pkl']\n",
"Removing goal_for_time < 15 mins\n",
"Removed 1 total rows\n"
]
}
],
"source": [
"df = load_data()\n",
"df = clean_df(df)"
]
},
{
"cell_type": "code",
"execution_count": 84,
"metadata": {},
"outputs": [],
"source": [
"def load_training_samples(\n",
" df,\n",
" cols,\n",
" masks=[],\n",
" dtype='timedelta64[s]'\n",
") -> np.ndarray:\n",
" '''\n",
" Return buckets of training data.\n",
" '''\n",
" if not masks:\n",
" masks = [None] * len(cols)\n",
" out = []\n",
" for col, m in zip(cols, masks):\n",
" if m is None:\n",
" d = df[col].dropna().astype(dtype).values\n",
" else:\n",
" d = df[col][m].dropna().astype(dtype).values\n",
" out.append(d)\n",
" print(f'Loaded {len(d)} samples for col {col}')\n",
"\n",
" out = np.array(out)\n",
" print(f'Training data shape = {out.shape}')\n",
" return out"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Model 1 - Time elapsed"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Load data"
]
},
{
"cell_type": "code",
"execution_count": 85,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Loaded 270 samples for col pull_time\n",
"Loaded 554 samples for col pull_time\n",
"Loaded 1305 samples for col pull_time\n",
"Training data shape = (3,)\n"
]
}
],
"source": [
"# Load time of pull for eventual outcomes:\n",
"feature_names = ['goal_for', 'goal_against', 'no_goals']\n",
"\n",
"# Logic for loading the data\n",
"features = ['pull_time', 'pull_time', 'pull_time']\n",
"masks = [\n",
" ~(df.goal_for_time.isnull()),\n",
" ~(df.goal_against_time.isnull()),\n",
" ~(df.game_end_timedelta.isnull()),\n",
"]\n",
"training_samples = load_training_samples(df, features, masks)"
]
},
{
"cell_type": "code",
"execution_count": 86,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(array([1119., 1114., 1109., 1111., 1081., 1121., 1100., 1103., 1145.,\n",
" 1142.]),\n",
" array([1184., 1143., 1168., 1133., 1133., 1140., 1118., 1130., 1148.,\n",
" 1104.]),\n",
" array([1198., 1125., 1068., 1151., 1119., 1137., 1131., 1140., 1192.,\n",
" 1137.]))"
]
},
"execution_count": 86,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"(training_samples[0][:10],\n",
"training_samples[1][:10],\n",
"training_samples[2][:10],)"
]
},
{
"cell_type": "code",
"execution_count": 87,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['goal_for', 'goal_against', 'no_goals']"
]
},
"execution_count": 87,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"feature_names"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### PyMC3 Model"
]
},
{
"cell_type": "code",
"execution_count": 88,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Multiprocess sampling (4 chains in 4 jobs)\n",
"CompoundStep\n",
">Metropolis: [p_no_goal]\n",
">Metropolis: [p_goal_against]\n",
">Metropolis: [p_goal_for]\n",
">Metropolis: [mu_no_goal]\n",
">Metropolis: [mu_goal_against]\n",
">Metropolis: [mu_goal_for]\n",
"Sampling 4 chains: 100%|██████████| 74000/74000 [00:29<00:00, 2512.99draws/s]\n",
"The number of effective samples is smaller than 10% for some parameters.\n"
]
},
{
"data": {
"text/latex": [
"$$\n",
" \\begin{array}{rcl}\n",
" \\text{mu_goal_for} &\\sim & \\text{Uniform}(\\mathit{lower}=900.0,~\\mathit{upper}=1200.0)\\\\\\text{mu_goal_against} &\\sim & \\text{Uniform}(\\mathit{lower}=900.0,~\\mathit{upper}=1200.0)\\\\\\text{mu_no_goal} &\\sim & \\text{Uniform}(\\mathit{lower}=900.0,~\\mathit{upper}=1200.0)\\\\\\text{obs_goal_for} &\\sim & \\text{Poisson}(\\mathit{mu}=\\text{mu_goal_for})\\\\\\text{obs_goal_against} &\\sim & \\text{Poisson}(\\mathit{mu}=\\text{mu_goal_against})\\\\\\text{obs_no_goal} &\\sim & \\text{Poisson}(\\mathit{mu}=\\text{mu_no_goal})\n",
" \\end{array}\n",
" $$"
],
"text/plain": [
"