| p | N=1 | N=2 | N=3 |
|---|---|---|---|
| 68.3% | $\\Delta \\chi^2$ = 1.00 | $\\Delta \\chi^2$ = 2.30 | $\\Delta \\chi^2$ = 3.53 |
| 95.4% | $\\Delta \\chi^2$ = 4.00 | $\\Delta \\chi^2$ = 6.17 | $\\Delta \\chi^2$ = 8.02 |
| 99.73% | $\\Delta \\chi^2$ = 9.00 | $\\Delta \\chi^2$ = 11.8 | $\\Delta \\chi^2$ = 14.2 |
\n",
" \n",
"
\n",
"
| \n", " | \n", " Low Variance\n", " | \n", "\n", " High Variance\n", " | \n", "
|---|---|---|
| \n", " Low Bias\n", " | \n", "\n", " \n", " | \n", "\n", " \n", " | \n", "
| \n", " High Bias\n", " | \n", "\n", " \n", " | \n", "\n", " \n", " | \n", "
Graphical illustration of bias and variance (http://scott.fortmann-roe.com)
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"When determining model complexity, there are other metrics to use than $\\Delta \\chi^2$, especially when comparing different clases of models. E.g. mean squared error: \n",
"\n",
"MSE = bias$^2$ + variance\n",
"\n",
"![](\"files/images/biasvariance.png\"/)
\n",
"\n",
"*http://scott.fortmann-roe.com*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There are principled methods that help you decide the order of your models by adding a **penalty** to the calculation of $\\chi^2$. These include the Bayesian Information Criterion (BIC) or Akaike Information Criterion (AIC):\n",
"\n",
"BIC = $\\chi^2$ + *Nterm* $\\cdot$ ln(*Ndata*)\n",
"\n",
"AIC = $\\chi^2$ + *Nterm* $\\cdot$ 2 "
]
},
{
"cell_type": "heading",
"level": 2,
"metadata": {},
"source": [
"Real World Example: Fourier Series Fit"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As an example, we will fit a Fourier series to a folded periodic variable star lightcurve from the LINEAR survey.\n",
"\n",
"The code below requires the [astroML](http://www.astroML.org) package, which you can easily install by running\n",
"\n",
"```\n",
"$ pip install astroML\n",
"```\n",
"\n",
"We will examine how $\\chi^2$, BIC, and AIC vary as we change the number of terms in the Fourier series. If we do not get through this whole exercise, your homework will be to finish the notebook, and then repeat the analysis for another variable star from the data. I would like to see as a function of the number of terms in your analysis: $\\chi^2$, $\\Delta \\chi^2$ compared to the previous model, the significance of $\\Delta \\chi^2$, AIC, and BIC. Indicate the order of the series you would choose based on each of the above numbers (they may be different for $\\Delta \\chi^2$, AIC, and BIC)."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"# Uncomment and run this to install astroML\n",
"# !pip install astroML\n",
"# !pip install astroML_addons"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 20
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"%matplotlib inline\n",
"import astroML.datasets\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 21
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"lcs = astroML.datasets.fetch_LINEAR_sample()"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 22
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"print lcs"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"