{
 "metadata": {
  "name": ""
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "View the assignment description [here](http://www.cs.ubc.ca/~nando/540-2013/lectures/homework2.pdf)"
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Exercise 1"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "$\\newcommand{\\vect}[1]{\\boldsymbol{\\mathbf{#1}}}$\n",
      "The MSE (= Mean Squared Error) of the estimator $\\hat{\\vect\\theta}$ is \n",
      "$$MSE\\left(\\hat{\\vect\\theta}\\right) = \\mathbb E _{p\\left( \\mathcal D | \\vect \\theta_0 \\right)} \\left[ \n",
      "\\left(\\hat{\\vect\\theta} - \\vect \\theta_0\\right)^2 \\right]$$\n",
      "where $\\vect \\theta_0$ is the true parameter."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We can rewrite this by subtracting and adding the squared bias term $bias\\left(\\hat{\\vect\\theta}\\right)^2=\n",
      "\\left(\\mathbb E _{p\\left( \\mathcal D | \\vect \\theta_0 \\right)} \\left[\\hat{\\vect \\theta}\\right] - \\vect \\theta_0\\right)^2\n",
      "=\\mathbb E _{p\\left( \\mathcal D | \\vect \\theta_0 \\right)} \\left[\\hat{\\vect \\theta} - \\vect \\theta_0\\right]^2$\n",
      "$$MSE\\left(\\hat{\\vect\\theta}\\right) = \\mathbb E _{p\\left( \\mathcal D | \\vect \\theta_0 \\right)} \\left[ \n",
      "\\left(\\hat{\\vect\\theta} - \\vect \\theta_0\\right)^2 \\right] \n",
      "- \\mathbb E _{p\\left( \\mathcal D | \\vect \\theta_0 \\right)} \\left[\n",
      "        \\hat{\\vect \\theta} - \\vect \\theta_0\\right]^2\n",
      "+ \\mathbb E _{p\\left( \\mathcal D | \\vect \\theta_0 \\right)} \\left[\n",
      "        \\hat{\\vect \\theta} - \\vect \\theta_0\\right]^2$$"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "And now since $Var\\left[ X \\right] = \\mathbb E\\left[ X^2 \\right] - \\mathbb E\\left[ X \\right]^2$\n",
      "$$MSE\\left(\\hat{\\vect\\theta}\\right) = Var\\left[\\hat{\\vect\\theta} - \\vect\\theta_0\\right] + bias\\left(\\hat{\\vect\\theta}\\right)^2$$"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "finally since $Var\\left[ X + c \\right] = Var\\left[X \\right]$\n",
      "$$MSE\\left(\\hat{\\vect\\theta}\\right) = Var\\left[\\hat{\\vect\\theta}\\right] + bias\\left(\\hat{\\vect\\theta}\\right)^2$$"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}