{ "metadata": { "name": "" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "View the assignment description [here](http://www.cs.ubc.ca/~nando/540-2013/lectures/homework2.pdf)" ] }, { "cell_type": "heading", "level": 3, "metadata": {}, "source": [ "Exercise 1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "$\\newcommand{\\vect}[1]{\\boldsymbol{\\mathbf{#1}}}$\n", "The MSE (= Mean Squared Error) of the estimator $\\hat{\\vect\\theta}$ is \n", "$$MSE\\left(\\hat{\\vect\\theta}\\right) = \\mathbb E _{p\\left( \\mathcal D | \\vect \\theta_0 \\right)} \\left[ \n", "\\left(\\hat{\\vect\\theta} - \\vect \\theta_0\\right)^2 \\right]$$\n", "where $\\vect \\theta_0$ is the true parameter." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can rewrite this by subtracting and adding the squared bias term $bias\\left(\\hat{\\vect\\theta}\\right)^2=\n", "\\left(\\mathbb E _{p\\left( \\mathcal D | \\vect \\theta_0 \\right)} \\left[\\hat{\\vect \\theta}\\right] - \\vect \\theta_0\\right)^2\n", "=\\mathbb E _{p\\left( \\mathcal D | \\vect \\theta_0 \\right)} \\left[\\hat{\\vect \\theta} - \\vect \\theta_0\\right]^2$\n", "$$MSE\\left(\\hat{\\vect\\theta}\\right) = \\mathbb E _{p\\left( \\mathcal D | \\vect \\theta_0 \\right)} \\left[ \n", "\\left(\\hat{\\vect\\theta} - \\vect \\theta_0\\right)^2 \\right] \n", "- \\mathbb E _{p\\left( \\mathcal D | \\vect \\theta_0 \\right)} \\left[\n", " \\hat{\\vect \\theta} - \\vect \\theta_0\\right]^2\n", "+ \\mathbb E _{p\\left( \\mathcal D | \\vect \\theta_0 \\right)} \\left[\n", " \\hat{\\vect \\theta} - \\vect \\theta_0\\right]^2$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And now since $Var\\left[ X \\right] = \\mathbb E\\left[ X^2 \\right] - \\mathbb E\\left[ X \\right]^2$\n", "$$MSE\\left(\\hat{\\vect\\theta}\\right) = Var\\left[\\hat{\\vect\\theta} - \\vect\\theta_0\\right] + bias\\left(\\hat{\\vect\\theta}\\right)^2$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "finally since $Var\\left[ X + c \\right] = Var\\left[X \\right]$\n", "$$MSE\\left(\\hat{\\vect\\theta}\\right) = Var\\left[\\hat{\\vect\\theta}\\right] + bias\\left(\\hat{\\vect\\theta}\\right)^2$$" ] } ], "metadata": {} } ] }