{
 "metadata": {
  "name": ""
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "View the assignment description [here](http://www.cs.ubc.ca/~nando/540-2013/lectures/homework2.pdf)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "$\\newcommand{\\vect}[1]{\\boldsymbol{\\mathbf{#1}}}$\n",
      "Let $\\vect x \\in \\mathbb R^n$ and consider the multivariate Gaussian distribution\n",
      "$$p\\left( \\vect x | \\vect \\mu \\vect, \\vect \\Sigma \\right) = \n",
      "| 2 \\pi \\vect \\Sigma |^{-1/2} \\exp \\left\\{ -\\frac{1}{2} \\left( \\vect x - \\vect \\mu \\right)^T \\vect \\Sigma^{-1} \\left(\\vect x - \\vect \\mu \\right) \\right\\}$$"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The differential entropy of this distribution is given by\n",
      "$$H\\left[ p \\right] = \n",
      "- \\int p\\left(\\vect x\\right) \\log p\\left( \\vect x \\right)\\mathrm d\\vect x$$\n",
      "$$=- \\int p\\left(\\vect x\\right) \\log \\left(\n",
      "| 2 \\pi \\vect \\Sigma |^{-1/2} \\exp \\left\\{ -\\frac{1}{2} \\left( \\vect x - \\vect \\mu \\right)^T \\vect \\Sigma^{-1} \\left(\\vect x - \\vect \\mu \\right) \\right\\} \n",
      "\\right)\\mathrm d\\vect x$$\n",
      "$$ = \\frac{1}{2} \\log |2 \\pi \\vect \\Sigma | \\int p\\left(\\vect x \\right) \\mathrm d \\vect x\n",
      "+ \\frac{1}{2} \\int p\\left(\\vect x \\right) \\left( \\vect x - \\vect \\mu \\right)^T \\vect \\Sigma^{-1} \\left(\\vect x - \\vect \\mu \\right) \\mathrm d \\vect x$$\n",
      "$$ = \\frac{1}{2} \\log |2 \\pi \\vect \\Sigma |\n",
      "+ \\frac{1}{2} \\mathbb E \\left[ \\left( \\vect x - \\vect \\mu \\right)^T \\vect \\Sigma^{-1} \\left(\\vect x - \\vect \\mu \\right) \\right]$$\n",
      "$$ = \\frac{1}{2} \\log |2 \\pi \\vect \\Sigma |\n",
      "+ \\frac{1}{2} \\mathbb E \\left[ \\sum_{i=1}^n \\sum_{j=1}^n \\vect \\Sigma_{ij}^{-1} \\left(\\vect x - \\vect \\mu\\right)_j \\left(\\vect x - \\vect \\mu\\right)_i \\right]$$\n",
      "$$ = \\frac{1}{2} \\log |2 \\pi \\vect \\Sigma |\n",
      "+ \\frac{1}{2} \\sum_{i=1}^n \\sum_{j=1}^n \\mathbb E \\left[  \\left(\\vect x - \\vect \\mu\\right)_j \\left(\\vect x - \\vect \\mu\\right)_i \\right] \\vect \\Sigma_{ij}^{-1}$$\n",
      "$$ = \\frac{1}{2} \\log |2 \\pi \\vect \\Sigma |\n",
      "+ \\frac{1}{2} \\sum_{j=1}^n \\sum_{i=1}^n \\vect \\Sigma_{ji} \\vect \\Sigma_{ij}^{-1} $$\n",
      "$$ = \\frac{1}{2} \\log |2 \\pi \\vect \\Sigma |\n",
      "+ \\frac{1}{2} \\sum_{j=1}^n \\left( \\vect \\Sigma \\vect \\Sigma^{-1}\\right)_{jj} $$\n",
      "$$ = \\frac{1}{2} \\log |2 \\pi \\vect \\Sigma |\n",
      "+ \\frac{1}{2} n $$\n",
      "$$ = \\frac{1}{2} \\log |2 \\pi \\vect \\Sigma |\n",
      "+ \\frac{1}{2} \\log e^n = \\frac{1}{2} \\log |2 \\pi e \\Sigma| $$"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}