{ "metadata": { "name": "" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "View the assignment description [here](http://www.cs.ubc.ca/~nando/540-2013/lectures/homework2.pdf)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "$\\newcommand{\\vect}[1]{\\boldsymbol{\\mathbf{#1}}}$\n", "Let $\\vect x \\in \\mathbb R^n$ and consider the multivariate Gaussian distribution\n", "$$p\\left( \\vect x | \\vect \\mu \\vect, \\vect \\Sigma \\right) = \n", "| 2 \\pi \\vect \\Sigma |^{-1/2} \\exp \\left\\{ -\\frac{1}{2} \\left( \\vect x - \\vect \\mu \\right)^T \\vect \\Sigma^{-1} \\left(\\vect x - \\vect \\mu \\right) \\right\\}$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The differential entropy of this distribution is given by\n", "$$H\\left[ p \\right] = \n", "- \\int p\\left(\\vect x\\right) \\log p\\left( \\vect x \\right)\\mathrm d\\vect x$$\n", "$$=- \\int p\\left(\\vect x\\right) \\log \\left(\n", "| 2 \\pi \\vect \\Sigma |^{-1/2} \\exp \\left\\{ -\\frac{1}{2} \\left( \\vect x - \\vect \\mu \\right)^T \\vect \\Sigma^{-1} \\left(\\vect x - \\vect \\mu \\right) \\right\\} \n", "\\right)\\mathrm d\\vect x$$\n", "$$= \\frac{1}{2} \\log |2 \\pi \\vect \\Sigma | \\int p\\left(\\vect x \\right) \\mathrm d \\vect x\n", "+ \\frac{1}{2} \\int p\\left(\\vect x \\right) \\left( \\vect x - \\vect \\mu \\right)^T \\vect \\Sigma^{-1} \\left(\\vect x - \\vect \\mu \\right) \\mathrm d \\vect x$$\n", "$$= \\frac{1}{2} \\log |2 \\pi \\vect \\Sigma |\n", "+ \\frac{1}{2} \\mathbb E \\left[ \\left( \\vect x - \\vect \\mu \\right)^T \\vect \\Sigma^{-1} \\left(\\vect x - \\vect \\mu \\right) \\right]$$\n", "$$= \\frac{1}{2} \\log |2 \\pi \\vect \\Sigma |\n", "+ \\frac{1}{2} \\mathbb E \\left[ \\sum_{i=1}^n \\sum_{j=1}^n \\vect \\Sigma_{ij}^{-1} \\left(\\vect x - \\vect \\mu\\right)_j \\left(\\vect x - \\vect \\mu\\right)_i \\right]$$\n", "$$= \\frac{1}{2} \\log |2 \\pi \\vect \\Sigma |\n", "+ \\frac{1}{2} \\sum_{i=1}^n \\sum_{j=1}^n \\mathbb E \\left[ \\left(\\vect x - \\vect \\mu\\right)_j \\left(\\vect x - \\vect \\mu\\right)_i \\right] \\vect \\Sigma_{ij}^{-1}$$\n", "$$= \\frac{1}{2} \\log |2 \\pi \\vect \\Sigma |\n", "+ \\frac{1}{2} \\sum_{j=1}^n \\sum_{i=1}^n \\vect \\Sigma_{ji} \\vect \\Sigma_{ij}^{-1}$$\n", "$$= \\frac{1}{2} \\log |2 \\pi \\vect \\Sigma |\n", "+ \\frac{1}{2} \\sum_{j=1}^n \\left( \\vect \\Sigma \\vect \\Sigma^{-1}\\right)_{jj}$$\n", "$$= \\frac{1}{2} \\log |2 \\pi \\vect \\Sigma |\n", "+ \\frac{1}{2} n$$\n", "$$= \\frac{1}{2} \\log |2 \\pi \\vect \\Sigma |\n", "+ \\frac{1}{2} \\log e^n = \\frac{1}{2} \\log |2 \\pi e \\Sigma|$$" ] } ], "metadata": {} } ] }