{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4d05d6b8",
   "metadata": {},
   "source": [
    "# Continuous Data and the Gaussian Distribution\n",
    "\n",
    "\n",
    "- **[1]** (##) We are given an IID data set $D = \\{x_1,x_2,\\ldots,x_N\\}$, where $x_n \\in \\mathbb{R}^M$. Let's assume that the data were drawn from a multivariate Gaussian (MVG),\n",
    "$$\\begin{align*}\n",
    "p(x_n|\\theta) = \\mathcal{N}(x_n|\\,\\mu,\\Sigma) = \\frac{1}{\\sqrt{(2 \\pi)^{M} |\\Sigma|}} \\exp\\left\\{-\\frac{1}{2}(x_n-\\mu)^T\n",
    "\\Sigma^{-1} (x_n-\\mu) \\right\\}\n",
    "\\end{align*}$$      \n",
    "  (a) Derive the log-likelihood of the parameters for these data.       \n",
    "  (b) Derive the maximum likelihood estimates for the mean $\\mu$ and variance $\\Sigma$ by setting the derivative of the log-likelihood to zero.\n",
    "\n",
    "\n",
    "- **[2]** (#) Shortly explain why the Gaussian distribution is often preferred as a prior distribution over other distributions with the same support?\n",
    "\n",
    "- **[3]** (###) We make $N$ IID observations $D=\\{x_1 \\dots x_N\\}$ and assume the following model\n",
    "$$\\begin{aligned}\n",
    "x_k &= A + \\epsilon_k \\\\\n",
    "A &\\sim \\mathcal{N}(m_A,v_A) \\\\\n",
    "\\epsilon_k &\\sim \\mathcal{N}(0,\\sigma^2) \\,.\n",
    "\\end{aligned}$$\n",
    "We assume that $\\sigma$ has a known value and are interested in deriving an estimator for $A$ .   \n",
    "  (a) Derive the Bayesian (posterior) estimate $p(A|D)$.     \n",
    "  (b) (##) Derive the Maximum Likelihood estimate for $A$.      \n",
    "  (c) Derive the MAP estimates for $A$.    \n",
    "  (d) Now assume that we do not know the variance of the noise term? Describe the procedure for Bayesian estimation of both $A$ and $\\sigma^2$ (No need to fully work out to closed-form estimates). \n",
    "\n",
    "- **[4]** (##) Proof that a linear transformation $z=Ax+b$ of a Gaussian variable $\\mathcal{N}(x|\\mu,\\Sigma)$ is Gaussian distributed as\n",
    "$$\n",
    "p(z) = \\mathcal{N} \\left(z \\,|\\, A\\mu+b, A\\Sigma A^T \\right) \n",
    "$$    \n",
    "\n",
    "\n",
    "- **[5]** (#) Given independent variables\n",
    "$x \\sim \\mathcal{N}(\\mu_x,\\sigma_x^2)$ and $y \\sim \\mathcal{N}(\\mu_y,\\sigma_y^2)$, what is the PDF for $z = A\\cdot(x -y) + b$?    \n",
    "\n",
    "\n",
    "\n",
    "- **[6]** (###) Compute\n",
    "\n",
    "\\begin{equation*}\n",
    "        \\int_{-\\infty}^{\\infty} \\exp(-x^2)\\mathrm{d}x \\,.\n",
    "    \\end{equation*}\n",
    "   \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "<!---\n",
    "The following computation is due to Poisson. First we note that the integrand is an even function so we can write\n",
    "$$ \\int_{-\\infty}^{\\infty} \\exp(-x^2)\\mathrm{d}x = 2\\int_{0}^{\\infty} \\exp(-x^2)\\mathrm{d}x \\,.$$ Let the right hand side be called $J$. Then we can write\n",
    "$$\\begin{align*}\n",
    "J^2 &= \\int_{0}^{\\infty} \\int_{0}^{\\infty}\\exp\\left(-(x^2+y^2\\right)\\mathrm{d}x\\mathrm{d}y \\\\\n",
    "&= \\int_0^{\\infty}r\\exp(-r^2)\\mathrm{d}r\\int_0^{\\frac{\\pi}{2}}\\mathrm{d}\\theta \\\\\n",
    "&= \\frac{\\pi}{2}\\frac{1}{2}\\left.\\exp(-r^2)\\right\\vert_{0}^{\\infty} \\\\\n",
    "&= \\frac{\\pi}{4}\\end{align*}$$    and consequently\n",
    "$$\\begin{equation*}\n",
    "\\int_{-\\infty}^{\\infty} \\exp(-x^2)\\mathrm{d}x = \\sqrt{\\pi}\n",
    "\\end{equation*}$$\n",
    "where the polar coordinate transformation is used and the 2d integral is taken over the first quadrant.\n",
    "\n",
    "- Derive the conditional distribution $p(x_a|x_b)$ and the marginal distribution $p(x_a)$ given that \n",
    "$$\n",
    "\\begin{bmatrix} x_a \\\\ x_b \\end{bmatrix} \\sim \\mathcal{N}\\left(\\begin{bmatrix} m_a \\\\m_b \\end{bmatrix} , \\begin{bmatrix} \\Sigma_a ~ \\Sigma_{ab} \\\\ \\Sigma_{ba} ~  \\Sigma_b\\end{bmatrix}\\right),\n",
    "$$\n",
    "where $x_a$ and $x_b$ are vectors. You may make use of \n",
    "$$\n",
    "       \\begin{bmatrix} A ~ B \\\\ C ~ D \\end{bmatrix}^{-1} = \\begin{bmatrix} M \\qquad -MBD^{-1} \\\\ -D^{-1}CM \\qquad D^{-1}+D^{-1}CMBD^{-1} \\end{bmatrix}\n",
    "$$\n",
    "where $M = (A - BD^{-1}C)^{-1}$ is called the Schur's complement.\n",
    "\n",
    "    \n",
    "- In the notebook, for the model \n",
    "$$\\begin{align*}\n",
    "p(x_t |\\theta) &= \\mathcal{N}(x_t\\,|\\,\\theta,\\sigma^2) \\\\\n",
    "p(\\theta) &= \\mathcal{N}(\\theta\\,|\\,\\mu_0,\\sigma_0^2)\n",
    "\\end{align*}$$\n",
    "we found the following posteestimator for the hidden states:\n",
    "$$\\begin{align*}\n",
    "p(\\theta|x) &= \\mathcal{N} \\left( \\theta\\,|\\,\\mu_1, \\sigma_1^2 \\right)\\,,\n",
    "\\end{align*}$$\n",
    "with\n",
    "$$\\begin{align*}\n",
    "K &= \\frac{\\sigma_0^2}{\\sigma_0^2+\\sigma^2} \\qquad \\text{($K$ is called: Kalman gain)}\\\\\n",
    "\\mu_1 &= \\mu_0 + K \\cdot (x-\\mu_0)\\\\\n",
    "\\sigma_1^2 &= \\left( 1-K \\right) \\sigma_0^2  \n",
    "\\end{align*}$$\n",
    " \n",
    "\n",
    "\n",
    "- Show that Eq.SRG-8 is a special case of Eq.SRG-4a. \n",
    "\n",
    "- Proof\n",
    "$$\n",
    "p(x,\\theta) = \\mathcal{N} \\left( \\begin{bmatrix} x\\\\ \n",
    "  \\theta  \\end{bmatrix} \n",
    "  \\,\\left|\\, \\begin{bmatrix} \\mu_0\\\\ \n",
    "  \\mu_0\\end{bmatrix}, \n",
    "         \\begin{bmatrix} \\sigma_0^2+\\sigma^2  & \\sigma_0^2\\\\ \n",
    "         \\sigma_0^2 &\\sigma_0^2 \n",
    "  \\end{bmatrix} \n",
    "  \\right. \\right)\n",
    "$$\n",
    "- Look up conditioning and marginalization in canonical coordinates and compare to the formulas for the moment parameterization of the Gaussian. Any conclusions?\n",
    "\n",
    "- Derive the conditional distribution $p(x_a|x_b)$ and the marginal distribution $p(x_a)$ given that \n",
    "$$\n",
    "\\begin{bmatrix} x_a \\\\ x_b \\end{bmatrix} \\sim \\mathcal{N}\\left(\\begin{bmatrix} m_a \\\\m_b \\end{bmatrix} , \\begin{bmatrix} \\Sigma_a ~ \\Sigma_{ab} \\\\ \\Sigma_{ba} ~  \\Sigma_b\\end{bmatrix}\\right),\n",
    "$$\n",
    "where $x_a$ and $x_b$ are vectors. You may make use of \n",
    "$$\n",
    "       \\begin{bmatrix} A ~ B \\\\ C ~ D \\end{bmatrix}^{-1} = \\begin{bmatrix} M \\qquad -MBD^{-1} \\\\ -D^{-1}CM \\qquad D^{-1}+D^{-1}CMBD^{-1} \\end{bmatrix}\n",
    "$$\n",
    "where $M = (A - BD^{-1}C)^{-1}$ is called the Schur's complement.\n",
    "\n",
    "    \n",
    "- In the notebook, for the model \n",
    "$$\\begin{align*}\n",
    "p(x_t |\\theta) &= \\mathcal{N}(x_t\\,|\\,\\theta,\\sigma^2) \\\\\n",
    "p(\\theta) &= \\mathcal{N}(\\theta\\,|\\,\\mu_0,\\sigma_0^2)\n",
    "\\end{align*}$$\n",
    "we found the following posteestimator for the hidden states:\n",
    "$$\\begin{align*}\n",
    "p(\\theta|x) &= \\mathcal{N} \\left( \\theta\\,|\\,\\mu_1, \\sigma_1^2 \\right)\\,,\n",
    "\\end{align*}$$\n",
    "with\n",
    "$$\\begin{align*}\n",
    "K &= \\frac{\\sigma_0^2}{\\sigma_0^2+\\sigma^2} \\qquad \\text{($K$ is called: Kalman gain)}\\\\\n",
    "\\mu_1 &= \\mu_0 + K \\cdot (x-\\mu_0)\\\\\n",
    "\\sigma_1^2 &= \\left( 1-K \\right) \\sigma_0^2  \n",
    "\\end{align*}$$\n",
    " \n",
    "\n",
    "\n",
    "- Show that Eq.SRG-8 is a special case of Eq.SRG-4a. \n",
    "\n",
    "- Proof\n",
    "$$\n",
    "p(x,\\theta) = \\mathcal{N} \\left( \\begin{bmatrix} x\\\\ \n",
    "  \\theta  \\end{bmatrix} \n",
    "  \\,\\left|\\, \\begin{bmatrix} \\mu_0\\\\ \n",
    "  \\mu_0\\end{bmatrix}, \n",
    "         \\begin{bmatrix} \\sigma_0^2+\\sigma^2  & \\sigma_0^2\\\\ \n",
    "         \\sigma_0^2 &\\sigma_0^2 \n",
    "  \\end{bmatrix} \n",
    "  \\right. \\right)\n",
    "$$\n",
    "- Look up conditioning and marginalization in canonical coordinates and compare to the formulas for the moment parameterization of the Gaussian. Any conclusions?\n",
    "--->\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ca67ac1",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.8.2",
   "language": "julia",
   "name": "julia-1.8"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}