{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extension to Multiple Linear Regression\n",
    "=======================================\n",
    "\n",
    "Consider the multiple regression model\n",
    "\n",
    "$$y_{i}=\\beta_{0}+\\sum_{j}x_{ij}\\beta_{j}+e_{i}\\tag{\n",
    "2}$$\n",
    "\n",
    "which extends model (1) to include multiple covariates $x_{ij}$. In\n",
    "matrix notation, this model can be written as\n",
    "\n",
    "$$\n",
    "\\mathbf{y}=\\mathbf{X}\\boldsymbol{\\beta}+\\mathbf{e},\n",
    "$$ \n",
    "\n",
    "where **$\\boldsymbol{\\beta}'=[\\beta_{0,}\\beta_{1},\\beta_{2},\\ldots,\\beta_{k}]$\n",
    "**and the matrix $\\mathbf{X}$ contains the corresponding covariates."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model with Normal Prior for Regression Coefficients \n",
    "----------------------------------------------------\n",
    "\n",
    "Here we consider a model with a flat prior for $\\beta_{0}$ and iid\n",
    "normal priors for the slopes:\n",
    "\n",
    "$$\n",
    "\\beta_{j}\\sim N(0,\\sigma_{\\beta}^{2})\\:\\text{for}\\, j=1,2,\\ldots,k,\n",
    "$$\n",
    "\n",
    "where $\\sigma_{\\beta}^{2}$ is assumed to be known. The residuals are\n",
    "assumed iid normal with null mean and variance $\\sigma_{e}^{2}$, which\n",
    "itself is assigned a scaled inverted chi-square prior. Then, the joint\n",
    "posterior for $\\boldsymbol{\\theta}$ is\n",
    "\n",
    "$$\\begin{aligned}\n",
    "f(\\boldsymbol{\\theta}|\\mathbf{y}) \n",
    "& \\propto f(\\mathbf{y}|\\boldsymbol{\\theta})f(\\boldsymbol{\\theta})\\\\\n",
    "& \\propto \\left(\\sigma_{e}^{2}\\right)^{-n/2}\\exp\\left\\{ -\\frac{(\\mathbf{y}-\\mathbf{\\mathbf{X}}\\boldsymbol{\\beta})'(\\mathbf{y}-\\mathbf{X}\\boldsymbol{\\beta})}{2\\sigma_{e}^{2}}\\right\\} \\\\\n",
    "& \\times \\left(\\sigma_{\\beta}^{2}\\right)^{-k/2}\\exp\\left\\{ -\\frac{\\sum_{j=1}^{k}\\beta_{j}^{2}}{2\\sigma_{\\beta}^{2}}\\right\\} \\\\\n",
    "& \\times (\\sigma_{e}^{2})^{-(2+\\nu_{e})/2}\\exp\\left\\{ -\\frac{\\nu_{e}S_{e}^{2}}{2\\sigma_{e}^{2}}\\right\\} .\\\\\n",
    "\\\\\\end{aligned}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The posterior distribution for $\\boldsymbol{\\beta}$ can be written as\n",
    "\n",
    "$$\\begin{aligned}\n",
    "f(\\boldsymbol{\\beta}|\\mathbf{y},\\sigma_{\\beta}^{2},\\sigma_{e}^{2})\n",
    "&  =\\frac{f(\\mathbf{y}|\\boldsymbol{\\beta},\\sigma_{\\beta}^{2},\\sigma_{e}^{2})f(\\boldsymbol{\\beta}|\\sigma_{\\beta}^{2})f(\\sigma_{e}^{2})}{f(\\mathbf{y},\\sigma_{\\beta}^{2},\\sigma_{e}^{2})}\\\\ \n",
    "& \\propto  f(\\mathbf{y}|\\boldsymbol{\\beta},\\sigma_{\\beta}^{2},\\sigma_{e}^{2})f(\\boldsymbol{\\beta}|\\sigma_{\\beta}^{2})f(\\sigma_{e}^{2})\\\\\n",
    "& \\propto f(\\mathbf{y}|\\boldsymbol{\\beta},\\sigma_{\\beta}^{2},\\sigma_{e}^{2})f(\\boldsymbol{\\beta}|\\sigma_{\\beta}^{2})\\\\\n",
    "& \\propto \\left(\\sigma_{e}^{2}\\right)^{-n/2}\\exp\\left\\{ -\\frac{(\\mathbf{y}-\\mathbf{\\mathbf{X}}\\boldsymbol{\\beta})'(\\mathbf{y}-\\mathbf{X}\\boldsymbol{\\beta})}{2\\sigma_{e}^{2}}\\right\\} \\\\\n",
    "& \\times \\left(\\sigma_{\\beta}^{2}\\right)^{-k/2}\\exp\\left\\{ -\\frac{\\sum_{j=1}^{k}\\beta_{j}^{2}}{2\\sigma_{\\beta}^{2}}\\right\\} \\\\\n",
    "& \\propto \\exp\\left\\{ -\\frac{(\\mathbf{y}-\\mathbf{\\mathbf{X}}\\boldsymbol{\\beta})'(\\mathbf{y}-\\mathbf{X}\\boldsymbol{\\beta})+\\sum_{j=1}^{k}\\beta_{j}^{2}\\frac{\\sigma_{e}^{2}}{\\sigma_{\\beta}^{2}}}{2\\sigma_{e}^{2}}\\right\\} \\\\\n",
    "& \\propto \\exp\\left\\{ -\\frac{\\mathbf{y}'\\mathbf{y}-2\\mathbf{y}'\\mathbf{X}\\boldsymbol{\\beta}+\\boldsymbol{\\beta}'(\\mathbf{X}'\\mathbf{X}+\\mathbf{D}\\frac{\\sigma_{e}^{2}}{\\sigma_{\\beta}^{2}})\\boldsymbol{\\beta}}{2\\sigma_{e}^{2}}\\right\\} \\\\\n",
    "& \\propto  \\exp\\left\\{ -\\frac{\\mathbf{y}'\\mathbf{y}-(\\boldsymbol{\\beta}-\\hat{\\boldsymbol{\\beta})}'(\\mathbf{X}'\\mathbf{X}+\\mathbf{D}\\frac{\\sigma_{e}^{2}}{\\sigma_{\\beta}^{2}})(\\boldsymbol{\\beta}-\\hat{\\boldsymbol{\\beta})}-\\hat{\\boldsymbol{\\beta}}'(\\mathbf{X}'\\mathbf{X}+\\mathbf{D}\\frac{\\sigma_{e}^{2}}{\\sigma_{\\beta}^{2}})\\hat{\\boldsymbol{\\beta}}}{2\\sigma_{e}^{2}}\\right\\} \\\\\n",
    "& \\propto  \\exp\\left\\{ -\\frac{(\\boldsymbol{\\beta}-\\hat{\\boldsymbol{\\beta})}'(\\mathbf{X}'\\mathbf{X}+\\mathbf{D}\\frac{\\sigma_{e}^{2}}{\\sigma_{\\beta}^{2}})(\\boldsymbol{\\beta}-\\hat{\\boldsymbol{\\beta})}}{2\\sigma_{e}^{2}}\\right\\} ,\n",
    "\\end{aligned}$$\n",
    "\n",
    "for\n",
    "\n",
    "$$(\\mathbf{X}'\\mathbf{X}+\\mathbf{D}\\frac{\\sigma_{e}^{2}}{\\sigma_{\\beta}^{2}})\\hat{\\boldsymbol{\\beta}}=\\mathbf{X}'\\mathbf{y},\\tag{3}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where $\\mathbf{D}$ is a diagonal matrix with zero on the first diagonal\n",
    "and ones on the remaining diagonals. Thus, the full-conditional\n",
    "posterior for $\\boldsymbol{\\beta}$ is a normal distribution with mean\n",
    "given by (3) and variance\n",
    "$(\\mathbf{X}'\\mathbf{X}+\\mathbf{D}\\frac{\\sigma_{e}^{2}}{\\sigma_{\\beta}^{2}})^{-1}\\sigma_{e}^{2}.$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Full-conditionals:\n",
    "\n",
    "The full conditionals for $\\beta_{0}$ and $\\sigma_{e}^{2}$ are identical\n",
    "to those in simple linear regression.\n",
    "\n",
    "### Full-conditional for $\\beta_{j}$\n",
    "\n",
    "The full-conditional for $\\beta_{j}$ is obtained by dropping from the\n",
    "joint posterior all terms and factors that do not involve $\\beta_{j}$:\n",
    "\n",
    "$$\\begin{aligned}\n",
    "f(\\beta_{j}|\\text{ELSE}) \n",
    "& \\propto  \\exp\\left\\{ -\\frac{(\\mathbf{w}_{j}-\\mathbf{x}_{j}\\beta_{j})'(\\mathbf{w}_{j}-\\mathbf{x}_{j}\\beta_{j})}{2\\sigma_{e}^{2}}\\right\\} \\\\\n",
    "& \\times  \\exp\\left\\{ -\\frac{\\beta_{j}^{2}}{2\\sigma_{\\beta}^{2}}\\right\\} \\\\\n",
    "& \\propto  \\exp\\left\\{ -\\frac{\\mathbf{w}_{j}'\\mathbf{w}_{j}-2\\mathbf{w}_{j}'\\mathbf{x}_{j}\\beta_{j}+\\beta_{j}^{2}(\\mathbf{x}_{j}'\\mathbf{x}_{j}+\\sigma_{e}^{2}/\\sigma_{\\beta}^{2})}{2\\sigma_{e}^{2}}\\right\\} \\\\\n",
    "& \\propto  \\exp\\left\\{ -\\frac{\\mathbf{w}_{j}'\\mathbf{w}_{j}-(\\beta_{j}-\\hat{\\beta}_{j})^{2}(\\mathbf{x}_{j}'\\mathbf{x}_{j}+\\sigma_{e}^{2}/\\sigma_{\\beta}^{2})-\\hat{\\beta}_{j}^{2}(\\mathbf{x}_{j}'\\mathbf{x}_{j}+\\sigma_{e}^{2}/\\sigma_{\\beta}^{2})}{2\\sigma_{e}^{2}}\\right\\} \\\\\n",
    "& \\propto  \\exp\\left\\{ -\\frac{(\\beta_{j}-\\hat{\\beta}_{j})^{2}}{\\frac{2\\sigma_{e}^{2}}{(\\mathbf{x}_{j}'\\mathbf{x}_{j}+\\sigma_{e}^{2}/\\sigma_{\\beta}^{2})}}\\right\\} ,\n",
    "\\end{aligned}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where\n",
    "$$\n",
    "\\hat{\\beta}_{j}=\\frac{\\mathbf{x}_{j}'\\mathbf{w}_{j}}{(\\mathbf{x}_{j}'\\mathbf{x}_{j}+\\sigma_{e}^{2}/\\sigma_{\\beta}^{2})},\n",
    "$$\n",
    "and $\\mathbf{w}_{j}=\\mathbf{y}- \\mathbf{1}\\beta_0 - \\sum_{l\\neq j}\\mathbf{x}_{l}\\beta_{l}.$\n",
    "So, the full-conditional posterior for $\\beta_{j}$ is a normal\n",
    "distribution with mean $\\hat{\\beta}_{j}$ and variance\n",
    "$\\frac{\\sigma_{e}^{2}}{(\\mathbf{x}_{j}'\\mathbf{x}_{j}+\\sigma_{e}^{2}/\\sigma_{\\beta}^{2})}.$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Exercise\n",
    "--------\n",
    "\n",
    "1.  Use $\\beta_0=1$, $\\sigma^2_{\\beta}=0.1$ and $\\sigma^2_e = 1.0$ to \n",
    "generate a data set with 10 observations from model (2) with $k=15$ covariates. \n",
    "\n",
    "2.  Setup and solve the mixed model equations given by (3).\n",
    "\n",
    "3.  Sample the elements of $\\boldsymbol{\\beta}$ using Gibbs.\n",
    "\n",
    "4.  Compute the posterior mean of $\\boldsymbol{\\beta}$ from the samples\n",
    "    and compare with the mixed model solutions.\n",
    "\n",
    "5.  Compute the posterior covariance matrix from the sampled values.\n",
    "    Compare results with inverse of the mixed-model coefficient matrix."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model with unknown $\\sigma_{\\beta}^{2}$\n",
    "---------------------------------------\n",
    "\n",
    "In the previous section, we assumed that $\\sigma_{\\beta}^{2}$ in the\n",
    "prior of the slopes was known. Here, we will consider this variance to\n",
    "be unknown with a scaled inverted chi-square prior with scale parameter\n",
    "$S_{\\beta}^{2}$ and degrees of freedom $\\nu_{\\beta}.$ The joint\n",
    "posterior for this model is\n",
    "\n",
    "$$\\begin{aligned}\n",
    "f(\\boldsymbol{\\theta}|\\mathbf{y}) \n",
    "& \\propto f(\\mathbf{y}|\\boldsymbol{\\theta})f(\\boldsymbol{\\theta})\\\\\n",
    "& \\propto \\left(\\sigma_{e}^{2}\\right)^{-n/2}\\exp\\left\\{ -\\frac{(\\mathbf{y}-\\mathbf{\\mathbf{X}}\\boldsymbol{\\beta})'(\\mathbf{y}-\\mathbf{X}\\boldsymbol{\\beta})}{2\\sigma_{e}^{2}}\\right\\} \\\\\n",
    "& \\times \\left(\\sigma_{\\beta}^{2}\\right)^{-k/2}\\exp\\left\\{ -\\frac{\\sum_{j=1}^{k}\\beta_{j}^{2}}{2\\sigma_{\\beta}^{2}}\\right\\} \\\\\n",
    "& \\times \\left(\\sigma_{\\beta}^{2}\\right)^{-(2+\\nu_{\\beta})/2}\\exp\\left\\{ -\\frac{\\nu_{\\beta}S_{\\beta}^{2}}{2\\sigma_{\\beta}^{2}}\\right\\} \\\\\n",
    "& \\times (\\sigma_{e}^{2})^{-(2+\\nu_{e})/2}\\exp\\left\\{ -\\frac{\\nu_{e}S_{e}^{2}}{2\\sigma_{e}^{2}}\\right\\} .\\\\\\end{aligned}$$\n",
    "\n",
    "Then, the full-conditional posterior for $\\sigma_{\\beta}^{2}$ is\n",
    "\n",
    "$$\\begin{aligned}\n",
    "f(\\sigma_{\\beta}^{2}|\\mathbf{y},\\boldsymbol{\\beta},\\sigma_{e}^{2}) \n",
    "& \\propto \\left(\\sigma_{\\beta}^{2}\\right)^{-k/2}\\exp\\left\\{ -\\frac{\\sum_{j=1}^{k}\\beta_{j}^{2}}{2\\sigma_{\\beta}^{2}}\\right\\} \\\\\n",
    "& \\times  \\left(\\sigma_{\\beta}^{2}\\right)^{-(2+\\nu_{\\beta})/2}\\exp\\left\\{ -\\frac{\\nu_{\\beta}S_{\\beta}^{2}}{2\\sigma_{\\beta}^{2}}\\right\\} \\\\\n",
    "& \\propto  \\left(\\sigma_{\\beta}^{2}\\right)^{-(2+k+\\nu_{\\beta})/2}\\exp\\left\\{ -\\frac{\\sum_{j=1}^{k}\\beta_{j}^{2}+\\nu_{\\beta}S_{\\beta}^{2}}{2\\sigma_{\\beta}^{2}}\\right\\} ,\n",
    "\\end{aligned}$$\n",
    "\n",
    "which can be recognized as a scaled inverted chi-square distribution\n",
    "with $\\tilde{\\nu}_{\\beta}=k+\\nu_{\\beta}$ degrees of freedom and scale\n",
    "parameter\n",
    "$\\tilde{S}_{\\beta}^{2}=(\\sum_{j=1}^{k}\\beta_{j}^{2}+\\nu_{\\beta}S_{\\beta}^{2})/\\tilde{\\nu}_{\\beta}.$\n",
    "A sample from this posterior can be obtained as\n",
    "$\\frac{\\sum_{j=1}^{k}\\beta_{j}^{2}+\\nu_{\\beta}S_{\\beta}^{2}}{\\chi_{\\tilde{\\nu}_{\\beta}}^{2}}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Extend the sampler used in the previous section to treat\n",
    "$\\sigma_{\\beta}^{2}$ as an unknown. Plot the posterior distribution for\n",
    "$\\sigma_{\\beta}^{2}$.\n",
    "\n",
    "Model with unknown covariate-specific variances\n",
    "-----------------------------------------------\n",
    "\n",
    "Here we consider a model where the prior for the slope corresponding to\n",
    "covariate $j$ is normal with mean 0 and variance $\\sigma_{j}^{2}$, where\n",
    "$\\sigma_{j}^{2}$ has scaled inverted chi-square prior with scale\n",
    "parameter $S_{\\beta}^{2}$ and degrees of freedom $\\nu_{\\beta}.$ The\n",
    "joint posterior for this model is\n",
    "\n",
    "$$\\begin{aligned}\n",
    "f(\\boldsymbol{\\theta}|\\mathbf{y}) \n",
    "& \\propto f(\\mathbf{y}|\\boldsymbol{\\theta})f(\\boldsymbol{\\theta})\\\\\n",
    "& \\propto  \\left(\\sigma_{e}^{2}\\right)^{-n/2}\\exp\\left\\{ -\\frac{(\\mathbf{y}-\\mathbf{\\mathbf{X}}\\boldsymbol{\\beta})'(\\mathbf{y}-\\mathbf{X}\\boldsymbol{\\beta})}{2\\sigma_{e}^{2}}\\right\\} \\\\\n",
    "& \\times  \\prod_{j=1}^{k}\\left(\\sigma_{j}^{2}\\right)^{-1/2}\\exp\\left\\{ -\\frac{\\beta_{j}^{2}}{2\\sigma_{j}^{2}}\\right\\} \\\\\n",
    "& \\times  \\prod_{j=1}^{k}\\left(\\sigma_{j}^{2}\\right)^{-(2+\\nu_{\\beta})/2}\\exp\\left\\{ -\\frac{\\nu_{\\beta}S_{\\beta}^{2}}{2\\sigma_{j}^{2}}\\right\\} \\\\\n",
    "& \\times  (\\sigma_{e}^{2})^{-(2+\\nu_{e})/2}\\exp\\left\\{ -\\frac{\\nu_{e}S_{e}^{2}}{2\\sigma_{e}^{2}}\\right\\} .\\\\\\end{aligned}$$\n",
    "\n",
    "It can be shown that:\n",
    "\n",
    "1.  The full-conditional posterior for $\\beta_{j}$ is normal with mean\n",
    "    $$\\hat{\\beta}_{j}=\\frac{\\mathbf{x}_{j}'\\mathbf{w}_{j}}{(\\mathbf{x}_{j}'\\mathbf{x}_{j}+\\sigma_{e}^{2}/\\sigma_{j}^{2})},$$\n",
    "    and variance\n",
    "    $\\frac{\\sigma_{e}^{2}}{(\\mathbf{x}_{j}'\\mathbf{x}_{j}+\\sigma_{e}^{2}/\\sigma_{j}^{2})}$.\n",
    "\n",
    "2.  The full-conditional posterior for $\\sigma_{j}^{2}$ is a scaled\n",
    "    inverted chi-square distribution with\n",
    "    $\\tilde{\\nu}_{\\beta}=1+\\nu_{\\beta}$ degrees of freedom and scale\n",
    "    parameter\n",
    "    $\\tilde{S}_{\\beta}^{2}=(\\beta_{j}^{2}+\\nu_{\\beta}S_{\\beta}^{2})/\\tilde{\\nu}_{\\beta}.$\n",
    "    A sample from this posterior can be obtained as\n",
    "    $\\frac{\\beta_{j}^{2}+\\nu_{\\beta}S_{\\beta}^{2}}{\\chi_{\\tilde{\\nu}_{\\beta}}^{2}}$.\n",
    "\n",
    "3.  Marginally, the prior for $\\beta_{j}$ is a scaled $t$ distribution\n",
    "    with $\\nu_{\\beta}$ degrees of freedom, mean 0 and scale parameter\n",
    "    $S_{\\beta}^{2}.$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "1.  Derive the full-conditional posterior for $\\beta_{j}.$\n",
    "\n",
    "2.  Derive the full-conditional posterior for $\\sigma_{j}^{2}.$\n",
    "\n",
    "3.  Use a Gibbs sampler to compute the posterior mean of\n",
    "    $\\boldsymbol{\\beta}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model with Mixture Prior for Regression Coefficients\n",
    "----------------------------------------------------\n",
    "\n",
    "As before, a flat prior is used for the intercept, $\\mu$. The prior for\n",
    "slope $j$ is a mixture: \n",
    "\n",
    "$$\n",
    "\\beta_{j}=\\begin{cases}\n",
    "0 & \\text{probability}\\,\\pi\\\\\n",
    "\\sim N(0,\\sigma_{\\beta}^{2}) & \\text{probability}\\,(1-\\pi)\n",
    "\\end{cases},\n",
    "$$ \n",
    "\n",
    "where $\\sigma_{\\beta}^{2}$ has a scaled inverted\n",
    "chi-square prior with scale parameter $S_{\\beta}^{2}$ and degrees of\n",
    "freedom $\\nu_{\\beta}.$ In order to use the Gibbs sampler, it is\n",
    "convenient to write $\\beta_{j}$ as \n",
    "\n",
    "$$\\beta_{j}=\\delta_{j}\\gamma_{j},$$\n",
    "\n",
    "where $\\delta_{j}$ is a Bernoulli variable with probability $1-\\pi$ of\n",
    "being 1: \n",
    "\n",
    "$$\\delta_{j}=\\begin{cases}\n",
    "0 & \\text{probability}\\,\\pi\\\\\n",
    "1 & \\text{probability}\\,(1-\\pi)\n",
    "\\end{cases},$$ \n",
    "\n",
    "and $\\gamma_{j}$ is normally distributed with mean zero\n",
    "and variance $\\sigma_{\\beta}^{2}.$ Then, the model for the phenotypic\n",
    "values can be written as\n",
    "$$y_{i}=\\mu+\\sum_{j=1}X_{ij}\\gamma_{j}\\delta_{j}+e_{i}.$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Full-conditionals:\n",
    "\n",
    "The joint posterior for all the parameters is proportional to\n",
    "\n",
    "$$\\begin{aligned}\n",
    "f(\\boldsymbol{\\theta}|\\mathbf{y}) \n",
    "& \\propto  f(\\mathbf{y}|\\boldsymbol{\\theta})f(\\boldsymbol{\\theta})\\\\\n",
    "& \\propto  \\left(\\sigma_{e}^{2}\\right)^{-n/2}\\exp\\left\\{ -\n",
    "\\frac{(\\mathbf{y}-\\mathbf{1}\\mu-\\sum\\mathbf{X}_{j}\\gamma_{j}\\delta_{j})'(\\mathbf{y}-\\mathbf{1}\\mu-\\sum\\mathbf{X}_{j}\\gamma_{j}\\delta_{j})}{2\\sigma_{e}^{2}}\\right\\} \\\\\n",
    "& \\times  \\prod_{j=1}^{k}\\left(\\sigma_{\\beta}^{2}\\right)^{-1/2}\\exp\\left\\{ -\\frac{\\gamma_{j}^{2}}{2\\sigma_{\\beta}^{2}}\\right\\} \\\\\n",
    "& \\times  \\prod_{j=1}^{k}\\pi^{(1-\\delta_{j})}(1-\\pi)^{\\delta_{j}}\\\\\n",
    "& \\times  (\\sigma_{\\beta}^{2})^{-(\\nu_{\\beta}+2)/2}\\exp\\left\\{ -\\frac{\\nu_{\\beta}S_{\\beta}^{2}}{2\\sigma_{\\beta}^{2}}\\right\\} \\\\\n",
    "& \\times  (\\sigma_{e}^{2})^{-(2+\\nu_{e})/2}\\exp\\left\\{ -\\frac{\\nu_{e}S_{e}^{2}}{2\\sigma_{e}^{2}}\\right\\} ,\n",
    "\\end{aligned}$$\n",
    "\n",
    "where $\\boldsymbol{\\theta}$ denotes all the unknowns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Full-conditional for $\\mu$\n",
    "\n",
    "The full-conditional for $\\mu$ is a normal distribution with mean\n",
    "$\\hat{\\mu}$ and variance $\\frac{\\sigma_{e}^{2}}{n}$, where $\\hat{\\mu}$\n",
    "is the least-squares estimate of $\\mu$ in the model\n",
    "$$\\mathbf{y-\\sum_{j=1}^{k}}\\mathbf{X}_{j}\\gamma_{j}\\delta_{j}=\\mathbf{1}\\mu+\\mathbf{e},$$\n",
    "and $\\frac{\\sigma_{e}^{2}}{n}$ is the variance of this estimator ($n$ is\n",
    "the number of observations)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Full-conditional for $\\gamma_{j}$\n",
    "\n",
    "$$\\begin{aligned}\n",
    "f(\\gamma{}_{j}|\\text{ELSE}) \n",
    "& \\propto  \\exp\\left\\{ -\\frac{(\\mathbf{w}_{j}-\\mathbf{X}_{j}\\gamma_{j}\\delta_{j})'(\\mathbf{w}_{j}-\\mathbf{X}_{j}\\gamma_{j}\\delta_{j})}{2\\sigma_{e}^{2}}\\right\\} \\\\\n",
    "& \\times  \\exp\\left\\{ -\\frac{\\gamma_{j}^{2}}{2\\sigma_{\\beta}^{2}}\\right\\} \\\\\n",
    "& \\propto  \\exp\\left\\{ -\\frac{[\\mathbf{w}'_{j}\\mathbf{w}_{j}-2\\mathbf{w}'_{j}\\mathbf{X}_{j}\\gamma_{j}\\delta_{j}+\\gamma_{j}^{2}(\\mathbf{x}'_{j}\\mathbf{x}_{j}\\delta_{j}+\\sigma_{e}^{2}/\\sigma_{\\beta}^{2})]}{2\\sigma_{e}^{2}}\\right\\} \\\\\n",
    "& \\propto  \\exp\\left\\{ -\\frac{(\\gamma_{j}-\\hat{\\gamma_{j}})^{2}}{\\frac{2\\sigma_{e}^{2}}{(\\mathbf{x}'_{j}\\mathbf{x}_{j}\\delta_{j}+\\sigma_{e}^{2}/\\sigma_{\\beta}^{2})}}\\right\\} ,\\end{aligned}$$\n",
    "\n",
    "where\n",
    "\n",
    "$$\\mathbf{w}_{j}=\\mathbf{y}-\\mathbf{1}\\mu-\\sum_{l\\neq j}\\mathbf{X}_{l}\\gamma_{l}\\delta_{l}.$$\n",
    "\n",
    "So, the full-conditional for $\\gamma_{j}$ is a normal distribution with\n",
    "mean\n",
    "\n",
    "$$\\hat{\\gamma}_{j}=\\frac{\\mathbf{X}'_{j}\\mathbf{w}_{j}\\delta_{j}}{(\\mathbf{x}'_{j}\\mathbf{x}_{j}\\delta_{j}+\\sigma_{e}^{2}/\\sigma_{\\beta}^{2})}$$\n",
    "and variance\n",
    "$\\frac{\\sigma_{e}^{2}}{(\\mathbf{x}'_{j}\\mathbf{x}_{j}\\delta_{j}+\\sigma_{e}^{2}/\\sigma_{\\beta}^{2})}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Full-conditional for $\\delta_{j}$\n",
    "\n",
    "$$\\Pr(\\delta_{j}=1|\\text{ELSE})\\propto\\frac{h(\\delta_{j}=1)}{h(\\delta_{j}=1)+h(\\delta_{j}=0)},$$\n",
    "where\n",
    "$$h(\\delta_{j})=\\pi^{(1-\\delta_{j})}(1-\\pi)^{\\delta_{j}}\\exp\\left\\{ -\\frac{\n",
    "(\\mathbf{w}_{j}-\\mathbf{X}_{j}\\gamma_{j}\\delta_{j})'\n",
    "(\\mathbf{w}_{j}-\\mathbf{X}_{j}\\gamma_{j}\\delta_{j})\n",
    "}{2\\sigma_{e}^{2}}\\right\\} .$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Full-conditional for $\\sigma_{\\beta}^{2}$\n",
    "\n",
    "$$\\begin{aligned}\n",
    "f(\\sigma_{\\beta}^{2}|\\text{ELSE}) \n",
    "&\\propto \\left(\\sigma_{\\beta}^{2}\\right)^{-k/2}\\exp\\left\\{ -\\frac{\\sum_{j=1}^{k}\\gamma_{j}^{2}}{2\\sigma_{\\beta}^{2}}\\right\\} \\\\\n",
    "&\\times  (\\sigma_{\\beta}^{2})^{-(\\nu_{\\beta}+2)/2}\\exp\\left\\{ -\\frac{\\nu_{\\beta}S_{\\beta}^{2}}{2\\sigma_{\\beta}^{2}}\\right\\} \\\\\n",
    "&\\propto  (\\sigma_{\\beta}^{2})^{-(k+\\nu_{\\beta}+2)/2}\\exp\\left\\{ -\\frac{\\sum_{j=1}^{k}\\gamma_{j}^{2}+\\nu_{\\beta}S_{\\beta}^{2}}{2\\sigma_{j}^{2}}\\right\\} ,\\end{aligned}$$\n",
    "\n",
    "and this is proportional to a scaled inverted chi-square distribution\n",
    "with $\\tilde{\\nu}_{\\beta}=\\nu_{\\beta}+k$ and scale parameter\n",
    "$\\tilde{S}_{\\beta}^{2}=(\\sum_{j=1}^{k}\\gamma_{j}^{2}+\\nu_{\\beta}S_{\\beta}^{2})/\\tilde{\\nu}_{\\beta}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Full-conditional for $\\pi$\n",
    "\n",
    "$$f(\\pi|ELSE)\\propto\\pi^{(k-\\sum_{j=1}^{k}\\delta_{j})}(1-\\pi)^{\\sum_{j=1}^{k}\\delta_{j}},$$\n",
    "\n",
    "which is proportional to a Beta distribution with parameters\n",
    "$a=k-\\sum_{j=1}^{k}\\delta_{j}+1$ and $b=\\sum\\delta_{j}+1.$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Full-conditional for $\\sigma_{e}^{2}$\n",
    "\n",
    "$$\\begin{aligned}\n",
    "f(\\sigma_{e}^{2}|\\text{ELSE}) \n",
    "& \\propto  \\left(\\sigma_{e}^{2}\\right)^{-n/2}\\exp\\left\\{ -\\frac{\n",
    "(\\mathbf{y}-\\mathbf{1}\\mu-\\sum\\mathbf{X}_{j}\\gamma_{j}\\delta_{j})'\n",
    "(\\mathbf{y}-\\mathbf{1}\\mu-\\sum\\mathbf{X}_{j}\\gamma_{j}\\delta_{j})\n",
    "}{2\\sigma_{e}^{2}}\\right\\} \\\\\n",
    "& \\times  (\\sigma_{e}^{2})^{-(2+\\nu_{e})/2}\\exp\\left\\{ -\\frac{\\nu_{e}S_{e}^{2}}{2\\sigma_{e}^{2}}\\right\\} \\\\\n",
    "& \\propto  (\\sigma_{e}^{2})^{-(n+2+\\nu_{e})/2}\\exp\\left\\{ -\\frac{(\\mathbf{y}-\\mathbf{1}\\mu-\\sum\\mathbf{X}_{j}\\gamma_{j}\\delta_{j})'(\\mathbf{y}-\\mathbf{1}\\mu-\\sum\\mathbf{X}_{j}\\gamma_{j}\\delta_{j})+\\nu_{e}S_{e}^{2}}{2\\sigma_{e}^{2}}\\right\\} ,\\end{aligned}$$\n",
    "\n",
    "which is proportional to a scaled inverted chi-square density with\n",
    "$\\tilde{\\nu}_{e}=n+\\nu_{e}$ degrees of freedom and\n",
    "$\\tilde{S_{e}^{2}}=\\frac{(\\mathbf{y}-\\mathbf{1}\\mu-\\sum\\mathbf{X}_{j}\\gamma_{j}\\delta_{j})'(\\mathbf{y}-\\mathbf{1}\\mu-\\sum\\mathbf{X}_{j}\\gamma_{j}\\delta_{j})+\\nu_{e}S_{e}^{2}}{\\tilde{\\nu_{e}}}$\n",
    "scale parameter."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 0.3.7",
   "language": "julia",
   "name": "julia 0.3"
  },
  "language": "Julia",
  "language_info": {
   "name": "julia",
   "version": "0.3.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}