{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d3027ced",
   "metadata": {},
   "source": [
    "# Bayesian Machine Learning\n",
    "\n",
    "\n",
    "- **[1]** (#) (a) Explain shortly the relation between machine learning and Bayes rule.     \n",
    "   (b) How are Maximum a Posteriori (MAP) and Maximum Likelihood (ML) estimation related to Bayes rule and machine learning?\n",
    "\n",
    "\n",
    "- **[2]** (#) What are the four stages of the Bayesian design approach?\n",
    "\n",
    "\n",
    "- **[3]** (##) The Bayes estimate is a summary of a posterior distribution by a delta distribution on its mean, i.e., \n",
    "$$\n",
    "\\hat \\theta_{bayes}  = \\int \\theta \\, p\\left( \\theta |D \\right)\n",
    "\\,\\mathrm{d}{\\theta}\n",
    "$$\n",
    "Proof that the Bayes estimate minimizes the expected mean-squared error, i.e., proof that\n",
    "$$\n",
    "\\hat \\theta_{bayes} = \\arg\\min_{\\hat \\theta} \\int_\\theta (\\hat \\theta -\\theta)^2 p \\left( \\theta |D \\right) \\,\\mathrm{d}{\\theta}\n",
    "$$\n",
    "\n",
    "\n",
    "- **[4]** (###) We make $N$ IID observations $D=\\{x_1 \\dots x_N\\}$ and assume the following model\n",
    "$$\n",
    "x_k = A + \\epsilon_k \n",
    "$$\n",
    "  where $\\epsilon_k = \\mathcal{N}(\\epsilon_k | 0,\\sigma^2)$ with known $\\sigma^2=1$. We are interested in deriving an estimator for $A$.   \n",
    "  (a) Make a reasonable assumption for a prior on $A$ and derive a Bayesian (posterior) estimate.     \n",
    "  (b) (##) Derive the Maximum Likelihood estimate for $A$.      \n",
    "  (c) Derive the MAP estimates for $A$.    \n",
    "  (d) Now assume that we do not know the variance of the noise term? Describe the procedure for Bayesian estimation of both $A$ and $\\sigma^2$ (No need to fully work out to closed-form estimates). \n",
    "\n",
    "   \n",
    "- **[5]** (##) We consider the coin toss example from the notebook and use a conjugate prior for a Bernoulli likelihood function.    \n",
    "  (a) Derive the Maximum Likelihood estimate.    \n",
    "  (b) Derive the MAP estimate.          \n",
    "  (c) Do these two estimates ever coincide (if so under what circumstances)?    \n",
    "\n",
    "<!---\n",
    "- **[6]** (###) Given a single observation $x_0$ from a uniform distribution $\\mathrm{Unif}[0,1/\\theta]$, where $\\theta > 0$.  \n",
    "  (a) Show that $\\mathbb{E}[g(x_0)] = \\theta$  if and only if $\\int_0^{1/\\theta} g(u) du =1$.     \n",
    "  (b) Show that there is no function $g$ that satisfies the condition for all $\\theta > 0$.     \n",
    "--->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27db69a5",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.6.3",
   "language": "julia",
   "name": "julia-1.6"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}