{ "cells": [ { "cell_type": "markdown", "id": "45e51d4b", "metadata": {}, "source": [ "# Generative Classification\n", "\n", "- **[1]** You have a machine that measures property $x$, the \"orangeness\" of liquids. You wish to discriminate between $C_1 = \\text{`Fanta'}$ and $C_2 = \\text{`Orangina'}$. It is known that\n", "\n", "$$\\begin{align*}\n", "p(x|C_1) &= \\begin{cases} 10 & 1.0 \\leq x \\leq 1.1\\\\\n", " 0 & \\text{otherwise}\n", " \\end{cases}\\\\\n", "p(x|C_2) &= \\begin{cases} 200(x - 1) & 1.0 \\leq x \\leq 1.1\\\\\n", "0 & \\text{otherwise}\n", "\\end{cases}\n", "\\end{align*}$$\n", "\n", "The prior probabilities $p(C_1) = 0.6$ and $p(C_2) = 0.4$ are also known from experience. \n", " (a) (##) A \"Bayes Classifier\" is given by\n", " \n", "$$ \\text{Decision} = \\begin{cases} C_1 & \\text{if } p(C_1|x)>p(C_2|x) \\\\\n", " C_2 & \\text{otherwise}\n", " \\end{cases}\n", "$$\n", "\n", "Derive the optimal Bayes classifier. \n", " (b) (###) The probability of making the wrong decision, given $x$, is\n", " \n", "$$\n", "p(\\text{error}|x)= \\begin{cases} p(C_1|x) & \\text{if we decide $C_2$}\\\\\n", " p(C_2|x) & \\text{if we decide $C_1$}\n", "\\end{cases}\n", "$$\n", "\n", "Compute the **total** error probability $p(\\text{error})$ for the Bayes classifier in this example.\n", "\n", "\n", "- **[2]** (#) (see Bishop exercise 4.8): Using (4.57) and (4.58) (from Bishop's book), derive the result (4.65) for the posterior class probability in the two-class generative model with Gaussian densities, and verify the results (4.66) and (4.67) for the parameters $w$ and $w0$. \n", "\n", "\n", "- **[3]** (###) (see Bishop exercise 4.9). \n", "\n", "\n", "- **[4]** (##) (see Bishop exercise 4.10). \n", "\n", "\n", "" ] }, { "cell_type": "markdown", "id": "0bf3a01b", "metadata": {}, "source": [ "# Discriminative Classification\n", "\n", "- **[1]** Given a data set $D=\\{(x_1,y_1),\\ldots,(x_N,y_N)\\}$, where $x_n \\in \\mathbb{R}^M$ and $y_n \\in \\{0,1\\}$. The probabilistic classification method known as *logistic regression* attempts to model these data as\n", "$$p(y_n=1|x_n) = \\sigma(\\theta^T x_n + b)$$\n", "where $\\sigma(x) = 1/(1+e^{-x})$ is the *logistic function*. Let's introduce shorthand notation $\\mu_n=\\sigma(\\theta^T x_n + b)$. So, for every input $x_n$, we have a model output $\\mu_n$ and an actual data output $y_n$. \n", " (a) Express $p(y_n|x_n)$ as a Bernoulli distribution in terms of $\\mu_n$ and $y_n$. \n", " (b) If furthermore is given that the data set is IID, show that the log-likelihood is given by\n", "$$\n", "L(\\theta) \\triangleq \\log p(D|\\theta) = \\sum_n \\left\\{y_n \\log \\mu_n + (1-y_n)\\log(1-\\mu_n)\\right\\}\n", "$$ \n", " (c) Prove that the derivative of the logistic function is given by\n", "$$\n", "\\sigma^\\prime(\\xi) = \\sigma(\\xi)\\cdot\\left(1-\\sigma(\\xi)\\right)\n", "$$ \n", " (d) Show that the derivative of the log-likelihood is\n", "$$\n", "\\nabla_\\theta L(\\theta) = \\sum_{n=1}^N \\left( y_n - \\sigma(\\theta^T x_n +b)\\right)x_n\n", "$$ \n", " (e) Design a gradient-ascent algorithm for maximizing $L(\\theta)$ with respect to $\\theta$. \n", "\n", "\n", "- **[2]** Describe shortly the similarities and differences between the discriminative and generative approach to classification.\n", "\n", "\n", "- **[3]** (Bishop ex.4.7) (#) Show that the logistic sigmoid function $\\sigma(a) = \\frac{1}{1+\\exp(-a)}$ satisfies the property $\\sigma(-a) = 1-\\sigma(a)$ and that its inverse is given by $\\sigma^{-1}(y) = \\log\\{y/(1-y)\\}$.\n", "\n", " \n", "- **[4]** (Bishop ex.4.16) (###) Consider a binary classification problem in which each observation $x_n$ is known to belong to one of two classes, corresponding to $y_n = 0$ and $y_n = 1$. Suppose that the procedure for collecting training data is imperfect, so that training points are sometimes mislabelled. For every data point $x_n$, instead of having a value $y_n$ for the class label, we have instead a value $\\pi_n$ representing the probability that $y_n = 1$. Given a probabilistic model $p(y_n = 1|x_n,\\theta)$, write down the log-likelihood function appropriate to such a data set.\n", " \n", "\n", "- **[5]** (###) Let $X$ be a real valued random variable with probability density\n", "$$\n", "p_X(x) = \\frac{e^{-x^2/2}}{\\sqrt{2\\pi}},\\quad\\text{for all $x$}.\n", "$$\n", "Also $Y$ is a real valued random variable with conditional density\n", "$$\n", "p_{Y|X}(y|x) = \\frac{e^{-(y-x)^2/2}}{\\sqrt{2\\pi}},\\quad\\text{for all $x$ and $y$}. \n", "$$\n", " (a) Give an (integral) expression for $p_Y(y)$. Do not try to evaluate the integral. \n", " (b) Approximate $p_Y(y)$ using the Laplace approximation.\n", " Give the detailed derivation, not just the answer.\n", "Hint: You may use the following results.\n", "Let \n", "$$g(x) = \\frac{e^{-x^2/2}}{\\sqrt{2\\pi}}$$\n", "and\n", "$$\n", "h(x) = \\frac{e^{-(y-x)^2/2}}{\\sqrt{2\\pi}}$$\n", "for some real value $y$. Then:\n", "$$\\begin{align*}\n", "\\frac{\\partial}{\\partial x} g(x) &= -xg(x) \\\\\n", "\\frac{\\partial^2}{\\partial x^2} g(x) &= (x^2-1)g(x) \\\\\n", "\\frac{\\partial}{\\partial x} h(x) &= (y-x)h(x) \\\\\n", "\\frac{\\partial^2}{\\partial x^2} h(x) &= ((y-x)^2-1)h(x) \n", "\\end{align*}$$ \n", "\n", "\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "id": "e455e95b", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Julia 1.5.2", "language": "julia", "name": "julia-1.5" }, "language_info": { "file_extension": ".jl", "mimetype": "application/julia", "name": "julia", "version": "1.5.2" } }, "nbformat": 4, "nbformat_minor": 5 }