{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "二项逻辑斯谛回归模型是如下的条件概率分布：\n",
    "\\begin{align*} \\\\&  P \\left( Y = 1 | x \\right) =  \\dfrac{1}{1+\\exp{-\\left(w \\cdot x + b \\right)}}\n",
    "\\\\ & \\quad\\quad\\quad\\quad = \\dfrac{\\exp{\\left(w \\cdot x + b \\right)}}{\\left( 1+\\exp{-\\left(w \\cdot x + b \\right)}\\right) \\cdot \\exp{\\left(w \\cdot x + b \\right)}}\n",
    "\\\\ & \\quad\\quad\\quad\\quad = \\dfrac{\\exp{\\left(w \\cdot x + b \\right)}}{1+\\exp{\\left( w \\cdot x + b \\right)}}\\\\&  P \\left( Y = 0 | x \\right) =  1- P \\left( Y = 1 | x \\right)\n",
    "\\\\ &  \\quad\\quad\\quad\\quad=1- \\dfrac{\\exp{\\left(w \\cdot x + b \\right)}}{1+\\exp{\\left( w \\cdot x + b \\right)}}\n",
    "\\\\ & \\quad\\quad\\quad\\quad=\\dfrac{1}{1+\\exp{\\left( w \\cdot x + b \\right)}}\\end{align*}\n",
    "其中，$x \\in R^{n}$是输入，$Y \\in \\left\\{ 0, 1 \\right\\}$是输出，$w \\in R^{n}$和$b \\in R$是参数，$w$称为权值向量，$b$称为偏置，$w \\cdot x$为$w$和$b$的内积。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "可将权值权值向量和输入向量加以扩充，即$w = \\left( w^{\\left(1\\right)},w^{\\left(2\\right)},\\cdots,w^{\\left(n\\right)},b \\right)^{T}$，$x = \\left( x^{\\left(1\\right)},x^{\\left(2\\right)},\\cdots,x^{\\left(n\\right)},1 \\right)^{T}$，则逻辑斯谛回归模型：\n",
    "\\begin{align*} \\\\&  P \\left( Y = 1 | x \\right) = \\dfrac{\\exp{\\left(w \\cdot x \\right)}}{1+\\exp{\\left( w \\cdot x  \\right)}}\\\\&  P \\left( Y = 0 | x \\right) =\\dfrac{1}{1+\\exp{\\left( w \\cdot x  \\right)}}\\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一个事件的几率是指事件发生的概率$p$与事件不发生的概率$1-p$的比值，即\n",
    "\\begin{align*} \\\\&  \\dfrac{p}{1-p}\\end{align*}  \n",
    "该事件的对数几率（logit函数）\n",
    "\\begin{align*} \\\\&  logit\\left( p \\right) = \\log \\dfrac{p}{1-p}\\end{align*} "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于逻辑斯谛回归模型\n",
    "\\begin{align*} \\\\&  \\log \\dfrac{P \\left( Y = 1 | x \\right)}{1-P \\left( Y = 1 | x \\right)} = w \\cdot x\\end{align*}   \n",
    "即输出$Y=1$的对数几率是输入$x$的线性函数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "给定训练数据集\n",
    "\\begin{align*} \\\\& T = \\left\\{ \\left( x_{1}, y_{1} \\right), \\left( x_{2}, y_{2} \\right), \\cdots, \\left( x_{N}, y_{N} \\right) \\right\\} \\end{align*}   \n",
    "其中，$x_{i} \\in R^{n+1}, y_{i} \\in \\left\\{ 0, 1 \\right\\}, i = 1, 2, \\cdots, N$。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "设：\n",
    "\\begin{align*} \\\\& P \\left( Y =1 | x \\right) = \\pi \\left( x \\right) ,\\quad P \\left( Y =0 | x \\right) = 1 - \\pi \\left( x \\right)  \\end{align*}   \n",
    "似然函数\n",
    "\\begin{align*} \\\\& l \\left( w \\right) = \\prod_{i=1}^{N} P \\left( y_{i} | x_{i} \\right) \n",
    "\\\\ & = P \\left( Y = 1 | x_{i} , w \\right) \\cdot P \\left( Y = 0 | x_{i}, w \\right) \n",
    "\\\\ & = \\prod_{i=1}^{N} \\left[ \\pi \\left( x_{i} \\right) \\right]^{y_{i}}\\left[ 1 - \\pi \\left( x_{i} \\right) \\right]^{1 - y_{i}}\\end{align*} "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对数似然函数\n",
    "\\begin{align*} \\\\& L \\left( w \\right) = \\log l \\left( w \\right) \n",
    "\\\\ & = \\sum_{i=1}^{N} \\left[ y_{i} \\log \\pi \\left( x_{i} \\right) + \\left( 1 - y_{i} \\right) \\log \\left( 1 - \\pi \\left( x_{i} \\right) \\right) \\right]\n",
    "\\\\ & = \\sum_{i=1}^{N} \\left[ y_{i} \\log \\dfrac{\\pi \\left( x_{i} \\right)}{1- \\pi \\left( x_{i} \\right)} + \\log \\left( 1 - \\pi \\left( x_{i} \\right) \\right) \\right]\n",
    "\\\\ & = \\sum_{i=1}^{N} \\left[ y_{i} \\left( w \\cdot x_{i} \\right) - \\log \\left( 1 + \\exp \\left( w \\cdot x \\right)  \\right) \\right]\\end{align*} "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "假设$w$的极大似然估计值是$\\hat{w}$，则学得得逻辑斯谛回归模型\n",
    "\\begin{align*} \\\\&  P \\left( Y = 1 | x \\right) = \\dfrac{\\exp{\\left(\\hat{w} \\cdot x \\right)}}{1+\\exp{\\left( \\hat{w} \\cdot x  \\right)}}\\\\&  P \\left( Y = 0 | x \\right) =\\dfrac{1}{1+\\exp{\\left( \\hat{w} \\cdot x  \\right)}}\\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "假设离散型随机变量$Y$的取值集合$\\left\\{ 1, 2, \\cdots, K \\right\\}$，则多项逻辑斯谛回归模型\n",
    "\\begin{align*} \\\\&  P \\left( Y = k | x \\right) = \\dfrac{\\exp{\\left(w_{k} \\cdot x \\right)}}{1+ \\sum_{k=1}^{K-1}\\exp{\\left( w_{k} \\cdot x  \\right)}}, \\quad k=1,2,\\cdots,K-1\n",
    "\\\\ &  P \\left( Y = K | x \\right) = 1 - \\sum_{k=1}^{K-1} P \\left( Y = k | x \\right)\n",
    "\\\\ & = 1 - \\sum_{k=1}^{K-1} \\dfrac{\\exp{\\left(w_{k} \\cdot x \\right)}}{1+ \\sum_{k=1}^{K-1}\\exp{\\left( w_{k} \\cdot x  \\right)}}\n",
    "\\\\ & = \\dfrac{1}{1+ \\sum_{k=1}^{K-1}\\exp{\\left( w_{k} \\cdot x  \\right)}}\\end{align*}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}