{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optimization: introduction\n",
    "\n",
    "* Optimization considers the problem\n",
    "$$\n",
    "\\begin{eqnarray*}\n",
    "    \\text{minimize } f(\\mathbf{x}) \\\\\n",
    "    \\text{subject to constraints on } \\mathbf{x}\n",
    "\\end{eqnarray*}\n",
    "$$\n",
    "\n",
    "* Possible confusion:\n",
    "     * We (statisticians) talk about **maximization**: $\\max \\, L(\\mathbf{\\theta})$.\n",
    "     * People talk about **minimization** in the field of optimization: $\\min \\, f(\\mathbf{x})$.\n",
    "\n",
    "* **Why** is optimization important in statistics? \n",
    "    * Maximum likelihood estimation (MLE). \n",
    "    * Maximum a posteriori (MAP) estimation in Bayesian framework.  \n",
    "    * Machine learning: minimize a loss + certain regularization.  \n",
    "    * ...\n",
    "    \n",
    "* Our major **goal** (or learning objectives) is to\n",
    "    * have a working knowledge of some commonly used optimization methods: \n",
    "        * Newton type algorithms\n",
    "        * expectation-maximization (EM) algorithm \n",
    "        * majorization-minimization (MM) algorithm  \n",
    "        * quasi-Newton algorithm\n",
    "        * convex programming with emphasis in statistical applications\n",
    "    * implement some of them in homework\n",
    "    * get to know some optimization tools in Julia\n",
    "\n",
    "* What's **not** covered in this course:\n",
    "    * Optimality conditions  \n",
    "    * Convergence theory \n",
    "    * Convex analysis  \n",
    "    * Modern algorithms for large scale problems (ADMM, CD, proximal gradient, stochastic gradient, ...)\n",
    "    * Combinatorial optimization \n",
    "    * Stochastic algorithms\n",
    "    * Many others\n",
    "    \n",
    "* You **must** take advantage of the great resources at UCLA. \n",
    "    * Lieven Vandenberghe: EE236A (Linear Programming), **EE236B** (Convex Optimization), **EE236C** (Optimization Methods for Large-scale Systems). One of the best places to learn convex programming.  \n",
    "    * Kenneth Lange: Biomath 210 (Optimization Methods in Biology). **The** best place to learn MM type algorithms.\n",
    "    * Wotao Yin in math.\n",
    "    \n",
    "<img src=\"http://stanford.edu/~boyd/cvxbook/bv_cvxbook_cover.jpg\" width=\"200\" align=\"center\"/>\n",
    "\n",
    "<img src=\"https://images.springer.com/sgw/books/medium/9781461458371.jpg\" width=\"200\" align=\"center\"/>\n",
    "\n",
    "<img src=\"http://cdn3.bigcommerce.com/s-q8xt814/products/608/images/955/OT147large__19280.1468256169.380.500.jpg?c=2\" width=\"200\" align=\"center\"/>\n"
   ]
  }
 ],
 "metadata": {
  "@webio": {
   "lastCommId": null,
   "lastKernelId": null
  },
  "kernelspec": {
   "display_name": "Julia 1.1.0",
   "language": "julia",
   "name": "julia-1.1"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.1.0"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "30px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false,
   "widenNotebook": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}