{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Optimization: introduction\n",
"\n",
"* Optimization considers the problem\n",
"$$\n",
"\\begin{eqnarray*}\n",
" \\text{minimize } f(\\mathbf{x}) \\\\\n",
" \\text{subject to constraints on } \\mathbf{x}\n",
"\\end{eqnarray*}\n",
"$$\n",
"\n",
"* Possible confusion:\n",
" * We (statisticians) talk about **maximization**: $\\max \\, L(\\mathbf{\\theta})$.\n",
" * People talk about **minimization** in the field of optimization: $\\min \\, f(\\mathbf{x})$.\n",
"\n",
"* **Why** is optimization important in statistics? \n",
" * Maximum likelihood estimation (MLE). \n",
" * Maximum a posteriori (MAP) estimation in Bayesian framework. \n",
" * Machine learning: minimize a loss + certain regularization. \n",
" * ...\n",
" \n",
"* Our major **goal** (or learning objectives) is to\n",
" * have a working knowledge of some commonly used optimization methods: \n",
" * Newton type algorithms\n",
" * expectation-maximization (EM) algorithm \n",
" * majorization-minimization (MM) algorithm \n",
" * quasi-Newton algorithm\n",
" * conjugate gradient (CG) type algorithms \n",
" * convex programming with emphasis in statistical applications\n",
" * implement some of them in homework\n",
" * get to know some optimization tools in Julia\n",
"\n",
"* What's **not** covered in this course:\n",
" * Optimality conditions \n",
" * Convergence theory \n",
" * Convex analysis \n",
" * Modern algorithms for large scale problems (ADMM, CD, proximal gradient, stochastic gradient, ...)\n",
" * Combinatorial optimization \n",
" * Stochastic algorithms\n",
" * Many others\n",
" \n",
"* You **must** take advantage of the great resources at UCLA. \n",
" * Lieven Vandenberghe: EE236A (Linear Programming), **EE236B** (Convex Optimization), **EE236C** (Optimization Methods for Large-scale Systems). One of the best places to learn convex programming. \n",
" * Kenneth Lange: Biomath 210 (Optimization Methods in Biology). **The** best place to learn MM type algorithms.\n",
" * Wotao Yin in math.\n",
" \n",
"\n",
"\n",
"\n",
"\n",
"\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Julia 0.6.2",
"language": "julia",
"name": "julia-0.6"
},
"language_info": {
"file_extension": ".jl",
"mimetype": "application/julia",
"name": "julia",
"version": "0.6.2"
},
"toc": {
"colors": {
"hover_highlight": "#DAA520",
"running_highlight": "#FF0000",
"selected_highlight": "#FFD700"
},
"moveMenuLeft": true,
"nav_menu": {
"height": "30px",
"width": "252px"
},
"navigate_menu": true,
"number_sections": true,
"sideBar": true,
"threshold": 4,
"toc_cell": false,
"toc_section_display": "block",
"toc_window_display": false,
"widenNotebook": false
}
},
"nbformat": 4,
"nbformat_minor": 2
}