{ "cells": [ { "cell_type": "markdown", "metadata": { "toc": "true" }, "source": [ "# Table of Contents\n", "

1  Disciplined Convex Programming
1.1  Convex optimization problems
1.2  Optimization software
1.3  LP examples
1.4  QP and SOCP examples
1.5  SDP
1.6  GP
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Disciplined Convex Programming\n", "\n", "## Convex optimization problems\n", "\n", "* A **mathematical optimization problem**, or just **optimization problem**, has the form\n", "$$\n", "\\begin{eqnarray*}\n", "\t&\\text{minimize}& f_0(\\mathbf{x}) \\\\\n", "\t&\\text{subject to}& f_i(\\mathbf{x}) \\le b_i, \\quad i = 1,\\ldots,m.\n", "\\end{eqnarray*}\n", "$$\n", "Here $f_0: \\mathbb{R}^n \\mapsto \\mathbb{R}$ is the **objective** function and $f_i:\\mathbb{R}^n \\mapsto \\mathbb{R}$, $i=1,\\ldots,m$, are the **constraint** functions.\n", "\n", "* An equality constraint $f_i(\\mathbf{x}) = b_i$ can be absorbed into inequality constraints $f_i(\\mathbf{x}) \\le b_i$ and $- f_i(\\mathbf{x}) \\le - b_i$.\n", "\n", "* If the objective and constraint functions are convex, then it is called a **convex optimization problem**. \n", "In a convex optimization problem, only linear equality constraint of form $\\mathbf{A} \\mathbf{x} = \\mathbf{b}$ is allowed (why?).\n", "\n", "* Convex programming (LS, LP, QP, GP, SOCP, SDP) is becoming a **technology** (`Mosek`, `Gurobi`, `Cplex`, `cvx`, `Convex.jl`, ...), just like numerical linear algebra libraries BLAS and LAPACK. Current technology can solve convex problems with up to thousands of variables and constraints.\n", "\n", "\n", "\n", "* A definite resource is the book _Convex Optimization_ by Boyd and Vandenberghe, which is freely available at http://stanford.edu/~boyd/cvxbook/. Same website has links to slides, code, and lecture videos. \n", " \n", "Lecture videos: \n", " \n", "\n", "\n", "* UCLA courses by **Lieven Vandenberghe**: EE236A (Linear Programming), EE236B (Convex Optimization), EE236C (Optimization Methods for Large-scale Systems)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Optimization software\n", "\n", "* Like numerical linear algebra, getting familiar with **good** optimization softwares broadens the scope and scale of problems we are able to solve in statistics.\n", "\n", "* Following table lists some of the best convex optimization softwares. Use of modeling tools `cvx` (for Matlab) or `Convex.jl` (for Julia), coupled with solvers `Mosek` and/or `Gurobi`, is highly recommended.\n", " \n", "| | | LP | MILP | SOCP | MISOCP | SDP | GP | NLP | MINLP | | R | Matlab | Julia | Python | | Cost |\n", "|:---------:|:-:|:--:|:----:|:----:|:--------------:|:---:|:--:|:---:|:-----:|:-:|:-:|:------:|:-----:|:------:|:-:|:----:|\n", "| **modeling tools** | | | | | | | | | | | | | | | | |\n", "| MathProgBase.jl | | ✔️ | ✔️ | ✔️ | ✔️ | | | ✔️ | ✔️ | | | | ✔️ | | | O |\n", "| Convex.jl | | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | | | | | | | ✔️ | | | O |\n", "| cvx | | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | | | | | ✔️ | | ✔️ | | A |\n", "| **convex solvers** | | | | | | | | | | | | | | | | |\n", "| Mosek | | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | | | ✔️ | ✔️ | ✔️ | ✔️ | | A |\n", "| Gurobi | | ✔️ | ✔️ | ✔️ | ✔️ | | | | | | ✔️ | ✔️ | ✔️ | ✔️ | | A |\n", "| CPLEX | | ✔️ | ✔️ | ✔️ | ✔️ | | | | | | ? | ✔️ | ✔️ | ✔️ | | A |\n", "| SCS | | ✔️ | | ✔️ | | ✔️ | | | | | | ✔️ | ✔️ | ✔️ | | O |\n", "| SeDuMi | | ✔️ | | ✔️ | | ✔️ | ? | | | | | ✔️ | | | | O |\n", "| SDPT3 | | ✔️ | | ✔️ | | ✔️ | ? | | | | | ✔️ | | | | O |\n", "| **NLP solvers** | | | | | | | | | | | | | | | | |\n", "| KNITRO | | ✔️ | ✔️ | | | | | ✔️ | ✔️ | | ✔️ | ✔️ | ✔️ | ✔️ | | $ |\n", "| NLopt | | ✔️ | | | | | | ✔️ | | | | ✔️ | ✔️ | ✔️ | | O |\n", "| Ipopt | | ✔️ | | | | | | ✔️ | | | | ✔️ | ✔️ | ✔️ | | O |\n", "\n", "* O: open source \n", "* A: free academic license \n", "* $: commercial \n", "\n", "* Difference between **modeling tool** and **solvers**.\n", "\n", " * Solvers (`Mosek`, `Gurobi`, `Cplex`, ...) are concrete software implementation of optimization algorithms. \n", " \n", " * Modeling tools such as `cvx` (for Matlab) and `Convex.jl` (Julia analog of cvx) implement the disciplined convex programming (DCP) paradigm proposed by Grant and Boyd (2008) http://stanford.edu/~boyd/papers/disc_cvx_prog.html. DCP prescribes a set of simple rules from which users can construct convex optimization problems easily. \n", "\n", " * Modeling tools usually have the capability to use a variety of solvers. But modeling tools are solver agnostic so users do not have to worry about specific solver interface." ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "## LP examples\n", "\n", "* [Compressive sensing (Candes, Tao, Donoho)](https://en.wikipedia.org/wiki/Compressed_sensing). \n", "http://hua-zhou.github.io/teaching/biostatm280-2016winter/cs.html\n", "\n", "* $L_1$, $L_\\infty$ regression, Quantile regression, Dantzig selector (Candes, Tao), 1-norm svm. \n", "http://hua-zhou.github.io/teaching/st790-2015spr/ST790-2015-HW4.pdf" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## QP and SOCP examples\n", "\n", "* Nonnegative least squares (NNLS), Lasso regression, Elastic net, generalized lasso, support vector machine (svm), Huber loss regression. \n", "\n", "* Group lasso, square root lasso, image denoising.\n", "\n", "http://hua-zhou.github.io/teaching/biostatm280-2016winter/lasso.html \n", "http://hua-zhou.github.io/teaching/st790-2015spr/ST790-2015-HW5.pdf" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## SDP\n", "\n", "* Expriment design, matrix completion.\n", "\n", "http://hua-zhou.github.io/teaching/st790-2015spr/ST790-2015-HW6.pdf" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## GP\n", "\n", "* Logistic regression, Bradley-Terry." ] } ], "metadata": { "kernelspec": { "display_name": "Julia 0.6.3", "language": "julia", "name": "julia-0.6" }, "language_info": { "file_extension": ".jl", "mimetype": "application/julia", "name": "julia", "version": "0.6.3" }, "toc": { "colors": { "hover_highlight": "#DAA520", "running_highlight": "#FF0000", "selected_highlight": "#FFD700" }, "moveMenuLeft": true, "nav_menu": { "height": "65px", "width": "252px" }, "navigate_menu": true, "number_sections": false, "sideBar": false, "threshold": 4, "toc_cell": true, "toc_position": { "height": "289px", "left": "1px", "right": "1057.4000244140625px", "top": "106px", "width": "212px" }, "toc_section_display": "block", "toc_window_display": false, "widenNotebook": false } }, "nbformat": 4, "nbformat_minor": 2 }