{ "cells": [ { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## EECS 545 - Machine Learning\n", "## Lecture 3: Convex Optimization\n", "### Date: January 13, 2016\n", "### Instructor: Jacob Abernethy\n", "\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false, "slideshow": { "slide_type": "skip" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Populating the interactive namespace from numpy and matplotlib\n" ] } ], "source": [ "from IPython.core.display import HTML, Image\n", "from IPython.display import YouTubeVideo\n", "from sympy import init_printing, Matrix, symbols, Rational\n", "import sympy as sym\n", "from warnings import filterwarnings\n", "init_printing(use_latex = 'mathjax')\n", "filterwarnings('ignore')\n", "\n", "%pylab inline\n", "\n", "import numpy as np" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Some important notes\n", "* HW1 is out! Due January 25th at 11pm\n", "* Homework will be submitted via *Gradescope*. Please see Piazza for precise instructions. Do it soon, not at the last minute!!\n", "* There is an *optional* tutorial **this evening**, 5pm, in Dow 1013. Come see Daniel go over some tough problems.\n", "* No class on Monday January 18, MLK day!" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Python: We recommend Anacdona\n", "\n", "* Anaconda is standalone Python distribution that includes all the most important scientific packages: *numpy, scipy, matplotlib, sympy, sklearn, etc.*\n", "* Easy to install, available for OS X, Windows, Linux.\n", "* Small warning: it's kind of large (250MB)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Some notes on using Python\n", "* HW1 has only a very simple programming exercise, just as a warmup. We don't expect you to submit code this time\n", "* This is a good time to start learning Python basics\n", "* There are **a ton** of good places on the web to learn python, we'll post some\n", "* Highly recommended: **ipython**; it's a much more user friendly terminal interface to python\n", "* Even better: **jupyter notebook**, a web based interface. This is how I'm making these slides!" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Checking if all is installed, and HelloWorld\n", "If you got everything installed, this should run:\n", "```python\n", "\n", "# numpy is crucial for vectors, matrices, etc.\n", "import numpy as np \n", "# Lots of cool plotting tools with matplotlib\n", "import matplotlib.pyplot as plt\n", "# For later: scipy has a ton of stats tools\n", "import scipy as sp\n", "# For later: sklearn has many standard ML algs\n", "import sklearn\n", "# Here we go!\n", "print(\"Hello World!\")\n", "```" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## More on learning python\n", "* We will have one tutorial devoted to this\n", "* If you're new to Python, go slow!\n", " * First learn the basics (lists, dicts, for loops, etc.)\n", " * Then spend a couple days playing with numpy\n", " * Then explore matplotlib\n", " * etc.\n", "* Piazza = your friend. We have a designated python instructor (IA Ben Bray) who has lots of answers." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Outline for this Lecture\n", "- Convexity\n", " - Convex Set\n", " - Convex Function\n", "- Introduction to Optimization\n", "- Introduction to Lagrange Duality" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "> In this lecture, we will first introduce convex set, convex function and optimization problem. One approach to solve optimization problem is to solve its dual problem. We will briefly cover some basics of duality in this lecture. More about optimization and duality will come when we study support vector machine (SVM)." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Convexity\n", "### Convex Sets\n", "- $C \\subseteq \\mathbb{R}^n$ is **convex** if\n", "$$t x + (1-t)y \\in C$$\n", "for any $x, y \\in C$ and $0 \\leq t \\leq 1$\n", "- that is, a set is convex if the line connecting **any** two points in the set is entirely inside the set\n", "\n", "