{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scipy Optimization [Source](https://docs.scipy.org/doc/scipy/reference/optimize.html#module-scipy.optimize)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SciPy optimize provides functions for minimizing (or maximizing) objective functions, possibly subject to constraints. It includes solvers for nonlinear problems (with support for both local and global optimization algorithms), linear programing, constrained and nonlinear least-squares, root finding and curve fitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Scalar Functions Optimization\n",
    "2. Local (Multivariate) Optimization\n",
    "```\n",
    "    ‘Nelder-Mead’ ,‘Powell’ ,‘CG’ ,‘BFGS’ ,‘Newton-CG’ \n",
    "    ‘L-BFGS-B’  ,‘TNC’  ,‘COBYLA’  ,‘SLSQP’  ,‘trust-constr’ \n",
    "    ‘dogleg’  ,‘trust-ncg’  ,‘trust-exact’  ,‘trust-krylov’  \n",
    "```\n",
    "3. Global Optimization\n",
    "4. Least-squares and Curve Fitting\n",
    "5. Root finding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Powell Method\n",
    "Method Powell is a modification of Powell’s method [3], [4] which is a conjugate direction method. It performs sequential one-dimensional minimizations along each vector of the directions set (direc field in options and info), which is updated at each iteration of the main minimization loop. The function need not be differentiable, and no derivatives are taken. If bounds are not provided, then an unbounded line search will be used. If bounds are provided and the initial guess is within the bounds, then every function evaluation throughout the minimization procedure will be within the bounds. If bounds are provided, the initial guess is outside the bounds, and direc is full rank (default has full rank), then some function evaluations during the first iteration may be outside the bounds, but every function evaluation after the first iteration will be within the bounds. If direc is not full rank, then some parameters may not be optimized and the solution is not guaranteed to be within the bounds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import numpy.linalg as la\n",
    "import scipy.optimize as sopt\n",
    "from scipy.optimize import minimize\n",
    "\n",
    "import matplotlib.pyplot as pt\n",
    "from mpl_toolkits.mplot3d import axes3d\n",
    "%matplotlib inline\n",
    "import seaborn as sns\n",
    "sns.set()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The minimum value of this function is 0 which is achieved when x =1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![img](img/0.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rosen(x):\n",
    "    \"\"\"The Rosenbrock function\"\"\"\n",
    "    return sum(100.0*(x[1:]-x[:-1]**2.0)**2.0 + (1-x[:-1])**2.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![img](img/1.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "x0 = np.array([1.3, 0.7, 0.8, 1.9, 1.2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.000000\n",
      "         Iterations: 19\n",
      "         Function evaluations: 1622\n"
     ]
    }
   ],
   "source": [
    "res = minimize(rosen, x0, method='Powell',options={'xtol': 1e-8, 'disp': True})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1., 1., 1., 1., 1.])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res.x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. https://andreask.cs.illinois.edu/cs357-s15/public/demos/12-optimization/Steepest%20Descent.html\n",
    "2. https://scipy-lectures.org/advanced/mathematical_optimization/auto_examples/plot_gradient_descent.html\n",
    "3. https://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html\n",
    "4. https://scipy-cookbook.readthedocs.io/index.html\n",
    "5. http://folk.ntnu.no/leifh/teaching/tkt4140/._main000.html"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}