{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# OPTaaS: Surrogate Prediction\n",
    "\n",
    "The surrogate model is what the optimizer *thinks* the scoring function looks like. It is part of the mechanism used to choose optimal configurations.\n",
    "\n",
    "You can generate predictions from the surrogate model (effectively asking OPTaaS to guess what the scoring function may be at a certain point) at any set of arbitrary configuration points."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Connect to OPTaaS using your API Key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mindfoundry.optaas.client.client import OPTaaSClient\n",
    "\n",
    "client = OPTaaSClient('https://optaas.mindfoundry.ai', '<Your OPTaaS API key>')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a simple task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mindfoundry.optaas.client.parameter import FloatParameter\n",
    "from mindfoundry.optaas.client.client import Goal\n",
    "\n",
    "task = client.create_task(\n",
    "    title='Basic 2D Example',\n",
    "    parameters=[\n",
    "        FloatParameter('x', minimum=-3, maximum=1),\n",
    "        FloatParameter('y', minimum=-6, maximum=21)\n",
    "    ],\n",
    "    goal=Goal.min,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define your scoring function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def scoring_function(x, y):\n",
    "    ''' A simple well with min at 0, 0'''\n",
    "    score = x**2 + y**2\n",
    "    return score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run your task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running task \"Basic 2D Example\" for 20 iterations\n",
      "(no score threshold set)\n",
      "\n",
      "Iteration: 0    Score: 57.25\n",
      "Configuration: {'x': -1.0, 'y': 7.5}\n",
      "\n",
      "Iteration: 1    Score: 207.0625\n",
      "Configuration: {'x': -2.0, 'y': 14.25}\n",
      "\n",
      "Iteration: 2    Score: 0.5625\n",
      "Configuration: {'x': 0.0, 'y': 0.75}\n",
      "\n",
      "Iteration: 3    Score: 19.265625\n",
      "Configuration: {'x': -1.5, 'y': 4.125}\n",
      "\n",
      "Iteration: 4    Score: 310.890625\n",
      "Configuration: {'x': 0.5, 'y': 17.625}\n",
      "\n",
      "Iteration: 5    Score: 124.515625\n",
      "Configuration: {'x': -2.5, 'y': 10.875}\n",
      "\n",
      "Iteration: 6    Score: 7.140625\n",
      "Configuration: {'x': -0.5, 'y': -2.625}\n",
      "\n",
      "Iteration: 7    Score: 3.94140625\n",
      "Configuration: {'x': -1.75, 'y': -0.9375}\n",
      "\n",
      "Iteration: 8    Score: 157.87890625\n",
      "Configuration: {'x': 0.25, 'y': 12.5625}\n",
      "\n",
      "Iteration: 9    Score: 380.53515625\n",
      "Configuration: {'x': -2.75, 'y': 19.3125}\n",
      "\n",
      "Iteration: 10    Score: 1.0128486056914057\n",
      "Configuration: {'x': 0.99999998, 'y': 0.11335186673101204}\n",
      "\n",
      "Iteration: 11    Score: 0.00927530250495255\n",
      "Configuration: {'x': 0.03888422866549611, 'y': 0.0881097001813192}\n",
      "\n",
      "Iteration: 12    Score: 0.0012117009114401805\n",
      "Configuration: {'x': 0.0050892493390089985, 'y': 0.03443545342529378}\n",
      "\n",
      "Iteration: 13    Score: 0.0006806662021513174\n",
      "Configuration: {'x': 0.006729942231381237, 'y': 0.025206627694191637}\n",
      "\n",
      "Iteration: 14    Score: 0.0006364356966787191\n",
      "Configuration: {'x': 0.0010303479825333319, 'y': 0.02520662769419203}\n",
      "\n",
      "Iteration: 15    Score: 0.00022768157398298682\n",
      "Configuration: {'x': -0.0005074972550283101, 'y': 0.015080584223402142}\n",
      "\n",
      "Iteration: 16    Score: 0.00024225926376497916\n",
      "Configuration: {'x': 0.0005289767480935436, 'y': 0.015555688585368233}\n",
      "\n",
      "Iteration: 17    Score: 0.0002342686143728285\n",
      "Configuration: {'x': 6.452081614416779e-05, 'y': 0.015305699965604729}\n",
      "\n",
      "Iteration: 18    Score: 0.00021932618498741224\n",
      "Configuration: {'x': -0.0006874418906318507, 'y': 0.014793701654231666}\n",
      "\n",
      "Iteration: 19    Score: 0.00021932619083313206\n",
      "Configuration: {'x': -0.0006874425416874602, 'y': 0.014793701821552655}\n",
      "\n",
      "Task Completed\n",
      "\n",
      "Best Result: { 'configuration': { 'type': 'exploitation',\n",
      "  'values': {'x': -0.0006874418906318507, 'y': 0.014793701654231666}},\n",
      "  'score': 0.00021932618498741224,\n",
      "  'user_defined_data': None}\n"
     ]
    }
   ],
   "source": [
    "best_result = task.run(scoring_function, max_iterations=20)\n",
    "print(\"Best Result:\", best_result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating the surrogate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ask the surrogate for a prediction at the known best point (x=0, y=0)\n",
    "The surrogate model should predict a fairly low score with high confidence, since it has been exploring the vicinity of this point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "interesting_configs = [{'x': 0.0, 'y': 0.0}]  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = task.get_surrogate_predictions(interesting_configs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0.0047830747120087835, 0.00012424807215192135)]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(p.mean, p.variance) for p in predictions]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ask the surrogate about a couple of points far away from the explored area (x=1, y=20) and (x=-3, y=-6)\n",
    "The surrogate model should be significantly less confident, as there were no evaluations near this point. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "far_away_points = [{'x': 1.0, 'y': 20.0}, {'x': -1.0, 'y': -6.0}]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = task.get_surrogate_predictions(far_away_points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(399.7652778647905, 0.04206749064962412),\n",
       " (35.073995608525436, 0.0782837406899546)]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(p.mean, p.variance) for p in predictions]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observation\n",
    "The predictions are quite accurate, as the function is quite simple, so the surrogate is able to learn it fairly quickly. The increased variance reflect the lower certainty, as expected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Want to know more?\n",
    "Here's an article we wrote on how the surrogate works: https://towardsdatascience.com/the-intuitions-behind-bayesian-optimization-with-gaussian-processes-7e00fcc898a0"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}