{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# OPTaaS Surrogate Prediction\n",
    "\n",
    "### <span style=\"color:red\">Note:</span> To run this notebook, you need an API Key. You can get one <a href=\"mailto:info@mindfoundry.ai\">here</a>.\n",
    "\n",
    "The surrogate model is what the optimizer *thinks* the scoring function looks like. It is part of the mechanism used to choose optimal configurations.\n",
    "\n",
    "You can generate predictions from the surrogate model (effectively asking OPTaaS to guess what the scoring function may be at a certain point) at any set of arbitrary configuration points."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Connect to OPTaaS using your API Key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(optaas.client)\n",
    "\n",
    "client <- OPTaaSClient$new(\"https://optaas.mindfoundry.ai\", \"Your OPTaaS API Key\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a simple Task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "task <- client$create_task(\n",
    "    title=\"Basic 2D Example\",\n",
    "    parameters=list(\n",
    "        FloatParameter('x', minimum = -3, maximum = 1),\n",
    "        FloatParameter('y', minimum = -6, maximum = 21)\n",
    "    ),\n",
    "    goal=\"min\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define your scoring function\n",
    "Just a simple well with the minimum at (0, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "scoring_function <- function(x, y) {\n",
    "    x**2 + y**2\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run your task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1] \"Iteration: 1   Score: 57.25\"\n",
      "[1] \"Iteration: 2   Score: 207.0625\"\n",
      "[1] \"Iteration: 3   Score: 0.5625\"\n",
      "[1] \"Iteration: 4   Score: 19.265625\"\n",
      "[1] \"Iteration: 5   Score: 310.890625\"\n",
      "[1] \"Iteration: 6   Score: 124.515625\"\n",
      "[1] \"Iteration: 7   Score: 7.140625\"\n",
      "[1] \"Iteration: 8   Score: 3.94140625\"\n",
      "[1] \"Iteration: 9   Score: 157.87890625\"\n",
      "[1] \"Iteration: 10   Score: 380.53515625\"\n",
      "[1] \"Iteration: 11   Score: 1.0128225401033\"\n",
      "[1] \"Iteration: 12   Score: 0.00860669147744739\"\n",
      "[1] \"Iteration: 13   Score: 0.000837724442447517\"\n",
      "[1] \"Iteration: 14   Score: 0.00061925191746992\"\n",
      "[1] \"Iteration: 15   Score: 0.000603156957389812\"\n"
     ]
    }
   ],
   "source": [
    "best_result <- task$run(scoring_function, 15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ask the surrogate for a prediction at the known best point (x=0, y=0)\n",
    "The surrogate model should predict a fairly low score with high confidence, since it has been exploring the vicinity of this point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1] \"Score: 0.0116939091971489 Variance: 0.000152567573711395\"\n"
     ]
    }
   ],
   "source": [
    "known_best <- list(x=0, y=0)\n",
    "predictions <- task$get_surrogate_predictions(list(known_best))\n",
    "print(paste(\"Score:\", predictions[[1]]$mean, \"Variance:\", predictions[[1]]$variance))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ask the surrogate about a couple of points far away from the explored area (x=1, y=20) and (x=-3, y=-6)\n",
    "The surrogate model should be significantly less confident, as there were no evaluations near this point. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1] \"Score: 399.699123738722 Variance: 0.0407275300936555\"\n",
      "[1] \"Score: 42.0550946488813 Variance: 0.188253127450461\"\n"
     ]
    }
   ],
   "source": [
    "far_away_points <- list(list(x=1, y=20), list(x=-3, y=-6))\n",
    "predictions <- task$get_surrogate_predictions(far_away_points)\n",
    "print(paste(\"Score:\", predictions[[1]]$mean, \"Variance:\", predictions[[1]]$variance))\n",
    "print(paste(\"Score:\", predictions[[2]]$mean, \"Variance:\", predictions[[2]]$variance))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Want to know more?\n",
    "Here's an article we wrote on how the surrogate works: https://towardsdatascience.com/the-intuitions-behind-bayesian-optimization-with-gaussian-processes-7e00fcc898a0"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.4.1"
  },
  "nav_menu": {},
  "toc": {
   "navigate_menu": true,
   "number_sections": false,
   "sideBar": true,
   "threshold": 6,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}