{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How can we sample *some* of the variables in a pyll configuration space, while assigning values to the others?\n",
    "\n",
    "Let's look at a simple example involving 2 variables 'a' and 'b'.\n",
    "The 'a' variable controls whether our space returns -1 or some random number, 'b'.\n",
    "\n",
    "If we just run optimization normally, then we'll find that 'a' should be 0 (the index of the choice that gives the lowest\n",
    "return value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|██████████| 100/100 [00:00<00:00, 1413.57trial/s, best loss: -1.0]\n",
      "{'a': 0}\n"
     ]
    }
   ],
   "source": [
    "from hyperopt import hp, fmin, rand\n",
    "space = hp.choice('a', [-1, hp.uniform('b', 0, 1)])\n",
    "best = fmin(fn=lambda x: x, space=space, algo=rand.suggest, max_evals=100)\n",
    "print(best)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But what if someone else already set up the space, and we just run the search over the other part of the space, which corresponds to the uniform draw?\n",
    "\n",
    "The easiest way to do this is probably to *clone* the search space, while making some substitutions while we're at it.\n",
    "We can just make a new search space in which 'a' is no longer a hyperparameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 switch\n",
      "1   hyperopt_param\n",
      "2     Literal{a}\n",
      "3     randint\n",
      "4       Literal{2}\n",
      "5   Literal{-1}\n",
      "6   float\n",
      "7     hyperopt_param\n",
      "8       Literal{b}\n",
      "9       uniform\n",
      "10         Literal{0}\n",
      "11         Literal{1}\n"
     ]
    }
   ],
   "source": [
    "# put the configuration space in a local var\n",
    "# so that we can work on it.\n",
    "print(space)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The transformation we want to make on the search space is to replace the `randint` with a constant value of 1, \n",
    "corresponding to always choosing hyperparameter a to be the second element of the list of choices.\n",
    "\n",
    "Now, if you don't have access to the code that generated a search space, then you'll have to go digging around for the\n",
    "node you need to replace. There are two approaches you can use to do this: navigation and search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "by navigation:\n",
      "0 randint\n",
      "1   Literal{2}\n",
      "by search:\n",
      "0 randint\n",
      "1   Literal{2}\n"
     ]
    }
   ],
   "source": [
    "from hyperopt import pyll\n",
    "\n",
    "# The \"navigation\" approach to finding an internal\n",
    "# search space node:\n",
    "randint_node_nav = space.pos_args[0].pos_args[1]\n",
    "print(\"by navigation:\")\n",
    "print(randint_node_nav)\n",
    "\n",
    "# The \"search\" approach to finding an internal\n",
    "# search space node:\n",
    "randint_nodes = [node for node in pyll.dfs(space) if node.name == 'randint']\n",
    "randint_node_srch, = randint_nodes\n",
    "print(\"by search:\")\n",
    "print(randint_node_srch)\n",
    "\n",
    "assert randint_node_nav == randint_node_srch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 switch\n",
      "1   hyperopt_param\n",
      "2     Literal{a}\n",
      "3     Literal{1}\n",
      "4   Literal{-1}\n",
      "5   float\n",
      "6     hyperopt_param\n",
      "7       Literal{b}\n",
      "8       uniform\n",
      "9         Literal{0}\n",
      "10         Literal{1}\n"
     ]
    }
   ],
   "source": [
    "space_with_fixed_a = pyll.clone(space, memo={randint_node_nav: pyll.as_apply(1)})\n",
    "print(space_with_fixed_a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, having cloned the space with a new term for the randint, we can search the new space.  I wasn't sure if this would work because I haven't really tested the use of hyperopt_params that wrap around non-random nodes (here we replaced the randint with a constant) but it works for random search:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|██████████| 100/100 [00:00<00:00, 1149.65trial/s, best loss: 0.0020179230078352095]\n",
      "{'a': 1, 'b': 0.0020179230078352095}\n"
     ]
    }
   ],
   "source": [
    "best = fmin(fn=lambda x: x, space=space_with_fixed_a, algo=rand.suggest, max_evals=100)\n",
    "print(best)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yep, sure enough: The TPE implementation is broken by a hyperparameter that turns out to be a constant. At implementation time, that was not part of the plan."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/100 [00:00<?, ?trial/s, best loss=?]\n"
     ]
    },
    {
     "ename": "KeyError",
     "evalue": "'asarray'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-6-8b19efbb73ec>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mhyperopt\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mtpe\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mbest\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfmin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfn\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mlambda\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mspace\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mspace_with_fixed_a\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0malgo\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtpe\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msuggest\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmax_evals\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbest\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/hyperopt/fmin.py\u001b[0m in \u001b[0;36mfmin\u001b[0;34m(fn, space, algo, max_evals, timeout, loss_threshold, trials, rstate, allow_trials_fmin, pass_expr_memo_ctrl, catch_eval_exceptions, verbose, return_argmin, points_to_evaluate, max_queue_len, show_progressbar)\u001b[0m\n\u001b[1;32m    507\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    508\u001b[0m     \u001b[0;31m# next line is where the fmin is actually executed\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 509\u001b[0;31m     \u001b[0mrval\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexhaust\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    510\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    511\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mreturn_argmin\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/hyperopt/fmin.py\u001b[0m in \u001b[0;36mexhaust\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    328\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mexhaust\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    329\u001b[0m         \u001b[0mn_done\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrials\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 330\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmax_evals\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mn_done\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mblock_until_done\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masynchronous\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    331\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrials\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrefresh\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    332\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/hyperopt/fmin.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, N, block_until_done)\u001b[0m\n\u001b[1;32m    264\u001b[0m                     \u001b[0;31m# processes orchestration\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    265\u001b[0m                     new_trials = algo(\n\u001b[0;32m--> 266\u001b[0;31m                         \u001b[0mnew_ids\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdomain\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrials\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrstate\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrandint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m2\u001b[0m \u001b[0;34m**\u001b[0m \u001b[0;36m31\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    267\u001b[0m                     )\n\u001b[1;32m    268\u001b[0m                     \u001b[0;32massert\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnew_ids\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnew_trials\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/hyperopt/tpe.py\u001b[0m in \u001b[0;36msuggest\u001b[0;34m(new_ids, domain, trials, seed, prior_weight, n_startup_jobs, n_EI_candidates, gamma, verbose)\u001b[0m\n\u001b[1;32m    866\u001b[0m     \u001b[0;31m# use build_posterior_wrapper to create the pyll nodes\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    867\u001b[0m     observed, observed_loss, posterior = build_posterior_wrapper(\n\u001b[0;32m--> 868\u001b[0;31m         \u001b[0mdomain\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprior_weight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgamma\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    869\u001b[0m     )\n\u001b[1;32m    870\u001b[0m     \u001b[0mtt\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtime\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mt0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/hyperopt/tpe.py\u001b[0m in \u001b[0;36mbuild_posterior_wrapper\u001b[0;34m(domain, prior_weight, gamma)\u001b[0m\n\u001b[1;32m    830\u001b[0m         \u001b[0mobserved_loss\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"vals\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    831\u001b[0m         \u001b[0mpyll\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mLiteral\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgamma\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 832\u001b[0;31m         \u001b[0mpyll\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mLiteral\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfloat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprior_weight\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    833\u001b[0m     )\n\u001b[1;32m    834\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/hyperopt/tpe.py\u001b[0m in \u001b[0;36mbuild_posterior\u001b[0;34m(specs, prior_idxs, prior_vals, obs_idxs, obs_vals, obs_loss_idxs, obs_loss_vals, oloss_gamma, prior_weight)\u001b[0m\n\u001b[1;32m    708\u001b[0m                 \u001b[0mobs_below\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mobs_above\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mobs_memo\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mnode\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    709\u001b[0m                 \u001b[0maa\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mmemo\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0ma\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mnode\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpos_args\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 710\u001b[0;31m                 \u001b[0mfn\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0madaptive_parzen_samplers\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mnode\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    711\u001b[0m                 \u001b[0mb_args\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mobs_below\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprior_weight\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0maa\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    712\u001b[0m                 \u001b[0mnamed_args\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mmemo\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mnode\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnamed_args\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyError\u001b[0m: 'asarray'"
     ]
    }
   ],
   "source": [
    "from hyperopt import tpe\n",
    "best = fmin(fn=lambda x: x, space=space_with_fixed_a, algo=tpe.suggest, max_evals=100)\n",
    "print(best)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The TPE algorithm works if we make a different replacement in the graph. If we replace the entire \"hyperopt_param\" node corresponding to hyperparameter \"a\", then it works fine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|██████████| 100/100 [00:00<00:00, 352.69trial/s, best loss: 0.00037832452435874396]\n",
      "{'b': 0.00037832452435874396}\n"
     ]
    }
   ],
   "source": [
    "space_with_no_a = pyll.clone(space, memo={space.pos_args[0]: pyll.as_apply(1)})\n",
    "best = fmin(fn=lambda x: x, space=space_with_no_a, algo=tpe.suggest, max_evals=100)\n",
    "print(best)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}