![](\"../slides/diagrams/IBM_Blue_Gene_P_supercomputer.jpg\")
\n",
"![](\"../slides/diagrams/ClaudeShannon_MFO3807.jpg\")
\n",
"![](\"../slides/diagrams/Gartner_Hype_Cycle.png\")
\n",
"\n",
"The [Gartner Hype Cycle](https://en.wikipedia.org/wiki/Hype_cycle) tries\n",
"to assess where an idea is in terms of maturity and adoption. It splits\n",
"the evolution of technology into a technological trigger, a peak of\n",
"expectations followed by a trough of disillusionment and a final\n",
"ascension into a useful technology. It looks rather like a classical\n",
"control response to a final set point."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pods\n",
"from ipywidgets import IntSlider"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pods.notebook.display_plots('ai-bd-dm-dl-ml-google-trends{sample:0>3}.svg', \n",
" '../slides/diagrams/data-science/', sample=IntSlider(1, 1, 4, 1))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](\"../slides/diagrams/Portrait_of_Karl_Pearson.jpg\")
\n",
"\n",
"[Karl Pearson](https://en.wikipedia.org/wiki/Karl_Pearson) (1857-1936),\n",
"[Ronald Fisher](https://en.wikipedia.org/wiki/Ronald_Fisher) (1890-1962)\n",
"and others considered the question of what conclusions can truly be\n",
"drawn from data. Their mathematical studies act as a restraint on our\n",
"tendency to over-interpret and see patterns where there are none. They\n",
"introduced concepts such as randomized control trials that form a\n",
"mainstay of the our decision making today, from government, to\n",
"clinicians to large scale A/B testing that determines the nature of the\n",
"web interfaces we interact with on social media and shopping.\n",
"\n",
"Today the statement \"There are three types of lies: lies, damned lies\n",
"and 'big data'\" may be more apt. We are revisiting many of the mistakes\n",
"made in interpreting data from the 19th century. Big data is laid down\n",
"by happenstance, rather than actively collected with a particular\n",
"question in mind. That means it needs to be treated with care when\n",
"conclusions are being drawn. For data science to succede it needs the\n",
"same form of rigour that Pearson and Fisher brought to statistics, a\n",
"\"mathematical data science\" is needed.\n",
"\n",
"## What is Machine Learning?\n",
"\n",
"What is machine learning? At its most basic level machine learning is a\n",
"combination of\n",
"\n",
"$$ \\text{data} + \\text{model} \\xrightarrow{\\text{compute}} \\text{prediction}$$\n",
"\n",
"where *data* is our observations. They can be actively or passively\n",
"acquired (meta-data). The *model* contains our assumptions, based on\n",
"previous experience. That experience can be other data, it can come from\n",
"transfer learning, or it can merely be our beliefs about the\n",
"regularities of the universe. In humans our models include our inductive\n",
"biases. The *prediction* is an action to be taken or a categorization or\n",
"a quality score. The reason that machine learning has become a mainstay\n",
"of artificial intelligence is the importance of predictions in\n",
"artificial intelligence. The data and the model are combined through\n",
"computation.\n",
"\n",
"In practice we normally perform machine learning using two functions. To\n",
"combine data with a model we typically make use of:\n",
"\n",
"**a prediction function** a function which is used to make the\n",
"predictions. It includes our beliefs about the regularities of the\n",
"universe, our assumptions about how the world works, e.g. smoothness,\n",
"spatial similarities, temporal similarities.\n",
"\n",
"**an objective function** a function which defines the cost of\n",
"misprediction. Typically it includes knowledge about the world's\n",
"generating processes (probabilistic objectives) or the costs we pay for\n",
"mispredictions (empiricial risk minimization).\n",
"\n",
"The combination of data and model through the prediction function and\n",
"the objectie function leads to a *learning algorithm*. The class of\n",
"prediction functions and objective functions we can make use of is\n",
"restricted by the algorithms they lead to. If the prediction function or\n",
"the objective function are too complex, then it can be difficult to find\n",
"an appropriate learning algorithm. Much of the acdemic field of machine\n",
"learning is the quest for new learning algorithms that allow us to bring\n",
"different types of models and data together.\n",
"\n",
"A useful reference for state of the art in machine learning is the UK\n",
"Royal Society Report, [Machine Learning: Power and Promise of Computers\n",
"that Learn by\n",
"Example](https://royalsociety.org/~/media/policy/projects/machine-learning/publications/machine-learning-report.pdf).\n",
"\n",
"You can also check my blog post on [\"What is Machine\n",
"Learning?\"](http://inverseprobability.com/2017/07/17/what-is-machine-learning)\n",
"\n",
"### Artificial Intelligence and Data Science\n",
"\n",
"Machine learning technologies have been the driver of two related, but\n",
"distinct disciplines. The first is *data science*. Data science is an\n",
"emerging field that arises from the fact that we now collect so much\n",
"data by happenstance, rather than by *experimental design*. Classical\n",
"statistics is the science of drawing conclusions from data, and to do so\n",
"statistical experiments are carefully designed. In the modern era we\n",
"collect so much data that there's a desire to draw inferences directly\n",
"from the data.\n",
"\n",
"As well as machine learning, the field of data science draws from\n",
"statistics, cloud computing, data storage (e.g. streaming data),\n",
"visualization and data mining.\n",
"\n",
"In contrast, artificial intelligence technologies typically focus on\n",
"emulating some form of human behaviour, such as understanding an image,\n",
"or some speech, or translating text from one form to another. The recent\n",
"advances in artifcial intelligence have come from machine learning\n",
"providing the automation. But in contrast to data science, in artifcial\n",
"intelligence the data is normally collected with the specific task in\n",
"mind. In this sense it has strong relations to classical statistics.\n",
"\n",
"Classically artificial intelligence worried more about *logic* and\n",
"*planning* and focussed less on data driven decision making. Modern\n",
"machine learning owes more to the field of *Cybernetics*\n",
"[@Wiener:cybernetics48] than artificial intelligence. Related fields\n",
"include *robotics*, *speech recognition*, *language understanding* and\n",
"*computer vision*.\n",
"\n",
"There are strong overlaps between the fields, the wide availability of\n",
"data by happenstance makes it easier to collect data for designing AI\n",
"systems. These relations are coming through wide availability of sensing\n",
"technologies that are interconnected by celluar networks, WiFi and the\n",
"internet. This phenomenon is sometimes known as the *Internet of\n",
"Things*, but this feels like a dangerous misnomer. We must never forget\n",
"that we are interconnecting people, not things.\n",
"\n",
"## Natural and Artificial Intelligence: Embodiment Factors\n",
"\n",
"| \n", " | \n", "\n",
"\n",
" | \n",
"\n",
"\n",
" | \n",
"
| \n", "compute\n", " | \n", "\n", "$$\\approx 100 \\text{ gigaflops}$$\n", " | \n", "\n", "$$\\approx 16 \\text{ petaflops}$$\n", " | \n", "
| \n", "communicate\n", " | \n", "\n", "$$1 \\text{ gigbit/s}$$\n", " | \n", "\n", "$$100 \\text{ bit/s}$$\n", " | \n", "
| \n", "(compute/communicate)\n", " | \n", "\n", "$$10^{4}$$\n", " | \n", "\n", "$$10^{14}$$\n", " | \n", "
![](\"../slides/diagrams/640px-Marcel_Renault_1903.jpg\")
\n",
"\n",
"In contrast, our computers have less computational power, but they can\n",
"communicate far more fluidly. They are more like a go-kart, less well\n",
"powered, but with tires that allow them to deploy that power.\n",
"\n",
"![](\"../slides/diagrams/Caleb_McDuff_WIX_Silence_Racing_livery.jpg\")
\n",
"\n",
"For humans, that means much of our computation should be dedicated to\n",
"considering *what* we should compute. To do that efficiently we need to\n",
"model the world around us. The most complex thing in the world around us\n",
"is other humans. So it is no surprise that we model them. We second\n",
"guess what their intentions are, and our communication is only necessary\n",
"when they are departing from how we model them. Naturally, for this to\n",
"work well, we need to understand those we work closely with. So it is no\n",
"surprise that social communication, social bonding, forms so much of a\n",
"part of our use of our limited bandwidth.\n",
"\n",
"There is a second effect here, our need to anthropomorphise objects\n",
"around us. Our tendency to model our fellow humans extends to when we\n",
"interact with other entities in our environment. To our pets as well as\n",
"inanimate objects around us, such as computers or even our cars. This\n",
"tendency to overinterpret could be a consequence of our limited ability\n",
"to communicate.\n",
"\n",
"For more details see this paper [\"Living Together: Mind and Machine\n",
"Intelligence\"](https://arxiv.org/abs/1705.07996), and this [TEDx\n",
"talk](http://inverseprobability.com/talks/lawrence-tedx17/living-together.html).\n",
"\n",
"## Evolved Relationship with Information\n",
"\n",
"The high bandwidth of computers has resulted in a close relationship\n",
"between the computer and data. Large amounts of information can flow\n",
"between the two. The degree to which the computer is mediating our\n",
"relationship with data means that we should consider it an intermediary.\n",
"\n",
"Originaly our low bandwith relationship with data was affected by two\n",
"characteristics. Firstly, our tendency to over-interpret driven by our\n",
"need to extract as much knowledge from our low bandwidth information\n",
"channel as possible. Secondly, by our improved understanding of the\n",
"domain of *mathematical* statistics and how our cognitive biases can\n",
"mislead us.\n",
"\n",
"With this new set up there is a potential for assimilating far more\n",
"information via the computer, but the computer can present this to us in\n",
"various ways. If it's motives are not aligned with ours then it can\n",
"misrepresent the information. This needn't be nefarious it can be simply\n",
"as a result of the computer pursuing a different objective from us. For\n",
"example, if the computer is aiming to maximize our interaction time that\n",
"may be a different objective from ours which may be to summarize\n",
"information in a representative manner in the *shortest* possible length\n",
"of time.\n",
"\n",
"For example, for me it was a common experience to pick up my telephone\n",
"with the intention of checking when my next appointment was, but to soon\n",
"find myself distracted by another application on the phone, and end up\n",
"reading something on the internet. By the time I'd finished reading, I\n",
"would often have forgotten the reason I picked up my phone in the first\n",
"place.\n",
"\n",
"There are great benefits to be had from the huge amount of information\n",
"we can unlock from this evolved relationship between us and data. In\n",
"biology, large scale data sharing has been driven by a revolution in\n",
"genomic, transcriptomic and epigenomic measurement. The improved\n",
"inferences that that can be drawn through summarizing data by computer\n",
"have fundamentally changed the nature of biological science, now this\n",
"phenomenon is also infuencing us in our daily lives as data measured by\n",
"*happenstance* is increasingly used to characterize us.\n",
"\n",
"Better mediation of this flow actually requires a better understanding\n",
"of human-computer interaction. This in turn involves understanding our\n",
"own intelligence better, what its cognitive biases are and how these\n",
"might mislead us.\n",
"\n",
"For further thoughts see [this Guardian\n",
"article](https://www.theguardian.com/media-network/2015/jul/23/data-driven-economy-marketing)\n",
"from 2015 on marketing in the internet era and [this blog\n",
"post](http://inverseprobability.com/2015/12/04/what-kind-of-ai) on\n",
"System Zero.\n",
"\n",
"![](\"../slides/diagrams/deepface_neg.png\")
\n",
"![](\"../slides/diagrams/576px-Early_Pinball.jpg\")
\n",
"![](\"../slides/diagrams/pinball001.svg\")
\n",
"\n",
"![](\"../slides/diagrams/pinball002.svg\")
\n",
"![](\"../slides/diagrams/SteamEngine_Boulton&Watt_1784_neg.png\")
\n",
"![](\"../slides/diagrams/uq/mountaincar.png\")
\n",
"\n",
"The goal is to define a sequence of actions (push the car right or left\n",
"with certain intensity) to make the car reach the flag after a number\n",
"$T$ of time steps.\n",
"\n",
"At each time step $t$, the car is characterized by a vector\n",
"$\\inputVector_{t} = (p_t,v_t)$ of states which are respectively the the\n",
"position and velocity of the car at time $t$. For a sequence of states\n",
"(an episode), the dynamics of the car is given by\n",
"\n",
"$$\\inputVector_{t+1} = \\mappingFunction(\\inputVector_{t},\\textbf{u}_{t})$$\n",
"\n",
"where $\\textbf{u}_{t}$ is the value of an action force, which in this\n",
"example corresponds to push car to the left (negative value) or to the\n",
"right (positive value). The actions across a full episode are\n",
"represented in a policy $\\textbf{u}_{t} = \\pi(\\inputVector_{t},\\theta)$\n",
"that acts according to the current state of the car and some parameters\n",
"$\\theta$. In the following examples we will assume that the policy is\n",
"linear which allows us to write $\\pi(\\inputVector_{t},\\theta)$ as\n",
"\n",
"$$\\pi(\\inputVector,\\theta)= \\theta_0 + \\theta_p p + \\theta_vv.$$\n",
"\n",
"For $t=1,\\dots,T$ now given some initial state $\\inputVector_{0}$ and\n",
"some some values of each $\\textbf{u}_{t}$, we can **simulate** the full\n",
"dynamics of the car for a full episode using\n",
"[Gym](https://gym.openai.com/envs/). The values of $\\textbf{u}_{t}$ are\n",
"fully determined by the parameters of the linear controller.\n",
"\n",
"After each episode of length $T$ is complete, a reward function\n",
"$R_{T}(\\theta)$ is computed. In the mountain car example the reward is\n",
"computed as 100 for reaching the target of the hill on the right hand\n",
"side, minus the squared sum of actions (a real negative to push to the\n",
"left and a real positive to push to the right) from start to goal. Note\n",
"that our reward depend on $\\theta$ as we make it dependent on the\n",
"parameters of the linear controller.\n",
"\n",
"### Emulate the Mountain Car"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import gym"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"env = gym.make('MountainCarContinuous-v0')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Our goal in this section is to find the parameters $\\theta$ of the\n",
"linear controller such that\n",
"\n",
"$$\\theta^* = arg \\max_{\\theta} R_T(\\theta).$$\n",
"\n",
"In this section, we directly use Bayesian optimization to solve this\n",
"problem. We will use [GPyOpt](https://sheffieldml.github.io/GPyOpt/) so\n",
"we first define the objective function:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import mountain_car as mc\n",
"import GPyOpt"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"obj_func = lambda x: mc.run_simulation(env, x)[0]\n",
"objective = GPyOpt.core.task.SingleObjective(obj_func)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For each set of parameter values of the linear controller we can run an\n",
"episode of the simulator (that we fix to have a horizon of $T=500$) to\n",
"generate the reward. Using as input the parameters of the controller and\n",
"as outputs the rewards we can build a Gaussian process emulator of the\n",
"reward.\n",
"\n",
"We start defining the input space, which is three-dimensional:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"## --- We define the input space of the emulator\n",
"\n",
"space= [{'name':'postion_parameter', 'type':'continuous', 'domain':(-1.2, +1)},\n",
" {'name':'velocity_parameter', 'type':'continuous', 'domain':(-1/0.07, +1/0.07)},\n",
" {'name':'constant', 'type':'continuous', 'domain':(-1, +1)}]\n",
"\n",
"design_space = GPyOpt.Design_space(space=space)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we initizialize a Gaussian process emulator."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model = GPyOpt.models.GPModel(optimize_restarts=5, verbose=False, exact_feval=True, ARD=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In Bayesian optimization an acquisition function is used to balance\n",
"exploration and exploitation to evaluate new locations close to the\n",
"optimum of the objective. In this notebook we select the expected\n",
"improvement (EI). For further details have a look to the review paper of\n",
"[Shahriari et al\n",
"(2015)](http://www.cs.ox.ac.uk/people/nando.defreitas/publications/BayesOptLoop.pdf)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"aquisition_optimizer = GPyOpt.optimization.AcquisitionOptimizer(design_space)\n",
"acquisition = GPyOpt.acquisitions.AcquisitionEI(model, design_space, optimizer=aquisition_optimizer)\n",
"evaluator = GPyOpt.core.evaluators.Sequential(acquisition) # Collect points sequentially, no parallelization."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To initalize the model we start sampling some initial points (25) for\n",
"the linear controler randomly."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from GPyOpt.experiment_design.random_design import RandomDesign"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"n_initial_points = 25\n",
"random_design = RandomDesign(design_space)\n",
"initial_design = random_design.get_samples(n_initial_points)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Before we start any optimization, lets have a look to the behavior of\n",
"the car with the first of these initial points that we have selected\n",
"randomly."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"random_controller = initial_design[0,:]\n",
"_, _, _, frames = mc.run_simulation(env, np.atleast_2d(random_controller), render=True)\n",
"anim=mc.animate_frames(frames, 'Random linear controller')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from IPython.core.display import HTML"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"HTML(anim.to_jshtml())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"As we can see the random linear controller does not manage to push the\n",
"car to the top of the mountain. Now, let's optimize the regret using\n",
"Bayesian optimization and the emulator for the reward. We try 50 new\n",
"parameters chosen by the EI."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"max_iter = 50\n",
"bo = GPyOpt.methods.ModularBayesianOptimization(model, design_space, objective, acquisition, evaluator, initial_design)\n",
"bo.run_optimization(max_iter = max_iter )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we visualize the result for the best controller that we have found\n",
"with Bayesian optimization."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"_, _, _, frames = mc.run_simulation(env, np.atleast_2d(bo.x_opt), render=True)\n",
"anim=mc.animate_frames(frames, 'Best controller after 50 iterations of Bayesian optimization')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"HTML(anim.to_jshtml())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"he car can now make it to the top of the mountain! Emulating the reward\n",
"function and using the EI helped as to find a linear controller that\n",
"solves the problem.\n",
"\n",
"### Data Efficient Emulation\n",
"\n",
"In the previous section we solved the mountain car problem by directly\n",
"emulating the reward but no considerations about the dynamics\n",
"$\\inputVector_{t+1} = \\mappingFunction(\\inputVector_{t},\\textbf{u}_{t})$\n",
"of the system were made. Note that we had to run 75 episodes of 500\n",
"steps each to solve the problem, which required to call the simulator\n",
"$500\\times 75 =37500$ times. In this section we will show how it is\n",
"possible to reduce this number by building an emulator for $f$ that can\n",
"later be used to directly optimize the control.\n",
"\n",
"The inputs of the model for the dynamics are the velocity, the position\n",
"and the value of the control so create this space accordingly."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import gym"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"env = gym.make('MountainCarContinuous-v0')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import GPyOpt"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"space_dynamics = [{'name':'position', 'type':'continuous', 'domain':[-1.2, +0.6]},\n",
" {'name':'velocity', 'type':'continuous', 'domain':[-0.07, +0.07]},\n",
" {'name':'action', 'type':'continuous', 'domain':[-1, +1]}]\n",
"design_space_dynamics = GPyOpt.Design_space(space=space_dynamics)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The outputs are the velocity and the position. Indeed our model will\n",
"capture the change in position and velocity on time. That is, we will\n",
"model\n",
"\n",
"$$\\Delta v_{t+1} = v_{t+1} - v_{t}$$\n",
"\n",
"$$\\Delta x_{t+1} = p_{t+1} - p_{t}$$\n",
"\n",
"with Gaussian processes with prior mean $v_{t}$ and $p_{t}$\n",
"respectively. As a covariance function, we use a Matern52. We need\n",
"therefore two models to capture the full dynamics of the system."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"position_model = GPyOpt.models.GPModel(optimize_restarts=5, verbose=False, exact_feval=True, ARD=True)\n",
"velocity_model = GPyOpt.models.GPModel(optimize_restarts=5, verbose=False, exact_feval=True, ARD=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, we sample some input parameters and use the simulator to compute\n",
"the outputs. Note that in this case we are not running the full\n",
"episodes, we are just using the simulator to compute\n",
"$\\inputVector_{t+1}$ given $\\inputVector_{t}$ and $\\textbf{u}_{t}$."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"from GPyOpt.experiment_design.random_design import RandomDesign\n",
"import mountain_car as mc"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"### --- Random locations of the inputs\n",
"n_initial_points = 500\n",
"random_design_dynamics = RandomDesign(design_space_dynamics)\n",
"initial_design_dynamics = random_design_dynamics.get_samples(n_initial_points)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"### --- Simulation of the (normalized) outputs\n",
"y = np.zeros((initial_design_dynamics.shape[0], 2))\n",
"for i in range(initial_design_dynamics.shape[0]):\n",
" y[i, :] = mc.simulation(initial_design_dynamics[i, :])\n",
"\n",
"# Normalize the data from the simulation\n",
"y_normalisation = np.std(y, axis=0)\n",
"y_normalised = y/y_normalisation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In general we might use much smarter strategies to design our emulation\n",
"of the simulator. For example, we could use the variance of the\n",
"predictive distributions of the models to collect points using\n",
"uncertainty sampling, which will give us a better coverage of the space.\n",
"For simplicity, we move ahead with the 500 randomly selected points.\n",
"\n",
"Now that we have a data set, we can update the emulators for the\n",
"location and the velocity."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"position_model.updateModel(initial_design_dynamics, y[:, [0]], None, None)\n",
"velocity_model.updateModel(initial_design_dynamics, y[:, [1]], None, None)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can now have a look to how the emulator and the simulator match.\n",
"First, we show a contour plot of the car aceleration for each pair of\n",
"can position and velocity. You can use the bar bellow to play with the\n",
"values of the controler to compare the emulator and the simulator."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from IPython.html.widgets import interact"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"We can see how the emulator is doing a fairly good job approximating the\n",
"simulator. On the edges, however, it struggles to captures the dynamics\n",
"of the simulator.\n",
"\n",
"Given some input parameters of the linear controlling, how do the\n",
"dynamics of the emulator and simulator match? In the following figure we\n",
"show the position and velocity of the car for the 500 time steps of an\n",
"episode in which the parameters of the linear controller have been fixed\n",
"beforehand. The value of the input control is also shown."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"controller_gains = np.atleast_2d([0, .6, 1]) # change the valus of the linear controller to observe the trayectories."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](\"../slides/diagrams/uq/emu_sim_comparison.svg\")
\n",
"\n",
"We now make explicit use of the emulator, using it to replace the\n",
"simulator and optimize the linear controller. Note that in this\n",
"optimization, we don't need to query the simulator anymore as we can\n",
"reproduce the full dynamics of an episode using the emulator. For\n",
"illustrative purposes, in this example we fix the initial location of\n",
"the car.\n",
"\n",
"We define the objective reward function in terms of the simulator."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"### --- Optimize control parameters with emulator\n",
"car_initial_location = np.asarray([-0.58912799, 0]) \n",
"\n",
"### --- Reward objective function using the emulator\n",
"obj_func_emulator = lambda x: mc.run_emulation([position_model, velocity_model], x, car_initial_location)[0]\n",
"objective_emulator = GPyOpt.core.task.SingleObjective(obj_func_emulator)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And as before, we use Bayesian optimization to find the best possible\n",
"linear controller."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"### --- Elements of the optimization that will use the multi-fidelity emulator\n",
"model = GPyOpt.models.GPModel(optimize_restarts=5, verbose=False, exact_feval=True, ARD=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The design space is the three continuous variables that make up the\n",
"linear controller."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"space= [{'name':'linear_1', 'type':'continuous', 'domain':(-1/1.2, +1)},\n",
" {'name':'linear_2', 'type':'continuous', 'domain':(-1/0.07, +1/0.07)},\n",
" {'name':'constant', 'type':'continuous', 'domain':(-1, +1)}]\n",
"\n",
"design_space = GPyOpt.Design_space(space=space)\n",
"aquisition_optimizer = GPyOpt.optimization.AcquisitionOptimizer(design_space)\n",
"\n",
"random_design = RandomDesign(design_space)\n",
"initial_design = random_design.get_samples(25)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We set the acquisition function to be expected improvement using\n",
"`GPyOpt`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"acquisition = GPyOpt.acquisitions.AcquisitionEI(model, design_space, optimizer=aquisition_optimizer)\n",
"evaluator = GPyOpt.core.evaluators.Sequential(acquisition)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"bo_emulator = GPyOpt.methods.ModularBayesianOptimization(model, design_space, objective_emulator, acquisition, evaluator, initial_design)\n",
"bo_emulator.run_optimization(max_iter=50)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"_, _, _, frames = mc.run_simulation(env, np.atleast_2d(bo_emulator.x_opt), render=True)\n",
"anim=mc.animate_frames(frames, 'Best controller using the emulator of the dynamics')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from IPython.core.display import HTML"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"HTML(anim.to_jshtml())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"And the problem is again solved, but in this case we have replaced the\n",
"simulator of the car dynamics by a Gaussian process emulator that we\n",
"learned by calling the simulator only 500 times. Compared to the 37500\n",
"calls that we needed when applying Bayesian optimization directly on the\n",
"simulator this is a great gain.\n",
"\n",
"In some scenarios we have simulators of the same environment that have\n",
"different fidelities, that is that reflect with different level of\n",
"accuracy the dynamics of the real world. Running simulations of the\n",
"different fidelities also have a different cost: hight fidelity\n",
"simulations are more expensive the cheaper ones. If we have access to\n",
"these simulators we can combine high and low fidelity simulations under\n",
"the same model.\n",
"\n",
"So let's assume that we have two simulators of the mountain car\n",
"dynamics, one of high fidelity (the one we have used) and another one of\n",
"low fidelity. The traditional approach to this form of multi-fidelity\n",
"emulation is to assume that\n",
"\n",
"$$\\mappingFunction_i\\left(\\inputVector\\right) = \\rho\\mappingFunction_{i-1}\\left(\\inputVector\\right) + \\delta_i\\left(\\inputVector \\right)$$\n",
"\n",
"where $\\mappingFunction_{i-1}\\left(\\inputVector\\right)$ is a low\n",
"fidelity simulation of the problem of interest and\n",
"$\\mappingFunction_i\\left(\\inputVector\\right)$ is a higher fidelity\n",
"simulation. The function $\\delta_i\\left(\\inputVector \\right)$ represents\n",
"the difference between the lower and higher fidelity simulation, which\n",
"is considered additive. The additive form of this covariance means that\n",
"if $\\mappingFunction_{0}\\left(\\inputVector\\right)$ and\n",
"$\\left\\{\\delta_i\\left(\\inputVector \\right)\\right\\}_{i=1}^m$ are all\n",
"Gaussian processes, then the process over all fidelities of simuation\n",
"will be a joint Gaussian process.\n",
"\n",
"But with Deep Gaussian processes we can consider the form\n",
"\n",
"$$\\mappingFunction_i\\left(\\inputVector\\right) = \\mappingFunctionTwo_{i}\\left(\\mappingFunction_{i-1}\\left(\\inputVector\\right)\\right) + \\delta_i\\left(\\inputVector \\right),$$\n",
"\n",
"where the low fidelity representation is non linearly transformed by\n",
"$\\mappingFunctionTwo(\\cdot)$ before use in the process. This is the\n",
"approach taken in @Perdikaris:multifidelity17. But once we accept that\n",
"these models can be composed, a highly flexible framework can emerge. A\n",
"key point is that the data enters the model at different levels, and\n",
"represents different aspects. For example these correspond to the two\n",
"fidelities of the mountain car simulator.\n",
"\n",
"We start by sampling both of them at 250 random input locations."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import gym"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"env = gym.make('MountainCarContinuous-v0')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import GPyOpt"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"### --- Collect points from low and high fidelity simulator --- ###\n",
"\n",
"space = GPyOpt.Design_space([\n",
" {'name':'position', 'type':'continuous', 'domain':(-1.2, +1)},\n",
" {'name':'velocity', 'type':'continuous', 'domain':(-0.07, +0.07)},\n",
" {'name':'action', 'type':'continuous', 'domain':(-1, +1)}])\n",
"\n",
"n_points = 250\n",
"random_design = GPyOpt.experiment_design.RandomDesign(space)\n",
"x_random = random_design.get_samples(n_points)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, we evaluate the high and low fidelity simualtors at those\n",
"locations."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import mountain_car as mc"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"d_position_hf = np.zeros((n_points, 1))\n",
"d_velocity_hf = np.zeros((n_points, 1))\n",
"d_position_lf = np.zeros((n_points, 1))\n",
"d_velocity_lf = np.zeros((n_points, 1))\n",
"\n",
"# --- Collect high fidelity points\n",
"for i in range(0, n_points):\n",
" d_position_hf[i], d_velocity_hf[i] = mc.simulation(x_random[i, :])\n",
"\n",
"# --- Collect low fidelity points \n",
"for i in range(0, n_points):\n",
" d_position_lf[i], d_velocity_lf[i] = mc.low_cost_simulation(x_random[i, :])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It is time to build the multi-fidelity model for both the position and\n",
"the velocity.\n",
"\n",
"As we did in the previous section we use the emulator to optimize the\n",
"simulator. In this case we use the high fidelity output of the emulator."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"### --- Optimize controller parameters \n",
"obj_func = lambda x: mc.run_simulation(env, x)[0]\n",
"obj_func_emulator = lambda x: mc.run_emulation([position_model, velocity_model], x, car_initial_location)[0]\n",
"objective_multifidelity = GPyOpt.core.task.SingleObjective(obj_func)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And we optimize using Bayesian optimzation."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from GPyOpt.experiment_design.random_design import RandomDesign"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model = GPyOpt.models.GPModel(optimize_restarts=5, verbose=False, exact_feval=True, ARD=True)\n",
"space= [{'name':'linear_1', 'type':'continuous', 'domain':(-1/1.2, +1)},\n",
" {'name':'linear_2', 'type':'continuous', 'domain':(-1/0.07, +1/0.07)},\n",
" {'name':'constant', 'type':'continuous', 'domain':(-1, +1)}]\n",
"\n",
"design_space = GPyOpt.Design_space(space=space)\n",
"aquisition_optimizer = GPyOpt.optimization.AcquisitionOptimizer(design_space)\n",
"\n",
"n_initial_points = 25\n",
"random_design = RandomDesign(design_space)\n",
"initial_design = random_design.get_samples(n_initial_points)\n",
"acquisition = GPyOpt.acquisitions.AcquisitionEI(model, design_space, optimizer=aquisition_optimizer)\n",
"evaluator = GPyOpt.core.evaluators.Sequential(acquisition)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"bo_multifidelity = GPyOpt.methods.ModularBayesianOptimization(model, design_space, objective_multifidelity, acquisition, evaluator, initial_design)\n",
"bo_multifidelity.run_optimization(max_iter=50)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"_, _, _, frames = mc.run_simulation(env, np.atleast_2d(bo_multifidelity.x_opt), render=True)\n",
"anim=mc.animate_frames(frames, 'Best controller with multi-fidelity emulator')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from IPython.core.display import HTML"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"HTML(anim.to_jshtml())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"And problem solved! We see how the problem is also solved with 250\n",
"observations of the high fidelity simulator and 250 of the low fidelity\n",
"simulator.\n",
"\n",
"- Artificial Intelligence and Data Science are fundamentally\n",
" different.\n",
"- In one you are dealing with data collected by happenstance.\n",
"- In the other you are trying to build systems in the real world,\n",
" often by actively collecting data.\n",
"- Our approaches to systems design are building powerful machines that\n",
" will be deployed in evolving environments.\n",
"\n",
"[^1]: the challenge of understanding what information pertains to is\n",
" known as knowledge representation."
]
}
],
"metadata": {},
"nbformat": 4,
"nbformat_minor": 2
}