{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Low Emission Building Control with Probabilistic Deep Reinforcement Learning\n",
    "\n",
    "## [Scott Jeen](https://enjeeneer.io/)\n",
    "\n",
    "### Refficiency Lunch, 2nd November 2021"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 1. Motivation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## 1.1 Cool (and the gang)\n",
    "<img class=\"\" src=\"images/global_sankey_1a.png\" style=\"width:75%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## 1.2 Idealised Building Control\n",
    "\n",
    "<img class=\"\" src=\"images/intermittency1.png\" style=\"width:50%\">\n",
    "<img class=\"\" src=\"images/intermittency2.png\" style=\"width:50%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 2. Current Approaches to RL for Building Control"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## 2.1 Model-free RL\n",
    "\n",
    "Rollouts             |  Network\n",
    ":-------------------------:|:-------------------------:\n",
    "|\n",
    "![](images/model_free.gif)  |  ![](images/model_free_network1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## 2.2 Model-based RL\n",
    "Rollouts             |  Network\n",
    ":-------------------------:|:-------------------------:\n",
    "|\n",
    "![](images/model_based.gif)  |  ![](images/model_based_network.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## 2.3 Qualitative Comparison\n",
    "\n",
    "|                | **Pros** 💚             | **Cons** 💔                         |\n",
    "|----------------|------------------|------------------------------|\n",
    "|**Model-free**  | Performance | Data inefficiency    |\n",
    "|**Model-based** | Data efficiency       | Performance; Model bias      |\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## 2.4 Quantitative Comparison\n",
    "|                | **SOA Energy Efficiency**             | **SOA Data Efficiency**         | **Authors** |\n",
    "|----------------|------------------|------------------------------|----------------------|\n",
    "|**Model-free**  | **10-30%**  |  1-2 years *simulated* data   | [1,2] |\n",
    "|**Model-based** | 5-10%       | **3-12 hours** *live* data      | [3,4] |\n",
    "\n",
    "\n",
    "- To scale, agents must learn quickly *online*. Can we get model-free performance with model-based agents?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 3. Probabilistic, Model-based Reinforcement Learning\n",
    "<img class=\"\" src=\"images/prob_network.png\" style=\"width:60%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## 3.1 Mitigating Model Bias\n",
    "<img class=\"\" src=\"images/model_bias_1.png\" style=\"width:80%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## 3.1 Mitigating Model Bias\n",
    "<img class=\"\" src=\"images/model_bias_2.png\" style=\"width:80%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## 3.1 Mitigating Model Bias\n",
    "<img class=\"\" src=\"images/model_bias_3a.png\" style=\"width:80%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## 3.1 Mitigating Model Bias\n",
    "<img class=\"\" src=\"images/model_bias_4a.png\" style=\"width:80%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## 3.1 Mitigating Model Bias\n",
    "<img class=\"\" src=\"images/model_bias_5a.png\" style=\"width:80%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## 3.2 Probabilistic Trajectory Sampling\n",
    "<img class=\"\" src=\"images/prob_rl.gif\" style=\"width:50%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 4. Experiments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## 4.1 Building Simulations [5]\n",
    "<span style=\"color:white\">some *blue* text</span>             |  <span style=\"color:white\">some *blue* text</span> \n",
    ":-------------------------:|:-------------------------:\n",
    "|\n",
    "<img class=\"\" src=\"images/mixed-use.png\" style=\"width:70%\">  |  <img class=\"\" src=\"images/apartment.png\" style=\"width:70%\">\n",
    "|\n",
    "<img class=\"\" src=\"images/offices.png\" style=\"width:70%\">  |  <img class=\"\" src=\"images/seminar_centre.png\" style=\"width:70%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## 4.2 Annual Carbon Emissions\n",
    "<img class=\"\" src=\"images/cum_emissions.png\" style=\"width:50%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## 4.3 Temperature Control\n",
    "\n",
    "<img class=\"\" src=\"images/mixed_temp.png\" style=\"width:80%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## 4.4 Load Shifting\n",
    "<img class=\"\" src=\"images/pets_grid.png\" style=\"width:80%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## 4.4 Load Shifting\n",
    "<img class=\"\" src=\"images/ppo_grid.png\" style=\"width:80%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## 4.4 Load Shifting\n",
    "<img class=\"\" src=\"images/mpc_grid.png\" style=\"width:80%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## 4.5 Mixed-Use Results\n",
    "\n",
    "|                           | Annual Energy Consumption (kWh) ⚡ | Average Temperature (oC) 🌡️ | Emissions (tCO2)   |\n",
    "|---------------------------|--------------------------------|--------------------------|--------------------------|\n",
    "| Current                   | 132, 776                       | 20.11                    |       65.15                |\n",
    "| Model-free                | 103, 363                       | 21.99                    | 50.91                  |\n",
    "| Model-based               | 137, 616                      | 20.59                    | 67.61                   |\n",
    "| Probabilistic Model-based | **97, 149**                             | 20.41                        | **47.73**      |                  |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 5. Future Work\n",
    "\n",
    "- In-situ verification\n",
    "- What is optimal?\n",
    "- What is the necessary feature space?\n",
    "- Can we share learning between buildings?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Thanks!"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}