{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# How to see what your agent did ?\n", "Try this notebook out interactively with: [![Binder](./img/badge_logo.svg)](https://mybinder.org/v2/gh/rte-france/Grid2Op/master)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "Execute the cell below by removing the # character if you use google colab !\n", "\n", "Cell will look like:\n", "```python\n", "!pip install grid2op[optional] # for use with google colab (grid2Op is not installed by default)\n", "```\n", "" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# !pip install grid2op[optional] # for use with google colab (grid2Op is not installed by default)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## I - General plotting utilities\n", "\n", "With the module \"grid2op.PlotGrid\" (more information on the official documentation [here](https://grid2op.readthedocs.io/en/latest/plotgrid.html)) it is possible to gather visual insights (plots) about the state of the powergrid.\n", "\n", "This module currently involves 2 \"base\" classes:\n", "- `PlotMatplot` which uses the well-known matplotlib python library to render the plot. Matplotlib being the most used plotting library in python. Therefore, we decided to add its support in grid2op.\n", "- `PlotPlotly` that uses the plotply library. Plotly is particularly suited for studying more deeply what happens at some particular time step.\n", "However, we do not recommend to use any of these. \n", "\n", "Instead, we developed two higher-level classes:\n", "- `EpisodeReplay` which uses the \"PlotMatplotlib\" class internally and is used to render a video as a gif or a mp4 file. This is mainly designed to communicate the results of your agent.\n", "- `env.render` which is similar to the gym method `env.render` that was mostly used in the open ai gym framework, hence users of this framework will be familiar with this.\n", "\n", "Last but not least, we developed a package called `grid2viz` to help you to inspect the behaviour of your agent in-depth. This package is more advanced than all the other methods presented above and therefore, we highly recommend you to use this package to get the best out of your agents!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import matplotlib\n", "import shutil\n", "%matplotlib inline\n", "import matplotlib.pyplot as plt # pip install matplotlib\n", "import seaborn as sns # pip install seaborn\n", "import plotly.graph_objects as go # pip install plotly\n", "import imageio # pip install imageio\n", "\n", "max_iter = 5 # to save time we only assess performance on 30 iterations" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This notebook will not work if one of the 4 packages above cannot be imported. We highly recommend that you install them on your machine. We provide you with the pip commands so that you can install them easily." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import grid2op\n", "env = grid2op.make(test=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## II - Plot.PlotGrid utility\n", "\n", "The \"Plot.Plotting\" module can help render a powergrid using 2 different methods: matplotlib or plotly. The display method is defined when you create a \"plotting\" object as shown below.\n", "\n", "All functions introduced in the following are available for plotly and matplotlib. Feel free to switch from one to the other in order to see the differences. The following will provide examples on how to plot static information, and single obvserations.\n", "\n", "\n", "### II A) Plot Static informations\n", "The following cell will plot the names of each object on the powergrid, as well as their id." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from grid2op.PlotGrid import PlotMatplot\n", "plot_helper = PlotMatplot(env.observation_space)\n", "line_ids = [int(i) for i in range(env.n_line)]\n", "fig_layout = plot_helper.plot_layout()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It is also possible to display some \"external\" information on this layout. For example, you can plot the thermal limit of each powerline:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig_info = plot_helper.plot_info(line_values=env.get_thermal_limit())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The argument \"line_values\" tells the information on the powerlines that you wish to display. Here, the thermal limits of the powerlines are displayed.\n", "\n", "It is also possible to display information abouts loads and generators in the same way." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### II - B) Plot a few observations\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "obs = env.reset()\n", "fig_obs = plot_helper.plot_obs(obs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here you can see that the powerlines are colored with respect to their flow (in % of the thermal limit). Blue lines are the powerlines where the flow is rather low, while orange lines have higher flows. Loads consumptions and generator powers are provided as positive or negative injections given in MW (+ for generators, - for loads).\n", "\n", "All of the above information can be modified as showed in the following cell, where we plot the active power flow for the powerlines and the load voltage magnitude for the loads. Note that the units will be different for the loads since we now display voltages instead of consumptions." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig_obs2 = plot_helper.plot_obs(obs, line_info=\"p\", load_info=\"v\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, the topology at each substation can also be plotted. \n", "\n", "For example, let's consider a topological action at substation 1. There, we will move the load there (the load 'load_1_0') and the powerlines '0_1_0' and '1_3_3' to bus number 2. You can see the names of the loads, generators, and powerlines on the first plot in this notebook. This can be done easily, as shown in the following cell (see notebook [03_Action](03_Action.ipynb) for more information)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "action = env.action_space({\"set_bus\": {\"loads_id\": [(0,2)], \"lines_or_id\": [(3,2)], \"lines_ex_id\": [(0,2)]}})\n", "print(action)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `print` utility helps us check that the action we implemented was indeed the one that we wanted to implement. `print` is also a useful method to see what happened sometimes." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's perform that action and plot the new observation :" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "new_obs, reward, done, info = env.step(action)\n", "fig_obs3 = plot_helper.plot_obs(new_obs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "On the previous observation, all the objects (loads, generators, and powerlines) were connected to bus 1, shown in red.\n", "\n", "Here, the 3 objects that we wanted to connect to bus 2 (the load 'load_1_0', the extremity of '0_1_0', and the origin of '1_3_3') are indeed connected to the second bus, which is shown in green.\n", "\n", "This plotting utility is a very useful tool to detect what happened, especially just before a game over." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## III - env.render\n", "\n", "Another way to inspect what is going on during the episode is to use the renderer. The renderer can be used in the same way as in any open ai gym environment. \n", "\n", "In the next cell, we will reset the environment created above, create an agent that takes random actions every 10 timesteps, and subsequently see how it goes.\n", "\n", "**NB** It is not recommended to train your agent with any renderer turned on. The preferred way is to train your agent without any renderer first, and then to evaluate your agent on a fixed set of observations while possibly turning the renderer on (more information about this in the next section)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from grid2op.Agent import RandomAgent\n", "from IPython import display # for proper display in jupyter notebook\n", "\n", "class CustomRandom(RandomAgent):\n", " def __init__(self, action_space):\n", " RandomAgent.__init__(self, action_space)\n", " self.i = 1\n", "\n", " def my_act(self, transformed_observation, reward, done=False):\n", " if (self.i % 10) != 0:\n", " res = 0\n", " else:\n", " res = self.action_space.sample()\n", " self.i += 1\n", " return res\n", " \n", "myagent = CustomRandom(env.action_space)\n", "obs = env.reset()\n", "reward = env.reward_range[0]\n", "done = False\n", "nb_step = 0\n", "while not done:\n", " fig = env.render()\n", " display.clear_output(wait=True)\n", " display.display(plt.gcf())\n", " act = myagent.act(obs, reward, done)\n", " obs, reward, done, info = env.step(act)\n", " nb_step += 1\n", " if nb_step >= max_iter:\n", " break" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## IV - ReplayEpisode\n", "\n", "This tool allows you to save a gif figure of the rendering of your agent. We recommend, as stated above, to get rid of any renderer in the training phase, and then to use the runner to assess the performance of your agent. Then, once the results are saved by the runner, you can use the `ReplayEpisode` class or study it more in-depth with grid2viz (see next section). \n", "\n", "But first things first, let's mimic what we think is a good process. Suppose you are happy with the result of your agent (for the sake of simplicity we will not train any agent here, we will use an untrained instance of our CustomRandom agent). What are the next steps?\n", "\n", "First, you need to create an environment in which it will be evaluated, and use a runner for the evaluation:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from grid2op.Runner import Runner\n", "env = grid2op.make(test=True)\n", "my_awesome_agent = CustomRandom(env.action_space)\n", "runner = Runner(**env.get_params_for_runner(), agentClass=None, agentInstance=my_awesome_agent)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Second, you must start the runner and save the result in a directory (here we limit the runner to perform only 30 iterations so that this notebook runs quickly):" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "path_agents = \"path_agents\" # this is mandatory for grid2viz to have a directory with only agents\n", "# that is why we have it here. It is aboslutely not mandatory for this more simple class.\n", "if not os.path.exists(path_agents):\n", " os.mkdir(path_agents)\n", "path_awesome_agent_log = os.path.join(path_agents, \"awesome_agent_logs\")\n", "shutil.rmtree(path_awesome_agent_log, ignore_errors=True)\n", "res = runner.run(nb_episode=2,\n", " path_save=path_awesome_agent_log,\n", " max_iter=max_iter,\n", " agent_seeds=[0, 1],\n", " env_seeds=[2, 3])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Third, you must use the results of the runner and save the results as gif for example (you can also visualize it offline on the screen if you prefer : in order to do that, you simply need to switch the \"display\" argument to ``True``)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from grid2op.Episode import EpisodeReplay\n", "\n", "gif_name = \"episode\"\n", "ep_replay = EpisodeReplay(agent_path=path_awesome_agent_log)\n", "for _, chron_name, cum_reward, nb_time_step, max_ts in res:\n", " ep_replay.replay_episode(chron_name, # which chronic was started\n", " gif_name=gif_name, # Name of the gif file\n", " display=False, # dont wait before rendering each frames\n", " fps=3.0) # limit to 3 frames per second" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And you can even see the gif in the jupyter notebook afterwards\n", "\n", "This only works if:\n", "- you haven't changed path_awesome_agent_log\n", "- you haven't changed the name of gif_name\n", "- all the above cells have be run properly\n", "\n", "![img](./path_agents/awesome_agent_logs/000/episode.gif)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## V - Grid2Viz\n", "\n", "This tool is really useful to dive deep into the analysis of your agent. We highly recommend that you use it to develop stronger agents and score higher in the competition." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Grid2viz is a package that we developed to help you visualize the behavior of your agent. \n", "\n", "It is available in the Github repository [grid2viz](https://github.com/mjothy/grid2viz). In the following cells, we will demonstrate how to use Grid2viz to inspect more deeply the log of our agent that was generated by the runner in the second cell of this notebook.\n", "\n", "In this section, we first run some other agents to show the full potential of grid2viz (optional). Then, we explain that the folder tree must respect a certain architecture for Grid2Viz to be used. Finally, we show how to install it and how to use it on the data generated in this notebook.\n", "\n", "![](https://raw.githubusercontent.com/mjothy/grid2viz/master/grid2viz/assets/screenshots/scenario_overview.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### V - 1) More agents to compare\n", "\n", "This section is not mandatory, but it is recommended to read it to discover the full capabilities of Grid2viz. We will first run 2 others agents: the `DoNothing` agent, and the `TopologyGreedy` agent." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# make a runner for this agent\n", "from grid2op.Agent import DoNothingAgent, TopologyGreedy\n", "import shutil\n", "\n", "for agentClass, agentName in zip([DoNothingAgent], # , TopologyGreedy\n", " [\"DoNothingAgent\"]): # , \"TopologyGreedy\"\n", " path_this_agent = os.path.join(path_agents, agentName)\n", " shutil.rmtree(os.path.abspath(path_this_agent), ignore_errors=True)\n", " runner = Runner(**env.get_params_for_runner(),\n", " agentClass=agentClass\n", " )\n", " res = runner.run(path_save=path_this_agent, nb_episode=2, \n", " max_iter=max_iter)\n", " print(\"The results for the {} agent are:\".format(agentName))\n", " for _, chron_id, cum_reward, nb_time_step, max_ts in res:\n", " msg_tmp = \"\\tFor chronics with id {}\\n\".format(chron_id)\n", " msg_tmp += \"\\t\\t - cumulative reward: {:.6f}\\n\".format(cum_reward)\n", " msg_tmp += \"\\t\\t - number of time steps completed: {:.0f} / {:.0f}\".format(nb_time_step, max_ts)\n", " print(msg_tmp)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### V - 2) Installation\n", "\n", "Grid2Viz is not yet on pypi, but it is maintained as a Github repository, so you need a specific command to install it. It can be done easily by running the cell bellow (more information can be found on the grid2iz github)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import sys\n", "print(\"To install it, either uncomment the cell bellow, or type, in a command prompt:\\n{}\".format(\n", " (\"\\t{} -m pip install -m pip install -U git+https://github.com/mjothy/grid2viz --user\".format(sys.executable))))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# !$sys.executable -m pip install -U git+https://github.com/mjothy/grid2viz --user" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### V - 3) Usage\n", "\n", "Once the above package is installed, you can start studying what your agent did (**NB** the agent must have been run with a runner and the \"path_save\" argument must have been provided in order for grid2viz to work properly.\n", "\n", "For performance optimization, grid2viz uses a cache. This notebook being only an example, it is recommended to clear the cache before starting the grid2viz app. Of course, if you decide to study different generations of your agent, it is NOT recommended to clear the cache before any study." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "shutil.rmtree(os.path.join(os.path.abspath(path_agents), \"_cache\"), ignore_errors=True)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(\"To run grid2viz type, in a command prompt:\\n{}\".format(\n", " (\"\\t{} -m grid2viz.main --agents_path {}\".format(sys.executable, os.path.abspath(path_agents)))))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "# If you want to start grid2viz for better plotting, you may try the following :\n", "# (remove the \"if False:\" or replace it with \"if True:\")\n", "if False:\n", " !$sys.executable -m grid2viz.main --agents_path $path_agents\n", "else:\n", " print(\"You need to copy paste the command in the cell above to run grid2viz\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" } }, "nbformat": 4, "nbformat_minor": 2 }