{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# This Notebook will develop how to build an Agent and assess its performance.\n", "Try me out interactively with: [![Binder](./img/badge_logo.svg)](https://mybinder.org/v2/gh/rte-france/Grid2Op/master)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Objective**\n", "\n", "This notebook covers the basics of the information that can be retrieved about the state of the powergrid. The basics are illustrated on examples of \"expert agents\" that can take actions based on some fixed rules. More generic types of *Agents*, relying for example on machine learning / deep learning will be covered in the notebook [04_TrainingAnAgent](04_TrainingAnAgent.ipynb).\n", "\n", "This notebook will also cover the description of the *Observation* class, which is useful to take some actions." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "Execute the cell below by removing the # character if you use google colab !\n", "\n", "Cell will look like:\n", "```python\n", "!pip install grid2op[optional] # for use with google colab (grid2Op is not installed by default)\n", "```\n", "" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# !pip install grid2op[optional] # for use with google colab (grid2Op is not installed by default)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "import sys\n", "import numpy as np\n", "import grid2op" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "res = None\n", "try:\n", " from jyquickhelper import add_notebook_menu\n", " res = add_notebook_menu()\n", "except ModuleNotFoundError:\n", " print(\"Impossible to automatically add a menu / table of content to this notebook.\\nYou can download \\\"jyquickhelper\\\" package with: \\n\\\"pip install jyquickhelper\\\"\")\n", "res" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## I) Description of the observations" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**NB** In this paragraph we will cover the basics of the observation class. Please visit the official documentation for more detailed information, or [here](https://grid2op.readthedocs.io/en/latest/observation.html) or in the [Observations.py](grid2op/Observation/Observation.py) files for more information. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### I.A) Obtaining an observation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "An observation can be accessed by calling `env.step()`. The next cell is dedicated to creating an environment and obtaining one instance of an observation. For illustration purposed, we use the default `rte_case14_realistic` environment from Grid2Op framework." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "env = grid2op.make(test=True)\n", "obs = env.reset() " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`obs` now contains the initial state of the grid." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### I.B) Information contained in an Observation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`Grid2Op` allows to model different kinds of observations. For example, some observations could have incomplete data, or noisy data, etc. As follows we will detail only the \"CompleteObservation\". `CompleteObservation` gives the full state of the powergrid, without any noise. It's the default type of observation used." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### a) Some attributes of complete observation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "An observation has calendar data (eg the time stamp of the observation):" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "obs.year, obs.month, obs.day, obs.hour_of_day, obs.minute_of_hour, obs.day_of_week" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "An observation has some powergrid generic information (that are static: the same environment has always these attributes, that have always th same values, but of course different environments can have different values)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(\"Number of generators of the powergrid: {}\".format(obs.n_gen))\n", "print(\"Number of loads of the powergrid: {}\".format(obs.n_load))\n", "print(\"Number of powerline of the powergrid: {}\".format(obs.n_line))\n", "print(\"Number of elements connected to each substations in the powergrid: {}\".format(obs.sub_info))\n", "print(\"Total number of elements: {}\".format(obs.dim_topo))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "An observation has some information about the generators (each generator can be viewed as a point in a 3-dimensional space)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(\"Generators active production: {}\".format(obs.gen_p))\n", "print(\"Generators reactive production: {}\".format(obs.gen_q))\n", "print(\"Generators voltage setpoint : {}\".format(obs.gen_v))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "An observation has some information about the loads (each load is a point in a 3-dimensional space, too)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(\"Loads active consumption: {}\".format(obs.load_p))\n", "print(\"Loads reactive consumption: {}\".format(obs.load_q))\n", "print(\"Loads voltage (voltage magnitude of the bus to which it is connected) : {}\".format(obs.load_v))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this setting, a powerline can be viewed as a point in an 8-dimensional space:\n", " * active flow\n", " * reactive flow\n", " * voltage magnitude\n", " * current flow\n", " \n", "for both its origin and its extremity.\n", "\n", "For example, suppose the powerline `line1` is connecting two node `A` and `B`. There are two separate values for the active flow on `line1` : the active flow from `A` to `B` (origin) and the active flow from `B` to `A` (extremity).\n", "\n", "These powerline features can be accessed with :" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(\"Origin active flow: {}\".format(obs.p_or))\n", "print(\"Origin reactive flow: {}\".format(obs.q_or))\n", "print(\"Origin current flow: {}\".format(obs.a_or))\n", "print(\"Origin voltage (voltage magnitude to the bus to which the origin end is connected): {}\".format(obs.v_or))\n", "print(\"Extremity active flow: {}\".format(obs.p_ex))\n", "print(\"Extremity reactive flow: {}\".format(obs.q_ex))\n", "print(\"Extremity current flow: {}\".format(obs.a_ex))\n", "print(\"Extremity voltage (voltage magnitude to the bus to which the origin end is connected): {}\".format(obs.v_ex))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Another powerline feature is the $\\rho$ ratio, the ratio between the current flow in the powerline and its thermal limit, *ie.* for each powerline. This feature $\\rho$ can be accessed with:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "obs.rho" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The observation (*obs*) also stores information on the topology and the state of the powerline." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "obs.timestep_overflow # the number of timestep each of the powerline is in overflow (1 powerline per component)\n", "obs.line_status # the status of each powerline: True connected, False disconnected\n", "obs.topo_vect # the topology vector the each element (generator, load, each end of a powerline) to which the object\n", "# is connected: 1 = bus 1, 2 = bus 2." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In `grid2op`, all objects (end of a powerline, load or generator) can be either disconnected, connected to the first bus of its substation, or connected to the second bus of its substation.\n", "\n", "`topo_vect` is the vector containing the connection information, it is part of the observation.\n", "If an object is disconnected, then its corresponding component in `topo_vect` will be `-1`. If it's connected to the first bus of its substation, its component will be `1` and if it's connected to the second bus, its component will be `2`.\n", "\n", "For example, if you want to know at which bus the \"load 2\" is connected, you can do:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "load_id = 2\n", "obs.topo_vect[obs.load_pos_topo_vect[load_id]]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Or alternatively" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "obs.load_bus[load_id]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**NB** of course the `obs.gen_bus`, `obs.line_or_bus`, `obs.line_ex_bus` or `obs.storage_bus` are also defined." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "More information about this topology vector is given in the documentation [here](https://grid2op.readthedocs.io/en/latest/observation.html). More information about this topology vector will be given in the notebook dedicated to vizualisation. \n", "\n", "#### b) Representation as \"a graph\"\n", "\n", "The powergrid can be represented as a graph. The \"topology vector\" is an efficient way to store and use the graph in terms of memory and speed. Some utilities functions in grid2op allows to build these graphs rapidly. \n", "\n", "We say \"these graphs\" because \"the\" graph of the grid can mean different things. Yes, you have a lot of information with something like:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from grid2op.PlotGrid import PlotMatplot\n", "plot_helper = PlotMatplot(env.observation_space)\n", "_ = plot_helper.plot_obs(obs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### i) connectivity matrix\n", "\n", "One way to represent the `graph` of the power system is the `connectivity matrix`. The `connectivity matrix` has as many rows / columns as the number of elements in the powergrid (remember that an element is either an end of a powerline, or a generator or a load) and that tells if 2 elements are connected to one another or not:\n", "\n", "$$\n", "\\left\\{\n", "\\begin{aligned}\n", "\\text{conn mat}[i,j] = 0 & ~\\text{element i and j are NOT connected to the same bus}\\\\\n", "\\text{conn mat}[i,j] = 1 & ~\\text{element i and j are connected to the same bus, or i and j are both ends of the same powerline}\\\\\n", "\\end{aligned}\n", "\\right.\n", "$$\n", "\n", "**NB** If two objects are not connected at the same substation, they are necessarily not connected together, except if these elements are the two side of the same powerline." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# if \"as_csr_matrix\" is set to True, then the result will be given as a csr scipy sparse matrix\n", "mat = obs.connectivity_matrix(as_csr_matrix=False)\n", "mat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And if you want to know to which object a given element is connected you can do the following (we take the exemple of trying to recover to which element load 5 is connected)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "load_id = 5\n", "which_connected = mat[obs.load_pos_topo_vect[load_id]] == 1\n", "obs.grid_objects_types[which_connected,]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This will give you a matrix with as many rows as the number of elements that are connected to the element (in this case 5) with:\n", "- the first column indicate the substation. All the objects connected to a given element necessarily belong to the same substation, in our example substation 8.\n", "- the second column encodes for \"load\". If there is a \"-1\" it means the element is NOT a load, otherwise it gives the load. Here, looking at the last row, when can know that currently load 5 is... connected to load 5. It will always be the case, except if the element is disconnected of course.\n", "- the third column encodes for \"generator\". In this case, all this column is \"-1\" it means the load 5 is not currently directly connected to a generator.\n", "- fourth column encodes for \"origin side of powerline\". Here we see that our load 5 is connected to powerline (origin side) with ids 10, 11 and 19\n", "- fifth column encodes for \"extremity side of powerline\". We can see that our load 5 is connecte to powerline (extremity side) 16\n", "- finally last (sixth) column encodes for storage units. This load 5 is not directly connected to any storage unit here. \n", "\n", "You can check that everything is consistent with the plot above." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Advantages**: \n", "\n", " - has always the same dimension, regardless of the topology of the powergrid\n", " - binary matrix\n", " - all elements are represented and easily accessible\n", " - symmetric matrix\n", " \n", "**Drawbacks**:\n", "\n", " - do not contain any information regarding the flows, generations, loads, storage units etc.\n", " - large matrix\n", " \n", "#### ii) bus connectivity matrix\n", "\n", "Another way to represent the \"graph\" of the powergrid is the \"bus-bus\" matrix, that says if at least one powerline connect two bus together or not. \n", "\n", "In grid2op this is called \"bus_connectivity_matrix\". \n", "\n", "This `bus connectivity matrix` has as many rows / columns as the number of active buses of the powergrid. It should be understood as follows:\n", "\n", "$$\n", "\\left\\{\n", "\\begin{aligned}\n", "\\text{bus conn mat}[i,j] = 0 & ~\\text{if no powerline connects bus i to bus j}\\\\\n", "\\text{bus conn mat}[i,j] = 1 & ~\\text{if at least one powerline connects bus i to bus j (or i == j)}\\\\\n", "\\end{aligned}\n", "\\right.\n", "$$" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "obs.bus_connectivity_matrix(as_csr_matrix=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Advantages**: \n", "\n", "- \"small\" matrix\n", "- binary matrix\n", "- symmetric matrix\n", " \n", "**Drawbacks**:\n", "\n", "- do not contain any information regarding the flows, generations, loads, storage units etc.\n", "- it's not easy to know which powerline connects which buses\n", "- it's not easy to know which element is connected to which buses (though the `obs.bus_connectivity_matrix(return_lines_index=True)` might help in this case)\n", "- its dimension changes: you can have 2, or 1 bus at any given substation.\n", " \n", "#### iii) \"flow bus\" matrix\n", "Finally, the third way to represent the graph is to introduce informations about flows (*eg* label) on your graph.\n", "\n", "This \"flow bus matrix\" has the following properties:\n", "\n", "- like the above \"bus connectivity matrix\" it has as many rows / columns as the number of different bus on a power\n", "- the diagonal coefficients are the power injected at this bus, which is defined at the sum of the generators connected at this bus minus the sum of the loads connected at this bus\n", "- the non diagonal coefficient `[i, j]` (i different from j then) is the amount of power flowing from bus i to bus j.\n", "\n", "In grid2op, you can do the following:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "mat, (load_bus, gen_bus, stor_bus, lor_bus, lex_bus) = obs.flow_bus_matrix(active_flow=True, as_csr_matrix=False)\n", "mat, (load_bus, gen_bus, stor_bus, lor_bus, lex_bus) = obs.flow_bus_matrix(active_flow=True, as_csr_matrix=False)\n", "mat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's dive a bit on the arguments there:\n", "\n", "- `mat` is the flow bus matrix\n", "- `load_bus` indicates for each load, to which bus it is connected\n", "- `gen_bus` indicates for each generator, to which bus it is connected\n", "- `stor_bus` indicates for each load, to which bus it is connected\n", "- `lor_bus` indicates for each line (origin side), to which bus it is connected\n", "- `lex_bus` indicates for each line (extremity side), to which bus it is connected\n", "\n", "Let's take the same example as above with the load id 5." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "load_id = 5\n", "this_load_bus = load_bus[load_id]\n", "lines_or_id = np.where(lor_bus == this_load_bus)[0]\n", "lines_ex_id = np.where(lex_bus == this_load_bus)[0]\n", "line_id = lines_or_id[0]\n", "print(f\"The load {load_id} absorbs: {obs.load_p[load_id]:.2f}MW\")\n", "print(f\"It is connected to the bus id {this_load_bus} of the matrix\")\n", "print(f\"And we can see the diagonal coefficient of the \\\"flow bus matrix\\\" \"\\\n", " f\"at this bus is: {mat[this_load_bus,this_load_bus]:.2f}\")\n", "print(f\"Also we can see that powerlines {lines_or_id} have their origin side connected at this bus.\")\n", "print(f\"And powerlines {lines_ex_id} have their extremity side connected at this bus.\")\n", "print(f\"And, for example, if we look at powerline {line_id}, that connects bus {lor_bus[line_id]} \"\\\n", " f\"to bus {lex_bus[line_id]} we can see that:\")\n", "print(f\"The flow from bus {lor_bus[line_id]} to {lex_bus[line_id]} is \"\\\n", " f\"{mat[lor_bus[line_id], lex_bus[line_id]]:.2f}MW\")\n", "print(f\"For information, the flow at the origin side of this line {line_id} is {obs.p_or[line_id]:.2f}MW. \"\\\n", " f\"And this is not a coincidence.\")\n", "print()\n", "print(f\"Also, we want to emphasize that this matrix is NOT symmetrical, for example:\")\n", "print(f\"\\t the power from bus {lor_bus[line_id]} to {lex_bus[line_id]} is \"\\\n", " f\"{mat[lor_bus[line_id], lex_bus[line_id]]:.5f} MW\")\n", "print(f\"\\t the power from bus {lex_bus[line_id]} to {lor_bus[line_id]} is \"\\\n", " f\"{mat[lex_bus[line_id], lor_bus[line_id]]:.5f} MW\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So you can see that the power from a bus to another has a sign (if it's positive from 8 to 9 it means that the flow is going from 9 to 8: power is injected at bus 8 from bus 9)\n", "\n", "And also, the two values (power injected at one side, from another) do not sum at 0. This is because there are losses on the grid, mainly due to Joule's effect." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Advantages**: \n", "\n", "- \"small\" matrix\n", "- contain lots of information regarding the flows, generators, loads, storage units etc.\n", "\n", "**Drawbacks**:\n", "\n", "- it's not easy to know which powerline connects which buses, even though the \"lor_bus\", \"lex_bus\", \"stor_bus\" etc. can be used to retrieve it [bus in any case it will be agregated]\n", "- it's not easy to know which elements are connected to which others\n", "- its dimension changes: you can have 2, or 1 bus at any given substation.\n", "- real number matrix\n", "- non symmetric matrix\n", "\n", "**NB** The power in alternative current contains two dimension called \"active\" and \"reactive\". We detailed here the example of the \"active flow\" matrix that is accessed with `obs.flow_bus_matrix(active_flow=True)` (active flows uses `obs.p_or`, `obs.p_ex`, `obs.load_p`, `obs.gen_p` and `obs.storage_power`). You can retrieve the \"reactive flow\" matrix with `obs.flow_bus_matrix(active_flow=False)` (reactive flows uses `obs.q_or`, `obs.q_ex`, `obs.load_q`, `obs.gen_q` and nothing for the storage reactive power)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### iv) networkx graph\n", "\n", "Lastly, the grid2op framework also offers the possibility to \"convert\" a given observation as a graph (representing as a networkx graph).\n", "\n", "This graphs has the following properties:\n", "\n", "- it counts as many nodes as the number of buses of the grid\n", "- it counts less edges than the number of lines of the grid (two lines connecting the same buses are \"merged\"\n", " into one single edges)\n", "- nodes have attributes:\n", "\n", " - \"p\": the active power produced at this node (negative means the sum of power produce minus power absorbed\n", " is negative)\n", " - \"q\": the reactive power produced at this node\n", " - \"v\": the voltage magnitude at this node\n", " - \"cooldown\": how much longer you need to wait before being able to merge / split or change this node\n", "\n", "- edges have attributes too:\n", "\n", " - \"rho\": the relative flow on this powerline\n", " - \"cooldown\": the number of step you need to wait before being able to act on this powerline\n", " - \"status\": whether this powerline is connected or not\n", " - \"thermal_limit\": maximum flow allowed on the the powerline (this is the \"a_or\" flow)\n", " - \"timestep_overflow\": number of time steps during which the powerline is on overflow\n", " - \"p_or\": active power injected at this node at the \"origin side\".\n", " - \"p_ex\": active power injected at this node at the \"extremity side\".\n", " - \"q_or\": reactive power injected at this node at the \"origin side\".\n", " - \"q_ex\": reactive power injected at this node at the \"extremity side\".\n", " - \"a_or\": current flow injected at this node at the \"origin side\".\n", " - \"a_ex\": current flow injected at this node at the \"extremity side\".\n", "\n", "**IMPORTANT NOTE** the \"origin\" and \"extremity\" of the networkx graph is not necessarily the same as the one\n", "in grid2op. The \"origin\" side will always be the nodes with the lowest id. For example, if an edges connects\n", "the bus 6 to the bus 8, then the \"origin\" of this powerline is bus 6 (**eg** p_or of this edge is the power\n", "injected at bus 6) and the \"extremity\" side is bus 8.\n", "\n", "\n", "An example is given in:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "import networkx\n", "graph = obs.as_networkx()\n", "graph" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And you can use any networkx methods you want with this graph, for example, you can plot it" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "networkx.draw_networkx(graph,\n", " with_labels=False,\n", " # I want to plot the \"rho\" value for edges\n", " edge_color=[graph.edges[el][\"rho\"] for el in graph.edges], \n", " # i use the position computed with grid2op\n", " # NB: this code only works if the number of bus per substation is 1 !\n", " pos=[plot_helper._grid_layout[sub_nm] for sub_nm in obs.name_sub]\n", " )" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "graph = obs.as_networkx()\n", "for node_id in graph.nodes:\n", " # check the or / ex convention is ok (kirchoff's law)\n", " p_ = graph.nodes[node_id][\"p\"]\n", " q_ = graph.nodes[node_id][\"q\"]\n", " \n", " # get the edges\n", " edges = graph.edges(node_id)\n", " p_line = 0 # all active power injected at this nodes on all powerlines\n", " q_line = 0 # all reactive power injected at this nodes on all powerlines\n", " for (k1, k2) in edges:\n", " # now retrieve the active / reactive power injected at this node (looking at either *_or or *_ex\n", " # depending on the direction of the powerline: remember that the \"origin\" is always the lowest\n", " # bus id.\n", " if k1 < k2:\n", " # the current inspected node is the lowest, so on the \"origin\" side\n", " p_line += graph.edges[(k1, k2)][\"p_or\"]\n", " q_line += graph.edges[(k1, k2)][\"q_or\"]\n", " else:\n", " # the current node is the largest, so on the \"extremity\" side\n", " p_line += graph.edges[(k1, k2)][\"p_ex\"]\n", " q_line += graph.edges[(k1, k2)][\"q_ex\"]\n", " assert abs(p_line - p_) <= 1e-5\n", " assert abs(q_line - q_) <= 1e-5" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### c) Some other handy methods" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The observation can be converted to / from a flat numpy array. This conversion is useful for interacting with machine learning libraries or to store it, but it probably makes it less readable for a human. The function proceeds by stacking all the features mentionned above in a single `numpy.float64` vector." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "vector_representation_of_observation = obs.to_vect()\n", "vector_representation_of_observation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "An observation can be copied, of course:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "obs2 = obs.copy()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Or reset:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "obs2.reset()\n", "print(obs2.gen_p)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Or loaded from a vector:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "obs2.from_vect(vector_representation_of_observation)\n", "obs2.gen_p" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It is also possible to assess whether two observations are equal or not:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "obs == obs2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### d) Simulate" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As opposed to most reinforcement learning (RL) problems, in this framework we add the possibility to \"simulate\" the impact of a possible action on the power grid. This helps calculating roll-outs in the RL setting, and can be close to \"model-based\" RL approaches (except that nothing more has to be learned).\n", "\n", "This \"simulate\" function uses the available forecast data (forecasts are made available by the same way we loaded the data here, with the class `GridStateFromFileWithForecasts`. For this class, only forecasts for 1 time step are provided, but this might be adapted in the future).\n", "\n", "Note that this `simulate` function can use a different simulator than the one used by the Environment. Fore more information, we encourage you to read the official documentation, or if it has been built locally (recommended), to consult [this page](https://grid2op.readthedocs.io/en/latest/observation.html#grid2op.Observation.Observation.simulate).\n", "\n", "This function will:\n", "\n", "1. apply the forecasted injection on the powergrid\n", "2. run a powerflow with the decidated `simulate` powerflow simulator\n", "3. return:\n", " 1. the anticipated observation (after the action has been taken)\n", " 2. the anticipated reward (of this simulated action)\n", " 3. whether or not there has been an error\n", " 4. some more informations\n", " \n", "From a user point of view, this is the main difference with the previous [pypownet](https://github.com/MarvinLer/pypownet) framework. In pypownet, this \"simulation\" used to be performed directly by the environment, thus giving direct access of the environment's future data to the agent, which could break the RL framework since the agent is only supposed to know about the current state of the environment (it was not the case in the first edition of the Learning to Run A Power Network as the Environment was fully observable). In grid2op, the simulation is now performed from the current state of the environment and it is imperfect since it does not have access to future information." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here is an example of some features of the observation, in the current state and in the simulated next state :" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "do_nothing_act = env.action_space({})\n", "obs_sim, reward_sim, is_done_sim, info_sim = obs.simulate(do_nothing_act)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "obs.gen_p" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "obs_sim.gen_p" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### e) obs + act\n", "\n", "In gri2op >= 1.5 we also introduced a method that can be used to rapidly assess the effect of a given \"action\" (see next notebook for a more detailed explanation of the actions) on the grid.\n", "\n", "To use it, you can use the `obs.connectivity_matrix()` and `obs.bus_connectivity_matrix()` function combined with the grid2op `obs + act` implementation.\n", "\n", "An example is given here:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "action_description = {}\n", "act = env.action_space(action_description)\n", "resulting_partial_obs = obs + act\n", "resulting_partial_obs.connectivity_matrix()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**NB** This is not commutative: `obs + act` is not the same as `act + obs` (the later will crash by the way).\n", "\n", "**NB** This `obs + act` method is much faster than the `env.step(act)` method or the `obs.simulate(act)` method, but the resulting \"partial observation\" is not complete at all. For example, it do not contains any information on the flows, nor the loads or generators etc. do not have anything regaring the time (date, hour of day etc.). We recommend to use it only combined with the `partial_obs.connectivity_matrix()` or `partial_obs.bus_connectivity_matrix()`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## II) Taking actions based on the observation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this section we will make our first *Agent* that will act based on these observations.\n", "\n", "All *Agents* must derive from the grid2op.Agent class. The main function to implement for the Agents is the \"act\" function (more information can be found on the official documentation or [here](https://grid2op.readthedocs.io/en/latest/agent.html) ). \n", "\n", "Basically, the Agent receives a reward and an observation, and chooses a new action. Some different *Agents* are pre-defined in the `grid2op` package. We won't talk about them here (for more information, see the documentation or the [Agent.py](grid2op/Agent/Agent.py) file), but rather we will make a custom Agent.\n", "\n", "This *Agent* will select among:\n", "\n", "- doing nothing \n", "- disconnecting the powerline having the higher relative flows\n", "- reconnecting a powerline disconnected\n", "- disconnecting the powerline having the lower relative flows\n", "\n", "by using `simulate` on the corresponding actions, and choosing the one that has the highest predicted reward.\n", "\n", "Note that this kind of Agent is not particularly smart and is given only as an example for illustration purposes.\n", "\n", "More information about the creation / manipulation of *Action* will be given in the notebook [2_Action_GridManipulation](2_Action_GridManipulation.ipynb)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from grid2op.Agent import BaseAgent\n", "import numpy as np\n", "\n", "\n", "class MyAgent(BaseAgent):\n", " def __init__(self, action_space):\n", " # python required method to code\n", " BaseAgent.__init__(self, action_space)\n", " self.do_nothing = self.action_space({})\n", " self.print_next = False\n", " \n", " def act(self, observation, reward, done=False):\n", " i_max = np.argmax(observation.rho)\n", " new_status_max = np.zeros(observation.rho.shape, dtype=int)\n", " new_status_max[i_max] = -1\n", " act_max = self.action_space({\"set_line_status\": new_status_max})\n", " \n", " i_min = np.argmin(observation.rho)\n", " new_status_min = np.zeros(observation.rho.shape, dtype=int)\n", " if observation.rho[i_min] > 0:\n", " # all powerlines are connected, i try to disconnect this one\n", " new_status_min[i_min] = -1\n", " act_min = self.action_space({\"set_line_status\": new_status_min})\n", " else:\n", " # at least one powerline is disconnected, i try to reconnect it\n", " new_status_min[i_min] = 1\n", "# act_min = self.action_space({\"set_status\": new_status_min})\n", " act_min = self.action_space({\"set_line_status\": new_status_min,\n", " \"set_bus\": {\"lines_or_id\": [(i_min, 1)], \"lines_ex_id\": [(i_min, 1)]}})\n", " \n", " _, reward_sim_dn, *_ = observation.simulate(self.do_nothing)\n", " _, reward_sim_max, *_ = observation.simulate(act_max)\n", " _, reward_sim_min, *_ = observation.simulate(act_min)\n", " \n", " if reward_sim_dn >= reward_sim_max and reward_sim_dn >= reward_sim_min:\n", " self.print_next = False\n", " res = self.do_nothing\n", " elif reward_sim_max >= reward_sim_min:\n", " self.print_next = True\n", " res = act_max\n", " else:\n", " self.print_next = True\n", " res = act_min\n", " return res" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We compare this Agent with the `Donothing` agent (already coded) on the 3 episodes made available with this package. To make this comparison more interesting, it's better to use the L2RPN rewards (`L2RPNReward`)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from grid2op.Runner import Runner\n", "from grid2op.Agent import DoNothingAgent\n", "from grid2op.Reward import L2RPNReward\n", "from grid2op.Chronics import GridStateFromFileWithForecasts\n", "\n", "max_iter = 10 # to make computation much faster we will only consider 50 time steps instead of 287\n", "runner = Runner(**env.get_params_for_runner(),\n", " agentClass=DoNothingAgent\n", " )\n", "res = runner.run(nb_episode=1, max_iter=max_iter)\n", "\n", "print(\"The results for DoNothing agent are:\")\n", "for _, chron_name, cum_reward, nb_time_step, max_ts in res:\n", " msg_tmp = \"\\tFor chronics with id {}\\n\".format(chron_name)\n", " msg_tmp += \"\\t\\t - cumulative reward: {:.6f}\\n\".format(cum_reward)\n", " msg_tmp += \"\\t\\t - number of time steps completed: {:.0f} / {:.0f}\".format(nb_time_step, max_ts)\n", " print(msg_tmp)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "runner = Runner(**env.get_params_for_runner(),\n", " agentClass=MyAgent\n", " )\n", "res = runner.run(nb_episode=1, max_iter=max_iter)\n", "print(\"The results for the custom agent are:\")\n", "for _, chron_name, cum_reward, nb_time_step, max_ts in res:\n", " msg_tmp = \"\\tFor chronics with id {}\\n\".format(chron_name)\n", " msg_tmp += \"\\t\\t - cumulative reward: {:.6f}\\n\".format(cum_reward)\n", " msg_tmp += \"\\t\\t - number of time steps completed: {:.0f} / {:.0f}\".format(nb_time_step, max_ts)\n", " print(msg_tmp)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we can see, both agents obtain the same score here, but there would be a difference if we didn't limit the episode length to 10 time steps.\n", "\n", "**NB** Disabling the time limit for the episode can be done by setting `max_iter=-1` in the previous cells. Here, setting `max_iter=10` is only done so that this notebook can run faster, but increasing or disabling the time limit would allow us to spot differences in the agents' performances." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The same can be done for the `PowerLineSwitch` agent :" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from grid2op.Agent import PowerLineSwitch\n", "runner = Runner(**env.get_params_for_runner(),\n", " agentClass=PowerLineSwitch\n", " )\n", "res = runner.run(nb_episode=1, max_iter=max_iter)\n", "print(\"The results for the PowerLineSwitch agent are:\")\n", "for _, chron_name, cum_reward, nb_time_step, max_ts in res:\n", " msg_tmp = \"\\tFor chronics with id {}\\n\".format(chron_name)\n", " msg_tmp += \"\\t\\t - cumulative reward: {:.6f}\\n\".format(cum_reward)\n", " msg_tmp += \"\\t\\t - number of time steps completed: {:.0f} / {:.0f}\".format(nb_time_step, max_ts)\n", " print(msg_tmp)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**NB** As of grid2op version 1.5, you can also retrieve all the state of all the step of the episodes by passing the flag `add_detailed_output=True` to the `runner.run(..., add_detailed_output=True)`" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" } }, "nbformat": 4, "nbformat_minor": 2 }