{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# This Notebook will develop how to explain an Agent and assess its performance."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It is recommended to have a look at the [0_basic_functionalities](0_basic_functionalities.ipynb) notebook before getting into this one."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Objective**\n",
"\n",
"This notebook will cover the basics of how to \"code\" an Agent that takes action on the powergrid. Examples will be given of \"expert agent\" that can take actions based on some fixed rules. More generic type of *Agent*, relying for example on machine learning / deep learning will be covered in the notebook [3_TrainingAnAgent](3_TrainingAnAgent.ipynb).\n",
"\n",
"This notebook will also cover the description of the *Observation* class, usefull to take some actions."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"import sys\n",
"import grid2op"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
""
],
"text/plain": [
""
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"res = None\n",
"try:\n",
" from jyquickhelper import add_notebook_menu\n",
" res = add_notebook_menu()\n",
"except ModuleNotFoundError:\n",
" print(\"Impossible to automatically add a menu / table of content to this notebook.\\nYou can download \\\"jyquickhelper\\\" package with: \\n\\\"pip install jyquickhelper\\\"\")\n",
"res"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## I) Description of the observations"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this paragraph we will cover the observation class. For more information about it, we recommend to have a look at the official documentation, or [here](https://grid2op.readthedocs.io/en/latest/observation.html) or in the [Observations.py](grid2op/Observation/Observation.py) files for more information. Only basic concepts are detailed in this notebook."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### I.A) Getting an observation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"An observationn of the environment at current time step can be accessed when calling `env.step()`. The next cell is dedicated to create an environment, and to get an observation instance. We use the default `rte_case14_realistic` from Grid2Op framework."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/tezirg/Code/Grid2Op.BDonnot/getting_started/grid2op/MakeEnv/Make.py:223: UserWarning: You are using a development environment. This environment is not intended for training agents.\n",
" warnings.warn(_MAKE_DEV_ENV_WARN)\n"
]
}
],
"source": [
"env = grid2op.make(test=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To perform a step, as stated on the short description above, we need an action. More information about them is given in the [2_ActionRepresentation](2_ActionRepresentation.ipynb) notebook. Here we use a *DoNothingAgent* that does no thing. *obs* is the observation of the environment."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"do_nothing_act = env.helper_action_player({})\n",
"obs, reward, done, info = env.step(do_nothing_act)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### I.B) Information present in an Observation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this notebook we will detail only the \"CompleteObservation\". `Grid2Op` allows to modeled different kind of observations, for example some with incomplete data, or with noisy data etc. `CompleteObservation` gives the full state of the powergrid, without any noise. It's the default observation used."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### a) some of its attributes"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"An observation has calendar data (eg the time stamp of the observation):"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(2019, 1, 1, 0, 10, 1)"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"obs.year, obs.month, obs.day, obs.hour_of_day, obs.minute_of_hour, obs.day_of_week"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It has some powegrid generic information:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Number of generators of the powergrid: 5\n",
"Number of loads of the powergrid: 11\n",
"Number of powerline of the powergrid: 20\n",
"Number of elements connected to each substations in the powergrid: [3 6 4 6 5 6 3 2 5 3 3 3 4 3]\n",
"Total number of elements: 56\n"
]
}
],
"source": [
"print(\"Number of generators of the powergrid: {}\".format(obs.n_gen))\n",
"print(\"Number of loads of the powergrid: {}\".format(obs.n_load))\n",
"print(\"Number of powerline of the powergrid: {}\".format(obs.n_line))\n",
"print(\"Number of elements connected to each substations in the powergrid: {}\".format(obs.sub_info))\n",
"print(\"Total number of elements: {}\".format(obs.dim_topo))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It has some information about the generators (each generator can be viewed as a point in a 3 dimensional space)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Generators active production: [81.6 81.1 12.9 0. 77.7201]\n",
"Generators reactive production: [ 21.790668 70.214264 48.05804 24.508774 -16.541656]\n",
"Generators voltage setpoint : [142.1 142.1 22. 13.200001 142.1 ]\n"
]
}
],
"source": [
"print(\"Generators active production: {}\".format(obs.prod_p))\n",
"print(\"Generators reactive production: {}\".format(obs.prod_q))\n",
"print(\"Generators voltage setpoint : {}\".format(obs.prod_v))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It has some information about the loads (each load is a point in a 3 dimensional space too)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Loads active consumption: [25.4 84.8 45. 6.8 12.7 28.8 9.5 3.4 5.6 11.9 15.4]\n",
"Loads reactive consumption: [ 21.790668 70.214264 48.05804 24.508774 -16.541656]\n",
"Loads voltage (voltage magnitude of the bus to which it is connected) : [142.1 142.1 138.70158 139.39479 22. 21.092138\n",
" 21.085659 21.453535 21.569204 21.430756 20.69996 ]\n"
]
}
],
"source": [
"print(\"Loads active consumption: {}\".format(obs.load_p))\n",
"print(\"Loads reactive consumption: {}\".format(obs.prod_q))\n",
"print(\"Loads voltage (voltage magnitude of the bus to which it is connected) : {}\".format(obs.load_v))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this setting a powerline can be viewed as a point in an 8 dimensional space:\n",
" * active flow\n",
" * reactive flow\n",
" * voltage magnitude\n",
" * current flow\n",
" \n",
"from both of its end.\n",
"\n",
"For example, suppose line1 is denoted by connecting two node A and B. Active flow on line1 has two values, flow from node A to node B (origin) and flow from node B to node A (extremity).\n",
"\n",
"It is then:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Origin active flow: [ 3.9922398e+01 3.7797703e+01 2.1780769e+01 4.0161697e+01\n",
" 3.3859901e+01 1.7860813e+01 -2.8052214e+01 9.7739801e+00\n",
" 7.7287230e+00 1.8051615e+01 3.3593693e+00 7.7858996e+00\n",
" -6.1440678e+00 2.0364611e+00 7.8760457e+00 2.5437183e+01\n",
" 1.4508085e+01 3.5354317e+01 1.5543122e-14 -2.5437183e+01]\n",
"Origin reactive flow: [-15.334058 -1.2075976 -7.0024953 0.663526 -0.383117\n",
" 7.329237 -2.9436882 10.462834 5.576318 14.927625\n",
" -0.85521334 4.0310593 -7.464343 1.4842943 7.410275\n",
" -15.625636 -2.7032974 -5.641245 -23.634314 -5.573329 ]\n",
"Origin current flow: [173.75766 153.64987 92.956 163.19867 137.5811 78.4405\n",
" 117.409485 375.74683 250.10808 614.7267 94.88822 239.99174\n",
" 264.71527 67.45319 291.3341 124.26482 61.42983 148.28416\n",
" 918.84076 712.8031 ]\n",
"Origin voltage (voltage magnitude to the bus to which the origin end is connected): [142.1 142.1 142.1 142.1 142.1 142.1\n",
" 138.70158 22. 22. 22. 21.092138 21.092138\n",
" 21.085659 21.569204 21.430756 138.70158 138.70158 139.39479\n",
" 14.850536 21.092138]\n",
"Extremity active flow: [-3.9602367e+01 -3.7068695e+01 -2.1560818e+01 -3.9274380e+01\n",
" -3.3242977e+01 -1.7618677e+01 2.8157352e+01 -9.6130629e+00\n",
" -7.6364613e+00 -1.7751646e+01 -3.3559325e+00 -7.6980476e+00\n",
" 6.2130628e+00 -2.0243990e+00 -7.7019520e+00 -2.5437183e+01\n",
" -1.4508085e+01 -3.5354317e+01 -1.5987212e-14 2.5437183e+01]\n",
"Extremity reactive flow: [ 10.712756 -0.90132874 3.285029 -1.4910296 -1.3327575\n",
" -8.036348 3.2753313 -10.125853 -5.3842945 -14.336893\n",
" 0.86434317 -3.8441856 7.625853 -1.473381 -7.0558143\n",
" 17.390247 3.8292 8.391265 24.508774 6.244066 ]\n",
"Extremity current flow: [ 166.68697 153.57782 88.61224 163.59877 137.79755 80.60723\n",
" 117.409485 375.74683 250.10808 614.7267 94.88822 239.99174\n",
" 264.71527 67.45319 291.3341 1197.9484 410.72595 953.58575\n",
" 1071.9808 1018.29016 ]\n",
"Extremity voltage (voltage magnitude to the bus to which the origin end is connected): [142.1 139.39479 142.1 138.70158 139.39479 138.70158\n",
" 139.39479 21.453535 21.569204 21.430756 21.085659 20.69996\n",
" 21.453535 21.430756 20.69996 14.850536 21.092138 22.\n",
" 13.200001 14.850536]\n"
]
}
],
"source": [
"print(\"Origin active flow: {}\".format(obs.p_or))\n",
"print(\"Origin reactive flow: {}\".format(obs.q_or))\n",
"print(\"Origin current flow: {}\".format(obs.a_or))\n",
"print(\"Origin voltage (voltage magnitude to the bus to which the origin end is connected): {}\".format(obs.v_or))\n",
"print(\"Extremity active flow: {}\".format(obs.p_ex))\n",
"print(\"Extremity reactive flow: {}\".format(obs.q_ex))\n",
"print(\"Extremity current flow: {}\".format(obs.a_ex))\n",
"print(\"Extremity voltage (voltage magnitude to the bus to which the origin end is connected): {}\".format(obs.v_ex))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The last informations about the powerlines is the $\\rho$ ratio, *ie.* the ratio between the current flow on each powerlines and the its thermal limits. It can be accessed with:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([0.45143566, 0.3991941 , 0.24462105, 0.42947015, 0.87631273,\n",
" 0.20642236, 0.30897233, 0.34864962, 0.54149985, 0.7985534 ,\n",
" 0.35218123, 0.62351686, 0.34830955, 0.1775084 , 0.38333437,\n",
" 0.32284948, 0.26599896, 0.868177 , 0.27006912, 0.20950978],\n",
" dtype=float32)"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"obs.rho"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Observation (*obs*) of the environment also store information of the topology and the state of the powerline."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
" 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
" 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], dtype=int32)"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"obs.timestep_overflow # the number of timestep each of the powerline is in overflow (1 powerline per component)\n",
"obs.line_status # the status of each powerline: True connected, False disconnected\n",
"obs.topo_vect # the topology vector the each element (generator, load, each end of a powerline) to which the object\n",
"# is connected: 1 = bus 1, 2 = bus 2."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In grid2op all objects (end of a powerline, load or generator) can be either disconnected, connected to the first bus at its substation, or connected to the second bus of its substation.\n",
" \n",
"If an object is disconnected, then the component of the `topo_vect` vector corresponding to this object will be `-1`. If it's connected to the first bus it will be `1` and `2` if it's connected to the second bus.\n",
"\n",
" More information about this `topology vector` is given in the documentation [here](https://grid2op.readthedocs.io/en/latest/observation.html).\n",
" \n",
" More information about this topology vector will be given in the notebook dedicated to vizualisation. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### b) some of its methods"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It can be converted to / from flat numpy vector. This function is usefull for interacting with machine learning or to store it, but probably less human readable. It consists in stacking all the above-mentionned information in a single `numpy.float64` vector."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([ 2.01900000e+03, 1.00000000e+00, 1.00000000e+00, 0.00000000e+00,\n",
" 1.00000000e+01, 1.00000000e+00, 8.15999985e+01, 8.10999985e+01,\n",
" 1.28999996e+01, 0.00000000e+00, 7.77201004e+01, 2.17906685e+01,\n",
" 7.02142639e+01, 4.80580406e+01, 2.45087738e+01, -1.65416565e+01,\n",
" 1.42100006e+02, 1.42100006e+02, 2.20000000e+01, 1.32000008e+01,\n",
" 1.42100006e+02, 2.53999996e+01, 8.48000031e+01, 4.50000000e+01,\n",
" 6.80000019e+00, 1.26999998e+01, 2.87999992e+01, 9.50000000e+00,\n",
" 3.40000010e+00, 5.59999990e+00, 1.18999996e+01, 1.53999996e+01,\n",
" 1.77999992e+01, 5.95999985e+01, 3.07999992e+01, 4.59999990e+00,\n",
" 8.69999981e+00, 1.97000008e+01, 6.59999990e+00, 2.50000000e+00,\n",
" 3.90000010e+00, 8.39999962e+00, 1.08999996e+01, 1.42100006e+02,\n",
" 1.42100006e+02, 1.38701584e+02, 1.39394791e+02, 2.20000000e+01,\n",
" 2.10921383e+01, 2.10856590e+01, 2.14535351e+01, 2.15692043e+01,\n",
" 2.14307556e+01, 2.06999607e+01, 3.99223976e+01, 3.77977028e+01,\n",
" 2.17807693e+01, 4.01616974e+01, 3.38599014e+01, 1.78608131e+01,\n",
" -2.80522137e+01, 9.77398014e+00, 7.72872305e+00, 1.80516148e+01,\n",
" 3.35936928e+00, 7.78589964e+00, -6.14406776e+00, 2.03646111e+00,\n",
" 7.87604570e+00, 2.54371834e+01, 1.45080853e+01, 3.53543167e+01,\n",
" 1.55431223e-14, -2.54371834e+01, -1.53340578e+01, -1.20759761e+00,\n",
" -7.00249529e+00, 6.63525999e-01, -3.83116990e-01, 7.32923698e+00,\n",
" -2.94368815e+00, 1.04628344e+01, 5.57631779e+00, 1.49276247e+01,\n",
" -8.55213344e-01, 4.03105927e+00, -7.46434307e+00, 1.48429430e+00,\n",
" 7.41027498e+00, -1.56256361e+01, -2.70329738e+00, -5.64124489e+00,\n",
" -2.36343136e+01, -5.57332897e+00, 1.42100006e+02, 1.42100006e+02,\n",
" 1.42100006e+02, 1.42100006e+02, 1.42100006e+02, 1.42100006e+02,\n",
" 1.38701584e+02, 2.20000000e+01, 2.20000000e+01, 2.20000000e+01,\n",
" 2.10921383e+01, 2.10921383e+01, 2.10856590e+01, 2.15692043e+01,\n",
" 2.14307556e+01, 1.38701584e+02, 1.38701584e+02, 1.39394791e+02,\n",
" 1.48505363e+01, 2.10921383e+01, 1.73757660e+02, 1.53649872e+02,\n",
" 9.29560013e+01, 1.63198669e+02, 1.37581100e+02, 7.84404984e+01,\n",
" 1.17409485e+02, 3.75746826e+02, 2.50108078e+02, 6.14726685e+02,\n",
" 9.48882217e+01, 2.39991745e+02, 2.64715271e+02, 6.74531937e+01,\n",
" 2.91334106e+02, 1.24264816e+02, 6.14298286e+01, 1.48284164e+02,\n",
" 9.18840759e+02, 7.12803101e+02, -3.96023674e+01, -3.70686951e+01,\n",
" -2.15608177e+01, -3.92743797e+01, -3.32429771e+01, -1.76186771e+01,\n",
" 2.81573524e+01, -9.61306286e+00, -7.63646126e+00, -1.77516460e+01,\n",
" -3.35593247e+00, -7.69804764e+00, 6.21306276e+00, -2.02439904e+00,\n",
" -7.70195198e+00, -2.54371834e+01, -1.45080853e+01, -3.53543167e+01,\n",
" -1.59872116e-14, 2.54371834e+01, 1.07127562e+01, -9.01328743e-01,\n",
" 3.28502893e+00, -1.49102962e+00, -1.33275747e+00, -8.03634834e+00,\n",
" 3.27533126e+00, -1.01258526e+01, -5.38429451e+00, -1.43368931e+01,\n",
" 8.64343166e-01, -3.84418559e+00, 7.62585306e+00, -1.47338104e+00,\n",
" -7.05581427e+00, 1.73902473e+01, 3.82920003e+00, 8.39126492e+00,\n",
" 2.45087738e+01, 6.24406624e+00, 1.42100006e+02, 1.39394791e+02,\n",
" 1.42100006e+02, 1.38701584e+02, 1.39394791e+02, 1.38701584e+02,\n",
" 1.39394791e+02, 2.14535351e+01, 2.15692043e+01, 2.14307556e+01,\n",
" 2.10856590e+01, 2.06999607e+01, 2.14535351e+01, 2.14307556e+01,\n",
" 2.06999607e+01, 1.48505363e+01, 2.10921383e+01, 2.20000000e+01,\n",
" 1.32000008e+01, 1.48505363e+01, 1.66686966e+02, 1.53577820e+02,\n",
" 8.86122437e+01, 1.63598770e+02, 1.37797546e+02, 8.06072311e+01,\n",
" 1.17409485e+02, 3.75746826e+02, 2.50108078e+02, 6.14726685e+02,\n",
" 9.48882217e+01, 2.39991745e+02, 2.64715271e+02, 6.74531937e+01,\n",
" 2.91334106e+02, 1.19794836e+03, 4.10725952e+02, 9.53585754e+02,\n",
" 1.07198083e+03, 1.01829016e+03, 4.51435655e-01, 3.99194092e-01,\n",
" 2.44621053e-01, 4.29470152e-01, 8.76312733e-01, 2.06422359e-01,\n",
" 3.08972329e-01, 3.48649621e-01, 5.41499853e-01, 7.98553407e-01,\n",
" 3.52181226e-01, 6.23516858e-01, 3.48309547e-01, 1.77508399e-01,\n",
" 3.83334368e-01, 3.22849482e-01, 2.65998960e-01, 8.68176997e-01,\n",
" 2.70069122e-01, 2.09509775e-01, 1.00000000e+00, 1.00000000e+00,\n",
" 1.00000000e+00, 1.00000000e+00, 1.00000000e+00, 1.00000000e+00,\n",
" 1.00000000e+00, 1.00000000e+00, 1.00000000e+00, 1.00000000e+00,\n",
" 1.00000000e+00, 1.00000000e+00, 1.00000000e+00, 1.00000000e+00,\n",
" 1.00000000e+00, 1.00000000e+00, 1.00000000e+00, 1.00000000e+00,\n",
" 1.00000000e+00, 1.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
" 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
" 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
" 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
" 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
" 0.00000000e+00, 0.00000000e+00, 1.00000000e+00, 1.00000000e+00,\n",
" 1.00000000e+00, 1.00000000e+00, 1.00000000e+00, 1.00000000e+00,\n",
" 1.00000000e+00, 1.00000000e+00, 1.00000000e+00, 1.00000000e+00,\n",
" 1.00000000e+00, 1.00000000e+00, 1.00000000e+00, 1.00000000e+00,\n",
" 1.00000000e+00, 1.00000000e+00, 1.00000000e+00, 1.00000000e+00,\n",
" 1.00000000e+00, 1.00000000e+00, 1.00000000e+00, 1.00000000e+00,\n",
" 1.00000000e+00, 1.00000000e+00, 1.00000000e+00, 1.00000000e+00,\n",
" 1.00000000e+00, 1.00000000e+00, 1.00000000e+00, 1.00000000e+00,\n",
" 1.00000000e+00, 1.00000000e+00, 1.00000000e+00, 1.00000000e+00,\n",
" 1.00000000e+00, 1.00000000e+00, 1.00000000e+00, 1.00000000e+00,\n",
" 1.00000000e+00, 1.00000000e+00, 1.00000000e+00, 1.00000000e+00,\n",
" 1.00000000e+00, 1.00000000e+00, 1.00000000e+00, 1.00000000e+00,\n",
" 1.00000000e+00, 1.00000000e+00, 1.00000000e+00, 1.00000000e+00,\n",
" 1.00000000e+00, 1.00000000e+00, 1.00000000e+00, 1.00000000e+00,\n",
" 1.00000000e+00, 1.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
" 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
" 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
" 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
" 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
" 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
" 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
" 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
" 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
" 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
" 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
" 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
" 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
" 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
" -1.00000000e+00, -1.00000000e+00, -1.00000000e+00, -1.00000000e+00,\n",
" -1.00000000e+00, -1.00000000e+00, -1.00000000e+00, -1.00000000e+00,\n",
" -1.00000000e+00, -1.00000000e+00, -1.00000000e+00, -1.00000000e+00,\n",
" -1.00000000e+00, -1.00000000e+00, -1.00000000e+00, -1.00000000e+00,\n",
" -1.00000000e+00, -1.00000000e+00, -1.00000000e+00, -1.00000000e+00,\n",
" 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
" 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
" 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
" 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
" 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
" 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
" 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
" 0.00000000e+00, 0.00000000e+00], dtype=float32)"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"vector_representation_of_observation = obs.to_vect()\n",
"vector_representation_of_observation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"An observation can be copied, of course:"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"obs2 = obs.copy()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Or reset:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[nan nan nan nan nan]\n"
]
}
],
"source": [
"obs2.reset()\n",
"print(obs2.prod_p)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Or loaded from a vector:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([81.6 , 81.1 , 12.9 , 0. , 77.7201], dtype=float32)"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"obs2.from_vect(vector_representation_of_observation)\n",
"obs2.prod_p"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It is also possible to assess whether two observations are equals or not:"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"obs == obs2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For this type of observation, it is also possible to retrieve the topology as a matrix. The topology matrix can be obtained in two different format."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"*Format 1*: the `connectivity matrix` which has as many row / columns as the number of elements in the powergrid (remember an element is either an end of a powerline, or a generator or a load) and that says if 2 elements are connected to one another or not:\n",
"\n",
"$$\n",
"\\left\\{\n",
"\\begin{aligned}\n",
"\\text{conn mat}[i,j] = 0 & ~\\text{element i and j are NOT connected to the same bus}\\\\\n",
"\\text{conn mat}[i,j] = 1 & ~\\text{element i and j are connected to the same bus, or i and j are both end of the same powerline}\\\\\n",
"\\end{aligned}\n",
"\\right.\n",
"$$"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[0., 1., 1., ..., 0., 0., 0.],\n",
" [1., 0., 1., ..., 0., 0., 0.],\n",
" [1., 1., 0., ..., 0., 0., 0.],\n",
" ...,\n",
" [0., 0., 0., ..., 0., 1., 1.],\n",
" [0., 0., 0., ..., 1., 0., 1.],\n",
" [0., 0., 0., ..., 1., 1., 0.]], dtype=float32)"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"obs.connectivity_matrix()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This representation has the advantage to give a matrix wil always the same dimension, regardless of the topology of the powergrid."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"*Format 2*: the `bus connectivity matrix` has as many row / columns as the number of active buses of the powergrid. It should be understood as followed:\n",
"\n",
"$$\n",
"\\left\\{\n",
"\\begin{aligned}\n",
"\\text{bus conn mat}[i,j] = 0 & ~\\text{no powerline connect bus i to bus j}\\\\\n",
"\\text{bus conn mat}[i,j] = 1 & ~\\text{at least a powerline connectes bus i to bus j (or i == j)}\\\\\n",
"\\end{aligned}\n",
"\\right.\n",
"$$"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[1., 1., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
" [1., 1., 1., 1., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
" [0., 1., 1., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
" [0., 1., 1., 1., 1., 0., 1., 0., 1., 0., 0., 0., 0., 0.],\n",
" [1., 1., 0., 1., 1., 1., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
" [0., 0., 0., 0., 1., 1., 0., 0., 0., 0., 1., 1., 1., 0.],\n",
" [0., 0., 0., 1., 0., 0., 1., 1., 1., 0., 0., 0., 0., 0.],\n",
" [0., 0., 0., 0., 0., 0., 1., 1., 0., 0., 0., 0., 0., 0.],\n",
" [0., 0., 0., 1., 0., 0., 1., 0., 1., 1., 0., 0., 0., 1.],\n",
" [0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 1., 0., 0., 0.],\n",
" [0., 0., 0., 0., 0., 1., 0., 0., 0., 1., 1., 0., 0., 0.],\n",
" [0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 1., 1., 0.],\n",
" [0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 1., 1., 1.],\n",
" [0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 1., 1.]],\n",
" dtype=float32)"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"obs.bus_connectivity_matrix()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### c) Simulate"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As opposed to most RL problems, in this framework we add the possibility to \"simulate\" the impact of a possible action on the power grid. This helps to calculate roll outs in RL setting, and can be close to \"model based\" reinforcement learning approaches (except that nothing more has to be learned).\n",
"\n",
"This \"simulate\" method uses the avalable forecast data (forecasts are made available by the way we loaded the data here, with the class `GridStateFromFileWithForecasts`. For this class, only 1 time step ahead forecasts are provided, but this might be adapted in the future).\n",
"\n",
"Note that this `simulate` function can use a different simulator than the one used by the Environment. Fore more information, we encourage you to read the official documentation or if it has been built locally (recommended) to consult [this page](https://grid2op.readthedocs.io/en/latest/observation.html#grid2op.Observation.Observation.simulate).\n",
"\n",
"This function will:\n",
"\n",
"1. apply the forecasted injection on the powergrid\n",
"2. run a powerflow with the decidated `simulate` powerflow simulator\n",
"3. return:\n",
" 1. the anticipated observation (after the action has been taken)\n",
" 2. the anticipated reward (of this simulated action)\n",
" 3. whether or not there has been an error\n",
" 4. some more informations\n",
" \n",
"From a user point of view, this is the main difference with the previous [pypownet](https://github.com/MarvinLer/pypownet) framework. This \"simulation\" used to be performed directly by the environment, thus giving a direct access to the Agent to the Environment, which could break the RL framework (it was not the case in the first edition of the Learning to Run A Power Network as the Environment was fully observable)."
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [],
"source": [
"do_nothing_act = env.helper_action_player({})\n",
"obs_sim, reward_sim, is_done_sim, info_sim = obs.simulate(do_nothing_act)"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([81.5 , 79.7 , 12.9 , 0. , 79.5781], dtype=float32)"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"obs_sim.prod_p"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([81.6 , 81.1 , 12.9 , 0. , 77.7201], dtype=float32)"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"obs.prod_p"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## II) Taking actions based on these"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this section we will make our first *Agent* that will act based on these observations.\n",
"\n",
"All *Agent* must derived from the grid2op.Agent class. The main function to code for the Agents is the \"act\" function (more information on the official documentation or [here](https://grid2op.readthedocs.io/en/latest/agent.html) ). \n",
"\n",
"Basically, the Agent receive a reward and an observation, and suggest a new action. Some different *Agent* are pre-define in the grid2op package. We won't expose them here (for more information see the documantation or the [Agent.py](grid2op/Agent/Agent.py) file), but rather we will make a custom Agent.\n",
"\n",
"This *Agent* will select among:\n",
"\n",
"- do nothing \n",
"- disconnecting the powerline having the higher relative flows\n",
"- reconnecting a powerline disconnected\n",
"- disconnecting the powerline having the lower relative flows\n",
"\n",
"by using `simulate` on the corresponding actions, and choosing the one that has the highest predicted reward.\n",
"\n",
"Note that this kind of Agent is not particularly smart and is given only as an example.\n",
"\n",
"More information about the creation / manipulation of *Action* will be given in the notebook [2_Action_GridManipulation](2_Action_GridManipulation.ipynb)"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [],
"source": [
"from grid2op.Agent import BaseAgent\n",
"import numpy as np\n",
"import pdb\n",
"\n",
"\n",
"class MyAgent(BaseAgent):\n",
" def __init__(self, action_space):\n",
" # python required method to code\n",
" BaseAgent.__init__(self, action_space)\n",
" self.do_nothing = self.action_space({})\n",
" self.print_next = False\n",
" \n",
" def act(self, observation, reward, done=False):\n",
" i_max = np.argmax(observation.rho)\n",
" new_status_max = np.zeros(observation.rho.shape)\n",
" new_status_max[i_max] = -1\n",
" act_max = self.action_space({\"set_line_status\": new_status_max})\n",
" \n",
" i_min = np.argmin(observation.rho)\n",
" new_status_min = np.zeros(observation.rho.shape)\n",
" if observation.rho[i_min] > 0:\n",
" # all powerlines are connected, i try to disconnect this one\n",
" new_status_min[i_min] = -1\n",
" act_min = self.action_space({\"set_line_status\": new_status_min})\n",
" else:\n",
" # at least one powerline is disconnected, i try to reconnect it\n",
" new_status_min[i_min] = 1\n",
"# act_min = self.action_space({\"set_status\": new_status_min})\n",
" act_min = self.action_space({\"set_line_status\": new_status_min,\n",
" \"set_bus\": {\"lines_or_id\": [(i_min, 1)], \"lines_ex_id\": [(i_min, 1)]}})\n",
" \n",
" _, reward_sim_dn, *_ = observation.simulate(self.do_nothing)\n",
" _, reward_sim_max, *_ = observation.simulate(act_max)\n",
" _, reward_sim_min, *_ = observation.simulate(act_min)\n",
" \n",
" if reward_sim_dn >= reward_sim_max and reward_sim_dn >= reward_sim_min:\n",
" self.print_next = False\n",
" res = self.do_nothing\n",
" elif reward_sim_max >= reward_sim_min:\n",
" self.print_next = True\n",
" res = act_max\n",
" print(res)\n",
" else:\n",
" self.print_next = True\n",
" res = act_min\n",
" print(res)\n",
" return res"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We compare this Agent with the Donothing agent (already coded) on the 3 episode made available with this package. To make the comparison more interesting, it's better to use the predefined reward class that has L2RPN rewards."
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The results for DoNothing agent are:\n",
"\tFor chronics with id 000\n",
"\t\t - cumulative reward: 11076.768763\n",
"\t\t - number of time steps completed: 10 / 10\n"
]
}
],
"source": [
"from grid2op.Runner import Runner\n",
"from grid2op.Agent import DoNothingAgent\n",
"from grid2op.Reward import L2RPNReward\n",
"from grid2op.Chronics import GridStateFromFileWithForecasts\n",
"\n",
"max_iter = 10 # to make computation much faster we will only consider 50 time steps instead of 287\n",
"runner = Runner(**env.get_params_for_runner(),\n",
" agentClass=DoNothingAgent\n",
" )\n",
"res = runner.run(nb_episode=1, max_iter=max_iter)\n",
"\n",
"print(\"The results for DoNothing agent are:\")\n",
"for _, chron_name, cum_reward, nb_time_step, max_ts in res:\n",
" msg_tmp = \"\\tFor chronics with id {}\\n\".format(chron_name)\n",
" msg_tmp += \"\\t\\t - cumulative reward: {:.6f}\\n\".format(cum_reward)\n",
" msg_tmp += \"\\t\\t - number of time steps completed: {:.0f} / {:.0f}\".format(nb_time_step, max_ts)\n",
" print(msg_tmp)"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The results for the custom agent are:\n",
"\tFor chronics with id 000\n",
"\t\t - cumulative reward: 11076.768763\n",
"\t\t - number of time steps completed: 10 / 10\n"
]
}
],
"source": [
"runner = Runner(**env.get_params_for_runner(),\n",
" agentClass=MyAgent\n",
" )\n",
"res = runner.run(nb_episode=1, max_iter=max_iter)\n",
"print(\"The results for the custom agent are:\")\n",
"for _, chron_name, cum_reward, nb_time_step, max_ts in res:\n",
" msg_tmp = \"\\tFor chronics with id {}\\n\".format(chron_name)\n",
" msg_tmp += \"\\t\\t - cumulative reward: {:.6f}\\n\".format(cum_reward)\n",
" msg_tmp += \"\\t\\t - number of time steps completed: {:.0f} / {:.0f}\".format(nb_time_step, max_ts)\n",
" print(msg_tmp)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As we can see, there is no change in the performance for both agent (there would be if we didn't limit the episode length to 10 time steps)\n",
"\n",
"This agent is NOT recommended.\n",
"\n",
"**NB** These scores are given if setting `max_iter=-1` in the previous cells. Here, when `max_iter=10` we don't see any difference in the score (max_iter=10 has been set to make the tests of these notebooks run faster)"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The results for the PowerLineSwitch agent are:\n",
"\tFor chronics with ID 000\n",
"\t\t - cumulative reward: 2164.011485\n",
"\t\t - number of time steps completed: 3 / 10\n"
]
}
],
"source": [
"from grid2op.Agent import PowerLineSwitch\n",
"runner = Runner(**env.get_params_for_runner(),\n",
" agentClass=PowerLineSwitch\n",
" )\n",
"res = runner.run(nb_episode=1, max_iter=max_iter)\n",
"print(\"The results for the PowerLineSwitch agent are:\")\n",
"for _, chron_name, cum_reward, nb_time_step, max_ts in res:\n",
" msg_tmp = \"\\tFor chronics with ID {}\\n\".format(chron_name)\n",
" msg_tmp += \"\\t\\t - cumulative reward: {:.6f}\\n\".format(cum_reward)\n",
" msg_tmp += \"\\t\\t - number of time steps completed: {:.0f} / {:.0f}\".format(nb_time_step, max_ts)\n",
" print(msg_tmp)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.9"
}
},
"nbformat": 4,
"nbformat_minor": 2
}