{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic Agent Sudy\n",
    "Try me out interactively with: [![Binder](./img/badge_logo.svg)](https://mybinder.org/v2/gh/rte-france/Grid2Op/master)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is recommended to have a look at the [0_basic_functionalities](0_basic_functionalities.ipynb), [1_Observation_Agents](1_Observation_Agents.ipynb), [2_Action_GridManipulation](3_TrainingAnAgent.ipynb) and [3_TrainingAnAgent](3_TrainingAnAgent.ipynb) notebooks before getting into this one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Objectives**\n",
    "\n",
    "In this notebook we will show how to study an Agent. We will use a dummy agent and then look at how to study its behaviour from the saved file.\n",
    "\n",
    "This notebook will also show you how to use the Graphical User Interface built for analyzing grid2Op agents, called \"Grid2Viz\".\n",
    "\n",
    "It is more than recommended to know how to define an Agent and use a Runner before doing this tutorial!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate the performance of a simple Agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import grid2op\n",
    "import copy\n",
    "import numpy as np\n",
    "import shutil\n",
    "import plotly.graph_objects as go\n",
    "\n",
    "from tqdm.notebook import tqdm\n",
    "from grid2op.Agent import PowerLineSwitch\n",
    "from grid2op.Reward import L2RPNReward\n",
    "from grid2op.Runner import Runner\n",
    "from grid2op.Chronics import GridStateFromFileWithForecasts, Multifolder\n",
    "path_agents = \"study_agent_getting_started\"\n",
    "max_iter = 30"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next cell we evaluate the agent \"PowerLineSwitch\" and save the results of this evaluation in the \"study_agent_getting_started\" directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/benjamin/Documents/grid2op_dev/getting_started/grid2op/MakeEnv/Make.py:267: UserWarning: You are using a development environment. This environment is not intended for training agents. It might not be up to date and its primary use if for tests (hence the \"test=True\" you passed as argument). Use at your own risk.\n",
      "  warnings.warn(_MAKE_DEV_ENV_WARN)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d32a3fb777f848efb6536728d8dab0fc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='episode', max=2.0, style=ProgressStyle(description_width=…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d6c47f251ab84f2da56345b38a579817",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='episode', max=30.0, style=ProgressStyle(description_width…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e5699ef8bdfb4756b19709661f2063e6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='episode', max=30.0, style=ProgressStyle(description_width…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "The results for the evaluated agent are:\n",
      "\tFor chronics with id 000\n",
      "\t\t - cumulative reward: 497.740265\n",
      "\t\t - number of time steps completed: 30 / 30\n",
      "\tFor chronics with id 001\n",
      "\t\t - cumulative reward: 515.659302\n",
      "\t\t - number of time steps completed: 30 / 30\n"
     ]
    }
   ],
   "source": [
    "scoring_function = L2RPNReward\n",
    "env = grid2op.make(reward_class=L2RPNReward, test=True)\n",
    "# env.chronics_handler.set_max_iter(max_iter)\n",
    "shutil.rmtree(os.path.abspath(path_agents), ignore_errors=True)\n",
    "if not os.path.exists(path_agents):\n",
    "    os.mkdir(path_agents)\n",
    "\n",
    "# make a runner for this agent\n",
    "path_agent = os.path.join(path_agents, \"PowerLineSwitch\")\n",
    "shutil.rmtree(os.path.abspath(path_agent), ignore_errors=True)\n",
    "\n",
    "runner = Runner(**env.get_params_for_runner(),\n",
    "                agentClass=PowerLineSwitch\n",
    "                )\n",
    "res = runner.run(path_save=path_agent, nb_episode=2, \n",
    "                max_iter=max_iter,\n",
    "                pbar=tqdm)\n",
    "print(\"The results for the evaluated agent are:\")\n",
    "for _, chron_id, cum_reward, nb_time_step, max_ts in res:\n",
    "    msg_tmp = \"\\tFor chronics with id {}\\n\".format(chron_id)\n",
    "    msg_tmp += \"\\t\\t - cumulative reward: {:.6f}\\n\".format(cum_reward)\n",
    "    msg_tmp += \"\\t\\t - number of time steps completed: {:.0f} / {:.0f}\".format(nb_time_step, max_ts)\n",
    "    print(msg_tmp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Looking at the results and understanding the behaviour of the Agent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The content of the folder is the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['000',\n",
       " 'dict_env_modification_space.json',\n",
       " '001',\n",
       " 'dict_action_space.json',\n",
       " 'dict_observation_space.json',\n",
       " 'dict_attack_space.json']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir(path_agent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can load the actions and observations corresponding to the episode 1 for example, and de-serialize them into proper objects: This is now automatically done with the class `EpisodeData` that can be used as follow:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from grid2op.Episode import EpisodeData\n",
    "episode_studied = \"001\"\n",
    "this_episode = EpisodeData.from_disk(path_agent, episode_studied)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime.datetime(2019, 1, 6, 0, 0)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from datetime import datetime, timedelta\n",
    "episode_data = this_episode\n",
    "this_episode.observations[0].hour_of_day\n",
    "datetime(year=episode_data.observations[0].year,\n",
    "         month=episode_data.observations[0].month,\n",
    "         day=episode_data.observations[0].day,\n",
    "         hour=episode_data.observations[0].hour_of_day,\n",
    "         minute=episode_data.observations[0].minute_of_hour)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(this_episode.observations).minute_of_hour"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspect the actions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can study the agent. For example, let's inspect its actions and see how many powerlines it has disconnected (this is probably not the best thing to do here)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "line_disc = 0\n",
    "line_reco = 0\n",
    "line_changed = 0\n",
    "for act in this_episode.actions:\n",
    "    dict_ = act.as_dict()\n",
    "    if \"set_line_status\" in dict_:\n",
    "        line_reco += dict_[\"set_line_status\"][\"nb_connected\"]\n",
    "        line_disc += dict_[\"set_line_status\"][\"nb_disconnected\"]\n",
    "    if \"change_line_status\" in dict_:\n",
    "        line_changed += dict_[\"change_line_status\"][\"nb_changed\"]\n",
    "print(f'Total lines set to connected : {line_reco}')\n",
    "print(f'Total lines set to disconnected : {line_disc}')\n",
    "print(f'Total lines changed: {line_changed}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also wonder how many times this agent acted on the powerline with id $14$, and inspect how many times it changed its status:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "id_line_inspected = 13\n",
    "actions_on_line_14 = 0\n",
    "for act in this_episode.actions:\n",
    "    dict_ = act.effect_on(line_id=id_line_inspected) # which effect has this action action on the substation with given id\n",
    "    # other objects are: load_id, gen_id, line_id or substation_id\n",
    "    if dict_['change_line_status'] or dict_[\"set_line_status\"] != 0:\n",
    "        actions_on_line_14 += 1\n",
    "print(f'Total actions on powerline 14 : {actions_on_line_14}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspect the modifications of the environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, we might want to inspect the number of hazards and maintenances in a total scenario, to have an idea of how difficult it was."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nb_hazards = 0\n",
    "nb_maintenance = 0\n",
    "for act in this_episode.env_actions:\n",
    "    dict_ = act.as_dict() # representation of an action as a dictionnary, see the documentation for more information\n",
    "    if \"nb_hazards\" in dict_:\n",
    "        nb_hazards += 1\n",
    "    if \"nb_maintenance\" in dict_:\n",
    "        nb_maintenance += 1\n",
    "print(f'Total hazards : {nb_hazards}')\n",
    "print(f'Total maintenances : {nb_maintenance}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspect the observations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, let's look at the consumption of load 1. For this cell to work, plotly is required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import plotly.graph_objects as go\n",
    "load_id = 1\n",
    "# extract the data\n",
    "val_load1 = np.zeros(len(this_episode.observations))\n",
    "for i, obs in enumerate(this_episode.observations):\n",
    "    dict_ = obs.state_of(load_id=load_id) # which effect has this action action on the substation with id 1\n",
    "    # other objects are: load_id, gen_id, line_id or substation_id\n",
    "    # see the documentation for more information.\n",
    "    val_load1[i] = dict_['p']\n",
    "\n",
    "# plot it\n",
    "fig = go.Figure(data=[go.Scatter(x=[i for i in range(len(val_load1))],\n",
    "                                 y=val_load1)])\n",
    "fig.update_layout(title=\"Consumption of load {}\".format(load_id),\n",
    "                 xaxis_title=\"Time step\",\n",
    "                 yaxis_title=\"Load (MW)\")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We may also want to plot the power generated by generator 3 (it represents a solar energy source) :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gen_id = 4\n",
    "# extract the data\n",
    "val_lgen3 = np.zeros(len(this_episode.observations))\n",
    "for i, obs in enumerate(this_episode.observations):\n",
    "    dict_ = obs.state_of(gen_id=gen_id) # which effect has this action action on the substation with id 1\n",
    "    # other objects are: load_id, gen_id, line_id or substation_id\n",
    "    # see the documentation for more information.\n",
    "    val_lgen3[i] = dict_['p']\n",
    "\n",
    "# plot it\n",
    "fig = go.Figure(data=[go.Scatter(x=[i for i in range(len(val_lgen3))],\n",
    "                                 y=val_lgen3)])\n",
    "fig.update_layout(title=\"Production of generator {}\".format(gen_id),\n",
    "                 xaxis_title=\"Time step\",\n",
    "                 yaxis_title=\"Production (MW)\")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the same fashion, we might want to get the flows on powerline connecting bus 3 to bus 4 (without knowing its id by using the appropriate method of the observation_space):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from_ = 3\n",
    "to_ = 4\n",
    "found_ids = this_episode.observations.helper.get_lines_id(from_=from_, to_=to_)\n",
    "line_id = found_ids[0]\n",
    "\n",
    "# extract the data\n",
    "val_l3_4 = np.zeros(len(this_episode.observations))\n",
    "for i, obs in enumerate(this_episode.observations):\n",
    "    dict_ = obs.state_of(line_id=line_id) # which effect has this action action on the substation with id 1\n",
    "    # other objects are: load_id, gen_id, line_id or substation_id\n",
    "    # see the documentation for more information.\n",
    "    val_l3_4[i] = dict_[\"origin\"]['a']\n",
    "\n",
    "# plot it\n",
    "fig = go.Figure(data=[go.Scatter(x=[i for i in range(len(val_l3_4))],\n",
    "                                 y=val_l3_4)])\n",
    "fig.update_layout(title=\"Flow on powerline {} (going from {} to {})\".format(line_id, from_, to_),\n",
    "                 xaxis_title=\"Time step\",\n",
    "                 yaxis_title=\"Production (MW)\")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quick display of a grid using an observation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, you can find an example of how to plot an observation and the underlying powergrid. This is still in development so the appearance will be improved later. It uses plotly and requires the layout of the grid (eg the coordinates of the substations) to be specified.\n",
    "\n",
    "Note also that this code is not optimized at all."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from grid2op.PlotGrid import PlotMatplot\n",
    "obs = copy.deepcopy(this_episode.observations[-1])\n",
    "# and change the topology (just to have something to represent)\n",
    "obs.topo_vect[3:9] = [2,2,1,1,2,1]\n",
    "\n",
    "plot_helper = PlotMatplot(observation_space=this_episode.observation_space, width=900, height=600)\n",
    "plot_helper._line_bus_radius = 7\n",
    "fig = plot_helper.plot_obs(obs)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Synchronizing Observations and Actions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As stated in the documentation, at row i, we can read the observation at time \"i\" (before he took the action at time \"i\"), and the action at time \"i\". This means that at row i of the numpy arrays, we can see what the agent saw at that time and what action he chose from that observation. We have \"an agent view\".\n",
    "\n",
    "In case we want to see the impact of an Action (see the action the agent took and the observation from the environment **after** the action has been taken), it is therefore then necessary to:\n",
    "\n",
    "- look at action i\n",
    "- look at observation i+1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the dedicated grid2viz framework\n",
    "\n",
    "Grid2viz is a package that has been developped to help you visualize the behaviour of your agent. This will be detailed in the notebook [7_PlottingCapabilities](7_PlottingCapabilities.ipynb)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}