{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Res1D - Strategies for handling large files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-06T14:20:17.325700Z",
     "iopub.status.busy": "2024-11-06T14:20:17.325166Z",
     "iopub.status.idle": "2024-11-06T14:20:18.733293Z",
     "shell.execute_reply": "2024-11-06T14:20:18.732798Z"
    }
   },
   "outputs": [],
   "source": [
    "import mikeio1d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All static network data is loaded into memory by default. Dynamic result data is only loaded when required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-06T14:20:18.735490Z",
     "iopub.status.busy": "2024-11-06T14:20:18.735010Z",
     "iopub.status.idle": "2024-11-06T14:20:18.966191Z",
     "shell.execute_reply": "2024-11-06T14:20:18.965360Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res = mikeio1d.open(\"../tests/testdata/network.res1d\")\n",
    "loaded_time_steps = res.data.NumberOfTimeSteps\n",
    "loaded_time_steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calling read() loads dynamic data for the entire network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-06T14:20:18.968808Z",
     "iopub.status.busy": "2024-11-06T14:20:18.968236Z",
     "iopub.status.idle": "2024-11-06T14:20:19.174037Z",
     "shell.execute_reply": "2024-11-06T14:20:19.173601Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "110"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res.read()\n",
    "loaded_time_steps = res.data.NumberOfTimeSteps\n",
    "loaded_time_steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All network dynamic data is loaded into memory regardless of how read() is called. In the future, it would nice to make this work lazily."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-06T14:20:19.175793Z",
     "iopub.status.busy": "2024-11-06T14:20:19.175496Z",
     "iopub.status.idle": "2024-11-06T14:20:19.330255Z",
     "shell.execute_reply": "2024-11-06T14:20:19.329460Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>WaterLevel:1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1994-08-07 16:35:00.000</th>\n",
       "      <td>195.052994</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1994-08-07 16:36:01.870</th>\n",
       "      <td>195.052994</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1994-08-07 16:37:07.560</th>\n",
       "      <td>195.052994</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1994-08-07 16:38:55.828</th>\n",
       "      <td>195.052994</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1994-08-07 16:39:55.828</th>\n",
       "      <td>195.052994</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         WaterLevel:1\n",
       "1994-08-07 16:35:00.000    195.052994\n",
       "1994-08-07 16:36:01.870    195.052994\n",
       "1994-08-07 16:37:07.560    195.052994\n",
       "1994-08-07 16:38:55.828    195.052994\n",
       "1994-08-07 16:39:55.828    195.052994"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res = mikeio1d.open(\"../tests/testdata/network.res1d\")\n",
    "# this still reads all network data, but only creates a DataFrame with the requested data\n",
    "res.nodes[\"1\"].WaterLevel.read().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To avoid loading the entire network's dynamic data into memory, you can specify which IDs to load when opening the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-06T14:20:19.331943Z",
     "iopub.status.busy": "2024-11-06T14:20:19.331754Z",
     "iopub.status.idle": "2024-11-06T14:20:19.370047Z",
     "shell.execute_reply": "2024-11-06T14:20:19.369465Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "&lt;ResultNodes&gt;\n",
       "    <style>\n",
       "        ul {\n",
       "            margin: 0px;\n",
       "            padding: 0px;\n",
       "            padding-left: 2em;\n",
       "        }\n",
       "    </style>\n",
       "    <details><summary>Names (1)</summary><ul><li>1</li></ul></details><details><summary>Quantities (1)</summary><ul><li>Water level (m)</li></ul></details><details><summary>Derived Quantities (3)</summary><ul><li>NodeFlooding</li><li>NodeWaterDepth</li><li>NodeWaterLevelAboveCritical</li></ul></details>"
      ],
      "text/plain": [
       "<ResultNodes>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res = mikeio1d.open(\"../tests/testdata/network.res1d\", nodes=[\"1\"], reaches=[\"100l1\"])\n",
    "res.nodes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-06T14:20:19.372057Z",
     "iopub.status.busy": "2024-11-06T14:20:19.371872Z",
     "iopub.status.idle": "2024-11-06T14:20:19.376270Z",
     "shell.execute_reply": "2024-11-06T14:20:19.375673Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "&lt;ResultReaches&gt;\n",
       "    <style>\n",
       "        ul {\n",
       "            margin: 0px;\n",
       "            padding: 0px;\n",
       "            padding-left: 2em;\n",
       "        }\n",
       "    </style>\n",
       "    <details><summary>Names (1)</summary><ul><li>100l1</li></ul></details><details><summary>Quantities (2)</summary><ul><li>Water level (m)</li><li>Discharge (m^3/s)</li></ul></details><details><summary>Derived Quantities (6)</summary><ul><li>ReachAbsoluteDischarge</li><li>ReachFilling</li><li>ReachFlooding</li><li>ReachQQManning</li><li>ReachWaterDepth</li><li>ReachWaterLevelAboveCritical</li></ul></details>"
      ],
      "text/plain": [
       "<ResultReaches>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res.reaches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-06T14:20:19.378065Z",
     "iopub.status.busy": "2024-11-06T14:20:19.377689Z",
     "iopub.status.idle": "2024-11-06T14:20:19.422404Z",
     "shell.execute_reply": "2024-11-06T14:20:19.421662Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>WaterLevel:1</th>\n",
       "      <th>WaterLevel:100l1:0</th>\n",
       "      <th>WaterLevel:100l1:47.6827</th>\n",
       "      <th>Discharge:100l1:23.8414</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1994-08-07 16:35:00.000</th>\n",
       "      <td>195.052994</td>\n",
       "      <td>195.441498</td>\n",
       "      <td>194.661499</td>\n",
       "      <td>0.000006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1994-08-07 16:36:01.870</th>\n",
       "      <td>195.052994</td>\n",
       "      <td>195.441498</td>\n",
       "      <td>194.661621</td>\n",
       "      <td>0.000006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1994-08-07 16:37:07.560</th>\n",
       "      <td>195.052994</td>\n",
       "      <td>195.441498</td>\n",
       "      <td>194.661728</td>\n",
       "      <td>0.000006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1994-08-07 16:38:55.828</th>\n",
       "      <td>195.052994</td>\n",
       "      <td>195.441498</td>\n",
       "      <td>194.661804</td>\n",
       "      <td>0.000006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1994-08-07 16:39:55.828</th>\n",
       "      <td>195.052994</td>\n",
       "      <td>195.441498</td>\n",
       "      <td>194.661972</td>\n",
       "      <td>0.000006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1994-08-07 18:30:07.967</th>\n",
       "      <td>195.119919</td>\n",
       "      <td>195.455109</td>\n",
       "      <td>194.689072</td>\n",
       "      <td>0.000588</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1994-08-07 18:31:07.967</th>\n",
       "      <td>195.118607</td>\n",
       "      <td>195.455063</td>\n",
       "      <td>194.688934</td>\n",
       "      <td>0.000583</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1994-08-07 18:32:07.967</th>\n",
       "      <td>195.117310</td>\n",
       "      <td>195.455002</td>\n",
       "      <td>194.688812</td>\n",
       "      <td>0.000579</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1994-08-07 18:33:07.967</th>\n",
       "      <td>195.115753</td>\n",
       "      <td>195.453049</td>\n",
       "      <td>194.688354</td>\n",
       "      <td>0.000526</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1994-08-07 18:35:00.000</th>\n",
       "      <td>195.112534</td>\n",
       "      <td>195.450409</td>\n",
       "      <td>194.686172</td>\n",
       "      <td>0.000343</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>110 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                         WaterLevel:1  WaterLevel:100l1:0  \\\n",
       "1994-08-07 16:35:00.000    195.052994          195.441498   \n",
       "1994-08-07 16:36:01.870    195.052994          195.441498   \n",
       "1994-08-07 16:37:07.560    195.052994          195.441498   \n",
       "1994-08-07 16:38:55.828    195.052994          195.441498   \n",
       "1994-08-07 16:39:55.828    195.052994          195.441498   \n",
       "...                               ...                 ...   \n",
       "1994-08-07 18:30:07.967    195.119919          195.455109   \n",
       "1994-08-07 18:31:07.967    195.118607          195.455063   \n",
       "1994-08-07 18:32:07.967    195.117310          195.455002   \n",
       "1994-08-07 18:33:07.967    195.115753          195.453049   \n",
       "1994-08-07 18:35:00.000    195.112534          195.450409   \n",
       "\n",
       "                         WaterLevel:100l1:47.6827  Discharge:100l1:23.8414  \n",
       "1994-08-07 16:35:00.000                194.661499                 0.000006  \n",
       "1994-08-07 16:36:01.870                194.661621                 0.000006  \n",
       "1994-08-07 16:37:07.560                194.661728                 0.000006  \n",
       "1994-08-07 16:38:55.828                194.661804                 0.000006  \n",
       "1994-08-07 16:39:55.828                194.661972                 0.000006  \n",
       "...                                           ...                      ...  \n",
       "1994-08-07 18:30:07.967                194.689072                 0.000588  \n",
       "1994-08-07 18:31:07.967                194.688934                 0.000583  \n",
       "1994-08-07 18:32:07.967                194.688812                 0.000579  \n",
       "1994-08-07 18:33:07.967                194.688354                 0.000526  \n",
       "1994-08-07 18:35:00.000                194.686172                 0.000343  \n",
       "\n",
       "[110 rows x 4 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, you can also specify the time range to read."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-06T14:20:19.424865Z",
     "iopub.status.busy": "2024-11-06T14:20:19.424679Z",
     "iopub.status.idle": "2024-11-06T14:20:19.463938Z",
     "shell.execute_reply": "2024-11-06T14:20:19.463175Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "start, end = \"1994-08-07 18:30:07.967\", \"1994-08-07 18:33:07.967\"\n",
    "res = mikeio1d.open(\n",
    "    \"../tests/testdata/network.res1d\", nodes=[\"1\"], reaches=[\"100l1\"], time=(start, end)\n",
    ")\n",
    "loaded_time_steps = res.data.NumberOfTimeSteps\n",
    "loaded_time_steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-06T14:20:19.466847Z",
     "iopub.status.busy": "2024-11-06T14:20:19.466606Z",
     "iopub.status.idle": "2024-11-06T14:20:19.506227Z",
     "shell.execute_reply": "2024-11-06T14:20:19.505755Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res.read()\n",
    "loaded_time_steps = res.data.NumberOfTimeSteps\n",
    "loaded_time_steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-06T14:20:19.507619Z",
     "iopub.status.busy": "2024-11-06T14:20:19.507477Z",
     "iopub.status.idle": "2024-11-06T14:20:19.517141Z",
     "shell.execute_reply": "2024-11-06T14:20:19.516602Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>WaterLevel:1</th>\n",
       "      <th>WaterLevel:100l1:0</th>\n",
       "      <th>WaterLevel:100l1:47.6827</th>\n",
       "      <th>Discharge:100l1:23.8414</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1994-08-07 18:30:07.967</th>\n",
       "      <td>195.119919</td>\n",
       "      <td>195.455109</td>\n",
       "      <td>194.689072</td>\n",
       "      <td>0.000588</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1994-08-07 18:31:07.967</th>\n",
       "      <td>195.118607</td>\n",
       "      <td>195.455063</td>\n",
       "      <td>194.688934</td>\n",
       "      <td>0.000583</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1994-08-07 18:32:07.967</th>\n",
       "      <td>195.117310</td>\n",
       "      <td>195.455002</td>\n",
       "      <td>194.688812</td>\n",
       "      <td>0.000579</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1994-08-07 18:33:07.967</th>\n",
       "      <td>195.115753</td>\n",
       "      <td>195.453049</td>\n",
       "      <td>194.688354</td>\n",
       "      <td>0.000526</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         WaterLevel:1  WaterLevel:100l1:0  \\\n",
       "1994-08-07 18:30:07.967    195.119919          195.455109   \n",
       "1994-08-07 18:31:07.967    195.118607          195.455063   \n",
       "1994-08-07 18:32:07.967    195.117310          195.455002   \n",
       "1994-08-07 18:33:07.967    195.115753          195.453049   \n",
       "\n",
       "                         WaterLevel:100l1:47.6827  Discharge:100l1:23.8414  \n",
       "1994-08-07 18:30:07.967                194.689072                 0.000588  \n",
       "1994-08-07 18:31:07.967                194.688934                 0.000583  \n",
       "1994-08-07 18:32:07.967                194.688812                 0.000579  \n",
       "1994-08-07 18:33:07.967                194.688354                 0.000526  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res.read().head()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}