{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Some examples of how to use the SEVIR generators\n", "\n", "The SEVIR dataset is approximately 1TB in size and thus is too large to load directly into memory. This tutorial shows the `SEVIRGenerator` class which can be used to generate samples from SEVIR.\n", "\n", "SEVIR is based on HDF5 data files. In general, streaming data directly from these files into a model for training is quite slow if you are doing a lot of random reads from the HDF files. It is recommended that you first rewrite your desired dataset into contiguous blocks using `SEVIRGenerator` before training.\n", "\n", "For a more general introduction to SEVIR, see the SEVIR_Tutorial notebook also in this directory.\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "%load_ext autoreload\n", "%autoreload 2\n", "import warnings\n", "warnings.filterwarnings(\"ignore\")" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# Make sure you add SEVIR module to your path\n", "import sys\n", "sys.path.append('..') # enter path to sevir module if not installed." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "# A keras.Sequece class for SEVIR\n", "import numpy as np\n", "from sevir.generator import SEVIRGenerator" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Get sequences from SEVIR" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "# Start by extracting just VIL sequences\n", "# (The sequence generator typically takes several seconds to initialize because it is busy parsing the SEVIR catalog)\n", "vil_seq = SEVIRGenerator(x_img_types=['vil'],batch_size=16)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1259\n" ] } ], "source": [ "# See how many batches of movie samples are available\n", "# The total number of movies is this times the batch_size\n", "print(vil_seq.__len__())" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(16, 384, 384, 49)" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Get a batch\n", "X = vil_seq.get_batch(1234) # returns list the same size as x_img_types passed to constructor\n", "X[0].shape" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "# View some frames\n", "import matplotlib.pyplot as plt\n", "from sevir.display import get_cmap\n", "fig,axs=plt.subplots(1,5,figsize=(15,5))\n", "cmap,norm,vmin,vmax = get_cmap('vil')\n", "for i in [0,10,20,30,40]:\n", " axs[i//10].imshow( X[0][0,:,:,i],cmap=cmap,norm=norm,vmin=vmin,vmax=vmax)\n", " axs[i//10].set_xticks([], [])\n", " axs[i//10].set_yticks([], []) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Get metadata of patch\n", "\n", "To get information about the SEVIR events in a batch, including the `event_id`, timestamp, and georeferencing information, you can also get the metadata" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "X,meta = vil_seq.get_batch(1234,return_meta=True)" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", " | id | \n", "time_utc | \n", "episode_id | \n", "event_id | \n", "event_type | \n", "minute_offsets | \n", "llcrnrlat | \n", "llcrnrlon | \n", "urcrnrlat | \n", "urcrnrlon | \n", "proj | \n", "height_m | \n", "width_m | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
50080 | \n", "S842418 | \n", "2019-09-05 22:10:00 | \n", "140025.0 | \n", "842418.0 | \n", "Hail | \n", "-120:-115:-110:-105:-100:-95:-90:-85:-80:-75:-... | \n", "40.637885 | \n", "-124.091295 | \n", "44.844771 | \n", "-120.838130 | \n", "+proj=laea +lat_0=38 +lon_0=-98 +units=m +a=63... | \n", "384000.0 | \n", "384000.0 | \n", "
18449 | \n", "S842458 | \n", "2019-07-20 22:43:00 | \n", "140026.0 | \n", "842458.0 | \n", "Thunderstorm Wind | \n", "-119:-114:-109:-104:-99:-94:-89:-84:-79:-74:-6... | \n", "39.691114 | \n", "-95.943051 | \n", "42.971940 | \n", "-91.114084 | \n", "+proj=laea +lat_0=38 +lon_0=-98 +units=m +a=63... | \n", "384000.0 | \n", "384000.0 | \n", "
18232 | \n", "S842585 | \n", "2019-07-30 20:00:00 | \n", "139866.0 | \n", "842585.0 | \n", "Thunderstorm Wind | \n", "-121:-116:-111:-106:-101:-96:-91:-86:-81:-76:-... | \n", "40.660904 | \n", "-78.422998 | \n", "43.119147 | \n", "-72.692830 | \n", "+proj=laea +lat_0=38 +lon_0=-98 +units=m +a=63... | \n", "384000.0 | \n", "384000.0 | \n", "
18187 | \n", "S842595 | \n", "2019-07-07 19:30:00 | \n", "140055.0 | \n", "842595.0 | \n", "Thunderstorm Wind | \n", "-121:-116:-111:-106:-101:-96:-91:-86:-81:-76:-... | \n", "31.239502 | \n", "-92.072786 | \n", "34.385802 | \n", "-87.645415 | \n", "+proj=laea +lat_0=38 +lon_0=-98 +units=m +a=63... | \n", "384000.0 | \n", "384000.0 | \n", "
17581 | \n", "S842645 | \n", "2019-08-01 19:09:00 | \n", "140060.0 | \n", "842645.0 | \n", "Hail | \n", "-120:-115:-110:-105:-100:-95:-90:-85:-80:-75:-... | \n", "46.080509 | \n", "-105.773773 | \n", "49.750429 | \n", "-100.991513 | \n", "+proj=laea +lat_0=38 +lon_0=-98 +units=m +a=63... | \n", "384000.0 | \n", "384000.0 | \n", "