{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# VirtualDatasets\n",
    "\n",
    "A VirtualDataset in ConX allows you to load just part of a dataset at a time so that you don't require all of the data to be in memory at once.\n",
    "\n",
    "You can construct a VirtualDataset as follows:\n",
    "\n",
    "```python\n",
    "cx.VirtualDataset(FUNCTION, \n",
    "                  LENGTH, \n",
    "                  INPUTS_SHAPES, \n",
    "                  TARGET_SHAPES, \n",
    "                  INPUT_RANGES, \n",
    "                  TARGET_RANGES,\n",
    "                  generator_ordered=True|False,\n",
    "                  load_cache_direct=True|False,\n",
    "                  cache_size=SIZE)\n",
    "```\n",
    "\n",
    "Where:\n",
    "\n",
    "* FUNCTION is a:\n",
    "  * a generator that yields a input/target pair\n",
    "  * a generator that yields a low-level set of batch data (use load_cache_direct=True)\n",
    "  * a function that returns an input/target pair\n",
    "  * a function that returns a low-level set of batch data (use load_cache_direct=True)\n",
    "* LENGTH is total count of data\n",
    "* INPUTS_SHAPES is shape of input banks\n",
    "* TARGET_SHAPES is shape of target banks\n",
    "* INPUT_RANGES is (min, max) for each input banks\n",
    "* TARGET_RANGES is (min, max) for each target banks\n",
    "* SIZE is size of batch\n",
    "\n",
    "The SIZE determines how many input/target pairs are generated at a time. Usually this should match the cache_size used in the training of the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n",
      "ConX, version 3.7.5\n"
     ]
    }
   ],
   "source": [
    "import conx as cx\n",
    "import numpy as np\n",
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "cache_size = 8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##   load_cache_direct = False, with function(position)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(self, pos):\n",
    "    print(\"Generating position:\", pos)\n",
    "    return ([pos/100, pos/100], [pos/100])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating position: 50\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([0.5, 0.5], [0.5])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f(None, 50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating position: 0\n",
      "Generating position: 1\n",
      "Generating position: 2\n",
      "Generating position: 3\n",
      "Generating position: 4\n",
      "Generating position: 5\n",
      "Generating position: 6\n",
      "Generating position: 7\n"
     ]
    }
   ],
   "source": [
    "dataset = cx.VirtualDataset(f, 100, [(2,)], [(1,)], [(0,1)], [(0,1)],\n",
    "                            load_cache_direct=False,\n",
    "                            cache_size=cache_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.0, 0.0]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.0, 0.0]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see from the above, retrieving input/target patterns from the current batch does not regenerate the batch.\n",
    "\n",
    "However, moving beyond the range does generate a new batch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating position: 48\n",
      "Generating position: 49\n",
      "Generating position: 50\n",
      "Generating position: 51\n",
      "Generating position: 52\n",
      "Generating position: 53\n",
      "Generating position: 54\n",
      "Generating position: 55\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.5, 0.5]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[50]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating position: 0\n",
      "Generating position: 1\n",
      "Generating position: 2\n",
      "Generating position: 3\n",
      "Generating position: 4\n",
      "Generating position: 5\n",
      "Generating position: 6\n",
      "Generating position: 7\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.0, 0.0]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## load_cache_direct = True, with function(batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(self, batch):\n",
    "    print(\"Generating batch:\", batch)\n",
    "    i = batch * cache_size\n",
    "    while True:\n",
    "        all_inputs = [[]]\n",
    "        all_targets = [[]]\n",
    "        while i < (batch + 1) * cache_size:\n",
    "            all_inputs[0].append([i/100, i/100])\n",
    "            all_targets[0].append([i/100])\n",
    "            i += 1\n",
    "        return ([np.array(inputs) for inputs in all_inputs],\n",
    "                [np.array(targets) for targets in all_targets])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating batch: 0\n"
     ]
    }
   ],
   "source": [
    "dataset = cx.VirtualDataset(f, 100, [(2,)], [(1,)], [(0,1)], [(0,1)],\n",
    "                            cache_size=cache_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.0, 0.0]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating batch: 6\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.5, 0.5]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[50]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating batch: 0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.0, 0.0]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.0, 0.0]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## generator_ordered = True, load_cache_direct = False, with generator function()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(self):\n",
    "    i = 0\n",
    "    while True:\n",
    "        print(\"Generating position:\", i)\n",
    "        yield ([i/100, i/100], [i/100])\n",
    "        i += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating position: 0\n",
      "Generating position: 1\n",
      "Generating position: 2\n",
      "Generating position: 3\n",
      "Generating position: 4\n",
      "Generating position: 5\n",
      "Generating position: 6\n",
      "Generating position: 7\n"
     ]
    }
   ],
   "source": [
    "dataset = cx.VirtualDataset(f, 100, [(2,)], [(1,)], [(0,1)], [(0,1)],\n",
    "                            generator_ordered=True,\n",
    "                            load_cache_direct=False,\n",
    "                            cache_size=cache_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.0, 0.0]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating position: 0\n",
      "Generating position: 1\n",
      "Generating position: 2\n",
      "Generating position: 3\n",
      "Generating position: 4\n",
      "Generating position: 5\n",
      "Generating position: 6\n",
      "Generating position: 7\n",
      "Generating position: 8\n",
      "Generating position: 9\n",
      "Generating position: 10\n",
      "Generating position: 11\n",
      "Generating position: 12\n",
      "Generating position: 13\n",
      "Generating position: 14\n",
      "Generating position: 15\n",
      "Generating position: 16\n",
      "Generating position: 17\n",
      "Generating position: 18\n",
      "Generating position: 19\n",
      "Generating position: 20\n",
      "Generating position: 21\n",
      "Generating position: 22\n",
      "Generating position: 23\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.20000000298023224, 0.20000000298023224]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating position: 24\n",
      "Generating position: 25\n",
      "Generating position: 26\n",
      "Generating position: 27\n",
      "Generating position: 28\n",
      "Generating position: 29\n",
      "Generating position: 30\n",
      "Generating position: 31\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.23999999463558197, 0.23999999463558197]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[24]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating position: 0\n",
      "Generating position: 1\n",
      "Generating position: 2\n",
      "Generating position: 3\n",
      "Generating position: 4\n",
      "Generating position: 5\n",
      "Generating position: 6\n",
      "Generating position: 7\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.0, 0.0]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## generator_ordered = True, load_cache_direct = True, with generator function()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(self):\n",
    "    i = 0\n",
    "    while True:\n",
    "        print(\"Generating positions:\", i, \"-\", i + cache_size)\n",
    "        i_end = i + cache_size\n",
    "        all_inputs = [[]]\n",
    "        all_targets = [[]]\n",
    "        while i < i_end:\n",
    "            all_inputs[0].append([i/100, i/100])\n",
    "            all_targets[0].append([i/100])\n",
    "            i += 1\n",
    "        yield ([np.array(inputs) for inputs in all_inputs],\n",
    "               [np.array(targets) for targets in all_targets])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating positions: 0 - 8\n"
     ]
    }
   ],
   "source": [
    "dataset = cx.VirtualDataset(f, 100, [(2,)], [(1,)], [(0,1)], [(0,1)],\n",
    "                            generator_ordered=True,\n",
    "                            load_cache_direct=True,\n",
    "                            cache_size=cache_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.0, 0.0]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating positions: 0 - 8\n",
      "Generating positions: 8 - 16\n",
      "Generating positions: 16 - 24\n",
      "Generating positions: 24 - 32\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.25, 0.25]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[25]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating positions: 0 - 8\n",
      "Generating positions: 8 - 16\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.08, 0.08]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[8]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## generator_ordered = True, load_cache_direct = False, with generator function() (showing another function style)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(self):\n",
    "    for i in range(100):\n",
    "        print(\"generating position:\", i)\n",
    "        yield ([i/100, i/100], [i/100])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "generating position: 0\n",
      "generating position: 1\n",
      "generating position: 2\n",
      "generating position: 3\n",
      "generating position: 4\n",
      "generating position: 5\n",
      "generating position: 6\n",
      "generating position: 7\n"
     ]
    }
   ],
   "source": [
    "dataset = cx.VirtualDataset(f, 100, [(2,)], [(1,)], [(0,1)], [(0,1)],\n",
    "                            generator_ordered=True,\n",
    "                            load_cache_direct=False,\n",
    "                            cache_size=cache_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.0, 0.0]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "generating position: 0\n",
      "generating position: 1\n",
      "generating position: 2\n",
      "generating position: 3\n",
      "generating position: 4\n",
      "generating position: 5\n",
      "generating position: 6\n",
      "generating position: 7\n",
      "generating position: 8\n",
      "generating position: 9\n",
      "generating position: 10\n",
      "generating position: 11\n",
      "generating position: 12\n",
      "generating position: 13\n",
      "generating position: 14\n",
      "generating position: 15\n",
      "generating position: 16\n",
      "generating position: 17\n",
      "generating position: 18\n",
      "generating position: 19\n",
      "generating position: 20\n",
      "generating position: 21\n",
      "generating position: 22\n",
      "generating position: 23\n",
      "generating position: 24\n",
      "generating position: 25\n",
      "generating position: 26\n",
      "generating position: 27\n",
      "generating position: 28\n",
      "generating position: 29\n",
      "generating position: 30\n",
      "generating position: 31\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.25, 0.25]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[25]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "generating position: 0\n",
      "generating position: 1\n",
      "generating position: 2\n",
      "generating position: 3\n",
      "generating position: 4\n",
      "generating position: 5\n",
      "generating position: 6\n",
      "generating position: 7\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.0, 0.0]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## generator_ordered = False, load_cache_direct = False, with generator function()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(self):\n",
    "    while True:\n",
    "        print(\"Generating a position!\")\n",
    "        r = random.random()\n",
    "        yield ([r, r], [r])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating a position!\n",
      "Generating a position!\n",
      "Generating a position!\n",
      "Generating a position!\n",
      "Generating a position!\n",
      "Generating a position!\n",
      "Generating a position!\n",
      "Generating a position!\n"
     ]
    }
   ],
   "source": [
    "dataset = cx.VirtualDataset(f, 100, [(2,)], [(1,)], [(0,1)], [(0,1)],\n",
    "                            generator_ordered=False,\n",
    "                            load_cache_direct=False,\n",
    "                            cache_size=cache_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.7236223220825195, 0.7236223220825195]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.7236223220825195, 0.7236223220825195]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating a position!\n",
      "Generating a position!\n",
      "Generating a position!\n",
      "Generating a position!\n",
      "Generating a position!\n",
      "Generating a position!\n",
      "Generating a position!\n",
      "Generating a position!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.661766767501831, 0.661766767501831]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating a position!\n",
      "Generating a position!\n",
      "Generating a position!\n",
      "Generating a position!\n",
      "Generating a position!\n",
      "Generating a position!\n",
      "Generating a position!\n",
      "Generating a position!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.2500929534435272, 0.2500929534435272]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## generator_ordered = False, load_cache_direct = True, with generator function()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(self):\n",
    "    while True:\n",
    "        print(\"Generating a batch!\")\n",
    "        all_inputs = [[]]\n",
    "        all_targets = [[]]\n",
    "        for i in range(cache_size):\n",
    "            r = random.random()\n",
    "            all_inputs[0].append([r, r])\n",
    "            all_targets[0].append([r])\n",
    "        yield ([np.array(inputs) for inputs in all_inputs],\n",
    "               [np.array(targets) for targets in all_targets])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating a batch!\n"
     ]
    }
   ],
   "source": [
    "dataset = cx.VirtualDataset(f, 100, [(2,)], [(1,)], [(0,1)], [(0,1)],\n",
    "                            generator_ordered=False,\n",
    "                            load_cache_direct=True,\n",
    "                            cache_size=cache_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.5143756422352775, 0.5143756422352775]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating a batch!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.14642449150160008, 0.14642449150160008]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[50]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating a batch!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.5630570620543651, 0.5630570620543651]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating a batch!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.5517772337207292, 0.5517772337207292]"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[99]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating a VirtualDataset from a directory of files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting test0.dat\n"
     ]
    }
   ],
   "source": [
    "%%file test0.dat\n",
    "[[0/3], [0/3]], [0/3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting test1.dat\n"
     ]
    }
   ],
   "source": [
    "%%file test1.dat\n",
    "[[1/3], [1/3]], [1/3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting test2.dat\n"
     ]
    }
   ],
   "source": [
    "%%file test2.dat\n",
    "[[2/3], [2/3]], [2/3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting test3.dat\n"
     ]
    }
   ],
   "source": [
    "%%file test3.dat\n",
    "[[3/3], [3/3]], [3/3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "filenames = sorted(glob.glob(\"./*.dat\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(self, pos):\n",
    "    # To get a specific order, always number from beginning:\n",
    "    print(\"Generating position:\", pos)\n",
    "    return eval(open(filenames[pos]).read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['./test0.dat', './test1.dat', './test2.dat', './test3.dat']"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filenames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating position: 0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([[0.0], [0.0]], [0.0])"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f(None, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating position: 0\n",
      "Generating position: 1\n",
      "Generating position: 2\n"
     ]
    }
   ],
   "source": [
    "dataset = cx.VirtualDataset(f, len(filenames), [(2,)], [(1,)], [(0,1)], [(0,1)],\n",
    "                            load_cache_direct=False,\n",
    "                            cache_size=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[0.0], [0.0]]"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[0.0], [0.0]]"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating position: 3\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[[1.0], [1.0]]"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.inputs[3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}