{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NLP datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [],
   "source": [
    "from fastai.gen_doc.nbdoc import *\n",
    "from fastai.text import * \n",
    "from fastai.gen_doc.nbdoc import *\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This module contains the [`TextDataset`](/text.data.html#TextDataset) class, which is the main dataset you should use for your NLP tasks. It automatically does the preprocessing steps described in [`text.transform`](/text.transform.html#text.transform). It also contains all the functions to quickly get a [`TextDataBunch`](/text.data.html#TextDataBunch) ready."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quickly assemble your data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should get your data in one of the following formats to make the most of the fastai library and use one of the factory methods of one of the [`TextDataBunch`](/text.data.html#TextDataBunch) classes:\n",
    "- raw text files in folders train, valid, test in an ImageNet style,\n",
    "- a csv where some column(s) gives the label(s) and the following one the associated text,\n",
    "- a dataframe structured the same way,\n",
    "- tokens and labels arrays,\n",
    "- ids, vocabulary (correspondence id to word) and labels.\n",
    "\n",
    "If you are assembling the data for a language model, you should define your labels as always 0 to respect those formats. The first time you create a [`DataBunch`](/basic_data.html#DataBunch) with one of those functions, your data will be preprocessed automatically. You can save it, so that the next time you call it is almost instantaneous. \n",
    "\n",
    "Below are the classes that help assembling the raw data in a [`DataBunch`](/basic_data.html#DataBunch) suitable for NLP."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h3 id=\"TextLMDataBunch\"><code>class</code> <code>TextLMDataBunch</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L244\" class=\"source_link\">[source]</a></h3>\n",
       "\n",
       "> <code>TextLMDataBunch</code>(**`train_dl`**:[`DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader), **`valid_dl`**:[`DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader), **`fix_dl`**:[`DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader)=***`None`***, **`test_dl`**:`Optional`\\[[`DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader)\\]=***`None`***, **`device`**:[`device`](https://pytorch.org/docs/stable/tensor_attributes.html#torch-device)=***`None`***, **`dl_tfms`**:`Optional`\\[`Collection`\\[`Callable`\\]\\]=***`None`***, **`path`**:`PathOrStr`=***`'.'`***, **`collate_fn`**:`Callable`=***`'data_collate'`***, **`no_check`**:`bool`=***`False`***) :: [`TextDataBunch`](/text.data.html#TextDataBunch)\n",
       "\n",
       "Create a [`TextDataBunch`](/text.data.html#TextDataBunch) suitable for training a language model.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TextLMDataBunch, title_level=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the texts in the [`datasets`](/datasets.html#datasets) are concatenated and the labels are ignored. Instead, the target is the next word in the sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"TextLMDataBunch.create\"><code>create</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L246\" class=\"source_link\">[source]</a></h4>\n",
       "\n",
       "> <code>create</code>(**`train_ds`**, **`valid_ds`**, **`test_ds`**=***`None`***, **`path`**:`PathOrStr`=***`'.'`***, **`no_check`**:`bool`=***`False`***, **`bs`**=***`64`***, **`val_bs`**:`int`=***`None`***, **`num_workers`**:`int`=***`0`***, **`device`**:[`device`](https://pytorch.org/docs/stable/tensor_attributes.html#torch-device)=***`None`***, **`collate_fn`**:`Callable`=***`'data_collate'`***, **`dl_tfms`**:`Optional`\\[`Collection`\\[`Callable`\\]\\]=***`None`***, **`bptt`**:`int`=***`70`***, **`backwards`**:`bool`=***`False`***) → [`DataBunch`](/basic_data.html#DataBunch)\n",
       "\n",
       "Create a [`TextDataBunch`](/text.data.html#TextDataBunch) in `path` from the `datasets` for language modelling.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TextLMDataBunch.create)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h3 id=\"TextClasDataBunch\"><code>class</code> <code>TextClasDataBunch</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L259\" class=\"source_link\">[source]</a></h3>\n",
       "\n",
       "> <code>TextClasDataBunch</code>(**`train_dl`**:[`DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader), **`valid_dl`**:[`DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader), **`fix_dl`**:[`DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader)=***`None`***, **`test_dl`**:`Optional`\\[[`DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader)\\]=***`None`***, **`device`**:[`device`](https://pytorch.org/docs/stable/tensor_attributes.html#torch-device)=***`None`***, **`dl_tfms`**:`Optional`\\[`Collection`\\[`Callable`\\]\\]=***`None`***, **`path`**:`PathOrStr`=***`'.'`***, **`collate_fn`**:`Callable`=***`'data_collate'`***, **`no_check`**:`bool`=***`False`***) :: [`TextDataBunch`](/text.data.html#TextDataBunch)\n",
       "\n",
       "Create a [`TextDataBunch`](/text.data.html#TextDataBunch) suitable for training an RNN classifier.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TextClasDataBunch, title_level=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"TextClasDataBunch.create\"><code>create</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L261\" class=\"source_link\">[source]</a></h4>\n",
       "\n",
       "> <code>create</code>(**`train_ds`**, **`valid_ds`**, **`test_ds`**=***`None`***, **`path`**:`PathOrStr`=***`'.'`***, **`bs`**:`int`=***`32`***, **`val_bs`**:`int`=***`None`***, **`pad_idx`**=***`1`***, **`pad_first`**=***`True`***, **`device`**:[`device`](https://pytorch.org/docs/stable/tensor_attributes.html#torch-device)=***`None`***, **`no_check`**:`bool`=***`False`***, **`backwards`**:`bool`=***`False`***, **\\*\\*`kwargs`**) → [`DataBunch`](/basic_data.html#DataBunch)\n",
       "\n",
       "Function that transform the `datasets` in a [`DataBunch`](/basic_data.html#DataBunch) for classification.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TextClasDataBunch.create)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the texts are grouped by length (with a bit of randomness for the training set) then padded so that the samples have the same length to get in a batch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h3 id=\"TextDataBunch\"><code>class</code> <code>TextDataBunch</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L146\" class=\"source_link\">[source]</a></h3>\n",
       "\n",
       "> <code>TextDataBunch</code>(**`train_dl`**:[`DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader), **`valid_dl`**:[`DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader), **`fix_dl`**:[`DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader)=***`None`***, **`test_dl`**:`Optional`\\[[`DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader)\\]=***`None`***, **`device`**:[`device`](https://pytorch.org/docs/stable/tensor_attributes.html#torch-device)=***`None`***, **`dl_tfms`**:`Optional`\\[`Collection`\\[`Callable`\\]\\]=***`None`***, **`path`**:`PathOrStr`=***`'.'`***, **`collate_fn`**:`Callable`=***`'data_collate'`***, **`no_check`**:`bool`=***`False`***) :: [`DataBunch`](/basic_data.html#DataBunch)\n",
       "\n",
       "General class to get a [`DataBunch`](/basic_data.html#DataBunch) for NLP. Subclassed by [`TextLMDataBunch`](/text.data.html#TextLMDataBunch) and [`TextClasDataBunch`](/text.data.html#TextClasDataBunch).  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TextDataBunch, title_level=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<div markdown=\"span\" class=\"alert alert-danger\" role=\"alert\"><i class=\"fa fa-danger-circle\"></i> <b>Warning: </b>This class can only work directly if all the texts have the same length.</div>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "jekyll_warn(\"This class can only work directly if all the texts have the same length.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Factory methods (TextDataBunch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All those classes have the following factory methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"TextDataBunch.from_folder\"><code>from_folder</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L230\" class=\"source_link\">[source]</a></h4>\n",
       "\n",
       "> <code>from_folder</code>(**`path`**:`PathOrStr`, **`train`**:`str`=***`'train'`***, **`valid`**:`str`=***`'valid'`***, **`test`**:`Optional`\\[`str`\\]=***`None`***, **`classes`**:`ArgStar`=***`None`***, **`tokenizer`**:[`Tokenizer`](/text.transform.html#Tokenizer)=***`None`***, **`vocab`**:[`Vocab`](/text.transform.html#Vocab)=***`None`***, **`chunksize`**:`int`=***`10000`***, **`max_vocab`**:`int`=***`60000`***, **`min_freq`**:`int`=***`2`***, **`mark_fields`**:`bool`=***`False`***, **\\*\\*`kwargs`**)\n",
       "\n",
       "Create a [`TextDataBunch`](/text.data.html#TextDataBunch) from text files in folders.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TextDataBunch.from_folder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The floders are scanned in `path` with a <code>train</code>, `valid` and maybe `test` folders. Text files in the <code>train</code> and `valid` folders should be places in subdirectories according to their classes (not applicable for a language model). `tokenizer` will be used to parse those texts into tokens.\n",
    "\n",
    "You can pass a specific `vocab` for the numericalization step (if you are building a classifier from a language model you fine-tuned for instance). kwargs will be split between the [`TextDataset`](/text.data.html#TextDataset) function and to the class initialization, you can precise there parameters such as `max_vocab`, `chunksize`, `min_freq`, `n_labels` (see the [`TextDataset`](/text.data.html#TextDataset) documentation) or `bs`, `bptt` and `pad_idx` (see the sections LM data and classifier data)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"TextDataBunch.from_csv\"><code>from_csv</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L215\" class=\"source_link\">[source]</a></h4>\n",
       "\n",
       "> <code>from_csv</code>(**`path`**:`PathOrStr`, **`csv_name`**, **`valid_pct`**:`float`=***`0.2`***, **`test`**:`Optional`\\[`str`\\]=***`None`***, **`tokenizer`**:[`Tokenizer`](/text.transform.html#Tokenizer)=***`None`***, **`vocab`**:[`Vocab`](/text.transform.html#Vocab)=***`None`***, **`classes`**:`StrList`=***`None`***, **`header`**=***`'infer'`***, **`text_cols`**:`IntsOrStrs`=***`1`***, **`label_cols`**:`IntsOrStrs`=***`0`***, **`label_delim`**:`str`=***`None`***, **`chunksize`**:`int`=***`10000`***, **`max_vocab`**:`int`=***`60000`***, **`min_freq`**:`int`=***`2`***, **`mark_fields`**:`bool`=***`False`***, **\\*\\*`kwargs`**) → [`DataBunch`](/basic_data.html#DataBunch)\n",
       "\n",
       "Create a [`TextDataBunch`](/text.data.html#TextDataBunch) from texts in csv files. `kwargs` are passed to the dataloader creation.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TextDataBunch.from_csv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This method will look for `csv_name` in  `path`, and maybe a `test` csv file opened with `header`. You can specify `text_cols` and `label_cols`. If there are several `text_cols`, the texts will be concatenated together with an optional field token. If there are several `label_cols`, the labels will be assumed to be one-hot encoded and `classes` will default to `label_cols` (you can ignore that argument for a language model). `tokenizer` will be used to parse those texts into tokens.\n",
    "\n",
    "You can pass a specific `vocab` for the numericalization step (if you are building a classifier from a language model you fine-tuned for instance). kwargs will be split between the [`TextDataset`](/text.data.html#TextDataset) function and to the class initialization, you can precise there parameters such as `max_vocab`, `chunksize`, `min_freq`, `n_labels` (see the [`TextDataset`](/text.data.html#TextDataset) documentation) or `bs`, `bptt` and `pad_idx` (see the sections LM data and classifier data)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"TextDataBunch.from_df\"><code>from_df</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L199\" class=\"source_link\">[source]</a></h4>\n",
       "\n",
       "> <code>from_df</code>(**`path`**:`PathOrStr`, **`train_df`**:`DataFrame`, **`valid_df`**:`DataFrame`, **`test_df`**:`OptDataFrame`=***`None`***, **`tokenizer`**:[`Tokenizer`](/text.transform.html#Tokenizer)=***`None`***, **`vocab`**:[`Vocab`](/text.transform.html#Vocab)=***`None`***, **`classes`**:`StrList`=***`None`***, **`text_cols`**:`IntsOrStrs`=***`1`***, **`label_cols`**:`IntsOrStrs`=***`0`***, **`label_delim`**:`str`=***`None`***, **`chunksize`**:`int`=***`10000`***, **`max_vocab`**:`int`=***`60000`***, **`min_freq`**:`int`=***`2`***, **`mark_fields`**:`bool`=***`False`***, **\\*\\*`kwargs`**) → [`DataBunch`](/basic_data.html#DataBunch)\n",
       "\n",
       "Create a [`TextDataBunch`](/text.data.html#TextDataBunch) from DataFrames. `kwargs` are passed to the dataloader creation.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TextDataBunch.from_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This method will use `train_df`, `valid_df` and maybe `test_df` to build the [`TextDataBunch`](/text.data.html#TextDataBunch) in `path`. You can specify `text_cols` and `label_cols`. If there are several `text_cols`, the texts will be concatenated together with an optional field token. If there are several `label_cols`, the labels will be assumed to be one-hot encoded and `classes` will default to `label_cols` (you can ignore that argument for a language model). `tokenizer` will be used to parse those texts into tokens.\n",
    "\n",
    "You can pass a specific `vocab` for the numericalization step (if you are building a classifier from a language model you fine-tuned for instance). kwargs will be split between the [`TextDataset`](/text.data.html#TextDataset) function and to the class initialization, you can precise there parameters such as `max_vocab`, `chunksize`, `min_freq`, `n_labels` (see the [`TextDataset`](/text.data.html#TextDataset) documentation) or `bs`, `bptt` and `pad_idx` (see the sections LM data and classifier data)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"TextDataBunch.from_tokens\"><code>from_tokens</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L186\" class=\"source_link\">[source]</a></h4>\n",
       "\n",
       "> <code>from_tokens</code>(**`path`**:`PathOrStr`, **`trn_tok`**:`Tokens`, **`trn_lbls`**:`Collection`\\[`Union`\\[`int`, `float`\\]\\], **`val_tok`**:`Tokens`, **`val_lbls`**:`Collection`\\[`Union`\\[`int`, `float`\\]\\], **`vocab`**:[`Vocab`](/text.transform.html#Vocab)=***`None`***, **`tst_tok`**:`Tokens`=***`None`***, **`classes`**:`ArgStar`=***`None`***, **`max_vocab`**:`int`=***`60000`***, **`min_freq`**:`int`=***`3`***, **\\*\\*`kwargs`**) → [`DataBunch`](/basic_data.html#DataBunch)\n",
       "\n",
       "Create a [`TextDataBunch`](/text.data.html#TextDataBunch) from tokens and labels. `kwargs` are passed to the dataloader creation.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TextDataBunch.from_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function will create a [`DataBunch`](/basic_data.html#DataBunch) from `trn_tok`, `trn_lbls`, `val_tok`, `val_lbls` and maybe `tst_tok`.\n",
    "\n",
    "You can pass a specific `vocab` for the numericalization step (if you are building a classifier from a language model you fine-tuned for instance). kwargs will be split between the [`TextDataset`](/text.data.html#TextDataset) function and to the class initialization, you can precise there parameters such as `max_vocab`, `chunksize`, `min_freq`, `n_labels`, `tok_suff` and `lbl_suff` (see the [`TextDataset`](/text.data.html#TextDataset) documentation) or `bs`, `bptt` and `pad_idx` (see the sections LM data and classifier data)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"TextDataBunch.from_ids\"><code>from_ids</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L161\" class=\"source_link\">[source]</a></h4>\n",
       "\n",
       "> <code>from_ids</code>(**`path`**:`PathOrStr`, **`vocab`**:[`Vocab`](/text.transform.html#Vocab), **`train_ids`**:`Collection`\\[`Collection`\\[`int`\\]\\], **`valid_ids`**:`Collection`\\[`Collection`\\[`int`\\]\\], **`test_ids`**:`Collection`\\[`Collection`\\[`int`\\]\\]=***`None`***, **`train_lbls`**:`Collection`\\[`Union`\\[`int`, `float`\\]\\]=***`None`***, **`valid_lbls`**:`Collection`\\[`Union`\\[`int`, `float`\\]\\]=***`None`***, **`classes`**:`ArgStar`=***`None`***, **`processor`**:[`PreProcessor`](/data_block.html#PreProcessor)=***`None`***, **\\*\\*`kwargs`**) → [`DataBunch`](/basic_data.html#DataBunch)\n",
       "\n",
       "Create a [`TextDataBunch`](/text.data.html#TextDataBunch) from ids, labels and a `vocab`. `kwargs` are passed to the dataloader creation.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TextDataBunch.from_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Texts are already preprocessed into `train_ids`, `train_lbls`, `valid_ids`, `valid_lbls` and maybe `test_ids`. You can specify the corresponding `classes` if applicable. You must specify a `path` and the `vocab` so that the [`RNNLearner`](/text.learner.html#RNNLearner) class can later infer the corresponding sizes in the model it will create. kwargs will be passed to the class initialization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load and save"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To avoid losing time preprocessing the text data more than once, you should save/load your [`TextDataBunch`](/text.data.html#TextDataBunch) using thse methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"TextDataBunch.load\"><code>load</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L175\" class=\"source_link\">[source]</a></h4>\n",
       "\n",
       "> <code>load</code>(**`path`**:`PathOrStr`, **`cache_name`**:`PathOrStr`=***`'tmp'`***, **`processor`**:[`PreProcessor`](/data_block.html#PreProcessor)=***`None`***, **\\*\\*`kwargs`**)\n",
       "\n",
       "Load a [`TextDataBunch`](/text.data.html#TextDataBunch) from `path/cache_name`. `kwargs` are passed to the dataloader creation.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TextDataBunch.load)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"TextDataBunch.save\"><code>save</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L149\" class=\"source_link\">[source]</a></h4>\n",
       "\n",
       "> <code>save</code>(**`cache_name`**:`PathOrStr`=***`'tmp'`***)\n",
       "\n",
       "Save the [`DataBunch`](/basic_data.html#DataBunch) in `self.path/cache_name` folder.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TextDataBunch.save)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Untar the IMDB sample dataset if not already done:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PosixPath('/home/ubuntu/.fastai/data/imdb_sample')"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = untar_data(URLs.IMDB_SAMPLE)\n",
    "path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since it comes in the form of csv files, we will use the corresponding `text_data` method. Here is an overview of what your file you should look like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>text</th>\n",
       "      <th>is_valid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>negative</td>\n",
       "      <td>Un-bleeping-believable! Meg Ryan doesn't even ...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>positive</td>\n",
       "      <td>This is a extremely well-made film. The acting...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>negative</td>\n",
       "      <td>Every once in a long while a movie will come a...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>positive</td>\n",
       "      <td>Name just says it all. I watched this movie wi...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>negative</td>\n",
       "      <td>This movie succeeds at being one of the most u...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      label                                               text  is_valid\n",
       "0  negative  Un-bleeping-believable! Meg Ryan doesn't even ...     False\n",
       "1  positive  This is a extremely well-made film. The acting...     False\n",
       "2  negative  Every once in a long while a movie will come a...     False\n",
       "3  positive  Name just says it all. I watched this movie wi...     False\n",
       "4  negative  This movie succeeds at being one of the most u...     False"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv(path/'texts.csv').head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here is a simple way of creating your [`DataBunch`](/basic_data.html#DataBunch) for language modelling or classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm = TextLMDataBunch.from_csv(Path(path), 'texts.csv')\n",
    "data_clas = TextClasDataBunch.from_csv(Path(path), 'texts.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The TextList input classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Behind the scenes, the previous functions will create a training, validation and maybe test [`TextList`](/text.data.html#TextList) that will be tokenized and numericalized (if needed) using [`PreProcessor`](/data_block.html#PreProcessor)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h3 id=\"Text\"><code>class</code> <code>Text</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L281\" class=\"source_link\">[source]</a></h3>\n",
       "\n",
       "> <code>Text</code>(**`ids`**, **`text`**) :: [`ItemBase`](/core.html#ItemBase)\n",
       "\n",
       "Basic item for <code>text</code> data in numericalized `ids`.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(Text, title_level=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h3 id=\"TextList\"><code>class</code> <code>TextList</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L318\" class=\"source_link\">[source]</a></h3>\n",
       "\n",
       "> <code>TextList</code>(**`items`**:`Iterator`\\[`T_co`\\], **`vocab`**:[`Vocab`](/text.transform.html#Vocab)=***`None`***, **`pad_idx`**:`int`=***`1`***, **\\*\\*`kwargs`**) :: [`ItemList`](/data_block.html#ItemList)\n",
       "\n",
       "Basic [`ItemList`](/data_block.html#ItemList) for text data.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TextList, title_level=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`vocab` contains the correspondence between ids and tokens, `pad_idx` is the id used for padding. You can pass a custom `processor` in the `kwargs` to change the defaults for tokenization or numericalization. It should have the following form:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "processor = [TokenizeProcessor(tokenizer=SpacyTokenizer('en')), NumericalizeProcessor(max_vocab=30000)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See below for all the arguments those tokenizers can take."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"TextList.label_for_lm\"><code>label_for_lm</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L333\" class=\"source_link\">[source]</a></h4>\n",
       "\n",
       "> <code>label_for_lm</code>(**\\*\\*`kwargs`**)\n",
       "\n",
       "A special labelling method for language models.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TextList.label_for_lm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"TextList.from_folder\"><code>from_folder</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L342\" class=\"source_link\">[source]</a></h4>\n",
       "\n",
       "> <code>from_folder</code>(**`path`**:`PathOrStr`=***`'.'`***, **`extensions`**:`StrList`=***`{'.txt'}`***, **`vocab`**:[`Vocab`](/text.transform.html#Vocab)=***`None`***, **`processor`**:[`PreProcessor`](/data_block.html#PreProcessor)=***`None`***, **\\*\\*`kwargs`**) → `TextList`\n",
       "\n",
       "Get the list of files in `path` that have a text suffix. `recurse` determines if we search subfolders.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TextList.from_folder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"TextList.show_xys\"><code>show_xys</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L349\" class=\"source_link\">[source]</a></h4>\n",
       "\n",
       "> <code>show_xys</code>(**`xs`**, **`ys`**, **`max_len`**:`int`=***`70`***)\n",
       "\n",
       "Show the `xs` (inputs) and `ys` (targets). `max_len` is the maximum number of tokens displayed.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TextList.show_xys)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"TextList.show_xyzs\"><code>show_xyzs</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L358\" class=\"source_link\">[source]</a></h4>\n",
       "\n",
       "> <code>show_xyzs</code>(**`xs`**, **`ys`**, **`zs`**, **`max_len`**:`int`=***`70`***)\n",
       "\n",
       "Show `xs` (inputs), `ys` (targets) and `zs` (predictions). `max_len` is the maximum number of tokens displayed.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TextList.show_xyzs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h3 id=\"OpenFileProcessor\"><code>class</code> <code>OpenFileProcessor</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L313\" class=\"source_link\">[source]</a></h3>\n",
       "\n",
       "> <code>OpenFileProcessor</code>(**`ds`**:`Collection`\\[`T_co`\\]=***`None`***) :: [`PreProcessor`](/data_block.html#PreProcessor)\n",
       "\n",
       "[`PreProcessor`](/data_block.html#PreProcessor) that opens the filenames and read the texts.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(OpenFileProcessor, title_level=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"open_text\"><code>open_text</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L277\" class=\"source_link\">[source]</a></h4>\n",
       "\n",
       "> <code>open_text</code>(**`fn`**:`PathOrStr`, **`enc`**=***`'utf-8'`***)\n",
       "\n",
       "Read the text in `fn`.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(open_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h3 id=\"TokenizeProcessor\"><code>class</code> <code>TokenizeProcessor</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L286\" class=\"source_link\">[source]</a></h3>\n",
       "\n",
       "> <code>TokenizeProcessor</code>(**`ds`**:[`ItemList`](/data_block.html#ItemList)=***`None`***, **`tokenizer`**:[`Tokenizer`](/text.transform.html#Tokenizer)=***`None`***, **`chunksize`**:`int`=***`10000`***, **`mark_fields`**:`bool`=***`False`***) :: [`PreProcessor`](/data_block.html#PreProcessor)\n",
       "\n",
       "[`PreProcessor`](/data_block.html#PreProcessor) that tokenizes the texts in `ds`.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TokenizeProcessor, title_level=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`tokenizer` is used on bits of `chunksize`. If `mark_fields=True`, add field tokens between each parts of the texts (given when the texts are read in several columns of a dataframe). See more about tokenizers in the [transform documentation](/text.transform.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h3 id=\"NumericalizeProcessor\"><code>class</code> <code>NumericalizeProcessor</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L301\" class=\"source_link\">[source]</a></h3>\n",
       "\n",
       "> <code>NumericalizeProcessor</code>(**`ds`**:[`ItemList`](/data_block.html#ItemList)=***`None`***, **`vocab`**:[`Vocab`](/text.transform.html#Vocab)=***`None`***, **`max_vocab`**:`int`=***`60000`***, **`min_freq`**:`int`=***`3`***) :: [`PreProcessor`](/data_block.html#PreProcessor)\n",
       "\n",
       "[`PreProcessor`](/data_block.html#PreProcessor) that numericalizes the tokens in `ds`.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(NumericalizeProcessor, title_level=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Uses `vocab` for this (if not None), otherwise create one with `max_vocab` and `min_freq` from tokens."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Language Model data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A language model is trained to guess what the next word is inside a flow of words. We don't feed it the different texts separately but concatenate them all together in a big array. To create the batches, we split this array into `bs` chunks of continuous texts. Note that in all NLP tasks, we don't use the usual convention of sequence length being the first dimension so batch size is the first dimension and sequence length is the second. Here you can read the chunks of texts in lines. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxmaj</td>\n",
       "      <td>name</td>\n",
       "      <td>just</td>\n",
       "      <td>says</td>\n",
       "      <td>it</td>\n",
       "      <td>all</td>\n",
       "      <td>.</td>\n",
       "      <td>i</td>\n",
       "      <td>watched</td>\n",
       "      <td>this</td>\n",
       "      <td>movie</td>\n",
       "      <td>with</td>\n",
       "      <td>my</td>\n",
       "      <td>dad</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>going</td>\n",
       "      <td>to</td>\n",
       "      <td>see</td>\n",
       "      <td>anything</td>\n",
       "      <td>you</td>\n",
       "      <td>'ll</td>\n",
       "      <td>remember</td>\n",
       "      <td>.</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>xxmaj</td>\n",
       "      <td>evidently</td>\n",
       "      <td>when</td>\n",
       "      <td>you</td>\n",
       "      <td>offer</td>\n",
       "      <td>a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>of</td>\n",
       "      <td>the</td>\n",
       "      <td>xxunk</td>\n",
       "      <td>xxunk</td>\n",
       "      <td>xxmaj</td>\n",
       "      <td>superbly</td>\n",
       "      <td>shot</td>\n",
       "      <td>,</td>\n",
       "      <td>this</td>\n",
       "      <td>thrilling</td>\n",
       "      <td>adult</td>\n",
       "      <td>adventure</td>\n",
       "      <td>certainly</td>\n",
       "      <td>contains</td>\n",
       "      <td>some</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\n\\n</td>\n",
       "      <td>xxmaj</td>\n",
       "      <td>one</td>\n",
       "      <td>bright</td>\n",
       "      <td>light</td>\n",
       "      <td>in</td>\n",
       "      <td>the</td>\n",
       "      <td>midst</td>\n",
       "      <td>of</td>\n",
       "      <td>this</td>\n",
       "      <td>is</td>\n",
       "      <td>xxmaj</td>\n",
       "      <td>fred</td>\n",
       "      <td>xxmaj</td>\n",
       "      <td>xxunk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>leave</td>\n",
       "      <td>you</td>\n",
       "      <td>xxunk</td>\n",
       "      <td>that</td>\n",
       "      <td>you</td>\n",
       "      <td>watched</td>\n",
       "      <td>it</td>\n",
       "      <td>.</td>\n",
       "      <td>i</td>\n",
       "      <td>feel</td>\n",
       "      <td>really</td>\n",
       "      <td>bad</td>\n",
       "      <td>for</td>\n",
       "      <td>those</td>\n",
       "      <td>xxmaj</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>xxmaj</td>\n",
       "      <td>ginger</td>\n",
       "      <td>'s</td>\n",
       "      <td>blonde</td>\n",
       "      <td>hair</td>\n",
       "      <td>)</td>\n",
       "      <td>has</td>\n",
       "      <td>a</td>\n",
       "      <td>couple</td>\n",
       "      <td>of</td>\n",
       "      <td>emotional</td>\n",
       "      <td>solo</td>\n",
       "      <td>numbers</td>\n",
       "      <td>,</td>\n",
       "      <td>including</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>.</td>\n",
       "      <td>xxmaj</td>\n",
       "      <td>this</td>\n",
       "      <td>is</td>\n",
       "      <td>what</td>\n",
       "      <td>really</td>\n",
       "      <td>attracted</td>\n",
       "      <td>me</td>\n",
       "      <td>to</td>\n",
       "      <td>this</td>\n",
       "      <td>film</td>\n",
       "      <td>.</td>\n",
       "      <td>i</td>\n",
       "      <td>was</td>\n",
       "      <td>impressed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>into</td>\n",
       "      <td>the</td>\n",
       "      <td>hype</td>\n",
       "      <td>that</td>\n",
       "      <td>one</td>\n",
       "      <td>you</td>\n",
       "      <td>are</td>\n",
       "      <td>somehow</td>\n",
       "      <td>white</td>\n",
       "      <td>or</td>\n",
       "      <td>superior</td>\n",
       "      <td>...</td>\n",
       "      <td>you</td>\n",
       "      <td>are</td>\n",
       "      <td>not</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>come</td>\n",
       "      <td>unintentionally</td>\n",
       "      <td>,</td>\n",
       "      <td>like</td>\n",
       "      <td>when</td>\n",
       "      <td>they</td>\n",
       "      <td>try</td>\n",
       "      <td>to</td>\n",
       "      <td>explain</td>\n",
       "      <td>that</td>\n",
       "      <td>an</td>\n",
       "      <td>invisible</td>\n",
       "      <td>man</td>\n",
       "      <td>'s</td>\n",
       "      <td>xxunk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>just</td>\n",
       "      <td>watch</td>\n",
       "      <td>the</td>\n",
       "      <td>movie</td>\n",
       "      <td>,</td>\n",
       "      <td>and</td>\n",
       "      <td>i</td>\n",
       "      <td>dear</td>\n",
       "      <td>say</td>\n",
       "      <td>you</td>\n",
       "      <td>'ll</td>\n",
       "      <td>see</td>\n",
       "      <td>things</td>\n",
       "      <td>a</td>\n",
       "      <td>bit</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>for</td>\n",
       "      <td>the</td>\n",
       "      <td>reform</td>\n",
       "      <td>movement</td>\n",
       "      <td>and</td>\n",
       "      <td>meets</td>\n",
       "      <td>xxmaj</td>\n",
       "      <td>eponine</td>\n",
       "      <td>.</td>\n",
       "      <td>xxmaj</td>\n",
       "      <td>except</td>\n",
       "      <td>...</td>\n",
       "      <td>not</td>\n",
       "      <td>xxmaj</td>\n",
       "      <td>eponine</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>xxmaj</td>\n",
       "      <td>well</td>\n",
       "      <td>,</td>\n",
       "      <td>given</td>\n",
       "      <td>the</td>\n",
       "      <td>target</td>\n",
       "      <td>audience</td>\n",
       "      <td>,</td>\n",
       "      <td>that</td>\n",
       "      <td>may</td>\n",
       "      <td>not</td>\n",
       "      <td>have</td>\n",
       "      <td>been</td>\n",
       "      <td>too</td>\n",
       "      <td>bad</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>gets</td>\n",
       "      <td>roles</td>\n",
       "      <td>in</td>\n",
       "      <td>movies</td>\n",
       "      <td>,</td>\n",
       "      <td>in</td>\n",
       "      <td>my</td>\n",
       "      <td>opinion</td>\n",
       "      <td>though</td>\n",
       "      <td>she</td>\n",
       "      <td>should</td>\n",
       "      <td>stick</td>\n",
       "      <td>to</td>\n",
       "      <td>movies</td>\n",
       "      <td>of</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>its</td>\n",
       "      <td>share</td>\n",
       "      <td>,</td>\n",
       "      <td>though</td>\n",
       "      <td>far</td>\n",
       "      <td>smaller</td>\n",
       "      <td>than</td>\n",
       "      <td>xxunk</td>\n",
       "      <td>even</td>\n",
       "      <td>including</td>\n",
       "      <td>a</td>\n",
       "      <td>basic</td>\n",
       "      <td>view</td>\n",
       "      <td>of</td>\n",
       "      <td>the</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>ruins</td>\n",
       "      <td>-</td>\n",
       "      <td>scene</td>\n",
       "      <td>is</td>\n",
       "      <td>xxup</td>\n",
       "      <td>almost</td>\n",
       "      <td>european</td>\n",
       "      <td>-</td>\n",
       "      <td>like</td>\n",
       "      <td>cinema</td>\n",
       "      <td>(</td>\n",
       "      <td>the</td>\n",
       "      <td>movie</td>\n",
       "      <td>is</td>\n",
       "      <td>eager</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       0                1       2         3      4         5          6   \\\n",
       "0   xxbos            xxmaj    name      just   says        it        all   \n",
       "1   going               to     see  anything    you       'll   remember   \n",
       "2      of              the   xxunk     xxunk  xxmaj  superbly       shot   \n",
       "3    \\n\\n            xxmaj     one    bright  light        in        the   \n",
       "4   leave              you   xxunk      that    you   watched         it   \n",
       "5   xxmaj           ginger      's    blonde   hair         )        has   \n",
       "6       .            xxmaj    this        is   what    really  attracted   \n",
       "7    into              the    hype      that    one       you        are   \n",
       "8    come  unintentionally       ,      like   when      they        try   \n",
       "9    just            watch     the     movie      ,       and          i   \n",
       "10    for              the  reform  movement    and     meets      xxmaj   \n",
       "11  xxmaj             well       ,     given    the    target   audience   \n",
       "12   gets            roles      in    movies      ,        in         my   \n",
       "13    its            share       ,    though    far   smaller       than   \n",
       "14  ruins                -   scene        is   xxup    almost   european   \n",
       "\n",
       "         7        8          9          10         11         12        13  \\\n",
       "0         .        i    watched       this      movie       with        my   \n",
       "1         .    xxbos      xxmaj  evidently       when        you     offer   \n",
       "2         ,     this  thrilling      adult  adventure  certainly  contains   \n",
       "3     midst       of       this         is      xxmaj       fred     xxmaj   \n",
       "4         .        i       feel     really        bad        for     those   \n",
       "5         a   couple         of  emotional       solo    numbers         ,   \n",
       "6        me       to       this       film          .          i       was   \n",
       "7   somehow    white         or   superior        ...        you       are   \n",
       "8        to  explain       that         an  invisible        man        's   \n",
       "9      dear      say        you        'll        see     things         a   \n",
       "10  eponine        .      xxmaj     except        ...        not     xxmaj   \n",
       "11        ,     that        may        not       have       been       too   \n",
       "12  opinion   though        she     should      stick         to    movies   \n",
       "13    xxunk     even  including          a      basic       view        of   \n",
       "14        -     like     cinema          (        the      movie        is   \n",
       "\n",
       "           14  \n",
       "0         dad  \n",
       "1           a  \n",
       "2        some  \n",
       "3       xxunk  \n",
       "4       xxmaj  \n",
       "5   including  \n",
       "6   impressed  \n",
       "7         not  \n",
       "8       xxunk  \n",
       "9         bit  \n",
       "10    eponine  \n",
       "11        bad  \n",
       "12         of  \n",
       "13        the  \n",
       "14      eager  "
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = untar_data(URLs.IMDB_SAMPLE)\n",
    "data = TextLMDataBunch.from_csv(path, 'texts.csv')\n",
    "x,y = next(iter(data.train_dl))\n",
    "example = x[:15,:15].cpu()\n",
    "texts = pd.DataFrame([data.train_ds.vocab.textify(l).split(' ') for l in example])\n",
    "texts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<div markdown=\"span\" class=\"alert alert-danger\" role=\"alert\"><i class=\"fa fa-danger-circle\"></i> <b>Warning: </b>If you are used to another convention, beware! fastai always uses batch as a first dimension, even in NLP.</div>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "jekyll_warn(\"If you are used to another convention, beware! fastai always uses batch as a first dimension, even in NLP.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is all done internally when we use [`TextLMDataBunch`](/text.data.html#TextLMDataBunch), by wrapping the dataset in the following pre-loader before calling a [`DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h2 id=\"LanguageModelPreLoader\"><code>class</code> <code>LanguageModelPreLoader</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L16\" class=\"source_link\">[source]</a></h2>\n",
       "\n",
       "> <code>LanguageModelPreLoader</code>(**`dataset`**:[`LabelList`](/data_block.html#LabelList), **`lengths`**:`Collection`\\[`int`\\]=***`None`***, **`bs`**:`int`=***`32`***, **`bptt`**:`int`=***`70`***, **`backwards`**:`bool`=***`False`***, **`shuffle`**:`bool`=***`False`***) :: [`Callback`](/callback.html#Callback)\n",
       "\n",
       "Transforms the tokens in `dataset` to a stream of contiguous batches for language modelling.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(LanguageModelPreLoader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LanguageModelPreLoader is an internal class uses for training a language model. It takes the sentences passed as a jagged array of numericalised sentences in `dataset` and returns contiguous batches to the pytorch dataloader with batch size `bs` and a sequence length `bptt`. \n",
    "- `lengths` can be provided for the jagged training data else lengths is calculated internally \n",
    "- `backwards=True` will reverses the sentences. \n",
    "- `shuffle=True`, will shuffle the order of the sentences, at the start of each epoch - except the first\n",
    "\n",
    "The following description is usefull for understanding the implementation of [`LanguageModelPreLoader`](/text.data.html#LanguageModelPreLoader):\n",
    "- idx: instance of CircularIndex that indexes items while taking the following into account 1) shuffle, 2) direction of indexing, 3) wraps around to head (reading forward) or tail (reading backwards) of the ragged array as needed in order to fill the last batch(s)\n",
    "\n",
    "- ro: index of the first rag of each row in the batch to be extract. Returns as index to the next rag to be extracted\n",
    "\n",
    "- ri: Reading forward: index to the first token to be extracted in the current rag (ro). Reading backwards: one position after the last token to be extracted in the rag\n",
    "\n",
    "- overlap: overlap between batches is 1, because we only predict the next token        \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classifier data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When preparing the data for a classifier, we keep the different texts separate, which poses another challenge for the creation of batches: since they don't all have the same length, we can't easily collate them together in batches. To help with this we use two different techniques:\n",
    "- padding: each text is padded with the `PAD` token to get all the ones we picked to the same size\n",
    "- sorting the texts (ish): to avoid having together a very long text with a very short one (which would then have a lot of `PAD` tokens), we regroup the texts by order of length. For the training set, we still add some randomness to avoid showing the same batches at every step of the training.\n",
    "\n",
    "Here is an example of batch with padding (the padding index is 1, and the padding is applied before the sentences start)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[   1,    1,    1,    1,    1,    1,    1,    2,   18,  310,    9,    0,\n",
       "           11,    0,    9,   48,    8,    0,   11, 2301],\n",
       "        [   1,    1,    1,    1,    1,    1,    1,    2,    4, 1427,   15,    8,\n",
       "          521,   10,    4,   90,  131,    9, 1427,  242],\n",
       "        [   1,    1,    1,    1,    1,    1,    1,    2,   18,  175,   55, 2063,\n",
       "         4677,   14,    8,  209,   22, 1343,   26,   20],\n",
       "        [   1,    1,    1,    1,    1,    1,    1,    1,    2,    4,   20,   30,\n",
       "           24,    8,  110,  616,   30,  164,  745,   18],\n",
       "        [   1,    1,    1,    1,    1,    1,    1,    1,    2,   18,   24, 3560,\n",
       "           14,  130,    8,   30,   26,   85,  193,    9],\n",
       "        [   1,    1,    1,    1,    1,    1,    1,    1,    2,   18,  101,    0,\n",
       "           20,  153,   71,   18,   24, 4055,   17,    4],\n",
       "        [   1,    1,    1,    1,    1,    1,    1,    1,    2,    4, 1998,  256,\n",
       "            4,    0,    4,    0,  273,   34,    8,    0],\n",
       "        [   1,    1,    1,    1,    1,    1,    1,    1,    1,    2,    4,   20,\n",
       "           24,   12,  119,   30,   19,   83,   12,  202],\n",
       "        [   1,    1,    1,    1,    1,    1,    1,    1,    1,    2,    4,    8,\n",
       "           79, 1031,  185,   13,   20,   30,   24,    8],\n",
       "        [   1,    1,    1,    1,    1,    1,    1,    1,    1,    1,    2,   18,\n",
       "           61,   36,  143,  104,   20,   30, 1408,   51]], device='cuda:0')"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = untar_data(URLs.IMDB_SAMPLE)\n",
    "data = TextClasDataBunch.from_csv(path, 'texts.csv')\n",
    "iter_dl = iter(data.train_dl)\n",
    "_ = next(iter_dl)\n",
    "x,y = next(iter_dl)\n",
    "x[-10:,:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is all done internally when we use [`TextClasDataBunch`](/text.data.html#TextClasDataBunch), by using the following classes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h2 id=\"SortSampler\"><code>class</code> <code>SortSampler</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L99\" class=\"source_link\">[source]</a></h2>\n",
       "\n",
       "> <code>SortSampler</code>(**`data_source`**:`NPArrayList`, **`key`**:`KeyFunc`) :: [`Sampler`](https://pytorch.org/docs/stable/data.html#torch.utils.data.Sampler)\n",
       "\n",
       "Go through the text data by order of length.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(SortSampler)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This pytorch [`Sampler`](https://pytorch.org/docs/stable/data.html#torch.utils.data.Sampler) is used for the validation and (if applicable) the test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h2 id=\"SortishSampler\"><code>class</code> <code>SortishSampler</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L107\" class=\"source_link\">[source]</a></h2>\n",
       "\n",
       "> <code>SortishSampler</code>(**`data_source`**:`NPArrayList`, **`key`**:`KeyFunc`, **`bs`**:`int`) :: [`Sampler`](https://pytorch.org/docs/stable/data.html#torch.utils.data.Sampler)\n",
       "\n",
       "Go through the text data by order of length with a bit of randomness.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(SortishSampler)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This pytorch [`Sampler`](https://pytorch.org/docs/stable/data.html#torch.utils.data.Sampler) is generally used for the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"pad_collate\"><code>pad_collate</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L128\" class=\"source_link\">[source]</a></h4>\n",
       "\n",
       "> <code>pad_collate</code>(**`samples`**:`BatchSamples`, **`pad_idx`**:`int`=***`1`***, **`pad_first`**:`bool`=***`True`***, **`backwards`**:`bool`=***`False`***) → `Tuple`\\[`LongTensor`, `LongTensor`\\]\n",
       "\n",
       "Function that collect samples and adds padding. Flips token order if needed  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(pad_collate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will collate the `samples` in batches while adding padding with `pad_idx`. If `pad_first=True`, padding is applied at the beginning (before the sentence starts) otherwise it's applied at the end."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Undocumented Methods - Methods moved below this line will intentionally be hidden"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"ItemList.new\"><code>new</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/data_block.py#L90\" class=\"source_link\">[source]</a></h4>\n",
       "\n",
       "> <code>new</code>(**`items`**:`Iterator`\\[`T_co`\\], **`processor`**:`Union`\\[[`PreProcessor`](/data_block.html#PreProcessor), `Collection`\\[[`PreProcessor`](/data_block.html#PreProcessor)\\]\\]=***`None`***, **\\*\\*`kwargs`**) → `ItemList`\n",
       "\n",
       "Create a new [`ItemList`](/data_block.html#ItemList) from `items`, keeping the same attributes.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TextList.new)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"TextList.get\"><code>get</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L329\" class=\"source_link\">[source]</a></h4>\n",
       "\n",
       "> <code>get</code>(**`i`**)\n",
       "\n",
       "Subclass if you want to customize how to create item `i` from `self.items`.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TextList.get)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"TokenizeProcessor.process_one\"><code>process_one</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L291\" class=\"source_link\">[source]</a></h4>\n",
       "\n",
       "> <code>process_one</code>(**`item`**)"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TokenizeProcessor.process_one)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"TokenizeProcessor.process\"><code>process</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L294\" class=\"source_link\">[source]</a></h4>\n",
       "\n",
       "> <code>process</code>(**`ds`**)"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TokenizeProcessor.process)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"OpenFileProcessor.process_one\"><code>process_one</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L315\" class=\"source_link\">[source]</a></h4>\n",
       "\n",
       "> <code>process_one</code>(**`item`**)"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(OpenFileProcessor.process_one)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"NumericalizeProcessor.process\"><code>process</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L308\" class=\"source_link\">[source]</a></h4>\n",
       "\n",
       "> <code>process</code>(**`ds`**)"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(NumericalizeProcessor.process)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"NumericalizeProcessor.process_one\"><code>process_one</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L307\" class=\"source_link\">[source]</a></h4>\n",
       "\n",
       "> <code>process_one</code>(**`item`**)"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(NumericalizeProcessor.process_one)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"TextList.reconstruct\"><code>reconstruct</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L338\" class=\"source_link\">[source]</a></h4>\n",
       "\n",
       "> <code>reconstruct</code>(**`t`**:`Tensor`)\n",
       "\n",
       "Reconstruct one of the underlying item for its data `t`.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TextList.reconstruct)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"LanguageModelPreLoader.on_epoch_begin\"><code>on_epoch_begin</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L52\" class=\"source_link\">[source]</a></h4>\n",
       "\n",
       "> <code>on_epoch_begin</code>(**\\*\\*`kwargs`**)\n",
       "\n",
       "At the beginning of each epoch.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(LanguageModelPreLoader.on_epoch_begin)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"LanguageModelPreLoader.on_epoch_end\"><code>on_epoch_end</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/data.py#L69\" class=\"source_link\">[source]</a></h4>\n",
       "\n",
       "> <code>on_epoch_end</code>(**\\*\\*`kwargs`**)\n",
       "\n",
       "Called at the end of an epoch.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(LanguageModelPreLoader.on_epoch_end)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## New Methods - Please document or move to the undocumented section"
   ]
  }
 ],
 "metadata": {
  "jekyll": {
   "keywords": "fastai",
   "summary": "Basic dataset for NLP tasks and helper functions to create a DataBunch",
   "title": "text.data"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}