{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## datasets" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This module has the necessary functions to be able to download several useful datasets that we might be interested in using in our models." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [], "source": [ "from fastai.gen_doc.nbdoc import *\n", "from fastai.datasets import * \n", "from fastai.datasets import Config\n", "from pathlib import Path" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

class URLs[source][test]

\n", "\n", "> URLs()\n", "\n", "
×

No tests found for URLs. To contribute a test please refer to this guide and this discussion.

\n", "\n", "Global constants for dataset and model URLs. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(URLs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This contains all the datasets' and models' URLs, and some classmethods to help use them - you don't create objects of this class. The supported datasets are (with their calling name): `S3_NLP`, `S3_COCO`, `MNIST_SAMPLE`, `MNIST_TINY`, `IMDB_SAMPLE`, `ADULT_SAMPLE`, `ML_SAMPLE`, `PLANET_SAMPLE`, `CIFAR`, `PETS`, `MNIST`. To get details on the datasets you can see the [fast.ai datasets webpage](http://course.fast.ai/datasets). Datasets with SAMPLE in their name are subsets of the original datasets. In the case of MNIST, we also have a TINY dataset which is even smaller than MNIST_SAMPLE.\n", "\n", "Models is now limited to `WT103` but you can expect more in the future!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'http://files.fast.ai/data/examples/mnist_sample'" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "URLs.MNIST_SAMPLE" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Downloading Data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For the rest of the datasets you will need to download them with [`untar_data`](/datasets.html#untar_data) or [`download_data`](/datasets.html#download_data). [`untar_data`](/datasets.html#untar_data) will decompress the data file and download it while [`download_data`](/datasets.html#download_data) will just download and save the compressed file in `.tgz` format. \n", "\n", "By default, data will be downloaded to `~/.fastai/data` folder. \n", "Configure the default `data_path` by editing `~/.fastai/config.yml`. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

untar_data[source][test]

\n", "\n", "> untar_data(**`url`**:`str`, **`fname`**:`PathOrStr`=***`None`***, **`dest`**:`PathOrStr`=***`None`***, **`data`**=***`True`***, **`force_download`**=***`False`***) → `Path`\n", "\n", "
×

Tests found for untar_data:

  • pytest -sv tests/test_vision_data.py::test_trunc_download [source]

To run tests please refer to this guide.

\n", "\n", "Download `url` to `fname` if it doesn't exist, and un-tgz to folder `dest`. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(untar_data)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "PosixPath('/home/ubuntu/.fastai/data/planet_sample')" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "untar_data(URLs.PLANET_SAMPLE)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

download_data[source][test]

\n", "\n", "> download_data(**`url`**:`str`, **`fname`**:`PathOrStr`=***`None`***, **`data`**:`bool`=***`True`***, **`ext`**:`str`=***`'.tgz'`***) → `Path`\n", "\n", "
×

No tests found for download_data. To contribute a test please refer to this guide and this discussion.

\n", "\n", "Download `url` to destination `fname`. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(download_data)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note: If the data file already exists in a data directory inside the notebook, that data file will be used instead of ~/.fasta/data. Paths are resolved by calling the function [`datapath4file`](/datasets.html#datapath4file) - which checks if data exists locally (`data/`) first, before downloading to `~/.fastai/data` home directory.\n", "\n", "Example:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "PosixPath('/home/ubuntu/.fastai/data/planet_sample.tgz')" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "download_data(URLs.PLANET_SAMPLE)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

datapath4file[source][test]

\n", "\n", "> datapath4file(**`filename`**, **`ext`**:`str`=***`'.tgz'`***)\n", "\n", "
×

No tests found for datapath4file. To contribute a test please refer to this guide and this discussion.

\n", "\n", "Return data path to `filename`, checking locally first then in the config file. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(datapath4file)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "All the downloading functions use this to decide where to put the tgz and expanded folder. If `filename` already exists in a data directory in the same place as the calling notebook/script, that is used as the parent directly, otherwise `~/.fastai/config.yml` is read to see what path to use, which defaults to ~/.fastai/data is used. To override this default, simply modify the value in your `~/.fastai/config.yml`:\n", "\n", " data_path: ~/.fastai/data" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

url2path[source][test]

\n", "\n", "> url2path(**`url`**, **`data`**=***`True`***, **`ext`**:`str`=***`'.tgz'`***)\n", "\n", "
×

No tests found for url2path. To contribute a test please refer to this guide and this discussion.

\n", "\n", "Change `url` to a path. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(url2path)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

class Config[source][test]

\n", "\n", "> Config()\n", "\n", "
×

Tests found for Config:

  • pytest -sv tests/test_datasets.py::test_creates_config [source]

To run tests please refer to this guide.

\n", "\n", "Creates a default config file at `~/.fastai/config.yml` " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(Config)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You probably won't need to use this yourself - it's used by `URLs.datapath4file`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

get_path[source][test]

\n", "\n", "> get_path(**`path`**)\n", "\n", "
×

No tests found for get_path. To contribute a test please refer to this guide and this discussion.

\n", "\n", "Get the `path` in the config file. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(Config.get_path)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Get the key corresponding to `path` in the [`Config`](/datasets.html#Config)." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

data_path[source][test]

\n", "\n", "> data_path()\n", "\n", "
×

No tests found for data_path. To contribute a test please refer to this guide and this discussion.

\n", "\n", "Get the path to data in the config file. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(Config.data_path)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Get the `Path` where the data is stored." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

model_path[source][test]

\n", "\n", "> model_path()\n", "\n", "
×

No tests found for model_path. To contribute a test please refer to this guide and this discussion.

\n", "\n", "Get the path to fastai pretrained models in the config file. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(Config.model_path)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Undocumented Methods - Methods moved below this line will intentionally be hidden" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "

create[source][test]

\n", "\n", "> create(**`fpath`**)\n", "\n", "
×

No tests found for create. To contribute a test please refer to this guide and this discussion.

\n", "\n", "Creates a [`Config`](/datasets.html#Config) from `fpath`. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(Config.create)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "

url2name[source][test]

\n", "\n", "> url2name(**`url`**)\n", "\n", "
×

No tests found for url2name. To contribute a test please refer to this guide and this discussion.

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(url2name)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "

get_key[source][test]

\n", "\n", "> get_key(**`key`**)\n", "\n", "
×

No tests found for get_key. To contribute a test please refer to this guide and this discussion.

\n", "\n", "Get the path to `key` in the config file. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(Config.get_key)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "

get[source][test]

\n", "\n", "> get(**`fpath`**=***`None`***, **`create_missing`**=***`True`***)\n", "\n", "
×

Tests found for get:

Some other tests where get is used:

  • pytest -sv tests/test_datasets.py::test_creates_config [source]

To run tests please refer to this guide.

\n", "\n", "Retrieve the [`Config`](/datasets.html#Config) in `fpath`. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(Config.get)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## New Methods - Please document or move to the undocumented section" ] } ], "metadata": { "jekyll": { "keywords": "fastai" }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 2 }