{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Torch Core"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This module contains all the basic functions we need in other modules of the fastai library (split with [`core`](/core.html#core) that contains the ones not requiring pytorch). Its documentation can easily be skipped at a first read, unless you want to know what a given fuction does."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"hide_input": true
},
"outputs": [],
"source": [
"from fastai.gen_doc.nbdoc import *\n",
"from fastai.torch_core import * "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Global constants"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`AdamW = partial(optim.Adam, betas=(0.9,0.99))` "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`bn_types = (nn.BatchNorm1d, nn.BatchNorm2d, nn.BatchNorm3d)` "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`default_device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')` "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Functions that operate conversions"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> flatten(`m`)"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(flatten_model, full_name='flatten')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Flattens all the layers of `m`. "
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> model2half(`model`:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)) → [`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)\n",
"\n",
"Convert `model` to half precision except the batchnorm layers. "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(model2half)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> requires_grad(`m`:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module), `b`:`Optional`\\[`bool`\\]=`None`) → `Optional`\\[`bool`\\]"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(requires_grad, doc_string=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If `b` is None, returns the [`requires_grad`](/torch_core.html#requires_grad) state of the first layer of `m`. Otherwise, sets `requires_grad=b` in all children of `m`."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> tensor(`x`:`Any`, `rest`) → `Tensor`\n",
"\n",
"Like `torch.as_tensor`, but handle lists too, and can pass multiple vector elements directly "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(tensor)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Ensures `x` is a torch `Tensor`."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> to_data(`b`:`ItemsList`)\n",
"\n",
"Recursively map lists of items in `b ` to their wrapped data "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(to_data)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> to_detach(`b`:`Tensors`)\n",
"\n",
"Recursively detach lists of tensors in `b ` "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(to_detach)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> to_device(`b`:`Tensors`, `device`:[`device`](https://pytorch.org/docs/stable/tensor_attributes.html#torch-device))\n",
"\n",
"Ensure `b` is on `device`. "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(to_device)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> to_half(`b`:`Collection`\\[`Tensor`\\]) → `Collection`\\[`Tensor`\\]"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(to_half, doc_string=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Put the input of the batch `b` in half precision."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> to_np(`x`)"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(to_np)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Convert `x` to a numpy array."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Functions to deal with model initialization"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> apply_init(`m`, `init_func`:`LayerFunc`)\n",
"\n",
"Initialize all non-batchnorm layers of `m` with `init_func`. "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(apply_init)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> apply_leaf(`m`:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module), `f`:`LayerFunc`)\n",
"\n",
"Apply `f` to children of `m`. "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(apply_leaf)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> cond_init(`m`:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module), `init_func`:`LayerFunc`)\n",
"\n",
"Initialize the non-batchnorm layers of `m` with `init_func` "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(cond_init)"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> in_channels(`m`:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)) → `List`\\[`int`\\]\n",
"\n",
"Return the shape of the first weight layer in `m`. "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(in_channels)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Functions to get information of a model"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> children(`m`:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)) → `ModuleList`\n",
"\n",
"Get children of module `m`. "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(children)"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> first_layer(`m`:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)) → [`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)\n",
"\n",
"Retrieve first layer in a module `m`. "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(first_layer)"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> num_children(`m`:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)) → `int`\n",
"\n",
"Get number of children modules in module `m`. "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(num_children)"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> range_children(`m`:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)) → `Iterator`\\[`int`\\]\n",
"\n",
"Return iterator of len of children of `m`. "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(range_children)"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> trainable_params(`m`:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)) → `ParamList`\n",
"\n",
"Return list of trainable params in `m`. "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(trainable_params)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Functions to deal with BatchNorm layers"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> bn2float(`module`:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)) → [`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)\n",
"\n",
"If `module` is batchnorm don't use half precision. "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(bn2float)"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> set_bn_eval(`m`:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module))\n",
"\n",
"Set bn layers in eval mode for all recursive children of `m`. "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(set_bn_eval)"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> split_bn_bias(`layer_groups`:`ModuleList`) → `ModuleList`\n",
"\n",
"Sort each layer in `layer_groups` into batchnorm (`bn_types`) and non-batchnorm groups. "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(split_bn_bias)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Other functions"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> calc_loss(`y_pred`:`Tensor`, `y_true`:`Tensor`, `loss_func`:`LossFunction`)\n",
"\n",
"Calculate loss between `y_pred` and `y_true` using `loss_class` and `bs`. "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(calc_loss)"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> data_collate(`batch`:`ItemsList`) → `Tensor`\n",
"\n",
"Convert `batch` items to tensor data. "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(data_collate)"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> split_model(`model`:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module), `splits`:`Collection`\\[`Union`\\[[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module), `ModuleList`\\]\\], `want_idxs`:`bool`=`False`)"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(split_model, doc_string=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Splits the `model` according to the layer in `splits`. If `splits` are layers, the model is split at those (not included) sequentially. If `want_idxs` is True, the corresponding indexes are returned. If `splits` are lists of layers, the model is split according to those."
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> split_model_idx(`model`:[`Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module), `idxs`:`Collection`\\[`int`\\]) → `ModuleList`\n",
"\n",
"Split `model` according to the indices in `idxs`. "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(split_model_idx)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Undocumented Methods - Methods moved below this line will intentionally be hidden"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## New Methods - Please document or move to the undocumented section"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> np2model_tensor(`a`)"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(np2model_tensor)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": []
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> np_address(`x`:`ndarray`) → `int`\n",
"\n",
"Address of `x` in memory "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(np_address)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": []
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> trange_of(`x`)"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(trange_of)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": []
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"hide_input": true
},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> model_type(`dtype`)"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(model_type)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": []
}
],
"metadata": {
"jekyll": {
"keywords": "fastai",
"summary": "Basic functions using pytorch",
"title": "torch_core"
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.5"
}
},
"nbformat": 4,
"nbformat_minor": 2
}