{
"cells": [
{
"cell_type": "code",
"metadata": {},
"execution_count": null, "outputs": [],
"source": ["// Google Colab only: uncomment and run the following to install dotnet and the F# kernel\n",
"// !bash \u003c(curl -Ls https://raw.githubusercontent.com/gbaydin/scripts/main/colab_dotnet6.sh)\n"]
}
,
{
"cell_type": "code",
"metadata": {},
"execution_count": null, "outputs": [],
"source": ["// Import DiffSharp package\n",
"#r \"nuget: DiffSharp-lite,1.0.7\"\n",
"\n",
"// Set dotnet interactive formatter to plaintext\n",
"Formatter.SetPreferredMimeTypesFor(typeof\u003cobj\u003e, \"text/plain\")\n",
"Formatter.Register(fun (x:obj) (writer: TextWriter) -\u003e fprintfn writer \"%120A\" x )\n"]
}
,
{
"cell_type": "markdown",
"metadata": {},
"source": ["[](https://colab.research.google.com/github/DiffSharp/diffsharp.github.io/blob/master/index.ipynb)\u0026emsp;\n",
"[](https://mybinder.org/v2/gh/diffsharp/diffsharp.github.io/master?filepath=index.ipynb)\u0026emsp;\n",
"[](index.fsx)\u0026emsp;\n",
"[](index.ipynb)\n",
"\n",
"# DiffSharp: Differentiable Tensor Programming Made Simple\n",
"\n",
"DiffSharp is a tensor library with support for [differentiable programming](https://en.wikipedia.org/wiki/Differentiable_programming).\n",
"It is designed for use in machine learning, probabilistic programming, optimization and other domains.\n",
"\n"]
}
,
{
"cell_type": "code",
"metadata": {},
"execution_count": null, "outputs": [],
"source": ["\u003cbutton class=\"button\" style=\"vertical-align:middle\" onclick=\"window.location.href=\u0027https://diffsharp.github.io/install.html\u0027\"\u003e\u003cspan\u003eInstall »\u003c/span\u003e\u003c/button\u003e\n"]
}
,
{
"cell_type": "markdown",
"metadata": {},
"source": ["## Key Features\n",
"\n",
"🗹 Nested and mixed-mode differentiation\n",
"\n",
"🗹 Common optimizers, model elements, differentiable probability distributions\n",
"\n",
"🗹 F# for robust functional programming\n",
"\n",
"🗹 PyTorch familiar naming and idioms, efficient LibTorch CUDA/C++ tensors with GPU support\n",
"\n",
"🗹 Linux, macOS, Windows supported\n",
"\n",
"🗹 Use interactive notebooks in Jupyter and Visual Studio Code\n",
"\n",
"🗹 100% open source\n",
"\n",
"## Differentiable Programming\n",
"\n",
"DiffSharp provides world-leading automatic differentiation capabilities for tensor code, including composable gradients, Hessians, Jacobians, directional derivatives, and matrix-free Hessian- and Jacobian-vector products over arbitrary user code. This goes beyond conventional tensor libraries such as PyTorch and TensorFlow, allowing the use of nested forward and reverse differentiation up to any level.\n",
"\n",
"With DiffSharp, you can compute higher-order derivatives efficiently and differentiate functions that are internally making use of differentiation and gradient-based optimization.\n",
"\n"]
}
,
{
"cell_type": "code",
"metadata": {},
"execution_count": null, "outputs": [],
"source": ["\u003c/br\u003e\n",
"\u003cimg src=\"img/anim-intro-2.gif\" width=\"85%\" /\u003e\n"]
}
,
{
"cell_type": "markdown",
"metadata": {},
"source": ["## Practical, Familiar and Efficient\n",
"\n",
"DiffSharp comes with a [LibTorch](https://pytorch.org/cppdocs/) backend, using the same C++ and CUDA implementations for tensor computations that power [PyTorch](https://pytorch.org/). On top of these raw tensors (LibTorch\u0027s ATen, excluding autograd), DiffSharp implements its own computation graph and differentiation capabilities. It is tested on Linux, macOS, and Windows, and it supports CUDA and GPUs.\n",
"\n",
"The DiffSharp API is designed to be similar to [the PyTorch Python API](https://pytorch.org/docs/stable/index.html) through very similar naming and idioms, and where elements have similar names the PyTorch documentation can generally be used as a guide.\n",
"\n",
"DiffSharp uses [the incredible F# programming language](https://dot.net/fsharp) for tensor programming. F# code is generally faster and more robust than equivalent Python code, while still being succinct and compact like Python, making it an ideal modern AI and machine learning implementation language. This allows fluent and productive code for tensor programming.\n",
"\n"]
}
,
{
"cell_type": "code",
"metadata": {},
"execution_count": null, "outputs": [],
"source": ["\u003c/br\u003e\n",
"\u003ciframe width=\"85%\" src=\"https://www.youtube.com/embed/_QnbV6CAWXc\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen\u003e\u003c/iframe\u003e\n"]
}
,
{
"cell_type": "markdown",
"metadata": {},
"source": ["## Interactive Notebooks\n",
"\n",
"All documentation pages in this website are interactive notebooks which you can execute directly in your browser without installing anything in your local machine.\n",
"\n",
"Using the buttons [](https://colab.research.google.com/github/DiffSharp/diffsharp.github.io/blob/master/index.ipynb) [](https://mybinder.org/v2/gh/diffsharp/diffsharp.github.io/master?filepath=index.ipynb) on the top of each page, you can execute the page as an interactive notebook running on cloud servers provided by [Google Colab](https://colab.research.google.com/) and [Binder](https://mybinder.org/).\n",
"\n",
"Using the buttons [](index.fsx)\n",
"[](index.ipynb) you can also download a page as a script or an interactive notebook, which you can execute locally in [Jupyter](https://jupyter.org/) or [Visual Studio Code](https://code.visualstudio.com/) using [dotnet interactive](https://github.com/dotnet/interactive).\n",
"\n",
"## Example\n",
"\n",
"Define and add two tensors:\n",
"\n"]
}
,
{
"cell_type": "code",
"metadata": {},
"execution_count": 2, "outputs": [
{
"data": {
"text/plain": ["tensor([1.0000, 2.2000, 3.4000, 4.6000, 5.8000, 7.0000])"]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}],
"source": ["open DiffSharp\n",
"\n",
"let t1 = dsharp.tensor [ 0.0 ..0.2.. 1.0 ] // Gives [0., 0.2, 0.4, 0.6, 0.8, 1.]\n",
"let t2 = dsharp.tensor [ 1, 2, 3, 4, 5, 6 ]\n",
"\n",
"t1 + t2\n"]
}
,
{
"cell_type": "markdown",
"metadata": {},
"source": ["Compute a convolution:\n",
"\n"]
}
,
{
"cell_type": "code",
"metadata": {},
"execution_count": 3, "outputs": [
{
"data": {
"text/plain": ["tensor([[[[38.5000]]]])"]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}],
"source": ["let t3 = dsharp.tensor [[[[0.0 .. 10.0]]]]\n",
"let t4 = dsharp.tensor [[[[0.0 ..0.1.. 1.0]]]]\n",
"\n",
"t3.conv2d(t4)\n"]
}
,
{
"cell_type": "markdown",
"metadata": {},
"source": ["Take the gradient of a vector-to-scalar function:\n",
"\n"]
}
,
{
"cell_type": "code",
"metadata": {},
"execution_count": 4, "outputs": [
{
"data": {
"text/plain": ["tensor([ 6.0496, 12.1825])"]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}],
"source": ["let f (x: Tensor) = x.exp().sum()\n",
"\n",
"dsharp.grad f (dsharp.tensor([1.8, 2.5]))\n"]
}
,
{
"cell_type": "markdown",
"metadata": {},
"source": ["Compute a nested derivative (checking for [perturbation confusion](https://doi.org/10.1007/s10990-008-9037-1)):\n",
"\n"]
}
,
{
"cell_type": "code",
"metadata": {},
"execution_count": 5, "outputs": [
{
"data": {
"text/plain": ["tensor(2.)"]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}],
"source": ["let x0 = dsharp.tensor(1.)\n",
"let y0 = dsharp.tensor(2.)\n",
"dsharp.diff (fun x -\u003e x * dsharp.diff (fun y -\u003e x * y) y0) x0\n"]
}
,
{
"cell_type": "markdown",
"metadata": {},
"source": ["Define a model and optimize it:\n",
"\n"]
}
,
{
"cell_type": "code",
"metadata": {},
"execution_count": null, "outputs": [],
"source": ["open DiffSharp.Data\n",
"open DiffSharp.Model\n",
"open DiffSharp.Compose\n",
"open DiffSharp.Util\n",
"open DiffSharp.Optim\n",
"\n",
"let epochs = 2\n",
"let batchSize = 32\n",
"let numBatches = 5\n",
"\n",
"let trainSet = MNIST(\"../data\", train=true, transform=id)\n",
"let trainLoader = trainSet.loader(batchSize=batchSize, shuffle=true)\n",
"\n",
"let validSet = MNIST(\"../data\", train=false, transform=id)\n",
"let validLoader = validSet.loader(batchSize=batchSize, shuffle=false)\n",
"\n",
"let encoder =\n",
" Conv2d(1, 32, 4, 2)\n",
" --\u003e dsharp.relu\n",
" --\u003e Conv2d(32, 64, 4, 2)\n",
" --\u003e dsharp.relu\n",
" --\u003e Conv2d(64, 128, 4, 2)\n",
" --\u003e dsharp.flatten(1)\n",
"\n",
"let decoder =\n",
" dsharp.unflatten(1, [128;1;1])\n",
" --\u003e ConvTranspose2d(128, 64, 4, 2)\n",
" --\u003e dsharp.relu\n",
" --\u003e ConvTranspose2d(64, 32, 4, 3)\n",
" --\u003e dsharp.relu\n",
" --\u003e ConvTranspose2d(32, 1, 4, 2)\n",
" --\u003e dsharp.sigmoid\n",
"\n",
"let model = VAE([1;28;28], 64, encoder, decoder)\n",
"\n",
"let lr = dsharp.tensor(0.001)\n",
"let optimizer = Adam(model, lr=lr)\n",
"\n",
"for epoch = 1 to epochs do\n",
" let batches = trainLoader.epoch(numBatches)\n",
" for i, x, _ in batches do\n",
" model.reverseDiff()\n",
" let l = model.loss(x)\n",
" l.reverse()\n",
" optimizer.step()\n",
" print $\"Epoch: {epoch} minibatch: {i} loss: {l}\" \n",
"\n",
"let validLoss = \n",
" validLoader.epoch() \n",
" |\u003e Seq.sumBy (fun (_, x, _) -\u003e model.loss(x, normalize=false))\n",
"\n",
"print $\"Validation loss: {validLoss/validSet.length}\"\n"]
}
,
{
"cell_type": "markdown",
"metadata": {},
"source": ["Numerous other model definition, differentiation, and training patterns are supported. See the tutorials in the left-hand menu and [examples](https://github.com/DiffSharp/DiffSharp/tree/dev/examples) on GitHub.\n",
"\n",
"## More Information\n",
"\n",
"DiffSharp is developed by [Atılım Güneş Baydin](http://www.robots.ox.ac.uk/~gunes/), [Don Syme](https://www.microsoft.com/en-us/research/people/dsyme/)\n",
"and other contributors, having started as a project supervised by the automatic differentiation wizards [Barak Pearlmutter](https://scholar.google.com/citations?user=AxFrw0sAAAAJ\u0026hl=en) and [Jeffrey Siskind](https://scholar.google.com/citations?user=CgSBtPYAAAAJ\u0026hl=en).\n",
"\n",
"Please join us [on GitHub](https://github.com/DiffSharp/DiffSharp)!\n",
"\n"]
}],
"metadata": {
"kernelspec": {"display_name": ".NET (F#)", "language": "F#", "name": ".net-fsharp"},
"langauge_info": {
"file_extension": ".fs",
"mimetype": "text/x-fsharp",
"name": "C#",
"pygments_lexer": "fsharp",
"version": "4.5"
}
},
"nbformat": 4,
"nbformat_minor": 1
}