{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "name": "automatic_differentiation.ipynb", "version": "0.3.2", "provenance": [], "private_outputs": true, "collapsed_sections": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" } }, "cells": [ { "metadata": { "colab_type": "text", "id": "t09eeeR5prIJ" }, "cell_type": "markdown", "source": [ "##### Copyright 2018 The TensorFlow Authors." ] }, { "metadata": { "cellView": "form", "colab_type": "code", "id": "GCCk8_dHpuNf", "colab": {} }, "cell_type": "code", "source": [ "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ], "execution_count": 0, "outputs": [] }, { "metadata": { "colab_type": "text", "id": "xh8WkEwWpnm7" }, "cell_type": "markdown", "source": [ "# Automatic differentiation and gradient tape" ] }, { "metadata": { "colab_type": "text", "id": "idv0bPeCp325" }, "cell_type": "markdown", "source": [ "\n", " \n", " \n", " \n", "
\n", " View on TensorFlow.org\n", " \n", " Run in Google Colab\n", " \n", " View source on GitHub\n", "
" ] }, { "metadata": { "colab_type": "text", "id": "vDJ4XzMqodTy" }, "cell_type": "markdown", "source": [ "In the previous tutorial we introduced `Tensor`s and operations on them. In this tutorial we will cover [automatic differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation), a key technique for optimizing machine learning models." ] }, { "metadata": { "colab_type": "text", "id": "GQJysDM__Qb0" }, "cell_type": "markdown", "source": [ "## Setup\n" ] }, { "metadata": { "colab_type": "code", "id": "OiMPZStlibBv", "colab": {} }, "cell_type": "code", "source": [ "import tensorflow as tf\n", "\n", "tf.enable_eager_execution()" ], "execution_count": 0, "outputs": [] }, { "metadata": { "colab_type": "text", "id": "1CLWJl0QliB0" }, "cell_type": "markdown", "source": [ "## Gradient tapes\n", "\n", "TensorFlow provides the [tf.GradientTape](https://www.tensorflow.org/api_docs/python/tf/GradientTape) API for automatic differentiation - computing the gradient of a computation with respect to its input variables. Tensorflow \"records\" all operations executed inside the context of a `tf.GradientTape` onto a \"tape\". Tensorflow then uses that tape and the gradients associated with each recorded operation to compute the gradients of a \"recorded\" computation using [reverse mode differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation).\n", "\n", "For example:" ] }, { "metadata": { "colab_type": "code", "id": "bAFeIE8EuVIq", "colab": {} }, "cell_type": "code", "source": [ "x = tf.ones((2, 2))\n", " \n", "with tf.GradientTape() as t:\n", " t.watch(x)\n", " y = tf.reduce_sum(x)\n", " z = tf.multiply(y, y)\n", "\n", "# Derivative of z with respect to the original input tensor x\n", "dz_dx = t.gradient(z, x)\n", "for i in [0, 1]:\n", " for j in [0, 1]:\n", " assert dz_dx[i][j].numpy() == 8.0" ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "N4VlqKFzzGaC", "colab_type": "text" }, "cell_type": "markdown", "source": [ "You can also request gradients of the output with respect to intermediate values computed during a \"recorded\" `tf.GradientTape` context." ] }, { "metadata": { "id": "7XaPRAwUyYms", "colab_type": "code", "colab": {} }, "cell_type": "code", "source": [ "x = tf.ones((2, 2))\n", " \n", "with tf.GradientTape() as t:\n", " t.watch(x)\n", " y = tf.reduce_sum(x)\n", " z = tf.multiply(y, y)\n", "\n", "# Use the tape to compute the derivative of z with respect to the\n", "# intermediate value y.\n", "dz_dy = t.gradient(z, y)\n", "assert dz_dy.numpy() == 8.0" ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "ISkXuY7YzIcS", "colab_type": "text" }, "cell_type": "markdown", "source": [ "By default, the resources held by a GradientTape are released as soon as GradientTape.gradient() method is called. To compute multiple gradients over the same computation, create a `persistent` gradient tape. This allows multiple calls to the `gradient()` method. as resources are released when the tape object is garbage collected. For example:" ] }, { "metadata": { "id": "zZaCm3-9zVCi", "colab_type": "code", "colab": {} }, "cell_type": "code", "source": [ "x = tf.constant(3.0)\n", "with tf.GradientTape(persistent=True) as t:\n", " t.watch(x)\n", " y = x * x\n", " z = y * y\n", "dz_dx = t.gradient(z, x) # 108.0 (4*x^3 at x = 3)\n", "dy_dx = t.gradient(y, x) # 6.0\n", "del t # Drop the reference to the tape" ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "6kADybtQzYj4", "colab_type": "text" }, "cell_type": "markdown", "source": [ "### Recording control flow\n", "\n", "Because tapes record operations as they are executed, Python control flow (using `if`s and `while`s for example) is naturally handled:" ] }, { "metadata": { "colab_type": "code", "id": "9FViq92UX7P8", "colab": {} }, "cell_type": "code", "source": [ "def f(x, y):\n", " output = 1.0\n", " for i in range(y):\n", " if i > 1 and i < 5:\n", " output = tf.multiply(output, x)\n", " return output\n", "\n", "def grad(x, y):\n", " with tf.GradientTape() as t:\n", " t.watch(x)\n", " out = f(x, y)\n", " return t.gradient(out, x) \n", "\n", "x = tf.convert_to_tensor(2.0)\n", "\n", "assert grad(x, 6).numpy() == 12.0\n", "assert grad(x, 5).numpy() == 12.0\n", "assert grad(x, 4).numpy() == 4.0\n" ], "execution_count": 0, "outputs": [] }, { "metadata": { "colab_type": "text", "id": "DK05KXrAAld3" }, "cell_type": "markdown", "source": [ "### Higher-order gradients\n", "\n", "Operations inside of the `GradientTape` context manager are recorded for automatic differentiation. If gradients are computed in that context, then the gradient computation is recorded as well. As a result, the exact same API works for higher-order gradients as well. For example:" ] }, { "metadata": { "colab_type": "code", "id": "cPQgthZ7ugRJ", "colab": {} }, "cell_type": "code", "source": [ "x = tf.Variable(1.0) # Create a Tensorflow variable initialized to 1.0\n", "\n", "with tf.GradientTape() as t:\n", " with tf.GradientTape() as t2:\n", " y = x * x * x\n", " # Compute the gradient inside the 't' context manager\n", " # which means the gradient computation is differentiable as well.\n", " dy_dx = t2.gradient(y, x)\n", "d2y_dx2 = t.gradient(dy_dx, x)\n", "\n", "assert dy_dx.numpy() == 3.0\n", "assert d2y_dx2.numpy() == 6.0" ], "execution_count": 0, "outputs": [] }, { "metadata": { "colab_type": "text", "id": "4U1KKzUpNl58" }, "cell_type": "markdown", "source": [ "## Next Steps\n", "\n", "In this tutorial we covered gradient computation in TensorFlow. With that we have enough of the primitives required to build and train neural networks." ] } ] }