{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "name": "Visualization Regularization - Feature Visualization", "version": "0.3.2", "provenance": [] }, "kernelspec": { "name": "python2", "display_name": "Python 2" }, "accelerator": "GPU" }, "cells": [ { "metadata": { "id": "JndnmDMp66FL", "colab_type": "text" }, "cell_type": "markdown", "source": [ "##### Copyright 2018 Google LLC.\n", "\n", "Licensed under the Apache License, Version 2.0 (the \"License\");" ] }, { "metadata": { "id": "hMqWDc_m6rUC", "colab_type": "code", "cellView": "both", "colab": {} }, "cell_type": "code", "source": [ "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "_vAVmphMywZR", "colab_type": "text" }, "cell_type": "markdown", "source": [ "# Visualization Regularization - Feature Visualization\n", "\n", "This notebook uses [**Lucid**](https://github.com/tensorflow/lucid) to reproduce some of the results in the section [\"The Enemy of Feature Visualization\"](https://distill.pub/2017/feature-visualization/#enemy-of-feature-vis) of [Feature Visualization](https://distill.pub/2017/feature-visualization/).\n", "\n", "This notebook doesn't introduce the abstractions behind lucid; you may wish to also read the [Lucid tutorial](https://colab.research.google.com/github/tensorflow/lucid/blob/master/notebooks/tutorial.ipynb).\n", "\n", "**Note**: The easiest way to use this tutorial is as a colab notebook, which allows you to dive in with no setup. We recommend you enable a free GPU by going:\n", "\n", "> **Runtime**   →   **Change runtime type**   →   **Hardware Accelerator: GPU**" ] }, { "metadata": { "id": "FsFc1mE51tCd", "colab_type": "text" }, "cell_type": "markdown", "source": [ "## Install, Import, Load Model" ] }, { "metadata": { "id": "RBr8QbboRAdU", "colab_type": "code", "colab": {} }, "cell_type": "code", "source": [ "!pip install --quiet lucid\n", "\n", "import numpy as np\n", "import scipy.ndimage as nd\n", "import tensorflow as tf\n", "\n", "import lucid.modelzoo.vision_models as models\n", "from lucid.misc.io import show\n", "import lucid.optvis.objectives as objectives\n", "import lucid.optvis.param as param\n", "import lucid.optvis.render as render\n", "import lucid.optvis.transform as transform" ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "yNALaA0QRJVT", "colab_type": "code", "colab": {} }, "cell_type": "code", "source": [ "# Let's import a model from the Lucid modelzoo!\n", "\n", "model = models.InceptionV1()\n", "model.load_graphdef()" ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "d3ZKjNizGWtD", "colab_type": "text" }, "cell_type": "markdown", "source": [ "# Naive Feature Visualization\n", "\n", "The code reproducing the following diagrams uses `CONSTANTS` to provide input values.\n" ] }, { "metadata": { "id": "THlwQw-1GWtD", "colab_type": "text" }, "cell_type": "markdown", "source": [ "\n", "" ] }, { "metadata": { "id": "3d6xz3CvTLu5", "colab_type": "code", "outputId": "095296c4-10d7-4e4c-fac0-bb912c578700", "colab": { "base_uri": "https://localhost:8080/", "height": 171 } }, "cell_type": "code", "source": [ "LEARNING_RATE = 0.05\n", "\n", "optimizer = tf.train.AdamOptimizer(LEARNING_RATE)\n", "\n", "imgs = render.render_vis(model, \"mixed4b_pre_relu:452\",\n", " optimizer=optimizer,\n", " transforms=[],\n", " param_f=lambda: param.image(64, fft=False, decorrelate=False), \n", " thresholds=(1, 32, 128, 256, 2048), verbose=False)\n", "\n", "\n", "# Note that we're doubling the image scale to make artifacts more obvious\n", "show([nd.zoom(img[0], [2,2,1], order=0) for img in imgs])" ], "execution_count": 3, "outputs": [ { "output_type": "display_data", "data": { "text/plain": [ "" ], "text/html": [ "
\n", " 0
\n", " \n", "
\n", " 1
\n", " \n", "
\n", " 2
\n", " \n", "
\n", " 3
\n", " \n", "
\n", " 4
\n", " \n", "
" ] }, "metadata": { "tags": [] } } ] }, { "metadata": { "id": "zlIWBnagGd4q", "colab_type": "text" }, "cell_type": "markdown", "source": [ "# Frequency Penalization" ] }, { "metadata": { "id": "1HdYBKktGd4s", "colab_type": "text" }, "cell_type": "markdown", "source": [ "\n", "" ] }, { "metadata": { "id": "_dwIwFdmTHgd", "colab_type": "code", "outputId": "a19f17f6-0bcd-4aa9-fd4f-78860330b451", "colab": { "base_uri": "https://localhost:8080/", "height": 171 } }, "cell_type": "code", "source": [ "L1 = -0.05\n", "TV = -0.25\n", "BLUR = -1.0\n", "\n", "obj = objectives.channel(\"mixed4b_pre_relu\", 452)\n", "obj += L1 * objectives.L1(constant=.5)\n", "obj += TV * objectives.total_variation()\n", "obj += BLUR * objectives.blur_input_each_step()\n", "\n", "imgs = render.render_vis(model, obj,\n", " transforms=[],\n", " param_f=lambda: param.image(64, fft=False, decorrelate=False), \n", " thresholds=(1, 32, 128, 256, 2048), verbose=False)\n", "\n", "\n", "# Note that we're doubling the image scale to make artifacts more obvious\n", "show([nd.zoom(img[0], [2,2,1], order=0) for img in imgs])" ], "execution_count": 14, "outputs": [ { "output_type": "display_data", "data": { "text/plain": [ "" ], "text/html": [ "
\n", " 0
\n", " \n", "
\n", " 1
\n", " \n", "
\n", " 2
\n", " \n", "
\n", " 3
\n", " \n", "
\n", " 4
\n", " \n", "
" ] }, "metadata": { "tags": [] } } ] }, { "metadata": { "id": "mSDJKMf5GeZy", "colab_type": "text" }, "cell_type": "markdown", "source": [ "# Transformation Robustness" ] }, { "metadata": { "id": "Za8zeP8HGeZ6", "colab_type": "text" }, "cell_type": "markdown", "source": [ "\n", "" ] }, { "metadata": { "id": "1rRrTzSBNrmP", "colab_type": "code", "outputId": "931dbf1f-7aea-4142-cbc4-6d55df5e58af", "colab": { "base_uri": "https://localhost:8080/", "height": 171 } }, "cell_type": "code", "source": [ "JITTER = 1\n", "ROTATE = 5\n", "SCALE = 1.1\n", "\n", "transforms = [\n", " transform.pad(2*JITTER),\n", " transform.jitter(JITTER),\n", " transform.random_scale([SCALE ** (n/10.) for n in range(-10, 11)]),\n", " transform.random_rotate(range(-ROTATE, ROTATE+1))\n", "]\n", "\n", "imgs = render.render_vis(model, \"mixed4b_pre_relu:452\", transforms=transforms,\n", " param_f=lambda: param.image(64), \n", " thresholds=(1, 32, 128, 256, 2048), verbose=False)\n", "\n", "\n", "# Note that we're doubling the image scale to make artifacts more obvious\n", "show([nd.zoom(img[0], [2,2,1], order=0) for img in imgs])" ], "execution_count": 5, "outputs": [ { "output_type": "display_data", "data": { "text/plain": [ "" ], "text/html": [ "
\n", " 0
\n", " \n", "
\n", " 1
\n", " \n", "
\n", " 2
\n", " \n", "
\n", " 3
\n", " \n", "
\n", " 4
\n", " \n", "
" ] }, "metadata": { "tags": [] } } ] }, { "metadata": { "id": "gvF6j5h4GkDe", "colab_type": "text" }, "cell_type": "markdown", "source": [ "# Preconditioning" ] }, { "metadata": { "id": "r4o7DzPXGkDg", "colab_type": "text" }, "cell_type": "markdown", "source": [ "\n", "" ] }, { "metadata": { "id": "1E5O1S3DR8YC", "colab_type": "code", "outputId": "5009acb9-1814-4957-f652-6e202d42e493", "colab": { "base_uri": "https://localhost:8080/", "height": 171 } }, "cell_type": "code", "source": [ "LEARNING_RATE = 0.05\n", "DECORRELATE = True\n", "ROBUSTNESS = True\n", "\n", "# `fft` parameter controls spatial decorrelation\n", "# `decorrelate` parameter controls channel decorrelation\n", "param_f = lambda: param.image(64, fft=DECORRELATE, decorrelate=DECORRELATE)\n", "\n", "if ROBUSTNESS:\n", " transforms = transform.standard_transforms\n", "else:\n", " transforms = []\n", "\n", "optimizer = tf.train.AdamOptimizer(LEARNING_RATE)\n", "\n", "imgs = render.render_vis(model, \"mixed4b_pre_relu:452\",\n", " optimizer=optimizer,\n", " transforms=transforms,\n", " param_f=param_f, \n", " thresholds=(1, 32, 128, 256, 2048), verbose=False)\n", "\n", "\n", "# Note that we're doubling the image scale to make artifacts more obvious\n", "show([nd.zoom(img[0], [2,2,1], order=0) for img in imgs])" ], "execution_count": 6, "outputs": [ { "output_type": "display_data", "data": { "text/plain": [ "" ], "text/html": [ "
\n", " 0
\n", " \n", "
\n", " 1
\n", " \n", "
\n", " 2
\n", " \n", "
\n", " 3
\n", " \n", "
\n", " 4
\n", " \n", "
" ] }, "metadata": { "tags": [] } } ] } ] }