{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "name": "Interaction Between Neurons - Feature Visualization", "version": "0.3.2", "views": {}, "default_view": {}, "provenance": [] }, "kernelspec": { "name": "python2", "display_name": "Python 2" }, "accelerator": "GPU" }, "cells": [ { "metadata": { "id": "JndnmDMp66FL", "colab_type": "text" }, "cell_type": "markdown", "source": [ "##### Copyright 2018 Google LLC.\n", "\n", "Licensed under the Apache License, Version 2.0 (the \"License\");" ] }, { "metadata": { "id": "hMqWDc_m6rUC", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 } }, "cellView": "both" }, "cell_type": "code", "source": [ "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "_vAVmphMywZR", "colab_type": "text" }, "cell_type": "markdown", "source": [ "# Interaction Between Neurons - Feature Visualization\n", "\n", "This notebook uses [**Lucid**](https://github.com/tensorflow/lucid) to reproduce some of the results in [Feature Visualization](https://distill.pub/2017/feature-visualization/).\n", "\n", "This notebook doesn't introduce the abstractions behind lucid; you may wish to also read the [Lucid tutorial](https://colab.research.google.com/github/tensorflow/lucid/blob/master/notebooks/tutorial.ipynb).\n", "\n", "**Note**: The easiest way to use this tutorial is as a colab notebook, which allows you to dive in with no setup. We recommend you enable a free GPU by going:\n", "\n", "> **Runtime**   →   **Change runtime type**   →   **Hardware Accelerator: GPU**" ] }, { "metadata": { "id": "FsFc1mE51tCd", "colab_type": "text" }, "cell_type": "markdown", "source": [ "## Install, Import, Load Model" ] }, { "metadata": { "id": "tavMPe3KQ8Cs", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 } } }, "cell_type": "code", "source": [ "# Install Lucid\n", "\n", "!pip install --quiet lucid==0.0.5\n", "#!pip install --quiet --upgrade-strategy=only-if-needed git+https://github.com/tensorflow/lucid.git" ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "RBr8QbboRAdU", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 } } }, "cell_type": "code", "source": [ "# Imports\n", "\n", "import numpy as np\n", "import scipy.ndimage as nd\n", "import tensorflow as tf\n", "\n", "import lucid.modelzoo.vision_models as models\n", "from lucid.misc.io import show\n", "import lucid.optvis.objectives as objectives\n", "import lucid.optvis.param as param\n", "import lucid.optvis.render as render\n", "import lucid.optvis.transform as transform" ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "yNALaA0QRJVT", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 } } }, "cell_type": "code", "source": [ "# Let's import a model from the Lucid modelzoo!\n", "\n", "model = models.InceptionV1()\n", "model.load_graphdef()" ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "0x-Zi49jGULb", "colab_type": "text" }, "cell_type": "markdown", "source": [ "# Combining Objectives" ] }, { "metadata": { "id": "YsES8tRfGULc", "colab_type": "text" }, "cell_type": "markdown", "source": [ "\n", "" ] }, { "metadata": { "id": "hkpAuu-64zfE", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 } } }, "cell_type": "code", "source": [ "neuron1 = ('mixed4b_pre_relu', 111) # large fluffy\n", "# neuron1 = ('mixed3a_pre_relu', 139) # pointilist\n", "# neuron1 = ('mixed3b_pre_relu', 81) # brush trokes\n", "# neuron1 = ('mixed4a_pre_relu', 97) # wavy\n", "# neuron1 = ('mixed4a_pre_relu', 41) # frames\n", "# neuron1 = ('mixed4a_pre_relu', 479) # B/W\n", "\n", "neuron2 = ('mixed4a_pre_relu', 476) # art\n", "# neuron2 = ('mixed4b_pre_relu', 360) # lattices\n", "# neuron2 = ('mixed4b_pre_relu', 482) # arcs\n", "# neuron2 = ('mixed4c_pre_relu', 440) # small fluffy\n", "# neuron2 = ('mixed4d_pre_relu', 479) # bird beaks\n", "# neuron2 = ('mixed4e_pre_relu', 718) # shoulders" ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "uxfsCgZ-jQ9_", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 }, "base_uri": "https://localhost:8080/", "height": 464 }, "outputId": "24d2ed93-45cc-4333-d3c2-7258965c7358", "executionInfo": { "status": "ok", "timestamp": 1526674752492, "user_tz": 420, "elapsed": 200644, "user": { "displayName": "Christopher Olah", "photoUrl": "//lh5.googleusercontent.com/-GhJP0RTFLEs/AAAAAAAAAAI/AAAAAAAAEZ8/wDVK-lwJYfA/s50-c-k-no/photo.jpg", "userId": "104171973056281402320" } } }, "cell_type": "code", "source": [ "C = lambda neuron: objectives.channel(*neuron)\n", "\n", "_ = render.render_vis(model, C(neuron1))\n", "_ = render.render_vis(model, C(neuron2))\n", "_ = render.render_vis(model, C(neuron1) + C(neuron2))" ], "execution_count": 8, "outputs": [ { "output_type": "stream", "text": [ "512 791.29175\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "text/plain": [ "" ], "text/html": [ "" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "512 1146.8491\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "text/plain": [ "" ], "text/html": [ "" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "512 1272.907\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "text/plain": [ "" ], "text/html": [ "" ] }, "metadata": { "tags": [] } } ] }, { "metadata": { "id": "nxPbPfXzGWCD", "colab_type": "text" }, "cell_type": "markdown", "source": [ "# Random Directions" ] }, { "metadata": { "id": "YKoI2xj3GWCE", "colab_type": "text" }, "cell_type": "markdown", "source": [ "\n", "" ] }, { "metadata": { "id": "yvIEwpo0aZGs", "colab_type": "text" }, "cell_type": "markdown", "source": [ "Unfortunately, constraints on ImageNet mean we can't provide an easy way for you to reproduce the dataset examples. However, we can reproduce the random directions, although since they're random, you'll get a different result each time (and they won't match the ones in the article)." ] }, { "metadata": { "id": "Oavrm8eJaZ1d", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 }, "base_uri": "https://localhost:8080/", "height": 166 }, "outputId": "8a3bb0e1-65fc-477b-eb52-a71beab73576", "executionInfo": { "status": "ok", "timestamp": 1526666552680, "user_tz": 420, "elapsed": 17103, "user": { "displayName": "Christopher Olah", "photoUrl": "//lh5.googleusercontent.com/-GhJP0RTFLEs/AAAAAAAAAAI/AAAAAAAAEZ8/wDVK-lwJYfA/s50-c-k-no/photo.jpg", "userId": "104171973056281402320" } } }, "cell_type": "code", "source": [ "obj = objectives.direction(\"mixed4d_pre_relu\", np.random.randn(528))\n", "_ = render.render_vis(model, obj)" ], "execution_count": 0, "outputs": [ { "output_type": "stream", "text": [ "512 61.127705\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "text/plain": [ "" ], "text/html": [ "" ] }, "metadata": { "tags": [] } } ] }, { "metadata": { "id": "MxRmLCTlGWSs", "colab_type": "text" }, "cell_type": "markdown", "source": [ "# Aligned Interpolation" ] }, { "metadata": { "id": "WpbF_gDNGWSs", "colab_type": "text" }, "cell_type": "markdown", "source": [ "\n", "" ] }, { "metadata": { "id": "_HPM3tN5dSmn", "colab_type": "text" }, "cell_type": "markdown", "source": [ "We hope to explore and explain this trick in more detail in an upcoming article." ] }, { "metadata": { "id": "GYujByXLdap7", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 }, "base_uri": "https://localhost:8080/", "height": 166 }, "outputId": "be8fb4d8-3f63-40ad-b7ec-810f97b5610a", "executionInfo": { "status": "ok", "timestamp": 1526667595453, "user_tz": 420, "elapsed": 26039, "user": { "displayName": "Christopher Olah", "photoUrl": "//lh5.googleusercontent.com/-GhJP0RTFLEs/AAAAAAAAAAI/AAAAAAAAEZ8/wDVK-lwJYfA/s50-c-k-no/photo.jpg", "userId": "104171973056281402320" } } }, "cell_type": "code", "source": [ "def interpolate_param_f():\n", " unique = param.fft_image((6, 128, 128, 3))\n", " shared = [\n", " param.lowres_tensor((6, 128, 128, 3), (1, 128//2, 128//2, 3)),\n", " param.lowres_tensor((6, 128, 128, 3), (1, 128//4, 128//4, 3)),\n", " param.lowres_tensor((6, 128, 128, 3), (1, 128//8, 128//8, 3)),\n", " param.lowres_tensor((6, 128, 128, 3), (2, 128//8, 128//8, 3)),\n", " param.lowres_tensor((6, 128, 128, 3), (1, 128//16, 128//16, 3)),\n", " param.lowres_tensor((6, 128, 128, 3), (2, 128//16, 128//16, 3)),\n", " ]\n", " return param.to_valid_rgb(unique + sum(shared), decorrelate=True)\n", "\n", "obj = objectives.channel_interpolate(\"mixed4a_pre_relu\", 476, \"mixed4a_pre_relu\", 460)\n", "\n", "_ = render.render_vis(model, obj, interpolate_param_f)" ], "execution_count": 0, "outputs": [ { "output_type": "stream", "text": [ "512 7413.614\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "text/plain": [ "" ], "text/html": [ "" ] }, "metadata": { "tags": [] } } ] } ] }