{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "name": "Aligned Interpolation.ipynb", "version": "0.3.2", "views": {}, "default_view": {}, "provenance": [], "collapsed_sections": [ "ISGK0WTSOuhH" ] }, "kernelspec": { "name": "python3", "display_name": "Python 3" }, "accelerator": "GPU" }, "cells": [ { "metadata": { "id": "ISGK0WTSOuhH", "colab_type": "text" }, "cell_type": "markdown", "source": [ "##### Copyright 2018 Google LLC.\n", "\n", "Licensed under the Apache License, Version 2.0 (the \"License\");" ] }, { "metadata": { "id": "Fx3QlXznOwE1", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 } } }, "cell_type": "code", "source": [ "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "5nno2iRZXJbX", "colab_type": "text" }, "cell_type": "markdown", "source": [ "# Aligned Feature Visualization Interpolation\n", "\n", "\n", "![](https://storage.googleapis.com/tensorflow-lucid/static/img/notebook-interpolation-header.png =802x)\n", "\n", "This notebook uses [**Lucid**](https://github.com/tensorflow/lucid) to create a visualizations that interpolates between two feature visualizations. On the way, it introduces custom parameterizations in lucid, which we will use here to build a shared parameterization that encourages alignment of visual landmarks between visualizations.\n", "\n", "This notebook doesn't introduce the abstractions behind lucid; you may wish to also read the [Lucid tutorial](https://colab.research.google.com/github/tensorflow/lucid/blob/master/notebooks/tutorial.ipynb).\n", "\n", "**Note**: The easiest way to use this tutorial is [as a colab notebook](https://colab.research.google.com/drive/1XuxLjIZj9MV-lRCpXHBhLo5A41Zs0f8E), which allows you to dive in with no setup. We recommend you enable a free GPU by going:\n", "\n", "> **Runtime**   →   **Change runtime type**   →   **Hardware Accelerator: GPU**\n" ] }, { "metadata": { "id": "FsFc1mE51tCd", "colab_type": "text" }, "cell_type": "markdown", "source": [ "## Install, Import, Load Model" ] }, { "metadata": { "id": "tavMPe3KQ8Cs", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 } } }, "cell_type": "code", "source": [ "!pip install -q lucid>=0.2.3" ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "RBr8QbboRAdU", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 } } }, "cell_type": "code", "source": [ "# Imports\n", "\n", "import numpy as np\n", "import tensorflow as tf\n", "\n", "import lucid.modelzoo.vision_models as models\n", "from lucid.misc.io import load, save, show\n", "\n", "import lucid.optvis.objectives as objectives\n", "import lucid.optvis.param as param\n", "import lucid.optvis.render as render" ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "yNALaA0QRJVT", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 } } }, "cell_type": "code", "source": [ "# Let's import a model from the Lucid modelzoo!\n", "\n", "model = models.InceptionV1()\n", "model.load_graphdef()" ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "VcT6e1BxY90X", "colab_type": "text" }, "cell_type": "markdown", "source": [ "## Creating an Interpolation" ] }, { "metadata": { "id": "G5LDE5mraW9u", "colab_type": "text" }, "cell_type": "markdown", "source": [ "First, let's take a look at the two neurons we want to interpolate between. This code should feel familiar from the Lucid tutorial:" ] }, { "metadata": { "id": "QVYxwt84Y9HM", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 } } }, "cell_type": "code", "source": [ "neuron1 = (\"mixed4a_pre_relu\", 476)\n", "neuron2 = (\"mixed4a_pre_relu\", 460)" ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "LyRUVLQ9TuKU", "colab_type": "text" }, "cell_type": "markdown", "source": [ "Many objective methods in lucid will take a `string` layer name and an `int` channel. \n", "I like to save them as tuples and use the destructuring operator (`*`) to conveniently pass them as arguments:" ] }, { "metadata": { "id": "OX6wUu-sab_L", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 }, "base_uri": "https://localhost:8080/", "height": 315 }, "outputId": "528c3358-745f-4c1b-d33b-97a7afb2bfd8", "executionInfo": { "status": "ok", "timestamp": 1532532333248, "user_tz": -120, "elapsed": 24384, "user": { "displayName": "", "photoUrl": "", "userId": "" } } }, "cell_type": "code", "source": [ "# For each neuron, we create a simple input parameterization, an objective, \n", "# and render a visualization:\n", "\n", "for neuron in [neuron1, neuron2]:\n", " param_f = lambda: param.image(128)\n", " objective = objectives.channel(*neuron)\n", " _ = render.render_vis(model, objective, param_f)" ], "execution_count": 4, "outputs": [ { "output_type": "stream", "text": [ "512 1281.2391\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "512 1672.5046\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": { "tags": [] } } ] }, { "metadata": { "id": "_Id7Dfg-tB4I", "colab_type": "text" }, "cell_type": "markdown", "source": [ "*You may wonder why these look a little less saturated than the images at the header of this notebook.\n", "The images in the header have simply been optimized for longer. If you need to recreate that look, you can set `thresholds=[2560]` on all `render_vis` calls. That will make working with this notebook a little slow, so we'll use the default thresholds for now.*" ] }, { "metadata": { "id": "d9n7Cps3bmra", "colab_type": "text" }, "cell_type": "markdown", "source": [ "When I know I'll deal with more than one neuron, I sometimes want to optimize them at the same time, rather than having to call `render_vis` twice.\n", "Currently I had to run it twice because `param.image(128, ...)` will give us a parameterization of shape `(1, 128, 128, 3)`--a tensor with a batch size of 1, or a single image. So, if I want to optimize two images at the same time, I will need to create a parameterization that can hold two images.\n", "\n", "To do so, I can use the `batch` parameter on many of the built-in objectives and parameterizations of lucid. I tell the parameterization to procide two image by supplying `batch=2` in the `param.image` call. I then explicitly set `batch=0` and `batch=1` on the two neuron objectives.\n", "\n", "This creates a single parameterization with a batch dimension of 2. Its shape will be `(2, 128, 128, 3)`. During rendering, [`render_viz` will call `np.hstack ` on this tensor](https://github.com/tensorflow/lucid/blob/master/lucid/optvis/render.py#L98), so it can be displayed as one wide image:" ] }, { "metadata": { "id": "tKl2aUSqcRaZ", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 }, "base_uri": "https://localhost:8080/", "height": 166 }, "outputId": "d766c03e-63e6-4595-93bc-84c677bde4dd", "executionInfo": { "status": "ok", "timestamp": 1532532347218, "user_tz": -120, "elapsed": 13928, "user": { "displayName": "", "photoUrl": "", "userId": "" } } }, "cell_type": "code", "source": [ "param_f = lambda: param.image(128, batch=2)\n", "objective = objectives.channel(*neuron1, batch=0) + objectives.channel(*neuron2, batch=1)\n", "_ = render.render_vis(model, objective, param_f)" ], "execution_count": 5, "outputs": [ { "output_type": "stream", "text": [ "512 2810.6096\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": { "tags": [] } } ] }, { "metadata": { "id": "J0xnhbWGeZGF", "colab_type": "text" }, "cell_type": "markdown", "source": [ "Now that we've seen parameterizations with multiple images, you can already see how we will get to the interpolation--we will just add more images to the batch dimension, and for each of them will use a linear combination of the two neuron objectives! Conceptually, it will look something like this:\n", "\n", "![](https://storage.googleapis.com/tensorflow-lucid/static/img/notebook-interpolation-two-to-five.png =800x)\n", "\n", "That is going to be a lot of objectives! Let's give it a try:" ] }, { "metadata": { "id": "5cSiL4Z9odF3", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 }, "base_uri": "https://localhost:8080/", "height": 166 }, "outputId": "732ca0a4-2d34-4306-9e09-765de75e44f3", "executionInfo": { "status": "ok", "timestamp": 1532532370730, "user_tz": -120, "elapsed": 23476, "user": { "displayName": "", "photoUrl": "", "userId": "" } } }, "cell_type": "code", "source": [ "param_f = lambda: param.image(128, batch=5)\n", "objective = objectives.Objective.sum([\n", " # neuron 1 objectives, orange row:\n", " 1.00 * objectives.channel(*neuron1, batch=0),\n", " 0.75 * objectives.channel(*neuron1, batch=1),\n", " 0.50 * objectives.channel(*neuron1, batch=2),\n", " 0.25 * objectives.channel(*neuron1, batch=3),\n", " 0.00 * objectives.channel(*neuron1, batch=4),\n", " # neuron 2 objectives, green row:\n", " 0.00 * objectives.channel(*neuron2, batch=0),\n", " 0.25 * objectives.channel(*neuron2, batch=1),\n", " 0.50 * objectives.channel(*neuron2, batch=2),\n", " 0.75 * objectives.channel(*neuron2, batch=3),\n", " 1.00 * objectives.channel(*neuron2, batch=4),\n", "])\n", "_ = render.render_vis(model, objective, param_f)" ], "execution_count": 6, "outputs": [ { "output_type": "stream", "text": [ "512 5700.5576\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": { "tags": [] } } ] }, { "metadata": { "id": "cic1CfgDuZsz", "colab_type": "text" }, "cell_type": "markdown", "source": [ "Nice! You can already see some interesting interactions between these neurons. For example, in the second image take a look at how the art-pattern starts to include round ovals.\n", "\n", "To make specifying objectives like this easier, Lucid has [a built in objective](https://github.com/tensorflow/lucid/blob/master/lucid/optvis/objectives.py#L282) called `channel_interpolate`. Now that you've seen how we built this from scratch, I encourage you to take a look at the code! **When you understand the underlying primitives, there's surprisingly little magic going on in Lucid.**\n", "\n", "So here's a more terse version of the above. (It's also running for twice as many steps.):" ] }, { "metadata": { "id": "EI3W3HvlZayV", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 }, "base_uri": "https://localhost:8080/", "height": 166 }, "outputId": "7db59e64-7ddf-4081-e84c-a4488db1764b", "executionInfo": { "status": "ok", "timestamp": 1532532411804, "user_tz": -120, "elapsed": 41014, "user": { "displayName": "", "photoUrl": "", "userId": "" } } }, "cell_type": "code", "source": [ "objective = objectives.channel_interpolate(*neuron1, *neuron2)\n", "param_f = lambda: param.image(128, batch=5)\n", "_ = render.render_vis(model, objective, param_f=param_f, thresholds=[1024])" ], "execution_count": 7, "outputs": [ { "output_type": "stream", "text": [ "1024 6233.812\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": { "tags": [] } } ] }, { "metadata": { "id": "J300jaE9wZ8M", "colab_type": "text" }, "cell_type": "markdown", "source": [ "## Introducing Alignment\n", "\n", "The interpolation in the last segment already helps understand how neurons interact, but visuakl landmarks such as eyes change position from one frame to the next. \n", "This lack of alignment can make it harder to see the difference due to slightly different objectives, because they’re swamped by the much larger differences in layout.\n", "\n", "We can see the issue with independent optimization if we look at the interpolated frames as an animation:" ] }, { "metadata": { "id": "-lyYhhUMxnf9", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 }, "base_uri": "https://localhost:8080/", "height": 145 }, "cellView": "code", "outputId": "4622184c-1229-4229-fa31-82977bd6414a", "executionInfo": { "status": "ok", "timestamp": 1532532412599, "user_tz": -120, "elapsed": 760, "user": { "displayName": "", "photoUrl": "", "userId": "" } } }, "cell_type": "code", "source": [ "# You can use `Clear Output` if the animation gets annoying.\n", "%%html\n", "
" ], "execution_count": 8, "outputs": [ { "output_type": "display_data", "data": { "text/html": [ "
" ], "text/plain": [ "" ] }, "metadata": { "tags": [] } } ] }, { "metadata": { "id": "gc93tydWzTHz", "colab_type": "text" }, "cell_type": "markdown", "source": [ "Since each image was optimized independently, it should not surprise us too much that they don't look similar!\n", "\n", "In fact, this will happen even if we optimize for the same objective every time. Since we create these visualizations from random noise, every random initialization will look slightly different:" ] }, { "metadata": { "id": "yA165XrCZzS6", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 }, "base_uri": "https://localhost:8080/", "height": 169 }, "outputId": "b9c7aff8-ce79-40fb-f060-a193185149be", "executionInfo": { "status": "ok", "timestamp": 1532532452921, "user_tz": -120, "elapsed": 40269, "user": { "displayName": "", "photoUrl": "", "userId": "" } } }, "cell_type": "code", "source": [ "objective = sum([objectives.channel(*neuron2, batch=i) for i in range(5)])\n", "param_f = lambda: param.image(128, batch=5, sd=0.5) # setting sd to .5 increases the noise used to initialize the image parameterization\n", "images = render.render_vis(model, objective, param_f=param_f, thresholds=[1024], verbose=False)[-1]\n", "show(images)" ], "execution_count": 9, "outputs": [ { "output_type": "display_data", "data": { "text/html": [ "
\n", " 0
\n", " \n", "
\n", " 1
\n", " \n", "
\n", " 2
\n", " \n", "
\n", " 3
\n", " \n", "
\n", " 4
\n", " \n", "
" ], "text/plain": [ "" ] }, "metadata": { "tags": [] } } ] }, { "metadata": { "id": "jKwrR-jsah-1", "colab_type": "text" }, "cell_type": "markdown", "source": [ "There are many plausible ways to go about making these images more aligned. For example, we could use additional objectives that penalizes the difference between neighboring images. In this notebook, however, we want to introduce a different technique that will turn out quite elegant: **a shared parameterization**.\n", "\n", "\n", "### Parameterizations\n", "We first need to take a deeper look at `lucid.optvis.param`'s child modules. So far we always used a convenience method called `image` from the creatively named `images` module. Under the hood, `image` assembles a parameterization from primitives found in the first two of the following modules. Let's import them now:" ] }, { "metadata": { "id": "rawhChf1HGWR", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 } } }, "cell_type": "code", "source": [ "from lucid.optvis.param import spatial, color, lowres" ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "f_Nmzk5ZHIYl", "colab_type": "text" }, "cell_type": "markdown", "source": [ "As a reminder, this is the parameterization we used so far:\n", "\n", "```python\n", "param.image(128, batch=5)\n", "```\n", "\n", "[Taking a peek at the source code](), this assembles roughly the following pieces:" ] }, { "metadata": { "id": "gBGu8GB4Hglo", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 } } }, "cell_type": "code", "source": [ "def default_image_param_f(batch = 1, size = 128, channels = 3):\n", " shape = (batch, size, size, channels)\n", " fft_parameterization = spatial.fft_image(shape)\n", " rgb_parameterization = color.to_valid_rgb(fft_parameterization, decorrelate=True)\n", " return rgb_parameterization" ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "5qAFv0n7Mr-N", "colab_type": "text" }, "cell_type": "markdown", "source": [ "Let's just quickly sanity check that this still works:" ] }, { "metadata": { "id": "ptz1Y4fcLlzk", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 }, "base_uri": "https://localhost:8080/", "height": 166 }, "outputId": "0892fec3-f1bb-4a4a-cae9-c5f37c42de3f", "executionInfo": { "status": "ok", "timestamp": 1532532465846, "user_tz": -120, "elapsed": 11300, "user": { "displayName": "", "photoUrl": "", "userId": "" } } }, "cell_type": "code", "source": [ "objective = objectives.channel(*neuron1)\n", "_ = render.render_vis(model, objective, default_image_param_f)" ], "execution_count": 12, "outputs": [ { "output_type": "stream", "text": [ "512 1235.2769\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": { "tags": [] } } ] }, { "metadata": { "id": "LKahzCT1N5Pz", "colab_type": "text" }, "cell_type": "markdown", "source": [ "…looks like it does! So now that we can assemble custom parameterizations, let's look at how to use this skill to achieve better alignment between our interpolated neurons." ] }, { "metadata": { "id": "5_y2DtUkyzmG", "colab_type": "text" }, "cell_type": "markdown", "source": [ "### Introducing `lowres_tensor`\n", "\n", "What we mean by shared parameterization is that a part of the image parameterization will be shared between all images, and another part will be unique to each of them:\n", "\n", "![lowres_tensor](https://storage.googleapis.com/tensorflow-lucid/static/img/notebook-interpolation-lowres-tensor-1.png =489x)\n", "\n", "`lowres_tensor` takes two shapes as arguments, and creates a per-dimension bilinear interpolation between those shapes. Let's break this down in a simple case:\n", "\n", "Let's say we have an image parameterization like before, with a shape of (5, 128, 128, 3). Then we could ask for a `lowres_tensor` of shape (**1**, 128, 128, 3) to get a shared parameterization, like this:" ] }, { "metadata": { "id": "9SdwaHVQfZj-", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 } } }, "cell_type": "code", "source": [ "def simple_shared_param_f():\n", " shared = lowres.lowres_tensor((6, 128, 128, 3), (1, 128, 128, 3))\n", " return color.to_valid_rgb(shared, decorrelate=True)" ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "AMnjhrPhfbrH", "colab_type": "text" }, "cell_type": "markdown", "source": [ "If we used this in our `channel_interpolate` objective, what would we expect to see?" ] }, { "metadata": { "id": "mvkJL-MJfeFd", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 }, "base_uri": "https://localhost:8080/", "height": 166 }, "outputId": "4fd86a08-5d26-4c7c-fa44-124602d3973d", "executionInfo": { "status": "ok", "timestamp": 1532532489864, "user_tz": -120, "elapsed": 22817, "user": { "displayName": "", "photoUrl": "", "userId": "" } } }, "cell_type": "code", "source": [ "objective = objectives.channel_interpolate(*neuron1, *neuron2)\n", "_ = render.render_vis(model, objective, param_f=simple_shared_param_f)" ], "execution_count": 14, "outputs": [ { "output_type": "stream", "text": [ "512 6464.317\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": { "tags": [] } } ] }, { "metadata": { "id": "YTGps2PbgknR", "colab_type": "text" }, "cell_type": "markdown", "source": [ "So even though our *objective* here is interpolating between two neurons, our parameterization is *entirely shared*, so all images end up looking the same.\n", "\n", "At first glance this may seem entirely undesireable. However it turns out that this is very helpful for creating a *part* of our final parameterization: the part that will be shared between all images and cause the interpolation to align better.\n", "\n", "To get closer to our goal, consider that `lowres_tensor` can interpolate along any dimension, not just batch. For example, it can interpolate between spatial dimensions:\n", "\n", "![lowres_tensor_2](https://storage.googleapis.com/tensorflow-lucid/static/img/notebook-interpolation-lowres-tensor-2.png =489x)\n", "\n", "Let's take a look at what optimization result such a parameterization results in, too:\n" ] }, { "metadata": { "id": "b2nWzjbvyzEO", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 } } }, "cell_type": "code", "source": [ "def lowres_param_f():\n", " shared = lowres.lowres_tensor((6, 128, 128, 3), (1, 128//16, 128//16, 3))\n", " return color.to_valid_rgb(shared, decorrelate=True)" ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "timqOOcxzSa-", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 }, "base_uri": "https://localhost:8080/", "height": 166 }, "outputId": "e7530b3c-569e-4dee-c930-35dd7e0dc637", "executionInfo": { "status": "ok", "timestamp": 1532532512705, "user_tz": -120, "elapsed": 21687, "user": { "displayName": "", "photoUrl": "", "userId": "" } } }, "cell_type": "code", "source": [ "objective = objectives.channel_interpolate(*neuron1, *neuron2)\n", "_ = render.render_vis(model, objective, param_f=lowres_param_f)" ], "execution_count": 16, "outputs": [ { "output_type": "stream", "text": [ "512 363.29443\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": { "tags": [] } } ] }, { "metadata": { "id": "9rv2xWL8128o", "colab_type": "text" }, "cell_type": "markdown", "source": [ "We can also combine interpolation between batch and spatial dimensions:" ] }, { "metadata": { "id": "IZkKnzOi17oM", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 } } }, "cell_type": "code", "source": [ "def lowres_interpolate_param_f():\n", " shared = lowres.lowres_tensor((6, 128, 128, 3), (2, 128//16, 128//16, 3))\n", " return color.to_valid_rgb(shared, decorrelate=True)" ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "xaX5NY3V19Qk", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 }, "base_uri": "https://localhost:8080/", "height": 166 }, "outputId": "c7924bf1-e510-40f6-c815-59c135059279", "executionInfo": { "status": "ok", "timestamp": 1532532535879, "user_tz": -120, "elapsed": 22274, "user": { "displayName": "", "photoUrl": "", "userId": "" } } }, "cell_type": "code", "source": [ "objective = objectives.channel_interpolate(*neuron1, *neuron2)\n", "_ = render.render_vis(model, objective, param_f=lowres_interpolate_param_f)" ], "execution_count": 18, "outputs": [ { "output_type": "stream", "text": [ "512 448.99667\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": { "tags": [] } } ] }, { "metadata": { "id": "t9q6T9Q3y2Ae", "colab_type": "text" }, "cell_type": "markdown", "source": [ "## Final Parameterization\n", "\n", "Let's combine multiple of these shared representations with one that allows each image to be unique, like showjn in [Figure 3 of Differentiable Image Parameterizations](https://distill.pub/2018/differentiable-parameterizations/#figure-aligned-interpolation-examples):\n", "\n", "![](https://storage.googleapis.com/tensorflow-lucid/notebooks/aligned-interpolation/header.png =713x)" ] }, { "metadata": { "id": "3HE5cCSlZrwa", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 } } }, "cell_type": "code", "source": [ "def interpolate_f():\n", " unique = spatial.fft_image((6, 128, 128, 3))\n", " shared = [\n", " lowres.lowres_tensor((6, 128, 128, 3), (1, 128//2, 128//2, 3)),\n", " lowres.lowres_tensor((6, 128, 128, 3), (1, 128//4, 128//4, 3)),\n", " lowres.lowres_tensor((6, 128, 128, 3), (1, 128//8, 128//8, 3)),\n", " lowres.lowres_tensor((6, 128, 128, 3), (2, 128//8, 128//8, 3)),\n", " lowres.lowres_tensor((6, 128, 128, 3), (1, 128//16, 128//16, 3)),\n", " lowres.lowres_tensor((6, 128, 128, 3), (2, 128//16, 128//16, 3)),\n", " ]\n", " return color.to_valid_rgb(unique + sum(shared), decorrelate=True)" ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "JVqekjjIBupw", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 } } }, "cell_type": "code", "source": [ "objective = objectives.channel_interpolate(*neuron1, *neuron2)\n", "images = render.render_vis(model, objective, param_f=interpolate_f, verbose=False)" ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "tqvSKoaskbSu", "colab_type": "text" }, "cell_type": "markdown", "source": [ "Sometimes it can be helpful to show the results as individual images to better see where each frame ends, and how well these images already align.\n", "\n", "`render_viz` returns a list of optimization results whenever the step number is equal to a `threshold`. If you don't explicitly specify them, there's only one threshold. Let's get the last optimization results and use `show` to display them as a list of images. \n", "\n", "(`lucid.misc.io.show` inspects a tensor's shape interprets certain rank 4 tensors as batches of images automatically:)" ] }, { "metadata": { "id": "IzwdtVqrjvq2", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 }, "base_uri": "https://localhost:8080/", "height": 169 }, "outputId": "2ad5517e-6ba2-41b8-a401-c1a776ae1a9a", "executionInfo": { "status": "ok", "timestamp": 1532532565461, "user_tz": -120, "elapsed": 1651, "user": { "displayName": "", "photoUrl": "", "userId": "" } } }, "cell_type": "code", "source": [ "result = images[-1]\n", "show(result)" ], "execution_count": 21, "outputs": [ { "output_type": "display_data", "data": { "text/html": [ "
\n", " 0
\n", " \n", "
\n", " 1
\n", " \n", "
\n", " 2
\n", " \n", "
\n", " 3
\n", " \n", "
\n", " 4
\n", " \n", "
\n", " 5
\n", " \n", "
" ], "text/plain": [ "" ] }, "metadata": { "tags": [] } } ] }, { "metadata": { "id": "GKPoTuVBlE5-", "colab_type": "text" }, "cell_type": "markdown", "source": [ "Note how for example the yellow swirl in the top left corner changes and transitions to an eye!\n", "\n", "(If you have re-run that cell this description may not be exact anymore, but you should see similarly well-aligned feature visualizations.)\n", "\n", "We can play this one as an animation again. The result is still not perfectly aligned, but much better than what we started with." ] }, { "metadata": { "id": "0h7uO-593dX3", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 }, "base_uri": "https://localhost:8080/", "height": 145 }, "outputId": "5dcf245b-01d4-470a-9a94-6e1278a806b6", "executionInfo": { "status": "ok", "timestamp": 1532532566727, "user_tz": -120, "elapsed": 1191, "user": { "displayName": "", "photoUrl": "", "userId": "" } } }, "cell_type": "code", "source": [ "# You can use `Clear Output` if the animation gets annoying.\n", "%%html\n", "
" ], "execution_count": 22, "outputs": [ { "output_type": "display_data", "data": { "text/html": [ "
" ], "text/plain": [ "" ] }, "metadata": { "tags": [] } } ] }, { "metadata": { "id": "hzJla0LvnFtL", "colab_type": "text" }, "cell_type": "markdown", "source": [ "To get the last step to appear even smoother, we can even add an additional alignment objective that encourages the activations of layers to be similar when images are close together in the batch dimension. At that point we're not strictly optimizing for the same objective anymore (because neighbors in the batch dimension will influence each other), so take this as a mostly aesthetic experiment." ] }, { "metadata": { "id": "jOCYDhRrnPjp", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 }, "base_uri": "https://localhost:8080/", "height": 762 }, "outputId": "9119d879-c5a5-4c2f-e677-aa23d47d6e80", "executionInfo": { "status": "ok", "timestamp": 1532532628691, "user_tz": -120, "elapsed": 61885, "user": { "displayName": "", "photoUrl": "", "userId": "" } } }, "cell_type": "code", "source": [ "interpolation_objective = objectives.channel_interpolate(*neuron1, *neuron2)\n", "alignment_objective = objectives.alignment('mixed3b', decay_ratio=5) + objectives.alignment('mixed4a', decay_ratio=5)\n", "objective = interpolation_objective + 1e-1 * alignment_objective\n", "\n", "images = render.render_vis(model, objective, param_f=interpolate_f, verbose=True, thresholds=(32,64,256,512, 1024))" ], "execution_count": 23, "outputs": [ { "output_type": "stream", "text": [ "32 3412.4922\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "64 4702.838\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "256 6419.515\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "512 7276.9077\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "1024 6642.4263\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": { "tags": [] } } ] }, { "metadata": { "id": "TEsBK9x1pShO", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 }, "base_uri": "https://localhost:8080/", "height": 145 }, "outputId": "ba0b1369-4350-4f82-d5d6-06e6b46863ef", "executionInfo": { "status": "ok", "timestamp": 1532532630075, "user_tz": -120, "elapsed": 1342, "user": { "displayName": "", "photoUrl": "", "userId": "" } } }, "cell_type": "code", "source": [ "# You can use `Clear Output` if the animation gets annoying.\n", "%%html\n", "
" ], "execution_count": 24, "outputs": [ { "output_type": "display_data", "data": { "text/html": [ "
" ], "text/plain": [ "" ] }, "metadata": { "tags": [] } } ] }, { "metadata": { "id": "SxuIixzSC3HX", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 } } }, "cell_type": "code", "source": [ "" ], "execution_count": 0, "outputs": [] } ] }