{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "name": "Neuron Groups - Building Blocks of Interpretability", "version": "0.3.2", "views": {}, "default_view": {}, "provenance": [ { "file_id": "1QA7GEHXJYOWld7jitZei-Qy5US0iTeN_", "timestamp": 1518982029535 }, { "file_id": "1uRqpBNPg-aW3tRU-uo-mWg6cQxuAquHW", "timestamp": 1518822563463 } ], "collapsed_sections": [] }, "kernelspec": { "name": "python2", "display_name": "Python 2" }, "accelerator": "GPU" }, "cells": [ { "metadata": { "id": "JndnmDMp66FL", "colab_type": "text" }, "cell_type": "markdown", "source": [ "##### Copyright 2018 Google LLC.\n", "\n", "Licensed under the Apache License, Version 2.0 (the \"License\");" ] }, { "metadata": { "id": "hMqWDc_m6rUC", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 } }, "cellView": "both" }, "cell_type": "code", "source": [ "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "pNqKk1MmrakH", "colab_type": "text" }, "cell_type": "markdown", "source": [ "# Neuron Groups -- Building Blocks of Interpretability\n", "\n", "This colab notebook is part of our **Building Blocks of Intepretability** series exploring how intepretability techniques combine together to explain neural networks. If you haven't already, make sure to look at the [**corresponding paper**](https://distill.pub/2018/building-blocks) as well!\n", "\n", "This notebook demonstrates **Neuron Groups**, a technique for exploring how detectors a different spatial positions in the network effected its output.\n", "\n", "
\n", "\n", "\n", "\n", "
\n", "\n", "This tutorial is based on [**Lucid**](https://github.com/tensorflow/lucid), a network for visualizing neural networks. Lucid is a kind of spiritual successor to DeepDream, but provides flexible abstractions so that it can be used for a wide range of interpretability research.\n", "\n", "**Note**: The easiest way to use this tutorial is [as a colab notebook](), which allows you to dive in with no setup. We recommend you enable a free GPU by going:\n", "\n", "> **Runtime**   →   **Change runtime type**   →   **Hardware Accelerator: GPU**\n", "\n", "Thanks for trying Lucid!\n" ] }, { "metadata": { "id": "hOBBuzMaxU37", "colab_type": "text" }, "cell_type": "markdown", "source": [ "# Install / Import / Load" ] }, { "metadata": { "id": "UL1yOZtjqkcj", "colab_type": "text" }, "cell_type": "markdown", "source": [ "This code depends on [Lucid](https://github.com/tensorflow/lucid) (our visualization library), and [svelte](https://svelte.technology/) (a web framework). The following cell will install both of them, and dependancies such as TensorFlow. And then import them as appropriate." ] }, { "metadata": { "id": "AA17rJBLuyYH", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 }, "output_extras": [ {} ], "base_uri": "https://localhost:8080/", "height": 99 }, "outputId": "5e6b9516-0e6c-4a41-c877-925c6ec16d51", "executionInfo": { "status": "ok", "timestamp": 1520297343023, "user_tz": 480, "elapsed": 4989, "user": { "displayName": "Christopher Olah", "photoUrl": "//lh5.googleusercontent.com/-GhJP0RTFLEs/AAAAAAAAAAI/AAAAAAAAEZ8/wDVK-lwJYfA/s50-c-k-no/photo.jpg", "userId": "104171973056281402320" } } }, "cell_type": "code", "source": [ "!pip install --quiet lucid==0.0.5\n", "!npm install -g svelte-cli@2.2.0\n", "\n", "import numpy as np\n", "import tensorflow as tf\n", "\n", "import lucid.modelzoo.vision_models as models\n", "from lucid.misc.io import show\n", "import lucid.misc.io.showing as showing\n", "from lucid.misc.channel_reducer import ChannelReducer\n", "import lucid.optvis.param as param\n", "import lucid.optvis.objectives as objectives\n", "import lucid.optvis.render as render\n", "from lucid.misc.io import show, load\n", "from lucid.misc.io.reading import read\n", "from lucid.misc.io.showing import _image_url\n", "import lucid.scratch.web.svelte as lucid_svelte\n", "from lucid.misc.gradient_override import gradient_override_map" ], "execution_count": 0, "outputs": [ { "output_type": "stream", "text": [ "\u001b[K\u001b[?25h/tools/node/bin/svelte -> /tools/node/lib/node_modules/svelte-cli/bin.js\n", "/tools/node/lib\n", "└─┬ \u001b[40m\u001b[33msvelte-cli@2.2.0\u001b[39m\u001b[49m \n", " └── \u001b[40m\u001b[33msvelte@1.56.2\u001b[39m\u001b[49m \n", "\n" ], "name": "stdout" } ] }, { "metadata": { "id": "0cUPBCRyG9xE", "colab_type": "text" }, "cell_type": "markdown", "source": [ "# Attribution & UI Code" ] }, { "metadata": { "id": "FWHqimIqk2Bs", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 } } }, "cell_type": "code", "source": [ "model = models.InceptionV1()\n", "model.load_graphdef()" ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "xIDcG0vjaDtk", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 } } }, "cell_type": "code", "source": [ "labels_str = read(\"https://gist.githubusercontent.com/aaronpolhamus/964a4411c0906315deb9f4a3723aac57/raw/aa66dd9dbf6b56649fa3fab83659b2acbf3cbfd1/map_clsloc.txt\")\n", "labels = [line[line.find(\" \"):].strip() for line in labels_str.split(\"\\n\")]\n", "labels = [label[label.find(\" \"):].strip().replace(\"_\", \" \") for label in labels]\n", "labels = [\"dummy\"] + labels" ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "PNiXhK5AYR0d", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 }, "output_extras": [ {} ], "base_uri": "https://localhost:8080/", "height": 164 }, "outputId": "93326ac0-ebbf-4caf-c2fe-c39415edd2b6", "executionInfo": { "status": "ok", "timestamp": 1520297344449, "user_tz": 480, "elapsed": 571, "user": { "displayName": "Christopher Olah", "photoUrl": "//lh5.googleusercontent.com/-GhJP0RTFLEs/AAAAAAAAAAI/AAAAAAAAEZ8/wDVK-lwJYfA/s50-c-k-no/photo.jpg", "userId": "104171973056281402320" } } }, "cell_type": "code", "source": [ "%%html_define_svelte GroupWidget\n", "\n", "
\n", "
\n", " \n", " {{#if pres_n != undefined}}\n", " \n", " {{/if}}\n", "
\n", " \n", "
\n", "
\n", " {{#each range(n_groups) as n}}\n", " {{#if pres_n == undefined || pres_n == n}}\n", " \n", " {{/if}}\n", " {{/each}}\n", "
\n", " \n", "
\n", "
\n", " \n", "
\n", " {{#each range(n_groups) as n}}\n", "
\n", " \n", "
\n", " {{/each}}\n", "
\n", " \n", "
\n", "\n", "\n", "\n", "\n", "" ], "execution_count": 0, "outputs": [ { "output_type": "stream", "text": [ "Trying to build svelte component from html...\n", "svelte compile --format iife /tmp/svelte_Frqeru/GroupWidget_1cb0e0d.html > /tmp/svelte_Frqeru/GroupWidget_1cb0e0d.js\n", "svelte version 1.56.2\n", "compiling ../tmp/svelte_Frqeru/GroupWidget_1cb0e0d.html...\n", "(4:4) – A11y: element should have an alt attribute\n", "(6:4) – A11y: element should have an alt attribute\n", "(14:4) – A11y: element should have an alt attribute\n", "(27:6) – A11y: element should have an alt attribute\n", "\n" ], "name": "stdout" } ] }, { "metadata": { "id": "6jshLh5FisuC", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 } } }, "cell_type": "code", "source": [ "def raw_class_group_attr(img, layer, label, group_vecs, override=None):\n", " \"\"\"How much did spatial positions at a given layer effect a output class?\"\"\"\n", "\n", " # Set up a graph for doing attribution...\n", " with tf.Graph().as_default(), tf.Session(), gradient_override_map(override or {}):\n", " t_input = tf.placeholder_with_default(img, [None, None, 3])\n", " T = render.import_model(model, t_input, t_input)\n", " \n", " # Compute activations\n", " acts = T(layer).eval()\n", " \n", " if label is None: return np.zeros(acts.shape[1:-1])\n", " \n", " # Compute gradient\n", " score = T(\"softmax2_pre_activation\")[0, labels.index(label)]\n", " t_grad = tf.gradients([score], [T(layer)])[0] \n", " grad = t_grad.eval({T(layer) : acts})\n", " \n", " # Linear approximation of effect of spatial position\n", " return [np.sum(group_vec * grad) for group_vec in group_vecs]" ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "JW6FWx8rZMCO", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 } } }, "cell_type": "code", "source": [ "def neuron_groups(img, layer, n_groups=6, attr_classes=[]):\n", "\n", " # Compute activations\n", "\n", " with tf.Graph().as_default(), tf.Session():\n", " t_input = tf.placeholder_with_default(img, [None, None, 3])\n", " T = render.import_model(model, t_input, t_input)\n", " acts = T(layer).eval()\n", "\n", " # We'll use ChannelReducer (a wrapper around scikit learn's factorization tools)\n", " # to apply Non-Negative Matrix factorization (NMF).\n", "\n", " nmf = ChannelReducer(n_groups, \"NMF\")\n", " spatial_factors = nmf.fit_transform(acts)[0].transpose(2, 0, 1).astype(\"float32\")\n", " channel_factors = nmf._reducer.components_.astype(\"float32\")\n", "\n", " # Let's organize the channels based on their horizontal position in the image\n", "\n", " x_peak = np.argmax(spatial_factors.max(1), 1)\n", " ns_sorted = np.argsort(x_peak)\n", " spatial_factors = spatial_factors[ns_sorted]\n", " channel_factors = channel_factors[ns_sorted]\n", "\n", " # And create a feature visualziation of each group\n", "\n", " param_f = lambda: param.image(80, batch=n_groups)\n", " obj = sum(objectives.direction(layer, channel_factors[i], batch=i)\n", " for i in range(n_groups))\n", " group_icons = render.render_vis(model, obj, param_f, verbose=False)[-1]\n", " \n", " # We'd also like to know about attribution\n", " \n", " # First, let's turn each group into a vector over activations\n", " group_vecs = [spatial_factors[i, ..., None]*channel_factors[i]\n", " for i in range(n_groups)]\n", " \n", " attrs = np.asarray([raw_class_group_attr(img, layer, attr_class, group_vecs)\n", " for attr_class in attr_classes])\n", " \n", " print attrs\n", " \n", " # Let's render the visualization!\n", " \n", " lucid_svelte.GroupWidget({\n", " \"img\" : _image_url(img),\n", " \"n_groups\": n_groups,\n", " \"spatial_factors\": [_image_url(factor[..., None]/np.percentile(spatial_factors,99)*[1,0,0]) for factor in spatial_factors], \n", " \"group_icons\": [_image_url(icon) for icon in group_icons] \n", " })\n", " \n", " " ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "LE8e0ZkVajTz", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 }, "output_extras": [ {}, {} ], "base_uri": "https://localhost:8080/", "height": 388 }, "outputId": "ca55f78a-5ed5-4bf9-9424-9a2e81cf8efd", "executionInfo": { "status": "ok", "timestamp": 1520297389746, "user_tz": 480, "elapsed": 19287, "user": { "displayName": "Christopher Olah", "photoUrl": "//lh5.googleusercontent.com/-GhJP0RTFLEs/AAAAAAAAAAI/AAAAAAAAEZ8/wDVK-lwJYfA/s50-c-k-no/photo.jpg", "userId": "104171973056281402320" } } }, "cell_type": "code", "source": [ "img = load(\"https://storage.googleapis.com/lucid-static/building-blocks/examples/dog_cat.png\")\n", "neuron_groups(img, \"mixed4d\", 6, [\"Labrador retriever\", \"tiger cat\"])" ], "execution_count": 0, "outputs": [ { "output_type": "stream", "text": [ "[[ 3.7463202 3.7567782 -1.1973696 0.7144106 -0.5310478 -2.1201751 ]\n", " [-0.50556946 -0.31257683 -1.9789598 0.6788014 1.1792561 0.76901126]]\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "text/plain": [ "" ], "text/html": [ "\n", "
\n", " \n", " \n", " " ] }, "metadata": { "tags": [] } } ] }, { "metadata": { "id": "EKY9bPKUWYL8", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 }, "output_extras": [ {}, {} ], "base_uri": "https://localhost:8080/", "height": 371 }, "outputId": "04722ae4-31d3-4048-8463-368d4bf65266", "executionInfo": { "status": "ok", "timestamp": 1520297406466, "user_tz": 480, "elapsed": 16667, "user": { "displayName": "Christopher Olah", "photoUrl": "//lh5.googleusercontent.com/-GhJP0RTFLEs/AAAAAAAAAAI/AAAAAAAAEZ8/wDVK-lwJYfA/s50-c-k-no/photo.jpg", "userId": "104171973056281402320" } } }, "cell_type": "code", "source": [ "img = load(\"https://storage.googleapis.com/lucid-static/building-blocks/examples/flowers.png\")\n", "neuron_groups(img, \"mixed4d\", 5)" ], "execution_count": 0, "outputs": [ { "output_type": "stream", "text": [ "[]\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "text/plain": [ "" ], "text/html": [ "\n", "
\n", " \n", " \n", " " ] }, "metadata": { "tags": [] } } ] }, { "metadata": { "id": "clUQ4PmrXBXd", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 } } }, "cell_type": "code", "source": [ "" ], "execution_count": 0, "outputs": [] }, { "metadata": { "id": "IGZxhPneh79e", "colab_type": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 } } }, "cell_type": "code", "source": [ "" ], "execution_count": 0, "outputs": [] } ] }