{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Handle changes in prediction during robustness evaluation.\n",
    "[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/understandable-machine-intelligence-lab/Quantus/main?labpath=tutorials%2FTutorial_Handle_Changes_In_Predictions.ipynb)\n",
    "\n",
    "\n",
    "Typically, during robustness evaluation, we want model prediction to stay the same.\n",
    "This behaviour is, however, very sensitive to perturbation function and its hyperparameter choices.\n",
    "In this notebook we demonstrate how this could be handled in `quantus` using a simple motivating example with Average Sensitivity Metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU')]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import dependencies.\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "import tensorflow_datasets as tfds\n",
    "import quantus\n",
    "\n",
    "tf.config.list_physical_devices()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1) Preliminaries\n",
    "\n",
    "### 1.1 Load ImageNet subset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(32, 224, 224, 3)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IMG_SIZE = 224\n",
    "\n",
    "ds = tfds.load(\n",
    "    \"imagenet_v2\",\n",
    "    split=[\"test\"],\n",
    "    as_supervised=True,\n",
    "    try_gcs=True,\n",
    "    batch_size=32,\n",
    "    data_dir=\"/tmp/tensorflow_datasets/\",\n",
    ")\n",
    "\n",
    "x_batch, y_batch = ds[0].take(1).as_numpy_iterator().next()\n",
    "x_batch = tf.image.resize(x_batch, (IMG_SIZE, IMG_SIZE)).numpy()\n",
    "x_batch.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2. Load pre-trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<KerasTensor: shape=(None, 224, 224, 3) dtype=float32 (created by layer 'input_3')>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = tf.keras.applications.MobileNetV2()\n",
    "model.input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3. Generate batch of predictions and explanations using baseline method \"IntegratedGradients\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(32, 224, 224)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_predicted = model.predict(x_batch).argmax(axis=1)\n",
    "a_batch_intgrad = quantus.explain(\n",
    "    model, x_batch, y_predicted, method=\"IntegratedGradients\"\n",
    ")\n",
    "a_batch_intgrad.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2) Quantitative evaluation with Quantus\n",
    "\n",
    "We can evaluate the robustness of our explanations on a variety of quantitative criteria, but as a motivating example we test the Average Sensitivity (Yeh at el., 2019) of the explanations. This metric tests how the explanations change on average while subject to slight perturbations.\n",
    "\n",
    "All robustness metrics accept constructor keyword argument `return_nan_when_prediction_changes`, as the name suggests,\n",
    "when set to true, the metric will be evaluated to NaN if the prediction changes after the perturbation is applied."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "47a0ef0f36824da0b4430a53dd56e3e5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "results = quantus.evaluate(\n",
    "    metrics={\n",
    "        \"DefaultAvgSensitivity\": quantus.AvgSensitivity(\n",
    "            nr_samples=20, disable_warnings=True, display_progressbar=True\n",
    "        ),\n",
    "        \"AvgSensitivityWithNan\": quantus.AvgSensitivity(\n",
    "            nr_samples=20,\n",
    "            disable_warnings=True,\n",
    "            return_nan_when_prediction_changes=True,\n",
    "            display_progressbar=True,\n",
    "        ),\n",
    "    },\n",
    "    xai_methods={\"IntegratedGradients\": a_batch_intgrad},\n",
    "    model=model,\n",
    "    x_batch=x_batch,\n",
    "    y_batch=y_batch,\n",
    "    explain_func_kwargs={\"method\": \"IntegratedGradients\"},\n",
    "    call_kwargs={\"0\": {\"softmax\": True, \"channel_first\": True}},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Display results in tabular form."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
      "outputs": [
    {
     "data": {
      "text/plain": "                                  0         1         2         3         4   \\\nNo Prediction Change Check  0.003274  0.003258  0.008044  0.005117  0.007698   \nNan On Prediction Change         NaN       NaN  0.008044  0.005117  0.007699   \n\n                                  5         6        7         8         9   \\\nNo Prediction Change Check  0.004486  0.003926  0.00226  0.004641  0.009831   \nNan On Prediction Change    0.004486       NaN      NaN       NaN  0.009832   \n\n                            ...        22      23        24        25  \\\nNo Prediction Change Check  ...  0.005673  0.0055  0.004242  0.007323   \nNan On Prediction Change    ...  0.005673  0.0055       NaN  0.007323   \n\n                                  26        27        28        29        30  \\\nNo Prediction Change Check  0.005993  0.007763  0.009405  0.012347  0.004398   \nNan On Prediction Change    0.005996  0.007763  0.009405  0.012347       NaN   \n\n                                  31  \nNo Prediction Change Check  0.011894  \nNan On Prediction Change         NaN  \n\n[2 rows x 32 columns]",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>0</th>\n      <th>1</th>\n      <th>2</th>\n      <th>3</th>\n      <th>4</th>\n      <th>5</th>\n      <th>6</th>\n      <th>7</th>\n      <th>8</th>\n      <th>9</th>\n      <th>...</th>\n      <th>22</th>\n      <th>23</th>\n      <th>24</th>\n      <th>25</th>\n      <th>26</th>\n      <th>27</th>\n      <th>28</th>\n      <th>29</th>\n      <th>30</th>\n      <th>31</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>No Prediction Change Check</th>\n      <td>0.003274</td>\n      <td>0.003258</td>\n      <td>0.008044</td>\n      <td>0.005117</td>\n      <td>0.007698</td>\n      <td>0.004486</td>\n      <td>0.003926</td>\n      <td>0.00226</td>\n      <td>0.004641</td>\n      <td>0.009831</td>\n      <td>...</td>\n      <td>0.005673</td>\n      <td>0.0055</td>\n      <td>0.004242</td>\n      <td>0.007323</td>\n      <td>0.005993</td>\n      <td>0.007763</td>\n      <td>0.009405</td>\n      <td>0.012347</td>\n      <td>0.004398</td>\n      <td>0.011894</td>\n    </tr>\n    <tr>\n      <th>Nan On Prediction Change</th>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>0.008044</td>\n      <td>0.005117</td>\n      <td>0.007699</td>\n      <td>0.004486</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>0.009832</td>\n      <td>...</td>\n      <td>0.005673</td>\n      <td>0.0055</td>\n      <td>NaN</td>\n      <td>0.007323</td>\n      <td>0.005996</td>\n      <td>0.007763</td>\n      <td>0.009405</td>\n      <td>0.012347</td>\n      <td>NaN</td>\n      <td>NaN</td>\n    </tr>\n  </tbody>\n</table>\n<p>2 rows × 32 columns</p>\n</div>"
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(\n",
    "    [\n",
    "        results[\"IntegratedGradients\"][\"DefaultAvgSensitivity\"],\n",
    "        results[\"IntegratedGradients\"][\"AvgSensitivityWithNan\"],\n",
    "    ],\n",
    "    index=[\"No Prediction Change Check\", \"Nan On Prediction Change\"],\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}