{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "bukochgPFg7s" }, "source": [ "# Getting Started with tabular data!\n", "[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/understandable-machine-intelligence-lab/Quantus/main?labpath=tutorials%2FTutorial_Getting_Started_with_Tabular_Data.ipynb)\n", "\n", "\n", "This notebook shows how to get started with Quantus using tabular data. For this purpose, we use the classic Titanic tabular dataset (Frank E. Harrell Jr., Thomas Cason):\n", "\n", "https://www.openml.org/d/40945\n", "\n", "The model in this notebook is taken from \"Getting started with Captum - Titanic Data Analysis\" provided by Captum:\n", "\n", "https://captum.ai/tutorials/Titanic_Basic_Interpret" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "from IPython.display import clear_output" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "id": "4Y7_mNf9Bic0" }, "outputs": [], "source": [ "!pip install quantus torch captum tensorflow-datasets pandas\n", "\n", "clear_output()" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "id": "RV7X-Ss9-16F" }, "outputs": [], "source": [ "import pathlib\n", "import numpy as np\n", "import pandas as pd\n", "from sklearn.model_selection import train_test_split\n", "\n", "import quantus\n", "from captum.attr import IntegratedGradients\n", "\n", "import torch\n", "import torch.nn as nn\n", "\n", "torch.manual_seed(27)\n", "\n", "clear_output()\n", "\n", "np.random.seed(27)" ] }, { "cell_type": "markdown", "metadata": { "id": "mGhP4bTuoWYF" }, "source": [ "## 1) Preliminaries" ] }, { "cell_type": "markdown", "metadata": { "id": "XqKzag4VFjHT" }, "source": [ "### 1.1 Load datasets\n", "\n", "We load the dataset using the tensorflow-datasets library. Alternatively, it can be downloaded directly from the OpenML website: https://www.openml.org/d/40945" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "id": "TmsZxFhuc0mm" }, "outputs": [], "source": [ "# Load datasets\n", "df = pd.read_csv(\"assets/titanic3.csv\")\n", "df = df[[\"age\", \"embarked\", \"fare\", \"parch\", \"pclass\", \"sex\", \"sibsp\", \"survived\"]]\n", "df[\"age\"] = df[\"age\"].fillna(df[\"age\"].mean())\n", "df[\"fare\"] = df[\"fare\"].fillna(df[\"fare\"].mean())" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": " age fare parch pclass sibsp \\\ncount 1309.000000 1309.000000 1309.000000 1309.000000 1309.000000 \nmean 29.881138 33.295479 0.385027 2.294882 0.498854 \nstd 12.883193 51.738879 0.865560 0.837836 1.041658 \nmin 0.170000 0.000000 0.000000 1.000000 0.000000 \n25% 22.000000 7.895800 0.000000 2.000000 0.000000 \n50% 29.881138 14.454200 0.000000 3.000000 0.000000 \n75% 35.000000 31.275000 0.000000 3.000000 1.000000 \nmax 80.000000 512.329200 9.000000 3.000000 8.000000 \n\n survived \ncount 1309.000000 \nmean 0.381971 \nstd 0.486055 \nmin 0.000000 \n25% 0.000000 \n50% 0.000000 \n75% 1.000000 \nmax 1.000000 ", "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
agefareparchpclasssibspsurvived
count1309.0000001309.0000001309.0000001309.0000001309.0000001309.000000
mean29.88113833.2954790.3850272.2948820.4988540.381971
std12.88319351.7388790.8655600.8378361.0416580.486055
min0.1700000.0000000.0000001.0000000.0000000.000000
25%22.0000007.8958000.0000002.0000000.0000000.000000
50%29.88113814.4542000.0000003.0000000.0000000.000000
75%35.00000031.2750000.0000003.0000001.0000001.000000
max80.000000512.3292009.0000003.0000008.0000001.000000
\n
" }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Data statistics\n", "df.describe()" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "# One-hot encode categorical variables\n", "df_enc = pd.get_dummies(df, columns=[\"embarked\", \"pclass\", \"sex\"]).sample(frac=1)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "# Pandas dataframes to numpy arrays\n", "X = df_enc.drop([\"survived\"], axis=1).values\n", "Y = df_enc[\"survived\"].values" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "# Create train and test set\n", "train_features, test_features, train_labels, test_labels = train_test_split(\n", " X, Y, test_size=0.3\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "vmccxpA0n6MY" }, "source": [ "### 1.2 Train a model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The model is based on \"Getting started with Captum - Titanic Data Analysis\" provided by Captum:\n", "\n", "https://captum.ai/tutorials/Titanic_Basic_Interpret" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "class TitanicSimpleNNModel(nn.Module):\n", " def __init__(self):\n", " super().__init__()\n", " self.linear1 = nn.Linear(12, 12)\n", " self.sigmoid1 = nn.Sigmoid()\n", " self.linear2 = nn.Linear(12, 8)\n", " self.sigmoid2 = nn.Sigmoid()\n", " self.linear3 = nn.Linear(8, 2)\n", " self.softmax = nn.Softmax(dim=1)\n", "\n", " def forward(self, x):\n", " lin1_out = self.linear1(x)\n", " sigmoid_out1 = self.sigmoid1(lin1_out)\n", " sigmoid_out2 = self.sigmoid2(self.linear2(sigmoid_out1))\n", " return self.softmax(self.linear3(sigmoid_out2))" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1/200 => Loss: 0.72\n", "Epoch 21/200 => Loss: 0.57\n", "Epoch 41/200 => Loss: 0.50\n", "Epoch 61/200 => Loss: 0.48\n", "Epoch 81/200 => Loss: 0.48\n", "Epoch 101/200 => Loss: 0.47\n", "Epoch 121/200 => Loss: 0.47\n", "Epoch 141/200 => Loss: 0.49\n", "Epoch 161/200 => Loss: 0.47\n", "Epoch 181/200 => Loss: 0.47\n" ] } ], "source": [ "net = TitanicSimpleNNModel()\n", "\n", "criterion = nn.CrossEntropyLoss()\n", "num_epochs = 200\n", "\n", "optimizer = torch.optim.Adam(net.parameters(), lr=0.1)\n", "input_tensor = torch.from_numpy(train_features).type(torch.FloatTensor)\n", "label_tensor = torch.from_numpy(train_labels)\n", "for epoch in range(num_epochs):\n", " output = net(input_tensor)\n", " loss = criterion(output, label_tensor)\n", " optimizer.zero_grad()\n", " loss.backward()\n", " optimizer.step()\n", " if epoch % 20 == 0:\n", " print(\"Epoch {}/{} => Loss: {:.2f}\".format(epoch + 1, num_epochs, loss.item()))" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Train Accuracy: 0.8384279475982532\n" ] } ], "source": [ "out_probs = net(input_tensor).detach().numpy()\n", "out_classes = np.argmax(out_probs, axis=1)\n", "print(\"Train Accuracy:\", sum(out_classes == train_labels) / len(train_labels))" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Test Accuracy: 0.7684478371501272\n" ] } ], "source": [ "test_input_tensor = torch.from_numpy(test_features).type(torch.FloatTensor)\n", "out_probs = net(test_input_tensor).detach().numpy()\n", "out_classes = np.argmax(out_probs, axis=1)\n", "print(\"Test Accuracy:\", sum(out_classes == test_labels) / len(test_labels))" ] }, { "cell_type": "markdown", "metadata": { "id": "4vY9mZQanaxr" }, "source": [ "### 1.3 Generate explanations\n", "\n", "In this example, we rely on the `captum` library. We use the Integrated Gradients method." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "ig = IntegratedGradients(net)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "test_input_tensor.requires_grad_()\n", "attr, delta = ig.attribute(test_input_tensor, target=1, return_convergence_delta=True)\n", "attr = attr.detach().numpy()" ] }, { "cell_type": "markdown", "metadata": { "id": "tuBkEBv3mihR" }, "source": [ "## 2) Quantative evaluation using Quantus\n", "\n", "We can evaluate our explanations on a variety of quantuative criteria but as a motivating example we test the ModelParameterRandomisation scores by Adebayo et al., 2018 and Complexity Bhatt et al., 2020.\n", "\n", "The ModelParameterRandomisation metric measures the distance between the original attribution and a newly computed attribution throughout the process of cascadingly/independently randomizing the model parameters of one layer at a time.\n", "\n", "The Complexity of attributions is defined as the entropy of the fractional contribution of feature x_i to the total\n", "magnitude of the attribution." ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "id": "iq7qqDfSmIdj" }, "outputs": [ { "data": { "text/plain": " 0%| | 0/3 [00:00