{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Explainable Outlier Detection in Titanic dataset\n", "\n", "This short notebook illustrates basic usage of the [OutlierTree](https://github.com/david-cortes/outliertree) library for explainable outlier detection using the Titanic dataset. For more details, you can check the package's documentation [here](http://outliertree.readthedocs.io/en/latest/).\n", "\n", "The dataset is very popular and can be downloaded from different sources, such as Kaggle or many university webpages. This notebook took it from the following link: http://biostat.mc.vanderbilt.edu/wiki/pub/Main/DataSets/titanic3.csv\n", "** *\n", "\n", "### Loading the raw data" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
pclasssurvivednamesexagesibspparchticketfarecabinembarkedboatbodyhome.dest
011Allen, Miss. Elisabeth Waltonfemale29.000024160211.3375B5S2NaNSt Louis, MO
111Allison, Master. Hudson Trevormale0.9212113781151.5500C22 C26S11NaNMontreal, PQ / Chesterville, ON
210Allison, Miss. Helen Lorainefemale2.0012113781151.5500C22 C26SNaNNaNMontreal, PQ / Chesterville, ON
310Allison, Mr. Hudson Joshua Creightonmale30.0012113781151.5500C22 C26SNaN135.0Montreal, PQ / Chesterville, ON
410Allison, Mrs. Hudson J C (Bessie Waldo Daniels)female25.0012113781151.5500C22 C26SNaNNaNMontreal, PQ / Chesterville, ON
\n", "
" ], "text/plain": [ " pclass survived name sex \\\n", "0 1 1 Allen, Miss. Elisabeth Walton female \n", "1 1 1 Allison, Master. Hudson Trevor male \n", "2 1 0 Allison, Miss. Helen Loraine female \n", "3 1 0 Allison, Mr. Hudson Joshua Creighton male \n", "4 1 0 Allison, Mrs. Hudson J C (Bessie Waldo Daniels) female \n", "\n", " age sibsp parch ticket fare cabin embarked boat body \\\n", "0 29.00 0 0 24160 211.3375 B5 S 2 NaN \n", "1 0.92 1 2 113781 151.5500 C22 C26 S 11 NaN \n", "2 2.00 1 2 113781 151.5500 C22 C26 S NaN NaN \n", "3 30.00 1 2 113781 151.5500 C22 C26 S NaN 135.0 \n", "4 25.00 1 2 113781 151.5500 C22 C26 S NaN NaN \n", "\n", " home.dest \n", "0 St Louis, MO \n", "1 Montreal, PQ / Chesterville, ON \n", "2 Montreal, PQ / Chesterville, ON \n", "3 Montreal, PQ / Chesterville, ON \n", "4 Montreal, PQ / Chesterville, ON " ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import numpy as np, pandas as pd\n", "from outliertree import OutlierTree\n", "\n", "## Read the raw data, downloaded from here:\n", "## https://github.com/jbryer/CompStats/raw/master/Data/titanic3.csv\n", "titanic = pd.read_csv(\"titanic3.csv\")\n", "titanic.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Pre-processing the data" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PclassSurvivedSexAgeSibSpParchFareCabinEmbarkedBoatBody
01YesFemale29.0000211.3375B5S2NaN
11YesMale0.9212151.5500C22 C26S11NaN
21NoFemale2.0012151.5500C22 C26SNaNNaN
31NoMale30.0012151.5500C22 C26SNaN135.0
41NoFemale25.0012151.5500C22 C26SNaNNaN
\n", "
" ], "text/plain": [ " Pclass Survived Sex Age SibSp Parch Fare Cabin Embarked Boat \\\n", "0 1 Yes Female 29.00 0 0 211.3375 B5 S 2 \n", "1 1 Yes Male 0.92 1 2 151.5500 C22 C26 S 11 \n", "2 1 No Female 2.00 1 2 151.5500 C22 C26 S NaN \n", "3 1 No Male 30.00 1 2 151.5500 C22 C26 S NaN \n", "4 1 No Female 25.00 1 2 151.5500 C22 C26 S NaN \n", "\n", " Body \n", "0 NaN \n", "1 NaN \n", "2 NaN \n", "3 135.0 \n", "4 NaN " ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "## Capitalize column names and some values for easier reading\n", "titanic.columns = titanic.columns.str.capitalize()\n", "titanic = titanic.rename(columns = {\"Sibsp\" : \"SibSp\"})\n", "titanic[\"Sex\"] = titanic[\"Sex\"].str.capitalize()\n", "\n", "## Convert 'survived' to yes/no for easier reading\n", "titanic[\"Survived\"] = titanic[\"Survived\"].astype(\"category\").replace({1:\"Yes\", 0:\"No\"})\n", "\n", "## Some columns are not useful, such as name (an ID), ticket number (another ID),\n", "## or destination (too many values, many non-repeated)\n", "cols_drop = [\"Name\", \"Ticket\", \"Home.dest\"]\n", "titanic = titanic.drop(cols_drop, axis=1)\n", "\n", "## Ordinal columns need to be passed as ordered categoricals\n", "cols_ord = [\"Pclass\", \"Parch\", \"SibSp\"]\n", "for col in cols_ord:\n", " titanic[col] = pd.Categorical(titanic[col], ordered=True)\n", "\n", "titanic.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Fitting a model" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Reporting top 9 outliers [out of 9 found]\n", "\n", "\n", "row [170] - suspicious column: [Fare] - suspicious value: [0.00]\n", "\tdistribution: 98.571% >= 25.74 - [mean: 55.22] - [sd: 27.56] - [norm. obs: 69]\n", "\tgiven:\n", "\t\t[Pclass] = [1]\n", "\t\t[Boat] in [9, B, 5, 7, C, 5 9, 1, 15, 5 7, 8 10, 12, 16, 13 15 B, C D, 15 16, 13 15] (value: C)\n", "\n", "\n", "row [18] - suspicious column: [Age] - suspicious value: [32.00]\n", "\tdistribution: 96.000% >= 43.00 - [mean: 48.35] - [sd: 3.16] - [norm. obs: 24]\n", "\tgiven:\n", "\t\t[Cabin] in [E12, D15, B10, E31, E58, C86, A16, A20, E63, C92, B82 B84, D33, B52 B54 B56, C124, D17, C110, C116, C126, D46] (value: D15)\n", "\n", "\n", "row [896] - suspicious column: [Fare] - suspicious value: [0.00]\n", "\tdistribution: 99.216% >= 3.17 - [mean: 9.68] - [sd: 6.98] - [norm. obs: 506]\n", "\tgiven:\n", "\t\t[Pclass] = [3]\n", "\t\t[SibSp] = [0]\n", "\n", "\n", "row [898] - suspicious column: [Fare] - suspicious value: [0.00]\n", "\tdistribution: 99.216% >= 3.17 - [mean: 9.68] - [sd: 6.98] - [norm. obs: 506]\n", "\tgiven:\n", "\t\t[Pclass] = [3]\n", "\t\t[SibSp] = [0]\n", "\n", "\n", "row [963] - suspicious column: [Fare] - suspicious value: [0.00]\n", "\tdistribution: 99.216% >= 3.17 - [mean: 9.68] - [sd: 6.98] - [norm. obs: 506]\n", "\tgiven:\n", "\t\t[Pclass] = [3]\n", "\t\t[SibSp] = [0]\n", "\n", "\n", "row [1254] - suspicious column: [Fare] - suspicious value: [0.00]\n", "\tdistribution: 99.216% >= 3.17 - [mean: 9.68] - [sd: 6.98] - [norm. obs: 506]\n", "\tgiven:\n", "\t\t[Pclass] = [3]\n", "\t\t[SibSp] = [0]\n", "\n", "\n", "row [1044] - suspicious column: [Fare] - suspicious value: [15.50]\n", "\tdistribution: 96.774% <= 8.52 - [mean: 7.73] - [sd: 0.28] - [norm. obs: 30]\n", "\tgiven:\n", "\t\t[Pclass] = [3]\n", "\t\t[SibSp] = [0]\n", "\t\t[Boat] in [3, 10, 4, 9, 6, B, 8, A, 5, 7, 5 9, 1, 5 7, 8 10, 16, 13 15 B, 15 16, 13 15] (value: 16)\n", "\n", "\n", "row [1146] - suspicious column: [Fare] - suspicious value: [29.12]\n", "\tdistribution: 97.849% <= 15.50 - [mean: 7.89] - [sd: 1.17] - [norm. obs: 91]\n", "\tgiven:\n", "\t\t[Pclass] = [3]\n", "\t\t[SibSp] = [0]\n", "\t\t[Embarked] = [Q]\n", "\n", "\n", "row [1163] - suspicious column: [Fare] - suspicious value: [24.15]\n", "\tdistribution: 97.849% <= 15.50 - [mean: 7.89] - [sd: 1.17] - [norm. obs: 91]\n", "\tgiven:\n", "\t\t[Pclass] = [3]\n", "\t\t[SibSp] = [0]\n", "\t\t[Embarked] = [Q]\n", "\n", "\n" ] }, { "data": { "text/plain": [ "OutlierTree model\n", "\tNumeric variables: 3\n", "\tCategorical variables: 5\n", "\tOrdinal variables: 3\n", "\n", "Consists of 221 clusters, spread across 18 tree branches" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "## Fit model with default hyperparameters\n", "otree = OutlierTree()\n", "otree.fit(titanic)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Examining the results more closely" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PclassSurvivedSexAgeSibSpParchFareCabinEmbarkedBoatBody
11463NoFemale39.00529.125NaNQNaN327.0
11633NoMaleNaN0024.150NaNQNaNNaN
\n", "
" ], "text/plain": [ " Pclass Survived Sex Age SibSp Parch Fare Cabin Embarked Boat \\\n", "1146 3 No Female 39.0 0 5 29.125 NaN Q NaN \n", "1163 3 No Male NaN 0 0 24.150 NaN Q NaN \n", "\n", " Body \n", "1146 327.0 \n", "1163 NaN " ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "## Double-check the data (last 2 outliers)\n", "titanic.loc[[1146, 1163]]" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEgCAYAAACwxdQWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAnXklEQVR4nO3de5xcdX3/8debsFzMCgGBJQlgouIFaESzoFTFXW6l2Ar6U9RSGywaq2hV1Er9/X4KrW21WFBb2xouJfYHLogo0aIIkUWt5ZKgRi4i16C7kCDkwi4Cm/D5/fH9Lpkzmd2ds9mZ2c2+n4/HPM7MuX7mO2fO55zv+Z5zFBGYmZkN26HVAZiZ2eTixGBmZgVODGZmVuDEYGZmBU4MZmZW4MRgZmYF0z4xSOqV1LI2u5IulhSS5lX0m5f7XdyquHIcLS2biSLpQEnflPRwLtf1rY5pshvvOjiedSYvp7fMNI0i6dQcz6mtjqWVtovEkH/IytdTkh6RdKukCyT9oaQZDVr2A5IeaMS8G61WUtre5N/9W8AJwHeAs4HP1jFd9TpV/Tq1oYFPUtNhnWm2ybgDtmOrA5hgZ+fuDGAWcDDwTuA0YIWkUyLiV1XT/BnwnKZFuLW/Jm2o+loYw0haXTYTYT5wEHB+RCwex/Rnj9D/Z+OOaGroA14GbGjCsl4GPNGE5VidtqvEEBFnVfeT1AH8M/BW4DpJnRGxtmKaB5sX4dYi4iHgoVbGMJJWl80EmZO7/eOZuNY6NR1ExBDwyyYtqynLsRIiYsq/gEhfZcThOwDX5/G+UDWst3paQMAi4CfAI8CTwK+Ba4C35XG6hpdb43VxVWy9wL7ABaQ9sc3AqXn4xXmceRXTzBueD/BSUlXIY8Ag8GPguBrf8aw8TVeNYc/Or7rMarweGK1sKsrzL4BbgIEc1y3A+4AdRvh9eoG9gCWkRPgUcDvwrnH83guBbwBr83xWA/8KzK61XtR4nbWt61QeZw7wKeC/gYeBp0kJ6FLgoNF+B+DFwGX5OzxT+bsBfwBcDfw2f797gXOAWXWWz3vzct5T1f9duf8TwM5Vw24iree7TsQ6Q9rp/CRwd/4OvwY+B+w00vox0voMvAW4Ocf9GNADzB3HevM2YHmex5PAA8DXgM6KcU7Nyz11rBgrhl1M1X84939jXt7w+t4P3AC8v6qMa72qy2M/4F+A+/K8HgWWAYeNti0A/iT/tgOVv9NYr+3qiGEkEfGMpM+QCuodkj4SuQRH8HekKp77gctJh9OzgcNIRx6XkVaqs4EP52m+UDH9z6rmtydwI+nHuZK0IVhTR+jzgf8BfgF8JcfwNuC7kv4kIi6rYx4jORs4CXg58EVgfe6/vvboBf9JWuF+TUp2AbyJtHF+LXBKjWlmkTagTwNXADuTyvIiSc9ExNJ6gpb0R6SkoDyf1aRE8T7gREmvjYj7K77jPFKSv4G00aKiu62OBM4k7XR8g/T7HkjakL1R0msi4uc1pnsh6c/6K+ASYFdgY/5+nyb9sR8jnRNZCywAPgacIOmIiNg4RlzLc/do4PyK/kfn7q7AEeRykLQ7qQx/FBG/G2W+ZdaZS4HXAd/N3+0E4K+AfUgJql7vJ21gl5F+w1eR/gMvl3RoRDw11gwkCfgP0nrwW9J/8BHSxrYbuAtYUSKmMUlaTPrPPgx8Oy93H9Jv+S7Sf2U9qUxPBZ5PsdrygYp5vRL4Pmk7ck2Ofy/Sb/FjSW+KiKtrhPFR4Ni8/OuB3ev+AmWz7mR8Ud/e3c7AUB53fvUeTtW4jwK/AZ5TYz57VX1+gFEyMVv2AL4K7FjP3gbFPYlzqsbvzN9jHbBbrb2EGssYnt/F9ezpjFE278jT3Aq0V/SfSfpzBfAnI5TBBcCMiv4HAZuAO+r8ndvzb7MZeF3VsE/kZXy/qn8XdR4ljBDzWTVep+Zx9gGeW2Pal5OSxHdH+B0C+Psa03XnYT+h6uiALXuy59UZ/2pSUlFFv35S0tgM/G1F/xPzvP/vRK0zwEpgz6r145687H1rlHVvVb/h9Xkj8HtVwy7Nw06usywW5/FvBnavGjaDiiNNJuiIIX//p4B9aoxfvQ3pZYTtF+nI6x7SEc7rq4bNIdVAPETFEWBF2Q0Cryiz3g+/totWSfWItGfxaP64dx2TDJFW4ur5/HYci38a+FhEbCo53Qbgb6qWv4K0lzmLtJfebH+eu2dGxEBFXIOkjTPAu2tM9wRwRkRsrpjmDtJRxMsktdex7BNJe02XRcSPqob9EylJHyvpgHq+SJ0+XeN1KkBErI2Ix6sniHSU8AOgW1JbjXmuofZJ7b/M3fdExPqqeV5MOhKtdTRWyw9I6/nvAUg6iHTEeQUpqR9dMe7w++VMnE9ExGPDH/L6cQmpGrKzxHy+FBG/qOo3fBR0eJ3z+GDuvjciCifTI2JzpPN8jbCJtB0pKLkNeQPpCPOfI+KGqvn0A/9IqqY+usa0SyLipyWW9axpUZVUQbk7WjUSpBX4g8Adki4nHcL+T/VKVcIDUXHCu4Rba214SHsYi4BXAHVVwUygV5KqwnprDLuBlExfUWPY3VG7CuTXubsHaS97rGVD2ugVRMQmST8k7em+ApiQE+cRodGGS3oD6XxLJ+nwvvo/tRdbNy74edSuAjmCtCF5q6S31hi+E7C3pOdFxKM1hlf6ASmBHQ2sAo7K/ZeTyugMSc/N69dRpLK/eYx5llGraqbyt27KfCTNBA4B1ox3IzlOl5B2Vu6Q1EP6b/x3RDxScj5H5O7zJZ1VY/iBufsy0nmpSuP+PadNYpC0C2lvE1L94mg+QjrJ8y5SHfKZwCZJVwMfjYh7Si7+4ZLjDxvpPMTw/OqvM5w4uwOPRcTT1QPyxnm4LrXa+hHmN3wUVc91JsPfd6Q9vOH+s+qY1zaT9CHSuaV1wLWkZPQEacfjJFKV0s41Jh1pfXge6T/56TEWPVylNprK8wzn5e5vIuJXkpaT6vtfL2kFqVn31eM4oh1R9RFPVua3Hrat85mVu01tDh4R5+b/wvtJR4IfBkLSDcDH85F/PZ6Xu7V2FCrVOuIe73Zn+iQG0knRHUl7Dg+MNmKu7vgC8AVJ++Rp3076cQ6WdPAIe3wjznJcEUPHCP33zd3KI5hncrfWbzprnMuvZQOwp6S2SE0anyVpR9Ie8lgnR7dl2bDl+1ebXTVew+Tvehbpz/fK6uoISUfUmi4baX3YQGrVtecIw+sWEf2S7gKOlLQz6VzLVXnwj0nVm8cAu+V+Wx2FbSfW5+7cbZxPMPL2clbNCSK+CnxV0izg90lVv38OXCPppXUePQyvyydGxLJSEY9/uzM9zjFI2gH43/njpWWmzfXIV0bEyaQ/zwtJh6bDNlNuD6iMV0p6bo3+XblbeWi8Lnf3rzH+SHW6w/X9ZeL/KWm9ObLGsCPzvG4tMb8yhr9vV/WAvKF+Xf7YqOVX2ou0QfhJjaTQzpZqrzJuBPaQdPC2hweko4bnklpszcqfiYgn8rKOpljFVI/xrDMtk89t3AZ0SKpVxVmvddT4b+Ur6w8dI4b1EXF1RLyHdKJ6T4r/n80V86p2Y+6+rsawhtnuE0Pe4+8hbUweBP5+jPF3lvSaGv3b2FIVVXmV5qOket9dJyTgot1J7eQr4+gknYDcAHyzYtBwfeK78kZyePz9q+dRYbg6oszJ2oty9x8kPXtVdH4/fKuJC0vMr4xvkZpxvkPSq6uGfZjUvPe6aM6FeWtJ68HCyhPneT35IilxlHVe7p4vaU71QEkza3zv0QwfBfx17i6vGnYIqSnoo0CtZrW1jGedabUv5e5XctPcZ0naQdLsGtNUuxk4QNJxVf3/D6mpaYGk7txMttpwNWv1NgRql+lVpOtYTpd0Qq3AJB1R+V+cCNtVVVLFyZkd2HJLjNeSTtrdDJxSR4uAXUltg+8hNTlbDexCag/8MmBZRNxZMf5y0vUN38snP58inVz89gR8pR8C75b0KlLrneHrGHYgtbB4tsomIm7Kyz8SuFnSD0hVUX9Mavtc60hiOfBx0oboG8DjwPqI+JeRAoqISyWdCJwM3C7pW2ypU59PajF0yTZ965GXPSDpz4GvAzdI+jop2S8EjiNV67y3EcuuEcszkr5EOv/0C0lXkdazbtIOxPX5fZl5Lpd0JvAPwN35nNb9pPrj5wOvJ1UDHV/nLK8nVTHuA/wyt2IZtpxUFbY3cEXkdo51KL3OTAIXkPa430kq16tI5xnnkI6YLiKVxWg+T7rw8CpJl5F2UH6ftM73svVR7DeBAUk3klrLKcdwGGm7cl3FuMtJ1dRX5t/8d8DqiPjPiBiS9GbSf/i/JP2E1DrtCdJ/+jDgBaRtw8TdVmQ8bVwn24utrxp8inRByUpS07bjqXFFbtRoQwy0kU7MfZe00XmStBLdSGp9slPV9DOBfyNd97CJ2leL1mz/HCO3gZ43PB9SMrqKdCj7BClB/MEI85qVv+/wFcG3kdpwz6uOq2KaM4A78/hB/Vc+v5/UYuSJ/FoJnF6rnEcrg1rfv47f+zDSH+8RUl35g/k3mFNj3C624TqGMcbZMZffHaQ/88Oki/+eP9bvOsZ8X0u6sLI/f79HSBuDc6m4SrfO77EyL/PLVf3bSC2RAnhfjekmdJ3Jw06lzmsEGMd1OXWUxSmk1kEbSP/r+0mth145Vox52BvzOv8kaS+/Z5Tf+i/yOnofW67Y/ilp2/LcqvnOINVk3MeWa62qy2Mf0hH5bXl+A6Sryq8A/pSKa6RGK7t6X8ozMjMzA6bBOQYzMyvHicHMzAqcGMzMrMCJwczMCqZsc9W99tor5s2b1+owmmJwcJCZM2e2OoxJw+VR5PIocnkUVZfHypUrfxsRo95IdMomhnnz5rFixYTeQn3S6u3tpaurq9VhTBoujyKXR5HLo6i6PCStHmuaplclSfqIpNsl3Sbpa5J2kTRf0k2S7pF0maSdmh2XmZklTU0MkuaS7jTYGRGHkC7seDvpkX/nRcSLSBdyndbMuMzMbItWnHzeEdg138/nOaRbJR9FuoIP0vMFTmpBXGZmBs2/8jnfw/7vSLcQ+D7wIeDGfLQwfNO37+YjiuppF5Nu8UBHR8fCnp6epsXdSgMDA7S31/OAs+nB5VHk8ihyeRRVl0d3d/fKiBj1KXpNPfksaQ/S4xnnk+6T/nXqvyEYEbEEWALQ2dkZ0+UEk0+mFbk8ilweRS6PovGUR7Orko4B7o+IRyI95OVK4DXArIpbRe9Hk5+2ZGZmWzQ7MTwIvFrSc/K9yo8m3ZnyeuAteZxFbHnSlJmZNVlTE0NE3EQ6yXwr8Iu8/CXAJ0gPJ7+H9IzTRj3oxczMxtD0C9wi4tNs/bDz+4DDmx1LvTZs2MDg4OBW/WfOnMnuu+9eYwozs6lryl753CwbNmzggAPms3Hjuq2G7bbbHjz44P1ODma2XXFiGMPg4GBOCotJT1gcNsDGjUsYHBx0YjCz7YoTQ93agd1aHYSZWcP5tttmZlbgxGBmZgVODGZmVuDEYGZmBU4MZmZW4MRgZmYFTgxmZlbgxGBmZgVODGZmVuDEYGZmBU4MZmZW4MRgZmYFTgxmZlbgxGBmZgVODGZmVtDUxCDpJZJ+VvHaKOnDkvaUdK2ku3N3j2bGZWZmWzQ1MUTEXRFxaEQcCiwEngC+CZwJLI+IA4Hl+bOZmbVAK6uSjgbujYjVwInA0tx/KXBSq4IyM5vuFBGtWbB0EXBrRPyLpPURMSv3F7Bu+HPVNItJD1+mo6NjYU9PT8PjHBoaYtWqVUAHMKNiyGZgDQsWLKCtra2hMQwMDNDe3j72iNOEy6PI5VHk8iiqLo/u7u6VEdE52jQtSQySdgL6gYMjYk1lYsjD10XEqOcZOjs7Y8WKFQ2OFPr7+5k7dy5wBsVnPm8EzqWvr485c+Y0NIbe3l66uroauoypxOVR5PIocnkUVZeHpDETQ6uqkv6QdLSwJn9eI2k2QO6ubVFcZmbTXqsSwzuAr1V8XgYsyu8XAVc1PSIzMwNakBgkzQSOBa6s6P1Z4FhJdwPH5M9mZtYCOzZ7gRExCDyvqt+jpFZKZmbWYr7y2czMCpwYzMyswInBzMwKnBjMzKzAicHMzAqcGMzMrMCJwczMCpwYzMyswInBzMwKnBjMzKzAicHMzAqcGMzMrMCJwczMCpwYzMyswInBzMwKnBjMzKzAicHMzApa8WjPWZKukPRLSXdKOkLSnpKulXR37u7R7LjMzCxpxRHDF4HvRcRLgZcDdwJnAssj4kBgef5sZmYt0NTEIGl34EjgQoCIeDoi1gMnAkvzaEuBk5oZl5mZbdHsI4b5wCPAf0j6qaQLJM0EOiLioTzOw0BHk+MyM7NMEdG8hUmdwI3AayLiJklfBDYCH4yIWRXjrYuIrc4zSFoMLAbo6OhY2NPT0/CYh4aGWLVqFSlXzagYshlYw4IFC2hra2toDAMDA7S3tzd0GVOJy6PI5VHk8iiqLo/u7u6VEdE52jTNTgz7AjdGxLz8+XWk8wkvAroi4iFJs4HeiHjJaPPq7OyMFStWNDpk+vv7mTt3LnAGsFvFkI3AufT19TFnzpyGxtDb20tXV1dDlzGVuDyKXB5FLo+i6vKQNGZiaGpVUkQ8DPxa0vBG/2jgDmAZsCj3WwRc1cy4zMxsix1bsMwPApdI2gm4D3gXKUFdLuk0YDVwcgviMjMzWpAYIuJnQK3DmKObHIqZmdXgK5/NzKzAicHMzAqcGMzMrMCJwczMCpwYzMyswInBzMwKnBjMzKzAicHMzAqcGMzMrMCJwczMCkolBkm/16hAzMxscih7xPBzSbdIep+kWY0IyMzMWqtsYjiKdJvsfwT6JX1N0rGSNPGhmZlZK5RKDBHRGxGLgH2BDwBzgWuA1ZL+VtILGxCjmZk10bhOPkfEYERcFBFHAi8BHgA+CfxK0g2S3jSBMZqZWRONu1WSpHmSziIdMRwBXE16HvMa4DJJ501IhGZm1lRlWyU9R9KfSboeuAc4BTgfOCAi/jgiLoyIk4H3AqdNfLhmZtZoZZ/gtoaUTK4EjomI3hHGuwV4dBviMjOzFimbGP4KuDQiNow2UkTcBsyvNUzSA8DjwGZgU0R0StoTuAyYRzpfcXJErCsZm5mZTYCyrZL+baykUKfuiDg0Ioaf/XwmsDwiDgSW589mZtYCZc8xXCSpZ4RhX5N0/jjjOBFYmt8vBU4a53zMzGwbKSLqH1n6NXBGRHy9xrC3AOdGxAFjzON+YB0QwFciYomk9RExKw8XsG74c9W0i0ktn+jo6FjY01MzR02ooaEhVq1aBXQAMyqGbAbWsGDBAtra2hoaw8DAAO3t7Q1dxlTi8ihyeRS5PIqqy6O7u3tlRW1NTWXPMewNPDbCsHXAPnXM47UR0SdpH+BaSb+sHBgRIalmtoqIJcASgM7Ozujq6qo78PHq7+/nuOOOA84AdqsYshE4l76+PubMmdPQGHp7e2nGd50qXB5FLo8il0fReMqj7HUMq4EjRxh2JPCbsWYQEX25uxb4JnA4sEbSbIDcXVsyLjMzmyBlE8PFwCcknS6pHUBSu6T3k1osXTDaxJJmSnru8HvgOOA2YBmwKI+2CLiqZFxmZjZBylYlfQ54IfDPwJckDQIzAZGqeD43xvQdwDfzPfd2JDV9/Z6kW4DLJZ1GOio5uWRcZmY2QUolhoh4Bni3pHNId1rdk3Qh2w8i4ld1TH8f8PIa/R8Fji4Ti5mZNUbZIwYAIuIu4K4JjsXMzCaBcSUGSS8G9gN2qR4WEVdva1BmZtY6pRKDpIOAHuBg0nmFakGxsb+ZmU0xZY8YvgLsDLyZ9CS3pyc8IjMza6myieEVwNsj4juNCMbMzFqv7HUM91LjvIKZmW0/yiaGjwKflPSCRgRjZmatV7Yq6R+AucAv83MV1lePEBGHb3tYZmbWKmUTw235ZWZm26myVz6/q1GBmJnZ5FD2HAOQnpkgaX9Jv59vhmdmZtuJ0okh30m1j3Szux8BL8n9r5T04QmNzszMmq7soz0/DpwLnE+6iV7l1c+9wNsmLDIzM2uJsiefTwc+FRH/KKn61hd3AS+emLDMzKxVylYl7QusHGHYM/jiNzOzKa9sYrgHeP0Iw44k3T/JzMymsLJVSV8A/lXS08AVud8++clrZwDvmcDYzMysBcpex3CBpD2ATwFn595XA08AZ0XEpfXMJ5+fWAH0RcQfSZpPup3380hVVe+MCN+51cysBUo3V42Ic4A5wAnAn+bu3Ny/Xh8C7qz4/DngvIh4EbAOOK1sXGZmNjHGdYFbRDweEddExKUR8b2I2FDvtJL2A94AXJA/i9T0dbhqailw0njiMjOzbVf2CW7vH2uciPjXMUb5AvBXwHPz5+cB6yNiU/78G9KN+szMrAUUEfWPLD0zyuAAiIgRH+0p6Y+AEyLi/ZK6gI8BpwI35mokJO0PfDciDqkx/WJgMUBHR8fCnp6eumMfr6GhIVatWgV0UHxq6WZgDQsWLKCtra2hMQwMDNDe3t7QZUwlLo8il0eRy6Ooujy6u7tXRkTnaNOUPfm8VdWTpFnAHwCfAN4xxixeA7xR0gmkax52A74IzJK0Yz5q2I90y41ay18CLAHo7OyMrq6uMuGPS39/P8cddxyp0dVuFUM2AufS19fHnDlzGhpDb28vzfiuU4XLo8jlUeTyKBpPeYzrHEOliFgfEZcB/056JvRo4/51ROwXEfOAtwM/iIhTgOuBt+TRFgFXbWtcZmY2PtucGCrcD4x6eDKKTwBnSLqHdM7hwgmLyszMSil7gVtNkmaTHvt5f73TREQv6cZ7RMR9gJ/8ZmY2CZRtlfQI+SRzhZ1ILYyeBN48QXGZmVmLlD1i+DJbJ4YnSU1MvxcRj05IVGZm1jJlWyWd1aA4zMxskpjIk89mZrYdKHuO4X62rkoaUUS8oHREZmbWUmXPMVxBuv7gOcC1wFpgH+BYYBC4bEKjMzOzpiubGNYB9wJviIjB4Z6S2oHvABsi4jMTGJ+ZmTVZ2XMMpwPnVCYFgIgYAD6fh5uZ2RRWNjHsRrqbXC37Ar5zlZnZFFe2KunbwDmSNgLLIuJpSTsBJ5IetvPtiQ7QzMyaq2xieB9wMXA5EJIeJ131LGBZHm5mZlNY2QvcNgBvknQwcBipWulh4JaIuKMB8ZmZWZON6yZ6EXE7cPsEx2JmZpNA6SufJe0j6XOSlku6Kx89IOlDko6Y+BDNzKyZSiUGSYcDdwP/C3gAeBGwcx48fOttMzObwsoeMZxHetrai4H3kk46D7sZP1PBzGzKK3uO4ZXAiRHxjCRVDXuUdHsMMzObwsoeMWwA9h5h2AuANdsWjpmZtVrZxLAMOFtS5V1TQ9JewMeAK0ebWNIukm6W9HNJt0s6O/efL+kmSfdIuixfNGdmZi1QNjF8AtgI3AH8MPf7d+Au4HfAp8aY/ingqIh4OXAocLykV5Oumj4vIl5EulHfaSXjMjOzCVIqMUTEOuDVpJvlrQauA+4HzgReExGPjzF95BvuAbTlVwBHkW7pDbAUOKlMXGZmNnEUUd9zdyTtQqpK+vuI6B33AqUZwEpSU9cvA+cAN+ajBSTtD3w3Ig6pMe1iYDFAR0fHwp6envGGUbehoSFWrVpFush7RsWQzcAaFixYQFtbW0NjGBgYoL3d9ycc5vIocnkUuTyKqsuju7t7ZUR0jjZN3a2SIuJJSYdR3DqWFhGbgUMlzQK+Cby0xLRLgCUAnZ2d0dXVtS2h1KW/v5/jjjsOOIN0c9lhG4Fz6evrY86cOQ2Nobe3l2Z816nC5VHk8ihyeRSNpzzGc/L5pJLT1BQR60nXRBwBzJI0nKT2A/omYhlmZlZe2esYriHddns2cDWpeWqhLioirh5pYkl7A0MRsV7SrqRHgn6OlCDeAvQAi4CrSsZlZmYTpGxi+H+5++b8qhaMXtU0G1iazzPsAFweEd+RdAfQI+kzwE+BC0vGZWZmE2TMxCDp+8AHI+IuYD7pNhhHAzcBo7ZCqhYRq4BX1Oh/H76dhpnZpFDPEcMxwO4AEbE67+0vAQ6LiNWNDM7MzJqv9G23s+r7JJmZ2XZivInBzMy2U/UmhlpXwdV3ZZyZmU0p9bZKukbSpqp+y2v0IyJ8620zsymsnsRwdsOjMDOzSWPMxBARTgxmZtOITz6bmVmBE4OZmRU4MZiZWYETg5mZFTgxmJlZgRODmZkVODGYmVmBE4OZmRU4MZiZWYETg5mZFTgxmJlZQVMTg6T9JV0v6Q5Jt0v6UO6/p6RrJd2du3s0My4zM9ui2UcMm4CPRsRBwKuB0yUdBJwJLI+IA4Hl+bOZmbVAUxNDRDwUEbfm948DdwJzgROBpXm0pcBJzYzLzMy2UERrHsQmaR7wQ+AQ4MGImJX7C1g3/LlqmsXAYoCOjo6FPT09DY9zaGiIVatWAR3AjIohm4E1LFiwgLa2tobGMDAwQHt7e0OXMZW4PIpcHkUuj6Lq8uju7l4ZEZ2jTdOSxCCpHbgB+LuIuFLS+spEIGldRIx6nqGzszNWrFjR4Eihv7+fuXPnAmcAu1UM2QicS19fH3PmzGloDL29vXR1dTV0GVOJy6PI5VHk8iiqLg9JYyaGprdKktQGfAO4JCKuzL3XSJqdh88G1jY7LjMzS5rdKknAhcCdEXFuxaBlwKL8fhFwVTPjMjOzLep55vNEeg3wTuAXkn6W+30S+CxwuaTTgNXAyU2Oy8zMsqYmhoj4MaARBh/dzFjMzKw2X/lsZmYFTgxmZlbgxGBmZgVODGZmVuDEYGZmBU4MZmZW4MRgZmYFTgxmZlbgxGBmZgVODGZmVuDEYGZmBU4MZmZW4MRgZmYFTgxmZlbgxGBmZgVODGZmVuDEYGZmBc1+5vNFktZKuq2i356SrpV0d+7u0cyYzMysqNlHDBcDx1f1OxNYHhEHAsvzZzMza5GmJoaI+CHwWFXvE4Gl+f1S4KRmxmRmZkWKiOYuUJoHfCciDsmf10fErPxewLrhzzWmXQwsBujo6FjY09PT8HiHhoZYtWoV0AHMqBiyGVjDggULaGtra2gMAwMDtLe3N3QZU4nLo8jlUeTyKKouj+7u7pUR0TnaNJMqMeTP6yJizPMMnZ2dsWLFiobFOay/v5+5c+cCZwC7VQzZCJxLX18fc+bMaWgMvb29dHV1NXQZU4nLo8jlUeTyKKouD0ljJobJ0CppjaTZALm7tsXxmJlNa5MhMSwDFuX3i4CrWhiLmdm01+zmql8D/gd4iaTfSDoN+CxwrKS7gWPyZzMza5Edm7mwiHjHCIOObmYcZmY2sslQlWRmZpOIE4OZmRU4MZiZWYETg5mZFTgxmJlZgRODmZkVODGYmVlBU69jmCw2bNjA4ODgVv1nzpzJ7rvv3oKIzMwmj2mXGDZs2MABB8xn48Z1Ww3bbbc9ePDB+50czGxam3aJYXBwMCeFxUDlrXkH2LhxCffeey/77rvvs30ffvjhZodoZtZS0y4xbNFO8TbaTwE7sHDhwhbFY2Y2OUzjxFBtCHiGrY8k1gCXtCQiM7NWcGLYSvWRxECrAjEzawk3VzUzswIfMWyjWien3ey1udz82GxiOTGM28gnq93stXnc/Nhs4jkxjNtIJ6tTs9fBwUFvkJpgrObH/h3MynNi2GbVJ6utNfw7bA9cLbhFK8ti0iQGSccDXwRmABdEhJ/9bDaNuFpwi1aXxaRIDJJmAF8GjgV+A9wiaVlE3NHayMysWVwtuEWry2JSJAbgcOCeiLgPQFIPcCLQwMRQfX3C4AT1T58n8lYaQ0ND9Pf3T9j8prrK8thSzo3/HSar7WX9GOu3qve33B7Ko9XrrSKipQEASHoLcHxEvDt/fifwqoj4QNV4i0kpFOAlwF1NDbR19gJ+2+ogJhGXR5HLo8jlUVRdHs+PiL1Hm2CyHDHUJSKWAEtaHUezSVoREZ2tjmOycHkUuTyKXB5F4ymPyXLlcx+wf8Xn/XI/MzNrssmSGG4BDpQ0X9JOwNuBZS2OycxsWpoUVUkRsUnSB4BrSM1VL4qI21sc1mQy7arPxuDyKHJ5FLk8ikqXx6Q4+WxmZpPHZKlKMjOzScKJwczMCpwYJhlJF0laK+m2in57SrpW0t25u0crY2ymEcrjLEl9kn6WXye0MsZmkrS/pOsl3SHpdkkfyv2n5ToySnlMy3VE0i6Sbpb081weZ+f+8yXdJOkeSZflRj4jcmKYfC4Gjq/qdyawPCIOBJbnz9PFxWxdHgDnRcSh+XV1k2NqpU3ARyPiIODVwOmSDmL6riMjlQdMz3XkKeCoiHg5cChwvKRXA58jlceLgHXAaaPNxIlhkomIHwKPVfU+EVia3y8FTmpmTK00QnlMWxHxUETcmt8/DtwJzGWariOjlMe0FMnw/WHa8iuAo4Arcv8x1w8nhqmhIyIeyu8fBjpaGcwk8QFJq3JV07SoNqkmaR7wCuAmvI5UlwdM03VE0gxJPwPWAtcC9wLrI2JTHuU3jJE8nRimmEjti6d7G+N/A15IOlR+CPinlkbTApLagW8AH46IjZXDpuM6UqM8pu06EhGbI+JQ0h0kDgdeWnYeTgxTwxpJswFyd22L42mpiFiTV/5ngPNJK/+0IamNtBG8JCKuzL2n7TpSqzym+zoCEBHrgeuBI4BZkoYvaB7zlkNODFPDMmBRfr8IuKqFsbTc8AYwexNw20jjbm8kCbgQuDMizq0YNC3XkZHKY7quI5L2ljQrv9+V9IybO0kJ4i15tDHXD1/5PMlI+hrQRbpV7hrg08C3gMuBA4DVwMkRMS1OyI5QHl2kKoIAHgDeW1G/vl2T9FrgR8AvSA8dB/gkqV592q0jo5THO5iG64ikBaSTyzNIO/6XR8TfSHoB0APsCfwU+NOIeGrE+TgxmJlZJVclmZlZgRODmZkVODGYmVmBE4OZmRU4MZiZWYETg9kI8h06o8brulbHZtZIk+LRnmaT2Aa2vrvrhlYEYtYsTgxmo9sUETdOxIwk7RoRv5uIeZk1kquSzEqSNDvfsfM+Sb+T9CtJn6l8+Imkebna6RRJX5W0Hvh2HranpCWS1kh6UtJPJL2qVd/HrJqPGMzGUHHzsWF7kZ4RcQbpoScvBs4C9gbeWzXu54ErgbcCmyXtDFwHzAI+TrrZ3fuA6yQdGBEPN+ZbmNXPicFsdM8Dhqr6HRsRHxv+IOm/gUHgIkkfjIinK8a9MSJOrxj3NOAQ4OCIuDv3uw64C/goKVmYtZQTg9noNgDHVPW7S9KHgcXAfGCXimEHAPdUfP6vqmmPAVYC91cdidwAdE5EwGbbyonBbHSbImJFZQ9JHwHOIT1H9wZSddJhwJcpJglId4SttBfp2cTVRyGQnrRl1nJODGblvRW4IiL+93CPigfQV6u+ffFjwArSeYVqI94G2ayZnBjMytuVrTfip9Q57XLgOODBiJg2T1mzqcWJway8a4G/lHQTqfrnFOBFdU77VeAvgF5JnwfuI53gPhx4OCLOa0C8ZqU4MZiV9zekpqmfyZ+vBP6SfJ3CaCLiSUndeR5nAx2kJqs3kx7PadZyfoKbmZkV+MpnMzMrcGIwM7MCJwYzMytwYjAzswInBjMzK3BiMDOzAicGMzMrcGIwM7OC/w/6y8eHCOWPigAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "## Distribution of the group from which those two outliers were flagged\n", "%matplotlib inline\n", "import matplotlib.pyplot as plt\n", "titanic.loc[\n", " (titanic.Pclass == 3) &\n", " (titanic.SibSp == 0) &\n", " (titanic.Embarked == \"Q\")\n", "] .Fare.hist(bins=50, color=\"navy\", edgecolor='black', linewidth=1.2)\n", "plt.xlabel(\"Fare\", fontsize=15)\n", "plt.ylabel(\"Frequency\", fontsize=15)\n", "plt.title(\"Distribution of Fare within cluster\", fontsize=20)\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
suspicious_valuegroup_statisticsconditionstree_depthuses_NA_branchoutlier_score
1146{'column': 'Fare', 'value': 29.125, 'decimals'...{'upper_thr': 15.5, 'pct_below': 0.97849462365...[{'column': 'Embarked', 'comparison': '=', 'va...4.0False0.003805
1163{'column': 'Fare', 'value': 24.15, 'decimals': 0}{'upper_thr': 15.5, 'pct_below': 0.97849462365...[{'column': 'Embarked', 'comparison': '=', 'va...4.0False0.005227
\n", "
" ], "text/plain": [ " suspicious_value \\\n", "1146 {'column': 'Fare', 'value': 29.125, 'decimals'... \n", "1163 {'column': 'Fare', 'value': 24.15, 'decimals': 0} \n", "\n", " group_statistics \\\n", "1146 {'upper_thr': 15.5, 'pct_below': 0.97849462365... \n", "1163 {'upper_thr': 15.5, 'pct_below': 0.97849462365... \n", "\n", " conditions tree_depth \\\n", "1146 [{'column': 'Embarked', 'comparison': '=', 'va... 4.0 \n", "1163 [{'column': 'Embarked', 'comparison': '=', 'va... 4.0 \n", "\n", " uses_NA_branch outlier_score \n", "1146 False 0.003805 \n", "1163 False 0.005227 " ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "## Get the outliers in a manipulable format\n", "otree.predict(titanic).loc[[1146, 1163]]" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
suspicious_valuegroup_statisticsconditionstree_depthuses_NA_branchoutlier_score
18{'column': 'Age', 'value': 32.0, 'decimals': 0}{'lower_thr': 43.0, 'pct_above': 0.96, 'mean':...[{'column': 'Cabin', 'comparison': 'in', 'valu...3.0False0.007545
170{'column': 'Fare', 'value': 0.0, 'decimals': 0}{'lower_thr': 25.7417, 'pct_above': 0.98571428...[{'column': 'Boat', 'comparison': 'in', 'value...2.0False0.015339
896{'column': 'Fare', 'value': 0.0, 'decimals': 0}{'lower_thr': 3.1708, 'pct_above': 0.992156862...[{'column': 'Pclass', 'comparison': '=', 'valu...3.0False0.011148
898{'column': 'Fare', 'value': 0.0, 'decimals': 0}{'lower_thr': 3.1708, 'pct_above': 0.992156862...[{'column': 'Pclass', 'comparison': '=', 'valu...3.0False0.011148
963{'column': 'Fare', 'value': 0.0, 'decimals': 0}{'lower_thr': 3.1708, 'pct_above': 0.992156862...[{'column': 'Pclass', 'comparison': '=', 'valu...3.0False0.011148
1044{'column': 'Fare', 'value': 15.5, 'decimals': 0}{'upper_thr': 8.5167, 'pct_below': 0.967741935...[{'column': 'Boat', 'comparison': 'in', 'value...4.0False0.002018
1146{'column': 'Fare', 'value': 29.125, 'decimals'...{'upper_thr': 15.5, 'pct_below': 0.97849462365...[{'column': 'Embarked', 'comparison': '=', 'va...4.0False0.003805
1163{'column': 'Fare', 'value': 24.15, 'decimals': 0}{'upper_thr': 15.5, 'pct_below': 0.97849462365...[{'column': 'Embarked', 'comparison': '=', 'va...4.0False0.005227
1254{'column': 'Fare', 'value': 0.0, 'decimals': 0}{'lower_thr': 3.1708, 'pct_above': 0.992156862...[{'column': 'Pclass', 'comparison': '=', 'valu...3.0False0.011148
\n", "
" ], "text/plain": [ " suspicious_value \\\n", "18 {'column': 'Age', 'value': 32.0, 'decimals': 0} \n", "170 {'column': 'Fare', 'value': 0.0, 'decimals': 0} \n", "896 {'column': 'Fare', 'value': 0.0, 'decimals': 0} \n", "898 {'column': 'Fare', 'value': 0.0, 'decimals': 0} \n", "963 {'column': 'Fare', 'value': 0.0, 'decimals': 0} \n", "1044 {'column': 'Fare', 'value': 15.5, 'decimals': 0} \n", "1146 {'column': 'Fare', 'value': 29.125, 'decimals'... \n", "1163 {'column': 'Fare', 'value': 24.15, 'decimals': 0} \n", "1254 {'column': 'Fare', 'value': 0.0, 'decimals': 0} \n", "\n", " group_statistics \\\n", "18 {'lower_thr': 43.0, 'pct_above': 0.96, 'mean':... \n", "170 {'lower_thr': 25.7417, 'pct_above': 0.98571428... \n", "896 {'lower_thr': 3.1708, 'pct_above': 0.992156862... \n", "898 {'lower_thr': 3.1708, 'pct_above': 0.992156862... \n", "963 {'lower_thr': 3.1708, 'pct_above': 0.992156862... \n", "1044 {'upper_thr': 8.5167, 'pct_below': 0.967741935... \n", "1146 {'upper_thr': 15.5, 'pct_below': 0.97849462365... \n", "1163 {'upper_thr': 15.5, 'pct_below': 0.97849462365... \n", "1254 {'lower_thr': 3.1708, 'pct_above': 0.992156862... \n", "\n", " conditions tree_depth \\\n", "18 [{'column': 'Cabin', 'comparison': 'in', 'valu... 3.0 \n", "170 [{'column': 'Boat', 'comparison': 'in', 'value... 2.0 \n", "896 [{'column': 'Pclass', 'comparison': '=', 'valu... 3.0 \n", "898 [{'column': 'Pclass', 'comparison': '=', 'valu... 3.0 \n", "963 [{'column': 'Pclass', 'comparison': '=', 'valu... 3.0 \n", "1044 [{'column': 'Boat', 'comparison': 'in', 'value... 4.0 \n", "1146 [{'column': 'Embarked', 'comparison': '=', 'va... 4.0 \n", "1163 [{'column': 'Embarked', 'comparison': '=', 'va... 4.0 \n", "1254 [{'column': 'Pclass', 'comparison': '=', 'valu... 3.0 \n", "\n", " uses_NA_branch outlier_score \n", "18 False 0.007545 \n", "170 False 0.015339 \n", "896 False 0.011148 \n", "898 False 0.011148 \n", "963 False 0.011148 \n", "1044 False 0.002018 \n", "1146 False 0.003805 \n", "1163 False 0.005227 \n", "1254 False 0.011148 " ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "## To programatically get all the outliers that were flagged\n", "pred = otree.predict(titanic)\n", "pred.loc[~pred.outlier_score.isnull()]" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Reporting top 1 outliers [out of 1 found]\n", "\n", "\n", "row [1146] - suspicious column: [Fare] - suspicious value: [29.12]\n", "\tdistribution: 97.849% <= 15.50 - [mean: 7.89] - [sd: 1.17] - [norm. obs: 91]\n", "\tgiven:\n", "\t\t[Pclass] = [3]\n", "\t\t[SibSp] = [0]\n", "\t\t[Embarked] = [Q]\n", "\n", "\n" ] } ], "source": [ "## To print selected rows only\n", "otree.print_outliers(pred.loc[[1146]])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Trying different hyperparameters" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Reporting top 5 outliers [out of 20 found]\n", "\n", "\n", "row [363] - suspicious column: [Fare] - suspicious value: [0.00]\n", "\tdistribution: 98.555% >= 3.17 - [mean: 11.66] - [sd: 9.02] - [norm. obs: 682]\n", "\tgiven:\n", "\t\t[Pclass] in [2, 3] (value: 2)\n", "\t\t[SibSp] = [0]\n", "\n", "\n", "row [384] - suspicious column: [Fare] - suspicious value: [0.00]\n", "\tdistribution: 98.555% >= 3.17 - [mean: 11.66] - [sd: 9.02] - [norm. obs: 682]\n", "\tgiven:\n", "\t\t[Pclass] in [2, 3] (value: 2)\n", "\t\t[SibSp] = [0]\n", "\n", "\n", "row [410] - suspicious column: [Fare] - suspicious value: [0.00]\n", "\tdistribution: 98.555% >= 3.17 - [mean: 11.66] - [sd: 9.02] - [norm. obs: 682]\n", "\tgiven:\n", "\t\t[Pclass] in [2, 3] (value: 2)\n", "\t\t[SibSp] = [0]\n", "\n", "\n", "row [473] - suspicious column: [Fare] - suspicious value: [0.00]\n", "\tdistribution: 98.555% >= 3.17 - [mean: 11.66] - [sd: 9.02] - [norm. obs: 682]\n", "\tgiven:\n", "\t\t[Pclass] in [2, 3] (value: 2)\n", "\t\t[SibSp] = [0]\n", "\n", "\n", "row [528] - suspicious column: [Fare] - suspicious value: [0.00]\n", "\tdistribution: 98.555% >= 3.17 - [mean: 11.66] - [sd: 9.02] - [norm. obs: 682]\n", "\tgiven:\n", "\t\t[Pclass] in [2, 3] (value: 2)\n", "\t\t[SibSp] = [0]\n", "\n", "\n" ] }, { "data": { "text/plain": [ "OutlierTree model\n", "\tNumeric variables: 3\n", "\tCategorical variables: 5\n", "\tOrdinal variables: 3\n", "\n", "Consists of 217 clusters, spread across 18 tree branches" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "## In order to flag more outliers, one can also experiment\n", "## with lowering the threshold hyperparameters\n", "OutlierTree(z_outlier=6.).fit(titanic, outliers_print=5)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Reporting top 5 outliers [out of 27 found]\n", "\n", "\n", "row [545] - suspicious column: [SibSp] - suspicious value: [3]\n", "\tdistribution: 99.701% in [0, 1, 2, 5, 8]\n", "\t( [norm. obs: 999] - [prior_prob: 1.528%] - [next smallest: 2.595%] )\n", "\tgiven:\n", "\t\t[Parch] = [0]\n", "\n", "\n", "row [656] - suspicious column: [SibSp] - suspicious value: [3]\n", "\tdistribution: 99.701% in [0, 1, 2, 5, 8]\n", "\t( [norm. obs: 999] - [prior_prob: 1.528%] - [next smallest: 2.595%] )\n", "\tgiven:\n", "\t\t[Parch] = [0]\n", "\n", "\n", "row [1274] - suspicious column: [SibSp] - suspicious value: [3]\n", "\tdistribution: 99.701% in [0, 1, 2, 5, 8]\n", "\t( [norm. obs: 999] - [prior_prob: 1.528%] - [next smallest: 2.595%] )\n", "\tgiven:\n", "\t\t[Parch] = [0]\n", "\n", "\n", "row [363] - suspicious column: [Fare] - suspicious value: [0.00]\n", "\tdistribution: 98.555% >= 3.17 - [mean: 11.66] - [sd: 9.02] - [norm. obs: 682]\n", "\tgiven:\n", "\t\t[Pclass] in [2, 3] (value: 2)\n", "\t\t[SibSp] = [0]\n", "\n", "\n", "row [384] - suspicious column: [Fare] - suspicious value: [0.00]\n", "\tdistribution: 98.555% >= 3.17 - [mean: 11.66] - [sd: 9.02] - [norm. obs: 682]\n", "\tgiven:\n", "\t\t[Pclass] in [2, 3] (value: 2)\n", "\t\t[SibSp] = [0]\n", "\n", "\n" ] }, { "data": { "text/plain": [ "OutlierTree model\n", "\tNumeric variables: 3\n", "\tCategorical variables: 5\n", "\tOrdinal variables: 3\n", "\n", "Consists of 283 clusters, spread across 23 tree branches" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "## One can also lower the gain threshold, but this tends\n", "## to result in more spurious outliers which come from\n", "## not-so-good splits (not recommended)\n", "OutlierTree(z_outlier=6, min_gain=1e-6).fit(titanic, outliers_print=5)" ] } ], "metadata": { "kernelspec": { "display_name": "Python (OpenBLAS)", "language": "python", "name": "py3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.7" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": false } }, "nbformat": 4, "nbformat_minor": 2 }