{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Phi_K advanced tutorial\n", "\n", "This notebook guides you through the more advanced functionality of the phik package. This notebook will not cover all the underlying theory, but will just attempt to give an overview of all the options that are available. For a theoretical description the user is referred to our paper.\n", "\n", "The package offers functionality on three related topics:\n", "\n", "1. Phik correlation matrix\n", "2. Significance matrix\n", "3. Outlier significance matrix" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "%%capture\n", "# install phik (if not installed yet)\n", "import sys\n", "\n", "!\"{sys.executable}\" -m pip install phik" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# import standard packages\n", "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "import itertools\n", "\n", "import phik\n", "\n", "from phik import resources\n", "from phik.binning import bin_data\n", "from phik.decorators import *\n", "from phik.report import plot_correlation_matrix\n", "\n", "%matplotlib inline" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "# if one changes something in the phik-package one can automatically reload the package or module\n", "%load_ext autoreload\n", "%autoreload 2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Load data\n", "\n", "A simulated dataset is part of the phik-package. The dataset concerns car insurance data. Load the dataset here:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "data = pd.read_csv( resources.fixture('fake_insurance_data.csv.gz') )" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
car_colordriver_ageareamileagecar_size
0black26.377219suburbs156806.288398XXL
1black58.976840suburbs74400.323559XL
2multicolor55.744988downtown267856.748015XXL
3metalic57.629139downtown259028.249060XXL
4green21.490637downtown110712.216080XL
\n", "
" ], "text/plain": [ " car_color driver_age area mileage car_size\n", "0 black 26.377219 suburbs 156806.288398 XXL\n", "1 black 58.976840 suburbs 74400.323559 XL\n", "2 multicolor 55.744988 downtown 267856.748015 XXL\n", "3 metalic 57.629139 downtown 259028.249060 XXL\n", "4 green 21.490637 downtown 110712.216080 XL" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Specify bin types\n", "\n", "The phik-package offers a way to calculate correlations between variables of mixed types. Variable types can be inferred automatically although we recommend to variable types to be specified by the user. \n", "\n", "Because interval type variables need to be binned in order to calculate phik and the significance, a list of interval variables is created." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['driver_age', 'mileage']" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_types = {'severity': 'interval',\n", " 'driver_age':'interval',\n", " 'satisfaction':'ordinal',\n", " 'mileage':'interval',\n", " 'car_size':'ordinal',\n", " 'car_use':'ordinal',\n", " 'car_color':'categorical',\n", " 'area':'categorical'}\n", "\n", "interval_cols = [col for col, v in data_types.items() if v=='interval' and col in data.columns]\n", "interval_cols\n", "# interval_cols is used below" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Phik correlation matrix\n", "\n", "Now let's start calculating the correlation phik between pairs of variables. \n", "\n", "Note that the original dataset is used as input, the binning of interval variables is done automatically." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
car_colordriver_ageareamileagecar_size
car_color1.0000000.3896710.5904560.0000000.000000
driver_age0.3896711.0000000.1055060.0000000.000000
area0.5904560.1055061.0000000.0000000.000000
mileage0.0000000.0000000.0000001.0000000.768589
car_size0.0000000.0000000.0000000.7685891.000000
\n", "
" ], "text/plain": [ " car_color driver_age area mileage car_size\n", "car_color 1.000000 0.389671 0.590456 0.000000 0.000000\n", "driver_age 0.389671 1.000000 0.105506 0.000000 0.000000\n", "area 0.590456 0.105506 1.000000 0.000000 0.000000\n", "mileage 0.000000 0.000000 0.000000 1.000000 0.768589\n", "car_size 0.000000 0.000000 0.000000 0.768589 1.000000" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "phik_overview = data.phik_matrix(interval_cols=interval_cols)\n", "phik_overview" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Specify binning per interval variable\n", "\n", "Binning can be set per interval variable individually. One can set the number of bins, or specify a list of bin edges. Note that the measured phik correlation is dependent on the chosen binning. \n", "The default binning is uniform between the min and max values of the interval variable." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
car_colordriver_ageareamileagecar_size
car_color1.0000000.3883500.5904560.0000000.000000
driver_age0.3883501.0000000.0711890.0000000.000000
area0.5904560.0711891.0000000.0000000.000000
mileage0.0000000.0000000.0000001.0000000.665845
car_size0.0000000.0000000.0000000.6658451.000000
\n", "
" ], "text/plain": [ " car_color driver_age area mileage car_size\n", "car_color 1.000000 0.388350 0.590456 0.000000 0.000000\n", "driver_age 0.388350 1.000000 0.071189 0.000000 0.000000\n", "area 0.590456 0.071189 1.000000 0.000000 0.000000\n", "mileage 0.000000 0.000000 0.000000 1.000000 0.665845\n", "car_size 0.000000 0.000000 0.000000 0.665845 1.000000" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "bins = {'mileage':5, 'driver_age':[18,25,35,45,55,65,125]}\n", "phik_overview = data.phik_matrix(interval_cols=interval_cols, bins=bins)\n", "phik_overview" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Do not apply noise correction\n", "\n", "For low statistics samples often a correlation larger than zero is measured when no correlation is actually present in the true underlying distribution. This is not only the case for phik, but also for the pearson correlation and Cramer's phi (see figure 4 in XX ). In the phik calculation a noise correction is applied by default, to take into account erroneous correlation values as a result of low statistics. To switch off this noise cancellation (not recommended), do:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
car_colordriver_ageareamileagecar_size
car_color1.0000000.4078600.5941720.1362670.096629
driver_age0.4078601.0000000.1903900.1996060.121585
area0.5941720.1903901.0000000.1496790.067452
mileage0.1362670.1996060.1496791.0000000.770836
car_size0.0966290.1215850.0674520.7708361.000000
\n", "
" ], "text/plain": [ " car_color driver_age area mileage car_size\n", "car_color 1.000000 0.407860 0.594172 0.136267 0.096629\n", "driver_age 0.407860 1.000000 0.190390 0.199606 0.121585\n", "area 0.594172 0.190390 1.000000 0.149679 0.067452\n", "mileage 0.136267 0.199606 0.149679 1.000000 0.770836\n", "car_size 0.096629 0.121585 0.067452 0.770836 1.000000" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "phik_overview = data.phik_matrix(interval_cols=interval_cols, noise_correction=False)\n", "phik_overview" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Using a different expectation histogram\n", "\n", "By default phik compares the 2d distribution of two (binned) variables with the distribution that assumes no dependency between them. One can also change the expected distribution though. Phi_K is calculated in the same way, but using the other expectation distribution. " ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "from phik.binning import auto_bin_data\n", "from phik.phik import phik_observed_vs_expected_from_rebinned_df, phik_from_hist2d\n", "from phik.statistics import get_dependent_frequency_estimates" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "# get observed 2d histogram of two variables\n", "cols = [\"mileage\", \"car_size\"]\n", "icols = [\"mileage\"]\n", "observed = data[cols].hist2d(interval_cols=icols).values" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.768588829489185\n" ] } ], "source": [ "# default phik evaluation from observed distribution\n", "phik_value = phik_from_hist2d(observed)\n", "print (phik_value)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.768588829489185\n" ] } ], "source": [ "# phik evaluation from an observed and expected distribution\n", "expected = get_dependent_frequency_estimates(observed)\n", "phik_value = phik_from_hist2d(observed=observed, expected=expected)\n", "print (phik_value)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "# one can also compare two datasets against each other, and get a full phik matrix that way.\n", "# this needs binned datasets though. \n", "# (the user needs to make sure the binnings of both datasets are identical.) \n", "data_binned, _ = auto_bin_data(data, interval_cols=interval_cols)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "# here we are comparing data_binned against itself\n", "phik_matrix = phik_observed_vs_expected_from_rebinned_df(data_binned, data_binned)" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
car_colordriver_ageareamileagecar_size
car_color1.00.00.00.00.0
driver_age0.01.00.00.00.0
area0.00.01.00.00.0
mileage0.00.00.01.00.0
car_size0.00.00.00.01.0
\n", "
" ], "text/plain": [ " car_color driver_age area mileage car_size\n", "car_color 1.0 0.0 0.0 0.0 0.0\n", "driver_age 0.0 1.0 0.0 0.0 0.0\n", "area 0.0 0.0 1.0 0.0 0.0\n", "mileage 0.0 0.0 0.0 1.0 0.0\n", "car_size 0.0 0.0 0.0 0.0 1.0" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# all off-diagonal entries are zero, meaning the all 2d distributions of both datasets are identical.\n", "# (by construction the diagonal is one.)\n", "phik_matrix" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Statistical significance of the correlation\n", "\n", "When assessing correlations it is good practise to evaluate both the correlation and the significance of the correlation: a large correlation may be statistically insignificant, and vice versa a small correlation may be very significant. For instance, scipy.stats.pearsonr returns both the pearson correlation and the p-value. Similarly, the phik package offers functionality the calculate a significance matrix. Significance is defined as:\n", "\n", "$$Z = \\Phi^{-1}(1-p)\\ ;\\quad \\Phi(z)=\\frac{1}{\\sqrt{2\\pi}} \\int_{-\\infty}^{z} e^{-t^{2}/2}\\,{\\rm d}t $$\n", "\n", "Several corrections to the 'standard' p-value calculation are taken into account, making the method more robust for low statistics and sparse data cases. The user is referred to our paper for more details.\n", "\n", "Due to the corrections, the significance calculation can take a few seconds." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
car_colordriver_ageareamileagecar_size
car_color85.49865519.83672037.623764-0.559532-0.483387
driver_age19.83672084.3705421.852524-0.572284-0.459980
area37.6237641.85252472.415600-0.560672-0.273138
mileage-0.559532-0.572284-0.56067291.26267749.285368
car_size-0.483387-0.459980-0.27313849.28536869.064056
\n", "
" ], "text/plain": [ " car_color driver_age area mileage car_size\n", "car_color 85.498655 19.836720 37.623764 -0.559532 -0.483387\n", "driver_age 19.836720 84.370542 1.852524 -0.572284 -0.459980\n", "area 37.623764 1.852524 72.415600 -0.560672 -0.273138\n", "mileage -0.559532 -0.572284 -0.560672 91.262677 49.285368\n", "car_size -0.483387 -0.459980 -0.273138 49.285368 69.064056" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "significance_overview = data.significance_matrix(interval_cols=interval_cols)\n", "significance_overview" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Specify binning per interval variable\n", "Binning can be set per interval variable individually. One can set the number of bins, or specify a list of bin edges. Note that the measure phik correlation is dependent on the chosen binning." ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
car_colordriver_ageareamileagecar_size
car_color85.48087020.54440037.613135-0.214896-0.447747
driver_age20.54440083.3441682.478032-0.563892-0.534263
area37.6131352.47803272.428355-0.309349-0.260994
mileage-0.214896-0.563892-0.30934977.78408647.010736
car_size-0.447747-0.534263-0.26099447.01073669.081712
\n", "
" ], "text/plain": [ " car_color driver_age area mileage car_size\n", "car_color 85.480870 20.544400 37.613135 -0.214896 -0.447747\n", "driver_age 20.544400 83.344168 2.478032 -0.563892 -0.534263\n", "area 37.613135 2.478032 72.428355 -0.309349 -0.260994\n", "mileage -0.214896 -0.563892 -0.309349 77.784086 47.010736\n", "car_size -0.447747 -0.534263 -0.260994 47.010736 69.081712" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "bins = {'mileage':5, 'driver_age':[18,25,35,45,55,65,125]}\n", "significance_overview = data.significance_matrix(interval_cols=interval_cols, bins=bins)\n", "significance_overview" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Specify significance method\n", "\n", "The recommended method to calculate the significance of the correlation is a hybrid approach, which uses the G-test statistic. The number of degrees of freedom and an analytical, empirical description of the $\\chi^2$ distribution are sed, based on Monte Carlo simulations. This method works well for both high as low statistics samples.\n", "\n", "Other approaches to calculate the significance are implemented:\n", "- asymptotic: fast, but over-estimates the number of degrees of freedom for low statistics samples, leading to erroneous values of the significance\n", "- MC: Many simulated samples are needed to accurately measure significances larger than 3, making this method computationally expensive.\n" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
car_colordriver_ageareamileagecar_size
car_color85.52657419.68156437.661844-0.385023-0.333340
driver_age19.68156484.0146541.742050-0.947153-0.793434
area37.6618441.74205072.440209-0.465002-0.123678
mileage-0.385023-0.947153-0.46500291.30112949.332305
car_size-0.333340-0.793434-0.12367849.33230569.107448
\n", "
" ], "text/plain": [ " car_color driver_age area mileage car_size\n", "car_color 85.526574 19.681564 37.661844 -0.385023 -0.333340\n", "driver_age 19.681564 84.014654 1.742050 -0.947153 -0.793434\n", "area 37.661844 1.742050 72.440209 -0.465002 -0.123678\n", "mileage -0.385023 -0.947153 -0.465002 91.301129 49.332305\n", "car_size -0.333340 -0.793434 -0.123678 49.332305 69.107448" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "significance_overview = data.significance_matrix(interval_cols=interval_cols, significance_method='asymptotic')\n", "significance_overview" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Simulation method\n", "\n", "The chi2 of a contingency table is measured using a comparison of the expected frequencies with the true frequencies in a contingency table. The expected frequencies can be simulated in a variety of ways. The following methods are implemented:\n", "\n", " - multinominal: Only the total number of records is fixed. (default)\n", " - row_product_multinominal: The row totals fixed in the sampling.\n", " - col_product_multinominal: The column totals fixed in the sampling.\n", " - hypergeometric: Both the row or column totals are fixed in the sampling. (Note that this type of sampling is only available when row and column totals are integers, which is usually the case.)" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "scrolled": true }, "outputs": [], "source": [ "# --- Warning, can be slow\n", "# turned off here by default for unit testing purposes\n", "\n", "#significance_overview = data.significance_matrix(interval_cols=interval_cols, simulation_method='hypergeometric')\n", "#significance_overview" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Expected frequencies" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "from phik.simulation import sim_2d_data_patefield, sim_2d_product_multinominal, sim_2d_data" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
areacountry_sidedowntownhillssuburbsunpaved_roads
driver_age
111.086.0123.0147.021.0
29.077.0137.0125.031.0
37.0102.0131.0130.018.0
417.083.0130.095.014.0
513.068.0120.072.08.0
67.030.051.047.09.0
71.011.023.014.07.0
80.04.07.08.02.0
90.00.01.01.00.0
100.01.01.00.00.0
\n", "
" ], "text/plain": [ "area country_side downtown hills suburbs unpaved_roads\n", "driver_age \n", "1 11.0 86.0 123.0 147.0 21.0\n", "2 9.0 77.0 137.0 125.0 31.0\n", "3 7.0 102.0 131.0 130.0 18.0\n", "4 17.0 83.0 130.0 95.0 14.0\n", "5 13.0 68.0 120.0 72.0 8.0\n", "6 7.0 30.0 51.0 47.0 9.0\n", "7 1.0 11.0 23.0 14.0 7.0\n", "8 0.0 4.0 7.0 8.0 2.0\n", "9 0.0 0.0 1.0 1.0 0.0\n", "10 0.0 1.0 1.0 0.0 0.0" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "inputdata = data[['driver_age', 'area']].hist2d(interval_cols=['driver_age'])\n", "inputdata" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Multinominal" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "data total: 2000.0\n", "sim total: 2000\n", "data row totals: [ 65. 462. 724. 639. 110.]\n", "sim row totals: [ 75 468 748 586 123]\n", "data column totals: [388. 379. 388. 339. 281. 144. 56. 21. 2. 2.]\n", "sim column totals: [378 380 375 335 281 164 59 25 1 2]\n" ] } ], "source": [ "simdata = sim_2d_data(inputdata.values)\n", "print('data total:', inputdata.sum().sum())\n", "print('sim total:', simdata.sum().sum())\n", "print('data row totals:', inputdata.sum(axis=0).values)\n", "print('sim row totals:', simdata.sum(axis=0))\n", "print('data column totals:', inputdata.sum(axis=1).values)\n", "print('sim column totals:', simdata.sum(axis=1))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### product multinominal" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "data total: 2000.0\n", "sim total: 2000\n", "data row totals: [ 65 462 724 639 110]\n", "sim row totals: [ 65 462 724 639 110]\n", "data column totals: [388 379 388 339 281 144 56 21 2 2]\n", "sim column totals: [399 353 415 349 272 139 45 22 4 2]\n" ] } ], "source": [ "simdata = sim_2d_product_multinominal(inputdata.values, axis=0)\n", "print('data total:', inputdata.sum().sum())\n", "print('sim total:', simdata.sum().sum())\n", "print('data row totals:', inputdata.sum(axis=0).astype(int).values)\n", "print('sim row totals:', simdata.sum(axis=0).astype(int))\n", "print('data column totals:', inputdata.sum(axis=1).astype(int).values)\n", "print('sim column totals:', simdata.sum(axis=1).astype(int))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### hypergeometric (\"patefield\")" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "data total: 2000.0\n", "sim total: 2000\n", "data row totals: [ 65 462 724 639 110]\n", "sim row totals: [ 65 462 724 639 110]\n", "data column totals: [388 379 388 339 281 144 56 21 2 2]\n", "sim column totals: [388 379 388 339 281 144 56 21 2 2]\n" ] } ], "source": [ "# patefield simulation needs compiled c++ code.\n", "# only run this if the python binding to the (compiled) patefiled simulation function is found.\n", "try:\n", " from phik.simcore import _sim_2d_data_patefield\n", " CPP_SUPPORT = True\n", "except ImportError:\n", " CPP_SUPPORT = False\n", "\n", "if CPP_SUPPORT:\n", " simdata = sim_2d_data_patefield(inputdata.values)\n", " print('data total:', inputdata.sum().sum())\n", " print('sim total:', simdata.sum().sum())\n", " print('data row totals:', inputdata.sum(axis=0).astype(int).values)\n", " print('sim row totals:', simdata.sum(axis=0))\n", " print('data column totals:', inputdata.sum(axis=1).astype(int).values)\n", " print('sim column totals:', simdata.sum(axis=1))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Outlier significance\n", "\n", "The normal pearson correlation between two interval variables is easy to interpret. However, the phik correlation between two variables of mixed type is not always easy to interpret, especially when it concerns categorical variables. Therefore, functionality is provided to detect \"outliers\": excesses and deficits over the expected frequencies in the contingency table of two variables. \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Example 1: mileage versus car_size" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For the categorical variable pair mileage - car_size we measured:\n", "\n", "$$\\phi_k = 0.77 \\, ,\\quad\\quad \\mathrm{significance} = 46.3$$\n", "\n", "Let's use the outlier significance functionality to gain a better understanding of this significance correlation between mileage and car size.\n" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [], "source": [ "c0 = 'mileage'\n", "c1 = 'car_size'\n", "\n", "tmp_interval_cols = ['mileage']" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
car_sizeLMSXLXSXXL
53.5_30047.06.88215521.48347618.076204-8.20953610.820863-22.423985
30047.0_60040.520.034528-0.251737-3.4084092.534277-1.973628-8.209536
60040.5_90033.91.627610-3.043497-2.26580910.215936-1.246784-8.209536
90033.9_120027.4-3.711579-3.827278-2.88547512.999048-1.638288-7.185622
120027.4_150020.9-7.665861-6.173001-4.7467629.629145-2.841508-0.504521
150020.9_180014.4-7.533189-6.063786-4.6600491.559370-2.7850496.765549
180014.4_210007.8-5.541940-4.425929-3.360023-4.802787-1.94246910.520540
210007.8_240001.3-3.496905-2.745103-2.030802-5.850529-1.1008738.723925
240001.3_269994.8-5.275976-4.207164-3.186534-8.616464-1.83094413.303101
269994.8_299988.2-8.014016-6.458253-4.973240-12.868389-2.98905520.992824
\n", "
" ], "text/plain": [ "car_size L M S XL XS \\\n", "53.5_30047.0 6.882155 21.483476 18.076204 -8.209536 10.820863 \n", "30047.0_60040.5 20.034528 -0.251737 -3.408409 2.534277 -1.973628 \n", "60040.5_90033.9 1.627610 -3.043497 -2.265809 10.215936 -1.246784 \n", "90033.9_120027.4 -3.711579 -3.827278 -2.885475 12.999048 -1.638288 \n", "120027.4_150020.9 -7.665861 -6.173001 -4.746762 9.629145 -2.841508 \n", "150020.9_180014.4 -7.533189 -6.063786 -4.660049 1.559370 -2.785049 \n", "180014.4_210007.8 -5.541940 -4.425929 -3.360023 -4.802787 -1.942469 \n", "210007.8_240001.3 -3.496905 -2.745103 -2.030802 -5.850529 -1.100873 \n", "240001.3_269994.8 -5.275976 -4.207164 -3.186534 -8.616464 -1.830944 \n", "269994.8_299988.2 -8.014016 -6.458253 -4.973240 -12.868389 -2.989055 \n", "\n", "car_size XXL \n", "53.5_30047.0 -22.423985 \n", "30047.0_60040.5 -8.209536 \n", "60040.5_90033.9 -8.209536 \n", "90033.9_120027.4 -7.185622 \n", "120027.4_150020.9 -0.504521 \n", "150020.9_180014.4 6.765549 \n", "180014.4_210007.8 10.520540 \n", "210007.8_240001.3 8.723925 \n", "240001.3_269994.8 13.303101 \n", "269994.8_299988.2 20.992824 " ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "outlier_signifs, binning_dict = data[[c0,c1]].outlier_significance_matrix(interval_cols=tmp_interval_cols, \n", " retbins=True)\n", "outlier_signifs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Specify binning per interval variable\n", "Binning can be set per interval variable individually. One can set the number of bins, or specify a list of bin edges. \n", "\n", "Note: in case a bin is created without any records this bin will be automatically dropped in the phik and (outlier) significance calculations. However, in the outlier significance calculation this will currently lead to an error as the number of provided bin edges does not match the number of bins anymore." ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
car_sizeLMSXLXSXXL
0.0_100.0-0.223635-0.153005-0.096640-0.5041672.150837-1.337308
100.0_1000.0-0.742899-0.5332112.164954-1.4699965.704340-3.272689
1000.0_10000.0-3.4896683.49985618.061724-6.83106211.617394-13.063085
10000.0_100000.025.08672315.956527-0.2518775.162309-3.896807-8.209536
100000.0_1000000.0-8.209536-17.223164-13.626621-2.140870-8.68884444.933133
\n", "
" ], "text/plain": [ "car_size L M S XL XS \\\n", "0.0_100.0 -0.223635 -0.153005 -0.096640 -0.504167 2.150837 \n", "100.0_1000.0 -0.742899 -0.533211 2.164954 -1.469996 5.704340 \n", "1000.0_10000.0 -3.489668 3.499856 18.061724 -6.831062 11.617394 \n", "10000.0_100000.0 25.086723 15.956527 -0.251877 5.162309 -3.896807 \n", "100000.0_1000000.0 -8.209536 -17.223164 -13.626621 -2.140870 -8.688844 \n", "\n", "car_size XXL \n", "0.0_100.0 -1.337308 \n", "100.0_1000.0 -3.272689 \n", "1000.0_10000.0 -13.063085 \n", "10000.0_100000.0 -8.209536 \n", "100000.0_1000000.0 44.933133 " ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "bins = [0,1E2, 1E3, 1E4, 1E5, 1E6]\n", "outlier_signifs, binning_dict = data[[c0,c1]].outlier_significance_matrix(interval_cols=tmp_interval_cols, \n", " bins=bins, retbins=True)\n", "outlier_signifs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Specify binning per interval variable -- dealing with underflow and overflow\n", "\n", "When specifying custom bins as situation can occur when the minimal (maximum) value in the data is smaller (larger) than the minimum (maximum) bin edge. Data points outside the specified range will be collected in the underflow (UF) and overflow (OF) bins. One can choose how to deal with these under/overflow bins, by setting the drop_underflow and drop_overflow variables.\n", "\n", "Note that the drop_underflow and drop_overflow options are also available for the calculation of the phik matrix and the significance matrix." ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
car_sizeLMSXLXSXXL
100.0_1000.0-0.742899-0.5332112.164954-1.4699965.704340-3.272689
1000.0_10000.0-3.4896683.49985618.061724-6.83106211.617394-13.063085
10000.0_100000.025.08672315.956527-0.2518775.162309-3.896807-8.209536
OF-8.209536-17.223164-13.626621-2.140870-8.68884444.933133
UF-0.223635-0.153005-0.096640-0.5041672.150837-1.337308
\n", "
" ], "text/plain": [ "car_size L M S XL XS \\\n", "100.0_1000.0 -0.742899 -0.533211 2.164954 -1.469996 5.704340 \n", "1000.0_10000.0 -3.489668 3.499856 18.061724 -6.831062 11.617394 \n", "10000.0_100000.0 25.086723 15.956527 -0.251877 5.162309 -3.896807 \n", "OF -8.209536 -17.223164 -13.626621 -2.140870 -8.688844 \n", "UF -0.223635 -0.153005 -0.096640 -0.504167 2.150837 \n", "\n", "car_size XXL \n", "100.0_1000.0 -3.272689 \n", "1000.0_10000.0 -13.063085 \n", "10000.0_100000.0 -8.209536 \n", "OF 44.933133 \n", "UF -1.337308 " ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "bins = [1E2, 1E3, 1E4, 1E5]\n", "outlier_signifs, binning_dict = data[[c0,c1]].outlier_significance_matrix(interval_cols=tmp_interval_cols, \n", " bins=bins, retbins=True, \n", " drop_underflow=False,\n", " drop_overflow=False)\n", "outlier_signifs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Dealing with NaN's in the data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's add some missing values to our data" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [], "source": [ "data.loc[np.random.choice(range(len(data)), size=10), 'car_size'] = np.nan\n", "data.loc[np.random.choice(range(len(data)), size=10), 'mileage'] = np.nan" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Sometimes there can be information in the missing values and in which case you might want to consider the NaN values as a separate category. This can be achieved by setting the dropna argument to False." ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
car_sizeLMNaNSXLXSXXL
100.0_1000.0-0.742899-0.533211-0.0536202.185319-1.4673225.704340-3.254118
1000.0_10000.0-3.4896683.4998561.63243817.591610-6.82151111.617394-13.000691
10000.0_100000.024.90916415.798682-1.078812-0.0812424.943028-3.875525-8.209536
NaN0.1326490.488424-0.073439-0.455333-0.132365-0.211155-0.012896
OF-8.209536-17.158980-0.283391-13.396642-1.909226-8.65180043.560131
UF-0.223635-0.153005-0.013130-0.094218-0.5030512.150837-1.328194
\n", "
" ], "text/plain": [ "car_size L M NaN S XL \\\n", "100.0_1000.0 -0.742899 -0.533211 -0.053620 2.185319 -1.467322 \n", "1000.0_10000.0 -3.489668 3.499856 1.632438 17.591610 -6.821511 \n", "10000.0_100000.0 24.909164 15.798682 -1.078812 -0.081242 4.943028 \n", "NaN 0.132649 0.488424 -0.073439 -0.455333 -0.132365 \n", "OF -8.209536 -17.158980 -0.283391 -13.396642 -1.909226 \n", "UF -0.223635 -0.153005 -0.013130 -0.094218 -0.503051 \n", "\n", "car_size XS XXL \n", "100.0_1000.0 5.704340 -3.254118 \n", "1000.0_10000.0 11.617394 -13.000691 \n", "10000.0_100000.0 -3.875525 -8.209536 \n", "NaN -0.211155 -0.012896 \n", "OF -8.651800 43.560131 \n", "UF 2.150837 -1.328194 " ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "bins = [1E2, 1E3, 1E4, 1E5]\n", "outlier_signifs, binning_dict = data[[c0,c1]].outlier_significance_matrix(interval_cols=tmp_interval_cols, \n", " bins=bins, retbins=True, \n", " drop_underflow=False,\n", " drop_overflow=False,\n", " dropna=False)\n", "outlier_signifs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here OF and UF are the underflow and overflow bin of car_size, respectively.\n", "\n", "To just ignore records with missing values set dropna to True (default)." ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
car_sizeLMSXLXSXXL
100.0_1000.0-0.745805-0.5341792.177522-1.4736025.695755-3.268662
1000.0_10000.0-3.4517933.55970517.674546-6.77080711.651568-12.916946
10000.0_100000.025.03589615.868135-0.1211914.904070-3.896177-8.209536
OF-8.209536-17.164792-13.459625-1.934622-8.69554744.449479
UF-0.224643-0.153312-0.095154-0.5056612.146765-1.335316
\n", "
" ], "text/plain": [ "car_size L M S XL XS \\\n", "100.0_1000.0 -0.745805 -0.534179 2.177522 -1.473602 5.695755 \n", "1000.0_10000.0 -3.451793 3.559705 17.674546 -6.770807 11.651568 \n", "10000.0_100000.0 25.035896 15.868135 -0.121191 4.904070 -3.896177 \n", "OF -8.209536 -17.164792 -13.459625 -1.934622 -8.695547 \n", "UF -0.224643 -0.153312 -0.095154 -0.505661 2.146765 \n", "\n", "car_size XXL \n", "100.0_1000.0 -3.268662 \n", "1000.0_10000.0 -12.916946 \n", "10000.0_100000.0 -8.209536 \n", "OF 44.449479 \n", "UF -1.335316 " ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "bins = [1E2, 1E3, 1E4, 1E5]\n", "outlier_signifs, binning_dict = data[[c0,c1]].outlier_significance_matrix(interval_cols=tmp_interval_cols, \n", " bins=bins, retbins=True, \n", " drop_underflow=False,\n", " drop_overflow=False,\n", " dropna=True)\n", "outlier_signifs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that the dropna option is also available for the calculation of the phik matrix and the significance matrix." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.8" }, "pycharm": { "stem_cell": { "cell_type": "raw", "metadata": { "collapsed": false }, "source": [] } } }, "nbformat": 4, "nbformat_minor": 2 }