{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## Protodash: NHANES (CDC) data example\n", "- This notebook shows an example of how to use the ProtodashExplainer defined in [AIX360](https://github.com/IBM/AIX360/) to generate prototypes from (training/test) data. The notebook uses one of the [NHANES CDC questionnaire dataset](https://wwwn.cdc.gov/nchs/nhanes/search/datapage.aspx?Component=Questionnaire&CycleBeginYear=2013) related to incomes of individuals.\n", "- ProtodashExplainer is an implementation of the [Protodash algorithm](https://arxiv.org/abs/1707.01212)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Protodash Explainer examples" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Import statements" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "from sklearn.preprocessing import OneHotEncoder\n", "\n", "from aix360.algorithms.protodash import ProtodashExplainer, get_Gaussian_Data\n", "from aix360.datasets import CDCDataset" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Load NHANES dataset from CDC " ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "nhanes = CDCDataset()\n", "nhanes_files = nhanes.get_csv_file_names()\n", "(nhanesinfo, _, _) = nhanes._cdc_files_info()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "#### Explore NHANES Income questionnaire dataset\n", "\n", "Now let us explore the income questionnaire dataset and find out the types of responses received in the survey. Each column in this dataset corresponds to a question and each row denotes the answers given by a respondent to those questions. Both column names and answers by respondents are encoded. For example, 'SEQN' denotes the sequence number assigned to a respondent and 'IND235' corresponds to a question about monthly family income. As seen below, in most cases a value of 1 implies \"Yes\" to the question, while a value of 2 implies \"No\". More details about the income questionaire and how questions and answers are encoded can be seen [here](https://wwwn.cdc.gov/Nchs/Nhanes/2013-2014/INQ_H.htm)\n", "\n", "|Column |Description | Values and Meaning|\n", "|-------|----------------------------|---------|\n", "|SEQN | Respondent sequence number |\n", "|INQ020 | Income from wages/salaries |1->Yes, 2->No, 7->Refused, 9->Don't know|\n", "|INQ012 | Income from self employment|1->Yes, 2->No, 7->Refused, 9->Don't know|\n", "|INQ030 | Income from Social Security or RR |1->Yes, 2->No, 7->Refused, 9->Don't know|\n", "|INQ060 | Income from other disability pension |1->Yes, 2->No, 7->Refused, 9->Don't know|\n", "|INQ080 | Income from retirement/survivor pension |1->Yes, 2->No, 7->Refused, 9->Don't know|\n", "|INQ090 | Income from Supplemental Security Income |1->Yes, 2->No, 7->Refused, 9->Don't know|\n", "|INQ132 | Income from state/county cash assistance |1->Yes, 2->No, 7->Refused, 9->Don't know|\n", "|INQ140 | Income from interest/dividends or rental |1->Yes, 2->No, 7->Refused, 9->Don't know|\n", "|INQ150 | Income from other sources |1->Yes, 2->No, 7->Refused, 9->Don't know|\n", "|IND235 | Monthly family income |1-12->Increasing income brackets, 77->Refused, 99->Don't know|\n", "|INDFMMPI | Family monthly poverty level index |0-5->Higher value more affluent|\n", "|INDFMMPC | Family monthly poverty level category |1-3->Increasing INDFMMPI brackets, 7->Refused, 9->Don't know|\n", "|INQ244 | Family has savings more than $5000 |1->Yes, 2->No, 7->Refused, 9->Don't know|\n", "|IND247 | Total savings/cash assets for the family |1-6->Increasing savings brackets, 77->Refused, 99->Don't know|" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Answers given by some respondents to the income questionnaire:\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Respondent sequence numberIncome from wages/salariesIncome from self employmentIncome from Social Security or RRIncome from other disability pensionIncome from retirement/survivor pensionIncome from Supplemental Security IncomeIncome from state/county cash assistanceIncome from interest/dividends or rentalIncome from other sourcesMonthly family incomeFamily monthly poverty level indexFamily monthly poverty level categoryFamily has savings more than $5000Total savings/cash assets for the family
073557.02.02.01.02.02.02.02.02.02.04.00.861.09.0NaN
173558.01.01.01.02.02.02.02.01.02.05.00.921.01.0NaN
273559.02.02.01.02.01.02.02.01.02.010.04.373.0NaNNaN
373560.01.02.02.02.02.02.02.02.01.09.02.523.0NaNNaN
473561.02.02.01.02.02.02.02.02.02.011.05.003.0NaNNaN
\n", "
" ], "text/plain": [ " Respondent sequence number Income from wages/salaries \\\n", "0 73557.0 2.0 \n", "1 73558.0 1.0 \n", "2 73559.0 2.0 \n", "3 73560.0 1.0 \n", "4 73561.0 2.0 \n", "\n", " Income from self employment Income from Social Security or RR \\\n", "0 2.0 1.0 \n", "1 1.0 1.0 \n", "2 2.0 1.0 \n", "3 2.0 2.0 \n", "4 2.0 1.0 \n", "\n", " Income from other disability pension \\\n", "0 2.0 \n", "1 2.0 \n", "2 2.0 \n", "3 2.0 \n", "4 2.0 \n", "\n", " Income from retirement/survivor pension \\\n", "0 2.0 \n", "1 2.0 \n", "2 1.0 \n", "3 2.0 \n", "4 2.0 \n", "\n", " Income from Supplemental Security Income \\\n", "0 2.0 \n", "1 2.0 \n", "2 2.0 \n", "3 2.0 \n", "4 2.0 \n", "\n", " Income from state/county cash assistance \\\n", "0 2.0 \n", "1 2.0 \n", "2 2.0 \n", "3 2.0 \n", "4 2.0 \n", "\n", " Income from interest/dividends or rental Income from other sources \\\n", "0 2.0 2.0 \n", "1 1.0 2.0 \n", "2 1.0 2.0 \n", "3 2.0 1.0 \n", "4 2.0 2.0 \n", "\n", " Monthly family income Family monthly poverty level index \\\n", "0 4.0 0.86 \n", "1 5.0 0.92 \n", "2 10.0 4.37 \n", "3 9.0 2.52 \n", "4 11.0 5.00 \n", "\n", " Family monthly poverty level category Family has savings more than $5000 \\\n", "0 1.0 9.0 \n", "1 1.0 1.0 \n", "2 3.0 NaN \n", "3 3.0 NaN \n", "4 3.0 NaN \n", "\n", " Total savings/cash assets for the family \n", "0 NaN \n", "1 NaN \n", "2 NaN \n", "3 NaN \n", "4 NaN " ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# replace encoded column names by the associated question text. \n", "df_inc = nhanes.get_csv_file('INQ_H.csv')\n", "df_inc.columns[0]\n", "dict_inc = {\n", "'SEQN': 'Respondent sequence number', \n", "'INQ020': 'Income from wages/salaries',\n", "'INQ012': 'Income from self employment',\n", "'INQ030':'Income from Social Security or RR',\n", "'INQ060': 'Income from other disability pension', \n", "'INQ080': 'Income from retirement/survivor pension',\n", "'INQ090': 'Income from Supplemental Security Income',\n", "'INQ132': 'Income from state/county cash assistance', \n", "'INQ140': 'Income from interest/dividends or rental', \n", "'INQ150': 'Income from other sources',\n", "'IND235': 'Monthly family income',\n", "'INDFMMPI': 'Family monthly poverty level index', \n", "'INDFMMPC': 'Family monthly poverty level category',\n", "'INQ244': 'Family has savings more than $5000',\n", "'IND247': 'Total savings/cash assets for the family'\n", "}\n", "qlist = []\n", "for i in range(len(df_inc.columns)):\n", " qlist.append(dict_inc[df_inc.columns[i]])\n", "df_inc.columns = qlist\n", "print(\"Answers given by some respondents to the income questionnaire:\")\n", "df_inc.head(5)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of respondents to Income questionnaire: 10175\n", "Distribution of answers to 'monthly family income' and 'Family savings' questions:\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlwAAAE8CAYAAAAVAG93AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAhAElEQVR4nO3df7RdZX3n8feXRFC0kiBphiahYWqsxc6AmIF0aKeMWAg/lqEdi2BHItJJZxVbO7aV0LqGLpVZsdMllWmlzZho6ChIUYdUsJAi1OW0IAlQfgUkxWCS8iMawLa02uB3/thP4Hi9N/fcu89z7jkn79daZ929n72fvZ99z93nfO7ez947MhNJkiTVc9BMN0CSJGnUGbgkSZIqM3BJkiRVZuCSJEmqzMAlSZJUmYFLkiSpMgOXJFUSEdsj4r6IuCciNpeywyNiU0Q8Un7OLeUREVdExLaIuDciju9Yzsoy/yMRsXKmtkfS9MUg34friCOOyMWLF890M6QD0pYtW76RmfNmuh3DLCK2A0sz8xsdZb8L7MnMNRGxGpibmRdHxBnArwBnACcCH8nMEyPicGAzsBRIYAvwhsx8eqL1+tkpzYz9fW7O7ndjpmLx4sVs3rx5ppshHZAi4rGZbsOIWgGcXIY3ALcBF5fyq7L5L/j2iJgTEUeWeTdl5h6AiNgELAeunmgFfnZKM2N/n5ueUpSkehK4OSK2RMSqUjY/Mx8vw08A88vwAmBHR92dpWyi8u8REasiYnNEbN69e3cvt0FSDwz0ES5JGnI/mZm7IuIHgU0R8VDnxMzMiOhJv47MXAusBVi6dOng9hWRDlAe4ZKkSjJzV/n5FPA54ATgyXKqkPLzqTL7LmBRR/WFpWyicklDxMAlSRVExMsj4gf2DQOnAvcDG4F9VxquBK4vwxuB88vVisuAZ8upx5uAUyNibrmi8dRSJmmIeEpRkuqYD3wuIqD5rP1UZv55RNwJXBsRFwKPAeeU+W+kuUJxG/AccAFAZu6JiA8Ad5b53r+vA72k4WHgkqQKMvNR4Nhxyr8JnDJOeQIXTbCs9cD6XrdRUv94SlGSJKkyA5ckSVJlBi5JkqTKDFySJEmVGbgkSZIqG6qrFBevvmG/07evObNPLZGk0TPZZ2yv+ZmtA8mkR7giYn1EPBUR93eU/c+IeCgi7o2Iz0XEnI5pl0TEtoh4OCJO6yhfXsq2RcTqnm+JJEnSgOrmlOInaJ5M32kT8OOZ+W+BrwKXAETEMcC5wOtKnY9GxKyImAX8IXA6cAxwXplXkiRp5E0auDLzS8CeMWU3Z+beMno7zbO9AFYA12TmtzPzazR3TD6hvLZl5qOZ+R3gmjKvJEnSyOtFp/l3Al8owwuAHR3TdpayicolSZJGXqvAFRG/DewFPtmb5kBErIqIzRGxeffu3b1arCRJ0oyZduCKiHcAZwG/UJ4BBrALWNQx28JSNlH598nMtZm5NDOXzps3b7rNkyRJGhjTClwRsRx4L/DmzHyuY9JG4NyIOCQijgaWAF+hecr9kog4OiIOpulYv7Fd0yVJkobDpPfhioirgZOBIyJiJ3ApzVWJhwCbIgLg9sz8r5n5QERcCzxIc6rxosx8viznXcBNwCxgfWY+UGF7JEmSBs6kgSszzxuneN1+5r8MuGyc8huBG6fUOkmSpBHgo30kSZIqM3BJkiRVZuCSJEmqzMAlSZJUmYFLkiSpMgOXJElSZQYuSZKkygxckiRJlRm4JEmSKjNwSZIkVWbgkiRJqszAJUmSVJmBS5IkqTIDlyRJUmUGLkmSpMoMXJIkSZUZuCRJkiozcEmSJFVm4JIkSarMwCVJklSZgUuSJKkyA5ckSVJlBi5JkqTKDFySJEmVGbgkSZIqM3BJkiRVZuCSJEmqzMAlSZJUmYFLkiSpMgOXJElSZQYuSaooImZFxN0R8fkyfnRE3BER2yLi0xFxcCk/pIxvK9MXdyzjklL+cEScNkObIqkFA5ck1fVuYGvH+IeAyzPz1cDTwIWl/ELg6VJ+eZmPiDgGOBd4HbAc+GhEzOpT2yX1iIFLkiqJiIXAmcDHyngAbwSuK7NsAM4uwyvKOGX6KWX+FcA1mfntzPwasA04oS8bIKlnDFySVM/vA+8FvlvGXwU8k5l7y/hOYEEZXgDsACjTny3zv1A+Tp0XRMSqiNgcEZt3797d482Q1NbsyWaIiPXAWcBTmfnjpexw4NPAYmA7cE5mPl3+G/sIcAbwHPCOzLyr1FkJvK8s9oOZuYE+W7z6hv1O377mzD61RNKoi4h9n5tbIuLk2uvLzLXAWoClS5dm7fVJmppJAxfwCeAPgKs6ylYDt2TmmohYXcYvBk4HlpTXicCVwIkloF0KLAUS2BIRGzPz6V5tSD8Y2CRNwUnAmyPiDOClwCtp/iGdExGzy1GshcCuMv8uYBGwMyJmA4cB3+wo36ezjqQhMekpxcz8ErBnTHFnX4OxfRCuysbtNB8sRwKnAZsyc08JWZtoOn9K0kjKzEsyc2FmLqbp9P7FzPwF4FbgLWW2lcD1ZXhjGadM/2JmZik/t1zFeDTNP7Rf6dNmSOqRbo5wjWd+Zj5ehp8A5pfhifoadNUHAZp+CMAqgKOOOmqazZOkgXUxcE1EfBC4G1hXytcBfxIR22j+yT0XIDMfiIhrgQeBvcBFmfl8/5stqY3pBq4XZGZGRM/6C9gPQdKoyczbgNvK8KOMc5VhZv4z8PMT1L8MuKxeCyXVNt2rFJ8spwopP58q5RP1NbAPgiRJOmBNN3B19jUY2wfh/GgsA54tpx5vAk6NiLkRMRc4tZRJkiSNvG5uC3E1cDJwRETspLnacA1wbURcCDwGnFNmv5HmlhDbaG4LcQFAZu6JiA8Ad5b53p+ZYzvijzyvcpQk6cA0aeDKzPMmmHTKOPMmcNEEy1kPrJ9S6yRJkkaAd5qXJEmqzMAlSZJUmYFLkiSpMgOXJElSZQYuSZKkygxckiRJlRm4JEmSKjNwSZIkVWbgkiRJqszAJUmSVJmBS5IkqTIDlyRJUmUGLkmSpMoMXJIkSZUZuCRJkiozcEmSJFVm4JIkSarMwCVJklSZgUuSJKkyA5ckSVJlBi5JkqTKDFySJEmVGbgkSZIqmz3TDVD3Fq++Yb/Tt685s08tkSRJU+ERLkmSpMoMXJIkSZUZuCRJkiozcEmSJFVm4JIkSarMwCVJklSZgUuSJKkyA5ckSVJlBi5JkqTKWgWuiPhvEfFARNwfEVdHxEsj4uiIuCMitkXEpyPi4DLvIWV8W5m+uCdbIEmSNOCmHbgiYgHwq8DSzPxxYBZwLvAh4PLMfDXwNHBhqXIh8HQpv7zMJ0mSNPLanlKcDbwsImYDhwKPA28ErivTNwBnl+EVZZwy/ZSIiJbrlyRJGnjTDlyZuQv4PeDrNEHrWWAL8Exm7i2z7QQWlOEFwI5Sd2+Z/1VjlxsRqyJic0Rs3r1793SbJ0mSNDDanFKcS3PU6mjgh4CXA8vbNigz12bm0sxcOm/evLaLkyRJmnFtTim+CfhaZu7OzH8BPgucBMwppxgBFgK7yvAuYBFAmX4Y8M0W65ckSRoKsyefZUJfB5ZFxKHAPwGnAJuBW4G3ANcAK4Hry/wby/hfl+lfzMxssX5N0eLVN+x3+vY1Z/apJZIkHVja9OG6g6bz+13AfWVZa4GLgfdExDaaPlrrSpV1wKtK+XuA1S3aLUmSNDTaHOEiMy8FLh1T/Chwwjjz/jPw823WJ0nDIiJeCnwJOITms/a6zLw0Io6mOQPwKpoLjd6emd+JiEOAq4A30HS3eGtmbi/LuoTm1jrPA7+amTf1e3skteOd5iWpjm8Db8zMY4HjgOURsYwp3qswIo6hucfh62guTPpoRMzq54ZIaq/VES4dWOwDJnWv9FH9hzL6kvJKmnsVvq2UbwB+B7iS5qrv3ynl1wF/UO5VuAK4JjO/DXytdMs4gaY/rKQh4REuSaokImZFxD3AU8Am4G+Z+r0KXygfp07nuryHoTTADFySVElmPp+Zx9HcIucE4LUV1+U9DKUBZuCSpMoy8xmaW+b8BFO/V+EL5ePUkTQkDFySVEFEzIuIOWX4ZcDPAFt58V6FMP69CuF771W4ETg3Ig4pVzguAb7Sl42Q1DN2mpekOo4ENpQrCg8Crs3Mz0fEg8A1EfFB4G6+916Ff1I6xe+huTKRzHwgIq4FHgT2Ahdl5vN93hZJLRm4JKmCzLwXeP045VO+V2FmXgZc1us2SuofTylKkiRVZuCSJEmqzMAlSZJUmYFLkiSpMjvNq298NJAk6UDlES5JkqTKDFySJEmVGbgkSZIqM3BJkiRVZuCSJEmqzMAlSZJUmYFLkiSpMgOXJElSZQYuSZKkygxckiRJlRm4JEmSKjNwSZIkVWbgkiRJqszAJUmSVJmBS5IkqTIDlyRJUmUGLkmSpMoMXJIkSZUZuCRJkiozcEmSJFXWKnBFxJyIuC4iHoqIrRHxExFxeERsiohHys+5Zd6IiCsiYltE3BsRx/dmEyRJkgZb2yNcHwH+PDNfCxwLbAVWA7dk5hLgljIOcDqwpLxWAVe2XLckSdJQmHbgiojDgP8ArAPIzO9k5jPACmBDmW0DcHYZXgFclY3bgTkRceR01y9JkjQs2hzhOhrYDXw8Iu6OiI9FxMuB+Zn5eJnnCWB+GV4A7Oiov7OUSZIkjbQ2gWs2cDxwZWa+HvhHXjx9CEBmJpBTWWhErIqIzRGxeffu3S2aJ0mSNBjaBK6dwM7MvKOMX0cTwJ7cd6qw/HyqTN8FLOqov7CUfY/MXJuZSzNz6bx581o0T5IkaTBMO3Bl5hPAjoj40VJ0CvAgsBFYWcpWAteX4Y3A+eVqxWXAsx2nHiVJkkbW7Jb1fwX4ZEQcDDwKXEAT4q6NiAuBx4Bzyrw3AmcA24DnyrySJEkjr1Xgysx7gKXjTDplnHkTuKjN+iRJkoaRd5qXJEmqzMAlSZJUmYFLkiSpMgOXJElSZQYuSZKkygxckiRJlRm4JEmSKmt741OpbxavvmG/07evObNPLZEkaWo8wiVJklSZgUuSJKkyA5ckSVJlBi5JqiAiFkXErRHxYEQ8EBHvLuWHR8SmiHik/JxbyiMiroiIbRFxb0Qc37GslWX+RyJi5Uxtk6TpM3BJUh17gV/PzGOAZcBFEXEMsBq4JTOXALeUcYDTgSXltQq4EpqABlwKnAicAFy6L6RJGh4GLkmqIDMfz8y7yvDfA1uBBcAKYEOZbQNwdhleAVyVjduBORFxJHAasCkz92Tm08AmYHn/tkRSLxi4JKmyiFgMvB64A5ifmY+XSU8A88vwAmBHR7WdpWyi8rHrWBURmyNi8+7du3u7AZJaM3BJUkUR8QrgM8CvZea3OqdlZgLZi/Vk5trMXJqZS+fNm9eLRUrqIQOXJFUSES+hCVufzMzPluIny6lCys+nSvkuYFFH9YWlbKJySUPEwCVJFUREAOuArZn54Y5JG4F9VxquBK7vKD+/XK24DHi2nHq8CTg1IuaWzvKnljJJQ8RH++iA4aOB1GcnAW8H7ouIe0rZbwFrgGsj4kLgMeCcMu1G4AxgG/AccAFAZu6JiA8Ad5b53p+Ze/qyBZJ6xsAlSRVk5peBmGDyKePMn8BFEyxrPbC+d62T1G+eUpQkSarMwCVJklSZgUuSJKkyA5ckSVJldpqXuuRVjpKk6fIIlyRJUmUGLkmSpMoMXJIkSZUZuCRJkiozcEmSJFVm4JIkSarMwCVJklSZgUuSJKkyA5ckSVJlrQNXRMyKiLsj4vNl/OiIuCMitkXEpyPi4FJ+SBnfVqYvbrtuSZKkYdCLI1zvBrZ2jH8IuDwzXw08DVxYyi8Eni7ll5f5JEmSRl6rwBURC4EzgY+V8QDeCFxXZtkAnF2GV5RxyvRTyvySJEkjre3Dq38feC/wA2X8VcAzmbm3jO8EFpThBcAOgMzcGxHPlvm/0bnAiFgFrAI46qijWjZPGhw+/FqSDlzTPsIVEWcBT2Xmlh62h8xcm5lLM3PpvHnzerloSZKkGdHmCNdJwJsj4gzgpcArgY8AcyJidjnKtRDYVebfBSwCdkbEbOAw4Jst1i9JkjQUpn2EKzMvycyFmbkYOBf4Ymb+AnAr8JYy20rg+jK8sYxTpn8xM3O665ckSRoWNe7DdTHwnojYRtNHa10pXwe8qpS/B1hdYd2SJEkDp22neQAy8zbgtjL8KHDCOPP8M/DzvVifJEnSMPFO85IkSZUZuCRJkiozcEmSJFVm4JIkSarMwCVJklSZgUuSJKkyA5ckSVJlBi5JkqTKDFySJEmVGbgkSZIqM3BJkiRV1pNnKUqqb/HqG/Y7ffuaM/vUEknSVHmES5IkqTIDlyRJUmUGLkmSpMoMXJIkSZUZuCRJkiozcEmSJFVm4JIkSarMwCVJklSZgUuSJKkyA5ckSVJlBi5JqiAi1kfEUxFxf0fZ4RGxKSIeKT/nlvKIiCsiYltE3BsRx3fUWVnmfyQiVs7Etkhqz8AlSXV8Alg+pmw1cEtmLgFuKeMApwNLymsVcCU0AQ24FDgROAG4dF9IkzRcDFySVEFmfgnYM6Z4BbChDG8Azu4ovyobtwNzIuJI4DRgU2buycyngU18f4iTNAQMXJLUP/Mz8/Ey/AQwvwwvAHZ0zLezlE1U/n0iYlVEbI6Izbt37+5tqyW1ZuCSpBmQmQlkD5e3NjOXZubSefPm9WqxknrEwCVJ/fNkOVVI+flUKd8FLOqYb2Epm6hc0pAxcElS/2wE9l1puBK4vqP8/HK14jLg2XLq8Sbg1IiYWzrLn1rKJA2Z2TPdAEn9sXj1Dfudvn3NmX1qyYEhIq4GTgaOiIidNFcbrgGujYgLgceAc8rsNwJnANuA54ALADJzT0R8ALizzPf+zBzbEV/SEDBwSVIFmXneBJNOGWfeBC6aYDnrgfU9bJqkGeApRUmSpMoMXJIkSZVNO3BFxKKIuDUiHoyIByLi3aV8yo+ukCRJGmVtjnDtBX49M48BlgEXRcQxTPHRFZIkSaNu2oErMx/PzLvK8N8DW2nugDzVR1dIkiSNtJ704YqIxcDrgTuY+qMrxi7Lx1NIkqSR0jpwRcQrgM8Av5aZ3+qcNp1HV/h4CkmSNGpaBa6IeAlN2PpkZn62FE/10RWSJEkjrc1VigGsA7Zm5oc7Jk310RWSJEkjrc2d5k8C3g7cFxH3lLLfYoqPrpAkSRp10w5cmfllICaYPKVHV0iSJI0y7zQvSZJUmYFLkiSpMgOXJElSZQYuSZKkygxckiRJlRm4JEmSKjNwSZIkVWbgkiRJqszAJUmSVJmBS5IkqbI2z1KUJGloLF59Q1/Xt33NmX1dnwabR7gkSZIqM3BJkiRVZuCSJEmqzMAlSZJUmYFLkiSpMgOXJElSZQYuSZKkygxckiRJlRm4JEmSKjNwSZIkVWbgkiRJqszAJUmSVJmBS5IkqTIDlyRJUmUGLkmSpMoMXJIkSZUZuCRJkiozcEmSJFU2e6YbIEmS2lu8+oa+rm/7mjP7ur5h5xEuSZKkygxckiRJlXlKUZIkDbx+njKtcbq070e4ImJ5RDwcEdsiYnW/1y9Jw8jPTmm49TVwRcQs4A+B04FjgPMi4ph+tkGSho2fndLw6/cRrhOAbZn5aGZ+B7gGWNHnNkjSsPGzUxpy/Q5cC4AdHeM7S5kkaWJ+dkpDLjKzfyuLeAuwPDN/sYy/HTgxM9/VMc8qYFUZ/VHg4f0s8gjgGy2aZH3rW39iP5yZ81osXz1S4bOz19r+LQ46t2+49XP7Jvzc7PdViruARR3jC0vZCzJzLbC2m4VFxObMXDrdxljf+taffn31VU8/O3tt1P+W3L7hNijb1+9TincCSyLi6Ig4GDgX2NjnNkjSsPGzUxpyfT3ClZl7I+JdwE3ALGB9Zj7QzzZI0rDxs1Mafn2/8Wlm3gjc2KPFtT18bn3rW19Docefnb026n9Lbt9wG4jt62uneUmSpAORz1KUJEmqzMAlSZJUmYFLkiSpsr53mh8EEXE4QGbumUbd+bx4h+ddmfnkMNUvy5j29veifhuD8PuT5L407Eb5/RvUbTtgOs1HxFHA7wKnAM8AAbwS+CKwOjO3T1L/OOCPgMN48YaDC8uyfjkz7xrw+m23v1X9tgbg9/dammfXvbATAxszc+sUtuEwYPmYZdyUmc9MYRnT+iDpRfslaL8vDYtB/dJua5Tfv4HftswcqhfwWuBi4Iryuhj4sS7q/TXwVmBWR9ksmhsI3t5F/XtoHqUxtnwZ8DdDUL/t9req3+a9m+nfX2nnPcBq4D+X1+p9ZV22/3zgb4ErgfeV1x+VsvO7qH8ccDuwFfiL8nqolB1fu/2+fO17td0XB/3VZl8bhtcov3+Dvm1DdYQrIi4GzgOuoXl4KzTp9Vzgmsxcs5+6j2TmkqlO67L+tsx89RDXb7v93dSf9nvXxfqr/v4i4qvA6zLzX8aUHww8MNm2l3kfpvkgeGZM+Vzgjsx8zST17wF+KTPvGFO+DPjjzDy2Zvulfdrui4Ouzb42DEb5/Rv0bRu2PlwXMv4Xx4eBB4D9fWlviYiPAhuAHaVsEbASuLuLdX8hIm4ArhpT/3zgz4egftvtb1u/zXsHM/v7+y7wQ8BjY8qPLNO6EcB4/918t0ybzMvHfgEAZObtEfHySer2ov3SPm33xUHXZl8bBqP8/g30tg3bEa6HgNMy87Ex5T8M3JyZP7qfugfTfOl39mPZCfwZsC4zv93F+k9n/H4wXd39eSbrt93+HtSf9nvXMe+M/P4iYjnwB8AjvLgTHwW8GnhXZk66I0fESuC/AzePWcbPAB/IzE9MUv8K4EcY/4Pka5n5rprtlzq13RcHWZt9bViM+Ps3sNs2bIHLL44hNezvXUQcBJzA9+7Ed2bm81NYxlzgNL6/0/zTXdZvE7hbt186UAzyl7aG11AFLqjzxRERZ2Xm51vUX5WZ035W0wDUb7v9XdWv9aU/078/SQ33peE2yu/fIGzb0N34NDO/m5m3Z+Znyuv2HvyX/u9a1u+mD84g12+7/V3Vr/TewQz+/iJi2kG1YxmtPgQiYlWLuq3bL3Vouy8OtDb72pAY5fdvxrdt6I5wTSQiPp+ZZ00yzwlAZuadEXEMzT2RHurylMyvAp/LzB2TzTtB/ROBrZn5rYh4Gc1l+ccDDwL/IzOfneLyfpLmaNH9mXnzNNt0VWae3+W8B9NcUfh3mfkXEfE24N/TXDq9dmxn+Cm2Y9L3rsz3r4Gfo+lP8TzwVeBTmfmtLtfzWpqja3dk5j90lC+f7inNiDgyMx+fTt2OZbwhM7e0qP9LmfnH06zbuv068NTYl4ZBm31tULT9Lht0bb8nahqlwLXfL46IuBQ4nebKzE3AicCtNJ2Wb8rMyyZZ/rPAP9LcN+lq4E8zc/cU2vcAcGxm7i1HNJ4DrqO5keixmflzk9T/SmaeUIb/C3AR8DngVODPuritwsaxRcB/pLlxKZn55knqf5Lmd3cozU3kXgF8trSfzHzH/upPsuxJv/TLh8RZwJeAM2iujHwG+FmaG9rd1kX9i2gC4nHAuzPz+jLtrsw8frrtn2kRcUFmfnym26EDwyjvS5MZhX2t7XfZIGv7PVFdDsDNyvrxAu6juVHnocC3gFeW8pcB93ZR/26aU7CnAuuA3TSXma4EfqCL+ls7hu8aM+2ebtbfMXwnMK8Mvxy4r4v6dwH/BzgZ+Ony8/Ey/NNd1L+3/JwNPEm5ASpNcJv099er968MHwrcVoaP6vzdTFL/FWV4MbCZ5ouCyerThMv309y+4tny3t8OvGMK7T+M5tYXDwF7gG/SfGGtAea0/N18vYv3/n3Aj9R+n3yN/qvNvjTsr8n2tWF4tf0uG+RX2++J2q+hug9XRLwSuITmhplfyMxPdUz7aGb+8n6q782mv9BzEfG3WQ4vZuY/RUQ39yLKzPwuzWX9N0fES2iOmJ0H/B4wb5L693f8d/Q3EbE0MzdHxGuAbk7HHVSucjuI5sjk7tKof4yIvV3UXwq8G/ht4Dcz856I+KfM/Msu6u5b/8E0Ae9QmgCxBzgEeEmXyxhXRHwhM0/vYtbZNIeID6EJQWTm18t7MZmDspz6yMztEXEycF25LcVk5/Y/SXM08TTgHJrfwTXA+yLiNZn5W12s/1qao4knZ+YTABHxr2g+5K6l+fCbUETcO9EkYP4k654LzAFujYgnaP6r/XRm/l0X7ZbGarMvDbyW+9owaPtdNujafE9UNVSBC/g4zW0FPgO8MyL+E/C2bO4BtWySut+JiEMz8zngDfsKo3m+XTeB63s+SLLps7QR2BgRh3ZR/xeBj0TE+4BvAH8dETtobpHwi13UPwzYUtqR+07DRcQrxrZtPGUHuzwi/rT8fJKpvf/raI7OzKIJbX8aEY/S/N6vmaxyREx0miFoTktM5mPAnRFxB/BTwIfKcufRBL/JPBkRx2XmPQCZ+Q8RcRawHvg3k9RdnC/eJ+vDEXFnZn4gIi6g6YPXTeBanJkf6iwowetDEfHOLurPpwl8Y28hEcBfTVL36cz8DeA3IuKnaD5Y74qIrcDVOaJXJamaNvvSMGizrw2Dtt9lg6zt90RVQ9WHKyLuyczjOsZ/m+Y87ZuBTbmfvgMRcUiOc3POiDgCODIz75tk3a/JzK9Ou/EvLueVwNE0YWdntnwgatlB5mfm16ZY70zgpC6Pzuyr80MAmfl3ETEHeBPNIfavdFH3eeAvGT8cLsvMl3WxjNcBP0ZzocBD3ba71F1Ic5TziXGmnZSZ/28/df8KeG9mfjkiVtD0BTitTHs4u7tp6800z2TbsO89j+bhuO8AfiYz3zRJ/XXAxzPzy+NM+1Rmvm0/de/OzNePKZtF03/xrZl5wWTtl/Zpsy8Ngzb72jDo1XfZoGrzPVHbsAWurTSPh/luR9k7gN+k6VPwwzPVNu1fRNwP/GxmPjLOtB2ZuWgGmtWViDgW+N/AEpp+XO/MzK+W/5rOy8wruljGXJorU1cAP1iKn6T5z3JNdnnz0+mIiGsy89xay5ckTW7YAtfv0jwG5i/GlC8H/lf6EN6BFRFvoenc//A4087OzP/b/1Z1LyJ+hBcvNd5LDy81rn3l06hfBi5Jw2CoAtf+jMLlugeqQX/val9qHBFfz8yjWjZzf8sf2cvAJWlYjFLgqvqlpXoG/b2LiPuA4zLz+dJn7sbMPDkijgKuH9s/aoJl7O/Kp9dk5iE9bPLYdd9Nc6HIm4C30vR53EITvj6bmX9fa92SpMZQXaV4AFyuO7JG4L1re6nxTF75NOqXgUvSwBuqwMXoX647yob5vevFpcafp7mw456xEyLitt40c0KjfBm4JA2FoTqlOOqX646yYX/vBvlS48mM+mXgkjQMhipwSZIkDaODZroBkiRJo87AJUmSVJmBS5IkqTIDlyRJUmX/HxVHwhJkOxj/AAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "print(\"Number of respondents to Income questionnaire:\", df_inc.shape[0])\n", "print(\"Distribution of answers to \\'monthly family income\\' and \\'Family savings\\' questions:\")\n", "\n", "fig, axes = plt.subplots(1, 2, figsize=(10,5))\n", "fig.subplots_adjust(wspace=0.5)\n", "hist1 = df_inc['Monthly family income'].value_counts().plot(kind='bar', ax=axes[0])\n", "hist2 = df_inc['Family has savings more than $5000'].value_counts().plot(kind='bar', ax=axes[1])\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "#### Summarize NHANES Income Questionnaire dataset using Prototypes\n", "\n", "Consider a social scientist who would like to quickly obtain a summary report of this dataset in terms of types of people that span this dataset. Is it possible to summarize this dataset by looking at answers given by a few representative/prototypical respondents? \n", "\n", "We now show how the ProtodashExplainer can be used to obtain a few prototypical respondents (about 10 in this example) that span the diverse set of individuals answering the income questionnaire making it easy for the social scientist to summarize the dataset." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "# convert pandas dataframe to numpy\n", "data = df_inc.to_numpy()\n", "\n", "#sort the rows by sequence numbers in 1st column \n", "idx = np.argsort(data[:, 0]) \n", "data = data[idx, :]\n", "\n", "# replace nan's (missing values) with 0's\n", "original = data\n", "original[np.isnan(original)] = 0\n", "\n", "# delete 1st column (sequence numbers)\n", "original = original[:, 1:]\n", "\n", "# one hot encode all features as they are categorical\n", "onehot_encoder = OneHotEncoder(sparse=False)\n", "onehot_encoded = onehot_encoder.fit_transform(original)\n", "\n", "explainer = ProtodashExplainer()\n", "\n", "# call protodash explainer\n", "# S contains indices of the selected prototypes\n", "# W contains importance weights associated with the selected prototypes \n", "(W, S, _) = explainer.explain(onehot_encoded, onehot_encoded, m=10) " ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Respondent sequence numberIncome from wages/salariesIncome from self employmentIncome from Social Security or RRIncome from other disability pensionIncome from retirement/survivor pensionIncome from Supplemental Security IncomeIncome from state/county cash assistanceIncome from interest/dividends or rentalIncome from other sourcesMonthly family incomeFamily monthly poverty level indexFamily monthly poverty level categoryFamily has savings more than $5000Total savings/cash assets for the familyWeights of Prototypes
873565.01.02.02.02.02.02.02.02.02.012.05.003.0NaNNaN0.18
747581032.02.02.01.02.02.02.02.01.02.03.01.081.02.01.00.10
244976006.01.01.02.02.02.02.02.02.01.01.00.001.02.01.00.15
291276469.01.02.02.01.01.02.02.02.02.06.01.322.01.0NaN0.07
689580452.01.02.02.02.02.01.01.02.02.05.00.861.02.01.00.09
389977456.01.02.01.02.02.02.02.02.02.07.02.713.0NaNNaN0.12
147575032.01.02.02.02.02.02.02.01.02.04.01.652.02.03.00.06
69074247.02.02.02.02.02.02.02.02.02.08.03.053.0NaNNaN0.09
507778634.01.02.02.02.01.02.02.02.02.02.00.441.02.01.00.07
13273689.01.02.02.02.02.02.02.01.02.011.04.303.0NaNNaN0.07
\n", "
" ], "text/plain": [ " Respondent sequence number Income from wages/salaries \\\n", "8 73565.0 1.0 \n", "7475 81032.0 2.0 \n", "2449 76006.0 1.0 \n", "2912 76469.0 1.0 \n", "6895 80452.0 1.0 \n", "3899 77456.0 1.0 \n", "1475 75032.0 1.0 \n", "690 74247.0 2.0 \n", "5077 78634.0 1.0 \n", "132 73689.0 1.0 \n", "\n", " Income from self employment Income from Social Security or RR \\\n", "8 2.0 2.0 \n", "7475 2.0 1.0 \n", "2449 1.0 2.0 \n", "2912 2.0 2.0 \n", "6895 2.0 2.0 \n", "3899 2.0 1.0 \n", "1475 2.0 2.0 \n", "690 2.0 2.0 \n", "5077 2.0 2.0 \n", "132 2.0 2.0 \n", "\n", " Income from other disability pension \\\n", "8 2.0 \n", "7475 2.0 \n", "2449 2.0 \n", "2912 1.0 \n", "6895 2.0 \n", "3899 2.0 \n", "1475 2.0 \n", "690 2.0 \n", "5077 2.0 \n", "132 2.0 \n", "\n", " Income from retirement/survivor pension \\\n", "8 2.0 \n", "7475 2.0 \n", "2449 2.0 \n", "2912 1.0 \n", "6895 2.0 \n", "3899 2.0 \n", "1475 2.0 \n", "690 2.0 \n", "5077 1.0 \n", "132 2.0 \n", "\n", " Income from Supplemental Security Income \\\n", "8 2.0 \n", "7475 2.0 \n", "2449 2.0 \n", "2912 2.0 \n", "6895 1.0 \n", "3899 2.0 \n", "1475 2.0 \n", "690 2.0 \n", "5077 2.0 \n", "132 2.0 \n", "\n", " Income from state/county cash assistance \\\n", "8 2.0 \n", "7475 2.0 \n", "2449 2.0 \n", "2912 2.0 \n", "6895 1.0 \n", "3899 2.0 \n", "1475 2.0 \n", "690 2.0 \n", "5077 2.0 \n", "132 2.0 \n", "\n", " Income from interest/dividends or rental Income from other sources \\\n", "8 2.0 2.0 \n", "7475 1.0 2.0 \n", "2449 2.0 1.0 \n", "2912 2.0 2.0 \n", "6895 2.0 2.0 \n", "3899 2.0 2.0 \n", "1475 1.0 2.0 \n", "690 2.0 2.0 \n", "5077 2.0 2.0 \n", "132 1.0 2.0 \n", "\n", " Monthly family income Family monthly poverty level index \\\n", "8 12.0 5.00 \n", "7475 3.0 1.08 \n", "2449 1.0 0.00 \n", "2912 6.0 1.32 \n", "6895 5.0 0.86 \n", "3899 7.0 2.71 \n", "1475 4.0 1.65 \n", "690 8.0 3.05 \n", "5077 2.0 0.44 \n", "132 11.0 4.30 \n", "\n", " Family monthly poverty level category \\\n", "8 3.0 \n", "7475 1.0 \n", "2449 1.0 \n", "2912 2.0 \n", "6895 1.0 \n", "3899 3.0 \n", "1475 2.0 \n", "690 3.0 \n", "5077 1.0 \n", "132 3.0 \n", "\n", " Family has savings more than $5000 \\\n", "8 NaN \n", "7475 2.0 \n", "2449 2.0 \n", "2912 1.0 \n", "6895 2.0 \n", "3899 NaN \n", "1475 2.0 \n", "690 NaN \n", "5077 2.0 \n", "132 NaN \n", "\n", " Total savings/cash assets for the family Weights of Prototypes \n", "8 NaN 0.18 \n", "7475 1.0 0.10 \n", "2449 1.0 0.15 \n", "2912 NaN 0.07 \n", "6895 1.0 0.09 \n", "3899 NaN 0.12 \n", "1475 3.0 0.06 \n", "690 NaN 0.09 \n", "5077 1.0 0.07 \n", "132 NaN 0.07 " ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Display the prototypes along with their computed weights\n", "inc_prototypes = df_inc.iloc[S, :].copy()\n", "# Compute normalized importance weights for prototypes\n", "inc_prototypes[\"Weights of Prototypes\"] = np.around(W/np.sum(W), 2) \n", "inc_prototypes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Explanation:\n", "The 10 people shown above (i.e. 5 prototypes) are representative of the income questionnaire according to Protodash. Firstly, in the distribution plot for family finance related questions we saw that there roughly were 5 times as many people not having savings in excess of $5000 compared with others. Our prototypes also have a similar spread which is reassuring. Also for monthly family income we get a more even spread over the more commonly occuring categories. This is kind of a spot check to see if our prototypes actually match the distribution of values in the dataset.\n", "\n", "Looking at the other questions in the questionnaire and the corresponding answers given by the prototypical people above the social scientist realizes that most people are employeed (3rd question) and work for an organization earning through salary/wages (1st two questions). Most of them are also young (5th question) and fit to work (4th question). However, they don't seem to have much savings (last question). These insights that the social scientist has acquired from studying the prototypes could be conveyed also to the appropriate government authorities that affect future public policy decisions." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Summarize Gaussian (simulated) data using prototypes" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(300, 100) (4000, 100)\n" ] } ], "source": [ "# generate normalized gaussian data X, Y with 100 features and 300 & 4000 observations respectively\n", "(X, Y) = get_Gaussian_Data(100, 300, 4000)\n", "print(X.shape, Y.shape)" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "(W, S, setValues) = explainer.explain(X, Y, m=5, kernelType='Gaussian', sigma=2)" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[3940 2539 2168 2189 1170] [0.20611992 0.24524021 0.19131892 0.17175183 0.16123833]\n" ] } ], "source": [ "print(S, W)" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "#Print prototypes\n", "#print(Y[S, :]) " ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.15" } }, "nbformat": 4, "nbformat_minor": 2 }