{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## Preface\n", "([article source](https://towardsdatascience.com/feature-engineering-examples-binning-categorical-features-9f8d582455da)) **How to use NumPy or Pandas to quickly bin categorical features**\n", "\n", "**Working with categorical data for machine learning** (ML) **purposes can sometimes present tricky issues**. Ultimately these features need to be numerically encoded in some way so that an ML algorithm can actually work with them.\n", "\n", "You’ll also want to consider additional methods for getting your categorical features ready for modeling. For example, your model performance may benefit from **binning categorical features**. This essentially means lumping multiple categories together into a single category. By applying domain knowledge, you may be able to engineer new categories and features that better represent the structure of your data.\n", "\n", "**In this post, we’ll briefly cover why binning categorical features can be beneficial. Then we’ll walk through three different methods for binning categorical features with specific examples using NumPy and Pandas.**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Agenda\n", "* [**Why Bin Categories?**](#sect1)\n", "* [**Using np.where() to Bin Categories**](#sect2)\n", "* [**Mapping Categories into New Groups with map()**](#sect3)\n", "* [**Applying a Custom Function with apply()**](#sect4)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## Why Bin Categories?\n", "With categorical features, you may encounter problems with rare labels, categories/groups that are extremely uncommon within your dataset. This issue is often related to features having **high cardinality — in other words, many different categories.**\n", "\n", "Having too many categories, and especially rare categories, leads to a noisy dataset. It can be difficult for an ML algorithm to cut through this noise and learn from the more meaningful signals in the data.\n", "\n", "**High cardinality can also exacerbate the [curse of dimensionality](https://towardsdatascience.com/the-curse-of-dimensionality-minus-the-curse-of-jargon-520da109fc87) if you choose to [one hot encode](https://towardsdatascience.com/categorical-encoding-using-label-encoding-and-one-hot-encoder-911ef77fb5bd) your categorical features.** If the original variable has 50 different categories, you’re basically adding 49 columns to your dataset.\n", "\n", "**Having too many categories can also lead to issues when training and testing your model.** It’s completely possible that a category will show up in the test set, but not in the training set. Your model would have no idea how to handle that category because it has never “seen” it before.\n", "\n", "**One way to address these problems is by [engineering new features](https://maxsteele731.medium.com/what-is-feature-engineering-bfd25b2b26b2) that have fewer categories. This can be accomplished through binning** (grouping) **multiple categories into a single category.**\n", "\n", "In the following examples, we’ll be exploring and engineering features from a dataset with information about voter demographics and participation. I’ve selected 3 categorical variables to work with:\n", "* **party_cd:** a registered voter’s political party affiliation\n", "* **voting_method:** how a registered voter cast their ballot in the election\n", "* **birth_state:** the U.S. state or territory where a registered voter was born" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np\n", "import random\n", "import matplotlib.pyplot as plt\n", "from collections import OrderedDict" ] }, { "cell_type": "code", "execution_count": 61, "metadata": {}, "outputs": [], "source": [ "def gen_fake_df(k=20000):\n", " datas = []\n", " # https://en.wikipedia.org/wiki/List_of_states_and_territories_of_the_United_States_by_population\n", " # https://www.ssa.gov/international/coc-docs/states.html\n", " us_states_weights = OrderedDict({\n", " 'AL': 4921532, \n", " 'AK': 731158, \n", " 'AS': 19437, \n", " 'AZ': 7421401, \n", " 'AR': 3030522, \n", " 'CA': 39368, \n", " 'CO': 5807719,\n", " 'CT': 3557006,\n", " 'DE': 986809,\n", " 'DC': 712816,\n", " 'FL': 21733312,\n", " 'GA': 10710017,\n", " 'GU': 68485,\n", " 'HI': 1407006,\n", " 'ID': 1826913,\n", " 'IL': 12587530,\n", " 'IN': 6754953,\n", " 'IA': 3163561,\n", " 'KS': 2913805,\n", " 'KY': 4477251,\n", " 'LA': 4645318,\n", " 'ME': 1350141,\n", " 'MD': 6055802,\n", " 'MA': 6893574,\n", " 'MI': 9966555,\n", " 'MN': 5657342,\n", " 'MS': 2966786,\n", " 'MO': 6151548,\n", " 'MT': 1080577,\n", " 'NE': 1937552,\n", " 'NV': 3138259,\n", " 'NH': 1366275,\n", " 'NJ': 8882371,\n", " 'NM': 2106319,\n", " 'NY': 10600823*5,\n", " 'NC': 10600823*9,\n", " 'Missing': 10600823*7,\n", " 'ND': 765309,\n", " 'NP': 51433,\n", " 'OH': 11693217,\n", " 'OC': 10600823*2,\n", " 'OK': 3980783,\n", " 'OR': 4241507,\n", " 'PA': 12783254,\n", " 'PR': 189068,\n", " 'SC': 5218040,\n", " 'TX': 29360759,\n", " 'UT': 3249879,\n", " 'VA': 8590563,\n", " 'WA': 7693612,\n", " })\n", " vote_method_weights = OrderedDict({\n", " 'ABSENTEE ONESTOP': 8300, \n", " 'NO VOTE': 3900,\n", " 'IN PERSON': 2000,\n", " 'ABSENTEE BY MAIL': 1900,\n", " 'ABSENTEE CURBSIDE': 500,\n", " 'PROVISIONAL': 60,\n", " 'TRANSFER': 40,\n", " 'CURBSIDE': 30,\n", " })\n", " for v, p, b in zip(\n", " random.choices(list(vote_method_weights.keys()), weights=tuple(vote_method_weights.values()), k=k),\n", " random.choices(['REP', 'UNA', 'DEM', 'LIB', 'CST', 'GRE'], weights=(6900, 5700, 4000, 100, 100, 90), k=k),\n", " random.choices(list(us_states_weights.keys()), weights=tuple(list(us_states_weights.values())), k=k),\n", " ):\n", " datas.append((v, p, b))\n", " df = pd.DataFrame(datas, columns = [\"voting_method\", \"party_cd\", \"birth_state\"])\n", " return df" ] }, { "cell_type": "code", "execution_count": 55, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
voting_methodparty_cdbirth_state
0ABSENTEE ONESTOPLIBMissing
1ABSENTEE BY MAILDEMOC
2ABSENTEE ONESTOPREPWA
3ABSENTEE BY MAILUNAIL
4NO VOTEREPMissing
\n", "
" ], "text/plain": [ " voting_method party_cd birth_state\n", "0 ABSENTEE ONESTOP LIB Missing\n", "1 ABSENTEE BY MAIL DEM OC\n", "2 ABSENTEE ONESTOP REP WA\n", "3 ABSENTEE BY MAIL UNA IL\n", "4 NO VOTE REP Missing" ] }, "execution_count": 55, "metadata": {}, "output_type": "execute_result" } ], "source": [ "fake_df = gen_fake_df()\n", "fake_df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## Using np.where() to Bin Categories\n", "First, let’s check out why I chose `party_cd`. The image below shows how many individual voters belong to each political party." ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAWZ0lEQVR4nO3df5BdZ33f8fcHK8aGFCTjRRjJg1xQyBgKxtHYTglJsECWDUVOBxy5HaxQddTpiCakQGMy7YhfTsyU1sFNMKPBCjIJGNfBtQJujCrDhHSC8RobgQyM1r+wNP6xWLJTMD8i8+0f91m4LLveu9LuXdnn/ZrZued8z3PueZ65q889eu65e1JVSJK64WkL3QFJ0vAY+pLUIYa+JHWIoS9JHWLoS1KHLFroDjyRE088sVasWLHQ3ZCkJ5Vbb731O1U1MtW2ozr0V6xYwejo6EJ3Q5KeVJLcO902p3ckqUMMfUnqEENfkjrE0JekDjH0JalDDH1J6hBDX5I6xNCXpA4x9CWpQ47qb+QerhUXf3aox7vn0tcN9XiSdLg805ekDhko9JP8fpI9Sb6e5JNJjktySpKbk4wl+VSSY1vbp7f1sbZ9Rd/zvKvVv5XknHkakyRpGjOGfpJlwO8Cq6rqpcAxwHrgA8BlVfUi4CCwse2yETjY6pe1diQ5te33EmAt8OEkx8ztcCRJT2TQ6Z1FwPFJFgHPAO4Hzgaubdu3A+e35XVtnbZ9dZK0+tVV9cOquhsYA8444hFIkgY2Y+hX1X7gg8C36YX9o8CtwCNVdag12wcsa8vLgPvavoda++f016fY5yeSbEoymmR0fHz8cMYkSZrGINM7S+idpZ8CPB94Jr3pmXlRVVuralVVrRoZmfIeAJKkwzTI9M5rgLuraryq/hH4NPBKYHGb7gFYDuxvy/uBkwHa9mcDD/fXp9hHkjQEg4T+t4Gzkjyjzc2vBu4APg+8sbXZAFzflne0ddr2m6qqWn19u7rnFGAl8OW5GYYkaRAzfjmrqm5Oci3wFeAQcBuwFfgscHWS97falW2XK4GPJxkDDtC7Yoeq2pPkGnpvGIeAzVX1+ByPR5L0BAb6Rm5VbQG2TCrfxRRX31TVD4A3TfM8lwCXzLKPkqQ54jdyJalDDH1J6hBDX5I6xNCXpA4x9CWpQwx9SeoQQ1+SOsTQl6QOMfQlqUMMfUnqEENfkjrE0JekDjH0JalDDH1J6hBDX5I6xNCXpA4Z5MboL05ye9/PPyR5W5ITkuxMsrc9Lmntk+TyJGNJdic5ve+5NrT2e5NsmP6okqT5MGPoV9W3quq0qjoN+BXgMeA64GJgV1WtBHa1dYBz6d3/diWwCbgCIMkJ9O6+dSa9O25tmXijkCQNx2ynd1YDd1bVvcA6YHurbwfOb8vrgKuq50vA4iQnAecAO6vqQFUdBHYCa490AJKkwc029NcDn2zLS6vq/rb8ALC0LS8D7uvbZ1+rTVeXJA3JwKGf5FjgDcD/nLytqgqouehQkk1JRpOMjo+Pz8VTSpKa2Zzpnwt8paoebOsPtmkb2uNDrb4fOLlvv+WtNl39Z1TV1qpaVVWrRkZGZtE9SdJMZhP6F/LTqR2AHcDEFTgbgOv76he1q3jOAh5t00A3AmuSLGkf4K5pNUnSkCwapFGSZwKvBf5dX/lS4JokG4F7gQta/QbgPGCM3pU+bwGoqgNJ3gfc0tq9t6oOHPEIJEkDGyj0q+p7wHMm1R6mdzXP5LYFbJ7mebYB22bfTUnSXPAbuZLUIQOd6evosuLizw71ePdc+rqhHk/S/PFMX5I6xNCXpA4x9CWpQwx9SeoQQ1+SOsTQl6QOMfQlqUMMfUnqEENfkjrE0JekDjH0JalDDH1J6hBDX5I6xNCXpA4x9CWpQwYK/SSLk1yb5JtJvpHkV5OckGRnkr3tcUlrmySXJxlLsjvJ6X3Ps6G135tkw/RHlCTNh0HP9D8E/E1V/TLwcuAbwMXArqpaCexq6wDnAivbzybgCoAkJwBbgDOBM4AtE28UkqThmDH0kzwb+HXgSoCq+lFVPQKsA7a3ZtuB89vyOuCq6vkSsDjJScA5wM6qOlBVB4GdwNo5HIskaQaDnOmfAowDf57ktiQfTfJMYGlV3d/aPAAsbcvLgPv69t/XatPVf0aSTUlGk4yOj4/PbjSSpCc0SOgvAk4HrqiqVwDf46dTOQBUVQE1Fx2qqq1VtaqqVo2MjMzFU0qSmkFCfx+wr6pubuvX0nsTeLBN29AeH2rb9wMn9+2/vNWmq0uShmTG0K+qB4D7kry4lVYDdwA7gIkrcDYA17flHcBF7Sqes4BH2zTQjcCaJEvaB7hrWk2SNCSLBmz3H4C/THIscBfwFnpvGNck2QjcC1zQ2t4AnAeMAY+1tlTVgSTvA25p7d5bVQfmZBSSpIEMFPpVdTuwaopNq6doW8DmaZ5nG7BtFv2TJM0hv5ErSR1i6EtShxj6ktQhhr4kdYihL0kdYuhLUocY+pLUIYa+JHWIoS9JHWLoS1KHGPqS1CGGviR1iKEvSR1i6EtShxj6ktQhhr4kdchAoZ/kniRfS3J7ktFWOyHJziR72+OSVk+Sy5OMJdmd5PS+59nQ2u9NsmG640mS5sdszvRfXVWnVdXEHbQuBnZV1UpgV1sHOBdY2X42AVdA700C2AKcCZwBbJl4o5AkDceRTO+sA7a35e3A+X31q6rnS8DiJCcB5wA7q+pAVR0EdgJrj+D4kqRZGjT0C/hckluTbGq1pVV1f1t+AFjalpcB9/Xtu6/Vpqv/jCSbkowmGR0fHx+we5KkQQx0Y3Tg16pqf5LnAjuTfLN/Y1VVkpqLDlXVVmArwKpVq+bkOSVJPQOd6VfV/vb4EHAdvTn5B9u0De3xodZ8P3By3+7LW226uiRpSGYM/STPTPJPJpaBNcDXgR3AxBU4G4Dr2/IO4KJ2Fc9ZwKNtGuhGYE2SJe0D3DWtJkkakkGmd5YC1yWZaP+JqvqbJLcA1yTZCNwLXNDa3wCcB4wBjwFvAaiqA0neB9zS2r23qg7M2UgkSTOaMfSr6i7g5VPUHwZWT1EvYPM0z7UN2Db7bkqS5oLfyJWkDhn06h1paFZc/NmhHu+eS1831ONJC8kzfUnqEENfkjrE0JekDjH0JalDDH1J6hBDX5I6xNCXpA4x9CWpQwx9SeoQQ1+SOsTQl6QOMfQlqUMMfUnqEENfkjpk4NBPckyS25J8pq2fkuTmJGNJPpXk2FZ/elsfa9tX9D3Hu1r9W0nOmfPRSJKe0GzO9H8P+Ebf+geAy6rqRcBBYGOrbwQOtvplrR1JTgXWAy8B1gIfTnLMkXVfkjQbA4V+kuXA64CPtvUAZwPXtibbgfPb8rq2Ttu+urVfB1xdVT+sqrvp3UP3jDkYgyRpQIOe6f8J8J+AH7f15wCPVNWhtr4PWNaWlwH3AbTtj7b2P6lPsc9PJNmUZDTJ6Pj4+OAjkSTNaMbQT/J64KGqunUI/aGqtlbVqqpaNTIyMoxDSlJnDHKP3FcCb0hyHnAc8CzgQ8DiJIva2fxyYH9rvx84GdiXZBHwbODhvvqE/n0kSUMw45l+Vb2rqpZX1Qp6H8TeVFX/Gvg88MbWbANwfVve0dZp22+qqmr19e3qnlOAlcCX52wkkqQZDXKmP50/AK5O8n7gNuDKVr8S+HiSMeAAvTcKqmpPkmuAO4BDwOaqevwIji9JmqVZhX5VfQH4Qlu+iymuvqmqHwBvmmb/S4BLZttJSdLc8Bu5ktQhhr4kdYihL0kdYuhLUocY+pLUIYa+JHWIoS9JHWLoS1KHGPqS1CGGviR1iKEvSR1i6EtShxj6ktQhhr4kdYihL0kdYuhLUocMcmP045J8OclXk+xJ8p5WPyXJzUnGknwqybGt/vS2Pta2r+h7rne1+reSnDNvo5IkTWmQM/0fAmdX1cuB04C1Sc4CPgBcVlUvAg4CG1v7jcDBVr+stSPJqfRunfgSYC3w4STHzOFYJEkzGOTG6FVV322rv9B+CjgbuLbVtwPnt+V1bZ22fXWStPrVVfXDqrobGGOK2y1KkubPQHP6SY5JcjvwELATuBN4pKoOtSb7gGVteRlwH0Db/ijwnP76FPv0H2tTktEko+Pj47MekCRpegOFflU9XlWnAcvpnZ3/8nx1qKq2VtWqqlo1MjIyX4eRpE6a1dU7VfUI8HngV4HFSRa1TcuB/W15P3AyQNv+bODh/voU+0iShmCQq3dGkixuy8cDrwW+QS/839iabQCub8s72jpt+01VVa2+vl3dcwqwEvjyHI1DkjSARTM34SRge7vS5mnANVX1mSR3AFcneT9wG3Bla38l8PEkY8ABelfsUFV7klwD3AEcAjZX1eNzOxxJ0hOZMfSrajfwiinqdzHF1TdV9QPgTdM81yXAJbPvpiRpLviNXEnqEENfkjrE0JekDjH0JalDDH1J6hBDX5I6xNCXpA4x9CWpQwx9SeoQQ1+SOsTQl6QOMfQlqUMMfUnqEENfkjrE0JekDjH0JalDBrld4slJPp/kjiR7kvxeq5+QZGeSve1xSasnyeVJxpLsTnJ633NtaO33Jtkw3TElSfNjkDP9Q8Dbq+pU4Cxgc5JTgYuBXVW1EtjV1gHOpXf/25XAJuAK6L1JAFuAM+ndcWvLxBuFJGk4Zgz9qrq/qr7Slv8fvZuiLwPWAdtbs+3A+W15HXBV9XwJWJzkJOAcYGdVHaiqg8BOYO1cDkaS9MRmNaefZAW9++XeDCytqvvbpgeApW15GXBf3277Wm26+uRjbEoymmR0fHx8Nt2TJM1g4NBP8ovAXwFvq6p/6N9WVQXUXHSoqrZW1aqqWjUyMjIXTylJagYK/SS/QC/w/7KqPt3KD7ZpG9rjQ62+Hzi5b/flrTZdXZI0JINcvRPgSuAbVfXf+zbtACauwNkAXN9Xv6hdxXMW8GibBroRWJNkSfsAd02rSZKGZNEAbV4JvBn4WpLbW+0PgUuBa5JsBO4FLmjbbgDOA8aAx4C3AFTVgSTvA25p7d5bVQfmYhCSpMHMGPpV9XdAptm8eor2BWye5rm2Adtm00FJ0tzxG7mS1CGGviR1iKEvSR1i6EtShxj6ktQhhr4kdYihL0kdYuhLUocY+pLUIYa+JHWIoS9JHWLoS1KHGPqS1CGGviR1iKEvSR1i6EtShwxyu8RtSR5K8vW+2glJdibZ2x6XtHqSXJ5kLMnuJKf37bOhtd+bZMNUx5Ikza9BzvQ/BqydVLsY2FVVK4FdbR3gXGBl+9kEXAG9NwlgC3AmcAawZeKNQpI0PDOGflX9LTD5XrbrgO1teTtwfl/9qur5ErA4yUnAOcDOqjpQVQeBnfz8G4kkaZ4d7pz+0qq6vy0/ACxty8uA+/ra7Wu16eqSpCE64g9y243Qaw76AkCSTUlGk4yOj4/P1dNKkjj80H+wTdvQHh9q9f3AyX3tlrfadPWfU1Vbq2pVVa0aGRk5zO5JkqZyuKG/A5i4AmcDcH1f/aJ2Fc9ZwKNtGuhGYE2SJe0D3DWtJkkaokUzNUjySeA3gROT7KN3Fc6lwDVJNgL3Ahe05jcA5wFjwGPAWwCq6kCS9wG3tHbvrarJHw5LkubZjKFfVRdOs2n1FG0L2DzN82wDts2qd5KkOeU3ciWpQwx9SeoQQ1+SOsTQl6QOMfQlqUMMfUnqEENfkjrE0JekDjH0JalDDH1J6hBDX5I6xNCXpA4x9CWpQwx9SeoQQ1+SOsTQl6QOMfQlqUNmvHPWXEuyFvgQcAzw0aq6dNh9kBbSios/O9Tj3XPp64Z2rKfy2OCpMb6hnuknOQb4M+Bc4FTgwiSnDrMPktRlw57eOQMYq6q7qupHwNXAuiH3QZI6K717mQ/pYMkbgbVV9W/b+puBM6vqrX1tNgGb2uqLgW8NrYNwIvCdIR5v2Bzfk9tTeXxP5bHB8Mf3gqoamWrD0Of0Z1JVW4GtC3HsJKNVtWohjj0Mju/J7ak8vqfy2ODoGt+wp3f2Ayf3rS9vNUnSEAw79G8BViY5JcmxwHpgx5D7IEmdNdTpnao6lOStwI30LtncVlV7htmHGSzItNIQOb4nt6fy+J7KY4OjaHxD/SBXkrSw/EauJHWIoS9JHdKp0E/yeJLbk3w9yV8nWdzqK5J8v22b+LmobbsnydeS7E7yuSTPW9BBTKH1/+uTau9O8o4kH0uyP8nTW/3EJPdMavu2JD9I8uwhdntgfa/bniRfTfL2JE9r234zyaOTXrvXtG2V5C/6nmdRkvEkn1moscwkyXenqL07yTva8seS3N3G+c0kW4bfy9lL8rwkVye5M8mtSW5I8ktJLm//Hr+W5JZ2kcfNbXzfbq/XxOu6YqHHMZUkS5N8IsldbWx/n+S3Jv1ufjPJB/v2+Z1JY7t9WH+d4Ki7Tn+efb+qTgNIsh3YDFzStt05sW0Kr66q7yT5I+APgd+d747OsceBfwNcMc32C+ldWfUvgT8fVqdmof91ey7wCeBZwETgfbGqXj/Fft8DXprk+Kr6PvBanhqXCL+zqq5NchxwR5Krquruhe7UdJIEuA7YXlXrW+3lwG8DzwdeVlU/TrIc+F5Vndna/A6wqv/Lm0ebNrb/RW9s/6rVXgC8AThI+91McjxwW5Lrqur/tt0/tRBj69SZ/iR/Dyyb5T5/C7xoHvoy3/4E+P0kP/cmn+SFwC8C/5le+B/Vquohet/Yfmv7BzeTG4CJv1p1IfDJ+erbAjiuPX5vQXsxs1cD/1hVH5koVNVX6fX7/qr6cavtq6qDC9THw3U28KNJY7u3qv5Hf6N20nE7s8+cOdfJ0E/vD7+t5me/I/DCSf/VetUUu74e+NpQOjm3vg38HfDmKbatp/c3kL4IvDjJ0mF27HBU1V30Lvl9biu9atJr98K+5lcD69tZ8cuAm4fc3fnwX5PcDuwDrm5vhEezlwK3TlG/BvgX7TX7b0leMeR+zYWXAF+ZqVGSJcBKeieOE3570u/t8fPVyX5dC/3j2z+WB4ClwM6+bXdW1Wl9P1/s2/b5tt+zgD8eWm8HN911t/31Pwbeyc+/5hfSC44fA38FvGnuuzfvvjjptbtzYkNV7QZW0BvnDQvVwTn2zjbd9TxgdZJ/vsD9OSxVtY/e39d6F/BjYFeS1QvbqyOT5M/a5063tNKrknyV3rTijVX1QF/zT036vf3+MPrYtdCfmBt+ARB6c/qDeHV7US6qqkfmq3NH4GFgyaTaCfT9gaeq2kvvv5cXTNSS/DN6Zx8724e763kSTPEk+af0PqcY9Ax3B/BBnlpTO1TVd4EvAL+2wF2ZyR7gV6baUFU/rKr/XVXvBP4IOH+YHZsDe4DTJ1aqajO9WYSJP3b2xap6Ob3/EWxMctrQezhJ10IfgKp6jN6HsW+fap77yab9478/ydkASU4A1tKb0ul3CfCOvvULgXdX1Yr283zg+e2DqKNSkhHgI8Cf1uDfLNwGvKeqnoxTc9Nqv7tnAnfO1HaB3QQ8Pb2/oAtAkpcl+Y0kz2/rT6M3/XbvAvXxcN0EHJfk3/fVnjG5Ufug/VLgD4bVsel0MvQBquo2YDc/PbOdPKf/ZLtC5yLgv7RpqJvohdzPhEH7kxf984/r6V1V0e+6Vj+aHN9ekz3A/wE+B7ynb/vkOf039u/cPiC8fJgdPgLPSLKv7+c/TtFmYk5/N73PmD491B7OUntz/i3gNe2SzT30phtfBvx1epcb7wYOAX+6cD2dvTa284HfaJfSfhnYztTh/hHg1/suPZ08pz+UaTr/DIMkdUhnz/QlqYsMfUnqEENfkjrE0JekDjH0JalDDH1J6hBDX5I65P8DM7/Ph621/XwAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.rcParams['figure.figsize'] = [6, 4]\n", "fake_df['party_cd'].value_counts().plot(kind='bar', rot=0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are so few registered Libertarians, Constitutionalists, and members of the Green Party that we can barely see them on the graph. These would be good examples of rare labels. **For the purposes of this post, we’ll define rare labels as those that make up less than 5% of observations. This is a common threshold for defining rare labels, but ultimately that’s up to your discretion.**\n", "\n", "Let’s look at a breakdown of the actual numbers:" ] }, { "cell_type": "code", "execution_count": 46, "metadata": {}, "outputs": [], "source": [ "def show_column_dist(df, column_name, top_n=-1):\n", " # https://stackoverflow.com/questions/50558458/pandas-get-frequency-of-item-occurrences-in-a-column-as-percentage\n", " a, fake_df_row_count = fake_df[column_name].value_counts(), fake_df.shape[0]\n", " dist_df = pd.DataFrame(a.tolist(), columns = [column_name], index=a.index)\n", " dist_df['%'] = dist_df.apply(lambda e: 100*e[column_name]/fake_df_row_count, axis=1)\n", " if top_n > 0:\n", " return dist_df.head(n=15)\n", " else:\n", " return dist_df" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "REP 8223\n", "UNA 6733\n", "DEM 4716\n", "LIB 116\n", "CST 114\n", "GRE 98\n", "Name: party_cd, dtype: int64" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "fake_df['party_cd'].value_counts()" ] }, { "cell_type": "code", "execution_count": 44, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
party_cd%
REP822341.115
UNA673333.665
DEM471623.580
LIB1160.580
CST1140.570
GRE980.490
\n", "
" ], "text/plain": [ " party_cd %\n", "REP 8223 41.115\n", "UNA 6733 33.665\n", "DEM 4716 23.580\n", "LIB 116 0.580\n", "CST 114 0.570\n", "GRE 98 0.490" ] }, "execution_count": 44, "metadata": {}, "output_type": "execute_result" } ], "source": [ "show_column_dist(fake_df, 'party_cd')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Those three categories each make up far less than 5% of the population. Even if we lumped them all together into a single category, that new category would still represent less than 1% of voters.\n", "\n", "`REP` and `DEM` represent the two major political parties, whereas `UNA` represents voters that registered as unaffiliated with a political party. So here, it could make sense to lump in our three rare labels into that unaffiliated group so that we have three categories: one for each of the two major parties, and a third representing individuals that chose not to align with either major party.\n", "\n", "This can be accomplished very easily with [**np**.where()](https://numpy.org/doc/stable/reference/generated/numpy.where.html) which takes 3 arguments:\n", "1. a condition\n", "2. what to return if the condition is met\n", "3. what to return if the condition is not met\n", "\n", "The following code creates a new feature, `party_grp`, from the original `party_cd` variable using [**np**.where()](https://numpy.org/doc/stable/reference/generated/numpy.where.html):" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [], "source": [ "fake_df['party_grp'] = np.where(\n", " fake_df['party_cd'].isin(['REP', 'DEM']),\n", " fake_df['party_cd'].str.title(),\n", " 'Other'\n", ")" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
party_cdparty_grp
0DEMDem
1DEMDem
2DEMDem
3REPRep
4REPRep
.........
19995UNAOther
19996REPRep
19997DEMDem
19998REPRep
19999REPRep
\n", "

20000 rows × 2 columns

\n", "
" ], "text/plain": [ " party_cd party_grp\n", "0 DEM Dem\n", "1 DEM Dem\n", "2 DEM Dem\n", "3 REP Rep\n", "4 REP Rep\n", "... ... ...\n", "19995 UNA Other\n", "19996 REP Rep\n", "19997 DEM Dem\n", "19998 REP Rep\n", "19999 REP Rep\n", "\n", "[20000 rows x 2 columns]" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "fake_df[['party_cd', 'party_grp']]" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
party_grp%
Rep822341.115
Other706135.305
Dem471623.580
\n", "
" ], "text/plain": [ " party_grp %\n", "Rep 8223 41.115\n", "Other 7061 35.305\n", "Dem 4716 23.580" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "show_column_dist(fake_df, 'party_grp')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## Mapping Categories into New Groups with map()\n", "Next up, let’s take a look at the distribution of `voting_method`:" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAFBCAYAAAB3mNjUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAA06UlEQVR4nO3debxc8/3H8dcniz3EEkoWsaRaa0OE0tpCFkHUrpbYmha1F1FUS+1UaZWmll+0FFU/tBSp0vVnCaq1NqldLdHYWrTC5/fH5zPuMW64987MPTO57+fjMY8758y5M9+ZOXM+5/v9fr7fY+6OiIhIr7ILICIizUEBQUREAAUEERFJCggiIgIoIIiISFJAEBERAPqUXYCuWmqppXzo0KFlF0NEpGXce++9L7v7gLk93rIBYejQoUyfPr3sYoiItAwze+qjHv/YJiMzu8TMXjKzBwvrljCzaWY2I/8unuvNzM4zs5lm9hczW7vwPxNz+xlmNrGwfh0z+2v+z3lmZl17qyIiUouO9CH8DzC2at1k4DZ3HwbclssA44BheZsEXAARQIATgPWAkcAJlSCS23yp8H/VryUiIt3gYwOCu/8OmF21egIwNe9PBbYtrL/Mw51AfzNbFhgDTHP32e7+CjANGJuPLerud3rMoXFZ4blERKQbdTXLaBl3fz7vvwAsk/cHAs8Utns2133U+mfbWS8iIt2s5rTTPLPvlhnyzGySmU03s+mzZs3qjpcUEekxuhoQXszmHvLvS7n+OWBwYbtBue6j1g9qZ3273H2Ku49w9xEDBsw1c0pERLqgqwHhBqCSKTQRuL6wfs/MNlofeC2blm4BRpvZ4tmZPBq4JR973czWz+yiPQvPJSIi3ehjxyGY2U+BTYClzOxZIlvoNOBqM9sXeArYKTe/CdgSmAm8CewN4O6zzewk4J7c7kR3r3RUH0BkMi0I/CpvIiLSzaxVL5AzYsQI18A0EZGOM7N73X3E3B5v2ZHKHTV08o11f84nTxtf9+cUESmbJrcTERFAAUFERJICgoiIAAoIIiKSFBBERARQQBARkaSAICIigAKCiIgkBQQREQEUEEREJCkgiIgIoIAgIiJJAUFERAAFBBERSQoIIiICKCCIiEhSQBAREUABQUREkgKCiIgACggiIpIUEEREBFBAEBGRpIAgIiKAAoKIiCQFBBERARQQREQkKSCIiAiggCAiIkkBQUREAAUEERFJCggiIgIoIIiISFJAEBERoMaAYGaHmdlDZvagmf3UzBYwsxXM7C4zm2lmV5nZfLnt/Lk8Mx8fWnieY3L9Y2Y2psb3JCIiXdDlgGBmA4GDgRHuvjrQG9gFOB04x91XBl4B9s1/2Rd4Jdefk9thZqvm/60GjAV+YGa9u1ouERHpmlqbjPoAC5pZH2Ah4HlgM+CafHwqsG3en5DL5OOjzMxy/ZXu/h93fwKYCYyssVwiItJJXQ4I7v4ccBbwNBEIXgPuBV519zm52bPAwLw/EHgm/3dObr9kcX07/yMiIt2kliajxYmz+xWA5YCFiSafhjGzSWY23cymz5o1q5EvJSLS49TSZLQ58IS7z3L3d4BrgQ2B/tmEBDAIeC7vPwcMBsjHFwP+WVzfzv98gLtPcfcR7j5iwIABNRRdRESq1RIQngbWN7OFsi9gFPAwcDuwQ24zEbg+79+Qy+Tjv3F3z/W7ZBbSCsAw4O4ayiUiIl3Q5+M3aZ+732Vm1wD3AXOA+4EpwI3AlWb27Vx3cf7LxcCPzWwmMJvILMLdHzKzq4lgMgc40N3f7Wq5RESka7ocEADc/QTghKrVj9NOlpC7vw3sOJfnORk4uZayiIhIbTRSWUREAAUEERFJCggiIgIoIIiISFJAEBERQAFBRESSAoKIiAAKCCIikhQQREQEUEAQEZGkgCAiIoACgoiIJAUEEREBFBBERCQpIIiICKCAICIiSQFBREQABQQREUkKCCIiAiggiIhIUkAQERFAAUFERJICgoiIAAoIIiKSFBBERARQQBARkaSAICIigAKCiIgkBQQREQEUEEREJCkgiIgIoIAgIiJJAUFERIAaA4KZ9Teza8zsUTN7xMw+a2ZLmNk0M5uRfxfPbc3MzjOzmWb2FzNbu/A8E3P7GWY2sdY3JSIinVdrDeFc4GZ3/xSwFvAIMBm4zd2HAbflMsA4YFjeJgEXAJjZEsAJwHrASOCEShAREZHu0+WAYGaLARsBFwO4+3/d/VVgAjA1N5sKbJv3JwCXebgT6G9mywJjgGnuPtvdXwGmAWO7Wi4REemaWmoIKwCzgEvN7H4zu8jMFgaWcffnc5sXgGXy/kDgmcL/P5vr5rZeRES6US0BoQ+wNnCBuw8H/k1b8xAA7u6A1/AaH2Bmk8xsuplNnzVrVr2eVkREqC0gPAs86+535fI1RIB4MZuCyL8v5ePPAYML/z8o181t/Ye4+xR3H+HuIwYMGFBD0UVEpFqXA4K7vwA8Y2ar5KpRwMPADUAlU2gicH3evwHYM7ON1gdey6alW4DRZrZ4diaPznUiItKN+tT4/wcBl5vZfMDjwN5EkLnazPYFngJ2ym1vArYEZgJv5ra4+2wzOwm4J7c70d1n11guERHppJoCgrv/GRjRzkOj2tnWgQPn8jyXAJfUUhYREamNRiqLiAiggCAiIkkBQUREAAUEERFJCggiIgIoIIiISFJAEBERQAFBRESSAoKIiAAKCCIikhQQREQEUEAQEZGkgCAiIoACgoiIJAUEEREBFBBERCQpIIiICKCAICIiSQFBREQABQQREUkKCCIiAiggiIhI6lN2ASQMnXxj3Z/zydPG1/05RWTepRqCiIgACggiIpIUEEREBFBAEBGRpIAgIiKAAoKIiCQFBBERARQQREQkKSCIiAiggCAiIqnmgGBmvc3sfjP7ZS6vYGZ3mdlMM7vKzObL9fPn8sx8fGjhOY7J9Y+Z2ZhayyQiIp1XjxrCIcAjheXTgXPcfWXgFWDfXL8v8EquPye3w8xWBXYBVgPGAj8ws951KJeIiHRCTQHBzAYB44GLctmAzYBrcpOpwLZ5f0Iuk4+Pyu0nAFe6+3/c/QlgJjCylnKJiEjn1VpD+C5wFPBeLi8JvOruc3L5WWBg3h8IPAOQj7+W27+/vp3/ERGRbtLlgGBmWwEvufu9dSzPx73mJDObbmbTZ82a1V0vKyLSI9RSQ9gQ2MbMngSuJJqKzgX6m1nlOguDgOfy/nPAYIB8fDHgn8X17fzPB7j7FHcf4e4jBgwYUEPRRUSkWpcDgrsf4+6D3H0o0Sn8G3ffDbgd2CE3mwhcn/dvyGXy8d+4u+f6XTILaQVgGHB3V8slIiJd04grph0NXGlm3wbuBy7O9RcDPzazmcBsIojg7g+Z2dXAw8Ac4EB3f7cB5RIRkY9Ql4Dg7ncAd+T9x2knS8jd3wZ2nMv/nwycXI+yiIhI12iksoiIAAoIIiKSFBBERARQQBARkaSAICIigAKCiIgkBQQREQEUEEREJCkgiIgIoIAgIiJJAUFERAAFBBERSQoIIiICKCCIiEhSQBAREUABQUREkgKCiIgACggiIpIUEEREBFBAEBGRpIAgIiKAAoKIiCQFBBERARQQREQkKSCIiAiggCAiIkkBQUREAAUEERFJfcougLSWoZNvrPtzPnna+Lo/p4h0nmoIIiICKCCIiEhSk5HMk9S0JdJ5qiGIiAhQQ0Aws8FmdruZPWxmD5nZIbl+CTObZmYz8u/iud7M7Dwzm2lmfzGztQvPNTG3n2FmE2t/WyIi0lm11BDmAEe4+6rA+sCBZrYqMBm4zd2HAbflMsA4YFjeJgEXQAQQ4ARgPWAkcEIliIiISPfpckBw9+fd/b68/wbwCDAQmABMzc2mAtvm/QnAZR7uBPqb2bLAGGCau89291eAacDYrpZLRES6pi59CGY2FBgO3AUs4+7P50MvAMvk/YHAM4V/ezbXzW29iIh0o5oDgpktAvwcONTdXy8+5u4OeK2vUXitSWY23cymz5o1q15PKyIi1BgQzKwvEQwud/drc/WL2RRE/n0p1z8HDC78+6BcN7f1H+LuU9x9hLuPGDBgQC1FFxGRKrVkGRlwMfCIu3+n8NANQCVTaCJwfWH9nplttD7wWjYt3QKMNrPFszN5dK4TEZFuVMvAtA2BPYC/mtmfc93XgdOAq81sX+ApYKd87CZgS2Am8CawN4C7zzazk4B7crsT3X12DeUSEZEu6HJAcPc/ADaXh0e1s70DB87luS4BLulqWUREpHYaqSwiIoACgoiIJAUEEREBFBBERCQpIIiICKCAICIiSQFBREQABQQREUkKCCIiAiggiIhIUkAQERFAAUFERJICgoiIAAoIIiKSFBBERARQQBARkaSAICIigAKCiIgkBQQREQEUEEREJCkgiIgIoIAgIiJJAUFERAAFBBERSQoIIiICQJ+yCyDSUw2dfGPdn/PJ08bX/Tml51ANQUREAAUEERFJCggiIgIoIIiISFJAEBERQAFBRESSAoKIiAAKCCIikpomIJjZWDN7zMxmmtnksssjItLTNEVAMLPewPnAOGBVYFczW7XcUomI9CzNMnXFSGCmuz8OYGZXAhOAh0stlYhoio0epFkCwkDgmcLys8B6JZVFRFpQqwSuZi6nuXtdnqimQpjtAIx19/1yeQ9gPXf/atV2k4BJubgK8Fidi7IU8HKdn7PeWqGMoHLWm8pZX61QzkaUcXl3HzC3B5ulhvAcMLiwPCjXfYC7TwGmNKoQZjbd3Uc06vnroRXKCCpnvamc9dUK5SyjjE3RqQzcAwwzsxXMbD5gF+CGksskItKjNEUNwd3nmNlXgVuA3sAl7v5QycUSEelRmiIgALj7TcBNJRejYc1RddQKZQSVs95UzvpqhXJ2exmbolNZRETK1yx9CAKYmTXgOXvX+zml5zGzfmWXQRpPAaEJVH5sXufqmpl9HtheQUFqYWZbAMea2SJll6UZNOLErV7MbGkzW6Cr/6+AUDIz2xKYambXmNkoM1u4Ts87BrgIeMrd363HczaamY00szMUwJqHmY0Gvgvc5O7/6onfjZl91sy2MrPNoP4nbvViZlsBFwJdDtwKCCUys/HAicA5wN+ALwEr1uF5twFOAL7k7neZ2SfMbJ1mPrNJTwFbACfOawceM9vIzHYquxydkfvRKcCe7v47MxsE7N1dNYVm2F/NbCxwMbAjcJSZHV5ykdqVgfsEYIq7v2xmXUoYUkAoiZkNAL4D3Onuv3f3rwPvANvV+LwLAicDT+aPeCDwa2ClJj6z6QPg7i8SExx+HjhtHgsKCwI/MLPtiyub4aD3Eb4IfMLd7zWzxYBfAr3c/V+NekEzG2ZmG0GciZf5+ZjZ5sC5wLbuPjHvD6rapvTvz8w+B1wDfMXdbzazlYALzWzxzj6XAkJ5/gVMBpYxs31z3UvA6119QjMbB+wDTAQGmdnpwFTgfHe/urDdfF0udZ3lj/9hMzvFzDZ29xeA8cBwoqbQNGXtCjMbYWZj3f0WYFfgLDPbMR/rVQnSZrZ6Mxxcitx9F+CvZnYfEQzOzdkCgPdPauomT2b2ALbLg1y7QaEbP6fBwOLEbxViAO06ZvaFbOotnZn1IpqI/gosnYH7UmCGu7/S6Sd0d9268QYsQ5wt9s/lrYFrgd/mX+vi844B7gM2zOW1gUeBW6q22xbYjzjTK/NzsLyNBx4E/gLcBfyAaEbbJMv/NaB32d9bDe/zy8CdwOjC9/QEsHNhm0nAb4ClmqC8KwKfBMYU1k0FXqzablfg+8CC9dgXCveHAd8ATgM+39423fAZbATskPePAR4ChgInEfOn/Sz319uAA5vktzSaGMf1MrB38TMDVgb6dei5yt4Be9IN2Aa4A/g58CdgT6BfHhT/Any5+CV34nnHAC8SVcbi+uF5MDoyl3chziSGlf1ZFMrYG9iSaP88DVie6Bg7AXgeeA84s+xy1vge98uDx5hcHp1BYZM8sD4JrNUE5dwKuBu4Ksv3U2D1fOwG4Pd5fxvgfuDTdXrd5aqWhwHfyv1h48L6Q4GrG/wZjMmD/oaFdUcB/wB+W7XtRGDZkr6r4cDOwN60nQSOJU4s9yhstw9wHXkC+rHPW/ZO2FNuwChgBrA+0Q45KoPAwcD8RE3hGuDgTj7v1sCfiSr9scS1JYpnXGsDvwduBv4ArNoEn8VGwNHAV4HtC+/jh8Behe02y+3qcuDpxve3IrBC1bqvVAWFMcBrRCBfswnKPCb3o5G5vADwI+ASYJVcd2OW+a567UfAWhl8jgXWJGschaBwRt7fg0i8WKeBn8HWxInaRrk8EBie9/cnrs8yuAm+qy2zLN8DLgNeBQ4tfI/XEyca2xAnhGt0+LnLfnPz+o22attZRNYPwHz5dwTwCLBbLm8HXA4s1pHnBfoTtY31ct15+QMaUbXtSKL9s8M7RgM/jy3zAHA00fn9MFEj6E3UlC6iEBTpxqaCenzXRPX833nw+iKwYuHx/YmmoUrz0bo0QW0NWB14GvhCLvfNv4sSzUU/KWw7pZ4BjLjuSSXInAL8Clg5H/sE0Xx0K9EU0rDASbTD/w24PJcHEk2Zuxa2mUzUWks7QSFqcX/ggzWnkVmug3J5NHAv8AKdDNyl7og94QYMyb9Tgf3zfm+y3ZHIqpkGLFy5dfB5Fyj+zftLf0RQmL8JPotBRD/HpoV1CxJnphfm8njgx5XPqhVv+R1cQ5ypXQh8r/DYzkRzy6iyy5nl6ZsH/huJs/RPVD2+GPA4hSaUBpRhUgaD5Yka84NERs9GGRSOIWspDXr9lTMgjCCaXE4nMvMqv9dijfvLRMZed39PvYgTwDnAqbmuD23BeyTwz3wPvYnadafLWfoOOS/fiLPhx/KL251otlkpH+ubX9yniDbbDnecEm2FPyeyCS7PHXrhfGyp/DGdCqxf9meQZarUkgYC1xTWz59/FyT6NsYSTRVbVR+Ymv0G9Cnc3z0Provke3sI+CPRBj4o3+fyTVDm1Ykz8CWJbJqfEycTSxO1nV65n15FVTt/ja+7BrBNYXlLshZCdN6+AhxHNKcdQeGkpwGfwZbECcnQXF47f7PVyRg7APuU+F0tmn/HE7WlnXK5F20tDtcSKbJdfh2lnTZIjhQ+k2gTn0McEB4A9jKzldz9HY8RxMOJM7SFOvi8Y4mzzsuJjJx/E2c0W5jZgu7+MpGlszQwzszmr/Nb64rKPDivAyvlIBrc/T9mtoC7v0V0ti/t7m8DN3qkn7aE/K4PM7NlctWNRFDYgjjAzU9M7b4UceZ5u7s/VUJRq70LrEbbVQj3A1YiMruWdvf3iGbMgcB/a30xC/MRzUQTzWxreH+m4z5m9n9EbXlvd/82sAFxAvF2ra89l/KMBY4nki6ezH3xPqKpb1EzOzK324VIcvhDI8rxMWU0M1seuNfM1nP3G4n+lIvMbIf8jubk5q9S6/dUVsSbl29EG96LRHrayoX1o4jMid8AhxM746N0oG2UOBNYiGiK2K7qsUOI5onlC+uWAJZpgs9iHFEzWj6XJwPfBlar2u5M4IC830r9Br2Ig/yzwGFknwGwMXFwe57CWRuweBOUeTNgYt5fjahpHkvUEio1heOAg4Dp1LnviQgwexMnNdvlusFELXGvXJ6vge/fgGWJA2ilWWgForlocC43RTIGbbXoyURT2ohcHkecYO2Yy3sSSSo1dXqXumPOizdgcyKbaE/iTOtUPpjCNpCofp4DfBP4VAeft0/+/TGFTJDC45cSFxaCJsnbJzIeHgAm0NbnMYLIJjq1cqAE9gL+ThN0sHbxfe5KBPnJRBPHQKI2cCttHX1980BUarDLA8nfiKaSfrnuM7n/HF8ICjcRHf71yib6fH4+SxLNaL2IGsn/0NaZfRnw1bzfsM+pcJA9nOh83TK/vyOqtis1GSPLdRE5PiX3rUergsLL+d39nkwRruk1y9w557VbHgjWBjbI5VWJs+FTKuu68JxGtLnel/cvBi4uPL5Q/t2MQudl2bc8IP4O2LJSTqLpZFB+RgcTbbdXExkmNe/M3fz+lqSt32Zp4uxyCpE5dQSRILAT0R5dSq56O2X+JHGWWdk/exUeWz4Pzsfmd7cIdeg3yH12gdx/3yPSWH8BbE/0peyUr7smkZL9INHE2JCA0M5B9tA8qJ5RLHPhfinJGFnOB4iTx96F9YfnPlUJCuOJ1oi6BC31IdRJThtxK/Ceu/8ppyV4GPgJ8UPYyszW6+zzevgrkao5mgguS5jZYfn4m7np6kB/M+vbJFMgvEpUYZ/ONtDjiSa0B4j38RPgs8SEfmPc/cGSytlpOa3CI8D5Zjbc3V8iDqT3EU0sKxLt8o8QzSLNMtvsksDjuX/2Iyaqu8rMfk6cvBxJ9GlNBN5093/U+oK5/75NBIA7gf8Qs6duBexLjM8YQpzlvkMkQrzhebSrp5xu4lSiGeiVLN93iX1zvJl9pp3y/6fe5fg4Of/Y0USa+jVAbzNb2MyWcffvEB3/l5nZ+h59CkPzGFGzprmEZiuzmC/+dOAwd/8zgEdnD+7+qJldQaQbftHM3nX36Z147j6AEwfSce5+qJldAuyXO/BlRDA4BNja3d+p3zvrPDPrm2V4lyj3MUTV9jrgCmKw0YXEgelq4K2SilqLN4nv47PAADO7jjjD7Uv8kF8k2sj/C5zn7q+WU8xgZvPnge1+4mTiCuDTxHt4jghipxGB+pvA7Mr+W+PrfpKoIfRx94dyDqdbgfvdfW8z60sEzlWIQWGzvEET51UdZO82s/ksrhuwiLtfYGb/BS43s73c/Z5GlKETXiHmNXMzW5Roel4PWMrMbqet6e08i2ue1K/TvYzq0LxyI3b2+Yiz3Ym5rh/RQbYjkWHSO9evTpxFDujgcw+uWl6WqCVUOpGGEM1HldGKq9X6furweWxO9A8cneU1os9g83y8MvbidArTdLTajWj/HkGcqV1LtDVfSAS3g3Ob0USmTtllHQNcAByfyysRB//DKDRlEc02HerP6uDrVuaoupho4jiTqH0sR3Qen1C1fYfm2qmhPAsTNdR1iay+E4nMr3uJQaN9iCkq7qK8ZqJ+ZNIBkdV0BXFycSlRaxtN1Dg3y22WrHsZynjj88qNOLsgD/QH5842JXe854mh/+ML2/ft4PMOzJ1hGpH9sHSu3z1/zAtXbd+n1vdSh89iHNEnsD9xFnjGXLbbMw8UK3dn+erw/harWu6bgeBS4NhctxENHEDVhTKPyc96V6KDeHLV45XxITsRnaeD6vS66xEnL5/P5TWIs9pLiakqBhPNid+uLksDPoONaDuJ+kYZB9kOlnOr/N38jkzCyOPJLhQyrojAvV+jPrPSd9pWvRFV3F/lgWH3PPi/mAFhC2KE5znAN7rw3AsQHZU/yJ310tyx1yDy9VfN7Spn3GVnrqxCnB3vmctDgP8jp9TIdQOJDrGHaILaTCff31rA6dWfNXFWuW4eZI4ru5xVZV6PGGG8bS5vQWSo7UtkFVVGvh6bB+eav5NCgNkXOCvvV2rIg/K1TsnlVYk+hYYegIn5fF4maq+9iCC+C4WTs0YfZDtQxtFEQN6ctpkLFmpnu12Ik64VG1aW7n7z88KNOPO6i8ygyXVGVrkLB+pDiCadPh3d0aq3yx/vl4kZMfcnUgava2+HKemzmC+D32lE01UlD/+6DGYn5U6+UB6UhpZd5i68xzXzhzigne+nN7BOvt/jyy5roVy7EhMerk2cSDxMXJDpSuJEYysiSJ9M/VJLK6nFOwIX5P3igXcdouawcvVjDf4svpCvu107jzX8IPsxZVsDeIO2GsqaRJPaD/P33o84wTqQbjiZKn3HbbUbcab+D9pmRFyZqB0sWLXdPkSnXZcmwqKqGYhof92PGPH8IDmUveTPYhvirLMP0QxwPDFl8kVEXvQOef82YuBctxwAGvRer2YuaZj5/ofP7fFuLmcxlfTLRO3lUdqmQF+YOEn5ZvX2Nb7up4g2+OUyCD1PpkISJw2V2sNUYGDeb1Qz0Siidj6ctuuOjCPGulRm1x1KzLZbeo2VSFk+Jw/8vyM69jcjZl49lGiF2IZuaGYtdedtxRvR7HENMSXvwPwyjyg8vixRZb6HDubWEx2Uo4lmqGJH34cGmBE1kWaYgnc0MXd+sZa0GJEWO4u22lJlQN3Assvcyfe3IXEmPSbf18Xk9SYadSCrQ5m3INrJTyys+2IG4y1o67A8KAN133q9F2BT4qz28NxH9ycymNYsbLM7kenUsBH0xKC3HxOp3rfk6+2bB9hNiZp2pW9jQnccZOdSzgVoG0NkRP/BbHJgXq5fjzgB7NCEl/W4Ke20g8xsFeLs6gLiC/wa8SM7y91/UNj0LSLKd2g+HjMbT5wd/IxoQ/yrmT3u7qe4+7tmZl7Za8x6e8x/9Ew931tnmdkmxAFyZ4+c9pWIrKGjzOz7xGdwrJmd5u4P5b/VnNPeXXL+p9eJaZm3I1JlKxPVXVj5PppJzstzJjHT6mQzW8Hd93D3K3LMwW7Au7kf70NMuV5zirKZbUAM8rrB4hrY2xC/jQtzk1+a2W+J1MhNiT6NF2t93blx97fM7ERivMMDxIHXgLOJOabeA6aZ2Sh3v75R5fgo+V0dDLxkZn9w94uA0WZ2PTC88Jtfibh8Z/ftb2VEx1a6ETvTQkRzyAXkRTyItL3rKIxAJgZZ/YIOXlaQ6FibQdvI0cWJqu0VwNFlv/e5lHk+ImXvZ8TlQJchmoe+VthmIDEAaCot1kxEpEv+CPhRYd0QohnkfrKppZluRDv008Duhe/oDmDtwjZ7AbcTTST16jMYm/vvVrRl3G1JBKUj87ezWm63M1UXDarzZ7Bk1fJwIhnjUGIK7X7ESOhv5f5ayjQptF3qdnciSN9P9h/k478malq7E4kZ3TptRuk7c6vciHEEXydGWa5NTMNwJHF2vz7ROfUnOjZRXaU9dVPgqrxfycZYIA9KU+jAhXK6+TPon+97wQyQ5wMzyTEFhfe1aAaK0q8R3Mn3N4bIuNmCGM16edXjQ/IH3DSX9CQ6tT9J1NgOqhzoiAy4M4iO5MEZJD5Xr4MykezwKLBJ8bvP+yOJ5rYjqFMq68eUZQNicN1RxKSOlc7tdTIoHMcHJ37stiaYqnIuT3Ru71VYdwTR51HsfP81Xbi4TT1umrriI5jZhma2Z06L+yBxkJ5NRPbViAPis8QgkpOJ1LW/dOCpK9NBvwQsaGYLAu9lVfFtom1+A2KHbgrZtHUjkSr4JaK2NIvIXvk/iGkKzKzy2CseU3G3BDNbnyj30e4+jQj+fc3sGDNbzcz6u/vTRMbKBma2VJnlhfdHAp9KpDufR0yZsY2Z/YgYyfoAUXs4n8g4+pO7P1Gnl18cuNXd7zCzAcQo/AtyFPSTREBYDfiCmfVp8HQqCxHjdSYR39vZZracu99LZL8NBb6UnxfESPNuZWaLeEx5/jNgVDbdQQSDs4Gfm9nRZraKu28OfMZj6pvuLWdGJCnInXc+oho+gMhT/i/ROfcWcXY0gOi8eoYY4HKru8/swHOvRGR/XE2kkP6GmN7gspz/6L3c7jzgCne/s65vrguyzfMsou35P0Sz2InEeziM+EH+mJgO4XCixtCRwNg0sl/kbOIs+2Xih3sTUSvqD0xx99tz2z4e17golZltSDTFvEEc+FYCDiBOJrat7I/Zh7CQ16Ht3syGEPPv9yVOBk4n+g0eIfpc5hDJEZ8maiQP1eN12ynH+31ruXwQbf0GmxFpmucS3+G/iNrDN9z9n/UuSwfK+kmiz+Bid7/fzCpB6t9E8+opRADfn+hr+1ppJ1NlVJ1a5UZU8R4iAsH6RDPCj4jh7rcCN9DJvHqiuel04ge8LDEfzqtEUFkit9mN+LEt3wSfwZJEX8kFtJ1ATKAtz3woUSW/kWg+KmXe+Dq91/FE1tij5EXLc/2pwNTCctNkGeX+c0aWcVFikOB3iTEwa9X5tRYlxpYcQzQdVkZqH0deKja3u5wOTtFSQ1l6Vy1/GfjfvD+ECAJnEDX4STTw+godKGs/4PtE83IlFfe4/I0Xs7AWoeTxRaohVDGzFYjawMLu/recFOtuYpj9BXm2dTiRdz0KGO7uz3Xgeft7TnJmZqsRA4cWpO3M7izijOHfRGfzjl6nGQxrlZOSjSSaAi4kAlpv4HB3dzMbSpypXuvuM8oqZ2dVamRVmVwbEWdspwO3ufubZrYnsAmRdlrzlcNqYWabAk+7+98L6zYAtiVmCz2JGFm9DzGH0Pe8PtlEmxH9Qg8Qc1X9mbj+xmtV2+1JnA2Pc/dZtb7uXMqyOTF54APAA+5+S66/ksgE/DSxb96QTTPvlb1f5nHjBKLl4WJ3f8DMTifGbZwD/NnrMKFgzcqMRs12I7Il7iOyY2YQUXy5vP0D+FbV9ot18Hk3J5qHziWmOliWyHz4JnFmN4TYkVciDrzNMn9+cR727Ykd9zbg14X1vaq3bYUb0ayyNW21nmKn6HgiS2dLIof9Pprkeg3E2flbVI2sJWoKZwOjc/nz1HFyPeLk5++5765LNBEeStu1iIcQgeIBGlhLJDr+7yWagM4gMnIqo+M3yt/ZuFwuZZK6fO1VgXOr1vUjUoO/Sw5iJPolL6HEGswHylh2AZrlRgxEepy2FNCRRGfcd4gzoyFEf0G7F9L4mOf+DJGH/ToxOnIGMer4rPwRnUkXRzQ36LPoV7hfDApbEf0Gh9KWFdU0zSedeH9jiLPndavWF4PClsQUAg828gDXiTKPBLbI+0cQWSiVA2ElqH0bOKeOr1l53t7EaOzTaEtt3YSYquTg/G0MIvraGjbqN7+3t4GNc3kYMdPwarncj7jc5e7F8pfwXY0gmqmmkXNgFR7rRwwKPL+wrvRZcSs3ZRm1GQn8xGOgVV93v5vI3FiESBN7mjhL2sbMlqru1PooHtdIWJvo9HqdGOX7HpFFtDPxA5+U88OXysw+BRxpZiMAPAbH9c77vwSuIjrCjjSzJTr6GTQLiwu7H080/9xjZktnEx7F9+Jx4fdJxDUmuj3bo8jMxhB9V8ua2VrufjZxsvInM1u+UO4ZRLZavQacLgKxD3h0oj9FfO8LuPsdxIFtODEa+i3iWgMPze3JapGfwUlEU9XpWa4ZxCSQ55jZqUSt7zxgTzNbqBHl6EA5xxFTg7xD9Lv1N7PvVrKs3P0NYpK/FcxssVz3UhllbVfZEalZbkR2xsl5v5gT/Dmis3RoLnd5qmmiqv0abRcS700Eisk0SQ2B6BuZQkyBMLywvlhT2JVo6ir9gvGdfG+LE3PsnJ/LQ4hawLZV29Vlfp86lXkDCoMXqx47njhI70xbu369Bp2tTKQY70RhPAnRYfwNPjiW5kIyIaJBn8GY/AzWzOWfEVPDnEmche9NjI5+lOi8LWWaFGDjLGdxlt/ViWB+bmHdrsS0Gh0awNqt76HsAjTLjRhN+RLZVkxkUfTK21Syv4Aaq6EZFF4BDij7PVeVa1HaLj7+SaK/4yQ+ONq1GBQaekGTBry/yoRqWxADCI/Og8lXq7Yr/doSVeU5gLxuAG39NX0Kj3+RGH17AfW9wM1CRLbbH/IE4cRcvz1xhl48aWrYgY2oTb9IzB+2WmH9pUQtu3itgEUosfmFSDY5JO/3rXxXRObXRcSUNkcRgxubok+q+tZj5zIq5vwDuPvNZnY28D0z+6pn1dfMdiO+0L65XU1NJB7NFJsD95jZ2+5+SS3PVw9mtjER9O7Oz+AFolp7KrBFzqF0j2fzkUcTwhtllrkzclDdN8xsgrtPM7M5RNvzne7+/cJ22wD9zOyKWr/nOppD26VuexEHwXezCWJ94DqPTKgON2F+lBygtxoxtuEuoolwQ+AUi2tjzyQymB4h+gxw94ZcBtXMRhFn/IcT/Xh7mNnN7n6HxyU4FwLuMLON3f0dj8tvNuQSnB9TzspnvwLRAgAwJ9fPAR4zszOJ2X+fJOYA+1t3l7MjemQfQqbpjW9n9OSFxFiD283sXDM7nzj7+pLXcaCIxwjKdYiZDJvBi8QBYGtiLqWriUEyrxKdhVua2aoQ7ckllbFLclDdccSgpBcsri98O7AHMNTM9sntdiNGm9/VRMEAokloHzNb3d3nZP+WZxlHUMfR7Bk4f0o0G25KnNEe7u6/dfcNiZrV28TYlAkW1yRupNeJ5tXLiXEu7wBjchAh7r4zcQD+dYPL8ZEK+8v/Auub2Tq5zsyscozdBPiFu1/drMEA6HlNRswlw6Rqm82IqvgkYKWyy9zAz2IUcGDeX5n40Z1GtLV/lUjpe4I4K72A1puo7jNEoBubyysRE7xVUv42pe0s+Pc0RzbRh/oviD6m2/lgk8luxLxLy9fpdYcR/SmfL6xblRjYdXzVtlvTjReUoa2pbBgxQv5U8nokub4pplYnUse/SaTDrlNYvwsx19KQssrW0VuPGpiWGSZHEzv47Wa2NDGisiGZEc3OzNYlOg4PcPcf5hD7qcSZzCm5zXpEE8KdXnK2TWdkjWYNIn30VmJw4YXALe5+RmG7LYj+kp295IGAWXNdEvilF36YmY3yFSIb7WpiFtEtgC/Ua9/N736yu++TWWW93P2d/BxvAr7o7n+qx2vVwsyGESdrSxITQ/6xXs1l9ZADWfclTramE9lXOwA7eMyH1tzKjkjdGL07lGHS025Ek8MrwP65vArRkXhq2WWr4T2NI7I4Pk10Sv6ASBg4LB+vnAh9mphdtvRsDzpWc92EyCjagzqfoROJBI9SmO6CtiSDqWQtqxluRJPWMTR4eowayrcgkZ34TWJKjU+WXaaO3npEp7KZDXT353JY/bfM7Ghi9PAP3f26wnZNMWlZd3L3e/MseZqZ4TE9xz7E7ItvufuJZZexMzJf/XSi3+cRM/s70eTVj+jom8/d/5vvcT9inEG3T3hWVKi5vj82gnZqrh65//V83eWIifz6ekzTch0wzsxmu/szxBQuEONn+tfztWvh7o+a2Vlehyk5GsGjk/0PeWsp83yncnaUXWtmn/CY1vhYYnTl6/7hDJOd2+lonue5+3SiCeIUM5vk0en1BWIkasvIDuSfEBdruQvAY+6hO4iLDg0D9jOzvYlMmUlNEAwWJ9I6H/BoxhxCTA8yrMGvO46YtXYqkYG1CFGrWokY2LWuu7uZ7UE0f9zVyPJ0VrMGg5ZXdhWlwVW3scRc/WNyuVIF3oyYD2WfXN6NaD4q5fqqzXIjmo/eq3wurXQjOpAfyr+nEhlcyxQe75P7wzXEtX5LvbB6lqmUsRFEp/AfiWaNLxCjeytzII0lxp88QZwQNM08Tro1/jbPdiqb2WeIzJEdPcYYrEQMDtnN3f9hMWvkaURe8HLEHP4t02naKGY2HHjT3R8ruywdZWarE7nyN3k0dZApw58hOl5fynV9crsnPKYiKU1lbAQwwSMddlPaxkZsX9huG6K5q+axEZkCOT9xsP+tR9om2YS6gLt/y9pmgB1MpHm+5800tYI01DzZZJSZEasA1wJLZmbCFOBX7v4PAI9c9OOITJQDFAyCu9/fYsFgEeKMd+Pienc/kJjK4X+zTR53n+ORU192MChrbEQfj/btzxL58sfm+pWBvc3sN8DNFtOd93L3FxQMepZ5roaQbaOH5m0wMU/8DkTWzDmVFDUz+zRxpmTeoJGW0j3MbE3ioj3LE9c7fqzw2HlEG/gm3qD5+TujrJprJg7sQ1yU5ZfEYMR7iWndnyDmA/oUMW32SOBgBYOeZ56qIRQyTL7h7o8QnYnXEp1llQwTzzOwi4mL4CgYtCAzW9fMtgXwuFznRcT05EdZzNhKPnYwkUe/cBnlLCqr5po1kpOJfor5iakgehET5y0L/NVjKpJ73f07wN4KBj3TPFNDyJ3+x8Ab7r5iYX0fotNuHJFn/RYxcOQr3goDReRDzGxJImd/CeJ6Ff2JE4GhxIXm1yUmhHuynBJ+WFk1VzNbgkgtneDuv8i+gbOAa9z9ZxmU7iAuEfr1/J+mGegl3WueqCFkNfxs4sB/lZn90cyWgWg3JjI3biKyi75NVMMVDFqQma3gkSq6NzGtwrPElMPfIqZAHkkEhVNy1Gjpyqy5uvtsoo/lNDNbNDvd3wGWsJiocAbRpLaDmS2pYNCztXwNoRUzTKRrMjPncKJJ42kz25UIBDsSVzZbg0ghHknMPDnS3V8oq7zQPDXXrKGcRwSh5YipKN6uDMa0mDRPuf09XEsHhMwwOYg4EBxdCQj52IeCgrQuMxtN1O6Od/dbzGwhj2mfJxIXitnfY2rrStrk0mV/71lzvZwIUjsT1/zdzt1fzMf7ECPm9yMyf0Z7A+fVsph2/VbgE+7+ksWVz97Ox1QzkNYOCNBaGSbSNRZz9P+ROOO/NzNzTgeOcvfHLaYk+ToxV9Gv8n9KPcA1a801awpnAZuWHTCl+bRkH0IrZphITV4krjK1kcXMn5cAd7v74wDufhkRIE6xvJZuycGgacdGZMD8OjHeoFdPnKpF5q7lagitmGEiXWNxJbc+7n6bmQ0lAsFniXTMSwvNQ+tkzWERj6tmla7Za67N9FlJ82ipGkIrZphI12TTxoXAkBzJ+yQxsOoeYgI2MhjsDZxpZsuUfYBrpZpr2Z+VNKeWqSG0YoaJdE32GVwI7OcxE2vxsaH52F3ESNtjiKmuS00jVs1V5gUtERBaMcNEus7MtgI2dvcjzWxR4uIxWwN/B64krut7PXGw3aAeo3lrkTXXJyyua3A+ERDmB9Yiruz1LHGS8gJwpLs/V1phRT5C0zcZ5dnir4iD/i2ZYXKZma3o7lOJa6x+z8zGuft7+W/KKGptbwCrZHPQr4hRvf8hLuV5iLs/BWwDDG+CYDAeuMjMhrj7L4gprA8AbiYmqzsamA30Jqabfressop8nFa4Yloxw2Qm0bF4YzHDxOIasKeY2W/d/U3lU7ceMxtFXI/hZXe/JNM2RwC/A/7H3R/LDKKbzWxwMwwuzJrrCUTN9emsuf7UzOYDfk5bzfUvqrlKK2jagFCVYbIDEQhOof0Mk0vN7Gfu/ma5pZauyAPrWcTFazY3s3+7+/ntbLo1cYH50jtECzXXD4yNMLOj3H2qmTlRc31/bASquUqTa8omo1bMMJGuyWAwFdje4/rNDwCrmtmYypgCMxtiZvsBk4mz7lfKK/H7WmpshEhHNF2ncitmmEjXWVwE5hziTPtJM3sMqHzv6xIds2sAXwLOLfu7buWxESIfpxmbjJYCprn79LlkmHyZyDA5hCbIMJHauPvl2bxys5nNDxzj7lcCmNmVwEHufoaZPeTu/y6zrFlz/Q5wRqXmajFD6WV8uOa6h5ntWpm3SKQVNGNAKGaY7EdcOepN2jJMDrC4zmxvd3+ivGJKV2UT3/sHSne/IqdQOB2YWdh0Btms2QTBYH3gVGCPYs01g8KewIVm9i2i5jqJqLkqGEhLaYqA0IoZJtI1OWL3YTM7F3jE3afA+zWFPsClOb6kPzCeGGzYDFRzlXle6QGhFTNMpCb/Ii7l+AJxUZbPEd/9HZmd8w5xAZlXgLEeF5RpBqq5yjyv1E7lQobJRu4+w8yOBhYiDhi/z9HIQ4DRwIFEdV0dyC3OzL4DDCTO/ncAdiWmfPgakb2zMvBc2cGgnZrrgcCqwOtU1VyB3bxwPQ6RVlR2DWEAMYKzcqWmfYgMk5WJHO61gE8A66Fg0PLM3r9GwWSiI3YpoqawBnHhlmOA54Ejyh5Topqr9ESlp52a2ReBbxBzv1RnmNyXGSYLl92pKPWRncd9iTmoViTOwCe7+3UWF3x/uexxBqq5Sk/V7QPTzGyZ4rK7XwGcRBwkmjLDROrHw3+BnxDXBLjc3a/Lx2aUHQxSezXXlYE9gT+b2YKo5irzoG5tMmrhDBOps2x/nwwMrcxeW3aZKlppbIRIPXV3H0KrZphIY9wJbFd2IaA1x0aI1Fu39yG0SoaJdI9mqB1Uaq7AB2qu+dhEYt+s1FzPIjKKtH/KPKfbagitlGEi3adJvmvVXEXo5hpCK2SYSM+kmqtISWmnZrYK8FvgfHc/qdsLIJIqNde8qM1lwKHAp4hZTG8lsolUc5UeoZSBac2cYSI9SwYDy8UZwNlEzfXwqpqr9lGZ55V5gZw7gbVLfH0RoGXGRog0XNlzGal2IE3FzPYChgJnaN+UnqbUS2jqBydNSDVX6bFKn8tIpNmo5io9lQKCiIgAJTcZiYhI81BAEBERQAFBRESSAoKIiAAKCCIikhQQREQEUEAQEZH0/yFcgpGPUDYpAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.rcParams['figure.figsize'] = [6, 4]\n", "fake_df['voting_method'].value_counts().plot(kind='bar', rot=45)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Not the prettiest of graphs, but we get the picture. **We have 8 different categories of voting method. I would hazard a guess that half of them meet our definition of rare labels.**" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
voting_method%
ABSENTEE ONESTOP980449.020
NO VOTE469723.485
IN PERSON245112.255
ABSENTEE BY MAIL232411.620
ABSENTEE CURBSIDE5802.900
PROVISIONAL650.325
TRANSFER410.205
CURBSIDE380.190
\n", "
" ], "text/plain": [ " voting_method %\n", "ABSENTEE ONESTOP 9804 49.020\n", "NO VOTE 4697 23.485\n", "IN PERSON 2451 12.255\n", "ABSENTEE BY MAIL 2324 11.620\n", "ABSENTEE CURBSIDE 580 2.900\n", "PROVISIONAL 65 0.325\n", "TRANSFER 41 0.205\n", "CURBSIDE 38 0.190" ] }, "execution_count": 37, "metadata": {}, "output_type": "execute_result" } ], "source": [ "show_column_dist(fake_df, 'voting_method')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Yup! Four of our categories are rare labels. **Now we could just group them all into an “Other” category and call it a day, but this may not be the most appropriate method.**\n", "\n", "Based on research I did into how these methods are coded, I know that `Absentee` means someone voted early. So we could group any `Absentee` method into an `Early` category, group `In-Person` and `Curbside` into an `Election Day` category, leave `No Vote` as its own category, and group `Provisional` and `Transfer` into an `Other` category.\n", "\n", "The following code accomplishes this by first defining a dictionary using the original `voting_method` categories as keys. The value for each key is the new category we actually want." ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [], "source": [ "vote_method_map = {'ABSENTEE ONESTOP': 'Early',\n", " 'IN PERSON': 'Election Day',\n", " 'ABSENTEE BY MAIL': 'Early',\n", " 'ABSENTEE CURBSIDE': 'Early',\n", " 'TRANSFER': 'Other',\n", " 'PROVISIONAL': 'Other',\n", " 'CURBSIDE': 'Election Day',\n", " 'NO VOTE': 'No Vote'}\n", "\n", "fake_df['vote_method_cat'] = fake_df['voting_method'].map(vote_method_map)" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
voting_methodvote_method_cat
0ABSENTEE ONESTOPEarly
1IN PERSONElection Day
2ABSENTEE ONESTOPEarly
3NO VOTENo Vote
4IN PERSONElection Day
.........
19995ABSENTEE ONESTOPEarly
19996ABSENTEE ONESTOPEarly
19997ABSENTEE BY MAILEarly
19998NO VOTENo Vote
19999NO VOTENo Vote
\n", "

20000 rows × 2 columns

\n", "
" ], "text/plain": [ " voting_method vote_method_cat\n", "0 ABSENTEE ONESTOP Early\n", "1 IN PERSON Election Day\n", "2 ABSENTEE ONESTOP Early\n", "3 NO VOTE No Vote\n", "4 IN PERSON Election Day\n", "... ... ...\n", "19995 ABSENTEE ONESTOP Early\n", "19996 ABSENTEE ONESTOP Early\n", "19997 ABSENTEE BY MAIL Early\n", "19998 NO VOTE No Vote\n", "19999 NO VOTE No Vote\n", "\n", "[20000 rows x 2 columns]" ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" } ], "source": [ "fake_df[['voting_method', 'vote_method_cat']]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "That last line creates a new column, `vote_method_cat`, based on the original values in the `voting_method` column. It does so by applying Pandas’ [map()](https://pandas.pydata.org/docs/reference/api/pandas.Series.map.html) method to the original column, and feeding in our `vote_method_map` to translate from key to corresponding value." ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
vote_method_cat%
Early1270863.540
No Vote469723.485
Election Day248912.445
Other1060.530
\n", "
" ], "text/plain": [ " vote_method_cat %\n", "Early 12708 63.540\n", "No Vote 4697 23.485\n", "Election Day 2489 12.445\n", "Other 106 0.530" ] }, "execution_count": 40, "metadata": {}, "output_type": "execute_result" } ], "source": [ "show_column_dist(fake_df, 'vote_method_cat')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we’ve gotten rid of all but one of our rare labels. Ultimately I chose to drop those 106 `Other` votes. **Voting method was actually the target variable I was trying to predict and what I was really interested in was how people chose to vote**. Provisional and transfer ballots are more reflective of the process and regulations surrounding voting, but my question was specifically about a voter’s active choice.\n", "\n", "**So not only can you think about engineering predictive features to better represent the underlying structure of your data, you can consider how best to represent your target variable relative to your specific question.**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## Applying a Custom Function with apply()\n", "Finally, we’re going to work on binning `birth_state`. This variable has 57 categories: one for each state, one for missing information, one for each U.S. territory, and a final category for individuals born outside the United States.\n", "\n", "So the graph looks comically terrible:" ] }, { "cell_type": "code", "execution_count": 62, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 62, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAswAAAFKCAYAAAAJ0AlCAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAvI0lEQVR4nO3dedxkRXno8d/DjCIuLMqIXEAHFWPAKOqIqDEiRkAwAY0xYILEYMYkEPFqEkG9ruAlUURx4QYDEbgGJBplBBQJSNyBYZFVw8giMxdkFFREJQGe+0fVC01Pd/Xy9rsNv+/n05+3u07VqerznuU5deqcjsxEkiRJUm8bzHUDJEmSpPnMgFmSJElqMGCWJEmSGgyYJUmSpAYDZkmSJKlh8Vw3oGXzzTfPpUuXznUzJEmStJ67+OKLf5yZS3pNm9cB89KlS1m5cuVcN0OSJEnruYi4sd+0oYdkRMSiiLg0Is6on7eNiAsiYlVEfCYiHlrTN6yfV9XpSzvmcVhN/35E7D6N7yRJkiTNilHGMB8CXNPx+e+BozPzycDtwIE1/UDg9pp+dM1HRGwP7AvsAOwBfCIiFk2v+ZIkSdLMGipgjoitgb2Af6qfA9gV+GzNciKwT32/d/1Mnf6Smn9v4NTMvCszrwdWATtN4DtIkiRJM2bYHuYPA38H3Fs/Pwb4aWbeXT+vBraq77cCbgKo039W89+X3qPMfSJieUSsjIiVa9euHf6bSJIkSTNgYMAcES8Hbs3Mi2ehPWTmcZm5LDOXLVnS80ZFSZIkadYM85SMFwC/HxF7Ag8DNgY+AmwaEYtrL/LWwJqafw2wDbA6IhYDmwA/6Uif0llGkiRJmpcG9jBn5mGZuXVmLqXctHdeZv4x8FXgVTXbAcDp9f2K+pk6/bzMzJq+b32KxrbAdsCFE/smkiRJ0gyYznOY3wqcGhGHA5cCx9f044GTI2IVcBslyCYzr4qI04CrgbuBgzLznmnUL0mSJM24KJ2/89OyZcvSHy6RJEnSTIuIizNzWa9pozyHWZIkSXrQMWCWJEmSGqYzhnnWLT30zJ7pNxy51yy3RJIkSQ8W9jBLkiRJDQbMkiRJUoMBsyRJktRgwCxJkiQ1GDBLkiRJDQbMkiRJUoMBsyRJktRgwCxJkiQ1GDBLkiRJDQbMkiRJUoMBsyRJktRgwCxJkiQ1GDBLkiRJDQbMkiRJUoMBsyRJktRgwCxJkiQ1GDBLkiRJDQbMkiRJUoMBsyRJktRgwCxJkiQ1GDBLkiRJDQMD5oh4WERcGBHfjYirIuI9Nf1TEXF9RFxWXzvW9IiIYyJiVURcHhHP6pjXARFxbX0dMGPfSpIkSZqQxUPkuQvYNTN/EREPAb4REV+q0/42Mz/blf9lwHb19VzgWOC5EfFo4F3AMiCBiyNiRWbePokvIkmSJM2EgT3MWfyifnxIfWWjyN7ASbXcd4BNI2JLYHfgnMy8rQbJ5wB7TK/5kiRJ0swaagxzRCyKiMuAWylB7wV10hF12MXREbFhTdsKuKmj+Oqa1i+9u67lEbEyIlauXbt2tG8jSZIkTdhQAXNm3pOZOwJbAztFxNOAw4CnAs8BHg28dRINyszjMnNZZi5bsmTJJGYpSZIkjW2kp2Rk5k+BrwJ7ZObNddjFXcA/AzvVbGuAbTqKbV3T+qVLkiRJ89YwT8lYEhGb1vcbAS8FvlfHJRMRAewDXFmLrABeW5+WsTPws8y8GTgb2C0iNouIzYDdapokSZI0bw3zlIwtgRMjYhElwD4tM8+IiPMiYgkQwGXAX9T8ZwF7AquAXwKvA8jM2yLifcBFNd97M/O2iX0TSZIkaQYMDJgz83LgmT3Sd+2TP4GD+kw7AThhxDZKkiRJc8Zf+pMkSZIaDJglSZKkBgNmSZIkqcGAWZIkSWowYJYkSZIaDJglSZKkBgNmSZIkqcGAWZIkSWowYJYkSZIaDJglSZKkBgNmSZIkqcGAWZIkSWowYJYkSZIaDJglSZKkBgNmSZIkqcGAWZIkSWowYJYkSZIaDJglSZKkBgNmSZIkqcGAWZIkSWowYJYkSZIaDJglSZKkBgNmSZIkqcGAWZIkSWowYJYkSZIaBgbMEfGwiLgwIr4bEVdFxHtq+rYRcUFErIqIz0TEQ2v6hvXzqjp9ace8Dqvp34+I3WfsW0mSJEkTMkwP813Arpn5DGBHYI+I2Bn4e+DozHwycDtwYM1/IHB7TT+65iMitgf2BXYA9gA+ERGLJvhdJEmSpIlbPChDZibwi/rxIfWVwK7Aa2r6icC7gWOBvet7gM8CH4uIqOmnZuZdwPURsQrYCfj2JL5IP0sPPbPvtBuO3Gsmq5YkSdJ6YKgxzBGxKCIuA24FzgF+APw0M++uWVYDW9X3WwE3AdTpPwMe05neo0xnXcsjYmVErFy7du3IX0iSJEmapKEC5sy8JzN3BLam9Ao/daYalJnHZeayzFy2ZMmSmapGkiRJGspIT8nIzJ8CXwWeB2waEVNDOrYG1tT3a4BtAOr0TYCfdKb3KCNJkiTNS8M8JWNJRGxa328EvBS4hhI4v6pmOwA4vb5fUT9Tp59Xx0GvAPatT9HYFtgOuHBC30OSJEmaEQNv+gO2BE6sT7TYADgtM8+IiKuBUyPicOBS4Pia/3jg5HpT322UJ2OQmVdFxGnA1cDdwEGZec9kv44kSZI0WcM8JeNy4Jk90q+jjGfuTv818Id95nUEcMTozZQkSZLmhr/0J0mSJDUYMEuSJEkNBsySJElSgwGzJEmS1GDALEmSJDUYMEuSJEkNBsySJElSgwGzJEmS1GDALEmSJDUYMEuSJEkNBsySJElSgwGzJEmS1GDALEmSJDUYMEuSJEkNBsySJElSgwGzJEmS1GDALEmSJDUYMEuSJEkNBsySJElSgwGzJEmS1GDALEmSJDUYMEuSJEkNBsySJElSgwGzJEmS1DAwYI6IbSLiqxFxdURcFRGH1PR3R8SaiLisvvbsKHNYRKyKiO9HxO4d6XvUtFURcejMfCVJkiRpchYPkedu4C2ZeUlEPAq4OCLOqdOOzswPdmaOiO2BfYEdgP8B/HtEPKVO/jjwUmA1cFFErMjMqyfxRSRJkqSZMDBgzsybgZvr+zsi4hpgq0aRvYFTM/Mu4PqIWAXsVKetyszrACLi1JrXgFmSJEnz1khjmCNiKfBM4IKadHBEXB4RJ0TEZjVtK+CmjmKra1q/9O46lkfEyohYuXbt2lGaJ0mSJE3c0AFzRDwS+Bzwpsz8OXAs8CRgR0oP9FGTaFBmHpeZyzJz2ZIlSyYxS0mSJGlsw4xhJiIeQgmWP52Z/waQmT/qmP5J4Iz6cQ2wTUfxrWsajXRJkiRpXhrmKRkBHA9ck5kf6kjfsiPbK4Ar6/sVwL4RsWFEbAtsB1wIXARsFxHbRsRDKTcGrpjM15AkSZJmxjA9zC8A9geuiIjLatrbgP0iYkcggRuANwBk5lURcRrlZr67gYMy8x6AiDgYOBtYBJyQmVdN7JtIkiRJM2CYp2R8A4gek85qlDkCOKJH+lmtcpIkSdJ84y/9SZIkSQ0GzJIkSVKDAbMkSZLUYMAsSZIkNRgwS5IkSQ0GzJIkSVKDAbMkSZLUYMAsSZIkNRgwS5IkSQ0GzJIkSVKDAbMkSZLUYMAsSZIkNRgwS5IkSQ0GzJIkSVKDAbMkSZLUYMAsSZIkNRgwS5IkSQ0GzJIkSVKDAbMkSZLUsHiuGzBfLT30zJ7pNxy51yy3RJIkSXPJHmZJkiSpwYBZkiRJajBgliRJkhoMmCVJkqSGgQFzRGwTEV+NiKsj4qqIOKSmPzoizomIa+vfzWp6RMQxEbEqIi6PiGd1zOuAmv/aiDhg5r6WJEmSNBnD9DDfDbwlM7cHdgYOiojtgUOBczNzO+Dc+hngZcB29bUcOBZKgA28C3gusBPwrqkgW5IkSZqvBgbMmXlzZl5S398BXANsBewNnFiznQjsU9/vDZyUxXeATSNiS2B34JzMvC0zbwfOAfaY5JeRJEmSJm2kMcwRsRR4JnABsEVm3lwn3QJsUd9vBdzUUWx1TeuXLkmSJM1bQwfMEfFI4HPAmzLz553TMjOBnESDImJ5RKyMiJVr166dxCwlSZKksQ0VMEfEQyjB8qcz899q8o/qUAvq31tr+hpgm47iW9e0fukPkJnHZeayzFy2ZMmSUb6LJEmSNHHDPCUjgOOBazLzQx2TVgBTT7o4ADi9I/219WkZOwM/q0M3zgZ2i4jN6s1+u9U0SZIkad5aPESeFwD7A1dExGU17W3AkcBpEXEgcCPw6jrtLGBPYBXwS+B1AJl5W0S8D7io5ntvZt42iS8hSZIkzZSBAXNmfgOIPpNf0iN/Agf1mdcJwAmjNFCSJEmaS/7SnyRJktRgwCxJkiQ1GDBLkiRJDQbMkiRJUoMBsyRJktRgwCxJkiQ1GDBLkiRJDQbMkiRJUoMBsyRJktRgwCxJkiQ1GDBLkiRJDQbMkiRJUoMBsyRJktRgwCxJkiQ1GDBLkiRJDQbMkiRJUoMBsyRJktRgwCxJkiQ1GDBLkiRJDQbMkiRJUoMBsyRJktRgwCxJkiQ1GDBLkiRJDQbMkiRJUoMBsyRJktQwMGCOiBMi4taIuLIj7d0RsSYiLquvPTumHRYRqyLi+xGxe0f6HjVtVUQcOvmvIkmSJE3eMD3MnwL26JF+dGbuWF9nAUTE9sC+wA61zCciYlFELAI+DrwM2B7Yr+aVJEmS5rXFgzJk5tciYumQ89sbODUz7wKuj4hVwE512qrMvA4gIk6tea8evcmSJEnS7JnOGOaDI+LyOmRjs5q2FXBTR57VNa1f+joiYnlErIyIlWvXrp1G8yRJkqTpGzdgPhZ4ErAjcDNw1KQalJnHZeayzFy2ZMmSSc1WkiRJGsvAIRm9ZOaPpt5HxCeBM+rHNcA2HVm3rmk00iVJkqR5a6we5ojYsuPjK4CpJ2isAPaNiA0jYltgO+BC4CJgu4jYNiIeSrkxcMX4zZYkSZJmx8Ae5og4BdgF2DwiVgPvAnaJiB2BBG4A3gCQmVdFxGmUm/nuBg7KzHvqfA4GzgYWASdk5lWT/jKSJEnSpA3zlIz9eiQf38h/BHBEj/SzgLNGap0kSZI0x/ylP0mSJKnBgFmSJElqMGCWJEmSGgyYJUmSpAYDZkmSJKnBgFmSJElqMGCWJEmSGgyYJUmSpAYDZkmSJKnBgFmSJElqMGCWJEmSGhbPdQPWJ0sPPbPvtBuO3GsWWyJJkqRJsYdZkiRJajBgliRJkhoMmCVJkqQGA2ZJkiSpwZv+5pg3CkqSJM1v9jBLkiRJDQbMkiRJUoMBsyRJktRgwCxJkiQ1GDBLkiRJDQbMkiRJUoMBsyRJktQwMGCOiBMi4taIuLIj7dERcU5EXFv/blbTIyKOiYhVEXF5RDyro8wBNf+1EXHAzHwdSZIkabKG6WH+FLBHV9qhwLmZuR1wbv0M8DJgu/paDhwLJcAG3gU8F9gJeNdUkC1JkiTNZwMD5sz8GnBbV/LewIn1/YnAPh3pJ2XxHWDTiNgS2B04JzNvy8zbgXNYNwiXJEmS5p1xxzBvkZk31/e3AFvU91sBN3XkW13T+qWvIyKWR8TKiFi5du3aMZsnSZIkTca0b/rLzARyAm2Zmt9xmbksM5ctWbJkUrOVJEmSxjJuwPyjOtSC+vfWmr4G2KYj39Y1rV+6JEmSNK+NGzCvAKaedHEAcHpH+mvr0zJ2Bn5Wh26cDewWEZvVm/12q2mSJEnSvLZ4UIaIOAXYBdg8IlZTnnZxJHBaRBwI3Ai8umY/C9gTWAX8EngdQGbeFhHvAy6q+d6bmd03EkqSJEnzzsCAOTP36zPpJT3yJnBQn/mcAJwwUuskSZKkOeYv/UmSJEkNBsySJElSw8AhGZqflh56Zs/0G47ca5ZbIkmStH6zh1mSJElqMGCWJEmSGgyYJUmSpAYDZkmSJKnBgFmSJElqMGCWJEmSGgyYJUmSpAYDZkmSJKnBgFmSJElqMGCWJEmSGgyYJUmSpAYDZkmSJKnBgFmSJElqMGCWJEmSGgyYJUmSpAYDZkmSJKnBgFmSJElqMGCWJEmSGgyYJUmSpAYDZkmSJKnBgFmSJElqMGCWJEmSGqYVMEfEDRFxRURcFhEra9qjI+KciLi2/t2spkdEHBMRqyLi8oh41iS+gCRJkjSTJtHD/OLM3DEzl9XPhwLnZuZ2wLn1M8DLgO3qazlw7ATqliRJkmbU4hmY597ALvX9icD5wFtr+kmZmcB3ImLTiNgyM2+egTaoh6WHntl32g1H7jVyuVYZSZKk9cV0e5gT+EpEXBwRy2vaFh1B8C3AFvX9VsBNHWVX17QHiIjlEbEyIlauXbt2ms2TJEmSpme6Pcy/nZlrIuKxwDkR8b3OiZmZEZGjzDAzjwOOA1i2bNlIZTV/jNubLUmSNN9Mq4c5M9fUv7cCnwd2An4UEVsC1L+31uxrgG06im9d0yRJkqR5a+yAOSIeERGPmnoP7AZcCawADqjZDgBOr+9XAK+tT8vYGfiZ45clSZI0301nSMYWwOcjYmo+/5KZX46Ii4DTIuJA4Ebg1TX/WcCewCrgl8DrplG3JEmSNCvGDpgz8zrgGT3SfwK8pEd6AgeNW5/Wf457liRJ85G/9CdJkiQ1zMRzmKVZ5XOiJUnSTLKHWZIkSWowYJYkSZIaHJKhByVvMJQkScMyYJZG4HhpSZIefAyYpRlmb7YkSQubY5glSZKkBnuYpXlo3F5ph4xIkjR5BszSg5xDRiRJajNgljQyg2xJ0oOJAbOkWWOgLUlaiAyYJc1rjueWJM01n5IhSZIkNdjDLEmVQ0YkSb0YMEvSNBhkS9L6z4BZkubAOIG247klaW4YMEuS1jGbAb0kzXcGzJKkOWUPuKT5zoBZkrTgzObwFHvOJRkwS5I0YY43l9YvBsySJC1g9oBLM8+AWZKkB6HZGp5iQK/1gQGzJEmadxyeovlk1gPmiNgD+AiwCPinzDxyttsgSZLWP/Ph+eaDymlhmtWAOSIWAR8HXgqsBi6KiBWZefVstkOSJGm2+XSXhWu2e5h3AlZl5nUAEXEqsDdgwCxJkjSHDLL7i8ycvcoiXgXskZmvr5/3B56bmQd35FkOLK8ffwP4fp/ZbQ78eIxmjFNutsrMZl3zvX2zWdd8b99s1jXf2zebdc339s1mXfO9fbNZl+1bOHXN9/bNZl3zvX2zWVerzBMyc0nPKZk5ay/gVZRxy1Of9wc+Nua8Vs5WudkqY/tcFnNd13xvn8vCZTHXddm+hVPXfG+fy2JhLYsNekbRM2cNsE3H561rmiRJkjQvzXbAfBGwXURsGxEPBfYFVsxyGyRJkqShzepNf5l5d0QcDJxNeazcCZl51ZizO24Wy81Wmdmsa763bzbrmu/tm8265nv7ZrOu+d6+2axrvrdvNuuyfQunrvnevtmsa763bzbrGqt9s3rTnyRJkrTQzPaQDEmSJGlBMWCWJEmSGgyYJUmSpAYD5jkSEY9vTHvhmPN87hhlHjJmXX8wTjkV4y53LXwRsWiW6um7jkXEtmPMr7nNR8QrR53nuCLicbNVlzQJEbFJY9qy2WyLxrMgbvqLiDcDP8vM47vSDwQelZkfnmBdO2fmd0Ys87DajrVd6UuAOzLz1z3KXAf8H+CozLynpm0BHAU8NTNH3oAi4oeZ2TcQ78gXwK7Aa4CXZ+YWk6pr0EEzM/9thDoeBvxeZv7rqO0bRUQ8ibIs9s3MHfrkeXNjFncBPwC+kpn3NuoZuNwj4jcys+evW0bECzLzm32mvbbRPjLzpNb0HvN7TmZe1Gdacx3LzB/2KfcblF/xfGpNugb4ZL/v21HuacDfAdvXpKso283lA8o9srbnF618Ne/ulG34s13pr6Lse84ZNI+ucn3X3Yi4DPjLzPz2KPNs1PX4Xss8Is4C9snM/+pKfwZwemYuHbGe5v4lIi7JzGeNOM+vZOZuo5Sp5W4BrgROAT6XmT8ddR4d89oA2C8zPz1Cmb7bYp/8jwF+B/hhZl48Yvu2oeybPjBE3ocATwPWZOato9QzxLw3zsyf95nWcx2s05rrRGZe0qNMa39LZn6oR5mnZub36vsNM/Oujml9j+sR8egBdd3Wo8z7M/NtrXI9ylwE7JaZt3el7wYcn5nb9Cm3a2aeV99vm5nXd0x7Za9j6jgxSUeex1COUZ376VMy8yd98nf/fxP4cWbe1KjjCcBPM/Nn9fOLgX2AGyk/Zvdf/cr2mNdQ20dEbJ6Z4/z64H0WSg/zHwO9DvgnA382qHBEfDEiVnS9To6IQ+qK1ekTEfGPEbHpCO07BujVK/zbwNF9yjwbeBJwWUTsGhGHABcC3wZ2GqHuTtGcGLFzRBxDWSlPB77G/RvFpOr6vY7XcV2fXz5wphGLImLPiDi5tvOPGnk/GhHH9HsNqOd/RMT/rDuxqyjbwr6NIo9qvB5HCQRP7VPXKMv9mog4cSrY6/LRRvue0+f1PuCERrnOdm4fEe+LiFXAsY2sZwJn1L9ndny+ALi+V4GIeB5wPnAHZb34JHAn8NWI2LnRpr2Bz9eyf1Zf/wF8rk7rVeavIuKHlOX9w4i4MSL+qvF9AN5Z59vtfOC9A8pO1TvsuvsG4CMR8cmI2GyYedf5Py8iXhURj62fnx4R/wL0C9wuAb4UEQ/vmMculP/Vnw9bb2cTxigzSO+foB1sK+ADlH3s9yPi9IjYNyI26lcgIjaOiMMi4mMRsVsUfw1cB7y6R/5FEbFfRPxNPWkjIl4eEd8CPtZqXESc0VFmS0pw/2fAyRHxpkFfLiKW1PX465R1sGenRkT8n4jYob7fBPgu5Vh5aUTsN6COAyLikoi4s75WRvvE+/yOsud2TftCo9xK4FPAB+vrqI7XB/uUae1vH9WnzL90vO8+Gf1Eo30X1zZeDPy/jvdT6b3s0ZhfP8dR9nf3rfMR8RrgH4G9GuU6l9Hnuqa9o0+ZcWISIuI3Kevqs4H/BK6lHEeuiIh+x6yjul4fAs6IiGsiYsc+ZU4DHlHr3BH4V+CHwDNo/6+m2jnU9lHz/l5ErK3fYXVEPH/Q/Psa5+cBZ/sFfLcx7Yohyn+EsjFNBW7/t/5TPg6c3JV3A+BNlJVl/yHbd3Fj2lUDyh4C3AusBrae5nL6YZ/091NW/HOB1wOPAa6fibq68lw6wvxeRNlx3ETZKdwCPHxAmQNarz5llgNfrf/fw4GnT3dZdMz78ukud+AK4Ijavp3HWZ6UwOZP6rw+Azy9kXcpcBhwOeUA8WNg6YjfeyklwL4W+Os+eb4E7NLn//6lxry/26s9tc519guUA8hZwBM70p4IfBF4R6Oevj+V2v1/ndC6G8BfUq5MfIxygDsGOKZP/g9Qe3ooPwB1eK3nEOBhjXreAXwdeCTwSspBadmY63dzmwd+Wdej7tcV/ZYhJVh9Zb/XkO16KLB3XTa3AJ/uk+90SuD2BsoB+3zKSdKOffJ/qm67/xs4j3Lc+B6l135Qm67qeP824KT6/lGNZfEoyr7rbMqJ51HA6hHqeRPwhfr+cTT2F7WeS4EXA5sAm1Kufl1Mn+Ne5/y65z2grjcB36CcWO8PPHKc9W+IZT5W+8bM911gM+DRvV6NcvvX7WHLuly+x4D97TjfizFjEuCzwKt7pP8B5UrOKP+PZcDX+ky7vOP9B4F/qO83mOT2MVUX5ao9wHOB/xh3HZvVHy6Zhg0iYovM/FFnYpQhDMN4fmY+p+PzFyPiosx8TkQ84IdTslxS/3BEfAX4dkR8gnKJIcrk3LjH/B/eI+2+tvdKrD3Yf0/5B+4B7EnpDTok6+WXPuW+WNuzziRKQNbL6ykB2LHAFzPzrogYOBYnIq5o1DXMsh9qvE9ErKYcyI8F/iYz74iI6zPzl82ZZ544zPy7fIzS+/CazFxZ6x9mWbyz3ZR8X2Y+vSt9nOX+35n59og4G/h0RJwIHF7Xy2bZiFgM/CnwN8B3gFdlY7hDRHwb2JjSM/4HmXltXe43DGjjVPntgLdT1uGjgDdm5n/3yf6kzDy/OzEz/yMiWg+RX9yrPZl5Q/Qeo7s/8IzsuOSYmddFxKspB7nD+9SzcUQszsy7OxNrHa1ey7HWXcqB9TnAWkqQ0ncoT7UX8MzM/HXtlb4JeNqg/1VmHh4Rv6x1BLBrZq5qfJ/pbPPXUzokRrEJ5cpTr97rBAYO4crM/4qIqyknFM8GfrNP1idm5m8BRMQ/ATcDj8/+l6eXUU42741yJfIWynrc89J0l87t4CWUKyrU9aPf//pWylXGdwDfyMyMiFcMqKfz0vVLKT11ZOYtEc0LAn8JvKJr/Tkvyjj1UylXb7tln/e9Pt8/oQyZ/HBEPJFyFe/ciLgReH9mXtarzKArhJn5xkm1b8x8T+X+barXPJ7Yc+aZJ0fEryknKz8EfjsHDxMY53uNHJNUv5WZr1qnAZmfi4j3N8qtIzNX9rlSCg9cbrtSOm2o21q/FXec7QPg7qxDdTLzgojod4VioIUSMH8AODMi3kK5zAhlx/gB+l/S6fTI6BhjFWUM5tQ/cp2xMlHGRh9KCQY+nvXUpOHWiNgpMy/sms/UAbGXiykH2YPqQfor9dLEJyLixszsdzmt9X37TduSsjPdj7Lj+iqwUa8AocvAIRQT8lnK+KU/Au6JiNMZYscVEc2fVc/M3++RvCXwh8BRUW4cOg0Y5ga8O3ukPQI4kHKi8r4+dY2z3MnMr0XEsynryNcj4o9b+SPiIEpv47nAHkMGvT+iXNregnJp/FqGW+5Po2wbOwD/AByYdRx+wx2Nab2W7ZS7o8f4yChj4Hotw+wVAGXmrxqBCpTA7JMRcXBm3lnreCSl17cVtI287kbEXwB/S9l/HTjE/gXg11PfKzNvj4hrB/2PO06ug/L/XQV8aOp41Gf76LXNB7AN9aDWcFdm3jggT7cbM3PgsLpeoo5dpGxfj6D0MP/+1MGxh/uC2My8JyJWN4JlgP+qJ6rUE5XrhgyWAW6qwz1WA88CvlzbvBH99zeH1e/zCeCUiPjMEPX8NCJeDqwBXkDZH02dPPc90QM2bpyI9uoUAnhslLHF0fEe7l+/muqJ6+m1XfsDTwEu65O9c5z3e4B3DZo/sHUNtKPj/VT7thqi/CiuzsxnjlKg42Q0KAHtYygnKQHcm5nP6FP0ifVYFx3vqZ/73bw7TkwC7X1xa9o6aodmv33beRFxGuWkdTPKFZyp4Uv9tslxtg944Lq6zufsMR6+nwVx0x9ARLyMEsQ+rSZdCRyZmV8aouyelBvsfsD9K9lfUS7J/Xl23DQYZXzaDcCbM/OWIdu2EyXw+hT3b+jLgNdSBqNf0KPMqsx8cp/5/XlmfrLPtL43VwzZ1g0pB8X9KGOczs3M14w7vx7z7zxIv5AyXvc+fQ7SUzfE7VLbtSflEuGBwJnZ56atOi7pJsqB8gK6zvYzc50xqRHxccoNDN+IiK0pgc7UAffzOcSNHPUM9ZDavtMoN6Ctc4NNlLGK36Kc5C1iiOUeEZd274gj4gDKMI2NMrPnVYQaDN5K2Rl2btRTV0a6e7+nym1Cufy9H7AdZbnv3r2j7SpzD2W5nwmsEyj36v2JiFvpPcY7KJcA+43R3IcSlL+fB25bhwJvzcwvdOU/l9JzdW5X+ksoQzJe3KeexZTe59dTxh8DPB44HvhfjZ7zXuvuJpR146xe625t43591pmXZ+YZPdJ/ygO3pd+pn6f+v+tsVxHxovp2I8r/NilB86+g9/bRVf6ZlJt//pDSe/y5zOw7djcifkEZCz4lKUN8vpEdNyp1lbmTciPUN7vSXwDckpk/6FPuW5Qg6F8p2/PAG+nqensn9+8nNqIMI+l59bD2zE/1xgflnpNV9X0rwCHKOPP3Uk6aP56ZX6npLwaenZl9Oz46emKntsl3UfZN/9kj71MoJ3WPAz6cmZ+q6btTlutb+tRxcWY+e5RpEdEMWjPzPQO+z96U/caplP36r1rz6yi/zj6xT74DBrSv5xXJrmDqzZQxuJ3let1gOFSbuso8oVcy9WQ0M/fsU+5FvdI72tfrODdyTFLLrabr+3e0803Z48bEiPgo6wbGjwaeDxySmV/sUSYox94tgdMyc01NfyHwqcx8Uq/21Tzd28c7KUOR1tk+av6x1tue81ooAfN01UBxatD69/v1LETE72bmv48x/8cCb6HsVKEE5z0DqZp/5DvKu8tFxOcyc+Dj3aJcTvwL4MmU8TwnZObdtSdhn+zzBIWIuIP+l2fXOcDUMtM6SNd5PATYnbJB7J6Zm/fJt4j7e3CfTgngTsnMq3rlr2UOoWxsW1J2KKdk5qVRhhfsl5l9b/CKcjf1myk3oZ4IfCS77njuyv9Byk7jqZRxa9+kBNDfpYznXeeyZ0T8VWauc9ND3ZG8rl9vXJ+d8X2G6fmrPQKvpiyfx/faOdZ8f0r7Euw6B6ZxD2a17DMo29bUE0yuBj6Ymd/tkXcHyljVb/DAA8ULgL37rRu152U18FPKdrILZXjB94B3Z4875fvMZ+C6GxHfr9Nu6Er/M+DtvQ4W42xXtS1HUG42mzrJ3oZyEH1br5OAGoDtV18/poyB/5vMbK5ftWyvA9OjKcvj3Zm5zglTRJxHOahe0ZX+W5QTn55DPCLid4CvD9k7P5ZxA5wJt+FplJOWV/fqYImIbbLP0wj6nXzVaZ0nAw+YRBm68ohpNLu7rnspx53TgZ/Tte8Y1MM37rFyhPaNHFBFxJ9OnZyMWedIJ6Md5ZbUNrV6iafybkHpGJzqZLyK8gSKvk9PGXNZdO/b7wV+Alw0ZDu7l8W/ZeY6N7hHxJOBLTpPrut+4iPAizJzxh/XuSAC5hhi/OgQ83g+5Uah+4ah9AoU6wrTb6H0rKv2Tr2fBx6YHk95OsHb+xyY+p3JTVXUc1rnme0IZ96foVyO/DrwMuCGzHzToHLjGPMgvTflhseP188XAI+tk9/ZK7DsMY8NKQf5DwDvGbTzqQfDfetrI0ov9SmNs9QPUHpij6P0GA18VFlH2YdSgrbnA8+rr59lZr+xllPlxtqpTkJEPGGYIHu+qTvVx1Eu93YG2N8Hbm70WF4C/G5m3laDsVOBvwZ2BH4ze4zrG6Ith2Xm/+6RvifwYWCvzLx2Ki/lf/2yzFzdo8w429XRlKFnb87MO2raxpShW7/stQ+owc3XKUNFVtW06zKz55jMYdQTzX/vFfREvZekT7krso457jFt5GNCv46DIb/DqL3t4wwXG1lEfI8eQ7BaJ191+sgn2OMehyPi3bRPsJs9fMMGzLO1zMeta9yT0dob+07K/mgDyknN3cBHWx08HeWHDrLHEeUq4FYdx+8LKUN0Evi77HpUZ80z8rKIiDMoJ6rdJ9dPp5xc9xxCOon48b55LZCAuddlpfvGj2Zmv4HlU+VPpj7CjfsvIWf2vnTcq66HU59y0KuuemB6FPA/exyYfpWZh/QoczNlfGrPAe79diJdPczD7kjuO/DU4P7CmTpjH/Mg/U3KZaKb6ufLKDfKPAL458x8SaO+DSk3RO1HOSFaQTkQrhmhzc+knNw8vd9Zag0k7qLsqHoNeeg37o8owx6eR+nlfB5l2MMVmfm6HnnH3amOczVgrAPMmAeLWamrsVMd1GP53ayX2KMM21mbme+uny/LzB1b7egzz77PLY4yROQfKeOfX095lORe2eeKxZjb1bXAU7p7YeuVme9l5nY9yuxDOYl8AWXc7anAP2Vmv7GSQ+l3ch/toWmtaePsp7s7Dm7stW/uyD+d3vZxhouNsw2PfPI1rukeh0esq3NZPJwydAbay2LkZV7LjXPyNc7/d6yT0ShDRl4GLM86tCnKsIRjgS9n5tE9ygRlKM9BlOGAUGKfZpA95rIY+fg9zrKYxsn1xNbbBREwd4ohx492lbkG2L77wDGpusY8MI07JKNzHN7UGDxo70geUNe4dQ/ZvnGWxQM2hIj4WGYeXN9/JzN7Pqc3Ik6iXG46Czg1M68coZ2LKTuhfSkb9/mUHubTh53HEHUcR+nlvIOyU/0O8J1+QVEtM/EevkZd4x5gxjlYzEpd09ipXkl5vNjdtddueWZ+bWpaZj6tV7mWiLgp+wxrqdNfSHnG9Lcol9xbPyYwznb1n5n5lD7z6zutTn8EZczpfpS72E+ijKP9Sr8yjXm9mDIOfNce004BzsuuezYi4vXASzOz73PYO/IOu58eqeNgOttijDFcbFyjnnzVMiMH513lhz4OT7KHb0Cbxlrm4wRU49Q17sloRFxK2RZ+3JW+hPJjWb1OREcOsmuecZbFyMfvcZZFlBud19nP1Wl9T6678o0cPz5Ajvk8utl+UcbCHU65HPZuYLMRyv4rsOVM1QX856jTGOEZxRNYdvdQxo79nBK83d3x/ucTrmucZbGqUeYHjWn31u9wR8f3a34vyk7uBMojolZQemIeMUPL/cvc/9D+5cBvUU9SG2X2oew8bqI8iuolTOg50T3qWkR5pOGJlMccHQ7sMBPlZqsu4NrGtNZ69nbKGPPTaz1TnQlPBr455vLt91z0qfX1DspTeu4cYr0dZ7v6AvDaHul/AqwY4XtsVtffcwfku4J1n8G8mvIoqKf2KbMF5YThfO7/4YP/oDz68XED6ht1P31J63OP/BPZFoENKY97XAscPM66NEQdL6T0gq+g8VzuCdQz8nGYcv9B9+udlJtrfzFD7RxrmVOuFL+jfr+/Bx476booAehrKM+Gv5MSxO7WyH/lqNPqPmzzHulLGP5Z00MtC8Y8fo+6LCidJn/eI/31wGcmvd72nM9MrKyTflHGpf4AeCtjPPSc8kMVt1MeeL1i6jWpuhjjwETj4eYL+TXmsvh0nw3hDZSz9km277y6gY21wYxRX1B6wZdTAueVwFco46xb5UbaqU6gneMeYEYuN5N1TXOnujPwCjpOoChjoZ/VKNN9svaAE9MJ/n/G2a62ovTKn88Dg9ELKWMOJ70OPaHr9XiGPBkFXkwZo/nXlGdFD8o/zn56rI6DcbfFur6+ktJhcxHwvya93Bnj5GsadU3rOFznMXJAOuL8x1rmjHciMO3/L0OcjNI4ses3jTGC7HGXBRM6fg9aFox5cj2J9XbqtSCGZExn/Ggt/6Je6dl/nNFIdUXEVpRntf6KB96ZvxHl4fBDj6dd6MZZFlGeMPIFynLvfM72hpSnePyou8xCE+URdi+g3Pj3csrlrU2HLLsZ5WajP8rGeO4x2zXWGPBxys1GXVHuDP88JXjoXP8eSln/hnpU5HwznX1MROxKxw2Q2fXIvYVouseEadQ71LY4neFi89V0lnmM+IShMds31jKPMW7ons3/b8cwzHUmUa4mrPNc79awywHTxlkWs3r8rkO87nvyRzZ+6K3mn9i+YkEEzAvF+nhgGtc4y6KrzMANYb6LiDdSAuTnU242+lbH64qsP4owV6ZxgBm53GzWVcuNtFNdKNzHLAz1ID0V5MxaQD8fjROEjVnPWMt8zE6yef3/HSfIruWmc1K0Xh2/e1mvA+aI+EZm/naPGxzmxUqt9VtEfIj67OXMvHmu29NtmgeYkcrNZl2S5o+5uhogTdp6HTBLkiRJ07XBXDdgNkTEk+o4SCJil4h4Y0RsOsfNkiRJ0gLwoAiYgc8B90T5FbDjKL+Q9S9z2yRJkiQtBA+WgPneLD+B+grKL938LbDlHLdJkiRJC8CDJWD+74jYDzgAOKOm9bxLVJIkSer0YAmYXwc8DzgiM6+PiG2Bk+e4TZIkSVoAHnRPyagPnt8mMy+f67ZIkiRp/ntQ9DBHxPkRsXH9taFLgE/WZ+RKkiRJTQ+KgBnYJDN/Tvm1oZMy87nA785xmyRJkrQAPFgC5sURsSXwau6/6U+SJEka6MESML8XOBtYlZkXRcQTgWvnuE2SJElaAB50N/1JkiRJo1g81w2YSRHxd5n5DxHxUWCdM4PMfOMcNEuSJEkLyHodMAPX1L8r57QVkiRJWrAckiFJkiQ1rNc9zBGxojU9M39/ttoiSZKkhWm9DpgpP4d9E3AKcAEQc9scSZIkLTTr9ZCMiFgEvBTYD3g6cCZwSmZeNacNkyRJ0oKxXj+HOTPvycwvZ+YBwM7AKuD8iDh4jpsmSZKkBWJ9H5JBRGwI7EXpZV4KHAN8fi7bJEmSpIVjfR+ScRLwNOAs4NTMvHKOmyRJkqQFZn0PmO8F7qwfO79oAJmZG89+qyRJkrSQrNcBsyRJkjRd6/VNf5IkSdJ0GTBLkiRJDQbMkiRJUoMBsyRJktTw/wGCrteCVyfIfgAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.rcParams['figure.figsize'] = [12, 5]\n", "fake_df['birth_state'].value_counts().plot(kind='bar', rot=90)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you ever see a graph like this while exploring categorical features, **that’s a good indication you should consider binning that variable if you intend to use it as a feature in your model.**\n", "\n", "Below is the breakdown of the 15 most common categories of `birth_state`:" ] }, { "cell_type": "code", "execution_count": 63, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
birth_state%
NC387419.370
Missing300915.045
NY212710.635
OC12386.190
TX11765.880
FL8864.430
PA5152.575
IL5092.545
OH4822.410
GA4232.115
MI3881.940
VA3531.765
NJ3321.660
WA3161.580
AZ2901.450
\n", "
" ], "text/plain": [ " birth_state %\n", "NC 3874 19.370\n", "Missing 3009 15.045\n", "NY 2127 10.635\n", "OC 1238 6.190\n", "TX 1176 5.880\n", "FL 886 4.430\n", "PA 515 2.575\n", "IL 509 2.545\n", "OH 482 2.410\n", "GA 423 2.115\n", "MI 388 1.940\n", "VA 353 1.765\n", "NJ 332 1.660\n", "WA 316 1.580\n", "AZ 290 1.450" ] }, "execution_count": 63, "metadata": {}, "output_type": "execute_result" } ], "source": [ "show_column_dist(fake_df, 'birth_state', top_n=15)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "North Carolina `NC` is the most common state, which makes sense since this data is for voters in a specific county in `NC`. Then we see lots of missing values `Missing`. New Yorkers `NY` and people born outside the U.S. `OC` also make up a decent portion of the population. The remaining 53 categories are rare labels based on our definition and will introduce a lot of noise into our modeling efforts.\n", "\n", "Let’s group states by [U.S. Census region](https://www.census.gov/geographies/reference-maps/2010/geo/2010-census-regions-and-divisions-of-the-united-states.html) (Northeast, South, Midwest, West). We’ll also group people born in U.S. territories or outside the country into an “`Other`” group, and leave “`Missing`” as its own category.\n", "\n", "We’ll do this by defining our own custom function to translate from state to region, then apply that function to our original variable to get our new feature. Here’s one way you could write a function to check each state and return the desired region/category:" ] }, { "cell_type": "code", "execution_count": 64, "metadata": {}, "outputs": [], "source": [ "## Define function for grouping birth state/country into categories\n", "def get_birth_reg(state):\n", " \n", " # check if U.S. territory or out of country\n", " if state in ['AS', 'GU', 'MP', 'PR', 'VI', 'OC']:\n", " return 'Other'\n", " \n", " # the rest of the categories are based on U.S. Census Bureau regions\n", " elif state in ['CT', 'ME', 'MA', 'NH', 'RI', 'VT',\n", " 'NJ', 'NY', 'PA']:\n", " return 'Northeast'\n", " \n", " elif state in ['DE', 'FL', 'GA', 'MD', 'NC', 'SC', 'VA', \n", " 'DC', 'WV', 'AL', 'KY', 'MS', 'TN', 'AR',\n", " 'LA', 'OK', 'TX']:\n", " return 'South'\n", " \n", " elif state in ['IL', 'IN', 'MI', 'OH', 'WI',\n", " 'IA', 'KS', 'MN', 'MO', 'NE', 'ND', 'SD']:\n", " return 'Midwest'\n", " \n", " elif state in ['AZ', 'CO', 'ID', 'MT', 'NV', 'NM', 'UT',\n", " 'WY', 'AK', 'CA', 'HI', 'OR', 'WA']:\n", " return 'West'\n", " \n", " else:\n", " return 'Missing'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And now to use Pandas’ [apply()](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.apply.html) method to create our new feature:" ] }, { "cell_type": "code", "execution_count": 65, "metadata": {}, "outputs": [], "source": [ "fake_df['birth_reg'] = fake_df['birth_state'].apply(get_birth_reg)" ] }, { "cell_type": "code", "execution_count": 66, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
birth_reg%
South811640.580
Northeast348517.425
Missing301015.050
Midwest244512.225
West15757.875
Other13696.845
\n", "
" ], "text/plain": [ " birth_reg %\n", "South 8116 40.580\n", "Northeast 3485 17.425\n", "Missing 3010 15.050\n", "Midwest 2445 12.225\n", "West 1575 7.875\n", "Other 1369 6.845" ] }, "execution_count": 66, "metadata": {}, "output_type": "execute_result" } ], "source": [ "show_column_dist(fake_df, 'birth_reg')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Much better! We’ve gone from 57 total categories with 53 rare labels to only 6 categories." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.7" } }, "nbformat": 4, "nbformat_minor": 4 }