{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Handling Categorical Data in Python"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### What does Nominal, Ordinal and Continuous features mean?\n",
"\n",
"Categorical features can only take on a limited, and usually fixed, number of possible values. For example, if a dataset is about information related to users, then you will typically find features like country, gender, age group, etc. Alternatively, if the data you're working with is related to products, you will find features like product type, manufacturer, seller and so on.\n",
"\n",
"These are all categorical features in your dataset. These features are typically stored as text values which represent various traits of the observations. For example, gender is described as Male (M) or Female (F), product type could be described as electronics, apparels, food etc.\n",
"\n",
"**Note that these type of features where the categories are only labeled without any order of precedence are called nominal features. Features which have some order associated with them are called ordinal features.** \n",
"\n",
"For example, a feature like economic status, with three categories: low, medium and high, which have an order associated with them.\n",
"\n",
"There are also continuous features. These are numeric variables that have an infinite number of values between any two values. A continuous variable can be numeric or a date/time.\n",
"\n",
"Regardless of what the value is used for, the challenge is determining how to use this data in the analysis because of the following constraints:\n",
"\n",
"- Categorical features may have a very large number of levels, known as high cardinality, (for example, cities or URLs), where most of the levels appear in a relatively small number of instances.\n",
"\n",
"- Many machine learning models, such as regression or SVM, are algebraic. This means that their input must be numerical. To use these models, categories must be transformed into numbers first, before you can apply the learning algorithm on them.\n",
"\n",
"- While some ML packages or libraries might transform categorical data to numeric automatically based on some default embedding method, many other ML packages don’t support such inputs.\n",
"\n",
"- For the machine, categorical data doesn’t contain the same context or information that humans can easily associate and understand. For example, when looking at a feature called ``City`` with three cities ``New York``, ``New Jersey`` and ``New Delhi``, humans can infer that ``New York`` is closely related to ``New Jersey`` as they are from same country, while ``New York`` and ``New Delhi`` are much different. But for the model, ``New York``, ``New Jersey`` and ``New Delhi``, are just three different levels (possible values) of the same feature ``City``. If you don’t specify the additional contextual information, it will be impossible for the model to differentiate between highly different levels.\n",
"\n",
"You therefore are faced with the challenge of figuring out how to turn these text values into numerical values for further processing and unmask lots of interesting information which these features might hide. Typically, any standard work-flow in feature engineering involves some form of transformation of these categorical values into numeric labels and then applying some encoding scheme on these values."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Encoding Categorical Data\n",
"Existing Encoding Methods (all modules for which code is available, see: http://contrib.scikit-learn.org/categorical-encoding/_modules/index.html)\n",
"- category_encoders.backward_difference\n",
"- category_encoders.basen\n",
"- category_encoders.binary\n",
"- category_encoders.hashing\n",
"- category_encoders.helmert\n",
"- category_encoders.leave_one_out\n",
"- category_encoders.one_hot\n",
"- category_encoders.ordinal\n",
"- category_encoders.polynomial\n",
"- category_encoders.sum_coding\n",
"- category_encoders.target_encoder\n",
"\n",
"\n",
"The techniques that you'll cover are the following:\n",
"\n",
"* Replacing values\n",
"* Encoding labels\n",
"* One-Hot encoding\n",
"* Binary encoding\n",
"* Backward difference encoding\n",
"* Polynomial encodings\n",
"* Miscellaneous features"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> ## 1. Replace Values"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"import copy\n",
"%matplotlib inline"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>year</th>\n",
" <th>month</th>\n",
" <th>day</th>\n",
" <th>dep_time</th>\n",
" <th>dep_delay</th>\n",
" <th>arr_time</th>\n",
" <th>arr_delay</th>\n",
" <th>carrier</th>\n",
" <th>tailnum</th>\n",
" <th>flight</th>\n",
" <th>origin</th>\n",
" <th>dest</th>\n",
" <th>air_time</th>\n",
" <th>distance</th>\n",
" <th>hour</th>\n",
" <th>minute</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>2014</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1.0</td>\n",
" <td>96.0</td>\n",
" <td>235.0</td>\n",
" <td>70.0</td>\n",
" <td>AS</td>\n",
" <td>N508AS</td>\n",
" <td>145</td>\n",
" <td>PDX</td>\n",
" <td>ANC</td>\n",
" <td>194.0</td>\n",
" <td>1542</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2014</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>4.0</td>\n",
" <td>-6.0</td>\n",
" <td>738.0</td>\n",
" <td>-23.0</td>\n",
" <td>US</td>\n",
" <td>N195UW</td>\n",
" <td>1830</td>\n",
" <td>SEA</td>\n",
" <td>CLT</td>\n",
" <td>252.0</td>\n",
" <td>2279</td>\n",
" <td>0.0</td>\n",
" <td>4.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2014</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>8.0</td>\n",
" <td>13.0</td>\n",
" <td>548.0</td>\n",
" <td>-4.0</td>\n",
" <td>UA</td>\n",
" <td>N37422</td>\n",
" <td>1609</td>\n",
" <td>PDX</td>\n",
" <td>IAH</td>\n",
" <td>201.0</td>\n",
" <td>1825</td>\n",
" <td>0.0</td>\n",
" <td>8.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>2014</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>28.0</td>\n",
" <td>-2.0</td>\n",
" <td>800.0</td>\n",
" <td>-23.0</td>\n",
" <td>US</td>\n",
" <td>N547UW</td>\n",
" <td>466</td>\n",
" <td>PDX</td>\n",
" <td>CLT</td>\n",
" <td>251.0</td>\n",
" <td>2282</td>\n",
" <td>0.0</td>\n",
" <td>28.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2014</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>34.0</td>\n",
" <td>44.0</td>\n",
" <td>325.0</td>\n",
" <td>43.0</td>\n",
" <td>AS</td>\n",
" <td>N762AS</td>\n",
" <td>121</td>\n",
" <td>SEA</td>\n",
" <td>ANC</td>\n",
" <td>201.0</td>\n",
" <td>1448</td>\n",
" <td>0.0</td>\n",
" <td>34.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" year month day dep_time dep_delay arr_time arr_delay carrier tailnum \\\n",
"0 2014 1 1 1.0 96.0 235.0 70.0 AS N508AS \n",
"1 2014 1 1 4.0 -6.0 738.0 -23.0 US N195UW \n",
"2 2014 1 1 8.0 13.0 548.0 -4.0 UA N37422 \n",
"3 2014 1 1 28.0 -2.0 800.0 -23.0 US N547UW \n",
"4 2014 1 1 34.0 44.0 325.0 43.0 AS N762AS \n",
"\n",
" flight origin dest air_time distance hour minute \n",
"0 145 PDX ANC 194.0 1542 0.0 1.0 \n",
"1 1830 SEA CLT 252.0 2279 0.0 4.0 \n",
"2 1609 PDX IAH 201.0 1825 0.0 8.0 \n",
"3 466 PDX CLT 251.0 2282 0.0 28.0 \n",
"4 121 SEA ANC 201.0 1448 0.0 34.0 "
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_flights = pd.read_csv('https://raw.githubusercontent.com/ismayc/pnwflights14/master/data/flights.csv')\n",
"\n",
"df_flights.head()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>carrier</th>\n",
" <th>tailnum</th>\n",
" <th>origin</th>\n",
" <th>dest</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>AS</td>\n",
" <td>N508AS</td>\n",
" <td>PDX</td>\n",
" <td>ANC</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>US</td>\n",
" <td>N195UW</td>\n",
" <td>SEA</td>\n",
" <td>CLT</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>UA</td>\n",
" <td>N37422</td>\n",
" <td>PDX</td>\n",
" <td>IAH</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>US</td>\n",
" <td>N547UW</td>\n",
" <td>PDX</td>\n",
" <td>CLT</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>AS</td>\n",
" <td>N762AS</td>\n",
" <td>SEA</td>\n",
" <td>ANC</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" carrier tailnum origin dest\n",
"0 AS N508AS PDX ANC\n",
"1 US N195UW SEA CLT\n",
"2 UA N37422 PDX IAH\n",
"3 US N547UW PDX CLT\n",
"4 AS N762AS SEA ANC"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cat_df_flights = df_flights.select_dtypes(include=['object']).copy()\n",
"cat_df_flights.head()"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"248\n"
]
}
],
"source": [
"print(cat_df_flights.isnull().values.sum())"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0\n"
]
}
],
"source": [
"cat_df_flights = cat_df_flights.fillna(cat_df_flights['tailnum'].value_counts().index[0])\n",
"print(cat_df_flights.isnull().values.sum())"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"replace_map = {'carrier': {'AA': 1, 'AS': 2, 'B6': 3, 'DL': 4, 'F9': 5, 'HA': 6, 'OO': 7 , 'UA': 8 , 'US': 9,'VX': 10,'WN': 11}}"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'carrier': {'AA': 1, 'AS': 2, 'B6': 3, 'DL': 4, 'F9': 5, 'HA': 6, 'OO': 7, 'UA': 8, 'US': 9, 'VX': 10, 'WN': 11}}\n"
]
}
],
"source": [
"labels = cat_df_flights['carrier'].astype('category').cat.categories.tolist()\n",
"replace_map_comp = {'carrier' : {k: v for k,v in zip(labels,list(range(1,len(labels)+1)))}}\n",
"\n",
"print(replace_map_comp)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"cat_df_flights_replace = cat_df_flights.copy()"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" carrier tailnum origin dest\n",
"0 2 N508AS PDX ANC\n",
"1 9 N195UW SEA CLT\n",
"2 8 N37422 PDX IAH\n",
"3 9 N547UW PDX CLT\n",
"4 2 N762AS SEA ANC\n"
]
}
],
"source": [
"cat_df_flights_replace.replace(replace_map_comp, inplace=True)\n",
"\n",
"print(cat_df_flights_replace.head())"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"int64\n"
]
}
],
"source": [
"print(cat_df_flights_replace['carrier'].dtypes)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"carrier category\n",
"tailnum object\n",
"origin category\n",
"dest object\n",
"dtype: object\n"
]
}
],
"source": [
"cat_df_flights_lc = cat_df_flights.copy()\n",
"cat_df_flights_lc['carrier'] = cat_df_flights_lc['carrier'].astype('category')\n",
"cat_df_flights_lc['origin'] = cat_df_flights_lc['origin'].astype('category') \n",
"\n",
"print(cat_df_flights_lc.dtypes)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"31.3 ms ± 1.07 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
]
}
],
"source": [
"import time\n",
"%timeit cat_df_flights.groupby(['origin','carrier']).count() #DataFrame with object dtype columns"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"21.6 ms ± 294 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
]
}
],
"source": [
"%timeit cat_df_flights_lc.groupby(['origin','carrier']).count() #DataFrame with category dtype columns"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> ## 2. Label Encoding"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>carrier</th>\n",
" <th>tailnum</th>\n",
" <th>origin</th>\n",
" <th>dest</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>AS</td>\n",
" <td>N508AS</td>\n",
" <td>PDX</td>\n",
" <td>ANC</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>US</td>\n",
" <td>N195UW</td>\n",
" <td>SEA</td>\n",
" <td>CLT</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>UA</td>\n",
" <td>N37422</td>\n",
" <td>PDX</td>\n",
" <td>IAH</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>US</td>\n",
" <td>N547UW</td>\n",
" <td>PDX</td>\n",
" <td>CLT</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>AS</td>\n",
" <td>N762AS</td>\n",
" <td>SEA</td>\n",
" <td>ANC</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" carrier tailnum origin dest\n",
"0 AS N508AS PDX ANC\n",
"1 US N195UW SEA CLT\n",
"2 UA N37422 PDX IAH\n",
"3 US N547UW PDX CLT\n",
"4 AS N762AS SEA ANC"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cat_df_flights_lc.head()"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [],
"source": [
"cat_df_flights_lc['carrier'] = cat_df_flights_lc['carrier'].cat.codes"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>carrier</th>\n",
" <th>tailnum</th>\n",
" <th>origin</th>\n",
" <th>dest</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>N508AS</td>\n",
" <td>PDX</td>\n",
" <td>ANC</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>8</td>\n",
" <td>N195UW</td>\n",
" <td>SEA</td>\n",
" <td>CLT</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>7</td>\n",
" <td>N37422</td>\n",
" <td>PDX</td>\n",
" <td>IAH</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>8</td>\n",
" <td>N547UW</td>\n",
" <td>PDX</td>\n",
" <td>CLT</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>1</td>\n",
" <td>N762AS</td>\n",
" <td>SEA</td>\n",
" <td>ANC</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" carrier tailnum origin dest\n",
"0 1 N508AS PDX ANC\n",
"1 8 N195UW SEA CLT\n",
"2 7 N37422 PDX IAH\n",
"3 8 N547UW PDX CLT\n",
"4 1 N762AS SEA ANC"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cat_df_flights_lc.head() #alphabetically labeled from 0 to 10"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>carrier</th>\n",
" <th>tailnum</th>\n",
" <th>origin</th>\n",
" <th>dest</th>\n",
" <th>US_code</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>AS</td>\n",
" <td>N508AS</td>\n",
" <td>PDX</td>\n",
" <td>ANC</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>US</td>\n",
" <td>N195UW</td>\n",
" <td>SEA</td>\n",
" <td>CLT</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>UA</td>\n",
" <td>N37422</td>\n",
" <td>PDX</td>\n",
" <td>IAH</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>US</td>\n",
" <td>N547UW</td>\n",
" <td>PDX</td>\n",
" <td>CLT</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>AS</td>\n",
" <td>N762AS</td>\n",
" <td>SEA</td>\n",
" <td>ANC</td>\n",
" <td>0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" carrier tailnum origin dest US_code\n",
"0 AS N508AS PDX ANC 0\n",
"1 US N195UW SEA CLT 1\n",
"2 UA N37422 PDX IAH 0\n",
"3 US N547UW PDX CLT 1\n",
"4 AS N762AS SEA ANC 0"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cat_df_flights_specific = cat_df_flights.copy()\n",
"cat_df_flights_specific['US_code'] = np.where(cat_df_flights_specific['carrier'].str.contains('US'), 1, 0)\n",
"\n",
"cat_df_flights_specific.head()"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>carrier</th>\n",
" <th>tailnum</th>\n",
" <th>origin</th>\n",
" <th>dest</th>\n",
" <th>carrier_code</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>AS</td>\n",
" <td>N508AS</td>\n",
" <td>PDX</td>\n",
" <td>ANC</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>US</td>\n",
" <td>N195UW</td>\n",
" <td>SEA</td>\n",
" <td>CLT</td>\n",
" <td>8</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>UA</td>\n",
" <td>N37422</td>\n",
" <td>PDX</td>\n",
" <td>IAH</td>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>US</td>\n",
" <td>N547UW</td>\n",
" <td>PDX</td>\n",
" <td>CLT</td>\n",
" <td>8</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>AS</td>\n",
" <td>N762AS</td>\n",
" <td>SEA</td>\n",
" <td>ANC</td>\n",
" <td>1</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" carrier tailnum origin dest carrier_code\n",
"0 AS N508AS PDX ANC 1\n",
"1 US N195UW SEA CLT 8\n",
"2 UA N37422 PDX IAH 7\n",
"3 US N547UW PDX CLT 8\n",
"4 AS N762AS SEA ANC 1"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cat_df_flights_sklearn = cat_df_flights.copy()\n",
"\n",
"from sklearn.preprocessing import LabelEncoder\n",
"\n",
"lb_make = LabelEncoder()\n",
"cat_df_flights_sklearn['carrier_code'] = lb_make.fit_transform(cat_df_flights['carrier'])\n",
"\n",
"cat_df_flights_sklearn.head() #Results in appending a new column to df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Label encoding is pretty much intuitive and straight-forward and may give you a good performance from your learning algorithm, but it has as disadvantage that the numerical values can be misinterpreted by the algorithm. Should the carrier US (encoded to 8) be given 8x more weight than the carrier AS (encoded to 1) ?**\n",
"\n",
"**To solve this issue there is another popular way to encode the categories via something called one-hot encoding.**\n",
"\n",
"> ## 3. One-Hot encoding"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>carrier</th>\n",
" <th>tailnum</th>\n",
" <th>origin</th>\n",
" <th>dest</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>AS</td>\n",
" <td>N508AS</td>\n",
" <td>PDX</td>\n",
" <td>ANC</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>US</td>\n",
" <td>N195UW</td>\n",
" <td>SEA</td>\n",
" <td>CLT</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>UA</td>\n",
" <td>N37422</td>\n",
" <td>PDX</td>\n",
" <td>IAH</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>US</td>\n",
" <td>N547UW</td>\n",
" <td>PDX</td>\n",
" <td>CLT</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>AS</td>\n",
" <td>N762AS</td>\n",
" <td>SEA</td>\n",
" <td>ANC</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" carrier tailnum origin dest\n",
"0 AS N508AS PDX ANC\n",
"1 US N195UW SEA CLT\n",
"2 UA N37422 PDX IAH\n",
"3 US N547UW PDX CLT\n",
"4 AS N762AS SEA ANC"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cat_df_flights.head()"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>tailnum</th>\n",
" <th>origin</th>\n",
" <th>dest</th>\n",
" <th>carrier_AA</th>\n",
" <th>carrier_AS</th>\n",
" <th>carrier_B6</th>\n",
" <th>carrier_DL</th>\n",
" <th>carrier_F9</th>\n",
" <th>carrier_HA</th>\n",
" <th>carrier_OO</th>\n",
" <th>carrier_UA</th>\n",
" <th>carrier_US</th>\n",
" <th>carrier_VX</th>\n",
" <th>carrier_WN</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>N508AS</td>\n",
" <td>PDX</td>\n",
" <td>ANC</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>N195UW</td>\n",
" <td>SEA</td>\n",
" <td>CLT</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>N37422</td>\n",
" <td>PDX</td>\n",
" <td>IAH</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>N547UW</td>\n",
" <td>PDX</td>\n",
" <td>CLT</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>N762AS</td>\n",
" <td>SEA</td>\n",
" <td>ANC</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" tailnum origin dest carrier_AA carrier_AS carrier_B6 carrier_DL \\\n",
"0 N508AS PDX ANC 0 1 0 0 \n",
"1 N195UW SEA CLT 0 0 0 0 \n",
"2 N37422 PDX IAH 0 0 0 0 \n",
"3 N547UW PDX CLT 0 0 0 0 \n",
"4 N762AS SEA ANC 0 1 0 0 \n",
"\n",
" carrier_F9 carrier_HA carrier_OO carrier_UA carrier_US carrier_VX \\\n",
"0 0 0 0 0 0 0 \n",
"1 0 0 0 0 1 0 \n",
"2 0 0 0 1 0 0 \n",
"3 0 0 0 0 1 0 \n",
"4 0 0 0 0 0 0 \n",
"\n",
" carrier_WN \n",
"0 0 \n",
"1 0 \n",
"2 0 \n",
"3 0 \n",
"4 0 "
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cat_df_flights_onehot = cat_df_flights.copy()\n",
"cat_df_flights_onehot = pd.get_dummies(cat_df_flights_onehot, columns=['carrier'], prefix = ['carrier'])\n",
"\n",
"cat_df_flights_onehot.head()"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>AA</th>\n",
" <th>AS</th>\n",
" <th>B6</th>\n",
" <th>DL</th>\n",
" <th>F9</th>\n",
" <th>HA</th>\n",
" <th>OO</th>\n",
" <th>UA</th>\n",
" <th>US</th>\n",
" <th>VX</th>\n",
" <th>WN</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" AA AS B6 DL F9 HA OO UA US VX WN\n",
"0 0 1 0 0 0 0 0 0 0 0 0\n",
"1 0 0 0 0 0 0 0 0 1 0 0\n",
"2 0 0 0 0 0 0 0 1 0 0 0\n",
"3 0 0 0 0 0 0 0 0 1 0 0\n",
"4 0 1 0 0 0 0 0 0 0 0 0"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cat_df_flights_onehot_sklearn = cat_df_flights.copy()\n",
"\n",
"from sklearn.preprocessing import LabelBinarizer\n",
"\n",
"lb = LabelBinarizer()\n",
"lb_results = lb.fit_transform(cat_df_flights_onehot_sklearn['carrier'])\n",
"lb_results_df = pd.DataFrame(lb_results, columns=lb.classes_)\n",
"\n",
"lb_results_df.head()"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>carrier</th>\n",
" <th>tailnum</th>\n",
" <th>origin</th>\n",
" <th>dest</th>\n",
" <th>AA</th>\n",
" <th>AS</th>\n",
" <th>B6</th>\n",
" <th>DL</th>\n",
" <th>F9</th>\n",
" <th>HA</th>\n",
" <th>OO</th>\n",
" <th>UA</th>\n",
" <th>US</th>\n",
" <th>VX</th>\n",
" <th>WN</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>AS</td>\n",
" <td>N508AS</td>\n",
" <td>PDX</td>\n",
" <td>ANC</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>US</td>\n",
" <td>N195UW</td>\n",
" <td>SEA</td>\n",
" <td>CLT</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>UA</td>\n",
" <td>N37422</td>\n",
" <td>PDX</td>\n",
" <td>IAH</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>US</td>\n",
" <td>N547UW</td>\n",
" <td>PDX</td>\n",
" <td>CLT</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>AS</td>\n",
" <td>N762AS</td>\n",
" <td>SEA</td>\n",
" <td>ANC</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" carrier tailnum origin dest AA AS B6 DL F9 HA OO UA US VX WN\n",
"0 AS N508AS PDX ANC 0 1 0 0 0 0 0 0 0 0 0\n",
"1 US N195UW SEA CLT 0 0 0 0 0 0 0 0 1 0 0\n",
"2 UA N37422 PDX IAH 0 0 0 0 0 0 0 1 0 0 0\n",
"3 US N547UW PDX CLT 0 0 0 0 0 0 0 0 1 0 0\n",
"4 AS N762AS SEA ANC 0 1 0 0 0 0 0 0 0 0 0"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"result_df = pd.concat([cat_df_flights_onehot_sklearn, lb_results_df], axis=1)\n",
"\n",
"result_df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> ## 4. Binary Encoding"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>carrier_0</th>\n",
" <th>carrier_1</th>\n",
" <th>carrier_2</th>\n",
" <th>carrier_3</th>\n",
" <th>carrier_4</th>\n",
" <th>tailnum</th>\n",
" <th>origin</th>\n",
" <th>dest</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>N508AS</td>\n",
" <td>PDX</td>\n",
" <td>ANC</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>N195UW</td>\n",
" <td>SEA</td>\n",
" <td>CLT</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>N37422</td>\n",
" <td>PDX</td>\n",
" <td>IAH</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>N547UW</td>\n",
" <td>PDX</td>\n",
" <td>CLT</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>N762AS</td>\n",
" <td>SEA</td>\n",
" <td>ANC</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" carrier_0 carrier_1 carrier_2 carrier_3 carrier_4 tailnum origin dest\n",
"0 0 0 0 0 1 N508AS PDX ANC\n",
"1 0 0 0 1 0 N195UW SEA CLT\n",
"2 0 0 0 1 1 N37422 PDX IAH\n",
"3 0 0 0 1 0 N547UW PDX CLT\n",
"4 0 0 0 0 1 N762AS SEA ANC"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cat_df_flights_ce = cat_df_flights.copy()\n",
"\n",
"import category_encoders as ce\n",
"\n",
"encoder = ce.BinaryEncoder(cols=['carrier'])\n",
"df_binary = encoder.fit_transform(cat_df_flights_ce)\n",
"\n",
"df_binary.head()"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>carrier_0</th>\n",
" <th>carrier_1</th>\n",
" <th>carrier_2</th>\n",
" <th>carrier_3</th>\n",
" <th>carrier_4</th>\n",
" <th>tailnum</th>\n",
" <th>origin</th>\n",
" <th>dest</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
"Empty DataFrame\n",
"Columns: [carrier_0, carrier_1, carrier_2, carrier_3, carrier_4, tailnum, origin, dest]\n",
"Index: []"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_binary[df_binary['carrier_0']==1]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> ## 5. Backward Difference Encoding"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>col_carrier_0</th>\n",
" <th>col_carrier_1</th>\n",
" <th>col_carrier_2</th>\n",
" <th>col_carrier_3</th>\n",
" <th>col_carrier_4</th>\n",
" <th>col_carrier_5</th>\n",
" <th>col_carrier_6</th>\n",
" <th>col_carrier_7</th>\n",
" <th>col_carrier_8</th>\n",
" <th>col_carrier_9</th>\n",
" <th>col_carrier_10</th>\n",
" <th>col_tailnum</th>\n",
" <th>col_origin</th>\n",
" <th>col_dest</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1.0</td>\n",
" <td>-0.909091</td>\n",
" <td>-0.818182</td>\n",
" <td>-0.727273</td>\n",
" <td>-0.636364</td>\n",
" <td>-0.545455</td>\n",
" <td>-0.454545</td>\n",
" <td>-0.363636</td>\n",
" <td>-0.272727</td>\n",
" <td>-0.181818</td>\n",
" <td>-0.090909</td>\n",
" <td>N508AS</td>\n",
" <td>PDX</td>\n",
" <td>ANC</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>1.0</td>\n",
" <td>0.090909</td>\n",
" <td>-0.818182</td>\n",
" <td>-0.727273</td>\n",
" <td>-0.636364</td>\n",
" <td>-0.545455</td>\n",
" <td>-0.454545</td>\n",
" <td>-0.363636</td>\n",
" <td>-0.272727</td>\n",
" <td>-0.181818</td>\n",
" <td>-0.090909</td>\n",
" <td>N195UW</td>\n",
" <td>SEA</td>\n",
" <td>CLT</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>1.0</td>\n",
" <td>0.090909</td>\n",
" <td>0.181818</td>\n",
" <td>-0.727273</td>\n",
" <td>-0.636364</td>\n",
" <td>-0.545455</td>\n",
" <td>-0.454545</td>\n",
" <td>-0.363636</td>\n",
" <td>-0.272727</td>\n",
" <td>-0.181818</td>\n",
" <td>-0.090909</td>\n",
" <td>N37422</td>\n",
" <td>PDX</td>\n",
" <td>IAH</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1.0</td>\n",
" <td>0.090909</td>\n",
" <td>-0.818182</td>\n",
" <td>-0.727273</td>\n",
" <td>-0.636364</td>\n",
" <td>-0.545455</td>\n",
" <td>-0.454545</td>\n",
" <td>-0.363636</td>\n",
" <td>-0.272727</td>\n",
" <td>-0.181818</td>\n",
" <td>-0.090909</td>\n",
" <td>N547UW</td>\n",
" <td>PDX</td>\n",
" <td>CLT</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>1.0</td>\n",
" <td>-0.909091</td>\n",
" <td>-0.818182</td>\n",
" <td>-0.727273</td>\n",
" <td>-0.636364</td>\n",
" <td>-0.545455</td>\n",
" <td>-0.454545</td>\n",
" <td>-0.363636</td>\n",
" <td>-0.272727</td>\n",
" <td>-0.181818</td>\n",
" <td>-0.090909</td>\n",
" <td>N762AS</td>\n",
" <td>SEA</td>\n",
" <td>ANC</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" col_carrier_0 col_carrier_1 col_carrier_2 col_carrier_3 col_carrier_4 \\\n",
"0 1.0 -0.909091 -0.818182 -0.727273 -0.636364 \n",
"1 1.0 0.090909 -0.818182 -0.727273 -0.636364 \n",
"2 1.0 0.090909 0.181818 -0.727273 -0.636364 \n",
"3 1.0 0.090909 -0.818182 -0.727273 -0.636364 \n",
"4 1.0 -0.909091 -0.818182 -0.727273 -0.636364 \n",
"\n",
" col_carrier_5 col_carrier_6 col_carrier_7 col_carrier_8 col_carrier_9 \\\n",
"0 -0.545455 -0.454545 -0.363636 -0.272727 -0.181818 \n",
"1 -0.545455 -0.454545 -0.363636 -0.272727 -0.181818 \n",
"2 -0.545455 -0.454545 -0.363636 -0.272727 -0.181818 \n",
"3 -0.545455 -0.454545 -0.363636 -0.272727 -0.181818 \n",
"4 -0.545455 -0.454545 -0.363636 -0.272727 -0.181818 \n",
"\n",
" col_carrier_10 col_tailnum col_origin col_dest \n",
"0 -0.090909 N508AS PDX ANC \n",
"1 -0.090909 N195UW SEA CLT \n",
"2 -0.090909 N37422 PDX IAH \n",
"3 -0.090909 N547UW PDX CLT \n",
"4 -0.090909 N762AS SEA ANC "
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"encoder = ce.BackwardDifferenceEncoder(cols=['carrier'])\n",
"df_bd = encoder.fit_transform(cat_df_flights_ce)\n",
"\n",
"df_bd.head()"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([-0.90909091, 0.09090909])"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.unique(df_bd['col_carrier_1'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> ## 6. Polynomial Encoding"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>col_carrier_0</th>\n",
" <th>col_carrier_1</th>\n",
" <th>col_carrier_2</th>\n",
" <th>col_carrier_3</th>\n",
" <th>col_carrier_4</th>\n",
" <th>col_carrier_5</th>\n",
" <th>col_carrier_6</th>\n",
" <th>col_carrier_7</th>\n",
" <th>col_carrier_8</th>\n",
" <th>col_carrier_9</th>\n",
" <th>col_carrier_10</th>\n",
" <th>col_tailnum</th>\n",
" <th>col_origin</th>\n",
" <th>col_dest</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1.0</td>\n",
" <td>-0.476731</td>\n",
" <td>0.512092</td>\n",
" <td>-0.458029</td>\n",
" <td>0.354787</td>\n",
" <td>-0.240192</td>\n",
" <td>0.141610</td>\n",
" <td>-0.071707</td>\n",
" <td>0.030334</td>\n",
" <td>-0.010141</td>\n",
" <td>0.002326</td>\n",
" <td>N508AS</td>\n",
" <td>PDX</td>\n",
" <td>ANC</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>1.0</td>\n",
" <td>-0.381385</td>\n",
" <td>0.204837</td>\n",
" <td>0.091606</td>\n",
" <td>-0.354787</td>\n",
" <td>0.480384</td>\n",
" <td>-0.453153</td>\n",
" <td>0.329853</td>\n",
" <td>-0.188069</td>\n",
" <td>0.081127</td>\n",
" <td>-0.023265</td>\n",
" <td>N195UW</td>\n",
" <td>SEA</td>\n",
" <td>CLT</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>1.0</td>\n",
" <td>-0.286039</td>\n",
" <td>-0.034139</td>\n",
" <td>0.335888</td>\n",
" <td>-0.354787</td>\n",
" <td>0.080064</td>\n",
" <td>0.273780</td>\n",
" <td>-0.473267</td>\n",
" <td>0.442872</td>\n",
" <td>-0.273805</td>\n",
" <td>0.104692</td>\n",
" <td>N37422</td>\n",
" <td>PDX</td>\n",
" <td>IAH</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1.0</td>\n",
" <td>-0.381385</td>\n",
" <td>0.204837</td>\n",
" <td>0.091606</td>\n",
" <td>-0.354787</td>\n",
" <td>0.480384</td>\n",
" <td>-0.453153</td>\n",
" <td>0.329853</td>\n",
" <td>-0.188069</td>\n",
" <td>0.081127</td>\n",
" <td>-0.023265</td>\n",
" <td>N547UW</td>\n",
" <td>PDX</td>\n",
" <td>CLT</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>1.0</td>\n",
" <td>-0.476731</td>\n",
" <td>0.512092</td>\n",
" <td>-0.458029</td>\n",
" <td>0.354787</td>\n",
" <td>-0.240192</td>\n",
" <td>0.141610</td>\n",
" <td>-0.071707</td>\n",
" <td>0.030334</td>\n",
" <td>-0.010141</td>\n",
" <td>0.002326</td>\n",
" <td>N762AS</td>\n",
" <td>SEA</td>\n",
" <td>ANC</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" col_carrier_0 col_carrier_1 col_carrier_2 col_carrier_3 col_carrier_4 \\\n",
"0 1.0 -0.476731 0.512092 -0.458029 0.354787 \n",
"1 1.0 -0.381385 0.204837 0.091606 -0.354787 \n",
"2 1.0 -0.286039 -0.034139 0.335888 -0.354787 \n",
"3 1.0 -0.381385 0.204837 0.091606 -0.354787 \n",
"4 1.0 -0.476731 0.512092 -0.458029 0.354787 \n",
"\n",
" col_carrier_5 col_carrier_6 col_carrier_7 col_carrier_8 col_carrier_9 \\\n",
"0 -0.240192 0.141610 -0.071707 0.030334 -0.010141 \n",
"1 0.480384 -0.453153 0.329853 -0.188069 0.081127 \n",
"2 0.080064 0.273780 -0.473267 0.442872 -0.273805 \n",
"3 0.480384 -0.453153 0.329853 -0.188069 0.081127 \n",
"4 -0.240192 0.141610 -0.071707 0.030334 -0.010141 \n",
"\n",
" col_carrier_10 col_tailnum col_origin col_dest \n",
"0 0.002326 N508AS PDX ANC \n",
"1 -0.023265 N195UW SEA CLT \n",
"2 0.104692 N37422 PDX IAH \n",
"3 -0.023265 N547UW PDX CLT \n",
"4 0.002326 N762AS SEA ANC "
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"encoder = ce.PolynomialEncoder(cols=['carrier'])\n",
"df_bd = encoder.fit_transform(cat_df_flights_ce)\n",
"\n",
"df_bd.head()"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([-4.76731295e-01, -3.81385036e-01, -2.86038777e-01, -1.90692518e-01,\n",
" -9.53462589e-02, -1.16688970e-17, 9.53462589e-02, 1.90692518e-01,\n",
" 2.86038777e-01, 3.81385036e-01, 4.76731295e-01])"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.unique(df_bd['col_carrier_1'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> ## 7. Miscellaneous Features"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>age</th>\n",
" <th>start</th>\n",
" <th>end</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0-20</td>\n",
" <td>0</td>\n",
" <td>20</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>20-40</td>\n",
" <td>20</td>\n",
" <td>40</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>40-60</td>\n",
" <td>40</td>\n",
" <td>60</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>60-80</td>\n",
" <td>60</td>\n",
" <td>80</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" age start end\n",
"0 0-20 0 20\n",
"1 20-40 20 40\n",
"2 40-60 40 60\n",
"3 60-80 60 80"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dummy_df_age = pd.DataFrame({'age': ['0-20', '20-40', '40-60','60-80']})\n",
"dummy_df_age['start'], dummy_df_age['end'] = zip(*dummy_df_age['age'].map(lambda x: x.split('-')))\n",
"\n",
"dummy_df_age.head()"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>age</th>\n",
" <th>age_mean</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0-20</td>\n",
" <td>10.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>20-40</td>\n",
" <td>30.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>40-60</td>\n",
" <td>50.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>60-80</td>\n",
" <td>70.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" age age_mean\n",
"0 0-20 10.0\n",
"1 20-40 30.0\n",
"2 40-60 50.0\n",
"3 60-80 70.0"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dummy_df_age = pd.DataFrame({'age': ['0-20', '20-40', '40-60','60-80']})\n",
"\n",
"def split_mean(x):\n",
" split_list = x.split('-')\n",
" mean = (float(split_list[0])+float(split_list[1]))/2\n",
" return mean\n",
"\n",
"dummy_df_age['age_mean'] = dummy_df_age['age'].apply(lambda x: split_mean(x))\n",
"\n",
"dummy_df_age.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Dealing with Categorical Features in Big Data with Spark"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- The first step in Spark programming is to create a SparkContext. SparkContext is required when you want to execute operations in a cluster. SparkContext tells Spark how and where to access a cluster. You'll start by importing SparkContext.\n",
"\n",
"- To start working with Spark DataFrames, you first have to create a SparkSession object from your SparkContext."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> #### 1st way"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#import findspark\n",
"\n",
"#findspark.init()\n",
"\n",
"#import pyspark\n",
"\n",
"#confspark = pyspark.SparkConf().setMaster(\"local[*]\").set(\"spark.cores.max\", \"4\").set(\"spark.executor.memory\", \"2G\").setAppName(\"--test--\")\n",
"\n",
"#sc = pyspark.SparkContext(conf=confspark)\n",
"\n",
"#sc._conf.getAll()\n",
"\n",
"#from pyspark.sql import SparkSession \n",
"\n",
"#spark = SparkSession(sc) \n",
"\n",
"#sc.stop()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> #### 2nd way"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [],
"source": [
"import findspark\n",
"findspark.init()\n",
"\n",
"import pyspark\n",
"from pyspark.sql import SparkSession \n",
"\n",
"#confspark = pyspark.SparkConf().setMaster(\"local[4]\").set(\"spark.cores.max\", \"4\").set(\"spark.executor.memory\", \"2G\").setAppName(\"--test--\")\n",
"#spark = SparkSession.builder.config(conf=confspark).getOrCreate()\n",
"\n",
"spark = SparkSession.builder.master(\"local[*]\").appName(\"--test--\").config(\"spark.some.config.option\", \"some-value\").getOrCreate()"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'2.3.1'"
]
},
"execution_count": 32,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"spark.version"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[]"
]
},
"execution_count": 33,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"spark.catalog.listTables()"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"+----+-----+---+--------+---------+--------+---------+-------+-------+------+------+----+--------+--------+----+------+\n",
"|year|month|day|dep_time|dep_delay|arr_time|arr_delay|carrier|tailnum|flight|origin|dest|air_time|distance|hour|minute|\n",
"+----+-----+---+--------+---------+--------+---------+-------+-------+------+------+----+--------+--------+----+------+\n",
"|2014| 1| 1| 1| 96| 235| 70| AS| N508AS| 145| PDX| ANC| 194| 1542| 0| 1|\n",
"|2014| 1| 1| 4| -6| 738| -23| US| N195UW| 1830| SEA| CLT| 252| 2279| 0| 4|\n",
"|2014| 1| 1| 8| 13| 548| -4| UA| N37422| 1609| PDX| IAH| 201| 1825| 0| 8|\n",
"+----+-----+---+--------+---------+--------+---------+-------+-------+------+------+----+--------+--------+----+------+\n",
"only showing top 3 rows\n",
"\n"
]
}
],
"source": [
"spark_flights = spark.read.format(\"csv\").option('header',True).load('data/flights.csv',inferSchema=True)\n",
"spark_flights.show(3)"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"root\n",
" |-- year: integer (nullable = true)\n",
" |-- month: integer (nullable = true)\n",
" |-- day: integer (nullable = true)\n",
" |-- dep_time: string (nullable = true)\n",
" |-- dep_delay: string (nullable = true)\n",
" |-- arr_time: string (nullable = true)\n",
" |-- arr_delay: string (nullable = true)\n",
" |-- carrier: string (nullable = true)\n",
" |-- tailnum: string (nullable = true)\n",
" |-- flight: integer (nullable = true)\n",
" |-- origin: string (nullable = true)\n",
" |-- dest: string (nullable = true)\n",
" |-- air_time: string (nullable = true)\n",
" |-- distance: integer (nullable = true)\n",
" |-- hour: string (nullable = true)\n",
" |-- minute: string (nullable = true)\n",
"\n"
]
}
],
"source": [
"spark_flights.printSchema()"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[]"
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"spark.catalog.listTables()"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [],
"source": [
"spark_flights.createOrReplaceTempView(\"flights_temp\")"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Table(name='flights_temp', database=None, description=None, tableType='TEMPORARY', isTemporary=True)]"
]
},
"execution_count": 38,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"spark.catalog.listTables()"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"+-------+\n",
"|carrier|\n",
"+-------+\n",
"| AS|\n",
"| US|\n",
"| UA|\n",
"| US|\n",
"| AS|\n",
"+-------+\n",
"only showing top 5 rows\n",
"\n"
]
}
],
"source": [
"carrier_df = spark_flights.select(\"carrier\")\n",
"carrier_df.show(5)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> ## StringIndexer"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"+-------+-------------+\n",
"|carrier|carrier_index|\n",
"+-------+-------------+\n",
"| AS| 0.0|\n",
"| US| 6.0|\n",
"| UA| 4.0|\n",
"| US| 6.0|\n",
"| AS| 0.0|\n",
"| DL| 3.0|\n",
"| UA| 4.0|\n",
"+-------+-------------+\n",
"only showing top 7 rows\n",
"\n"
]
}
],
"source": [
"from pyspark.ml.feature import StringIndexer\n",
"\n",
"carr_indexer = StringIndexer(inputCol=\"carrier\",outputCol=\"carrier_index\")\n",
"carr_indexed = carr_indexer.fit(carrier_df).transform(carrier_df)\n",
"\n",
"carr_indexed.show(7)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> ## OneHotEncoder"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"+-------+-------------+--------------+\n",
"|carrier|carrier_index| carrier_vec|\n",
"+-------+-------------+--------------+\n",
"| AS| 0.0|(11,[0],[1.0])|\n",
"| US| 6.0|(11,[6],[1.0])|\n",
"| UA| 4.0|(11,[4],[1.0])|\n",
"| US| 6.0|(11,[6],[1.0])|\n",
"| AS| 0.0|(11,[0],[1.0])|\n",
"| DL| 3.0|(11,[3],[1.0])|\n",
"| UA| 4.0|(11,[4],[1.0])|\n",
"+-------+-------------+--------------+\n",
"only showing top 7 rows\n",
"\n"
]
}
],
"source": [
"carrier_df_onehot = spark_flights.select(\"carrier\")\n",
"\n",
"from pyspark.ml.feature import OneHotEncoder, StringIndexer\n",
"\n",
"stringIndexer = StringIndexer(inputCol=\"carrier\", outputCol=\"carrier_index\")\n",
"model = stringIndexer.fit(carrier_df_onehot)\n",
"indexed = model.transform(carrier_df_onehot)\n",
"\n",
"encoder = OneHotEncoder(dropLast=False, inputCol=\"carrier_index\", outputCol=\"carrier_vec\")\n",
"encoded = encoder.transform(indexed)\n",
"\n",
"encoded.show(7)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> ## Example"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"+---+--------+-------------+-------------+\n",
"| id|category|categoryIndex| categoryVec|\n",
"+---+--------+-------------+-------------+\n",
"| 0| a| 0.0|(2,[0],[1.0])|\n",
"| 1| b| 2.0| (2,[],[])|\n",
"| 2| c| 1.0|(2,[1],[1.0])|\n",
"| 3| a| 0.0|(2,[0],[1.0])|\n",
"| 4| a| 0.0|(2,[0],[1.0])|\n",
"| 5| c| 1.0|(2,[1],[1.0])|\n",
"+---+--------+-------------+-------------+\n",
"\n"
]
}
],
"source": [
"from pyspark.ml.feature import OneHotEncoder, StringIndexer\n",
"\n",
"df1 = spark.createDataFrame([\n",
" (0, \"a\"),\n",
" (1, \"b\"),\n",
" (2, \"c\"),\n",
" (3, \"a\"),\n",
" (4, \"a\"),\n",
" (5, \"c\"),\n",
" (6, \"a\"),\n",
" (7, \"b\"),\n",
" (8, \"d\"),\n",
" (9, \"d\")\n",
"], [\"id\", \"category\"])\n",
"\n",
"df2 = spark.createDataFrame([\n",
" (0, \"a\"),\n",
" (1, \"b\"),\n",
" (2, \"c\"),\n",
" (3, \"a\"),\n",
" (4, \"a\"),\n",
" (5, \"c\")\n",
"], [\"id\", \"category\"])\n",
"\n",
"df = df2\n",
"\n",
"stringIndexer = StringIndexer(inputCol=\"category\", outputCol=\"categoryIndex\")\n",
"model = stringIndexer.fit(df)\n",
"indexed = model.transform(df)\n",
"\n",
"encoder = OneHotEncoder(dropLast=True, inputCol=\"categoryIndex\", outputCol=\"categoryVec\")\n",
"encoded = encoder.transform(indexed)\n",
"encoded.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> ## VectorIndexer"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chose 351 categorical features: 645, 69, 365, 138, 101, 479, 333, 249, 0, 555, 666, 88, 170, 115, 276, 308, 5, 449, 120, 247, 614, 677, 202, 10, 56, 533, 142, 500, 340, 670, 174, 42, 417, 24, 37, 25, 257, 389, 52, 14, 504, 110, 587, 619, 196, 559, 638, 20, 421, 46, 93, 284, 228, 448, 57, 78, 29, 475, 164, 591, 646, 253, 106, 121, 84, 480, 147, 280, 61, 221, 396, 89, 133, 116, 1, 507, 312, 74, 307, 452, 6, 248, 60, 117, 678, 529, 85, 201, 220, 366, 534, 102, 334, 28, 38, 561, 392, 70, 424, 192, 21, 137, 165, 33, 92, 229, 252, 197, 361, 65, 97, 665, 583, 285, 224, 650, 615, 9, 53, 169, 593, 141, 610, 420, 109, 256, 225, 339, 77, 193, 669, 476, 642, 637, 590, 679, 96, 393, 647, 173, 13, 41, 503, 134, 73, 105, 2, 508, 311, 558, 674, 530, 586, 618, 166, 32, 34, 148, 45, 161, 279, 64, 689, 17, 149, 584, 562, 176, 423, 191, 22, 44, 59, 118, 281, 27, 641, 71, 391, 12, 445, 54, 313, 611, 144, 49, 335, 86, 672, 172, 113, 681, 219, 419, 81, 230, 362, 451, 76, 7, 39, 649, 98, 616, 477, 367, 535, 103, 140, 621, 91, 66, 251, 668, 198, 108, 278, 223, 394, 306, 135, 563, 226, 3, 505, 80, 167, 35, 473, 675, 589, 162, 531, 680, 255, 648, 112, 617, 194, 145, 48, 557, 690, 63, 640, 18, 282, 95, 310, 50, 67, 199, 673, 16, 585, 502, 338, 643, 31, 336, 613, 11, 72, 175, 446, 612, 143, 43, 250, 231, 450, 99, 363, 556, 87, 203, 671, 688, 104, 368, 588, 40, 304, 26, 258, 390, 55, 114, 171, 139, 418, 23, 8, 75, 119, 58, 667, 478, 536, 82, 620, 447, 36, 168, 146, 30, 51, 190, 19, 422, 564, 305, 107, 4, 136, 506, 79, 195, 474, 664, 532, 94, 283, 395, 332, 528, 644, 47, 15, 163, 200, 68, 62, 277, 691, 501, 90, 111, 254, 227, 337, 122, 83, 309, 560, 639, 676, 222, 592, 364, 100\n",
"+-----+--------------------+--------------------+\n",
"|label| features| indexed|\n",
"+-----+--------------------+--------------------+\n",
"| 0.0|(692,[127,128,129...|(692,[127,128,129...|\n",
"| 1.0|(692,[158,159,160...|(692,[158,159,160...|\n",
"| 1.0|(692,[124,125,126...|(692,[124,125,126...|\n",
"| 1.0|(692,[152,153,154...|(692,[152,153,154...|\n",
"| 1.0|(692,[151,152,153...|(692,[151,152,153...|\n",
"| 0.0|(692,[129,130,131...|(692,[129,130,131...|\n",
"| 1.0|(692,[158,159,160...|(692,[158,159,160...|\n",
"| 1.0|(692,[99,100,101,...|(692,[99,100,101,...|\n",
"| 0.0|(692,[154,155,156...|(692,[154,155,156...|\n",
"| 0.0|(692,[127,128,129...|(692,[127,128,129...|\n",
"| 1.0|(692,[154,155,156...|(692,[154,155,156...|\n",
"| 0.0|(692,[153,154,155...|(692,[153,154,155...|\n",
"| 0.0|(692,[151,152,153...|(692,[151,152,153...|\n",
"| 1.0|(692,[129,130,131...|(692,[129,130,131...|\n",
"| 0.0|(692,[154,155,156...|(692,[154,155,156...|\n",
"| 1.0|(692,[150,151,152...|(692,[150,151,152...|\n",
"| 0.0|(692,[124,125,126...|(692,[124,125,126...|\n",
"| 0.0|(692,[152,153,154...|(692,[152,153,154...|\n",
"| 1.0|(692,[97,98,99,12...|(692,[97,98,99,12...|\n",
"| 1.0|(692,[124,125,126...|(692,[124,125,126...|\n",
"+-----+--------------------+--------------------+\n",
"only showing top 20 rows\n",
"\n"
]
}
],
"source": [
"from pyspark.ml.feature import VectorIndexer\n",
"\n",
"data = spark.read.format(\"libsvm\").load(\"data/sample_libsvm_data.txt\")\n",
"\n",
"indexer = VectorIndexer(inputCol=\"features\", outputCol=\"indexed\", maxCategories=10)\n",
"indexerModel = indexer.fit(data)\n",
"\n",
"categoricalFeatures = indexerModel.categoryMaps\n",
"print(\"Chose %d categorical features: %s\" %\n",
" (len(categoricalFeatures), \", \".join(str(k) for k in categoricalFeatures.keys())))\n",
"\n",
"# Create new column \"indexed\" with categorical values transformed to indices\n",
"indexedData = indexerModel.transform(data)\n",
"indexedData.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> https://www.datacamp.com/community/tutorials/categorical-data\n",
"\n",
"> http://contrib.scikit-learn.org/categorical-encoding/_modules/index.html"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.6"
}
},
"nbformat": 4,
"nbformat_minor": 2
}