{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open in Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/justmarkham/scikit-learn-tips/master?filepath=notebooks%2F07_handle_unknown_categories.ipynb)\n",
"\n",
"[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/justmarkham/scikit-learn-tips/blob/master/notebooks/07_handle_unknown_categories.ipynb)\n",
"\n",
"# 🤖⚡ scikit-learn tip #7 ([video](https://www.youtube.com/watch?v=bA6mYC1a_Eg&list=PL5-da3qGB5ID7YYAqireYEew2mWVvgmj6&index=7))\n",
"\n",
"Q: For a one-hot encoded feature, what can you do if new data contains categories that weren't seen during training?\n",
"\n",
"A: Set handle_unknown='ignore' to encode new categories as all zeros.\n",
"\n",
"See example 👇\n",
"\n",
"P.S. If you know all possible categories that might ever appear, you can instead specify the categories manually. handle_unknown='ignore' is useful specifically when you don't know all possible categories."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"X = pd.DataFrame({'col':['A', 'B', 'C', 'B']})\n",
"X_new = pd.DataFrame({'col':['A', 'C', 'D']})"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.preprocessing import OneHotEncoder\n",
"ohe = OneHotEncoder(sparse=False, handle_unknown='ignore')"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" col | \n",
"
\n",
" \n",
" \n",
" \n",
" 0 | \n",
" A | \n",
"
\n",
" \n",
" 1 | \n",
" B | \n",
"
\n",
" \n",
" 2 | \n",
" C | \n",
"
\n",
" \n",
" 3 | \n",
" B | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" col\n",
"0 A\n",
"1 B\n",
"2 C\n",
"3 B"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[1., 0., 0.],\n",
" [0., 1., 0.],\n",
" [0., 0., 1.],\n",
" [0., 1., 0.]])"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# three columns represent categories A, B, and C\n",
"ohe.fit_transform(X[['col']])"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" col | \n",
"
\n",
" \n",
" \n",
" \n",
" 0 | \n",
" A | \n",
"
\n",
" \n",
" 1 | \n",
" C | \n",
"
\n",
" \n",
" 2 | \n",
" D | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" col\n",
"0 A\n",
"1 C\n",
"2 D"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# category D was not learned by OneHotEncoder during the \"fit\" step\n",
"X_new"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[1., 0., 0.],\n",
" [0., 0., 1.],\n",
" [0., 0., 0.]])"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# category D is encoded as all zeros\n",
"ohe.transform(X_new[['col']])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Want more tips? [View all tips on GitHub](https://github.com/justmarkham/scikit-learn-tips) or [Sign up to receive 2 tips by email every week](https://scikit-learn.tips) 💌\n",
"\n",
"© 2020 [Data School](https://www.dataschool.io). All rights reserved."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.2"
}
},
"nbformat": 4,
"nbformat_minor": 4
}