{
"cells": [
{
"cell_type": "markdown",
"id": "f2dbfe83",
"metadata": {},
"source": [
"# Lesson 16 activity: feature engineering\n",
"\n",
"## Notebook set up\n",
"\n",
"Submit your notebook to the class leaderboard on HuggingFace at [huggingface.co/spaces/gperdrizet/leaderboard](https://huggingface.co/spaces/gperdrizet/leaderboard)\n",
"\n",
"**Your task**: Apply at least two different feature engineering techniques to the `housing_df` dataframe to improve the dataset. At the end of the notebook, your engineered dataset and the original dataset will be used to train a linear regression model to predict `MedHouseVal`. Your goal is to achieve better model performance via feature engineering.\n",
"\n",
"Don't change any of the code in the Model evaluation section of the notebook, especially the output saving. Otherwise the leaderboard scoring may not work!\n",
"\n",
"Before applying transformations, explore the dataset to understand what techniques might be most beneficial.\n",
"\n",
"### Import libraries"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "981d7f9f",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-30T16:48:26.781866Z",
"iopub.status.busy": "2026-06-30T16:48:26.781447Z",
"iopub.status.idle": "2026-06-30T16:48:28.907306Z",
"shell.execute_reply": "2026-06-30T16:48:28.905280Z"
}
},
"outputs": [],
"source": [
"from pathlib import Path\n",
"\n",
"import pandas as pd\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"from sklearn.linear_model import LinearRegression\n",
"from sklearn.model_selection import cross_val_score\n",
"\n",
"# Set random seed for reproducibility\n",
"np.random.seed(315)"
]
},
{
"cell_type": "markdown",
"id": "af33c7c8",
"metadata": {},
"source": [
"### Load dataset"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "fd5fd573",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-30T16:48:28.911387Z",
"iopub.status.busy": "2026-06-30T16:48:28.910964Z",
"iopub.status.idle": "2026-06-30T16:48:29.428903Z",
"shell.execute_reply": "2026-06-30T16:48:29.426860Z"
}
},
"outputs": [],
"source": [
"# Load California housing dataset\n",
"original_housing_df = pd.read_csv('https://media.githubusercontent.com/media/gperdrizet/fullstack-2605/refs/heads/main/data/california_housing.csv')\n",
"housing_df = original_housing_df.copy()"
]
},
{
"cell_type": "markdown",
"id": "7d0f6873",
"metadata": {},
"source": [
"## Task 1: Explore the dataset\n",
"\n",
"Before deciding what feature engineering techniques to apply, explore the dataset to understand its characteristics.\n",
"\n",
"**Things to investigate**:\n",
"- Display basic information about the dataset (`.info()`, `.describe()`)\n",
"- Check for missing values\n",
"- Examine feature distributions (histograms, box plots)\n",
"- Look at feature scales and ranges\n",
"- Look for relationships (or lack thereof) between features and between features and the label\n",
"\n",
"Use this exploration to inform your feature engineering decisions in the following tasks."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "7ce1254f",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-30T16:48:29.433863Z",
"iopub.status.busy": "2026-06-30T16:48:29.433546Z",
"iopub.status.idle": "2026-06-30T16:48:29.502112Z",
"shell.execute_reply": "2026-06-30T16:48:29.500229Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Rows: 20,640\n",
"Columns: 9\n",
"\n",
"Missing values by column:\n"
]
},
{
"data": {
"text/plain": [
"MedInc 0\n",
"HouseAge 0\n",
"AveRooms 0\n",
"AveBedrms 0\n",
"Population 0\n",
"AveOccup 0\n",
"Latitude 0\n",
"Longitude 0\n",
"MedHouseVal 0\n",
"dtype: int64"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" count \n",
" mean \n",
" std \n",
" min \n",
" 25% \n",
" 50% \n",
" 75% \n",
" max \n",
" \n",
" \n",
" \n",
" \n",
" MedInc \n",
" 20640.0 \n",
" 3.870671 \n",
" 1.899822 \n",
" 0.499900 \n",
" 2.563400 \n",
" 3.534800 \n",
" 4.743250 \n",
" 15.000100 \n",
" \n",
" \n",
" HouseAge \n",
" 20640.0 \n",
" 28.639486 \n",
" 12.585558 \n",
" 1.000000 \n",
" 18.000000 \n",
" 29.000000 \n",
" 37.000000 \n",
" 52.000000 \n",
" \n",
" \n",
" AveRooms \n",
" 20640.0 \n",
" 5.429000 \n",
" 2.474173 \n",
" 0.846154 \n",
" 4.440716 \n",
" 5.229129 \n",
" 6.052381 \n",
" 141.909091 \n",
" \n",
" \n",
" AveBedrms \n",
" 20640.0 \n",
" 1.096675 \n",
" 0.473911 \n",
" 0.333333 \n",
" 1.006079 \n",
" 1.048780 \n",
" 1.099526 \n",
" 34.066667 \n",
" \n",
" \n",
" Population \n",
" 20640.0 \n",
" 1425.476744 \n",
" 1132.462122 \n",
" 3.000000 \n",
" 787.000000 \n",
" 1166.000000 \n",
" 1725.000000 \n",
" 35682.000000 \n",
" \n",
" \n",
" AveOccup \n",
" 20640.0 \n",
" 3.070655 \n",
" 10.386050 \n",
" 0.692308 \n",
" 2.429741 \n",
" 2.818116 \n",
" 3.282261 \n",
" 1243.333333 \n",
" \n",
" \n",
" Latitude \n",
" 20640.0 \n",
" 35.631861 \n",
" 2.135952 \n",
" 32.540000 \n",
" 33.930000 \n",
" 34.260000 \n",
" 37.710000 \n",
" 41.950000 \n",
" \n",
" \n",
" Longitude \n",
" 20640.0 \n",
" -119.569704 \n",
" 2.003532 \n",
" -124.350000 \n",
" -121.800000 \n",
" -118.490000 \n",
" -118.010000 \n",
" -114.310000 \n",
" \n",
" \n",
" MedHouseVal \n",
" 20640.0 \n",
" 2.068558 \n",
" 1.153956 \n",
" 0.149990 \n",
" 1.196000 \n",
" 1.797000 \n",
" 2.647250 \n",
" 5.000010 \n",
" \n",
" \n",
"
\n",
"
\n",
"\n",
"
\n",
" \n",
" \n",
" Correlation with MedHouseVal \n",
" \n",
" \n",
" \n",
" \n",
" MedInc \n",
" 0.688075 \n",
" \n",
" \n",
" AveRooms \n",
" 0.151948 \n",
" \n",
" \n",
" Latitude \n",
" -0.144160 \n",
" \n",
" \n",
" HouseAge \n",
" 0.105623 \n",
" \n",
" \n",
" AveBedrms \n",
" -0.046701 \n",
" \n",
" \n",
" Longitude \n",
" -0.045967 \n",
" \n",
" \n",
" Population \n",
" -0.024650 \n",
" \n",
" \n",
" AveOccup \n",
" -0.023737 \n",
" \n",
" \n",
"
\n",
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"fig, axs = plt.subplots(2, 4, figsize=(12, 6))\n",
"\n",
"for ax, column in zip(axs.ravel(), feature_columns):\n",
" ax.hist(housing_df[column], bins=40)\n",
" ax.set_title(column)\n",
"\n",
"plt.tight_layout()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "1dfb2bd2",
"metadata": {},
"source": [
"## Task 2: Apply your first feature engineering technique\n",
"\n",
"Based on your exploration, apply your first feature engineering technique.\n",
"\n",
"**Example approaches**:\n",
"- Transform skewed features using log, sqrt, power, or quantile transformations\n",
"- Create bins/categories from continuous variables\n",
"- Create interaction features (e.g., rooms per household = total rooms / households)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "0a9f837e",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-30T16:48:31.199863Z",
"iopub.status.busy": "2026-06-30T16:48:31.199598Z",
"iopub.status.idle": "2026-06-30T16:48:31.225923Z",
"shell.execute_reply": "2026-06-30T16:48:31.224839Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" count \n",
" mean \n",
" std \n",
" min \n",
" 25% \n",
" 50% \n",
" 75% \n",
" max \n",
" \n",
" \n",
" \n",
" \n",
" BedroomsPerRoom \n",
" 20640.0 \n",
" 0.213075 \n",
" 0.058023 \n",
" 0.100000 \n",
" 0.175426 \n",
" 0.203181 \n",
" 0.239834 \n",
" 1.000000 \n",
" \n",
" \n",
" PopulationPerHousehold \n",
" 20640.0 \n",
" 499.539680 \n",
" 382.329753 \n",
" 1.000000 \n",
" 280.000000 \n",
" 409.000000 \n",
" 605.000000 \n",
" 6082.000000 \n",
" \n",
" \n",
" RoomsPerPerson \n",
" 20640.0 \n",
" 0.025028 \n",
" 0.174101 \n",
" 0.000746 \n",
" 0.008140 \n",
" 0.012683 \n",
" 0.019576 \n",
" 12.900826 \n",
" \n",
" \n",
"
\n",
"
\n",
"\n",
"
\n",
" \n",
" \n",
" count \n",
" mean \n",
" std \n",
" min \n",
" 25% \n",
" 50% \n",
" 75% \n",
" max \n",
" \n",
" \n",
" \n",
" \n",
" LogPopulation \n",
" 20640.0 \n",
" 7.025503 \n",
" 0.736238 \n",
" 1.386294 \n",
" 6.669498 \n",
" 7.062192 \n",
" 7.453562 \n",
" 10.482430 \n",
" \n",
" \n",
" LogAveOccup \n",
" 20640.0 \n",
" 1.351645 \n",
" 0.207530 \n",
" 0.526093 \n",
" 1.232485 \n",
" 1.339757 \n",
" 1.454481 \n",
" 7.126355 \n",
" \n",
" \n",
" LogAveRooms \n",
" 20640.0 \n",
" 1.828722 \n",
" 0.231938 \n",
" 0.613104 \n",
" 1.693911 \n",
" 1.829236 \n",
" 1.953365 \n",
" 4.962209 \n",
" \n",
" \n",
" LogAveBedrms \n",
" 20640.0 \n",
" 0.732035 \n",
" 0.105412 \n",
" 0.287682 \n",
" 0.696182 \n",
" 0.717245 \n",
" 0.741712 \n",
" 3.557251 \n",
" \n",
" \n",
"
\n",
"
\n",
"\n",
"
\n",
" \n",
" \n",
" MedInc \n",
" HouseAge \n",
" AveRooms \n",
" AveBedrms \n",
" Population \n",
" AveOccup \n",
" Latitude \n",
" Longitude \n",
" MedHouseVal \n",
" BedroomsPerRoom \n",
" ... \n",
" IncomeQ_4 \n",
" AgeBin_0 \n",
" AgeBin_1 \n",
" AgeBin_2 \n",
" AgeBin_3 \n",
" LatitudeSquared \n",
" LongitudeSquared \n",
" LatitudeLongitude \n",
" DistanceToSanFrancisco \n",
" DistanceToLosAngeles \n",
" \n",
" \n",
" \n",
" \n",
" 0 \n",
" 8.3252 \n",
" 41.0 \n",
" 6.984127 \n",
" 1.023810 \n",
" 322.0 \n",
" 2.555556 \n",
" 37.88 \n",
" -122.23 \n",
" 4.526 \n",
" 0.146591 \n",
" ... \n",
" 1 \n",
" 0 \n",
" 0 \n",
" 0 \n",
" 1 \n",
" 1434.8944 \n",
" 14940.1729 \n",
" -4630.0724 \n",
" 0.219545 \n",
" 5.530732 \n",
" \n",
" \n",
" 1 \n",
" 8.3014 \n",
" 21.0 \n",
" 6.238137 \n",
" 0.971880 \n",
" 2401.0 \n",
" 2.109842 \n",
" 37.86 \n",
" -122.22 \n",
" 3.585 \n",
" 0.155797 \n",
" ... \n",
" 1 \n",
" 0 \n",
" 0 \n",
" 1 \n",
" 0 \n",
" 1433.3796 \n",
" 14937.7284 \n",
" -4627.2492 \n",
" 0.219317 \n",
" 5.509673 \n",
" \n",
" \n",
" 2 \n",
" 7.2574 \n",
" 52.0 \n",
" 8.288136 \n",
" 1.073446 \n",
" 496.0 \n",
" 2.802260 \n",
" 37.85 \n",
" -122.24 \n",
" 3.521 \n",
" 0.129516 \n",
" ... \n",
" 1 \n",
" 0 \n",
" 0 \n",
" 0 \n",
" 1 \n",
" 1432.6225 \n",
" 14942.6176 \n",
" -4626.7840 \n",
" 0.196977 \n",
" 5.517246 \n",
" \n",
" \n",
" 3 \n",
" 5.6431 \n",
" 52.0 \n",
" 5.817352 \n",
" 1.073059 \n",
" 558.0 \n",
" 2.547945 \n",
" 37.85 \n",
" -122.25 \n",
" 3.413 \n",
" 0.184458 \n",
" ... \n",
" 1 \n",
" 0 \n",
" 0 \n",
" 0 \n",
" 1 \n",
" 1432.6225 \n",
" 14945.0625 \n",
" -4627.1625 \n",
" 0.187883 \n",
" 5.524500 \n",
" \n",
" \n",
" 4 \n",
" 3.8462 \n",
" 52.0 \n",
" 6.281853 \n",
" 1.081081 \n",
" 565.0 \n",
" 2.181467 \n",
" 37.85 \n",
" -122.25 \n",
" 3.422 \n",
" 0.172096 \n",
" ... \n",
" 0 \n",
" 0 \n",
" 0 \n",
" 0 \n",
" 1 \n",
" 1432.6225 \n",
" 14945.0625 \n",
" -4627.1625 \n",
" 0.187883 \n",
" 5.524500 \n",
" \n",
" \n",
"
\n",
"
5 rows × 30 columns
\n",
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"original_model = LinearRegression()\n",
"original_model.fit(original_housing_df.drop('MedHouseVal', axis=1), original_housing_df['MedHouseVal'])\n",
"original_predictions = original_model.predict(original_housing_df.drop('MedHouseVal', axis=1))\n",
"\n",
"model = LinearRegression()\n",
"model.fit(housing_df.drop('MedHouseVal', axis=1), housing_df['MedHouseVal'])\n",
"predictions = model.predict(housing_df.drop('MedHouseVal', axis=1))\n",
"\n",
"# Create boxplot comparing performance\n",
"data_to_plot = [scores_original, scores_engineered]\n",
"labels = ['Original', 'Engineered']\n",
"\n",
"fig, axs = plt.subplots(1, 2, figsize=(9,4.5))\n",
"\n",
"fig.suptitle(f'Model performance comparison\\nmean improvement: {mean_improvement:.2f}%')\n",
"\n",
"axs[0].set_title('Cross validation R² scores')\n",
"axs[0].boxplot(data_to_plot, tick_labels=labels)\n",
"axs[0].set_xlabel('Dataset')\n",
"axs[0].set_ylabel('R² score')\n",
"\n",
"axs[1].set_title('Predictions vs true values')\n",
"axs[1].plot(\n",
" original_housing_df['MedHouseVal'], original_predictions,\n",
" 'o', markersize=1, label='Original', alpha=0.25\n",
")\n",
"\n",
"axs[1].plot(\n",
" housing_df['MedHouseVal'], predictions,\n",
" 'o', markersize=1, label='Engineered', alpha=0.25\n",
")\n",
"\n",
"axs[1].set_xlabel('True Values')\n",
"axs[1].set_ylabel('Predictions')\n",
"\n",
"leg = axs[1].legend(loc='upper left', markerscale=8, framealpha=1)\n",
"\n",
"for lh in leg.legend_handles:\n",
" lh.set_alpha(1)\n",
"\n",
"plt.tight_layout()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "cb9c2962",
"metadata": {},
"source": [
"## 3. Reflection\n",
"\n",
"**Questions to consider**:\n",
"\n",
"1. Which feature engineering techniques had the biggest impact on model performance?\n",
"2. Did adding more features always improve performance, or did some hurt it?\n",
"3. How might you further improve the engineered dataset?\n",
"4. What trade-offs did you consider (e.g., interpretability vs performance, complexity vs gains)?\n",
"\n",
"Ratio features helped the model use existing columns in a more meaningful way, especially bedroom share and population per household. Log transforms made highly skewed count-like features less extreme. The optional bin and location features added non-linear structure while still keeping the final estimator as a plain linear regression model.\n",
"\n",
"Adding more features does not always improve performance. For this solution, the added features were chosen because they gave a better cross-validation score in the notebook's evaluation setup while staying reasonably interpretable. Further improvements might include testing polynomial features more systematically, using regularization to control extra feature complexity, or building richer location features from known California regions."
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv (3.12.3)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}