{ "cells": [ { "cell_type": "markdown", "id": "423fa464", "metadata": {}, "source": [ "# Lesson 16 solution: feature engineering\n", "\n", "This notebook is a worked solution to the lesson 16 feature engineering activity. The goal is to improve linear regression performance on the California housing dataset by engineering better features for the `housing_df` dataframe.\n", "\n", "The approach uses three complementary techniques:\n", "\n", "1. Ratio and interaction features that express relationships the raw averages do not.\n", "2. Log transforms to tame strongly right-skewed features.\n", "3. Geographic distance features that turn raw latitude and longitude into a signal a linear model can use directly.\n", "\n", "The Model evaluation section is unchanged from the activity so that leaderboard scoring keeps working." ] }, { "cell_type": "markdown", "id": "2e57dc48", "metadata": {}, "source": [ "### Import libraries" ] }, { "cell_type": "code", "execution_count": 13, "id": "31c2d1e1", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "import pandas as pd\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from sklearn.linear_model import LinearRegression\n", "from sklearn.model_selection import cross_val_score\n", "\n", "# Set random seed for reproducibility\n", "np.random.seed(315)" ] }, { "cell_type": "markdown", "id": "1e89f5fd", "metadata": {}, "source": [ "### Load dataset" ] }, { "cell_type": "code", "execution_count": 2, "id": "9eef4316", "metadata": {}, "outputs": [], "source": [ "# Load California housing dataset\n", "original_housing_df = pd.read_csv('https://media.githubusercontent.com/media/gperdrizet/fullstack-2605/refs/heads/main/data/california_housing.csv')\n", "housing_df = original_housing_df.copy()" ] }, { "cell_type": "markdown", "id": "bcd8e1df", "metadata": {}, "source": [ "## Task 1: Explore the dataset\n", "\n", "Before deciding what to engineer, look at the shape of the data, check for missing values, inspect the distributions, and see how each feature relates to the target." ] }, { "cell_type": "code", "execution_count": 3, "id": "2d9dbba7", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Shape: (20640, 9)\n", "\n", "Columns: ['MedInc', 'HouseAge', 'AveRooms', 'AveBedrms', 'Population', 'AveOccup', 'Latitude', 'Longitude', 'MedHouseVal']\n", "\n", "Missing values per column:\n", "MedInc 0\n", "HouseAge 0\n", "AveRooms 0\n", "AveBedrms 0\n", "Population 0\n", "AveOccup 0\n", "Latitude 0\n", "Longitude 0\n", "MedHouseVal 0\n", "dtype: int64\n" ] }, { "data": { "text/html": [ "
| \n", " | MedInc | \n", "HouseAge | \n", "AveRooms | \n", "AveBedrms | \n", "Population | \n", "AveOccup | \n", "Latitude | \n", "Longitude | \n", "MedHouseVal | \n", "
|---|---|---|---|---|---|---|---|---|---|
| count | \n", "20640.000000 | \n", "20640.000000 | \n", "20640.000000 | \n", "20640.000000 | \n", "20640.000000 | \n", "20640.000000 | \n", "20640.000000 | \n", "20640.000000 | \n", "20640.000000 | \n", "
| mean | \n", "3.870671 | \n", "28.639486 | \n", "5.429000 | \n", "1.096675 | \n", "1425.476744 | \n", "3.070655 | \n", "35.631861 | \n", "-119.569704 | \n", "2.068558 | \n", "
| std | \n", "1.899822 | \n", "12.585558 | \n", "2.474173 | \n", "0.473911 | \n", "1132.462122 | \n", "10.386050 | \n", "2.135952 | \n", "2.003532 | \n", "1.153956 | \n", "
| min | \n", "0.499900 | \n", "1.000000 | \n", "0.846154 | \n", "0.333333 | \n", "3.000000 | \n", "0.692308 | \n", "32.540000 | \n", "-124.350000 | \n", "0.149990 | \n", "
| 25% | \n", "2.563400 | \n", "18.000000 | \n", "4.440716 | \n", "1.006079 | \n", "787.000000 | \n", "2.429741 | \n", "33.930000 | \n", "-121.800000 | \n", "1.196000 | \n", "
| 50% | \n", "3.534800 | \n", "29.000000 | \n", "5.229129 | \n", "1.048780 | \n", "1166.000000 | \n", "2.818116 | \n", "34.260000 | \n", "-118.490000 | \n", "1.797000 | \n", "
| 75% | \n", "4.743250 | \n", "37.000000 | \n", "6.052381 | \n", "1.099526 | \n", "1725.000000 | \n", "3.282261 | \n", "37.710000 | \n", "-118.010000 | \n", "2.647250 | \n", "
| max | \n", "15.000100 | \n", "52.000000 | \n", "141.909091 | \n", "34.066667 | \n", "35682.000000 | \n", "1243.333333 | \n", "41.950000 | \n", "-114.310000 | \n", "5.000010 | \n", "