{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Univariate Linear Regression\n",
"\n",
"## Introduction\n",
"\n",
"In this exercise, we will implement linear regression with one\n",
"variable to predict profits for a food truck. Suppose you are the CEO of a\n",
"restaurant franchise and are considering different cities for opening a new\n",
"outlet. The chain already has trucks in various cities and you have data for\n",
"profits and populations from the cities.\n",
"\n",
" NOTE: The example and sample data is being taken from the \"Machine Learning course by Andrew Ng\" in Coursera.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# initial imports\n",
"import numpy as np\n",
"from matplotlib import pyplot as plt\n",
"%matplotlib notebook\n",
"import seaborn as sns\n",
"\n",
"# setting graph properties\n",
"plt.rcParams['figure.dpi'] = 300 # setting figure dpi for better quality graphs\n",
"plt.rcParams['figure.figsize'] = [10,8]\n",
"sns.set(context=\"notebook\", style=\"white\") # graph styling using seaborn\n",
"%config InlineBackend.figure_format = 'svg'"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"# imports from my models designed for these examples\n",
"from models.linear_regression import compute_cost, gradient_descent, compute_gradient\n",
"from models.data_preprocessing import add_bias_unit"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Plottig Data\n",
"\n",
"Before starting on any task, it is often useful to understand the data by\n",
"visualizing it. For this dataset, we are using a scatter plot to visualize the data, since it has only two properties to plot (profit and population). (Many other problems that you will encounter in real life are multi-dimensional and can’t be plotted on a 2-d plot.)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Loading data ...\n",
"The First 5 elements of the loaded data are \n",
"[[6.1101]\n",
" [5.5277]\n",
" [8.5186]\n",
" [7.0032]\n",
" [5.8598]]\n"
]
},
{
"data": {
"image/svg+xml": [
"\n",
"\n",
"\n",
"\n"
],
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# Loading X and y matrix\n",
"print('Loading data ...')\n",
"\n",
"data = np.loadtxt('data/ex1data1.txt', delimiter=',')\n",
"X = data[:, 0:-1]\n",
"y = data[:, -1:]\n",
"m = y.shape[0]\n",
"\n",
"# printing first 5 elements\n",
"print(\"The First 5 elements of the loaded data are \\n{}\".format(X[0:5]))\n",
"\n",
"# ----------------Plotting Data-----------------\n",
"fig = plt.figure(\"data\")\n",
"axes1 = fig.add_subplot()\n",
"axes1.scatter(X, y, marker=\"x\", c='r');"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Gradient Descent\n",
"\n",
"In this part we will fit the linear regression parameters $\\theta$ to our dataset using gradient descent.\n",
"\n",
"### Update equation\n",
"\n",
"The objective of linear regression is to minimize the cost function:\n",
"\n",
"$$ J(\\theta) = \\frac{1}{2m}\\sum_{i=1}^{m} \\left( h_{\\theta}\\left( x^{(i)} \\right) - y^{(i)} \\right) ^{2} $$\n",
"\n",
"where the hypothesis $h_{\\theta}(x)$ is given by the linear model :\n",
"$$ \n",
"h_{\\theta}(x) = \\theta^{T}x = \\theta_{0} + \\theta_{1}x_{1}\n",
"$$\n",
"\n",
"the parameters of our model are the $\\theta_{j}$ values. These are the values we will adjust to minimize cost J(θ). One way to do this is to use the batch gradient descent algorithm. In batch gradient descent, each\n",
"iteration performs the update \n",
"\n",
"$\n",
"\\theta_{j} := \\theta_{j} - \\alpha \\frac{1}{m} \\sum_{i=1}^{m} \\left(h_{\\theta} \\left(x^{(i)}\\right) -y^{(i)}\\right)x^{(i)}_{j}$ (simultaneously update $\\theta_{j}$ for all j) \n",
"\n",
"With each step of gradient descent, our parameters $\\theta_{j}$ come closer to the\n",
"optimal values that will achieve the lowest cost J(θ).\n",
"\n",
"### Implementation\n",
"\n",
"We have already set up the data needed for linear regression. In the\n",
"following lines, we add another dimension to our data to accommodate the $\\theta_{0}$ intercept term. We also initialize the initial parameters to 0 and the learning rate alpha to 0.01."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"# adding bias units to X\n",
"X = add_bias_unit(X)\n",
"\n",
"# Some gradient descent settings\n",
"iterations = 1500\n",
"alpha = 0.01\n",
"theta = np.zeros([X.shape[1], 1])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Computing the cost J($\\theta$)\n",
"\n",
"As you perform gradient descent to learn minimize the cost function J(θ), it is helpful to monitor the convergence by computing the cost. \n",
"In this section, we will run the compute_cost function to calculate J(θ) so you can check the convergence of your gradient descent implementation."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Testing the cost function ...\n",
"With theta = [0 ; 0]\n",
"Cost computed = [32.07273388]\n",
"Expected cost value (approx) 32.07\n",
"\n",
"With theta = [-1 ; 2]\n",
"Cost computed = [54.24245508]\n",
"Expected cost value (approx) 54.24\n"
]
}
],
"source": [
"print('Testing the cost function ...')\n",
"# compute and display initial cost\n",
"J = compute_cost(X, y, theta);\n",
"print('With theta = [0 ; 0]\\nCost computed = {}'.format(J))\n",
"print('Expected cost value (approx) 32.07')\n",
"\n",
"# further testing of the cost function\n",
"J = compute_cost(X, y, np.array([[-1], [2]]))\n",
"print('\\nWith theta = [-1 ; 2]\\nCost computed = {}'.format(J))\n",
"print('Expected cost value (approx) 54.24')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Gradient Descent"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Running Gradient Descent ...\n",
"Theta found by gradient descent:\n",
"[-3.63029144 1.16636235]\n",
"Expected theta values (approx)\n",
" [-3.6303 1.1664]\n"
]
}
],
"source": [
"print('\\nRunning Gradient Descent ...')\n",
"# run gradient descent\n",
"theta, _ = gradient_descent(X, y, theta, alpha, iterations)\n",
"\n",
"# print theta to screen\n",
"print('Theta found by gradient descent:')\n",
"print(theta.ravel())\n",
"print('Expected theta values (approx)')\n",
"print(' [-3.6303 1.1664]')"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"\n",
"\n",
"\n",
"\n"
],
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# plotting the linear model\n",
"plt.scatter(X[:, 1:], y, marker='x', c='r', lw=1)\n",
"plt.plot(X[:,1:], np.dot(X,theta), lw=2);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Visualizing J($\\theta$)\n",
"\n",
"To understand the cost function J(θ) better, you will now plot the cost over\n",
"a 2-dimensional grid of θ0 and θ1 values. \n",
"The purpose of these graphs is to show you that how J(θ) varies with\n",
"changes in θ0 and θ1. The cost function J(θ) is bowl-shaped and has a global\n",
"mininum. (This is easier to see in the contour plot than in the 3D surface\n",
"plot). This minimum is the optimal point for θ0 and θ1, and each step of\n",
"gradient descent moves closer to this point."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Visualizing J(theta_0, theta_1) ...\n",
"\n"
]
},
{
"data": {
"image/svg+xml": [
"\n",
"\n",
"\n",
"