{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "# Install the necessary dependencies\n", "\n", "import sys\n", "import os\n", "!{sys.executable} -m pip install --quiet pandas numpy matplotlib jupyterlab_myst ipython " ] }, { "cell_type": "markdown", "metadata": { "tags": [ "remove-cell" ] }, "source": [ "---\n", "license:\n", " code: MIT\n", " content: CC-BY-4.0\n", "github: https://github.com/ocademy-ai/machine-learning\n", "venue: By Ocademy\n", "open_access: true\n", "bibliography:\n", " - https://raw.githubusercontent.com/ocademy-ai/machine-learning/main/open-machine-learning-jupyter-book/references.bib\n", "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Loss function\n", "\n", "The function we want to minimize or maximize is called the objective function or criterion. When we are minimizing it, we may also call it the cost function, loss function, or error function.\n", "\n", "— Deep Learning, Ian Goodfellow, Yoshua Bengio, Aaron Courville" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Objective of this section\n", "\n", "We have already learned math and code for \"Gradient Descent\", as well as other optimization techniques.\n", "\n", "In this section, we will learn more about loss functions for Linear Regression and Logistic Regression." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## What’s the Difference between a Loss Function and a Cost Function?\n", "\n", "A loss function is for a single training example. It is also sometimes called an error function. A cost function, on the other hand, is the average loss over the entire training dataset. The optimization strategies aim at minimizing the cost function." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Regression Loss Functions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Squared Error Loss\n", "\n", "Squared Error loss for each training example, \n", "also known as **L2 Loss**, is the square of the \n", "difference between the actual and the predicted values:\n", "\n", "$$L = (y - f(x))^2$$\n", "\n", "The corresponding cost function is the \n", "Mean of these Squared Errors (MSE)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Absolute Error Loss\n", "\n", "Absolute Error for each training example \n", "is the distance between the predicted and the actual values, \n", "irrespective of the sign. Absolute Error is also \n", "known as the **L1 loss**:\n", "\n", "$$L = \\lvert y - f(x) \\rvert$$\n", "\n", "The corresponding cost function is the Mean of these Absolute Errors (MAE)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Classification Loss Functions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Binary Cross Entropy Loss\n", "\n", "Cross-entropy is the default loss function to use for binary classification problems.\n", "\n", "It is intended for use with binary classification where the target values are in the set {0, 1}.\n", "\n", "Mathematically, it is the preferred loss function \n", "under the inference framework of maximum likelihood. \n", "It is the loss function to be evaluated first and only \n", "changed if you have a good reason.\n", "\n", "Cross-entropy will calculate a score that summarizes \n", "the average difference between the actual and predicted \n", "probability distributions for predicting class 1. \n", "The score is minimized and a perfect cross-entropy value is 0.\n", "\n", "This YouTube video by Andrew Ng explains very well Binary Cross Entropy Loss (make sure \n", "that you have access to YouTube for this web page to render correctly):" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "tags": [ "hide-input" ] }, "outputs": [ { "data": { "text/html": [ "\n", "

\n", "\n", "video. [source]\n", "

\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from IPython.display import HTML\n", "\n", "display(\n", " HTML(\n", " \"\"\"\n", "

\n", "\n", "video. [source]\n", "

\n", "\"\"\"\n", " )\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Multi-Class Cross-Entropy Loss\n", "\n", "Cross-entropy is the default loss function to \n", "use for multi-class classification problems.\n", "\n", "In this case, it is intended for use with \n", "multi-class classification where the target values \n", "are in the set {0, 1, 3, …, n}, where each class is \n", "assigned a unique integer value.\n", "\n", "Mathematically, it is the preferred loss \n", "function under the inference framework of \n", "maximum likelihood. It is the loss function \n", "to be evaluated first and only changed if you have a good reason.\n", "\n", "Cross-entropy will calculate a score that \n", "summarizes the average difference between \n", "the actual and predicted probability distributions \n", "for all classes in the problem. The score is minimized \n", "and a perfect cross-entropy value is 0." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## [optional] At the frontier of Machine Learning Research" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "tags": [ "hide-input" ] }, "outputs": [ { "data": { "text/html": [ "\n", "

\n", "\n", "video. [source]\n", "

\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from IPython.display import HTML\n", "\n", "display(\n", " HTML(\n", " \"\"\"\n", "

\n", "\n", "video. [source]\n", "

\n", "\"\"\"\n", " )\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With its corresponding paper: [A General and Adaptive Robust Loss Function](https://arxiv.org/abs/1701.03077)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Bibliography\n", "\n", "- [ML cheat sheet for loss functions](https://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html)\n", "- [A Short Introduction to Entropy, Cross-Entropy and KL-Divergence](https://www.youtube.com/watch?v=ErfnhcEV1O8)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Your turn! 🚀\n", " \n", "Please complete the following tasks:\n", "[loss-function](../../assignments/ml-fundamentals/linear-regression/loss-function.ipynb)" ] } ], "metadata": { "kernelspec": { "display_name": "myconda1", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.16" } }, "nbformat": 4, "nbformat_minor": 2 }