{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "HYn5jBq2_Onh"
      },
      "outputs": [],
      "source": [
        "#@title Copyright 2023 Google LLC. Double-click for license information.\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "25T2QAXLJPso"
      },
      "source": [
        "# Colabs\n",
        "\n",
        "Machine Learning Crash Course uses Colaboratories (Colabs) for all programming exercises. Colab is Google's implementation of [Jupyter Notebook](https://jupyter.org/). For more information about Colabs and how to use them, go to [Welcome to Colaboratory](https://research.google.com/colaboratory).\n",
        "\n",
        "# Numerical data: Statistics on a dataset\n",
        "\n",
        "This Colab programming exercise (first of two) is part of the Machine Learning Crash Course module [Working with numerical data](https://developers.google.com/machine-learning/crash-course/numerical-data)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qiy_IL3AsWkA"
      },
      "source": [
        "## What to expect\n",
        "\n",
        "In the section, [First steps with numerical data](https://developers.google.com/machine-learning/crash-course/numerical-data/first-steps), you learned how to do the following:\n",
        "- Visualize your data in plots or graphs.\n",
        "- Evaluate potential features and labels mathematically.\n",
        "- Find [**outliers**](https://developers.google.com/machine-learning/glossary/#outliers) in the dataset.\n",
        "\n",
        "This exercise takes you through the process of finding columns that contain blatant outliers, which you can then decide to keep in or delete from the dataset."
      ]
    },
    {
      "metadata": {
        "id": "XfnQEaePhztH"
      },
      "cell_type": "code",
      "source": [
        "# @title Setup - Install relevant modules\n",
        "\n",
        "!pip install pandas~=2.2.0"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "MyGvT2U4QWmA"
      },
      "outputs": [],
      "source": [
        "# @title Setup - Import relevant modules\n",
        "\n",
        "# The following code imports relevant modules that\n",
        "# allow you to run the colab.\n",
        "# If you encounter technical issues running some of the code sections\n",
        "# that follow, try running this section again.\n",
        "\n",
        "import pandas as pd\n",
        "\n",
        "# The following lines adjust the granularity of reporting.\n",
        "pd.options.display.max_rows = 10\n",
        "pd.options.display.float_format = \"{:.1f}\".format"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "n-qYeaU9QgIA"
      },
      "outputs": [],
      "source": [
        "#@title Import the dataset\n",
        "\n",
        "# The following code imports the dataset that is used in the colab.\n",
        "\n",
        "training_df = pd.read_csv(filepath_or_buffer=\"https://download.mlcc.google.com/mledu-datasets/california_housing_train.csv\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9CfNPW4GRf09"
      },
      "source": [
        "## Get basic statistics\n",
        "\n",
        "In the following code section, the DataFrame `describe` method returns basic statistics on all the columns in the dataset, such as:\n",
        "\n",
        "* `count` is the number of populated elements in this column. Ideally, every column contains the same value for `count`, but that's not always the case.\n",
        "* `mean` is the traditional average of values in that column. We recommend comparing the `mean` to the median for each column. The **median** is the 50% row of the table.\n",
        "* `std` is the standard deviation of the values in this column.\n",
        "* `min`, `25%`, `50%`, `75%`, and `max` indicate values in the 0, 25, 50, 75, and 100th percentiles."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "faMaLr_4QgzP"
      },
      "outputs": [],
      "source": [
        "# Get statistics on the dataset.\n",
        "\n",
        "# The following code returns basic statistics about the data in the dataframe.\n",
        "\n",
        "training_df.describe()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vkkok1t-Rw1l"
      },
      "source": [
        "### Task: Identify possible outliers\n",
        "\n",
        "Based on the preceding statisics, do you see any columns that might contain outliers?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pzt1ZVNhvSUM"
      },
      "outputs": [],
      "source": [
        "# @title Solution (run this code block to view) { display-mode: \"form\" }\n",
        "\n",
        "print(\"\"\"The following columns might contain outliers:\n",
        "\n",
        "  * total_rooms\n",
        "  * total_bedrooms\n",
        "  * population\n",
        "  * households\n",
        "  * possibly, median_income\n",
        "\n",
        "In all of those columns:\n",
        "\n",
        "  * the standard deviation is almost as high as the mean\n",
        "  * the delta between 75% and max is much higher than the\n",
        "      delta between min and 25%.\"\"\")"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "private_outputs": true,
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}