{
  "cells": [
    {
      "metadata": {
        "id": "kYmgbnGytC9h",
        "cellView": "form"
      },
      "cell_type": "code",
      "source": [
        "#@title Copyright 2023 Google LLC. Double-click for license information.\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {
        "id": "CeNGK50ZP5pR"
      },
      "cell_type": "markdown",
      "source": [
        "# Colabs\n",
        "\n",
        "Machine Learning Crash Course uses Colaboratories (Colabs) for all programming exercises. Colab is Google's implementation of [Jupyter Notebook](https://jupyter.org/). For more information about Colabs and how to use them, go to [Welcome to Colaboratory](https://research.google.com/colaboratory).\n",
        "\n",
        "# Binary classification\n",
        "\n",
        "In this Colab, you'll complete the following tasks:\n",
        "- Examine a dataset containing measurements derived from images of two species of Turkish rice.\n",
        "- Create a binary classifier to sort grains of rice into the two species.\n",
        "- Evaluate the performance of the model.\n",
        "\n",
        "## Learning objectives\n",
        "\n",
        "By completing this Colab, you'll learn:\n",
        "- How to train a binary classifier.\n",
        "- How to calculate metrics for a binary classifier at different thresholds.\n",
        "- How to compare AUC and ROC of two different models.\n",
        "\n",
        "## Dataset\n",
        "\n",
        "This Colab uses the Cinar and Koklu 2019 Osmancik and Cammeo rice dataset.\n",
        "\n",
        "Provided with a CC0 license (see [Kaggle](https://www.kaggle.com/datasets/muratkokludataset/rice-dataset-commeo-and-osmancik) for more documentation; lengths and area are given in pixels). Cinar and Koklu also provide datasets for multiclass (5 species of rice), pistachios, raisins, grape leaves, and so on, at their [repository](https://www.muratkoklu.com/datasets/).\n",
        "\n",
        "### Citation\n",
        "\n",
        "Cinar, I. and Koklu, M., (2019). “Classification of Rice Varieties Using Artificial Intelligence Methods.” *International Journal of Intelligent Systems and Applications in Engineering*, 7(3), 188-194.\n",
        "\n",
        "DOI: https://doi.org/10.18201/ijisae.2019355381\n"
      ]
    },
    {
      "metadata": {
        "id": "D4JwY1X5iryL"
      },
      "cell_type": "markdown",
      "source": [
        "# Load Imports"
      ]
    },
    {
      "metadata": {
        "cellView": "form",
        "id": "cJt1sNSzb-6z"
      },
      "cell_type": "code",
      "source": [
        "# @title Install required libraries\n",
        "\n",
        "!pip install google-ml-edu==0.1.2 \\\n",
        "    keras~=3.8.0 \\\n",
        "    matplotlib~=3.10.0 \\\n",
        "    numpy~=2.0.0 \\\n",
        "    pandas~=2.2.0 \\\n",
        "    tensorflow~=2.18.0\n",
        "\n",
        "print('\\n\\nAll requirements successfully installed.')"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {
        "cellView": "form",
        "id": "irD_75BI97__"
      },
      "cell_type": "code",
      "source": [
        "# @title Load the imports\n",
        "\n",
        "import keras\n",
        "import ml_edu.experiment\n",
        "import ml_edu.results\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import plotly.express as px\n",
        "\n",
        "# The following lines adjust the granularity of reporting.\n",
        "pd.options.display.max_rows = 10\n",
        "pd.options.display.float_format = \"{:.1f}\".format\n",
        "\n",
        "print(\"Ran the import statements.\")"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {
        "cellView": "form",
        "id": "aDobhxERWPD1"
      },
      "cell_type": "code",
      "source": [
        "# @title Load the dataset\n",
        "rice_dataset_raw = pd.read_csv(\"https://download.mlcc.google.com/mledu-datasets/Rice_Cammeo_Osmancik.csv\")"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {
        "id": "1IqvqOvaQqlK"
      },
      "cell_type": "markdown",
      "source": [
        "Once the dataset has been loaded via the cell above, select specific columns to show summary statistics of the numerical features in the dataset.\n",
        "\n",
        "See the Kaggle [dataset documentation](https://www.kaggle.com/datasets/muratkokludataset/rice-dataset-commeo-and-osmancik), especially the **Provenance** section, for explanations of what each feature means and how they were calculated."
      ]
    },
    {
      "metadata": {
        "cellView": "form",
        "id": "XKakOMCmHp-E"
      },
      "cell_type": "code",
      "source": [
        "# @title\n",
        "# Read and provide statistics on the dataset.\n",
        "rice_dataset = rice_dataset_raw[[\n",
        "    'Area',\n",
        "    'Perimeter',\n",
        "    'Major_Axis_Length',\n",
        "    'Minor_Axis_Length',\n",
        "    'Eccentricity',\n",
        "    'Convex_Area',\n",
        "    'Extent',\n",
        "    'Class',\n",
        "]]\n",
        "\n",
        "rice_dataset.describe()"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {
        "id": "ynv9WwTPG9oU"
      },
      "cell_type": "markdown",
      "source": [
        "## Task 1: Describe the data\n",
        "\n",
        "From the summary statistics above, answer the following questions:\n",
        "- What are the min and max lengths (major axis length, given in pixels) of the rice grains?\n",
        "- What is the range of areas between the smallest and largest rice grains?\n",
        "- How many standard deviations (`std`) is the largest rice grain's perimeter from the mean?"
      ]
    },
    {
      "metadata": {
        "cellView": "form",
        "id": "y36kQm1vJ7n3"
      },
      "cell_type": "code",
      "source": [
        "# @title Solutions (run the cell to get the answers)\n",
        "\n",
        "print(\n",
        "    f'The shortest grain is {rice_dataset.Major_Axis_Length.min():.1f}px long,'\n",
        "    f' while the longest is {rice_dataset.Major_Axis_Length.max():.1f}px.'\n",
        ")\n",
        "print(\n",
        "    f'The smallest rice grain has an area of {rice_dataset.Area.min()}px, while'\n",
        "    f' the largest has an area of {rice_dataset.Area.max()}px.'\n",
        ")\n",
        "print(\n",
        "    'The largest rice grain, with a perimeter of'\n",
        "    f' {rice_dataset.Perimeter.max():.1f}px, is'\n",
        "    f' ~{(rice_dataset.Perimeter.max() - rice_dataset.Perimeter.mean())/rice_dataset.Perimeter.std():.1f} standard'\n",
        "    f' deviations ({rice_dataset.Perimeter.std():.1f}) from the mean'\n",
        "    f' ({rice_dataset.Perimeter.mean():.1f}px).'\n",
        ")\n",
        "print(\n",
        "    f'This is calculated as: ({rice_dataset.Perimeter.max():.1f} -'\n",
        "    f' {rice_dataset.Perimeter.mean():.1f})/{rice_dataset.Perimeter.std():.1f} ='\n",
        "    f' {(rice_dataset.Perimeter.max() - rice_dataset.Perimeter.mean())/rice_dataset.Perimeter.std():.1f}'\n",
        ")"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {
        "id": "Bc8JVgN2j6h3"
      },
      "cell_type": "markdown",
      "source": [
        "# Explore the dataset\n",
        "\n",
        "Plot some of the features against each other, including in 3D.\n"
      ]
    },
    {
      "metadata": {
        "id": "QNw5U7-4NFLR"
      },
      "cell_type": "code",
      "source": [
        "# Create five 2D plots of the features against each other, color-coded by class.\n",
        "for x_axis_data, y_axis_data in [\n",
        "    ('Area', 'Eccentricity'),\n",
        "    ('Convex_Area', 'Perimeter'),\n",
        "    ('Major_Axis_Length', 'Minor_Axis_Length'),\n",
        "    ('Perimeter', 'Extent'),\n",
        "    ('Eccentricity', 'Major_Axis_Length'),\n",
        "]:\n",
        "  px.scatter(rice_dataset, x=x_axis_data, y=y_axis_data, color='Class').show()"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {
        "id": "G6xJ0HQxLB4N"
      },
      "cell_type": "markdown",
      "source": [
        "## Task 2: Visualize samples in 3D\n",
        "\n",
        "Try graphing three of the features in 3D against each other."
      ]
    },
    {
      "metadata": {
        "cellView": "form",
        "id": "qvpUsZF1LDWM"
      },
      "cell_type": "code",
      "source": [
        "#@title Plot three features in 3D by entering their names and running this cell\n",
        "\n",
        "x_axis_data = 'Enter a feature name here'  # @param {type: \"string\"}\n",
        "y_axis_data = 'Enter a feature name here'  # @param {type: \"string\"}\n",
        "z_axis_data = 'Enter a feature name here'  # @param {type: \"string\"}\n",
        "\n",
        "px.scatter_3d(\n",
        "    rice_dataset,\n",
        "    x=x_axis_data,\n",
        "    y=y_axis_data,\n",
        "    z=z_axis_data,\n",
        "    color='Class',\n",
        ").show()"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {
        "cellView": "form",
        "id": "r5WBGiJChXt-"
      },
      "cell_type": "code",
      "source": [
        "# @title One possible solution\n",
        "\n",
        "# Plot major and minor axis length and eccentricity, with observations\n",
        "# color-coded by class.\n",
        "px.scatter_3d(\n",
        "    rice_dataset,\n",
        "    x='Eccentricity',\n",
        "    y='Area',\n",
        "    z='Major_Axis_Length',\n",
        "    color='Class',\n",
        ").show()"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {
        "id": "Ch82395CSBMR"
      },
      "cell_type": "markdown",
      "source": [
        "If we were to pick three features, it seems that major axis length, area, and eccentricity might contain most of the information that differentiates the two classes. Other combinations may work as well.\n",
        "\n",
        "Run the previous code cell to graph those three features if you haven't already.\n",
        "\n",
        "It seems like a distinct class boundary appears in the plane of these three features. We'll train a model on just these features, then another model on the complete set of features, and compare their performance."
      ]
    },
    {
      "metadata": {
        "id": "_G6y-XcEmk6r"
      },
      "cell_type": "markdown",
      "source": [
        "## Normalize data\n",
        "\n",
        "When creating a model with multiple features, the values of each feature should span roughly the same range.  If one feature's values range from 500 to 100,000 and another feature's values range from 2 to 12, the model will need to have weights of extremely low or extremely high values to be able to combine these features effectively. This could result in a low quality model.  To avoid this,\n",
        "[normalize](https://developers.google.com/machine-learning/glossary/#normalization) features in a multi-feature model.\n",
        "\n",
        "This can be done by converting each raw value to its Z-score. The **Z-score** for a given value is how many standard deviations away from the mean the value is.\n",
        "\n",
        "Consider a feature with a mean of 60 and a standard deviation of 10.\n",
        "\n",
        "The raw value 75 would have a Z-score of +1.5:\n",
        "\n",
        "```\n",
        "  Z-score = (75 - 60) / 10 = +1.5\n",
        "```\n",
        "\n",
        "The raw value 38 would have a Z-score of -2.2:\n",
        "\n",
        "```\n",
        "  Z-score = (38 - 60) / 10 = -2.2\n",
        "```\n",
        "\n",
        "Now normalize the numerical values in the rice dataset by converting them to Z-scores."
      ]
    },
    {
      "metadata": {
        "id": "hSUjPSwNiyBP"
      },
      "cell_type": "code",
      "source": [
        "# Calculate the Z-scores of each numerical column in the raw data and write\n",
        "# them into a new DataFrame named df_norm.\n",
        "\n",
        "feature_mean = rice_dataset.mean(numeric_only=True)\n",
        "feature_std = rice_dataset.std(numeric_only=True)\n",
        "numerical_features = rice_dataset.select_dtypes('number').columns\n",
        "normalized_dataset = (\n",
        "    rice_dataset[numerical_features] - feature_mean\n",
        ") / feature_std\n",
        "\n",
        "# Copy the class to the new dataframe\n",
        "normalized_dataset['Class'] = rice_dataset['Class']\n",
        "\n",
        "# Examine some of the values of the normalized training set. Notice that most\n",
        "# Z-scores fall between -2 and +2.\n",
        "normalized_dataset.head()"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {
        "id": "D5aXjXq-YIkL"
      },
      "cell_type": "markdown",
      "source": [
        "# Set the random seeds\n",
        "\n",
        "To make experiments reproducible, we set the seed of the random number generators. This means that the order in which the data is shuffled, the values of the random weight initializations, etc, will all be the same each time the colab is run."
      ]
    },
    {
      "metadata": {
        "id": "bu257GAFYH-N"
      },
      "cell_type": "code",
      "source": [
        "keras.utils.set_random_seed(42)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {
        "id": "p7M9I-ekT1dV"
      },
      "cell_type": "markdown",
      "source": [
        "## Label and split data\n",
        "\n",
        "To train the model, we'll arbritrarily assign the Cammeo species a label of '1' and the Osmancik species a label of '0'."
      ]
    },
    {
      "metadata": {
        "id": "F4_yTxWdvPqz"
      },
      "cell_type": "code",
      "source": [
        "# Create a column setting the Cammeo label to '1' and the Osmancik label to '0'\n",
        "# then show 10 randomly selected rows.\n",
        "normalized_dataset['Class_Bool'] = (\n",
        "    # Returns true if class is Cammeo, and false if class is Osmancik\n",
        "    normalized_dataset['Class'] == 'Cammeo'\n",
        ").astype(int)\n",
        "normalized_dataset.sample(10)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {
        "id": "VBY8b0akUqiQ"
      },
      "cell_type": "markdown",
      "source": [
        "We can then randomize and partition the dataset into train, validation, and test splits, consisting of 80%, 10%, and 10% of the dataset respectively.\n",
        "\n",
        "We will use the training data to learn the model parameters, then use the validation data to evaluate different models, and finally use the test data to compute the final metrics for the model that performed best on the validation data."
      ]
    },
    {
      "metadata": {
        "id": "XE-RAq0av1wv"
      },
      "cell_type": "code",
      "source": [
        "# Create indices at the 80th and 90th percentiles\n",
        "number_samples = len(normalized_dataset)\n",
        "index_80th = round(number_samples * 0.8)\n",
        "index_90th = index_80th + round(number_samples * 0.1)\n",
        "\n",
        "# Randomize order and split into train, validation, and test with a .8, .1, .1 split\n",
        "shuffled_dataset = normalized_dataset.sample(frac=1, random_state=100)\n",
        "train_data = shuffled_dataset.iloc[0:index_80th]\n",
        "validation_data = shuffled_dataset.iloc[index_80th:index_90th]\n",
        "test_data = shuffled_dataset.iloc[index_90th:]\n",
        "\n",
        "# Show the first five rows of the last split\n",
        "test_data.head()"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {
        "id": "7Iq_haqJYeSH"
      },
      "cell_type": "markdown",
      "source": [
        "It's important to prevent the model from getting the label as input during training, which is called label leakage. This can be done by storing features and labels as separate variables."
      ]
    },
    {
      "metadata": {
        "id": "_Gi0VaAAYiaO"
      },
      "cell_type": "code",
      "source": [
        "label_columns = ['Class', 'Class_Bool']\n",
        "\n",
        "train_features = train_data.drop(columns=label_columns)\n",
        "train_labels = train_data['Class_Bool'].to_numpy()\n",
        "validation_features = validation_data.drop(columns=label_columns)\n",
        "validation_labels = validation_data['Class_Bool'].to_numpy()\n",
        "test_features = test_data.drop(columns=label_columns)\n",
        "test_labels = test_data['Class_Bool'].to_numpy()"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {
        "id": "U-kTF2rTY-K8"
      },
      "cell_type": "markdown",
      "source": [
        "## Train the model\n",
        "\n",
        "### Choose the input features\n",
        "\n",
        "To start with, we'll train a model on `Eccentricity`, `Major_Axis_Length,` and `Area`."
      ]
    },
    {
      "metadata": {
        "id": "7v_UNIPBtjoz"
      },
      "cell_type": "code",
      "source": [
        "# Name of the features we'll train our model on.\n",
        "input_features = [\n",
        "    'Eccentricity',\n",
        "    'Major_Axis_Length',\n",
        "    'Area',\n",
        "]"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {
        "id": "cSBegHR5rSEn"
      },
      "cell_type": "markdown",
      "source": [
        "## Define functions that build and train a model\n",
        "\n",
        "The following code cell defines two functions:\n",
        "\n",
        "  * `create_model(inputs, learning_rate, metrics)`, which defines the model's architecture.\n",
        "  * `train_model(model, dataset, epochs, label_name, batch_size, shuffle)`, uses input features and labels to train the model.\n",
        "\n",
        "Note: create_model applies the sigmoid function to perform [logistic regression](https://developers.google.com/machine-learning/crash-course/logistic-regression).\n",
        "\n",
        "We also define two helpful data structures: `ExperimentSettings` and `Experiment`. We use these simple classes to keep track of our experiments, allowing us to know what hyperparameters were used and what the results were. In `ExperimentSettings`, we store all values describing an experiment (i.e., hyperparameters). Then, we store the results of a training run (i.e., the model and the training metrics) into an `Experiment` instance, along with the `ExperimentSettings` used for that experiment."
      ]
    },
    {
      "metadata": {
        "id": "8B2VArcKH6UX"
      },
      "cell_type": "code",
      "source": [
        "# @title Define the functions that create and train a model.\n",
        "\n",
        "\n",
        "def create_model(\n",
        "    settings: ml_edu.experiment.ExperimentSettings,\n",
        "    metrics: list[keras.metrics.Metric],\n",
        ") -> keras.Model:\n",
        "  \"\"\"Create and compile a simple classification model.\"\"\"\n",
        "  model_inputs = [\n",
        "      keras.Input(name=feature, shape=(1,))\n",
        "      for feature in settings.input_features\n",
        "  ]\n",
        "  # Use a Concatenate layer to assemble the different inputs into a single\n",
        "  # tensor which will be given as input to the Dense layer.\n",
        "  # For example: [input_1[0][0], input_2[0][0]]\n",
        "\n",
        "  concatenated_inputs = keras.layers.Concatenate()(model_inputs)\n",
        "  model_output = keras.layers.Dense(\n",
        "      units=1, name='dense_layer', activation=keras.activations.sigmoid\n",
        "  )(concatenated_inputs)\n",
        "  model = keras.Model(inputs=model_inputs, outputs=model_output)\n",
        "  # Call the compile method to transform the layers into a model that\n",
        "  # Keras can execute.  Notice that we're using a different loss\n",
        "  # function for classification than for regression.\n",
        "  model.compile(\n",
        "      optimizer=keras.optimizers.RMSprop(\n",
        "          settings.learning_rate\n",
        "      ),\n",
        "      loss=keras.losses.BinaryCrossentropy(),\n",
        "      metrics=metrics,\n",
        "  )\n",
        "  return model\n",
        "\n",
        "\n",
        "def train_model(\n",
        "    experiment_name: str,\n",
        "    model: keras.Model,\n",
        "    dataset: pd.DataFrame,\n",
        "    labels: np.ndarray,\n",
        "    settings: ml_edu.experiment.ExperimentSettings,\n",
        ") -> ml_edu.experiment.Experiment:\n",
        "  \"\"\"Feed a dataset into the model in order to train it.\"\"\"\n",
        "\n",
        "  # The x parameter of keras.Model.fit can be a list of arrays, where\n",
        "  # each array contains the data for one feature.\n",
        "  features = {\n",
        "      feature_name: np.array(dataset[feature_name])\n",
        "      for feature_name in settings.input_features\n",
        "  }\n",
        "\n",
        "  history = model.fit(\n",
        "      x=features,\n",
        "      y=labels,\n",
        "      batch_size=settings.batch_size,\n",
        "      epochs=settings.number_epochs,\n",
        "  )\n",
        "\n",
        "  return ml_edu.experiment.Experiment(\n",
        "      name=experiment_name,\n",
        "      settings=settings,\n",
        "      model=model,\n",
        "      epochs=history.epoch,\n",
        "      metrics_history=pd.DataFrame(history.history),\n",
        "  )\n",
        "\n",
        "\n",
        "print('Defined the create_model and train_model functions.')"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {
        "id": "Ak_TMAzGOIFq"
      },
      "cell_type": "markdown",
      "source": [
        "## Define a plotting function\n",
        "\n",
        "The following [matplotlib](https://developers.google.com/machine-learning/glossary/#matplotlib) function plots one or more curves, showing how various classification metrics change with each epoch."
      ]
    },
    {
      "metadata": {
        "id": "D-IXYVfvM4gD"
      },
      "cell_type": "markdown",
      "source": [
        "## Invoke the creating, training, and plotting functions\n",
        "\n",
        "The following code specifies the hyperparameters, invokes the\n",
        "functions to create and train the model, then plots the results, including accuracy, precision, and recall.\n",
        "\n",
        "Classification threshold is set at .35. Try playing with the threshold, then the learning rate, to see what changes."
      ]
    },
    {
      "metadata": {
        "id": "Q82E6tS13O_2"
      },
      "cell_type": "code",
      "source": [
        "# Let's define our first experiment settings.\n",
        "settings = ml_edu.experiment.ExperimentSettings(\n",
        "    learning_rate=0.001,\n",
        "    number_epochs=60,\n",
        "    batch_size=100,\n",
        "    classification_threshold=0.35,\n",
        "    input_features=input_features,\n",
        ")\n",
        "\n",
        "metrics = [\n",
        "    keras.metrics.BinaryAccuracy(\n",
        "        name='accuracy', threshold=settings.classification_threshold\n",
        "    ),\n",
        "    keras.metrics.Precision(\n",
        "        name='precision', thresholds=settings.classification_threshold\n",
        "    ),\n",
        "    keras.metrics.Recall(\n",
        "        name='recall', thresholds=settings.classification_threshold\n",
        "    ),\n",
        "    keras.metrics.AUC(num_thresholds=100, name='auc'),\n",
        "]\n",
        "\n",
        "# Establish the model's topography.\n",
        "model = create_model(settings, metrics)\n",
        "\n",
        "# Train the model on the training set.\n",
        "experiment = train_model(\n",
        "    'baseline', model, train_features, train_labels, settings\n",
        ")\n",
        "\n",
        "# Plot metrics vs. epochs\n",
        "ml_edu.results.plot_experiment_metrics(experiment, ['accuracy', 'precision', 'recall'])\n",
        "ml_edu.results.plot_experiment_metrics(experiment, ['auc'])"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {
        "id": "RfxkB-_vwUwq"
      },
      "cell_type": "markdown",
      "source": [
        "AUC is calculated across all possible thresholds (in practice in the code above, 100 thresholds), while accuracy, precision, and recall are calculated for only the specified threshold. For this reason they are shown separately above."
      ]
    },
    {
      "metadata": {
        "id": "u8y8vKBGsv0m"
      },
      "cell_type": "markdown",
      "source": [
        "## Evaluate the model against the validation set\n",
        "\n",
        "At the end of model training, you ended up with a certain accuracy against the *training set*. Invoke the following code cell to determine your model's accuracy against the *validation set*."
      ]
    },
    {
      "metadata": {
        "id": "bHh53BX44R94"
      },
      "cell_type": "code",
      "source": [
        "def compare_train_validation(experiment: ml_edu.experiment.Experiment, validation_metrics: dict[str, float]):\n",
        "  print('Comparing metrics between train and validation:')\n",
        "  for metric, validation_value in validation_metrics.items():\n",
        "    print('------')\n",
        "    print(f'Train {metric}: {experiment.get_final_metric_value(metric):.4f}')\n",
        "    print(f'Validation {metric}:  {validation_value:.4f}')\n",
        "\n",
        "\n",
        "# Evaluate validation metrics\n",
        "validation_metrics = experiment.evaluate(validation_features, validation_labels)\n",
        "compare_train_validation(experiment, validation_metrics)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {
        "id": "ku6nq9KDtL6u"
      },
      "cell_type": "markdown",
      "source": [
        "It appears that the model, which achieved ~92% accuracy on the training data, still shows an accuracy of about 90% on the validation data. Can we do better? Let's train a model using all seven available features and compare the AUC."
      ]
    },
    {
      "metadata": {
        "id": "72Mfkp3RxQii"
      },
      "cell_type": "code",
      "source": [
        "# Features used to train the model on.\n",
        "# Specify all features.\n",
        "all_input_features = [\n",
        "  'Eccentricity',\n",
        "  'Major_Axis_Length',\n",
        "  'Minor_Axis_Length',\n",
        "  ? Your code here\n",
        "]"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {
        "cellView": "form",
        "id": "xfdeJjoUNmTv"
      },
      "cell_type": "code",
      "source": [
        "#@title Solution\n",
        "# Features used to train the model on.\n",
        "# Specify all features.\n",
        "all_input_features = [\n",
        "  'Eccentricity',\n",
        "  'Major_Axis_Length',\n",
        "  'Minor_Axis_Length',\n",
        "  'Area',\n",
        "  'Convex_Area',\n",
        "  'Perimeter',\n",
        "  'Extent',\n",
        "]"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {
        "id": "Hql2nxXqxuBg"
      },
      "cell_type": "markdown",
      "source": [
        "## Train the full-featured model and calculate metrics"
      ]
    },
    {
      "metadata": {
        "id": "-85dcJ3ntocd"
      },
      "cell_type": "code",
      "source": [
        "settings_all_features = ml_edu.experiment.ExperimentSettings(\n",
        "    learning_rate=0.001,\n",
        "    number_epochs=60,\n",
        "    batch_size=100,\n",
        "    classification_threshold=0.5,\n",
        "    input_features=all_input_features,\n",
        ")\n",
        "\n",
        "# Modify the following definition of METRICS to generate\n",
        "# not only accuracy and precision, but also recall:\n",
        "metrics = [\n",
        "    keras.metrics.BinaryAccuracy(\n",
        "        name='accuracy',\n",
        "        threshold=settings_all_features.classification_threshold,\n",
        "    ),\n",
        "    keras.metrics.Precision(\n",
        "        name='precision',\n",
        "        thresholds=settings_all_features.classification_threshold,\n",
        "    ),\n",
        "    keras.metrics.Recall(\n",
        "        name='recall', thresholds=settings_all_features.classification_threshold\n",
        "    ),\n",
        "    keras.metrics.AUC(num_thresholds=100, name='auc'),\n",
        "]\n",
        "\n",
        "# Establish the model's topography.\n",
        "model_all_features = create_model(settings_all_features, metrics)\n",
        "\n",
        "# Train the model on the training set.\n",
        "experiment_all_features = train_model(\n",
        "    'all features',\n",
        "    model_all_features,\n",
        "    train_features,\n",
        "    train_labels,\n",
        "    settings_all_features,\n",
        ")\n",
        "\n",
        "# Plot metrics vs. epochs\n",
        "ml_edu.results.plot_experiment_metrics(\n",
        "    experiment_all_features, ['accuracy', 'precision', 'recall']\n",
        ")\n",
        "ml_edu.results.plot_experiment_metrics(experiment_all_features, ['auc'])"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {
        "id": "E5ndvrnjzXCo"
      },
      "cell_type": "markdown",
      "source": [
        "## Evaluate full-featured model on validation split"
      ]
    },
    {
      "metadata": {
        "id": "-BklcY6pyDrY"
      },
      "cell_type": "code",
      "source": [
        "validation_metrics_all_features = experiment_all_features.evaluate(\n",
        "    validation_features,\n",
        "    validation_labels,\n",
        ")\n",
        "compare_train_validation(experiment_all_features, validation_metrics_all_features)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {
        "id": "wTr_boLBze2k"
      },
      "cell_type": "markdown",
      "source": [
        "This second model has closer train and validation metrics, suggesting it overfit less to the training data."
      ]
    },
    {
      "metadata": {
        "id": "EqgyfbXXawq4"
      },
      "cell_type": "markdown",
      "source": [
        "# Comparing our two models\n",
        "\n",
        "With our simple experimentation framework, we can keep track of which experiments we ran, and what the results were. We also defined a helper function which allows us to easily compare two or more models, both during training and when evaluated on the validation set."
      ]
    },
    {
      "metadata": {
        "id": "JhbgA_FEayYU"
      },
      "cell_type": "code",
      "source": [
        "ml_edu.results.compare_experiment([experiment, experiment_all_features],\n",
        "                                  ['accuracy', 'auc'],\n",
        "                                  validation_features, validation_labels)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {
        "id": "sKIuJGOTbNWz"
      },
      "cell_type": "markdown",
      "source": [
        "Comparing the two models, both have AUC of ~.97-.98. There does not seem to be a large gain in model quality when adding the other four features, which makes sense, given that many of the features (area, perimeter, and convex area, for example) are interrelated."
      ]
    },
    {
      "metadata": {
        "id": "jRRAx9FRWEij"
      },
      "cell_type": "markdown",
      "source": [
        "# Computing the final test metrics\n",
        "\n",
        "To estimate the peformance of our model on unseen data, we can now compute the metrics of the best model on the test data. This final step must be done once experimentation is over and we have selected the model we want to use. Any model comparison must be done using the validation set, to avoid accidentally selecting a model which is tailored for our test set. \n",
        "\n",
        "This final step is also the opportunity to check for potential overfitting: if the validation and test metrics are very different, it might be a sign that the selection process done using the validation set led to a model that doesn't generalize well, possibly because the validation set is not representative of the overall data distribution. In that case, the best solution is to shuffle the data and re-assign the train, validation, and test sets, before running your experiments again."
      ]
    },
    {
      "metadata": {
        "id": "CVOCsf-rXSDp"
      },
      "cell_type": "code",
      "source": [
        "test_metrics_all_features = experiment_all_features.evaluate(\n",
        "    test_features,\n",
        "    test_labels,\n",
        ")\n",
        "for metric, test_value in test_metrics_all_features.items():\n",
        "  print(f'Test {metric}:  {test_value:.4f}')"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {
        "id": "_7NIsBQUXmw0"
      },
      "cell_type": "markdown",
      "source": [
        "In this case, we see that the test accuracy is about 92%, which is close to the validation accuracy we obtained above. This means that our model should perform equally well on new, unseen data!"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}