{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "zg02FZzDyEqd" }, "source": [ "##### Copyright 2019 The TensorFlow Authors.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "id": "2mapZ9afGJ69" }, "outputs": [], "source": [ "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "metadata": { "id": "sMYQvJuBi7MS" }, "source": [ "# Classify structured data using Keras Preprocessing Layers" ] }, { "cell_type": "markdown", "metadata": { "id": "8FaL4wnr22oy" }, "source": [ "\n", " \n", " \n", " \n", " \n", "
\n", " \n", " \n", " View on TensorFlow.org\n", " \n", " \n", " \n", " Run in Google Colab\n", " \n", " \n", " \n", " View source on GitHub\n", " \n", " Download notebook\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "Nna1tOKxyEqe" }, "source": [ "This tutorial demonstrates how to classify structured data (e.g. tabular data in a CSV). You will use [Keras](https://www.tensorflow.org/guide/keras) to define the model, and [preprocessing layers](https://keras.io/guides/preprocessing_layers/) as a bridge to map from columns in a CSV to features used to train the model. This tutorial contains complete code to:\n", "\n", "* Load a CSV file using [Pandas](https://pandas.pydata.org/).\n", "* Build an input pipeline to batch and shuffle the rows using [tf.data](https://www.tensorflow.org/guide/datasets).\n", "* Map from columns in the CSV to features used to train the model using Keras Preprocessing layers.\n", "* Build, train, and evaluate a model using Keras." ] }, { "cell_type": "markdown", "metadata": { "id": "h5xkXCicjFQD" }, "source": [ "Note: This tutorial is similar to [Classify structured data with feature columns](https://www.tensorflow.org/tutorials/structured_data/feature_columns). This version uses new experimental Keras [Preprocessing Layers](https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing) instead of `tf.feature_column`. Keras Preprocessing Layers are more intuitive, and can be easily included inside your model to simplify deployment." ] }, { "cell_type": "markdown", "metadata": { "id": "ZHxU1FMNpomc" }, "source": [ "## The Dataset\n", "\n", "You will use a simplified version of the PetFinder [dataset](https://www.kaggle.com/c/petfinder-adoption-prediction). There are several thousand rows in the CSV. Each row describes a pet, and each column describes an attribute. You will use this information to predict if the pet will be adopted.\n", "\n", "Following is a description of this dataset. Notice there are both numeric and categorical columns. There is a free text column which you will not use in this tutorial.\n", "\n", "Column | Description| Feature Type | Data Type\n", "------------|--------------------|----------------------|-----------------\n", "Type | Type of animal (Dog, Cat) | Categorical | string\n", "Age | Age of the pet | Numerical | integer\n", "Breed1 | Primary breed of the pet | Categorical | string\n", "Color1 | Color 1 of pet | Categorical | string\n", "Color2 | Color 2 of pet | Categorical | string\n", "MaturitySize | Size at maturity | Categorical | string\n", "FurLength | Fur length | Categorical | string\n", "Vaccinated | Pet has been vaccinated | Categorical | string\n", "Sterilized | Pet has been sterilized | Categorical | string\n", "Health | Health Condition | Categorical | string\n", "Fee | Adoption Fee | Numerical | integer\n", "Description | Profile write-up for this pet | Text | string\n", "PhotoAmt | Total uploaded photos for this pet | Numerical | integer\n", "AdoptionSpeed | Speed of adoption | Classification | integer" ] }, { "cell_type": "markdown", "metadata": { "id": "vjFbdBldyEqf" }, "source": [ "## Import TensorFlow and other libraries\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "S_BdyQlPjfDW" }, "outputs": [], "source": [ "!pip install -q sklearn" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "LklnLlt6yEqf" }, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import tensorflow as tf\n", "\n", "from sklearn.model_selection import train_test_split\n", "from tensorflow.keras import layers\n", "from tensorflow.keras.layers.experimental import preprocessing" ] }, { "cell_type": "markdown", "metadata": { "id": "UXvBvobayEqi" }, "source": [ "## Use Pandas to create a dataframe\n", "\n", "[Pandas](https://pandas.pydata.org/) is a Python library with many helpful utilities for loading and working with structured data. You will use Pandas to download the dataset from a URL, and load it into a dataframe." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "qJ4Ajn-YyEqj" }, "outputs": [], "source": [ "import pathlib\n", "\n", "dataset_url = 'http://storage.googleapis.com/download.tensorflow.org/data/petfinder-mini.zip'\n", "csv_file = 'datasets/petfinder-mini/petfinder-mini.csv'\n", "\n", "tf.keras.utils.get_file('petfinder_mini.zip', dataset_url,\n", " extract=True, cache_dir='.')\n", "dataframe = pd.read_csv(csv_file)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "3uiq4hoIGyXI" }, "outputs": [], "source": [ "dataframe.head()" ] }, { "cell_type": "markdown", "metadata": { "id": "C3zDbrozyEqq" }, "source": [ "## Create target variable\n", "\n", "The task in the Kaggle competition is to predict the speed at which a pet will be adopted (e.g., in the first week, the first month, the first three months, and so on). Let's simplify this for our tutorial. Here, you will transform this into a binary classification problem, and simply predict whether the pet was adopted, or not.\n", "\n", "After modifying the label column, 0 will indicate the pet was not adopted, and 1 will indicate it was." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "wmMDc46-yEqq" }, "outputs": [], "source": [ "# In the original dataset \"4\" indicates the pet was not adopted.\n", "dataframe['target'] = np.where(dataframe['AdoptionSpeed']==4, 0, 1)\n", "\n", "# Drop un-used columns.\n", "dataframe = dataframe.drop(columns=['AdoptionSpeed', 'Description'])" ] }, { "cell_type": "markdown", "metadata": { "id": "sp0NCbswyEqs" }, "source": [ "## Split the dataframe into train, validation, and test\n", "\n", "The dataset you downloaded was a single CSV file. You will split this into train, validation, and test sets." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "qT6HdyEwyEqt" }, "outputs": [], "source": [ "train, test = train_test_split(dataframe, test_size=0.2)\n", "train, val = train_test_split(train, test_size=0.2)\n", "print(len(train), 'train examples')\n", "print(len(val), 'validation examples')\n", "print(len(test), 'test examples')" ] }, { "cell_type": "markdown", "metadata": { "id": "C_7uVu-xyEqv" }, "source": [ "## Create an input pipeline using tf.data\n", "\n", "Next, you will wrap the dataframes with [tf.data](https://www.tensorflow.org/guide/datasets), in order to shuffle and batch the data. If you were working with a very large CSV file (so large that it does not fit into memory), you would use tf.data to read it from disk directly. That is not covered in this tutorial." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "7r4j-1lRyEqw" }, "outputs": [], "source": [ "# A utility method to create a tf.data dataset from a Pandas Dataframe\n", "def df_to_dataset(dataframe, shuffle=True, batch_size=32):\n", " dataframe = dataframe.copy()\n", " labels = dataframe.pop('target')\n", " ds = tf.data.Dataset.from_tensor_slices((dict(dataframe), labels))\n", " if shuffle:\n", " ds = ds.shuffle(buffer_size=len(dataframe))\n", " ds = ds.batch(batch_size)\n", " ds = ds.prefetch(batch_size)\n", " return ds" ] }, { "cell_type": "markdown", "metadata": { "id": "PYxIXH579uS9" }, "source": [ "Now that you have created the input pipeline, let's call it to see the format of the data it returns. You have used a small batch size to keep the output readable." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "tYiNH-QI96Jo" }, "outputs": [], "source": [ "batch_size = 5\n", "train_ds = df_to_dataset(train, batch_size=batch_size)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "nFYir6S8HgIJ" }, "outputs": [], "source": [ "[(train_features, label_batch)] = train_ds.take(1)\n", "print('Every feature:', list(train_features.keys()))\n", "print('A batch of ages:', train_features['Age'])\n", "print('A batch of targets:', label_batch )" ] }, { "cell_type": "markdown", "metadata": { "id": "geqHWW54Hmte" }, "source": [ "You can see that the dataset returns a dictionary of column names (from the dataframe) that map to column values from rows in the dataframe." ] }, { "cell_type": "markdown", "metadata": { "id": "-v50jBIuj4gb" }, "source": [ "## Demonstrate the use of preprocessing layers.\n", "\n", "The Keras preprocessing layers API allows you to build Keras-native input processing pipelines. You will use 3 preprocessing layers to demonstrate the feature preprocessing code.\n", "\n", "* [`Normalization`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing/Normalization) - Feature-wise normalization of the data.\n", "* [`CategoryEncoding`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing/CategoryEncoding) - Category encoding layer.\n", "* [`StringLookup`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing/StringLookup) - Maps strings from a vocabulary to integer indices.\n", "* [`IntegerLookup`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing/IntegerLookup) - Maps integers from a vocabulary to integer indices.\n", "\n", "You can find a list of available preprocessing layers [here](https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing)." ] }, { "cell_type": "markdown", "metadata": { "id": "twXBSxnT66o8" }, "source": [ "### Numeric columns\n", "For each of the Numeric feature, you will use a Normalization() layer to make sure the mean of each feature is 0 and its standard deviation is 1." ] }, { "cell_type": "markdown", "metadata": { "id": "OosUh4kTsK_q" }, "source": [ "`get_normalization_layer` function returns a layer which applies featurewise normalization to numerical features." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "D6OuEKMMyEq1" }, "outputs": [], "source": [ "def get_normalization_layer(name, dataset):\n", " # Create a Normalization layer for our feature.\n", " normalizer = preprocessing.Normalization()\n", "\n", " # Prepare a Dataset that only yields our feature.\n", " feature_ds = dataset.map(lambda x, y: x[name])\n", "\n", " # Learn the statistics of the data.\n", " normalizer.adapt(feature_ds)\n", "\n", " return normalizer" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "MpKgUDyk69bM" }, "outputs": [], "source": [ "photo_count_col = train_features['PhotoAmt']\n", "layer = get_normalization_layer('PhotoAmt', train_ds)\n", "layer(photo_count_col)" ] }, { "cell_type": "markdown", "metadata": { "id": "foWY00YBUx9N" }, "source": [ "Note: If you many numeric features (hundreds, or more), it is more efficient to concatenate them first and use a single [normalization](https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing/Normalization) layer." ] }, { "cell_type": "markdown", "metadata": { "id": "yVD--2WZ7vmh" }, "source": [ "### Categorical columns\n", "In this dataset, Type is represented as a string (e.g. 'Dog', or 'Cat'). You cannot feed strings directly to a model. The preprocessing layer takes care of representing strings as a one-hot vector." ] }, { "cell_type": "markdown", "metadata": { "id": "LWlkOPwMsxdv" }, "source": [ "`get_category_encoding_layer` function returns a layer which maps values from a vocabulary to integer indices and one-hot encodes the features." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "GmgaeRjlDoUO" }, "outputs": [], "source": [ "def get_category_encoding_layer(name, dataset, dtype, max_tokens=None):\n", " # Create a StringLookup layer which will turn strings into integer indices\n", " if dtype == 'string':\n", " index = preprocessing.StringLookup(max_tokens=max_tokens)\n", " else:\n", " index = preprocessing.IntegerLookup(max_values=max_tokens)\n", "\n", " # Prepare a Dataset that only yields our feature\n", " feature_ds = dataset.map(lambda x, y: x[name])\n", "\n", " # Learn the set of possible values and assign them a fixed integer index.\n", " index.adapt(feature_ds)\n", "\n", " # Create a Discretization for our integer indices.\n", " encoder = preprocessing.CategoryEncoding(max_tokens=index.vocab_size())\n", "\n", " # Prepare a Dataset that only yields our feature.\n", " feature_ds = feature_ds.map(index)\n", "\n", " # Learn the space of possible indices.\n", " encoder.adapt(feature_ds)\n", "\n", " # Apply one-hot encoding to our indices. The lambda function captures the\n", " # layer so we can use them, or include them in the functional model later.\n", " return lambda feature: encoder(index(feature))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "X2t2ff9K8PcT" }, "outputs": [], "source": [ "type_col = train_features['Type']\n", "layer = get_category_encoding_layer('Type', train_ds, 'string')\n", "layer(type_col)" ] }, { "cell_type": "markdown", "metadata": { "id": "j6eDongw8knz" }, "source": [ "Often, you don't want to feed a number directly into the model, but instead use a one-hot encoding of those inputs. Consider raw data that represents a pet's age." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "7FjBioQ38oNE" }, "outputs": [], "source": [ "type_col = train_features['Age']\n", "category_encoding_layer = get_category_encoding_layer('Age', train_ds,\n", " 'int64', 5)\n", "category_encoding_layer(type_col)" ] }, { "cell_type": "markdown", "metadata": { "id": "SiE0glOPkMyh" }, "source": [ "## Choose which columns to use\n", "You have seen how to use several types of preprocessing layers. Now you will use them to train a model. You will be using [Keras-functional API](https://www.tensorflow.org/guide/keras/functional) to build the model. The Keras functional API is a way to create models that are more flexible than the [tf.keras.Sequential](https://www.tensorflow.org/api_docs/python/tf/keras/Sequential) API.\n", "\n", "The goal of this tutorial is to show you the complete code (e.g. mechanics) needed to work with preprocessing layers. A few columns have been selected arbitrarily to train our model.\n", "\n", "Key point: If your aim is to build an accurate model, try a larger dataset of your own, and think carefully about which features are the most meaningful to include, and how they should be represented." ] }, { "cell_type": "markdown", "metadata": { "id": "Uj1GoHSZ9R3H" }, "source": [ "Earlier, you used a small batch size to demonstrate the input pipeline. Let's now create a new input pipeline with a larger batch size.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Rcv2kQTTo23h" }, "outputs": [], "source": [ "batch_size = 256\n", "train_ds = df_to_dataset(train, batch_size=batch_size)\n", "val_ds = df_to_dataset(val, shuffle=False, batch_size=batch_size)\n", "test_ds = df_to_dataset(test, shuffle=False, batch_size=batch_size)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Q3RBa51VkaAn" }, "outputs": [], "source": [ "all_inputs = []\n", "encoded_features = []\n", "\n", "# Numeric features.\n", "for header in ['PhotoAmt', 'Fee']:\n", " numeric_col = tf.keras.Input(shape=(1,), name=header)\n", " normalization_layer = get_normalization_layer(header, train_ds)\n", " encoded_numeric_col = normalization_layer(numeric_col)\n", " all_inputs.append(numeric_col)\n", " encoded_features.append(encoded_numeric_col)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "1FOMGfZflhoA" }, "outputs": [], "source": [ "# Categorical features encoded as integers.\n", "age_col = tf.keras.Input(shape=(1,), name='Age', dtype='int64')\n", "encoding_layer = get_category_encoding_layer('Age', train_ds, dtype='int64',\n", " max_tokens=5)\n", "encoded_age_col = encoding_layer(age_col)\n", "all_inputs.append(age_col)\n", "encoded_features.append(encoded_age_col)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "K8C8xyiXm-Ie" }, "outputs": [], "source": [ "# Categorical features encoded as string.\n", "categorical_cols = ['Type', 'Color1', 'Color2', 'Gender', 'MaturitySize',\n", " 'FurLength', 'Vaccinated', 'Sterilized', 'Health', 'Breed1']\n", "for header in categorical_cols:\n", " categorical_col = tf.keras.Input(shape=(1,), name=header, dtype='string')\n", " encoding_layer = get_category_encoding_layer(header, train_ds, dtype='string',\n", " max_tokens=5)\n", " encoded_categorical_col = encoding_layer(categorical_col)\n", " all_inputs.append(categorical_col)\n", " encoded_features.append(encoded_categorical_col)\n" ] }, { "cell_type": "markdown", "metadata": { "id": "YHSnhz2fyEq3" }, "source": [ "## Create, compile, and train the model\n" ] }, { "cell_type": "markdown", "metadata": { "id": "IDGyN_wpo0XS" }, "source": [ "Now you can create our end-to-end model." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "6Yrj-_pr6jyL" }, "outputs": [], "source": [ "all_features = tf.keras.layers.concatenate(encoded_features)\n", "x = tf.keras.layers.Dense(32, activation=\"relu\")(all_features)\n", "x = tf.keras.layers.Dropout(0.5)(x)\n", "output = tf.keras.layers.Dense(1)(x)\n", "model = tf.keras.Model(all_inputs, output)\n", "model.compile(optimizer='adam',\n", " loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),\n", " metrics=[\"accuracy\"])" ] }, { "cell_type": "markdown", "metadata": { "id": "f6mNMfG6yEq5" }, "source": [ "Let's visualize our connectivity graph:\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Y7Bkx4c7yEq5" }, "outputs": [], "source": [ "# rankdir='LR' is used to make the graph horizontal.\n", "tf.keras.utils.plot_model(model, show_shapes=True, rankdir=\"LR\")\n" ] }, { "cell_type": "markdown", "metadata": { "id": "CED6OStLyEq7" }, "source": [ "### Train the model\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "OQfE3PC6yEq8" }, "outputs": [], "source": [ "model.fit(train_ds, epochs=10, validation_data=val_ds)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "T8N2uAdU2Cni" }, "outputs": [], "source": [ "loss, accuracy = model.evaluate(test_ds)\n", "print(\"Accuracy\", accuracy)" ] }, { "cell_type": "markdown", "metadata": { "id": "LmZMnTKaCZda" }, "source": [ "## Inference on new data\n", "\n", "Key point: The model you have developed can now classify a row from a CSV file directly, because the preprocessing code is included inside the model itself.\n" ] }, { "cell_type": "markdown", "metadata": { "id": "4xkOlK8Zweeh" }, "source": [ "You can now save and reload the Keras model. Follow the tutorial [here](https://www.tensorflow.org/tutorials/keras/save_and_load) for more information on TensorFlow models." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "QH9Zy1sBvwOH" }, "outputs": [], "source": [ "model.save('my_pet_classifier')\n", "reloaded_model = tf.keras.models.load_model('my_pet_classifier')" ] }, { "cell_type": "markdown", "metadata": { "id": "D973plJrdwQ9" }, "source": [ "To get a prediction for a new sample, you can simply call `model.predict()`. There are just two things you need to do:\n", "\n", "1. Wrap scalars into a list so as to have a batch dimension (models only process batches of data, not single samples)\n", "2. Call `convert_to_tensor` on each feature" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "rKq4pxtdDa7i" }, "outputs": [], "source": [ "sample = {\n", " 'Type': 'Cat',\n", " 'Age': 3,\n", " 'Breed1': 'Tabby',\n", " 'Gender': 'Male',\n", " 'Color1': 'Black',\n", " 'Color2': 'White',\n", " 'MaturitySize': 'Small',\n", " 'FurLength': 'Short',\n", " 'Vaccinated': 'No',\n", " 'Sterilized': 'No',\n", " 'Health': 'Healthy',\n", " 'Fee': 100,\n", " 'PhotoAmt': 2,\n", "}\n", "\n", "input_dict = {name: tf.convert_to_tensor([value]) for name, value in sample.items()}\n", "predictions = reloaded_model.predict(input_dict)\n", "prob = tf.nn.sigmoid(predictions[0])\n", "\n", "print(\n", " \"This particular pet had a %.1f percent probability \"\n", " \"of getting adopted.\" % (100 * prob)\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "XJQQZEiH2FaB" }, "source": [ "Key point: You will typically see best results with deep learning with larger and more complex datasets. When working with a small dataset like this one, we recommend using a decision tree or random forest as a strong baseline. The goal of this tutorial is to demonstrate the mechanics of working with structured data, so you have code to use as a starting point when working with your own datasets in the future." ] }, { "cell_type": "markdown", "metadata": { "id": "k0QAY2Tb2HYG" }, "source": [ "## Next steps\n", "The best way to learn more about classifying structured data is to try it yourself. You may want to find another dataset to work with, and training a model to classify it using code similar to the above. To improve accuracy, think carefully about which features to include in your model, and how they should be represented." ] } ], "metadata": { "colab": { "collapsed_sections": [], "name": "preprocessing_layers.ipynb", "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 0 }