{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Part 3a - Starting From Predefined Models\n",
"\n",
"In this notebook we will cover the following topics:\n",
"\n",
"* Loading predefined networks that come with Keras\n",
"* Retraining from scratch\n",
"* Partial retraining"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Why Predefined Networks?\n",
"\n",
"Although we have made our own network from scratch to learn about deep learning components, in practice you will often want to use a standard network that has been found by deep learning researchers to be successful.\n",
"\n",
"Keras comes with several popular networks already defined, and can even load them with weights from standard datasets. Keras calls these premade networks [\"applications\"](https://keras.io/applications/). Many popular networks are included, like:\n",
"\n",
"* Xception\n",
"* VGG16\n",
"* VGG19\n",
"* ResNet, ResNetV2, ResNeXt\n",
"* InceptionV3\n",
"* InceptionResNetV2\n",
"* MobileNet\n",
"* MobileNetV2\n",
"* DenseNet\n",
"* NASNet\n",
"\n",
"Let's try out the *InceptionV3* network, which is a popular image recognition network."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"np.warnings.filterwarnings('ignore') # Hide np.floating warning\n",
"\n",
"import keras\n",
"\n",
"from keras.datasets import cifar10\n",
"from keras.models import Sequential\n",
"from keras.layers import Dense, Dropout, Flatten\n",
"from keras.layers import Conv2D, MaxPooling2D, Input, Lambda\n",
"\n",
"# Prevent TensorFlow from grabbing all the GPU memory\n",
"import tensorflow as tf\n",
"\n",
"gpu_devices = tf.config.experimental.list_physical_devices('GPU')\n",
"for device in gpu_devices:\n",
" tf.config.experimental.set_memory_growth(device, True)\n",
"\n",
"import holoviews as hv\n",
"hv.extension('bokeh')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Load the Data\n",
"\n",
"Same data preparation as before."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from keras.datasets import cifar10\n",
"import keras.utils\n",
"\n",
"(x_train, y_train), (x_test, y_test) = cifar10.load_data()\n",
"\n",
"# Save an unmodified copy of y_test for later, flattened to one column\n",
"y_test_true = y_test[:,0].copy()\n",
"\n",
"x_train = x_train.astype('float32')\n",
"x_test = x_test.astype('float32')\n",
"x_train /= 255\n",
"x_test /= 255\n",
"\n",
"num_classes = 10\n",
"y_train = keras.utils.to_categorical(y_train, num_classes)\n",
"y_test = keras.utils.to_categorical(y_test, num_classes)\n",
"\n",
"# The data only has numeric categories so we also have the string labels below \n",
"cifar10_labels = np.array(['airplane', 'automobile', 'bird', 'cat', 'deer', \n",
" 'dog', 'frog', 'horse', 'ship', 'truck'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Load InceptionV3\n",
"\n",
"When we load a network, we have a number of options we can set. Some of the more important ones are:\n",
"\n",
"* `input_shape`: Pretrained networks assume a particular image input size. If your data is not this shape, Keras will allow you to set it here, but some models have limitations. InceptionV3 cannot go below 75 by 75.\n",
"* `weights`: What weights to load with the model. Default is random weights or `'imagenet'`, which loads weights from training on the ImageNet dataset. We will try the pretrained weights in a later section.\n",
"* `include_top`: Include the dense layer at the end of the network? If you are loading pre-trained weights, you will likely need to replace the top layer with your own.\n",
"* `classes`: Number of classes to output. Needed if `include_top` is True and `weights` is None.\n",
"\n",
"For our first attempt, we'll start training the model from scratch. Because we are dealing with such small images, we'll need to built a custom first layer to rescale the image up by a factor of 3. We can do this using a `Lambda` layer, which lets us call backend (TensorFlow in this case) tensor manipulation functions. Keras provides a `resize_images` function which will scale up the images.\n",
"\n",
"We also use the \"functional\" API of Keras here, where we connect one layer to the next by treating each layer like a function and passing the preceding layer to it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from keras import backend as K\n",
"from keras.layers import Input, Lambda, GlobalAveragePooling2D\n",
"from keras.models import Model\n",
"\n",
"# Rescale input from 32x32 to 96x96\n",
"input_layer = Input(shape=(32,32,3), dtype=np.float32)\n",
"resize_layer = Lambda(lambda x: K.resize_images(x, 3, 3, 'channels_last', interpolation='nearest'))(input_layer)\n",
"\n",
"# Load InceptionV3 with random initial weights\n",
"inception = keras.applications.InceptionV3(\n",
" input_shape=(96,96,3), # must be larger than 75x75\n",
" weights=None, # random weights\n",
" include_top=True, \n",
" classes=num_classes, \n",
")(resize_layer)\n",
"\n",
"model = Model(inputs=[input_layer], outputs=[inception])\n",
"\n",
"model.compile(loss=keras.losses.categorical_crossentropy,\n",
" optimizer=keras.optimizers.Adadelta(),\n",
" metrics=['accuracy'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's see how many parameters the model has:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"model.summary()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The InceptionV3 model is signficantly deeper than our toy models before. Let's see how Keras handles it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"history = model.fit(x_train, y_train,\n",
" batch_size=256,\n",
" epochs=5,\n",
" verbose=1,\n",
" validation_data=(x_test, y_test))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"train_acc = hv.Curve((history.epoch, history.history['accuracy']), 'epoch', 'accuracy', label='training')\n",
"val_acc = hv.Curve((history.epoch, history.history['val_accuracy']), 'epoch', 'accuracy', label='validation')\n",
"\n",
"layout = (train_acc * val_acc).redim(accuracy=dict(range=(0.0, 1.1)))\n",
"\n",
"layout.opts(\n",
" hv.opts.Curve(width=400, height=300, line_width=3),\n",
" hv.opts.Overlay(legend_position='bottom_right')\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This model seems to be training reasonably well, even with completely random starting weights. Let's try seeding the model with a starting point."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Transfer Learning\n",
"\n",
"Given the relatively small size of our training dataset, it can be hard to retrain a complex predefined model entirely from scratch. Let's try to retrain a model starting from the ImageNet weights:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from keras import backend as K\n",
"from keras.layers import Input, Lambda, GlobalAveragePooling2D\n",
"from keras.models import Model\n",
"\n",
"# Rescale input from 32x32 to 96x96\n",
"input_layer = Input(shape=(32,32,3), dtype=np.float32)\n",
"resize_layer = Lambda(lambda x: K.resize_images(x, 3, 3, 'channels_last', interpolation='nearest'))(input_layer)\n",
"\n",
"# Load InceptionV3 with imagenet weights, but removing the top dense layers\n",
"inception = keras.applications.InceptionV3(\n",
" input_shape=(96,96,3), # our scaled up dimension >= 75\n",
" weights='imagenet', # random weights\n",
" include_top=False, # we are going to replace the top of the network with our own layers\n",
")\n",
"#inception.trainable = False # uncomment this to freeze the loaded weights \n",
"\n",
"# Add our own top layers to produce 10 categories, but also adding dropout to control overfitting\n",
"prediction = Flatten()(inception(resize_layer))\n",
"prediction = Dropout(0.25)(prediction)\n",
"prediction = Dense(num_classes, activation='softmax')(prediction)\n",
"\n",
"model2 = Model(inputs=[input_layer], outputs=[prediction])\n",
"\n",
"model2.compile(loss=keras.losses.categorical_crossentropy,\n",
" optimizer=keras.optimizers.Adadelta(),\n",
" metrics=['accuracy'])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model2.summary()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"history2 = model2.fit(x_train, y_train,\n",
" batch_size=256,\n",
" epochs=5,\n",
" verbose=1,\n",
" validation_data=(x_test, y_test))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"train_acc = hv.Curve((history2.epoch, history2.history['accuracy']), 'epoch', 'accuracy', label='training')\n",
"val_acc = hv.Curve((history2.epoch, history2.history['val_accuracy']), 'epoch', 'accuracy', label='validation')\n",
"\n",
"layout = (train_acc * val_acc).redim(accuracy=dict(range=(0.0, 1.1)))\n",
"\n",
"layout.opts(\n",
" hv.opts.Curve(width=400, height=300, line_width=3),\n",
" hv.opts.Overlay(legend_position='bottom_right')\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Experiments to Try\n",
"\n",
"* We allowed the ImageNet weights to vary in the transfer learning example. If we froze the imagenet weights (`inception.trainable = False`), does the training still work?\n",
"* Does the interpolation scheme used to scale up the image matter? Try `interpolation='bilinear'`.\n",
"* Take a look at the [other models Keras includes](https://keras.io/applications/). Try using some of the other ones. Note that not all of them will work!\n",
"\n",
"If you screw everything up, you can use File / Revert to Checkpoint to go back to the first version of the notebook and restart the Jupyter kernel with Kernel / Restart."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
}
},
"nbformat": 4,
"nbformat_minor": 4
}