{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# `Keras`\n", "\n", "[Keras](https://keras.io/) is a high-level neural networks API, written in Python and capable of running on top of [TensorFlow](https://www.tensorflow.org/), [CNTK](https://docs.microsoft.com/de-de/cognitive-toolkit/), or [Theano](http://www.deeplearning.net/software/theano/). It was developed with a focus on enabling fast experimentation. *Being able to go from idea to result with the least possible delay is key to doing good research.*\n", "\n", "**Note 1:** This is not an introduction to deep neural networks as this would explode the scope of this notebook. But we want to show you how you can implement a convoluted neural network to classify neuroimages, in our case fMRI images. \n", "**Note 2:** We want to thank [Anisha Keshavan](https://github.com/akeshavan) as a lot of the content in this notebook is coming from here [introduction notebook](http://nbviewer.jupyter.org/github/brainhack101/IntroDL/blob/master/IntroToKeras.ipynb) about Keras." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setup" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from nilearn import plotting\n", "%matplotlib inline\n", "import numpy as np\n", "import nibabel as nb\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "sns.set(style=\"darkgrid\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Load machine learning dataset\n", "\n", "Let's again load the dataset we prepared in the machine learning preparation notebook, plus the anatomical template image (we will need this for visualization)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "anat = nb.load('/home/neuro/workshop/notebooks/data/templates/MNI152_T1_1mm.nii.gz')\n", "func = nb.load('/home/neuro/workshop/notebooks/data/dataset_ML.nii.gz')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from nilearn.image import mean_img\n", "from nilearn.plotting import plot_anat" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot_anat(mean_img(func), cmap='magma', colorbar=False, display_mode='x', vmax=2, annotate=False,\n", " cut_coords=range(0, 49, 12), title='Mean value of machine learning dataset');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As a reminder, the shape of our machine learning dataset is as follows:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "func.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Specifying labels and chunks\n", "\n", "As in the `nilearn` and `PyMVPA` notebook, we need some chunks and label variables to train the neural network. The labels are important so that we can predict what we want to classify. And the chunks are just an easy way to make sure that the training and test dataset are split in an equal/balanced way.\n", "\n", "So, as before, we specify again which volumes of the dataset were recorded during eyes **closed** resting state and which ones were recorded during eyes **open** resting state recording.\n", "\n", "From the [Machine Learning Preparation](machine_learning_preparation.ipynb) notebook, we know that we have a total of 384 volumes in our `dataset_ML.nii.gz` file and that it's always 4 volumes of the condition `eyes closed`, followed by 4 volumes of the condition `eyes open`, etc. Therefore our labels should be as follows:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "labels = np.ravel([[['closed'] * 4, ['open'] * 4] for i in range(48)])\n", "labels[:20]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "***Second***, the `chunks` variable should not switch between subjects. So, as before, we can again specify 6 chunks of 64 volumes (8 subjects), each:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "chunks = np.ravel([[i] * 64 for i in range(6)])\n", "chunks[:150]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Keras - 2D Example\n", "\n", "Convoluted neural networks are very powerful (as you will see), but the computation power to train the models can be incredibly demanding. For this reason, it's sometimes recommended to try to reduce the input space if possible.\n", "\n", "In our case, we could try to not train the neural network only on one very thin slab (a few slices) of the brain. So, instead of taking the data matrix of the whole brain, we just take 2 slices in the region that we think is most likely to be predictive for the question at hand.\n", "\n", "We know (or suspect) that the regions with the most predictive power are probably somewhere around the eyes and in the visual cortex. So let's try to specify a few slices that cover those regions.\n", "\n", "So, let's try to just take a few slices around the eyes:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot_anat(mean_img(func).slicer[...,5:-25], cmap='magma', colorbar=False,\n", " display_mode='x', vmax=2, annotate=False, cut_coords=range(0, 49, 12),\n", " title='Slab of the machine learning mean image');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Hmm... That doesn't seem to work. We want to cover the eyes and the visual cortex. Like this, we're too far down in the back of the head (at the Cerebellum). One solution to this is to rotate the volume.\n", "\n", "So let's do that:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Rotation parameters\n", "phi = 0.35\n", "cos = np.cos(phi)\n", "sin = np.sin(phi)\n", "\n", "# Compute rotation matrix around x-axis\n", "rotation_affine = np.array([[1, 0, 0, 0],\n", " [0, cos, -sin, 0],\n", " [0, sin, cos, 0],\n", " [0, 0, 0, 1]])\n", "new_affine = rotation_affine.dot(func.affine)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Rotate and resample image to new orientation\n", "from nilearn.image import resample_img\n", "new_img = nb.Nifti1Image(func.get_data(), new_affine)\n", "img_rot = resample_img(new_img, func.affine, interpolation='continuous')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Delete zero-only rows and columns\n", "from nilearn.image import crop_img\n", "img_crop = crop_img(img_rot)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's check if the rotation worked." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot_anat(mean_img(img_crop), cmap='magma', colorbar=False, display_mode='x', vmax=2, annotate=False,\n", " cut_coords=range(-20, 30, 12), title='Rotated machine learning dataset');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Perfect! And which slab should we take? Let's try the slices 12 and 13." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from nilearn.plotting import plot_stat_map\n", "img_slab = img_crop.slicer[..., 12:14, :]\n", "plot_stat_map(mean_img(img_slab), cmap='magma', bg_img=mean_img(img_crop), colorbar=False,\n", " display_mode='x', vmax=2, annotate=False, cut_coords=range(-20, 30, 12),\n", " title='Slab of rotated machine learning dataset');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Perfect, the slab seems to contain exactly what we want. Now that the data is ready we can continue with the actual machine learning part." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Split data into a training and testing set\n", "\n", "First things first, we need to define a training and testing set. This is *really* important because we need to make sure that our model can generalize to new, unseen data. Here, we randomly shuffle our data, and reserve 80% of it for our training data, and the remaining 20% for testing.\n", "\n", "So let's first get the data in the right structure for keras. For this, we need to swap some of the dimensions of our data matrix." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data = np.rollaxis(img_slab.get_data(), 3, 0)\n", "data.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see, the goal is to have in the first dimension, the different volumes, and then the volume itself. Keep in mind, that the last dimension (here of size 2), are considered as `channels` in the keras model that we will be using below." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Note:** To make this notebook reproducible, i.e. always leading to the \"same\" results. Let's set a seed point for the random split of the dataset. This should only be done for teaching purposes, but not for real research as randomness and chance are a crucial part of machine learning." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from numpy.random import seed\n", "seed(0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As a next step, let's create a index list that we can use to split the data and labels into training and test sets:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Create list of indices and shuffle them\n", "N = data.shape[0]\n", "indices = np.arange(N)\n", "np.random.shuffle(indices)\n", "\n", "# Cut the dataset at 80% to create the training and test set\n", "N_80p = int(0.8 * N)\n", "indices_train = indices[:N_80p]\n", "indices_test = indices[N_80p:]\n", "\n", "# Split the data into training and test sets\n", "X_train = data[indices_train, ...]\n", "X_test = data[indices_test, ...]\n", "\n", "print(X_train.shape, X_test.shape)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Create outcome variable\n", "\n", "We need to define a variable that holds the outcome variable (1 or 0) that indicates whether or not the resting-state images were recorded with eyes opened or closed. Luckily we have this information already stored in the `labels` variable above. So let's split these labels in training and test set:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "y_train = labels[indices_train] == 'open'\n", "y_test = labels[indices_test] == 'open'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We need to reformat the shape of our outcome variables, `y_train` and `y_test`, because Keras needs the labels as a 2D array. Keras provides a function to do this:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from keras.utils import to_categorical\n", "y_train = to_categorical(y_train)\n", "y_test = to_categorical(y_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And now we're good to go." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Creating a Sequential Model\n", "\n", "Now come the fun and tricky part. We need to specify the structure of our convoluted neural network. As a quick reminder, a convoluted neural network consists of some convolution layers, pooling layers, some flattening layers and some full connect layers:\n", "\n", "\n", "\n", "Taken from: https://www.mathworks.com/videos/introduction-to-deep-learning-what-are-convolutional-neural-networks--1489512765771.html" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So as a first step, let's import all modules that we need to create the keras model:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import tensorflow as tf\n", "from keras.models import Sequential\n", "\n", "from keras.layers import Dense, Flatten, Dropout\n", "from keras.layers import Conv2D, MaxPooling2D, BatchNormalization\n", "\n", "from keras.optimizers import Adam, SGD\n", "\n", "from keras import backend as K" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As a next step, we should specify some of the model parameters that we want to be identical throughout the model:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Get shape of input data\n", "data_shape = tuple(X_train.shape[1:])\n", "\n", "# Specify shape of convolution kernel\n", "kernel_size = (3, 3)\n", "\n", "# Specify number of output categories\n", "n_classes = 2\n", "\n", "# Specify number of filters per layer\n", "filters = 16" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now comes the big part... the model, i.e. the structure of the neural network! We want to make clear that we're no experts in deep neural networks and therefore, the model below might not necessarily be a good model. But we chose it as it can be rather quickly estimated and has rather few parameters to estimate." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "K.clear_session()\n", "model = Sequential()\n", "\n", "model.add(Conv2D(filters, kernel_size, activation='relu', input_shape=data_shape))\n", "model.add(BatchNormalization())\n", "model.add(MaxPooling2D())\n", "\n", "model.add(Conv2D(filters * 2, kernel_size, activation='relu'))\n", "model.add(BatchNormalization())\n", "model.add(MaxPooling2D())\n", "\n", "model.add(Conv2D(filters * 4, kernel_size, activation='relu'))\n", "model.add(BatchNormalization())\n", "model.add(MaxPooling2D())\n", "\n", "model.add(Flatten())\n", "\n", "model.add(Dense(256, activation='relu'))\n", "model.add(Dropout(0.5))\n", "\n", "model.add(Dense(512, activation='relu'))\n", "model.add(Dropout(0.5))\n", "\n", "model.add(Dense(n_classes, activation='softmax'))\n", "\n", "# optimizer\n", "learning_rate = 1e-5\n", "adam = Adam(lr=learning_rate)\n", "sgd = SGD(lr=learning_rate)\n", "\n", "model.compile(loss='categorical_crossentropy',\n", " optimizer=adam, # swap out for sgd \n", " metrics=['accuracy'])\n", "\n", "model.summary()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "That's what our model looks like! Cool!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Fitting the Model\n", "\n", "The next step is now, of course, to fit our model to the training data. In our case we have two parameters that we can work with:\n", "\n", "*First*: How many iterations of the model fitting should be computed" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "nEpochs = 100 # Increase this value for better results (i.e., more training)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Second*: How many elements (volumes) should be considered at once for the updating of the weights?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "batch_size = 16 # Increasing this value might speed up fitting" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So let's test the model:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "%time fit = model.fit(X_train, y_train, epochs=nEpochs, batch_size=batch_size)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Performance during model fitting\n", "\n", "Let's take a look at the loss and accuracy values during the different epochs:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig = plt.figure(figsize=(10, 4))\n", "epoch = np.arange(nEpochs) + 1\n", "fontsize = 16\n", "plt.plot(epoch, fit.history['acc'], marker=\"o\", linewidth=2,\n", " color=\"steelblue\", label=\"accuracy\")\n", "plt.plot(epoch, fit.history['loss'], marker=\"o\", linewidth=2,\n", " color=\"orange\", label=\"loss\")\n", "plt.xlabel('epoch', fontsize=fontsize)\n", "plt.xticks(fontsize=fontsize)\n", "plt.yticks(fontsize=fontsize)\n", "plt.legend(frameon=False, fontsize=16);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Great, it seems that accuracy is constantly increasing and the loss is continuing to drop. But how well is our model doing on the test data?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Evaluating the model" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "evaluation = model.evaluate(X_test, y_test)\n", "print('Loss in Test set: %.02f' % (evaluation[0]))\n", "print('Accuracy in Test set: %.02f' % (evaluation[1] * 100))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Run more model iterations" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Not bad for just a few iterations. But let's see what we reach if we iterate a few hundred times more?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "nEpochs = 200" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "%time fit = model.fit(X_train, y_train, epochs=nEpochs, batch_size=batch_size)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig = plt.figure(figsize=(10, 4))\n", "epoch = np.arange(nEpochs) + 1\n", "fontsize = 16\n", "plt.plot(epoch, fit.history['acc'], marker=\"o\", linewidth=2,\n", " color=\"steelblue\", label=\"accuracy\")\n", "plt.plot(epoch, fit.history['loss'], marker=\"o\", linewidth=2,\n", " color=\"orange\", label=\"loss\")\n", "plt.xlabel('epoch', fontsize=fontsize)\n", "plt.xticks(fontsize=fontsize)\n", "plt.yticks(fontsize=fontsize)\n", "plt.legend(frameon=False, fontsize=16);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Wow, much better! At least on the training data. What about the test data?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "evaluation = model.evaluate(X_test, y_test)\n", "print('Loss in Test set: %.02f' % (evaluation[0]))\n", "print('Accuracy in Test set: %.02f' % (evaluation[1] * 100))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "That's amazing. But keep in mind that overfitting is a constant problem. We try to prevent this with the Dropout layers and by using `relu` activation function. Setting up the right model for convoluted neural networks can be a very tricky!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Analyze prediction values" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "What are the predicted values of the test set?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "y_pred = model.predict(X_test)\n", "y_pred[:10,:]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see, those values can be between 0 and 1." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig = plt.figure(figsize=(6, 4))\n", "fontsize = 16\n", "plt.hist(y_pred[:,0], bins=16, label='eyes closed')\n", "plt.hist(y_pred[:,1], bins=16, label='eyes open');\n", "plt.xticks(fontsize=fontsize)\n", "plt.yticks(fontsize=fontsize)\n", "plt.legend(frameon=False, fontsize=16);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The more both distributions are distributed around chance level, the weaker your model is." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Note:** Keep in mind that we trained the whole model only on one split of test and training data. Ideally, you would repeat this process many times so that your results become less dependent on what kind of split you did." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Visualizing Hidden Layers\n", "\n", "Finally, as a cool additional feature: We can now visualize the individual filters of the hidden layers. So let's get to it:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Aggregate the layers\n", "layer_dict = dict([(layer.name, layer) for layer in model.layers])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Specify a function that visualized the layers\n", "def show_activation(layer_name):\n", " \n", " layer_output = layer_dict[layer_name].output\n", "\n", " fn = K.function([model.input], [layer_output])\n", " \n", " inp = X_train[0:1]\n", " \n", " this_hidden = fn([inp])[0]\n", " \n", " # plot the activations, 8 filters per row\n", " plt.figure(figsize=(16,8))\n", " nFilters = this_hidden.shape[-1]\n", " nColumn = 8 if nFilters >= 8 else nFilters\n", " for i in range(nFilters):\n", " plt.subplot(nFilters / nColumn, nColumn, i+1)\n", " plt.imshow(this_hidden[0,:,:,i], cmap='magma', interpolation='nearest')\n", " plt.axis('off')\n", " \n", " return " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can plot the filters of the hidden layers:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "show_activation('conv2d_1')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "show_activation('conv2d_2')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "show_activation('conv2d_3')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Conclusion of 2D example\n", "\n", "The classification of the training set gets incredibly high, while the validation set also reaches a reasonable accuracy level above 80. Nonetheless, by only investigating a slab of our fMRI dataset, we might have missed out on some important additional parameters.\n", "\n", "An alternative solution might be to use 3D convoluted neural networks. But keep in mind that they will have even more parameters and probably take much longer to fit the model to the training data. Having said so, let's get to it." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Keras - 3D Example\n", "\n", "We could take the full brain for the 3D convoluted neural network. But the more voxels we include the longer the model takes to fit to the training data. Therefor it makes sense again to reduce the whole brain volume to a slab around the regions that we're interested in. But for this case, let's not just keep 3 slices, but a bit more:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from nilearn.plotting import plot_stat_map\n", "img_slab = img_crop.slicer[5:-5, 3:-3, 6:24, :]\n", "plot_stat_map(mean_img(img_slab), cmap='magma', bg_img=mean_img(img_crop), colorbar=False,\n", " display_mode='x', vmax=2, annotate=False, cut_coords=range(-20, 30, 12),\n", " title='Slab of rotated machine learning dataset');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This should do the trick. Now we just have to extract the data from this slab:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data = np.rollaxis(img_slab.get_data(), 3, 0)[...,None]\n", "data.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Note:** We added another dimension to our array with `[...,None]`. This is needed for Keras and Tensorflow as this represents the channel dimension." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Split data into a training and testing set\n", "\n", "The splitting of the data into training and test set is exactly the same as before." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Create list of indices and shuffle them\n", "N = data.shape[0]\n", "indices = np.arange(N)\n", "np.random.shuffle(indices)\n", "\n", "# Cut the dataset at 80% to create the training and test set\n", "N_80p = int(0.8 * N)\n", "indices_train = indices[:N_80p]\n", "indices_test = indices[N_80p:]\n", "\n", "# Split the data into training and test sets\n", "X_train = data[indices_train, ...]\n", "X_test = data[indices_test, ...]\n", "\n", "print(X_train.shape, X_test.shape)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Create outcome variable\n", "\n", "Also here, everything is the same as in the 2D example." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "y_train = labels[indices_train] == 'open'\n", "y_test = labels[indices_test] == 'open'\n", "\n", "from keras.utils import to_categorical\n", "y_train = to_categorical(y_train)\n", "y_test = to_categorical(y_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Creating a Sequential Model\n", "\n", "So, first, let's again import the necessary modules. Notice that we swithced `Conv2D` and `MaxPooling2D` with their 3-dimensional counter part." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import tensorflow as tf\n", "from keras.models import Sequential\n", "\n", "from keras.layers import Dense, Flatten, Dropout\n", "from keras.layers import Conv3D, MaxPooling3D, BatchNormalization\n", "\n", "from keras.optimizers import Adam, SGD\n", "\n", "from keras import backend as K" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, we need again to identify the important model parameters. In this case we need to extend the kernel size to 3 dimension:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Get shape of input data\n", "data_shape = tuple(X_train.shape[1:])\n", "\n", "# Specify shape of convolution kernel\n", "kernel_size = (2, 2, 2)\n", "\n", "# Specify number of output categories\n", "n_classes = 2\n", "\n", "# Specify number of filters per layer\n", "filters = 32 # For better results, increase this value to 8" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And we're good to go and can setup our model:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "K.clear_session()\n", "model = Sequential()\n", "\n", "model.add(Conv3D(filters, kernel_size, activation='relu', input_shape=data_shape))\n", "model.add(BatchNormalization())\n", "model.add(MaxPooling3D(pool_size=(2, 3, 2)))\n", "\n", "model.add(Conv3D(filters * 2, kernel_size, activation='relu'))\n", "model.add(BatchNormalization())\n", "model.add(MaxPooling3D(pool_size=(2, 3, 2)))\n", "\n", "model.add(Conv3D(filters * 4, kernel_size, activation='relu'))\n", "model.add(BatchNormalization())\n", "model.add(MaxPooling3D())\n", "\n", "model.add(Flatten())\n", "\n", "model.add(Dense(256, activation='relu'))\n", "model.add(Dropout(0.5))\n", "\n", "model.add(Dense(512, activation='relu'))\n", "model.add(Dropout(0.5))\n", "\n", "model.add(Dense(n_classes, activation='softmax'))\n", "\n", "# optimizer\n", "learning_rate = 1e-5\n", "adam = Adam(lr=learning_rate)\n", "sgd = SGD(lr=learning_rate)\n", "\n", "model.compile(loss='categorical_crossentropy',\n", " optimizer=adam, # swap out for sgd \n", " metrics=['accuracy'])\n", "\n", "model.summary()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Fitting the Model\n", "\n", "As before we need to specify how many epochs we want to run and what the batch size is. As you will see, the fitting of the 3D convoluted model takes much much longer than the one for the 2D model. Therefore we recommend to drastically decrease the number of epochs (at least for this workshop example)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "nEpochs = 100 # Increase this value for better results (i.e., more training)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "batch_size = 16 # Increasing this value might speed up fitting" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And we're good to go!\n", "\n", "***Please set `run_3D_convnet` to `True` if you want to fit the model and run the rest of the analysis.***" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "run_3D_convnet = False" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%time \n", "\n", "if run_3D_convnet:\n", " fit = model.fit(X_train, y_train, epochs=nEpochs, batch_size=batch_size)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Performance during model fitting\n", "\n", "Let's take a look at the loss and accuracy values during the different epochs:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "if run_3D_convnet:\n", " fig = plt.figure(figsize=(10, 4))\n", " epoch = np.arange(nEpochs) + 1\n", " fontsize = 16\n", " plt.plot(epoch, fit.history['acc'], marker=\"o\", linewidth=2,\n", " color=\"steelblue\", label=\"accuracy\")\n", " plt.plot(epoch, fit.history['loss'], marker=\"o\", linewidth=2,\n", " color=\"orange\", label=\"loss\")\n", " plt.xlabel('epoch', fontsize=fontsize)\n", " plt.xticks(fontsize=fontsize)\n", " plt.yticks(fontsize=fontsize)\n", " plt.legend(frameon=False, fontsize=16);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Evaluating the model\n", "\n", "How about the performance on the test data?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "if run_3D_convnet:\n", " evaluation = model.evaluate(X_test, y_test)\n", " print('Loss in Test set: %.02f' % (evaluation[0]))\n", " print('Accuracy in Test set: %.02f' % (evaluation[1] * 100))" ] } ], "metadata": { "anaconda-cloud": {}, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.7" } }, "nbformat": 4, "nbformat_minor": 1 }