{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"author: Kirill Panin"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Diabetic Retinopathy Detection"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"TOC: \n",
"[1. Feature and data explanation (2 points)](#1) \n",
"[2-3. Primary data analysis (4 points)](#2)[3. Primary visual data analysis (4 points)](#3) \n",
"[4-6. Insights and found dependencies (4 points)](#4) [5. Metrics selection (3 points)](#5)[6. Model selection (3 points)](#6): \n",
"[7-11. Data preprocessing (4 points)](#7)[8. Cross-validation and adjustment of model hyperparameters (4 points)](#8)[9. Creation of new features and description of this process (4 points)](#9)[10. Plotting training and validation curves (4 points)](#10)[11. Prediction for test or hold-out samples (2 points)](#11): \n",
"\n",
" * [Augmentation](#aug)\n",
" * [VGG-16 Model](#vgg)\n",
" * [Inception Model](#inception) \n",
" \n",
"[12. Conclusions (2 points)](#12) \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Feature and data explanation (2 points) \n",
"(+) The process of collecting data is described, a detailed explanation of the task is provided, its value is explained, target features are given; "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Foreword\n",
"Hi friends. I've been starting this project without practical knowledge in CV. I want help my friends in that sphere. They doing ml models associated with detection diabetic retinopathy disease on early stage. \n",
"I united some evaluation topic plans to one categories. Because approaches to image processing distinguished than processing with tabular data. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"__*Diabetic retinopathy*__ is the leading cause of blindness in the working-age population of the developed world. It is estimated to affect over 93 million people. \n",
"__Major aim__ of this task develop robust algorithms that can detection diabetic retinopathy disease on early stage and cope with presence of noise and variation. \n",
"\n",
"**Dataset** \n",
"Provided a extremely large dataset(82 GB) [link data](https://www.kaggle.com/c/diabetic-retinopathy-detection/data). Train set with labels(but some labels don't true)(35 GB). High-resolution retina images taken under a variety of imaging conditions. The images in the dataset come from different models and types of cameras, which can affect the visual appearance of left vs. right. A left and right eyes is provided for every patient. Like any real-world data set have encounter noise in both the images and labels. Images may contain artifacts, be out of focus, underexposed, or overexposed.\n",
"\n",
"Clinicians, doctors has __marked up the data__ using a web special tool.\n",
"\n",
"A clinician has rated the presence of diabetic retinopathy in each image on a scale of 0 to 4, according to the following scale(__multiclass classification, target__):\n",
"\n",
"0 - No DR \n",
"1 - Mild \n",
"2 - Moderate \n",
"3 - Severe \n",
"4 - Proliferative DR \n",
"\n",
"The methods __how to solve this task__:\n",
"\n",
"The goal is to make a retinopathy model by using a pretrained models inception v3(base and retraining some modified final layers with attention), VGG-16 and just Keras from scratch. \n",
"\n",
"This can be massively improved with\n",
"\n",
"* high-resolution images\n",
"* better data sampling\n",
"* ensuring there is no leaking between training and validation sets, sample(replace = True) is real dangerous\n",
"* better target variable (age) normalization\n",
"* pretrained models\n",
"* attention/related techniques to focus on areas, segmentation "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"import seaborn as sns\n",
"import os\n",
"import tensorflow as tf\n",
"from PIL import Image\n",
"from tensorflow.python.client import device_lib\n",
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#cd /media/gismart/sda/kirill_data/deepdee/\n",
"#cd ../../../workdir/Work/deepDee/\n",
"#ls"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Example of image.\n",
"im1 = Image.open('train/10210_right.jpeg')\n",
"im1"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Primary data analysis (4 points) & "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Primary visual data analysis (4 points) "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"Image.open('train/1233_left.jpeg')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Plotting Histograms"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import cv2\n",
"import numpy as np\n",
"from matplotlib import pyplot as plt\n",
"\n",
"img = cv2.imread('train/1052_left.jpeg',0)\n",
"plt.hist(img.ravel(),256,[0,256]); plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Train test has 35126 marked images"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Example trainLabels\n",
"trainLabels = pd.read_csv('trainLabels.csv')\n",
"trainLabels.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Number of Images\n",
"print(\"Train data with levels size:\",len(trainLabels))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Distribution of target feature \n",
"trainLabels.level.value_counts()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### The dataset is highly imbalanced with maximum 0 target, i.e., no Diabetic Retinopathy"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Distribution of target feature \n",
"sns.countplot(\"level\",data= trainLabels);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Balancing trainset"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"min_val = trainLabels.level.value_counts().min()\n",
"labels = trainLabels.level.unique()\n",
"new_trainLabels = pd.DataFrame()\n",
"for i in labels:\n",
" new_trainLabels = pd.concat([new_trainLabels, trainLabels[trainLabels.level==i][:min_val]], axis=0)\n",
"\n",
"# New distribution of target feature \n",
"new_trainLabels.level.value_counts()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Ok, looks great, we reduce dataset. And now dataset weighted 3.5 GB, but not 35 GB!!!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Replace all marked set on our balanced dataset\n",
"trainLabels = new_trainLabels"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Insights and found dependencies (4 points) \n",
"## 5. Metrics selection (3 points) \n",
"## 6. Model selection (3 points)) "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* (4) The assumptions about various correlations/omissions/regularities from the previous paragraphs are found and put forward. There is an explanation why they are important for the task being solved;\n",
"* (5) There is a reasonable justification for the selection of the model quality metrics. The issues that affect the choice of quality metrics (solving problem, decision goal, number of classes, class \n",
"* (6) A model selection is made. The process of selection and connection with the task being solved is described;\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Insights and found dependencies \n",
"Finding ideas and dependencies is very difficult. Retenopathy appears due to blockage of the capillaries by the blood supply to the retina. Over time, the lack of oxygen develops. Does not receive nutrients. In this regard, the tissue creates new vessels, which further burst (hemorrhages). The presence of such vessels will be detected by the algorithm."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Metric Selection\n",
"The dataset is highly imbalanced with no Diabetic Retinopathy images. Usually used Confusion Matrix, Roc AUC, F1 score for imbalances classes. If classes can be balanced we can use accuracy. The __quadratic weighted kappa__ should be best score for that task. But i don't know that metric so good and I will be using Confusion Matrix, AUC, F1 score."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Model selection \n",
"I will be using 2 pre-trained models __VGG-16__ and __Inception__ and __just keras__ from scratch. __VGG-16__ model pretrained on imagenet weights. __Inception/Attention Model__ The basic idea is that a Global Average Pooling is too simplistic since some of the regions are more relevant than others. So we build an attention mechanism to turn pixels in the GAP on an off before the pooling and then rescale (Lambda layer) the results based on the number of pixels. The model could be seen as a sort of 'global weighted average' pooling."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 7. Data preprocessing (4 points) "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* (7) Data for a particular model is preprocessed. If necessary, there is a description of scaling characteristics, filling in blanks, replacing strings with numbers, One-Hot Encoding, handling of outliers, selection of features with an explanation of the methods used for this. In general, the data is divided into training and hold-out sets;"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 8. Cross-validation and adjustment of model hyperparameters (4 points) "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 9. Creation of new features and description of this process (4 points) "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* (7) Data for a particular model is preprocessed. If necessary, there is a description of scaling characteristics, filling in blanks, replacing strings with numbers, One-Hot Encoding, handling of outliers, selection of features with an explanation of the methods used for this. In general, the data is divided into training and hold-out sets;"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* (8+) Cross-validation is performed technically correct, without data leaks. The number of folds is reasonably selected and the split (Random/Stratified or otherwise) is fixed by a seed. An explanation is given. Hyperparameters of the model and the method of their selection are explained. The choice is based on some study of the model’s hyperparameters for the task being solved;\n",
"* (8+/-) There are small errors (for example, there is no fixed seed) or there is no explanation for cross-validation. But the hyperparameters of the model and the way they are selected are explained;"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* (9) New features are created. Justification is given: logical (for example, birds have body temperature a few degrees higher than human, which means that virus XXX will not survive in environment like this), physical (for example, a rainbow means that the light source is located behind; calculation of the value according to physical law using these features) or another one (for example a feature is built after data visualization). Justification is reasonably described. The usefulness of new features is confirmed statistically or with the help of the corresponding model;"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 10. Plotting training and validation curves (4 points) "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* (+) Training and validation curves are built. The correct interpretation is provided;\n",
"* (+/-) Noticeable mistakes were made while building curves or the curves are not informative;\n",
"* (-/+) An incorrect interpretation is given or another serious mistakes can be found;\n",
"* (-) Training and validation curves are omitted."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 11. Prediction for test or hold-out samples (2 points) "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* (+) The results on the test sample or LB score are provided. The results on the test sample are comparable to the results on cross-validation. If the test sample was created by the author of the project, the creation mechanism should be unbiased and explained (a reasonable sampling mechanism was applied, in the simplest case - randomization);\n",
"* (+/-) The values of the metrics on the test sample do not differ enough from the values of the metrics on cross-validation and/or the test sample is biased, but there is a reasonable justification for this (example: the customer took a test sample from another distribution);\n",
"* (-/+) The values of the metrics on the test sample differ a lot from the values of the metrics on cross-validation and/or the test sample was biased;\n",
"* (-) No prediction for test or hold-out samples is provided."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### AUGMENATION \n",
"In CV for increasing and transformation dataset used some techniques are called Augmentation techniques:\n",
"* Scaling\n",
"* Rotation\n",
"* Translation\n",
"* Flipping\n",
"* Lighting condition\n",
"* Adding Salt and Pepper noise\n",
"* Perspective transform\n",
"* GANs\n",
"* Constant\n",
"* Edge \n",
"* Reflect and etc.."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"I've been using: rgb to greyscale, edge.\n",
"Gaussian blur, mask The Canny Edge Detector, Masks"
]
},
{
"cell_type": "raw",
"metadata": {
"collapsed": true
},
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"from subprocess import check_output\n",
"print(check_output([\"ls\", \"train/\"]).decode(\"utf8\"))\n",
"\n",
"import matplotlib.pyplot as plt\n",
"from skimage.io import imread\n",
"import os\n",
"from glob import glob\n",
"%matplotlib inline \n",
"\n",
"import tifffile as tiff\n",
"import cv2 as cv2\n",
"from skimage.segmentation import slic, mark_boundaries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"base_image_dir = os.path.join('train/')\n",
"base_image_dir"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"base_image_dir = os.path.join('train/')\n",
"retina_df = pd.read_csv('trainLabels.csv')\n",
"retina_df['PatientId'] = retina_df['image'].map(lambda x: x.split('_')[0])\n",
"retina_df['path'] = retina_df['image'].map(lambda x: os.path.join(base_image_dir,\n",
" '{}.jpeg'.format(x)))\n",
"retina_df['exists'] = retina_df['path'].map(os.path.exists)\n",
"print(retina_df['exists'].sum(), 'images found of', retina_df.shape[0], 'total')\n",
"retina_df['eye'] = retina_df['image'].map(lambda x: 1 if x.split('_')[-1]=='left' else 0)\n",
"from keras.utils.np_utils import to_categorical\n",
"retina_df['level_cat'] = retina_df['level'].map(lambda x: to_categorical(x, 1+retina_df['level'].max()))\n",
"\n",
"retina_df.dropna(inplace = True)\n",
"retina_df = retina_df[retina_df['exists']]\n",
"retina_df.sample(3)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"retina_df = retina_df[retina_df.level.isin([0,4])]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"retina_df = pd.concat([retina_df[retina_df.level==0].sample(1000),retina_df[retina_df.level==4]],axis=0)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.model_selection import train_test_split\n",
"rr_df = retina_df[['PatientId', 'level']].drop_duplicates()\n",
"train_ids, valid_ids = train_test_split(rr_df['PatientId'], \n",
" test_size = 0.2, \n",
" random_state = 2018,\n",
" stratify = rr_df['level'])\n",
"raw_train_df = retina_df[retina_df['PatientId'].isin(train_ids)]\n",
"valid_df = retina_df[retina_df['PatientId'].isin(valid_ids)]\n",
"print('train', raw_train_df.shape[0], 'validation', valid_df.shape[0])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#balance the train set:\n",
"train_df = raw_train_df.groupby(['level', 'eye']).apply(lambda x: x.sample(75, replace = True)\n",
" ).reset_index(drop = True)\n",
"print('New Data Size:', train_df.shape[0], 'Old Size:', raw_train_df.shape[0])\n",
"train_df[['level', 'eye']].hist(figsize = (10, 5))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"from keras import backend as K\n",
"from keras.applications.inception_v3 import preprocess_input\n",
"import numpy as np\n",
"IMG_SIZE = (512, 512) # slightly smaller than vgg16 normally expects\n",
"def tf_image_loader(out_size, \n",
" horizontal_flip = True, \n",
" vertical_flip = False, \n",
" random_brightness = True,\n",
" random_contrast = True,\n",
" random_saturation = True,\n",
" random_hue = True,\n",
" color_mode = 'rgb',\n",
" preproc_func = preprocess_input,\n",
" on_batch = False):\n",
" def _func(X):\n",
" with tf.name_scope('image_augmentation'):\n",
" with tf.name_scope('input'):\n",
" X = tf.image.decode_png(tf.read_file(X), channels = 3 if color_mode == 'rgb' else 0)\n",
" X = tf.image.resize_images(X, out_size)\n",
" with tf.name_scope('augmentation'):\n",
" if horizontal_flip:\n",
" X = tf.image.random_flip_left_right(X)\n",
" if vertical_flip:\n",
" X = tf.image.random_flip_up_down(X)\n",
" if random_brightness:\n",
" X = tf.image.random_brightness(X, max_delta = 0.1)\n",
" if random_saturation:\n",
" X = tf.image.random_saturation(X, lower = 0.75, upper = 1.5)\n",
" if random_hue:\n",
" X = tf.image.random_hue(X, max_delta = 0.15)\n",
" if random_contrast:\n",
" X = tf.image.random_contrast(X, lower = 0.75, upper = 1.5)\n",
" return preproc_func(X)\n",
" if on_batch: \n",
" # we are meant to use it on a batch\n",
" def _batch_func(X, y):\n",
" return tf.map_fn(_func, X), y\n",
" return _batch_func\n",
" else:\n",
" # we apply it to everything\n",
" def _all_func(X, y):\n",
" return _func(X), y \n",
" return _all_func\n",
" \n",
"def tf_augmentor(out_size,\n",
" intermediate_size = (640, 640),\n",
" intermediate_trans = 'crop',\n",
" batch_size = 16,\n",
" horizontal_flip = True, \n",
" vertical_flip = False, \n",
" random_brightness = True,\n",
" random_contrast = True,\n",
" random_saturation = True,\n",
" random_hue = True,\n",
" color_mode = 'rgb',\n",
" preproc_func = preprocess_input,\n",
" min_crop_percent = 0.001,\n",
" max_crop_percent = 0.005,\n",
" crop_probability = 0.5,\n",
" rotation_range = 10):\n",
" \n",
" load_ops = tf_image_loader(out_size = intermediate_size, \n",
" horizontal_flip=horizontal_flip, \n",
" vertical_flip=vertical_flip, \n",
" random_brightness = random_brightness,\n",
" random_contrast = random_contrast,\n",
" random_saturation = random_saturation,\n",
" random_hue = random_hue,\n",
" color_mode = color_mode,\n",
" preproc_func = preproc_func,\n",
" on_batch=False)\n",
" def batch_ops(X, y):\n",
" batch_size = tf.shape(X)[0]\n",
" with tf.name_scope('transformation'):\n",
" # code borrowed from https://becominghuman.ai/data-augmentation-on-gpu-in-tensorflow-13d14ecf2b19\n",
" # The list of affine transformations that our image will go under.\n",
" # Every element is Nx8 tensor, where N is a batch size.\n",
" transforms = []\n",
" identity = tf.constant([1, 0, 0, 0, 1, 0, 0, 0], dtype=tf.float32)\n",
" if rotation_range > 0:\n",
" angle_rad = rotation_range / 180 * np.pi\n",
" angles = tf.random_uniform([batch_size], -angle_rad, angle_rad)\n",
" transforms += [tf.contrib.image.angles_to_projective_transforms(angles, intermediate_size[0], intermediate_size[1])]\n",
"\n",
" if crop_probability > 0:\n",
" crop_pct = tf.random_uniform([batch_size], min_crop_percent, max_crop_percent)\n",
" left = tf.random_uniform([batch_size], 0, intermediate_size[0] * (1.0 - crop_pct))\n",
" top = tf.random_uniform([batch_size], 0, intermediate_size[1] * (1.0 - crop_pct))\n",
" crop_transform = tf.stack([\n",
" crop_pct,\n",
" tf.zeros([batch_size]), top,\n",
" tf.zeros([batch_size]), crop_pct, left,\n",
" tf.zeros([batch_size]),\n",
" tf.zeros([batch_size])\n",
" ], 1)\n",
" coin = tf.less(tf.random_uniform([batch_size], 0, 1.0), crop_probability)\n",
" transforms += [tf.where(coin, crop_transform, tf.tile(tf.expand_dims(identity, 0), [batch_size, 1]))]\n",
" if len(transforms)>0:\n",
" X = tf.contrib.image.transform(X,\n",
" tf.contrib.image.compose_transforms(*transforms),\n",
" interpolation='BILINEAR') # or 'NEAREST'\n",
" if intermediate_trans=='scale':\n",
" X = tf.image.resize_images(X, out_size)\n",
" elif intermediate_trans=='crop':\n",
" X = tf.image.resize_image_with_crop_or_pad(X, out_size[0], out_size[1])\n",
" else:\n",
" raise ValueError('Invalid Operation {}'.format(intermediate_trans))\n",
" return X, y\n",
" def _create_pipeline(in_ds):\n",
" batch_ds = in_ds.map(load_ops, num_parallel_calls=4).batch(batch_size)\n",
" return batch_ds.map(batch_ops)\n",
" return _create_pipeline\n",
"\n",
"def flow_from_dataframe(idg, \n",
" in_df, \n",
" path_col,\n",
" y_col, \n",
" shuffle = True, \n",
" color_mode = 'rgb'):\n",
" files_ds = tf.data.Dataset.from_tensor_slices((in_df[path_col].values, \n",
" np.stack(in_df[y_col].values,0)))\n",
" in_len = in_df[path_col].values.shape[0]\n",
" while True:\n",
" if shuffle:\n",
" files_ds = files_ds.shuffle(in_len) # shuffle the whole dataset\n",
" \n",
" next_batch = idg(files_ds).repeat().make_one_shot_iterator().get_next()\n",
" for i in range(max(in_len//32,1)):\n",
" # NOTE: if we loop here it is 'thread-safe-ish' if we loop on the outside it is completely unsafe\n",
" yield K.get_session().run(next_batch)\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"batch_size = 48\n",
"core_idg = tf_augmentor(out_size = IMG_SIZE, \n",
" color_mode = 'rgb', \n",
" vertical_flip = True,\n",
" crop_probability=0.0, # crop doesn't work yet\n",
" batch_size = batch_size) \n",
"valid_idg = tf_augmentor(out_size = IMG_SIZE, color_mode = 'rgb', \n",
" crop_probability=0.0, \n",
" horizontal_flip = False, \n",
" vertical_flip = False, \n",
" random_brightness = False,\n",
" random_contrast = False,\n",
" random_saturation = False,\n",
" random_hue = False,\n",
" rotation_range = 0,\n",
" batch_size = batch_size)\n",
"\n",
"train_gen = flow_from_dataframe(core_idg, train_df, \n",
" path_col = 'path',\n",
" y_col = 'level_cat')\n",
"\n",
"valid_gen = flow_from_dataframe(valid_idg, valid_df, \n",
" path_col = 'path',\n",
" y_col = 'level_cat') # we can use much larger batches for evaluation"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#display images from validation set\n",
"t_x, t_y = next(valid_gen)\n",
"fig, m_axs = plt.subplots(2, 4, figsize = (16, 8))\n",
"for (c_x, c_y, c_ax) in zip(t_x, t_y, m_axs.flatten()):\n",
" c_ax.imshow(np.clip(c_x*127+127, 0, 255).astype(np.uint8))\n",
" c_ax.set_title('Severity {}'.format(np.argmax(c_y, -1)))\n",
" c_ax.axis('off')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#we focus on a healthy eye and display it alone\n",
"print('t_x shape: ', t_x.shape)\n",
"fig, m_axs = plt.subplots(2, 1, figsize = (16, 8))\n",
"for (c_x, c_y, c_ax) in zip(t_x, t_y, m_axs.flatten()):\n",
" c_ax.imshow(np.clip(c_x*127+127, 0, 255).astype(np.uint8))\n",
" c_ax.set_title('Severity {}'.format(np.argmax(c_y, -1)))\n",
" c_ax.axis('off')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"edges = cv2.Canny(np.clip(c_x*127+127, 0, 255).astype(np.uint8),70,130)\n",
"plt.subplot(121),plt.imshow(np.clip(c_x*127+127, 0, 255).astype(np.uint8),cmap = 'gray')\n",
"plt.title('Original Image'), plt.xticks([]), plt.yticks([])\n",
"plt.subplot(122),plt.imshow(edges,cmap = 'gray')\n",
"plt.title('Edge Image'), plt.xticks([]), plt.yticks([])\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#https://www.pyimagesearch.com/2015/04/06/zero-parameter-automatic-canny-edge-detection-with-python-and-opencv/\n",
"img = cv2.imread(os.path.join(base_image_dir, '15_right.jpeg'), cv2.IMREAD_COLOR)\n",
"print('img type ', type(img), img.shape)\n",
"print('img min max ', img.min(), img.max())\n",
"gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n",
"blurred = cv2.GaussianBlur(gray, (3, 3), 0)\n",
"v = np.median(img)\n",
"sigma = 0.33 #0.93 \n",
"# apply automatic Canny edge detection using the computed median\n",
"lower = int(max(0, (1.0 - sigma) * v))\n",
"upper = int(min(255, (1.0 + sigma) * v))\n",
"#imgProd = img*127+127\n",
"imgProd = gray*127 #+127\n",
"imgClip = np.clip(imgProd, 0, 255).astype(np.uint8)\n",
"print('imgProd min max ', imgProd.min(), imgProd.max())\n",
"\n",
"# apply Canny edge detection using a wide threshold, tight\n",
"# threshold, and automatically determined threshold\n",
"\n",
"wide = cv2.Canny(blurred, 10, 200)\n",
"tight = cv2.Canny(imgProd, 200, 250)\n",
"auto = cv2.Canny(imgProd, lower, upper)\n",
" \n",
"#edges = cv2.Canny(np.clip(imgClip, 0, 255).astype(np.uint8), lower, upper)\n",
"#edges = cv.Canny(img,lower,upper)\n",
"fig, m_axs = plt.subplots(1, 1, figsize = (16, 8))\n",
"#plt.imshow(img) #,cmap = 'gray')\n",
"#plt.title('Original Image'), plt.xticks([]), plt.yticks([])\n",
"\n",
"plt.imshow(tight,cmap = 'gray')\n",
"#plt.title('Edge Image'), plt.xticks([]), plt.yticks([])\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#another path to segmentation\n",
"def stretch_8bit(bands, lower_percent=2, higher_percent=98):\n",
" out = np.zeros_like(bands)\n",
" for i in range(3):\n",
" a = 0 #np.min(band)\n",
" b = 255 #np.max(band)\n",
" c = np.percentile(bands[:,:,i], lower_percent)\n",
" d = np.percentile(bands[:,:,i], higher_percent) \n",
" t = a + (bands[:,:,i] - c) * (b - a) / (d - c) \n",
" t[tb] = b\n",
" out[:,:,i] =t\n",
" return out.astype(np.uint8) \n",
" \n",
"def RGB(image_id):\n",
" filename = os.path.join(base_image_dir, '{}.jpeg'.format(image_id))\n",
" #img = tiff.imread(filename)\n",
" img = cv2.imread(filename, cv2.IMREAD_COLOR)\n",
" img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n",
" #img = np.rollaxis(img, 0, 3) \n",
" return img\n",
"\n",
"def GRAY(image_id):\n",
" filename = os.path.join(base_image_dir, '{}.jpeg'.format(image_id))\n",
" img = cv2.imread(filename, cv2.IMREAD_COLOR)\n",
" img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n",
" blurred = cv2.GaussianBlur(img, (3, 3), 0)\n",
" # invert the image\n",
" img = cv2.bitwise_not(blurred)\n",
" return img\n",
" \n",
"def M(image_id):\n",
" filename = os.path.join('..', 'input', 'sixteen_band', '{}_M.tif'.format(image_id))\n",
" img = tiff.imread(filename) \n",
" img = np.rollaxis(img, 0, 3)\n",
" return img"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"image_id = '15_right'\n",
"rgb = GRAY(image_id)\n",
"print('shape rgb: ', rgb.shape, rgb.min(), rgb.max())\n",
"#rgb1 = stretch_8bit(rgb)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"y1,y2,x1,x2 = 1900, 2364, 3000, 4000\n",
"region = rgb[y1:y2, x1:x2]\n",
"plt.figure()\n",
"plt.imshow(region)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#blue_mask = cv2.inRange(region, np.array([175,70,20]), np.array([189,140,150])) #img minthresh maxthre (BGR) \n",
"blue_mask = cv2.inRange(region, 19, 113) #img minthresh maxthre (BGR) \n",
"print('blue_mask shape: ', blue_mask.shape, ' region shape: ', region.shape, 'min max bmask', blue_mask.min(), blue_mask.max())\n",
"mask = cv2.bitwise_and(region, region, mask=blue_mask)\n",
"#mask = cv2.cvtColor(mask, cv2.COLOR_RGB2GRAY) \n",
"#print('mask shape ', mask.shape, mask.max(), region[:,:,2].max())\n",
"plt.figure()\n",
"plt.imshow(blue_mask, cmap='gray');"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"y1,y2,x1,x2 = 100, 3364, 500, 4000\n",
"region = rgb[y1:y2, x1:x2]\n",
"plt.figure()\n",
"plt.imshow(region);"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#https://docs.opencv.org/3.4/d3/db4/tutorial_py_watershed.html\n",
"ret, thresh = cv2.threshold(region,125, 255,cv2.THRESH_TOZERO_INV) #_INV+cv2.THRESH_OTSU)\n",
"'''\n",
"ret,thresh1 = cv.threshold(img,127,255,cv.THRESH_BINARY)\n",
"ret,thresh2 = cv.threshold(img,127,255,cv.THRESH_BINARY_INV)\n",
"ret,thresh3 = cv.threshold(img,127,255,cv.THRESH_TRUNC)\n",
"ret,thresh4 = cv.threshold(img,127,255,cv.THRESH_TOZERO)\n",
"ret,thresh5 = cv.threshold(img,127,255,cv.THRESH_TOZERO_INV)\n",
"'''\n",
"print('threshold ret shapes ', thresh.min(), thresh.max())\n",
"plt.figure()\n",
"plt.imshow(thresh);"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"th = cv2.adaptiveThreshold(region, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 115, 1)\n",
"plt.figure()\n",
"plt.imshow(th);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## VGG-16"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import os\n",
"import time\n",
"from keras.applications.vgg16 import VGG16\n",
"from keras.preprocessing import image\n",
"from keras.applications.vgg16 import preprocess_input\n",
"from keras.layers import Dense, Activation, Flatten\n",
"from keras.layers import merge, Input\n",
"from keras.models import Model\n",
"from keras.utils import np_utils\n",
"from sklearn.utils import shuffle\n",
"from sklearn.model_selection import train_test_split"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Load the base model, which is a VGG-16 model pretrained on imagenet weights. We then move on to freeze all the layers except the last three."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#loading base model\n",
"base_model = VGG16(weights='imagenet', include_top=True, input_shape=(224, 224, 3))\n",
"#freeze_layers(base_model)\n",
"base_model.summary()\n",
"#model = Model(input=base_model.input, output=base_model.get_layer('fc1').output)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Freeze the layers except the last 4 layers\n",
"for layer in base_model.layers[:-3]:\n",
" layer.trainable = False\n",
"# Check the trainable status of the individual layers\n",
"for layer in base_model.layers:\n",
" print(layer, layer.trainable)\n",
"base_model.summary()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from keras.utils.vis_utils import plot_model\n",
"from IPython.display import SVG\n",
"from keras.utils.vis_utils import model_to_dot"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Before we go about pre processing data and training our loaded model, we fix the following:\n",
"\n",
"* __Batch Size__ of the data required for training\n",
"* __nb_classes__ -> indicates the number of output classes\n",
"* __nb_epoch__ -> induicates the number of iterations during training"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#batch_size to train\n",
"batch_size = 32\n",
"# number of output classes\n",
"nb_classes = 5\n",
"# number of epochs to train\n",
"nb_epoch = 10"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Fine Tune : VGG-16\n",
"\n",
"We move on to add customised layers on top of our pre-loaded model for purpose of fine-tuning. The following layers were added :\n",
"\n",
"Dense Relu\n",
"Dropout\n",
"Dense Softmax"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from keras import models\n",
"from keras import layers\n",
"from keras import optimizers\n",
" \n",
"# Create the model\n",
"model = models.Sequential()\n",
" \n",
"# Add the vgg convolutional base model\n",
"model.add(base_model)\n",
" \n",
"# Add new layers\n",
"model.add(layers.Dense(1024, activation='relu'))\n",
"model.add(layers.Dropout(0.5))\n",
"model.add(layers.Dense(nb_classes, activation='softmax', name ='output'))\n",
" \n",
"# Show a summary of the model. Check the number of trainable parameters\n",
"model.summary()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Preparing Data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Trying to predict 0 and 4 label"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"new_trainL = trainLabels[trainLabels.level.isin([0,4])]\n",
"listing = os.listdir(\"train/\") \n",
"print(\"Number of images in train:\",np.size(listing))\n",
"listing = [i for i in listing if os.path.splitext(i)[0] in new_trainL.image.values]\n",
"print(\"len new listing with our data should be 1416;answer or 3540:\",len(listing))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Image to Numpy Array\n",
"from PIL import Image\n",
"from sklearn.utils import shuffle\n",
"\n",
"# input image dimensions\n",
"img_rows, img_cols = 224, 224\n",
"\n",
"immatrix = []\n",
"imlabel = []\n",
"\n",
"for file in listing:\n",
" base = os.path.basename(\"train/\" + file)\n",
" fileName = os.path.splitext(base)[0]\n",
" imlabel.append(trainLabels.loc[trainLabels.image==fileName, 'level'].values[0])\n",
" im = Image.open(\"train/\" + file) \n",
" img = im.resize((img_rows,img_cols))\n",
" rgb = img.convert('RGB')\n",
" immatrix.append(np.array(rgb).flatten())"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#converting images & labels to numpy arrays\n",
"immatrix = np.asarray(immatrix)\n",
"imlabel = np.asarray(imlabel)\n",
"\n",
"data,Label = shuffle(immatrix,imlabel, random_state=2)\n",
"train_data = [data,Label]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Splitting Dataset to training and test samples"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.cross_validation import train_test_split\n",
"(X, y) = (train_data[0],train_data[1])\n",
"# STEP 1: split X and y into training and testing sets\n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=4)\n",
"\n",
"X_train = X_train.reshape(X_train.shape[0], img_cols, img_rows, 3)\n",
"X_test = X_test.reshape(X_test.shape[0], img_cols, img_rows, 3)\n",
"print(X_train.shape),print(y_train.shape),print(X_test.shape),print(y_test.shape)\n",
"\n",
"# Test set target variable distribution\n",
"unique, counts = np.unique(y_test, return_counts=True)\n",
"dict(zip(unique, counts))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train = X_train.astype('float32')\n",
"X_test = X_test.astype('float32')\n",
"\n",
"# Scaling\n",
"X_train /= 255\n",
"X_test /= 255\n",
"\n",
"print('X_train shape:', X_train.shape)\n",
"print(X_train.shape[0], 'train samples')\n",
"print(X_test.shape[0], 'test samples')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from keras.utils import np_utils\n",
"\n",
"# convert class vectors to binary class matrices\n",
"Y_train = np_utils.to_categorical(y_train, nb_classes)\n",
"Y_test = np_utils.to_categorical(y_test, nb_classes)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Performing Image augmentation"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from keras.preprocessing.image import ImageDataGenerator\n",
"# create generators - training data will be augmented imagees\n",
"validationdatagenerator = ImageDataGenerator()\n",
"traindatagenerator = ImageDataGenerator(width_shift_range=0.1,height_shift_range=0.1,rotation_range=15,zoom_range=0.1 )\n",
"batchsize=8\n",
"train_generator=traindatagenerator.flow(X_train, Y_train, batch_size=batchsize) \n",
"validation_generator=validationdatagenerator.flow(X_test, Y_test,batch_size=batchsize)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.compile(loss='categorical_crossentropy',\n",
" optimizer='adam',\n",
" metrics=['acc'])"
]
},
{
"cell_type": "raw",
"metadata": {
"collapsed": true
},
"source": [
"history= model.fit_generator(train_generator, steps_per_epoch=int(len(X_train)/batchsize), \n",
" epochs=10, validation_data=validation_generator, \n",
" validation_steps=int(len(X_test)/batchsize))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# summarize history for accuracy\n",
"plt.plot(history.history['acc'])\n",
"plt.plot(history.history['val_acc'])\n",
"plt.title('model accuracy')\n",
"plt.ylabel('accuracy')\n",
"plt.xlabel('epoch')\n",
"plt.legend(['train', 'test'], loc='upper left')\n",
"plt.show()\n",
"#plt.savefig('model_accuracy.png')\n",
"# summarize history for loss\n",
"plt.plot(history.history['loss'])\n",
"plt.plot(history.history['val_loss'])\n",
"plt.title('model loss')\n",
"plt.ylabel('loss')\n",
"plt.xlabel('epoch')\n",
"plt.legend(['train', 'test'], loc='upper left')\n",
"plt.show()\n",
"#plt.savefig('model_loss.png')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"score = model.evaluate(X_test, Y_test, verbose=0)\n",
"print(score)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Get a bad model"
]
},
{
"cell_type": "raw",
"metadata": {},
"source": [
"#Save model/ Don't have prediction\n",
"from keras.models import load_model\n",
"#model.save('retinopathy_predict.h5')\n",
"\n",
"print(X_train.shape)\n",
"print(X_test[0].shape)\n",
"y_pred = []\n",
"p = 100\n",
"for i in range (200):\n",
" image = X_train[i]\n",
" imagematrix = np.asarray(image)\n",
" #img1=imagematrix.reshape(img_rows,img_cols,3)\n",
" #plt.imshow(img1)\n",
" #print(imagematrix.shape)\n",
" imagepredict = np.expand_dims(imagematrix, axis=0)\n",
" #print(imagepredict.shape)\n",
" y_pred1 = model.predict(imagepredict)\n",
" y_pred2 = [x * p for x in y_pred1]\n",
" y_pred3 = np.max(y_pred2)\n",
" y_pred.append(y_pred3)\n",
" print(y_pred3)\n",
"y_pred = np.asarray(y_pred)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## INCEPTION \n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.model_selection import train_test_split\n",
"rr_df = retina_df[['PatientId', 'level']].drop_duplicates()\n",
"train_ids, valid_ids = train_test_split(rr_df['PatientId'], \n",
" test_size = 0.25, \n",
" random_state = 2018,\n",
" stratify = rr_df['level'])\n",
"raw_train_df = retina_df[retina_df['PatientId'].isin(train_ids)]\n",
"valid_df = retina_df[retina_df['PatientId'].isin(valid_ids)]\n",
"print('train', raw_train_df.shape[0], 'validation', valid_df.shape[0])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"retina_df[['level', 'eye']].hist(figsize = (10, 5))"
]
},
{
"cell_type": "raw",
"metadata": {
"scrolled": true
},
"source": [
"#Balance the distribution in the training set\n",
"train_df = raw_train_df.groupby(['level', 'eye']).apply(lambda x: x.sample(75, replace = True)\n",
" ).reset_index(drop = True)\n",
"print('New Data Size:', train_df.shape[0], 'Old Size:', raw_train_df.shape[0])\n",
"train_df[['level', 'eye']].hist(figsize = (10, 5))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"from keras import backend as K\n",
"from keras.applications.inception_v3 import preprocess_input\n",
"import numpy as np\n",
"IMG_SIZE = (512, 512) # slightly smaller than vgg16 normally expects\n",
"def tf_image_loader(out_size, \n",
" horizontal_flip = True, \n",
" vertical_flip = False, \n",
" random_brightness = True,\n",
" random_contrast = True,\n",
" random_saturation = True,\n",
" random_hue = True,\n",
" color_mode = 'rgb',\n",
" preproc_func = preprocess_input,\n",
" on_batch = False):\n",
" def _func(X):\n",
" with tf.name_scope('image_augmentation'):\n",
" with tf.name_scope('input'):\n",
" X = tf.image.decode_png(tf.read_file(X), channels = 3 if color_mode == 'rgb' else 0)\n",
" X = tf.image.resize_images(X, out_size)\n",
" with tf.name_scope('augmentation'):\n",
" if horizontal_flip:\n",
" X = tf.image.random_flip_left_right(X)\n",
" if vertical_flip:\n",
" X = tf.image.random_flip_up_down(X)\n",
" if random_brightness:\n",
" X = tf.image.random_brightness(X, max_delta = 0.1)\n",
" if random_saturation:\n",
" X = tf.image.random_saturation(X, lower = 0.75, upper = 1.5)\n",
" if random_hue:\n",
" X = tf.image.random_hue(X, max_delta = 0.15)\n",
" if random_contrast:\n",
" X = tf.image.random_contrast(X, lower = 0.75, upper = 1.5)\n",
" return preproc_func(X)\n",
" if on_batch: \n",
" # we are meant to use it on a batch\n",
" def _batch_func(X, y):\n",
" return tf.map_fn(_func, X), y\n",
" return _batch_func\n",
" else:\n",
" # we apply it to everything\n",
" def _all_func(X, y):\n",
" return _func(X), y \n",
" return _all_func\n",
" \n",
"def tf_augmentor(out_size,\n",
" intermediate_size = (640, 640),\n",
" intermediate_trans = 'crop',\n",
" batch_size = 16,\n",
" horizontal_flip = True, \n",
" vertical_flip = False, \n",
" random_brightness = True,\n",
" random_contrast = True,\n",
" random_saturation = True,\n",
" random_hue = True,\n",
" color_mode = 'rgb',\n",
" preproc_func = preprocess_input,\n",
" min_crop_percent = 0.001,\n",
" max_crop_percent = 0.005,\n",
" crop_probability = 0.5,\n",
" rotation_range = 10):\n",
" \n",
" load_ops = tf_image_loader(out_size = intermediate_size, \n",
" horizontal_flip=horizontal_flip, \n",
" vertical_flip=vertical_flip, \n",
" random_brightness = random_brightness,\n",
" random_contrast = random_contrast,\n",
" random_saturation = random_saturation,\n",
" random_hue = random_hue,\n",
" color_mode = color_mode,\n",
" preproc_func = preproc_func,\n",
" on_batch=False)\n",
" def batch_ops(X, y):\n",
" batch_size = tf.shape(X)[0]\n",
" with tf.name_scope('transformation'):\n",
" # code borrowed from https://becominghuman.ai/data-augmentation-on-gpu-in-tensorflow-13d14ecf2b19\n",
" # The list of affine transformations that our image will go under.\n",
" # Every element is Nx8 tensor, where N is a batch size.\n",
" transforms = []\n",
" identity = tf.constant([1, 0, 0, 0, 1, 0, 0, 0], dtype=tf.float32)\n",
" if rotation_range > 0:\n",
" angle_rad = rotation_range / 180 * np.pi\n",
" angles = tf.random_uniform([batch_size], -angle_rad, angle_rad)\n",
" transforms += [tf.contrib.image.angles_to_projective_transforms(angles, intermediate_size[0], intermediate_size[1])]\n",
"\n",
" if crop_probability > 0:\n",
" crop_pct = tf.random_uniform([batch_size], min_crop_percent, max_crop_percent)\n",
" left = tf.random_uniform([batch_size], 0, intermediate_size[0] * (1.0 - crop_pct))\n",
" top = tf.random_uniform([batch_size], 0, intermediate_size[1] * (1.0 - crop_pct))\n",
" crop_transform = tf.stack([\n",
" crop_pct,\n",
" tf.zeros([batch_size]), top,\n",
" tf.zeros([batch_size]), crop_pct, left,\n",
" tf.zeros([batch_size]),\n",
" tf.zeros([batch_size])\n",
" ], 1)\n",
" coin = tf.less(tf.random_uniform([batch_size], 0, 1.0), crop_probability)\n",
" transforms += [tf.where(coin, crop_transform, tf.tile(tf.expand_dims(identity, 0), [batch_size, 1]))]\n",
" if len(transforms)>0:\n",
" X = tf.contrib.image.transform(X,\n",
" tf.contrib.image.compose_transforms(*transforms),\n",
" interpolation='BILINEAR') # or 'NEAREST'\n",
" if intermediate_trans=='scale':\n",
" X = tf.image.resize_images(X, out_size)\n",
" elif intermediate_trans=='crop':\n",
" X = tf.image.resize_image_with_crop_or_pad(X, out_size[0], out_size[1])\n",
" else:\n",
" raise ValueError('Invalid Operation {}'.format(intermediate_trans))\n",
" return X, y\n",
" def _create_pipeline(in_ds):\n",
" batch_ds = in_ds.map(load_ops, num_parallel_calls=4).batch(batch_size)\n",
" return batch_ds.map(batch_ops)\n",
" return _create_pipeline"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def flow_from_dataframe(idg, \n",
" in_df, \n",
" path_col,\n",
" y_col, \n",
" shuffle = True, \n",
" color_mode = 'rgb'):\n",
" files_ds = tf.data.Dataset.from_tensor_slices((in_df[path_col].values, \n",
" np.stack(in_df[y_col].values,0)))\n",
" in_len = in_df[path_col].values.shape[0]\n",
" while True:\n",
" if shuffle:\n",
" files_ds = files_ds.shuffle(in_len) # shuffle the whole dataset\n",
" \n",
" next_batch = idg(files_ds).repeat().make_one_shot_iterator().get_next()\n",
" for i in range(max(in_len//32,1)):\n",
" # NOTE: if we loop here it is 'thread-safe-ish' if we loop on the outside it is completely unsafe\n",
" yield K.get_session().run(next_batch)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"batch_size = 48\n",
"core_idg = tf_augmentor(out_size = IMG_SIZE, \n",
" color_mode = 'rgb', \n",
" vertical_flip = True,\n",
" crop_probability=0.0, # crop doesn't work yet\n",
" batch_size = batch_size) \n",
"valid_idg = tf_augmentor(out_size = IMG_SIZE, color_mode = 'rgb', \n",
" crop_probability=0.0, \n",
" horizontal_flip = False, \n",
" vertical_flip = False, \n",
" random_brightness = False,\n",
" random_contrast = False,\n",
" random_saturation = False,\n",
" random_hue = False,\n",
" rotation_range = 0,\n",
" batch_size = batch_size)\n",
"\n",
"train_gen = flow_from_dataframe(core_idg, train_df, \n",
" path_col = 'path',\n",
" y_col = 'level_cat')\n",
"\n",
"valid_gen = flow_from_dataframe(valid_idg, valid_df, \n",
" path_col = 'path',\n",
" y_col = 'level_cat') # we can use much larger batches for evaluation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Validation Set\n",
"We do not perform augmentation at all on these images"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"t_x, t_y = next(valid_gen)\n",
"fig, m_axs = plt.subplots(2, 4, figsize = (16, 8))\n",
"for (c_x, c_y, c_ax) in zip(t_x, t_y, m_axs.flatten()):\n",
" c_ax.imshow(np.clip(c_x*127+127, 0, 255).astype(np.uint8))\n",
" c_ax.set_title('Severity {}'.format(np.argmax(c_y, -1)))\n",
" c_ax.axis('off')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Training Set\n",
"These are augmented and a real mess"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"t_x, t_y = next(train_gen)\n",
"fig, m_axs = plt.subplots(2, 4, figsize = (16, 8))\n",
"for (c_x, c_y, c_ax) in zip(t_x, t_y, m_axs.flatten()):\n",
" c_ax.imshow(np.clip(c_x*127+127, 0, 255).astype(np.uint8))\n",
" c_ax.set_title('Severity {}'.format(np.argmax(c_y, -1)))\n",
" c_ax.axis('off')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Attention Model\n",
"The basic idea is that a Global Average Pooling is too simplistic since some of the regions are more relevant than others. So we build an attention mechanism to turn pixels in the GAP on an off before the pooling and then rescale (Lambda layer) the results based on the number of pixels. The model could be seen as a sort of 'global weighted average' pooling. There is probably something published about it and it is very similar to the kind of attention models used in NLP."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from keras.applications.vgg16 import VGG16 as PTModel\n",
"from keras.applications.inception_resnet_v2 import InceptionResNetV2 as PTModel\n",
"from keras.applications.inception_v3 import InceptionV3 as PTModel\n",
"from keras.layers import GlobalAveragePooling2D, Dense, Dropout, Flatten, Input, Conv2D, multiply, LocallyConnected2D, Lambda\n",
"from keras.models import Model\n",
"in_lay = Input(t_x.shape[1:])\n",
"base_pretrained_model = PTModel(input_shape = t_x.shape[1:], include_top = False, weights = 'imagenet')\n",
"base_pretrained_model.trainable = False\n",
"pt_depth = base_pretrained_model.get_output_shape_at(0)[-1]\n",
"pt_features = base_pretrained_model(in_lay)\n",
"from keras.layers import BatchNormalization\n",
"bn_features = BatchNormalization()(pt_features)\n",
"\n",
"# here we do an attention mechanism to turn pixels in the GAP on an off\n",
"\n",
"attn_layer = Conv2D(64, kernel_size = (1,1), padding = 'same', activation = 'relu')(Dropout(0.5)(bn_features))\n",
"attn_layer = Conv2D(16, kernel_size = (1,1), padding = 'same', activation = 'relu')(attn_layer)\n",
"attn_layer = Conv2D(8, kernel_size = (1,1), padding = 'same', activation = 'relu')(attn_layer)\n",
"attn_layer = Conv2D(1, \n",
" kernel_size = (1,1), \n",
" padding = 'valid', \n",
" activation = 'sigmoid')(attn_layer)\n",
"# fan it out to all of the channels\n",
"up_c2_w = np.ones((1, 1, 1, pt_depth))\n",
"up_c2 = Conv2D(pt_depth, kernel_size = (1,1), padding = 'same', \n",
" activation = 'linear', use_bias = False, weights = [up_c2_w])\n",
"up_c2.trainable = False\n",
"attn_layer = up_c2(attn_layer)\n",
"\n",
"mask_features = multiply([attn_layer, bn_features])\n",
"gap_features = GlobalAveragePooling2D()(mask_features)\n",
"gap_mask = GlobalAveragePooling2D()(attn_layer)\n",
"# to account for missing values from the attention model\n",
"gap = Lambda(lambda x: x[0]/x[1], name = 'RescaleGAP')([gap_features, gap_mask])\n",
"gap_dr = Dropout(0.25)(gap)\n",
"dr_steps = Dropout(0.25)(Dense(128, activation = 'relu')(gap_dr))\n",
"out_layer = Dense(t_y.shape[-1], activation = 'softmax')(dr_steps)\n",
"retina_model = Model(inputs = [in_lay], outputs = [out_layer])\n",
"from keras.metrics import top_k_categorical_accuracy\n",
"def top_2_accuracy(in_gt, in_pred):\n",
" return top_k_categorical_accuracy(in_gt, in_pred, k=2)\n",
"\n",
"retina_model.compile(optimizer = 'adam', loss = 'categorical_crossentropy',\n",
" metrics = ['categorical_accuracy', top_2_accuracy])\n",
"retina_model.summary()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from keras.callbacks import ModelCheckpoint, LearningRateScheduler, EarlyStopping, ReduceLROnPlateau\n",
"weight_path=\"{}_weights.best.hdf5\".format('retina')\n",
"\n",
"checkpoint = ModelCheckpoint(weight_path, monitor='val_loss', verbose=1, \n",
" save_best_only=True, mode='min', save_weights_only = True)\n",
"\n",
"reduceLROnPlat = ReduceLROnPlateau(monitor='val_loss', factor=0.8, patience=3, verbose=1, mode='auto', epsilon=0.0001, cooldown=5, min_lr=0.0001)\n",
"early = EarlyStopping(monitor=\"val_loss\", \n",
" mode=\"min\", \n",
" patience=6) # probably needs to be more patient, but kaggle time is limited\n",
"callbacks_list = [checkpoint, early, reduceLROnPlat]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#!rm -rf ~/.keras # clean up before starting training"
]
},
{
"cell_type": "raw",
"metadata": {
"collapsed": true
},
"source": [
"retina_model.fit_generator(train_gen, \n",
" steps_per_epoch = train_df.shape[0]//batch_size,\n",
" validation_data = valid_gen, \n",
" validation_steps = valid_df.shape[0]//batch_size,\n",
" epochs = 25, \n",
" callbacks = callbacks_list,\n",
" workers = 0, # tf-generators are not thread-safe\n",
" use_multiprocessing=False, \n",
" max_queue_size = 0\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"##### create one fixed dataset for evaluating\n",
"from tqdm import tqdm_notebook\n",
"# fresh valid gen\n",
"valid_gen = flow_from_dataframe(valid_idg, valid_df, \n",
" path_col = 'path',\n",
" y_col = 'level_cat') \n",
"vbatch_count = (valid_df.shape[0]//batch_size-1)\n",
"out_size = vbatch_count*batch_size\n",
"test_X = np.zeros((out_size,)+t_x.shape[1:], dtype = np.float32)\n",
"test_Y = np.zeros((out_size,)+t_y.shape[1:], dtype = np.float32)\n",
"for i, (c_x, c_y) in zip(tqdm_notebook(range(vbatch_count)), \n",
" valid_gen):\n",
" j = i*batch_size\n",
" test_X[j:(j+c_x.shape[0])] = c_x\n",
" test_Y[j:(j+c_x.shape[0])] = c_y"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Show Attention\n",
"Did our attention model learn anything useful?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# get the attention layer since it is the only one with a single output dim\n",
"for attn_layer in retina_model.layers:\n",
" c_shape = attn_layer.get_output_shape_at(0)\n",
" if len(c_shape)==4:\n",
" if c_shape[-1]==1:\n",
" print(attn_layer)\n",
" break"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import keras.backend as K\n",
"rand_idx = np.random.choice(range(len(test_X)), size = 6)\n",
"attn_func = K.function(inputs = [retina_model.get_input_at(0), K.learning_phase()],\n",
" outputs = [attn_layer.get_output_at(0)]\n",
" )\n",
"fig, m_axs = plt.subplots(len(rand_idx), 2, figsize = (8, 4*len(rand_idx)))\n",
"[c_ax.axis('off') for c_ax in m_axs.flatten()]\n",
"for c_idx, (img_ax, attn_ax) in zip(rand_idx, m_axs):\n",
" cur_img = test_X[c_idx:(c_idx+1)]\n",
" attn_img = attn_func([cur_img, 0])[0]\n",
" img_ax.imshow(np.clip(cur_img[0,:,:,:]*127+127, 0, 255).astype(np.uint8))\n",
" attn_ax.imshow(attn_img[0, :, :, 0]/attn_img[0, :, :, 0].max(), cmap = 'viridis', \n",
" vmin = 0, vmax = 1, \n",
" interpolation = 'lanczos')\n",
" real_cat = np.argmax(test_Y[c_idx, :])\n",
" img_ax.set_title('Eye Image\\nCat:%2d' % (real_cat))\n",
" pred_cat = retina_model.predict(cur_img)\n",
" attn_ax.set_title('Attention Map\\nPred:%2.2f%%' % (100*pred_cat[0,real_cat]))\n",
"#fig.savefig('attention_map.png', dpi = 300)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Evaluate the results\n",
"Here we evaluate the results the best version of the model and seeing how the predictions look on the results. We then visualize spec"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.metrics import accuracy_score, classification_report\n",
"pred_Y = retina_model.predict(test_X, batch_size = 32, verbose = True)\n",
"pred_Y_cat = np.argmax(pred_Y, -1)\n",
"test_Y_cat = np.argmax(test_Y, -1)\n",
"print('Accuracy on Test Data: %2.2f%%' % (accuracy_score(test_Y_cat, pred_Y_cat)))\n",
"print(classification_report(test_Y_cat, pred_Y_cat))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import seaborn as sns\n",
"from sklearn.metrics import confusion_matrix\n",
"sns.heatmap(confusion_matrix(test_Y_cat, pred_Y_cat), \n",
" annot=True, fmt=\"d\", cbar = False, cmap = plt.cm.Blues, vmax = test_X.shape[0]//16)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## ROC Curve for healthy vs sick\n",
"Here we make an ROC curve for healthy (severity = 0) and heavy sick (severity=4) to see how well the model works at just identifying the disease"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.metrics import roc_curve, roc_auc_score\n",
"sick_vec = test_Y_cat>0\n",
"sick_score = np.sum(pred_Y[:,1:],1)\n",
"fpr, tpr, _ = roc_curve(sick_vec, sick_score)\n",
"fig, ax1 = plt.subplots(1,1, figsize = (6, 6), dpi = 150)\n",
"ax1.plot(fpr, tpr, 'b.-', label = 'Model Prediction (AUC: %2.2f)' % roc_auc_score(sick_vec, sick_score))\n",
"ax1.plot(fpr, fpr, 'g-', label = 'Random Guessing')\n",
"ax1.legend()\n",
"ax1.set_xlabel('False Positive Rate')\n",
"ax1.set_ylabel('True Positive Rate');"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"AUC metric say that the model works well. But it only for binary classification. Healthy or heavy sick"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"fig, m_axs = plt.subplots(2, 4, figsize = (32, 20))\n",
"for (idx, c_ax) in enumerate(m_axs.flatten()):\n",
" c_ax.imshow(np.clip(test_X[idx]*127+127,0 , 255).astype(np.uint8), cmap = 'bone')\n",
" c_ax.set_title('Actual Severity: {}\\n{}'.format(test_Y_cat[idx], \n",
" '\\n'.join(['Predicted %02d (%04.1f%%): %s' % (k, 100*v, '*'*int(10*v)) for k, v in sorted(enumerate(pred_Y[idx]), key = lambda x: -1*x[1])])), loc='left')\n",
" c_ax.axis('off')\n",
"#fig.savefig('trained_img_predictions.png', dpi = 300)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Keras "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Image to Numpy Array\n",
"from PIL import Image\n",
"\n",
"# input image dimensions\n",
"img_rows, img_cols = 224, 224\n",
"### TRPLACE !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!1\n",
"#img_rows, img_cols = 200, 200\n",
"\n",
"immatrix = []\n",
"imlabel = []\n",
"\n",
"for file in listing:\n",
" base = os.path.basename(\"train/\" + file)\n",
" fileName = os.path.splitext(base)[0]\n",
" imlabel.append(trainLabels.loc[trainLabels.image==fileName, 'level'].values[0])\n",
" im = Image.open(\"train/\" + file) \n",
" img = im.resize((img_rows,img_cols))\n",
" #rgb = img.convert('RGB')\n",
" gray = img.convert('L')\n",
" immatrix.append(np.array(gray).flatten())"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"immatrix = np.asarray(immatrix)\n",
"imlabel = np.asarray(imlabel)\n",
"from sklearn.utils import shuffle\n",
"\n",
"data,Label = shuffle(immatrix,imlabel, random_state=2)\n",
"train_data = [data,Label]\n",
"type(train_data)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"import matplotlib\n",
"\n",
"img=immatrix[167].reshape(img_rows,img_cols)\n",
"plt.imshow(img)\n",
"plt.imshow(img,cmap='gray')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#batch_size to train\n",
"batch_size = 32\n",
"# number of output classes\n",
"nb_classes = 5\n",
"# number of epochs to train\n",
"nb_epoch = 5\n",
"# number of convolutional filters to use\n",
"nb_filters = 32\n",
"# size of pooling area for max pooling\n",
"nb_pool = 2\n",
"# convolution kernel size\n",
"nb_conv = 3"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"(X, y) = (train_data[0],train_data[1])\n",
"from sklearn.cross_validation import train_test_split\n",
"\n",
"# STEP 1: split X and y into training and testing sets\n",
"\n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=4)\n",
"\n",
"print(X_train.shape)\n",
"print(X_test.shape)\n",
"\n",
"X_train = X_train.reshape(X_train.shape[0], img_cols, img_rows, 1)\n",
"X_test = X_test.reshape(X_test.shape[0], img_cols, img_rows, 1)\n",
"\n",
"X_train = X_train.astype('float32')\n",
"X_test = X_test.astype('float32')\n",
"\n",
"X_train /= 255\n",
"X_test /= 255\n",
"\n",
"print('X_train shape:', X_train.shape)\n",
"print(X_train.shape[0], 'train samples')\n",
"print(X_test.shape[0], 'test samples')\n",
"\n",
"from keras.utils import np_utils\n",
"# convert class vectors to binary class matrices\n",
"Y_train = np_utils.to_categorical(y_train, nb_classes)\n",
"Y_test = np_utils.to_categorical(y_test, nb_classes)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from keras.models import Sequential\n",
"from keras.layers.core import Dense, Dropout, Activation, Flatten\n",
"from keras.layers.convolutional import Convolution2D, MaxPooling2D\n",
"from keras.optimizers import SGD,RMSprop,adam\n",
"\n",
"model = Sequential()\n",
"\n",
"model.add(Convolution2D(nb_filters, nb_conv, nb_conv,\n",
" border_mode='valid',\n",
" input_shape=(img_cols, img_rows, 1)))\n",
"convout1 = Activation('relu')\n",
"model.add(convout1)\n",
"model.add(Convolution2D(nb_filters, nb_conv, nb_conv))\n",
"convout2 = Activation('relu')\n",
"model.add(convout2)\n",
"model.add(MaxPooling2D(pool_size=(nb_pool, nb_pool)))\n",
"model.add(Dropout(0.5))\n",
"\n",
"model.add(Flatten())\n",
"model.add(Dense(128))\n",
"model.add(Activation('relu'))\n",
"model.add(Dropout(0.5))\n",
"model.add(Dense(nb_classes))\n",
"model.add(Activation('softmax'))\n",
"model.compile(loss='categorical_crossentropy', optimizer='adadelta')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from keras.preprocessing.image import ImageDataGenerator\n",
"\n",
"# create generators - training data will be augmented images\n",
"validationdatagenerator = ImageDataGenerator()\n",
"traindatagenerator = ImageDataGenerator(width_shift_range=0.1,height_shift_range=0.1,rotation_range=15,zoom_range=0.1 )\n",
"\n",
"batchsize=8\n",
"train_generator=traindatagenerator.flow(X_train, Y_train, batch_size=batchsize) \n",
"validation_generator=validationdatagenerator.flow(X_test, Y_test,batch_size=batchsize)\n",
"\n",
"#hist = model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch, verbose=1, validation_data=(X_test, Y_test))\n",
"\n",
"model.fit_generator(train_generator, steps_per_epoch=int(len(X_train)/batchsize), epochs=3, validation_data=validation_generator, validation_steps=int(len(X_test)/batchsize))\n",
"\n",
"#hist = model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch,\n",
"# show_accuracy=True, verbose=1, validation_split=0.2)\n",
"\n",
"score = model.evaluate(X_test, Y_test, verbose=0)\n",
"print(score)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### OTHER"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"import cv2\n",
"from matplotlib import pyplot as plt\n",
"import os\n",
"from subprocess import check_output\n",
"import cv2\n",
"from PIL import Image\n",
"import glob\n",
"\n",
"img_data = []\n",
"img_label = []\n",
"img_r = 224\n",
"img_c = 224\n",
"for file in listing:\n",
" file = os.path.join(\"train/\" + file)\n",
" tmp = cv2.imread(file)\n",
" tmp = cv2.resize(tmp,(img_r, img_c), interpolation = cv2.INTER_CUBIC)\n",
" tmp = cv2.cvtColor(tmp, cv2.COLOR_BGR2GRAY)\n",
" tmp = cv2.normalize(tmp, None, alpha=0, beta=1, norm_type=cv2.NORM_MINMAX, dtype=cv2.CV_32F)\n",
" img_data.append(np.array(tmp).flatten())\n",
" tmpfn = file\n",
" tmpfn = tmpfn.replace(\"train/\",\"\")\n",
" tmpfn = tmpfn.replace(\".jpeg\",\"\")\n",
" img_label.append(trainLabels.loc[trainLabels.image==tmpfn, 'level'].values[0])\n",
" \n",
"fileName = []\n",
"eye = []\n",
"for file in listing:\n",
" tmpfn = file\n",
" tmpfn = tmpfn.replace(\"train/\",\"\")\n",
" tmpfn = tmpfn.replace(\".jpeg\",\"\")\n",
" fileName.append(tmpfn)\n",
" if \"left\" in tmpfn:\n",
" eye.append(1)\n",
" else:\n",
" eye.append(0)\n",
" \n",
"#data = pd.DataFrame({'fileName':fileName,'eye':eye,'img_data':img_data,'label':img_label}) # keyerror 10\n",
"data = pd.DataFrame({'eye':eye,'img_data':img_data,'label':img_label})\n",
"data.sample(3)\n",
"\n",
"from sklearn.model_selection import train_test_split\n",
"X = data['img_data']\n",
"y = data['label']\n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from keras.models import Sequential\n",
"from keras.layers.core import Dense, Dropout, Activation, Flatten\n",
"from keras.layers.convolutional import Convolution2D, MaxPooling2D\n",
"from keras.optimizers import SGD,RMSprop,adam\n",
"from keras.layers import Conv2D, MaxPooling2D\n",
"from keras.optimizers import SGD\n",
"from keras.layers import Dense, Dropout, Flatten\n",
"# Create CNN model\n",
"model = Sequential()\n",
"model.add(Conv2D(64, (3, 3), activation='relu', input_shape = (img_r,img_c,1)))\n",
"model.add(Conv2D(128, (3, 3), activation='relu'))\n",
"model.add(Dropout(0.5))\n",
"model.add(Conv2D(128, (3, 3), activation='relu'))\n",
"model.add(MaxPooling2D(pool_size=(2, 2)))\n",
"model.add(Conv2D(256, (3, 3), activation='relu'))\n",
"model.add(Dropout(0.75))\n",
"model.add(Conv2D(128, (3, 3), activation='relu'))\n",
"model.add(MaxPooling2D(pool_size=(2, 2)))\n",
"model.add(Conv2D(128, (3, 3), activation='relu'))\n",
"model.add(Dropout(0.5))\n",
"model.add(Conv2D(64, (3, 3), activation='relu'))\n",
"model.add(MaxPooling2D(pool_size=(2, 2)))\n",
"model.add(Dropout(0.25))\n",
"\n",
"model.add(Flatten())\n",
"model.add(Dense(256, activation='relu'))\n",
"model.add(Dense(5, activation='softmax'))\n",
"\n",
"# Calculate the class weights for unbalanced data\n",
"from sklearn.utils.class_weight import compute_class_weight\n",
"classes = np.unique(y_train)\n",
"class_weight = compute_class_weight(\"balanced\", classes, y_train)\n",
"\n",
"sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)\n",
"# Compile model\n",
"model.compile(loss='categorical_crossentropy', optimizer='sgd', metrics=['accuracy'])\n",
"# convert class vectors to binary class matrices\n",
"nb_classes = 5\n",
"Y_train = np_utils.to_categorical(y_train, nb_classes)\n",
"Y_test = np_utils.to_categorical(y_test, nb_classes)\n",
"# Fit the model\n",
"model.fit(X_train_resh, Y_train, batch_size = 16, epochs=30, verbose=1,class_weight=class_weight)\n",
"score = model.evaluate(X_test_resh, Y_test, verbose=0)\n",
"print(\"%s: %.2f%%\" % (model.metrics_names[1], score[1]*100))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 12. Conclusions (2 points) "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"* (+) The value of the solution, possible cases for its application and further ways of developing and improving the solution are described;"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Sorry for this mess, unstructured data etc and little description"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Inception v3 as a base and retraining some modified final layers with attention seems good for simple binary classification(healthy=0 or heavy sick=4). But really important task that find a disease on early stage, target==1(Mild). \n",
"Further: \n",
"* increase size of images\n",
"* multiclassification\n",
"* improve attention/related techniques to focus on areas, segmentation\n",
"* learn"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.4"
}
},
"nbformat": 4,
"nbformat_minor": 2
}