{"cells":[{"attachments":{},"cell_type":"markdown","metadata":{},"source":["# NN Classify 15 Fruits Assignment"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["![Fruit Example](https://static-1300131294.cos.ap-shanghai.myqcloud.com/images/assignment/deep-learning/nn/examplefruit.png)"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["# Data collection\n","The database used in this study is comprising of 44406 fruit images, which we collected\n","in a period of 6 months. The images where made with in our lab’s environment under different\n","scenarios which we mention below. We captured all the images on a clear background with\n","resolution of 320×258 pixels. We used HD Logitech web camera to took the pictures. During\n","collecting this database, we created all kind of challenges, which, we have to face in real-world\n","recognition scenarios in supermarket and fruit shops such as light, shadow, sunshine, pose\n","variation, to make our model robust for, it might be necessary to cope with illumination\n","variation, camera capturing artifacts, specular reflection shading and shadows. We tested our\n","model’s robustness in all scenarios and it perform quit well.\n","All of images were stored in RGB color-space at 8 bits per channel. The images were\n","gathered at various day times of the day and in different days for the same category. These\n","features increase the dataset variability and represent more realistic scenario. The Images had\n","large variation in quality and lighting. Illumination is one of those variations in imagery. In fact,\n","illumination can make two images of same fruit less similar than two images of different kind\n","of fruits. We were used our own intelligent weight machine and camera to captured all images.\n","The fruit dataset was collected under relatively unconstrained conditions. There are also images\n","with the room light on and room lights off, moved the camera and intelligent weight machine\n","near to the windows of our lab than open windows, closed windows, open window curtains,\n","closed curtains. For a real application in a supermarket, it might be necessary to cope with\n","illumination variation, camera capturing artifacts, specular reflection shading and shadows.\n","Below are the few conditions which we were considered during collected dataset.\n","- Pose Variations with different categories of fruits\n","- Variability on the number of elements of fruits\n","- Used HD camera with 5-megapixel snapshots\n","- Same color but different Category fruits images with illumination variation\n","- Cropping and partial occlusion\n","- Different color same category fruit images\n","- Different lighting conditions (e.g. fluorescent, natural light some of the fruits shops\n","- and supermarkets are without sunshine so it can easily affect the recognition system\n","- Six different kind of apple fruit images\n","- Three categories of mango fruit with specular reflecting shading and shadows\n","- Three categories of Kiwi fruit images\n","- Natural and artificial lighting effect on images\n","- Partial occlusion with hand\n","\n"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["# Load and visualize the dataset\n","\n","The database used in this study is comprising of 70549 fruit images, which were collected in a period of 6 months. The images where made with in a lab’s environment under different scenarios which we mention below. All the images were captured on a clear background with resolution of 320×258 pixels.\n","\n","Type of fruits in the dataset:\n","- Apple\n","- Banana\n","- Carambola\n","- Guava\n","- Kiwi\n","- Mango\n","- Orange\n","- Peach\n","- Pear\n","- Persimmon\n","- Pitaya\n","- Plum\n","- Pomegranate\n","- Tomatoes\n","- muskmelon"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2023-07-13T08:28:49.706912Z","iopub.status.busy":"2023-07-13T08:28:49.706499Z","iopub.status.idle":"2023-07-13T08:29:00.108211Z","shell.execute_reply":"2023-07-13T08:29:00.106943Z","shell.execute_reply.started":"2023-07-13T08:28:49.706878Z"},"trusted":true},"outputs":[],"source":["# Load the libraries\n","import pandas as pd\n","import numpy as np\n","import seaborn as sns\n","import os\n","import cv2\n","import matplotlib.pyplot as plt\n","import random\n","import time\n","from sklearn.metrics import classification_report, confusion_matrix, accuracy_score\n","from sklearn.model_selection import train_test_split\n","import keras\n","from keras import Sequential\n","from keras.layers import Activation, Dropout, Flatten, Dense, Conv2D, MaxPooling2D\n","from keras.utils import to_categorical\n","from keras.callbacks import EarlyStopping, ModelCheckpoint\n","import gc\n","import requests\n","import zipfile\n","from IPython.display import Markdown, display\n","def printmd(string):\n","    # Print with Markdowns    \n","    display(Markdown(string))\n","    \n","np.random.seed(0) # Add random seed of training for reproducibility\n","\n","def load_images_from_folder(folder,only_path = False, label = \"\"):\n","# Load the paths to the images in a directory\n","# or load the images\n","    if only_path == False:\n","        images = []\n","        for filename in os.listdir(folder):\n","            img = plt.imread(os.path.join(folder,filename))\n","            if img is not None:\n","                images.append(img)\n","        return images\n","    else:\n","        path = []\n","        for filename in os.listdir(folder):\n","            img_path = os.path.join(folder,filename)\n","            if img_path is not None:\n","                path.append([label,img_path])\n","        return path"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["datasets_url = \"https://static-1300131294.cos.ap-shanghai.myqcloud.com/data/deep-learning/nn/nn-classify-15-fruits.zip\""]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["notebook_path = os.getcwd()\n","\n","tmp_folder_path = os.path.join(notebook_path, \"tmp\")\n","\n","if not os.path.exists(tmp_folder_path):\n","    os.makedirs(tmp_folder_path)\n","\n","tmp_zip_path = os.path.join(tmp_folder_path,\"zip-store\")\n","\n","if not os.path.exists(tmp_zip_path):\n","    os.makedirs(tmp_zip_path)\n","    \n","datasets_response = requests.get(datasets_url)\n","\n","datasets_name = os.path.basename(datasets_url)\n","\n","datasets_save_path = os.path.join(tmp_zip_path, datasets_name)\n","\n","with open(datasets_save_path, \"wb\") as file:\n","    file.write(datasets_response.content)\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["zip_file_path = f\"./tmp/zip-store/{datasets_name}\"\n","extract_path = \"./tmp/\"\n","\n","zip_ref = zipfile.ZipFile(zip_file_path, 'r')\n","zip_ref.extractall(extract_path)\n","zip_ref.close()"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2023-07-13T08:29:00.111374Z","iopub.status.busy":"2023-07-13T08:29:00.110410Z","iopub.status.idle":"2023-07-13T08:29:03.419617Z","shell.execute_reply":"2023-07-13T08:29:03.418419Z","shell.execute_reply.started":"2023-07-13T08:29:00.111324Z"},"trusted":true},"outputs":[],"source":["# Load the paths on the images\n","images = []\n","DATASET_PATH =\"./tmp/\"\n","for f in os.listdir(DATASET_PATH):\n","    if \"png\" in os.listdir(DATASET_PATH+f)[0]:\n","        images += load_images_from_folder(DATASET_PATH+f,True,label = f)\n","    else: \n","        for d in os.listdir(DATASET_PATH+f):\n","            if os.path.isdir(os.path.join(DATASET_PATH+f, d)):  # Check if it is a directory\n","                images += load_images_from_folder(os.path.join(DATASET_PATH+f, d), True, label=f)            \n","# Create a dataframe with the paths and the label for each fruit\n","df = pd.DataFrame(images, columns = [\"fruit\", \"path\"])\n","\n","# Shuffle the dataset\n","from sklearn.utils import shuffle\n","df = shuffle(df, random_state = 0)\n","df = df.reset_index(drop=True)\n","\n","# Assign to each fruit a specific number\n","fruit_names = sorted(df.fruit.unique())\n","mapper_fruit_names = dict(zip(fruit_names, [t for t in range(len(fruit_names))]))\n","df[\"label\"] = df[\"fruit\"].map(mapper_fruit_names)\n","print(mapper_fruit_names)\n","\n","# Visualize the resulting dataframe\n","df.head()"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2023-07-13T08:29:03.421193Z","iopub.status.busy":"2023-07-13T08:29:03.420857Z","iopub.status.idle":"2023-07-13T08:29:03.883114Z","shell.execute_reply":"2023-07-13T08:29:03.881932Z","shell.execute_reply.started":"2023-07-13T08:29:03.421164Z"},"trusted":true},"outputs":[],"source":["# Display the number of pictures of each category\n","vc = df[\"fruit\"].value_counts()\n","plt.figure(figsize=(10,5))\n","sns.barplot(x = vc.index, y = vc, palette = \"rocket\")\n","plt.title(\"Number of pictures of each category\", fontsize = 15)\n","plt.xticks(rotation=90)\n","plt.show()"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2023-07-13T08:29:03.886022Z","iopub.status.busy":"2023-07-13T08:29:03.885670Z","iopub.status.idle":"2023-07-13T08:29:07.009447Z","shell.execute_reply":"2023-07-13T08:29:07.008558Z","shell.execute_reply.started":"2023-07-13T08:29:03.885990Z"},"trusted":true},"outputs":[],"source":["# Display some pictures of the dataset\n","fig, axes = plt.subplots(nrows=4, ncols=5, figsize=(15, 15),\n","                        subplot_kw={'xticks': [], 'yticks': []})\n","\n","for i, ax in enumerate(axes.flat):\n","    ax.imshow(plt.imread(df.path[i]))\n","    ax.set_title(df.fruit[i], fontsize = 12)\n","plt.tight_layout(pad=0.0)\n","plt.show()"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["# Train the neural network from scratch with Keras and w/o generator"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2023-07-13T08:29:07.010799Z","iopub.status.busy":"2023-07-13T08:29:07.010467Z","iopub.status.idle":"2023-07-13T08:29:07.885382Z","shell.execute_reply":"2023-07-13T08:29:07.884203Z","shell.execute_reply.started":"2023-07-13T08:29:07.010770Z"},"trusted":true},"outputs":[],"source":["# The pictures will be resized to have the same size for the neural network\n","img = plt.imread(df.path[0])\n","plt.imshow(img)\n","plt.title(\"Original image\")\n","plt.show()\n","\n","plt.imshow(cv2.resize(img, (150,150)))\n","plt.title(\"After resizing\")\n","plt.show()"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["## Create and train the NN Model"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2023-07-13T08:29:07.887594Z","iopub.status.busy":"2023-07-13T08:29:07.887138Z","iopub.status.idle":"2023-07-13T08:29:07.900278Z","shell.execute_reply":"2023-07-13T08:29:07.898894Z","shell.execute_reply.started":"2023-07-13T08:29:07.887552Z"},"trusted":true},"outputs":[],"source":["def cut_df(df, number_of_parts, part):\n","# Return a part of the dataframe\n","# For example, if a dataframe has 10 rows and we want to return a part of them\n","# if it is cut in two, it will return the first 5 rows or the last 5 rows depending the part wanted\n","\n","# Args:\n","#     df (pandas.DataFrame): The dataframe to cut a part of\n","#     number_of_parts (int): In how many parts should the dataframe be cut\n","#     part (int): The part of the dataframe to return\n","\n","    if part < 1:\n","        print(\"Error, the part should be at least 1\")\n","    elif part > number_of_parts:\n","        print(\"Error, the part cannot be higher than the number_of_parts\")\n","        \n","    number_imgs_each_part = int(df.shape[0]/number_of_parts)\n","    idx1 = (part-1) * number_imgs_each_part\n","    idx2 = part * number_imgs_each_part\n","    return df.iloc[idx1:idx2]\n","\n","def load_img(df):\n","# Load the images using their contained in the dataframe df\n","# Return a list of images and a list with the labels of the images\n","    img_paths = df[\"path\"].values\n","    img_labels = df[\"label\"].values\n","    X = []\n","    y = []\n","    \n","    for i,path in enumerate(img_paths):\n","        img =  plt.imread(path)\n","        img = cv2.resize(img, (150,150))\n","        label = img_labels[i]\n","        X.append(img)\n","        y.append(label)\n","    return np.array(X),np.array(y)"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# CNN model:\n","def create_model():\n","    shape_img = (150,150,3)\n","    \n","    model = Sequential()\n","\n","    model.add(Conv2D(filters=32, kernel_size=(3,3),input_shape=shape_img, activation='relu', padding = 'same'))\n","    model.add(MaxPooling2D(pool_size=(2, 2)))\n","\n","    model.add(Conv2D(filters=64, kernel_size=(3,3),input_shape=shape_img, activation='relu', padding = 'same'))\n","    model.add(MaxPooling2D(pool_size=(2, 2)))\n","\n","    model.add(Conv2D(filters=64, kernel_size=(3,3),input_shape=shape_img, activation='relu', padding = 'same'))\n","    model.add(MaxPooling2D(pool_size=(2, 2)))\n","\n","    model.add(Conv2D(filters=64, kernel_size=(3,3),input_shape=shape_img, activation='relu', padding = 'same'))\n","    model.add(MaxPooling2D(pool_size=(2, 2)))\n","\n","    model.add(Conv2D(filters=64, kernel_size=(3,3),input_shape=shape_img, activation='relu', padding = 'same'))\n","    model.add(MaxPooling2D(pool_size=(2, 2)))\n","\n","    model.add(Conv2D(filters=64, kernel_size=(3,3),input_shape=shape_img, activation='relu', padding = 'same'))\n","    model.add(MaxPooling2D(pool_size=(2, 2)))\n","\n","    model.add(Flatten())\n","\n","    model.add(Dense(256))\n","    model.add(Activation('relu'))\n","    model.add(Dropout(0.5))\n","\n","    model.add(Dense(len(mapper_fruit_names)))\n","    model.add(Activation('softmax'))\n","\n","    model.compile(loss='categorical_crossentropy',optimizer='adam',metrics=['accuracy'])\n","    \n","    return model"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2023-07-13T08:29:08.191903Z","iopub.status.busy":"2023-07-13T08:29:08.191551Z","iopub.status.idle":"2023-07-13T08:29:08.206644Z","shell.execute_reply":"2023-07-13T08:29:08.205303Z","shell.execute_reply.started":"2023-07-13T08:29:08.191873Z"},"trusted":true},"outputs":[],"source":["def from_categorical(lst):\n","    \"\"\"\n","    Inverse of to_categorical\n","    Example: [[0,0,0,1,0], [1,0,0,0,0]] => [3,0]\n","    \"\"\"\n","    \n","    lst = lst.tolist()\n","    lst2 = []\n","    for x in lst:\n","        lst2.append(x.index(max(x)))\n","    return lst2\n","\n","def display_stats(y_test, pred):\n","# Display prediction statistics\n","    print(f\"### Result of the predictions using {len(y_test)} test data ###\\n\")\n","    y_test_class = from_categorical(y_test)\n","    print(\"Classification Report:\\n\")\n","    print(classification_report(y_test_class, pred))\n","    print(\"\\nConfusion Matrix:\\n\\n\")\n","    print(confusion_matrix(y_test_class, pred))\n","    print(\"\\n\")\n","    printmd(f\"# Accuracy: {round(accuracy_score(y_test_class, pred),5)}\")\n","    \n","def plot_training(model):\n","    history = pd.DataFrame(model.history.history)\n","    history[[\"accuracy\",\"val_accuracy\"]].plot()\n","    plt.title(\"Training results\")\n","    plt.xlabel(\"# epoch\")\n","    plt.show()"]},{"cell_type":"code","execution_count":null,"metadata":{"_kg_hide-output":true,"execution":{"iopub.execute_input":"2023-07-13T08:29:08.208486Z","iopub.status.busy":"2023-07-13T08:29:08.208048Z"},"trusted":true},"outputs":[],"source":["model = create_model()\n","hists = []\n","\n","# The model will be trained with one part of the data.\n","# There isn't enough RAM on Kaggle to handle all the data.\n","# In the next chapter a generator will be used\n","# to \"feed\" the ANN step by step.\n","# For Kaggle set divisor <= 5. 1/5 of the data will be used\n","divisor = 5\n","\n","start_time = time.time()\n","X_train, y_train = load_img(cut_df(df,divisor,1))\n","y_train = to_categorical(y_train)\n","\n","# If the ANN doesn't increase its prediction accuracy on the validation data after \n","# 10 epochs, stop the training and take the best of the ANN.\n","callbacks = [EarlyStopping(monitor='val_loss', patience=20),\n","             ModelCheckpoint(filepath='best_model.h5', monitor='val_loss', save_best_only=True)]\n","\n","model.fit(X_train, y_train, batch_size=128, epochs=25, callbacks=callbacks, validation_split = 0.1, verbose = 1)\n","hists.append(model.history.history)\n"]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":true},"outputs":[],"source":["# Run the garbage collector\n","gc.collect()"]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":true},"outputs":[],"source":["time_model = time.time() - start_time\n","print(f\"Time to train the model: {int(time_model)} seconds\")"]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":true},"outputs":[],"source":["acc = []\n","val_acc = []\n","for i in range(len(hists)):\n","    acc += hists[i][\"accuracy\"]\n","    val_acc += hists[i][\"val_accuracy\"]\n","hist_df = pd.DataFrame({\"# Epoch\": [e for e in range(1,len(acc)+1)],\"Accuracy\": acc, \"Val_accuracy\": val_acc})\n","hist_df.plot(x = \"# Epoch\", y = [\"Accuracy\",\"Val_accuracy\"])\n","plt.title(\"Accuracy vs Validation Accuracy\")\n","plt.show()"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["## Predictions<a class=\"anchor\" id=\"3\"></a>"]},{"cell_type":"code","execution_count":null,"metadata":{"_kg_hide-output":false,"trusted":true},"outputs":[],"source":["import warnings\n","warnings.filterwarnings(\"ignore\")\n","\n","# Make predictions with the model using the last 1/20 part of the dataset\n","X, y = load_img(cut_df(df, 20, 20))\n","pred = model.predict(X)\n","pred_classes = np.argmax(pred, axis=1)\n","y_test = to_categorical(y)\n","\n","# Display statistics\n","display_stats(y_test, pred_classes)"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["import tensorflow as tf\n","from tensorflow.keras.models import load_model\n","from tensorflow.keras.utils import get_file\n","\n","model_url = \"https://static-1300131294.cos.ap-shanghai.myqcloud.com/data/deep-learning/nn/best_model_cnn.h5\"\n","model_path = get_file(\"best_model_cnn.h5\", model_url)\n","\n","model = load_model(model_path)"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["import warnings\n","warnings.filterwarnings(\"ignore\")\n","\n","# Make predictions with the model using the last 1/20 part of the dataset\n","X, y = load_img(cut_df(df, 20, 20))\n","pred = model.predict(X)\n","pred_classes = np.argmax(pred, axis=1)\n","y_test = to_categorical(y)\n","\n","# Display statistics\n","display_stats(y_test, pred_classes)"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["## Visualize the result with pictures of fruits"]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":true},"outputs":[],"source":["fig, axes = plt.subplots(nrows=4, ncols=4, figsize=(10, 10),\n","                        subplot_kw={'xticks': [], 'yticks': []})\n","\n","for i, ax in enumerate(axes.flat):\n","    ax.imshow(X[-i])\n","    ax.set_title(f\"True label: {fruit_names[y[-i]]}\\nPredicted label: {fruit_names[int(pred_classes[-i])]}\")\n","\n","plt.tight_layout()\n","plt.show()"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["<!-- <strong>The predictions are very good with around 97% accuracy using only 1/5 of the dataset to train the model.</strong> -->"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["# Acknowledgments\n","\n","Thanks to [ DATALIRA ](https://www.kaggle.com/databeru) for creating the open-source course [ Classify 15 Fruits with TensorFlow ](https://www.kaggle.com/code/databeru/classify-15-fruits-with-tensorflow-acc-99-6). It inspires the majority of the content in this chapter."]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.9.16"}},"nbformat":4,"nbformat_minor":4}