{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Packaging the AI with Docker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Docker is a software that is used to create containers. These containers are independent units, usually with an installed software that can be run inside the container. Thus it is akin to a virtual maschine, however docker goes one step further and is more independent than a virtual maschine as it creates a standardized brige between an operating system and the software installed in the container. As benefits, the software contained in a container can be run as long as the docker software is installed in the operating system. This frees up users from the complexities and nightmares usually associated with installing dependencies, and compabilities of different operating systems, settings, and many more varieties within each user's computer. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.  Making a Docker Image for Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we need to clone the github repository containing the model using: \n",
    "```console\n",
    "git clone https://github.com/igemsoftware2019/iGemMarburg2019.git \n",
    "cd iGemMarburg2019/AI <br>\n",
    "```\n",
    "Now we build the docker image using the included process.dockerfile below. The purpose is to build all the required dependencies and run the train.sh script, which does all the steps elaborated in the AI documentation file. The default training steps is 100 and can be changed by changing the NUM\\_STEPS parameter in the train.sh file. Images must be labeled (the label needs to be colony), have corresponding .xml files, and be put in their corresponding folders (either in &lt;path to test images&gt; or &lt;path to train images&gt;). Subsequently, the following command needs to be executed: \n",
    "```console\n",
    "docker build -f training.dockerfile -t <image tag> . \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# training.dockerfile\n",
    "# Copyright 2019, iGEM Marburg 2019\n",
    "# This program is free software: you can redistribute it and/or modify\n",
    "# it under the terms of the GNU General Public License as published by\n",
    "# the Free Software Foundation, either version 3 of the License, or\n",
    "# (at your option) any later version. This program is distributed in the hope that it will be useful,\n",
    "# but WITHOUT ANY WARRANTY; without even the implied warranty of\n",
    "# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n",
    "# GNU General Public License for more details.\n",
    "# You should have received a copy of the GNU General Public License\n",
    "# along with this program.  If not, see <https://www.gnu.org/licenses/>.\n",
    "\n",
    "FROM tensorflow/tensorflow:1.14.0-py3\n",
    "\n",
    "RUN pip install --user Cython contextlib2 pillow lxml matplotlib pandas && pip install --user pycocotools\n",
    "\n",
    "COPY models /tf/models\n",
    "COPY train.sh /train.sh\n",
    "RUN chmod +x /train.sh\n",
    "\n",
    "RUN export PYTHONPATH=$PYTHONPATH:/tf/models/research:/tf/models/research/object_detection:/tf/models/research/slim\n",
    "RUN cd /tf/models/research && python setup.py build && python setup.py install\n",
    "RUN cd /tf/models/research/slim && python setup.py build && python setup.py install\n",
    "\n",
    "ENTRYPOINT /train.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#train.sh\n",
    "#!/usr/bin/env bash\n",
    "\n",
    "# Copyright 2019, iGEM Marburg 2019\n",
    "# This program is free software: you can redistribute it and/or modify\n",
    "# it under the terms of the GNU General Public License as published by\n",
    "# the Free Software Foundation, either version 3 of the License, or\n",
    "# (at your option) any later version. This program is distributed in the hope that it will be useful,\n",
    "# but WITHOUT ANY WARRANTY; without even the implied warranty of\n",
    "# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n",
    "# GNU General Public License for more details.\n",
    "# You should have received a copy of the GNU General Public License\n",
    "# along with this program.  If not, see <https://www.gnu.org/licenses/>.\n",
    "\n",
    "mkdir /tf/trained_model\n",
    "\n",
    "cd /tf/models/research/object_detection && python xml_to_csv.py\n",
    "\n",
    "cd /tf/models/research && python generate_tfrecord.py \\\n",
    "  --csv_input=object_detection/images/train_labels.csv \\\n",
    "  --image_dir=object_detection/images/train \\\n",
    "  --output_path=mscoco_train.record\n",
    "cd /tf/models/research && python generate_tfrecord.py \\\n",
    "  --csv_input=object_detection/images/test_labels.csv \\\n",
    "  --image_dir=object_detection/images/test \\\n",
    "  --output_path=mscoco_val.record\n",
    "\n",
    "NUM_STEPS=${NUM_STEPS:-100}\n",
    "\n",
    "cd /tf/models/research\n",
    "\n",
    "python model_main.py \\\n",
    "  --logtostderr \\\n",
    "  --model_dir=/tf/trained_model \\\n",
    "  --num_train_steps=${NUM_STEPS} \\\n",
    "  --train_dir=object_detection/training/ \\\n",
    "  --pipeline_config_path=object_detection/training/faster_rcnn_resnet101_coco.config\n",
    "\n",
    "suffix=$(ls /tf/trained_model | grep \".index\" | tail -1 | cut -d '.' -f 2 | cut -d '-' -f 2)\n",
    "\n",
    "cd /tf/models/research/object_detection\n",
    "python export_inference_graph.py \\\n",
    "  --input_type=image_tensor \\\n",
    "  --pipeline_config_path=training/faster_rcnn_resnet101_coco.config \\\n",
    "  --trained_checkpoint_prefix=/tf/trained_model/model.ckpt-${suffix} \\\n",
    "  --output_directory=inference_graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Running a Docker Container"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Subsequently the image can be run as a container using the following command. \n",
    "```console\n",
    "docker run \\ \n",
    "  --rm \\ \n",
    "  -v <path to train images>:/tf/models/research/object_detection/images/train \\ \n",
    "  -v <path to test images>:/tf/models/research/object_detection/images/test \\ \n",
    "  -v <output path of inference graph>:/tf/models/research/object_detection/inference_graph \\ \n",
    "  <image tag> \n",
    "```\n",
    "Once running the model will be trained on your local computer. The inference image will be put in &lt;output path of inference graph&gt;\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Processing a Colony Image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once trained, a frozen inference image is then created in the corresponding folder. To use this with an arbritary colony image we need to put the image in &lt;path of test images folder&gt; and create a docker image as before using process.dockerfile with the command below. The command build all the dependencies and run process.py file which takes the frozen inference file and use it on the arbritary colony image.  \n",
    "```console\n",
    "docker build -f process.dockerfile -t <image tag> .\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# process.dockerfile\n",
    "# Copyright 2019, iGEM Marburg 2019\n",
    "# This program is free software: you can redistribute it and/or modify\n",
    "# it under the terms of the GNU General Public License as published by\n",
    "# the Free Software Foundation, either version 3 of the License, or\n",
    "# (at your option) any later version. This program is distributed in the hope that it will be useful,\n",
    "# but WITHOUT ANY WARRANTY; without even the implied warranty of\n",
    "# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n",
    "# GNU General Public License for more details.\n",
    "# You should have received a copy of the GNU General Public License\n",
    "# along with this program.  If not, see <https://www.gnu.org/licenses/>.\n",
    "\n",
    "FROM tensorflow/tensorflow:1.14.0-py3\n",
    "\n",
    "RUN pip install --user Cython contextlib2 pillow lxml matplotlib pandas && pip install --user pycocotools\n",
    "RUN mkdir /processed\n",
    "\n",
    "COPY models /tf/models\n",
    "COPY process.py /tf/models/research/object_detection/process.py\n",
    "\n",
    "RUN export PYTHONPATH=$PYTHONPATH:/tf/models/research:/tf/models/research/object_detection:/tf/models/research/slim\n",
    "RUN cd /tf/models/research && python setup.py build && python setup.py install\n",
    "RUN cd /tf/models/research/slim && python setup.py build && python setup.py install\n",
    "\n",
    "WORKDIR /tf/models/research/object_detection\n",
    "ENTRYPOINT [\"python\", \"process.py\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# process.py file\n",
    "import numpy as np\n",
    "import os\n",
    "import six.moves.urllib as urllib\n",
    "import sys\n",
    "import tarfile\n",
    "import tensorflow.compat.v1 as tf\n",
    "import zipfile\n",
    "\n",
    "from distutils.version import StrictVersion\n",
    "from collections import defaultdict\n",
    "from io import StringIO\n",
    "from matplotlib import pyplot as plt\n",
    "from PIL import Image\n",
    "\n",
    "tf.disable_v2_behavior()\n",
    "\n",
    "# This is needed since the notebook is stored in the object_detection folder.\n",
    "sys.path.append(\"..\")\n",
    "from object_detection.utils import ops as utils_ops\n",
    "\n",
    "if StrictVersion(tf.__version__) < StrictVersion('1.12.0'):\n",
    "  raise ImportError('Please upgrade your TensorFlow installation to v1.12.*.')\n",
    "\n",
    "from utils import label_map_util\n",
    "\n",
    "from utils import visualization_utils as vis_util\n",
    "\n",
    "# What model to download.\n",
    "MODEL_NAME = 'inference_graph'\n",
    "#MODEL_FILE = MODEL_NAME + '.tar.gz'\n",
    "#DOWNLOAD_BASE = 'http://download.tensorflow.org/models/object_detection/'\n",
    "\n",
    "# Path to frozen detection graph. This is the actual model that is used for the object detection.\n",
    "PATH_TO_FROZEN_GRAPH = MODEL_NAME + '/frozen_inference_graph.pb'\n",
    "\n",
    "# List of the strings that is used to add correct label for each box.\n",
    "#PATH_TO_LABELS = os.path.join('data', 'mscoco_label_map.pbtxt')\n",
    "PATH_TO_LABELS = 'training/mscoco_label_map.pbtxt'\n",
    "\n",
    "detection_graph = tf.Graph()\n",
    "with detection_graph.as_default():\n",
    "  od_graph_def = tf.GraphDef()\n",
    "  with tf.io.gfile.GFile(PATH_TO_FROZEN_GRAPH, 'rb') as fid:\n",
    "    serialized_graph = fid.read()\n",
    "    od_graph_def.ParseFromString(serialized_graph)\n",
    "    tf.import_graph_def(od_graph_def, name='')\n",
    "\n",
    "category_index = label_map_util.create_category_index_from_labelmap(\n",
    "    PATH_TO_LABELS, use_display_name=True)\n",
    "\n",
    "def load_image_into_numpy_array(image):\n",
    "  (im_width, im_height) = image.size\n",
    "  return np.array(image.getdata()).reshape(\n",
    "      (im_height, im_width, 3)).astype(np.uint8)\n",
    "\n",
    "\n",
    "def absoluteFilePaths(directory):\n",
    "   for dirpath, _, filenames in os.walk(directory):\n",
    "       for f in filenames:\n",
    "           yield os.path.abspath(os.path.join(dirpath, f))\n",
    "\n",
    "PATH_TO_TEST_IMAGES_DIR = 'test_images'\n",
    "TEST_IMAGE_PATHS = absoluteFilePaths(PATH_TO_TEST_IMAGES_DIR)\n",
    "\n",
    "# Size, in inches, of the output images.\n",
    "IMAGE_SIZE = (12, 8)\n",
    "\n",
    "def run_inference_for_single_image(image, graph):\n",
    "  with graph.as_default():\n",
    "    with tf.Session() as sess:\n",
    "      # Get handles to input and output tensors\n",
    "      ops = tf.get_default_graph().get_operations()\n",
    "      all_tensor_names = {output.name for op in ops for output in op.outputs}\n",
    "      tensor_dict = {}\n",
    "      for key in [\n",
    "          'num_detections', 'detection_boxes', 'detection_scores',\n",
    "          'detection_classes', 'detection_masks'\n",
    "      ]:\n",
    "        tensor_name = key + ':0'\n",
    "        if tensor_name in all_tensor_names:\n",
    "          tensor_dict[key] = tf.get_default_graph().get_tensor_by_name(\n",
    "              tensor_name)\n",
    "      if 'detection_masks' in tensor_dict:\n",
    "        # The following processing is only for single image\n",
    "        detection_boxes = tf.squeeze(tensor_dict['detection_boxes'], [0])\n",
    "        detection_masks = tf.squeeze(tensor_dict['detection_masks'], [0])\n",
    "        # Reframe is required to translate mask from box coordinates to image coordinates and fit the image size.\n",
    "        real_num_detection = tf.cast(\n",
    "            tensor_dict['num_detections'][0], tf.int32)\n",
    "        detection_boxes = tf.slice(detection_boxes, [0, 0], [\n",
    "                                   real_num_detection, -1])\n",
    "        detection_masks = tf.slice(detection_masks, [0, 0, 0], [\n",
    "                                   real_num_detection, -1, -1])\n",
    "        detection_masks_reframed = utils_ops.reframe_box_masks_to_image_masks(\n",
    "            detection_masks, detection_boxes, image.shape[1], image.shape[2])\n",
    "        detection_masks_reframed = tf.cast(\n",
    "            tf.greater(detection_masks_reframed, 0.5), tf.uint8)\n",
    "        # Follow the convention by adding back the batch dimension\n",
    "        tensor_dict['detection_masks'] = tf.expand_dims(\n",
    "            detection_masks_reframed, 0)\n",
    "      image_tensor = tf.get_default_graph().get_tensor_by_name('image_tensor:0')\n",
    "\n",
    "      # Run inference\n",
    "      output_dict = sess.run(tensor_dict,\n",
    "                             feed_dict={image_tensor: image})\n",
    "\n",
    "      # all outputs are float32 numpy arrays, so convert types as appropriate\n",
    "      output_dict['num_detections'] = int(output_dict['num_detections'][0])\n",
    "      output_dict['detection_classes'] = output_dict[\n",
    "          'detection_classes'][0].astype(np.int64)\n",
    "      output_dict['detection_boxes'] = output_dict['detection_boxes'][0]\n",
    "      output_dict['detection_scores'] = output_dict['detection_scores'][0]\n",
    "      if 'detection_masks' in output_dict:\n",
    "        output_dict['detection_masks'] = output_dict['detection_masks'][0]\n",
    "  return output_dict\n",
    "\n",
    "\n",
    "c = 0\n",
    "for image_path in TEST_IMAGE_PATHS:\n",
    "  image = Image.open(image_path)\n",
    "  # the array based representation of the image will be used later in order to prepare the\n",
    "  # result image with boxes and labels on it.\n",
    "  image_np = load_image_into_numpy_array(image)\n",
    "  # Expand dimensions since the model expects images to have shape: [1, None, None, 3]\n",
    "  image_np_expanded = np.expand_dims(image_np, axis=0)\n",
    "  # Actual detection.\n",
    "  output_dict = run_inference_for_single_image(\n",
    "      image_np_expanded, detection_graph)\n",
    "  # Visualization of the results of a detection.\n",
    "  vis_util.visualize_boxes_and_labels_on_image_array(\n",
    "      image_np,\n",
    "      output_dict['detection_boxes'],\n",
    "      output_dict['detection_classes'],\n",
    "      output_dict['detection_scores'],\n",
    "      category_index,\n",
    "      instance_masks=output_dict.get('detection_masks'),\n",
    "      use_normalized_coordinates=True,\n",
    "      line_thickness=4)\n",
    "  plt.figure(figsize=IMAGE_SIZE)\n",
    "  plt.imshow(image_np)\n",
    "  plt.savefig('/processed/{0}.jpg'.format(c))\n",
    "  np.savetxt('/processed/boxes{0}.csv'.format(c),\n",
    "                output_dict['detection_boxes'], delimiter=\",\")\n",
    "  np.savetxt('/processed/scores{0}.csv'.format(c),\n",
    "             output_dict['detection_scores'], delimiter=\",\")\n",
    "  c += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally the docker image can be run with the command: \n",
    "```console\n",
    "docker run \\\n",
    "  --rm \\\n",
    "  -v <path of inference graph>:/tf/models/research/object_detection/inference_graph \\\n",
    "  -v <path of test images folder>:/tf/models/research/object_detection/test_images \\\n",
    "  -v <output path>:/processed \\\n",
    "  <image tag>\n",
    "```\n",
    "The processed image will be in the folder &lt;output path&gt;. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}