{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Faster R-CNN for Colony Picking" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The Artificial Intelligence (AI) algorithm plays an integral part for our colony picking project. As already elaborated, we chose an AI-based approach due to its high degree of flexibility: we can easily tune the training data in order to achieve desired results. For example every user almost likely wants to pick colonies differently: some want to only pick green ones, some want to pick bigger colonies, some smaller. Hard coding colony picking will require changing the code to each case, however by using AI changing results will merely mean changing training data. This lower the technicality requirement for users to do colony picking" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Tensorflow framework" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For our purpose we used the readily available faster_rcnn_resnet101 model from tensorflow [object detection model zoo]( https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md ), which is trained using the [COCO dataset](http://cocodataset.org/#home ). We chose model trained using COCO dataset, because it provides most basic shape of an object such as straight or curved lines, and for our case most importantly round shapes. We then train the algorithm using our colony pictures at the end to set up the AI to detect colony. The set up is based on [this guide](https://towardsdatascience.com/creating-your-own-object-detector-ad69dda69c85) and it goes as follows:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Collecting and Labelling Training Data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We collected our colony pictures using two methods: first the pictures come from our own lab everytime our team member were doing cultures, and second via [our colony picking competition](https://2019.igem.org/Teams/Collaborations#collaboration_3). The data is labelled using the tool [ImgLab](https://github.com/NaturalIntelligence/imglab), which is hosted on [a server](https://vinca.de/igem/label/). Once enough pictures are collected, which is around 150 pictures in our case (we collected around 300 pictures in total), their resolutions need to be reduced ( to 800x600 in our case) and the pictures need to be labelled. The code is given in the cell below (adapted from [the guide](https://towardsdatascience.com/creating-your-own-object-detector-ad69dda69c85))." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Script for reducing image resolution \n", "# Save as transform_image_resolution.py \n", "# Use by python transform_image_resolution.py -d images/ -s 800 600\n", "from PIL import Image\n", "import os\n", "import argparse\n", "def rescale_images(directory, size):\n", " for img in os.listdir(directory):\n", " im = Image.open(directory+img)\n", " im_resized = im.resize(size, Image.ANTIALIAS)\n", " im_resized.save(directory+img)\n", "if __name__ == '__main__':\n", " parser = argparse.ArgumentParser(description=\"Rescale images\")\n", " parser.add_argument('-d', '--directory', type=str, required=True, help='Directory containing the images')\n", " parser.add_argument('-s', '--size', type=int, nargs=2, required=True, metavar=('width', 'height'), help='Image size')\n", " args = parser.parse_args()\n", " rescale_images(args.directory, args.size)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Preparing the model zoo repository" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we need to prepare the repository of object detection model from Tensorflow. The workflow is adapted from [here](https://gilberttanner.com/blog/installing-the-tensorflow-object-detection-api).
\n", "1. Firstly we need to clone the [repository](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md): \n", "```console\n", "git clone https://github.com/tensorflow/models\n", "```\n", "2. Then we can install dependencies:\n", "```console\n", "pip install --user Cython \n", "pip install --user contextlib2 \n", "pip install --user pillow \n", "pip install --user lxml \n", "pip install --user jupyter \n", "pip install --user matplotlib \n", "```\n", "3. After that we need to prepare .py files that is prepared by google in form of .proto protobufs files. The protos files is located in _models/research_ folder and needs to be put into _models/research/object_\\__detection_ folder. Protobuf first needs to be installed and then the following command can be used: \n", "```console\n", "./bin/protoc object_detection/protos/*.proto --python_out=. \n", "```\n", "4. Subsequently, we need to export some path variables: \n", "```console\n", "export PYTHONPATH=\\\\$PYTHONPATH\\:/TensorFlow/models/research \n", "export PYTHONPATH=\\\\$PYTHONPATH\\:/TensorFlow/models/research/object_detection \n", "export PYTHONPATH=\\\\$PYTHONPATH\\:/TensorFlow/models/research/slim \n", "```\n", "\n", "5. And build the model from within _TensorFlow/models/research/_ \n", "```console\n", "python setup.py build \n", "python setup.py install \n", "```\n", "6. Finally build _slim_ from _TensorFlow/models/research/slim_ with \n", "```console\n", "python setup.py build \n", "python setup.py install \n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Preparing training file" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we need to create a folder for the training and validation images. We will name the _images_ and put it in _models/research/object_\\__detection_. The folder contains images in .jpg format and a corresponding label in .xml format and the folder needs to have the following structure:
\n", "-images
\n", " ─> test
\n", " ─> train
\n", " Then from _models/research/object_\\__detection_ we run the following script (adapted from [this repository](https://github.com/datitran/raccoon_dataset )):" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# xml_to_csv.py\n", "# run using python xml_to_csv.py\n", "# it will result as train_labels.csv and test_labels.csv im /images/ folder\n", "import os\n", "import glob\n", "import pandas as pd\n", "import xml.etree.ElementTree as ET\n", "\n", "\n", "def xml_to_csv(path):\n", " xml_list = []\n", " for xml_file in glob.glob(path + '/*.xml'):\n", " tree = ET.parse(xml_file)\n", " root = tree.getroot()\n", " for member in root.findall('object'):\n", " value = (root.find('filename').text,\n", " int(root.find('size')[0].text),\n", " int(root.find('size')[1].text),\n", " member[0].text,\n", " int(float(member[4][0].text)),\n", " int(float(member[4][1].text)),\n", " int(float(member[4][2].text)),\n", " int(float(member[4][3].text))\n", " )\n", " xml_list.append(value)\n", " column_name = ['filename', 'width', 'height', 'class', 'xmin', 'ymin', 'xmax', 'ymax']\n", " xml_df = pd.DataFrame(xml_list, columns=column_name)\n", " return xml_df\n", "\n", "'''\n", "def main():\n", " image_path = os.path.join(os.getcwd(), 'annotations')\n", " xml_df = xml_to_csv(image_path)\n", " xml_df.to_csv('raccoon_labels.csv', index=None)\n", " print('Successfully converted xml to csv.')\n", "'''\n", "\n", "def main():\n", " for folder in ['train','test']:\n", " image_path = os.path.join(os.getcwd(), ('images/' + folder))\n", " xml_df = xml_to_csv(image_path)\n", " xml_df.to_csv(('images/'+folder+'_labels.csv'), index=None)\n", " print('Successfully converted xml to csv.')\n", "\n", "\n", "main()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Generating tf.record file" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we need to create a tf.record file to package the training data as an input for the algorithm. We do it using the following script (adapted from [this repository](https://github.com/datitran/raccoon_dataset)): " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "\"\"\"\n", "Usage:\n", " # From tensorflow/models/research\n", " # Create train data:\n", " python generate_tfrecord.py --csv_input=./object_detection/images/train_labels.csv --output_path=train.record\n", "\n", " # Create test data:\n", " python generate_tfrecord.py --csv_input=./object_detection/images/test_labels.csv --output_path=test.record\n", " It will result as train.record and test.record in the ./research/ folder \n", "\"\"\"\n", "from __future__ import division\n", "from __future__ import print_function\n", "from __future__ import absolute_import\n", "\n", "import os\n", "import io\n", "import pandas as pd\n", "import tensorflow as tf\n", "\n", "from PIL import Image\n", "from object_detection.utils import dataset_util\n", "from collections import namedtuple, OrderedDict\n", "\n", "flags = tf.app.flags\n", "flags.DEFINE_string('csv_input', '', 'Path to the CSV input')\n", "flags.DEFINE_string('output_path', '', 'Path to output TFRecord')\n", "flags.DEFINE_string('image_dir', '', 'Path to images')\n", "FLAGS = flags.FLAGS\n", "\n", "\n", "# TO-DO replace this with label map\n", "def class_text_to_int(row_label):\n", " if row_label == 'colony':\n", " return 1\n", " else:\n", " None\n", "\n", "\n", "def split(df, group):\n", " data = namedtuple('data', ['filename', 'object'])\n", " gb = df.groupby(group)\n", " return [data(filename, gb.get_group(x)) for filename, x in zip(gb.groups.keys(), gb.groups)]\n", "\n", "\n", "def create_tf_example(group, path):\n", " with tf.gfile.GFile(os.path.join(path, '{}'.format(group.filename)), 'rb') as fid:\n", " encoded_jpg = fid.read()\n", " encoded_jpg_io = io.BytesIO(encoded_jpg)\n", " image = Image.open(encoded_jpg_io)\n", " width, height = image.size\n", "\n", " filename = group.filename.encode('utf8')\n", " image_format = b'jpg'\n", " xmins = []\n", " xmaxs = []\n", " ymins = []\n", " ymaxs = []\n", " classes_text = []\n", " classes = []\n", "\n", " for index, row in group.object.iterrows():\n", " xmins.append(row['xmin'] / width)\n", " xmaxs.append(row['xmax'] / width)\n", " ymins.append(row['ymin'] / height)\n", " ymaxs.append(row['ymax'] / height)\n", " classes_text.append(row['class'].encode('utf8'))\n", " classes.append(class_text_to_int(row['class']))\n", "\n", " tf_example = tf.train.Example(features=tf.train.Features(feature={\n", " 'image/height': dataset_util.int64_feature(height),\n", " 'image/width': dataset_util.int64_feature(width),\n", " 'image/filename': dataset_util.bytes_feature(filename),\n", " 'image/source_id': dataset_util.bytes_feature(filename),\n", " 'image/encoded': dataset_util.bytes_feature(encoded_jpg),\n", " 'image/format': dataset_util.bytes_feature(image_format),\n", " 'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),\n", " 'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),\n", " 'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),\n", " 'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),\n", " 'image/object/class/text': dataset_util.bytes_list_feature(classes_text),\n", " 'image/object/class/label': dataset_util.int64_list_feature(classes),\n", " }))\n", " return tf_example\n", "\n", "\n", "def main(_):\n", " writer = tf.python_io.TFRecordWriter(FLAGS.output_path)\n", " path = os.path.join(FLAGS.image_dir)\n", " examples = pd.read_csv(FLAGS.csv_input)\n", " grouped = split(examples, 'filename')\n", " for group in grouped:\n", " tf_example = create_tf_example(group, path)\n", " writer.write(tf_example.SerializeToString())\n", "\n", " writer.close()\n", " output_path = os.path.join(os.getcwd(), FLAGS.output_path)\n", " print('Successfully created the TFRecords: {}'.format(output_path))\n", "\n", "\n", "if __name__ == '__main__':\n", " tf.app.run()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Configuring training" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To configure training create a label map file and a config file and put it into the _./object_\\__detection/training_ folder. The files are given below: " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#save as label_map.pbtxt in training folder\n", "item {\n", " id: 1\n", " name: 'colony'\n", "}" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#save as faster_rcnn_resnet101_coco.config in training folder\n", "# Faster R-CNN with Resnet-101 (v1)\n", "# Trained on KITTI dataset (cars and pedestrian), initialized from COCO\n", "# detection checkpoint.\n", "# Users should configure the fine_tune_checkpoint field in the train config as\n", "# well as the label_map_path and input_path fields in the train_input_reader and\n", "# eval_input_reader. Search for \"PATH_TO_BE_CONFIGURED\" to find the fields that\n", "# should be configured.\n", "\n", "model {\n", " faster_rcnn {\n", " num_classes: 1\n", " image_resizer {\n", " keep_aspect_ratio_resizer {\n", " # Raw KITTI images have a resolution of 1242x375, if we wish to resize\n", " # them to have a height of 600 then their width should be\n", " # 1242/(375/600)=1987.2\n", " min_dimension: 600\n", " max_dimension: 1987\n", " }\n", " }\n", " feature_extractor {\n", " type: 'faster_rcnn_resnet101'\n", " first_stage_features_stride: 16\n", " }\n", " first_stage_anchor_generator {\n", " grid_anchor_generator {\n", " scales: [0.25, 0.5, 1.0, 2.0]\n", " aspect_ratios: [0.5, 1.0, 2.0]\n", " height_stride: 16\n", " width_stride: 16\n", " }\n", " }\n", " first_stage_box_predictor_conv_hyperparams {\n", " op: CONV\n", " regularizer {\n", " l2_regularizer {\n", " weight: 0.0\n", " }\n", " }\n", " initializer {\n", " truncated_normal_initializer {\n", " stddev: 0.01\n", " }\n", " }\n", " }\n", " first_stage_nms_score_threshold: 0.0\n", " first_stage_nms_iou_threshold: 0.7\n", " first_stage_max_proposals: 300\n", " first_stage_localization_loss_weight: 2.0\n", " first_stage_objectness_loss_weight: 1.0\n", " initial_crop_size: 14\n", " maxpool_kernel_size: 2\n", " maxpool_stride: 2\n", " second_stage_box_predictor {\n", " mask_rcnn_box_predictor {\n", " use_dropout: false\n", " dropout_keep_probability: 1.0\n", " fc_hyperparams {\n", " op: FC\n", " regularizer {\n", " l2_regularizer {\n", " weight: 0.0\n", " }\n", " }\n", " initializer {\n", " variance_scaling_initializer {\n", " factor: 1.0\n", " uniform: true\n", " mode: FAN_AVG\n", " }\n", " }\n", " }\n", " }\n", " }\n", " second_stage_post_processing {\n", " batch_non_max_suppression {\n", " score_threshold: 0.0\n", " iou_threshold: 0.6\n", " max_detections_per_class: 100\n", " max_total_detections: 300\n", " }\n", " score_converter: SOFTMAX\n", " }\n", " second_stage_localization_loss_weight: 2.0\n", " second_stage_classification_loss_weight: 1.0\n", " }\n", "}\n", "\n", "train_config: {\n", " batch_size: 1\n", " optimizer {\n", " momentum_optimizer: {\n", " learning_rate: {\n", " manual_step_learning_rate {\n", " initial_learning_rate: 0.0001\n", " schedule {\n", " step: 500000\n", " learning_rate: .00001\n", " }\n", " schedule {\n", " step: 700000\n", " learning_rate: .000001\n", " }\n", " }\n", " }\n", " momentum_optimizer_value: 0.9\n", " }\n", " use_moving_average: false\n", " }\n", " gradient_clipping_by_norm: 10.0\n", " fine_tune_checkpoint: \"/Users/wihan/tensorflow/models/research/object_detection/faster_rcnn_resnet101_kitti_2018_01_28/model.ckpt\"\n", " from_detection_checkpoint: true\n", " num_steps: 800000\n", " data_augmentation_options {\n", " random_horizontal_flip {\n", " }\n", " }\n", "}\n", "\n", "train_input_reader: {\n", " label_map_path: \"/Users/wihan/tensorflow/models/research/object_detection/training/kitti_label_map.pbtxt\"\n", " tf_record_input_reader: {\n", " input_path: \"/Users/wihan/tensorflow/models/research/kitti_train.record\"\n", " }\n", "}\n", "\n", "eval_config: {\n", " use_moving_averages: false\n", " num_examples: 5\n", "}\n", "\n", "eval_input_reader: {\n", " label_map_path: \"/Users/wihan/tensorflow/models/research/object_detection/training/kitti_label_map.pbtxt\"\n", " tf_record_input_reader: {\n", " input_path: \"/Users/wihan/tensorflow/models/research/kitti_val.record\"\n", " }\n", "}\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 6. Start training" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, the training can be start by going to _/models/research/_ folder and start the script model_main.py with command: \n", "```console\n", "python model_main.py --logtostderr --train_dir=training/ --pipeline_config_path=training/faster_rcnn_inception_v2_pets.config \n", "Training status can be check using Tensorboard with the command \n", "tensorboard --logdir=training \n", "```\n", "Stop the training once the loss is small enough" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#model_main.py \n", "# start at /models/research/ folder \n", "# Copyright 2017 The TensorFlow Authors. All Rights Reserved.\n", "#\n", "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# http://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License.\n", "# ==============================================================================\n", "\"\"\"Binary to run train and evaluation on object detection model.\"\"\"\n", "\n", "from __future__ import absolute_import\n", "from __future__ import division\n", "from __future__ import print_function\n", "\n", "from absl import flags\n", "\n", "import tensorflow as tf\n", "\n", "from object_detection import model_hparams\n", "from object_detection import model_lib\n", "\n", "flags.DEFINE_string(\n", " 'model_dir', None, 'Path to output model directory '\n", " 'where event and checkpoint files will be written.')\n", "flags.DEFINE_string('pipeline_config_path', None, 'Path to pipeline config '\n", " 'file.')\n", "flags.DEFINE_integer('num_train_steps', None, 'Number of train steps.')\n", "flags.DEFINE_boolean('eval_training_data', False,\n", " 'If training data should be evaluated for this job. Note '\n", " 'that one call only use this in eval-only mode, and '\n", " '`checkpoint_dir` must be supplied.')\n", "flags.DEFINE_integer('sample_1_of_n_eval_examples', 1, 'Will sample one of '\n", " 'every n eval input examples, where n is provided.')\n", "flags.DEFINE_integer('sample_1_of_n_eval_on_train_examples', 5, 'Will sample '\n", " 'one of every n train input examples for evaluation, '\n", " 'where n is provided. This is only used if '\n", " '`eval_training_data` is True.')\n", "flags.DEFINE_string(\n", " 'hparams_overrides', None, 'Hyperparameter overrides, '\n", " 'represented as a string containing comma-separated '\n", " 'hparam_name=value pairs.')\n", "flags.DEFINE_string(\n", " 'checkpoint_dir', None, 'Path to directory holding a checkpoint. If '\n", " '`checkpoint_dir` is provided, this binary operates in eval-only mode, '\n", " 'writing resulting metrics to `model_dir`.')\n", "flags.DEFINE_boolean(\n", " 'run_once', False, 'If running in eval-only mode, whether to run just '\n", " 'one round of eval vs running continuously (default).'\n", ")\n", "FLAGS = flags.FLAGS\n", "\n", "\n", "def main(unused_argv):\n", " flags.mark_flag_as_required('model_dir')\n", " flags.mark_flag_as_required('pipeline_config_path')\n", " config = tf.estimator.RunConfig(model_dir=FLAGS.model_dir)\n", "\n", " train_and_eval_dict = model_lib.create_estimator_and_inputs(\n", " run_config=config,\n", " hparams=model_hparams.create_hparams(FLAGS.hparams_overrides),\n", " pipeline_config_path=FLAGS.pipeline_config_path,\n", " train_steps=FLAGS.num_train_steps,\n", " sample_1_of_n_eval_examples=FLAGS.sample_1_of_n_eval_examples,\n", " sample_1_of_n_eval_on_train_examples=(\n", " FLAGS.sample_1_of_n_eval_on_train_examples))\n", " estimator = train_and_eval_dict['estimator']\n", " train_input_fn = train_and_eval_dict['train_input_fn']\n", " eval_input_fns = train_and_eval_dict['eval_input_fns']\n", " eval_on_train_input_fn = train_and_eval_dict['eval_on_train_input_fn']\n", " predict_input_fn = train_and_eval_dict['predict_input_fn']\n", " train_steps = train_and_eval_dict['train_steps']\n", "\n", " if FLAGS.checkpoint_dir:\n", " if FLAGS.eval_training_data:\n", " name = 'training_data'\n", " input_fn = eval_on_train_input_fn\n", " else:\n", " name = 'validation_data'\n", " # The first eval input will be evaluated.\n", " input_fn = eval_input_fns[0]\n", " if FLAGS.run_once:\n", " estimator.evaluate(input_fn,\n", " steps=None,\n", " checkpoint_path=tf.train.latest_checkpoint(\n", " FLAGS.checkpoint_dir))\n", " else:\n", " model_lib.continuous_eval(estimator, FLAGS.checkpoint_dir, input_fn,\n", " train_steps, name)\n", " else:\n", " train_spec, eval_specs = model_lib.create_train_and_eval_specs(\n", " train_input_fn,\n", " eval_input_fns,\n", " eval_on_train_input_fn,\n", " predict_input_fn,\n", " train_steps,\n", " eval_on_train_data=False)\n", "\n", " # Currently only a single Eval Spec is allowed.\n", " tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])\n", "\n", "\n", "if __name__ == '__main__':\n", " tf.app.run()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 7. Creating frozen inference image" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally after the training is deemed sufficient, we can import the frozen model for usage using the following command and the export_inference_graph.py script: \n", "```console\n", "python export_inference_graph.py --input_type image_tensor --pipeline_config_path training/faster_rcnn_inception_v2_pets.config --trained_checkpoint_prefix training/model.ckpt-XXXX --output_directory inference_graph \n", "```\n", "Where XXXX is the number of training step. Once made, the frozen model can be loaded and used " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Copyright 2017 The TensorFlow Authors. All Rights Reserved.\n", "#\n", "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# http://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License.\n", "# ==============================================================================\n", "\n", "r\"\"\"Tool to export an object detection model for inference.\n", "\n", "Prepares an object detection tensorflow graph for inference using model\n", "configuration and a trained checkpoint. Outputs inference\n", "graph, associated checkpoint files, a frozen inference graph and a\n", "SavedModel (https://tensorflow.github.io/serving/serving_basic.html).\n", "\n", "The inference graph contains one of three input nodes depending on the user\n", "specified option.\n", " * `image_tensor`: Accepts a uint8 4-D tensor of shape [None, None, None, 3]\n", " * `encoded_image_string_tensor`: Accepts a 1-D string tensor of shape [None]\n", " containing encoded PNG or JPEG images. Image resolutions are expected to be\n", " the same if more than 1 image is provided.\n", " * `tf_example`: Accepts a 1-D string tensor of shape [None] containing\n", " serialized TFExample protos. Image resolutions are expected to be the same\n", " if more than 1 image is provided.\n", "\n", "and the following output nodes returned by the model.postprocess(..):\n", " * `num_detections`: Outputs float32 tensors of the form [batch]\n", " that specifies the number of valid boxes per image in the batch.\n", " * `detection_boxes`: Outputs float32 tensors of the form\n", " [batch, num_boxes, 4] containing detected boxes.\n", " * `detection_scores`: Outputs float32 tensors of the form\n", " [batch, num_boxes] containing class scores for the detections.\n", " * `detection_classes`: Outputs float32 tensors of the form\n", " [batch, num_boxes] containing classes for the detections.\n", " * `raw_detection_boxes`: Outputs float32 tensors of the form\n", " [batch, raw_num_boxes, 4] containing detection boxes without\n", " post-processing.\n", " * `raw_detection_scores`: Outputs float32 tensors of the form\n", " [batch, raw_num_boxes, num_classes_with_background] containing class score\n", " logits for raw detection boxes.\n", " * `detection_masks`: (Optional) Outputs float32 tensors of the form\n", " [batch, num_boxes, mask_height, mask_width] containing predicted instance\n", " masks for each box if its present in the dictionary of postprocessed\n", " tensors returned by the model.\n", " * detection_multiclass_scores: (Optional) Outputs float32 tensor of shape\n", " [batch, num_boxes, num_classes_with_background] for containing class\n", " score distribution for detected boxes including background if any.\n", " * detection_features: (Optional) float32 tensor of shape\n", " [batch, num_boxes, roi_height, roi_width, depth]\n", " containing classifier features\n", "\n", "Notes:\n", " * This tool uses `use_moving_averages` from eval_config to decide which\n", " weights to freeze.\n", "\n", "Example Usage:\n", "--------------\n", "python export_inference_graph \\\n", " --input_type image_tensor \\\n", " --pipeline_config_path path/to/ssd_inception_v2.config \\\n", " --trained_checkpoint_prefix path/to/model.ckpt \\\n", " --output_directory path/to/exported_model_directory\n", "\n", "The expected output would be in the directory\n", "path/to/exported_model_directory (which is created if it does not exist)\n", "with contents:\n", " - inference_graph.pbtxt\n", " - model.ckpt.data-00000-of-00001\n", " - model.ckpt.info\n", " - model.ckpt.meta\n", " - frozen_inference_graph.pb\n", " + saved_model (a directory)\n", "\n", "Config overrides (see the `config_override` flag) are text protobufs\n", "(also of type pipeline_pb2.TrainEvalPipelineConfig) which are used to override\n", "certain fields in the provided pipeline_config_path. These are useful for\n", "making small changes to the inference graph that differ from the training or\n", "eval config.\n", "\n", "Example Usage (in which we change the second stage post-processing score\n", "threshold to be 0.5):\n", "\n", "python export_inference_graph \\\n", " --input_type image_tensor \\\n", " --pipeline_config_path path/to/ssd_inception_v2.config \\\n", " --trained_checkpoint_prefix path/to/model.ckpt \\\n", " --output_directory path/to/exported_model_directory \\\n", " --config_override \" \\\n", " model{ \\\n", " faster_rcnn { \\\n", " second_stage_post_processing { \\\n", " batch_non_max_suppression { \\\n", " score_threshold: 0.5 \\\n", " } \\\n", " } \\\n", " } \\\n", " }\"\n", "\"\"\"\n", "import tensorflow as tf\n", "from google.protobuf import text_format\n", "from object_detection import exporter\n", "from object_detection.protos import pipeline_pb2\n", "\n", "slim = tf.contrib.slim\n", "flags = tf.app.flags\n", "\n", "flags.DEFINE_string('input_type', 'image_tensor', 'Type of input node. Can be '\n", " 'one of [`image_tensor`, `encoded_image_string_tensor`, '\n", " '`tf_example`]')\n", "flags.DEFINE_string('input_shape', None,\n", " 'If input_type is `image_tensor`, this can explicitly set '\n", " 'the shape of this input tensor to a fixed size. The '\n", " 'dimensions are to be provided as a comma-separated list '\n", " 'of integers. A value of -1 can be used for unknown '\n", " 'dimensions. If not specified, for an `image_tensor, the '\n", " 'default shape will be partially specified as '\n", " '`[None, None, None, 3]`.')\n", "flags.DEFINE_string('pipeline_config_path', None,\n", " 'Path to a pipeline_pb2.TrainEvalPipelineConfig config '\n", " 'file.')\n", "flags.DEFINE_string('trained_checkpoint_prefix', None,\n", " 'Path to trained checkpoint, typically of the form '\n", " 'path/to/model.ckpt')\n", "flags.DEFINE_string('output_directory', None, 'Path to write outputs.')\n", "flags.DEFINE_string('config_override', '',\n", " 'pipeline_pb2.TrainEvalPipelineConfig '\n", " 'text proto to override pipeline_config_path.')\n", "flags.DEFINE_boolean('write_inference_graph', False,\n", " 'If true, writes inference graph to disk.')\n", "tf.app.flags.mark_flag_as_required('pipeline_config_path')\n", "tf.app.flags.mark_flag_as_required('trained_checkpoint_prefix')\n", "tf.app.flags.mark_flag_as_required('output_directory')\n", "FLAGS = flags.FLAGS\n", "\n", "\n", "def main(_):\n", " pipeline_config = pipeline_pb2.TrainEvalPipelineConfig()\n", " with tf.gfile.GFile(FLAGS.pipeline_config_path, 'r') as f:\n", " text_format.Merge(f.read(), pipeline_config)\n", " text_format.Merge(FLAGS.config_override, pipeline_config)\n", " if FLAGS.input_shape:\n", " input_shape = [\n", " int(dim) if dim != '-1' else None\n", " for dim in FLAGS.input_shape.split(',')\n", " ]\n", " else:\n", " input_shape = None\n", " exporter.export_inference_graph(\n", " FLAGS.input_type, pipeline_config, FLAGS.trained_checkpoint_prefix,\n", " FLAGS.output_directory, input_shape=input_shape,\n", " write_inference_graph=FLAGS.write_inference_graph)\n", "\n", "\n", "if __name__ == '__main__':\n", " tf.app.run()\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.16" } }, "nbformat": 4, "nbformat_minor": 2 }