{ "cells": [ { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "6bYaCABobL5q" }, "source": [ "##### Copyright 2018 The TensorFlow Authors." ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "cellView": "form", "colab": {}, "colab_type": "code", "id": "FlUw7tSKbtg4" }, "outputs": [], "source": [ "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "xc1srSc51n_4" }, "source": [ "# Using the SavedModel format" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "-nBUqG2rchGH" }, "source": [ "\n", " \n", " \n", " \n", " \n", "
\n", " View on TensorFlow.org\n", " \n", " Run in Google Colab\n", " \n", " View source on GitHub\n", " \n", " Download notebook\n", "
" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "CPE-fshLTsXU" }, "source": [ "A SavedModel contains a complete TensorFlow program, including weights and computation. It does not require the original model building code to run, which makes it useful for sharing or deploying (with [TFLite](https://tensorflow.org/lite), [TensorFlow.js](https://js.tensorflow.org/), [TensorFlow Serving](https://www.tensorflow.org/tfx/serving/tutorials/Serving_REST_simple), or [TensorFlow Hub](https://tensorflow.org/hub)).\n", "\n", "This document dives into some of the details of how to use the low-level `tf.saved_model` api:\n", "\n", "- If you are using a `tf.keras.Model` the `keras.Model.save(output_path)` method may be all you need: See the [Keras save and serialize](keras/save_and_serialize.ipynb)\n", "\n", "- If you just want to save/load weights during training see the [guide to training checkpoints](./checkpoint.ipynb).\n" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "9SuIC7FiI9g8" }, "source": [ "## Creating a SavedModel from Keras\n", "\n", "For a quick introduction, this section exports a pre-trained Keras model and serves image classification requests with it. The rest of the guide will fill in details and discuss other ways to create SavedModels." ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "Le5OB-fBHHW7" }, "outputs": [], "source": [ "import os\n", "import tempfile\n", "\n", "from matplotlib import pyplot as plt\n", "import numpy as np\n", "import tensorflow as tf\n", "\n", "tmpdir = tempfile.mkdtemp()" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "wlho4HEWoHUT" }, "outputs": [], "source": [ "physical_devices = tf.config.experimental.list_physical_devices('GPU')\n", "if physical_devices:\n", " tf.config.experimental.set_memory_growth(physical_devices[0], True)" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "SofdPKo0G8Lb" }, "outputs": [], "source": [ "file = tf.keras.utils.get_file(\n", " \"grace_hopper.jpg\",\n", " \"https://storage.googleapis.com/download.tensorflow.org/example_images/grace_hopper.jpg\")\n", "img = tf.keras.preprocessing.image.load_img(file, target_size=[224, 224])\n", "plt.imshow(img)\n", "plt.axis('off')\n", "x = tf.keras.preprocessing.image.img_to_array(img)\n", "x = tf.keras.applications.mobilenet.preprocess_input(\n", " x[tf.newaxis,...])" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "sqVcFL10JkF0" }, "source": [ "We'll use an image of Grace Hopper as a running example, and a Keras pre-trained image classification model since it's easy to use. Custom models work too, and are covered in detail later." ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "JhVecdzJTsKE" }, "outputs": [], "source": [ "labels_path = tf.keras.utils.get_file(\n", " 'ImageNetLabels.txt',\n", " 'https://storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt')\n", "imagenet_labels = np.array(open(labels_path).read().splitlines())" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "aEHSYjW6JZHV" }, "outputs": [], "source": [ "pretrained_model = tf.keras.applications.MobileNet()\n", "result_before_save = pretrained_model(x)\n", "\n", "decoded = imagenet_labels[np.argsort(result_before_save)[0,::-1][:5]+1]\n", "\n", "print(\"Result before saving:\\n\", decoded)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "r4KIsQDZJ5PS" }, "source": [ "The top prediction for this image is \"military uniform\"." ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "8nfznDmHCW6F" }, "outputs": [], "source": [ "mobilenet_save_path = os.path.join(tmpdir, \"mobilenet/1/\")\n", "tf.saved_model.save(pretrained_model, mobilenet_save_path)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "pyX-ETE3wX63" }, "source": [ "The save-path follows a convention used by TensorFlow Serving where the last path component (`1/` here) is a version number for your model - it allows tools like Tensorflow Serving to reason about the relative freshness." ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "VCZZ8avqLF1g" }, "source": [ "We can load the SavedModel back into Python with `tf.saved_model.load` and see how Admiral Hopper's image is classified." ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "NP2UpVFRV7N_" }, "outputs": [], "source": [ "loaded = tf.saved_model.load(mobilenet_save_path)\n", "print(list(loaded.signatures.keys())) # [\"serving_default\"]" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "K5srGzowfWff" }, "source": [ "Imported signatures always return dictionaries. To customize signature names and output dictionary keys, see [Specifying signatures during export](#specifying_signatures_during_export)." ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "ChFLpegYfQGR" }, "outputs": [], "source": [ "infer = loaded.signatures[\"serving_default\"]\n", "print(infer.structured_outputs)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "cJYyZnptfuru" }, "source": [ "Running inference from the SavedModel gives the same result as the original model." ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "9WjGEaS3XfX7" }, "outputs": [], "source": [ "labeling = infer(tf.constant(x))[pretrained_model.output_names[0]]\n", "\n", "decoded = imagenet_labels[np.argsort(labeling)[0,::-1][:5]+1]\n", "\n", "print(\"Result after saving and loading:\\n\", decoded)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "SJEkdXjTWbtl" }, "source": [ "## Running a SavedModel in TensorFlow Serving\n", "\n", "SavedModels are usable from Python (more on that below), but production environments typically use a dedicated service for inference without running Python code. This is easy to set up from a SavedModel using TensorFlow Serving.\n", "\n", "See the [TensorFlow Serving REST tutorial](https://www.tensorflow.org/tfx/tutorials/serving/rest_simple) for more details about serving, including instructions for installing `tensorflow_model_server` in a notebook or on your local machine. As a quick sketch, to serve the `mobilenet` model exported above just point the model server at the SavedModel directory:\n", "\n", "```bash\n", "nohup tensorflow_model_server \\\n", " --rest_api_port=8501 \\\n", " --model_name=mobilenet \\\n", " --model_base_path=\"/tmp/mobilenet\" >server.log 2>\u00261\n", "```\n", "\n", " Then send a request.\n", "\n", "```python\n", "!pip install requests\n", "import json\n", "import numpy\n", "import requests\n", "data = json.dumps({\"signature_name\": \"serving_default\",\n", " \"instances\": x.tolist()})\n", "headers = {\"content-type\": \"application/json\"}\n", "json_response = requests.post('http://localhost:8501/v1/models/mobilenet:predict',\n", " data=data, headers=headers)\n", "predictions = numpy.array(json.loads(json_response.text)[\"predictions\"])\n", "```\n", "\n", "The resulting `predictions` are identical to the results from Python." ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "Bi0ILzu1XdWw" }, "source": [ "## The SavedModel format on disk\n", "\n", "A SavedModel is a directory containing serialized signatures and the state needed to run them, including variable values and vocabularies.\n" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "6u3YZuYZXyTO" }, "outputs": [], "source": [ "!ls {mobilenet_save_path}" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "ple4X5utX8ue" }, "source": [ "The `saved_model.pb` file stores the actual TensorFlow program, or model, and a set of named signatures, each identifying a function that accepts tensor inputs and produces tensor outputs.\n", "\n", "SavedModels may contain multiple variants of the model (multiple `v1.MetaGraphDefs`, identified with the `--tag_set` flag to `saved_model_cli`), but this is rare. APIs which create multiple variants of a model include [`tf.Estimator.experimental_export_all_saved_models`](https://www.tensorflow.org/api_docs/python/tf/estimator/Estimator#experimental_export_all_saved_models) and in TensorFlow 1.x `tf.saved_model.Builder`." ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "Pus0dOYTYXbI" }, "outputs": [], "source": [ "!saved_model_cli show --dir {mobilenet_save_path} --tag_set serve" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "eALHpGvRZOhk" }, "source": [ "The `variables` directory contains a standard training checkpoint (see the [guide to training checkpoints](./checkpoint.ipynb))." ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "EDYqhDlNZAC2" }, "outputs": [], "source": [ "!ls {mobilenet_save_path}/variables" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "VKmaZQpHahGh" }, "source": [ "The `assets` directory contains files used by the TensorFlow graph, for example text files used to initialize vocabulary tables. It is unused in this example.\n", "\n", "SavedModels may have an `assets.extra` directory for any files not used by the TensorFlow graph, for example information for consumers about what to do with the SavedModel. TensorFlow itself does not use this directory." ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "zIceoF_CYmaF" }, "source": [ "## Saving a custom model\n", "\n", "`tf.saved_model.save` supports saving `tf.Module` objects and its subclasses, like `tf.keras.Layer` and `tf.keras.Model`.\n", "\n", "Let's look at an example of saving and restoring a `tf.Module`.\n" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "6EPvKiqXMm3d" }, "outputs": [], "source": [ "class CustomModule(tf.Module):\n", "\n", " def __init__(self):\n", " super(CustomModule, self).__init__()\n", " self.v = tf.Variable(1.)\n", "\n", " @tf.function\n", " def __call__(self, x):\n", " print('Tracing with', x)\n", " return x * self.v\n", "\n", " @tf.function(input_signature=[tf.TensorSpec([], tf.float32)])\n", " def mutate(self, new_v):\n", " self.v.assign(new_v)\n", "\n", "module = CustomModule()" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "J4FcP-Co3Fnw" }, "source": [ "When you save a `tf.Module`, any `tf.Variable` attributes, `tf.function`-decorated methods, and `tf.Module`s found via recursive traversal are saved. (See the [Checkpoint tutorial](./checkpoint.ipynb) for more about this recursive traversal.) However, any Python attributes, functions, and data are lost. This means that when a `tf.function` is saved, no Python code is saved.\n", "\n", "If no Python code is saved, how does SavedModel know how to restore the function?\n", "\n", "Briefly, `tf.function` works by tracing the Python code to generate a ConcreteFunction (a callable wrapper around `tf.Graph`). When saving a `tf.function`, you're really saving the `tf.function`'s cache of ConcreteFunctions.\n", "\n", "To learn more about the relationship between `tf.function` and ConcreteFunctions, see the [tf.function guide](../../guide/concrete_function). This [section on tracing](../../tutorials/customization/performance#side_effects_in_tffunction) also elaborates on which Python code is captured by a ConcreteFunction." ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "85PUO9iWH7xn" }, "outputs": [], "source": [ "module_no_signatures_path = os.path.join(tmpdir, 'module_no_signatures')\n", "module(tf.constant(0.))\n", "print('Saving model...')\n", "tf.saved_model.save(module, module_no_signatures_path)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "2ujwmMQg7OUo" }, "source": [ "## Loading and using a custom model" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "QpxQy5Eb77qJ" }, "source": [ "When you load a SavedModel in Python, all `tf.Variable` attributes, `tf.function`-decorated methods, and `tf.Module`s are restored in the same object structure as the original saved `tf.Module`." ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "EMASjADPxPso" }, "outputs": [], "source": [ "imported = tf.saved_model.load(module_no_signatures_path)\n", "assert imported(tf.constant(3.)).numpy() == 3\n", "imported.mutate(tf.constant(2.))\n", "assert imported(tf.constant(3.)).numpy() == 6" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "CDiauvb_99uk" }, "source": [ "Because no Python code is saved, calling a `tf.function` with a new input signature will fail:\n", "\n", "```python\n", "imported(tf.constant([3.]))\n", "```\n", "\n", "
\n",
        "ValueError: Could not find matching function to call for canonicalized inputs ((,), {}). Only existing signatures are [((TensorSpec(shape=(), dtype=tf.float32, name=u'x'),), {})].\n",
        "
" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "4Vsva3UZ-2sf" }, "source": [ "### Basic fine-tuning\n", "\n", "Variable objects are available, and we can backprop through imported functions. That is enough to fine-tune (i.e. retrain) a SavedModel in simple cases." ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "PEkQNarJ-7nT" }, "outputs": [], "source": [ "optimizer = tf.optimizers.SGD(0.05)\n", "\n", "def train_step():\n", " with tf.GradientTape() as tape:\n", " loss = (10. - imported(tf.constant(2.))) ** 2\n", " variables = tape.watched_variables()\n", " grads = tape.gradient(loss, variables)\n", " optimizer.apply_gradients(zip(grads, variables))\n", " return loss" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "p41NM6fF---3" }, "outputs": [], "source": [ "for _ in range(10):\n", " # \"v\" approaches 5, \"loss\" approaches 0\n", " print(\"loss={:.2f} v={:.2f}\".format(train_step(), imported.v.numpy()))" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "XuXtkHSD_KSW" }, "source": [ "### General fine-tuning\n", "\n", "A SavedModel from Keras provides [more details](https://github.com/tensorflow/community/blob/master/rfcs/20190509-keras-saved-model.md#serialization-details) than a plain `__call__` to address more advanced cases of fine-tuning. TensorFlow Hub recommends to provide the following of those, if applicable, in SavedModels shared for the purpose of fine-tuning:\n", "\n", " * If the model uses dropout or another technique in which the forward pass differs between training and inference (like batch normalization), the `__call__` method takes an optional, Python-valued `training=` argument that defaults to `False` but can be set to `True`.\n", " * Next to the `__call__` attribute, there are `.variable` and `.trainable_variable` attributes with the corresponding lists of variables. A variable that was originally trainable but is meant to be frozen during fine-tuning is omitted from `.trainable_variables`.\n", " * For the sake of frameworks like Keras that represent weight regularizers as attributes of layers or sub-models, there can also be a `.regularization_losses` attribute. It holds a list of zero-argument functions whose values are meant for addition to the total loss.\n", "\n", "Going back to the initial MobileNet example, we can see some of those in action:" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "Y6EUFdY8_PRD" }, "outputs": [], "source": [ "loaded = tf.saved_model.load(mobilenet_save_path)\n", "print(\"MobileNet has {} trainable variables: {}, ...\".format(\n", " len(loaded.trainable_variables),\n", " \", \".join([v.name for v in loaded.trainable_variables[:5]])))" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "B-mQJ8iP_R0h" }, "outputs": [], "source": [ "trainable_variable_ids = {id(v) for v in loaded.trainable_variables}\n", "non_trainable_variables = [v for v in loaded.variables\n", " if id(v) not in trainable_variable_ids]\n", "print(\"MobileNet also has {} non-trainable variables: {}, ...\".format(\n", " len(non_trainable_variables),\n", " \", \".join([v.name for v in non_trainable_variables[:3]])))" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "qGlHlbd3_eyO" }, "source": [ "## Specifying signatures during export\n", "\n", "Tools like TensorFlow Serving and `saved_model_cli` can interact with SavedModels. To help these tools determine which ConcreteFunctions to use, we need to specify serving signatures. `tf.keras.Model`s automatically specify serving signatures, but we'll have to explicitly declare a serving signature for our custom modules.\n", "\n", "By default, no signatures are declared in a custom `tf.Module`." ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "h-IB5Xa0NxLa" }, "outputs": [], "source": [ "assert len(imported.signatures) == 0" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "BiNtaMZSI8Tb" }, "source": [ "To declare a serving signature, specify a ConcreteFunction using the `signatures` kwarg. When specifying a single signature, its signature key will be `'serving_default'`, which is saved as the constant `tf.saved_model.DEFAULT_SERVING_SIGNATURE_DEF_KEY`." ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "_pAdgIORR2yH" }, "outputs": [], "source": [ "module_with_signature_path = os.path.join(tmpdir, 'module_with_signature')\n", "call = module.__call__.get_concrete_function(tf.TensorSpec(None, tf.float32))\n", "tf.saved_model.save(module, module_with_signature_path, signatures=call)" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "nAzRHR0UT4hv" }, "outputs": [], "source": [ "imported_with_signatures = tf.saved_model.load(module_with_signature_path)\n", "list(imported_with_signatures.signatures.keys())\n" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "_gH91j1IR4tq" }, "source": [ "To export multiple signatures, pass a dictionary of signature keys to ConcreteFunctions. Each signature key corresponds to one ConcreteFunction." ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "6VYAiQmLUiox" }, "outputs": [], "source": [ "module_multiple_signatures_path = os.path.join(tmpdir, 'module_with_multiple_signatures')\n", "signatures = {\"serving_default\": call,\n", " \"array_input\": module.__call__.get_concrete_function(tf.TensorSpec([None], tf.float32))}\n", "\n", "tf.saved_model.save(module, module_multiple_signatures_path, signatures=signatures)" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "8IPx_0RWEx07" }, "outputs": [], "source": [ "imported_with_multiple_signatures = tf.saved_model.load(module_multiple_signatures_path)\n", "list(imported_with_multiple_signatures.signatures.keys())" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "43_Qv2W_DJZZ" }, "source": [ "By default, the output tensor names are fairly generic, like `output_0`. To control the names of outputs, modify your `tf.function` to return a dictionary that maps output names to outputs. The names of inputs are derived from the Python function arg names." ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "ACKPl1X8G1gw" }, "outputs": [], "source": [ "class CustomModuleWithOutputName(tf.Module):\n", " def __init__(self):\n", " super(CustomModuleWithOutputName, self).__init__()\n", " self.v = tf.Variable(1.)\n", "\n", " @tf.function(input_signature=[tf.TensorSpec([], tf.float32)])\n", " def __call__(self, x):\n", " return {'custom_output_name': x * self.v}\n", "\n", "module_output = CustomModuleWithOutputName()\n", "call_output = module_output.__call__.get_concrete_function(tf.TensorSpec(None, tf.float32))\n", "module_output_path = os.path.join(tmpdir, 'module_with_output_name')\n", "tf.saved_model.save(module_output, module_output_path,\n", " signatures={'serving_default': call_output})" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "1yGVy4MuH-V0" }, "outputs": [], "source": [ "imported_with_output_name = tf.saved_model.load(module_output_path)\n", "imported_with_output_name.signatures['serving_default'].structured_outputs" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "Dk5wWyuMpuHx" }, "source": [ "## SavedModels from Estimators\n", "\n", "Estimators export SavedModels through [`tf.Estimator.export_saved_model`](https://www.tensorflow.org/api_docs/python/tf/estimator/Estimator#export_saved_model). See the [guide to Estimator](https://www.tensorflow.org/guide/estimator) for details." ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "B9KQq5qzpzbK" }, "outputs": [], "source": [ "input_column = tf.feature_column.numeric_column(\"x\")\n", "estimator = tf.estimator.LinearClassifier(feature_columns=[input_column])\n", "\n", "def input_fn():\n", " return tf.data.Dataset.from_tensor_slices(\n", " ({\"x\": [1., 2., 3., 4.]}, [1, 1, 0, 0])).repeat(200).shuffle(64).batch(16)\n", "estimator.train(input_fn)\n", "\n", "serving_input_fn = tf.estimator.export.build_parsing_serving_input_receiver_fn(\n", " tf.feature_column.make_parse_example_spec([input_column]))\n", "estimator_base_path = os.path.join(tmpdir, 'from_estimator')\n", "estimator_path = estimator.export_saved_model(estimator_base_path, serving_input_fn)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "XJ4PJ-Cl4060" }, "source": [ "This SavedModel accepts serialized `tf.Example` protocol buffers, which are useful for serving. But we can also load it with `tf.saved_model.load` and run it from Python." ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "c_BUBBNB1UH9" }, "outputs": [], "source": [ "imported = tf.saved_model.load(estimator_path)\n", "\n", "def predict(x):\n", " example = tf.train.Example()\n", " example.features.feature[\"x\"].float_list.value.extend([x])\n", " return imported.signatures[\"predict\"](\n", " examples=tf.constant([example.SerializeToString()]))" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "C1ylWZCQ1ahG" }, "outputs": [], "source": [ "print(predict(1.5))\n", "print(predict(3.5))" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "_IrCCm0-isqA" }, "source": [ "`tf.estimator.export.build_raw_serving_input_receiver_fn` allows you to create input functions which take raw tensors rather than `tf.train.Example`s." ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "Co6fDbzw_UnD" }, "source": [ "## Load a SavedModel in C++\n", "\n", "The C++ version of the SavedModel [loader](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/cc/saved_model/loader.h) provides an API to load a SavedModel from a path, while allowing SessionOptions and RunOptions. You have to specify the tags associated with the graph to be loaded. The loaded version of SavedModel is referred to as SavedModelBundle and contains the MetaGraphDef and the session within which it is loaded.\n", "\n", "```C++\n", "const string export_dir = ...\n", "SavedModelBundle bundle;\n", "...\n", "LoadSavedModel(session_options, run_options, export_dir, {kSavedModelTagTrain},\n", " \u0026bundle);\n", "```" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "b33KuyEuAO3Z" }, "source": [ "\n", "\n", "## Details of the SavedModel command line interface\n", "\n", "You can use the SavedModel Command Line Interface (CLI) to inspect and\n", "execute a SavedModel.\n", "For example, you can use the CLI to inspect the model's `SignatureDef`s.\n", "The CLI enables you to quickly confirm that the input\n", "Tensor dtype and shape match the model. Moreover, if you\n", "want to test your model, you can use the CLI to do a sanity check by\n", "passing in sample inputs in various formats (for example, Python\n", "expressions) and then fetching the output.\n", "\n", "\n", "### Install the SavedModel CLI\n", "\n", "Broadly speaking, you can install TensorFlow in either of the following\n", "two ways:\n", "\n", "* By installing a pre-built TensorFlow binary.\n", "* By building TensorFlow from source code.\n", "\n", "If you installed TensorFlow through a pre-built TensorFlow binary,\n", "then the SavedModel CLI is already installed on your system\n", "at pathname `bin/saved_model_cli`.\n", "\n", "If you built TensorFlow from source code, you must run the following\n", "additional command to build `saved_model_cli`:\n", "\n", "```\n", "$ bazel build tensorflow/python/tools:saved_model_cli\n", "```\n", "\n", "### Overview of commands\n", "\n", "The SavedModel CLI supports the following two commands on a SavedModel:\n", "\n", "* `show`, which shows the computations available from a SavedModel.\n", "* `run`, which runs a computation from a SavedModel.\n", "\n", "\n", "### `show` command\n", "\n", "A SavedModel contains one or more model variants (technically, `v1.MetaGraphDef`s), identified by their tag-sets. To serve a model, you might wonder what kind of `SignatureDef`s are in each model variant, and what are their inputs and outputs. The `show` command let you examine the contents of the SavedModel in hierarchical order. Here's the syntax:\n", "\n", "```\n", "usage: saved_model_cli show [-h] --dir DIR [--all]\n", "[--tag_set TAG_SET] [--signature_def SIGNATURE_DEF_KEY]\n", "```\n", "\n", "For example, the following command shows all available tag-sets in the SavedModel:\n", "\n", "```\n", "$ saved_model_cli show --dir /tmp/saved_model_dir\n", "The given SavedModel contains the following tag-sets:\n", "serve\n", "serve, gpu\n", "```\n", "\n", "The following command shows all available `SignatureDef` keys for a tag set:\n", "\n", "```\n", "$ saved_model_cli show --dir /tmp/saved_model_dir --tag_set serve\n", "The given SavedModel `MetaGraphDef` contains `SignatureDefs` with the\n", "following keys:\n", "SignatureDef key: \"classify_x2_to_y3\"\n", "SignatureDef key: \"classify_x_to_y\"\n", "SignatureDef key: \"regress_x2_to_y3\"\n", "SignatureDef key: \"regress_x_to_y\"\n", "SignatureDef key: \"regress_x_to_y2\"\n", "SignatureDef key: \"serving_default\"\n", "```\n", "\n", "If there are *multiple* tags in the tag-set, you must specify\n", "all tags, each tag separated by a comma. For example:\n", "\n", "
\n",
        "$ saved_model_cli show --dir /tmp/saved_model_dir --tag_set serve,gpu\n",
        "
\n", "\n", "To show all inputs and outputs TensorInfo for a specific `SignatureDef`, pass in\n", "the `SignatureDef` key to `signature_def` option. This is very useful when you\n", "want to know the tensor key value, dtype and shape of the input tensors for\n", "executing the computation graph later. For example:\n", "\n", "```\n", "$ saved_model_cli show --dir \\\n", "/tmp/saved_model_dir --tag_set serve --signature_def serving_default\n", "The given SavedModel SignatureDef contains the following input(s):\n", " inputs['x'] tensor_info:\n", " dtype: DT_FLOAT\n", " shape: (-1, 1)\n", " name: x:0\n", "The given SavedModel SignatureDef contains the following output(s):\n", " outputs['y'] tensor_info:\n", " dtype: DT_FLOAT\n", " shape: (-1, 1)\n", " name: y:0\n", "Method name is: tensorflow/serving/predict\n", "```\n", "\n", "To show all available information in the SavedModel, use the `--all` option.\n", "For example:\n", "\n", "
\n",
        "$ saved_model_cli show --dir /tmp/saved_model_dir --all\n",
        "MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:\n",
        "\n",
        "signature_def['classify_x2_to_y3']:\n",
        "  The given SavedModel SignatureDef contains the following input(s):\n",
        "    inputs['inputs'] tensor_info:\n",
        "        dtype: DT_FLOAT\n",
        "        shape: (-1, 1)\n",
        "        name: x2:0\n",
        "  The given SavedModel SignatureDef contains the following output(s):\n",
        "    outputs['scores'] tensor_info:\n",
        "        dtype: DT_FLOAT\n",
        "        shape: (-1, 1)\n",
        "        name: y3:0\n",
        "  Method name is: tensorflow/serving/classify\n",
        "\n",
        "...\n",
        "\n",
        "signature_def['serving_default']:\n",
        "  The given SavedModel SignatureDef contains the following input(s):\n",
        "    inputs['x'] tensor_info:\n",
        "        dtype: DT_FLOAT\n",
        "        shape: (-1, 1)\n",
        "        name: x:0\n",
        "  The given SavedModel SignatureDef contains the following output(s):\n",
        "    outputs['y'] tensor_info:\n",
        "        dtype: DT_FLOAT\n",
        "        shape: (-1, 1)\n",
        "        name: y:0\n",
        "  Method name is: tensorflow/serving/predict\n",
        "
\n", "\n", "\n", "### `run` command\n", "\n", "Invoke the `run` command to run a graph computation, passing\n", "inputs and then displaying (and optionally saving) the outputs.\n", "Here's the syntax:\n", "\n", "```\n", "usage: saved_model_cli run [-h] --dir DIR --tag_set TAG_SET --signature_def\n", " SIGNATURE_DEF_KEY [--inputs INPUTS]\n", " [--input_exprs INPUT_EXPRS]\n", " [--input_examples INPUT_EXAMPLES] [--outdir OUTDIR]\n", " [--overwrite] [--tf_debug]\n", "```\n", "\n", "The `run` command provides the following three ways to pass inputs to the model:\n", "\n", "* `--inputs` option enables you to pass numpy ndarray in files.\n", "* `--input_exprs` option enables you to pass Python expressions.\n", "* `--input_examples` option enables you to pass `tf.train.Example`.\n", "\n", "#### `--inputs`\n", "\n", "To pass input data in files, specify the `--inputs` option, which takes the\n", "following general format:\n", "\n", "```bsh\n", "--inputs \n", "```\n", "\n", "where *INPUTS* is either of the following formats:\n", "\n", "* `=`\n", "* `=[]`\n", "\n", "You may pass multiple *INPUTS*. If you do pass multiple inputs, use a semicolon\n", "to separate each of the *INPUTS*.\n", "\n", "`saved_model_cli` uses `numpy.load` to load the *filename*.\n", "The *filename* may be in any of the following formats:\n", "\n", "* `.npy`\n", "* `.npz`\n", "* pickle format\n", "\n", "A `.npy` file always contains a numpy ndarray. Therefore, when loading from\n", "a `.npy` file, the content will be directly assigned to the specified input\n", "tensor. If you specify a *variable_name* with that `.npy` file, the\n", "*variable_name* will be ignored and a warning will be issued.\n", "\n", "When loading from a `.npz` (zip) file, you may optionally specify a\n", "*variable_name* to identify the variable within the zip file to load for\n", "the input tensor key. If you don't specify a *variable_name*, the SavedModel\n", "CLI will check that only one file is included in the zip file and load it\n", "for the specified input tensor key.\n", "\n", "When loading from a pickle file, if no `variable_name` is specified in the\n", "square brackets, whatever that is inside the pickle file will be passed to the\n", "specified input tensor key. Otherwise, the SavedModel CLI will assume a\n", "dictionary is stored in the pickle file and the value corresponding to\n", "the *variable_name* will be used.\n", "\n", "\n", "#### `--input_exprs`\n", "\n", "To pass inputs through Python expressions, specify the `--input_exprs` option.\n", "This can be useful for when you don't have data\n", "files lying around, but still want to sanity check the model with some simple\n", "inputs that match the dtype and shape of the model's `SignatureDef`s.\n", "For example:\n", "\n", "```bsh\n", "`=[[1],[2],[3]]`\n", "```\n", "\n", "In addition to Python expressions, you may also pass numpy functions. For\n", "example:\n", "\n", "```bsh\n", "`=np.ones((32,32,3))`\n", "```\n", "\n", "(Note that the `numpy` module is already available to you as `np`.)\n", "\n", "\n", "#### `--input_examples`\n", "\n", "To pass `tf.train.Example` as inputs, specify the `--input_examples` option.\n", "For each input key, it takes a list of dictionary, where each dictionary is an\n", "instance of `tf.train.Example`. The dictionary keys are the features and the\n", "values are the value lists for each feature.\n", "For example:\n", "\n", "```bsh\n", "`=[{\"age\":[22,24],\"education\":[\"BS\",\"MS\"]}]`\n", "```\n", "\n", "#### Save output\n", "\n", "By default, the SavedModel CLI writes output to stdout. If a directory is\n", "passed to `--outdir` option, the outputs will be saved as `.npy` files named after\n", "output tensor keys under the given directory.\n", "\n", "Use `--overwrite` to overwrite existing output files.\n" ] } ], "metadata": { "colab": { "collapsed_sections": [], "name": "saved_model.ipynb", "private_outputs": true, "provenance": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 0 }