{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# BentoML Example: Keras Text Classification\n", "\n", "**BentoML makes moving trained ML models to production easy:**\n", "\n", "* Package models trained with **any ML framework** and reproduce them for model serving in production\n", "* **Deploy anywhere** for online API serving or offline batch serving\n", "* High-Performance API model server with *adaptive micro-batching* support\n", "* Central hub for managing models and deployment process via Web UI and APIs\n", "* Modular and flexible design making it *adaptable to your infrastrcuture*\n", "\n", "BentoML is a framework for serving, managing, and deploying machine learning models. It is aiming to bridge the gap between Data Science and DevOps, and enable teams to deliver prediction services in a fast, repeatable, and scalable way.\n", "\n", "Before reading this example project, be sure to check out the [Getting started guide](https://github.com/bentoml/BentoML/blob/master/guides/quick-start/bentoml-quick-start-guide.ipynb) to learn about the basic concepts in BentoML. \n", "\n", "This notebook demonstrates how to use BentoML to turn a Keras model into a docker image containing a REST API server serving this model, how to use your ML service built with BentoML as a CLI tool, and how to distribute it a pypi package.\n", "\n", "This notebook is built based on Keras's IMDB LSTM tutorial [here](https://github.com/keras-team/keras/blob/master/examples/imdb_lstm.py).\n", "\n", "![Impression](https://www.google-analytics.com/collect?v=1&tid=UA-112879361-3&cid=555&t=event&ec=keras&ea=keras-text-classification&dt=keras-text-classification)" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "%reload_ext autoreload\n", "%autoreload 2\n", "%matplotlib inline" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: bentoml in /home/bentoml/BentoML (0.8.3+47.g7af5a8f)\n", "Requirement already satisfied: aiohttp in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (3.6.2)\n", "Requirement already satisfied: alembic in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (1.4.2)\n", "Requirement already satisfied: boto3 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (1.14.30)\n", "Requirement already satisfied: cerberus in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (1.3.2)\n", "Requirement already satisfied: certifi in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (2020.6.20)\n", "Requirement already satisfied: click>=7.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (7.1.2)\n", "Requirement already satisfied: configparser in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (5.0.0)\n", "Requirement already satisfied: docker in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (4.2.0)\n", "Requirement already satisfied: flask in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (1.1.2)\n", "Requirement already satisfied: grpcio<=1.27.2 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (1.27.2)\n", "Requirement already satisfied: gunicorn in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (20.0.4)\n", "Requirement already satisfied: humanfriendly in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (8.2)\n", "Requirement already satisfied: numpy in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (1.16.6)\n", "Requirement already satisfied: packaging in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (20.4)\n", "Requirement already satisfied: prometheus_client in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (0.7.1)\n", "Requirement already satisfied: protobuf>=3.6.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (3.11.4)\n", "Requirement already satisfied: psutil in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (5.7.2)\n", "Requirement already satisfied: py_zipkin in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (0.20.0)\n", "Requirement already satisfied: python-dateutil<2.8.1,>=2.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (2.8.0)\n", "Requirement already satisfied: python-json-logger in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (0.1.11)\n", "Requirement already satisfied: requests in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (2.23.0)\n", "Requirement already satisfied: ruamel.yaml>=0.15.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (0.16.10)\n", "Requirement already satisfied: sqlalchemy-utils in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (0.36.8)\n", "Requirement already satisfied: sqlalchemy>=1.3.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (1.3.13)\n", "Requirement already satisfied: tabulate in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (0.8.7)\n", "Requirement already satisfied: multidict in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (4.7.6)\n", "Requirement already satisfied: contextvars in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (2.4)\n", "Requirement already satisfied: idna-ssl>=1.0; python_version < \"3.7\" in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from aiohttp->bentoml) (1.1.0)\n", "Requirement already satisfied: async-timeout<4.0,>=3.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from aiohttp->bentoml) (3.0.1)\n", "Requirement already satisfied: typing-extensions>=3.6.5; python_version < \"3.7\" in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from aiohttp->bentoml) (3.7.4.2)\n", "Requirement already satisfied: yarl<2.0,>=1.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from aiohttp->bentoml) (1.5.0)\n", "Requirement already satisfied: chardet<4.0,>=2.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from aiohttp->bentoml) (3.0.4)\n", "Requirement already satisfied: attrs>=17.3.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from aiohttp->bentoml) (19.3.0)\n", "Requirement already satisfied: Mako in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from alembic->bentoml) (1.1.2)\n", "Requirement already satisfied: python-editor>=0.3 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from alembic->bentoml) (1.0.4)\n", "Requirement already satisfied: botocore<1.18.0,>=1.17.30 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from boto3->bentoml) (1.17.30)\n", "Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from boto3->bentoml) (0.3.3)\n", "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from boto3->bentoml) (0.10.0)\n", "Requirement already satisfied: setuptools in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from cerberus->bentoml) (46.2.0.post20200511)\n", "Requirement already satisfied: six>=1.4.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from docker->bentoml) (1.14.0)\n", "Requirement already satisfied: websocket-client>=0.32.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from docker->bentoml) (0.57.0)\n", "Requirement already satisfied: Jinja2>=2.10.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from flask->bentoml) (2.11.2)\n", "Requirement already satisfied: Werkzeug>=0.15 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from flask->bentoml) (1.0.1)\n", "Requirement already satisfied: itsdangerous>=0.24 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from flask->bentoml) (1.1.0)\n", "Requirement already satisfied: pyparsing>=2.0.2 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from packaging->bentoml) (2.4.7)\n", "Requirement already satisfied: thriftpy2>=0.4.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from py_zipkin->bentoml) (0.4.11)\n", "Requirement already satisfied: idna<3,>=2.5 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from requests->bentoml) (2.9)\n", "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from requests->bentoml) (1.25.9)\n", "Requirement already satisfied: ruamel.yaml.clib>=0.1.2; platform_python_implementation == \"CPython\" and python_version < \"3.9\" in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from ruamel.yaml>=0.15.0->bentoml) (0.2.0)\n", "Requirement already satisfied: immutables>=0.9 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from contextvars->bentoml) (0.14)\n", "Requirement already satisfied: MarkupSafe>=0.9.2 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from Mako->alembic->bentoml) (1.1.1)\n", "Requirement already satisfied: docutils<0.16,>=0.10 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from botocore<1.18.0,>=1.17.30->boto3->bentoml) (0.15.2)\n", "Requirement already satisfied: ply<4.0,>=3.4 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from thriftpy2>=0.4.0->py_zipkin->bentoml) (3.11)\n", "Requirement already satisfied: tensorflow==1.14.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (1.14.0)\n", "Requirement already satisfied: termcolor>=1.1.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.1.0)\n", "Requirement already satisfied: keras-applications>=1.0.6 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.0.8)\n", "Requirement already satisfied: tensorboard<1.15.0,>=1.14.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.14.0)\n", "Requirement already satisfied: keras-preprocessing>=1.0.5 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.1.0)\n", "Requirement already satisfied: gast>=0.2.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (0.3.3)\n", "Requirement already satisfied: numpy<2.0,>=1.14.5 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.16.6)\n", "Requirement already satisfied: astor>=0.6.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (0.8.0)\n", "Requirement already satisfied: six>=1.10.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.14.0)\n", "Requirement already satisfied: wrapt>=1.11.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.12.1)\n", "Requirement already satisfied: tensorflow-estimator<1.15.0rc0,>=1.14.0rc0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.14.0)\n", "Requirement already satisfied: grpcio>=1.8.6 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.27.2)\n", "Requirement already satisfied: google-pasta>=0.1.6 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (0.2.0)\n", "Requirement already satisfied: wheel>=0.26 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (0.34.2)\n", "Requirement already satisfied: absl-py>=0.7.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (0.9.0)\n", "Requirement already satisfied: protobuf>=3.6.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (3.11.4)\n", "Requirement already satisfied: h5py in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from keras-applications>=1.0.6->tensorflow==1.14.0) (2.10.0)\n", "Requirement already satisfied: markdown>=2.6.8 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (3.1.1)\n", "Requirement already satisfied: setuptools>=41.0.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (46.2.0.post20200511)\n", "Requirement already satisfied: werkzeug>=0.11.15 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (1.0.1)\n", "Requirement already satisfied: numpy in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (1.16.6)\n" ] } ], "source": [ "!pip install -q bentoml tensorflow==1.14.0 numpy>=1.16.6" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Tensorflow Version: 1.14.0\n", "BentoML Version: 0.9.0.pre+7.g8af1c8b\n" ] } ], "source": [ "from __future__ import absolute_import, division, print_function\n", "\n", "import numpy as np\n", "import tensorflow as tf\n", "print(\"Tensorflow Version: %s\" % tf.__version__)\n", "\n", "from tensorflow import keras\n", "from tensorflow.keras.preprocessing import sequence\n", "from tensorflow.keras.models import Sequential\n", "from tensorflow.keras.layers import Dense, Embedding\n", "from tensorflow.keras.layers import LSTM\n", "from tensorflow.keras.datasets import imdb\n", "\n", "import bentoml\n", "print(\"BentoML Version: %s\" % bentoml.__version__)" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "max_features = 1000\n", "maxlen = 80 # cut texts after this number of words (among top max_features most common words)\n", "batch_size = 300\n", "index_from=3 # word index offset" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "iAsKG535pHep" }, "source": [ "# Prepare Dataset\n", "Download the IMDB dataset" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/opt/conda/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/keras/datasets/imdb.py:128: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n", " x_train, y_train = np.array(xs[:idx]), np.array(labels[:idx])\n", "/opt/conda/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/keras/datasets/imdb.py:129: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n", " x_test, y_test = np.array(xs[idx:]), np.array(labels[idx:])\n" ] } ], "source": [ "# A dictionary mapping words to an integer index\n", "imdb.load_data(num_words=max_features)\n", "word_index = imdb.get_word_index()\n", "\n", "# The first indices are reserved\n", "word_index = {k:(v+index_from) for k,v in word_index.items()} \n", "word_index[\"\"] = 0\n", "word_index[\"\"] = 1\n", "word_index[\"\"] = 2 # unknown\n", "\n", "# Use decode_review to look at original review text in training/testing data\n", "reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])\n", "def decode_review(encoded_text):\n", " return ' '.join([reverse_word_index.get(i, '?') for i in encoded_text])" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features, index_from=index_from)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "x_train = sequence.pad_sequences(x_train,\n", " value=word_index[\"\"],\n", " padding='post',\n", " maxlen=maxlen)\n", "\n", "x_test = sequence.pad_sequences(x_test,\n", " value=word_index[\"\"],\n", " padding='post',\n", " maxlen=maxlen)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "LLC02j2g-llC" }, "source": [ "# Model Training & Evaluation" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "colab": {}, "colab_type": "code", "id": "xpKOoWgu-llD" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /opt/conda/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/keras/initializers.py:119: calling RandomUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "WARNING:tensorflow:From /opt/conda/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "Model: \"sequential\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "embedding (Embedding) (None, None, 128) 128000 \n", "_________________________________________________________________\n", "lstm (LSTM) (None, 128) 131584 \n", "_________________________________________________________________\n", "dense (Dense) (None, 1) 129 \n", "=================================================================\n", "Total params: 259,713\n", "Trainable params: 259,713\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] } ], "source": [ "model = Sequential()\n", "model.add(Embedding(max_features, 128))\n", "model.add(LSTM(128, dropout=0.2, recurrent_dropout=0.2))\n", "model.add(Dense(1, activation='sigmoid'))\n", "\n", "model.summary()" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "colab": {}, "colab_type": "code", "id": "tXSGrjWZ-llW" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /opt/conda/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.where in 2.0, which has the same broadcast rule as np.where\n", "Train on 25000 samples, validate on 25000 samples\n", "Epoch 1/3\n", "25000/25000 [==============================] - 10s 388us/sample - loss: 0.5804 - acc: 0.6948 - val_loss: 0.4279 - val_acc: 0.8050\n", "Epoch 2/3\n", "25000/25000 [==============================] - 9s 357us/sample - loss: 0.4410 - acc: 0.8017 - val_loss: 0.4285 - val_acc: 0.8053\n", "Epoch 3/3\n", "25000/25000 [==============================] - 9s 358us/sample - loss: 0.4205 - acc: 0.8133 - val_loss: 0.4086 - val_acc: 0.8152\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.compile(loss='binary_crossentropy',\n", " optimizer='adam',\n", " metrics=['accuracy'])\n", "\n", "model.fit(x_train, y_train,\n", " batch_size=batch_size,\n", " epochs=3, # for demo purpose :P\n", " validation_data=(x_test, y_test))" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "colab": {}, "colab_type": "code", "id": "zOMKywn4zReN" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "25000/25000 [==============================] - 2s 69us/sample - loss: 0.4086 - acc: 0.8152\n", "Test score: 0.40862307107448576\n", "Test accuracy: 0.81524\n" ] } ], "source": [ "score, acc = model.evaluate(x_test, y_test,\n", " batch_size=batch_size)\n", "\n", "print('Test score:', score)\n", "print('Test accuracy:', acc)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Define BentoService for model serving" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Overwriting keras_text_classification_service.py\n" ] } ], "source": [ "%%writefile keras_text_classification_service.py\n", "from typing import List\n", "\n", "import pandas as pd\n", "import numpy as np\n", "from tensorflow import keras\n", "from tensorflow.keras.preprocessing import sequence, text\n", "from bentoml import api, env, BentoService, artifacts\n", "from bentoml.frameworks.keras import KerasModelArtifact\n", "from bentoml.service.artifacts.common import PickleArtifact\n", "from bentoml.adapters import JsonInput\n", "from bentoml.types import JsonSerializable\n", "\n", "\n", "max_features = 1000\n", "\n", "@artifacts([\n", " KerasModelArtifact('model'),\n", " PickleArtifact('word_index')\n", "])\n", "@env(pip_packages=['tensorflow==1.14.0', 'numpy', 'pandas'])\n", "class KerasTextClassificationService(BentoService):\n", " \n", " def word_to_index(self, word):\n", " if word in self.artifacts.word_index and self.artifacts.word_index[word] <= max_features:\n", " return self.artifacts.word_index[word]\n", " else:\n", " return self.artifacts.word_index[\"\"]\n", " \n", " def preprocessing(self, text_str):\n", " sequence = text.text_to_word_sequence(text_str)\n", " return list(map(self.word_to_index, sequence))\n", " \n", " @api(input=JsonInput(), batch=True)\n", " def predict(self, parsed_jsons: List[JsonSerializable]):\n", " input_datas = [self.preprocessing(parsed_json['text']) for parsed_json in parsed_jsons]\n", " input_datas = sequence.pad_sequences(input_datas,\n", " value=self.artifacts.word_index[\"\"],\n", " padding='post',\n", " maxlen=80)\n", "\n", " return self.artifacts.model.predict_classes(input_datas).T[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Save BentoService to file archive" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[2020-09-23 13:27:53,173] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.\n", "[2020-09-23 13:27:53,201] WARNING - pip package requirement tensorflow already exist\n", "WARNING:tensorflow:From /home/bentoml/BentoML/bentoml/frameworks/keras.py:123: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.\n", "\n", "[2020-09-23 13:27:55,262] INFO - Detected non-PyPI-released BentoML installed, copying local BentoML modulefiles to target saved bundle path..\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/opt/conda/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/setuptools/dist.py:452: UserWarning: Normalizing '0.9.0.pre+7.g8af1c8b' to '0.9.0rc0+7.g8af1c8b'\n", " warnings.warn(tmpl.format(**locals()))\n", "warning: no previously-included files matching '*~' found anywhere in distribution\n", "warning: no previously-included files matching '*.pyo' found anywhere in distribution\n", "warning: no previously-included files matching '.git' found anywhere in distribution\n", "warning: no previously-included files matching '.ipynb_checkpoints' found anywhere in distribution\n", "warning: no previously-included files matching '__pycache__' found anywhere in distribution\n", "warning: no directories found matching 'bentoml/yatai/web/dist'\n", "no previously-included directories found matching 'e2e_tests'\n", "no previously-included directories found matching 'tests'\n", "no previously-included directories found matching 'benchmark'\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "UPDATING BentoML-0.9.0rc0+7.g8af1c8b/bentoml/_version.py\n", "set BentoML-0.9.0rc0+7.g8af1c8b/bentoml/_version.py to '0.9.0.pre+7.g8af1c8b'\n", "[2020-09-23 13:27:56,076] INFO - BentoService bundle 'KerasTextClassificationService:20200923132753_A6330E' saved to: /home/bentoml/bentoml/repository/KerasTextClassificationService/20200923132753_A6330E\n" ] } ], "source": [ "# 1) import the custom BentoService defined above\n", "from keras_text_classification_service import KerasTextClassificationService\n", "\n", "# 2) `pack` it with required artifacts\n", "bento_svc = KerasTextClassificationService()\n", "bento_svc.pack('model', model)\n", "bento_svc.pack('word_index', word_index)\n", "\n", "# 3) save your BentoSerivce\n", "saved_path = bento_svc.save()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## REST API Model Serving\n", "\n", "\n", "To start a REST API model server with the BentoService saved above, use the bentoml serve command:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[2020-09-23 13:28:28,754] INFO - Getting latest version KerasTextClassificationService:20200923132753_A6330E\n", "[2020-09-23 13:28:28,754] INFO - Starting BentoML API server in development mode..\n", "[2020-09-23 13:28:29,911] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.\n", "[2020-09-23 13:28:29,926] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+7.g8af1c8b\n", "/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n", "/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n", "/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n", "/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n", "/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n", "/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n", "/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n", "/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n", "/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n", "/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n", "/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n", "/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n", "2020-09-23 13:28:31.030552: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1\n", "2020-09-23 13:28:31.045371: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", "2020-09-23 13:28:31.045743: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: \n", "name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705\n", "pciBusID: 0000:01:00.0\n", "2020-09-23 13:28:31.045919: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1\n", "2020-09-23 13:28:31.047219: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10\n", "2020-09-23 13:28:31.048643: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10\n", "2020-09-23 13:28:31.048942: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10\n", "2020-09-23 13:28:31.050474: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10\n", "2020-09-23 13:28:31.051345: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10\n", "2020-09-23 13:28:31.054455: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7\n", "2020-09-23 13:28:31.054595: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", "2020-09-23 13:28:31.055003: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", "2020-09-23 13:28:31.055361: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0\n", "2020-09-23 13:28:31.055618: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA\n", "2020-09-23 13:28:31.304050: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", "2020-09-23 13:28:31.304852: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55c930571740 executing computations on platform CUDA. Devices:\n", "2020-09-23 13:28:31.304915: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): GeForce GTX 1060, Compute Capability 6.1\n", "2020-09-23 13:28:31.305159: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", "2020-09-23 13:28:31.305755: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: \n", "name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705\n", "pciBusID: 0000:01:00.0\n", "2020-09-23 13:28:31.307161: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1\n", "2020-09-23 13:28:31.307187: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10\n", "2020-09-23 13:28:31.307201: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10\n", "2020-09-23 13:28:31.307214: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10\n", "2020-09-23 13:28:31.307227: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10\n", "2020-09-23 13:28:31.307240: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10\n", "2020-09-23 13:28:31.307253: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7\n", "2020-09-23 13:28:31.307324: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", "2020-09-23 13:28:31.308336: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", "2020-09-23 13:28:31.308658: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0\n", "2020-09-23 13:28:31.308749: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1\n", "2020-09-23 13:28:31.309279: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:\n", "2020-09-23 13:28:31.309309: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0 \n", "2020-09-23 13:28:31.309319: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N \n", "2020-09-23 13:28:31.309439: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", "2020-09-23 13:28:31.309818: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", "2020-09-23 13:28:31.310218: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5683 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)\n", "2020-09-23 13:28:31.332882: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2699905000 Hz\n", "2020-09-23 13:28:31.333235: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55c9335dad30 executing computations on platform Host. Devices:\n", "2020-09-23 13:28:31.333256: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): , \n", "WARNING:tensorflow:From /home/bentoml/BentoML/bentoml/frameworks/keras.py:137: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.\n", "\n", "WARNING:tensorflow:From /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/keras/initializers.py:119: calling RandomUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "WARNING:tensorflow:From /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:97: calling GlorotUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "WARNING:tensorflow:From /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "WARNING:tensorflow:From /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:97: calling Orthogonal.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "WARNING:tensorflow:From /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:97: calling Zeros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "WARNING:tensorflow:From /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.where in 2.0, which has the same broadcast rule as np.where\n", "WARNING:tensorflow:From /home/bentoml/BentoML/bentoml/frameworks/keras.py:123: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.\n", "\n", "[2020-09-23 13:28:33,134] WARNING - pip package requirement tensorflow already exist\n", " * Serving Flask app \"KerasTextClassificationService\" (lazy loading)\n", " * Environment: production\n", "\u001b[31m WARNING: This is a development server. Do not use it in a production deployment.\u001b[0m\n", "\u001b[2m Use a production WSGI server instead.\u001b[0m\n", " * Debug mode: off\n", " * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)\n", "2020-09-23 13:28:42.243146: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10\n", "[2020-09-23 13:28:42,414] INFO - {'service_name': 'KerasTextClassificationService', 'service_version': '20200923132753_A6330E', 'api': 'predict', 'task': {'data': '{\"text\": \"best movie ever\"}', 'task_id': '649408a6-f446-4c46-b8aa-ed203b4f5ead', 'http_headers': (('Host', 'localhost:5000'), ('User-Agent', 'curl/7.72.0'), ('Content-Length', '27'), ('Accept', '*/*'), ('Content-Type', 'application/json'))}, 'result': {'data': '0', 'http_status': 200, 'http_headers': (('Content-Type', 'application/json'),)}, 'request_id': '649408a6-f446-4c46-b8aa-ed203b4f5ead'}\n", "127.0.0.1 - - [23/Sep/2020 13:28:42] \"\u001b[37mPOST /predict HTTP/1.1\u001b[0m\" 200 -\n", "[2020-09-23 13:28:44,526] INFO - {'service_name': 'KerasTextClassificationService', 'service_version': '20200923132753_A6330E', 'api': 'predict', 'task': {'data': '{\"text\": \"best movie ever\"}', 'task_id': '4518a10f-ea5b-4e48-b93c-6a263591d691', 'http_headers': (('Host', 'localhost:5000'), ('User-Agent', 'curl/7.72.0'), ('Content-Length', '27'), ('Accept', '*/*'), ('Content-Type', 'application/json'))}, 'result': {'data': '0', 'http_status': 200, 'http_headers': (('Content-Type', 'application/json'),)}, 'request_id': '4518a10f-ea5b-4e48-b93c-6a263591d691'}\n", "127.0.0.1 - - [23/Sep/2020 13:28:44] \"\u001b[37mPOST /predict HTTP/1.1\u001b[0m\" 200 -\n", "^C\n" ] } ], "source": [ "!bentoml serve KerasTextClassificationService:latest" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you are running this notebook from Google Colab, you can start the dev server with `--run-with-ngrok` option, to gain acccess to the API endpoint via a public endpoint managed by [ngrok](https://ngrok.com/):" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!bentoml serve KerasTextClassificationService:latest --run-with-ngrok" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Send prediction request to REST API server\n", "\n", "*Run the following command in terminal to make a HTTP request to the API server*\n", "```bash\n", "curl -i \\\n", "--header \"Content-Type: application/json\" \\\n", "--request POST \\\n", "--data '{\"text\": \"best movie ever\"}' \\\n", "localhost:5000/predict\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Containerize model server with Docker\n", "\n", "\n", "One common way of distributing this model API server for production deployment, is via Docker containers. And BentoML provides a convenient way to do that.\n", "\n", "Note that docker is **not available in Google Colab**. You will need to download and run this notebook locally to try out this containerization with docker feature.\n", "\n", "If you already have docker configured, simply run the follow command to product a docker container serving the IrisClassifier prediction service created above:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[2020-09-23 13:29:13,718] INFO - Getting latest version KerasTextClassificationService:20200923132753_A6330E\n", "\u001b[39mFound Bento: /home/bentoml/bentoml/repository/KerasTextClassificationService/20200923132753_A6330E\u001b[0m\n", "[2020-09-23 13:29:13,736] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.\n", "[2020-09-23 13:29:13,751] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+7.g8af1c8b\n", "Building Docker image kerastextclassificationservice:latest from KerasTextClassificationService:latest \n", "-\u001b[39mStep 1/15 : FROM bentoml/model-server:0.9.0.pre-py36\u001b[0m\n", "\u001b[39m ---> 4aac43d10e50\u001b[0m\n", "\u001b[39mStep 2/15 : ARG EXTRA_PIP_INSTALL_ARGS=\u001b[0m\n", "\u001b[39m ---> Using cache\u001b[0m\n", "\u001b[39m ---> 790054f5ad85\u001b[0m\n", "\u001b[39mStep 3/15 : ENV EXTRA_PIP_INSTALL_ARGS $EXTRA_PIP_INSTALL_ARGS\u001b[0m\n", "\u001b[39m ---> Using cache\u001b[0m\n", "\u001b[39m ---> 85b0a1b40542\u001b[0m\n", "\u001b[39mStep 4/15 : COPY environment.yml requirements.txt setup.sh* bentoml-init.sh python_version* /bento/\u001b[0m\n", "\b|\u001b[39m ---> 7b194657ef63\u001b[0m\n", "\u001b[39mStep 5/15 : WORKDIR /bento\u001b[0m\n", "\u001b[39m ---> Running in 71f0d3c7e9c5\u001b[0m\n", "\b\\\u001b[39m ---> f08b2b76d924\u001b[0m\n", "\u001b[39mStep 6/15 : RUN chmod +x /bento/bentoml-init.sh\u001b[0m\n", "\u001b[39m ---> Running in 04750f5c4d86\u001b[0m\n", "\b|\u001b[39m ---> bdc789e68d2d\u001b[0m\n", "\u001b[39mStep 7/15 : RUN if [ -f /bento/bentoml-init.sh ]; then bash -c /bento/bentoml-init.sh; fi\u001b[0m\n", "\u001b[39m ---> Running in 8b708da945a7\u001b[0m\n", "\b|\u001b[39m\u001b[91m+++ dirname /bento/bentoml-init.sh\n", "\u001b[0m\u001b[0m\n", "\u001b[39m\u001b[91m++ cd /bento\n", "++ pwd -P\n", "\u001b[0m\u001b[0m\n", "\u001b[39m\u001b[91m+ SAVED_BUNDLE_PATH=/bento\n", "+ cd /bento\n", "+ '[' -f ./setup.sh ']'\n", "+ '[' -f ./python_version ']'\n", "\u001b[0m\u001b[0m\n", "\u001b[39m\u001b[91m++ cat ./python_version\n", "\u001b[0m\u001b[0m\n", "\u001b[39m\u001b[91m+ PY_VERSION_SAVED=3.6.10\n", "+ DESIRED_PY_VERSION=3.6\n", "\u001b[0m\u001b[0m\n", "\u001b[39m\u001b[91m++ python -c 'import sys; print(f\"{sys.version_info.major}.{sys.version_info.minor}\")'\n", "\u001b[0m\u001b[0m\n", "\u001b[39m\u001b[91m+ CURRENT_PY_VERSION=3.6\n", "+ [[ 3.6 == \\3\\.\\6 ]]\n", "+ echo 'Python Version in docker base image 3.6 matches requirement python=3.6. Skipping.'\n", "\u001b[0m\u001b[0m\n", "\u001b[39mPython Version in docker base image 3.6 matches requirement python=3.6. Skipping.\u001b[0m\n", "\u001b[39m\u001b[91m+ command -v conda\n", "+ echo 'Updating conda base environment with environment.yml'\n", "+ conda env update -n base -f ./environment.yml\n", "\u001b[0m\u001b[0m\n", "\u001b[39mUpdating conda base environment with environment.yml\u001b[0m\n", "\b|\u001b[39mCollecting package metadata (repodata.json): ...working... \u001b[0m\n", "\b\\\u001b[39mdone\n", "Solving environment: ...working... \u001b[0m\n", "\b\\\u001b[39mdone\u001b[0m\n", "\u001b[39m\n", "Downloading and Extracting Packages\n", "python_abi-3.6 | 4 KB | | 0% \u001b[0m\n", "python_abi-3.6 | 4 KB | ########## | 100% \u001b[0m\n", "python_abi-3.6 | 4 KB | ########## | 100% \u001b[0m\n", "\u001b[39m\n", "certifi-2020.6.20 | 151 KB | | 0% \u001b[0m\n", "certifi-2020.6.20 | 151 KB | # | 11% \u001b[0m\n", "certifi-2020.6.20 | 151 KB | ####2 | 42% \u001b[0m\n", "certifi-2020.6.20 | 151 KB | #######4 | 74% \u001b[0m\n", "certifi-2020.6.20 | 151 KB | #########5 | 95% \u001b[0m\n", "certifi-2020.6.20 | 151 KB | ########## | 100% \u001b[0m\n", "\u001b[39m\n", "pip-20.2.3 | 1.1 MB | | 0% \u001b[0m\n", "pip-20.2.3 | 1.1 MB | 1 | 1% \u001b[0m\n", "pip-20.2.3 | 1.1 MB | 4 | 4% \u001b[0m\n", "pip-20.2.3 | 1.1 MB | 5 | 6% \u001b[0m\n", "pip-20.2.3 | 1.1 MB | 8 | 9% \u001b[0m\n", "pip-20.2.3 | 1.1 MB | # | 10% \u001b[0m\n", "pip-20.2.3 | 1.1 MB | ## | 20% \u001b[0m\n", "pip-20.2.3 | 1.1 MB | ### | 31% \u001b[0m\n", "pip-20.2.3 | 1.1 MB | ###5 | 35% \u001b[0m\n", "pip-20.2.3 | 1.1 MB | ###7 | 38% \u001b[0m\n", "pip-20.2.3 | 1.1 MB | ####5 | 45% \u001b[0m\n", "pip-20.2.3 | 1.1 MB | ####9 | 50% \u001b[0m\n", "pip-20.2.3 | 1.1 MB | #####5 | 56% \u001b[0m\n", "pip-20.2.3 | 1.1 MB | ######2 | 63% \u001b[0m\n", "pip-20.2.3 | 1.1 MB | ######5 | 66% \u001b[0m\n", "pip-20.2.3 | 1.1 MB | ######8 | 69% \u001b[0m\n", "pip-20.2.3 | 1.1 MB | #######5 | 76% \u001b[0m\n", "pip-20.2.3 | 1.1 MB | ########1 | 82% \u001b[0m\n", "pip-20.2.3 | 1.1 MB | ########4 | 85% \u001b[0m\n", "pip-20.2.3 | 1.1 MB | ########7 | 88% \u001b[0m\n", "pip-20.2.3 | 1.1 MB | #########2 | 92% \u001b[0m\n", "pip-20.2.3 | 1.1 MB | #########4 | 95% \u001b[0m\n", "pip-20.2.3 | 1.1 MB | #########6 | 96% \u001b[0m\n", "pip-20.2.3 | 1.1 MB | ########## | 100% \u001b[0m\n", "pip-20.2.3 | 1.1 MB | ########## | 100% \u001b[0m\n", "\u001b[39m\n", "ca-certificates-2020 | 145 KB | | 0% \u001b[0m\n", "ca-certificates-2020 | 145 KB | #1 | 11% \u001b[0m\n", "ca-certificates-2020 | 145 KB | ####4 | 44% \u001b[0m\n", "ca-certificates-2020 | 145 KB | ########8 | 88% \u001b[0m\n", "ca-certificates-2020 | 145 KB | ########## | 100% \u001b[0m\n", "\u001b[39m\n", "openssl-1.1.1h | 2.1 MB | | 0% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | | 1% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | 2 | 2% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | 2 | 3% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | 7 | 7% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | 8 | 9% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | 9 | 10% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | #3 | 13% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | #6 | 17% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | #8 | 18% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | #9 | 20% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ##2 | 22% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ##4 | 24% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ##8 | 29% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ### | 30% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ###1 | 32% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ###3 | 33% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ###4 | 35% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ###5 | 35% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ###6 | 36% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | #### | 41% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ####1 | 42% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ####6 | 46% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ####7 | 48% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ####9 | 49% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ##### | 51% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | #####2 | 52% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | #####5 | 55% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | #####8 | 58% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | #####9 | 60% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ###### | 60% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ######1 | 61% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ######2 | 63% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ######4 | 65% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ######5 | 66% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ######8 | 69% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ######9 | 70% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | #######1 | 71% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | #######2 | 73% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | #######4 | 74% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | #######5 | 75% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | #######9 | 80% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ######## | 80% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ########1 | 81% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ########3 | 83% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ########4 | 85% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ########5 | 85% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ########7 | 88% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ########8 | 88% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ########9 | 90% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | #########1 | 91% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | #########5 | 95% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | #########5 | 96% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | #########9 | 99% \u001b[0m\n", "openssl-1.1.1h | 2.1 MB | ########## | 100% \u001b[0m\n", "\u001b[39m\n", "Preparing transaction: ...working... \u001b[0m\n", "\b\\\u001b[39mdone\u001b[0m\n", "\u001b[39mVerifying transaction: ...working... \u001b[0m\n", "\u001b[39mdone\u001b[0m\n", "\u001b[39mExecuting transaction: ...working... \u001b[0m\n", "\b/\u001b[39mdone\u001b[0m\n", "\b|\u001b[39m#\n", "# To activate this environment, use\n", "#\n", "# $ conda activate base\n", "#\n", "# To deactivate an active environment, use\n", "#\n", "# $ conda deactivate\u001b[0m\n", "\b\\\u001b[39m\u001b[91m+ pip install -r ./requirements.txt --no-cache-dir\n", "\u001b[0m\u001b[0m\n", "\b\\\u001b[39mRequirement already satisfied: bentoml==0.9.0.pre in /opt/conda/lib/python3.6/site-packages (from -r ./requirements.txt (line 1)) (0.9.0rc0)\u001b[0m\n", "\b|\u001b[39mCollecting tensorflow==1.14.0\u001b[0m\n", "\b/\u001b[39m Downloading tensorflow-1.14.0-cp36-cp36m-manylinux1_x86_64.whl (109.2 MB)\u001b[0m\n", "\b-\u001b[39mRequirement already satisfied: numpy==1.19.2 in /opt/conda/lib/python3.6/site-packages (from -r ./requirements.txt (line 3)) (1.19.2)\u001b[0m\n", "\b/\u001b[39mCollecting pandas==1.0.3\u001b[0m\n", "\b|\u001b[39m Downloading pandas-1.0.3-cp36-cp36m-manylinux1_x86_64.whl (10.0 MB)\u001b[0m\n", "\b|\u001b[39mRequirement already satisfied: contextvars; python_version < \"3.7\" in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.4)\u001b[0m\n", "\u001b[39mRequirement already satisfied: psutil in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (5.7.2)\u001b[0m\n", "\u001b[39mRequirement already satisfied: prometheus-client in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.8.0)\u001b[0m\n", "\u001b[39mRequirement already satisfied: dataclasses; python_version < \"3.7\" in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.7)\u001b[0m\n", "\u001b[39mRequirement already satisfied: click>=7.0 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (7.1.2)\u001b[0m\n", "\u001b[39mRequirement already satisfied: py-zipkin in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.20.0)\u001b[0m\n", "\u001b[39mRequirement already satisfied: certifi in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2020.6.20)\n", "Requirement already satisfied: sqlalchemy-utils<0.36.8 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.36.7)\u001b[0m\n", "\b\\\u001b[39mRequirement already satisfied: aiohttp in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.6.2)\u001b[0m\n", "\u001b[39mRequirement already satisfied: packaging in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.4)\u001b[0m\n", "\u001b[39mRequirement already satisfied: docker in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (4.3.1)\u001b[0m\n", "\u001b[39mRequirement already satisfied: cerberus in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.3.2)\u001b[0m\n", "\u001b[39mRequirement already satisfied: humanfriendly in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (8.2)\u001b[0m\n", "\u001b[39mRequirement already satisfied: alembic in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.4.3)\u001b[0m\n", "\u001b[39mRequirement already satisfied: python-dateutil<3.0.0,>=2.7.3 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.8.1)\u001b[0m\n", "\u001b[39mRequirement already satisfied: tabulate in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.8.7)\u001b[0m\n", "\u001b[39mRequirement already satisfied: boto3 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.15.2)\u001b[0m\n", "\u001b[39mRequirement already satisfied: sqlalchemy>=1.3.0 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.3.19)\u001b[0m\n", "\b-\u001b[39mRequirement already satisfied: multidict in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (4.7.6)\u001b[0m\n", "\u001b[39mRequirement already satisfied: flask in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.2)\u001b[0m\n", "\u001b[39mRequirement already satisfied: python-json-logger in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.1.11)\u001b[0m\n", "\u001b[39mRequirement already satisfied: requests in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.24.0)\u001b[0m\n", "\u001b[39mRequirement already satisfied: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.15.87)\n", "Requirement already satisfied: grpcio<=1.27.2 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.27.2)\u001b[0m\n", "\u001b[39mRequirement already satisfied: configparser in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (5.0.0)\u001b[0m\n", "\u001b[39mRequirement already satisfied: gunicorn in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.0.4)\u001b[0m\n", "\u001b[39mRequirement already satisfied: protobuf>=3.6.0 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.13.0)\u001b[0m\n", "\b/\u001b[39mCollecting keras-applications>=1.0.6\u001b[0m\n", "\b|\u001b[39m Downloading Keras_Applications-1.0.8-py3-none-any.whl (50 kB)\u001b[0m\n", "\b-\u001b[39mCollecting termcolor>=1.1.0\u001b[0m\n", "\b|\u001b[39m Downloading termcolor-1.1.0.tar.gz (3.9 kB)\u001b[0m\n", "\b|\u001b[39mCollecting google-pasta>=0.1.6\u001b[0m\n", "\b\\\u001b[39m Downloading google_pasta-0.2.0-py3-none-any.whl (57 kB)\u001b[0m\n", "\b/\u001b[39mCollecting tensorboard<1.15.0,>=1.14.0\u001b[0m\n", "\b|\u001b[39m Downloading tensorboard-1.14.0-py3-none-any.whl (3.1 MB)\u001b[0m\n", "\b/\u001b[39mCollecting astor>=0.6.0\u001b[0m\n", "\b|\u001b[39m Downloading astor-0.8.1-py2.py3-none-any.whl (27 kB)\u001b[0m\n", "\b/\u001b[39mCollecting absl-py>=0.7.0\u001b[0m\n", "\b|\u001b[39m Downloading absl_py-0.10.0-py3-none-any.whl (127 kB)\u001b[0m\n", "\b\\\u001b[39mCollecting keras-preprocessing>=1.0.5\u001b[0m\n", "\b-\u001b[39m Downloading Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB)\u001b[0m\n", "\b/\u001b[39mRequirement already satisfied: wheel>=0.26 in /opt/conda/lib/python3.6/site-packages (from tensorflow==1.14.0->-r ./requirements.txt (line 2)) (0.35.1)\u001b[0m\n", "\b\\\u001b[39mCollecting tensorflow-estimator<1.15.0rc0,>=1.14.0rc0\u001b[0m\n", "\b-\u001b[39m Downloading tensorflow_estimator-1.14.0-py2.py3-none-any.whl (488 kB)\u001b[0m\n", "\b/\u001b[39mRequirement already satisfied: six>=1.10.0 in /opt/conda/lib/python3.6/site-packages (from tensorflow==1.14.0->-r ./requirements.txt (line 2)) (1.15.0)\u001b[0m\n", "\b|\u001b[39mCollecting wrapt>=1.11.1\u001b[0m\n", "\b-\u001b[39m Downloading wrapt-1.12.1.tar.gz (27 kB)\u001b[0m\n", "\b-\u001b[39mCollecting gast>=0.2.0\u001b[0m\n", "\b/\u001b[39m Downloading gast-0.4.0-py3-none-any.whl (9.8 kB)\u001b[0m\n", "\b-\u001b[39mCollecting pytz>=2017.2\u001b[0m\n", "\b/\u001b[39m Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB)\u001b[0m\n", "\b/\u001b[39mRequirement already satisfied: immutables>=0.9 in /opt/conda/lib/python3.6/site-packages (from contextvars; python_version < \"3.7\"->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.14)\u001b[0m\n", "\u001b[39mRequirement already satisfied: thriftpy2>=0.4.0 in /opt/conda/lib/python3.6/site-packages (from py-zipkin->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.4.11)\u001b[0m\n", "\u001b[39mRequirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.5.1)\u001b[0m\n", "\u001b[39mRequirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.2.0)\u001b[0m\n", "\b|\u001b[39mRequirement already satisfied: idna-ssl>=1.0; python_version < \"3.7\" in /opt/conda/lib/python3.6/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.0)\u001b[0m\n", "\u001b[39mRequirement already satisfied: typing-extensions>=3.6.5; python_version < \"3.7\" in /opt/conda/lib/python3.6/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.7.4.3)\u001b[0m\n", "\u001b[39mRequirement already satisfied: async-timeout<4.0,>=3.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.0.1)\u001b[0m\n", "\u001b[39mRequirement already satisfied: chardet<4.0,>=2.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.0.4)\u001b[0m\n", "\u001b[39mRequirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.6/site-packages (from packaging->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.4.7)\u001b[0m\n", "\u001b[39mRequirement already satisfied: websocket-client>=0.32.0 in /opt/conda/lib/python3.6/site-packages (from docker->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.57.0)\u001b[0m\n", "\u001b[39mRequirement already satisfied: setuptools in /opt/conda/lib/python3.6/site-packages (from cerberus->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (49.6.0.post20200814)\u001b[0m\n", "\u001b[39mRequirement already satisfied: python-editor>=0.3 in /opt/conda/lib/python3.6/site-packages (from alembic->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.0.4)\u001b[0m\n", "\u001b[39mRequirement already satisfied: Mako in /opt/conda/lib/python3.6/site-packages (from alembic->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.3)\u001b[0m\n", "\u001b[39mRequirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /opt/conda/lib/python3.6/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.3.3)\u001b[0m\n", "\u001b[39mRequirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.6/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.10.0)\u001b[0m\n", "\u001b[39mRequirement already satisfied: botocore<1.19.0,>=1.18.2 in /opt/conda/lib/python3.6/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.18.2)\u001b[0m\n", "\u001b[39mRequirement already satisfied: Jinja2>=2.10.1 in /opt/conda/lib/python3.6/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.11.2)\u001b[0m\n", "\u001b[39mRequirement already satisfied: itsdangerous>=0.24 in /opt/conda/lib/python3.6/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.0)\u001b[0m\n", "\u001b[39mRequirement already satisfied: Werkzeug>=0.15 in /opt/conda/lib/python3.6/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.0.1)\u001b[0m\n", "\u001b[39mRequirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.6/site-packages (from requests->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.10)\u001b[0m\n", "\u001b[39mRequirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.6/site-packages (from requests->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.25.10)\u001b[0m\n", "\b-\u001b[39mCollecting h5py\u001b[0m\n", "\b/\u001b[39m Downloading h5py-2.10.0-cp36-cp36m-manylinux1_x86_64.whl (2.9 MB)\u001b[0m\n", "\b/\u001b[39mCollecting markdown>=2.6.8\u001b[0m\n", "\b|\u001b[39m Downloading Markdown-3.2.2-py3-none-any.whl (88 kB)\u001b[0m\n", "\b-\u001b[39mRequirement already satisfied: ply<4.0,>=3.4 in /opt/conda/lib/python3.6/site-packages (from thriftpy2>=0.4.0->py-zipkin->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.11)\u001b[0m\n", "\u001b[39mRequirement already satisfied: MarkupSafe>=0.9.2 in /opt/conda/lib/python3.6/site-packages (from Mako->alembic->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.1)\u001b[0m\n", "\b|\u001b[39mCollecting importlib-metadata; python_version < \"3.8\"\u001b[0m\n", "\b\\\u001b[39m Downloading importlib_metadata-2.0.0-py2.py3-none-any.whl (31 kB)\u001b[0m\n", "\b/\u001b[39mCollecting zipp>=0.5\u001b[0m\n", "\b\\\u001b[39m Downloading zipp-3.2.0-py3-none-any.whl (5.1 kB)\u001b[0m\n", "\u001b[39mBuilding wheels for collected packages: termcolor, wrapt\u001b[0m\n", "\u001b[39m Building wheel for termcolor (setup.py): started\u001b[0m\n", "\b|\u001b[39m Building wheel for termcolor (setup.py): finished with status 'done'\u001b[0m\n", "\u001b[39m Created wheel for termcolor: filename=termcolor-1.1.0-py3-none-any.whl size=4830 sha256=536e701b6ff11f70e0ddd33ebc4e067534831a257c7f2049439841cc7dd459dd\u001b[0m\n", "\u001b[39m Stored in directory: /tmp/pip-ephem-wheel-cache-pibtyjg5/wheels/93/2a/eb/e58dbcbc963549ee4f065ff80a59f274cc7210b6eab962acdc\u001b[0m\n", "\u001b[39m Building wheel for wrapt (setup.py): started\u001b[0m\n", "\b|\u001b[39m Building wheel for wrapt (setup.py): finished with status 'done'\u001b[0m\n", "\u001b[39m Created wheel for wrapt: filename=wrapt-1.12.1-cp36-cp36m-linux_x86_64.whl size=75219 sha256=959d6882e2c26a483447c83bfa95658228c224ddd243c611884f063aaf5111e7\u001b[0m\n", "\u001b[39m Stored in directory: /tmp/pip-ephem-wheel-cache-pibtyjg5/wheels/32/42/7f/23cae9ff6ef66798d00dc5d659088e57dbba01566f6c60db63\u001b[0m\n", "\u001b[39mSuccessfully built termcolor wrapt\u001b[0m\n", "\b-\u001b[39mInstalling collected packages: h5py, keras-applications, termcolor, google-pasta, absl-py, zipp, importlib-metadata, markdown, tensorboard, astor, keras-preprocessing, tensorflow-estimator, wrapt, gast, tensorflow, pytz, pandas\u001b[0m\n", "\b\\\u001b[39mSuccessfully installed absl-py-0.10.0 astor-0.8.1 gast-0.4.0 google-pasta-0.2.0 h5py-2.10.0 importlib-metadata-2.0.0 keras-applications-1.0.8 keras-preprocessing-1.1.2 markdown-3.2.2 pandas-1.0.3 pytz-2020.1 tensorboard-1.14.0 tensorflow-1.14.0 tensorflow-estimator-1.14.0 termcolor-1.1.0 wrapt-1.12.1 zipp-3.2.0\u001b[0m\n", "\b\\\u001b[39m ---> 1dd84b19fec2\u001b[0m\n", "\u001b[39mStep 8/15 : COPY . /bento\u001b[0m\n", "\b-\u001b[39m ---> 4909012adc5d\u001b[0m\n", "\u001b[39mStep 9/15 : RUN if [ -d /bento/bundled_pip_dependencies ]; then pip install -U bundled_pip_dependencies/* ;fi\u001b[0m\n", "\b/\u001b[39m ---> Running in a2e2d71e0735\u001b[0m\n", "\b/\u001b[39mProcessing ./bundled_pip_dependencies/BentoML-0.9.0rc0+7.g8af1c8b.tar.gz\u001b[0m\n", "\b|\u001b[39m Installing build dependencies: started\u001b[0m\n", "\b/\u001b[39m Installing build dependencies: finished with status 'done'\u001b[0m\n", "\u001b[39m Getting requirements to build wheel: started\u001b[0m\n", "\b/\u001b[39m Getting requirements to build wheel: finished with status 'done'\u001b[0m\n", "\u001b[39m Preparing wheel metadata: started\u001b[0m\n", "\b|\u001b[39m Preparing wheel metadata: finished with status 'done'\u001b[0m\n", "\b-\u001b[39mRequirement already satisfied, skipping upgrade: python-dateutil<3.0.0,>=2.7.3 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (2.8.1)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: aiohttp in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (3.6.2)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: flask in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.1.2)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: docker in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (4.3.1)\u001b[0m\n", "\b/\u001b[39mRequirement already satisfied, skipping upgrade: boto3 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.15.2)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: gunicorn in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (20.0.4)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: python-json-logger in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.1.11)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: certifi in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (2020.6.20)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: py-zipkin in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.20.0)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: configparser in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (5.0.0)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: contextvars; python_version < \"3.7\" in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (2.4)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: dataclasses; python_version < \"3.7\" in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.7)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: protobuf>=3.6.0 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (3.13.0)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: sqlalchemy-utils<0.36.8 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.36.7)\u001b[0m\n", "\b|\u001b[39mRequirement already satisfied, skipping upgrade: psutil in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (5.7.2)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.15.87)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: alembic in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.4.3)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: cerberus in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.3.2)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: requests in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (2.24.0)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: tabulate in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.8.7)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: grpcio<=1.27.2 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.27.2)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: numpy in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.19.2)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: prometheus-client in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.8.0)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: humanfriendly in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (8.2)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: click>=7.0 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (7.1.2)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: multidict in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (4.7.6)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: packaging in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (20.4)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: sqlalchemy>=1.3.0 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.3.19)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: six>=1.5 in /opt/conda/lib/python3.6/site-packages (from python-dateutil<3.0.0,>=2.7.3->BentoML==0.9.0rc0+7.g8af1c8b) (1.15.0)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: chardet<4.0,>=2.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->BentoML==0.9.0rc0+7.g8af1c8b) (3.0.4)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: typing-extensions>=3.6.5; python_version < \"3.7\" in /opt/conda/lib/python3.6/site-packages (from aiohttp->BentoML==0.9.0rc0+7.g8af1c8b) (3.7.4.3)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: idna-ssl>=1.0; python_version < \"3.7\" in /opt/conda/lib/python3.6/site-packages (from aiohttp->BentoML==0.9.0rc0+7.g8af1c8b) (1.1.0)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: yarl<2.0,>=1.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->BentoML==0.9.0rc0+7.g8af1c8b) (1.5.1)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: attrs>=17.3.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->BentoML==0.9.0rc0+7.g8af1c8b) (20.2.0)\u001b[0m\n", "\b\\\u001b[39mRequirement already satisfied, skipping upgrade: async-timeout<4.0,>=3.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->BentoML==0.9.0rc0+7.g8af1c8b) (3.0.1)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: itsdangerous>=0.24 in /opt/conda/lib/python3.6/site-packages (from flask->BentoML==0.9.0rc0+7.g8af1c8b) (1.1.0)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: Werkzeug>=0.15 in /opt/conda/lib/python3.6/site-packages (from flask->BentoML==0.9.0rc0+7.g8af1c8b) (1.0.1)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: Jinja2>=2.10.1 in /opt/conda/lib/python3.6/site-packages (from flask->BentoML==0.9.0rc0+7.g8af1c8b) (2.11.2)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: websocket-client>=0.32.0 in /opt/conda/lib/python3.6/site-packages (from docker->BentoML==0.9.0rc0+7.g8af1c8b) (0.57.0)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: botocore<1.19.0,>=1.18.2 in /opt/conda/lib/python3.6/site-packages (from boto3->BentoML==0.9.0rc0+7.g8af1c8b) (1.18.2)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: s3transfer<0.4.0,>=0.3.0 in /opt/conda/lib/python3.6/site-packages (from boto3->BentoML==0.9.0rc0+7.g8af1c8b) (0.3.3)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.6/site-packages (from boto3->BentoML==0.9.0rc0+7.g8af1c8b) (0.10.0)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: setuptools>=3.0 in /opt/conda/lib/python3.6/site-packages (from gunicorn->BentoML==0.9.0rc0+7.g8af1c8b) (49.6.0.post20200814)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: thriftpy2>=0.4.0 in /opt/conda/lib/python3.6/site-packages (from py-zipkin->BentoML==0.9.0rc0+7.g8af1c8b) (0.4.11)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: immutables>=0.9 in /opt/conda/lib/python3.6/site-packages (from contextvars; python_version < \"3.7\"->BentoML==0.9.0rc0+7.g8af1c8b) (0.14)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: Mako in /opt/conda/lib/python3.6/site-packages (from alembic->BentoML==0.9.0rc0+7.g8af1c8b) (1.1.3)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: python-editor>=0.3 in /opt/conda/lib/python3.6/site-packages (from alembic->BentoML==0.9.0rc0+7.g8af1c8b) (1.0.4)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: idna<3,>=2.5 in /opt/conda/lib/python3.6/site-packages (from requests->BentoML==0.9.0rc0+7.g8af1c8b) (2.10)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.6/site-packages (from requests->BentoML==0.9.0rc0+7.g8af1c8b) (1.25.10)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /opt/conda/lib/python3.6/site-packages (from packaging->BentoML==0.9.0rc0+7.g8af1c8b) (2.4.7)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: MarkupSafe>=0.23 in /opt/conda/lib/python3.6/site-packages (from Jinja2>=2.10.1->flask->BentoML==0.9.0rc0+7.g8af1c8b) (1.1.1)\u001b[0m\n", "\u001b[39mRequirement already satisfied, skipping upgrade: ply<4.0,>=3.4 in /opt/conda/lib/python3.6/site-packages (from thriftpy2>=0.4.0->py-zipkin->BentoML==0.9.0rc0+7.g8af1c8b) (3.11)\u001b[0m\n", "\u001b[39mBuilding wheels for collected packages: BentoML\u001b[0m\n", "\u001b[39m Building wheel for BentoML (PEP 517): started\u001b[0m\n", "\b|\u001b[39m Building wheel for BentoML (PEP 517): finished with status 'done'\u001b[0m\n", "\u001b[39m Created wheel for BentoML: filename=BentoML-0.9.0rc0+7.g8af1c8b-py3-none-any.whl size=620483 sha256=278295e2f90a947339804c628cc6ee92d0aff94b14ef66479cca623c34ef7a3d\u001b[0m\n", "\u001b[39m Stored in directory: /root/.cache/pip/wheels/0f/c2/e2/64640eb406a012df3abf685e4e5fa441b87b3e609903d041b6\u001b[0m\n", "\u001b[39mSuccessfully built BentoML\u001b[0m\n", "\b\\\u001b[39mInstalling collected packages: BentoML\u001b[0m\n", "\u001b[39m Attempting uninstall: BentoML\u001b[0m\n", "\u001b[39m Found existing installation: BentoML 0.9.0rc0\u001b[0m\n", "\b-\u001b[39m Uninstalling BentoML-0.9.0rc0:\u001b[0m\n", "\b/\u001b[39m Successfully uninstalled BentoML-0.9.0rc0\u001b[0m\n", "\b-\u001b[39mSuccessfully installed BentoML-0.9.0rc0+7.g8af1c8b\u001b[0m\n", "\b-\u001b[39m ---> 6d8fe8c65a1b\u001b[0m\n", "\u001b[39mStep 10/15 : ENV PORT 5000\u001b[0m\n", "\u001b[39m ---> Running in 578f2c6e0fc1\u001b[0m\n", "\b/\u001b[39m ---> 79ea1f81da3c\u001b[0m\n", "\u001b[39mStep 11/15 : EXPOSE $PORT\u001b[0m\n", "\u001b[39m ---> Running in d91152d5d6ed\u001b[0m\n", "\b|\u001b[39m ---> 13da05012ec1\u001b[0m\n", "\u001b[39mStep 12/15 : COPY docker-entrypoint.sh /usr/local/bin/\u001b[0m\n", "\b\\\u001b[39m ---> 25bbd2879490\u001b[0m\n", "\u001b[39mStep 13/15 : RUN chmod +x /usr/local/bin/docker-entrypoint.sh\u001b[0m\n", "\u001b[39m ---> Running in ea1faa2ef27d\u001b[0m\n", "\b\\\u001b[39m ---> f73d9449c4b3\u001b[0m\n", "\u001b[39mStep 14/15 : ENTRYPOINT [ \"docker-entrypoint.sh\" ]\u001b[0m\n", "\u001b[39m ---> Running in fabf2c37aa1c\u001b[0m\n", "\u001b[39m ---> 7f57e8360238\u001b[0m\n", "\u001b[39mStep 15/15 : CMD [\"bentoml\", \"serve-gunicorn\", \"/bento\"]\u001b[0m\n", "\u001b[39m ---> Running in 492cc9d6469f\u001b[0m\n", "\b-\u001b[39m ---> 3a09b42bd49f\u001b[0m\n", "\u001b[39mSuccessfully built 3a09b42bd49f\u001b[0m\n", "\u001b[39mSuccessfully tagged kerastextclassificationservice:latest\u001b[0m\n", "\u001b[32mFinished building kerastextclassificationservice:latest from KerasTextClassificationService:latest\u001b[0m\n" ] } ], "source": [ "!bentoml containerize KerasTextClassificationService:latest -t kerastextclassificationservice:latest" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[2020-09-23 05:36:30,701] INFO - Starting BentoML API server in production mode..\n", "[2020-09-23 05:36:30,921] INFO - get_gunicorn_num_of_workers: 3, calculated by cpu count\n", "[2020-09-23 05:36:30 +0000] [1] [INFO] Starting gunicorn 20.0.4\n", "[2020-09-23 05:36:30 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)\n", "[2020-09-23 05:36:30 +0000] [1] [INFO] Using worker: sync\n", "[2020-09-23 05:36:30 +0000] [13] [INFO] Booting worker with pid: 13\n", "[2020-09-23 05:36:30 +0000] [14] [INFO] Booting worker with pid: 14\n", "[2020-09-23 05:36:31 +0000] [15] [INFO] Booting worker with pid: 15\n", "[2020-09-23 05:36:31,942] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/bentoml.git@{branch}'\n", "[2020-09-23 05:36:31,983] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/bentoml.git@{branch}'\n", "[2020-09-23 05:36:31,986] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+7.g8af1c8b\n", "[2020-09-23 05:36:31,986] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.6.10, but current environment version is 3.6.12.\n", "[2020-09-23 05:36:32,004] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+7.g8af1c8b\n", "[2020-09-23 05:36:32,004] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.6.10, but current environment version is 3.6.12.\n", "[2020-09-23 05:36:32,032] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/bentoml.git@{branch}'\n", "[2020-09-23 05:36:32,051] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+7.g8af1c8b\n", "[2020-09-23 05:36:32,052] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.6.10, but current environment version is 3.6.12.\n", "2020-09-23 05:36:33.325834: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\n", "2020-09-23 05:36:33.328236: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\n", "2020-09-23 05:36:33.331867: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2699905000 Hz\n", "2020-09-23 05:36:33.332062: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55ebdc298780 executing computations on platform Host. Devices:\n", "2020-09-23 05:36:33.332079: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): , \n", "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n", "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/bentoml/frameworks/keras.py:137: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.\n", "\n", "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/keras/initializers.py:119: calling RandomUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "2020-09-23 05:36:33.346211: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\n", "2020-09-23 05:36:33.348862: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2699905000 Hz\n", "2020-09-23 05:36:33.349103: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55ebdc297ac0 executing computations on platform Host. Devices:\n", "2020-09-23 05:36:33.349129: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): , \n", "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n", "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/bentoml/frameworks/keras.py:137: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.\n", "\n", "2020-09-23 05:36:33.350851: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2699905000 Hz\n", "2020-09-23 05:36:33.351168: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55ebdc298bf0 executing computations on platform Host. Devices:\n", "2020-09-23 05:36:33.351349: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): , \n", "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n", "/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", " np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n", "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/bentoml/frameworks/keras.py:137: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.\n", "\n", "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/keras/initializers.py:119: calling RandomUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/keras/initializers.py:119: calling RandomUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:97: calling GlorotUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:97: calling Orthogonal.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:97: calling Zeros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:97: calling GlorotUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:97: calling Orthogonal.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:97: calling GlorotUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:97: calling Zeros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:97: calling Orthogonal.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:97: calling Zeros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "2020-09-23 05:36:33.895032: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.\n", "2020-09-23 05:36:33.899674: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.\n", "2020-09-23 05:36:33.902993: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.\n", "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.where in 2.0, which has the same broadcast rule as np.where\n", "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.where in 2.0, which has the same broadcast rule as np.where\n", "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.where in 2.0, which has the same broadcast rule as np.where\n", "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/bentoml/frameworks/keras.py:123: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.\n", "\n", "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/bentoml/frameworks/keras.py:123: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.\n", "\n", "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/bentoml/frameworks/keras.py:123: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.\n", "\n", "[2020-09-23 05:36:35,219] WARNING - pip package requirement tensorflow already exist\n", "[2020-09-23 05:36:35,227] WARNING - pip package requirement tensorflow already exist\n", "[2020-09-23 05:36:35,229] WARNING - pip package requirement tensorflow already exist\n", "[2020-09-23 05:36:44,045] INFO - {'service_name': 'KerasTextClassificationService', 'service_version': '20200923132753_A6330E', 'api': 'predict', 'task': {'data': '{\"text\": \"best movie ever\"}', 'task_id': '55eb695b-eb0d-4aae-98e7-399eafdf9000', 'http_headers': (('Host', 'localhost:5000'), ('User-Agent', 'curl/7.72.0'), ('Content-Length', '27'), ('Accept', '*/*'), ('Content-Type', 'application/json'))}, 'result': {'data': '0', 'http_status': 200, 'http_headers': (('Content-Type', 'application/json'),)}, 'request_id': '55eb695b-eb0d-4aae-98e7-399eafdf9000'}\n", "^C\n", "[2020-09-23 05:37:08 +0000] [1] [INFO] Handling signal: int\n", "[2020-09-23 05:37:08 +0000] [14] [INFO] Worker exiting (pid: 14)\n", "[2020-09-23 05:37:08 +0000] [15] [INFO] Worker exiting (pid: 15)\n", "[2020-09-23 05:37:08 +0000] [13] [INFO] Worker exiting (pid: 13)\n" ] } ], "source": [ "!docker run -p 5000:5000 kerastextclassificationservice" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Launch inference job from CLI\n", "\n", "BentoML cli supports loading and running a packaged model from CLI. With the DataframeInput adapter, the CLI command supports reading input Dataframe data from CLI argument or local csv or json files:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[2020-08-04 12:54:48,270] INFO - Getting latest version KerasTextClassificationService:20200804125206_4639A5\n", "[2020-08-04 12:54:49,229] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.\n", "[2020-08-04 12:54:49,260] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.8.3, but loading from BentoML version 0.8.3+47.g7af5a8f\n", "2020-08-04 12:54:50.476102: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1\n", "2020-08-04 12:54:50.491479: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", "2020-08-04 12:54:50.491867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: \n", "name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705\n", "pciBusID: 0000:01:00.0\n", "2020-08-04 12:54:50.492035: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1\n", "2020-08-04 12:54:50.493344: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10\n", "2020-08-04 12:54:50.494672: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10\n", "2020-08-04 12:54:50.494922: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10\n", "2020-08-04 12:54:50.496422: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10\n", "2020-08-04 12:54:50.497297: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10\n", "2020-08-04 12:54:50.500471: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7\n", "2020-08-04 12:54:50.500639: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", "2020-08-04 12:54:50.501045: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", "2020-08-04 12:54:50.502606: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0\n", "2020-08-04 12:54:50.502938: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA\n", "2020-08-04 12:54:50.528646: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2699905000 Hz\n", "2020-08-04 12:54:50.529115: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55b8f266e630 executing computations on platform Host. Devices:\n", "2020-08-04 12:54:50.529144: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): , \n", "2020-08-04 12:54:50.529324: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", "2020-08-04 12:54:50.529758: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: \n", "name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705\n", "pciBusID: 0000:01:00.0\n", "2020-08-04 12:54:50.529812: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1\n", "2020-08-04 12:54:50.529840: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10\n", "2020-08-04 12:54:50.529854: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10\n", "2020-08-04 12:54:50.529891: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10\n", "2020-08-04 12:54:50.529904: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10\n", "2020-08-04 12:54:50.529941: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10\n", "2020-08-04 12:54:50.529955: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7\n", "2020-08-04 12:54:50.530008: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", "2020-08-04 12:54:50.530349: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", "2020-08-04 12:54:50.530656: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0\n", "2020-08-04 12:54:50.530698: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1\n", "2020-08-04 12:54:50.636832: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:\n", "2020-08-04 12:54:50.636878: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0 \n", "2020-08-04 12:54:50.636885: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N \n", "2020-08-04 12:54:50.637047: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", "2020-08-04 12:54:50.637448: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", "2020-08-04 12:54:50.637806: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", "2020-08-04 12:54:50.638144: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4600 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)\n", "2020-08-04 12:54:50.639792: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55b8f63ea160 executing computations on platform CUDA. Devices:\n", "2020-08-04 12:54:50.639813: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): GeForce GTX 1060, Compute Capability 6.1\n", "WARNING:tensorflow:From /home/bentoml/BentoML/bentoml/artifact/keras_model_artifact.py:151: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.\n", "\n", "WARNING:tensorflow:From /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/keras/initializers.py:119: calling RandomUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "WARNING:tensorflow:From /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:97: calling GlorotUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "WARNING:tensorflow:From /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "WARNING:tensorflow:From /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:97: calling Orthogonal.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "WARNING:tensorflow:From /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:97: calling Zeros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n", "WARNING:tensorflow:From /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.where in 2.0, which has the same broadcast rule as np.where\n", "WARNING:tensorflow:From /home/bentoml/BentoML/bentoml/artifact/keras_model_artifact.py:137: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.\n", "\n", "2020-08-04 12:54:53.857399: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10\n", "0\n" ] } ], "source": [ "!bentoml run KerasTextClassificationService:latest predict --input '{\"text\": \"bad movie\"}'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Deployment Options\n", "\n", "If you are at a small team with limited engineering or DevOps resources, try out automated deployment with BentoML CLI, currently supporting AWS Lambda, AWS SageMaker, and Azure Functions:\n", "- [AWS Lambda Deployment Guide](https://docs.bentoml.org/en/latest/deployment/aws_lambda.html)\n", "- [AWS SageMaker Deployment Guide](https://docs.bentoml.org/en/latest/deployment/aws_sagemaker.html)\n", "- [Azure Functions Deployment Guide](https://docs.bentoml.org/en/latest/deployment/azure_functions.html)\n", "\n", "If the cloud platform you are working with is not on the list above, try out these step-by-step guide on manually deploying BentoML packaged model to cloud platforms:\n", "- [AWS ECS Deployment](https://docs.bentoml.org/en/latest/deployment/aws_ecs.html)\n", "- [Google Cloud Run Deployment](https://docs.bentoml.org/en/latest/deployment/google_cloud_run.html)\n", "- [Azure container instance Deployment](https://docs.bentoml.org/en/latest/deployment/azure_container_instance.html)\n", "- [Heroku Deployment](https://docs.bentoml.org/en/latest/deployment/heroku.html)\n", "\n", "Lastly, if you have a DevOps or ML Engineering team who's operating a Kubernetes or OpenShift cluster, use the following guides as references for implementating your deployment strategy:\n", "- [Kubernetes Deployment](https://docs.bentoml.org/en/latest/deployment/kubernetes.html)\n", "- [Knative Deployment](https://docs.bentoml.org/en/latest/deployment/knative.html)\n", "- [Kubeflow Deployment](https://docs.bentoml.org/en/latest/deployment/kubeflow.html)\n", "- [KFServing Deployment](https://docs.bentoml.org/en/latest/deployment/kfserving.html)\n", "- [Clipper.ai Deployment Guide](https://docs.bentoml.org/en/latest/deployment/clipper.html)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "colab": { "collapsed_sections": [], "name": "basic-text-classification.ipynb", "private_outputs": true, "provenance": [], "toc_visible": true, "version": "0.3.2" }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.3" } }, "nbformat": 4, "nbformat_minor": 4 }