{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## State-of-the-Art Sentiment Classification in TensorFlow" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the previous tutorial ([1-Construct-TensorFlow-Data-Pipeline.ipynb](https://github.com/ralphbrooks/tensorflow-tutorials/blob/master/1-Construct-TensorFlow-Data-Pipeline.ipynb)), I showed how you can create a data pipeline for a TensorFlow model. Once you have this data pipeline prepared, you are now ready to ingest this data and create a state-of-the-art classifier of emotion in language (a sentiment classifier). \n", "\n", "To this this we are going to make use of the following technologies:\n", "\n", "1) Transfer Learning: In 2019, Transfer Learning showed that it could achieve good performance on relatively low amounts of data. The concept is that a model is trained on a large amount of data beforehand, and this \"pre-trained\" model is then used in a second stage for classification. \n", "\n", "2) [Google BERT:](https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html) This is model that was pretrained by Google off of information from Wikipedia. It attempts to predict a word in a sentence given the words that PRECEDE the missing word and the words that FOLLOW the missing word. The model combines this prediction task with a [Transformer model architecture](https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html) which builds a neural network that looks at the emphasis of each word in relation to all other words in the sentence (a concept called self-attention).\n", "\n", "3) [Huggingface Transformers:](https://github.com/huggingface/transformers) - A startup company called Huggingface related in 2019 \"wrapper code\" that simplies using BERT as part of a data pipeline. \n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 1: Import and Setup" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We will get started by importing in the GPU version of Tensorflow and the Transformers library from Huggingface.\n", "\n", "Notes:\n", "\n", "* This exercise is going to require running this code with a GPU. I ran this code in AWS Sagemaker using a ml.p3.2xlarge GPU. Keep in mind that this GPU (as of Nov. 2019) costs a little\n", "over $4 / hour to operate.\n", "\n", "* Said differently, make sure that you stop or terminate your AWS notebook after completing this example so that you do not incur unnecessary cloud compute fees.\n", "\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "%load_ext autoreload\n", "% autoreload 2" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already up-to-date: pip in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (19.3.1)\n" ] } ], "source": [ "!pip install --upgrade pip" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since this exercise involves training a model, it is important that you use the GPU version of tensorflow for the training." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[33mWARNING: Skipping tensorflow as it is not installed.\u001b[0m\n", "yes: standard output: Broken pipe\n", "yes: write error\n", "Requirement already satisfied: tensorflow-gpu==2.0.0 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (2.0.0)\n", "Requirement already satisfied: termcolor>=1.1.0 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorflow-gpu==2.0.0) (1.1.0)\n", "Requirement already satisfied: wrapt>=1.11.1 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorflow-gpu==2.0.0) (1.11.2)\n", "Requirement already satisfied: six>=1.10.0 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorflow-gpu==2.0.0) (1.11.0)\n", "Requirement already satisfied: gast==0.2.2 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorflow-gpu==2.0.0) (0.2.2)\n", "Requirement already satisfied: tensorboard<2.1.0,>=2.0.0 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorflow-gpu==2.0.0) (2.0.2)\n", "Requirement already satisfied: google-pasta>=0.1.6 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorflow-gpu==2.0.0) (0.1.7)\n", "Requirement already satisfied: tensorflow-estimator<2.1.0,>=2.0.0 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorflow-gpu==2.0.0) (2.0.1)\n", "Requirement already satisfied: keras-applications>=1.0.8 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorflow-gpu==2.0.0) (1.0.8)\n", "Requirement already satisfied: opt-einsum>=2.3.2 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorflow-gpu==2.0.0) (3.1.0)\n", "Requirement already satisfied: protobuf>=3.6.1 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorflow-gpu==2.0.0) (3.7.1)\n", "Requirement already satisfied: wheel>=0.26 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorflow-gpu==2.0.0) (0.31.1)\n", "Requirement already satisfied: grpcio>=1.8.6 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorflow-gpu==2.0.0) (1.10.1)\n", "Requirement already satisfied: numpy<2.0,>=1.16.0 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorflow-gpu==2.0.0) (1.16.4)\n", "Requirement already satisfied: keras-preprocessing>=1.0.5 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorflow-gpu==2.0.0) (1.1.0)\n", "Requirement already satisfied: absl-py>=0.7.0 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorflow-gpu==2.0.0) (0.8.1)\n", "Requirement already satisfied: astor>=0.6.0 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorflow-gpu==2.0.0) (0.8.0)\n", "Requirement already satisfied: requests<3,>=2.21.0 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorboard<2.1.0,>=2.0.0->tensorflow-gpu==2.0.0) (2.22.0)\n", "Requirement already satisfied: werkzeug>=0.11.15 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorboard<2.1.0,>=2.0.0->tensorflow-gpu==2.0.0) (0.14.1)\n", "Requirement already satisfied: setuptools>=41.0.0 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorboard<2.1.0,>=2.0.0->tensorflow-gpu==2.0.0) (41.5.1)\n", "Requirement already satisfied: google-auth<2,>=1.6.3 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorboard<2.1.0,>=2.0.0->tensorflow-gpu==2.0.0) (1.10.0)\n", "Requirement already satisfied: markdown>=2.6.8 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorboard<2.1.0,>=2.0.0->tensorflow-gpu==2.0.0) (3.1.1)\n", "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorboard<2.1.0,>=2.0.0->tensorflow-gpu==2.0.0) (0.4.1)\n", "Requirement already satisfied: h5py in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from keras-applications>=1.0.8->tensorflow-gpu==2.0.0) (2.8.0)\n", "Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from requests<3,>=2.21.0->tensorboard<2.1.0,>=2.0.0->tensorflow-gpu==2.0.0) (2019.9.11)\n", "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from requests<3,>=2.21.0->tensorboard<2.1.0,>=2.0.0->tensorflow-gpu==2.0.0) (3.0.4)\n", "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from requests<3,>=2.21.0->tensorboard<2.1.0,>=2.0.0->tensorflow-gpu==2.0.0) (1.23)\n", "Requirement already satisfied: idna<2.9,>=2.5 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from requests<3,>=2.21.0->tensorboard<2.1.0,>=2.0.0->tensorflow-gpu==2.0.0) (2.6)\n", "Requirement already satisfied: rsa<4.1,>=3.1.4 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.1.0,>=2.0.0->tensorflow-gpu==2.0.0) (3.4.2)\n", "Requirement already satisfied: pyasn1-modules>=0.2.1 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.1.0,>=2.0.0->tensorflow-gpu==2.0.0) (0.2.7)\n", "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.1.0,>=2.0.0->tensorflow-gpu==2.0.0) (4.0.0)\n", "Requirement already satisfied: requests-oauthlib>=0.7.0 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.1.0,>=2.0.0->tensorflow-gpu==2.0.0) (1.3.0)\n", "Requirement already satisfied: pyasn1>=0.1.3 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from rsa<4.1,>=3.1.4->google-auth<2,>=1.6.3->tensorboard<2.1.0,>=2.0.0->tensorflow-gpu==2.0.0) (0.4.8)\n", "Requirement already satisfied: oauthlib>=3.0.0 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.1.0,>=2.0.0->tensorflow-gpu==2.0.0) (3.1.0)\n" ] } ], "source": [ "!yes | pip uninstall tensorflow\n", "! pip install tensorflow-gpu==2.0.0" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "!pip install -q transformers==2.1.1" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "import tensorflow as tf\n", "from transformers import *\n", "from transformers import BertTokenizer, TFBertForSequenceClassification, glue_convert_examples_to_features\n" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'2.0.0'" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tf.__version__" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 2: Create tf.data.Dataset from TFRecord" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the previous tutorial, we had put together the encoded data as TFRecords. Now, we will set up a transformation pipeline with a ```tf.data.Dataset``` .\n", "\n", "A ```tf.data.Dataset``` allows you to construct the pipeline without having to ingest and process all of the data in each step before examining the next transformation step. This is massively critical if you are testing out your pipeline. \n", "\n", "Said differently, the last thing in the world that you want to do is to wait 3-4 minutes during an iteration of code because you are waiting for a stage of preprocessing to complete. \n", "\n", "  \n", " \n", "The following code extracts the data that will be used to train the model (the tr_ds or train dataset), the data that will be used to validate the accuracy of the model (val_ds) and the holdout set of data that is used for the final verification of model accuracy." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "tr_ds = tf.data.TFRecordDataset(\"data/yelp_train.tfrecord\")\n", "val_ds = tf.data.TFRecordDataset(\"data/yelp_validate.tfrecord\")\n", "test_ds = tf.data.TFRecordDataset(\"data/yelp_test.tfrecord\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The following code will now extract tensors (the basic building blocks in TensorFlow) from the TFRecord. In essence, these tensors have the potential to be used as inputs into the model. \n", "\n", "Notes: \n", "\n", "* When you are extracting features from the TFRecord Dataset, it is important to note that the keys in the feature spec must match the keys that that were used in the TFRecord encoding process (the process that was covered in the previous tutorial). Also note that you don't have to use all of the tensors that are encoded within TFRecord (you could use just a subset of the Tensors). " ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "# convert the encoded string tensor into the separate tensors that will feed into the model\n", "\n", "feature_spec = {\n", " 'idx': tf.io.FixedLenFeature([], tf.int64),\n", " 'sentence': tf.io.FixedLenFeature([], tf.string),\n", " 'label': tf.io.FixedLenFeature([], tf.int64)\n", "}\n", "\n", "def parse_example(example_proto):\n", " # Parse the input tf.Example proto using the dictionary above.\n", " return tf.io.parse_single_example(example_proto, feature_spec)\n", "\n", "tr_parse_ds = tr_ds.map(parse_example)\n", "val_parse_ds = val_ds.map(parse_example)\n", "test_parse_ds = test_ds.map(parse_example)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "One approach to cleaning up a pipeline is to map a function to the dataset. In this way, the function gets applied to each example. The following code uses this approach to clean up the sentence tensor." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "\n", "def clean_yelp_string(features):\n", " revised_sentence = tf.strings.regex_replace(features['sentence'], \"\\.\\.\\.\", \"\", replace_global=True)\n", " revised_sentence = tf.strings.regex_replace(revised_sentence, \"\\\\'\", \"'\", replace_global=True)\n", " revised_sentence = tf.strings.regex_replace(revised_sentence, \"\\\\n\", \"\", replace_global=True)\n", " features['sentence'] = revised_sentence\n", " return features" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "tr_clean_ds = tr_parse_ds.map(lambda features: clean_yelp_string(features))\n", "val_clean_ds = val_parse_ds.map(lambda features: clean_yelp_string(features))\n", "test_clean_ds = test_parse_ds.map(lambda features: clean_yelp_string(features))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 3: Train the model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Recent research( [https://arxiv.org/abs/1804.07612](https://arxiv.org/abs/1804.07612)) suggests that smaller batch training improves generalization. Based on this, I set the batch size for training at 32. " ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "BATCH_SIZE = 32\n", "\n", "EVAL_BATCH_SIZE = BATCH_SIZE * 2\n", "\n", "# XLA is the optimizing compiler for machine learning\n", "# It can potentially increase speed by 15% with no source code changes\n", "USE_XLA = False\n", "\n", "# mixed precision results on https://github.com/huggingface/transformers/tree/master/examples\n", "# Mixed precision can help to speed up training time\n", "USE_AMP = False" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now you need to pull in the lengths of the different datasets from the JSON file that we created in the last tutorial. " ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(738, 82, 82)" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Steps is determined by the number of examples\n", "import json\n", "\n", "with open('data/yelp_info.json') as json_file:\n", " data_info = json.load(json_file)\n", " \n", "train_examples = data_info['train_length']\n", "valid_examples = data_info['validation_length']\n", "test_examples = data_info['test_length']\n", "\n", "train_examples, valid_examples, test_examples" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "tf.config.optimizer.set_jit(USE_XLA)\n", "tf.config.optimizer.set_experimental_options({\"auto_mixed_precision\": USE_AMP})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we have a pipeline setup, we need to start the process of converting words into numbers so that they can be processed by the BERT transfer learning backbone. This process is commonly called Tokenization, and Huggingface includes a tokenizer that helps with this process.\n", "\n", "The tokenizers are based on the underlying research code. For example, the following are different BERT models that can be utilized within the BERT framework:\n", "\n", " * ``bert-base-uncased``: 12-layer, 768-hidden, 12-heads, 110M parameters\n", " * ``bert-large-uncased``: 24-layer, 1024-hidden, 16-heads, 340M parameters\n", " * ``bert-base-cased``: 12-layer, 768-hidden, 12-heads , 110M parameters\n", " * ``bert-large-cased``: 24-layer, 1024-hidden, 16-heads, 340M parameters\n", " \n", "As seen above, the different numbers have different levels of complexity and are associated either with uncapitalized text (uncased) or text that has capitalization. I selected the bert-base-cased underlying model because the Yelp utterances have capitalization. I also selected it because in general models that are less complex tend to run faster than models which are more complex.\n" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "tokenizer = BertTokenizer.from_pretrained('bert-base-cased')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The Transformers framework can use a configuration dictionary in order to set up the hyperparameters for the model. In this case, I explictly use the config to make sure that the model\n", "is looking at num_labels=2. If we had been going through an example with three categories ('Positive', 'Negative', and 'Neutral') as opposed to just two cases ('Positive' and 'Negative') then we would have wanted to use num_labels=3 instead." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "# Make use of the following config parameters which are stored in bert_config.json.\n", "\n", "# {\n", "# \"attention_probs_dropout_prob\": 0.1,\n", "# \"hidden_act\": \"gelu\",\n", "# \"hidden_dropout_prob\": 0.1,\n", "# \"hidden_size\": 768,\n", "# \"initializer_range\": 0.02,\n", "# \"intermediate_size\": 3072,\n", "# \"layer_norm_eps\": 1e-12,\n", "# \"max_position_embeddings\": 512,\n", "# \"num_attention_heads\": 12,\n", "# \"num_hidden_layers\": 12,\n", "# \"num_labels\": 2,\n", "# \"output_attentions\": false,\n", "# \"output_hidden_states\": false,\n", "# \"output_past\": true,\n", "# \"pruned_heads\": {},\n", "# \"torchscript\": false,\n", "# \"type_vocab_size\": 2,\n", "# \"use_bfloat16\": false,\n", "# \"vocab_size\": 28996\n", "# }" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "from transformers.configuration_bert import BertConfig\n", "config = BertConfig(\"bert_config.json\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "TensorFlow uses layers of abstraction when putting together a model. Operations (such as matrix algebra) can occur at a low level, and an abstraction of a neural network layer can occur at a higher level. One of these higher levels of abstraction is called a Keras model, and Huggingface uses this model as a way to abstract some of the granular details involved in using BERT for transfer learning.\n", "\n", "The following is the syntax to instantiate a keras model with the Huggingface framework. " ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "model = TFBertForSequenceClassification.from_pretrained('bert-base-cased', config=config)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "Now that we have a tokenizer and the model with the right configurations, we need to take our parsed tensors (the tr_parse_ds or train parsed dataset) and feed them into the Huggingface framework. To do this, we are going to make a slight modification to the glue_convert_examples_to_features code found in the\n", "HuggingFace transformers repo. Here we are going to use the sst-2 task (the Stanford Sentiment Treebank binary classification task) because this task also works with binary classification. Because our example uses '1' for 'Negative' and '3' for Positive, we need to make this explicit by including the label_list keyword when doing the conversion. \n", "\n", "\n", "Notes:\n", "* Huggingface uses the similar strategy of taking the TFExamples and using a dataset in order to convert the \"sentence\" and \"labels\" into inputs that are needed by BERT (inputs such as 'input_ids', 'attention_mask', and 'token_type_ids'). As disscussed earlier in the workbook, this transformation process makes it quick to test out the conversion on a couple of data points and to move onto the next step without waiting for the full conversion to complete. " ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "train_dataset = glue_convert_examples_to_features(examples=tr_clean_ds, tokenizer=tokenizer\n", " , max_length=128, task='sst-2'\n", " , label_list =['1', '3'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Repeate the same transformation that you did for the training set on the information that will be used to validate the performance of the model. " ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "valid_dataset = glue_convert_examples_to_features(examples=val_clean_ds, tokenizer=tokenizer\n", " , max_length=128, task='sst-2'\n", " , label_list =['1', '3'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "One of the advantage of using the dataset approach is that you can specify the size of your batches, and you can shuffle the data." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "train_dataset = train_dataset.shuffle(train_examples).batch(BATCH_SIZE).repeat(-1)\n", "\n", "valid_dataset = valid_dataset.batch(EVAL_BATCH_SIZE)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this next section, we need to configure the loss function, the optimizer, and any additional metrics that we want to capture.\n", "\n", "Loss: \n", "\n", "This is the objective that the model is trying to minimize. In our example, the HuggingFace framework converts our Negatives ('1') and our Positives ('3') into '0' and '1' and compares this against the distribution of predicted classes. Said differently, we are trying to compare how similar the actual distribution is to the predicted distribution, and this is captured in the loss function called ```SparseCategoricalCrossentropy``` . \n", "\n", "A good discussion on cross entropy can be found at [The Gradient](https://thegradient.pub/understanding-evaluation-metrics-for-language-models/)\n", "\n", "Optimizer:\n", "\n", "Deep learning is the process of minimizing a loss function. The process that determines what steps to try out in each iteration is commonly referred to as the optimizer. For this exercise, I used the [Adam optimization algorithm](https://arxiv.org/pdf/1412.6980.pdf) which tends to work well in a variety of situations. \n", "\n", " Metric: \n", "\n", "As a basic metric, we should look at the number of times that the actual class is identical to what is predicted. \n", "\n", "Unfortunately ,the model currently generates unscaled outputs for each example (an unscaled output for the negative class and another unscaled output for the positive class). Said differently, the model generates outputs before they are converted into probabilities (the conversion happens with a softmax function). Because of all of this, the appropriate metric to use would be ```SparseCategoricalAccuracy```.\n", "\n", "  " ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "opt = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08)\n", "\n", "if USE_AMP:\n", " # loss scaling is currently required when using mixed precision\n", " opt = tf.keras.mixed_precision.experimental.LossScaleOptimizer(opt, 'dynamic')\n", "\n", "loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n", "metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')\n", "model.compile(optimizer=opt, loss=loss, metrics=[metric])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The TensorFlow documentation states the following:\n", "\n", "If x is a tf.data dataset, and 'steps_per_epoch' is None, the epoch will run until the input dataset is exhausted.\n", "\n", "Because these datasets can be a precursor to training on TFRecords of significant size, it is best practice to specificially state the number of steps that will be processed per epoch in the train and validation stage. \n", "\n" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": [ "train_steps = train_examples//BATCH_SIZE\n", "valid_steps = valid_examples//EVAL_BATCH_SIZE" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "GPUs run up to 27x faster that CPUs for model training. Because of this, it is critical that the following preconditions are in place:\n", "\n", "* You are using the version of Tensflow for GPUs\n", "* The code has access to a GPU\n", "\n", "To confirm the preconditions, I run the following code to detect GPUs and to see the physical devices that are available." ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Num GPUs Available: 1\n" ] } ], "source": [ "# GPU USAGE\n", "print(\"Num GPUs Available: \", len(tf.config.experimental.list_physical_devices('GPU')))" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'),\n", " PhysicalDevice(name='/physical_device:XLA_CPU:0', device_type='XLA_CPU'),\n", " PhysicalDevice(name='/physical_device:XLA_GPU:0', device_type='XLA_GPU'),\n", " PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tf.config.experimental.list_physical_devices()" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model: \"tf_bert_for_sequence_classification\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "bert (TFBertMainLayer) multiple 108310272 \n", "_________________________________________________________________\n", "dropout_37 (Dropout) multiple 0 \n", "_________________________________________________________________\n", "classifier (Dense) multiple 1538 \n", "=================================================================\n", "Total params: 108,311,810\n", "Trainable params: 108,311,810\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] } ], "source": [ "model.summary()" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Train for 23 steps, validate for 1 steps\n", "23/23 [==============================] - 35s 2s/step - loss: 0.3049 - accuracy: 0.8899 - val_loss: 0.2429 - val_accuracy: 0.9062\n" ] } ], "source": [ "history = model.fit(train_dataset, epochs=1, steps_per_epoch=train_steps,\n", " validation_data=valid_dataset, validation_steps=valid_steps)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "The above output shows the following \n", "\n", "* In the first epoch, the train accuracy is 88% and the validation accuracy is 90%.\n", "\n", "At this point, you see that the model has accuracy of 90%. This is fairly good considering that the Yelp rating captures the full sentiment of the user, but that in many cases, the Yelp API is only giving you part of the actual text review.\n", "\n", "Notes:\n", "1) In different runs of this notebook, the validation accuracy typically hits 90% with either 1 or 2 epochs. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 4: Evaluate the results of the model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The results above seem to look good. At this point, it is worth doing one final check before calling it a day." ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [], "source": [ "test_dataset = glue_convert_examples_to_features(examples=test_clean_ds, tokenizer=tokenizer\n", " , max_length=128, task='sst-2'\n", " , label_list =['1', '3'])" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [], "source": [ "test_dataset = test_dataset.batch(EVAL_BATCH_SIZE)" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2/2 [==============================] - 1s 683ms/step - loss: 0.3934 - accuracy: 0.8537\n" ] }, { "data": { "text/plain": [ "[0.39343981444835663, 0.85365856]" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.evaluate(test_dataset)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here we see that the model has 85% accuracy in the test set. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 5: Create a Confusion Matrix" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "  \n", "\n", "We can also visualize the evaluation in terms of how many true positives, true negatives, false positives, and false negatives occur. This visualization is commonly called a Confusion Matrix.\n", "\n", "Creation of the confusion matrix involves the following steps:\n", "\n", "1) Take the unnormalized outputs from the model (the logits) and compress them into probabilities (that by definition are between 0 and 1). The function that converts logits into probabilities is the softmax function.\n", "\n", "2) Once you have probabilities for each prediction, identify predicted emotion ('Negative' or 'Positive') by selecting the probability with the largest value. This is accomplished with the argmax function.\n" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [], "source": [ "y_pred = tf.nn.softmax(model.predict(test_dataset))" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y_pred[:10]" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [], "source": [ "y_pred_argmax = tf.math.argmax(y_pred, axis=1)" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 37, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y_pred_argmax[:10]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "3) Use a simple for loop in order to pull the true values from the test dataset. \n", "\n", "Be aware though that the Huggingface framework converted the labels that we had ('1' for negative and '3' for positive) into '0' for negative and '1' for positive." ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [], "source": [ "y_true = tf.Variable([], dtype=tf.int64)\n", "\n", "for features, label in test_dataset.take(-1):\n", " y_true = tf.concat([y_true, label], 0)" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y_true[:30]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "4) Use tf.math.confusion.matrix in order to determine true positives, true negatives, false positives, and false negatives from the true labels (y_true) and the predictions (y_pred_argmax). \n", "\n", "5) Use seaborn and matplotlib to visualize the confusion matrix. " ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAEmCAYAAAC0zD1TAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAFyVJREFUeJzt3XuYXXV97/H3d2aAAAlXlQZCDWKQAy0QLlGrbUEQ8FKgVDkgRaQ8pqAUkJaKl/MoPaXSWuEgWG3KLVa5yaWkKlpMschVwlUoyk2uBpIaSBAkkOR7/piFDENmZu/J/s1es+b98llP9l57zW99kyfmw3f91v6tyEwkSSqpp9sFSJKaz7CRJBVn2EiSijNsJEnFGTaSpOIMG0lScYaNJKk4w0aSVJxhI0kqrq/bBQxl3ZnHuLSBxtTTt5zV7RI0AU3qIzo5Xrv/dv769rM6ev6h1DZsJEmjEPW8YGXYSFKTxJg0Km0zbCSpSexsJEnF2dlIkoqzs5EkFWdnI0kqzs5GklScnY0kqTg7G0lScXY2kqTi7GwkScXZ2UiSirOzkSQVZ9hIkorr8TKaJKk0OxtJUnHeICBJKs7ORpJUnJ2NJKk4OxtJUnF2NpKk4np6u13Bahk2ktQkXkaTJBXnZTRJUnF2NpKk4gwbSVJxXkaTJBVnZyNJKs7ORpJUnJ2NJKk4OxtJUmlRIGwi4mHgWWAlsCIzd42ITYCLgenAw8BBmfn0UGPUs9+SJI1KRLS1tWGPzNwpM3et3p8EzM/MGcD86v2QDBtJapJocxu9/YG51eu5wAHDHWzYSFKDFOpsEviPiLg1ImZX+zbLzIXV6yeBzYYbwDkbSWqQdudsqvCYPWDXnMycM+iwd2bmExHxBuDqiPjpwA8zMyMihzuPYSNJDdJu2FTBMjhcBh/zRPXrooi4ApgFPBURUzNzYURMBRYNN4aX0SSpQTp9GS0i1o+IKS+/BvYG7gbmAYdXhx0OXDncOHY2ktQknb/zeTPgiiqY+oALMvN7EXELcElEHAk8Ahw03CCGjSQ1SKe/Z5OZDwE7rmb/L4E9Wx3HsJGkBinxpc5OMGwkqUEMG0lScYaNJKm8emaNYSNJTWJnI0kqzrCRJBUXPYaNJKkwOxtJUnGGjSSpOMNGklScYSNJKq+eWWPYSFKT2NlIkoozbCRJxRk2kqTy6pk1hs1499PvnMyzzy1n5apVrFi5ince+g/ssM0WnPmZg1lnnbVYsXIVx//dxSy455Ful6oGWr58OUd8+FBeevFFVqxcybv33oePHXNst8ua0OxsVMy+s8/gl88895v3pxx/AKfMuYr/uP6/2eed23HK8Qewz0fP6GKFaqq1116bs8+dy3rrr89LL73ERw77EO/8/T9ghx136nZpE9aEC5uI2BbYH9ii2vUEMC8z7y11TvXLhA3WnwTAhpPXZeHipV2uSE0VEay3/voArFixghUrVkBN/7GbKCZU2ETEJ4FDgIuAH1e7pwEXRsRFmXlqifNORJnJv//TMWQm51x2Pedefj0n/uOl/PtXPs4XPvHH9PQEe3zkS90uUw22cuVKDvnggTz66KP870M+xA47vOZx9RpDEypsgCOB7TPzpYE7I+I04B5gtWETEbOB2QB903an73XbFyqvOfY84nR+sXgpr994Mt/+2jH87OEnOXCvmfz1ly7n3+bfwZ+8eyZf/dyhvO+os7pdqhqqt7eXSy6/kmXLlvGJYz/O/fffx4wZ23S7rImrnllDT6FxVwGbr2b/1Oqz1crMOZm5a2buatC05hfVJbLFT/+Kef95F7ttP51D3/9W/m3+HQBcdvXt7Lr9G7tZoiaIDTbYgN1mvZUbrvtRt0uZ0CKirW2slAqb44H5EXFVRMyptu8B84HjCp1zwllv0tpMXm+d37ze6+3bcs+Dv2Dh4qX8/i4zANh91jY88OjibpapBluyZAnLli0D4IUXXuCmG29g+lZv6nJVE1tdw6bIZbTM/F5EbAPM4tU3CNySmStLnHMiesOmU7j4tI8C0Nfby8VXLeDqG+7l489fwBdP/AB9fT0sX76CY/72wi5Xqqb6n8WL+OynT2LVqpWsWpXsvc++/OHue3S7rAmtplM2RGZ2u4bVWnfmMfUsTI319C3Oa2nsTerr7CzLjBO/19a/nfd/cd8xiSe/ZyNJDVLXzsawkaQGmWi3PkuSuqCmWWPYSFKT9PTUM20MG0lqEDsbSVJxdjaSpOK8QUCSVJxhI0kqrqZZY9hIUpPY2UiSiqtp1hRb9VmS1AUlVn2OiN6IuD0ivl293yoibo6IByLi4ohYe6QxDBtJapCI9rYWHQfcO+D93wOnZ+abgafpf2DmsAwbSWqQTnc2ETENeB9wdvU+gHcBl1aHzAUOGGkcw0aSGqTdziYiZkfEggHb7EFD/j/gr3nlKcubAs9k5orq/eO88tyyIXmDgCQ1SLt3o2XmHGDOEGO9H1iUmbdGxO5rUpdhI0kN0uG70d4B7BcR7wUmARsAZwAbRURf1d1Mo/9JzMPyMpokNUgn52wy81OZOS0zpwMHA/+ZmYcC1wAfqA47HLhypLoMG0lqkEJ3ow32SeCEiHiA/jmcc0b6AS+jSVKDlFpBIDN/CPywev0QMKudnzdsJKlB6rqCgGEjSQ3i2miSpOIMG0lScTXNGsNGkprEzkaSVFxNs8awkaQmsbORJBVX06wxbCSpSXpqmjaGjSQ1SE+PYSNJKqymWWPYSFKTeIOAJKm4mmaNYSNJTRLUM22GDJuI2GC4H8zMZZ0vR5K0JsbjnM09QMKrYvLl9wn8dsG6JEmjMO7mbDJzy7EsRJK05mqaNa09FjoiDo6IT1evp0XELmXLkiSNRk9EW9uY1TXSARFxFrAHcFi163ngayWLkiSNTkR721hp5W6038vMnSPidoDMXBIRaxeuS5I0CuNuzmaAlyKih/6bAoiITYFVRauSJI1KTbOmpbD5CnAZ8PqIOBk4CDi5aFWSpFEZtwtxZubXI+JWYK9q1wcz8+6yZUmSRqOeUdP6CgK9wEv0X0pr6Q42SdLYq+ucTSt3o30GuBDYHJgGXBARnypdmCSpfT3R3jZWWulsPgzMzMznASLiFOB24AslC5Mkta+unU0rYbNw0HF91T5JUs3UNGuGXYjzdPrnaJYA90TE96v3ewO3jE15kqR2jMfO5uU7zu4BvjNg/03lypEkrYlxt+pzZp4zloVIktbceOxsAIiIrYFTgO2ASS/vz8xtCtYlSRqFekZNa9+ZOR84j/7fw3uAS4CLC9YkSRqlcbvqM7BeZn4fIDMfzMzP0h86kqSaGc+rPi+vFuJ8MCKOAp4AppQtS5I0GuN2zgb4BLA+cCz9czcbAn9WsihJ0uj01vR2tFYW4ry5evksrzxATZJUQzVtbIb9UucVVM+wWZ3MPLBIRZX75n+p5PDSazy8+Plul6AJaNup63V0vE5fRouIScC1wDr0Z8almfm5iNgKuAjYFLgVOCwzXxxqnOE6m7M6WK8kaQwUWJZ/OfCuzPxVRKwFXBcRVwEnAKdn5kUR8TXgSOCrQw0y3Jc653e6YklSWZ3ubDIzgV9Vb9eqtgTeBXyo2j8X+DzDhI3PppGkBmn3EQMRMTsiFgzYZg8eMyJ6I+IOYBFwNfAg8ExmrqgOeRzYYri6Wn14miRpHGj3ZrTMnAPMGeGYlcBOEbERcAWwbbt1tRw2EbFOZi5v9wSSpLFT8ns2mflMRFwDvB3YKCL6qu5mGv3fwRxSK0/qnBURPwHur97vGBFndqBuSVKHdfpJnRHx+qqjISLWBd4N3AtcA3ygOuxw4Mph62qh9i8D7wd+CZCZdwJ7tPBzkqQxVmC5mqnANRFxF/3PMrs6M78NfBI4ISIeoP/252GfFNDKZbSezHxkUGu2sqUSJUljqtOLa2bmXcDM1ex/CJjV6jithM1jETELyIjoBf4CuK/VE0iSxk5dbzFuJWyOpv9S2m8DTwE/qPZJkmpm3C1X87LMXAQcPAa1SJLW0Fg+o6YdrTyp819YzRppmfmaL/5IkrqrplnT0mW0Hwx4PQn4Y+CxMuVIktZETZ8w0NJltFc9Ajoi/hW4rlhFkqRRG7eX0VZjK2CzThciSVpzNc2aluZsnuaVOZseYAlwUsmiJEmjMy4vo0X/Nzl35JU1b1ZVy01LkmooqGfaDPv9nypYvpuZK6vNoJGkGuv02mgdq6uFY+6IiNcsVSBJqp+6hs2Ql9EGLB09E7glIh4EngOC/qZn5zGqUZLUopKPGFgTw83Z/BjYGdhvjGqRJK2h8XiDQABk5oNjVIskaQ311jRthgub10fECUN9mJmnFahHkrQGapo1w4ZNLzAZanofnSTpNWo6ZTNs2CzMzL8Zs0okSWusp6b9wYhzNpKk8WM8djZ7jlkVkqSOGHdzNpm5ZCwLkSStuSat+ixJqqmaZo1hI0lNYmcjSSqupllj2EhSk7SyunI3GDaS1CDjcSFOSdI4U8+oMWwkqVG8QUCSVFw9o8awkaRGqWljY9hIUpN4g4AkqThvfZYkFWdnI0kqrp5RY9hIUqPY2UiSinPORpJUXF07m7qGoCRpFKLNbcTxIraMiGsi4r8j4p6IOK7av0lEXB0R91e/bjzcOIaNJDVIb0RbWwtWAH+ZmdsBbwM+HhHbAScB8zNzBjC/ej8kw0aSGiSivW0kmbkwM2+rXj8L3AtsAewPzK0OmwscMNw4ho0kNUi0+7+I2RGxYMA2e8ixI6YDM4Gbgc0yc2H10ZPAZsPV5Q0CktQg7d4fkJlzgDkjjxuTgcuA4zNz2cAbETIzIyKH+3nDRpIapKfA1zojYi36g+abmXl5tfupiJiamQsjYiqwaPi6JEmN0ek5m+hvYc4B7s3M0wZ8NA84vHp9OHDlcOPY2UhSgxT4ms07gMOAn0TEHdW+TwOnApdExJHAI8BBww1i2EhSg0SHL6Nl5nUM/ZWcPVsdx7CRpAbpqecCAoaNJDVJpzubTjFsJKlBaro0mmEjSU1iZ6PiLr/4G3x33mVkwnv3O5A/OfiwbpekBvry33+eBTdey4YbbcKZ518KwHlfPZ1bbriWvrXW4rc2n8axnzyZyVOmdLnSiamuczZ+z6Yhfv7g/Xx33mWcdc4FzPn6t7jp+mt54rFHu12WGmjPff+Iz/3DV161b6dd38aZ532LL597CVts+UYuu+DcLlWndperGSuGTUM8+vDP2Xa7HZg0aV16+/rYceauXPdfP+h2WWqg7XfchclTNnzVvpm7vZ3evv4LJdts97v8z+KnulGa6PyXOjvFsGmI6Vu/mZ/ceRtLlz7DCy/8mptv/BGLnvL/8Bp78797JbvMeke3y5iwOv08m04Z87CJiCOG+ew3q49+c+7ZY1nWuPfG6W/i4D89gpOO+3M+9Ymj2XrGW+jt8b8lNLYu+dez6ent5Q/f/d5ulzJh9US0tY2VbtwgcDJw3uo+GLj66GNLlg+7gqhe6z37Hch79jsQgHO+egave8OwK35LHTX/qnksuPFa/u9p/1zbRxNPBHX9ky8SNhFx11AfMcIzDzR6Ty/5JRtvsilPPbmQ6344nzPP/ka3S9IEcdvN13P5Refzd2eczTqT1u12ORNbTdMmMjvfQETEU8A+wNODPwJuyMzNRxrDzqZ9xx91OMuWLqWvr4+jjv0rdt7tbd0uaVx5bvnKbpcwLvzj35zE3XfcyrKlz7DRxptwyBFHcek3z+Oll15kgw36bxzYZrvf5WN/+dkuVzo+bDt1vY7Gw80PLm3r3863br3hmMRTqbA5BzivWsBt8GcXZOaHRhrDsNFYM2zUDZ0Omx8/1F7YzHrT2IRNkctomXnkMJ+NGDSSpNGp6VU0VxCQpEapadoYNpLUIK6NJkkqrq53nRs2ktQgho0kqTgvo0mSirOzkSQVV9OsMWwkqVFqmjaGjSQ1iHM2kqTinLORJBVX06wxbCSpUWqaNoaNJDWIczaSpOKcs5EkFVfTrDFsJKlRapo2ho0kNYhzNpKk4pyzkSQVV9OsMWwkqVFqmjaGjSQ1SF3nbHq6XYAkqXMi2ttGHi/OjYhFEXH3gH2bRMTVEXF/9evGI41j2EhSg0SbWwvOB/YdtO8kYH5mzgDmV++HZdhIUpN0OG0y81pgyaDd+wNzq9dzgQNGGsc5G0lqkDGas9ksMxdWr58ENhvpB+xsJKlB2p2ziYjZEbFgwDa7nfNlZgI50nF2NpLUIO32NZk5B5jT5o89FRFTM3NhREwFFo30A3Y2ktQgEdHWNkrzgMOr14cDV470A3Y2ktQgnV6uJiIuBHYHXhcRjwOfA04FLomII4FHgINGGsewkaQG6fTtAZl5yBAf7dnOOIaNJDWIC3FKksZAPdPGsJGkBrGzkSQVV9OsMWwkqUnsbCRJxdX1EQOGjSQ1ST2zxrCRpCapadYYNpLUJM7ZSJKKc85GklRePbPGsJGkJqlp1hg2ktQkztlIkopzzkaSVFxdOxuf1ClJKs7ORpIapK6djWEjSQ3inI0kqTg7G0lScTXNGsNGkhqlpmlj2EhSg/TU9DqaYSNJDVLPqDFsJKlZapo2ho0kNYi3PkuSiqvplA2Rmd2uQR0WEbMzc06369DE4d85jcS10ZppdrcL0ITj3zkNy7CRJBVn2EiSijNsmslr5xpr/p3TsLxBQJJUnJ2NJKk4w0aSVJxh0yARsW9E/CwiHoiIk7pdj5ovIs6NiEURcXe3a1G9GTYNERG9wFeA9wDbAYdExHbdrUoTwPnAvt0uQvVn2DTHLOCBzHwoM18ELgL273JNarjMvBZY0u06VH+GTXNsATw24P3j1T5J6jrDRpJUnGHTHE8AWw54P63aJ0ldZ9g0xy3AjIjYKiLWBg4G5nW5JkkCDJvGyMwVwDHA94F7gUsy857uVqWmi4gLgRuBt0TE4xFxZLdrUj25XI0kqTg7G0lScYaNJKk4w0aSVJxhI0kqzrCRJBVn2KhrImJlRNwREXdHxLciYr01GGv3iPh29Xq/4Va9joiNIuJjozjH5yPir1rdP+iY8yPiA22ca7orKatJDBt1068zc6fM/B3gReCogR9Gv7b/jmbmvMw8dZhDNgLaDhtJo2fYqC5+BLy5+i/6n0XE14G7gS0jYu+IuDEibqs6oMnwm+f3/DQibgMOfHmgiPhIRJxVvd4sIq6IiDur7feAU4Gtq67qi9VxJ0bELRFxV0ScPGCsz0TEfRFxHfCWkX4TEfHRapw7I+KyQd3aXhGxoBrv/dXxvRHxxQHn/vM1/YOU6siwUddFRB/9z+H5SbVrBvBPmbk98BzwWWCvzNwZWACcEBGTgH8B/gjYBfitIYb/MvBfmbkjsDNwD3AS8GDVVZ0YEXtX55wF7ATsEhF/EBG70L/sz07Ae4HdWvjtXJ6Zu1XnuxcY+I366dU53gd8rfo9HAkszczdqvE/GhFbtXAeaVzp63YBmtDWjYg7qtc/As4BNgceycybqv1vo/9hcNdHBMDa9C+Psi3w88y8HyAivgHMXs053gV8GCAzVwJLI2LjQcfsXW23V+8n0x8+U4ArMvP56hytrDX3OxHxt/RfqptM//JBL7skM1cB90fEQ9XvYW9ghwHzORtW576vhXNJ44Zho276dWbuNHBHFSjPDdwFXJ2Zhww67lU/t4YC+EJm/vOgcxw/irHOBw7IzDsj4iPA7gM+G7w2VFbn/ovMHBhKRMT0UZxbqi0vo6nubgLeERFvBoiI9SNiG+CnwPSI2Lo67pAhfn4+cHT1s70RsSHwLP1dy8u+D/zZgLmgLSLiDcC1wAERsW5ETKH/kt1IpgALI2It4NBBn30wInqqmt8E/Kw699HV8UTENhGxfgvnkcYVOxvVWmYurjqECyNinWr3ZzPzvoiYDXwnIp6n/zLclNUMcRwwp1qNeCVwdGbeGBHXV7cWX1XN2/wv4Maqs/oV8KeZeVtEXAzcCSyi/zEOI/k/wM3A4urXgTU9CvwY2AA4KjNfiIiz6Z/LuS36T74YOKC1Px1p/HDVZ0lScV5GkyQVZ9hIkoozbCRJxRk2kqTiDBtJUnGGjSSpOMNGklTc/wdH7hy/lUnhWwAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "%matplotlib inline \n", "import seaborn as sns\n", "import matplotlib.pyplot as plt\n", "import pandas as pd\n", "\n", "def visualize_confusion_matrix(y_pred_argmax, y_true):\n", " \"\"\"\n", "\n", " :param y_pred_arg: This is an array with values that are 0 or 1\n", " :param y_true: This is an array with values that are 0 or 1\n", " :return:\n", " \"\"\"\n", "\n", " cm = tf.math.confusion_matrix(y_true, y_pred_argmax).numpy()\n", " con_mat_df = pd.DataFrame(cm)\n", "\n", " sns.heatmap(con_mat_df, annot=True, fmt='g', cmap=plt.cm.Blues)\n", " plt.tight_layout()\n", " plt.ylabel('True label')\n", " plt.xlabel('Predicted label')\n", " plt.show()\n", "\n", "\n", "visualize_confusion_matrix(y_pred_argmax, y_true)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Analysis of the above shows the following:\n", "\n", "* The analysis of the above shows that if the customer is expressing negative sentiment. Then the classifier gets it right 95% of the time. \n", "* If the model is asked to evaluate positive sentiment, then the model gets it right 57% of the time.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 6: Create the Saved Model" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "Once you have a model with decent accuracy, " ] }, { "cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1781: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "If using Keras pass *_constraint arguments to layers.\n" ] }, { "ename": "PermissionDeniedError", "evalue": "/20191227; Permission denied", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mPermissionDeniedError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msaved_model\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msave\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'/20191227'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/python/saved_model/save.py\u001b[0m in \u001b[0;36msave\u001b[0;34m(obj, export_dir, signatures, options)\u001b[0m\n\u001b[1;32m 897\u001b[0m \u001b[0;31m# the checkpoint, copy assets into the assets directory, and write out the\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 898\u001b[0m \u001b[0;31m# SavedModel proto itself.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 899\u001b[0;31m \u001b[0mutils_impl\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_or_create_variables_dir\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mexport_dir\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 900\u001b[0m \u001b[0mobject_saver\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msave\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mutils_impl\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_variables_path\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mexport_dir\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 901\u001b[0m builder_impl.copy_assets_to_destination_dir(asset_info.asset_filename_map,\n", "\u001b[0;32m~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/python/saved_model/utils_impl.py\u001b[0m in \u001b[0;36mget_or_create_variables_dir\u001b[0;34m(export_dir)\u001b[0m\n\u001b[1;32m 181\u001b[0m \u001b[0mvariables_dir\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_variables_dir\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mexport_dir\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 182\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mfile_io\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfile_exists\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvariables_dir\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 183\u001b[0;31m \u001b[0mfile_io\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecursive_create_dir\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvariables_dir\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 184\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mvariables_dir\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 185\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/python/lib/io/file_io.py\u001b[0m in \u001b[0;36mrecursive_create_dir\u001b[0;34m(dirname)\u001b[0m\n\u001b[1;32m 436\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOpError\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mIf\u001b[0m \u001b[0mthe\u001b[0m \u001b[0moperation\u001b[0m \u001b[0mfails\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 437\u001b[0m \"\"\"\n\u001b[0;32m--> 438\u001b[0;31m \u001b[0mrecursive_create_dir_v2\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdirname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 439\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 440\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/python/lib/io/file_io.py\u001b[0m in \u001b[0;36mrecursive_create_dir_v2\u001b[0;34m(path)\u001b[0m\n\u001b[1;32m 451\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOpError\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mIf\u001b[0m \u001b[0mthe\u001b[0m \u001b[0moperation\u001b[0m \u001b[0mfails\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 452\u001b[0m \"\"\"\n\u001b[0;32m--> 453\u001b[0;31m \u001b[0mpywrap_tensorflow\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mRecursivelyCreateDir\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcompat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_bytes\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 454\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 455\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mPermissionDeniedError\u001b[0m: /20191227; Permission denied" ] } ], "source": [ "tf.saved_model.save(model, '/20191227')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ " \n", "\n", "Congratulations. You made it to the end of the tutorial and now you have a process that takes you from building a data pipeline to having an accurate model." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Also, for the first time in 10 years, I am back on the job market looking for consulting opportunities or full time employment. If you think I can be of help to you, feel free to reach out. I am on twitter at [@ralphbrooks](https://twitter.com/ralphbrooks) ." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "conda_tensorflow_p36", "language": "python", "name": "conda_tensorflow_p36" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.5" } }, "nbformat": 4, "nbformat_minor": 4 }