"
]
},
"execution_count": 98,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"Y_pred = tf.argmax(Y_probas, axis=1)\n",
"Y_pred # 0 = contradiction, 1 = entailment, 2 = neutral"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This TensorFlow code no longer works, so I commented it out:"
]
},
{
"cell_type": "code",
"execution_count": 99,
"metadata": {},
"outputs": [],
"source": [
"# sentences = [(\"Sky is blue\", \"Sky is red\"), (\"I love her\", \"She loves me\")]\n",
"# X_train = tokenizer(sentences, padding=True, return_tensors=\"tf\").data\n",
"# y_train = tf.constant([0, 2]) # contradiction, neutral\n",
"# loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n",
"# model.compile(loss=loss, optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
"# history = model.fit(X_train, y_train, epochs=2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The following code does the same thing using PyTorch and the Transformer library's `Trainer` API:"
]
},
{
"cell_type": "code",
"execution_count": 100,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
"
\n",
" [2/2 00:21, Epoch 2/2]\n",
"
\n",
" \n",
" \n",
" | Step | \n",
" Training Loss | \n",
"
\n",
" \n",
" \n",
" \n",
"
"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"TrainOutput(global_step=2, training_loss=0.0020126409363001585, metrics={'train_runtime': 21.8279, 'train_samples_per_second': 0.183, 'train_steps_per_second': 0.092, 'total_flos': 9314280072.0, 'train_loss': 0.0020126409363001585, 'epoch': 2.0})"
]
},
"execution_count": 100,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments\n",
"from torch.utils.data import TensorDataset\n",
"import torch\n",
"\n",
"sentences = [(\"Sky is blue\", \"Sky is red\"), (\"I love her\", \"She loves me\")]\n",
"X_train = tokenizer(sentences, padding=True, return_tensors=\"pt\").data\n",
"y_train = torch.tensor([0, 2]) # 0=contradiction, 2=neutral\n",
"\n",
"dataset = TensorDataset(X_train[\"input_ids\"], X_train[\"attention_mask\"],\n",
" y_train)\n",
"\n",
"def collate_fn(batch):\n",
" input_ids, attention_mask, labels = zip(*batch)\n",
" return {\n",
" \"input_ids\": torch.stack(input_ids),\n",
" \"attention_mask\": torch.stack(attention_mask),\n",
" \"labels\": torch.stack(labels)\n",
" }\n",
"\n",
"args = TrainingArguments(output_dir=\"./results\", num_train_epochs=2,\n",
" per_device_train_batch_size=2, report_to=\"none\")\n",
"trainer = Trainer(model=model, args=args, train_dataset=dataset,\n",
" data_collator=collate_fn)\n",
"trainer.train()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Exercise solutions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. to 7."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. Stateless RNNs can only capture patterns whose length is less than, or equal to, the size of the windows the RNN is trained on. Conversely, stateful RNNs can capture longer-term patterns. However, implementing a stateful RNN is much harder—especially preparing the dataset properly. Moreover, stateful RNNs do not always work better, in part because consecutive batches are not independent and identically distributed (IID). Gradient Descent is not fond of non-IID datasets.\n",
"2. In general, if you translate a sentence one word at a time, the result will be terrible. For example, the French sentence \"Je vous en prie\" means \"You are welcome,\" but if you translate it one word at a time, you get \"I you in pray.\" Huh? It is much better to read the whole sentence first and then translate it. A plain sequence-to-sequence RNN would start translating a sentence immediately after reading the first word, while an Encoder–Decoder RNN will first read the whole sentence and then translate it. That said, one could imagine a plain sequence-to-sequence RNN that would output silence whenever it is unsure about what to say next (just like human translators do when they must translate a live broadcast).\n",
"3. Variable-length input sequences can be handled by padding the shorter sequences so that all sequences in a batch have the same length, and using masking to ensure the RNN ignores the padding token. For better performance, you may also want to create batches containing sequences of similar sizes. Ragged tensors can hold sequences of variable lengths, and Keras now supports them, which simplifies handling variable-length input sequences (at the time of this writing, it still does not handle ragged tensors as targets on the GPU, though). Regarding variable-length output sequences, if the length of the output sequence is known in advance (e.g., if you know that it is the same as the input sequence), then you just need to configure the loss function so that it ignores tokens that come after the end of the sequence. Similarly, the code that will use the model should ignore tokens beyond the end of the sequence. But generally the length of the output sequence is not known ahead of time, so the solution is to train the model so that it outputs an end-of-sequence token at the end of each sequence.\n",
"4. Beam search is a technique used to improve the performance of a trained Encoder–Decoder model, for example in a neural machine translation system. The algorithm keeps track of a short list of the _k_ most promising output sentences (say, the top three), and at each decoder step it tries to extend them by one word; then it keeps only the _k_ most likely sentences. The parameter _k_ is called the _beam width_: the larger it is, the more CPU and RAM will be used, but also the more accurate the system will be. Instead of greedily choosing the most likely next word at each step to extend a single sentence, this technique allows the system to explore several promising sentences simultaneously. Moreover, this technique lends itself well to parallelization. You can implement beam search by writing a custom memory cell. Alternatively, TensorFlow Addons's seq2seq API provides an implementation.\n",
"5. An attention mechanism is a technique initially used in Encoder–Decoder models to give the decoder more direct access to the input sequence, allowing it to deal with longer input sequences. At each decoder time step, the current decoder's state and the full output of the encoder are processed by an alignment model that outputs an alignment score for each input time step. This score indicates which part of the input is most relevant to the current decoder time step. The weighted sum of the encoder output (weighted by their alignment score) is then fed to the decoder, which produces the next decoder state and the output for this time step. The main benefit of using an attention mechanism is the fact that the Encoder–Decoder model can successfully process longer input sequences. Another benefit is that the alignment scores make the model easier to debug and interpret: for example, if the model makes a mistake, you can look at which part of the input it was paying attention to, and this can help diagnose the issue. An attention mechanism is also at the core of the Transformer architecture, in the Multi-Head Attention layers. See the next answer.\n",
"6. The most important layer in the Transformer architecture is the Multi-Head Attention layer (the original Transformer architecture contains 18 of them, including 6 Masked Multi-Head Attention layers). It is at the core of language models such as BERT and GPT-2. Its purpose is to allow the model to identify which words are most aligned with each other, and then improve each word's representation using these contextual clues.\n",
"7. Sampled softmax is used when training a classification model when there are many classes (e.g., thousands). It computes an approximation of the cross-entropy loss based on the logit predicted by the model for the correct class, and the predicted logits for a sample of incorrect words. This speeds up training considerably compared to computing the softmax over all logits and then estimating the cross-entropy loss. After training, the model can be used normally, using the regular softmax function to compute all the class probabilities based on all the logits."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 8.\n",
"_Exercise:_ Embedded Reber grammars _were used by Hochreiter and Schmidhuber in [their paper](https://homl.info/93) about LSTMs. They are artificial grammars that produce strings such as \"BPBTSXXVPSEPE.\" Check out Jenny Orr's [nice introduction](https://homl.info/108) to this topic. Choose a particular embedded Reber grammar (such as the one represented on Jenny Orr's page), then train an RNN to identify whether a string respects that grammar or not. You will first need to write a function capable of generating a training batch containing about 50% strings that respect the grammar, and 50% that don't._"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First we need to build a function that generates strings based on a grammar. The grammar will be represented as a list of possible transitions for each state. A transition specifies the string to output (or a grammar to generate it) and the next state."
]
},
{
"cell_type": "code",
"execution_count": 101,
"metadata": {},
"outputs": [],
"source": [
"default_reber_grammar = [\n",
" [(\"B\", 1)], # (state 0) =B=>(state 1)\n",
" [(\"T\", 2), (\"P\", 3)], # (state 1) =T=>(state 2) or =P=>(state 3)\n",
" [(\"S\", 2), (\"X\", 4)], # (state 2) =S=>(state 2) or =X=>(state 4)\n",
" [(\"T\", 3), (\"V\", 5)], # and so on...\n",
" [(\"X\", 3), (\"S\", 6)],\n",
" [(\"P\", 4), (\"V\", 6)],\n",
" [(\"E\", None)]] # (state 6) =E=>(terminal state)\n",
"\n",
"embedded_reber_grammar = [\n",
" [(\"B\", 1)],\n",
" [(\"T\", 2), (\"P\", 3)],\n",
" [(default_reber_grammar, 4)],\n",
" [(default_reber_grammar, 5)],\n",
" [(\"T\", 6)],\n",
" [(\"P\", 6)],\n",
" [(\"E\", None)]]\n",
"\n",
"def generate_string(grammar):\n",
" state = 0\n",
" output = []\n",
" while state is not None:\n",
" index = np.random.randint(len(grammar[state]))\n",
" production, state = grammar[state][index]\n",
" if isinstance(production, list):\n",
" production = generate_string(grammar=production)\n",
" output.append(production)\n",
" return \"\".join(output)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's generate a few strings based on the default Reber grammar:"
]
},
{
"cell_type": "code",
"execution_count": 102,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"BTXXTTVPXTVPXTTVPSE BPVPSE BTXSE BPVVE BPVVE BTSXSE BPTVPXTTTVVE BPVVE BTXSE BTXXVPSE BPTTTTTTTTVVE BTXSE BPVPSE BTXSE BPTVPSE BTXXTVPSE BPVVE BPVVE BPVVE BPTTVVE BPVVE BPVVE BTXXVVE BTXXVVE BTXXVPXVVE "
]
}
],
"source": [
"np.random.seed(42)\n",
"\n",
"for _ in range(25):\n",
" print(generate_string(default_reber_grammar), end=\" \")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Looks good. Now let's generate a few strings based on the embedded Reber grammar:"
]
},
{
"cell_type": "code",
"execution_count": 103,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"BTBPTTTVPXTVPXTTVPSETE BPBPTVPSEPE BPBPVVEPE BPBPVPXVVEPE BPBTXXTTTTVVEPE BPBPVPSEPE BPBTXXVPSEPE BPBTSSSSSSSXSEPE BTBPVVETE BPBTXXVVEPE BPBTXXVPSEPE BTBTXXVVETE BPBPVVEPE BPBPVVEPE BPBTSXSEPE BPBPVVEPE BPBPTVPSEPE BPBTXXVVEPE BTBPTVPXVVETE BTBPVVETE BTBTSSSSSSSXXVVETE BPBTSSSXXTTTTVPSEPE BTBPTTVVETE BPBTXXTVVEPE BTBTXSETE "
]
}
],
"source": [
"np.random.seed(42)\n",
"\n",
"for _ in range(25):\n",
" print(generate_string(embedded_reber_grammar), end=\" \")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Okay, now we need a function to generate strings that do not respect the grammar. We could generate a random string, but the task would be a bit too easy, so instead we will generate a string that respects the grammar, and we will corrupt it by changing just one character:"
]
},
{
"cell_type": "code",
"execution_count": 104,
"metadata": {},
"outputs": [],
"source": [
"POSSIBLE_CHARS = \"BEPSTVX\"\n",
"\n",
"def generate_corrupted_string(grammar, chars=POSSIBLE_CHARS):\n",
" good_string = generate_string(grammar)\n",
" index = np.random.randint(len(good_string))\n",
" good_char = good_string[index]\n",
" bad_char = np.random.choice(sorted(set(chars) - set(good_char)))\n",
" return good_string[:index] + bad_char + good_string[index + 1:]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's look at a few corrupted strings:"
]
},
{
"cell_type": "code",
"execution_count": 105,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"BTBPTTTPPXTVPXTTVPSETE BPBTXEEPE BPBPTVVVEPE BPBTSSSSXSETE BPTTXSEPE BTBPVPXTTTTTTEVETE BPBTXXSVEPE BSBPTTVPSETE BPBXVVEPE BEBTXSETE BPBPVPSXPE BTBPVVVETE BPBTSXSETE BPBPTTTPTTTTTVPSEPE BTBTXXTTSTVPSETE BBBTXSETE BPBTPXSEPE BPBPVPXTTTTVPXTVPXVPXTTTVVEVE BTBXXXTVPSETE BEBTSSSSSXXVPXTVVETE BTBXTTVVETE BPBTXSTPE BTBTXXTTTVPSBTE BTBTXSETX BTBTSXSSTE "
]
}
],
"source": [
"np.random.seed(42)\n",
"\n",
"for _ in range(25):\n",
" print(generate_corrupted_string(embedded_reber_grammar), end=\" \")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We cannot feed strings directly to an RNN, so we need to encode them somehow. One option would be to one-hot encode each character. Another option is to use embeddings. Let's go for the second option (but since there are just a handful of characters, one-hot encoding would probably be a good option as well). For embeddings to work, we need to convert each string into a sequence of character IDs. Let's write a function for that, using each character's index in the string of possible characters \"BEPSTVX\":"
]
},
{
"cell_type": "code",
"execution_count": 106,
"metadata": {},
"outputs": [],
"source": [
"def string_to_ids(s, chars=POSSIBLE_CHARS):\n",
" return [chars.index(c) for c in s]"
]
},
{
"cell_type": "code",
"execution_count": 107,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[0, 4, 4, 4, 6, 6, 5, 5, 1, 4, 1]"
]
},
"execution_count": 107,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"string_to_ids(\"BTTTXXVVETE\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can now generate the dataset, with 50% good strings, and 50% bad strings:"
]
},
{
"cell_type": "code",
"execution_count": 108,
"metadata": {},
"outputs": [],
"source": [
"def generate_dataset(size):\n",
" good_strings = [\n",
" string_to_ids(generate_string(embedded_reber_grammar))\n",
" for _ in range(size // 2)\n",
" ]\n",
" bad_strings = [\n",
" string_to_ids(generate_corrupted_string(embedded_reber_grammar))\n",
" for _ in range(size - size // 2)\n",
" ]\n",
" all_strings = good_strings + bad_strings\n",
" X = tf.ragged.constant(all_strings, ragged_rank=1)\n",
" y = np.array([[1.] for _ in range(len(good_strings))] +\n",
" [[0.] for _ in range(len(bad_strings))])\n",
" return X, y"
]
},
{
"cell_type": "code",
"execution_count": 109,
"metadata": {},
"outputs": [],
"source": [
"np.random.seed(42)\n",
"\n",
"X_train, y_train = generate_dataset(10000)\n",
"X_valid, y_valid = generate_dataset(2000)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's take a look at the first training sequence:"
]
},
{
"cell_type": "code",
"execution_count": 110,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 110,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X_train[0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"What class does it belong to?"
]
},
{
"cell_type": "code",
"execution_count": 111,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([1.])"
]
},
"execution_count": 111,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"y_train[0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Perfect! We are ready to create the RNN to identify good strings. We build a simple sequence binary classifier:"
]
},
{
"cell_type": "code",
"execution_count": 112,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/20\n",
"313/313 [==============================] - 4s 8ms/step - loss: 0.6910 - accuracy: 0.5095 - val_loss: 0.6825 - val_accuracy: 0.5645\n",
"Epoch 2/20\n",
"313/313 [==============================] - 2s 7ms/step - loss: 0.6678 - accuracy: 0.5659 - val_loss: 0.6635 - val_accuracy: 0.6105\n",
"Epoch 3/20\n",
"313/313 [==============================] - 2s 7ms/step - loss: 0.6504 - accuracy: 0.5766 - val_loss: 0.6521 - val_accuracy: 0.6110\n",
"Epoch 4/20\n",
"313/313 [==============================] - 2s 8ms/step - loss: 0.6347 - accuracy: 0.5980 - val_loss: 0.6224 - val_accuracy: 0.6445\n",
"Epoch 5/20\n",
"313/313 [==============================] - 2s 7ms/step - loss: 0.6054 - accuracy: 0.6361 - val_loss: 0.5779 - val_accuracy: 0.6980\n",
"Epoch 6/20\n",
"313/313 [==============================] - 2s 7ms/step - loss: 0.5414 - accuracy: 0.7093 - val_loss: 0.4695 - val_accuracy: 0.7795\n",
"Epoch 7/20\n",
"313/313 [==============================] - 2s 7ms/step - loss: 0.3756 - accuracy: 0.8418 - val_loss: 0.2685 - val_accuracy: 0.9115\n",
"Epoch 8/20\n",
"313/313 [==============================] - 2s 7ms/step - loss: 0.2601 - accuracy: 0.9044 - val_loss: 0.1534 - val_accuracy: 0.9615\n",
"Epoch 9/20\n",
"313/313 [==============================] - 2s 7ms/step - loss: 0.1774 - accuracy: 0.9427 - val_loss: 0.1063 - val_accuracy: 0.9735\n",
"Epoch 10/20\n",
"313/313 [==============================] - 2s 7ms/step - loss: 0.0624 - accuracy: 0.9826 - val_loss: 0.0219 - val_accuracy: 0.9975\n",
"Epoch 11/20\n",
"313/313 [==============================] - 2s 7ms/step - loss: 0.0371 - accuracy: 0.9914 - val_loss: 0.0055 - val_accuracy: 1.0000\n",
"Epoch 12/20\n",
"313/313 [==============================] - 2s 7ms/step - loss: 0.0029 - accuracy: 0.9995 - val_loss: 8.7265e-04 - val_accuracy: 1.0000\n",
"Epoch 13/20\n",
"313/313 [==============================] - 2s 7ms/step - loss: 6.7552e-04 - accuracy: 1.0000 - val_loss: 4.9408e-04 - val_accuracy: 1.0000\n",
"Epoch 14/20\n",
"313/313 [==============================] - 2s 7ms/step - loss: 4.4514e-04 - accuracy: 1.0000 - val_loss: 3.6322e-04 - val_accuracy: 1.0000\n",
"Epoch 15/20\n",
"313/313 [==============================] - 2s 7ms/step - loss: 3.3943e-04 - accuracy: 1.0000 - val_loss: 2.8524e-04 - val_accuracy: 1.0000\n",
"Epoch 16/20\n",
"313/313 [==============================] - 2s 7ms/step - loss: 2.7723e-04 - accuracy: 1.0000 - val_loss: 2.3880e-04 - val_accuracy: 1.0000\n",
"Epoch 17/20\n",
"313/313 [==============================] - 2s 7ms/step - loss: 2.3477e-04 - accuracy: 1.0000 - val_loss: 2.0363e-04 - val_accuracy: 1.0000\n",
"Epoch 18/20\n",
"313/313 [==============================] - 2s 7ms/step - loss: 2.0382e-04 - accuracy: 1.0000 - val_loss: 1.7760e-04 - val_accuracy: 1.0000\n",
"Epoch 19/20\n",
"313/313 [==============================] - 2s 7ms/step - loss: 1.8077e-04 - accuracy: 1.0000 - val_loss: 1.5916e-04 - val_accuracy: 1.0000\n",
"Epoch 20/20\n",
"313/313 [==============================] - 2s 8ms/step - loss: 1.6246e-04 - accuracy: 1.0000 - val_loss: 1.4362e-04 - val_accuracy: 1.0000\n"
]
}
],
"source": [
"np.random.seed(42)\n",
"tf.random.set_seed(42)\n",
"\n",
"embedding_size = 5\n",
"\n",
"model = tf.keras.Sequential([\n",
" tf.keras.layers.InputLayer(input_shape=[None], dtype=tf.int32, ragged=True),\n",
" tf.keras.layers.Embedding(input_dim=len(POSSIBLE_CHARS),\n",
" output_dim=embedding_size),\n",
" tf.keras.layers.GRU(30),\n",
" tf.keras.layers.Dense(1, activation=\"sigmoid\")\n",
"])\n",
"optimizer = tf.keras.optimizers.SGD(learning_rate=0.02, momentum = 0.95,\n",
" nesterov=True)\n",
"model.compile(loss=\"binary_crossentropy\", optimizer=optimizer,\n",
" metrics=[\"accuracy\"])\n",
"history = model.fit(X_train, y_train, epochs=20,\n",
" validation_data=(X_valid, y_valid))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's test our RNN on two tricky strings: the first one is bad while the second one is good. They only differ by the second to last character. If the RNN gets this right, it shows that it managed to notice the pattern that the second letter should always be equal to the second to last letter. That requires a fairly long short-term memory (which is the reason why we used a GRU cell)."
]
},
{
"cell_type": "code",
"execution_count": 113,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Estimated probability that these are Reber strings:\n",
"BPBTSSSSSSSXXTTVPXVPXTTTTTVVETE: 0.02%\n",
"BPBTSSSSSSSXXTTVPXVPXTTTTTVVEPE: 99.99%\n"
]
}
],
"source": [
"test_strings = [\"BPBTSSSSSSSXXTTVPXVPXTTTTTVVETE\",\n",
" \"BPBTSSSSSSSXXTTVPXVPXTTTTTVVEPE\"]\n",
"X_test = tf.ragged.constant([string_to_ids(s) for s in test_strings], ragged_rank=1)\n",
"\n",
"y_proba = model.predict(X_test)\n",
"print()\n",
"print(\"Estimated probability that these are Reber strings:\")\n",
"for index, string in enumerate(test_strings):\n",
" print(\"{}: {:.2f}%\".format(string, 100 * y_proba[index][0]))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Ta-da! It worked fine. The RNN found the correct answers with very high confidence. :)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 9.\n",
"_Exercise: Train an Encoder–Decoder model that can convert a date string from one format to another (e.g., from \"April 22, 2019\" to \"2019-04-22\")._"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's start by creating the dataset. We will use random days between 1000-01-01 and 9999-12-31:"
]
},
{
"cell_type": "code",
"execution_count": 114,
"metadata": {},
"outputs": [],
"source": [
"from datetime import date\n",
"\n",
"# cannot use strftime()'s %B format since it depends on the locale\n",
"MONTHS = [\"January\", \"February\", \"March\", \"April\", \"May\", \"June\",\n",
" \"July\", \"August\", \"September\", \"October\", \"November\", \"December\"]\n",
"\n",
"def random_dates(n_dates):\n",
" min_date = date(1000, 1, 1).toordinal()\n",
" max_date = date(9999, 12, 31).toordinal()\n",
"\n",
" ordinals = np.random.randint(max_date - min_date, size=n_dates) + min_date\n",
" dates = [date.fromordinal(ordinal) for ordinal in ordinals]\n",
"\n",
" x = [MONTHS[dt.month - 1] + \" \" + dt.strftime(\"%d, %Y\") for dt in dates]\n",
" y = [dt.isoformat() for dt in dates]\n",
" return x, y"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here are a few random dates, displayed in both the input format and the target format:"
]
},
{
"cell_type": "code",
"execution_count": 115,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Input Target \n",
"--------------------------------------------------\n",
"September 20, 7075 7075-09-20 \n",
"May 15, 8579 8579-05-15 \n",
"January 11, 7103 7103-01-11 \n"
]
}
],
"source": [
"np.random.seed(42)\n",
"\n",
"n_dates = 3\n",
"x_example, y_example = random_dates(n_dates)\n",
"print(\"{:25s}{:25s}\".format(\"Input\", \"Target\"))\n",
"print(\"-\" * 50)\n",
"for idx in range(n_dates):\n",
" print(\"{:25s}{:25s}\".format(x_example[idx], y_example[idx]))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's get the list of all possible characters in the inputs:"
]
},
{
"cell_type": "code",
"execution_count": 116,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"' ,0123456789ADFJMNOSabceghilmnoprstuvy'"
]
},
"execution_count": 116,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"INPUT_CHARS = \"\".join(sorted(set(\"\".join(MONTHS) + \"0123456789, \")))\n",
"INPUT_CHARS"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And here's the list of possible characters in the outputs:"
]
},
{
"cell_type": "code",
"execution_count": 117,
"metadata": {},
"outputs": [],
"source": [
"OUTPUT_CHARS = \"0123456789-\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's write a function to convert a string to a list of character IDs, as we did in the previous exercise:"
]
},
{
"cell_type": "code",
"execution_count": 118,
"metadata": {},
"outputs": [],
"source": [
"def date_str_to_ids(date_str, chars=INPUT_CHARS):\n",
" return [chars.index(c) for c in date_str]"
]
},
{
"cell_type": "code",
"execution_count": 119,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[19, 23, 31, 34, 23, 28, 21, 23, 32, 0, 4, 2, 1, 0, 9, 2, 9, 7]"
]
},
"execution_count": 119,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"date_str_to_ids(x_example[0], INPUT_CHARS)"
]
},
{
"cell_type": "code",
"execution_count": 120,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[7, 0, 7, 5, 10, 0, 9, 10, 2, 0]"
]
},
"execution_count": 120,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"date_str_to_ids(y_example[0], OUTPUT_CHARS)"
]
},
{
"cell_type": "code",
"execution_count": 121,
"metadata": {},
"outputs": [],
"source": [
"def prepare_date_strs(date_strs, chars=INPUT_CHARS):\n",
" X_ids = [date_str_to_ids(dt, chars) for dt in date_strs]\n",
" X = tf.ragged.constant(X_ids, ragged_rank=1)\n",
" return (X + 1).to_tensor() # using 0 as the padding token ID\n",
"\n",
"def create_dataset(n_dates):\n",
" x, y = random_dates(n_dates)\n",
" return prepare_date_strs(x, INPUT_CHARS), prepare_date_strs(y, OUTPUT_CHARS)"
]
},
{
"cell_type": "code",
"execution_count": 122,
"metadata": {},
"outputs": [],
"source": [
"np.random.seed(42)\n",
"\n",
"X_train, Y_train = create_dataset(10000)\n",
"X_valid, Y_valid = create_dataset(2000)\n",
"X_test, Y_test = create_dataset(2000)"
]
},
{
"cell_type": "code",
"execution_count": 123,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 123,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"Y_train[0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### First version: a very basic seq2seq model"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's first try the simplest possible model: we feed in the input sequence, which first goes through the encoder (an embedding layer followed by a single LSTM layer), which outputs a vector, then it goes through a decoder (a single LSTM layer, followed by a dense output layer), which outputs a sequence of vectors, each representing the estimated probabilities for all possible output character.\n",
"\n",
"Since the decoder expects a sequence as input, we repeat the vector (which is output by the encoder) as many times as the longest possible output sequence."
]
},
{
"cell_type": "code",
"execution_count": 124,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/20\n",
"313/313 [==============================] - 10s 23ms/step - loss: 1.8150 - accuracy: 0.3489 - val_loss: 1.3726 - val_accuracy: 0.4939\n",
"Epoch 2/20\n",
"313/313 [==============================] - 7s 22ms/step - loss: 1.2447 - accuracy: 0.5510 - val_loss: 1.0725 - val_accuracy: 0.6115\n",
"Epoch 3/20\n",
"313/313 [==============================] - 7s 23ms/step - loss: 1.0937 - accuracy: 0.6125 - val_loss: 1.0548 - val_accuracy: 0.6130\n",
"Epoch 4/20\n",
"313/313 [==============================] - 7s 23ms/step - loss: 1.0032 - accuracy: 0.6413 - val_loss: 3.8747 - val_accuracy: 0.1788\n",
"Epoch 5/20\n",
"313/313 [==============================] - 8s 26ms/step - loss: 0.8159 - accuracy: 0.7023 - val_loss: 0.6623 - val_accuracy: 0.7474\n",
"Epoch 6/20\n",
"313/313 [==============================] - 8s 26ms/step - loss: 0.5645 - accuracy: 0.7795 - val_loss: 0.5005 - val_accuracy: 0.8032\n",
"Epoch 7/20\n",
"313/313 [==============================] - 8s 26ms/step - loss: 0.5037 - accuracy: 0.8103 - val_loss: 0.3798 - val_accuracy: 0.8500\n",
"Epoch 8/20\n",
"313/313 [==============================] - 8s 26ms/step - loss: 0.3131 - accuracy: 0.8795 - val_loss: 0.2582 - val_accuracy: 0.9043\n",
"Epoch 9/20\n",
"313/313 [==============================] - 8s 26ms/step - loss: 0.2141 - accuracy: 0.9280 - val_loss: 0.1637 - val_accuracy: 0.9498\n",
"Epoch 10/20\n",
"313/313 [==============================] - 9s 28ms/step - loss: 0.1282 - accuracy: 0.9650 - val_loss: 0.0918 - val_accuracy: 0.9774\n",
"Epoch 11/20\n",
"313/313 [==============================] - 9s 28ms/step - loss: 0.0669 - accuracy: 0.9871 - val_loss: 0.3368 - val_accuracy: 0.8871\n",
"Epoch 12/20\n",
"313/313 [==============================] - 10s 32ms/step - loss: 0.1551 - accuracy: 0.9662 - val_loss: 0.0398 - val_accuracy: 0.9949\n",
"Epoch 13/20\n",
"313/313 [==============================] - 9s 29ms/step - loss: 0.0291 - accuracy: 0.9969 - val_loss: 0.0240 - val_accuracy: 0.9984\n",
"Epoch 14/20\n",
"313/313 [==============================] - 9s 30ms/step - loss: 0.0182 - accuracy: 0.9986 - val_loss: 0.0161 - val_accuracy: 0.9993\n",
"Epoch 15/20\n",
"313/313 [==============================] - 9s 30ms/step - loss: 0.0119 - accuracy: 0.9995 - val_loss: 0.0112 - val_accuracy: 0.9997\n",
"Epoch 16/20\n",
"313/313 [==============================] - 10s 32ms/step - loss: 0.0082 - accuracy: 0.9998 - val_loss: 0.0083 - val_accuracy: 0.9999\n",
"Epoch 17/20\n",
"313/313 [==============================] - 10s 33ms/step - loss: 0.0059 - accuracy: 0.9999 - val_loss: 0.0058 - val_accuracy: 0.9999\n",
"Epoch 18/20\n",
"313/313 [==============================] - 11s 34ms/step - loss: 0.0042 - accuracy: 1.0000 - val_loss: 0.0043 - val_accuracy: 0.9999\n",
"Epoch 19/20\n",
"313/313 [==============================] - 10s 33ms/step - loss: 0.0031 - accuracy: 1.0000 - val_loss: 0.0034 - val_accuracy: 0.9999\n",
"Epoch 20/20\n",
"313/313 [==============================] - 12s 40ms/step - loss: 0.0024 - accuracy: 1.0000 - val_loss: 0.0026 - val_accuracy: 1.0000\n"
]
}
],
"source": [
"embedding_size = 32\n",
"max_output_length = Y_train.shape[1]\n",
"\n",
"np.random.seed(42)\n",
"tf.random.set_seed(42)\n",
"\n",
"encoder = tf.keras.Sequential([\n",
" tf.keras.layers.Embedding(input_dim=len(INPUT_CHARS) + 1,\n",
" output_dim=embedding_size,\n",
" input_shape=[None]),\n",
" tf.keras.layers.LSTM(128)\n",
"])\n",
"\n",
"decoder = tf.keras.Sequential([\n",
" tf.keras.layers.LSTM(128, return_sequences=True),\n",
" tf.keras.layers.Dense(len(OUTPUT_CHARS) + 1, activation=\"softmax\")\n",
"])\n",
"\n",
"model = tf.keras.Sequential([\n",
" encoder,\n",
" tf.keras.layers.RepeatVector(max_output_length),\n",
" decoder\n",
"])\n",
"\n",
"optimizer = tf.keras.optimizers.Nadam()\n",
"model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer,\n",
" metrics=[\"accuracy\"])\n",
"history = model.fit(X_train, Y_train, epochs=20,\n",
" validation_data=(X_valid, Y_valid))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Looks great, we reach 100% validation accuracy! Let's use the model to make some predictions. We will need to be able to convert a sequence of character IDs to a readable string:"
]
},
{
"cell_type": "code",
"execution_count": 125,
"metadata": {},
"outputs": [],
"source": [
"def ids_to_date_strs(ids, chars=OUTPUT_CHARS):\n",
" return [\"\".join([(\"?\" + chars)[index] for index in sequence])\n",
" for sequence in ids]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we can use the model to convert some dates"
]
},
{
"cell_type": "code",
"execution_count": 126,
"metadata": {},
"outputs": [],
"source": [
"X_new = prepare_date_strs([\"September 17, 2009\", \"July 14, 1789\"])"
]
},
{
"cell_type": "code",
"execution_count": 127,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2009-09-17\n",
"1789-07-14\n"
]
}
],
"source": [
"ids = model.predict(X_new).argmax(axis=-1)\n",
"for date_str in ids_to_date_strs(ids):\n",
" print(date_str)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Perfect! :)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"However, since the model was only trained on input strings of length 18 (which is the length of the longest date), it does not perform well if we try to use it to make predictions on shorter sequences:"
]
},
{
"cell_type": "code",
"execution_count": 128,
"metadata": {},
"outputs": [],
"source": [
"X_new = prepare_date_strs([\"May 02, 2020\", \"July 14, 1789\"])"
]
},
{
"cell_type": "code",
"execution_count": 129,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2020-02-02\n",
"1789-01-14\n"
]
}
],
"source": [
"ids = model.predict(X_new).argmax(axis=-1)\n",
"for date_str in ids_to_date_strs(ids):\n",
" print(date_str)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Oops! We need to ensure that we always pass sequences of the same length as during training, using padding if necessary. Let's write a little helper function for that:"
]
},
{
"cell_type": "code",
"execution_count": 130,
"metadata": {},
"outputs": [],
"source": [
"max_input_length = X_train.shape[1]\n",
"\n",
"def prepare_date_strs_padded(date_strs):\n",
" X = prepare_date_strs(date_strs)\n",
" if X.shape[1] < max_input_length:\n",
" X = tf.pad(X, [[0, 0], [0, max_input_length - X.shape[1]]])\n",
" return X\n",
"\n",
"def convert_date_strs(date_strs):\n",
" X = prepare_date_strs_padded(date_strs)\n",
" ids = model.predict(X).argmax(axis=-1)\n",
" return ids_to_date_strs(ids)"
]
},
{
"cell_type": "code",
"execution_count": 131,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['2020-05-02', '1789-07-14']"
]
},
"execution_count": 131,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"convert_date_strs([\"May 02, 2020\", \"July 14, 1789\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Cool! Granted, there are certainly much easier ways to write a date conversion tool (e.g., using regular expressions or even basic string manipulation), but you have to admit that using neural networks is way cooler. ;-)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"However, real-life sequence-to-sequence problems will usually be harder, so for the sake of completeness, let's build a more powerful model."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Second version: feeding the shifted targets to the decoder (teacher forcing)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Instead of feeding the decoder a simple repetition of the encoder's output vector, we can feed it the target sequence, shifted by one time step to the right. This way, at each time step the decoder will know what the previous target character was. This should help is tackle more complex sequence-to-sequence problems.\n",
"\n",
"Since the first output character of each target sequence has no previous character, we will need a new token to represent the start-of-sequence (sos).\n",
"\n",
"During inference, we won't know the target, so what will we feed the decoder? We can just predict one character at a time, starting with an sos token, then feeding the decoder all the characters that were predicted so far (we will look at this in more details later in this notebook).\n",
"\n",
"But if the decoder's LSTM expects to get the previous target as input at each step, how shall we pass it it the vector output by the encoder? Well, one option is to ignore the output vector, and instead use the encoder's LSTM state as the initial state of the decoder's LSTM (which requires that encoder's LSTM must have the same number of units as the decoder's LSTM).\n",
"\n",
"Now let's create the decoder's inputs (for training, validation and testing). The sos token will be represented using the last possible output character's ID + 1."
]
},
{
"cell_type": "code",
"execution_count": 132,
"metadata": {},
"outputs": [],
"source": [
"sos_id = len(OUTPUT_CHARS) + 1\n",
"\n",
"def shifted_output_sequences(Y):\n",
" sos_tokens = tf.fill(dims=(len(Y), 1), value=sos_id)\n",
" return tf.concat([sos_tokens, Y[:, :-1]], axis=1)\n",
"\n",
"X_train_decoder = shifted_output_sequences(Y_train)\n",
"X_valid_decoder = shifted_output_sequences(Y_valid)\n",
"X_test_decoder = shifted_output_sequences(Y_test)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's take a look at the decoder's training inputs:"
]
},
{
"cell_type": "code",
"execution_count": 133,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 133,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X_train_decoder"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's build the model. It's not a simple sequential model anymore, so let's use the functional API:"
]
},
{
"cell_type": "code",
"execution_count": 134,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/10\n",
"313/313 [==============================] - 11s 27ms/step - loss: 1.6824 - accuracy: 0.3734 - val_loss: 1.4054 - val_accuracy: 0.4681\n",
"Epoch 2/10\n",
"313/313 [==============================] - 8s 26ms/step - loss: 1.1935 - accuracy: 0.5550 - val_loss: 0.8868 - val_accuracy: 0.6750\n",
"Epoch 3/10\n",
"313/313 [==============================] - 8s 26ms/step - loss: 0.6403 - accuracy: 0.7700 - val_loss: 0.3493 - val_accuracy: 0.8978\n",
"Epoch 4/10\n",
"313/313 [==============================] - 8s 26ms/step - loss: 0.2292 - accuracy: 0.9423 - val_loss: 0.1254 - val_accuracy: 0.9782\n",
"Epoch 5/10\n",
"313/313 [==============================] - 8s 26ms/step - loss: 0.0694 - accuracy: 0.9932 - val_loss: 0.0441 - val_accuracy: 0.9982\n",
"Epoch 6/10\n",
"313/313 [==============================] - 9s 29ms/step - loss: 0.0576 - accuracy: 0.9923 - val_loss: 0.0280 - val_accuracy: 0.9988\n",
"Epoch 7/10\n",
"313/313 [==============================] - 8s 26ms/step - loss: 0.0179 - accuracy: 0.9998 - val_loss: 0.0143 - val_accuracy: 0.9999\n",
"Epoch 8/10\n",
"313/313 [==============================] - 6s 18ms/step - loss: 0.0107 - accuracy: 0.9999 - val_loss: 0.0092 - val_accuracy: 0.9999\n",
"Epoch 9/10\n",
"313/313 [==============================] - 6s 20ms/step - loss: 0.0070 - accuracy: 1.0000 - val_loss: 0.0065 - val_accuracy: 0.9999\n",
"Epoch 10/10\n",
"313/313 [==============================] - 6s 18ms/step - loss: 0.0050 - accuracy: 1.0000 - val_loss: 0.0047 - val_accuracy: 0.9999\n"
]
}
],
"source": [
"encoder_embedding_size = 32\n",
"decoder_embedding_size = 32\n",
"lstm_units = 128\n",
"\n",
"np.random.seed(42)\n",
"tf.random.set_seed(42)\n",
"\n",
"encoder_input = tf.keras.layers.Input(shape=[None], dtype=tf.int32)\n",
"encoder_embedding = tf.keras.layers.Embedding(\n",
" input_dim=len(INPUT_CHARS) + 1,\n",
" output_dim=encoder_embedding_size)(encoder_input)\n",
"_, encoder_state_h, encoder_state_c = tf.keras.layers.LSTM(\n",
" lstm_units, return_state=True)(encoder_embedding)\n",
"encoder_state = [encoder_state_h, encoder_state_c]\n",
"\n",
"decoder_input = tf.keras.layers.Input(shape=[None], dtype=tf.int32)\n",
"decoder_embedding = tf.keras.layers.Embedding(\n",
" input_dim=len(OUTPUT_CHARS) + 2,\n",
" output_dim=decoder_embedding_size)(decoder_input)\n",
"decoder_lstm_output = tf.keras.layers.LSTM(lstm_units, return_sequences=True)(\n",
" decoder_embedding, initial_state=encoder_state)\n",
"decoder_output = tf.keras.layers.Dense(len(OUTPUT_CHARS) + 1,\n",
" activation=\"softmax\")(decoder_lstm_output)\n",
"\n",
"model = tf.keras.Model(inputs=[encoder_input, decoder_input],\n",
" outputs=[decoder_output])\n",
"\n",
"optimizer = tf.keras.optimizers.Nadam()\n",
"model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer,\n",
" metrics=[\"accuracy\"])\n",
"history = model.fit([X_train, X_train_decoder], Y_train, epochs=10,\n",
" validation_data=([X_valid, X_valid_decoder], Y_valid))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This model also reaches 100% validation accuracy, but it does so even faster."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's once again use the model to make some predictions. This time we need to predict characters one by one."
]
},
{
"cell_type": "code",
"execution_count": 135,
"metadata": {},
"outputs": [],
"source": [
"sos_id = len(OUTPUT_CHARS) + 1\n",
"\n",
"def predict_date_strs(date_strs):\n",
" X = prepare_date_strs_padded(date_strs)\n",
" Y_pred = tf.fill(dims=(len(X), 1), value=sos_id)\n",
" for index in range(max_output_length):\n",
" pad_size = max_output_length - Y_pred.shape[1]\n",
" X_decoder = tf.pad(Y_pred, [[0, 0], [0, pad_size]])\n",
" Y_probas_next = model.predict([X, X_decoder])[:, index:index+1]\n",
" Y_pred_next = tf.argmax(Y_probas_next, axis=-1, output_type=tf.int32)\n",
" Y_pred = tf.concat([Y_pred, Y_pred_next], axis=1)\n",
" return ids_to_date_strs(Y_pred[:, 1:])"
]
},
{
"cell_type": "code",
"execution_count": 136,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['1789-07-14', '2020-05-01']"
]
},
"execution_count": 136,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"predict_date_strs([\"July 14, 1789\", \"May 01, 2020\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Works fine! Next, feel free to write a Transformer version. :)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 10.\n",
"_Exercise: Go through Keras's tutorial for [Natural language image search with a Dual Encoder](https://homl.info/dualtuto). You will learn how to build a model capable of representing both images and text within the same embedding space. This makes it possible to search for images using a text prompt, like in the [CLIP model](https://openai.com/blog/clip/) by OpenAI._ "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Just click the link and follow the instructions."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 11.\n",
"_Exercise: Use the Transformers library to download a pretrained language model capable of generating text (e.g., GPT), and try generating more convincing Shakespearean text. You will need to use the model's `generate()` method—see Hugging Face's documentation for more details._"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**WARNING**: I've updated this solution to use PyTorch instead of TensorFlow since the Transformers library no longer supports TensorFlow (or JAX)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First, let's load a pretrained model. In this example, we will use OpenAI's GPT model, with an additional Language Model on top (just a linear layer with weights tied to the input embeddings). Let's import it and load the pretrained weights (this will download about 445MB of data to `~/.cache/torch/transformers`):"
]
},
{
"cell_type": "code",
"execution_count": 137,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "2cf2278e9105443ba7cb1ab12e91ac54",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"generation_config.json: 0%| | 0.00/74.0 [00:00
![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
\n",
" ![](\"https://kaggle.com/static/images/open-in-kaggle.svg\")
\n",
"