"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_random_elements(datasets[\"train\"])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "n9qywopnIrJH"
},
"source": [
"## Preprocessing the training data"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YVx71GdAIrJH"
},
"source": [
"Before we can feed those texts to our model, we need to preprocess them. This is done by a π€ Transformers `Tokenizer` which will (as the name indicates) tokenize the inputs (including converting the tokens to their corresponding IDs in the pretrained vocabulary) and put it in a format the model expects, as well as generate the other inputs that model requires.\n",
"\n",
"To do all of this, we instantiate our tokenizer with the `AutoTokenizer.from_pretrained` method, which will ensure:\n",
"\n",
"- we get a tokenizer that corresponds to the model architecture we want to use,\n",
"- we download the vocabulary used when pretraining this specific checkpoint.\n",
"\n",
"That vocabulary will be cached, so it's not downloaded again the next time we run the cell."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "eXNLu_-nIrJI"
},
"outputs": [],
"source": [
"from transformers import AutoTokenizer\n",
" \n",
"tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Vl6IidfdIrJK"
},
"source": [
"The following assertion ensures that our tokenizer is a fast tokenizers (backed by Rust) from the π€ Tokenizers library. Those fast tokenizers are available for almost all models, and we will need some of the special features they have for our preprocessing."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import transformers\n",
"assert isinstance(tokenizer, transformers.PreTrainedTokenizerFast)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can check which type of models have a fast tokenizer available and which don't on the [big table of models](https://huggingface.co/transformers/index.html#bigtable)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rowT4iCLIrJK"
},
"source": [
"You can directly call this tokenizer on two sentences (one for the answer, one for the context):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "a5hBlsrHIrJL",
"outputId": "acdaa98a-a8cd-4a20-89b8-cc26437bbe90"
},
"outputs": [
{
"data": {
"text/plain": [
"{'input_ids': [101, 2054, 2003, 2115, 2171, 1029, 102, 2026, 2171, 2003, 25353, 22144, 2378, 1012, 102], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tokenizer(\"What is your name?\", \"My name is Sylvain.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Depending on the model you selected, you will see different keys in the dictionary returned by the cell above. They don't matter much for what we're doing here (just know they are required by the model we will instantiate later), you can learn more about them in [this tutorial](https://huggingface.co/transformers/preprocessing.html) if you're interested.\n",
"\n",
"Now one specific thing for the preprocessing in question answering is how to deal with very long documents. We usually truncate them in other tasks, when they are longer than the model maximum sentence length, but here, removing part of the the context might result in losing the answer we are looking for. To deal with this, we will allow one (long) example in our dataset to give several input features, each of length shorter than the maximum length of the model (or the one we set as a hyper-parameter). Also, just in case the answer lies at the point we split a long context, we allow some overlap between the features we generate controlled by the hyper-parameter `doc_stride`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"max_length = 384 # The maximum length of a feature (question and context)\n",
"doc_stride = 128 # The authorized overlap between two part of the context when splitting it is needed."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's find one long example in our dataset:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"for i, example in enumerate(datasets[\"train\"]):\n",
" if len(tokenizer(example[\"question\"], example[\"context\"])[\"input_ids\"]) > 384:\n",
" break\n",
"example = datasets[\"train\"][i]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Without any truncation, we get the following length for the input IDs:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"396"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(tokenizer(example[\"question\"], example[\"context\"])[\"input_ids\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, if we just truncate, we will lose information (and possibly the answer to our question):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"384"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(tokenizer(example[\"question\"], example[\"context\"], max_length=max_length, truncation=\"only_second\")[\"input_ids\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that we never want to truncate the question, only the context, else the `only_second` truncation picked. Now, our tokenizer can automatically return us a list of features capped by a certain maximum length, with the overlap we talked above, we just have to tell it with `return_overflowing_tokens=True` and by passing the stride:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"tokenized_example = tokenizer(\n",
" example[\"question\"],\n",
" example[\"context\"],\n",
" max_length=max_length,\n",
" truncation=\"only_second\",\n",
" return_overflowing_tokens=True,\n",
" stride=doc_stride\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we don't have one list of `input_ids`, but several: "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[384, 157]"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"[len(x) for x in tokenized_example[\"input_ids\"]]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And if we decode them, we can see the overlap:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[CLS] how many wins does the notre dame men's basketball team have? [SEP] the men's basketball team has over 1, 600 wins, one of only 12 schools who have reached that mark, and have appeared in 28 ncaa tournaments. former player austin carr holds the record for most points scored in a single game of the tournament with 61. although the team has never won the ncaa tournament, they were named by the helms athletic foundation as national champions twice. the team has orchestrated a number of upsets of number one ranked teams, the most notable of which was ending ucla's record 88 - game winning streak in 1974. the team has beaten an additional eight number - one teams, and those nine wins rank second, to ucla's 10, all - time in wins against the top team. the team plays in newly renovated purcell pavilion ( within the edmund p. joyce center ), which reopened for the beginning of the 2009 β 2010 season. the team is coached by mike brey, who, as of the 2014 β 15 season, his fifteenth at notre dame, has achieved a 332 - 165 record. in 2009 they were invited to the nit, where they advanced to the semifinals but were beaten by penn state who went on and beat baylor in the championship. the 2010 β 11 team concluded its regular season ranked number seven in the country, with a record of 25 β 5, brey's fifth straight 20 - win season, and a second - place finish in the big east. during the 2014 - 15 season, the team went 32 - 6 and won the acc conference tournament, later advancing to the elite 8, where the fighting irish lost on a missed buzzer - beater against then undefeated kentucky. led by nba draft picks jerian grant and pat connaughton, the fighting irish beat the eventual national champion duke blue devils twice during the season. the 32 wins were [SEP]\n",
"[CLS] how many wins does the notre dame men's basketball team have? [SEP] championship. the 2010 β 11 team concluded its regular season ranked number seven in the country, with a record of 25 β 5, brey's fifth straight 20 - win season, and a second - place finish in the big east. during the 2014 - 15 season, the team went 32 - 6 and won the acc conference tournament, later advancing to the elite 8, where the fighting irish lost on a missed buzzer - beater against then undefeated kentucky. led by nba draft picks jerian grant and pat connaughton, the fighting irish beat the eventual national champion duke blue devils twice during the season. the 32 wins were the most by the fighting irish team since 1908 - 09. [SEP]\n"
]
}
],
"source": [
"for x in tokenized_example[\"input_ids\"][:2]:\n",
" print(tokenizer.decode(x))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now this will give us some work to properly treat the answers: we need to find in which of those features the answer actually is, and where exactly in that feature. The models we will use require the start and end positions of these answers in the tokens, so we will also need to to map parts of the original context to some tokens. Thankfully, the tokenizer we're using can help us with that by returning an `offset_mapping`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[(0, 0), (0, 3), (4, 8), (9, 13), (14, 18), (19, 22), (23, 28), (29, 33), (34, 37), (37, 38), (38, 39), (40, 50), (51, 55), (56, 60), (60, 61), (0, 0), (0, 3), (4, 7), (7, 8), (8, 9), (10, 20), (21, 25), (26, 29), (30, 34), (35, 36), (36, 37), (37, 40), (41, 45), (45, 46), (47, 50), (51, 53), (54, 58), (59, 61), (62, 69), (70, 73), (74, 78), (79, 86), (87, 91), (92, 96), (96, 97), (98, 101), (102, 106), (107, 115), (116, 118), (119, 121), (122, 126), (127, 138), (138, 139), (140, 146), (147, 153), (154, 160), (161, 165), (166, 171), (172, 175), (176, 182), (183, 186), (187, 191), (192, 198), (199, 205), (206, 208), (209, 210), (211, 217), (218, 222), (223, 225), (226, 229), (230, 240), (241, 245), (246, 248), (248, 249), (250, 258), (259, 262), (263, 267), (268, 271), (272, 277), (278, 281), (282, 285), (286, 290), (291, 301), (301, 302), (303, 307), (308, 312), (313, 318), (319, 321), (322, 325), (326, 330), (330, 331), (332, 340), (341, 351), (352, 354), (355, 363), (364, 373), (374, 379), (379, 380), (381, 384), (385, 389), (390, 393), (394, 406), (407, 408), (409, 415), (416, 418)]\n"
]
}
],
"source": [
"tokenized_example = tokenizer(\n",
" example[\"question\"],\n",
" example[\"context\"],\n",
" max_length=max_length,\n",
" truncation=\"only_second\",\n",
" return_overflowing_tokens=True,\n",
" return_offsets_mapping=True,\n",
" stride=doc_stride\n",
")\n",
"print(tokenized_example[\"offset_mapping\"][0][:100])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This gives, for each index of our input IDS, the corresponding start and end character in the original text that gave our token. The very first token (`[CLS]`) has (0, 0) because it doesn't correspond to any part of the question/answer, then the second token is the same as the characters 0 to 3 of the question:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"how How\n"
]
}
],
"source": [
"first_token_id = tokenized_example[\"input_ids\"][0][1]\n",
"offsets = tokenized_example[\"offset_mapping\"][0][1]\n",
"print(tokenizer.convert_ids_to_tokens([first_token_id])[0], example[\"question\"][offsets[0]:offsets[1]])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So we can use this mapping to find the position of the start and end tokens of our answer in a given feature. We just have to distinguish which parts of the offsets correspond to the question and which part correspond to the context, this is where the `sequence_ids` method of our `tokenized_example` can be useful:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[None, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, None, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, None]\n"
]
}
],
"source": [
"sequence_ids = tokenized_example.sequence_ids()\n",
"print(sequence_ids)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It returns `None` for the special tokens, then 0 or 1 depending on whether the corresponding token comes from the first sentence past (the question) or the second (the context). Now with all of this, we can find the first and last token of the answer in one of our input feature (or if the answer is not in this feature):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"23 26\n"
]
}
],
"source": [
"answers = example[\"answers\"]\n",
"start_char = answers[\"answer_start\"][0]\n",
"end_char = start_char + len(answers[\"text\"][0])\n",
"\n",
"# Start token index of the current span in the text.\n",
"token_start_index = 0\n",
"while sequence_ids[token_start_index] != 1:\n",
" token_start_index += 1\n",
"\n",
"# End token index of the current span in the text.\n",
"token_end_index = len(tokenized_example[\"input_ids\"][0]) - 1\n",
"while sequence_ids[token_end_index] != 1:\n",
" token_end_index -= 1\n",
"\n",
"# Detect if the answer is out of the span (in which case this feature is labeled with the CLS index).\n",
"offsets = tokenized_example[\"offset_mapping\"][0]\n",
"if (offsets[token_start_index][0] <= start_char and offsets[token_end_index][1] >= end_char):\n",
" # Move the token_start_index and token_end_index to the two ends of the answer.\n",
" # Note: we could go after the last offset if the answer is the last word (edge case).\n",
" while token_start_index < len(offsets) and offsets[token_start_index][0] <= start_char:\n",
" token_start_index += 1\n",
" start_position = token_start_index - 1\n",
" while offsets[token_end_index][1] >= end_char:\n",
" token_end_index -= 1\n",
" end_position = token_end_index + 1\n",
" print(start_position, end_position)\n",
"else:\n",
" print(\"The answer is not in this feature.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And we can double check that it is indeed the theoretical answer:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"over 1, 600\n",
"over 1,600\n"
]
}
],
"source": [
"print(tokenizer.decode(tokenized_example[\"input_ids\"][0][start_position: end_position+1]))\n",
"print(answers[\"text\"][0])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For this notebook to work with any kind of models, we need to account for the special case where the model expects padding on the left (in which case we switch the order of the question and the context):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pad_on_right = tokenizer.padding_side == \"right\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's put everything together in one function we will apply to our training set. In the case of impossible answers (the answer is in another feature given by an example with a long context), we set the cls index for both the start and end position. We could also simply discard those examples from the training set if the flag `allow_impossible_answers` is `False`. Since the preprocessing is already complex enough as it is, we've kept is simple for this part."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def prepare_train_features(examples):\n",
" # Some of the questions have lots of whitespace on the left, which is not useful and will make the\n",
" # truncation of the context fail (the tokenized question will take a lots of space). So we remove that\n",
" # left whitespace\n",
" examples[\"question\"] = [q.lstrip() for q in examples[\"question\"]]\n",
"\n",
" # Tokenize our examples with truncation and padding, but keep the overflows using a stride. This results\n",
" # in one example possible giving several features when a context is long, each of those features having a\n",
" # context that overlaps a bit the context of the previous feature.\n",
" tokenized_examples = tokenizer(\n",
" examples[\"question\" if pad_on_right else \"context\"],\n",
" examples[\"context\" if pad_on_right else \"question\"],\n",
" truncation=\"only_second\" if pad_on_right else \"only_first\",\n",
" max_length=max_length,\n",
" stride=doc_stride,\n",
" return_overflowing_tokens=True,\n",
" return_offsets_mapping=True,\n",
" padding=\"max_length\",\n",
" )\n",
"\n",
" # Since one example might give us several features if it has a long context, we need a map from a feature to\n",
" # its corresponding example. This key gives us just that.\n",
" sample_mapping = tokenized_examples.pop(\"overflow_to_sample_mapping\")\n",
" # The offset mappings will give us a map from token to character position in the original context. This will\n",
" # help us compute the start_positions and end_positions.\n",
" offset_mapping = tokenized_examples.pop(\"offset_mapping\")\n",
"\n",
" # Let's label those examples!\n",
" tokenized_examples[\"start_positions\"] = []\n",
" tokenized_examples[\"end_positions\"] = []\n",
"\n",
" for i, offsets in enumerate(offset_mapping):\n",
" # We will label impossible answers with the index of the CLS token.\n",
" input_ids = tokenized_examples[\"input_ids\"][i]\n",
" cls_index = input_ids.index(tokenizer.cls_token_id)\n",
"\n",
" # Grab the sequence corresponding to that example (to know what is the context and what is the question).\n",
" sequence_ids = tokenized_examples.sequence_ids(i)\n",
"\n",
" # One example can give several spans, this is the index of the example containing this span of text.\n",
" sample_index = sample_mapping[i]\n",
" answers = examples[\"answers\"][sample_index]\n",
" # If no answers are given, set the cls_index as answer.\n",
" if len(answers[\"answer_start\"]) == 0:\n",
" tokenized_examples[\"start_positions\"].append(cls_index)\n",
" tokenized_examples[\"end_positions\"].append(cls_index)\n",
" else:\n",
" # Start/end character index of the answer in the text.\n",
" start_char = answers[\"answer_start\"][0]\n",
" end_char = start_char + len(answers[\"text\"][0])\n",
"\n",
" # Start token index of the current span in the text.\n",
" token_start_index = 0\n",
" while sequence_ids[token_start_index] != (1 if pad_on_right else 0):\n",
" token_start_index += 1\n",
"\n",
" # End token index of the current span in the text.\n",
" token_end_index = len(input_ids) - 1\n",
" while sequence_ids[token_end_index] != (1 if pad_on_right else 0):\n",
" token_end_index -= 1\n",
"\n",
" # Detect if the answer is out of the span (in which case this feature is labeled with the CLS index).\n",
" if not (offsets[token_start_index][0] <= start_char and offsets[token_end_index][1] >= end_char):\n",
" tokenized_examples[\"start_positions\"].append(cls_index)\n",
" tokenized_examples[\"end_positions\"].append(cls_index)\n",
" else:\n",
" # Otherwise move the token_start_index and token_end_index to the two ends of the answer.\n",
" # Note: we could go after the last offset if the answer is the last word (edge case).\n",
" while token_start_index < len(offsets) and offsets[token_start_index][0] <= start_char:\n",
" token_start_index += 1\n",
" tokenized_examples[\"start_positions\"].append(token_start_index - 1)\n",
" while offsets[token_end_index][1] >= end_char:\n",
" token_end_index -= 1\n",
" tokenized_examples[\"end_positions\"].append(token_end_index + 1)\n",
"\n",
" return tokenized_examples"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0lm8ozrJIrJR"
},
"source": [
"This function works with one or several examples. In the case of several examples, the tokenizer will return a list of lists for each key:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "-b70jh26IrJS",
"outputId": "acd3a42d-985b-44ee-9daa-af5d944ce1d9"
},
"outputs": [],
"source": [
"features = prepare_train_features(datasets['train'][:5])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zS-6iXTkIrJT"
},
"source": [
"To apply this function on all the sentences (or pairs of sentences) in our dataset, we just use the `map` method of our `dataset` object we created earlier. This will apply the function on all the elements of all the splits in `dataset`, so our training, validation and testing data will be preprocessed in one single command. Since our preprocessing changes the number of samples, we need to remove the old columns when applying it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "DDtsaJeVIrJT",
"outputId": "aa4734bf-4ef5-4437-9948-2c16363da719"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Loading cached processed dataset at /home/sgugger/.cache/huggingface/datasets/squad/plain_text/1.0.0/4c81550d83a2ac7c7ce23783bd8ff36642800e6633c1f18417fb58c3ff50cdd7/cache-a5c71e98733887b0.arrow\n",
"Loading cached processed dataset at /home/sgugger/.cache/huggingface/datasets/squad/plain_text/1.0.0/4c81550d83a2ac7c7ce23783bd8ff36642800e6633c1f18417fb58c3ff50cdd7/cache-14932a8c6aecc96d.arrow\n"
]
}
],
"source": [
"tokenized_datasets = datasets.map(prepare_train_features, batched=True, remove_columns=datasets[\"train\"].column_names)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "voWiw8C7IrJV"
},
"source": [
"Even better, the results are automatically cached by the π€ Datasets library to avoid spending time on this step the next time you run your notebook. The π€ Datasets library is normally smart enough to detect when the function you pass to map has changed (and thus requires to not use the cache data). For instance, it will properly detect if you change the task in the first cell and rerun the notebook. π€ Datasets warns you when it uses cached files, you can pass `load_from_cache_file=False` in the call to `map` to not use the cached files and force the preprocessing to be applied again.\n",
"\n",
"Note that we passed `batched=True` to encode the texts by batches together. This is to leverage the full benefit of the fast tokenizer we loaded earlier, which will use multi-threading to treat the texts in a batch concurrently."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "545PP3o8IrJV"
},
"source": [
"## Fine-tuning the model"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FBiW8UpKIrJW"
},
"source": [
"Now that our data is ready for training, we can download the pretrained model and fine-tune it. Since our task is question answering, we use the `AutoModelForQuestionAnswering` class. Like with the tokenizer, the `from_pretrained` method will download and cache the model for us:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "TlqNaB8jIrJW",
"outputId": "84916cf3-6e6c-47f3-d081-032ec30a4132"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForQuestionAnswering: ['vocab_transform.weight', 'vocab_transform.bias', 'vocab_layer_norm.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight', 'vocab_projector.bias']\n",
"- This IS expected if you are initializing DistilBertForQuestionAnswering from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
"- This IS NOT expected if you are initializing DistilBertForQuestionAnswering from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
"Some weights of DistilBertForQuestionAnswering were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['qa_outputs.weight', 'qa_outputs.bias']\n",
"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
]
}
],
"source": [
"from transformers import AutoModelForQuestionAnswering, TrainingArguments, Trainer\n",
"\n",
"model = AutoModelForQuestionAnswering.from_pretrained(model_checkpoint)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CczA5lJlIrJX"
},
"source": [
"The warning is telling us we are throwing away some weights (the `vocab_transform` and `vocab_layer_norm` layers) and randomly initializing some other (the `pre_classifier` and `classifier` layers). This is absolutely normal in this case, because we are removing the head used to pretrain the model on a masked language modeling objective and replacing it with a new head for which we don't have pretrained weights, so the library warns us we should fine-tune this model before using it for inference, which is exactly what we are going to do."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_N8urzhyIrJY"
},
"source": [
"To instantiate a `Trainer`, we will need to define three more things. The most important is the [`TrainingArguments`](https://huggingface.co/transformers/main_classes/trainer.html#transformers.TrainingArguments), which is a class that contains all the attributes to customize the training. It requires one folder name, which will be used to save the checkpoints of the model, and all other arguments are optional:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Bliy8zgjIrJY"
},
"outputs": [],
"source": [
"model_name = model_checkpoint.split(\"/\")[-1]\n",
"args = TrainingArguments(\n",
" f\"{model_name}-finetuned-squad\",\n",
" evaluation_strategy = \"epoch\",\n",
" learning_rate=2e-5,\n",
" per_device_train_batch_size=batch_size,\n",
" per_device_eval_batch_size=batch_size,\n",
" num_train_epochs=3,\n",
" weight_decay=0.01,\n",
" push_to_hub=True,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "km3pGVdTIrJc"
},
"source": [
"Here we set the evaluation to be done at the end of each epoch, tweak the learning rate, use the `batch_size` defined at the top of the notebook and customize the number of epochs for training, as well as the weight decay.\n",
"\n",
"The last argument to setup everything so we can push the model to the [Hub](https://huggingface.co/models) regularly during training. Remove it if you didn't follow the installation steps at the top of the notebook. If you want to save your model locally in a name that is different than the name of the repository it will be pushed, or if you want to push your model under an organization and not your name space, use the `hub_model_id` argument to set the repo name (it needs to be the full name, including your namespace: for instance `\"sgugger/bert-finetuned-squad\"` or `\"huggingface/bert-finetuned-squad\"`)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then we will need a data collator that will batch our processed examples together, here the default one will work:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from transformers import default_data_collator\n",
"\n",
"data_collator = default_data_collator"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rXuFTAzDIrJe"
},
"source": [
"We will evaluate our model and compute metrics in the next section (this is a very long operation, so we will only compute the evaluation loss during training).\n",
"\n",
"Then we just need to pass all of this along with our datasets to the `Trainer`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "imY1oC3SIrJf"
},
"outputs": [],
"source": [
"trainer = Trainer(\n",
" model,\n",
" args,\n",
" train_dataset=tokenized_datasets[\"train\"],\n",
" eval_dataset=tokenized_datasets[\"validation\"],\n",
" data_collator=data_collator,\n",
" tokenizer=tokenizer,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CdzABDVcIrJg"
},
"source": [
"We can now finetune our model by just calling the `train` method:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "uNx5pyRlIrJh",
"outputId": "077e661e-d36c-469b-89b8-7ff7f73541ec"
},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" \n",
"
\n",
" [16599/16599 58:06, Epoch 3/3]\n",
"
\n",
" \n",
" \n",
" | Epoch | \n",
" Training Loss | \n",
" Validation Loss | \n",
" Runtime | \n",
" Samples Per Second | \n",
"
\n",
" \n",
" \n",
" \n",
" | 1 | \n",
" 1.220600 | \n",
" 1.160322 | \n",
" 39.574900 | \n",
" 272.496000 | \n",
"
\n",
" \n",
" | 2 | \n",
" 0.945200 | \n",
" 1.121690 | \n",
" 39.706000 | \n",
" 271.596000 | \n",
"
\n",
" \n",
" | 3 | \n",
" 0.773000 | \n",
" 1.157358 | \n",
" 39.734000 | \n",
" 271.405000 | \n",
"
\n",
" \n",
"
"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"TrainOutput(global_step=16599, training_loss=1.1112074395519933, metrics={'train_runtime': 3487.0114, 'train_samples_per_second': 4.76, 'total_flos': 40606919924189184, 'epoch': 3.0})"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"trainer.train()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Since this training is particularly long, let's save the model just in case we need to restart."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"trainer.save_model(\"test-squad-trained\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Evaluation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Evaluating our model will require a bit more work, as we will need to map the predictions of our model back to parts of the context. The model itself predicts logits for the start and en position of our answers: if we take a batch from our validation datalaoder, here is the output our model gives us:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"odict_keys(['loss', 'start_logits', 'end_logits'])"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import torch\n",
"\n",
"for batch in trainer.get_eval_dataloader():\n",
" break\n",
"batch = {k: v.to(trainer.args.device) for k, v in batch.items()}\n",
"with torch.no_grad():\n",
" output = trainer.model(**batch)\n",
"output.keys()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The output of the model is a dict-like object that contains the loss (since we provided labels), the start and end logits. We won't need the loss for our predictions, let's have a look a the logits:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(torch.Size([16, 384]), torch.Size([16, 384]))"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"output.start_logits.shape, output.end_logits.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We have one logit for each feature and each token. The most obvious thing to predict an answer for each featyre is to take the index for the maximum of the start logits as a start position and the index of the maximum of the end logits as an end position."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(tensor([ 46, 57, 78, 43, 118, 15, 72, 35, 15, 34, 73, 41, 80, 91,\n",
" 156, 35], device='cuda:0'),\n",
" tensor([ 47, 58, 81, 55, 118, 110, 75, 37, 110, 36, 76, 53, 83, 94,\n",
" 158, 35], device='cuda:0'))"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"output.start_logits.argmax(dim=-1), output.end_logits.argmax(dim=-1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This will work great in a lot of cases, but what if this prediction gives us something impossible: the start position could be greater than the end position, or point to a span of text in the question instead of the answer. In that case, we might want to look at the second best prediction to see if it gives a possible answer and select that instead.\n",
"\n",
"However, picking the second best answer is not as easy as picking the best one: is it the second best index in the start logits with the best index in the end logits? Or the best index in the start logits with the second best index in the end logits? And if that second best answer is not possible either, it gets even trickier for the third best answer.\n",
"\n",
"\n",
"To classify our answers, we will use the score obtained by adding the start and end logits. We won't try to order all the possible answers and limit ourselves to with a hyper-parameter we call `n_best_size`. We'll pick the best indices in the start and end logits and gather all the answers this predicts. After checking if each one is valid, we will sort them by their score and keep the best one. Here is how we would do this on the first feature in the batch:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"n_best_size = 20"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"start_logits = output.start_logits[0].cpu().numpy()\n",
"end_logits = output.end_logits[0].cpu().numpy()\n",
"# Gather the indices the best start/end logits:\n",
"start_indexes = np.argsort(start_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
"end_indexes = np.argsort(end_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
"valid_answers = []\n",
"for start_index in start_indexes:\n",
" for end_index in end_indexes:\n",
" if start_index <= end_index: # We need to refine that test to check the answer is inside the context\n",
" valid_answers.append(\n",
" {\n",
" \"score\": start_logits[start_index] + end_logits[end_index],\n",
" \"text\": \"\" # We need to find a way to get back the original substring corresponding to the answer in the context\n",
" }\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And then we can sort the `valid_answers` according to their `score` and only keep the best one. The only point left is how to check a given span is inside the context (and not the question) and how to get back the text inside. To do this, we need to add two things to our validation features:\n",
"- the ID of the example that generated the feature (since each example can generate several features, as seen before);\n",
"- the offset mapping that will give us a map from token indices to character positions in the context.\n",
"\n",
"That's why we will re-process the validation set with the following function, slightly different from `prepare_train_features`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def prepare_validation_features(examples):\n",
" # Some of the questions have lots of whitespace on the left, which is not useful and will make the\n",
" # truncation of the context fail (the tokenized question will take a lots of space). So we remove that\n",
" # left whitespace\n",
" examples[\"question\"] = [q.lstrip() for q in examples[\"question\"]]\n",
"\n",
" # Tokenize our examples with truncation and maybe padding, but keep the overflows using a stride. This results\n",
" # in one example possible giving several features when a context is long, each of those features having a\n",
" # context that overlaps a bit the context of the previous feature.\n",
" tokenized_examples = tokenizer(\n",
" examples[\"question\" if pad_on_right else \"context\"],\n",
" examples[\"context\" if pad_on_right else \"question\"],\n",
" truncation=\"only_second\" if pad_on_right else \"only_first\",\n",
" max_length=max_length,\n",
" stride=doc_stride,\n",
" return_overflowing_tokens=True,\n",
" return_offsets_mapping=True,\n",
" padding=\"max_length\",\n",
" )\n",
"\n",
" # Since one example might give us several features if it has a long context, we need a map from a feature to\n",
" # its corresponding example. This key gives us just that.\n",
" sample_mapping = tokenized_examples.pop(\"overflow_to_sample_mapping\")\n",
"\n",
" # We keep the example_id that gave us this feature and we will store the offset mappings.\n",
" tokenized_examples[\"example_id\"] = []\n",
"\n",
" for i in range(len(tokenized_examples[\"input_ids\"])):\n",
" # Grab the sequence corresponding to that example (to know what is the context and what is the question).\n",
" sequence_ids = tokenized_examples.sequence_ids(i)\n",
" context_index = 1 if pad_on_right else 0\n",
"\n",
" # One example can give several spans, this is the index of the example containing this span of text.\n",
" sample_index = sample_mapping[i]\n",
" tokenized_examples[\"example_id\"].append(examples[\"id\"][sample_index])\n",
"\n",
" # Set to None the offset_mapping that are not part of the context so it's easy to determine if a token\n",
" # position is part of the context or not.\n",
" tokenized_examples[\"offset_mapping\"][i] = [\n",
" (o if sequence_ids[k] == context_index else None)\n",
" for k, o in enumerate(tokenized_examples[\"offset_mapping\"][i])\n",
" ]\n",
"\n",
" return tokenized_examples"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And like before, we can apply that function to our validation set easily:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "32ba04d6240149f49eb48c8d8b7f9aae",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"HBox(children=(FloatProgress(value=0.0, max=11.0), HTML(value='')))"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"source": [
"validation_features = datasets[\"validation\"].map(\n",
" prepare_validation_features,\n",
" batched=True,\n",
" remove_columns=datasets[\"validation\"].column_names\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we can grab the predictions for all features by using the `Trainer.predict` method:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" \n",
"
\n",
" [674/674 00:39]\n",
"