"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_random_elements(raw_datasets[\"train\"])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lnjDIuQ3IrI-"
},
"source": [
"The metric is an instance of [`datasets.Metric`](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Metric):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "5o4rUteaIrI_",
"outputId": "18038ef5-554c-45c5-e00a-133b02ec10f1"
},
"outputs": [
{
"data": {
"text/plain": [
"Metric(name: \"sacrebleu\", features: {'predictions': Value(dtype='string', id='sequence'), 'references': Sequence(feature=Value(dtype='string', id='sequence'), length=-1, id='references')}, usage: \"\"\"\n",
"Produces BLEU scores along with its sufficient statistics\n",
"from a source against one or more references.\n",
"\n",
"Args:\n",
" predictions: The system stream (a sequence of segments)\n",
" references: A list of one or more reference streams (each a sequence of segments)\n",
" smooth: The smoothing method to use\n",
" smooth_value: For 'floor' smoothing, the floor to use\n",
" force: Ignore data that looks already tokenized\n",
" lowercase: Lowercase the data\n",
" tokenize: The tokenizer to use\n",
"Returns:\n",
" 'score': BLEU score,\n",
" 'counts': Counts,\n",
" 'totals': Totals,\n",
" 'precisions': Precisions,\n",
" 'bp': Brevity penalty,\n",
" 'sys_len': predictions length,\n",
" 'ref_len': reference length,\n",
"Examples:\n",
"\n",
" >>> predictions = [\"hello there general kenobi\", \"foo bar foobar\"]\n",
" >>> references = [[\"hello there general kenobi\", \"hello there !\"], [\"foo bar foobar\", \"foo bar foobar\"]]\n",
" >>> sacrebleu = datasets.load_metric(\"sacrebleu\")\n",
" >>> results = sacrebleu.compute(predictions=predictions, references=references)\n",
" >>> print(list(results.keys()))\n",
" ['score', 'counts', 'totals', 'precisions', 'bp', 'sys_len', 'ref_len']\n",
" >>> print(round(results[\"score\"], 1))\n",
" 100.0\n",
"\"\"\", stored examples: 0)"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"metric"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jAWdqcUBIrJC"
},
"source": [
"You can call its `compute` method with your predictions and labels, which need to be list of decoded strings (list of list for the labels):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6XN1Rq0aIrJC",
"outputId": "a4405435-a8a9-41ff-9f79-a13077b587c7"
},
"outputs": [
{
"data": {
"text/plain": [
"{'score': 0.0,\n",
" 'counts': [4, 2, 0, 0],\n",
" 'totals': [4, 2, 0, 0],\n",
" 'precisions': [100.0, 100.0, 0.0, 0.0],\n",
" 'bp': 1.0,\n",
" 'sys_len': 4,\n",
" 'ref_len': 4}"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fake_preds = [\"hello there\", \"general kenobi\"]\n",
"fake_labels = [[\"hello there\"], [\"general kenobi\"]]\n",
"metric.compute(predictions=fake_preds, references=fake_labels)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "n9qywopnIrJH"
},
"source": [
"## Preprocessing the data"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YVx71GdAIrJH"
},
"source": [
"Before we can feed those texts to our model, we need to preprocess them. This is done by a 🤗 Transformers `Tokenizer` which will (as the name indicates) tokenize the inputs (including converting the tokens to their corresponding IDs in the pretrained vocabulary) and put it in a format the model expects, as well as generate the other inputs that model requires.\n",
"\n",
"To do all of this, we instantiate our tokenizer with the `AutoTokenizer.from_pretrained` method, which will ensure:\n",
"\n",
"- we get a tokenizer that corresponds to the model architecture we want to use,\n",
"- we download the vocabulary used when pretraining this specific checkpoint.\n",
"\n",
"That vocabulary will be cached, so it's not downloaded again the next time we run the cell."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "eXNLu_-nIrJI"
},
"outputs": [],
"source": [
"from transformers import AutoTokenizer\n",
" \n",
"tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For the mBART tokenizer (like we have here), we need to set the source and target languages (so the texts are preprocessed properly). You can check the language codes [here](https://huggingface.co/facebook/mbart-large-cc25) if you are using this notebook on a different pairs of languages."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"if \"mbart\" in model_checkpoint:\n",
" tokenizer.src_lang = \"en-XX\"\n",
" tokenizer.tgt_lang = \"ro-RO\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Vl6IidfdIrJK"
},
"source": [
"By default, the call above will use one of the fast tokenizers (backed by Rust) from the 🤗 Tokenizers library."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rowT4iCLIrJK"
},
"source": [
"You can directly call this tokenizer on one sentence or a pair of sentences:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "a5hBlsrHIrJL",
"outputId": "acdaa98a-a8cd-4a20-89b8-cc26437bbe90"
},
"outputs": [
{
"data": {
"text/plain": [
"{'input_ids': [125, 778, 3, 63, 141, 9191, 23, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1]}"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tokenizer(\"Hello, this one sentence!\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qo_0B1M2IrJM"
},
"source": [
"Depending on the model you selected, you will see different keys in the dictionary returned by the cell above. They don't matter much for what we're doing here (just know they are required by the model we will instantiate later), you can learn more about them in [this tutorial](https://huggingface.co/transformers/preprocessing.html) if you're interested.\n",
"\n",
"Instead of one sentence, we can pass along a list of sentences:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'input_ids': [[125, 778, 3, 63, 141, 9191, 23, 0], [187, 32, 716, 9191, 2, 0]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1]]}"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tokenizer([\"Hello, this one sentence!\", \"This is another sentence.\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To prepare the targets for our model, we need to tokenize them inside the `as_target_tokenizer` context manager. This will make sure the tokenizer uses the special tokens corresponding to the targets:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'input_ids': [[10334, 1204, 3, 15, 8915, 27, 452, 59, 29579, 581, 23, 0], [235, 1705, 11, 32, 8, 1205, 5305, 59, 29579, 581, 2, 0]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]}\n"
]
}
],
"source": [
"with tokenizer.as_target_tokenizer():\n",
" print(tokenizer([\"Hello, this one sentence!\", \"This is another sentence.\"]))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2C0hcmp9IrJQ"
},
"source": [
"If you are using one of the five T5 checkpoints that require a special prefix to put before the inputs, you should adapt the following cell."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"if model_checkpoint in [\"t5-small\", \"t5-base\", \"t5-larg\", \"t5-3b\", \"t5-11b\"]:\n",
" prefix = \"translate English to Romanian: \"\n",
"else:\n",
" prefix = \"\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can then write the function that will preprocess our samples. We just feed them to the `tokenizer` with the argument `truncation=True`. This will ensure that an input longer that what the model selected can handle will be truncated to the maximum length accepted by the model. The padding will be dealt with later on (in a data collator) so we pad examples to the longest length in the batch and not the whole dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "vc0BSBLIIrJQ"
},
"outputs": [],
"source": [
"max_input_length = 128\n",
"max_target_length = 128\n",
"source_lang = \"en\"\n",
"target_lang = \"ro\"\n",
"\n",
"def preprocess_function(examples):\n",
" inputs = [prefix + ex[source_lang] for ex in examples[\"translation\"]]\n",
" targets = [ex[target_lang] for ex in examples[\"translation\"]]\n",
" model_inputs = tokenizer(inputs, max_length=max_input_length, truncation=True)\n",
"\n",
" # Setup the tokenizer for targets\n",
" with tokenizer.as_target_tokenizer():\n",
" labels = tokenizer(targets, max_length=max_target_length, truncation=True)\n",
"\n",
" model_inputs[\"labels\"] = labels[\"input_ids\"]\n",
" return model_inputs"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0lm8ozrJIrJR"
},
"source": [
"This function works with one or several examples. In the case of several examples, the tokenizer will return a list of lists for each key:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "-b70jh26IrJS",
"outputId": "acd3a42d-985b-44ee-9daa-af5d944ce1d9"
},
"outputs": [
{
"data": {
"text/plain": [
"{'input_ids': [[393, 4462, 14, 1137, 53, 216, 28636, 0], [24385, 14, 28636, 14, 4646, 4622, 53, 216, 28636, 0]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]], 'labels': [[42140, 494, 1750, 53, 8, 59, 903, 3543, 9, 15202, 0], [36199, 6612, 9, 15202, 122, 568, 35788, 21549, 53, 8, 59, 903, 3543, 9, 15202, 0]]}"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"preprocess_function(raw_datasets['train'][:2])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zS-6iXTkIrJT"
},
"source": [
"To apply this function on all the pairs of sentences in our dataset, we just use the `map` method of our `dataset` object we created earlier. This will apply the function on all the elements of all the splits in `dataset`, so our training, validation and testing data will be preprocessed in one single command."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "DDtsaJeVIrJT",
"outputId": "aa4734bf-4ef5-4437-9948-2c16363da719"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Loading cached processed dataset at /home/sgugger/.cache/huggingface/datasets/wmt16/ro-en/1.0.0/9dc00622c30446e99c4c63d12a484ea4fb653f2f37c867d6edcec839d7eae50f/cache-126b7925258542cf.arrow\n",
"Loading cached processed dataset at /home/sgugger/.cache/huggingface/datasets/wmt16/ro-en/1.0.0/9dc00622c30446e99c4c63d12a484ea4fb653f2f37c867d6edcec839d7eae50f/cache-995e36d1e1745505.arrow\n",
"Loading cached processed dataset at /home/sgugger/.cache/huggingface/datasets/wmt16/ro-en/1.0.0/9dc00622c30446e99c4c63d12a484ea4fb653f2f37c867d6edcec839d7eae50f/cache-313c89c543a3c175.arrow\n"
]
}
],
"source": [
"tokenized_datasets = raw_datasets.map(preprocess_function, batched=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "voWiw8C7IrJV"
},
"source": [
"Even better, the results are automatically cached by the 🤗 Datasets library to avoid spending time on this step the next time you run your notebook. The 🤗 Datasets library is normally smart enough to detect when the function you pass to map has changed (and thus requires to not use the cache data). For instance, it will properly detect if you change the task in the first cell and rerun the notebook. 🤗 Datasets warns you when it uses cached files, you can pass `load_from_cache_file=False` in the call to `map` to not use the cached files and force the preprocessing to be applied again.\n",
"\n",
"Note that we passed `batched=True` to encode the texts by batches together. This is to leverage the full benefit of the fast tokenizer we loaded earlier, which will use multi-threading to treat the texts in a batch concurrently."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "545PP3o8IrJV"
},
"source": [
"## Fine-tuning the model"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FBiW8UpKIrJW"
},
"source": [
"Now that our data is ready, we can download the pretrained model and fine-tune it. Since our task is of the sequence-to-sequence kind, we use the `AutoModelForSeq2SeqLM` class. Like with the tokenizer, the `from_pretrained` method will download and cache the model for us."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "TlqNaB8jIrJW",
"outputId": "84916cf3-6e6c-47f3-d081-032ec30a4132"
},
"outputs": [],
"source": [
"from transformers import AutoModelForSeq2SeqLM, DataCollatorForSeq2Seq, Seq2SeqTrainingArguments, Seq2SeqTrainer\n",
"\n",
"model = AutoModelForSeq2SeqLM.from_pretrained(model_checkpoint)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CczA5lJlIrJX"
},
"source": [
"Note that we don't get a warning like in our classification example. This means we used all the weights of the pretrained model and there is no randomly initialized head in this case."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_N8urzhyIrJY"
},
"source": [
"To instantiate a `Seq2SeqTrainer`, we will need to define three more things. The most important is the [`Seq2SeqTrainingArguments`](https://huggingface.co/transformers/main_classes/trainer.html#transformers.Seq2SeqTrainingArguments), which is a class that contains all the attributes to customize the training. It requires one folder name, which will be used to save the checkpoints of the model, and all other arguments are optional:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Bliy8zgjIrJY"
},
"outputs": [],
"source": [
"batch_size = 16\n",
"model_name = model_checkpoint.split(\"/\")[-1]\n",
"args = Seq2SeqTrainingArguments(\n",
" f\"{model_name}-finetuned-{source_lang}-to-{target_lang}\",\n",
" evaluation_strategy = \"epoch\",\n",
" learning_rate=2e-5,\n",
" per_device_train_batch_size=batch_size,\n",
" per_device_eval_batch_size=batch_size,\n",
" weight_decay=0.01,\n",
" save_total_limit=3,\n",
" num_train_epochs=1,\n",
" predict_with_generate=True,\n",
" fp16=True,\n",
" push_to_hub=True,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "km3pGVdTIrJc"
},
"source": [
"Here we set the evaluation to be done at the end of each epoch, tweak the learning rate, use the `batch_size` defined at the top of the cell and customize the weight decay. Since the `Seq2SeqTrainer` will save the model regularly and our dataset is quite large, we tell it to make three saves maximum. Lastly, we use the `predict_with_generate` option (to properly generate summaries) and activate mixed precision training (to go a bit faster).\n",
"\n",
"The last argument to setup everything so we can push the model to the [Hub](https://huggingface.co/models) regularly during training. Remove it if you didn't follow the installation steps at the top of the notebook. If you want to save your model locally in a name that is different than the name of the repository it will be pushed, or if you want to push your model under an organization and not your name space, use the `hub_model_id` argument to set the repo name (it needs to be the full name, including your namespace: for instance `\"sgugger/marian-finetuned-en-to-ro\"` or `\"huggingface/marian-finetuned-en-to-ro\"`).\n",
"\n",
"Then, we need a special kind of data collator, which will not only pad the inputs to the maximum length in the batch, but also the labels:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data_collator = DataCollatorForSeq2Seq(tokenizer, model=model)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7sZOdRlRIrJd"
},
"source": [
"The last thing to define for our `Seq2SeqTrainer` is how to compute the metrics from the predictions. We need to define a function for this, which will just use the `metric` we loaded earlier, and we have to do a bit of pre-processing to decode the predictions into texts:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "UmvbnJ9JIrJd"
},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"def postprocess_text(preds, labels):\n",
" preds = [pred.strip() for pred in preds]\n",
" labels = [[label.strip()] for label in labels]\n",
"\n",
" return preds, labels\n",
"\n",
"def compute_metrics(eval_preds):\n",
" preds, labels = eval_preds\n",
" if isinstance(preds, tuple):\n",
" preds = preds[0]\n",
" decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)\n",
"\n",
" # Replace -100 in the labels as we can't decode them.\n",
" labels = np.where(labels != -100, labels, tokenizer.pad_token_id)\n",
" decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)\n",
"\n",
" # Some simple post-processing\n",
" decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)\n",
"\n",
" result = metric.compute(predictions=decoded_preds, references=decoded_labels)\n",
" result = {\"bleu\": result[\"score\"]}\n",
"\n",
" prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds]\n",
" result[\"gen_len\"] = np.mean(prediction_lens)\n",
" result = {k: round(v, 4) for k, v in result.items()}\n",
" return result"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rXuFTAzDIrJe"
},
"source": [
"Then we just need to pass all of this along with our datasets to the `Seq2SeqTrainer`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "imY1oC3SIrJf"
},
"outputs": [],
"source": [
"trainer = Seq2SeqTrainer(\n",
" model,\n",
" args,\n",
" train_dataset=tokenized_datasets[\"train\"],\n",
" eval_dataset=tokenized_datasets[\"validation\"],\n",
" data_collator=data_collator,\n",
" tokenizer=tokenizer,\n",
" compute_metrics=compute_metrics\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CdzABDVcIrJg"
},
"source": [
"We can now finetune our model by just calling the `train` method:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "uNx5pyRlIrJh",
"outputId": "077e661e-d36c-469b-89b8-7ff7f73541ec",
"scrolled": false
},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" \n",
"
\n",
" [38145/38145 1:18:58, Epoch 1/1]\n",
"
\n",
" \n",
" \n",
" | Epoch | \n",
" Training Loss | \n",
" Validation Loss | \n",
" Bleu | \n",
" Gen Len | \n",
" Runtime | \n",
" Samples Per Second | \n",
"
\n",
" \n",
" \n",
" \n",
" | 1 | \n",
" 0.740100 | \n",
" 1.290665 | \n",
" 28.059300 | \n",
" 34.051500 | \n",
" 135.611700 | \n",
" 14.741000 | \n",
"
\n",
" \n",
"
"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"TrainOutput(global_step=38145, training_loss=0.7717826230230017, metrics={'train_runtime': 4738.3882, 'train_samples_per_second': 8.05, 'total_flos': 3.62019881263104e+16, 'epoch': 1.0, 'init_mem_cpu_alloc_delta': 337319, 'init_mem_gpu_alloc_delta': 300833792, 'init_mem_cpu_peaked_delta': 18306, 'init_mem_gpu_peaked_delta': 0, 'train_mem_cpu_alloc_delta': 1028051, 'train_mem_gpu_alloc_delta': 899235328, 'train_mem_cpu_peaked_delta': 267371462, 'train_mem_gpu_peaked_delta': 3136797184})"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"trainer.train()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can now upload the result of the training to the Hub, just execute this instruction:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"trainer.push_to_hub()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can now share this model with all your friends, family, favorite pets: they can all load it with the identifier `\"your-username/the-name-you-picked\"` so for instance:\n",
"\n",
"```python\n",
"from transformers import AutoModelForSeq2SeqLM\n",
"\n",
"model = AutoModelForSeq2SeqLM.from_pretrained(\"sgugger/my-awesome-model\")\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"colab": {
"name": "Translation",
"provenance": []
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 1
}