{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Transformers installation\n",
"! pip install transformers datasets\n",
"# To install from source instead of the last release, comment the command above and uncomment the following one.\n",
"# ! pip install git+https://github.com/huggingface/transformers.git"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Fine-tuning a pretrained model"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this tutorial, we will show you how to fine-tune a pretrained model from the Transformers library. In TensorFlow,\n",
"models can be directly trained using Keras and the `fit` method. In PyTorch, there is no generic training loop so\n",
"the 🤗 Transformers library provides an API with the class [Trainer](https://huggingface.co/docs/transformers/master/en/main_classes/trainer#transformers.Trainer) to let you fine-tune or train\n",
"a model from scratch easily. Then we will show you how to alternatively write the whole training loop in PyTorch.\n",
"\n",
"Before we can fine-tune a model, we need a dataset. In this tutorial, we will show you how to fine-tune BERT on the\n",
"[IMDB dataset](https://www.imdb.com/interfaces/): the task is to classify whether movie reviews are positive or\n",
"negative. For examples of other tasks, refer to the [additional-resources](#additional-resources) section!\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Preparing the datasets"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"hide_input": true
},
"outputs": [
{
"data": {
"text/html": [
""
],
"text/plain": [
""
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#@title\n",
"from IPython.display import HTML\n",
"\n",
"HTML('')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We will use the [🤗 Datasets](https://github.com/huggingface/datasets/) library to download and preprocess the IMDB\n",
"datasets. We will go over this part pretty quickly. Since the focus of this tutorial is on training, you should refer\n",
"to the 🤗 Datasets [documentation](https://huggingface.co/docs/datasets/) or the [preprocessing](https://huggingface.co/docs/transformers/master/en/preprocessing) tutorial for\n",
"more information.\n",
"\n",
"First, we can use the `load_dataset` function to download and cache the dataset:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from datasets import load_dataset\n",
"\n",
"raw_datasets = load_dataset(\"imdb\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This works like the `from_pretrained` method we saw for the models and tokenizers (except the cache directory is\n",
"_~/.cache/huggingface/dataset_ by default).\n",
"\n",
"The `raw_datasets` object is a dictionary with three keys: `\"train\"`, `\"test\"` and `\"unsupervised\"`\n",
"(which correspond to the three splits of that dataset). We will use the `\"train\"` split for training and the\n",
"`\"test\"` split for validation.\n",
"\n",
"To preprocess our data, we will need a tokenizer:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from transformers import AutoTokenizer\n",
"\n",
"tokenizer = AutoTokenizer.from_pretrained(\"bert-base-cased\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As we saw in [preprocessing](https://huggingface.co/docs/transformers/master/en/preprocessing), we can prepare the text inputs for the model with the following command (this is an\n",
"example, not a command you can execute):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"inputs = tokenizer(sentences, padding=\"max_length\", truncation=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This will make all the samples have the maximum length the model can accept (here 512), either by padding or truncating\n",
"them.\n",
"\n",
"However, we can instead apply these preprocessing steps to all the splits of our dataset at once by using the\n",
"`map` method:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def tokenize_function(examples):\n",
" return tokenizer(examples[\"text\"], padding=\"max_length\", truncation=True)\n",
"\n",
"\n",
"tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can learn more about the map method or the other ways to preprocess the data in the 🤗 Datasets [documentation](https://huggingface.co/docs/datasets/).\n",
"\n",
"Next we will generate a small subset of the training and validation set, to enable faster training:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"small_train_dataset = tokenized_datasets[\"train\"].shuffle(seed=42).select(range(1000))\n",
"small_eval_dataset = tokenized_datasets[\"test\"].shuffle(seed=42).select(range(1000))\n",
"full_train_dataset = tokenized_datasets[\"train\"]\n",
"full_eval_dataset = tokenized_datasets[\"test\"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In all the examples below, we will always use `small_train_dataset` and `small_eval_dataset`. Just replace\n",
"them by their _full_ equivalent to train or evaluate on the full dataset.\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Fine-tuning in PyTorch with the Trainer API"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"hide_input": true
},
"outputs": [
{
"data": {
"text/html": [
""
],
"text/plain": [
""
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#@title\n",
"from IPython.display import HTML\n",
"\n",
"HTML('')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Since PyTorch does not provide a training loop, the 🤗 Transformers library provides a [Trainer](https://huggingface.co/docs/transformers/master/en/main_classes/trainer#transformers.Trainer)\n",
"API that is optimized for 🤗 Transformers models, with a wide range of training options and with built-in features like\n",
"logging, gradient accumulation, and mixed precision.\n",
"\n",
"First, let's define our model:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from transformers import AutoModelForSequenceClassification\n",
"\n",
"model = AutoModelForSequenceClassification.from_pretrained(\"bert-base-cased\", num_labels=2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This will issue a warning about some of the pretrained weights not being used and some weights being randomly\n",
"initialized. That's because we are throwing away the pretraining head of the BERT model to replace it with a\n",
"classification head which is randomly initialized. We will fine-tune this model on our task, transferring the knowledge\n",
"of the pretrained model to it (which is why doing this is called transfer learning).\n",
"\n",
"Then, to define our [Trainer](https://huggingface.co/docs/transformers/master/en/main_classes/trainer#transformers.Trainer), we will need to instantiate a\n",
"[TrainingArguments](https://huggingface.co/docs/transformers/master/en/main_classes/trainer#transformers.TrainingArguments). This class contains all the hyperparameters we can tune for the\n",
"[Trainer](https://huggingface.co/docs/transformers/master/en/main_classes/trainer#transformers.Trainer) or the flags to activate the different training options it supports. Let's begin by\n",
"using all the defaults, the only thing we then have to provide is a directory in which the checkpoints will be saved:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from transformers import TrainingArguments\n",
"\n",
"training_args = TrainingArguments(\"test_trainer\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then we can instantiate a [Trainer](https://huggingface.co/docs/transformers/master/en/main_classes/trainer#transformers.Trainer) like this:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from transformers import Trainer\n",
"\n",
"trainer = Trainer(model=model, args=training_args, train_dataset=small_train_dataset, eval_dataset=small_eval_dataset)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To fine-tune our model, we just need to call"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"trainer.train()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"which will start a training that you can follow with a progress bar, which should take a couple of minutes to complete\n",
"(as long as you have access to a GPU). It won't actually tell you anything useful about how well (or badly) your model\n",
"is performing however as by default, there is no evaluation during training, and we didn't tell the\n",
"[Trainer](https://huggingface.co/docs/transformers/master/en/main_classes/trainer#transformers.Trainer) to compute any metrics. Let's have a look on how to do that now!\n",
"\n",
"To have the [Trainer](https://huggingface.co/docs/transformers/master/en/main_classes/trainer#transformers.Trainer) compute and report metrics, we need to give it a `compute_metrics`\n",
"function that takes predictions and labels (grouped in a namedtuple called [EvalPrediction](https://huggingface.co/docs/transformers/master/en/internal/trainer_utils#transformers.EvalPrediction)) and\n",
"return a dictionary with string items (the metric names) and float values (the metric values).\n",
"\n",
"The 🤗 Datasets library provides an easy way to get the common metrics used in NLP with the `load_metric` function.\n",
"here we simply use accuracy. Then we define the `compute_metrics` function that just convert logits to predictions\n",
"(remember that all 🤗 Transformers models return the logits) and feed them to `compute` method of this metric."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"from datasets import load_metric\n",
"\n",
"metric = load_metric(\"accuracy\")\n",
"\n",
"\n",
"def compute_metrics(eval_pred):\n",
" logits, labels = eval_pred\n",
" predictions = np.argmax(logits, axis=-1)\n",
" return metric.compute(predictions=predictions, references=labels)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The compute function needs to receive a tuple (with logits and labels) and has to return a dictionary with string keys\n",
"(the name of the metric) and float values. It will be called at the end of each evaluation phase on the whole arrays of\n",
"predictions/labels.\n",
"\n",
"To check if this works on practice, let's create a new [Trainer](https://huggingface.co/docs/transformers/master/en/main_classes/trainer#transformers.Trainer) with our fine-tuned model:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"trainer = Trainer(\n",
" model=model,\n",
" args=training_args,\n",
" train_dataset=small_train_dataset,\n",
" eval_dataset=small_eval_dataset,\n",
" compute_metrics=compute_metrics,\n",
")\n",
"trainer.evaluate()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"which showed an accuracy of 87.5% in our case.\n",
"\n",
"If you want to fine-tune your model and regularly report the evaluation metrics (for instance at the end of each\n",
"epoch), here is how you should define your training arguments:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from transformers import TrainingArguments\n",
"\n",
"training_args = TrainingArguments(\"test_trainer\", evaluation_strategy=\"epoch\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"See the documentation of [TrainingArguments](https://huggingface.co/docs/transformers/master/en/main_classes/trainer#transformers.TrainingArguments) for more options.\n",
"\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Fine-tuning with Keras"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"hide_input": true
},
"outputs": [
{
"data": {
"text/html": [
""
],
"text/plain": [
""
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#@title\n",
"from IPython.display import HTML\n",
"\n",
"HTML('')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Models can also be trained natively in TensorFlow using the Keras API. First, let's define our model:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"from transformers import TFAutoModelForSequenceClassification\n",
"\n",
"model = TFAutoModelForSequenceClassification.from_pretrained(\"bert-base-cased\", num_labels=2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then we will need to convert our datasets from before in standard `tf.data.Dataset`. Since we have fixed shapes,\n",
"it can easily be done like this. First we remove the _\"text\"_ column from our datasets and set them in TensorFlow\n",
"format:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"tf_train_dataset = small_train_dataset.remove_columns([\"text\"]).with_format(\"tensorflow\")\n",
"tf_eval_dataset = small_eval_dataset.remove_columns([\"text\"]).with_format(\"tensorflow\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then we convert everything in big tensors and use the `tf.data.Dataset.from_tensor_slices` method:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"train_features = {x: tf_train_dataset[x] for x in tokenizer.model_input_names}\n",
"train_tf_dataset = tf.data.Dataset.from_tensor_slices((train_features, tf_train_dataset[\"label\"]))\n",
"train_tf_dataset = train_tf_dataset.shuffle(len(tf_train_dataset)).batch(8)\n",
"\n",
"eval_features = {x: tf_eval_dataset[x] for x in tokenizer.model_input_names}\n",
"eval_tf_dataset = tf.data.Dataset.from_tensor_slices((eval_features, tf_eval_dataset[\"label\"]))\n",
"eval_tf_dataset = eval_tf_dataset.batch(8)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"With this done, the model can then be compiled and trained as any Keras model:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.compile(\n",
" optimizer=tf.keras.optimizers.Adam(learning_rate=5e-5),\n",
" loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
" metrics=tf.metrics.SparseCategoricalAccuracy(),\n",
")\n",
"\n",
"model.fit(train_tf_dataset, validation_data=eval_tf_dataset, epochs=3)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"With the tight interoperability between TensorFlow and PyTorch models, you can even save the model and then reload it\n",
"as a PyTorch model (or vice-versa):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from transformers import AutoModelForSequenceClassification\n",
"\n",
"model.save_pretrained(\"my_imdb_model\")\n",
"pytorch_model = AutoModelForSequenceClassification.from_pretrained(\"my_imdb_model\", from_tf=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Fine-tuning in native PyTorch"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"hide_input": true
},
"outputs": [
{
"data": {
"text/html": [
""
],
"text/plain": [
""
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#@title\n",
"from IPython.display import HTML\n",
"\n",
"HTML('')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You might need to restart your notebook at this stage to free some memory, or execute the following code:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"del model\n",
"del pytorch_model\n",
"del trainer\n",
"torch.cuda.empty_cache()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's now see how to achieve the same results as in [trainer section](#trainer) in PyTorch. First we need to\n",
"define the dataloaders, which we will use to iterate over batches. We just need to apply a bit of post-processing to\n",
"our `tokenized_datasets` before doing that to:\n",
"\n",
"- remove the columns corresponding to values the model does not expect (here the `\"text\"` column)\n",
"- rename the column `\"label\"` to `\"labels\"` (because the model expect the argument to be named `labels`)\n",
"- set the format of the datasets so they return PyTorch Tensors instead of lists.\n",
"\n",
"Our _tokenized_datasets_ has one method for each of those steps:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"tokenized_datasets = tokenized_datasets.remove_columns([\"text\"])\n",
"tokenized_datasets = tokenized_datasets.rename_column(\"label\", \"labels\")\n",
"tokenized_datasets.set_format(\"torch\")\n",
"\n",
"small_train_dataset = tokenized_datasets[\"train\"].shuffle(seed=42).select(range(1000))\n",
"small_eval_dataset = tokenized_datasets[\"test\"].shuffle(seed=42).select(range(1000))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that this is done, we can easily define our dataloaders:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from torch.utils.data import DataLoader\n",
"\n",
"train_dataloader = DataLoader(small_train_dataset, shuffle=True, batch_size=8)\n",
"eval_dataloader = DataLoader(small_eval_dataset, batch_size=8)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, we define our model:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from transformers import AutoModelForSequenceClassification\n",
"\n",
"model = AutoModelForSequenceClassification.from_pretrained(\"bert-base-cased\", num_labels=2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We are almost ready to write our training loop, the only two things are missing are an optimizer and a learning rate\n",
"scheduler. The default optimizer used by the [Trainer](https://huggingface.co/docs/transformers/master/en/main_classes/trainer#transformers.Trainer) is [AdamW](https://huggingface.co/docs/transformers/master/en/main_classes/optimizer_schedules#transformers.AdamW):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from transformers import AdamW\n",
"\n",
"optimizer = AdamW(model.parameters(), lr=5e-5)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, the learning rate scheduler used by default is just a linear decay from the maximum value (5e-5 here) to 0:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from transformers import get_scheduler\n",
"\n",
"num_epochs = 3\n",
"num_training_steps = num_epochs * len(train_dataloader)\n",
"lr_scheduler = get_scheduler(\"linear\", optimizer=optimizer, num_warmup_steps=0, num_training_steps=num_training_steps)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"One last thing, we will want to use the GPU if we have access to one (otherwise training might take several hours\n",
"instead of a couple of minutes). To do this, we define a `device` we will put our model and our batches on."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import torch\n",
"\n",
"device = torch.device(\"cuda\") if torch.cuda.is_available() else torch.device(\"cpu\")\n",
"model.to(device)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We now are ready to train! To get some sense of when it will be finished, we add a progress bar over our number of\n",
"training steps, using the _tqdm_ library."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tqdm.auto import tqdm\n",
"\n",
"progress_bar = tqdm(range(num_training_steps))\n",
"\n",
"model.train()\n",
"for epoch in range(num_epochs):\n",
" for batch in train_dataloader:\n",
" batch = {k: v.to(device) for k, v in batch.items()}\n",
" outputs = model(**batch)\n",
" loss = outputs.loss\n",
" loss.backward()\n",
"\n",
" optimizer.step()\n",
" lr_scheduler.step()\n",
" optimizer.zero_grad()\n",
" progress_bar.update(1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that if you are used to freezing the body of your pretrained model (like in computer vision) the above may seem a\n",
"bit strange, as we are directly fine-tuning the whole model without taking any precaution. It actually works better\n",
"this way for Transformers model (so this is not an oversight on our side). If you're not familiar with what \"freezing\n",
"the body\" of the model means, forget you read this paragraph.\n",
"\n",
"Now to check the results, we need to write the evaluation loop. Like in the [trainer section](#trainer) we will\n",
"use a metric from the datasets library. Here we accumulate the predictions at each batch before computing the final\n",
"result when the loop is finished."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"metric = load_metric(\"accuracy\")\n",
"model.eval()\n",
"for batch in eval_dataloader:\n",
" batch = {k: v.to(device) for k, v in batch.items()}\n",
" with torch.no_grad():\n",
" outputs = model(**batch)\n",
"\n",
" logits = outputs.logits\n",
" predictions = torch.argmax(logits, dim=-1)\n",
" metric.add_batch(predictions=predictions, references=batch[\"labels\"])\n",
"\n",
"metric.compute()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Additional resources"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To look at more fine-tuning examples you can refer to:\n",
"\n",
"- [🤗 Transformers Examples](https://github.com/huggingface/transformers/tree/master/examples) which includes scripts\n",
" to train on all common NLP tasks in PyTorch and TensorFlow.\n",
"\n",
"- [🤗 Transformers Notebooks](https://huggingface.co/docs/transformers/master/en/notebooks) which contains various notebooks and in particular one per task (look for\n",
" the _how to finetune a model on xxx_)."
]
}
],
"metadata": {},
"nbformat": 4,
"nbformat_minor": 4
}