{ "cells": [ { "cell_type": "markdown", "id": "03b19adf-76fa-4938-af32-ae11453bf68e", "metadata": {}, "source": [ "# Fine-tunining DeBERTa model on a question answering task with ORTTrainer" ] }, { "cell_type": "markdown", "id": "d133d240-7a72-4f24-b0c1-55fd8fd7ee77", "metadata": { "tags": [] }, "source": [ "In this notebook, we will see how to fine-tune the [DeBERTa base](https://huggingface.co/microsoft/deberta-base/tree/main) model to a question answering task, which is the task of extracting the answer to a question from a given context. We will use the `ORTTrainer` API in Optimum library to leverage ONNX Runtime backend to accelerate the training." ] }, { "cell_type": "markdown", "id": "7e816d8e-85b8-4225-948a-0814ba19e6bd", "metadata": {}, "source": [ "## Start by setting up the environment\n", "\n", "To use ONNX Runtime for training, you need a machine with at least one NVIDIA or AMD GPU.\n", "\n", "ONNX Runtime training module need to be properly installed before launching the notebook! Please follow the instruction in [Optimum's documentation](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/trainer#start-by-setting-up-the-environment) to set up your environment, or use directly the [dockerfiles in Optimum](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/trainer#start-by-setting-up-the-environment) " ] }, { "cell_type": "code", "execution_count": 1, "id": "1786df19-07a0-4f9d-aacd-0d100aad0fae", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Thu Apr 6 08:49:04 2023 \n", "+-----------------------------------------------------------------------------+\n", "| NVIDIA-SMI 510.108.03 Driver Version: 510.108.03 CUDA Version: 11.6 |\n", "|-------------------------------+----------------------+----------------------+\n", "| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |\n", "| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |\n", "| | | MIG M. |\n", "|===============================+======================+======================|\n", "| 0 Tesla V100-SXM2... Off | 00000000:00:1D.0 Off | 0 |\n", "| N/A 38C P0 39W / 300W | 4MiB / 16384MiB | 0% Default |\n", "| | | N/A |\n", "+-------------------------------+----------------------+----------------------+\n", " \n", "+-----------------------------------------------------------------------------+\n", "| Processes: |\n", "| GPU GI CI PID Type Process name GPU Memory |\n", "| ID ID Usage |\n", "|=============================================================================|\n", "+-----------------------------------------------------------------------------+\n" ] } ], "source": [ "# Check your GPU\n", "!nvidia-smi" ] }, { "cell_type": "markdown", "id": "2be03174-3fc7-4b50-bb7f-28673a2758d2", "metadata": {}, "source": [ "If you're opening this Notebook on colab, you will probably need to install 🤗 Optimum, 🤗 Transformers, 🤗 Datasets and 🤗 evaluate. Uncomment the following cell and run it." ] }, { "cell_type": "code", "execution_count": 2, "id": "a758e26c-0847-4978-9cfe-6e27c7e9cce8", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Collecting optimum\n", " Downloading optimum-1.7.3-py3-none-any.whl (300 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m300.5/300.5 kB\u001b[0m \u001b[31m9.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hRequirement already satisfied: transformers in /usr/local/lib/python3.8/dist-packages (4.27.0.dev0)\n", "Requirement already satisfied: datasets in /usr/local/lib/python3.8/dist-packages (2.10.0)\n", "Requirement already satisfied: evaluate in /usr/local/lib/python3.8/dist-packages (0.4.0)\n", "Requirement already satisfied: huggingface_hub in /usr/local/lib/python3.8/dist-packages (0.12.1)\n", "Requirement already satisfied: torchvision in /usr/local/lib/python3.8/dist-packages (from optimum) (0.14.1+cu116)\n", "Requirement already satisfied: sympy in /usr/local/lib/python3.8/dist-packages (from optimum) (1.11.1)\n", "Requirement already satisfied: torch>=1.9 in /usr/local/lib/python3.8/dist-packages (from optimum) (1.13.1+cu116)\n", "Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from optimum) (23.0)\n", "Requirement already satisfied: coloredlogs in /usr/local/lib/python3.8/dist-packages (from optimum) (15.0.1)\n", "Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from optimum) (1.24.2)\n", "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.8/dist-packages (from transformers) (4.64.1)\n", "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.8/dist-packages (from transformers) (6.0)\n", "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.8/dist-packages (from transformers) (2022.10.31)\n", "Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from transformers) (3.9.0)\n", "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.8/dist-packages (from transformers) (0.13.2)\n", "Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from transformers) (2.28.2)\n", "Requirement already satisfied: multiprocess in /usr/local/lib/python3.8/dist-packages (from datasets) (0.70.14)\n", "Requirement already satisfied: responses<0.19 in /usr/local/lib/python3.8/dist-packages (from datasets) (0.18.0)\n", "Requirement already satisfied: dill<0.3.7,>=0.3.0 in /usr/local/lib/python3.8/dist-packages (from datasets) (0.3.6)\n", "Requirement already satisfied: pandas in /usr/local/lib/python3.8/dist-packages (from datasets) (1.5.3)\n", "Requirement already satisfied: aiohttp in /usr/local/lib/python3.8/dist-packages (from datasets) (3.8.4)\n", "Requirement already satisfied: pyarrow>=6.0.0 in /usr/local/lib/python3.8/dist-packages (from datasets) (11.0.0)\n", "Requirement already satisfied: xxhash in /usr/local/lib/python3.8/dist-packages (from datasets) (3.2.0)\n", "Requirement already satisfied: fsspec[http]>=2021.11.1 in /usr/local/lib/python3.8/dist-packages (from datasets) (2023.1.0)\n", "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.8/dist-packages (from huggingface_hub) (4.5.0)\n", "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets) (22.2.0)\n", "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets) (1.8.2)\n", "Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets) (3.0.1)\n", "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets) (6.0.4)\n", "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets) (1.3.3)\n", "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets) (1.3.1)\n", "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets) (4.0.2)\n", "Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests->transformers) (2.8)\n", "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (1.26.14)\n", "Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests->transformers) (2019.11.28)\n", "Requirement already satisfied: protobuf<=3.20.2 in /usr/local/lib/python3.8/dist-packages (from transformers) (3.20.2)\n", "Requirement already satisfied: sentencepiece!=0.1.92,>=0.1.91 in /usr/local/lib/python3.8/dist-packages (from transformers) (0.1.97)\n", "Requirement already satisfied: humanfriendly>=9.1 in /usr/local/lib/python3.8/dist-packages (from coloredlogs->optimum) (10.0)\n", "Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.8/dist-packages (from pandas->datasets) (2.8.2)\n", "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.8/dist-packages (from pandas->datasets) (2022.7.1)\n", "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.8/dist-packages (from sympy->optimum) (1.2.1)\n", "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /usr/local/lib/python3.8/dist-packages (from torchvision->optimum) (9.4.0)\n", "Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.8.1->pandas->datasets) (1.14.0)\n", "Installing collected packages: optimum\n", "Successfully installed optimum-1.7.3\n", "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n", "\u001b[0m" ] } ], "source": [ "!pip install optimum transformers datasets evaluate huggingface_hub" ] }, { "cell_type": "code", "execution_count": 3, "id": "2b0b586b-d710-4ce5-8491-b2a3aa4096da", "metadata": { "tags": [] }, "outputs": [], "source": [ "# This flag is the difference between SQUAD v1 or 2 (if you're using another dataset, it indicates if impossible\n", "# answers are allowed or not).\n", "squad_v2 = False\n", "model_checkpoint = \"microsoft/deberta-base\"\n", "batch_size = 16" ] }, { "cell_type": "markdown", "id": "6c7b9c65-d07f-4ab1-b6ab-5dd480177f9c", "metadata": {}, "source": [ "## Loading the dataset" ] }, { "cell_type": "markdown", "id": "c347b1cb-83b4-47dc-9cd2-f4d99221fc9f", "metadata": {}, "source": [ "We will use the 🤗 Datasets library to download the data and get the metric we need to use for evaluation. This can be easily done with the functions load_dataset and load_metric." ] }, { "cell_type": "code", "execution_count": 4, "id": "14f8d218-f659-43d3-8538-1bb2b393eecf", "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/usr/local/lib/python3.8/dist-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", " from .autonotebook import tqdm as notebook_tqdm\n", "Downloading builder script: 100%|███████████████████████████████████████████████████████████| 5.27k/5.27k [00:00<00:00, 5.12MB/s]\n", "Downloading metadata: 100%|█████████████████████████████████████████████████████████████████| 2.36k/2.36k [00:00<00:00, 2.29MB/s]\n", "Downloading readme: 100%|███████████████████████████████████████████████████████████████████| 7.67k/7.67k [00:00<00:00, 5.37MB/s]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Downloading and preparing dataset squad/plain_text to /root/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Downloading data files: 0%| | 0/2 [00:00, ?it/s]\n", "Downloading data: 0%| | 0.00/8.12M [00:00, ?B/s]\u001b[A\n", "Downloading data: 88%|████████████████████████████████████████████████████████████▋ | 7.14M/8.12M [00:00<00:00, 71.4MB/s]\u001b[A\n", "Downloading data: 14.8MB [00:00, 74.6MB/s] \u001b[A\n", "Downloading data: 22.4MB [00:00, 75.3MB/s]\u001b[A\n", "Downloading data: 30.3MB [00:00, 74.6MB/s]\u001b[A\n", "Downloading data files: 50%|███████████████████████████████████ | 1/2 [00:00<00:00, 1.83it/s]\n", "Downloading data: 4.85MB [00:00, 70.1MB/s] \u001b[A\n", "Downloading data files: 100%|██████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 2.52it/s]\n", "Extracting data files: 100%|█████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 1732.47it/s]\n", " \r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Dataset squad downloaded and prepared to /root/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453. Subsequent calls will reuse this data.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "100%|█████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 278.17it/s]\n" ] } ], "source": [ "from datasets import load_dataset, load_metric\n", "\n", "datasets = load_dataset(\"squad_v2\" if squad_v2 else \"squad\")" ] }, { "cell_type": "markdown", "id": "d1cdcfe7-c4ba-4905-a472-9c77c942c444", "metadata": { "tags": [] }, "source": [ "We can see the training, validation and test sets all have a column for the context, the question and the answers to those questions." ] }, { "cell_type": "code", "execution_count": 5, "id": "9acf8428-3f1f-493e-8792-7cbf648badfb", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "DatasetDict({\n", " train: Dataset({\n", " features: ['id', 'title', 'context', 'question', 'answers'],\n", " num_rows: 87599\n", " })\n", " validation: Dataset({\n", " features: ['id', 'title', 'context', 'question', 'answers'],\n", " num_rows: 10570\n", " })\n", "})" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "datasets" ] }, { "cell_type": "markdown", "id": "1038fa6a-0ff7-40b3-93a1-0a37e99f5d81", "metadata": { "tags": [] }, "source": [ "We can access an actual element to see what the data looks like:" ] }, { "cell_type": "code", "execution_count": 6, "id": "2da9ab03-4bce-4bc7-b0ce-6128042e7d00", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "{'id': '5733be284776f41900661182',\n", " 'title': 'University_of_Notre_Dame',\n", " 'context': 'Architecturally, the school has a Catholic character. Atop the Main Building\\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.',\n", " 'question': 'To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?',\n", " 'answers': {'text': ['Saint Bernadette Soubirous'], 'answer_start': [515]}}" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "datasets[\"train\"][0]" ] }, { "cell_type": "markdown", "id": "0a830e35-34d8-4531-a9d1-e9ee68b69d91", "metadata": {}, "source": [ "## Preprocessing the training data" ] }, { "cell_type": "markdown", "id": "08cb075c-dd7a-4327-acd8-aaefab4b850b", "metadata": {}, "source": [ "Before we can feed those texts to our model, we need to preprocess them. This is done by a 🤗 Transformers Tokenizer which will tokenize the inputs (including converting the tokens to their corresponding IDs in the pretrained vocabulary) and put it in a format the model expects, as well as generate the other inputs that model requires.\n", "\n", "To do all of this, we instantiate our tokenizer with the `AutoTokenizer.from_pretrained` method, which will ensure:\n", "\n", "we get a tokenizer that corresponds to the model architecture we want to use,\n", "we download the vocabulary used when pretraining this specific checkpoint.\n", "That vocabulary will be cached, so it's not downloaded again the next time we run the cell." ] }, { "cell_type": "code", "execution_count": 7, "id": "c8b8daf2-b90f-48e8-9e4e-4724dd92603c", "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Downloading (…)okenizer_config.json: 100%|████████████████████████████████████████████████████| 52.0/52.0 [00:00<00:00, 17.2kB/s]\n", "Downloading (…)lve/main/config.json: 100%|███████████████████████████████████████████████████████| 474/474 [00:00<00:00, 406kB/s]\n", "Downloading (…)olve/main/vocab.json: 100%|████████████████████████████████████████████████████| 899k/899k [00:00<00:00, 11.8MB/s]\n", "Downloading (…)olve/main/merges.txt: 100%|████████████████████████████████████████████████████| 456k/456k [00:00<00:00, 3.10MB/s]\n" ] } ], "source": [ "from transformers import AutoTokenizer\n", " \n", "tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)" ] }, { "cell_type": "markdown", "id": "8bbf3812-f38f-4f31-9e7e-5bc3a19d4bc8", "metadata": {}, "source": [ "Now one specific thing for the preprocessing in question answering is how to deal with very long documents. We usually truncate them in other tasks, when they are longer than the model maximum sentence length, but here, removing part of the the context might result in losing the answer we are looking for. To deal with this, we will allow one (long) example in our dataset to give several input features, each of length shorter than the maximum length of the model (or the one we set as a hyper-parameter). Also, just in case the answer lies at the point we split a long context, we allow some overlap between the features we generate controlled by the hyper-parameter `doc_stride`:" ] }, { "cell_type": "code", "execution_count": 8, "id": "8940abb8-4f98-4326-bed7-eeb0a2e6a548", "metadata": { "tags": [] }, "outputs": [], "source": [ "max_length = 384 # The maximum length of a feature (question and context)\n", "doc_stride = 128 # The authorized overlap between two part of the context when splitting it is needed." ] }, { "cell_type": "markdown", "id": "49545982-b5d5-4656-96d9-217f0dc5b4df", "metadata": {}, "source": [ "Note that we never want to truncate the question, only the context, else the `only_second` truncation picked. Now, our tokenizer can automatically return us a list of features capped by a certain maximum length, with the overlap we talked above, we just have to tell it with `return_overflowing_tokens=True` and by passing the stride." ] }, { "cell_type": "code", "execution_count": 9, "id": "f9aa3597-3a59-4205-bc2f-3f11745ff0e3", "metadata": { "tags": [] }, "outputs": [], "source": [ "pad_on_right = tokenizer.padding_side == \"right\"" ] }, { "cell_type": "markdown", "id": "f18fc077-166d-4182-b5cd-2d7819f903e6", "metadata": {}, "source": [ "Now let's put everything together in one function we will apply to our training set. In the case of impossible answers (the answer is in another feature given by an example with a long context), we set the cls index for both the start and end position. We could also simply discard those examples from the training set if the flag allow_impossible_answers is False. Since the preprocessing is already complex enough as it is, we've kept is simple for this part." ] }, { "cell_type": "code", "execution_count": 10, "id": "05838576-a552-4ca4-b21d-2078041317c9", "metadata": { "tags": [] }, "outputs": [], "source": [ "def prepare_train_features(examples):\n", " # Some of the questions have lots of whitespace on the left, which is not useful and will make the\n", " # truncation of the context fail (the tokenized question will take a lots of space). So we remove that\n", " # left whitespace\n", " examples[\"question\"] = [q.lstrip() for q in examples[\"question\"]]\n", "\n", " # Tokenize our examples with truncation and padding, but keep the overflows using a stride. This results\n", " # in one example possible giving several features when a context is long, each of those features having a\n", " # context that overlaps a bit the context of the previous feature.\n", " tokenized_examples = tokenizer(\n", " examples[\"question\" if pad_on_right else \"context\"],\n", " examples[\"context\" if pad_on_right else \"question\"],\n", " truncation=\"only_second\" if pad_on_right else \"only_first\",\n", " max_length=max_length,\n", " stride=doc_stride,\n", " return_overflowing_tokens=True,\n", " return_offsets_mapping=True,\n", " padding=\"max_length\",\n", " )\n", "\n", " # Since one example might give us several features if it has a long context, we need a map from a feature to\n", " # its corresponding example. This key gives us just that.\n", " sample_mapping = tokenized_examples.pop(\"overflow_to_sample_mapping\")\n", " # The offset mappings will give us a map from token to character position in the original context. This will\n", " # help us compute the start_positions and end_positions.\n", " offset_mapping = tokenized_examples.pop(\"offset_mapping\")\n", "\n", " # Let's label those examples!\n", " tokenized_examples[\"start_positions\"] = []\n", " tokenized_examples[\"end_positions\"] = []\n", "\n", " for i, offsets in enumerate(offset_mapping):\n", " # We will label impossible answers with the index of the CLS token.\n", " input_ids = tokenized_examples[\"input_ids\"][i]\n", " cls_index = input_ids.index(tokenizer.cls_token_id)\n", "\n", " # Grab the sequence corresponding to that example (to know what is the context and what is the question).\n", " sequence_ids = tokenized_examples.sequence_ids(i)\n", "\n", " # One example can give several spans, this is the index of the example containing this span of text.\n", " sample_index = sample_mapping[i]\n", " answers = examples[\"answers\"][sample_index]\n", " # If no answers are given, set the cls_index as answer.\n", " if len(answers[\"answer_start\"]) == 0:\n", " tokenized_examples[\"start_positions\"].append(cls_index)\n", " tokenized_examples[\"end_positions\"].append(cls_index)\n", " else:\n", " # Start/end character index of the answer in the text.\n", " start_char = answers[\"answer_start\"][0]\n", " end_char = start_char + len(answers[\"text\"][0])\n", "\n", " # Start token index of the current span in the text.\n", " token_start_index = 0\n", " while sequence_ids[token_start_index] != (1 if pad_on_right else 0):\n", " token_start_index += 1\n", "\n", " # End token index of the current span in the text.\n", " token_end_index = len(input_ids) - 1\n", " while sequence_ids[token_end_index] != (1 if pad_on_right else 0):\n", " token_end_index -= 1\n", "\n", " # Detect if the answer is out of the span (in which case this feature is labeled with the CLS index).\n", " if not (offsets[token_start_index][0] <= start_char and offsets[token_end_index][1] >= end_char):\n", " tokenized_examples[\"start_positions\"].append(cls_index)\n", " tokenized_examples[\"end_positions\"].append(cls_index)\n", " else:\n", " # Otherwise move the token_start_index and token_end_index to the two ends of the answer.\n", " # Note: we could go after the last offset if the answer is the last word (edge case).\n", " while token_start_index < len(offsets) and offsets[token_start_index][0] <= start_char:\n", " token_start_index += 1\n", " tokenized_examples[\"start_positions\"].append(token_start_index - 1)\n", " while offsets[token_end_index][1] >= end_char:\n", " token_end_index -= 1\n", " tokenized_examples[\"end_positions\"].append(token_end_index + 1)\n", "\n", " return tokenized_examples" ] }, { "cell_type": "markdown", "id": "ab6bb2d9-bdad-4796-b1eb-fe06f8c7e3a3", "metadata": {}, "source": [ "Now apply this function on all the sentences (or pairs of sentences) in our dataset:" ] }, { "cell_type": "code", "execution_count": 11, "id": "83874d1e-80f0-44e3-98da-bd8ed9916b14", "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ " \r" ] } ], "source": [ "tokenized_datasets = datasets.map(prepare_train_features, batched=True, remove_columns=datasets[\"train\"].column_names)" ] }, { "cell_type": "markdown", "id": "e5b3f1fb-14f0-4d29-acc2-e7a1d5494076", "metadata": {}, "source": [ "## Fine-tuning DeBERTa model" ] }, { "cell_type": "markdown", "id": "f4a39f2f-66bf-45fa-8f21-ff5cca95313e", "metadata": {}, "source": [ "Now that our data is ready for training, we can download the pretrained model and fine-tune it. Since our task is question answering, we use the `AutoModelForQuestionAnswering` class. Like with the tokenizer, the from_pretrained method will download and cache the model for us:" ] }, { "cell_type": "code", "execution_count": 23, "id": "ffcb6fdf-ae3a-4d2a-bd4c-bf161ff51af2", "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Some weights of the model checkpoint at microsoft/deberta-base were not used when initializing DebertaForQuestionAnswering: ['lm_predictions.lm_head.bias', 'lm_predictions.lm_head.dense.bias', 'lm_predictions.lm_head.LayerNorm.bias', 'lm_predictions.lm_head.LayerNorm.weight', 'deberta.embeddings.position_embeddings.weight', 'lm_predictions.lm_head.dense.weight']\n", "- This IS expected if you are initializing DebertaForQuestionAnswering from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n", "- This IS NOT expected if you are initializing DebertaForQuestionAnswering from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n", "Some weights of DebertaForQuestionAnswering were not initialized from the model checkpoint at microsoft/deberta-base and are newly initialized: ['qa_outputs.weight', 'qa_outputs.bias']\n", "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n" ] } ], "source": [ "from transformers import AutoModelForQuestionAnswering\n", "from optimum.onnxruntime import ORTTrainer, ORTTrainingArguments\n", "\n", "from transformers import default_data_collator\n", "\n", "model = AutoModelForQuestionAnswering.from_pretrained(model_checkpoint)\n", "data_collator = default_data_collator" ] }, { "cell_type": "markdown", "id": "3efbacf5-6e8b-48d7-b3d7-69b42e65c933", "metadata": {}, "source": [ "Before setting up the training arguments, we will download a configuration of deepspeed stage 1. ONNX Runtime training supports ZeRO stage 1(optimizer state partitioning)." ] }, { "cell_type": "code", "execution_count": 13, "id": "3b3832f5-552a-4cb7-96a3-ba9ca3a9a360", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", "To disable this warning, you can either:\n", "\t- Avoid using `tokenizers` before the fork if possible\n", "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n", "--2023-04-06 08:56:02-- https://raw.githubusercontent.com/huggingface/optimum/main/tests/onnxruntime/ds_configs/ds_config_zero_stage_1.json\n", "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.108.133, 185.199.109.133, ...\n", "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 1224 (1.2K) [text/plain]\n", "Saving to: ‘ds_config_zero_stage_1.json.1’\n", "\n", "ds_config_zero_stag 100%[===================>] 1.20K --.-KB/s in 0s \n", "\n", "2023-04-06 08:56:02 (60.4 MB/s) - ‘ds_config_zero_stage_1.json.1’ saved [1224/1224]\n", "\n" ] } ], "source": [ "!wget https://raw.githubusercontent.com/huggingface/optimum/main/tests/onnxruntime/ds_configs/ds_config_zero_stage_1.json" ] }, { "cell_type": "code", "execution_count": 24, "id": "f3e1cf9e-72eb-4cdb-a10a-5cdbf252eb8f", "metadata": { "tags": [] }, "outputs": [], "source": [ "model_name = model_checkpoint.split(\"/\")[-1]\n", "args = ORTTrainingArguments(\n", " output_dir=f\"{model_name}-finetuned-squad\",\n", " evaluation_strategy = \"epoch\",\n", " learning_rate=2e-5,\n", " per_device_train_batch_size=batch_size,\n", " per_device_eval_batch_size=batch_size,\n", " num_train_epochs=3,\n", " weight_decay=0.01,\n", " optim=\"adamw_ort_fused\",\n", " deepspeed=\"ds_config_zero_stage_1.json\",\n", ")" ] }, { "cell_type": "markdown", "id": "864cf4aa-5285-4b43-a18a-3f79f989adea", "metadata": {}, "source": [ "Instantiate `ORTTrainer`, here we will use ORT fused optimizer to have better performance on latency:" ] }, { "cell_type": "code", "execution_count": 25, "id": "1c9da87b-698b-4351-a4e4-122c532318ea", "metadata": { "tags": [] }, "outputs": [], "source": [ "trainer = ORTTrainer(\n", " model=model,\n", " args=args,\n", " train_dataset=tokenized_datasets[\"train\"],\n", " eval_dataset=tokenized_datasets[\"validation\"],\n", " data_collator=data_collator,\n", " tokenizer=tokenizer,\n", " feature=\"question-answering\",\n", ")" ] }, { "cell_type": "markdown", "id": "115c93b8-9870-49ad-870c-45cc6c53a238", "metadata": {}, "source": [ "Launch the training (it will leverage one single GPU for the training)" ] }, { "cell_type": "code", "execution_count": 24, "id": "846ee1dc-6c29-4a94-888a-6b60112e75e7", "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Wrap ORTModule for ONNX Runtime training.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2023-04-05 16:41:07,941] [WARNING] [config_utils.py:74:_process_deprecated_field] Config parameter cpu_offload is deprecated use offload_optimizer instead\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py:1794: FutureWarning: The first argument to symbolic functions is deprecated in 1.13 and will be removed in the future. Please annotate treat the first argument (g) as GraphContext and use context information from the object instead.\n", " warnings.warn(\n", "/usr/local/lib/python3.8/dist-packages/onnxruntime/training/ortmodule/_utils.py:250: UserWarning: User Module's attribute name float collides with ORTModule's attribute name. User Module's method may not be called upon invocation through ORTModule.\n", " warnings.warn(\n", "/usr/local/lib/python3.8/dist-packages/onnxruntime/training/ortmodule/_utils.py:250: UserWarning: User Module's attribute name half collides with ORTModule's attribute name. User Module's method may not be called upon invocation through ORTModule.\n", " warnings.warn(\n", "/usr/local/lib/python3.8/dist-packages/onnxruntime/training/ortmodule/_utils.py:250: UserWarning: User Module's attribute name to collides with ORTModule's attribute name. User Module's method may not be called upon invocation through ORTModule.\n", " warnings.warn(\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", "To disable this warning, you can either:\n", "\t- Avoid using `tokenizers` before the fork if possible\n", "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n", "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", "To disable this warning, you can either:\n", "\t- Avoid using `tokenizers` before the fork if possible\n", "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n", "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", "To disable this warning, you can either:\n", "\t- Avoid using `tokenizers` before the fork if possible\n", "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n", "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", "To disable this warning, you can either:\n", "\t- Avoid using `tokenizers` before the fork if possible\n", "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Using /root/.cache/torch_extensions/py38_cu116 as PyTorch extensions root...\n", "Creating extension directory /root/.cache/torch_extensions/py38_cu116/fused_adam...\n", "Detected CUDA files, patching ldflags\n", "Emitting ninja build file /root/.cache/torch_extensions/py38_cu116/fused_adam/build.ninja...\n", "Building extension module fused_adam...\n", "Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", "To disable this warning, you can either:\n", "\t- Avoid using `tokenizers` before the fork if possible\n", "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n", "[1/3] /usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=fused_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\\\"_gcc\\\" -DPYBIND11_STDLIB=\\\"_libstdcpp\\\" -DPYBIND11_BUILD_ABI=\\\"_cxxabi1011\\\" -I/usr/local/lib/python3.8/dist-packages/deepspeed/ops/csrc/includes -I/usr/local/lib/python3.8/dist-packages/deepspeed/ops/csrc/adam -isystem /usr/local/lib/python3.8/dist-packages/torch/include -isystem /usr/local/lib/python3.8/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/lib/python3.8/dist-packages/torch/include/TH -isystem /usr/local/lib/python3.8/dist-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_70,code=compute_70 -gencode=arch=compute_70,code=sm_70 --compiler-options '-fPIC' -O3 -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -lineinfo --use_fast_math -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_70,code=compute_70 -std=c++14 -c /usr/local/lib/python3.8/dist-packages/deepspeed/ops/csrc/adam/multi_tensor_adam.cu -o multi_tensor_adam.cuda.o \n", "[2/3] c++ -MMD -MF fused_adam_frontend.o.d -DTORCH_EXTENSION_NAME=fused_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\\\"_gcc\\\" -DPYBIND11_STDLIB=\\\"_libstdcpp\\\" -DPYBIND11_BUILD_ABI=\\\"_cxxabi1011\\\" -I/usr/local/lib/python3.8/dist-packages/deepspeed/ops/csrc/includes -I/usr/local/lib/python3.8/dist-packages/deepspeed/ops/csrc/adam -isystem /usr/local/lib/python3.8/dist-packages/torch/include -isystem /usr/local/lib/python3.8/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/lib/python3.8/dist-packages/torch/include/TH -isystem /usr/local/lib/python3.8/dist-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -std=c++14 -g -Wno-reorder -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -c /usr/local/lib/python3.8/dist-packages/deepspeed/ops/csrc/adam/fused_adam_frontend.cpp -o fused_adam_frontend.o \n", "[3/3] c++ fused_adam_frontend.o multi_tensor_adam.cuda.o -shared -L/usr/local/lib/python3.8/dist-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda_cu -ltorch_cuda_cpp -ltorch -ltorch_python -L/usr/local/cuda/lib64 -lcudart -o fused_adam.so\n", "Time to load fused_adam op: 35.75363326072693 seconds\n", "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", "To disable this warning, you can either:\n", "\t- Avoid using `tokenizers` before the fork if possible\n", "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n", "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", "To disable this warning, you can either:\n", "\t- Avoid using `tokenizers` before the fork if possible\n", "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n", "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", "To disable this warning, you can either:\n", "\t- Avoid using `tokenizers` before the fork if possible\n", "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Loading extension module fused_adam...\n", "Using /root/.cache/torch_extensions/py38_cu116 as PyTorch extensions root...\n", "Creating extension directory /root/.cache/torch_extensions/py38_cu116/utils...\n", "Emitting ninja build file /root/.cache/torch_extensions/py38_cu116/utils/build.ninja...\n", "Building extension module utils...\n", "Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", "To disable this warning, you can either:\n", "\t- Avoid using `tokenizers` before the fork if possible\n", "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n", "[1/2] c++ -MMD -MF flatten_unflatten.o.d -DTORCH_EXTENSION_NAME=utils -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\\\"_gcc\\\" -DPYBIND11_STDLIB=\\\"_libstdcpp\\\" -DPYBIND11_BUILD_ABI=\\\"_cxxabi1011\\\" -isystem /usr/local/lib/python3.8/dist-packages/torch/include -isystem /usr/local/lib/python3.8/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/lib/python3.8/dist-packages/torch/include/TH -isystem /usr/local/lib/python3.8/dist-packages/torch/include/THC -isystem /usr/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /usr/local/lib/python3.8/dist-packages/deepspeed/ops/csrc/utils/flatten_unflatten.cpp -o flatten_unflatten.o \n", "[2/2] c++ flatten_unflatten.o -shared -L/usr/local/lib/python3.8/dist-packages/torch/lib -lc10 -ltorch_cpu -ltorch -ltorch_python -o utils.so\n", "Time to load utils op: 20.83265709877014 seconds\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Loading extension module utils...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Rank: 0 partition count [1] and sizes[(138603266, False)] \n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Using /root/.cache/torch_extensions/py38_cu116 as PyTorch extensions root...\n", "No modifications detected for re-loaded extension module utils, skipping build step...\n", "Loading extension module utils...\n", "***** Running training *****\n", " Num examples = 88550\n", " Num Epochs = 3\n", " Instantaneous batch size per device = 16\n", " Total train batch size (w. parallel, distributed & accumulation) = 16\n", " Gradient Accumulation steps = 1\n", " Total optimization steps = 16605\n", " Number of trainable parameters = 138603266\n", "/usr/local/lib/python3.8/dist-packages/onnxruntime/training/ortmodule/_training_manager.py:192: UserWarning: Fast path enabled - skipping checks. Rebuild graph: True, Execution agent: True, Device check: True\n", " warnings.warn(\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Time to load utils op: 0.0007810592651367188 seconds\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Warning: Checker does not support models with experimental ops: ATen\n", "/usr/local/lib/python3.8/dist-packages/onnxruntime/training/ortmodule/_logger.py:52: UserWarning: There were one or more warnings or errors raised while exporting the PyTorch model. Please enable INFO level logging to view all warnings and errors.\n", " warnings.warn(\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n" ] }, { "data": { "text/html": [ "\n", "
| Epoch | \n", "Training Loss | \n", "Validation Loss | \n", "
|---|---|---|
| 1 | \n", "0.828100 | \n", "0.801068 | \n", "
| 2 | \n", "0.659800 | \n", "0.793299 | \n", "
| 3 | \n", "0.508100 | \n", "0.895923 | \n", "
"
],
"text/plain": [
"